The former Trident Networks/Speedyweb server "Neptune" has been experiencing issues lately. In order to ensure future stability and performance of the sites served from this machine we've decided to migrate them to more reliable servers. Accounts will be moved to newer and faster servers, either running UNIX or Windows, depending on the if the website relies on FrontPage extensions. E-mail and any MySQL databases will be migrated to UNIX servers.

We apologize for any inconvenience that this migration may cause you and will be more then happy to answer any questions you may have; we will be working with you to resolve any issues that may arise due to this migration.

Users on Neptune will be contacted directly via our helpdesk ticketing system with more details. Please respond to the helpdesk ticket and/or call us at 877-720-0483 option #3. We will have staff onsite 24 hours a day, 7 days a week during this migration and they will be able to help you with any problems that you may have. If you believe that your e-mail address with us may be out of date we highly recommend that you respond to this ticket or call us at 877-720-0483 option #2 during business hours and update your contact information with an Account Manager.

Thank you for your patience.

Just a reminder, tonight at 11 PM we will be performing an emergency maintenance on part of our electrical system. This will have an impact on a portion of our shared web and database servers, and eight dedicated and colocated servers. The shared hosting servers affected are:

acacia, acorn, alder, aralia, arrowwood, ash, aspen, avocado, balsa, balsam, bamboo, banana, banyan, bayberry, beech, bigleaf, blackberry, boojum, boxwood, bubinga, buckeye, cactus, cedar, cherry, chestnut, cholla, cinnamon, clover, columbia, commerce2, commerce3, cork, cottonwood, db, deerwood, dogwood, ebony, elm, evergreen, ficus, fig, filbert, fuji, gingko, grape, grapevine, hackberry, hazel, hemlock, hickory, ironbark, ivy, kentia, kola, kudzu, larch, laurel, lilac, lime, madrona, magnolia, mango, mangrove, maple, mesquite, mimosa, moringa, mulberry, myrtle, newninewire, ninewire2, olive, orchid, palmetto, papaya, pear, pecan, plum, poplar, privet, quince, redbud, sassafras, savin, sequoia, sherwood, snowberry, spiceberry, spruce, strawberry, sycamore, tamarack, teak, truffula, tutsan, walnut, woodpecker, yucca, yulan/fern

Every effort will be made to minimize the downtime. Our electricians have estimated the time required to complete the task at 2 hours.

Thank you for your patience and understanding while we strive to build a better facility.

Update 1:15 AM: The maintenance went very smoothly and was completed in under 90 minutes. We were able to supply alternate power to the few dedicated and colo servers involved in the maintenance. We also took the opportunity to replace some failed parts in a couple of servers (acorn for example whose fan inside the power supply had stopped working.)

Again, thank you so much for your patience during this critical maintenance interval.

Tuesday night March 11th, Wednesday morning March 12th we will be performing emergency maintenance on our power infrastructure related to the installation of our new UPS system. This maintenance will impact a small portion of our shared hosting clients. We will provide specific times in the next couple days as they firm up.

We will strive to keep this outage as short as possible but we are allowing up to two hours of downtime.

Thanks for your understanding while we grow to serve you better.

Tonight during our scheduled maintenance window, the servers palm.forest.net and tangerine.forest.net will undergo RAM upgrades. Each server will be down for approximately 5 to 15 minutes between the hours of 11:00 PM and 1:00 AM Pacific Standard Time. Every effort will be made to minimize the servers' offline time.

Please note: Palm is a mail server. Email will be unavailable while the server is offline. Inbound mail will be spooled on sending servers are delivered after the maintenance interval. Palm does not relay outbound mail itself, it works with one of our outbound mail hubs here so this event should have no impact on outbound email.

Tangerine is a database server running mySQLv4. Dynamic content on several of our other hosting servers may be affected during the maintenance interval.

Performance of both servers should be improved after their upgrades.

Most of the planned software updates have been completed; no downtime occurred. People who were having trouble with GnuPG should have more success now; if not, please let us know ASAP. We will be updating the remaining software ports this weekend.

Souari.forest.net will have some software libraries updated tonight, beginning at 10pm pacific time. We anticipate no downtime, but there is a possibility that some functionality (such as PHP) will experience interruption for a few minutes during the process. We will update this page again when the upgrades are complete.

We will be taking the FileMaker 9 Server named rosemary offline this morning to perform some emergency maintenance on the FileMaker Instant Web Publishing. We do not currently have an ETA for when this work will be completed but we will update this entry as we have more information.

Thank You

11/07/2007 10:25AM: Rosemary is now back online and the error in IWP has been corrected.

Souari is back up and running with upgraded versions of Apache and PHP. Most users should not see any difference with the new software, but a few people may find problems with their site that our processes were not able to catch in advance. if you notice a problem with your site, please contact us right away and we will get it resolved for you.

The next step in this process will begin tomorrow, as we run a script to locate sites that may be affected by the upcoming suexec/suPHP implementation. Any such sites we find, we will summarize the issues and notify the users. We expect to be able to give at least a week to address these issues (which are usually very simple), and will of course evaluate that expected time frame if necessary.

Thank you for your patience during these upgrades.

As part of our changes to the FreeBSD hosting servers, Souari will be offline late Wednesday night for an upgrade to the Apache web server software. We estimate that the downtime will be less than 2 hours, during which the current configuration will be backed up, and the new configuration will be installed. There will be substantial modifications to the virtual host configuration files. In the vast majority of cases, everything should continue to function after the upgrade as it has until now. However, we ask that you test your sites thoroughly, and report to us as soon as possible if they find any problems so that we can correct them.

The configuration changes to Butternut are complete; all CGI and PHP scripts should now be running as the owner, rather than as the web server. The server is in the process of scanning the home directory for all user files and changing their ownership and permissions to match the new requirements.

Note that from this time forward, if you have scripts that require the web server to write files to your account, those scripts will be running as you, rather than as "nobody." This means you need to make sure the files/directories you want to write to are owned by and writable by you, rather than the web server or everybody.

Please feel free to contact us if you have any questions or any problems arise. Thank you for your patience.

We are now beginning the upgrades to suPHP and suexec on Butternut. We estimate about two hours of downtime, and will post again when it's back up and running.

On Monday evening, October 1, we will be making the final changes to Butternut to enable suexec/suPHP for enhanced security. This will require approximately two hours of downtime late Monday evening.

Customers who are likely to need code changes have been notified through our ticket system and by phone. Everyone else shouldn't notice any changes after we've modified the server. If you do notice a problem on Tuesday morning, October 2, please let us know ASAP so we can resolve it.

As part of our changes to the FreeBSD hosting servers, Butternut will be offline late tonight for an upgrade to the Apache web server software. We estimate that the downtime will be less than 2 hours, during which the current configuration will be backed up, and the new configuration will be installed. There will be substantial modifications to the virtual host configuration files. In the vast majority of cases, everything should continue to function after the upgrade as it has until now. However, we ask that you test your sites thoroughly, and report to us as soon as possible if they find any problems so that we can correct them.

We're pleased to announce that digital.forest has opened a new technical support forum. This new forum is designed to give digital.forest customers the ability to ask questions of each other related to the services that digital.forest provides such as Filemaker databases, Lasso, PHP, etc. These forums will have some limited interaction from the dgital.forest staff however it is mainly a forum in which users can exchange ideas, ask questions, and interact as a community. We know that we have some smart folks among our clients and we hope to have the forum be a great place to exchange knowledge, both between us and our clients, but more importantly from client to client.

You can find this new service at forums.forest.net. There is also a link in the "Tools" menu over there. -->

This will not replace our trouble ticket system, which will remain the primary method of one-to-one technical support for our clients. Nor will it replace this "support blog" as a way for us to announce events, maintenance, and news about digital.forest. Instead we hope the forum will be a great place to share ideas among many people at once. Have a look. Create a login. Start a discussion. Help somebody out, or get some help for yourself. Let us know what you think.

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

Starting tonight at about 10:00 PM PST, Silverpine will be taken down for the next step in its configuration changes. suexec and suPHP will be enabled on the server, and file ownership and permissions will be adjusted for all files on the server to match the requirements of this change.

Several weeks ago we opened tickets for any users we could find who were likely to be affected by this change. If you did not receive such a ticket, then it is unlikely you will experience any problems from this change. If you did receive a ticket, and you have made the changes we described, you should also be in the clear. If you do experience problems with your site after this change, please let us know ASAP so we can investigate and correct the problem.

As part of our ongoing process of upgrading the software and configuration on our FreeBSD hosting servers, Souari and Butternut will be taken down on Sunday evening for an operating system upgrade. We expect the maintenance to begin around 9pm Sunday evening, and should last less than two hours.

As part of our changes to the FreeBSD hosting servers, Silverpine will be offline late tonight for an upgrade to the Apache web server software. We estimate that the downtime will be less than 2 hours, during which the current configuration will be backed up, and the new configuration will be installed. There will be substantial modifications to the virtual host configuration files. In the vast majority of cases, everything should continue to function after the upgrade as it has until now. However, we ask that you test your sites thoroughly, and report to us as soon as possible if they find any problems so that we can correct them.

digital.forest is in the beginning stages of a significant change in the way our FreeBSD hosting servers will function. This change is intended to increase the security of your sites, and to improve our ability to quickly determine the source of a problem, such as an insecure web form being used to send spam.

We are taking care to minimize the impact these changes will have on you, but there may be some cases where problems will arise. In such a situation, we will assist you in making any changes necessary to get your site working with the new configuration.

The first stage of the changes is upgrading the server to the latest version of the FreeBSD operating system. This stage has been completed on Silverpine. Souari and Butternut will follow by the end of this month. We do not anticipate that customers will notice any changes from this upgrade.

The second stage is upgrading Apache, the web server software, to the latest version. This is a more significant change, from a hosting standpoint. Silverpine is scheduled to be upgraded tonight. There will be a period of downtime, estimated to be less than 2 hours, during which the current configuration will be backed up, and the new configuration will be installed. There will be substantial modifications to the virtual host configuration files. In the vast majority of cases, everything should continue to function after the upgrade as it has until now. However, customers will need to test their sites thoroughly, and report to us as soon as possible if they find any problems so that we can correct them.

The third stage will be activating two key configuration changes for each virtual host: suexec and suPHP. Once these are activated, customer CGI and PHP scripts will execute under their own usernames, rather than under the server's username. We will change the ownership on all files and folders in your home directory to your own username, so that if your site needs to write files into your directory, it can still do so. We will again need people to test their sites carefully and make sure everything is working as expected.

The fourth and final stage will be activating suexec and suPHP for the default server, outside the virtual hosts. Before this stage is completed, any customers who are using the server's built-in SSL certificate, rather than one purchased for their own domain, will need to work with us to make changes to their site to work with an alternate secure URL. You'll still be able to use our certificate, but we'll need to set up a subdomain specific to your virtual host to permit you to do so.

We will be moving very slowly and carefully on the Silverpine changes so that we can refine the process for the remaining servers. Your testing and feedback will be a great help in getting any unanticipated problems handled as quickly as possible.

The shared hosting named "durian" has experienced a hardware-related failure and is undergoing repair. We will post an update as soon as we have an ETA for it being back online.

Update: Durian is fixed and running again. Thanks for your patience.

The former Trident Networks/Speedyweb server "Neptune" is having some issues this morning. We are working on it and hope to have it back up and running again ASAP. Thank you for your patience.

Update 8:45 AM PST: Neptune is back up.

Mercury is currently experiencing some trouble with it's PHP and SQL. We are investigating and expect a resolution shortly.

Mars is currently down for an emergency maintenance due to a failed hardware component. Unfortunately at this time Mars is still down due to this problem. If you are a former Trident/Speedyweb customer and your website and email are currently not working, you are affected by this problem.

Update @ 3:30am PDT: The proper technicans have been notified and we are currently evaluating the situation in order to get this resolved as soon as possible. We anticipate that Mars will be back online around 9:30 AM PDT.

Update @ 8:30am PDT: We are executing our plan to get the machine fully back up.

On Sunday May 21st, 2006 we will be patching all our OS X 10.3 & 10.4 servers with the latest security update (Security Update 2006-003) released by Apple on May 8th, 2006. This maintenance will take place at 12pm PDT. The update will require a reboot of the server. Downtime should be no less then 5 to 10 minutes depending on the server configuration.

You can find more information regarding Security Update 2006-003 on Apple's website or by clicking this link: http://docs.info.apple.com/article.html?artnum=303737

Update @ 2:04am on 05/06/2006: Both upgrades have been successfully completed

We will be performing a hardware upgrade (specifically a processor upgrade) on columbia (columbia.forest.net) and Ninewire's mail server (mail.ninewire.com) this coming Friday (05/05/2006). We anticipate the maintenance to last approximately 3 hours and it will begin at 10pm PDT and conclude at 1am PDT.

During this time streaming on Columbia will not be available & email service on mail.ninewire.com will also not be available.

This upgrade will increase the performance on Columbia & mail.Ninewire.com.

If you have any questions, feel free to open up a trouble ticket.

Update @ 6:07pm: thyme.forest.net has been updated successfully

This Wednesday, 05/03/2006, we will be updating FileMaker Server on thyme.forest.net from 8.0v2 to 8.0v3. All FileMaker Services, (IWP, XML, Remote) will be affected. The upgrade is scheduled to take place at 5:30 PM PDT and should last about 10 minutes.

FileMaker considers this a maintenance release and here are the change notes from 8.0v2:

1. New in FileMaker Server Advanced 8.0v3

1.1. Finds on an unstored calculation field

1.1.1. Fixed a problem that caused the entire record set to be returned if a client performed a find on an unstored calculation field if an 8.0v1 Web Publishing Engine was connected to an 8.0v2 Server.

1.1.2 Fixed a problem that could cause a FileMaker host to quit if a client performed a find on an unstored calculation field using either the Get(PrivilegeSetName) or Get(ExtendedPrivileges) functions.

1.1.3 Fixed a problem that caused unstored calculation finds to always be performed at the Web Publishing Engine which negatively affected performance. These finds are now performed appropriately at the Server or the WPE, which improves the performance of these types of finds.

Further information regarding this update is available on FileMaker's website.

Update @ 2:03pm PDT: Laurel is now once again accepting FTP connections
Update @ 12:11pm PDT: Laurel is now back up and processing Lasso

Currently Laurel is experiencing a problem regarding the processing of Lasso 3.6. It seems to 'time-out' whenever it is thrown certain Lasso code. Our system administrator is researching what is causing this problem and anticipates that this problem should be resolved by noon.

We will keep this post updated if new information becomes available.

-digital.forest technical support

Update @ 12:41am: Maintenance has been completed successfully on both banana & cactus.

Tomorrow night, Saturday April 8th, maintenance will be performed on banana.forest.net & cactus.forest.net. This will be the third, and final, maintenance performed on both of the machines. In the final maintenance we will be replacing the hard drives in both machines with faster and bigger drives.

The maintenance will begin at 9pm and will conclude at midnight.

If you have any questions or concerns feel free to open up a support ticket.

Final Update:
Mango is now back online. All files are recovered and it has received a major upgrade from what the hardware was located on. If you are hosted on Mango and you are experiencing problems, please notify our technical support immediately. -Yvo

UPDATE:
Mango suffered a catastrophic hardware failure. We are currently finding alternative hardware for the server now.

Currently, mango.forest.net is down and is being repaired. downtime should not be more than 1 hour.

We will be taking souari's webserver offline briefly this evening to update the Apache configuration. Downtime should be less than 30 minutes.

Update: Maintenance was completed successfully and well before 6pm. If you experience any problems on your Ninewire website don't hestitate to call or open a support ticket stating the problem and/or error message.

This is a reminder that today Ninewire.Ninewire.com (216.168.42.146) will be unavailable between 3pm and 6pm PST. During this time connectivity to the sites on this machine may be unavailable and/or not rendered properly.

Thank you

This weekend we will be making some changes to a Ninewire machine named www2.ninewire.com, ip address 216.168.42.146. The machine will be unavailable from 3pm PST to 6pm PST. During this time the websites hosted on the machine may be unavailable.

The reason for the maintenance is because the operating system on this machine is exhibiting some symptons that we like to get taken care of before it becomes a major problem.

If you have any questions or concerns about this maintenance please let us know by phone or opening up a ticket via our support system.

Thank you

The following servers will be going down for routine maintenance tonight between 9pm and 6am PST, estimated at 20-30 minutes each: ara, callisto, celestial, europa, gold, mars, mercury, mimas, neptune, oberon, phobos, pluto, and sol.

The following Windows hosting servers are scheduled to be patched the evening of 11 January 2006. Reboots will be needed.

Shared Hosting Servers:
jimbo, kenny, mrscrabtee, dax, frink, mrgarrison, wendy, minnie, pikachu, checkov, mccoy and imail.

Dedicated Hosting Servers:
(users on dedicated servers have been individually contacted)
sd, archiv, prime, uhura, genesis

Thank you for your understanding

At approximately 1:45 AM PST this morning we experienced a power overload condition on a single electrical circuit, which services half of two racks in our facility. This tripped a breaker in one of our Power Distribution Units. We reset the breaker and used power monitoring equipment to measure the load on that circuit as servers rebooted. With the data collected rerouted some power cables in those two racks to spread the load in a manner which should prevent this from happening again.

Most of the servers affected are digital.forest shared hosting servers, however one of the two racks contained some colocated and one dedicated server. We will be contacting the affected clients during the business day with a follow-up.

Server downtime was limited to about 20 minutes maximum, with most servers being down less that 15 minutes.

We apologize for any inconvenience.

Chuck Goolsbee
VP Technical Operations

Please note that an exploit in PHP is being widely abused by spammers to generate untraceable bulk mail. The spammer's identity is completely concealed, while the spam itself is identified as coming from the exploited website. If you use PHP in your website contruction you need to be aware of how to prevent your website from being abused in this way.

The greatest risk from this exploit (beyond being associated with being a spammer) is having all mail from your domain, and ultimately our network rejected. If you rely on email to communicate, and use PHP in your website construction, then you should take this issue very seriously and take all steps to prevent that possibility.

In a nutshell, the issue is injecting mail headers into a PHP form, complete with carriage returns and linefeeds in the right places, to force the web server's mail transport agent to relay thousands of messages per minute off your website. You can read some excellent coverage of the problem and cure at the following links:

www.anders.com

php.net

www.phpit.net

securePHP

This site describes a method to test your forms:

www.developertutorials.com

An additional method, and one we highly recommend, is preventing DIRECT access to form data. Don't put form entry fields on your home page, and limit access to form pages from referrers outside your domain... i.e. only allow access to your pages containing forms from other pages on your website. This can be done with .htaccess files. You can find a tutorial for this here:

apache-server.com

This is a serious risk, and digital.forest must take this seriously as it can have widespread implications regarding the acceptability of outbound mail from our network. As such if we notify you about exploitable forms on your website, whether it is on our servers, or your server colocated in our facilities, please take steps to immediately correct your website code. If we receive and relay to you repeated reports of your forms being exploited, and you have not taken steps to correct the problem we will have no choice but to suspend your service.

We appreciate your swift action with regards to this situation.

Regards,
Chuck Goolsbee
V.P. Technical Operations
digital.forest, Inc.

Update: this is complete as of 22.10hrs

Starting at 21.00 hours on Wednesday, September 21 these servers:

buzz.forest.net
power.forest.net

will be rebooted to apply current MicroSoft security patches. Two reboots will be needed, and downtime is expected to last about fime miuntes each.

Palmetto.forest.net has suffered a hardware failure; we are in the process of installing the disk into new hardware, and will have the server back online ASAP.

bigleaf.forest.net is experiencing some disk problems. It is offline while we perform some disk maintenance work to try to clear them up. We will post again when service has been restored.

We have completed the moves of the following servers:

acacia, ash, bamboo, banyan, commerce, commerce2, commerce3, cork, fm4.infoasis, ironbark, ivy, magnolia, mangrove, maple, mulberry, orchid, redbud, teak, and yucca.

We are taking Cinnamon down to run some disk maintenance on it in an attempt to deal with some performance problems. The server will be back up as soon as the disk operation is complete.

Tonight the following hosting servers were successfully moved to our new facility in Seattle:

aralia, aspen, bayberry, beech, boxwood, cactus, cedar, cherry, chestnut, cholla, deerwood, dogwood, elm, fern, gingko, hackberry, hawthorne, kola, moringa, olive, papaya. plum, quince, savin, sequoia, strawberry, tiawe, tutsan, and yulan.

Moves were done in two waves, the first started with 8 servers at 9:30 PM PST, and completed at 11:00 PM PST. The second started at 1 AM PST, and was complete by 2:45 AM PST.

Sorry, no pictures from tonight. Between the issues at the Bothell facility earlier in the evening and the mail server problem that sidetracked a portion of the technical staff involved in the move, we had no time for pictures.

We have completed the moves of:

skil
jig-old
power
chain
circular
iron
sabre
buzz
radial
jig

We were delayed in starting (actual shutdowns did not begin until close to midnight.) All were up by 2 AM, with the exceptions of sabre and iron. We are working on those now. We will update you when we have news. Thanks for your patience.

UPDATE: 2:55 AM
sabre.forest.net is now UP.

Still working on iron.

UPDATE: 3:30 AM
Iron suffered a motherboard failure. We are installing its disks into a spare server (an equivalent dual-processor design) now. We'll update with an ETA soon.

UPDATE: 5:30 AM
iron.forest.net is UP.

Sorry that took so long. Here are some photos we grabbed before the whole iron issue flared up:

Above: Kyle Murray loads servers into the van for the run over to Seattle.

Above: Chris Kilbourn (in the van) and Kyle Murray, unloading at the Intergate.West parking garage.

Above: The servers we moved last night, all racked and running (except iron.)

The following servers will be moved on Monday night, Feb 7, starting around 11 pm.

skil
jig-old
power
chain
circular
iron
sabre
buzz
radial
jig

Expected downtime for any server move should be around two hours. We will do everything we can to minimize this as much as possible. All servers listed should be back online by 1 AM.

This will be our first large scale server move, so downtimes will only get shorter, and number of servers moved should increase as the nights go by.

The next servers scheduled to move are:

boysenberry
sage
date
crabapple
woodpecker
tangerine
souari
butternut

These will move on Tuesday, Feb 8, either starting around 2 AM, or 11 PM. Specifics will be posted here as the time approaches.

PLEASE NOTE: As all these devices are rack-mount servers in very tight quarters with each other, we can not specifically call out exact shut-down times. Additionally we may elect to NOT move some particular servers due to space or time constraints. Details will be posted here with specifics at the time of move.

Thanks for your patience.

On Wednesday, December 1st 2004 at 7:30 pm we will be upgrading FileMaker Server Advanced on sage.forest.net from version 7.0v1 to 7.0v2.

According to FileMaker, the update "includes changes to stability and performance and External Authentication security."

No updates or changes are required on the client side, but we always recommend customers using FileMaker Client to access their databases keep their software up to date -- see http://www.filemaker.com/support/updaters.html for the latest updates.

Thank you!

We are performing an emergency hardware swap on banana.forest.net in an attempt to resolve some recurring failures that began last night. We hope that the downtime will be minimal, measured in the half hour range, and will post again when the server is back up or if it's going to take longer.

On Wednesday September 22nd at 7:30 PM PST, we will be upgrading Lasso on grapevine.forest.net from version 7.0.4 to version 7.1.

The new version provides a few useful bug fixes and adds support for FileMaker Pro 7 databases. Click here for more details. While no code changes should be necessary, you may want to check your site after the upgrade.

For interested customers, FileMaker Pro 7 Hosting is now available. Please feel free to contact your account manager or e-mail sales@forest.net for more details.

On Wednesday, July 28th 2004 at 10:30 PM PST we will be upgrading the operating system on banana.forest.net to Mac OS X Server 10.3. The upgrade should provide enhanced performance and stability. Lasso configuration and all other server settings will remain the same.

The upgrade should take between 1 and 2 hours to complete. Web sites on banana may be periodically unavailable while the upgrade takes place.

Buzz.forest.net may be periodically unavailable during the next 10-15 minutes while we perform some software and hardware upgrades.

A "zero-day" Windows exploit has been discovered as of yesterday. You can track the discovery and analysis of this on various security and Internet-operations websites. From our observations, this exploit can affect FULLY PATCHED servers, despite claims to the contrary by Microsoft. It was discovered on one of our hosting servers yesterday, which was fully patched (including update KB835732).

There are no current virus scanning signatures for this exploit (on the server side) so it must be manually discovered. We strongly suggest that owners of colocated Microsoft Windows servers take whatever precautions they can, as they become available. We also suggest that server owners read the above referenced web pages to learn how to spot the exploit, and disable it. Take this knowledge and closely search and monitor your systems, both here at digital.forest and on your local networks.

We are updating our intrusion detection systems to monitor for this exploit, but given that it is still unknown how this exploit gets onto servers we'll only be able to spot already infected machines. If we find them, the server owners will be notified.

Thank you for your cooperation and understanding.

We are installing some security patches on Souari. This will briefly prevent users from making FTP connections; however, the web service will continue uninterrupted apart from a brief reboot.

Please note that Apple has released some security-related updates in the past weeks, including a new OS version (10.3.4) which includes these updates. Experience has shown that staying up-to-date with security-related patches can limit exposure if an exploit is released "into the wild." While we have yet to see a MacOS X worm or similar malware, we still prefer to update when the vendor releases a patch. OS updates are handled a little differently, as they usually involve changes outside the realm of security, and require some testing prior to deployment.

NOTE: If we have administrative access to a client-owned server we almost always install security patches when they are released. The nature of our network, directly connected to very high bandwidth "backbone" connections, means that we have a much greater risk and exposure to newly released malware, especially of the "worm" type, as they spread automatically.

These recent patches from Apple cover a set of vulnerabilities which are classified as "Trojan Horses" which means they require user intervention to activate. However we felt it necessary to apply the patches, if only to set a precedence for our MacOS X using clients with regards to how we handle security-related patches.

If you have a server colocated here running MacOS X, or MacOS X Server that you manage yourself, we strongly suggest that you run Software Update on a regular schedule. Install any security-related patches as soon as you are comfortable. Based on our experience with other platforms, it is better to be patched prior to the release of an actual exploit.

Our methodology for handling unpatched machines if there is a known exploit "in the wild" is to remove them from our network until patched. This is how we were able to survive high-profile issues such as CodeRed, SQLSlammer, etc with minimal downtime and very low infection rates. Our Windows and UNIX using clients already know this, but given the recent widespread publicity about these MacOS X issues, we thought we should make the rest of our clients aware of this policy.

Thank you for your attention to this matter.

On Wednesday, April 29th 2004, at 7:30 PM PST, we will be upgrading Lasso on balsa.forest.net and banana.forest.net to version 6.0.6, the latest version of Lasso Professional 6. The new version provides a fix to the error handling functionality in Lasso, which had been creating some problems for customers designing and debugging new sites. The new version should also further improve performance and reliability.

Update: The upgrade on balsa.forest.net has been delayed until next Wednesday, May 5th 2004.

As a reminder, Lasso Professional 7 hosting is available for interested customers. click here for more details.

No code changes should be necessary, but we recommend checking your site after the upgrade to verify that your site is functioning as expected.

We started a new backup set on Friday evening. The initial backup usually takes about 48-56 hours to complete. Nightly incremental backups will resume again Monday night.

Thanks for your patience.

Our electrician is installing some new wiring in the datacenter today for our new row of racks. This work will require the brief shutdown of the two rows immediately adjacent in Server Room 2. One row is home to our oldest MacOS-based computers ("BandW G3's and older "beige" units.) The other is home to the servers we acquired last summer from FMPHosting.net. At noon PST we will perform an orderly shutdown/reboot of the Mac servers to accommodate the work. We will delay the FMPHosting.net reboot until after 2PM PST (to match 5 PM EST)

NO OTHER servers will be affected.

Thanks for your patience.

Jig is experiencing some intermitent disk troubles. If you are hosted on Jig you may experience some downtime as we fix the problem.

Over the next 24 hours we will be doing OS updates to our Windows servers. If you are hosted on a Windows box you may experience a brief, 5 min, downtime while the server reboots.

This is due to a critical vulnerability announced by Microsoft earlier today.

This Friday (February 13, 2004), at 9:00 PST, we will be upgrading Lasso Professional on balsa.forest.net and banana.forest.net from version 6.0.4 to 6.05. A brief restart of the server may be required.

Lasso Professional 6.0.5 includes fixes for security, reliability, and tag functionality issues. No code changes should be necessary, but we do recommend checking your site as soon as possible after the upgrade. Please feel free to contact client support if you have any questions or concerns.

Lasso Professional 7 Hosting is now available. Please contact your account manager if you're interested in upgrading.

Crabapple has been upgraded to FreeBSD 4.9-RELEASE. The upgrade went smoothly; we had a minor hiccup on the first reboot, but it was quickly resolved. Downtime was less than 5 minutes.

On Wednesday, January 14 (that's tomorrow), starting at 5:00pm, we will be upgrading FreeBSD on Crabapple. As this is a minor upgrade, we intend to attempt a "live upgrade," meaning the OS will be upgraded as the system runs. The only downtime should be a reboot at the end of the process.

We will post another update when the upgrade is complete.

Tonight during our scheduled maintenance window we will be performing an upgrade (adding disk space) to skil. Expect downtime of about 15 minutes, starting at 5:30 PM PST.

Our outgoing mail hub for both hosting and colocation servers, smtp.forest.net had a problem tonight, which slowed outgoing mail for about 4 hours. This has been fixed and mail is flowing again. It appears no mail was lost, as the queues just backed up, so there should be no need to resend.

Power will be brought down briefly this morning while we perform a series of disk upgrades. Expected downtime is approximately 15 minutes.

08:51 PDT
The new hardware is installed, and Power is back online. FTP services will be disabled for an additional period of time while we consolidate data and migrate it to the new disk.

11:27 PDT
Power will be down briefly to allow us to complete the upgrade. Expected downtime is again in the neighborhood of about 15 minutes.

11:50 PDT
All upgrades have been completed.

We have experienced a significant increase in ICMP traffic, from all corners of the 'Net recently. Today it caused a slowdown and some packet loss on our core switch due to the rapid growth of the forwarding table. We attempted to mitigate this with different switch configurations, but were unable to achieve a proper balance. At 2:30 PM we elected to just block ICMP echo traffic at our boundary router. We realize that this may "break" some monitoring systems, and make some troubleshooting more difficult - but the overall performance of our network is far more critical. We apologize for any inconvenience you may experience.

Should the "worms du jour" that generate this kind of traffic subside, we will enable ICMP echo packets again.

power.forest.net was briefly down (about 5 minutes) for a quick reboot due to a disk space related issue. Everything is now back up and running, but we will need to take the machine down again at 10:00 PM PST for roughly 15 minutes to install an additional hard drive. Please feel free to contact us if you have any questions or concerns.

September 17th, 9pm through 12:00 AM

During our normal scheduled maintenance window (Wednesday night,) we will be migrating several hosting & colocation subnets to newer, better ethernet switches in Server Room1. The subnets affected are:

216.168.37.0/24
216.168.47.0/24
216.168.60.0/24

All servers in these subnets will experience very brief outages (less than 30 seconds) as they are patched into the new switches.

No configuration changes will be required.
No servers in Server Room 2 will be affected.

Kudzu.forest.net has been experiencing some downtime lately; in an attempt to rectify the situation, we will be doing a two-stage upgrade. Tonight, we will upgrade the operating system to the latest release. Wednesday, we will upgrade WebSTAR to the latest release.

We will post again when the OS upgrade is complete.

Laurel is currently down for Tech Tool maintenance, and will be up as soon as possible.

Update:
Techtools turned up some software incompatibilities, we are now running a OS reinstall.... user files will not be affected.

Update:
Software reinstalled.. now making last minute configs.

All ASP.NET clients will experience intermittent periods of brief downtime between 5:00 and 6:00 PM PDT today, Thursday, July 31, 2003 as we perform hardware and operating system upgrades on the server buzz.forest.net.

We anticipate a quick and trouble-free transition to the new system; all database connections and customizations will be preserved. This entry will be updated when the upgrade is complete.

Update: The server migration is complete and tested. Please let us know if you have any questions.

Customers on Windows-based hosting servers may see a very brief period of downtime this morning while we install several security related updates to correct a severe vulnerability found yesterday in all Windows NT-based operating systems.

We strongly urge all co-location customers running any variant of Windows NT 4, Windows 2000, Windows XP, or Windows Server 2003 to update their systems immediately.

Additional information on this vulnerability is available here:

During the normal scheduled maintenance window tonight just after midnight (PDT) we will be performing the following procedures:

1. Swapping an Ethernet card on our boundary router.
This serves the 204.142.69.0/23 network that includes our newly acquired clients from Exceedia Inc. Users on this network will experience a brief service interruption, lasting approximately 15 seconds.

2. Installing a new switches in the 216.168.37.0/24, 216.168.58.0/24 & 216.168.62.0/24 networks. Users on these networks will experience a brief service interruption, lasting approximately 30 seconds, as each host is moved from the old switches to the new.

We will be performing an upgrade of the .NET frameworks on buzz.forest.net to version 1.1 at 5:00pm, Thursday, June 12.

During our normal scheduled maintenance window on Wednesday, May 21, 2003 at 4:30 pm, we will be transferring iron.forest.net to a new server.

This will involve:

* Shutting down all database, FTP, and remote access to iron.forest.net
* Transferring client data to the new machine
* Reconfiguring startup procedures on the new machine to properly start client data.

Work should take from one to two hours to complete. All data will be transferred in the server change, and will result in better response times, increased server stability, and improved connection reliability. Thank you for your patience in this matter.

Update: The transfer is complete. We have done our best to test functionality of many of the web sites hosted through iron, but cannot test everything. If you find anything unexpected, please alert us via our trouble ticket system, or contact emergency if a major component is down. Thank you again!

Cholla has been having a bit of trouble this morning. It's presently down while we run some diagnostic and repair tools to hopefully remedy the issue.

Downtime will be approximately 30 minutes.

1:25 PM PDT
Service has been restored, performance should be improved.

WebStar seems to be crashing routinely on Pear for some reason. We've taken the machine down for a quick tech tools run to try to determine the source of the problem.

10:10 PST
Unfortunately, running tech tools didn't correct the problem. We'll continue to work on it until we figure out what's wrong.

10:45 PST
Pear is back up and running.

Bigleaf is undergoing a quick TechTools run to hopefully resolve some problems that cropped up this morning. Shouldn't be much more than 30 minutes.

We're currently experiencing a minor interruption in network connectivity which is affecting the following servers:

Butternut
Catalpa (smtp.forest.net)
Chain
Circular
Power

Your patience is appreciated. We'll have everything back up as soon as possible.

11:27 PST
Service has been restored.

On Wednesday March 12 between the hours of 5PM and 9PM PST we will be performing physical moves of the servers in the list below.

Please expect 15-30 minute outages during this move.

These moves are to being done to improve datacenter and network efficiency.

Thanks for hosting with digital.forest.

sherwood.hst
bristlecone.hst
skil.hst
shrubbery.int
crabapple.hst
jig.hst
power.hst
souari.hst
chain.hst
circular.hst
catalpa.int
newchain.hst
logging.int
sabre.hst
rosetta.int
butternut.hst
boysenberry.hst
mail.infoasis.hst
coen.hst
qt3.bopjet.hst
qt1.bopjet.hst
interact.int
iron.hst

Bayberry has been causing a problem ever since its upgrade to os 9.2.2 and webstar 4.5. Specifically the blueworld.lasso.zip file in the Java Modules folder seems to be the suspect. We removed it for a day or two, but found that it controls the other java modules in lasso. Reinstalling it only crashes the machine ever couple hours. We have once again removed it which will keep the server up and running. Friday evening we will be looking at this issue closer to see if we can find a solution.

Thank you for your patience.

Butternut's upgrade was completed as scheduled. Catalpa experienced problems related to a conflict between the version of FreeBSD to which we were upgrading and the motherboard of the server. Yesterday, at the cost of extra downtime, we upgraded Catalpa to FreeBSD 5.0, and it's now running normally. Please open a trouble ticket at support.forest.net if you experience problems with Catalpa.

The extra work on Catalpa has delayed our work on the other machines. Revised schedule:

- Souari: Tonight after 5:00pm
- Boysenberry: Monday after 5:00pm
- Crabapple: Monday after 3:00pm

Crabapple will be down for extra time to allow for the installation of an additional disk.

Bayberry is currently undergoing some brief maintenance.

Expected downtime is less than 30 minutes.

08:08 PST
A short period of additional downtime will be required; several software updates haven't quite finished.

Cedar has been exhibiting an unusual degree of instability since late this morning, which has resulted in periodic disruption of web and FTP services based off that machine.

The server is currently down, and a technician is working to resolve the issue. Estimated restoration of service is approximately 30 minutes.

14:01 PST
Cedar is back online.

Olive is presently down for a much needed disk upgrade.

Total downtime should be less than 15 minutes; however, FTP access to Olive will be disabled for a longer period of time while we migrate data to the new disk. Service will be fully restored by this afternoon.

12:18 PST
The upgrade is complete. FTP access to Olive has been restored.

Those who are hosted on hazel have noticed that there has been an issue with it and that it has been hanging a lot recently.

Diagnostics tools were run on it today, a clean copy of the operating system was installed (which included a FirmWare update).

No User files were changed.
In end effect, Hazel's uptime and stability should be a lot better from here on.

Those who are hosted on laurel have noticed that there has been an issue with it and that it has been hanging a lot recently. Diagnostics tools were run on it today, a clean copy of the operating system was installed (which included a FirmWare update) and Webstar has been upgraded from 4.3 to 4.5. No User files were changed and there should be no changes to lasso.
In end effect, laurel's uptime and stability should be a lot better from here on.

crabapple.forest.net will be taken down for maintenance at 5:00pm on Wednesday, Feb. 12. We will be replacing the /usr disk that failed a couple weeks back, thus clearing up some space on /home.

At approximately 9:30 PM PST January 24th, digital.forest was affected by the worldwide internet attack known as W32/SQLSlammer. We were alerted within fifteen minutes of the first evidence of this worm as it entered our network. Preventative steps were immediately taken to minimize the impact on our network performance, and prevent damage to network devices.

At this point, all systems are performing normally, with the exception of a handful of client servers that are unfortunately affected by this worm. We are coordinating with clients to have their software patched and brought back online as soon as possible.

Any additional information on this worm can be found at:
http://www.cnn.com/2003/TECH/internet/01/25/internet.attack/index.html
http://www.bayarea.com/mld/mercurynews/news/5030801.htm
http://vil.nai.com/vil/content/v_99992.htm

Thank you for your ongoing understanding and patience.

elm.forest.net is undergoing some emergency system maintenance to correct an issue that is causing frequent downtime. It should be back up very shortly.

chain.forest.net was rebooted and configured for about 10 minutes while we performed some major operating system maintenance and upgrades. We apologize for any inconvenience this may have caused.

The server mimosa is currently down for emergency maintenance.
No ETR at this time.

The server is now back up.

The server crabapple.forest.net has suffered a hardware failure. We are working to recover from the failure and will have crabapple online as soon as possible. No user data was affected.

Update: The failure has been resolved, and crabapple is now back online. No user data has been affected.

FTP, MySQL and Web services on balsa.forest.net in preparation of an operating system upgrade and Lasso maintenance.

We are currently having a configuration problem on Cassia which is causing SSL pages to fail to load. We currently do not have an estimated time of when this problem will be solved. Thank you for your patience.

Update: Problem solved.

Cassia, Kola and Fuji should all be working normally now. Some logging data was lost during the repair time; everything else should be fine. Please let us know if you run into trouble.

We have suffered corruption of settings files for WebSTAR on Cassia, Kola and Fuji, plus the Lasso Data file on Fuji, possibly caused by repeated attempts to break into those servers on the WebSTAR Admin port. We have the servers back up, and are working on restoring the missing settings, which include secure realm information and, on Fuji, the lasso settings. We will also be blocking the admin port.

The server kola is currently down for emergency maintenance.
We expect to have the server back up shortly.

The hosting server mango.forest.net is down. A tech is working on it now. No ETA at the moment for it's return to life.

Some hosting servers may experience a temporary disruption of service due to
scheduled maintenance. The disruptions should not be more than 15 minutes.
Maintenance will be completed by 8:00 PM Pacific Time. Thank you for your patience.

The mesquite server is currently down for emergency maintenance.
There is no ETR at this time.

UPDATE: The server is now back up. 6:45 AM

We are seeing some network issues at our Vancouver datacenter. More news will be posted as it becomes available.

UPDATE: 6:00 PM PST. A technician is onsite at this time. Upstream provider appears to be the target of a DDoS.

UPDATE: 9:15 PM PST. The decision has been made to move the remaining (8) servers in Vancouver to the Seattle datacenter. Downtime should be just over 2 hours.

UPDATE: 11:25 PM PST. The servers are being set up in Seattle right now. New IP's will be mailed to server tech contacts where required.

UPDATE: 12:10 AM PST. The servers are all back online.

Balsa will be upgraded to 10.2 during our normal maintenance schedule.

Normal maintenance is performed from 6P-12P (PST) Wednesday evenings.

We're running some emergency maintenance on Balsa. It will be back online within the next 5-10 minutes. Thank you for your patience.

**Update**

This has been completed. Thank you for your patience.

cactus.forest.net will be temporarily offline for an operating system upgrade at 2:00 PM PT on November 10th, 2002. The upgrade will take approximately 1.5 hours, and will result in faster web serving and file access.

balsa.forest.net had suffered severe catalog and directory corruption -- after trying three different disk utility programs, we were finally able to restore the directory structure and bring the machine fully back online.

In the near future, we will be swapping in a new hard drive as an additional precautionary measure. The swap will be scheduled and coordinated to minimize any downtime.

balsa.forest.net, the Lasso 5 server, is experiencing disk problems. A technician is coming in to the office to make repairs and bring it back online. We will post again when the problem has been resolved.

iron.forest.net is now back online.

The server iron.forest.net has experienced a component failure and is currently down. We are in the process of replacing the failed componet and will have iron back online as soon as possible.

We will be doing some emergency network maintenance this evening that will result in outages lasting between 5-15 minutes.

This work is being done to add redundany to our network in the case of a hardware failure.

This outage should affect hosting machines ONLY.

UPDATE
This work is now completed.

A Techtools maintenance run is being performed on sycamore.forest.net The machine should not be down for more than 30 min during this time.

After some unscheduled maintenance, the server ebony is now back up.
Thank you for your patience.

The server ebony is currently down for some unscheduled maintenance.
Currently there is no estimate as to when it will be back up.

Laurel has being hanging over the last couple of days. We are now running Techtools on it to resolve any file system errors. It should be offline for no more than 30 minutes. Thank you for your patience.

We have updated some software on crabapple.forest.net for security reasons. Crabapple is now running:

- Apache 1.3.26
- CURL 7.9
- mod_ssl 2.8.10
- mod_perl-1.27

We also plan to upgrade the OS to FreeBSD 4.6, after which we will upgrade PHP to 4.2.2; these updates will wait for a server maintenance day since they will involve some downtime. None of the changes made today should affect existing code. mod_perl was not previously available.

We have upgraded some software on butternut.forest.net for security reasons. Butternut is now running:

• Apache 1.3.26
  
• mod_perl 1.27
  
• PHP 4.2.2

We will be upgrading apache on butternut and crabapple today. Webhosting downtime on these two machines should not excede 3 minutes.

butternut.forest.net is now back up and running fully.

Our tech has been able to boot and gain access to the machine and get apache running again. This means that your webpages are back up and running. Users will not be able to log in via FTP yet.

The server butternut.forest.net has suffered a major directory structure error and is down for emergency maintenance. At the moment, we do not know what is causing this problem so an ETA is hard to come up with. As soon as our Tech knows what the issue is and how to solve it, i will update this page again with an ETA.