*****Service Impacting Maintenance*****

Thursday morning at 1AM PST (7/15/2010) In a continuing effort to maintain a stable and secure environment the shared hosting server BUTTERNUT will be taken offline and its core operating system components will be patched to current standards. The work is expected to include less than 1 hour of downtime.

Note: This update process is specifically for core OS components and does not affect the web server or its runtime components (i.e. PHP). Prior to any work on the system all customer data will be fully backed up.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

On Sunday, October 31st beginning at 22:00 hrs PST we will perform several software updates on spirea.forest.net and butternut.forest.net. We anticipate this outage to last approximately four to six hours on Spirea, and less than 30 minutes on Butternut.

The updates to Spirea include the base operating system and several core components. These are mostly minor code revisions and should not cause any significant impact beyond the maintenance window.

On both Spirea and Butternut, we will also be upgrading PHP from version 5.2.x to version 5.3.0. This is a more significant upgrade. There are a number of important changes in this version of PHP, which are detailed here:

http://php.net/migration53

Please pay particular attention to the "Backward Incompatible Changes" page:

http://www.php.net/manual/en/migration53.incompatible.php

It would also be wise to check the "Deprecated Features" page:

http://www.php.net/manual/en/migration53.deprecated.php

Deprecated features will still function, but will be removed in a future release.

If you wrote your PHP code yourself, please verify that it will be compatible with the new 5.3.x release. If you use PHP code written by somebody else, please contact the appropriate support channel for the software and verify that it is compatible with 5.3.x.

We will publish an upgrade schedule for the servers closer to the actual work date.

Thank you for your attention.

******NON SERVCE IMPACTING CONFIGURATION CHANGE******

As part of the ongoing process to improve and enhance the statistical reporting package on our FreeBSD servers, one small change has been made to the error logs found in your home directory.

Previously, the current log was named "error_log", and all older logs were named "error_log-YYYYMMDD.bz2", reflecting the date of the log and the fact that it had been compressed.

Starting today, the current log will be named "error_log-YYYYMMDD", using the current day's date. The older logs remain unchanged.

If you have any questions or concerns about this change, please contact digtal.forest Technical Services via a trouble ticket at http://helpdesk.forest.net or via phone at 206-838-1630 option 3

************

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PDT at 877-720-0483 Option 2.

Butternut's upgrades are complete and the system is running. Our standard tests have been successful, but if you discover anything on your site that is no longer working properly, please contact support and we will address your issue as quickly as possible.

Note that PHP was upgraded to 5.2.10, not the brand-new 5.3.0. As 5.3.0 is a more significant upgrade, we will announce several weeks in advance before performing that upgrade, so that you will have time to examine the release notes and determine whether you need to make any code changes.

Thank you for your patience during this upgrade.

As announced on July 6, butternut.forest.net is now going down for scheduled maintenance. Please expect intermittent downtime for the next several hours. We will post again when the upgrades are complete.

****** SERVICE IMPACTING UPDATE ******

On Monday, July 13th beginning at 22:00 hrs PST we will perform several software updates on butternut.forest.net. We anticipate this outage to last approximately four to six hours.

The updates include the base operating system and several core components. These are minor code revisions and should not cause any significant impact beyond the maintenance window.

Butternut will now be running its own stats, which means butternut users will need to use a new URL to access their stats and logs. Users should be able to see their stats at:

http://butternut.forest.net/stats/username/

...where "username" is, of course, replaced with their own username. As a side benefit, you will also find a file in your home directory named "webalizer.conf.sample". If you edit this file and upload a new copy named "webalizer.conf," our stats system will use your config file while generating your stats. Note that while we permit this, we don't support it; it's available to you if you want to decipher the webalizer config file, but we won't provide tech support to help you with that. However, it's pretty well documented in comments inside the file.

Butternut's stats need to be caught up, starting with November 2nd, before it can start running them automatically on a nightly basis. We're not sure how long this will take, as it depends heavily on the amount of log data and how many unique IP addresses are contained within it, but we'll be running those updates as quickly as possible.

Butternut is now up and running on the new hardware. In addition to the hardware upgrade, there have been some software upgrades; Butternut is now running exactly the same software as Souari. Your site can make full use of PHP, mod_perl, and we can now support SSL certificates on Butternut as well.

If you run into any problems with your site on Butternut, please contact us immediately and we'll resolve it.

As of 8:20am, we have disabled FTP connections on butternut. We will momentarily be disabling the web server, and will then begin the final stage of the upgrade. We'll post here again when it is complete.

One of our main hosting servers, butternut.forest.net, will be upgraded tomorrow morning to a much more powerful machine. We've been prepping the machine all day today; tomorrow morning, we will disable FTP access to the server to prevent files from changing, and synchronize the data. When this is complete, we will switch off the web server and perform a final synchronization, then switch on the web server on the new server. The downtime should be minimal, on the order of minutes.

We are performing this upgrade in the morning, rather than in the middle of the night, so that users of butternut can immediately detect any problems that occur due to the upgrade. While we are striving to duplicate the setup as closely as possible, performance considerations and the new hardware itself demand that some software upgrades accompany the hardware upgrade. Should any problems arise with the site after the upgrade, please call us immediate at 425-483-0483, and we'll fix the problem as quickly as possible.

Thank you for your patience! We think you'll appreciate the result.

We experienced another anomaly on the hosting network due to an unknown issue with Butternut. The machine was isolated from the network, and will be further tested. Hosting services should be returning to normal.

Attention all Butternut users: As part of our emergency repairs two weeks ago, we installed a temporary disk in Butternut, onto which we restored your data. We are now prepared to install a permanent disk.

We have already duplicated the data onto the permanent disk. At 5:00 this evening, we will shut down Butternut for approximately 15 minutes to sync the data changes from the temporary disk to the permanent one, install the permanent disk, and re-rack the server. We expect the server to be back up by 5:15pm.

Thank you for your patience with this issue.

At this time, all customer data has been recovered into a directory named "RECOVERED" inside your home directory. We will be moving data into the web root for those customers who requested it via ticket.

We have a full restore of our damaged hard drive, and we are copying the data back to user home directories now. This is a two-step process. First we will restore the entire home directory into a folder called "RESTORED". For those clients who have let us know they need their sites restored fully, we will be then moving this data into their appropriate directories within their home ("www," etc.)

If you fully restored your own data from a development server or other local copy, you will still find useful items in the "RESTORED" directory such as old log files. Feel free to move, download, or delete items found there.

For those that need to selectively restore certain fies (such as CGI's) you will be able to do this from the "RESTORED" directories as well.

Special requests submitted via trouble tickets since Sunday will be answered via the ticket system. If you have additional requests, you can add them to existing trouble tickets or start a new one if you haven't done so.

Normal backups of all our hosting servers, including butternut resumed about 48 hours ago.

Thanks for your patience and understanding.

First I must apologize for not posting an update yesterday. I spent the day on the phone, or answering emails about the situation.

First the Update: It looks like our expected FULL recovery time is mid-day Thursday (7/19), though we may begin to restore some directories as early as tonight. We have a list compiled via our help desk and emails of directories to make priority in recovery. The plan will be to create new directories in your home login areas with the recovered data. We WILL NOT overwrite any changes made by you since Saturday without your prior written permission. This will allow you to integrate the recovered data as needed.

Many of you asked if this has any affect on email. The answer is no, email is handled by different servers.

Many of you have asked how this happened, and why we do not have backups, or backups of our backups. The answer is that we do, but the backup systems (two of them in fact) failed prior to the disk trouble on butternut. To make a very long story short, our tape backup system suffered a mechanical failure last week, destroying a tape set in the process. Our "backup" backup system was then employed. It uses a large (1 terabyte) disk array which is one of a set we use to backup our colocation clients. It started backing up our hosting servers last Wednesday. The backup software reported an error on Friday night that required us to rebuild the catalog from source data. This is a very time-intensive process that can take many days. Of course, less than 24 hours later is when the drive in butternut failed. So, in the end two backup systems failed prior to the loss of one of butternut's drives.

Currently we are pursuing multiple avenues for data recovery, and have excellent faith at least two of them. As soon as we have good data to restore we will post an update here, and will not leave until butternut is fully restored.

This afternoon (Saturday, June 14th) at 15:48 PDT the hard drive containing the user directories of the web server "butternut.forest.net" crashed. The timing of this issue is particularly bad as the VXA drive in our tape library broke two days ago. A new tape drive has been on order since Thursday the 12th, but is not expected to arrive until Wednesday at the earliest.

We have several alternative recovery methods available to us, and will be exploring all of them. In the meantime however there are some steps you can take to mitigate the issue.

The server is up and running with a new set of drives. Your FTP login and web site configurations are still valid. If you have a local copy of your website, you can re-upload it immediately. Remember that is always a best practice to keep a local copy of your web site for just such an occasion. Most of our clients do their development in-house or with a web developer who maintains a staging server for local testing, so if you are one of these clients you are in good shape.

If you do not have a copy of your current website, it may be some time before we can fully recover your data, so you may want to upload a temporary site or page to "place hold" your site until that time comes. Additionally please let us know, either via email, or by opening a trouble ticket so we can make recovering your data a priority.

We appreciate your patience and understanding, and apologize for any inconvenience this has caused.

We're having problems with one of Butternut's disks. The machine is presently down while we assess the problem and perform the necessary repairs.

Your patience is appreciated.

We have upgraded some software on butternut.forest.net for security reasons. Butternut is now running:

• Apache 1.3.26
  
• mod_perl 1.27
  
• PHP 4.2.2

We will be upgrading apache on butternut and crabapple today. Webhosting downtime on these two machines should not excede 3 minutes.

butternut.forest.net is now back up and running fully.

Our tech has been able to boot and gain access to the machine and get apache running again. This means that your webpages are back up and running. Users will not be able to log in via FTP yet.

The server butternut.forest.net has suffered a major directory structure error and is down for emergency maintenance. At the moment, we do not know what is causing this problem so an ETA is hard to come up with. As soon as our Tech knows what the issue is and how to solve it, i will update this page again with an ETA.

A security flaw has been exploited on butternut that allows a web user to send high volumes (between 500,000 to a million per day) of 'spam' via a web form. We started receiving complaints from all over the Internet late last week about spam from butternut. Earlier this week we attempted to filter the spammer(s) with some success, but in so doing we were also filtering some legitimate mail as well. The exploit uses the PERL script 'formmail.pl', which is a very popular and widespread form-to-mail script. We have attempted to filter based on two inherent 'signatures' of the spam: a "Return-Path: " header and a "From: (anything)@aol.com" (or yahoo.com, or hotmail.com) address.

The filter was very successful, in that our outgoing spam dropped to zero, but this obviously caught several legitimate emails for our clients too. Not wanting to be an unwitting relay for spammers, but also not wanting to break any of our clients' website functionality has left me in an awkward position. I violate a policy which ever way I go. My goal is to restrict the use of formmail.pl-driven emails solely to the legitimate use of our clients, within the limits of our stated anti-spam and privacy policies.

As of now (Fri, Apr 5, 2002 3:48 PM) we have dropped the filter on the 'nobody@butternut.forest.net' header, but have kept the filters on the common 'spam return' ISPs of Hotmail, Yahoo, & AOL. This should trap the vast majority of the 'bad' mails, while allowing the majority of 'good' mails to go through. On Monday we will try and come up with a better spamtrap.

Until then you may want to edit the html of your pages to reflect the above changes and notify users of your forms to not use an address from one of these ISPs.

Best of all, you can edit the PERL script to specifically set a return or reply-to address from your domain that is valid, so that the script does not default these values to 'nobody@butternut.forest.net.

Thanks for your patience.