At 2am tomorrow morning we will be shutting down the mail server "treehouse" to install new memory in it. We have been seeing RAM-related errors which caused the server some problems last week. We figured tonight would be a good time to bring it down and perform the hardware installation. Downtime should be limited to 15 minutes or less.

One of our mail servers, treehouse.forest.net, is experiencing problems and is currently down. We are working to restore it as soon as possible.

Thanks for your patience.

Update 11:25AM: treehouse is back up and running.

Final Report 12:30PM: Treehouse was experiencing memory-related errors and when we rebooted it the hardware self-test showed a failed RAM card. This particular server uses RAM in pairs so we had to source a pair of equivalent cards from our inventory to get the mail server running again. We have ordered additional RAM for both treehouse and our inventory and will likely schedule a brief downtime for treehouse over the holidays to perform this work.

Currently our mail server, treehouse.forest.net, is experiencing some general slowness. This will be resolved tonight during our scheduled maintenance period which is at 11pm PDT. We apologize for any inconveniences this may cause and appreciate your understanding as we work to improve our services.

At 3pm today, our mail server Treehouse will be taken down for preventative maintenance. We expect that the maintenance will last approximately 20 min. During this time, mail will continue to be received ant queued for delivery following the maintenance. We apologize for any inconveniences this may cause and appreciate your understanding as we work to improve our services.

Update @ 2:33pm: At this time the problems have ceased to exist on treehouse, we will continue to monitor the situation

Currently, one of our mailservers treehouse.forest.net is running slower then usual. The mail server is not down and it is still processing email, however at a much slower pace then usual. Possible side effects could be your mail client experiencing a connection error when it does a check every 10 or so minutes or the webmail interface being much slower then usual. We have a plan in place that we will execute tonight during our scheduled maintenance window so we can properly resolve this issue.

Currently the mail server issue seems to be resolved. Please allow some time for all email to come in as email servers around the globe will start sending your email to our server again.

Thank you for continued patience.

We continue to experience elevated traffic on our server, no doubt due to SMTP servers out there attempting to re-deliver messages that queued up overnight during the hardware change.

In addition due to the load however, we have not been satisified with the performance of the server software. In an effort to improve the performance, we have been working directly with our software vendor. At this time, the vendor is unable to resolve the performance issue. We are moving ahead with a software version upgrade as recommended by our vendor. We intend to begin this process this morning and will update this site as more information is available.

The new Treehouse server appears to be functioning well, however it is handling a very high load as it "catches up" with messages that piled up during the down time. Due to the load, some users are experiencing intermittant timeouts and the webmail interface is a bit slower than usual, but mail reception and delivery appears to be occurring with only minimal delays.

We appreciate your continued patience.

Treehouse will be shutdown at 10pm PDT and we will be migrating all of the users to a new server on a different platform.

Stay tuned for updates by monitoring this space.

Thank you for your continued patience

digital.forest technical support

The mail server is currently back online and churning through the queue that has been build up due the downtime. It is still performing poorly at this time.

Connections via your mail client (such as Eudora, Outlook, Entourage, etc.) will most likely not be working at all due to the state of the mail server.

Right now your best bet will be to access the web mail portion by going to http://mail.(yourdomain) and then to log in using your email username and password. This may not as well, but will be more reliable then using a mail client.

Tonight we will replace the mail server, treehouse.forest.net, with a different mail server in order to restore complete functionality to our email.

We thank you for your patience,

digital.forest technical support

In the next 30 minutes we expect to have an update from our email system administrator on a potential return to service time.

More information will be posted to this website at that time.

While message loss is a concern to everyone, we don't anticipate any significant email losses.

Thank you for your continuing patience,

digital.forest technical support

In our continuing effort to get our mail server, treehouse.forest.net, fully back online, we will be taking it off line for emergency maintenance. It is down until further notice.

Please monitor this space for further updates.

Thank you for your continuing patience,

digital.forest technical support

Update: 8:45 AM Mail.ninewire.com will be fixed next. Downtime should be around 15 minutes, starting at 9 AM PST.

Update: 8:25 AM Treehouse is back up. Still some issues to work out (TLS/SSL related) stay tuned.

We are carrying out emergency maintenance on two of our five mailservers over the next few minutes. This is to address the problem which we noted last night which was crashing the server. "Treehouse.forest.net" will be the first, with "mail.ninewire.com" being the second. We apologize for any inconvenience this may cause, but these problems require immediate attention in order to provide stable and reliable email service going forward.

You will note that mail from two of our five mail servers has an incorrect timestamp. The affected servers are "treehouse.forest.net" and "mail.ninewire.com".

A crashing bug surfaced in the software running on both of these servers whose only short-term fix is to roll back their clocks. We figured that the inconvenience of an incorrect timestamp is better than a crashing mail server. The developers have promised to respond to our issue within 24 hours, so hopefully we will have a fix before a full day goes by. In the meantime, please accept our apologies for the inconvenience until then.

Tonight at midnight PDT we will be shutting down one of our mail servers "treehouse.forest.net" for maintenance. Downtime could be up to 2.5 hours.

We are taking this step to mitigate or remove the issues that have been affecting performance of this mailserver over the past week (see post below). We have been working very closely with the mail server software vendor Communigate Systems to analyze the issues and investigate possibilities for solving them. Unfortunately doing these operations on a "live" production server during the business day is impossible. We believe that this downtime will be long enough to get them all done at once.

Thanks for your patience.

We moved our main mail server, treehouse.forest.net to both our new Seattle facility, and onto new hardware. We are working out some issues with regards to performance. Things should be back to normal later this morning. Please be patient while we work on this.

We have almost completed server moves to our new Seattle facility. Our DNS and mail servers will be moving over the next few nights. At this rate, we may be able to shut down the Bothell facility as early as next week after the weekend moves complete.

We will post updates concerning the mail servers at least 12 hours before we move them.

Update: 12:30 AM: RAID recovery almost complete. We should know by 1 AM what the next step will be. Stay Tuned.

Our other mail servers, Palm, Catalpa, etc are doing fine. Treehouse is the only mailserver down right now.

The power event from earlier this evening has caused our mail server "treehouse" some problems. We will use this blog entry to provide updates as we rebuild it.

Update: 9:10 PM: RAID array rebuild is about 50% complete, so our ETA of completion is about 11:30 PM. If the data is OK, we'll be back up shortly after. If not, we may make the tough call to run with what we have so we can continue and avoid an extended outage. Stay tuned.

Update: 11:45 PM: RAID array rebuild is at 85%. Time just is not on our side tonight.

At 11pm on Wednesday evening, we will be taking down our primary mail server, Treehouse, for minor maintenance. We'll be installing software updates and a fibrechannel card, which will permit Treehouse to use our new Xraid for storage. We don't anticipate downtime lasting more than ten minutes, and will post again when the process is complete.

No mail will be lost; sending servers usually retry after several minutes, then again every few hours for 1-3 days before returning mail. Mail that misses delivery during this maintenance window will be delayed, but should arrive shortly afterward.

The frequency of the attacks on our mail servers has decreased, but we are still experiencing spikes in SMTP traffic that effect webmail and outgoing mail. These spikes appear to be of limited duration. We continue to take steps in an effort to minimize the effects of these attacks on our servers.

Our primary mail server, Treehouse, is being hit with a high volume of Dictionary Harvest attacks. This may result in slowdowns in webmail and SMTP responsiveness, and occasional connection failures.
Thank you for your patience as we take steps to reduce the impact of these attacks.

Our primary mail server, treehouse.forest.net, is getting hit by an unusually high number of directory harvest attacks today. This results in saturation of our inbound SMTP connections, preventing users from sending mail through our server.

Our spam protection service, Postini, is detecting and blocking these attacks as quickly as it can. As the attackers quickly shift to new IP addresses, however, we are experiencing surges of activity followed by periods of calm. Thus, until the attacks stop, we can expect intermittent connection problems to continue.

This may give you an idea what we're dealing with. You can see a number of attacks, followed by decreased activity when the IP address in question is blocked, over the course of an hour. It ends with a particularly strong attack that maxed out all 500 of our connections for over a minute.

One of our main mailservers (treehouse.forest.net 216.168.37.80) was blacklisted by a major spam filtering organization (Spamcop.net) earlier today. The server was on the blacklist for about 3 hours. We are investigating the source and working with spamcop.net staff to resolve this issue and prevent a repeat. Further details will be posted when we know more.

Since many secondary services use spamcop.net as a blacklist source, we expect a 'ripple effect' for the next 24 hours or so. Please be patient.

We shut down Treehouse for five minutes today at about noon to add additional RAM, in response to performance problems under heavy SMTP loads. The server is performing significantly better now.

Treehouse is now up and running, with its upgraded completed. Downtime for moving the data to the new disk began at 9:00pm and ended at 9:39pm. We'll be monitoring it closely to make certain it behaves itself.

The new disk has been installed; all went well. We will be shutting the server down at about 9pm Pacific time to transfer the data from the old disk to the new one.

To address occasional performance problems with Treehouse, our CommuniGate mail server, we will be upgrading the disk this evening. This will involve a brief downtime at about 5:00 this evening, pacific time, to install the new disk; the server will be down for about 15 minutes. Later, starting at about 9:00pm pacific, we will be shutting down mail services to transfer the data to the new disk. This downtime will probably last about an hour; we will update this note if it looks like that time will be exceeded.

Our primary mail server, treehouse.forest.net, is down for emergency maintenance. Unfortunately, we dont know what caused the issue as it sneaked up on us. We are investigating the issue which at this point seems to be a file system error. If this is the case, then a simple disk diagnostic and fix will take care of it. Please watch this entry for more updates as they are available.

Very little mail should be lost as new incoming mail will be sent to our back up mail server, willow.forest.net. Willow will hold mail for up to 2 days and try to send it back to treehouse in intervals.

11:14am
Update. Treehouse was successfully fixed with the diagnostic tools and was restarted. You may find that the server runs a little slower than normal as the server catches up and the backup mail server, willow, sends its mail to treehouse.

Our downtime lasted considerably longer than we'd hoped, totalling almost exactly five hours. But treehouse is now running on a server architecture that, we hope, will better handle the high threading demands being placed on it by the continuous dictionary attacks. We should have a good idea of the improvement by tomorrow afternoon.

Thank you for your patience.

Our primary mail server, treehouse.forest.net, will be going down for emergency maintenance at 5:00pm this evening. Downtime should be less than two hours (we expect it to be considerably less than that, in fact, possibly only a few minutes). We hope that this will improve mail server performance, which has been dramatically affected by dictionary attacks for some time. We will update this notice when the server is back up.

As many of you are still aware, our CommuniGate mail server treehouse.forest.net is still slughish during peak times. We want to assure you that we are working to get this resolved. Options are being discussed in your best interest to reduce server downtime. More info will be shown here on support.forest.net when available. Thanks for your patience.

At 5:00pm Pacific Time tonight, we will be upgrading the CommuniGate software on treehouse.forest.net to version 4.0.1. This is a major, but easy upgrade, and features some improvements to the software that we hope will address the performance problems we've been seeing lately. The upgrade will take only a few minutes, during which time your mail will be unavailable. No mail will be lost.