At 2am tomorrow morning we will be shutting down the mail server "treehouse" to install new memory in it. We have been seeing RAM-related errors which caused the server some problems last week. We figured tonight would be a good time to bring it down and perform the hardware installation. Downtime should be limited to 15 minutes or less.

On Saturday, December 29th during our scheduled maintenance window we will be making configuration changes to one of our upstream BGP peering sessions. This change will require a reset of the BGP session to complete. The reset will go unnoticed for the most part as our other peers will handle our traffic during the maintenance.

The maintenance should last about 30 seconds and will occur between 11:00 pm and 1:00 am.

One of our mail servers, treehouse.forest.net, is experiencing problems and is currently down. We are working to restore it as soon as possible.

Thanks for your patience.

Update 11:25AM: treehouse is back up and running.

Final Report 12:30PM: Treehouse was experiencing memory-related errors and when we rebooted it the hardware self-test showed a failed RAM card. This particular server uses RAM in pairs so we had to source a pair of equivalent cards from our inventory to get the mail server running again. We have ordered additional RAM for both treehouse and our inventory and will likely schedule a brief downtime for treehouse over the holidays to perform this work.

On Wednesday, December 19th we will be performing some maintenance on our UPS system. While this maintenance is being performed we will put the UPS system and grid power in bypass and operate the datacenter on generator power. This is standard procedure for work such as this. While we do not expect any change in power delivery, any time power is switched from one source to another, a risk of interruption is possible. Every effort and precaution will be taken to minimize this risk.

We will also be turning up an additional 100kVA PDU in Datacenter 2 to facilitate growth in that area.

Thank You,

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

Tonight (Thursday, December 13, 2007) is our annual digital.forest Christmas party. We're going to close the phones for the evening and ignore the trouble tickets for a little while and relax a bit. We will still be available for emergency issues, but please be patient. This is the ONE time of year we do this and it is just for a few hours. Normal support will resume around 10 tonight.

Thanks so much and all of us here at d.f wish you and yours a very merry Christmas!

--Chuck Goolsbee
VP Technical Operations
digital.forest, Inc

Sharp eyed observers may have noted that when you complete a trouble ticket in our Web Help Desk system it asks if you would like to take a survey. We value feedback from our clients and make use of these surveys all the time to improve our support.

The ticket system is very valuable to us since it ensures that no issue can "fall through the cracks" and our tech support staff can work around the clock on problems, handing them off when shifts change. Trouble tickets allow us to note trends and anticipate issues. The post-ticket surveys allow YOU to provide feedback to us about how satisfied you were with response times, quality of support, and length of time it took to resolve your issue.

We want to encourage our clients to fill out these surveys so we decided to pull names at random from the last two month's of survey responses and award them with a free gift. The names we pulled out were Daniela MacLeod from Rhythm Interactive, and her survey response on October 16th, and Dean Pachl from Photoworks for his survey response on November 11th. We're sending Daniela & Dean an Apple iPod in thanks for there participation in our trouble ticket survey.

You too can win an iPod. Just fill out a survey after your trouble ticket is completed. Provide us with feedback about your support expereience and you'll automatically be entered in our drawing. We appreciate your input and thank you for your time.

--Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

Most of the planned software updates have been completed; no downtime occurred. People who were having trouble with GnuPG should have more success now; if not, please let us know ASAP. We will be updating the remaining software ports this weekend.

Souari.forest.net will have some software libraries updated tonight, beginning at 10pm pacific time. We anticipate no downtime, but there is a possibility that some functionality (such as PHP) will experience interruption for a few minutes during the process. We will update this page again when the upgrades are complete.

Starting at around 2AM we started seeing temps rise in the datacenter, more so in DC 1. We've called our HVAC-systems vendor and they should have an emergency technician out here shortly. We've taken steps to mitigate temps in the meantime. We'll post updates as more information becomes available.

UPDATE: 03:55 Our HVAC vendor arrived within a few minutes of the posting above. They've corrected the issue and we are recovering nicely. Things should be back to normal in a few minutes. We'll post an update when we have definitive data on the cause.

We've had a few phone calls and emails from some of our valued clients around the world asking if we're OK, or under water. Have no worries! All the scenes on TV and pictures in the newspapers of flooding and damage are happening in other parts of the region. Your servers are high and dry, and we're comfy and warm. We do appreciate your concern, and our thoughts are with the people of the region who are in the thick of it, dealing with floodwaters & mudslides.

When we moved in late 2004/early 2005 we relocated to an area south of downtown Seattle and our building is safely up on a hill, far away from any water sources. Additionally our datacenter is on the top floor of the building, six stories above the ground. I do not have my camera today, otherwise I would post a photo of the facility and the environs to show how things look. The sun was even out a little while ago.

Ironically one of the areas hardest hit by the flooding is Bothell, specifically the area around North Creek Parkway, where we operated for seven years between 1997 and 2004. That area is home to many datacenters and other high-tech industries and we hope they are doing OK. In hindsight though we're very happy we relocated here, for many more reasons than just the flood risk.

Thanks again for your concern.

Chuck Goolsbee
VP Tech Ops
digital.forest, Inc.
Seattle, WA

One of our shared hosting servers, souari, is having trouble and is currently down. We apologize for the inconvenience.

Update 10:30AM: souari is back up.

One of our legacy Trident servers, celestial.wwwnexus.com, is currently experiencing technical difficulties and is not serving pages.

We apologize for the inconvenience this presents, and are currently working to bring it back to full operation. We'll edit the support blog when there's progress to report.

Update 9:20AM: celestial is now back online.

Earlier this month we had our backup generator's fuel quality tested and we found some algae in there. Given the size of the tank (5000 gallons/19,000 liters) and the wide temperature variations we've seen over the past year, it didn't really surprise us. Our previous test did not show any, but the recent one had enough to make us worry, especially as we're going into the critical months of the year for backup power availability. There must have been a recent "bloom" of the algae. We scheduled our friends at Fuel Care to come out and remove the contamination ASAP. They arrived on Friday.

The process involves parking their truck next to our fuel tank and cycling the fuel out of the tank and through special filters in the truck to remove the algal contamination. While that happens they scrub the tank walls and generally inspect the tank's interior while the access cover is off. Please note that we can still operate the generator if needed while this process is happening, as the fuel is constantly cycled through Fuel Care's filter system and no more than a few hundred gallons are out of our tank at a time. In many ways it is akin to kidney dialysis, but for Diesel fuel.

Thankfully the tank itself was very clean and appeared in great shape. We removed about a gallon and a half of water (5 liters) accumulated via condensation and all the algae and now our fuel supply is sparkling clean.

The whole point of this process is to ensure the smooth operation of our backup power supply. You rely upon us to maintain our facility to the highest standard and ensure the maximum uptime for your servers. We just like to share all the things we do with that in mind so you stay informed. We'll have more news and info about other steps we're taking in that direction soon.

Regards,
Chuck Goolsbee
VP Technical Operations
digital.forest