REMINDER

On July 14th, 2008 one of our upstream providers will be performing some maintenance on their equipment. There will be 2 separate outages of up to 45 minutes in duration. During these outage periods our other connections will carry all of our traffic. You may see some latency as routes shift. VPN connections running over this provider will drop and re-connect.

The maintenance will occur between 01:00 and 04:00.

digital.forest is in the process of replacing some equipment in our Vancouver, BC facility. As part of the hardware upgrades, we will be separating the secondary mail and DNS services currently provided by willow.forest.net.

Domains that currently use a non-digital.forest primary mail server, and use Willow as a secondary mail server, will be have their secondary MX service repointed at the replacement secondary mail server, weeping-willow.forest.net. We control the DNS service for most of these domains, and will be making the change later this afternoon; it will take up to 24 hours for the change to propagate. The old server will remain up and running at least into the middle of next week, so there will be no effect on secondary MX service while DNS propagates.

We will be contacting the owners of those domains that are controlled by customers and require similar changes.

We will be disabling the FTP server on date.forest.net for about 2-3 (11:30 - (13:30-14:30) PDT) hours in order to perform an active backup of the server. Once the backup is complete the FTP server will be turned back on.

Thank You

It is a perfect summer day here in Seattle, sunny at 65°F (18°C), and we're preparing the roof for the arrival of another AC unit. Lots of interesting things to share!

The new 70 Ton Aaon Air Handling Unit will arrive in less than two weeks. This will bring our total cooling capacity to 333 tons. Just like we did last time, the locations must be prepared ahead of time, first penetrating the roof to provide supply and return ducting, as well as mounting points attached to the building's steel frame. This latter task provides mounting for the steel superstructure that the air handling unit is secured onto. Unlike last time however, the weather is great and we don't have to plan our penetration work between rain squalls.

The first thing completed by our contractor MacDonald-Miller was the supply duct penetration. It was done in the very early hours of the day when it was still very cool outside to minimize the impact on datacenter temperature. This penetration provides a very large opening between the new HVAC system's supply and our facility's cold air plenum. The plenum runs the length of the datacenter and all the HVAC systems supply it with cold air. This allows a common air supply so that our HVAC units can share the duties of cooling the facility. By feeding a common plenum they can provide redundancy to each other.

Above: The completed and capped supply duct in the background. The return duct in the foreground is positioned and the blue paint shows where it will be countersunk into the roofing material.

In addition to the openings through the roof for air ducts, five smaller holes had to be prepared. Four will contain the steel supports for the superstructure, and one for the electrical service to operate the HVAC unit.

Above: One of the smaller penetrations in progress. Mt. Rainier in the background.

Above: Larry Mayrand (left) from MacDonald-Miller operates the sawzall on the roof's pandeck, while digital.forest Facilities Manager Kevin Teker runs the shop-vac to minimize the spread of collateral. Mike Klein from MacDonald-Miller (right) assists.

Directly below these support penetrations lie the steel I-beams that make up the skeletal structure of the building. Onto these will go the support columns for our HVAC superstructure.

Above: Larry Mayrand checks the plumb on the support. Downtown Seattle is in the background.

After these penetrations were completed a roofing crew arrived to re-seal the areas around the work. Our roof is insulated by a special foam, and sealed with a few layers of a tough but light/heat reflective material with tar in the joints. This keeps the heat, and rain out, while keeping the facility below nice and cool.

Other Changes:
Sharp-eyed observers will note a few subtle changes to digital.forest's rooftop landscape when they compare these photos to the ones referenced above concerning our last roof-related work. Kevin Teker took me around to show me these to share with all of you:

Above: Inlet screen filters.

Late Spring here brings on a veritable blizzard of pollen and airborne cottonwood seeds. We have found that these clog the HVAC filters very fast (sometimes within hours) reducing the efficiency of our cooling systems. To combat this we have constructed and installed various inlet screens on our air handling equipment. We keep a shop-vac on the roof and part of the regular schedule of the NOC staff 24 hours a day is to check the condition of the inlet screens and clean them off with the vacuum when they start to clog. This saves our filters which are better served handling routine airborne contaminants rather than a seasonal flora bloom. The screens will come off in another week or so, but now seems to be a good time to share them with you. The big box sections above are the large intakes for our original AC system. Our new systems are much more compact and easier to screen off. If you look closely at the top photo posted in this entry you can see a full rooftop shot that includes our twin systems installed last year. The inlets are the large horizontal areas near their tops. A close up view reveals that they are covered by a snap-on fabric inlet screen:

Above: A closeup view of the "cottonwood screen" on one of the inlets of our Aaon AC unit. Below: A wider shot of the installed screen.

We sweat all these details so you don't have to.

We'll have more updates as we get closer to completion, so stay tuned.

--Chuck Goolsbee
V.P. Technical Operations
digital.forest

Tonight at 7:28 PM we experienced an issue with one of our boundary routers. This caused brief interruption with two of our BGP peer sessions. This lasted until 7:33 PM. The traffic was not lost, as it switched to our other connections on our other boundary router. This is how the system is designed to work. Persistent connections, such as VPN tunnels may have been temporarily down as they reset themselves if they were routing over one of the lost connections.

This particular event is linked to our effort to assist a digital.forest colocation client whose server has been the subject of a denial of service attack. This attack has had no direct impact on any digital.forest customers other than this particular client. We'll post more news if required.

celestial.wwwnexus.com is currently offline for emergency maintenance. Unfortunately we do not have a time frame for its return. We will update the blog when we have more information. Sorry for the inconvenience.

DC3 is almost complete! In fact some customers have already started moving into the space. In early July our next AC system will be arriving on the roof. This will bring our total HVAC capacity up to 333 tons!

Work on this rooftop phase of the Datacenter 3 project starts on Monday, June 23rd. As you have come to expect, we'll be posting updates on the progress of the work, complete with photographs. Stay tuned.

Tomorrow morning (Tuesday, June 17th, 2008) at 02:00 Pacific Daylight Time, we will be performing some routine maintenance on our Helpdesk system, remote reboot system and phpmyadmin systems. This means that for a short time after 02:00 PDT helpdesk.forest.net, forest.net/support/tools/reboot.php and phpmyadmin.forest.net will not be available. If you need to contact the digital.forest support team during this time or at any other time, please feel free to call our phone line at 877-720-0483 option #3.

Thank You

Update 02:05 PDT: The server maintance is now complete and all services associated with this maintenance are now back online.

Thank You

At this time our shared hosting servers bristlecone and durian are experiencing some elevated traffic and are not responding to requests as quickly as normal. We are working to mitigate the extra load on these servers and expect to have server performance levels return to normal within the hour.

Further updates will be posted here as things develop.

Treehouse's maintenance has been completed; the server is up and running again with new RAM modules. This should stop the repeated crashes we've seen in the past two days.

Thank you for your patience.

We are moving the emergency maintenance on Treehouse up, and will be performing it immediately, as the RAM failure appears to be accelerating. Downtime should be brief.

The mail server "treehouse.forest.net" which acts as a primary mail server for a portion of our users has had some issues with stability over the past 24 hours. Our diagnosis points to memory-related errors. We have new RAM for this server and will replace it this evening around 7PM Pacific Daylight Time. The maintenance interval should be about 45 minutes in duration. No inbound mail will be lost, as it spools on secondary servers, however users will not be able to send or receive electronic mail during this maintenance interval.

Treehouse may be unreachable for short periods of time during the day today as we wait until the after-hours maintenance window. If it becomes untenable during business hours we may elect to perform the RAM replacement earlier, but will do everything in our power to avoid interrupting mail server during the business day.

Thank you for your patience while we address this critical issue.

On June 11th, 2008 starting at 00:00 PDT we will be performing an upgrade to the Lasso service on bristlecone.forest.net. This update will increase security, performance and stability of the lasso server. We will be upgrading bristlecone's Lasso from 8.5.4 to 8.5.5. We do not expect this update to take more then 1 hour and as such the downtime associated with this update should be over by 01:00 PDT.

If you wish to review the changes to Lasso involved in this update please reference: http://www.lassosoft.com/downloads/updates/index.lasso?9325

Update: June 11th, 2008 00:57, the update to the Lasso software on bristlecone.forest.net is complete and the server is again serving lasso web pages as expected.

Effective immediately we will be performing emergency maintenance on the date.forest.net Lasso web server in order to address some performance issues. This maintenance will impact all websites on the date.forest.net server and will cause them to be unavailable sporadically until this maintenance is complete.

We will strive to keep this outage as short as possible but we do not currently have an ETA for when this emergency maintenance will be complete.

Thanks for your understanding while we address this issue.

Update: date.forest.net is back online and we do not expect to have this server down anymore. This concludes the emergency maintenance for date.forest.net.