****** Update 2/26/2010 15:45 PST ******

Since the network events of this morning we have been monitoring the stability of our network and collecting data from the reported anomalies. Our network has been and continues to be stable and operating as expected.

We are continuing to evaluate all of the data related to this mornings network events and expect to report on the findings by Monday afternoon.

****** Update ******

During the scheduled network maintenance this morning, a couple of our clients reported experiencing some anomalies.

We are investigating each of these reports to understand the nature of the anomalies and will provide the appropriate feedback when finished.

****** SERVICE IMPACTING MAINTENANCE ******

Impact
• Very short-term increased network latency at certain times during our maintenance window.
• Clients using persistent VPN connections may experience a momentary loss of connectivity across the tunnel while traffic is rerouting between carriers.

Starting on Thursday, February 25th at 23:00 hrs PST and finishing on or before Friday, February 26th at 03:00 hrs PST we will migrate our upstream network connections to our new core routing infrastructure.

digital.forest's ongoing commitment to providing the most reliable, robust infrastructure requires a constant evaluation of our datacenter environment. In February of 2009, we commenced a comprehensive analysis of our current network infrastructure, comparing it to what are now the most trusted and efficient network technologies available. Appraising our current and projected future requirements alongside available mature technologies, we initiated the process of testing numerous network hardware and configurations. Today, we are pleased to announce the result of our efforts, and our decision to deploy a more robust, higher capacity network for digital.forest.

Utilizing current models of Cisco carrier-grade network equipment, our new core infrastructure is designed to be fault-tolerant, easily expandable, fast and flexible.

The first phase of implementation included installing the equipment and creating the configurations, all while performing tests and burning-in. Phase One has now been completed, and we are delighted with the results. Phase Two involves migrating our upstream ISP connections to the new equipment. During this part of the upgrade, some clients may experience some very short-term, sightly increased network latency. The duration of this increased latency should be no longer than two or three minutes, and well within the tolerance of Internet applications.

To further reduce client impact during this transition, all network traffic will be routed across a pair of new connections on the new network equipment. Once all network traffic is passing across the new connections we will migrate the other connections to the new equipment and enable them for service. This process will allow the new routers to perform all of the heavy lifting on our network which will significantly enhance performance.
digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

Last night we experienced a hardware failure on grapevine.forest.net. All customer data has been restored to the server and we are in the process of rebuilding it. We will update here when the server has been restored to operational status.

******2/14/2010 08:40******

The transformer upgrade has been postponed and no work will be performed today.

After a delay due to load issues for the other building service testing began at 07:30. At 07:50 one of the rented generators experienced a catastrophic failure. No persons were injured and there is no damage or impact to digital.forest critical infrastructure or back-up power equipment. All of our critical infrastructure is fully operational and in place to serve our facility.

This transformer upgrade will be rescheduled for another date in the next month and we will provide an updated schedule when that has happened.

******2/14/2010 04:30 Update******

Due to a non-impacting issue with one of the rental generators we were unable to allow Seattle City Light to proceed with the transformer upgrade on Saturday. One of the generators rented to provide redundant power to another tenant in the building did not function properly and required a replacement brought in from Portland.

The upgrade will take place on Sunday starting at 05:30 with a similar schedule to the original. As previously planned, digital.forest has a full compliment of staff on hand and has coordinated closely with all of our vendors for emergency on-call service for all of our critical infrastructure should the need arise.

The new schedule follows here:

05:00 to 05:30 digital.forest staff will pre-flight and warm up our generator

05:30 digital.forest staff will transfer critical load from utility power to generator power

06:00 Seattle City Light will de-energize the main transformers feeding our building

06:00 to 12:00 Seattle City Light will replace on of the main transformers feeding our building

13:00 Seattle City Light will re-energize the main transformers

13:00 to 13:30 Seattle City Light and Veca Electric will validate the voltage, frequency, and rotation of the power from the new transformer

13:45 digital.forest staff will transfer critical load from generator to utility

13:45 to 14:00 digital.forest staff will monitor power conditions from the new transformer and cool down our generator

14:00 to 16:00 digital.forest staff and Veca Electric will remove the wire connections from the second generator and clean up the operations area

******NON-SERVICE IMPACTING MAINTENANCE******

On Saturday, February 13th Seattle City Light will be performing an upgrade on the transformer that provides power to the digital.forest data center. During this maintenance our facility will operate on generator power and will have a second generator in place for redundancy. This maintenance will provide additional power to the building and our facility. Our facilities management team has planned the work with representatives from Seattle City Light, Veca Electric, Sabey Corporation, and our neighbor tenants. We have planned contingencies for staff shortages, foul weather, time over runs, and broken or damaged parts. As always, we are taking all necessary steps to ensure continuous service to our customers and life-safety for everyone working on this project. The process for this work is very similar to the process used during the September 4th, 2009 work which you can read about here and here.

****** The schedule for work is as follows:

04:00 to 04:45 digital.forest staff will pre-flight and warm up our generator

04:45 digital.forest staff will transfer critical load from utility power to generator power

05:00 Seattle City Light will de-energize the main transformers feeding our building

05:00 to 14:00 Seattle City Light will replace on of the main transformers feeding our building

16:00 Seattle City Light will re-energize the main transformers

16:00 to 16:30 Seattle City Light and Veca Electric will validate the voltage, frequency, and rotation of the power from the new transformer

16:30 digital.forest staff will transfer critical load from generator to utility

16:30 to 16:45 digital.forest staff will monitor power conditions from the new transformer and cool down our generator

16:45 to 18:30 digital.forest staff and Veca Electric will remove the wire connections from the second generator and clean up the operations area

************ digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.