******NON-SERVICE IMPACTING UPS SYSTEM MAINTENANCE******

On Monday, March 7th between 08:00 hrs and 17:00 hrs, we will perform preventive maintenance on two of our UPS systems. For the duration of this maintenance window, power will be supplied by our generator as the UPS systems will be taken offline. Power to client equipment will not be disrupted during this maintenance.

******************

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******Update 5/8/10 16:30 hrs******

As of 16:30 hours this work is complete and we are once again operating on grid power. During the work there were no issues and all systems functioned normally.

******NON-SERVICE IMPACTING MAINTENANCE******

On Saturday, May 8th, Seattle City Light will perform an upgrade on the transformer that provides power to the digital.forest data center. During this maintenance our facility will operate on generator power and will have a second generator in place for redundancy. This maintenance will provide additional power to the building and our facility. Our operations team has planned the work with representatives from Seattle City Light, Veca Electric, Sabey Corporation, and our neighbor tenants. We have planned contingencies for staff shortages, foul weather, time over runs, and broken or damaged parts. As always, we are taking all necessary steps to ensure continuous service to our customers and life-safety for everyone working on this project. The process for this work is very similar to the process used during the September 4th, 2009 work which you can read about here and here.

************

The schedule for work is as follows:

05:00 to 05:30 digital.forest staff will pre-flight and warm up our generator

05:30 digital.forest staff will transfer critical load from utility power to generator power

06:00 Seattle City Light will de-energize the main transformers feeding our building

06:00 to 17:00 Seattle City Light will replace on of the main transformers feeding our building (the schedule will flex from this point forward depending on how much time is required for Seattle City Light to perform their work)

17:00 Seattle City Light will re-energize the main transformers

17:00 to 17:30 Seattle City Light and Veca Electric will validate the voltage, frequency, and rotation of the power from the new transformer

17:30 digital.forest staff will transfer critical load from generator to utility

17:30 to 17:45 digital.forest staff will monitor power conditions from the new transformer and cool down our generator

17:45 to 19:30 digital.forest staff and Veca Electric will remove the wire connections from the second generator and clean up the operations area

We will provide an update here following the completion of this work to indicate that our facility is once again being serviced by grid power.

************

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******Update 12:30 Saturday 3/13******

We finished testing at 12:00 noon with no service interruptions. All of our critical infrastructure is operating normally.

The tests we performed once again demonstrate the unusual behavior and our electrical engineering firm was able to collect significant amounts of data from the generator and from multiple points on our electrical distribution system. They will evaluate it over the next week to identify the root cause of the unusual behavior we have observed and develop the necessary corrective actions.

We will reschedule the Seattle City Light high-voltage transformer upgrade once we have a clear understanding of the issue and have verified that the necessary corrective action has been taken and the result proven.

******Non-Impacting Facility Testing******

On Saturday, March 13th between 07:00 and 12:00 hrs. we will perform a number of tests on our emergency power system. These tests will not have any direct impact to the operation of the facility.

We have coordinated with our vendor partners, and will have a full compliment of df staff on-hand to investigate the issues we've experienced during Seattle City Light's attempts to upgrade the high-voltage transformer that supplies our facility.

We will cycle our automatic transfer switch (ATS) a couple of times during this window in order to provide facility load to our roll-up generator. Throughout this process, we will be monitoring our HVAC equipment and performing extremely detailed power quality monitoring on the generator power to help isolate the root cause of the unusual behavior we observed last Saturday.

We will provide an update here following the completion of these tests.

************ digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******Update 3/6/2010 14:10******

This maintenance has again been postponed due to rental generator related issues. We will reschedule and provide an update with the new date once it has been determined.

As a result of the generator related issues, some of our HVAC equipment experienced a power quality issue (client equipment would not be impacted as all of the power to the data center floor is scrubbed in our UPS systems). The result was some increased temperatures in some of our data center spaces. digital.forest operations staff was on-hand to mitigate and we resolved the issue quickly. Some clients may have noticed high temperature alerts on their monitoring systems but no supply temperatures should have exceeded 80° F. We are continuing to monitor our HVAC systems to ensure proper operation.

******Update 3/6/2010 06:15******

We have postponed the start of this maintenance until today at 12:00 noon due to issues with one of the rental generators for the building. The duration of the work will remain the same once we begin.

******Update******

The correct date for this maintenance is Saturday, March 6th.

******NON-SERVICE IMPACTING MAINTENANCE******

On Saturday, March 6th Seattle City Light will perrform an upgrade on the transformer that provides power to the digital.forest data center. During this maintenance our facility will operate on generator power and will have a second generator in place for redundancy. This maintenance will provide additional power to the building and our facility. Our facilities management team has planned the work with representatives from Seattle City Light, Veca Electric, Sabey Corporation, and our neighbor tenants. We have planned contingencies for staff shortages, foul weather, time over runs, and broken or damaged parts. As always, we are taking all necessary steps to ensure continuous service to our customers and life-safety for everyone working on this project. The process for this work is very similar to the process used during the September 4th, 2009 work which you can read about here and here.

****** The schedule for work is as follows:

04:00 to 04:45 digital.forest staff will pre-flight and warm up our generator

04:45 digital.forest staff will transfer critical load from utility power to generator power

05:00 Seattle City Light will de-energize the main transformers feeding our building

05:00 to 14:00 Seattle City Light will replace on of the main transformers feeding our building

16:00 Seattle City Light will re-energize the main transformers

16:00 to 16:30 Seattle City Light and Veca Electric will validate the voltage, frequency, and rotation of the power from the new transformer

16:30 digital.forest staff will transfer critical load from generator to utility

16:30 to 16:45 digital.forest staff will monitor power conditions from the new transformer and cool down our generator

16:45 to 18:30 digital.forest staff and Veca Electric will remove the wire connections from the second generator and clean up the operations area

************ digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******22:10 Update******

Above: Our building at 10 PM tonight.

We transferred back to grid power from our generator at 11:05 PM. The Secondary Generator never saw any use, but it did its job, namely providing us with a proper safety net. Now the task of packing up all the cables and putting everything back into normal operating conditions begins.

Our thanks go out to Sabey, for staying on Seattle City Light to get this maintenance completed tonight.

Thanks as well go to the digital.forest Technical Operations staff who have worked tirelessly over the past several weeks to prepare for this event. The fact that it went seamlessly is a testament to their professionalism.

******21:30 Update******

Grid power has been restored to the campus. We will be transferring back momentarily.

******20:00 Update******

Word from SCL is that grid power may be restored around 10 PM.

******19:00 Update******

Things are running well. SCL continues to do their work. We are still here keeping things running. Here is a quick video we shot showing a regular check of the backup power system as it is running. (Warning: It is very loud in the generator room, put on your ear protection.)

Above: Our building at 5 PM today.

******12:00 (Noon) Update******

The Good News: Everything is running fine.

The Bad News: Seattle City Light's original ETA of "around 3 PM" for completion of the work and restoration of grid power is now "around 6 AM tomorrow."

We are prepared. We have enough fuel on-site to run until Tuesday of next week, though we'll likely top off tanks later today.

Meanwhile Seaport Biofuels just stopped by to top off the tank of the generator running the building elevator and common-area lights:

******10:30 Update******

Above: digital.forest Facilities Manager Kevin Teker points to the fan relay he just replaced.

Why there is really no such thing as a "lights out datacenter" and why do we maintain all of our own facility systems? A short time ago Kevin Teker, our Facilities Manager performed a fan relay swap. On the right-hand side of the image above you see several large cooling fans. They supply air for our backup generator system and are driven from large electric motors. A short time ago they stopped working when their relay failed. Kevin noted this and opened up the box where the relay is located and replaced the failed relay with one of the spares he keeps right in that very box. We love Kevin, and you should too. Why? He always thinks of these things and like a Boy Scout, is always prepared. Had this happened in an unattended "lights out" facility the generator would have slowly overheated and perhaps eventually failed. Instead, because our lights are on and we're always here at digital.forest, the issue was noted, and resolved in a matter of minutes. It is never the big things that go wrong, it is the little things that eventually become big things. We keep an eye on the little things. Not the landlord. Not an outsourced facilities contractor. Not a real estate company. Our full-time staff here at digital.forest is always looking out for you.

Above: Kevin labels everything. The new relay is installed on the right, the new spare, replacing the old spare, is on the left side of the relay box.

Above: Kevin closes up the relay box after he's done with the repair.

******09:45 Update******

Grid power went offline around 09:00. All systems running normally.

Above: The reason for this event: Road construction on International Blvd. Seattle City Light is relocating the power feed for the campus as part of the road widening.

******08:45 Update******

Above: Generators.

The photo shows several generators. The large one on the left is the secondary generator for the digital.forest datacenter. The small green one in the middle is running backup power for the building, including one of the two elevators. The digital.forest offices are powered by our own generator but it is nice that the elevator is working today. In the distance you can see another portable generator in front of Building B. The entire campus will be off the grid during this maintenance event.

Above: More Generators.

The large generator on the left is another view of the secondary generator for our datacenter. The small one on the right is running the block heaters for the secondary generator. This minimizes the startup time for the secondary should we need to bring it online. In the background you can see the exhaust stack from our primary generator.

Above: The view of all the connections in the basement.

This is the scene at the convergence of all this activity in our building. Behind the sound baffles is our primary generator system. To the left of the door is our Bus Tie, where the "portable" generators tie into our power system. On the right is where the building backup generators to run lights, elevators, etc are tied in, as well as the other datacenter facility in our building.

******08:15 Update******

We transfered our facility load to our primary generator at 08:00. All services are operating normally and are expected to continue doing so. Seattle City Light crews will de-energize the campus feed sometime in the next hour or so. We will not notice any change at that time as we are already using our own generator power.

Above: The calm night before...

Today is the big day. Seattle City Light is relocating the power feed for our campus as part of the road construction ongoing on Tukwila International Blvd. We expect to be off the grid for several hours. In preparation we have brought in secondary backup power generators, as we always want to have at least two sources of electricity at all times. We expect to run the entire duration on our primary generator system, but have the secondary system in place. In the above photo you can see several "roll up" generators in place last night. We'll explain all of their purposes when we have time today. Check back often to read of status reports and more photos throughout the day Friday, September 4th.

******NON SERVICE IMPACTING MAINTENANCE******

On September 4th, as part of the widening of Tukwila International Boulevard, the electrical service to the Intergate.West campus will be shut down for several hours as it is relocated by Seattle City Light. Our facility will operate on generator power and will be further supported by secondary generator for the duration of the outage. We have taken every precaution available to ensure continuous power delivery to our customers, including pre-testing of secondary generator equipment. We will have a full compliment of staff and vendors on site, and will post updates here on the Support blog throughout the entire day. The following is a projected schedule of day's events:

************

0500 Hours PST - Veca Electric will begin relocating electrical circuits supplying some building common areas (Lobbies, hallways, restrooms, and elevators) to a backup generator to provide power through the duration of the maintenance. Anyone on site during this period may notice some dark areas in the building and a reduction to one operational elevator as the circuits are moved.

0600 Hours PST - Veca Electric will power the moved circuits with the backup generator and ensure life safety equipment, stairwell lights, and restroom lights are all functioning normally. digital.forest staff will prepare our secondary generator for standby for the duration of the day.

0700 Hours PST - digital.forest staff will preflight our main generator and verify all equipment is ready and operational for the day.

0730 Hours PST - digital.forest staff will start our main generator and begin run-up prior to transferring the data center load

0800 Hours PST - digtal.forest staff will manually transfer the data center load to the primary generator

0815 Hours PST - digtal.forest staff will notify project coordinator that load is successfully transfered and that we are ready for Seattle City Light to turn service off for the maintenance.

0815 to 1600 Hours PST - digital.forest will monitor generator status for signs of impending service degradation or failure and to initiate transfer to secondary generator if necessary.

Between 1400 and 1600 Hours PST - Seattle City Light will notify that maintenance has been completed and reinstate grid power to the campus.

30 minutes following stable grid power notification - digtal.forest staff will transfer data center load back to grid power and complete post flight procedures on the primary generator. Veca Electric will relocate electrical circuits supplying the building common areas (Lobbies, hallways, restrooms, and elevators) to their permanent panel positions to provide power from the grid. Anyone on site during this period may notice some dark areas in the building and a reduction to one operational elevator as the circuits are moved.

Following this maintenance we will post a notice here indicating that the data center is operating on grid power and that the maintenance is completed.

************

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******NON-SERVICE IMPACTING GRID POWER MAINTENANCE******

On Friday, September 4th Seattle City Light will relocate a power pole that is part of the grid feeding our facility. We will provide power to the facility using our primary generator while grid power to our facility is down. The expected duration for this maintenance is approximately 10 hours.

For the duration of this maintenance our datacenter electrical load will be transferred to generator power. We will have a second generator of the same size directly attached to our bus-tie panel and capable of providing power in the same way as our primary generator.

******************

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******NON-SERVICE IMPACTING UPS SYSTEM MAINTENANCE******
On Tuesday, June 23rd starting at 07:00 hrs PST and ending at 18:30 hrs PST, we will perform maintenance on our UPS systems. This maintenance will include adding a graphic interface to UPS's 1 and 2 and adding additional monitoring devices to all UPS equipment.

******************

For the duration of this maintenance our datacenter electrical load will be transferred to generator power.

This maintenance window has been selected based on factors to provide the greatest degree of protection for our clients. These factors include the availability of the most senior technicians from our UPS vendor and the ability to mobilize parts and additional service personnel in the very unlikely event of a component malfunction or failure during the maintenance. Additionally, by servicing both UPS systems during the same maintenance window, we eliminate an additional operation of our maintenance bypass switch thus limiting exposure to a potential voltage drop to datacenter critical equipment.

The schedule for this maintenance is as follows:

07:00 to 07:30 Generator pre-flight evaluation: Check fluids, connections, air inlets/exhaust, fuel supply, fuel lines, filters and separators. Log results and announce startup to internal staff.

07:30 Generator start-up and warm-up: Start the generator and monitor performance stats, check for fluid leaks, supply artificial load and evaluate voltage, amperage, frequency and engine stats against established baselines.

08:00 Transfer from grid power to generator power and wrap power around UPS: Manually operate the ATS and UPS maintenance bypass to force the datacenter electrical load to the generator. At this time our HVAC system will power cycle as described above.

08:00 to 10:30 Add additional monitoring equipment to all UPS systems.

10:00 to 17:00 Add graphic interface to UPS's 1 and 2

17:30 to 18:00 Power on UPS system and perform artificial load testing: Following the completion of work, the UPS systems will be powered on and connected to an artificial load (not datacenter equipment, servers or other infrastructure) for testing. With this equipment we will test the transfer process and mechanism as well as simulate a load of 400% of the UPS system capacity.

18:00 Transfer datacenter load back to UPS's: Once the process and mechanism has been validated we will transfer the datacenter load back to the UPS's though power will continue to be supplied to the UPS from the back-up generator.

18:15 Transfer Datacenter live load back to grid power: Once all testing has been completed and both UPS systems are online and functioning within specifications, we will transfer the datacenter live load back to grid power.

18:15 to 18:30 Generator Cool-Down and Post-Flight Evaluation: Following the operation of our generator we will perform the same evaluations we performed during the pre-flight as well as allow the generator to cool prior to shutting down.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

Above: The brand new batteries going into UPS1 today.

As posted previously, we are performing some scheduled maintenance on our UPS systems today. Two tasks are scheduled: Performing a battery swap on UPS1, and replacing capacitors on UPS3.

The first task is the battery replacement. UPS batteries should be replaced at regular intervals to ensure continued safe operation and our UPS systems are no exception to that rule. UPS1's batteries were installed in 2005 and are due for retirement.

We will be running the facility on generator power throughout the maintenance interval.

Above: Facilities Manager Kevin Teker performs a preflight check on the Generator.

We will post updates through the day. Up to the minute info can be found on Twitter @digitalforest.

Update 8:25 AM PDT: As of 8:05 AM PDT we are running on generator power.

Update 8:30 AM PDT: As of 0:25 AM PDT UPS1 & 2 are in Bypass Mode. They will be shut down and the old batteries removed over the next few hours.

Above: Battery technicians from the UPS manufacturer prepare to remove the old batteries from UPS1.

Above: Old batteries coming out, new batteries in the background going in soon.

Above: The old batteries are out, now the new ones are going in.

Once the new batteries are installed we'll test UPS1 under an artificial load. We've brought in a load bank for that purpose. It is basically a heat generator, taking in electricity and outputting heat via elements. It will be stationed outside the building and cabled to the UPS system. The UPS will be brought online, and this load added to it to ensure that it works properly before we transfer the live datacenter load onto it. The load bank itself can be seen here:

Above: The portable load bank.

Our generator has a built-in load bank as well, though it is much larger than this unit since it has to simulate the entire facility (datacenter, HVAC, etc) load whereas this portable unit only has to simulate the full load of one UPS system. We expect to be testing the UPS under this artificial load in the early afternoon. Stay tuned for updates.

Above: Facilities Manager Kevin Teker checks the new batteries with the Fluke Multimeter. They all tested out perfectly at around 12.6 V.

Here is a quick movie taken in the Generator Room. The little microphone on the digital camera cannot truly capture how teeth-rattlingly loud it is in here:

That is a 16 Cylinder, twin-turbo, 1850 horsepower, 50.3 liter Diesel engine.

The new batteries are now completely installed in UPS1. We're setting up the portable load bank to test the UPS under load.

Above: UPS1, all the new batteries are installed. Testing begins soon.

Update 12:30 PM PDT: Transfer tests have been completed successfully under artificial load. Next step is to prepare for live load transfer.

Above: As a safety precaution while the UPS technicians attach the load directly to the bypass switch's bus bars they open the UPS system's main breaker and lock it out.

Above: Preparing the load bank for artificial load testing.

Update 12:45 PM PDT: UPS1 (and UPS2) are back online and carrying live datacenter load. The technicians will enjoy a well-deserved lunch break and we'll commence on the UPS3 maintenance afterwards.

Above: Maintenance complete! UPS1 is back online.

The work scheduled for UPS3 involves replacing some capacitors. The UPS manufacturer has recalled these as similar units in other facilities have presented some issues. Since uptime is very important to us, we have elected to have this replacement performed as soon as possible. The sequence of events will closely mirror the maintenance this morning: The UPS will be bypassed, and then shut down. The technicians will open up the unit itself and remove the old capacitors, replace it with new ones, and then reassemble and test. We will conduct a simulated load transfer test with the artificial load bank. Once convinced that the UPS is operating properly we'll bring it back online.

When all the maintenance is complete we will transfer power back to the electrical grid and after a cool-down period the generator will shut down and be placed back into automatic startup mode.

Above: Maintenance on UPS3 begins. New capacitors are in the boxes at the bottom left of the photo. The technicians are removing cables in order to extract the old capacitors. Note: the blurry techs are a result of shooting the photographs without a flash. I learned a long time ago that UPS technicians do not appreciate flash photography when they're working around potentially high voltages. I'm happy to produce blurry pictures in exchange for their confidence and comfort! The technicians pictured are Steve Cretsinger (left) and Craig Kraft (right) from MGE.

Update 1:50 PM PDT: We've shut off the clean agent fire suppression system in DC1 and the UPS room. The new batteries shipped with a coating of dielectric grease on their terminals and we've noted that it is cooking off under the UPS' ~165,000 watt load. Just to be on the safe side we've disabled the system to prevent a false detection and discharge. The UPS manufacturer feels that it should be clear in a few hours.

Meanwhile the UPS3 maintenance continues.

Above: The old capacitors from UPS3 lie in the foreground as the technicians prepare to install the new ones.

Update 3:00 PM PDT: New capacitors are installed. The UPS is being reassembled. Our ETA for completion is now about 4:00 PM PDT.

Above: The technicians wrap up the installation of the new capacitors.

Above: UPS3 maintenance is finished. ETA for bringing it back online is about 3:50 PM PDT.

Update 3:45 PM PDT: UPS3 is now back online. We'll transfer back to the electrical grid momentarily.

Update 4:00 PM PDT: All Scheduled Maintenance activities are completed, digital.forest's datacenter has resumed normal operations.

******SERVICE IMPACTING UPS SYSTEM MAINTENANCE******

On Thursday, March 19th starting at 07:00 hrs PST and ending at 18:30 hrs PST, we will perform maintenance on our UPS systems. This maintenance will include replacing the batteries in UPS 1 and upgrading the capacitors in UPS 3.

******Impacted Services******

Our FileMaker 3, 4, 5, and 6, Lasso 3, 4, 5, and 6, and legacy Mac shared web hosting environments will be taken offline for 30 minutes in the morning at approximately 08:00 hrs PST and again for 30 minutes at approximately 18:00 hrs PST.

******************
For the duration of this maintenance our datacenter electrical load will be transferred to generator power.

This combined maintenance window has been selected based on factors to provide the greatest degree of protection for our clients. These factors include the availability of the most senior technicians from our UPS vendor and the ability to mobilize parts and additional service personnel in the very unlikely event of a component malfunction or failure during the maintenance. Additionally, by servicing both UPS systems during the same maintenance window, we eliminate an additional operation of our maintenance bypass switch thus limiting exposure to a potential voltage drop to datacenter critical equipment.

The schedule for this maintenance is as follows:

07:00 Datacenter Temperature Reduction: We will force the temperature of the datacenter toward the lower portion of the ASHRAE allowable envelope prior to transferring power to the back-up generator. When the transfer takes place our HVAC systems will power-cycle causing a small (2 to 4 degree Fahrenheit) thermal inclination in the datacenter. By lowering the overall space temperature we can be assured that equipment temperatures will not be adversely impacted by the 10-minute restart period of the HVAC systems.

07:00 to 07:30 Generator pre-flight evaluation: Check fluids, connections, air inlets/exhaust, fuel supply, fuel lines, filters and separators. Log results and announce startup to internal staff.

07:30 Generator start-up and warm-up: Start the generator and monitor performance stats, check for fluid leaks, supply artificial load and evaluate voltage, amperage, frequency and engine stats against established baselines.

07:45 FileMaker 3, 4, 5, and 6, Lasso 3, 4, 5, and 6, and legacy Mac shared web hosting environment shutdown: We will begin shutting down the shared hosting environment at this time. The shutdown will be complete by 08:00 hrs and server environment restarts will begin shortly after 08:00 hrs.

08:00 Transfer from grid power to generator power and wrap power around UPS: Manually operate the ATS and UPS maintenance bypass to force the datacenter electrical load to the generator. At this time our HVAC system will power cycle as described above.

08:00 to 12:00 Replace batteries in UPS 1: The UPS system must be completely powered off for life safety while the batteries are removed and replaced. During this operation UPS 3 will remain online and provide UPS power to clients with A+B power.

12:00 to 12:30 Power on UPS system and perform artificial load testing: Following the removal and replacement of components, the UPS systems will be powered on and connected to an artificial load (not datacenter equipment, servers or other infrastructure) for testing. With this equipment we will test the transfer process and mechanism as well as simulate a load of 80% of the UPS system capacity.

13:00 Transfer datacenter load back to UPS 1: Once the process and mechanism has been validated we will transfer the datacenter load back to UPS 1 though power will continue to be supplied to the UPS from the back-up generator.

13:30 to 17:30 Replace capacitors in UPS 3: The UPS system must be completely powered off for life safety while the capacitors are removed and replaced. During this operation UPS 1 will remain online and provide UPS power to clients with A+B power

17:30 to 18:00 Power on UPS 3 and transfer datacenter load: Once the capacitors have been replaced and charged we will transfer the datacenter load back onto UPS 3.

17:45 FileMaker 3, 4, 5, and 6, Lasso 3, 4, 5, and 6, and legacy Mac shared web hosting environment shutdown: We will begin shutting down the shared hosting environment at this time. The shutdown will be complete by 18:00 hrs and server environment restarts will begin shortly after 18:00 hrs

18:00 Transfer Datacenter live load back to grid power: Once all testing has been completed and both UPS systems are online and functioning within specifications, we will transfer the datacenter live load back to grid power.

18:00 to 18:30 Generator Cool-Down and Post-Flight Evaluation: Following the operation of our generator we will perform the same evaluations we performed during the pre-flight as well as allow the generator to cool prior to shutting down.

One to two weeks following this maintenance, we will perform a battery preventive maintenance on UPS system 1 to validate the condition of the new batteries and replace any questionable jars. This preventive maintenance will follow the same procedure as above but require significantly less time to complete. A notice will be posted to our support blog prior to this future maintenance.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

******NON-SERVICE IMPACTING MAINTENANCE******

On Tuesday, February 24th at 09:00 hrs PST, our local fire department will be onsite conducting an inspection of the fire alarm and double pre-action system in our newest datacenter, DC 3. During this inspection the fire alarm system for that space will be placed in standby and the automatic fire suppression system deactivated. Fire department personnel will be conducting the tests and monitoring the space for any fire hazards. Our alarm strobes, sirens, and bells will all operate at different times and one of our HVAC systems will power cycle during the test.

At no time will there be any client impact or temperature inclination outside of the ASHRAE allowable envelope.

If you have any questions or concerns, please contact your account manager directly. Our account management staff is available Monday through Friday from 08:00 to 17:00 hrs PST and can be reached at 877-720-0483 Option 2.

On Tuesday, February 10th, starting at 07:00 hrs and ending at approximately 14:00 hrs, we will be conducting our semi-annual fire alarm inspection and test. During the test bells, alarms and strobes will be operated to verify functionality and the various components of the fire suppression system will be inspected.

The fire system control panel will be placed in bypass by the vendor during this inspection but all manual pull stations will remain active. Monitoring will be performed by both d.f and vendor staff during the inspection.

This entry will be updated once the inspection has been concluded.

We'll be providing updates within this blog post today concerning our scheduled UPS Maintenance & Upgrade today. Check back for details as the work progresses.

Switch to Generator Power: Around 10 am PDT the electricians and UPS techs were ready to shut down the UPS systems for their work and Kevin Teker, our Facilities Manager cut us over to generator power. The transition went smoothly. The UPS systems were put into bypass mode and then shut down. We expect this work to take a few hours. Updates will be posted as work progresses.

A bit of background...
One of the procedures we are having performed today is a capacity upgrade on one of our UPS systems. It is an MGE EPS 7000 which is capable of this neat trick. With a software upgrade and the addition of a some batteries it can scale from 300kVA to 500kVA. We took delivery of the additional batteries last week. As we've noted on previous posts here due to some unusual circumstances we move UPS gear into the facility by crane.

Above: Preparing the cabinet for flight through the roof.

Above: A view from inside the UPS room looking out (and up).

Above: The new cabinet landed, safe and sound.

We've done this enough times that it has now become rather routine. The crane arrive before dawn, and within an hour the work was done and the roof re-sealed.

Update, Noon PDT: Work continues apace. The UPS maintenance is about halfway finished. The electricians are wiring up the new battery cabinet on UPS 3.

Above: Electricians from VECA installing some conduit for the new cabinet.


Above: Technicians from ECS checking the batteries on UPS1.

The only thing out of the ordinary so far is the relative quiet of the room with all the equipment there turned off.

Update, 1 PM PDT: The UPS maintenance on UPS 1 & 2 is complete. Those systems will come back online momentarily.

Please see this post for details on the incident involving UPS 1 & 2's turn-up.

Update, 3 PM PDT:UPS 1 & 2 remain offline. The facility remains on generator power. Work is continuing on UPS 3.

Update, 4:40 PM PDT: The battery cabinet installation to UPS 3 is complete. The upgrade process is now underway. We plan to have it back online after 6 PM. It is on a different static switch than UPS 1 & 2.

Update, 6:15 PM PDT: UPS 3 came online flawlessly.

We are scheduled to bring UPS 1 & 2 back online in a couple of hours. Our UPS vendor and electricians are pursuing a plan to minimize the risk of a repeat of the earlier event as much as possible. Check back for details.

Update 7:45 PM: We are planning to bring UPS 1 & 2 back online in about 5 minutes.

Update 9:10 PM: UPS 1 & 2 failed to transfer. After two attempts, and long conversations with the UPS manufacturer we have decided to keep them offline for the night. We will continue to run the facility on generator power until this UPS issue can be resolved. We have moved what loads we could onto UPS 3 which successfully completed its upgrade and has spare capacity.

Preliminary analysis suggests that at least one large capacitor in UPS 2 has likely failed. MGE will have technicians here in the morning to investigate and remedy this situation ASAP.

As we posted last week we will be performing our semiannual UPS maintenance tomorrow, Wednesday, October 22nd. Additionally we will be performing an upgrade on UPS 3. This involves software on the unit itself, along with the installation of an additional battery cabinet. The new battery cabinet arrived late last week (pictures coming soon) and was lifted into position.

As is normal during such maintenance intervals the UPS systems will be in bypass and the facility will be running on generator power. There should be no interruption of service during this procedure.

We'll post updates when the work starts and updates as it progresses.

On Wednesday, October 22nd we will be performing scheduled maintenance on our UPS Systems. UPS 1 & 2 will be undergoing their biannual preventative maintenance. UPS 3 will be undergoing an upgrade, which involves the installation of an additional battery cabinet.

As is normal during such maintenance intervals the UPS systems will be in bypass and the facility will be running on generator power.

We will post reminders as the date approaches, and updates as the maintenance progresses.

As summer draws to a close we're already preparing for autumn here at digital.forest. Part of that is topping off our fuel tank for the backup power system. Our generator runs on Diesel fuel from a 5000 gallon tank. This is enough fuel to keep us running for over four days at full load. While we don't see storms such as the hurricanes that have caused such problems in the southeastern US, we have had the occasional stormy day and the traditional season for this kind of weather here in the Seattle area is October through early December. Our policy is to always start the autumn with a full tank. Our monitoring systems told us we were mostly full already, having only used a few hundred gallons over the summer during our monthly generator maintenance & exercise runs. The first step is picking a supplier for delivery. Diesel fuel is a commodity and we maintain delivery contracts with several suppliers. When you buy a commodity fuel in a non-emergency you buy on price. Sort of like when you are buying fuel for your car, you pick the station based on the posted price rather than what is first available. Our Facilities Manager, Kevin Teker places calls to check prices and schedules a truck based on the best buy of the day.

Our generator uses non-road taxed Diesel fuel, which is used in power generation, marine, and agricultural applications. This fuel has a bright red dye added to it to distinguish it from road-taxed Diesel. Ultra-Low Sulphur Diesel (ULSD) for road use is also dyed, with a bright green tint, while also being more expensive as it is taxed at the retail level. BioDiesel remains its natural color and appears, as you would expect as a honey-colored liquid. This short lesson in fuel types is here because it is important to us that the right fuel goes into our tank:

Above: Kevin Teker performs the "Bucket Test" before taking delivery.

The next step happens when the truck arrives. We take a fuel sample from the bottom of the truck's tank and put it into a light-colored bucket. This allows us to check its type as well as quality in several ways, looking for contaminations in the form of water, dirt, etc. Diesel fuel is less refined than gasoline, being barely beyond crude oil, so it is subject to contamination a bit more than gasoline. While we have excellent and flexible filtration systems it is always best to start with the best quality fuel available.

Once accepted the truck will hook up to the tank and fill it up. A sensor tells us when to stop pumping.

Above: Kevin supervises the fill up.

Today we took on only about 450 gallons. It is just good insurance however and we feel ready for the change of seasons.

The UPS maintenance originally announced for April 25th has been rescheduled to Wednesday, April 30th

In other maintenance news, we took advantage of some cold early morning temperatures to do some coil cleaning and belt changes on our original HVAC unit, known as 'AC1'. Yesterday we cleaned the interior condenser coils, and this morning we cleaned the exterior evaporator coils and changed some belts. Keeping these coils clean is a vital part of maintaining our HVAC systems. Clean coils are efficient coils. The condenser coils can be cleaned without interruption of the system's operations. The evaporator coils required shutting down the AC1 for about 5 minutes at a time. Since we have three main AC units and one reserve unit, there was negligible operational impact on the datacenter. Belts are in constant duty cycle, much like in your car, and require changing at regular intervals.

Below: A view of AC1's interior condenser coils in the middle of their "Spring Cleaning." Usually these are not visible, as they are hidden behind doors.

On April 25th, 2008 we will be performing an upgrade on portions of our UPS system. These will bring these units up to a similar spec to our new UPS.

Anytime we perform UPS maintenance we put the UPS system in bypass, meaning that we run the facility on our own generator rather than the electrical grid. Anytime there is a transfer of power from one source to another, there is a risk of interruption. The risk is very small, and we will take every measure to minimize and eliminate that risk. Briefly, here is the process:

* Manually start the generator.

* Manually transfer the electrical load from grid to generator.
(The interruption here is measured in milliseconds, unlike a power failure, which usually lasts 7-10 seconds as the generator starts.)

* Manually put the UPS system in bypass mode.
(This routes power directly from the main power panel through to the servers, removing the UPS from the flow. This is a two-step process where the bypass bus is energized before the UPS is bypassed. This means there is no interruption.)

* Perform the required maintenance.
(This will take several hours.)

* Put the UPS system back on-line.

* Manually transfer from generator to grid power.

* Shut down the generator after a cool-down period.

We perform this procedure several times a year, so this is fairly routine. We also have over a week's supply of fuel for the generator, so runtime is not an issue. We just like to keep our clients informed of happenings here at the facility. As per usual we'll document the process here on the support blog so you can see what we're up to. We'll post reminders as the maintenance day approaches.

Earlier this month we had our backup generator's fuel quality tested and we found some algae in there. Given the size of the tank (5000 gallons/19,000 liters) and the wide temperature variations we've seen over the past year, it didn't really surprise us. Our previous test did not show any, but the recent one had enough to make us worry, especially as we're going into the critical months of the year for backup power availability. There must have been a recent "bloom" of the algae. We scheduled our friends at Fuel Care to come out and remove the contamination ASAP. They arrived on Friday.

The process involves parking their truck next to our fuel tank and cycling the fuel out of the tank and through special filters in the truck to remove the algal contamination. While that happens they scrub the tank walls and generally inspect the tank's interior while the access cover is off. Please note that we can still operate the generator if needed while this process is happening, as the fuel is constantly cycled through Fuel Care's filter system and no more than a few hundred gallons are out of our tank at a time. In many ways it is akin to kidney dialysis, but for Diesel fuel.

Thankfully the tank itself was very clean and appeared in great shape. We removed about a gallon and a half of water (5 liters) accumulated via condensation and all the algae and now our fuel supply is sparkling clean.

The whole point of this process is to ensure the smooth operation of our backup power supply. You rely upon us to maintain our facility to the highest standard and ensure the maximum uptime for your servers. We just like to share all the things we do with that in mind so you stay informed. We'll have more news and info about other steps we're taking in that direction soon.

Regards,
Chuck Goolsbee
VP Technical Operations
digital.forest

We successfully completed our UPS upgrade work today. We were able to transfer to generator power, bypass the UPS, shut down the UPS, perform the work, the reverse the whole process and restore grid power. Everything worked as planned and as expected. We spent approximately 2.3 hours on backup power.

Above you can see the UPS Technician from MGE performing some circuit board replacements and firmware upgrades on our bypass switch.

Below you can see a gauge that shows the output from our backup power generator system. Running our entire facility's load, which includes all the servers and all of our mechanical systems (air conditioners, humidity controls, fans, etc) uses only 22% of our backup power system's capacity. Lots of room to grow.

Stay tuned for more news about our datacenter facility expansion.

Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

Between the hours of 9am and 12pm, Thursday, September 20th we will be performing maintenance on our UPS system, which will include upgrading some control boards and updating to the latest firmware. This is the final step in our recent UPS upgrade.

While this maintenance is being performed we will put the UPS system and grid power in bypass and operate the datacenter on generator power. This is standard procedure for work such as this.

While we do not expect any change in power delivery, any time power is switched from one source to another, a risk of interruption is possible. Every effort and precaution will be taken to minimize this risk.

Thank You,

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

The Seattle area experienced a bit of a heat wave yesterday. We realize that the rest of the country has seen temperatures even higher, but it is rare for our cool marine & mountain location to see things get as hot as they did in the past few days. The peak came yesterday afternoon, around 5 pm, when we hit 98°F/37°C outside. I'm happy to say that our datacenter's peak temperature was 74°F/23°C.

Today we have high overcast, and expected highs in the low-80s. Datacenter temps should remain in the mid-high 60s.

We've been making continual improvements to our HVAC system. Adding and making modifications to existing duct work mostly. We've boosted our cooling capacity by about 20% in the past few weeks. As we've filled our facility we have been adapting the generic air diffusers with custom duct work. The aim of which is to direct cool air into specific areas, namely the front of the racks. Servers are generally designed to cool front to back. Visitors to our facility will note that we arrange rows of servers with the backs of servers facing into alternate rows. This creates "hot aisles" and "cold aisles", a standard industry practice in datacenters. Cold air from the HVAC system falls down from just above, is pulled into the servers, and hot exhaust air rises up to the top of the room where it is pulled out by the HVAC system's hot air return system. What we have been doing the past several months in preparation for summer is focussing the cold air. This involves custom duct work.

Above: A shipment of our custom ducting arrives.

Above: Custom ducting awaits installation in the datacenter.

Above: A custom duct installed in a client's cage. Note the room for their planned expansion. Note also the unmodified generic air diffuser to the left. It is from these that we mount the custom ducts.

For our general rackspace datacenter we have taken the generic diffusers and focused the new ducts down and into the cold aisles. In the cage space datacenter, we have extended custom ducts directly to the front of customer's servers. This work allows us to better control airflow and maintain the optimum air temps right where it matters. Later this year we'll be adding similar duct work, this time above the hot aisles, specifically to remove hot air and bring it back to the return of the HVAC systems.

Above: The business end of our auxiliary HVAC system.

We have also added a large custom diffuser with dampers in 14" conduit heads on our auxiliary HVAC system. Throughout most of the year, this unit does not even run. In warmer conditions we can open a subset of the dampers and provide supplementary cooling in a very general fashion to the entire facility. On very hot days like yesterday we open them all and provide the extra HVAC tonnage to keep the facility cool. In emergency situations such as last week we can attach temporary lightweight ducts to the 14" heads and run them through the ladder rack to the specific areas of the datacenter that need it.

Stay tuned for some more exciting news very soon about expanded capabilities and additional layers of redundancy!

Chuck Goolsbee
VP, Technical Operations
digital.forest

Last night around 19:30, one of the three primary compressors of one of our two HVAC systems failed. This was noted by NOC personnel who contacted our HVAC contractor who set the controls to bypass the failed unit. We did run on single-stage cooling from one unit for several hours, which lead to both datacenters reaching temperatures in the low-80s F/high-20s C. Once the unit was bypassed 3-stage cooling was achieved and temps dropped to their normal mid-60s F/high-teens C. With outside summertime temperatures in the high-80s F/low-30s C possible we require 4-stage cooling at peak hours.

We will be replacing the failed compressor tonight between 21:00 and 23:00. We have secured portable HVAC units to supplement our secondary HVAC while the primary unit is down for repairs. We will update this website with more information as the repair progresses.

Update: 8:00 AM The new compressor was installed last night. We've resumed normal operations.

Regards,
--Chuck Goolsbee
VP, Tech Ops
digital.forest

We have both our HVAC and Controls Systems vendors on site right now performing an investigation of our issue over the weekend. They will be shutting off the systems for short periods of time in order to perform diagnosis of various system components. Datacenter temperatures may rise as a result. Recover times should be brief however, as the outside temperature here in Seattle is 30F/17C cooler than it was on Sunday with cloudy skies, showers and 53F.

Springtime brings sunny weather and cleanup tasks! Our backup power generator comes equipped with a 5000 gallon, double walled fuel tank. We can run for almost a week on our own power source with this capacity. While fairly mundane in appearance (beyond of course being HUGE) it actually is a fairly high-tech bit of equipment. It has sensors that provide us feedback concerning fuel quality, quantity, and rate of consumption while the generator is running. It also provides data on water percentage in the fuel. Diesel fuel is basically oil, and oil and water as we all learned in grade school science class, do not mix. Large metal containers condense water from the atmosphere, both inside and out due to normal temperature and humidity changes. Diesel engines all have water separation equipment built into their fuel systems. Rudolf Diesel's invention is a compression engine meaning that unlike a gasoline engine it does not use spark plugs to ignite a fuel/air mixture; instead it compresses the mixture at such a high ratio that it heats up to an ignition point on its own. Unlike a fuel/air mixture, water does not lend itself to compression, so it must be removed from the fuel prior to injection into the compression cylinders.

The fuel monitoring system for our storage tank had an alarm go off last week. It signaled a interstitial leak. This indicates that there could be a possible leak of the main tank into the interior space between it and the outer tank.

Kevin Teker, our Facilities Manager responded by going into the interstitial space with testing equipment to see what was setting off the fluid sensor.

Above: The fluid sensor removed from the fuel tank.

Above: The high-tech method of getting the chemical testing paste to the lower reaches of the tank... a tape ruler.

Using special pastes which change color based on contact with either water or fuel, Kevin was able to determine that the "leak" was not in fact a leak, but instead water that had condensed within the interstitial space between the two tanks. Once satisfied that our primary tank's integrity was sound the next step was to remove the water. How does one drain less than a pint of water from a tank that weighs 50,000 pounds and only has openings at the top?

We solved that problem by lowering absorbent materials attached to that high-tech reach tool pictured above. Once the water was removed, we replaced the sensor and it indicated a dry interstitial space once again.

This is just one small example of the little things we do every day to maintain our facility. Every colocation provider has a facility to maintain, and we know that you choose digital.forest for our excellent service AND our facility. I hope that you see and recognize both every day, in the form of your servers' uptime, and when you have interaction with the Tech Support and Customer Service staff. The facility though is in many ways invisible to most of our clients, so it is nice to highlight things when they present themselves and allow you a glimpse into daily operations.

Regards,
Chuck Goolsbee
VP, Technical Operations
digital.forest

4. Wrap up & Summary

We're pleased to report that the repair on our HVAC system is complete, and finished without incident. The final bit of work required brazing & welding within the unit itself. To mitigate any risk of having the pre-action fire suppression system discharging its gasses, we had our vendor Fire Chief, come out and disable the system. Part of our annual maintenance procedure for the fire suppression system involves the shut down of the HVAC system anyway, so Fire Chief took advantage of the situation to perform that maintenance.

Above: Technicians from Fire Chief perform preventative maintenance on the Fire Detection and Suppression system.

During the HVAC system shutdown, digital.forest staff monitored temperatures in various locations around the datacenter, while our Facilities Manager bounced between the roof and the datacenter monitoring our vendors. Below you can see digital.forest Tech Support member Will Winslow and Facilities Manager Kevin Teker in the darkened datacenter just after the HVAC shutdown occurred. They're carrying their temperature monitors and about to spread out to their stations. You can see the high-CFM fans mentioned earlier today in the open door behind them.

All of our preparation paid off, plus a bit of luck from the weather (it stayed very cool, plus it didn't rain) so that the natural tendency for the facility to warm up was mitigated by the combination of pre-cooling and the fans pulling outside air into the facility. We're happy to report that our highest temperature reached was about what we see here on a "normal" day. Our temporary portable HVAC units never even needed to be turned on.

Interesting conclusions:
Electrical capacity is a hot topic in the datacenter management business these days. There are various rules of thumb concerning the estimataion of power usage split between "floor" (meaning the servers) and "mechanical" (meaning the HVAC systems to cool the servers.) The variable is the delta between outside and inside ambient temperatures. The hotter it is outside, the harder the HVAC systems have to work to chill the inside. We're blessed to be located in a very moderate climate here in Seattle. It rarely gets very hot here. Nor does it get very cold. Our average temperature is actually quite a bit lower than ideal datacenter temperature. Even in summer, it cools enough at night to keep our average right at ideal datacenter temperature. We monitor electricity usage at several points, along the flow for a lot of reasons, but on our main panel in the datacenter we can check at a glance and see how much power is being used in total. The ammeter for example read this way earlier today when we were running the rooftop HVAC and 100% using outside air:

That reads 274 Amps. That is 274 Amps of 3-phase power as it comes in off the grid. Our feed is 2000 Amps so as you can see we have a lot of room for growth with regards to electricity. This is one of the things that really attracted us to this facility when we moved here just over two years ago. With so many datacenter operations running at nearly 100% of their power capacity we felt it important to be able to accommodate our clients expanding needs and requirements. This maintenance interval provided us some real-time data concerning the power needs of our mechanical infrastructure. Those rules of thumb mentioned earlier say "for every 1 amp you feed the floor, you feed the mechanical 1 to 1.75 amps." This seems to have been proven in our experience, but rounded down due to our temperate, if not downright cool location here in Seattle. Here is a shot of the ammeter with the HVAC system shut down completely:

That is 219 Amps of 3-phase power. Looking at our monitoring history, we hit our maximum of 400 Amps last July when we had a week of temperatures in the 90-95° F (32-35°C) range. That means we are running at a roughly 1:1 floor:mechanical ratio in terms of electricity at our peak consumption. If anything we are favoring the floor, which is a great advantage in this industry.

Yet another benefit of colocation at digital.forest in cool Seattle!

3. Shut Down Interval.

At 12:07 the entire HVAC system was shut down. Datacenter temps are well within reasonable tolerances after 20 minutes on fans alone. We'll update again with more information after the HVAC is retuned to service.

Update: 12:32 PM PDT
HVAC systems are running again. We'll summarize the day's work soon.

2. Repair Work

Thankfully it has remained nice and cold outside today so our HVAC system, which is designed to use outside cool air if available to reduce compressor load, is running 100% on outside air. This allowed us to continue to run the HVAC while the technicians remove the old compressor and install the new one. So from the perspective of the datacenter things appear no different than a normal day here at digital.forest. All the action is happening up on the roof:

In the top image above the techs wrestle the new compressor up a temporary ramp and into place. In the bottom shot you can see the new compressor in place, and the old broken one on the handtruck, ready to be removed.

The Trane Intellipak is an excellent HVAC system that has a myriad of control options. Below you can catch a glimpse into the heart of the controls, which are usually locked behind a steel panel. We usually interface with these systems via software down in the office, but occasionally it is good to have a look at the atoms represented by the bits.

Above is a close up of the breakers and control units for the compressors. You can see that several breakers are in the "off" position, providing safety for the technicians while they work. Others remain "on" so that the system can still function and provide air handling for the datacenter.

Above: digital.forest Facilities Manager Kevin Teker explains how all of this works.

The next step requires the complete shutdown of the HVAC and Fire Suppression Systems, as the HVAC technicians braze some plumbing. Stay tuned.

Last week we had a single compressor unit in our Trane Intellipak cooling system fail during an unseasonably warm day. The system has built-in redundancies to handle such situations so we recovered quickly from the condition. In order to prepare for the warmer weather coming soon, we have elected to replace this failed unit now. So tomorrow (Wednesday, March 14) we will have a vendor here replacing the compressor. This will involve occasional, brief shutdowns of our HVAC system.

We have brought in industrial sized high-CFM fans, to maintain air circulation in the facility during the maintenance. Additionally we have several portable 1-ton HVAC systems which we can deploy on an as-needed basis should any areas of our datacenter exceed standard temperatures. We have deployed temperature probes throughout the datacenter to monitor this as the maintenance progresses. As such we are confident that this event will have minimal-to-no impact on operations, since we will be prepared to mitigate any heat issues should we see temperatures rise.

We apologize for the short notice, and we hope you understand the reasons why. We strive to maintain our facility to the highest standard, as well as keep you informed as we take steps to do so. We will post updates throughout the day tomorrow.

Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.