We transferred back to grid power from our generator at 11:05 PM. The Secondary Generator never saw any use, but it did its job, namely providing us with a proper safety net. Now the task of packing up all the cables and putting everything back into normal operating conditions begins.
Our thanks go out to Sabey, for staying on Seattle City Light to get this maintenance completed tonight.
Thanks as well go to the digital.forest Technical Operations staff who have worked tirelessly over the past several weeks to prepare for this event. The fact that it went seamlessly is a testament to their professionalism.
******21:30 Update******
Grid power has been restored to the campus. We will be transferring back momentarily.
******20:00 Update******
Word from SCL is that grid power may be restored around 10 PM.
******19:00 Update******
Things are running well. SCL continues to do their work. We are still here keeping things running. Here is a quick video we shot showing a regular check of the backup power system as it is running. (Warning: It is very loud in the generator room, put on your ear protection.)
Above: Our building at 5 PM today.
******12:00 (Noon) Update******
The Good News: Everything is running fine.
The Bad News: Seattle City Light's original ETA of "around 3 PM" for completion of the work and restoration of grid power is now "around 6 AM tomorrow."
We are prepared. We have enough fuel on-site to run until Tuesday of next week, though we'll likely top off tanks later today.
Meanwhile Seaport Biofuels just stopped by to top off the tank of the generator running the building elevator and common-area lights:
******10:30 Update******
Above: digital.forest Facilities Manager Kevin Teker points to the fan relay he just replaced.
Why there is really no such thing as a "lights out datacenter" and why do we maintain all of our own facility systems? A short time ago Kevin Teker, our Facilities Manager performed a fan relay swap. On the right-hand side of the image above you see several large cooling fans. They supply air for our backup generator system and are driven from large electric motors. A short time ago they stopped working when their relay failed. Kevin noted this and opened up the box where the relay is located and replaced the failed relay with one of the spares he keeps right in that very box. We love Kevin, and you should too. Why? He always thinks of these things and like a Boy Scout, is always prepared. Had this happened in an unattended "lights out" facility the generator would have slowly overheated and perhaps eventually failed. Instead, because our lights are on and we're always here at digital.forest, the issue was noted, and resolved in a matter of minutes. It is never the big things that go wrong, it is the little things that eventually become big things. We keep an eye on the little things. Not the landlord. Not an outsourced facilities contractor. Not a real estate company. Our full-time staff here at digital.forest is always looking out for you.
Above: Kevin labels everything. The new relay is installed on the right, the new spare, replacing the old spare, is on the left side of the relay box.
Above: Kevin closes up the relay box after he's done with the repair.
******09:45 Update******
Grid power went offline around 09:00. All systems running normally.
Above: The reason for this event: Road construction on International Blvd. Seattle City Light is relocating the power feed for the campus as part of the road widening.
******08:45 Update******
Above: Generators.
The photo shows several generators. The large one on the left is the secondary generator for the digital.forest datacenter. The small green one in the middle is running backup power for the building, including one of the two elevators. The digital.forest offices are powered by our own generator but it is nice that the elevator is working today. In the distance you can see another portable generator in front of Building B. The entire campus will be off the grid during this maintenance event.
Above: More Generators.
The large generator on the left is another view of the secondary generator for our datacenter. The small one on the right is running the block heaters for the secondary generator. This minimizes the startup time for the secondary should we need to bring it online. In the background you can see the exhaust stack from our primary generator.
Above: The view of all the connections in the basement.
This is the scene at the convergence of all this activity in our building. Behind the sound baffles is our primary generator system. To the left of the door is our Bus Tie, where the "portable" generators tie into our power system. On the right is where the building backup generators to run lights, elevators, etc are tied in, as well as the other datacenter facility in our building.
******08:15 Update******
We transfered our facility load to our primary generator at 08:00. All services are operating normally and are expected to continue doing so. Seattle City Light crews will de-energize the campus feed sometime in the next hour or so. We will not notice any change at that time as we are already using our own generator power.
Above: The calm night before...
Today is the big day. Seattle City Light is relocating the power feed for our campus as part of the road construction ongoing on Tukwila International Blvd. We expect to be off the grid for several hours. In preparation we have brought in secondary backup power generators, as we always want to have at least two sources of electricity at all times. We expect to run the entire duration on our primary generator system, but have the secondary system in place. In the above photo you can see several "roll up" generators in place last night. We'll explain all of their purposes when we have time today. Check back often to read of status reports and more photos throughout the day Friday, September 4th.
