digital.forest Technical Support
Update on our UPS situation

A briefing for you on the current situation regarding our power event yesterday:

* UPS 1 & 2 remain offline this morning, and the facility remains on generator power.

* UPS 3 is online and functioning normally. We migrated what circuits we could to this UPS system last night between 7 and 10 PM.

* We have enough fuel on-site for over four days of run time, but to be safe we have scheduled fuel deliveries for the next several days to ensure supply.

* We are performing our own hourly generator checks around the clock to ensure proper operation.

* Our generator maintenance vendor is also scheduled to arrive daily and inspect the operation of the system.

* Until we have a fully functioning UPS system we can not risk a power transfer back to the grid. We believe the fault is specific to UPS 2. Technicians will investigate further today.

* At 9 o'clock this morning we are meeting with our UPS manufacturer to determine next steps.


Update 11 AM: The UPS manufacturer has dispatched a high-level technician and is airfreighting a complete set of replacement parts for our UPS systems from California right now. ETA is mid-afternoon. We just received a shipment of load banks which are being installed on our roof. Here is the plan:

* Isolate UPS 1 & 2 from our facility input and load.

* Test every subsystem of each UPS, isolate and repair fault(s).

* Test each UPS under artificial load. If they fail, fix. If they pass...

* Sync the UPS' back into their parallel configuration, retest under artificial load. If they fail, fix. If they pass...

* Test again and confirm smooth transfer of artificial load.

* Plan for reinsertion of UPS 1 & 2 into our facility load.

Our initial ETA for full restoration of protected power is sometime after midnight tonight. We will update this schedule with a specific time once more details are known. Meanwhile we continue to run the facility on generator power. Even though we have only consumed about 20% of our available fuel we still plan on refueling early this afternoon.

Update 12:25 PM: Fuel delivery truck is here and we're topping up the fuel tank.

Update 1:30 PM: Senior-level technician from the UPS manufacturer has arrived on-site.

Update 3:45 PM: Root Cause Suspect...

This is part of the circuit board of UPS 2 that communicates with the static switch. A replacement board in en route.

We're not completely satisfied however, and continuing to give each part of UPS 2 close examination and test. Stay tuned for more details.

Update 7:15 PM: Both UPS units have been inspected and we're quite certain the item identified above is the only issue to be addressed. The replacement part is arriving at Sea-Tac airport very soon. Meanwhile we are working to test UPS 1 under artificial load. Once the replacement board arrives we will test UPS 2 as well. We should have a specific ETA for turn-up by the time we next update the site.

Informational Update 7:55 PM: We have successfully transferred a 100% artificial load on and off UPS 1 three times. We're certain that UPS 1 is operating properly. Repair on UPS 2 will begin soon.

Update 9:10 PM: Barring any new issues we are on track for a transfer back to grid power between midnight and 1 AM tonight. The replacement part has arrived and we should begin testing UPS 2 very soon.

Update 11:25 PM: Some further issues have been uncovered in testing UPS 2, so we have postponed a transfer back to grid power indefinitely. No transfer will take place tonight.


We will continue to update this post as we learn more information. Thank you for your continued patience and understanding as we deal with this emergency situation.

posted by Chuck G. at 11:26 AM on Thursday, October 23, 2008
Categories: Emergency Maintenance