We'll be providing updates within this blog post today concerning our scheduled UPS Maintenance & Upgrade today. Check back for details as the work progresses.

Switch to Generator Power: Around 10 am PDT the electricians and UPS techs were ready to shut down the UPS systems for their work and Kevin Teker, our Facilities Manager cut us over to generator power. The transition went smoothly. The UPS systems were put into bypass mode and then shut down. We expect this work to take a few hours. Updates will be posted as work progresses.

A bit of background...
One of the procedures we are having performed today is a capacity upgrade on one of our UPS systems. It is an MGE EPS 7000 which is capable of this neat trick. With a software upgrade and the addition of a some batteries it can scale from 300kVA to 500kVA. We took delivery of the additional batteries last week. As we've noted on previous posts here due to some unusual circumstances we move UPS gear into the facility by crane.

Above: Preparing the cabinet for flight through the roof.

Above: A view from inside the UPS room looking out (and up).

Above: The new cabinet landed, safe and sound.

We've done this enough times that it has now become rather routine. The crane arrive before dawn, and within an hour the work was done and the roof re-sealed.

Update, Noon PDT: Work continues apace. The UPS maintenance is about halfway finished. The electricians are wiring up the new battery cabinet on UPS 3.

Above: Electricians from VECA installing some conduit for the new cabinet.


Above: Technicians from ECS checking the batteries on UPS1.

The only thing out of the ordinary so far is the relative quiet of the room with all the equipment there turned off.

Update, 1 PM PDT: The UPS maintenance on UPS 1 & 2 is complete. Those systems will come back online momentarily.

Please see this post for details on the incident involving UPS 1 & 2's turn-up.

Update, 3 PM PDT:UPS 1 & 2 remain offline. The facility remains on generator power. Work is continuing on UPS 3.

Update, 4:40 PM PDT: The battery cabinet installation to UPS 3 is complete. The upgrade process is now underway. We plan to have it back online after 6 PM. It is on a different static switch than UPS 1 & 2.

Update, 6:15 PM PDT: UPS 3 came online flawlessly.

We are scheduled to bring UPS 1 & 2 back online in a couple of hours. Our UPS vendor and electricians are pursuing a plan to minimize the risk of a repeat of the earlier event as much as possible. Check back for details.

Update 7:45 PM: We are planning to bring UPS 1 & 2 back online in about 5 minutes.

Update 9:10 PM: UPS 1 & 2 failed to transfer. After two attempts, and long conversations with the UPS manufacturer we have decided to keep them offline for the night. We will continue to run the facility on generator power until this UPS issue can be resolved. We have moved what loads we could onto UPS 3 which successfully completed its upgrade and has spare capacity.

Preliminary analysis suggests that at least one large capacitor in UPS 2 has likely failed. MGE will have technicians here in the morning to investigate and remedy this situation ASAP.