Above: The brand new batteries going into UPS1 today.

As posted previously, we are performing some scheduled maintenance on our UPS systems today. Two tasks are scheduled: Performing a battery swap on UPS1, and replacing capacitors on UPS3.

The first task is the battery replacement. UPS batteries should be replaced at regular intervals to ensure continued safe operation and our UPS systems are no exception to that rule. UPS1's batteries were installed in 2005 and are due for retirement.

We will be running the facility on generator power throughout the maintenance interval.

Above: Facilities Manager Kevin Teker performs a preflight check on the Generator.

We will post updates through the day. Up to the minute info can be found on Twitter @digitalforest.

Update 8:25 AM PDT: As of 8:05 AM PDT we are running on generator power.

Update 8:30 AM PDT: As of 0:25 AM PDT UPS1 & 2 are in Bypass Mode. They will be shut down and the old batteries removed over the next few hours.

Above: Battery technicians from the UPS manufacturer prepare to remove the old batteries from UPS1.

Above: Old batteries coming out, new batteries in the background going in soon.

Above: The old batteries are out, now the new ones are going in.

Once the new batteries are installed we'll test UPS1 under an artificial load. We've brought in a load bank for that purpose. It is basically a heat generator, taking in electricity and outputting heat via elements. It will be stationed outside the building and cabled to the UPS system. The UPS will be brought online, and this load added to it to ensure that it works properly before we transfer the live datacenter load onto it. The load bank itself can be seen here:

Above: The portable load bank.

Our generator has a built-in load bank as well, though it is much larger than this unit since it has to simulate the entire facility (datacenter, HVAC, etc) load whereas this portable unit only has to simulate the full load of one UPS system. We expect to be testing the UPS under this artificial load in the early afternoon. Stay tuned for updates.

Above: Facilities Manager Kevin Teker checks the new batteries with the Fluke Multimeter. They all tested out perfectly at around 12.6 V.

Here is a quick movie taken in the Generator Room. The little microphone on the digital camera cannot truly capture how teeth-rattlingly loud it is in here:

That is a 16 Cylinder, twin-turbo, 1850 horsepower, 50.3 liter Diesel engine.

The new batteries are now completely installed in UPS1. We're setting up the portable load bank to test the UPS under load.

Above: UPS1, all the new batteries are installed. Testing begins soon.

Update 12:30 PM PDT: Transfer tests have been completed successfully under artificial load. Next step is to prepare for live load transfer.

Above: As a safety precaution while the UPS technicians attach the load directly to the bypass switch's bus bars they open the UPS system's main breaker and lock it out.

Above: Preparing the load bank for artificial load testing.

Update 12:45 PM PDT: UPS1 (and UPS2) are back online and carrying live datacenter load. The technicians will enjoy a well-deserved lunch break and we'll commence on the UPS3 maintenance afterwards.

Above: Maintenance complete! UPS1 is back online.

The work scheduled for UPS3 involves replacing some capacitors. The UPS manufacturer has recalled these as similar units in other facilities have presented some issues. Since uptime is very important to us, we have elected to have this replacement performed as soon as possible. The sequence of events will closely mirror the maintenance this morning: The UPS will be bypassed, and then shut down. The technicians will open up the unit itself and remove the old capacitors, replace it with new ones, and then reassemble and test. We will conduct a simulated load transfer test with the artificial load bank. Once convinced that the UPS is operating properly we'll bring it back online.

When all the maintenance is complete we will transfer power back to the electrical grid and after a cool-down period the generator will shut down and be placed back into automatic startup mode.

Above: Maintenance on UPS3 begins. New capacitors are in the boxes at the bottom left of the photo. The technicians are removing cables in order to extract the old capacitors. Note: the blurry techs are a result of shooting the photographs without a flash. I learned a long time ago that UPS technicians do not appreciate flash photography when they're working around potentially high voltages. I'm happy to produce blurry pictures in exchange for their confidence and comfort! The technicians pictured are Steve Cretsinger (left) and Craig Kraft (right) from MGE.

Update 1:50 PM PDT: We've shut off the clean agent fire suppression system in DC1 and the UPS room. The new batteries shipped with a coating of dielectric grease on their terminals and we've noted that it is cooking off under the UPS' ~165,000 watt load. Just to be on the safe side we've disabled the system to prevent a false detection and discharge. The UPS manufacturer feels that it should be clear in a few hours.

Meanwhile the UPS3 maintenance continues.

Above: The old capacitors from UPS3 lie in the foreground as the technicians prepare to install the new ones.

Update 3:00 PM PDT: New capacitors are installed. The UPS is being reassembled. Our ETA for completion is now about 4:00 PM PDT.

Above: The technicians wrap up the installation of the new capacitors.

Above: UPS3 maintenance is finished. ETA for bringing it back online is about 3:50 PM PDT.

Update 3:45 PM PDT: UPS3 is now back online. We'll transfer back to the electrical grid momentarily.

Update 4:00 PM PDT: All Scheduled Maintenance activities are completed, digital.forest's datacenter has resumed normal operations.