Starting a new support blog post on a new day to keep things easy to read.

Status as of 9:00 AM Friday, October 24, 2008:

* Facility remains on generator power. We have enough fuel on-site for a run until Monday.

* UPS 3 is online and functioning properly

* UPS 1 & 2 remain offline. UPS 1 is working properly but can not be brought online until UPS 2 is repaired.

* UPS 2 showed inverter errors whenever it was loaded last night. The spare parts package contained the wrong inverter. The correct one is on its way from California on a commercial flight this morning.

* When it arrives we will install and begin testing procedures again with an eye on going back to grid power sometime tonight.

Update 2:45 PM: Two bits of news...

* We have found a faulty connector inside of UPS 2. The fault is very minor, but it could have contributed to some of the issues we have experienced. We are replacing the whole assembly to be on the safe side. We'll post photos soon.

* We just finished topping off our fuel supply with 1300 gallons of Diesel and now have enough for a continuous generator run through Tuesday afternoon (October 28th).

Update 4:00 PM: Several updates...

* The replacement inverter is installed in UPS 2 and we have begun load testing that unit. If all goes well we will sync UPS 1 & 2 and begin load testing them together.

* The replacement connector component has an ETA of 7:30 PM. Once that arrives we should be able to proceed swiftly.

* We are planning to switch back to grid power this evening between 8 and 9 PM PDT.

* Cummins Northwest, our generator maintenance contractor just finished an inspection of our generator system and proclaimed it in excellent condition.

Update 4:30 PM: UPS 2 is confirmed operational by the MGE tech on-site. We are testing the units synced now.

Based on a client request, the timing for the switch back to grid power has been postponed until 10:00 PM PDT

Update 8:00 PM: All the required parts are here, we are finalizing tests and preparing to perform the switch back to grid power tonight just after 10 PM.

Update 8:35 PM: Here are some photos of the faulty connector. Earlier I stated that it was from the static switch, it is in fact from inside UPS 2. It was just my misunderstanding of what was reported to me, and I've corrected that statement above. This connector contains three large switches, and one of them has a slim gap which is making a poor connection.

Above: The connector, with a socket wrench for scale. The connector in the photo is in an "open" state.

Below: Here is a close up shot with the connector closed. The top red arrow is pointing to a solid connection. The bottom red arrow is pointing to the bad connector portion. Note the shadow cast from the camera's flash. There should be no shadow visible if the connector is tight.

We are certain that this is the root cause of our power event on Wednesday. The connection was loose enough to cause poor conduction, which is why we saw a voltage drop and then falling back to a bypass state. It would also explain the damage to the comm board in UPS 2 we showed you yesterday, and the damage to the inverter. UPS 2 was purchased just over a year ago and should not have failed in this manner. Since UPS 1 is an identical unit we have removed the same connector and inspected it. UPS 1's connector is good and solid. We tested UPS 1's connector in UPS 2 and it works perfectly. The technician from MGE is replacing this connector in UPS 1 right now with a new one, and we'll begin testing both of the units soon.

Update 9:30 PM: We are "a go." Both UPS units pass all tests, several times. Everything is working as it should under load. We are preparing for coming off generator power at just after 10 PM.

Update 10:00 PM: 10:10 PM is our specific target time for bringing the repaired UPS system back online.

We will continue to update this post as we learn more information. Thank you for your continued patience and understanding as we deal with this emergency situation.

Chuck Goolsbee
V.P. Technical Operations
digital.forest, Inc.