When Plan A won't work, you fall back on Plan B. What happens if that won't work?
It has been more than three weeks since our last mention of our new UPS. It has been a busy month, with a lot of glancing back into the mists of time and collective head scratching.
We love our building, we really do, but sometimes we wonder what the designers and builders were thinking back in those crazy "dot com boom" days. So first, a short history review: We are located in the largest datacenter campus in North America, Sabey's Intergate.Seattle complex. It was built in the heyday of the datacenter building boom between 1998 and 2001. The original tenants were Zama Networks, Netstream Inc. and Microsoft. Two of those companies vanished in 2001, I bet you can spot which two. We moved here back in early 2005. The building was purpose-built to be a datacenter and in many ways is ideal. The folks who built it had LOTS of money, and the best engineers and architects. The only thing they didn't have was time. Once in a while you look at things and realize how rush jobs can come back to haunt you... even indirectly.
Above: Panorama of our UPS room. On the extreme left is our existing UPS, a 180kW/225KVA MGE EPS6000. To the right of it is our wooden "wire frame" UPS mock-up, and in front of it some tape outlining the location of the battery cabinets. In the middle is the switching and bypass gear with all the conduit above for both the old and new UPS. In the background are our Facility Manager and a tech from our fire suppression systems vendor preparing for today's job. On the extreme right is the stairway and door into the UPS room from the main datacenter.
Our facility is on the top floor of the building and has a specifically-engineered UPS room, with a substantially reinforced floor to support the weight of many UPS'. We can place close to 60,000 lbs of equipment in there. Our new UPS weighs ~4500lbs. The two battery cabinets weigh in at ~3500lbs each. Plan A was to bring them up in the elevator but size and weight contraints, even of the larger of the two lifts here prevented that. Plan B was to crane them up to the deck outside the datacenter and roll them into the facility with industrial rollers and winch them up our purpose-built ramps. This is how the existing units arrived here when the facility was built, minus the ramps of course. Apparently they originally used jacks and it took all day. Our building owner, and their engineers ruled that out for structural reasons. The UPS room floor is more than sturdy enough to support them, but they worried about the point loads during the rolls across the regular floor. They admitted that it had been done once before, but without some way to distribute the point loads, rolling was out of the question. We could have built a superstructure of steel plate from the deck to the UPS room, but that route has two ninety-degree bends that would make the actual rolling part very tough if done on top of a structure. Then of course is the whole problem of getting the superstructure up into the datacenter as well! We now had to come up with Plan C. We spent more time with Structural Engineers, in fact the very same ones that originally designed the building and at one point, the Engineer Of Record said: "You know, we really didn't consider the problem of moving heavy equipment into and out of this room."
A shocking admission. But the legacy we have to manage. So our Facilities Manager and the Structural Engineers came up with not only a way to get heavy equipment in, but also a way to get it out: Through the roof.
Literally in the space of less than a week, we came up with the idea, prepared plans and structural drawings, fabricated parts, arranged a crane, scheduled a roofer, and got all the permissions and blessings required. That is a lot of work, many hurdles, and a few hoops jumped through in a short amount of time. Not only that but we ALSO effectively UNPAINTED ourselves out of the corner the original builders painted themselves into. We redesigned the placement of equipment in the UPS room to allow greater maintenance access, and more importantly the ability to move any single part in or out without impediment from the other equipment, or potential interruption of service!
We also have allowed ourselves room for larger UPS units than what could even be fit through the existing door. If we need or want larger units, say 500 or 750 KVA UPS gear, we can now get them into our facility.
Here is how we spent our day today:
Above: On the left is the curb and cap for our new access to the UPS room. The room's exhaust fan has been removed, and a tech from MacDonald-Miller is preparing the roof for work.
Above: Roofers have cut away the roofing material and insulation to expose the pan deck below. The hole is where the exhaust fan used to be. We positively pressurized the UPS room with our air handling equipment so that any small particles were forced out of the room. There was a strong wind coming out of that exhaust hole, but it prevented any insulation material from being pulled in.
Above: The Mac-Miller techs remove a section of pan decking. The UPS room is below. Facilities Manager Kevin Teker is assisting on the left. He's turning his head to keep the wind from inside the room out of his face. It is unusual to experience a pressurized room. Standing inside it, you have no sensation of pressure. Standing near the opening, like the tech on the ladder in the hole, you feel a mild breeze. Outside the hole, where Kevin and the tech running the saw are, it is like a blast of air. You can see the tech running the saw... his hair standing straight out, where the tech in the hole in unruffled. What this did was blow the debris of this job up and out, onto the roof, rather than down and in, putting equipment at risk. Access to the UPS room was limited to only when the no cutting or peeling was going on, otherwise the pressure would change enough to risk fine debris being pulled in. The big stuff is OK as it falls straight down and can be picked up. It is the lightweight, fine debris that you have to make sure goes out, rather than in.
Above: Kevin cleans up. Cleanliness is always a very important aspect of any work in and around an active datacenter. You must keep the work areas clean throughout the entire procedure.
Above: The view from inside the UPS room. This opening was specifically designed to be as large as possible. It is significantly larger than our new UPS. This will allow us to install larger UPS' if we need them. Here the Mac-Miller technicians prepare the edges of the opening for the installation of the curb.
Above: The Mac-Miller techs and the Roofer are almost done. The curb is installed, the cap is on, and the exhaust fan re-installed in the cap. The cap is insulated and water-proof, yet lightweight and removable by two people.
Above: Our new access door from inside. Tomorrow we'll open it up and drop the UPS and battery cabinets through it with a crane. Friday the electricians start the installation process.
Stay tuned for more updates soon.
--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc
