Between 4:45 am and 5:45 am on Monday, March 31st one of our upstream peers (Time Warner) will be replacing a faulty card in one of their routers.

This will cause an outage of up to 10 minutes on this circuit. Our other peers will handle all of our traffic during this maintenance.

We're proud to announce that our new UPS is installed and online. As I've said before it is an MGE EPS 7000 unit rated for 300-500 kVA. Our electrical contractors VECA Electric completed their job yesterday morning and a representative from MGE spent the afternoon yesterday performing the unit's turn-up. Everything went very well.

We captured the installation process in a time lapse movie:

As you can see it is a lot of work, over several days. Our guys know what they are doing though and the turn-up went well.

This is a premium bit of equipment with some very nice features. The most important of which is the capability to scale upwards in capacity as we grow. It also has a nice touch-screen LCD panel on the front that provides us with a lot of detail about what is going on inside:

Above: The user interface of our new UPS. The bottom image is a composite of three sample status screens.

We also purchased a pair of battery cabinets for the new UPS. Combined they provide 14 minutes of battery time at full load. Mind you our backup generator automatically starts up in 7-to-10 seconds if the grid power fails, but having that extra buffer is a good thing.

Above: The battery cabinets. The bottom image shows some of the batteries themselves through the open doors.

Now we turn our attention to the work in Datacenter Three. Stay tuned for more exciting news!

--Chuck Goolsbee
V.P. Technical Operations
digital.forest, Inc.

Sometime between 12:01am and 4:00am on 03/20/08 one of our upstream providers will be starting maintenance on our circuit. Once they start the maintenance should last about 30 minutes. They are going to be moving our connection from it's current port to a new port. This will cause an outage of about 5 minutes on our connection with them.

Our other providers will carry our traffic while this maintenance is occurring.

Just a reminder, tonight at 11 PM we will be performing an emergency maintenance on part of our electrical system. This will have an impact on a portion of our shared web and database servers, and eight dedicated and colocated servers. The shared hosting servers affected are:

acacia, acorn, alder, aralia, arrowwood, ash, aspen, avocado, balsa, balsam, bamboo, banana, banyan, bayberry, beech, bigleaf, blackberry, boojum, boxwood, bubinga, buckeye, cactus, cedar, cherry, chestnut, cholla, cinnamon, clover, columbia, commerce2, commerce3, cork, cottonwood, db, deerwood, dogwood, ebony, elm, evergreen, ficus, fig, filbert, fuji, gingko, grape, grapevine, hackberry, hazel, hemlock, hickory, ironbark, ivy, kentia, kola, kudzu, larch, laurel, lilac, lime, madrona, magnolia, mango, mangrove, maple, mesquite, mimosa, moringa, mulberry, myrtle, newninewire, ninewire2, olive, orchid, palmetto, papaya, pear, pecan, plum, poplar, privet, quince, redbud, sassafras, savin, sequoia, sherwood, snowberry, spiceberry, spruce, strawberry, sycamore, tamarack, teak, truffula, tutsan, walnut, woodpecker, yucca, yulan/fern

Every effort will be made to minimize the downtime. Our electricians have estimated the time required to complete the task at 2 hours.

Thank you for your patience and understanding while we strive to build a better facility.

Update 1:15 AM: The maintenance went very smoothly and was completed in under 90 minutes. We were able to supply alternate power to the few dedicated and colo servers involved in the maintenance. We also took the opportunity to replace some failed parts in a couple of servers (acorn for example whose fan inside the power supply had stopped working.)

Again, thank you so much for your patience during this critical maintenance interval.

Tuesday night March 11th, Wednesday morning March 12th we will be performing emergency maintenance on our power infrastructure related to the installation of our new UPS system. This maintenance will impact a small portion of our shared hosting clients. We will provide specific times in the next couple days as they firm up.

We will strive to keep this outage as short as possible but we are allowing up to two hours of downtime.

Thanks for your understanding while we grow to serve you better.

The gold.wwwnexus.com hosting server has experienced a hardware failure. We are working on repairing it, and will update as soon as we have more information.

Above: Ness Crane arrives to lift our battery cabinets.

As previously noted, yesterday was a fully packed day of activity here at digital.forest. The crane was here again, this time to lift our new UPS' battery cabinets into the datacenter. Additionally our electrical contractors, VECA were here to pull the electrical wire runs for the new UPS and its sub-panel in Datacenter Three.

Thankfully it was a beautiful day, unlike the drizzle earlier in the week...

Above: The view south towards Mt. Rainier from the roof. On the right are our HVAC units 2 & 3.

We captured the crane lifting one of the battery cabinets on video:

That is about 10 minutes of footage compressed down to 2 minutes and 35 seconds. The crane operator was very smooth and carefully slow, so if we'd present the video scaled in real-time it would be pretty dull to watch.

Above: The second battery cabinet arrives.

Above: Tim Beyer and Shawn Hammer check out the batteries

Above: Both cabinets safe and sound inside the UPS room, waiting to be moved to their final locations.

Congrats to our Facility Manager Kevin Teker for successfully landing another big shipment with the help of Ness Crane.

Meanwhile, as the preparations for the crane work were underway, the boys from VECA Electric were over in Datacenter Three making preparations to pull cable. The cable itself is very impressive, being a bit over an inch thick. As we noted yesterday each spool is custom wound with custom made cable cut to fit our facility. Four spools in all, two for the run from the UPS room to Datacenter Three, then two from our main electrical panel to the UPS room, one main feed, and another bypass feed. The latter allows us to continue to power the facility should we need to turn the UPS off, such as for maintenance. (If any of you follow the price of commodities such as copper, you know that this is a phenomenally expensive bit of work - let's just say you could trade a few feet of this cable for a tankful of gasoline. I joked with the electricians as they trimmed a few inches off that they could buy me lunch with what was lying on the floor.)

These cable spools weigh many thousands of pounds and pulling this cable through conduit is not as easy as giving it a tug. VECA brought a special cable pulling machine, which they bolted to the floor of the UPS room in front of the UPS itself.

Above: The cable puller, bolted to the floor. The rope has been fished through the conduit from the other end of the building. At the far end they strip the cables and bolt them onto the end loop of the rope, then use the puller to s l o w l y drag the cables off the spools and through the conduit.

Once things got moving it looked like this at the UPS room end of things:

...and this at the far end in Datacenter Three:

VECA was successful in pulling one set of cables before the crane arrived for the battery cabinet delivery. Pulling continued through today and will hopefully finish up tomorrow. More updates coming soon, so stay tuned.

Regards,
--Chuck Goolsbee
VP Technical Operations
digital.forest

Yesterday was a busy day here at digital.forest. In preparation for the turn up of our new UPS our electricians rolled in several custom-prepared spools of heavy gauge copper wire. Each of these spools weigh just over 3000 lbs (1400 kg). They've been placed in various locations required to complete the cable pulls, which are happening later today.

Also happening today is the arrival of our new UPS' battery cabinets, along with a crane to lift them into the datacenter. To say things are a bit busy around here at the moment is an understatement! I'm doing my best to capture the process and we'll post images later.

--Chuck Goolsbee
VP Technical Operations

Our brand spanking new MGE EPS 7000 UPS arrived this morning. As I explained last summer we crane our UPS equipment through the roof of the building due to the weight of the units. It is just the safest, easiest, and quickest way. So in the wee hours of the morning we cordoned off a section of the upper parking lot for the crane and prepared the units to be lifted.

The UPS arrived packaged in two pieces, and thankfully well wrapped in plastic since light rain was in the forecast. We removed the units from the delivery truck with a forklift and lined them up for the crane to pick.

Top: Ready for the pick.
Middle: Facilities Manager Kevin Teker rigging it up.
Bottom: Kevin guides the UPS into the roof opening.

The whole operation went very smoothly, without incident other than everyone getting a little wet. The new unit is in the UPS room resting on Hilman rollers waiting for our electricians to install it. The delivery of the battery cabinets will be a little later this month.

Above: Our new UPS arrives safe and sound.

This new UPS has over 30% more capacity than our existing units, and has the capability to increase its capacity through an upgrade and the addition of more battery cabinets. With this unit operational we'll be able to more than double our protected power output as we grow. Stay tuned for more information as we continue to expand our datacenter.

Regards,
--Chuck Goolsbee
VP Technical Operations