The UPS maintenance originally announced for April 25th has been rescheduled to Wednesday, April 30th

In other maintenance news, we took advantage of some cold early morning temperatures to do some coil cleaning and belt changes on our original HVAC unit, known as 'AC1'. Yesterday we cleaned the interior condenser coils, and this morning we cleaned the exterior evaporator coils and changed some belts. Keeping these coils clean is a vital part of maintaining our HVAC systems. Clean coils are efficient coils. The condenser coils can be cleaned without interruption of the system's operations. The evaporator coils required shutting down the AC1 for about 5 minutes at a time. Since we have three main AC units and one reserve unit, there was negligible operational impact on the datacenter. Belts are in constant duty cycle, much like in your car, and require changing at regular intervals.

Below: A view of AC1's interior condenser coils in the middle of their "Spring Cleaning." Usually these are not visible, as they are hidden behind doors.

On April 25th, 2008 we will be performing an upgrade on portions of our UPS system. These will bring these units up to a similar spec to our new UPS.

Anytime we perform UPS maintenance we put the UPS system in bypass, meaning that we run the facility on our own generator rather than the electrical grid. Anytime there is a transfer of power from one source to another, there is a risk of interruption. The risk is very small, and we will take every measure to minimize and eliminate that risk. Briefly, here is the process:

* Manually start the generator.

* Manually transfer the electrical load from grid to generator.
(The interruption here is measured in milliseconds, unlike a power failure, which usually lasts 7-10 seconds as the generator starts.)

* Manually put the UPS system in bypass mode.
(This routes power directly from the main power panel through to the servers, removing the UPS from the flow. This is a two-step process where the bypass bus is energized before the UPS is bypassed. This means there is no interruption.)

* Perform the required maintenance.
(This will take several hours.)

* Put the UPS system back on-line.

* Manually transfer from generator to grid power.

* Shut down the generator after a cool-down period.

We perform this procedure several times a year, so this is fairly routine. We also have over a week's supply of fuel for the generator, so runtime is not an issue. We just like to keep our clients informed of happenings here at the facility. As per usual we'll document the process here on the support blog so you can see what we're up to. We'll post reminders as the maintenance day approaches.

Earlier this month we had our backup generator's fuel quality tested and we found some algae in there. Given the size of the tank (5000 gallons/19,000 liters) and the wide temperature variations we've seen over the past year, it didn't really surprise us. Our previous test did not show any, but the recent one had enough to make us worry, especially as we're going into the critical months of the year for backup power availability. There must have been a recent "bloom" of the algae. We scheduled our friends at Fuel Care to come out and remove the contamination ASAP. They arrived on Friday.

The process involves parking their truck next to our fuel tank and cycling the fuel out of the tank and through special filters in the truck to remove the algal contamination. While that happens they scrub the tank walls and generally inspect the tank's interior while the access cover is off. Please note that we can still operate the generator if needed while this process is happening, as the fuel is constantly cycled through Fuel Care's filter system and no more than a few hundred gallons are out of our tank at a time. In many ways it is akin to kidney dialysis, but for Diesel fuel.

Thankfully the tank itself was very clean and appeared in great shape. We removed about a gallon and a half of water (5 liters) accumulated via condensation and all the algae and now our fuel supply is sparkling clean.

The whole point of this process is to ensure the smooth operation of our backup power supply. You rely upon us to maintain our facility to the highest standard and ensure the maximum uptime for your servers. We just like to share all the things we do with that in mind so you stay informed. We'll have more news and info about other steps we're taking in that direction soon.

Regards,
Chuck Goolsbee
VP Technical Operations
digital.forest

We successfully completed our UPS upgrade work today. We were able to transfer to generator power, bypass the UPS, shut down the UPS, perform the work, the reverse the whole process and restore grid power. Everything worked as planned and as expected. We spent approximately 2.3 hours on backup power.

Above you can see the UPS Technician from MGE performing some circuit board replacements and firmware upgrades on our bypass switch.

Below you can see a gauge that shows the output from our backup power generator system. Running our entire facility's load, which includes all the servers and all of our mechanical systems (air conditioners, humidity controls, fans, etc) uses only 22% of our backup power system's capacity. Lots of room to grow.

Stay tuned for more news about our datacenter facility expansion.

Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

Between the hours of 9am and 12pm, Thursday, September 20th we will be performing maintenance on our UPS system, which will include upgrading some control boards and updating to the latest firmware. This is the final step in our recent UPS upgrade.

While this maintenance is being performed we will put the UPS system and grid power in bypass and operate the datacenter on generator power. This is standard procedure for work such as this.

While we do not expect any change in power delivery, any time power is switched from one source to another, a risk of interruption is possible. Every effort and precaution will be taken to minimize this risk.

Thank You,

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

The Seattle area experienced a bit of a heat wave yesterday. We realize that the rest of the country has seen temperatures even higher, but it is rare for our cool marine & mountain location to see things get as hot as they did in the past few days. The peak came yesterday afternoon, around 5 pm, when we hit 98°F/37°C outside. I'm happy to say that our datacenter's peak temperature was 74°F/23°C.

Today we have high overcast, and expected highs in the low-80s. Datacenter temps should remain in the mid-high 60s.

We've been making continual improvements to our HVAC system. Adding and making modifications to existing duct work mostly. We've boosted our cooling capacity by about 20% in the past few weeks. As we've filled our facility we have been adapting the generic air diffusers with custom duct work. The aim of which is to direct cool air into specific areas, namely the front of the racks. Servers are generally designed to cool front to back. Visitors to our facility will note that we arrange rows of servers with the backs of servers facing into alternate rows. This creates "hot aisles" and "cold aisles", a standard industry practice in datacenters. Cold air from the HVAC system falls down from just above, is pulled into the servers, and hot exhaust air rises up to the top of the room where it is pulled out by the HVAC system's hot air return system. What we have been doing the past several months in preparation for summer is focussing the cold air. This involves custom duct work.

Above: A shipment of our custom ducting arrives.

Above: Custom ducting awaits installation in the datacenter.

Above: A custom duct installed in a client's cage. Note the room for their planned expansion. Note also the unmodified generic air diffuser to the left. It is from these that we mount the custom ducts.

For our general rackspace datacenter we have taken the generic diffusers and focused the new ducts down and into the cold aisles. In the cage space datacenter, we have extended custom ducts directly to the front of customer's servers. This work allows us to better control airflow and maintain the optimum air temps right where it matters. Later this year we'll be adding similar duct work, this time above the hot aisles, specifically to remove hot air and bring it back to the return of the HVAC systems.

Above: The business end of our auxiliary HVAC system.

We have also added a large custom diffuser with dampers in 14" conduit heads on our auxiliary HVAC system. Throughout most of the year, this unit does not even run. In warmer conditions we can open a subset of the dampers and provide supplementary cooling in a very general fashion to the entire facility. On very hot days like yesterday we open them all and provide the extra HVAC tonnage to keep the facility cool. In emergency situations such as last week we can attach temporary lightweight ducts to the 14" heads and run them through the ladder rack to the specific areas of the datacenter that need it.

Stay tuned for some more exciting news very soon about expanded capabilities and additional layers of redundancy!

Chuck Goolsbee
VP, Technical Operations
digital.forest

Last night around 19:30, one of the three primary compressors of one of our two HVAC systems failed. This was noted by NOC personnel who contacted our HVAC contractor who set the controls to bypass the failed unit. We did run on single-stage cooling from one unit for several hours, which lead to both datacenters reaching temperatures in the low-80s F/high-20s C. Once the unit was bypassed 3-stage cooling was achieved and temps dropped to their normal mid-60s F/high-teens C. With outside summertime temperatures in the high-80s F/low-30s C possible we require 4-stage cooling at peak hours.

We will be replacing the failed compressor tonight between 21:00 and 23:00. We have secured portable HVAC units to supplement our secondary HVAC while the primary unit is down for repairs. We will update this website with more information as the repair progresses.

Update: 8:00 AM The new compressor was installed last night. We've resumed normal operations.

Regards,
--Chuck Goolsbee
VP, Tech Ops
digital.forest

We have both our HVAC and Controls Systems vendors on site right now performing an investigation of our issue over the weekend. They will be shutting off the systems for short periods of time in order to perform diagnosis of various system components. Datacenter temperatures may rise as a result. Recover times should be brief however, as the outside temperature here in Seattle is 30F/17C cooler than it was on Sunday with cloudy skies, showers and 53F.

Springtime brings sunny weather and cleanup tasks! Our backup power generator comes equipped with a 5000 gallon, double walled fuel tank. We can run for almost a week on our own power source with this capacity. While fairly mundane in appearance (beyond of course being HUGE) it actually is a fairly high-tech bit of equipment. It has sensors that provide us feedback concerning fuel quality, quantity, and rate of consumption while the generator is running. It also provides data on water percentage in the fuel. Diesel fuel is basically oil, and oil and water as we all learned in grade school science class, do not mix. Large metal containers condense water from the atmosphere, both inside and out due to normal temperature and humidity changes. Diesel engines all have water separation equipment built into their fuel systems. Rudolf Diesel's invention is a compression engine meaning that unlike a gasoline engine it does not use spark plugs to ignite a fuel/air mixture; instead it compresses the mixture at such a high ratio that it heats up to an ignition point on its own. Unlike a fuel/air mixture, water does not lend itself to compression, so it must be removed from the fuel prior to injection into the compression cylinders.

The fuel monitoring system for our storage tank had an alarm go off last week. It signaled a interstitial leak. This indicates that there could be a possible leak of the main tank into the interior space between it and the outer tank.

Kevin Teker, our Facilities Manager responded by going into the interstitial space with testing equipment to see what was setting off the fluid sensor.

Above: The fluid sensor removed from the fuel tank.

Above: The high-tech method of getting the chemical testing paste to the lower reaches of the tank... a tape ruler.

Using special pastes which change color based on contact with either water or fuel, Kevin was able to determine that the "leak" was not in fact a leak, but instead water that had condensed within the interstitial space between the two tanks. Once satisfied that our primary tank's integrity was sound the next step was to remove the water. How does one drain less than a pint of water from a tank that weighs 50,000 pounds and only has openings at the top?

We solved that problem by lowering absorbent materials attached to that high-tech reach tool pictured above. Once the water was removed, we replaced the sensor and it indicated a dry interstitial space once again.

This is just one small example of the little things we do every day to maintain our facility. Every colocation provider has a facility to maintain, and we know that you choose digital.forest for our excellent service AND our facility. I hope that you see and recognize both every day, in the form of your servers' uptime, and when you have interaction with the Tech Support and Customer Service staff. The facility though is in many ways invisible to most of our clients, so it is nice to highlight things when they present themselves and allow you a glimpse into daily operations.

Regards,
Chuck Goolsbee
VP, Technical Operations
digital.forest

4. Wrap up & Summary

We're pleased to report that the repair on our HVAC system is complete, and finished without incident. The final bit of work required brazing & welding within the unit itself. To mitigate any risk of having the pre-action fire suppression system discharging its gasses, we had our vendor Fire Chief, come out and disable the system. Part of our annual maintenance procedure for the fire suppression system involves the shut down of the HVAC system anyway, so Fire Chief took advantage of the situation to perform that maintenance.

Above: Technicians from Fire Chief perform preventative maintenance on the Fire Detection and Suppression system.

During the HVAC system shutdown, digital.forest staff monitored temperatures in various locations around the datacenter, while our Facilities Manager bounced between the roof and the datacenter monitoring our vendors. Below you can see digital.forest Tech Support member Will Winslow and Facilities Manager Kevin Teker in the darkened datacenter just after the HVAC shutdown occurred. They're carrying their temperature monitors and about to spread out to their stations. You can see the high-CFM fans mentioned earlier today in the open door behind them.

All of our preparation paid off, plus a bit of luck from the weather (it stayed very cool, plus it didn't rain) so that the natural tendency for the facility to warm up was mitigated by the combination of pre-cooling and the fans pulling outside air into the facility. We're happy to report that our highest temperature reached was about what we see here on a "normal" day. Our temporary portable HVAC units never even needed to be turned on.

Interesting conclusions:
Electrical capacity is a hot topic in the datacenter management business these days. There are various rules of thumb concerning the estimataion of power usage split between "floor" (meaning the servers) and "mechanical" (meaning the HVAC systems to cool the servers.) The variable is the delta between outside and inside ambient temperatures. The hotter it is outside, the harder the HVAC systems have to work to chill the inside. We're blessed to be located in a very moderate climate here in Seattle. It rarely gets very hot here. Nor does it get very cold. Our average temperature is actually quite a bit lower than ideal datacenter temperature. Even in summer, it cools enough at night to keep our average right at ideal datacenter temperature. We monitor electricity usage at several points, along the flow for a lot of reasons, but on our main panel in the datacenter we can check at a glance and see how much power is being used in total. The ammeter for example read this way earlier today when we were running the rooftop HVAC and 100% using outside air:

That reads 274 Amps. That is 274 Amps of 3-phase power as it comes in off the grid. Our feed is 2000 Amps so as you can see we have a lot of room for growth with regards to electricity. This is one of the things that really attracted us to this facility when we moved here just over two years ago. With so many datacenter operations running at nearly 100% of their power capacity we felt it important to be able to accommodate our clients expanding needs and requirements. This maintenance interval provided us some real-time data concerning the power needs of our mechanical infrastructure. Those rules of thumb mentioned earlier say "for every 1 amp you feed the floor, you feed the mechanical 1 to 1.75 amps." This seems to have been proven in our experience, but rounded down due to our temperate, if not downright cool location here in Seattle. Here is a shot of the ammeter with the HVAC system shut down completely:

That is 219 Amps of 3-phase power. Looking at our monitoring history, we hit our maximum of 400 Amps last July when we had a week of temperatures in the 90-95° F (32-35°C) range. That means we are running at a roughly 1:1 floor:mechanical ratio in terms of electricity at our peak consumption. If anything we are favoring the floor, which is a great advantage in this industry.

Yet another benefit of colocation at digital.forest in cool Seattle!

3. Shut Down Interval.

At 12:07 the entire HVAC system was shut down. Datacenter temps are well within reasonable tolerances after 20 minutes on fans alone. We'll update again with more information after the HVAC is retuned to service.

Update: 12:32 PM PDT
HVAC systems are running again. We'll summarize the day's work soon.

2. Repair Work

Thankfully it has remained nice and cold outside today so our HVAC system, which is designed to use outside cool air if available to reduce compressor load, is running 100% on outside air. This allowed us to continue to run the HVAC while the technicians remove the old compressor and install the new one. So from the perspective of the datacenter things appear no different than a normal day here at digital.forest. All the action is happening up on the roof:

In the top image above the techs wrestle the new compressor up a temporary ramp and into place. In the bottom shot you can see the new compressor in place, and the old broken one on the handtruck, ready to be removed.

The Trane Intellipak is an excellent HVAC system that has a myriad of control options. Below you can catch a glimpse into the heart of the controls, which are usually locked behind a steel panel. We usually interface with these systems via software down in the office, but occasionally it is good to have a look at the atoms represented by the bits.

Above is a close up of the breakers and control units for the compressors. You can see that several breakers are in the "off" position, providing safety for the technicians while they work. Others remain "on" so that the system can still function and provide air handling for the datacenter.

Above: digital.forest Facilities Manager Kevin Teker explains how all of this works.

The next step requires the complete shutdown of the HVAC and Fire Suppression Systems, as the HVAC technicians braze some plumbing. Stay tuned.

Last week we had a single compressor unit in our Trane Intellipak cooling system fail during an unseasonably warm day. The system has built-in redundancies to handle such situations so we recovered quickly from the condition. In order to prepare for the warmer weather coming soon, we have elected to replace this failed unit now. So tomorrow (Wednesday, March 14) we will have a vendor here replacing the compressor. This will involve occasional, brief shutdowns of our HVAC system.

We have brought in industrial sized high-CFM fans, to maintain air circulation in the facility during the maintenance. Additionally we have several portable 1-ton HVAC systems which we can deploy on an as-needed basis should any areas of our datacenter exceed standard temperatures. We have deployed temperature probes throughout the datacenter to monitor this as the maintenance progresses. As such we are confident that this event will have minimal-to-no impact on operations, since we will be prepared to mitigate any heat issues should we see temperatures rise.

We apologize for the short notice, and we hope you understand the reasons why. We strive to maintain our facility to the highest standard, as well as keep you informed as we take steps to do so. We will post updates throughout the day tomorrow.

Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.