digital.forest Technical Support
News archive: September 2007

The bristlecone Lasso 8.5 server is currently undergoing emergency maintenance and may be be unavailable for a short time until it is completed.

posted by digital.forest at 10:35 AM on Thursday, September 27, 2007
Categories: bristlecone.forest.net

On Monday evening, October 1, we will be making the final changes to Butternut to enable suexec/suPHP for enhanced security. This will require approximately two hours of downtime late Monday evening.

Customers who are likely to need code changes have been notified through our ticket system and by phone. Everyone else shouldn't notice any changes after we've modified the server. If you do notice a problem on Tuesday morning, October 2, please let us know ASAP so we can resolve it.

posted by Bill D. at 09:06 AM on Friday, September 21, 2007
Categories: Hosting Servers

Our spam & virus filtering service, Postini, has been experiencing some performance issues this week. Many users are experiencing slow response times, or even time-outs when connecting to their Postini spam quarantine.

Postini is aware of this issue and are taking steps to address it. They expect the issues to be resolved early next week.

Thanks for your patience.

posted by Chuck G. at 07:20 AM on Friday, September 21, 2007
Categories: Postini spam filtering

We reached a major milestone in our datacenter expansion project this week. Our new rooftop air conditioner units arrived at the Fabrication Shop of MacDonald-Miller, our HVAC systems vendor. We had a lunchtime excursion to inspect them on Wednesday. They are impressive little units:

Of course "little" is a relative term here, as they are in reality quite large. I say "little" because of their overall compact design compared to our existing equipment, which is identical in terms of capacity, but much larger in terms of size. These new units will do the same job while taking up less space on our rooftop. Here is another photo with some people in it to give you an idea of scale:

Of course, just like kids on Christmas morning we had to open up all the doors and panels and look inside:

Larry Harbison (MacDonald-Miller) and Kevin Teker (digital.forest) have the tools out and are opening up the panels and doors. Shawn Hammer gets out of their way.

"ooooh.. aaahhh"

Yes, we know it is kind of weird to see grown men getting all thrilled about big machinery. But somebody has to care about this stuff, right? If you boil it down to the very basics, we are in the business of transforming electricity into bits. Electricity comes in, and all the computers turn that into bits. Bits which flow out of our facility via the network. The by-product of that transformation is heat. Computers generate lots of heat, but they can't operate in a hot environment so we need to protect them by creating a cool environment. This is what HVAC equipment does. It takes electricity, combines it with the heat expelled by the computers and makes cold air out of the two. This we pump into the datacenter to cool the computers.

HVAC units are measured by their capacity, which is expressed in tons. The system in your home or office is probably measured between 3 and 5 tons of capacity. In our case, things are just a bit bigger:

Both of these new units are rated at 70 tons each. That is not their weight, but a measure of their cooling capacity. Our existing system is a 75 ton unit, and by adding these two we're roughly tripling our cooling capacity to 215 tons. (In reality it will be 263 tons as we currently have 48 tons of reserve capacity in another, backup unit, but it is only really there as a safety net, so it does not count in the basic calculation.)

In addition to their compact size, these new air conditioners are very cool (pardon the pun) for their energy efficiency and excellent features. These new units can meet the same capacity as our old systems while using 33% less electricity. In the datacenter business that is vital as computers consume more and more power. Cooling is the hidden cost of the power usage; for every amp dedicated to running computers there is usually between one and two amps used to cool those computers. The more we can shift that ratio away from cooling and towards computers the better.

A great benefit to being here in Seattle is the predominantly cool weather we enjoy. This allows us to use units like these which have large "Make Up Air Handlers", meaning that when the outside temperatures are at or below ideal datacenter temperature, which in Seattle is most of the year, they can use outside air to cool the facility. This contributes significantly to their energy efficiency. It also allows us to avoid chilled water systems, raised floors, under-floor plumbing, and floor-space robbing CRAC (Computer Room Air Conditioner) units.

To complete the tour, here are some shots of the inside of our new HVAC systems:

Above: Supply fan motors and composite material fan blades

Above: Make Up Air louvers and filtration.

Above: Just a few fuses inside the electrical panel.

At this stage in our growth, we need about 80 tons of cooling on the hottest days of the year, so our plan is to have full-unit redundancy. In other words always have a single 70-ton unit in reserve. This means we can shut down an entire HVAC system for maintenance or repair while still being able to cool our datacenter on the hottest day of summer. The reason for our selection of these compact units is to allow us to grow and continue to add cooling capacity at that full-unit redundancy.

These units were custom-made for us by Aaon in Tulsa, Oklahoma. The next phase is to install them on our roof. More detail on that project will be coming soon.

Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

posted by Chuck G. at 01:35 AM on Friday, September 21, 2007
Categories: Datacenter Expansion

MGE Tech working

We successfully completed our UPS upgrade work today. We were able to transfer to generator power, bypass the UPS, shut down the UPS, perform the work, the reverse the whole process and restore grid power. Everything worked as planned and as expected. We spent approximately 2.3 hours on backup power.

Above you can see the UPS Technician from MGE performing some circuit board replacements and firmware upgrades on our bypass switch.

Below you can see a gauge that shows the output from our backup power generator system. Running our entire facility's load, which includes all the servers and all of our mechanical systems (air conditioners, humidity controls, fans, etc) uses only 22% of our backup power system's capacity. Lots of room to grow.

twenty-two percent

Stay tuned for more news about our datacenter facility expansion.

Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

posted by Chuck G. at 09:46 AM on Thursday, September 20, 2007
Categories: Datacenter Expansion, Facility Maintenance, Scheduled Maintenance

Between the hours of 9am and 12pm, Thursday, September 20th we will be performing maintenance on our UPS system, which will include upgrading some control boards and updating to the latest firmware. This is the final step in our recent UPS upgrade.

While this maintenance is being performed we will put the UPS system and grid power in bypass and operate the datacenter on generator power. This is standard procedure for work such as this.

While we do not expect any change in power delivery, any time power is switched from one source to another, a risk of interruption is possible. Every effort and precaution will be taken to minimize this risk.


Thank You,

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

posted by Chuck G. at 10:52 AM on Thursday, September 13, 2007
Categories: Datacenter Expansion, Facility Maintenance, Scheduled Maintenance

Our Lasso 8.5 server, bristlecone.forest.net, is going to have it's lasso installation upgraded tonight at 11 pm. There may be 10-15 minutes of downtime during this process.

posted by digital.forest at 07:05 AM on Tuesday, September 11, 2007
Categories: bristlecone.forest.net

This evening during our scheduled maintenance window we will be restarting IIS services on kataba.forest.net in order to update the PHP config on the server. The restart will cause all web sites on the server to unavailable for approximately 30 seconds.

The restart will occur at 11:00 pm.

posted by Kyle at 03:52 AM on Tuesday, September 11, 2007
Categories: Windows Hosting Servers

Tonight during our scheduled maintenance window we will be reseting one of our BGP sessions in order to balance our outbound traffic. Our other providers will take the traffic during the reset.

The reset will occur between 11:00 pm and 1:00 am.

posted by Kyle at 02:06 AM on Tuesday, September 4, 2007
Categories: Network