As of yesterday afternoon the roof penetration phase of the construction project is complete. The weather cooperated and after a foggy morning we enjoyed clear skies. The roofers were able to seal up and weatherproof the project. The next phase is completely independent of weather, so we can proceed at a pace dictated by ourselves and our contractors, not the national weeather service.

Today (another sunny day in Seattle, we're enjoying them while they last!) the electricians are on the roof preparing the circuits for the HVAC units which are scheduled to arrive early next week. On Tuesday the crane will arrive to install the steel superstructure to support the units, which will be welded onto the support columns installed this week. As soon as the steel frames are ready the two new HVAC units will be lifted up and placed on the roof. Our primary contractor for the project sent us this photo of the steel structure today:

Constructioncam is offline for this lull in rooftop action. Expect an update to the timelapse next week.

To bring you up to date, here are some photos of the project after our rain ceased and work continued:

A single penetration remained after the rain stopped, a return duct in what we call "datacenter two". This is the room we completed and opened exactly one year ago today. Since that time we have sold cage space to five new clients so only about 500sq' of floor space remaining. The final penetration is coincidentally located above this empty space.

The metal workers had two electrical conduits to work around but did so skillfully and without drama or delay. These guys know what they are doing. You can see above the beginnings of the penetration and the conduits they avoided. In the background of the photo is the cold air supply plenum and a stub duct we have in place.

Above: Two views as the hole is completed, the last one showing the work in context of the datacenter two room.

As I indicated in previous postings about roof penetrations we do our best to mitigate or eliminate any risk to the datacenter and its equipment below during this sort of work. We increase supply fan output to positively pressurize the room. This means that fine particles are forced out and upward. Heavy debris such as metal shavings fall straight down and onto a drop cloth we place under the work area with more coverage than required. Our Facilities Manager is everywhere at once, supervising work, ensuring that everything is just right, even operating the shop-vac to make sure as much debris is caught at the source as possible. He also made sure that each penetration was completed within 30 minutes, from first cut to capped. This meant that temperatures in the datacenter remained stable throughout the process. Thankfully outside temperature and humidity were ideal and made managing the inside temps easy.

The final steps involved inserting the curb, duct and capping it:

The roofers then sealed up all the penetrations and the rest of the day was spent cleaning up and getting all the tools and supplies ready for the crane to come and lift them off the roof this morning.

Above: Removed materials collected and ready to go. You can see the chunk of concrete we installed studs into, then later sawed off the slab at the direction of the building engineer. If you look back at the constructioncam time-lapse this was the work carried out under lights in the pre-dawn weekend darkness.

We're going to enjoy the lull in the project to pick up some other datacenter projects that have been on the back-burner while this HVAC install has been monopolizing our attention. Early next week though we're back to full-throttle with a projected HVAC startup date sometime in mid-November. Stay tuned for updates.

Chuck Goolsbee
VP Tech Ops
digital.forest, Inc

The time has come to reveal the "special feature" we mentioned earlier... here it is:

Constructioncam!

We've mounted a weatherproofed Apple Mac Mini computer up on our roof, and networked it to our datacenter below. It is running a timelapse software application and tracking the progress of our HVAC construction project. The above is October 24 through October 27... in under 2 minutes.

We'll update the timelapse and the site as work progresses.

Thanks to the magic of the Internet Protocol, we have a NOC equipped with Doppler Radar:

That photo is right from one of the big plasma screens in our Network Operations Center. It clearly shows the rain squall tracking towards our roof construction, so we inform the crews up there that it is coming. They cover up the in-progress work with tarps and erect their tent over the one they are working on.

During the above conference the roofers vote to keep going, while the metal workers decide to grab lunch, since they have to wait for the rain to stop. Three holes down, one to go, as soon as this squall goes somewhere else.

It has been a busy week!

Of course, as soon as I posted that our break in the weather had arrived, it rained. Thankfully at the start of the project it did not matter too much as it was just a day of hoisting materials up onto the roof and preparing for the next day.

True to the forecast, the next day was clear, bright, and dry. We got to work!

Above is a shot of the supply/return ducts with the Seattle skyline in the background. Work proceeds at a quick pace, with two teams on the roof working away. One team from Centimark Roofing preparing holes in the roof, followed by a team of metal workers from MacDonald-Miller who install the steel column supports on the top of the building's vertical structural members. The roofers then come back to the holes and re-roof around the newly installed support.

Above: A metal worker up to his elbows in roofing while he bolts a column support to the building. You can see a full column support off to the right.

Here is what the column supports looked like when compete:

Several of these were installed over last week, with the final one going in before dawn on Saturday morning. We uncovered an unexpected concrete slab. It was not on any of the building's as-built drawings and was as much a surprise to the building's engineer as it was to us. At first the engineer said we could rest the column support on top of it, but then he changed his mind and decided we should cut a portion of it off. (this is the same engineer who admitted to not planning for installing or removing UPS equipment in our UPS room!) To spare the rest of the building's tenants from the noise of the concrete saw, we did the work in the pre-dawn darkness on Saturday.

The weather is looking "iffy" today but we're rolling forward as fast as we can. Parallel sets of rain squalls are tracking north and south of us as we carry on with roof penetrations. (The rest of the week's forecast is much better though.)

The next job is the large opening for air supply and return. These are huge rectangular openings for the HVAC units to pull hot air out of the datacenter, and push cold air into our supply plenum. The supply plenum is basically a huge box suspended from the ceiling which traverses the length of the datacenter. Supply ducts branch off from the plenum to bring cold air to the racks. We initially place duct stubs along the plenum, then custom-build duct work to suit the rack installations that clients bring in. Unlike the build-a-big-empty-room approach of our competitors, we deliver custom ducting to suit our clients' particular designs. This is one of the reasons we can support much higher electrical densities than just about every other colocation facility in the Pacific Northwest. In addition to supply ducts we will also be installing custom hot air return ducting as well. These will go back to the main hot air return below the individual HVAC units.

Below is a sequence of photos showing the penetration of the roof and installation of the main supply duct. The subsequent supply and return installations are identical.

Step one is removing the roof material, which I did not photograph, sorry. Step two is cutting through the pan deck roof below it.

Cuts with the grain of the pan deck corrugations are done with a metal knife, which is quiet.

The pan deck comes out in sections.

The cuts against the grain have to be done with a sawzall, which is quite noisy.

Adhesive and insulation foam are placed around the exposed pan deck shelf to seal and hold the curb duct in place.

The curb duct is placed...

...then leveled, squared, and aligned properly.

Meanwhile the roofers are preparing the next hole through the roofing material.

Inspecting the quality of the seal before capping.

Capped and complete!

The roofers return and seal up the whole penetration to weatherproof and insulate it.

Here is what one of the hot air return penetrations looks like from inside the datacenter:

All this has to happen in a very short time as any penetration drops our air pressure in the datacenter and could have an impact on our cooling efficiency. Our Facilities Manager is closely supervising the procedures to ensure swift completion. We are also closely monitoring the conditions inside the datacenter.

More updates, and a surprise feature coming soon!

Chuck Goolsbee
VP Tech Ops
digital.forest, Inc

Last week the mail server "smtp.forest.net" crashed and required being rebuilt from backups. This server acts as an outbound relay for hosting servers' mail, the primary mail server for our domain "forest.net", as well as handling mail for many of the clients that used to be customers of Ninewire Digital Solutions, a company we acquired in 2005.

One of the features offered to Ninewire customers was custom, third-party "skins" on their webmail. We updated the mail server software last year and since then the skins have proven problematic and unstable. We made the decision during the rebuild to not re-install any skins beyond the half-dozen ones included in the default install. We apologize if your favorite skin is no longer supported, but overall we feel that the long-term stability of the mail server is more important than the look & feel of a minor component. We hope you understand and agree.

Regards,
Chuck Goolsbee
VP Tech Ops
digital.forest

Update 8:59 PM
new souari seems to be up and running now. All static sites should be working, and while there may be some issues with certain php sites, so far our tests have been very positive. We'll remain working on this through the night, and update as soon as we can put a definite "done" stamp on everything, but for the time being, it looks like we're all back in business and we'd like to apologize to all of our clients who have been inconvenienced.

---

souari.forest.net has suffered a major hardware failure. We have been preparing a replacement for Souari for the past three days, and had planned to deploy it next week, but this development has forced our hand; the replacement box is in the process of being deployed as this is written.

UPDATE: 10/26/07 04:00 smtp.forest.net has been rebuilt and is back up.

One of our mail servers, smtp.forest.net, is currently down.

We are working to bring it back online and would like to apologize for any inconvenience this might cause.

We will edit this entry as soon as there is an update to report.

Souari is back up and running with upgraded versions of Apache and PHP. Most users should not see any difference with the new software, but a few people may find problems with their site that our processes were not able to catch in advance. if you notice a problem with your site, please contact us right away and we will get it resolved for you.

The next step in this process will begin tomorrow, as we run a script to locate sites that may be affected by the upcoming suexec/suPHP implementation. Any such sites we find, we will summarize the issues and notify the users. We expect to be able to give at least a week to address these issues (which are usually very simple), and will of course evaluate that expected time frame if necessary.

Thank you for your patience during these upgrades.

As part of our changes to the FreeBSD hosting servers, Souari will be offline late Wednesday night for an upgrade to the Apache web server software. We estimate that the downtime will be less than 2 hours, during which the current configuration will be backed up, and the new configuration will be installed. There will be substantial modifications to the virtual host configuration files. In the vast majority of cases, everything should continue to function after the upgrade as it has until now. However, we ask that you test your sites thoroughly, and report to us as soon as possible if they find any problems so that we can correct them.

It has been a long time since we've updated you with progress on our datacenter expansion, mostly because the project has been delayed by weather. The last part of our current expansion involves work on our roof, and we've had roughly six straight weeks of rainy weather here in the Pacific Northwest. Not shocking to anyone, but usually September and early October are rather mild and dry around here. That is until we want to open up holes in our roof!

So here is what the roof looks like as of today:

Sharp eyed observers will note the lack of rainfall and unusually clear skies. The forecast calls for stable, mostly clear weather for the next several days. That means we're rolling again! Our primary contractor McDonald-Miller should be starting their roof penetrations very soon. We have to create openings in the roof for cold air supply, and hot air return for the datacenter. The supply goes into the existing cold air plenum, and the returns will be new openings in datacenter two. These should happen between now and Friday, along with roofing work to surround them.

After that we'll be erecting a steel superstructure across the section of rooftop you see above. This will be attached to the main vertical supports of the building. On top of that will be room for several HVAC units. We'll start with just two, which will more than double our current cooling capacity. The beauty of this design is that it will allow us to expand our cooling capacity easily, with minimal impact on the building itself. It also means we can add much more cooling capacity than was envisioned by the building's designers. They originally planned for HVAC units that sit directly on the roof itself. Here are photographs of our existing units:

Above: This is our main HVAC system. It is a 75 ton Trane Intellipak with a humidity control unit. It is hard to see from this angle due to the duct work hiding the main unit, but it is HUGE and covers virtually all of the south end of the building's roof.

Above: This is our Secondary HVAC system. It is also a Trane IntelliPak, but has a capacity of 48 tons. It is much smaller than our primary since it lacks a humidity control unit and has less capacity and no external duct work.

You can see how these units sit directly on the roof, requiring large curbs and tie-down systems. Our new units will be attached to the steel superstructure which will actually be above the roof itself by a meter or so. This takes the load bearing job off the roof and transmits it directly to the building's frame. The original design planned for four HVAC units, with this new design we'll be able to accommodate double that.

Stay tuned for updates as work progresses.

Tonight during our scheduled maintenance window (11:00pm-2:00am) we will be making some changes to our network configuration. This will momentarily interrupt service on one of our upstream network connections. There should no service impact as our other connections should carry all our traffic.

This is being done to address the Comcast routing issue posted yesterday.

We've noted an unusual network issue affecting Comcast customers in the Pacific Northwest. Packets coming into and out of our network are taking unusual paths to get to Comcast users, frequently via California, which is adding latency and occasionally causing timeouts when accessing services on our network.

The issue started some time late yesterday. We're keeping an eye on the situation and will update this notice when we have any further information.

We are currently experiencing issues with our Filemaker 9 server, rosemary.

We apologize for the inconvenience and will update as soon as the problems have been resolved.

EDIT: As of 9PM on 10/7, rosemary is back up and running.

UPDATE 2007-10-04 11:35 PM: Tonights maintenance will be rescheduled for a later date. No maintenance activity will occur tonight.

Tonight during our scheduled maintenance window we will be changing the switch processor engine in one of our distribution switches in order to resolve some intermittent issues with the switch. We will be doing this by moving each connection on the current switch to a new switch one at a time. Customers will experience an outage of up to 30 seconds as each connection is moved. In reality this outage will most likely be less than 5 seconds.

This maintenance will affect servers that are in rows 11,12 & 13 of the datacenter. The maintenance will occur between 11:00 pm and 1:00 am.

The configuration changes to Butternut are complete; all CGI and PHP scripts should now be running as the owner, rather than as the web server. The server is in the process of scanning the home directory for all user files and changing their ownership and permissions to match the new requirements.

Note that from this time forward, if you have scripts that require the web server to write files to your account, those scripts will be running as you, rather than as "nobody." This means you need to make sure the files/directories you want to write to are owned by and writable by you, rather than the web server or everybody.

Please feel free to contact us if you have any questions or any problems arise. Thank you for your patience.

We are now beginning the upgrades to suPHP and suexec on Butternut. We estimate about two hours of downtime, and will post again when it's back up and running.