That is a tower of power you see above. It is the new benchmark for electrical density here at digital.forest. 528 Watts per square foot'. We know that isn't a world record, but it is a very respectable number for a colocation facility here in the Seattle area. Few can approach it, much less match it.

We can't share the name of the client (shhh... they are a startup in "stealth mode") so we've blurred the pictures a bit and removed any identifying marks. They are however VERY happy that they found digital.forest.

Prior to being here, they were colocated at a competitor. They were NOT allowed to put this many servers into their rack at this competitor. In fact, in order to have this many servers, they had to buy FOUR RACKS.

We think that's nuts. So did our new client. Now they are happily colocated at digital.forest and can do it all in one rack.

At digital.forest we are here to assist our clients to succeed, not restrict them from what they need. It is just one part of the digital.forest difference.

Since late last summer, we've been posting occasional updates about our construction progress concerning our datacenter expansion. The reality is that construction was complete at the end of October. We had pre-sold the space to three separate clients, who required colocation cages ranging in size from 300sq' to 700sq'.

We spent November and December preparing those cages. The first client moved into their cage in December. The next moved into their colocation space in late January, in an all-nighter cut-and-move that went off with admirable precision. Our final pre-sale client actually relocated from a set of cabinets in our main facility into a custom cage, with plenty of room to handle their amazing rapid growth (more about them in an upcoming post.) Their move was completed just two weeks ago.

There is still some post-move cleanup to do, but were very proud of our new space, and feel the time is right to unveil it and open it up for further sale. We have over a thousand square feet available, which can be subdivided by cages from 100sq' to 1000sq'. Plenty of power and cooling is available, as we have barely tapped 10% of what our power feed can deliver.

Stay tuned for more expansion news, as we are set to soon take over the entire floor of our building. Meanwhile, enjoy the view of our freshly finished facility expansion:

It is an ironic fact that Network Geeks like us love to have access to more fiber optic cable, but have an almost irrational fear of the equipment that performs the task, namely the backhoe. Yet another fiber provider is landing in our building, but the process to get there involves trenching into one of the fiber vaults. Needless to say, we've been keeping a sharp eye on the crew out in front of the building. (Thankfully our facility enjoys multiple fiber paths so even if there were a backhoe disaster out front, only 33% of our current Internet bandwidth would be at risk.) We've been impressed with the precision and care of this particular crew so far. Their work should wrap up later today.

What does this mean for you, the current or potential digital.forest client? Choices of course! More choices. You can choose to connect to our well-peered BGP4 network. You can choose to connect directly to your preferred carrier, right here in our facility or building. You can choose to use our fiber network to connect to any major Seattle Exchange Point, such as the Westin Building. You can choose to connect your office directly to your servers at digital.forest via a Metropolitan Ethernet connection. You can mix and match any of the above!

We're happy to assist you in the process. Talk to your digital.forest Account Manager for more information. Not yet a digital.forest customer? Contact our Sales staff at 877-720-0483, option 2.

Yes, Seattle has been hit with a cold snap, but here at digital.forest we're still comfortable and the servers are running cool in the datacenter. The image above was taken just before dawn today. That is Mt. Rainier in the distance with the digital.forest world headquarters under the moon in the foreground.

No, it isn't really "breaking news" but we thought we'd share the view.

It has been a while since we updated the support pages with news of our datacenter expansion. Our first client for the space experienced a month delay, so work was paused for a while in September. They are now moving in November 1st, so the finishing touches are being applied. Here is a familiar view, taken from the northwest corner:

It is a composite panorama shot, with the first client's area at the left, their cage walls being constructed around them, the rest of the colocation space around it, in the foreground, and all the way back to the wall, Our Power Distribution Unit near the far door, and the ladder rack extending down the right side of the frame. Our existing datacenter is beyond the double doors at the far right.

Behind the photographer is a ~900sq' private datacenter suite. Beyond that is another ~1200sq' private datacenter suite, or perhaps subdivided cage suite area that will be developed as soon as we fill the existing space.

We have enough available electrical power to power several hundred racks at our current usage level, and the building can support cooling capacity between 109 to 130 Watts per square foot. These are amazing numbers, and we are thrilled to have found a facility with this sort of expansion capability. The work above is just the start of what digital.forest can grow to become.

Above: Facilities Manager Damian Amrhein installs seismic supports to the cage walls being installed.

In other news, we are offering a new line of server enclosures from APC. These units have some very nice features with regards to airflow, power distribution, and cable management. Shown below are the cabinets chosen by our latest client. They were delivered earlier this week and will be bolted to the floor over the weekend. They are very nice units.

Space is going fast, with about 40% of it already spoken for, so if you need datacenter space in the Seattle market, give your digital.forest sales professional a call today at 877-720-0483, option 2.

--Chuck Goolsbee
VP, Technical Operations
digital.forest

Just a brief update on our datacenter expansion. The city inspector was here today to inspect the fire alarm and related circuits. Everything went well. We also commenced the installation of the overhead ladder racking, which carries the network, power and supplemental lighting for the facility.

Here are some photos that show our progress:

Above: The empty shell, before we started work in June.

Above: Last week, after the PDU install but before we started "filling in the room."

Above: The first ladder rack support poles go in today.

Next up, pressure testing the room for the Ecaro25 system, and ladder rack and cage installs.

A major milestone was passed today as our new PDU (Power Distribution Unit) was installed and powered up. We'll have 160kVA available in phase one of the build, and an additional 160kVA in phase two, scheduled for later this year. This means we'll have over FOUR times more power in this section of the datacenter compared to our original facility next door. In our current space we have two 75kVA PDUs. We can pull circuits from 110V to 208V at 20 or 30 Amps. We're barely at 10% of our overall power capacity, with lots of room to grow, so we have the luxury of being very flexible with our potential clients' power needs.

In other news the fire suppression work continues. Unlike wiring a PDU, which is about bring a few really big fat wires from the UPS panel to the PDU itself, this is lots of little wires, sensors, conduit, dectors, alarms, bells, buttons, lights, buzzers, baffle servos, and of course the suppressant tanks themselves. Remember those games when you were a kid where you would spot the difference between pictures? Let's play!

These two photos were taken from roughly the same spot, about a week apart. In the top photo you can see the skilled electrician from VECA installing components of the suppression delay actuator by the facility doorway. You can see all of the detector and sensor wires hanging here and there, whereas in the lower photo you can see all the equipment installed, at least in the ceiling mounted units. The sensor and detector network is quite extensive, with 45 units placed in locations all over the facility. They are small and not very visible in the photos above, except where mounted on the HVAC plenum above the electrician, though a few are visible in the latticework of steel in the ceiling area. Speaking of fire suppression, we've opted for the newer technology of "Ecaro25" for this new facility.

Ecaro25, like FM200, is a non-lethal gas which performs the task of fire suppression without any damage to equipment. Our facility is rigged with a pre-action water system as well, which means that it is plumbed with water sprinklers, but their pipes are dry. In the case of an actual fire the Ecaro25 system seals the room and discharges the gas, only if this fails will the sprinkler system be charged with water. Building code requires the sprinklers be present, but we have mitigated the risk of their discharge by installing the Ecaro25 system as the first line of defence. It is a very cool system, and serves as an excellent safety net for your investment in equipment.

After the fire suppression work is complete, we'll be erecting the ladder rack system for cable management, network distribution, and lighting support. After that is in place we'll begin caging areas already set aside for waiting customers. We have already sold roughly one thousand square feet of this new facility. If you have the need for caged, private access floor space in a top tier datacenter with plenty of electrical power in the Seattle area call digital.forest's sales department soon, because it is going fast!

Chuck Goolsbee
V.P. Technical Operations
digital.forest

Apologies for not updating the d.f Support Blog with any news from our expansion. I've been real busy, but thankfully Facilities Manager Damian Amrhein has been photographing steps along the way.

A lot has happened since our last update:

* The HVAC Plenum & Ductwork is complete.

* The non-static floor tiles have been installed.

* The new PDU/PMM has been delivered and placed.

* The window glass has been blocked off.

* Paint & Drywall is complete.

* The rooms are sealed, and ready for pressure testing
(required for our fire suppression system)

It looks almost ready:

The remaining work consists of:

* Completion of the fire/smoke detection system
(visible as wires dangling here & there in the above photo)

* plumbing of the Ecaro25 tanks, which arrived today.

* Pull the 3-phase 480V feed from the UPS system into the PDU/PMM.

* Erect Ladder Racking System & cage materials.

* Move in the first clients!

This space will be used for custom-caged multi-rack clients. Given the lack of such space, at least with sufficient power & cooling, here in the Seattle market, we expect it could be sold swiftly. We are currently running at merely TEN PERCENT of our power capacity, and only 30% of our HVAC capacity (with plans to add another 110 tons of cooling in 2007) we've got lots of headroom here at digital.forest!

Stay tuned for more news, coming soon.

--Chuck Goolsbee
V.P., Technical Operations
digital.forest, Inc.

The pace of our datacenter expansion has really picked up this past week. Paint, Drywall, and Flooring are the projects underway.

First the room was sealed completely. This increases the effectiveness of the fire suppression system (more news about that coming soon.) Next the ceiling is painted:

Drywall work involves enclosing some beams and finishing a few areas. The major drywall effort is the HVAC system plenum.

Floor tiles are being installed as we speak. We use a special VCT flooring that electrically grounds to the building's extensive ground system. This prevents any buildup of static electricity by humans as we walk around; you discharge static at every step.

We also lost our wonderful view of the Duwamish Valley, as the windows received their frosted film in preparation for complete blackout by metal panels. Oh well, the decks are still there and accessible.

Next up: Our new PDU arrives.

Construction has started on our new addition.

Above: Drywall work extending the plenum for the HVAC system across the new datacenter space.

We're proud to announce that digital.forest has begun construction of our new datacenter addition. In 2004 when we found this facility, one of the significant attractions was the close availability of partially finished datacenter space. We looked forward to the day when we could grow into it. That day has arrived!

Stay tuned to our support website as we track the construction with at least weekly updates. We'll fill in all the details of what the space will become but here is a small sample:

• Roughly 5000sq' of floor space
• Custom sized caged colocation spaces available
• Over 200 Watts per square foot electrical capacity
• One fully-enclosed 1000sq' area available for a single potential client

Unlike most other datacenters in the Seattle area, digital.forest has electrical and cooling capacity to spare. Too hot or cramped where you are now? Can't get that extra 30 Amp circuit? Give our sales team a call. But hurry, 1000sq' has already been sold.

We expect to have the addition open for clients to move servers in by mid-summer. Stay tuned.

As previously promised, here is a support blog post about the backup generator at our new facility at Intergate.West. It is a unit dedicated solely to our datacenter, and securely housed in purpose-built rooms in the lower level of our building.

Above: It is big. It is loud. It is ours.

The generator pictured above is a 16 Cylinder, twin turbo, Cummins Diesel engine fitted with an Onan generator capable of producing 1.25 MegaWatts of electricity. At the far end is a 700Kw load bank. Since our datacenter consumes a fraction of what this machine can produce we must add some load to the genset when it runs to allow it to operate efficiently.

You can watch a movie of it being turned on here. It is VERY loud, which isn't conveyed by the video. It also creates a fair amount of wind, which is apparent. The whole room is vented with large fans which come on when the generator operates.

Above: Yes, it is loud.

Above: Facilities Manager Damian Amrhein operates the load bank.

We have a fully stocked 5000 gallon fuel tank. We'll be running a full load test soon to get an exact figure on our run rate, but we assume that should last us between 75 and 120 hours, perhaps more. We are located, as we were in Bothell, in a datacenter-rich environment (our neighbors are two Microsoft.com/MSN/Hotmail Datacenters, Washington Mutual Bank's Datacenter, and several fiber carrier switches.) all of which are equipped with backup generators. Should any large environmental issue, such as a storm, cause a regional power outage, the facilities here will be getting regular fuel supplies from local vendors. We have been a long-time customer of Nelson Petroleum, who can be here with 5000 gallons of fuel within a few hours. Our standard protocol has been to call Nelson at the start of any power issue that starts our backup generator.

A nice feature of this backup power system that we did not have at our old facility in Bothell is a bus tie system. This is a panel on the outside wall of the generator room that allows us to attach another generator in series with the existing system. This provides N+1 redundancy, as we can bring in a truck-trailer mounted generator unit and attach it to our power infrastructure. Should the existing genset require an overhaul or extended maintenance, we can have a second unit there to continue to provide backup power.

We test run the genset at least once every month, to keep it in good running condition. We will also test it and the transfer switch system on full load at least twice a year. Advanced notice of which will be posted here on our support blog.

Removing the Generator.

We've had this cozy little 150Kva Diesel Genset for several years, and on Friday we bid it farewell. It is heading off to Connecticut to carry on doing its job for its new owners. It was sad to see it go, but as we said in the previous blog post, we have an even more impressive new genset at the Seattle facility. We'll post info about it soon. In the weeks since we vacated the datacenter we've been prepping the genset for removal - draining the fuel and disconnecting it from the electrical infrastructure.

Above: Riggers prep the genset for removal. We're sure the little Western Red Cedar in the foreground will appreciate losing the neighbor that occasionally vented exhaust near it.

Above: Cutting through the bolts that held the generator down on the pad.

It was impressive to watch the riggers gingerly lift and remove the big self-contained generator and fuel tank off the concrete pad and out from under our building. It was always problematic to refill this genset, as we would have to re-route traffic on North Creek Parkway and run a fuel hose from a truck up through the landscaping and into the fuel tank. Our new genset has a fuel filler that is easily accessible. In hindsight, having watched the installation a few years ago, and the removal last week, I wish we could have hired Ballard Transfer for the installation. Their staff were amazing to watch. Unlike the installation, which was an all-day affair involving about six people... none of which seemed to have a firm grasp on the situation, the guys from Ballard Transfer were swift, accurate, and professional when removing the genset. If you have the need to move heavy equipment or machinery in the Seattle area, give these guys a call!

Above: The genset being lifted off the pad.

Above: The genset going through the parking garage and out from under the building.

The riggers then removed the rest of the large equipment from the datacenter and loaded it on a truck bound for the east coast. Plenty of work remains - removal of the racks, floor repair (from rack mounting), conduit removal, etc.

Above: Damian Amrhein walks through the nearly empty Bothell datacenter.

Above: The truck, loaded with the genset, UPS, HVAC, ATS, etc. Damian directs traffic.

While things in Seattle return to normal, work to decommission the Bothell datacenter continued this past week. In November of last year we put all of our datacenter infrastructure (Generator, UPS, electrical transformers, ATS, etc) up for sale on eBay, believing that it would require many months to sell. It actually sold within hours of our initial posting. The Internet is indeed a great equalizer and excellent commerce conduit! An east coast buyer agreed to acquire all that we could not use at our new facility, and we arranged for them to pick the equipment up and ship it to their location. All parties were satisfied - The buyer got an excellent deal and we were able to not worry about disposal.

It was sad to see our big green backup generator go... it had saved us many times and had been a very dependable part of digital.forest's infrastructure. However, we have an amazing generator at our new facility which we will present in a future posting.

The decommissioning work went quite well, with minimal hassle, but maximum effort from all involved. Big thanks to Damian Amrhein, recently promoted to Facilities Manager, and our excellent contractors: ECS Inc., Ballard Transfer, and Team Mechanical.

The process began with a general shutdown of all electricity to the facility. We performed this work after hours to prevent any possible power interruption to the rest of the building. Thankfully no such interruption was required, and we proceeded to remove the wiring and electrical panels. Before the general shutdown, Damian fulfilled a several-year-long desire and pushed the "big red button" (the EPO) which performed as designed and shut down the electrical in the datacenter.

Above: Heavy gauge copper wires on the floor after their removal from between the UPS and the breaker panels.

While the contractors worked on the electrical, d.f staff went to work removing the network and racking infrastructure.

Above: A few miles and a few hundred pounds of twisted pair.

After having been in our new facility for a while, the old one seems small. It is a wonder we were able to squeeze all those servers into this space! As the racks and other infrastructure came down it lost the cramped feel, but still felt quite small. Of course we also are now able to open the blinds and let sunlight in... something we never did in all the years we occupied the space - a refreshing change!

Above: Sunshine in Server Room Two. Electrical Contractors have removed the breaker panels.

Above: Sunshine in Server Room One. The racks here are about to be removed. We'll be selling them soon probably for about $30 a rack, if any clients are interested contact us.

Above: digital.forest Facilities Manager Damian Amrhein poses above vanquished racks, as he pauses between unbolting the seismic supports.

Above: The view from behind the OC-12 fiber termination and the UPS of the almost empty facility.

Above: Racks, removed and lined up. Plus some ladder rack (for overhead wiring) and our lone concession to the ".com era" excess, one of our 80's Vintage video games.

Above: The riggers from Ballard Transfer remove one of the two electrical transformers. The empty UPS battery cabinet sits in the foreground.

One of the benefits of our new facility in Seattle is that the potentially dangerous parts of the electrical infrastructure are physically removed from the datacenter. For example, the UPS is isolated in its own room, and the transformers are outside the building. While we never had any issues with our transformers over the years, we do know of other datacenter operators that have experienced some. Just one more thing that makes our new facility so nice.

Next: Removing the Generator.

When Apple announced the MacMini at Macworld Expo last January, an immediate discussion began in the technology community about the possibility of using it as a server. Just like when Apple introduced the G4 Cube, and the Xserve, companies even came out of the woodwork, basing their whole business model on MacMini Colocation. While the latter is a bit absurd, the former is worth discussing. While digital.forest has never been a "Mac Only" facility (our very first server in 1994 was a Sun Microsystems SparcStation 5) we have always been very open-minded about platform, and embraced the Mac as one of many we support. The Macintosh community, a fiercely loyal group of customers that any company would be wise to embrace, has shown digital.forest as much loyalty as they have shown Apple. This is how we have become the largest collection of MacOS servers on the Internet, at least outside of Apple's corporate datacenters.

People who run servers are always attracted to small form-factor computers because when you buy colocation, part of what you are buying is space. The more computer you can pack into the least amount of space the better... so the theory goes. It also helps that Apple is selling the MacMini for a very low price. A full-blown FreeBSD UNIX install on a sub-$500 computer.

Given the insane colocation market now the cost-per-rackspace argument is kind of moot. But the appeal of a spending under $500 for a server is hard to argue with... even if it may not be a significant performer. An Xserve is far better suited to serve moderate-to-high traffic sites, but as a low-volume mail server for a small office, or a "personal server" the bang-for-the-buck of the Mini isn't too bad.

So it didn't surprise us today when Phil Herring, a long-time client of digital.forest showed up today to replace his aging "blue & white" G3 mail server with a brand new MacMini.

Above: Phil Herring unpacks the lunchbox-sized container that the Mini ships in. This is the first server that has ever come into our datacenter carried one-handed.

Above: Phil Herring holds up his new server.

Above: This little computer is twice as powerful as the one it sits upon, and 3X more that the one it is replacing. We'll have to find out from Phil how well it performs.

I'm keen to find out myself actually, as we have racks and racks of G3 & G4 computers that carry pretty light loads. Mind you we also have racks and racks of G4 & G5 machines and Xserves that carry pretty high loads... but the combo of Xserves and Minis might be an interesting "Big & Little" option. Personally, I'd like to see an "Xserve lite" more tailored to the realities of the datacenter and Internet serving, (Apple should look at Dell's 1U & 2U servers... very nice) but until that happens, we may be looking at a lot of Minis... who knows.

--chuck goolsbee
vp of technical operations
digital.forest
seattle, wa

It has been exactly a week since our last servers moved out of Bothell. We have taken the past week as sort of a "breather" to stop the around-the-clock-frenzy that was our move, and get back to the around-the-clock "normal" that is involved in operating an Internet Datacenter. Move related work will continue next week, but it only involves decommissioning the old facility, and has zero operational impact on you, our valued clients.

Our last server moves were special, in that it was the last servers to leave Bothell, but also because of the client involved. We took pictures and would like to share them and the story here on the Support Blog, if for no other reason than to mark the historic occasion.

The final colocation client to move was our good friends "The Steves" from Cheatcodes.com... they were the last, (They are *always* late... for example they've been with us for over two years but still credit HostPro in their "about" section... what's up with that??) But as always they were entertaining. Steve Cook and Steve Jenkins showed up in Steve's Ferrari F40... not exactly the right tool for moving a cabinet full of servers, but thankfully the digital.forest cargo van was there driven by d.f's operations VP, Chuck Goolsbee.

Above: Steve Jenkins' 1991 Ferrari F40, not exactly the right tool for the job of moving servers, but what the heck.

Above: Interior of the d.f cargo van, loaded with the final servers and cabinet moving from Bothell to Seattle. Apologies for the blurry camera-phone shot.

Above: The ultimate accessory for any car, the digital.forest bumper sticker!

Above: Steve Cook racks a server in their cabinet.

Above: The entire executive staff of Cheatcodes.com engaged in server installation.

After an exciting drive through Seattle's notorious freeway traffic (including a detour over some rough railroad crossings in Seattle's Industrial area south of downtown... we don't know who was more nervous... Chuck for Steve's Ferrari, or Steve for the servers in the cargo van!) the unlikely pair of vehicles arrived at d.f's new facility with everything intact. Within a couple of hours Cheatcodes.com was back online and serving desperate gamers around the globe. The Steves with some knowledge and experience about these things, having found their fortune in the hosting business were impressed with our new facility. They rode off into the sunset, fire blazing from the Ferrari's tailpipes, while we set to the task of shutting down the network links that held Bothell and Seattle together into one seamless 'Local' area network. Few of our server moves were this entertaining, but we were happy to finish up our major migrations in style. Thanks to the Steves, and thanks to all our clients for "hanging in there" while this major move was undertaken. We'll post pictures from our Bothell decomissioning next week. See you then!

We experienced a BGP-related routing issue tonight that reduced our visibility to the outside world for about 5-15 minutes, depending upon where on the Internet you accessed us. It started about 8:15 PM PST, and was corrected just before 8:30 PM PST.

We are investigating and will post more details when they become available.

Update: The issue was the result of a small configuration error in our Seattle router. The slow DNS-like nature of BGP propagation is what caused it to not have immediate impact. Our FastEthernet circuit with NTT/Verio caused us some problems while we worked to migrate it to Seattle overnight, and our Network Management staff spent most of the day today dealing with wrapping up that process. This was completed at exactly 6:30 PM PST this evening. However, at some point during that process an error was introduced into our BGP configuration that caused this issue to arise later in the evening. Thankfully all of our senior network management were still here on site and could address the issue swiftly.

The good news is, that after almost three months of constant configuration changes to all of our routers, we have arrived at a point in our move where no significant network changes will be made for the foreseeable future. All servers, subnets, and connections to the Internet are now fully migrated to the Seattle facility and are here to stay. As I said in yesterday's move update, plenty of work remains to complete the new datacenter, and decommission the Bothell facility, but the server migrations are complete.

As of 5:20 AM PST Thursday, March 10, 2005 ... all servers (minus one colocated cabinet) have been moved to our Seattle facility. Our hard work in the wee hours is finally complete. Plenty of work remains (tie down, clean up, installation of the full reboot system, etc.) but at least now we can readjust to "normal" working hours. We will continue to update this blog with datacenter improvement news as we finish the move process.

Tonight we will be migrating our Ethernet circuit with NTT/Verio over to our Seattle facility. There should be no operational impact or associated downtime in conjunction with this migration. This circuit has been carrying very little traffic (less than 5% of our total capacity) since the majority of the servers have been in Seattle. As such it will not be noticeable as we migrate it. Our other carriers will carry the traffic while it is moved. Once it is here in Seattle we expect it to once again carry the usual 30-40% it has in the past.

The last colocated servers are moving out Thursday night, so the lights will go out in Bothell. Three months and three days ago we unveiled our planned move to the world, and as of tomorrow night, it will be essentially complete. Much work remains, as we must fully decommission the old datacenter (which will continue all this month), but operationally the Bothell facility will close in the wee hours of Friday morning, March 11, 2005.

The Bothell facility has been our home for seven of our almost eleven years of operation, so in some ways this is a milestone. However our new facility in Seattle is so much better - several orders of magnitude better - that we have no regrets in leaving Bothell behind.

We'll post photos of the final moves and decommissioning soon.

As part of the decommissioning of the Bothell facility we are migrating subnets off our core switch in Bothell, and onto our core switch in Seattle. We have moved several of these already, usually as the last machine leaves that subnet in Bothell. However over the next few nights we will migrate the remaining subnets over to Seattle. There should be nothing more than a momentary pause in connectivity as servers in those subnets find their new gateway.

The overall operational impact is close to zero, thanks to the magic of OSPF and iBGP, but we feel the need to share news of the event as it is a significant milestone in our move. We could be migrating our Verio circuit over to Seattle as soon as tomorrow night, leaving literally no more than cleanup and a couple of stragglers [You know who you are... Cliff & The Steves! =) ] as all that stands in the way of total shutdown of d.f Bothell.

It has been a long hard move, but like hitting the 24 mile mark on a marathon, we know it is almost over. Thanks for your patience through all of this.

Our valued clients have spoken - and we are providing what they want. Most of the move support blog feedback of late has been... "Yeah, yeah, that text is informative, but what happened to all the pictures? We want pictures again."

So here is an update, complete with pictures. We are about 95% complete with server moves. Last night was particularly productive. We moved over 40 servers from Bothell to our new Seattle facility. These included mostly single-server colocation machines. We also moved three half-rack clients with relatively complex server setups - such as a combination of firewalls, load balancers, large storage arrays, etc. It was a long night, but we had a full turnout and lots of help.

Above: digital.forest Sysadmin Extraordinaire Dave Rose and Overnight Shift Support Guy Christian Garland watch a server come up after being racked.

Above: Nirav Patel from Pragmatyxs brings up a server.

Above: digital.forest Network Manager Kyle Murray prepares to punch down a new network drop for two of our favorite colocation customers. Servers from Car Toys and Neptune.com are in the foreground.

Speaking of Car Toys... we heard from them that their online store's revenue grew in 2004, from the previous year - by 945%(!) The Car Toys Online group were named "Department Of The Year" in 2004 for this achievement. Congratulations from all of us at digital.forest!

This goes to prove something we have always believed: The Internet is an excellent place to conduct business, and the "dot com crash" was merely a speed bump on the road to success. Those businesses that kept their commitment to electronic commerce through the slowdown are beginning to see rewards for their efforts. The costs to operate a half-rack of servers and maintain a small web development staff must be a fraction of operating a traditional physical retail outlet. Car Toys' experience has shown that their online store does not "cannibalize" sales from their "brick and mortar" stores though, if anything it has opened up regions they don't yet serve with retail outlets. They operate stores from Seattle to Houston, yet their online store operates "coast to coast" 24 hours a day. Hard to argue with 945% growth too.

We only have a few servers left to move from Bothell. Once that work is complete we have quite a bit of finish work to do at the Seattle facility. The reboot system is not yet fully functional here, as we have some equipment to move and systems to re-program. We also have a lot of "tie-down" work to do. This involves cleaning up the inevitable wire mess that seems to gather around all computers. We'll illustrate how that works in an upcoming blog post.

We finally were able to empty enough racks out of Bothell to start building the last few rows for servers at the new facility.

Above: Damian Amrhein and Kyle Murray pull cable bundles into the latest row of racks.

After all the racks are removed from Bothell, we can pull the bracing plates from them and mount them between the overhead ladder rack and the rows of racks here in Seattle. This provides further structural rigidity to the rack system. After that completes, we decommission the electrical and HVAC systems at Bothell. That should wrap up by end of March.

We hope you enjoyed this latest news and we promise to post more pictures in upcoming blog updates. Thanks!

Our shared FibreChannel Array storage system is moving on Thursday night, along with our FibreChannel Switch, and any colocated customers attached to it. Since it will involve downtime, we are also taking the opportunity to upgrade the array's firmware. This should fix a known issue with LUN Masking not working with a certain operating system.

We will also retire an old 1GB FibreChannel hub at this time and integrate customers using it onto our 2GB FC Switch.

Our trouble ticket server (helpdesk.forest.net) will be moved to our new facility tonight. It will be unavailable for approximately two hours, from 11:30 PM until 1:30 AM PST.

We have almost completed server moves to our new Seattle facility. Our DNS and mail servers will be moving over the next few nights. At this rate, we may be able to shut down the Bothell facility as early as next week after the weekend moves complete.

We will post updates concerning the mail servers at least 12 hours before we move them.

The colocated server backup device, and large disk array that it uses as storage will be moving over the weekend. It will move during daytime, so there should be minimal disruption of server backups.

The servers we moved tonight are those behind our shared firewall. We had some delays leaving Bothell due to some struggles with rackmount Dell "Versa Rails"... stripped screws specifically. We also are having to rebuild our Firewall config from backup, as the device did not seem to like its new location. Please be patient while we sort these issues out.

Thanks.

All hosting and colocation server moves scheduled for tonight have been completed successfully.

We have a few more servers in the hosting network to move. Mostly we are moving colos now, but tonight we will also move the following servers:

band, logging, cactus2, and sherwood.

We have completed the moves of the following servers:

acacia, ash, bamboo, banyan, commerce, commerce2, commerce3, cork, fm4.infoasis, ironbark, ivy, magnolia, mangrove, maple, mulberry, orchid, redbud, teak, and yucca.

Tonight we are moving most of the remaining hosting servers. These are legacy boxes running older CGIs/apps such as Tango, & Lasso 2, and FileMaker 4. They were originally scheduled for last night (Sunday), but some priority colo moves pushed them out a day.

Moves will start later, around 1 AM.

Here are the servers going:

acacia, ash, bamboo, banyan, commerce, commerce2, commerce3, cork, fm4.infoasis, ironbark, ivy, magnolia, mangrove, maple, mulberry, orchid, redbud, teak, and yucca.

We will post results when finished.

Colocated servers started moving over the weekend, and will continue every night from here until we finish sometime in March. For obvious reasons, we will not be publishing the servers by name here on the support blog. However, the schedule remains pretty much the same. Between 8 and 30 servers moving a night, usually between 11 PM and 2 AM. Some clients have requested different schedules, but most prefer the "wee hours."

Our customer service staff have been in contact with most colo and dedicated server owners. If you haven't heard from them yet, you will be soon. I want to thank all of you, our clients for being so flexible with server move schedules overall, and understanding when it comes to this minimum downtime while we move your servers.

The remaining posts to the public blog will be when we transition this server (www.forest.net), our mail, and DNS servers, migrating the remaining shared resources. We'll also post the occasional picture of a server move if we have anything of note to show. Expect some shots of the decommissioning process as well... should be interesting.

We are taking a very short break from server moves tonight, and will likely only move the streaming server "columbia". No rest though, as we'll continue to handle infrastructure (racks, shelving, etc) relocation this evening and all through the weekend.

Saturday night will be a busy one, with the following servers scheduled to relocate:

boysenberry, bark, sage, date, crabapple, woodpecker, tangerine, souari, and butternut.

These are all rackmount servers and require extra time to remove and setup. To date we have averaged about two hours to shut down, remove, relocate, and setup servers. The tower model boxes we've moved this past week go very fast, and we can move 20-30 a night. The rackmounts take so much longer so we can only move a handful at a time.

Sunday night will see the relocation of the former Exceedia/FMPhosting.net servers and the remaining d.f hosting servers (mostly older boxes supporting legacy CGI's such as Lasso 2, and Tango) We will post a named list at least 24 hours before their move.

Colocated servers will start moving Monday. Our customer service group has been contacting colo clients over the past few weeks to schedule these moves to within a day or two. We can be as flexible as possible, and hope you can too. If you are located close to us and wish to participate in your server move, that is great - In fact we encourage it. We have a cargo van specifically for the job if you would like us to perform the actual transport. If you can not be here, please be prepared to either login remotely at the appointed time, or provide us some method for gracefully shutting down your server(s). For example, with some versions of Solaris, and all versions of OS X, you can enable a "shutdown" button on the login screen. With FreeBSD, Linux or similar, you will need to shut it down yourself, or provide us with a login to do so ("sudo" permissions are fine, we don't need full root access - and you can create/delete an account for us just for the move.) With Windows or pre-X MacOS, you can leave the GUI running. If you logout, or use a lockout screen saver with Windows you will need to either login at the appointed time, or grant us access to shut down the server.

Please Note: We are willing to VERY flexible with regards to time of day for your move. If you want to move your server mid-day, or evenings (for example if your users are mostly in Europe or Asia... or you just don't want to be up at midnight) let us know. About the only times we would discourage are during the Seattle Area's notorious rush hours. The distance between our two facilities is about 24 miles (39 Kilometers), which between 4 PM and 6 PM could take well over an 90 minutes.

Tonight's server moves are complete. Thank you!

The following servers will be moving tonight, starting at 11:45 PST:

avocado,bayberry,bubinga,buckeye,cassia,
cinnamon,clover,cottonwood,db,fig,filbert,
fuji,kentia,lilac,lime,madrona,mango,mesquite,
mimosa,myrtle,orange,poplar,spiceberry,
tamarack,truffula,walnut

Tonight the following hosting servers were successfully moved to our new facility in Seattle:

aralia, aspen, bayberry, beech, boxwood, cactus, cedar, cherry, chestnut, cholla, deerwood, dogwood, elm, fern, gingko, hackberry, hawthorne, kola, moringa, olive, papaya. plum, quince, savin, sequoia, strawberry, tiawe, tutsan, and yulan.

Moves were done in two waves, the first started with 8 servers at 9:30 PM PST, and completed at 11:00 PM PST. The second started at 1 AM PST, and was complete by 2:45 AM PST.

Sorry, no pictures from tonight. Between the issues at the Bothell facility earlier in the evening and the mail server problem that sidetracked a portion of the technical staff involved in the move, we had no time for pictures.

The following servers will be moving slightly earlier than expected tonight (around 9:30 - 10:00 PM PST):

cedar, aspen, beech, yulan, fern, strawberry, dogwood, tiawe

The following servers were originally scheduled to move on 2/10 but may instead move late tonight or early tomorrow morning:

deerwood, kola, cherry, kudzu, chestnut, boxwood, cholla, savin, hawthorn, cactus, aralia, elm, papaya

All other servers scheduled to move tonight will move as expected. Thank you!

The following servers were successfully moved tonight:

balsa, banana, grapevine, pear, laurel, sassafras, grape, moringa, privet, evergreen, pecan, snowberry, mango, blackberry, palmetto, bigleaf, and acorn.

Again, we got a late (after midnight) start with actual moves, as we had quite a bit of prep work to do. However, we managed to pre-prep for tomorrow and Thursday night's moves, so those should cover more servers and take less time.

Here are some photos from tonight's move:

Above: The Amazing Tom Kepler prepares documentation. Servers are queued for transport behind him.

Above: Sue Dart and Kyle Murray push the cart loaded with servers towards the van for the run over to Seattle.

Above: The van loaded and ready to go.

Above: Servers racked.

We have changed the schedule a bit, and will be moving these servers in the next few nights:

2/8/05:
balsa, banana, grapevine, pear, laurel, sassafras, grape, moringa, privet, evergreen, pecan, snowberry, mango, lilac, blackberry, palmetto, orchid, bigleaf, and acorn.

2/9/05:
cedar, hazel, aspen, olive, fern, yulan, beech, dogwood, yucca, larch, acacia, sycamore, hemlock, ebony, balsam, hickory, truffula, tiawe, ficus, filbert, and strawberry.

2/10/05:
orange, raspberry, clover, gingko, tutsan, sequoia, fig, cinnamon, hackberry, spiceberry, avacado, quince, lime, cactus, deerwood, savin, elm, kentia, tamarack, myrtle, cassia, buckeye, papaya, hawthorn, kola, chestnut, mimosa, boxwood, bubinga, cherry, blackberry, cholla, fuji, bayberry, columbia, sherwood, plum, and kudzu.

Again, shutdowns will begin roughly around 11 PM PST, and all servers should be back online between 1 AM to2 AM PST. The above list is subject to change due to space or time concerns. Thanks for your patience. We will post updates as required.

The moves of boysenberry, sage, date, crabapple, woodpecker, tangerine, souari, butternut, and bark will likely happen over the weekend or early next week. Stay tuned for updates with more specific schedules.

We have completed the moves of:

skil
jig-old
power
chain
circular
iron
sabre
buzz
radial
jig

We were delayed in starting (actual shutdowns did not begin until close to midnight.) All were up by 2 AM, with the exceptions of sabre and iron. We are working on those now. We will update you when we have news. Thanks for your patience.

UPDATE: 2:55 AM
sabre.forest.net is now UP.

Still working on iron.

UPDATE: 3:30 AM
Iron suffered a motherboard failure. We are installing its disks into a spare server (an equivalent dual-processor design) now. We'll update with an ETA soon.

UPDATE: 5:30 AM
iron.forest.net is UP.

Sorry that took so long. Here are some photos we grabbed before the whole iron issue flared up:

Above: Kyle Murray loads servers into the van for the run over to Seattle.

Above: Chris Kilbourn (in the van) and Kyle Murray, unloading at the Intergate.West parking garage.

Above: The servers we moved last night, all racked and running (except iron.)

The following servers will be moved on Monday night, Feb 7, starting around 11 pm.

skil
jig-old
power
chain
circular
iron
sabre
buzz
radial
jig

Expected downtime for any server move should be around two hours. We will do everything we can to minimize this as much as possible. All servers listed should be back online by 1 AM.

This will be our first large scale server move, so downtimes will only get shorter, and number of servers moved should increase as the nights go by.

The next servers scheduled to move are:

boysenberry
sage
date
crabapple
woodpecker
tangerine
souari
butternut

These will move on Tuesday, Feb 8, either starting around 2 AM, or 11 PM. Specifics will be posted here as the time approaches.

PLEASE NOTE: As all these devices are rack-mount servers in very tight quarters with each other, we can not specifically call out exact shut-down times. Additionally we may elect to NOT move some particular servers due to space or time constraints. Details will be posted here with specifics at the time of move.

Thanks for your patience.

The final regulatory, inspection, and configuration hurdles have been cleared. The new facility is fully wired, fully inspected, officially approved, redundantly connected, and ready to go. It is time to start moving servers. Hosting servers* will be the first to move and we will list them by hostname and schedule tomorrow, for moves starting Monday/Tuesday night 2/7-8/05. Stay tuned, and thanks for your patience.

* By "Hosting Servers" we mean shared web and database hosting servers owned by digital.forest. Dedicated and Client owned (colocated) servers will move later. Half-rack, full-rack, and multi-rack client servers will move last.

Please note our new contact information, effective Monday, January 31st, 2005:

Phone Number: 206-838-1630

Fax Number: 206-838-3749

Our toll-free (in North America) line remains the same: 877-720-0483

Mailing Address:
digital.forest, Inc.
12101 Tukwila International Boulevard,
Suite 410
Seattle, WA, 98168

We will be updating our web site's other pages asap. Telephone extensions remain the same.

Lots of activity at the new facility, but we've been remiss in updating these pages lately. Sorry for that. Hopefully this entry will get us up to date.

Our second fiber optic connection has arrived.

Above: Installers from XO Communications thread the fiber (thick yellow cable) through the ladder racks.

Splicing should be completed today. We'll turn up that circuit some time next week. In the mean time we look up and admire the colorful cabling while we work.

Above: Fiber (yellow) exits the four inch conduit coming up from the basement vault and onto our ladder rack system. Nice radius bends as it turns corners.

On the compliance front, we passed both our initial electrical and fire suppression system inspections. State Inspectors and the Fire Marshall spent several hours yesterday going over every system while we, and our electrical contractor watched in anticipation. When it was all done they stuck their approval stickers and we all celebrated. One final test remains, our EPO (Emergency Power Off), which has been delayed several times due to schedule issues with the State Inspector. Once that hurdle is crossed we start moving servers in earnest.

Above: Official Seal Of Approval from the State of Washington.

Above: Testing the FM200 fire supression system.

We actually have moved some servers. For example, the one that serves most of the images for this blog has been running at our new facility for a week. Stay tuned, as we expect next week to be as busy as this one.

The electrical inspector has finally provided a firm schedule for our EPO test. It will be conducted between 9:30 AM PST and 10:30 AM PST today.

We'll post results later today.

At some point early next week, the Electrical Inspector and Fire Department we will be testing our EPO (Emergency Power Off) and FM200 Fire Suppression systems. This is one of the final hurdles to getting the new facility officially blessed and ready for production. We will post exact times prior to, and photos of the event shortly after.

EPOs at colocation facilities have seen some news and discussion lately. But it serves to bring up an issue of relevance to digital.forest. We have no plans to allow "un-escorted" or "card-key" access to our new facility. We will always escort clients and visitors while they are in our colocation area. As much as we love and trust our clients, we also know that human beings are fallible and curious. "What does this button do?" is not something you want being answered by a curious push. We have cameras in our datacenter, but they can not prevent an event, only record it... at least up to the moment the EPO gets pushed.

EPOs are required by law for human safety in facilities where large amounts of electricity, and backup power systems exist. Common sense, in the form of knowledgeable staff escorting visitors is the only way to ensure that serious accidents do not happen.

We have always felt that service is what distinguishes us from the competition. While many consider unrestricted access a service benefit, we consider it a liability as it offers no protection from EPO-related outages, be they deliberate or accidental. We hope you understand this perspective and share our view.

We took delivery on a very important piece of new equipment this week. A new Cisco Router/Switch. It is a 6509 series router.

Above: It arrived in a huge box, shrink wrapped to a wooden pallet. It took three of us to wrestle it out onto a cart so we could wheel it into the datacenter. It weighs close to 300lbs (136 kgs). Our new router.

Since our move to Bothell from Seattle in 1998, we have employed Cisco 7206 routers for our boundary. We have upgraded them along the way to 7206VXR's. Our core routing was originally done with a Cisco 3640 router, which was eventually replaced with an Extreme Summit 48 (and later 48i) switches. We are taking to opportunity afforded by this move to upgrade again, this time consolidating the boundary routing and core switching functions into the well-proven Cisco Catalyst 6500-series hardware. The router pictured is the first of two we have acquired, which will be deployed as a redundant pair.

The move to a high-end carrier-class facility really requires similar carrier-class equipment. A large portion of the Internet's traffic is carried through Cisco 6500-series hardware. Very soon digital.forest's traffic will too. These will also allow us to expand our internal network core as well. Take one last look before it becomes obscured by a lot of wire.

Above: Racked and ready to go.

This week (1/10-1/15) will likely be a slow week on the move blog here while we deal with the "people" side of the office move. If you have feedback or questions about the move, or anything covered in the blog to date, feel free to drop me an email. Thanks.

We have received some inquiries from existing and potential colocation clients about the "locked cabinets" mentioned in previous move blog entries. They are very nice cabinets manufactured by Cooper B-line... their "E2 Zone 4 Seismic Cabinet." Made of steel with locking doors on either side, two removable/locking side panels - the front door has a tinted Plexiglas panel. The four server mounting rails can be mounted at any position fore/aft. The entire cabinet is bolted to the floor. The roof of the cabinets have three openings, one for network cable entry, one for power cable entry, and one large vented one for a fan installation. The cabinets are available in both 19" and 23" widths. Anyone interested in deploying their Full-Rack installation in one of these fine enclosures, contact your digital.forest account manager.

Above: Our B-line steel locking server cabinets.

Work at the new facility continues.

Above: Outside plant guy and digital.forest network manager Kyle Murray meet regarding a propsed change in the fiber installation.

Some delays have come up with regards to our second fiber optic installation. Our chosen provider has some facility in the building but "getting from there to here" is the primary issue at the moment. While that is going on we continue the installation of our Ethernet wiring. All the wire bundles are pulled and installed, so now we are punching them down into patch panels. Thankfully we have some skilled labor in-house:

Above: It is a punchdown party. Clockwise from top: Sue Dart laces cable, Kyle Murray, Chris Kilbourn and Damian "The Terminator" Amrhein terminate Cat-5.

The final step in the cable installation is "lacing." Cable lacing is something of a lost art. It was how the phone company arranged installed cables in old switch rooms prior to the invention of "zip ties." The terminology for a clean cable installation has always been "tie down" and in the case of cable lacing it is quite literal. There is a sequence of knots, using a waxed lace to secure the cable bundles to the supporting steel ladder rack. Lacing provides a better tie down than zip ties, which can damage the cables and destroy the integrity of the network signal. Zip ties also make it difficult to trace or remove individual cables. Given the opportunity to do it right from the start, we chose to lace our cable installation. Two shifts have been terminating and one shift lacing this past week. Here is a sample of the lace work:

Above: Looking up at the laced cable bundles (blue) leaving the network core out towards the racks. The orange conduit on the left is fiber. The thick white 4-inch conduits high above the light fixture are one of the two pairs of fiber conduits coming into the facility. The orange and silver conduits on the upper ladder racks are the power feeds. The black finger-shaped items in the lower-right are cable management devices. It all looks jumbled in the forced perspective, but it is all separated by reasonable vertical distances.

Next update: The arrival of our new router.

Apologies for a lack of updates this past week concerning our new facility in Seattle. We have been working two shifts a day doing the wiring for the network. This involves bundling wire in groups of 24 as it comes off of spools and then laying it up into the ladder-rack. It is very time-consuming and labor intensive, but like the rack installation, it is something we insist on doing ourselves.

Above: Four and a half miles of Category 5 Ethernet wire. When complete the new facility will have over thirty miles (48 Kilometers) of network wiring.

We built a jig and spent days pulling measured bundles of wire. Next we pulled these through the ladder racks, which are a grid of horizontal pathways suspended above the server racks. We much prefer overhead wiring to the alternative of "raised floor." Raised floor was designed for the mainframe systems of old, where there would be a low density of machines (and network wire) with circulation of cooling air being forced through the floor. With today's datacenters being very dense with servers and networks maintaining thirty miles of wire under the floor would become difficult and would inhibit air circulation. Wiring can also be messy and hidden under a raised floor. We prefer to invest the time to create a very clean and visible wiring plant, as you will soon see. Getting there of course is anything but clean. The wire bundles are difficult to manage before they get suspended. They must be kept clean and not allowed to be stepped on or tangled. We made use of the ample space available to us in this new building to arrange the bundles neatly prior to raising. The longer runs had to be delayed until New Years Eve when the building was officially 'closed' for the holiday and we could make use of corridors for laying them out.

Above: Our longest cable run, a 120+ foot 12 wire bundle, destined for our server building lab is pulled from the spools by Chad Quimby and Rus Pagenkopf. The above shot is a panorama of two photos taken at a ninety-degree angle, with the end of the run being down the hallway.

The bundles were pulled into the ladder rack, either with a rope or sometimes by hand. The next step is to drape the ends into their final locations in preparation for termination (where the individual wires inside the cables are punched down into a patch-panel at both ends of the run.) The final step is to "lace" the cable bundles to the ladder rack to keep them neat and straight.

Above: Wire runs completed, awaiting termination.

Termination work will begin Monday, January 3rd. We had hoped to start December 31st, but a delay in our second fiber installation has pushed back the entire schedule a few days. Server moves can not begin in volume until we have some network redundancy in place. This should happen sometime later this week.

Next up: Termination, cable lacing and initial server moves.

I'd also like to take this opportunity to wish a Happy and Prosperous New Year to all of our clients and friends.

--Chuck Goolsbee V.P., Technical Operations digital.forest, Inc.

We took a short break over the Christmas holiday. However, just before, on Thursday, December 23rd, we completed the new fiber circuit to our new facility and lit it up with network connectivity. It was thrilling to see the first packets in and out of our equipment there. This is a significant milestone, as we can begin to move servers soon. The first few test machines will be heading over this week. Wholesale migrations of our hosting servers should begin after the first of the year. We will post a schedule for hosting servers later this week. Mid January will see the start of single-server colocation clients, with rack-space clients moving in February.

We did do a bit of work on Sunday, December 26th, installing some shelves and prep work for the wire runs:

Above: Installing shelves.

Next Entry: Wiring and first servers.

It has been a busy week so far at our new facility in Seattle. Today's update is all about racks and Fiber.

Above: Tech Support Manager Damian Amrhein wields a hammer drill almost as well as he handles your technical support calls.

As primary contractors finish their work more and more responsibility falls on digital.forest staff to complete the preparations of our new datacenter. We would not dream of sub-contracting issues such as rack installation. For our entire ten years of operations we have always done this job ourselves. This is a job that must be done right the first time, and as the old adage says, if you want it done right - you have to do it yourself. Unlike past situations though this time we have a lot of racks to install. Previously we would do a row or three, this time we are installing well over 100 racks in twelve rows. It is tedious, repetitive work, but shows how well we work as a team.

Above: Hands blurred in high speed installation action, digital.forest Network Manager Kyle Murray hammers home concrete anchors while operations VP Chuck Goolsbee works the wrenches to install racks. This row will house our network, routers, and switches.

Two teams have worked almost around the clock since early Monday to install over 60 racks, and prep for the installation over 50 more coming over from our old facility. Some of the original digital.forest racks from our last move (summer 1998) are being retired. If you need 32 open relay racks, let us know and we'll give you first crack before they go on eBay.

Over the same near round-the-clock schedule this week, our first fiber-optic cable installation continued. OnFiber, our favorite supplier of Metropolitan Area Ethernet services completed much of the circuit from our current facility to our new one. Today they finished the last splice in the fiber optic vault outside our new building, as well as terminated the fiber on one of network racks we bolted in last night.

Above: The OnFiber splicing trailer.

Above: Mike Kim and Brian Medley from OnFiber hard at work splicing fiber in the trailer.

One of our other fiber providers performed a site survey on Monday, and is scheduled to bring their fiber from the first floor up to our datacenter sometime in the next two weeks. We are on schedule, and working hard to be ready for server moves which should start after New Years. We are hoping to have a webcam up at the new facility soon... then you can watch us work in real-time! Until then, you can watch some video:

Click here.

Next up: Networks, part two.

It was a busy week at our new facility in Seattle! Today's update is all about Power and Fiber.

Above: Technician from VECA at work terminating electrical wire in one of our two PDUs.

Primary electrical work completed on Friday. This involved taking AC power from the PDUs out to all of the racks. Seattle's VECA electric is performing the work. They've been at it for the past week and have done an excellent job. Big thanks to Orville and his crew, and Alan & Charlotte in Project Management.

The UPS batteries are now fully charged and operating. We should start installing network equipment soon.

Above: UPS status.

Most importantly, our first fiber-optic cable installation began on Thursday. OnFiber, our favorite supplier of Metropolitan Area Ethernet services started pulling a new installation from their POP at Intergate.East. We will use these connections to bridge our Local Area Network in the current Bothell facility to the new one in Seattle. We also will use OnFiber as we always have, a Gigabit Ethernet connection to Seattle's "Westin Building", the telecommunications hub for the Pacific Northwest. Like our Bothell facility has been since 2001, our new facility in Seattle will be "virtually in the Westin" by a Gig-E connection. We have ordered local connectivity at Intergate.West, so our facility will be directly connected to two of the three major POP's in Seattle.

Above: The crew from OnFiber measures out innerduct for a fiber pull from the ground floor up to our datacenter.

Thursday's work involved pulling innerduct (the orange plastic tubes that protects and identifies fiber-optic cable) up through the building's riser ducts as well as from the building to OnFiber's POP. The actual fiber will be installed next week.

Above: Willy Peterson and Brian Medley from OnFiber pull the last innerduct into place. Willy is standing on the spot where our fiber-optic termination rack will stand.

We love OnFiber, they are a great company to work with. (Reload that page enough times and our logo will eventually rotate through.) Huge thanks to Brian, his crew in Operations, as well as Melissa in Sales, and Jennifer in Project Management.

Next up: Racks and Network.

An update on our new facility construction progress.

The UPS had new batteries installed last week. This UPS has two complete battery cabinets - they can operate as a single unit, or independently. This allows us to have more capacity of course, but also provides battery backup power while we perform maintenance on either cabinet. For example when we do our annual preventative maintenance we can shut down one battery string for testing, while still having the other available. It is unlikely that we would experience a power failure during our annual maintenance - but at least we would be fully protected in that occurrence. This should also add a layer of redundancy that should prevent issues like we experienced a few months ago where a power event caused a battery string to fail momentarily. Just one of many things we have learned and are applying to improve our service and facility as we grow.

Above: Battery Technicians at work turning up our new UPS.

In other power-related news: Just prior to the UPS turnup we activated mains utility power out to the MGE PDUs (Power Distribution Units). These act as both circuit breakers and "intelligent routers" of AC power out to our racks. We have two of these units and they will serve the two sections of our datacenter. One (the near unit in the below photo) will cover the hosting and single-server colocation section. The other (the far unit in the below photo) will be dedicated to the rackspace colocation section. This is where our half-rack, full-rack, and locked cabinet colocation clients' equipment will be located.

Above: MGE PDUs, (the white boxes in the middle-ground) lit up for the first time.

Sharp eyed viewers will note the DC power distribution equipment in the foreground on the left. There is also a Lucent DC PDU in the far background. While we have no plans to immediately turn these up, there is infrastructure to support 48 VDC in the new facility. If a large client requires DC power, we could make arrangements to supply it.

We are on-track for our first servers moving in mid-late January. The next update should include racks and wiring, with fiber optics close behind. Stay tuned!

After almost seven years in our Bothell facility, digital.forest is moving to a new location - we are headed back to Seattle in early 2005. In late 2003 we recognized that we were beginning to reach the limits of what our Bothell facility could physically support without significant investment - we looked at expansion here, while also investigating possible alternative locations. What we found last summer was an opportunity we could not pass up, and now after months of negotiation, preparation, and just plain hard work, we are ready to share the news with you, our clients.

digital.forest's new home will be in Seattle's Intergate.West technology campus. Our new datacenter in Seattle is truly top-tier. It is a $5million dollar build-out with features such as:

• 3x the datacenter floor space of our current facility
• 14' ceilings
• Zone 3 Critical Seismic Construction
• 180kW MGE UPS (3x our current capacity)
• Twin 75kVA Power Distribution Units (3x our current capacity)
• 75 tons of HVAC (cooling - over 3x our current capacity)
• 1.25MW Diesel generator (8.3x our current capacity)
• 5000 gallon Diesel fuel tank (16.6x our current capacity)
• FM200 fire suppression
• Locking server cabinets
• High quality overhead ladder racking with dual levels for network and power

Plus all the things we have delivered to you from our current facility, such as:

• Redundant fiber-optic paths in and out of the building
• BGP4 connectivity to multiple major Internet backbone providers
• Secure facility, with cameras and staff escorted visits
• Onsite, knowledgeable support staff
• Interesting music-on-hold

This is a major move for us and something that will allow us to expand and grow through our second decade.

VERY LITTLE work will be required by you. NO IP ADDRESS CHANGES will be required, and downtimes will be minimal. In the coming days we will be posting details here in our support blog outlining the plan, and the progress. We will include pictures and updates as work continues. Specific schedules will be posted here for hosting and colocation server moves. For the latter group we will closely coordinate your server move, and will be in direct contact with colocation customers several weeks before your move date.

If you want to stay on top of this move, feel free to use the links here to subscribe your email or RSS reader to this page. You can also bookmark it and filter on the "Intergate.West Move" topic.

Here is a taste of things to come:

Above: digital.forest Developer Support Specialist Tom Kepler and Account Manager Thom Dimitriou tour the new facility.