Time for an update on our ongoing datacenter expansion project.

The new area is called "Datacenter Three" and is adjacent to our previous expansion, creatively named "Datacenter Two". It shares many attributes from the previous expansion, such as a common overhead cold air plenum, and a static-free, non-raised floor environment. We much prefer solid floors to raised, as they provide a far more ideal environment for today's high-density server installations. We are innovating in other ways for this expansion which will be revealed as we grow, for now though our work is fairly basic:

Here you see our electricians have begun with the power delivery for Datacenter Three. On the wall is the new main panel which is connected to the UPS (Uninterruptible Power Supply) room at the other end of the building. On the pallet-jack is the panel itself, being readied for installation. The innovation comes with our chosen method of power distribution from this new panel, stay tuned for more details on that.

Here is the northern half of our UPS room, being readied for the new UPS which is arriving soon. The technician shown here is relocating the FM200 clean agent fire suppression system to make room for the new battery cabinets. Our new UPS should be arriving shortly and has the capability of more than doubling our deliverable power to the datacenter. Unlike older UPS technologies this system is flexible in the amount of power it can deliver. When it arrives it will be configured for 300kVA. Later as we grow we can add capacity to it with additional batteries and software to raise it up to 500kVA.

Just like last time we'll be bringing the new UPS in via the roof and we'll document the process here on the support blog. Stay tuned for that, and other interesting updates from our expanding facility.

Regards,
--Chuck Goolsbee
VP Technical Operations
digital.forest

UPDATE 2/23/08 1:29 PM PST Work is complete and the circuit is stable again. We will now bring our BGP peer back up to restore full redundancy at our border.

This morning at approximately 10:00 am the transport provider to one of our BGP peers starting experiencing intermittent connectivity issues. They have narrowed the problem down to a bad card in one of their switches in the Westin building.

They have a tech en route and should have this resolved in a few hours. In order to minimize customer impact while they work we have shut down the affected BGP session. Our other peers will take up the slack in the mean time.

No, that is not Rodin's Thinker it is our fiber optic testing contractor running a certification check of the new fiber circuit we created for a customer yesterday. Clients of digital.forest not only benefit from our well-managed BGP-meshed network, they can also choose to directly connect to bandwidth providers here in the building, and in the Intergate.Seattle datacenter campus. In the business this is what is called being "carrier-neutral", so unlike for example an AT&T facility who only provides AT&T bandwidth, digital.forest clients have choice.

In this case our customer bought a small circuit from InterNAP, who has a network point of presence across the campus from us. We provided the connection to the campus fiber network and did end-to-end testing of the circuit after it was complete.

Very soon we will be formally announcing our own point of presence at The Westin Building in downtown Seattle. This will allow our customers to provision low cost dedicated circuits between their equipment at digital.forest and the myriad of providers found at the Westin. Contact your digital.forest sales or account manager today, or watch our support blog for more information.

In our continued efforts to upgrade and improve our network we will be moving the GigE connections to one of our distribution switches to the new GigE modules in the core switches.

This maintenance will take place during our scheduled maintenance window on Thursday, Feb. 14th between 11:00 pm and midnight. The expected impact of this will be about 1 minute of downtime. This will affect servers in rows 11, 12 & 13 of Datacenter 1.

Over two-thirds of our Technical Support staff are out sick at the moment with some sort of mild illness making the rounds here in Seattle. As a result we are a tad shorthanded. Please accept our apologies for any delays is handling incoming calls and support tickets. We appreciate your patience!

Tonight we installed two new 16-port fiber optic cards in our network core. One of the two is pictured above - it is the one with the empty ports slotted between the 8-port fiber card above and the 48-port copper card in the middle. We've seen a sharp increase in clients requesting a fiber connection to our network, as well as more sophisticated connectivity such as a BGP routing with our AS combined with fail over protocols such as HSRP. Additionally we are part of the way through a project to seriously upgrade and expand our network; better external connectivity, and more connectivity options for our clients. These cards are a small, but important part of that project. We'll have more news and some exciting announcements in the coming weeks. Stay tuned!

Tonight during our scheduled maintenance window, the servers palm.forest.net and tangerine.forest.net will undergo RAM upgrades. Each server will be down for approximately 5 to 15 minutes between the hours of 11:00 PM and 1:00 AM Pacific Standard Time. Every effort will be made to minimize the servers' offline time.

Please note: Palm is a mail server. Email will be unavailable while the server is offline. Inbound mail will be spooled on sending servers are delivered after the maintenance interval. Palm does not relay outbound mail itself, it works with one of our outbound mail hubs here so this event should have no impact on outbound email.

Tangerine is a database server running mySQLv4. Dynamic content on several of our other hosting servers may be affected during the maintenance interval.

Performance of both servers should be improved after their upgrades.

This morning at 10:37 Pacific Standard Time, one of our network peers experienced an unusual event on their network. They related to us the following information: "A router in Salt Lake City dropped its routing tables, which caused a router in Chicago to lose several BGP sessions."

The event was felt here on our network as a sudden loss of traffic going out in their direction. It lasted about five minutes, as things returned to normal by 10:47 AM. The traffic appeared to shift to our other connections as they increased proportionally as the one decreased. Some clients may have noticed this in the form of resets on persistent connections such as VPNs. It was likely invisible to the average "web surfer" however.

We are still awaiting an official explanation from the NOC of the provider in question, but we felt it important to state what we know now so that you are aware. When we here more, we'll update this post.

We finally turned up the new circuit over the weekend. Friday night to be specific. Close observation over the weekend proved that the circuit was performing exactly as we had anticipated. At this time we're happy to declare the installation a success and start focussing on our next project. Stay tuned for news on that very soon.

Thanks for your patience as we completed this installation.