When Plan A won't work, you fall back on Plan B. What happens if that won't work?

It has been more than three weeks since our last mention of our new UPS. It has been a busy month, with a lot of glancing back into the mists of time and collective head scratching.

We love our building, we really do, but sometimes we wonder what the designers and builders were thinking back in those crazy "dot com boom" days. So first, a short history review: We are located in the largest datacenter campus in North America, Sabey's Intergate.Seattle complex. It was built in the heyday of the datacenter building boom between 1998 and 2001. The original tenants were Zama Networks, Netstream Inc. and Microsoft. Two of those companies vanished in 2001, I bet you can spot which two. We moved here back in early 2005. The building was purpose-built to be a datacenter and in many ways is ideal. The folks who built it had LOTS of money, and the best engineers and architects. The only thing they didn't have was time. Once in a while you look at things and realize how rush jobs can come back to haunt you... even indirectly.

Above: Panorama of our UPS room. On the extreme left is our existing UPS, a 180kW/225KVA MGE EPS6000. To the right of it is our wooden "wire frame" UPS mock-up, and in front of it some tape outlining the location of the battery cabinets. In the middle is the switching and bypass gear with all the conduit above for both the old and new UPS. In the background are our Facility Manager and a tech from our fire suppression systems vendor preparing for today's job. On the extreme right is the stairway and door into the UPS room from the main datacenter.

Our facility is on the top floor of the building and has a specifically-engineered UPS room, with a substantially reinforced floor to support the weight of many UPS'. We can place close to 60,000 lbs of equipment in there. Our new UPS weighs ~4500lbs. The two battery cabinets weigh in at ~3500lbs each. Plan A was to bring them up in the elevator but size and weight contraints, even of the larger of the two lifts here prevented that. Plan B was to crane them up to the deck outside the datacenter and roll them into the facility with industrial rollers and winch them up our purpose-built ramps. This is how the existing units arrived here when the facility was built, minus the ramps of course. Apparently they originally used jacks and it took all day. Our building owner, and their engineers ruled that out for structural reasons. The UPS room floor is more than sturdy enough to support them, but they worried about the point loads during the rolls across the regular floor. They admitted that it had been done once before, but without some way to distribute the point loads, rolling was out of the question. We could have built a superstructure of steel plate from the deck to the UPS room, but that route has two ninety-degree bends that would make the actual rolling part very tough if done on top of a structure. Then of course is the whole problem of getting the superstructure up into the datacenter as well! We now had to come up with Plan C. We spent more time with Structural Engineers, in fact the very same ones that originally designed the building and at one point, the Engineer Of Record said: "You know, we really didn't consider the problem of moving heavy equipment into and out of this room."

A shocking admission. But the legacy we have to manage. So our Facilities Manager and the Structural Engineers came up with not only a way to get heavy equipment in, but also a way to get it out: Through the roof.

Literally in the space of less than a week, we came up with the idea, prepared plans and structural drawings, fabricated parts, arranged a crane, scheduled a roofer, and got all the permissions and blessings required. That is a lot of work, many hurdles, and a few hoops jumped through in a short amount of time. Not only that but we ALSO effectively UNPAINTED ourselves out of the corner the original builders painted themselves into. We redesigned the placement of equipment in the UPS room to allow greater maintenance access, and more importantly the ability to move any single part in or out without impediment from the other equipment, or potential interruption of service!

We also have allowed ourselves room for larger UPS units than what could even be fit through the existing door. If we need or want larger units, say 500 or 750 KVA UPS gear, we can now get them into our facility.

---

Here is how we spent our day today:

Above: On the left is the curb and cap for our new access to the UPS room. The room's exhaust fan has been removed, and a tech from MacDonald-Miller is preparing the roof for work.

Above: Roofers have cut away the roofing material and insulation to expose the pan deck below. The hole is where the exhaust fan used to be. We positively pressurized the UPS room with our air handling equipment so that any small particles were forced out of the room. There was a strong wind coming out of that exhaust hole, but it prevented any insulation material from being pulled in.

Above: The Mac-Miller techs remove a section of pan decking. The UPS room is below. Facilities Manager Kevin Teker is assisting on the left. He's turning his head to keep the wind from inside the room out of his face. It is unusual to experience a pressurized room. Standing inside it, you have no sensation of pressure. Standing near the opening, like the tech on the ladder in the hole, you feel a mild breeze. Outside the hole, where Kevin and the tech running the saw are, it is like a blast of air. You can see the tech running the saw... his hair standing straight out, where the tech in the hole in unruffled. What this did was blow the debris of this job up and out, onto the roof, rather than down and in, putting equipment at risk. Access to the UPS room was limited to only when the no cutting or peeling was going on, otherwise the pressure would change enough to risk fine debris being pulled in. The big stuff is OK as it falls straight down and can be picked up. It is the lightweight, fine debris that you have to make sure goes out, rather than in.

Above: Kevin cleans up. Cleanliness is always a very important aspect of any work in and around an active datacenter. You must keep the work areas clean throughout the entire procedure.

Above: The view from inside the UPS room. This opening was specifically designed to be as large as possible. It is significantly larger than our new UPS. This will allow us to install larger UPS' if we need them. Here the Mac-Miller technicians prepare the edges of the opening for the installation of the curb.

Above: The Mac-Miller techs and the Roofer are almost done. The curb is installed, the cap is on, and the exhaust fan re-installed in the cap. The cap is insulated and water-proof, yet lightweight and removable by two people.

Above: Our new access door from inside. Tomorrow we'll open it up and drop the UPS and battery cabinets through it with a crane. Friday the electricians start the installation process.

Stay tuned for more updates soon.

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc

It looks as if our spam filtering service, postini, is experiencing some problems with their login page at the moment. If you get an error message when attempting to log in to postini, don't be alarmed, we will update the support blog when the situation has been resolved.

UPDATE 8/23 (8:00AM PST): As of this morning, postini logins seem to be working. If you have any further problems, please contact us at support.

Currently, one of our mail servers (palm) is having issues and clients may be unable to send out or receive mail. Our techs are working on the server, and we will update as soon as there's new information to report.

Apologies for the inconvenience.

EDIT 12:45 : The server appears to be back up and running. If you are still having problems please call technical support.

Tonight we will be making some configuration changes to our border routers in our continued efforts to provide the most reliable and secure network possible. These changes will be non-service affecting.

This work will be performed between 9:00 pm and midnight.

Tonight during our scheduled maintenance window we will be making a minor configuration change to one of our upstream connections. This may cause a brief interruption of the connection, however, our other upstreams will handle the traffic during this time.

This maintenance will occur between 11:00pm and midnight tonight.

Our server sol (sol.wwwnexus.com) is currently undergoing some emergency maintenance and is temporarily unavailable. We will keep this space updated with the latest as we work to get it back up ASAP.

Update 9:25AM PST: sol is back up and operational

We're currently experiencing intermittent outages on our date.forest.net lasso server and are working to correct them, we will update this blog entry when we have further news.

Our new Uninterruptable Power Supply (UPS) is a rather large heavy box, with heavy being the operative word. Our UPS room has a purpose-built, reinforced floor specifically to bear the weight of several UPS units. The problem to solve of course is getting the UPS into the room. It is heavy enough that the point loads on the standard floor need to be minimized. It is also large enough to make negotiating hallways and doors very difficult. The final obstacle is lifting it up to the raised special-load floor of the UPS room itself through a door.

This UPS is a twin to the one we have now, so we know it can be done. We have spoken to the people who installed the first one and we'll be trying to benefit from that experience. Instead of jacks, we opted for a set of ramps. The question then becomes clearance. We did the math. We did the math again. We made diagrams and plans. The calculations said it would work, but it would be VERY close. Our Facilities Manager Kevin Teker had the ramps fabricated and we set them up on blocks in place. With our margin of clearance so close, we measured again several times and it was just too close to be absolutely confident that once we started to move the UPS that it would actually fit.

Nothing left to do but build a model and make a test run.

Kevin and our summer intern Chris set about building a wooden frame that would exactly match the dimensions of our new UPS. We would then use it, plus some wooden blocks that would do stand-in duty for the industrial rollers used to move the UPS up the ramps to check the clearances.

Above you can see the wooden frame UPS model taking shape behind Kevin (standing) and Chris (the intern, sitting on the floor). In the foreground right are our steel ramps. This view is through the door that the UPS must travel.

We set up the ramps next, and place the model frame on them to check clearances. The ramps will remain semi-installed until we are done moving all the new equipment into the UPS room, then go into storage until next time we have to move something in or out. Kevin designed them to bolt into the steel floor sub-frame for lateral stability, and use wooden dunnage for vertical support.

Above Kevin checks the clearance of the top of the model in relation to the door frame. It matches our initial calculations to within a centimeter. The overall clearance however is just over two centimeters, so you can see why we wanted to confirm our calculations. In the inset on the right he re-checks the measurements against the already installed UPS.

Next we have to check on the clearances as we roll the model UPS up the ramps and onto the floor above.

Here you see Kevin Teker sliding the model from level on the datacenter floor, up onto the ramps and through the door onto the UPS room floor. Kyle Murray our Network Manager looks on. Our multi-faceted trigonometry problem, which worked on paper, also works in real life. As you can see, the tolerances are close, but the model moved through the door frame with enough clearance to have us resting easy and ready for the real thing.

If Kevin is happy, we're happy.

Next comes moving the real thing. After that comes the installation and turn up. The goal is 360,000 Watts of redundant power. Stay tuned for more updates.

Final Summary

It is 3:00 PM PDT on Monday, August 6, as of 1:15 PM PDT everything here at digital.forest is back to normal. At that time we brought up our connection with 'Network X' and it has been stable ever since. As promised I will provide a short timeline of events over the weekend, a summary of what we did to mitigate this attack, and what we have done to prevent future attacks from having a similar effect. This data, together with what we posted last week (below) should serve as a total recap of the entire event.

We have had varying levels of success working with our peer networks. One has been excellent, one has been awful, and the other just "OK"... for this reason I have chosen to make them anonymous using the "Network *" as a substitute for their names. The one network which provided excellent support is one we have been working with for many years. The other two are both up for contract renewal before the end of the year and the one we are very unhappy with, "Network X" is unlikely to keep our business. Please keep that in mind as you read the following.

The attack which started in the early morning hours Friday was best mitigated by identifying the attack source and destination, and configuring the routers in between to ignore that traffic. This requires coordination with the networks we connect to directly, and ideally the networks that connect to the attack sources directly. We started by making those configuration changes on our routers, then contacting our network peers and requesting thy make a similar configuration change. The NOC staff at "Network Y" were VERY responsive and immediately made the changes we requested. This is what allowed us to come back online at 7:56 AM PDT Friday morning. Half of the attack traffic was coming in via the connection with "Network X" and we could not get a positive response from their NOC. We had opened a trouble ticket with them, but nothing was done on that ticket through all of Friday. The circuit stayed down until 7:15 PM PDT on Saturday. Unfortunately when that circuit came back up, the attack resumed. We once again shut it down on our end. "Network X" stayed offline over the entire weekend.

Our Network Manager, Kyle Murray performed some forensic analysis on the data we collected during the attack. The attack seemed to be coming from a single IP address allocated to a company in New York which appears to have been out of business since 2004. The source address was likely a forgery, as the amount of traffic we saw coming inbound was impossible to generate with a single computer. It was also coming into our network over several network peers. This is what lead us to believe that in reality it was a distributed attack. The source network of that IP was being announced to the Internet routing table by a network we'll call "Network C". We contacted their NOC to let them know about the attack we were seeing which theoretically came from their network. They agreed to null route that address as well. Kyle was hoping to hear back from them today with more information, but they are based on the east coast and their Security Staff have already left for the day.

Today at 1:15 PM PDT we finally brought up our BGP connection with "Network X" and the attack has been completely blocked.

We have fingerprinted the attack profile and created an alarm that pages us if any traffic matches the behavior of this attack traffic. We are building some automated systems to detect and null route such traffic.

---

digital.forest was hit with a massive distributed denial of service (ddos) attack this morning. We are working with our network peers to mitigate this as much as possible. Please be patient while we dedicate all available resources to resist this attack.

Update 8:25am As of 7:30 am we have good connectivity with one of our network peers. Another is partially up, with attack traffic coming in, but at a lower volume than before. One network connection is still down. We are working with the NOC staff of all our external networks to resolve this issue as best we can.

Update 9:51am PDT The ddos against our network this morning has been stopped and as such our network has returned to normal operating status. All sites and servers should again be functioning normally and accessible. If you are still experiencing any difficulties getting onto your website or server please give us a call at our technical support line 877-720-0483 option 3.

Final Analysis and Time Line:
Some specific details are still under investigation, however we have a very good understanding of what happened early this morning and are prepared to share in general terms the following information.

* Around 3:45 AM PDT a Denial of Service attack, directed at a single IP address inside our network began. At first it was not very large.

* By 4:10 AM PDT it had grown large enough to set off alarms in our network monitoring systems. Emergency pages went out to NOC staff and our Network Manager.

* At 4:27 AM PDT we lost the BGP session with one of our network peers, we'll call them "Network X".

* BGP was reestablished with Network X at 4:28 AM PDT.

* 4:35 AM PDT Network Manager was awake and gathering data from d.f NetFlow server. Recognized the traffic patterns as a Denial of Service Attack. Was in contact with NOC staff on site at digital.forest.

* 4:40 AM PDT Discovered the target of the attack via NetFlow reports. DoS traffic now at 30,900 flows per second.

* 4:44 AM PDT added route to black hole DoS target to Boundary Router 1

* 4:59 AM PDT added route to black hole DoS target to Boundary Router 2

In past experience, this step has stopped every other attempted denail of service attack on our network. By telling the world that the target does not exist, the attack usually stops. What followed instead was more of the same. The attack continued, and in fact intensified.

* 5:10 AM PDT BGP with Network X goes down.

* 5:11 AM PDT BGP with Network X reestablished.

* 5:16 AM PDT BGP with Network X goes down.

* 5:17 AM PDT BGP with Network X reestablished.

"Blackholing" the attack target has had no effect. Our attempts to get attack source data from our NetFlow server is fruitless, it is unable to keep up with processing as flows begin to exceed 50,000 flows per second/3,000,000 flows per minute. If we can get source data, we can start making attempts to block the source, or work with our peer networks to block the attack.

* 5:40 AM PDT Boundary Router 1 goes non-responsive.

* 5:43 AM PDT On site tech restarts BR1 under direction from Network Manager.

* 5:52 AM PDT BR1 up again.

* 5:52 AM PDT BGP session with another provider we'll call "Network Y" is lost. This network is terminated on a separate router, Boundary Router 2.

* 5:53 AM PDT BGP with Network Y restored.

While we maintained BGP connectivity with one of our three providers ("Network Z") throughout the event, the attack traffic at times consumed 100% of the CPU of one, or both routers, causing such high latency that we were, for all intents and purposes, not passing traffic. At this time Network Manager calls the Vice President of Technical Operations and informs him of the situation. Ops VP starts calling technical support staff to have them get to the office and assist with telephone calls. Also informs the CEO and VP of Sales.

* 6:08 AM PDT Boundary Router 2 goes non-responsive and it restarted by on site staff.

* 6:12 AM PDT Boundary Router 2 is back up. Attack traffic has effectively blinded both routers. NetFlow server records over 70,000 flows per second/4.2million flows per minute before it goes non-responsive as well.

* 6:20 - 7:30 AM PDT Network Manager and Ops VP contacting the NOCs of peer networks to have them assist in DoS Mitigation. Network Manager logs trouble tickets with Network Y and related Metro Ethernet provider, proceeds to datacenter from home. Ops VP logs tickets with Network X and Network Z... (after much phone tree navigation... way too much with "Network X")

* 7:37 AM PDT Network Manager on site at d.f and making very good progress with NOC staff of Network Y.

* 7:56 AM PDT BGP session back up with Networks Y & Z. We are back "on the air" again, though down by one provider. Attack continues, but is being mitigated actively by Network Y. Network Z is up, steady and not included in the attack. Network X is still down. Most of the tech & customer service staff is on site, taking calls from customers.

* 8:03 AM PDT BGP with Network Y lost.

* 8:10 AM PDT BGP with Network Y restored.

* 8:50 AM PDT Ops VP leaves home headed for digital.forest.

* 8:58 AM PDT BGP with Network Y lost.

* 9:03 AM PDT BGP with Network Y restored.

* 9:06 AM PDT BGP with Network Y lost.

* 9:07 AM PDT BGP with Network Y restored.

* 9:10 AM PDT BGP with Network Y lost.

* 9:11 AM PDT BGP with Network Y restored. Attack now completely blocked. Traffic on Network Y stabilizes and returns to normal. Network Z has remained up and stable since 7:56 AM PDT. Network X still down.

* ~10:00 AM PDT Server which was the target of the attack is brought back online.

As of now, 11:55 PM PDT, Network X, the first of our network peers to be lost, is still down. We have been calling their NOC and have a trouble ticket logged. We strongly suspect that the port on their equipment we connect to in downtown Seattle has failed an auto-negotiation. Hopefully we'll have this resolved soon.

Our other circuits, Networks Y & Z are stable and handling all our traffic normally.

We would like to thank our clients from their patience and understanding during this event. We will continue to work on this issue with the intent of learning as much as we can. We have been subjected to denial of service attacks before, but in each of those cases we have been able to successfully mitigate them, usually before they had any noticeable impact on our network. This was the first attack on our network since December 23, 2001 that had more than a few minutes impact on our ability to stay online. We've spoken to a number of DoS Mitigation experts today, and will continue to do so. We've made some configuration changes and will continue to harden our infrastructure against attacks.

As always, if you have questions or concerns, feel free to contact us.

Regards,
--Chuck Goolsbee
V.P. Technical Operations
digital.forest, Inc.

As part of our changes to the FreeBSD hosting servers, Butternut will be offline late tonight for an upgrade to the Apache web server software. We estimate that the downtime will be less than 2 hours, during which the current configuration will be backed up, and the new configuration will be installed. There will be substantial modifications to the virtual host configuration files. In the vast majority of cases, everything should continue to function after the upgrade as it has until now. However, we ask that you test your sites thoroughly, and report to us as soon as possible if they find any problems so that we can correct them.