As we've previously announced, we are performing biannual maintenance on our UPS system today, as well as performing some monitoring system upgrades on our older UPS systems. This involves taking the UPS system offline for portions of the work. While this is happening we'll have our backup power systems running to carry our load. We'll post more info as the work progresses.

Update - 12:10 PM PDT: The maintenance work is complete.

We've experienced ourselves, and have had some reports from our clients of a large amount of "backscatter" coming into our mail system. Backscatter is made up of bounced mail notifications, but not from any mail that you might have sent. It is from spam which has your mail address used as the sender, ie: in the "From: " header.

This is a technique used by spammers to mask their identity and increase their odds of successful delivery by using an address that is "real". The term for this spamming technique is a "Joe Job" (specifically in that wiki entry see the section titled "Joe-job-like automated spam".)

I'll delve into more technical aspects of this later, but the most important question I need to answer now is: "Can digital.forest make it stop?"

The short answer is unfortunately "no", mostly because we can not control the behavior of spammers. However there may be some things we can do to minimize the annoyance. That requires some consideration of unintended consequences though as it attacks symptoms, not causes. Let's break down the process into simple steps and examine what we can do and the results that would come of it:

1. The spammer sends out mail with your address as the sender.
Nothing can be done to prevent this unfortunately. Just as anyone can use your physical address on a piece of paper mail as the return, the same applies to e-mail. How did they get your address? Any number of ways: published in WHOIS records, harvested from the web, harvested from mailing lists, harvested through Microsoft Outlook viruses, etc, etc.

Further these mails are NOT being sent from digital.forest's servers. They are being generated and relayed from thousands of compromised hosts (usually infected Windows desktops) on broadband networks all over the globe. These computers are referred to as "zombies" or "bots" in the network security world, and are literally numbered in the hundreds of millions (called "botnets".)

There have been some technologies proposed, and some even partially adopted, to put some sort of check into the mail process that verifies that the sender is the actual sender. "SenderID", Sender Policy Framework (SPF), etc. We can implement some or all of these, but it would only serve to reduce the percentage of mail that bounces by a small amount, as these solutions are far from universally deployed, or even agreed upon.

2. The mail recipient's server accepts or rejects the spam, but then bounces it, or sends a challenge-response, or other auto-reply.
Again, this is something digital.forest has no control over. If these servers recognize the incoming message as spam, then they should not bounce it. It should be just ignored, filtered, discarded, etc. There used to be a school of thought that bouncing, rejecting, or sending a challenge-response would somehow convince the spammer to not send you any more mail. The reality is that the spammer doesn't care. In fact they have masked the true source and are redirecting these bounces, rejections, etc elsewhere! Unfortunately some percentage of the mail servers, and mail operators still want to bounce or reject spam, so all these bounces, rejections, challenge-response notices, etc go flooding back towards the supposed sender.

3. Here come the bounces, right at you!
So here is the point at which we can do something, because this is the first time digital.forest systems are directly involved. Unfortunately as I said earlier this is attacking a symptom, not the root cause, and will have unintended consequences. We can create server-side filters to discard bounce messages. Like SPF, this will only cut down on the backscatter by some percentage because not all bounces are crafted the same. Additionally much of the backscatter is not bounces, but various sorts of auto-replies, vacation messages, out-of-the-office notices, and challenge-response systems. Even if we filter, we can't stop them all. If we do filter, the consequence will be that you will not be notified if your legitimate sent mail has bounced. If mail you send does not reach the person you sent it to, you want to be notified. So we're stuck between the proverbial rock and hard place. If you choose to start filtering bounces, you can - usually by writing a rule on the mail server that DISCARDS (NOT rejects!) mail with a return-path of "<>" - a common bounce attribute. If you need some help with this process you can contact technical support or submit a trouble ticket and we can assist you. Just keep in mind the potential consequences of this action.

The good news is that these automated joe jobs rarely go on for very long, as the spammer needs to constantly cycle through senders to mask their identity. The backscatter should stop somewhere around 5-7 days. I realize that is small consolation, but please know that we are right there with you. Our long-published email addresses (like abuse@forest.net, support@forest.net, and many of the personal addresses like mine that have been in operation for the lifetime of digital.forest) can experience large volumes of backscatter, several thousand messages per hour. If it were within our power to stop these, we certainly would.

Regards,
Chuck Goolsbee
VP Technical Operations
digital.forest, Inc

The UPS maintenance originally announced for April 25th has been rescheduled to Wednesday, April 30th

In other maintenance news, we took advantage of some cold early morning temperatures to do some coil cleaning and belt changes on our original HVAC unit, known as 'AC1'. Yesterday we cleaned the interior condenser coils, and this morning we cleaned the exterior evaporator coils and changed some belts. Keeping these coils clean is a vital part of maintaining our HVAC systems. Clean coils are efficient coils. The condenser coils can be cleaned without interruption of the system's operations. The evaporator coils required shutting down the AC1 for about 5 minutes at a time. Since we have three main AC units and one reserve unit, there was negligible operational impact on the datacenter. Belts are in constant duty cycle, much like in your car, and require changing at regular intervals.

Below: A view of AC1's interior condenser coils in the middle of their "Spring Cleaning." Usually these are not visible, as they are hidden behind doors.

Our new Fiber Optic installation was terminated this afternoon. The contractor will be back tonight around 11 pm to test the circuit. We hope to have this circuit up and running by the end of the week.

In other connectivity news, we have a new Gigabit Ethernet connection arriving soon for a BGP session with another major provider. It was originally scheduled for January, but has been delayed by fiber and power issues at another location. That circuit may also be up and running by the end of this week. We'll share more details about that as the turn-up approaches.

We have a contractor here today pulling a new fiber optic connection into the building and our datacenter. Initially we'll be using this new fiber for a dedicated connection to the Westin Building in downtown Seattle. Over this connection we'll connect our network to the Seattle Internet eXchange aka "The SIX"... we were SIX members when we were located in Bothell and are looking forward to the peering opportunities there again now that we've settled into our new facility in Seattle. More importantly however, this circuit will allow relatively inexpensive direct cross-connects for our valued clients who are seeking low-to-moderate bandwidth (10mb-300mb) at the Westin. Acquiring high-bandwidth (1Gb+) at our location is very cost effective, but the backhaul and loop costs can be prohibitive for smaller scale purchases. We're seeking to remedy that via this installation and provide our clients with more and better choices for direct connectivity.

Installing the fiber meant running a new 24-strand bundle from the vault out in front of our building up to the network core in our datacenter six floors above. The contractors arrived and found the vaults with a bit of water in them, not surprising due to our rainy climate. They ran a pump and removed the water so as to make working in the vaults less... wet. The water was pumped out onto the parking lot, where it ran down into a nearby storm drain.

Above: Looking down into one of the two fiber vaults in front of the building. Just a bit of water down there. The vaults are designed in such a way to keep the fiber conduits above the level of pooled water, but even so, the fiber itself is well-protected. Each strand is insulated and the whole bundle is encased in a weatherproof jacket seal, and then the bundles are run through plastic "innerduct".

Above: A view of the work from a balcony on our floor. The grassy area and shoulder of Tukwila International Blvd (SR 99) at the top of the frame is where virtually all the fiber optics that run southbound out of the Seattle metropolitan area located. This is the principal reason why the Intergate.Seattle datacenter campus was built here.

Above: Kevin from digital.forest holds the ladder while Kevin from the cable contractor opens up the fiber junction box at the top of the conduit run from the vault six floors below.

Above: Preparing to run the innerduct.

Above: Cable Installer pulling the innerduct through to the network core. He is standing up on our ladder racking, which is about eight feet (2.5m) above the floor. All our previous fiber optic cable installations are on the left. If it comes from outside the building it arrives wrapped in innerduct. If it comes from within the building it is inside a simple jacket. We have some multi-pair bundles, as well as a few single-pair runs to other parts of the datacenter (usually for storage area networks.) All of our connectivity from the core out to the datacenter for IP networking is on the right hand side. Don't worry, no cables are ever stepped on, and we rarely climb up on the racking.

Above: A view down to the parking lot where the first pull from ground level is going up.

Above: The fiber bundle has arrived at the junction box (you can see the cable pulling harness hanging out of the innerduct to the left of the installer. He is on a radio to the installer at the other end of each innerduct. They pulled the full length of slack to here next, then made the final pull to the network core.

Above: Starting the last pull. They use cloth tape, which is pre-installed inside the innerduct to pull the cable itself though.

Once the pull is complete they leave large slack loops at either end. On Monday another team will come and terminate the fiber into a panel here in the datacenter, and splice the other end down in the vault. You can see three Fiber Termination Panels just to the left of the installer's head. Another one of these will appear Monday.

The final step was tying down the innerduct to the ladder rack and labeling the install...

Stay tuned for an update on Monday evening.

On April 25th, 2008 we will be performing an upgrade on portions of our UPS system. These will bring these units up to a similar spec to our new UPS.

Anytime we perform UPS maintenance we put the UPS system in bypass, meaning that we run the facility on our own generator rather than the electrical grid. Anytime there is a transfer of power from one source to another, there is a risk of interruption. The risk is very small, and we will take every measure to minimize and eliminate that risk. Briefly, here is the process:

* Manually start the generator.

* Manually transfer the electrical load from grid to generator.
(The interruption here is measured in milliseconds, unlike a power failure, which usually lasts 7-10 seconds as the generator starts.)

* Manually put the UPS system in bypass mode.
(This routes power directly from the main power panel through to the servers, removing the UPS from the flow. This is a two-step process where the bypass bus is energized before the UPS is bypassed. This means there is no interruption.)

* Perform the required maintenance.
(This will take several hours.)

* Put the UPS system back on-line.

* Manually transfer from generator to grid power.

* Shut down the generator after a cool-down period.

We perform this procedure several times a year, so this is fairly routine. We also have over a week's supply of fuel for the generator, so runtime is not an issue. We just like to keep our clients informed of happenings here at the facility. As per usual we'll document the process here on the support blog so you can see what we're up to. We'll post reminders as the maintenance day approaches.