****** SERVICE IMPACTING MAINTENANCE******

Tonight at 23:30 hrs PST treehouse.forest.net will be taken down for brief maintenance. The configuration changes made on Monday improved performance but not sufficiently to compensate for the increased load on the server. The mail services hosted on treehouse will be migrated to a higher performance environment to better serve our client's email needs.

The migration will require approximately 45 minutes of down time. During this period, connections to the server for sending or receiving mail will not be possible. In most cases, mail sent to you during this window will be resent by the originating server and delivered shortly after treehouse is brought back online.

digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

Treehouse was brought back online a short while ago. We expect to have a very brief downtime, lasting only a minute or two, later tonight, and then the system will be back to running normally; tomorrow we'll be able to gauge whether the changes have improved performance.

We are taking Treehouse down immediately to make some changes in an effort to improve performance. We expect downtime to be brief. We will post again as soon as it's back up.

The mail server treehouse.forest.net is experiencing some issues which are manifesting themselves as slow performance when checking or sending mail. We are aware of the issues and are taking steps to correct them as soon as possible. We will post an update when we have more information to share.

The project up on the roof for our new client is compete. To celebrate we held a champagne toast and short photo session at the end of the day. Congratulations to our new client and their team for a job well-done.

Previously we shared the preparations and groundwork for the installation of a new digital.forest client's equipment on our roof, and today we'll share how that installation went. As we revealed previously the equipment in question is a set of antennas, and those were craned up to the roof yesterday. The weather was cold and a bit wet, but thankfully calm and windless.

First up were the bases of the antennas themselves, which were bolted to the platforms; then the antenna mounts atop the bases:

Next came lifting the dishes themselves and attaching them to the mounts. This process required more time, but went very smoothly. The dishes were attached with a sling, then lifted to the roof, positioned, attached to the mounts in a vertical position, then dropped fully 90° for the remaining process, which involves installing the electronics required for their operation.

Above: digital.forest Facilities Manager Kevin Teker helping out.

Our new client is very pleased with how smoothly the installation went,. In fact the project's manger told us that this installation has been the smoothest he's done in a long time, largely due to the logistics and assistance provided by digital.forest and our contractors. They'll be here for another week or so to complete the assembly of their colocation cage, servers, and antenna turn-up.

Thursday Update: Spring weather has returned in time for the finish work.

In our last post we explained how digital.forest is assisting a new client with some unusual installation requirements, including adding some infrastructure to our roof. We showed some of the behind-the-scenes work that has taken place over the past several months and weeks leading up to this week's work, which is the client's installation.

Above: The two new platforms on the roof installed for our new client.

The client's equipment arrived this week in two large trailers, one enclosed, the other a flatbed. Like all things in life, there are tradeoffs that happen in terms of benefits. In the case of our facility it is located on some of the best real estate in the Seattle area for a datacenter. It is on a relatively steep hillside above the Duwamish River, out of any risk from flood, and according to the US Geological Survey Earthquake Hazards Program is located in one of the least-risk areas of the metropolitan area. But, in many ways due to the above, we don't have a loading dock capable of handling such large trucks. As a result, our Facilities Management staff have become very adept at off-loading items from large trucks with a heavy forklift:

Standard deliveries to our building usually come via the parking garage and then up the freight elevator. To add another challenge to this installation this client uses server cabinets which are quite a bit taller than average, and prior to their delivery our Facilities Manager created a 1:1 scale mock-up of them and tested clearances of various building doors and elevators to ensure that they would clear all obstacles on the way to the datacenter. As a result of these tests we learned that they would only be able to come into the building via the front door, and had to be carried on a special hand truck which could lean back on a second set of casters.

The whole process went seamlessly as we were prepared ahead of time to deal with the oddities and exceptions of this client's installation. Next came unloading the large flatbed, which contained more of the equipment for the rooftop installation which was scheduled for Tuesday morning by crane. Items needed to be unpacked and arranged for the "pick" by the crane as soon as it arrived the next morning. This meant some assembly and arrangement of items by the client's crew, assisted by d.f staff.

The final step was the unloading of the largest rooftop components:

In case it has been a mystery up to now, the equipment being placed on our roof by our new client are satellite dishes:

Stay tuned for more information as the project moves to the next step.

Several months ago, our roof played host for a few days to some unusual devices, at the request of a potential client with an unusual colocation need. These devices established several data points that were critical to this potential client: exact position of our building on our planet, and its orientation relative to true north. The results were promising and they have made the jump from potential, to current digital.forest client. In addition to some cage colocation space, this client requires access to the roof, and a connection between their equipment in the datacenter, and the equipment on the roof. Unlike many colocation facilities digital.forest is both flexible enough, and has the on-staff expertise to accommodate requests of this nature. Many months of hard work by the client, digital.forest, external engineers & contractors later, the project is coming together now.

Starting on April 1st the roof construction began, with the arrival of two purpose-built platforms. These had been fabricated by our contractor, MacDonald-Miller and were lifted to the roof by crane.

The weather did not cooperate, but despite the rain & snow the platforms were delivered exactly where designed to within a 64th of an inch.

It is amazing to watch these guys work, as they are able to maneuver and place large heavy objects, essentially on the "end of a string", with precision and grace.

In the intervening time, the platforms were welded in place, railings and ladders attached, and conduits run from them to the colocation space below.

Next up, is the arrival of the client's equipment, stay tuned.

The Internet as a whole has seen a large spike in SSH "Brute Force" attack traffic in the last several days, and we've seen it here at digital.forest as well. This is an automated attack where logins are attempted repeatedly via SSH using a list of usernames and passwords. This is the network equivalent of a burglar walking through an apartment building turning doorknobs, or more accurately a big key ring of random keys, looking for apartments into which they can gain access.

This sort of thing goes on all the time, which is why we always recommend locking down your servers here to the minimum number of user accounts via SSH and have very strong passwords on those accounts.

What is different about this latest wave is the intensity and scale of the attacks. They are far larger, and far more persistent than most previous ones. Some servers are being overwhelmed with attack traffic, to the point they become very slow, or even unresponsive to normal traffic.

If you haven't already take some precautions and lock down your servers. Limit SSH to specific external addresses, limit accounts accessible via a direct SSH login, make sure your passwords are strong. As always, make sure your systems are up to date with their security patches and updates.

date.forest.net is coming down immediately for emergency maintenance. Duration should be less than 10 minutes.
UPDATE 5:02PM: date.forest.net should be operating normally. As always, if you have any questions or issues, please contact us.

We are making some changes to how email flows outbound from our mail servers. It is now necessary for digital.forest to implement OUTBOUND SPAM FILTERING. To date this has not been required, since as a rule our network is not a source of spam. Lately though we have had some problems with our servers being blocked by other mail providers. In one case this was directly attributed to a client's home computer being compromised and used to send spam. More often however it is caused by our clients setting their mail accounts here to automatically forward to another mail provider. If their account receives spam here, and forwards it, to the remote server it appears as if we are the spam source.

We have warned our clients about this many times, and even have temporarily implemented outbound filtering before; now we must take this step to protect all of our users.

We have actually been running the outbound filtering system for the past several days, but not blocking any mail, just collecting data in order to make a judgement on what needs to be blocked. This does introduce some latency in the mail delivery process, but so far we've been able to keep this delay to usually less than 5 seconds. After analyzing the data collected from around half a million messages, and load testing the filtering system with all of our production mail servers, we have a very good idea of both where the blocking thresholds need to be set to start, and what levels of latency are acceptable. Effective immediately we will start blocking messages at a relatively high "spam score" threshold of "20" and gauge the impact of that action on the mail servers, the filtering systems, and our ability to deliver mail to domains that consistently give us problems (Comcast, Hotmail/MSN, Shaw, Yahoo, etc.) If all goes well we will slowly lower the thresholds until we achieve a balance of the right scores and performance.

The ultimate goal is to increase the reliability of mail delivery, and prevent our servers from being blacklisted by other mail providers. If you have any questions or mail-related problems that you believe are attributed to this change, please contact our Technical Support group via a trouble ticket.

At some point in the last day and a half, a client of digital.forest had their home computer compromised by some sort of malware which then used their logon credentials on one of our mail servers to send a large amount of "spam". We have been able to stop the flow of spam, and assist the client in changing passwords, but the result of the brief outbreak is that one of our main mailservers has been "blacklisted" (that is blocked from sending mail) by many other providers. These providers include AOL, Comcast, Hotmail, Juno Roadrunner, Shaw, and Yahoo. We are working with these providers to have the blocks removed as soon as possible, though this process may require several days to resolve completely. Additionally, we have re-routed most of our outgoing mail to not use the server that was affected. We appreciate your patience while we sort this situation out.

In the meantime, there are things that you can to to mitigate the risk of these sorts of blocks. Namely do not configure your mail here at digital.forest to automatically forward to another provider. The mechanism of auto-forwarding re-writes the "envelope" of email in such a way to make the mail appear to originate at digital.forest. If your account here receives spam from outside, and it automatically forwards to your account at Yahoo or Comcast, to them it appears as if the spam came from us. Frequently the error is compounded further if you then use that provider's "report as spam" webmail feature. Finally, please maintain the security of your home and office computers, and the integrity of your passwords, as their compromise can have widespread implications.

The domain name registrar Register.com has been experiencing a multi-hour outage of their DNS servers today. Clients who rely on Register.com for hosting their DNS may experience issues with anything that relies on DNS, meaning email, web, etc.

No digital.forest services are affected by this outage, but we know that some of our clients use Register.com, and to them it may appear that their data here is offline. That is certainly not the case.

At this time we do not have any more information about the cause, or the expected time of resolution to Regsiter.com's issues. So far Register.com has made no statements and has no notice on their website about the issues. We are aware of them through our constant monitoring of the Internet's operational communications channels. If we hear more definitive information we'll post it here on the support blog.

Update: 4:30 AM PDT At this time it appears Register.com is fully back online.