The Puget Sound area of western Washington awoke this morning to anywhere from a few inches to over a foot of snow. While this should have no impact at all on our network operations, it likely will have some impact on staffing levels at the office to handle your technical support, customer service, sales, and billing issues. Our region's transportation systems do not cope well with snowfall, so it may take some time, more time than usual, for people to get to work. Some of us may not make it in at all.

The storm is expected to end by tomorrow, so things should be back to normal soon. We appreciate your patience.

--Chuck Goolsbee
VP, Technical Operations
digital.forest
Seattle, WA

[UPDATE] Maintenance on the silverpine server has been completed.

At 5pm PDT we will be performing server maintenance on the silverpine shared hosting server. It should be down no more than an hour.

In observance of the Thanksgiving Holiday, digital.forest will be closed Thursday November 23rd and Friday November 24th. We will resume regular business hours at 8am PST Monday November 27th.

Technical Support staff will remain on-site 24 hours a day throughout the holiday period. Please be aware that we'll have limited staff coverage for telephone tech support during the next few days. Please note that our building will be locked throughout the weekend and clients requiring access to the datacenter to work on colocated servers will have to call or email first to be allowed access to the building. Additionally we will likely take advantage of the holiday lulls to perform maintenance and upgrades on core equipment. We will post notice of these beforehand. Finally, we are closing the phone support queue for a while on Wednesday, November 22nd in the afternoon for a Tech Department meeting. We appreciate your patience.

All of us at digital.forest wish you a happy Thanksgiving and a wonderful holiday season!

Chuck Goolsbee
VP, Tech Ops
digital.forest

For 6 hours today we tested a new technology for OUTBOUND mail filtering in an attempt to solve some of our mail deliverability issues. The theory is that other mail providers cause trouble making it hard for us to deliver your legitimate mail because they see enough "illegitimate" mail coming out of our network.

When operations like Yahoo!, AOL, Comcast, etc won't accept our mail, you complain to us. So we're trying to solve the problem. Here is what we do know:

1. We have far too many clients who have their domains here configured to FORWARD ALL MAIL to a non-d.f network mail address (Comcast, AOL, Yahoo!, etc)

2. We have far too many clients who have exploitable mail forms on their website and get hit by spammers. Daily.

#1 causes problems when that address receives spam from outside our network. When our servers forward it, they get tagged as the "spam source". This leads to us being greylisted, blacklisted, which leads to deliverability problems or bounces. In the past we were able to mitigate this by the old fashioned method of talking to the other providers and letting them know that we share customers - that we host their webservers and domains, and they used "you" (Comcast, Shaw, Verizon, whatever) for e-mail. This has worked well in the past, but this latest wave of spam on a global scale has made talking to people very difficult... the large providers are too busy talking to their own customers complaining about spam to take time to talk to us.

#2 is just a HUGE game of "whack-a-mole" and to be honest, we're tired of whacking. As much as we love our clients, we know that many of them haven't written their website code... but have instead used off-the-shelf PHP, or ASP, or complete CMS systems that they have NO IDEA of what is going on behind the scenes. Our past stance of "we'll disable all your mail scripts until you fix them" is not working. Too many of you just tell us it has been fixed when in reality, it isn't. Or the vulnerability is still there. We have always suspected that some percentage of the outbound mail from our netblock is "spammy" but what happened today shocked us.

Below is some sample data from a new device we are testing, which acts essentially in reverse to the Postini service we currently use to filter inbound mail. It filters OUTBOUND mail. Early today we noted a large outbreak of spam queued on one of our mail servers, specifically the one that acts as the mail relay for our hosting servers. That is when we turned on this new device. Within minutes, it stopped over TWENTY THOUSAND mails being relayed off a single client's web form. Once that ended, look and see what happened over the subsequent several hours...

On average, 50% of the mail OUTBOUND from two of our mailservers has the smell of canned meat about it. I find that disturbing. We don't host spammers, at least not knowingly. Can that much be drive-by form exploits and forwards?

We suspended the outbound filtering around 4:30 PM. We discovered that all our previous work of ensuring our servers were "whitelisted" with peer networks had not been completed with this new filtering device. We will spend the next few days getting those issues settled and turn it back on.

The ultimate goal here to ensure that we are being good network neighbors and NOT allowing junk out of our network. The result of that will be better and more reliable delivery of mail. Here is what you can do to help:

1. If you forward mail, stop. The risks far outweigh the benefits. Just configure your mail program to pick up mail on our servers via POP or IMAP. Virtually ALL mail software these days supports multiple accounts so the need to forward is no longer valid.

2. If you have forms on your website that send mail, do everything in your power to ensure that they cannot be abused. We're more than happy to assist you or your developers in this task. Please do not ignore this because it will not go away. It taints your domain name and causes severe damage to your reputation.

We appreciate you patience while we continually take steps to improve our service, and maintain one of the best hosting and colocation environments on the Internet.

Chuck Goolsbee
VP, Tech Ops,
digital.forest
Seattle, WA

The server www.ninewire.com is currently experiencing hardware problems. We are working on the server and will have the issues resolved as soon as possible.

Sorry for the wait everyone.

Celestial is back up and running. All webpages should be working again.

We've been experiencing problems with our "Celestial" server for the past few days, and today it's had to come down for hardware maintenance. Any user accounts hosted on celestial will therefore be unavailable until the maintenance is completed. Our apologies for the inconvenience, we'll try and have it back up as soon as possible.