The former Trident Networks/Speedyweb server "Neptune" has been experiencing issues lately. In order to ensure future stability and performance of the sites served from this machine we've decided to migrate them to more reliable servers. Accounts will be moved to newer and faster servers, either running UNIX or Windows, depending on the if the website relies on FrontPage extensions. E-mail and any MySQL databases will be migrated to UNIX servers.

We apologize for any inconvenience that this migration may cause you and will be more then happy to answer any questions you may have; we will be working with you to resolve any issues that may arise due to this migration.

Users on Neptune will be contacted directly via our helpdesk ticketing system with more details. Please respond to the helpdesk ticket and/or call us at 877-720-0483 option #3. We will have staff onsite 24 hours a day, 7 days a week during this migration and they will be able to help you with any problems that you may have. If you believe that your e-mail address with us may be out of date we highly recommend that you respond to this ticket or call us at 877-720-0483 option #2 during business hours and update your contact information with an Account Manager.

Thank you for your patience.

We've experienced ourselves, and have had some reports from our clients of a large amount of "backscatter" coming into our mail system. Backscatter is made up of bounced mail notifications, but not from any mail that you might have sent. It is from spam which has your mail address used as the sender, ie: in the "From: " header.

This is a technique used by spammers to mask their identity and increase their odds of successful delivery by using an address that is "real". The term for this spamming technique is a "Joe Job" (specifically in that wiki entry see the section titled "Joe-job-like automated spam".)

I'll delve into more technical aspects of this later, but the most important question I need to answer now is: "Can digital.forest make it stop?"

The short answer is unfortunately "no", mostly because we can not control the behavior of spammers. However there may be some things we can do to minimize the annoyance. That requires some consideration of unintended consequences though as it attacks symptoms, not causes. Let's break down the process into simple steps and examine what we can do and the results that would come of it:

1. The spammer sends out mail with your address as the sender.
Nothing can be done to prevent this unfortunately. Just as anyone can use your physical address on a piece of paper mail as the return, the same applies to e-mail. How did they get your address? Any number of ways: published in WHOIS records, harvested from the web, harvested from mailing lists, harvested through Microsoft Outlook viruses, etc, etc.

Further these mails are NOT being sent from digital.forest's servers. They are being generated and relayed from thousands of compromised hosts (usually infected Windows desktops) on broadband networks all over the globe. These computers are referred to as "zombies" or "bots" in the network security world, and are literally numbered in the hundreds of millions (called "botnets".)

There have been some technologies proposed, and some even partially adopted, to put some sort of check into the mail process that verifies that the sender is the actual sender. "SenderID", Sender Policy Framework (SPF), etc. We can implement some or all of these, but it would only serve to reduce the percentage of mail that bounces by a small amount, as these solutions are far from universally deployed, or even agreed upon.

2. The mail recipient's server accepts or rejects the spam, but then bounces it, or sends a challenge-response, or other auto-reply.
Again, this is something digital.forest has no control over. If these servers recognize the incoming message as spam, then they should not bounce it. It should be just ignored, filtered, discarded, etc. There used to be a school of thought that bouncing, rejecting, or sending a challenge-response would somehow convince the spammer to not send you any more mail. The reality is that the spammer doesn't care. In fact they have masked the true source and are redirecting these bounces, rejections, etc elsewhere! Unfortunately some percentage of the mail servers, and mail operators still want to bounce or reject spam, so all these bounces, rejections, challenge-response notices, etc go flooding back towards the supposed sender.

3. Here come the bounces, right at you!
So here is the point at which we can do something, because this is the first time digital.forest systems are directly involved. Unfortunately as I said earlier this is attacking a symptom, not the root cause, and will have unintended consequences. We can create server-side filters to discard bounce messages. Like SPF, this will only cut down on the backscatter by some percentage because not all bounces are crafted the same. Additionally much of the backscatter is not bounces, but various sorts of auto-replies, vacation messages, out-of-the-office notices, and challenge-response systems. Even if we filter, we can't stop them all. If we do filter, the consequence will be that you will not be notified if your legitimate sent mail has bounced. If mail you send does not reach the person you sent it to, you want to be notified. So we're stuck between the proverbial rock and hard place. If you choose to start filtering bounces, you can - usually by writing a rule on the mail server that DISCARDS (NOT rejects!) mail with a return-path of "<>" - a common bounce attribute. If you need some help with this process you can contact technical support or submit a trouble ticket and we can assist you. Just keep in mind the potential consequences of this action.

The good news is that these automated joe jobs rarely go on for very long, as the spammer needs to constantly cycle through senders to mask their identity. The backscatter should stop somewhere around 5-7 days. I realize that is small consolation, but please know that we are right there with you. Our long-published email addresses (like abuse@forest.net, support@forest.net, and many of the personal addresses like mine that have been in operation for the lifetime of digital.forest) can experience large volumes of backscatter, several thousand messages per hour. If it were within our power to stop these, we certainly would.

Regards,
Chuck Goolsbee
VP Technical Operations
digital.forest, Inc

Tonight during our scheduled maintenance window, the servers palm.forest.net and tangerine.forest.net will undergo RAM upgrades. Each server will be down for approximately 5 to 15 minutes between the hours of 11:00 PM and 1:00 AM Pacific Standard Time. Every effort will be made to minimize the servers' offline time.

Please note: Palm is a mail server. Email will be unavailable while the server is offline. Inbound mail will be spooled on sending servers are delivered after the maintenance interval. Palm does not relay outbound mail itself, it works with one of our outbound mail hubs here so this event should have no impact on outbound email.

Tangerine is a database server running mySQLv4. Dynamic content on several of our other hosting servers may be affected during the maintenance interval.

Performance of both servers should be improved after their upgrades.

The maintenance has been completed and the server is back online.

We are taking smtp.forest.net down again tonight to deal with problems identified by last night's maintenance. The server will be up again as soon as possible.

One of our mail servers, smtp.forest.net, is down to investigate recurring problems. We hope to have the server back up within an hour to ninety minutes. Thanks for your patience.

Update (4:38AM PST): smtp.forest.net is back up and running.

Treehouse's new memory modules have been installed, and the server is back up and running.

At 2am tomorrow morning we will be shutting down the mail server "treehouse" to install new memory in it. We have been seeing RAM-related errors which caused the server some problems last week. We figured tonight would be a good time to bring it down and perform the hardware installation. Downtime should be limited to 15 minutes or less.

One of our mail servers, treehouse.forest.net, is experiencing problems and is currently down. We are working to restore it as soon as possible.

Thanks for your patience.

Update 11:25AM: treehouse is back up and running.

Final Report 12:30PM: Treehouse was experiencing memory-related errors and when we rebooted it the hardware self-test showed a failed RAM card. This particular server uses RAM in pairs so we had to source a pair of equivalent cards from our inventory to get the mail server running again. We have ordered additional RAM for both treehouse and our inventory and will likely schedule a brief downtime for treehouse over the holidays to perform this work.

We've noted an unusual network issue affecting Comcast customers in the Pacific Northwest. Packets coming into and out of our network are taking unusual paths to get to Comcast users, frequently via California, which is adding latency and occasionally causing timeouts when accessing services on our network.

The issue started some time late yesterday. We're keeping an eye on the situation and will update this notice when we have any further information.

Currently, one of our mail servers (palm) is having issues and clients may be unable to send out or receive mail. Our techs are working on the server, and we will update as soon as there's new information to report.

Apologies for the inconvenience.

EDIT 12:45 : The server appears to be back up and running. If you are still having problems please call technical support.

One of our mail servers, smtp.forest.net, was tagged as a spam source yesterday when a spammer abused a latent mail form on a website inside our network. Here are the steps we've taken and will continue to take to resolve the issue as soon as possible:

* We've removed the web-to-mail form that was targeted.
* We've dumped the offending messages from the outgoing SMTP queues.
* We've re-routed outbound mail via another SMTP relay host for the time being.
* We're doing our best to get our server removed from the various blacklists.

This serves as a timely reminder to everyone that manages a website to have a look through all your code and make sure that all your forms are validated and protected. You can read more about that here. Even pages that have no active links to them are vulnerable! So clear out the deadwood folks, because with Google and other search robots, no HTML stone is left unturned these days. The form that was abused recently was a "dead link" that had not been active since 2002 or so, but was still there. Perhaps it is time for a little "spring cleaning" for webmasters.

Regards,
Chuck Goolsbee
VP, Tech Ops,
digital.forest

Update 4/2 12:21PM
We have successfully petitioned Comcast to take our mailserver off of their block list. If you're still having issues sending to Comcast, feel free to submit a trouble ticket or contact support toll-free at (877) 720-0483

A number of clients have contacted us regarding failed mail deliveries to Comcast accounts. If you're having trouble sending to Comcast, and you get a message that looks like this:

Failed to deliver to 'xxxxxxx@comcast.net'
SMTP module(domain comcast.net) reports:
return-path address 550 216.168.37.34 blocked by ldap:ou=rblmx,dc=comcast,dc=net -> BL003 Blocked for spam. Please see http://www.comcast.net/help/faq/index.jsp?faq=SecurityMail_

be advised that we are aware of the issue, and are currently attempting to work with Comcast to resolve it as quickly as possible.

Updates will be posted to this page.

Tonight, a little before midnight, a client created a forwarding mail loop, back to themselves via an external address. A single message queued on the server "treehouse" looped out via our mailhub, to an external mail address, which then looped back via postini, and into treehouse. You will note that this loop is asynchronous, which prevented the built-in mail loop detection features from stopping it.

Within seconds, this loop started clogging our outbound SMTP queues, and it was finally detected by our monitoring systems as the disks of our mail servers began to fill.

It took us well over an hour to get this under control, and required us to stop processing mail for several minutes at a time. As it was a forwarding loop, the message that looped grew in size every time it looped and so we had tens of thousands of looped messages each queued on multiple servers here. We were able to delete them from the SMTP queue on treehouse, but not on our outbound spam/virus filtering mail hub (due to limitations of that device's software.)

The loop will have lingering effects, and we're taking the following steps to mitigate them:

* We have removed the filtering mail hub out of its primary task of handling all of our outbound mail. It will take it some time to unload its outbound and inbound(bounces) queue.

* We have configured our mail servers to relay mail directly outbound. This will slow normal delivery as we can not filter outbound, and any pollution of the mail stream with spam (usually via forwards to external addresses) may cause remote servers to temporarily reject our mail via "greylisting".

* We have scoured the queues for copies of this looping message and deleted them. Some are inevitably still "out there" on external servers, so we have created filters to reject them.

* We will contact the client who created the mail loop and explain to them how NOT to do that in the future.

We apologize for any inconvenience this may cause you. NO MAIL WAS LOST during this event, but we do expect that delivery will be delayed throughout the day today as queues clear out. There is no way for us to prioritize some mail over others as it will really be up to the willingness of remote servers to accept our mail in a timely fashion, as the queues for external domains rotate to the head of the line.

In the "good old days" before the spam problem, this issue was solved via automated technical means, but now the ubiquitous deployment of spam filtering technologies has complicated the environment significantly. We took every possible measure to detect, and correct this issue before it escalated into an actual outage, or crash of the servers involved. We must now ask your patience as the resulting backlog clears itself.

Please note: For obvious reasons, I have elected not to use the email notification system of this blog. If you rely on email to be notified of digital.forest support updates, I suggest switching to an RSS reader. The link for doing so is just over to the right side of this page. -->

Regards,
Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

Update: 12-15-06 13:37 The mail server is doing much better now. The queue is decreasing and mail is going out at a normal pace.

treehouse, our main mail server may have trouble sending right now, we are working to correct it and should have an update for you shortly.

For 6 hours today we tested a new technology for OUTBOUND mail filtering in an attempt to solve some of our mail deliverability issues. The theory is that other mail providers cause trouble making it hard for us to deliver your legitimate mail because they see enough "illegitimate" mail coming out of our network.

When operations like Yahoo!, AOL, Comcast, etc won't accept our mail, you complain to us. So we're trying to solve the problem. Here is what we do know:

1. We have far too many clients who have their domains here configured to FORWARD ALL MAIL to a non-d.f network mail address (Comcast, AOL, Yahoo!, etc)

2. We have far too many clients who have exploitable mail forms on their website and get hit by spammers. Daily.

#1 causes problems when that address receives spam from outside our network. When our servers forward it, they get tagged as the "spam source". This leads to us being greylisted, blacklisted, which leads to deliverability problems or bounces. In the past we were able to mitigate this by the old fashioned method of talking to the other providers and letting them know that we share customers - that we host their webservers and domains, and they used "you" (Comcast, Shaw, Verizon, whatever) for e-mail. This has worked well in the past, but this latest wave of spam on a global scale has made talking to people very difficult... the large providers are too busy talking to their own customers complaining about spam to take time to talk to us.

#2 is just a HUGE game of "whack-a-mole" and to be honest, we're tired of whacking. As much as we love our clients, we know that many of them haven't written their website code... but have instead used off-the-shelf PHP, or ASP, or complete CMS systems that they have NO IDEA of what is going on behind the scenes. Our past stance of "we'll disable all your mail scripts until you fix them" is not working. Too many of you just tell us it has been fixed when in reality, it isn't. Or the vulnerability is still there. We have always suspected that some percentage of the outbound mail from our netblock is "spammy" but what happened today shocked us.

Below is some sample data from a new device we are testing, which acts essentially in reverse to the Postini service we currently use to filter inbound mail. It filters OUTBOUND mail. Early today we noted a large outbreak of spam queued on one of our mail servers, specifically the one that acts as the mail relay for our hosting servers. That is when we turned on this new device. Within minutes, it stopped over TWENTY THOUSAND mails being relayed off a single client's web form. Once that ended, look and see what happened over the subsequent several hours...

On average, 50% of the mail OUTBOUND from two of our mailservers has the smell of canned meat about it. I find that disturbing. We don't host spammers, at least not knowingly. Can that much be drive-by form exploits and forwards?

We suspended the outbound filtering around 4:30 PM. We discovered that all our previous work of ensuring our servers were "whitelisted" with peer networks had not been completed with this new filtering device. We will spend the next few days getting those issues settled and turn it back on.

The ultimate goal here to ensure that we are being good network neighbors and NOT allowing junk out of our network. The result of that will be better and more reliable delivery of mail. Here is what you can do to help:

1. If you forward mail, stop. The risks far outweigh the benefits. Just configure your mail program to pick up mail on our servers via POP or IMAP. Virtually ALL mail software these days supports multiple accounts so the need to forward is no longer valid.

2. If you have forms on your website that send mail, do everything in your power to ensure that they cannot be abused. We're more than happy to assist you or your developers in this task. Please do not ignore this because it will not go away. It taints your domain name and causes severe damage to your reputation.

We appreciate you patience while we continually take steps to improve our service, and maintain one of the best hosting and colocation environments on the Internet.

Chuck Goolsbee
VP, Tech Ops,
digital.forest
Seattle, WA

For the past several weeks we've noticed a growing problem in delivering mail to yahoo.com email addresses. On all of our servers, at any time of day or night, the mail queues going to yahoo.com are backlogged. It seems Yahoo! has taken the extreme step of "greylisting" all inbound mail in order to avoid spam. They are basically slowing down the rate at which they accept mail, and throwing up error messages telling our server to try again later. Our servers then have to wait some prescribed amount of time before trying to send their queued mail. In the meantime more mail queues up for delivery. When the prescribed time arrives, our server tries again, and is able to deliver a small portion of the queue, when yahoo.com's mail servers once again call a halt and ask to retry later. In the meantime even more mail queues up for delivery to yahoo.com.

lather, rinse, repeat.

This means that at any given moment there are hundreds, if not thousands of mails queued on our servers for yahoo.com, and they aren't willing to take them at a good enough rate to relieve the backlog. It is like a short-cycling stop light at rush hour, except here rush hour never ends.

To add injury to insult, they are being very critical of message contents. many of our customers have reported large numbers of mails being returned as undeliverable with error messages such as:

message text rejected by mx1.mail.yahoo.com: 451

We have no idea what criteria yahoo.com is using to determine what text they are rejecting. The messages we've had forwarded to us looked very "un-spammy"... no styled text, no HTML, no mentions of investment advice or pharmaceutical life enhancements. We're as confused and helpless as you are.

Yahoo.com has no human beings we can talk to in order to resolve this issue, unlike most ISPs. As an autonomous network operator, we often can pick up the phone and call other ISPs and network providers and ask to talk to the people who run mail ops, and actually talk to them. We realize that might be an odd concept in this highly networked and automated world, but it still holds true. We attend technical conferences and meet with our peers, and exchange business cards. We help each other out when bad things happen. We make friends and call in favors. In most cases, if there is a problem between two networks, you can find the person on the other side who can work with us to fix it.

Except with Yahoo.

They are seemingly impenetrable, at least with regards to mail operations. They have some web forms to fill out and hopefully start the process (we can completely understand their desire to hide from user-level support issues since they make no revenue from those users, but they shouldn't hide from network peers!) but the web forms go nowhere and apparently don't do anything.

We're not alone in this. Others have the same problem. While it is a bit comforting to know that we aren't being singled out for poor treatment, it does nothing to help the mail get moving again.

As yahoo!mail is a free service, it is a classic case of "getting what they are paying for."

If you are a digital.forest client and you are FORWARDING your mail to a yahoo.com account, we strongly suggest that you stop doing that - many of you do, which is only adding to the problem. You are likely not getting your mail at all, or if you are, it is VERY late. We suggest you configure your mail client to check your mail directly on our servers. We support POP, IMAP, and webmail. Our servers have a very configurable webmail that you will probably like just as much, if not more than Yahoo's.

If you are a digital.forest client and you are trying to send mail to an address in yahoo.com, we suggest that you contact those people another way, and get an alternative email from them, as the same situation applies, they are not getting your mail, or at the least not in a timely fashion.

If you are a digital.forest client and you maintain mailing lists for communicating with your customers, we strongly suggest that you make every effort to keep those lists "clean" and remove addresses that bounce or cause problems like this. If you are mailing to non-existing addresses then you are clogging mail queues with useless, time-consuming items that only serve to create or enlarge an existing problem.

We'll keep an eye on this, but we suspect it won't be getting better anytime soon. If it does, we'll update this site. Thanks for your patience.

--Chuck Goolsbee
V.P., Technical Operations
digital.forest

UPDATE: 10-26-06 16:19 - Mail issues have been resolved.

We are currently experiencing an issue with our primary mail server. Technicians are working on the server and expect a resolution within the hour.

The mail server "treehouse" will be undergoing emergency maintenance today.

We will perform some quick maintenance to the existing server which is happening now, requiring a shutdown of mail service. The server should be back up and online around 8:45 AM PDT.

We will post details about what is going on as soon as possible.

Update: noon PDT

We are once again experiencing the mysterious slowdown bug. The server slows to a crawl with the primary symptom of slow disk writes to local mailboxes. This acts as a sea-anchor of sorts and slows the whole server down, which leads to the intermittent connectivity and glacial response that users experience.

Some background on the issue:
Communigate Systems, (formerly Stalker Software) the makers of the mail server software have never been able to determine the cause of this. Their suggestion each time has been to put the mail data on a faster disk subsystem. We kept upgrading every year and eventually migrated the data to a very high-end, high-performance FibreChannel disk array. yet, the problem returned. In the interest of troubleshooting, we moved the mail data back to the internal ATA disk of the server one time and sure enough, the problem went away. So this meant that it that it was NOT a hardware/performance issue of the underlying machine, but a true software problem within Communigate.

So our "cure" of late has been to switch file systems as soon as the early symptoms appear. We have successfully done that twice over the past year.

This time however, that did not fix the issue. In fact it got worse.

Short-term attack plan:
This morning we shut down treehouse for about 30 minutes and moved the mail data back from the array to the local filesystem and it seems to be staying even instead of falling behind.

Now that the patient is stable, we are preparing the transplant. We are building a new server, and plan on migrating treehouse over to it later tonight. It will be a completely different CPU and OS platform. It will have a different, but still very high-performance disk subsystem. We will post here when it goes online. Our hope here is that the software bug in CGP is specific to its particular interaction with the underlying architecture.

The search for a replacement:
In November of 2005 another software bug shut down treehouse, and our frustration level with Communigate went through the roof. We committed at that time to replace it as soon as possible. We have spent the past five months testing various replacement alternatives. Our ideal replacement system would match the current one for functionality and add a layer of redundancy that we really want and need for our mail system. This means a mail cluster. Unfortunately, we have yet to find something that meets those requirements. We have come close, and we were within weeks of deploying a system when a flaw in IMAP handling revealed itself. The project is "on hold" at the moment but given the events of this past week, we will have start work on it again. The developer of the cluster solution has released a new version with some fixes, and it is time to give it another thorough test.

We appreciate your patience while we sort though this. Stay tuned for updates.

Currently our mail server, treehouse.forest.net, is experiencing some general slowness. This will be resolved tonight during our scheduled maintenance period which is at 11pm PDT. We apologize for any inconveniences this may cause and appreciate your understanding as we work to improve our services.

At 3pm today, our mail server Treehouse will be taken down for preventative maintenance. We expect that the maintenance will last approximately 20 min. During this time, mail will continue to be received ant queued for delivery following the maintenance. We apologize for any inconveniences this may cause and appreciate your understanding as we work to improve our services.

Update @ 2:33pm: At this time the problems have ceased to exist on treehouse, we will continue to monitor the situation

Currently, one of our mailservers treehouse.forest.net is running slower then usual. The mail server is not down and it is still processing email, however at a much slower pace then usual. Possible side effects could be your mail client experiencing a connection error when it does a check every 10 or so minutes or the webmail interface being much slower then usual. We have a plan in place that we will execute tonight during our scheduled maintenance window so we can properly resolve this issue.

Update @ 2:04am on 05/06/2006: Both upgrades have been successfully completed

We will be performing a hardware upgrade (specifically a processor upgrade) on columbia (columbia.forest.net) and Ninewire's mail server (mail.ninewire.com) this coming Friday (05/05/2006). We anticipate the maintenance to last approximately 3 hours and it will begin at 10pm PDT and conclude at 1am PDT.

During this time streaming on Columbia will not be available & email service on mail.ninewire.com will also not be available.

This upgrade will increase the performance on Columbia & mail.Ninewire.com.

If you have any questions, feel free to open up a trouble ticket.

Update: 21:35 PST This maintenance has been postponed. We will re-post when it is rescheduled.

Tonight shortly after midnight we will be making a hostname configuration change and update a software license on the server mail.ninewire.com. This mail host serves about 750 users who came to us via the acquisition of Ninewire Digital Solutions.

Downtime should be limited to about 15 minutes.

On Thursday night, catalpa.forest.net (smtp.forest.net) suffered a hard drive failure at approximately 7pm PST. Currently we are spooling the mail for the accounts hosted on this machine on another mail server and we have pointed smtp.forest.net to another mail server located at digital.forest. In the meantime we are migrating all the affected accounts over to another mail server.

Accounts on other email servers such as treehouse, palm (infoasis) and ninewire are not affected unless their mail client's outbound (SMTP) server is set to smtp.forest.net. If this is the case please change the outbound SMTP to the same as your incoming (POP3 or IMAP) mail server. Be sure to enable SMTP authentication in your mail client when you make this change.

We currently do not have an estimated time of repair, but we anticipate that we will have the affecting customers moved over quickly.

If there any updates concerning catalpa.forest.net we will post them here as soon as they are available.

digital.forest technical support

Treehouse.forest.net and mail.ninewire.com will experience some interruption in service this evening starting at midnight pacific time, and continuing until approximately 2:00 AM pacific time. We will be testing the mail software on both servers for potential problems related to future run dates. This will involve temporarily adjusting the clock on both servers, which will affect the displayed arrival date and time of incoming messages. We apologize for the inconvenience; this testing is necessary to ensure that software date issues do not interrupt service on a much greater scale in the future.

Woodpecker.forest.net has taken a hit from the power incident and is reporting errors. It is currently in not functioning for many lists, but mail is being spooled on our mail hub for those lists that are not working. We should have it back up and running early Sunday. Thank you for your patience.

Please note that an exploit in PHP is being widely abused by spammers to generate untraceable bulk mail. The spammer's identity is completely concealed, while the spam itself is identified as coming from the exploited website. If you use PHP in your website contruction you need to be aware of how to prevent your website from being abused in this way.

The greatest risk from this exploit (beyond being associated with being a spammer) is having all mail from your domain, and ultimately our network rejected. If you rely on email to communicate, and use PHP in your website construction, then you should take this issue very seriously and take all steps to prevent that possibility.

In a nutshell, the issue is injecting mail headers into a PHP form, complete with carriage returns and linefeeds in the right places, to force the web server's mail transport agent to relay thousands of messages per minute off your website. You can read some excellent coverage of the problem and cure at the following links:

www.anders.com

php.net

www.phpit.net

securePHP

This site describes a method to test your forms:

www.developertutorials.com

An additional method, and one we highly recommend, is preventing DIRECT access to form data. Don't put form entry fields on your home page, and limit access to form pages from referrers outside your domain... i.e. only allow access to your pages containing forms from other pages on your website. This can be done with .htaccess files. You can find a tutorial for this here:

apache-server.com

This is a serious risk, and digital.forest must take this seriously as it can have widespread implications regarding the acceptability of outbound mail from our network. As such if we notify you about exploitable forms on your website, whether it is on our servers, or your server colocated in our facilities, please take steps to immediately correct your website code. If we receive and relay to you repeated reports of your forms being exploited, and you have not taken steps to correct the problem we will have no choice but to suspend your service.

We appreciate your swift action with regards to this situation.

Regards,
Chuck Goolsbee
V.P. Technical Operations
digital.forest, Inc.

We are seeing very long delivery times to mail destined to "hotmail.com" and "msn.com" addresses. This is not limited to our servers as we are seeing similar reports from around the Internet. Either Microsoft's mail servers are very busy, or they have instituted some sort of "throttling" for inbound mail.

The larger the message, the longer the time for delivery - anything larger than a megabyte can take several hours.

If you can avoid sending messages to hotmail or MSN users, please do. Especially if they have large attachments. Do not be surprised if they bounce or timeout.

We will update this post if the conditions improve.

Our secondary DNS server and back-up colo mail server, willow.forest.net, has experienced a fatal system malfunction. We don’t expect any interruption from this failure and expect to have a new server deployed and operational by Wednesday. If you are a colo client who uses willow as a back-up to your email server and are experiencing any issues with your primary email server, please notify us immediately so we may assist you.

Yvo V.
digital.forest technical support

Currently the mail server issue seems to be resolved. Please allow some time for all email to come in as email servers around the globe will start sending your email to our server again.

Thank you for continued patience.

At 5:00pm pacific time, we will once again be taking the mail server down to make changes recommended by the vendor. We will be doing as much as possible in the meantime to prepare for the change so that we can minimize the downtime. We will, as always, post more info when it is available.

The new treehouse is up and running and online as of 4:20 AM PST.

The final sync of treehouse is taking longer than anticipated; we will be getting the server back up and running as soon as possible. In the meantime, we will update this page again when we have further info.

A pre-sync has been completed between the old Treehouse and the new, and we will now be taking the server offline for a final sync and cutover. We expect the downtime to be less than an hour. Watch this space for updates.

First let me say that we apologize deeply for the issues surrounding our mail servers over the past 24 hours. We TRULY do feel your pain since we use the very same servers for our own email. Secondly I want to tell you in detail what has transpired over the last 24 hours, and what steps we have taken to address the performance and reliability of the current mail systems. Finally, I'd like to take a moment to tell you what we are planning in order to prevent this or similar issues from happening again.

In short, a bug in the mail server software we use, Communigate Pro (CGP) caused two of our three CGP servers (and actually a few of the colocated CGP servers we manage for clients here) to crash last night at the rolling of the clock to November 1st (Midnight UTC). The software vendor suggested that the fix for this would be a downgrade to a previous version. We executed that change overnight for most of the servers with few issues, and were able to keep the other servers stable overnight by rolling their clocks back a day.

Treehouse is a large server, with a significant number of users on it, including ourselves. We hesitate to make changes to its configuration since we have had performance issues with it in the past and have had to deal with the vendor a lot to get it working properly when upgrades are made. But stuck between it crashing with certainty, or risking the downgrade, we did not have much choice. The downgrade unfortunately brought back some long-standing file-system related bugs that cause serious performance issues under heavy load. We have wrestled with this particular file-system bug several times over the past four years and finally thought we had it fixed. To say this is frustrating for us is a huge understatement.

We have built a new server, equal or greater in specification to the existing treehouse server, but running on a different platform (FreeBSD). We have performed a clean install of (the non-crashing bug version of) CGP on it, and will migrate our user and mail data to it tonight. I can not promise, or guarantee that this will fix everything, or not introduce some other software bug. Those are issues beyond our control. We do feel that it should stabilize the server enough for the next step.

What I CAN promise and guarantee is that we at digital.forest will dedicate all of our resources to replace the current mail system with something far more robust and scalable. By the end of this year, we will have tested, and begun to deploy on a wide scale a system to completely replace the one in use now. It must allow us to operate our servers in a cluster, so we have failover and load-balancing. It will support all the features you have come to expect: POP, IMAP and Webmail, as well as delegated administration of domains.

Again I apologize for this situation, and thank you for your patience today. As always we will post progress and updates here as we proceed.

Chuck Goolsbee
V.P. Technical Operations
digital.forest, Inc.

Treehouse will be shutdown at 10pm PDT and we will be migrating all of the users to a new server on a different platform.

Stay tuned for updates by monitoring this space.

Thank you for your continued patience

digital.forest technical support

The mail server is currently back online and churning through the queue that has been build up due the downtime. It is still performing poorly at this time.

Connections via your mail client (such as Eudora, Outlook, Entourage, etc.) will most likely not be working at all due to the state of the mail server.

Right now your best bet will be to access the web mail portion by going to http://mail.(yourdomain) and then to log in using your email username and password. This may not as well, but will be more reliable then using a mail client.

Tonight we will replace the mail server, treehouse.forest.net, with a different mail server in order to restore complete functionality to our email.

We thank you for your patience,

digital.forest technical support

In the next 30 minutes we expect to have an update from our email system administrator on a potential return to service time.

More information will be posted to this website at that time.

While message loss is a concern to everyone, we don't anticipate any significant email losses.

Thank you for your continuing patience,

digital.forest technical support

In our continuing effort to get our mail server, treehouse.forest.net, fully back online, we will be taking it off line for emergency maintenance. It is down until further notice.

Please monitor this space for further updates.

Thank you for your continuing patience,

digital.forest technical support

You will note that mail from two of our five mail servers has an incorrect timestamp. The affected servers are "treehouse.forest.net" and "mail.ninewire.com".

A crashing bug surfaced in the software running on both of these servers whose only short-term fix is to roll back their clocks. We figured that the inconvenience of an incorrect timestamp is better than a crashing mail server. The developers have promised to respond to our issue within 24 hours, so hopefully we will have a fix before a full day goes by. In the meantime, please accept our apologies for the inconvenience until then.

At 10:00pm this evening, we will be separating customer data from mail server internal data on our primary mail server, Treehouse. By spreading the disk i/o load over two disks, we hope to address recent sporadic performance issues with SMTP connections. The downtime should be less than an hour.

Tonight at midnight PDT we will be shutting down one of our mail servers "treehouse.forest.net" for maintenance. Downtime could be up to 2.5 hours.

We are taking this step to mitigate or remove the issues that have been affecting performance of this mailserver over the past week (see post below). We have been working very closely with the mail server software vendor Communigate Systems to analyze the issues and investigate possibilities for solving them. Unfortunately doing these operations on a "live" production server during the business day is impossible. We believe that this downtime will be long enough to get them all done at once.

Thanks for your patience.

We are aware that our primary mail server, treehouse.forest.net, has been suffering periods of very poor performance over the past week. What's going on is a dramatic increase in dictionary harvest activity: attempts to harvest good email addresses from a server by using a "shotgun" approach. Basically, the harvesting programs throw an enormous mix-n-match combination of usernames and domain names at a mail server to find out which ones work and which ones don't, thus allowing them to build a list of good addresses.

We use a spam-filtering service named Postini that provides a good measure of protection against such activity, but it only helps when the domains being harvested are actually routed through Postini. To date, we have had a number of customers whose mail has not been routed through Postini, either because they didn't want to use the service, or because they use a catch-all address on their domain, which is incompatible with Postini.

What we have been doing in the past week is moving as many domains as possible behind Postini, with spam filtering turned off -- this means the domains are protected from harvesting, but are not being filtered for spam (or billed for that service). There is still a fair number of domains to move, but we are working through them as quickly as we can. We expect that performance should improve incrementally as more domains are moved -- however, there will continue to be periods of poor performance when a domain that hasn't been moved yet is being harvested.

We thank you for your patience while we continue to work on this serious issue.

One of our mail servers, treehouse has been very slow this week. This is a known issue and we are working to correct it. Thanks for your patience.

We will post more detail as soon as possible.

Treehouse's transfer was complete just before midnight last night. We had a nagging issue affecting performance throughout the night, but it was resolved at about 7:45 this morning. Please let us know if you are experiencing any problems.

We moved our main mail server, treehouse.forest.net to both our new Seattle facility, and onto new hardware. We are working out some issues with regards to performance. Things should be back to normal later this morning. Please be patient while we work on this.

One of our clients behind our shared firewall was running an open mail relay. This was discovered by a spammer and has been exploited. Not only did they relay off that host, they are now attempting to relay through the entire firewalled subnet. We have had to block port 25 to that subnet in order to allow any "normal" traffic at all to and from the servers behind the firewall.

This block was in place most of the night.

We have lifted it (for all except the open relay server of course) as of 5:30 AM, though the SMTP traffic remains unusually high. We will monitor the situation and respond as required.

Please remember that a firewall is not a magical protection device. If you have vulnerable software on an open port, you can be compromised.

UPDATE 6:05 AM: We have been able to isolate the network (in Russia) performing the brute-force SMTP relay attack, and block it at our network boundary.

At approximately 11pm this evening, we will be taking Treehouse, our primary mailserver, offline for approximately 30 minutes to transfer it to new hardware in our new facility. We will post again when the transfer is complete.

As part of our move to the new Seattle datacenter, our three main mail servers will be taken offline this evening sometime after 11pm Pacific time.

palm.forest.net will be down for approximately one hour as it is transferred to the new datacenter.

catalpa.forest.net will be down for approximately one hour as it is transferred to the new datacenter. smtp.forest.net, normally served by catalpa for outgoing mail from CGI scripts, has been pointed to souari.forest.net for the time being. It will be switched back tomorrow.

treehouse.forest.net, our primary server, will be down for approximately 15 minutes while we transfer its data from the external array to the internal disks. We will then move the array to the new datacenter. Treehouse's performance may be somewhat degraded afterward; tomorrow, we will begin syncing treehouse's data with the new, more powerful server and the array at the new datacenter.

We have almost completed server moves to our new Seattle facility. Our DNS and mail servers will be moving over the next few nights. At this rate, we may be able to shut down the Bothell facility as early as next week after the weekend moves complete.

We will post updates concerning the mail servers at least 12 hours before we move them.

As of 12:47 AM PST, treehouse.forest.net is back online.

No data appears to be lost, however we suggest you check your mail, especially IMAP users. The server will be quite busy as it accepts mail spooled and waiting for it out on other mail servers on the Internet.

Thanks for your patience while we recovered this vital asset for us all.

Update: 12:30 AM: RAID recovery almost complete. We should know by 1 AM what the next step will be. Stay Tuned.

Our other mail servers, Palm, Catalpa, etc are doing fine. Treehouse is the only mailserver down right now.

The power event from earlier this evening has caused our mail server "treehouse" some problems. We will use this blog entry to provide updates as we rebuild it.

Update: 9:10 PM: RAID array rebuild is about 50% complete, so our ETA of completion is about 11:30 PM. If the data is OK, we'll be back up shortly after. If not, we may make the tough call to run with what we have so we can continue and avoid an extended outage. Stay tuned.

Update: 11:45 PM: RAID array rebuild is at 85%. Time just is not on our side tonight.

Treehouse has had a minor update to CommuniGate Pro followed by a reboot; the system now appears to be stable. We will be monitoring it closely to see if the problem recurs, and will continue to investigate the problem to see if we can determine another cause.

treehouse.forest.net is experiencing some intermittent failure. We're working on the problem currently and will update when we have more information or a resolution.

We completed the treehouse maintenance work last night just after midnight. Everything seems to have gone smoothly. Treehouse is now using a fibrechannel RAID to store the mail data, which should improve performance and reduce system load.

We will be performing 2-3 reboots on treehouse.forest.net this afternoon in an effort to improve performance on return-path lookups. We have been experiencing occasional problems where a backlog of DNS requests causes the incoming SMTP connections to slow down; this in turn increases the number of connections, until we hit our maximum and people begin experiencing connection failures.

We are going to try having Treehouse perform its DNS lookups locally and see if this mitigates the problem. The configuration changes will require two or three reboots, each lasting 2-3 minutes. They should be complete by 1:00pm Pacific time.

We are making these changes now, rather than after hours, so that we can monitor server performance throughout the busier parts of the day and make sure the changes have the desired effect.

As many of you are aware, we are experiencing major delays in delivering mail to yahoo.com addresses, verizon.net addresses, and some other large providers. We are not alone in this problem, but we've been doing a lot of investigation into it and believe we have an explanation, if not yet a solution.

First, as you can see from this post on Open Tech Support, other people are struggling with the same issue. Basically, if a large provider feels that they're receiving a lot of spam from a given server, they block that server automatically for a brief period, during which time no mail can be delivered from that server to that provider.

What seems to have changed recently is how those services are deciding to block the sending server. Many of our customers use mail forwarding to handle their mail -- rather than using mailboxes on our server, they set up a forwarder that directs their mail to a mailbox on another server. For instance, if we host the domain "foo.com," bob@foo.com may be configured to forward to bob@yahoo.com.

Unfortunately, new schemes used to stop spam are breaking traditional methods of forwarding. Some of them are, as described in the above post, checking the sender's info to make sure that the return address is valid. Unfortunately, a failure on the part of the original sender's server to respond in a timely fashion, or to respond at all, or if it turns out the address is actually forged, results in our server being blocked -- not the original sender's -- because as far as the receiving server is concerned, we're the ones sending the bad mail. We're not, of course; we're simply acting as a middleman. But there's no way for them to tell that with the traditional method of forwarding.

Another method some servers are using is called "SPF," which attempts to identify whether a given server actually has permission to send mail for a given domain. So if joe@bar.com sends mail to bob@foo.com, which is then forwarded to bob@yahoo.com, Yahoo's mail server will check the SPF records to see if our mail server is authorized to send mail from the bar.com domain. Which, of course, it isn't, because it's only acting as a forwarder -- so they decide we're spamming and block our server for a while.

There is a way around this problem, which is to use something called "SRS," or Sender Rewrite Scheme. This rewrites the return path of the message so that the destination server can tell that it has been forwarded, and handle it appropriately. The problem is that SRS is extremely new, and has not been implemented in many mail servers yet -- including CommuniGate Pro, the one we use.

We've upgraded to the latest version of CommuniGate, and are investigating methods of working around this problem, but ultimately, it won't be entirely solved until SRS is implemented in CommuniGate. Because of the serious nature of the problem, we will consider (very carefully) installing a beta version of the software when it supports this feature. We will update this space when we have more info.

In the meantime, you can help: wherever possible, create mailboxes and collect your mail from our server instead of forwarding it to other providers. The less mail that is forwarded, the fewer blocks will be triggered, and the more mail will be successfully delivered to these providers. Please feel free to open a trouble ticket or call if you have any questions.

We are experiencing a massive, distributed directory harvest attack which is being conducted in a way that makes it extremely difficult, if not impossible, to block the attack. We are monitoring the situation and considering options. In the meantime, SMTP service is slow, but not unavailable, so please be patient.

digital.forest clients who use Cox as their access provider are seeing issues connecting to services on our network.

We are receiving a lot of calls from our clients with connectivity issues, mostly checking/sending email. The common thread in all these calls is that the user has Internet connectivity through Cox Communications, a large US national broadband provider. We have contacted the Cox Network Operations Center and they report an issue with their connection to NTT/Verio. Since NTT/Verio is also one of our upstream connections, this would explain why Cox customers are seeing issues connecting to digital.forest. We do not have an ETR on this issue from Cox.

As soon as we hear more information from Cox or Verio, we will post it here.

As mentioned earlier, we are in the process of moving treehouse.forest.net to a more powerful server. We are going to make this move tonight; downtime will begin between 8:00 and 8:15pm and continue until the move is complete. We don't expect it to take more than two hours, but will update the support site when we can give a better estimate.

Thank you for your patience.

As many of you are aware, we have been experiencing increasing problems over the last week and a half with mail service. Traffic has increased dramatically, largely due to the spam worms that were recently released on the net, and our server has been having a lot of trouble coping.

We are taking two actions to deal with the problem: first, we are in an ongoing process to eliminate the last of the catch-all addresses on our server so that at we can move it behind our spam service and block connections from unauthorized systems. Second, we are going to move treehouse.forest.net onto more powerful hardware with a faster storage array so that it can better handle the illegitimate traffic that has become a reality of the modern Internet.

The new server is being prepared now, and we will be moving over to it sometime in the next 24 hours. We hope to do it at night for minimal disruption, but if service remains as impaired as it has been today, we may make the change earlier.

We will update again when we have further information.

The frequency of the attacks on our mail servers has decreased, but we are still experiencing spikes in SMTP traffic that effect webmail and outgoing mail. These spikes appear to be of limited duration. We continue to take steps in an effort to minimize the effects of these attacks on our servers.

In fact, there are two new mass-mailing viruses -- one (Bagle.B) started yesterday, the other (Netsky.b) today. As a result, even with the aid of our Postini filtering service, mail traffic is up approximately 1000%. This is saturating our incoming SMTP connections periodically, interfering with legitimate attempts to send mail through our server, and slowing down the webmail interface significantly.

I'm afraid there is little that can be done besides riding it out. Thank you for your patience.

Our primary mail server, Treehouse, is being hit with a high volume of Dictionary Harvest attacks. This may result in slowdowns in webmail and SMTP responsiveness, and occasional connection failures.
Thank you for your patience as we take steps to reduce the impact of these attacks.

(CNN) -- Hackers unleashed an agile worm Monday -- using a sneaky, fairly new
tactic to get unsuspecting computer users to diffuse their malicious code.

CNN link

Our primary mail server, treehouse.forest.net, is getting hit by an unusually high number of directory harvest attacks today. This results in saturation of our inbound SMTP connections, preventing users from sending mail through our server.

Our spam protection service, Postini, is detecting and blocking these attacks as quickly as it can. As the attackers quickly shift to new IP addresses, however, we are experiencing surges of activity followed by periods of calm. Thus, until the attacks stop, we can expect intermittent connection problems to continue.

This may give you an idea what we're dealing with. You can see a number of attacks, followed by decreased activity when the IP address in question is blocked, over the course of an hour. It ends with a particularly strong attack that maxed out all 500 of our connections for over a minute.

One of our main mailservers (treehouse.forest.net 216.168.37.80) was blacklisted by a major spam filtering organization (Spamcop.net) earlier today. The server was on the blacklist for about 3 hours. We are investigating the source and working with spamcop.net staff to resolve this issue and prevent a repeat. Further details will be posted when we know more.

Since many secondary services use spamcop.net as a blacklist source, we expect a 'ripple effect' for the next 24 hours or so. Please be patient.

We had an emergency maintenance on the mail server and it's currently functioning. Performance may be slow as people try to connect and while the maintenance is being performed.

Our outgoing mail hub for both hosting and colocation servers, smtp.forest.net had a problem tonight, which slowed outgoing mail for about 4 hours. This has been fixed and mail is flowing again. It appears no mail was lost, as the queues just backed up, so there should be no need to resend.

We have been working with AOL to ensure that mail servers in our netblock are on their whitelist, as they have become very aggressive about mail filtering lately. I don't blame them for this, as all of us have had to take extra measures to protect users from spam.

However, they have been blocking quite a large amount of legitimate mail from servers located at digital.forest. It has taken us some time to find the right people inside AOL to communicate with, but we finally have and are making headway.

The reason I am posting this is if any of you run your own mail servers on our network, and are having trouble delivering mail to "aol.com" addresses, please contact us via our trouble ticket system and let us know. We will do our best to pass the correct information to AOL to fix the problem.

Regards,
Chuck Goolsbee, VP, Technical Operations

We will be shutting down Treehouse for about ten minutes to install some additional RAM. We expect this to eliminate the periodic slowdowns we've been experiencing for the past week.

Catalpa is back up on its new server, which we hope will prove more reliable than the old one. SMTP, POP and listserv service were restored in about an hour and fifteen minutes; IMAP service was restored in about two hours. Please let us know if you have any problems.

On Monday evening, starting at about 4:30, we will be taking down our older mail server, catalpa.forest.net, for maintenance. Several weeks ago, we upgraded the operating system to FreeBSD 5.0 to address some security concerns; 5.0 has proven less reliable than we had hoped. FreeBSD 4.8 addresses the same concerns, but does not support Catalpa's current hardware. So we will be moving Catalpa to new hardware running FreeBSD 4.8. We expect roughly three hours of downtime, and will update the support site if it looks like we'll exceed that.

smtp.forest.net will be temporarily repointed to another server so that forms hosted on web servers will continue to function. Mail will be inaccessible during the downtime for any users with domains hosted on Catalpa.

The infoasis mail server, palm.forest.net, has been moved to its new Mac OS X server, and is now up and running. Please let us know if you experience any problems with it.

Our primary mail server, treehouse.forest.net, is presently down for emergency maintenance to diagnose and resolve our recent mail system slowdowns.

Expected downtime is less than thirty minutes.

12:35 PST
Service has been restored; mail performance should be significantly improved.

At 5:00pm this evening, we will be taking the Infoasis mail server, palm.forest.net, down for a hardware changeout and a software upgrade. We expect downtime to be only about an hour, but the actual time will depend on how long it takes to move the data. We will post updates as soon as we have more info.

Due to a serious security problem noted today in versions of sendmail installed on our FreeBSD servers, we will be performing system upgrades on the following servers. Downtime will be involved, but not much more than half an hour per server:

- Catalpa (pop.forest.net): Tonight after 5:00pm
- Butternut: Tonight after 5:00pm
- Souari: Tomorrow after 5:00pm
- Boysenberry: Tomorrow after 5:00pm
- Crabapple: Wednesday after 5:00pm

Update
Butternut is complete. Catalpa has had a complication; it is up and running, but we will need to do some more work on it to address issues with some mailing lists.

Update (12:35 AM, Tuesday, Mar. 4)
Working through mailing list issues. Some lists back to normal operation. All should be functional by later this morning.

Our primary mail server, treehouse.forest.net, is down for emergency maintenance. Unfortunately, we dont know what caused the issue as it sneaked up on us. We are investigating the issue which at this point seems to be a file system error. If this is the case, then a simple disk diagnostic and fix will take care of it. Please watch this entry for more updates as they are available.

Very little mail should be lost as new incoming mail will be sent to our back up mail server, willow.forest.net. Willow will hold mail for up to 2 days and try to send it back to treehouse in intervals.

11:14am
Update. Treehouse was successfully fixed with the diagnostic tools and was restarted. You may find that the server runs a little slower than normal as the server catches up and the backup mail server, willow, sends its mail to treehouse.

Our downtime lasted considerably longer than we'd hoped, totalling almost exactly five hours. But treehouse is now running on a server architecture that, we hope, will better handle the high threading demands being placed on it by the continuous dictionary attacks. We should have a good idea of the improvement by tomorrow afternoon.

Thank you for your patience.

Due to our current maintenance and upgrades on our treehouse.forest.net email server; in addition to certain client email functionality being temporarily unavailable, our emergency and support email addresses are currently offline.. If you have any technical issues, please dial 1–425–483–0483 (domestic) or 1–877–720–0483 (Toll Free) and select extension 5099 to leave a phone message for our Technical Support department.

We apologize for the delay and we are working as fast as we can in order to restore email services.

We will update this notice when all email services are up and running again; current estimates place this time at approximately 10:00 PM Pacific Time.

Our primary mail server, treehouse.forest.net, will be going down for emergency maintenance at 5:00pm this evening. Downtime should be less than two hours (we expect it to be considerably less than that, in fact, possibly only a few minutes). We hope that this will improve mail server performance, which has been dramatically affected by dictionary attacks for some time. We will update this notice when the server is back up.

As many of you are still aware, our CommuniGate mail server treehouse.forest.net is still slughish during peak times. We want to assure you that we are working to get this resolved. Options are being discussed in your best interest to reduce server downtime. More info will be shown here on support.forest.net when available. Thanks for your patience.

At 5:00pm Pacific Time tonight, we will be upgrading the CommuniGate software on treehouse.forest.net to version 4.0.1. This is a major, but easy upgrade, and features some improvements to the software that we hope will address the performance problems we've been seeing lately. The upgrade will take only a few minutes, during which time your mail will be unavailable. No mail will be lost.

The transition from mail.bopjet.net to treehouse.forest.net is complete. mail.bopjet.net has been shut down. It has passed on. It is no more. It has ceased to be. It has expired and gone to meet its maker. It's a stiff. Bereft of life, it rests in peace. If we hadn't bolted it to a rack, it'd be pushing up the daisies. Its various server processes are now history. It's off the twig. It's kicked the bucket, it's shuffled off its mortal coil, run down the curtain, and joined the bleedin' choir inivisible.

It is an ex-mail server.

You may be able to tell that we aren't too despondent about this. It is truly better off dead. As for those of you who may have been using it, all mail has been directed to the new server for a week now, so it should have been at least five days since any mail arrived at mail.bopjet.net. Feel free to delete your old account settings, and please call us if you're having any trouble with the new server.

mail.bopjet.net is now back online in our Bothell data center. It has a new IP of 216.168.37.201.

mail.bopjet.net has been shut down for a move to the Bothell data center. Having it local will make the transition much easier as we move customers off it and onto the CommuniGate mail server. We expect the server to be back up late this evening.

Some serious problems with the bopjet mail server have arisen in the past 24 hours. Due to the unique configuration of this server and the difficulty involved in reconstructing it after any necessary repairs, we have decided that instead of repairing the server we will be moving your domain(s) to our CommuniGate mail server. This change is urgent, and must be completed as quickly as possible. Our goal is to have all mail flowing to the new server by Tuesday, October 22, and have the old mail server completely shut down by Tuesday, October 29.

PLEASE NOTE that this move will not cause you to lose mail. There will be a transitional period during which you will retrieve mail from both the old server and the new server to ensure that you receive all your mail.

We will make the transition as easy for you as possible, but be aware that you will need to make some configuration changes on your machines. We will include all the information you need to make these changes, and you should feel free to contact us if you need help.

Further details will follow early next week.

Our CommuniGate mail server was rebooted. We noticed a lag in the mail service and with further investigation, the server needed to be rebooted.

Mail service was restored in 5 minutes.

The follwoing is a communication from Peter Lalor for Infoasis email clients:

To: all@infoasis.com
From: Peter Xavier Lalor
Subject: Apology to email clients

Dear Infoasis/digital.forest email customers,

As you should be aware by now, digital.forest has just implemented Postini spam filtering for all customers. As I waged a long war with spammers prior to digital.forest taking the baton, this is a move I wholeheartedly applaud.

Although digital.forest manages the servers and support on a day-to-day basis, I still made adjustments to the mail server from time to time to address email abuse that I was seeing.

Yesterday, I made one last adjustment that went very wrong. Seeing what appeared to be heavy abuse from a specific range of addresses, I told the server to ignore them. Unbeknownst to me, the range I blocked was Postini itself. The filter was put in place by myself around 5PM PDT yesterday evening, and was removed by digital.forest tech support at around 10 AM PDT today. Some of the mail sent between those hours may have bounced back to the sender or been held on digital.forest's secondary mail server. Anyone receiving a bounce can re-send their mail and it will come through today.

I want to apologize to you for this error on my part, and to assure you that it doesn't reflect in any way on digital.forest, their staff, or the Postini system itself. Mea culpa. I feel sure that the move to Postini will be a good one, and will improve the email experience for us all.

Sincerely,

Peter Lalor
President,
Infoasis, Inc.

The problem with the spam filtering on palm.forest.net has been discovered and corrected. A misconfiguration of the SMTP server's filtered IP addresses was the culprit; a more complete explanation will follow when our investigation is complete.

Due to the fact that we had to turn Postini off and back on again, and had to delete the accounts that had been created previously, your Postini accounts will be recreated and you will receive a new activation message from the system.

Due to unforseen problems with the Postini filtering on the Infoasis mail server, palm.forest.net, we have been forced to disable filtering for infoasis.com and delete the existing Postini accounts. We will restore filtering as soon as the problems have been resolved.

This affects only the infoasis.com domain.

This afternoon around 4:30 PM PDT, we disabled POP & IMAP access to the server for about 20 minutes, and manually managed the SMTP queues. The object of this effort was to clear a huge number (around 30,000+) of "invalid messages" that were clogging the queues and adversely affecting server performance. These messages were created as a result of earlier efforts to block the ongoing dictionary attacks on your main server, treehouse, and the attack subsequently falling back on willow, our secondary mail server.

A frustrating aspect of this attack is that it uses standard mail protocols which makes it very difficult to stop. The last thing we want to do is prevent actual legitimate mail from being delivered, so that limits us to only reacting, rather than preventing.

We are doing all we can to adapt, and will have a solution in place soon to either minimize the affect to you, or completely remove the issue entirely.

Thanks for your patience.

Our main mail server, treehouse.forest.net, is under a heavy amount of load right now. Our mail admin is working to clear out a large queue, and the server might need to be restarted. Please be advised that there will be intermittent access to this CommuniGate mail server until the issue has been resolved.

Update: POP and IMAP connections have been disabled for a short period of time as the mail server's queue scripts require precedence.

Many of our customers have reported problems retrieving their mail lately; specifically, that the mail server has been very slow. These performance problems are real, and they are being caused by a spamming method known as a "dictionary attack."

What happens is a spammer makes use of two dictionaries: one containing thousands of usernames that are known to be popular, and one containing thousands of domains names. A program the spammer runs then uses the two dictionaries to generate email addresses -- say, pulling "dave," "david," and "davis" from the name dictionary, and "foo.com" from the domain dictionary, thus creating "dave@foo.com," "david@foo.com," and "davis@foo.com."

The spammers are not attacking our server, specifically, but whenever the domain they pull from the dictionary matches one hosted on our server, the server is subjected to several thousand attempted mail deliveries. This has a serious impact on performance.

We do try to block the offending IP addresses when we detect them, but new ones pop up about as quickly as we can block old ones; many of the spammers work through compromised systems all over the network. We are working with the company that produces our mail server software to see what else we can do.

We are also now offering Postini mail filtering to all customers on our CommuniGate mail server. The cost is $1/mailbox/month, and it must be turned on for all mailboxes in a given domain. If you would like to consider adding this service to your account, please contact your account manager.

Please note that the dictionary attacks do not cause mail to be lost — they can cause a short delay, but all mail will get through.

We received the following message today from Postini, which affects all users of our Postini spam filtering service:

Postini is pleased to announce that the next release of Postini's email perimeter protection service is scheduled to rollout on Thursday, August 1st. This release will incorporate several enhancements to the Message Center and its settings, based on the feedback from you and your users. The rollout of this new functionality will not interrupt your Postini services. You will see these changes late in the day on Thursday, August 1st.

The key enhancements in this release include:

- Enhanced spam filter controls, giving users increased flexibility to personalize the filters.

- A more intuitive and streamlined process to automate approving senders and mailing lists after a message is delivered and read.

- Improved trash functionality, including the ability to empty the trash.

- New, cleaner message center layout.

- The ability to delete all messages at once.

We have updated some software on catalpa.forest.net to increase security on the server. In particular, Apache has been updated to version 1.3.26. This should have no effect on mail services running on catalpa.forest.net (aka "smtp.forest.net").