Between the hours of 12:01 AM and 1:00 AM Thursday, November 29th, and Friday, November 30th, we will be performing some hardware upgrades on portions of our network.

We will be replacing gigabit Ethernet modules on a few switches deployed in the colocation portions of our facility. Expect intermittent outages, likely less than a minute in duration, but possibly lasting up to five minutes while we install these improvements. Every effort will be made to minimize downtime.

Thank you for your patience.

That is a tower of power you see above. It is the new benchmark for electrical density here at digital.forest. 528 Watts per square foot'. We know that isn't a world record, but it is a very respectable number for a colocation facility here in the Seattle area. Few can approach it, much less match it.

We can't share the name of the client (shhh... they are a startup in "stealth mode") so we've blurred the pictures a bit and removed any identifying marks. They are however VERY happy that they found digital.forest.

Prior to being here, they were colocated at a competitor. They were NOT allowed to put this many servers into their rack at this competitor. In fact, in order to have this many servers, they had to buy FOUR RACKS.

We think that's nuts. So did our new client. Now they are happily colocated at digital.forest and can do it all in one rack.

At digital.forest we are here to assist our clients to succeed, not restrict them from what they need. It is just one part of the digital.forest difference.

In observance of the Thanksgiving Holiday, digital.forest will be closed Thursday November 23rd and Friday November 24th. We will resume regular business hours at 8am PST Monday November 27th.

Technical Support staff will remain on-site 24 hours a day throughout the holiday period. Please be aware that we'll have limited staff coverage for telephone tech support during the next few days. Please note that our building will be locked throughout the weekend and clients requiring access to the datacenter to work on colocated servers will have to call or email first to be allowed access to the building. Additionally we will likely take advantage of the holiday lulls to perform maintenance and upgrades on core equipment. We will post notice of these beforehand. Finally, we are closing the phone support queue for a while on Wednesday, November 22nd in the afternoon for a Tech Department meeting. We appreciate your patience.

All of us at digital.forest wish you a happy Thanksgiving and a wonderful holiday season!

Chuck Goolsbee
VP, Tech Ops
digital.forest

The Internet is buzzing with news concerning a potential malware threat from a Microsoft Windows vulnerability which was patched this past Tuesday. I'd like to take this opportunity to remind our valued clients of our policies and procedures in instances such as this.

* We do our best to protect the hosts inside our network from such threats, by both patching and port-blocking on our boundary network and firewall devices.

* We ask that you also stay current on your patches, not only on your servers here, but also on any internal hosts used to access them. This is crucial because many of our clients use VPN technology to communicate with servers in our datacenter. Our port blocking and firewalling efforts have NO AFFECT on the contents and payload of VPN-tunnelled/excrypted traffic. This means that even if we have successfully stopped the malware from entering our network from "the wild" you or your users can still "infect" your own servers via a VPN connection.

* If an outbreak of some malware does occur, our first priority will be to secure our network from further spread. If your servers are infected, and being used to spread further malware or similarly abusive traffic, we will have no choice but to disconnect them from the network. We reserve the right to block any malicious traffic, or remove any system from our network being used to generate malicious traffic.

* We are available to assist clients in patching or repairing systems, but be aware that our priorities in the midst of an event will be protecting those clients and systems that are NOT affected first. In other words we may not be available to assist immediately as our resources will be focussed on prevention of the spread before curing of the ill.

It is therefore in your best interest to patch your systems now.

For more information on this issue, please see:
http://www.microsoft.com/technet/security/bulletin/ms06-040.mspx

http://www.dhs.gov/dhspublic/display?content=5789

http://www.eweek.com/article2/0,1895,2002142,00.asp

Excellent sources of up-to-date information should an event occur are:
SANS' Internet Storm Center
CERT
US-CERT

Regards,
--Chuck Goolsbee
V.P., Technical Operations
digital.forest

If you are using IBM's Websphere 4.0 or 5.0 on your server, please be aware that an internal certificate will be expiring at 21.59.19 GMT today.

IBM has released more information at: http://www-1.ibm.com/support/docview.wss?uid=swg21236118

Thank you.

At approximately 1:45 AM PST this morning we experienced a power overload condition on a single electrical circuit, which services half of two racks in our facility. This tripped a breaker in one of our Power Distribution Units. We reset the breaker and used power monitoring equipment to measure the load on that circuit as servers rebooted. With the data collected rerouted some power cables in those two racks to spread the load in a manner which should prevent this from happening again.

Most of the servers affected are digital.forest shared hosting servers, however one of the two racks contained some colocated and one dedicated server. We will be contacting the affected clients during the business day with a follow-up.

Server downtime was limited to about 20 minutes maximum, with most servers being down less that 15 minutes.

We apologize for any inconvenience.

Chuck Goolsbee
VP Technical Operations

Currently the Colocated server reboot script that is used to remotely reboot your machine from our website (located here: http://www.forest.net/support/tools/reboot.php) is not working. We are aware of this issue and will have it resolved soon.

In the meantime if you wish to have your machine rebooted please give us a call at 206-838-1630 or shoot an email to our emergency email address.

UPDATE: (12-14-05) The Colocated server reboot script has been fixed. Thank you for your patience while we resolved this issue.

Please note that an exploit in PHP is being widely abused by spammers to generate untraceable bulk mail. The spammer's identity is completely concealed, while the spam itself is identified as coming from the exploited website. If you use PHP in your website contruction you need to be aware of how to prevent your website from being abused in this way.

The greatest risk from this exploit (beyond being associated with being a spammer) is having all mail from your domain, and ultimately our network rejected. If you rely on email to communicate, and use PHP in your website construction, then you should take this issue very seriously and take all steps to prevent that possibility.

In a nutshell, the issue is injecting mail headers into a PHP form, complete with carriage returns and linefeeds in the right places, to force the web server's mail transport agent to relay thousands of messages per minute off your website. You can read some excellent coverage of the problem and cure at the following links:

www.anders.com

php.net

www.phpit.net

securePHP

This site describes a method to test your forms:

www.developertutorials.com

An additional method, and one we highly recommend, is preventing DIRECT access to form data. Don't put form entry fields on your home page, and limit access to form pages from referrers outside your domain... i.e. only allow access to your pages containing forms from other pages on your website. This can be done with .htaccess files. You can find a tutorial for this here:

apache-server.com

This is a serious risk, and digital.forest must take this seriously as it can have widespread implications regarding the acceptability of outbound mail from our network. As such if we notify you about exploitable forms on your website, whether it is on our servers, or your server colocated in our facilities, please take steps to immediately correct your website code. If we receive and relay to you repeated reports of your forms being exploited, and you have not taken steps to correct the problem we will have no choice but to suspend your service.

We appreciate your swift action with regards to this situation.

Regards,
Chuck Goolsbee
V.P. Technical Operations
digital.forest, Inc.

We are performing a backup set rotation on our colocated and dedicated server backup system. It will likely run through the weekend. Normal daily incremental backups should resume early next week.

Our secondary DNS server and back-up colo mail server, willow.forest.net, has experienced a fatal system malfunction. We don’t expect any interruption from this failure and expect to have a new server deployed and operational by Wednesday. If you are a colo client who uses willow as a back-up to your email server and are experiencing any issues with your primary email server, please notify us immediately so we may assist you.

Yvo V.
digital.forest technical support

We had to stop the normal backup routine yesterday to perform a major data restore for a client. This process required extensive searching through older backup data sets which took over 18 hours to perform. It finished a few minutes ago. This has put the backup server very far behind. It will likely take us until the weekend to catch up. We appreciate your patience with regards to this issue.

Our backup set for colocated servers running OS X ran out of space prematurely. We have started a new backup set. All colo and dedicated backups will run a bit behind until we can catch up (probably mid day Saturday).

We are taking advantage of the long holiday weekend to perform a backup set rotation on our Colocated and Dedicated servers. Incremental backups should continue by Monday night.

We are posting this notice to make Colo and Dedicated server owners aware of a service-impacting issue.

The Windows Update posted by Microsoft yesterday will break FileMaker 7 Advanced Server installations, and cause FileMaker Server to not launch after reboot. To fix, it requires a re-install of FileMaker 7 AS, which of course requires all installation and license codes to be re-entered.

So far, we have only seen this on Windows 2003 Server, so it may not affect other versions. If you encounter this issue on a different version of Windows, or FMP, please let us know. We will post more information it becomes available.

We are performing a backup set rotation for colo & dedicated Windows and MacOS X servers this weekend. Full backups will run through Monday night, Incremental backups should resume by Tuesday.

Thanks.

We will be performing a backup set rotation for colocated and dedicated OS X servers starting tonight. Normal incremental backups will resume by the weekend. Backups for Windows Servers, Linux Servers and old ("classic") MacOS servers are unaffected.

This note only applies to those running MacOS X Server 10.3.x. Clients running other UNIX variants, or Windows are unaffected.

We have had a few reports of today's "Security Update 2005-003" for MacOS X Server causing POP mail services to fail, and in some cases crashing the server. This is only if you are are using the built-in Cyrus POP server in MacOS X Server 10.3, not a third-party POP server such as WebStar Mail or Communigate Pro. If you are running OS X Server 10.3, and the built-in POP server, we suggest that you delay the installation of "Security Update 2005-003" until more is known about this issue.

We are currently installing this update on a test server that is not running any mail services at all, and will update this post with results as soon as they are available.

UPDATE (23:10 PST): Our test server sems to be doing fine. So if you are NOT running the built-in POP server in OS X Server 10.3, it should be OK to run "Security Update 2005-003"... However, like all updates, it might be prudent to wait a day or so and let the reports come back from the field. It is always best practice to test these updates on NON PRODUCTION systems prior to deployment on live, production servers.

When Apple announced the MacMini at Macworld Expo last January, an immediate discussion began in the technology community about the possibility of using it as a server. Just like when Apple introduced the G4 Cube, and the Xserve, companies even came out of the woodwork, basing their whole business model on MacMini Colocation. While the latter is a bit absurd, the former is worth discussing. While digital.forest has never been a "Mac Only" facility (our very first server in 1994 was a Sun Microsystems SparcStation 5) we have always been very open-minded about platform, and embraced the Mac as one of many we support. The Macintosh community, a fiercely loyal group of customers that any company would be wise to embrace, has shown digital.forest as much loyalty as they have shown Apple. This is how we have become the largest collection of MacOS servers on the Internet, at least outside of Apple's corporate datacenters.

People who run servers are always attracted to small form-factor computers because when you buy colocation, part of what you are buying is space. The more computer you can pack into the least amount of space the better... so the theory goes. It also helps that Apple is selling the MacMini for a very low price. A full-blown FreeBSD UNIX install on a sub-$500 computer.

Given the insane colocation market now the cost-per-rackspace argument is kind of moot. But the appeal of a spending under $500 for a server is hard to argue with... even if it may not be a significant performer. An Xserve is far better suited to serve moderate-to-high traffic sites, but as a low-volume mail server for a small office, or a "personal server" the bang-for-the-buck of the Mini isn't too bad.

So it didn't surprise us today when Phil Herring, a long-time client of digital.forest showed up today to replace his aging "blue & white" G3 mail server with a brand new MacMini.

Above: Phil Herring unpacks the lunchbox-sized container that the Mini ships in. This is the first server that has ever come into our datacenter carried one-handed.

Above: Phil Herring holds up his new server.

Above: This little computer is twice as powerful as the one it sits upon, and 3X more that the one it is replacing. We'll have to find out from Phil how well it performs.

I'm keen to find out myself actually, as we have racks and racks of G3 & G4 computers that carry pretty light loads. Mind you we also have racks and racks of G4 & G5 machines and Xserves that carry pretty high loads... but the combo of Xserves and Minis might be an interesting "Big & Little" option. Personally, I'd like to see an "Xserve lite" more tailored to the realities of the datacenter and Internet serving, (Apple should look at Dell's 1U & 2U servers... very nice) but until that happens, we may be looking at a lot of Minis... who knows.

--chuck goolsbee
vp of technical operations
digital.forest
seattle, wa

As part of our ongoing commitment to improve your service, we are adding a new Backup Server to our network next week and will be migrating several of our client's servers over to it. This server will be running Windows, and will specifically be installed to backup clients running Windows. It will still be using EMC/Dantz's Retrospect backup software, so it is likely that no changes will be required for current clients. The benefits of this new service will be:

* Fewer clients per backup server.
* Better performance.
* Greater restore capabilities for some Windows-specific technologies.
* Better control over backup timing.

We will likely add just a few servers a night to the new system as each one will require a full backup to start the process. Some firewall settings will need to be changed as backups will be using a different server IP address. We will contact server administrators directly to implement required changes, if needed.

One of our clients behind our shared firewall was running an open mail relay. This was discovered by a spammer and has been exploited. Not only did they relay off that host, they are now attempting to relay through the entire firewalled subnet. We have had to block port 25 to that subnet in order to allow any "normal" traffic at all to and from the servers behind the firewall.

This block was in place most of the night.

We have lifted it (for all except the open relay server of course) as of 5:30 AM, though the SMTP traffic remains unusually high. We will monitor the situation and respond as required.

Please remember that a firewall is not a magical protection device. If you have vulnerable software on an open port, you can be compromised.

UPDATE 6:05 AM: We have been able to isolate the network (in Russia) performing the brute-force SMTP relay attack, and block it at our network boundary.

The colocated server backup device, and large disk array that it uses as storage will be moving over the weekend. It will move during daytime, so there should be minimal disruption of server backups.

The servers we moved tonight are those behind our shared firewall. We had some delays leaving Bothell due to some struggles with rackmount Dell "Versa Rails"... stripped screws specifically. We also are having to rebuild our Firewall config from backup, as the device did not seem to like its new location. Please be patient while we sort these issues out.

Thanks.

We are performing a backup set rotation this weekend for colocated and dedicated MacOS (8/9) and Linux clients. Normal incremental backups should continue by Sunday night.

We experienced an issue with our colo/dedicated backup server this morning. It appears our recently started backup set may be suspect. We have taken it offline and are starting a new backup set on fresh media. This means we will likely not be able to perform data restores on Windows and MacOS X servers for the time period between Monday Jan 10 and Today. Linux and MacOS 9 backups were not affected. Apologies for any inconvenience this may cause.

We performed a backup set rotation for our Windows and MacOS X colocation backups on Sunday night, Jan 9th. This was unscheduled (the next rotation was scheduled for January 21st) but required due to our addition of a new, larger capacity disk array.

This has placed a delay on colocation backups as Windows & MacOS X clients have gone through a full backup. We should be caught up and doing incrementals on all clients again by early tomorrow morning.

Apologies for any inconvenience or delay.

Taking advantage of the long holiday weekend, we have kicked off a backup set rotation on our backup servers. It is a week earlier than scheduled, but the low traffic volume and relative quiet of a four day weekend were too hard to resist. Normal incremental backups should resume Sunday/Monday.

We here at digital.forest hope you have a pleasant holiday season, and look forward to the new year, which will bring some very exciting news and significant improvements to your server environment. =)

We started a new backup set for colocated and dedicated servers last night (Friday, November 5th). Normal backups should resume Tuesday night.

We are beginning a new backup set for colo and dedicated servers this evening. Normal backups should resume Wednesday evening.

The backups for colo and dedicated servers are a bit behind schedule. A full backup was run last weekend, and a subsequent incremental was completed on Monday/Tuesday night. Since then we have had to interrupt backups to perform a few restores, so incremental backups are behind schedule. We hope to be caught up by the weekend. Thanks for your patience.

The backup for Colocation and Dedicated servers will have a set rotation this weekend. Expect normal incremental backups to resume Monday night.

Starting Wednesday evening, Aug 4, we will be starting a new backup set for our Colo & Dedicated servers. We have just deployed a new RAID array for the purpose, and we expect some better performance and more reliability.

We do not know what to expect with regards to speed, so we can not predict the rotation time. With our old array a full rotation took about 2.5 days. We expect this one will take less time.

Thanks for your patience while the set rotates.

We lost a member of the backup set for our OS X colo/dedicated clients. This should not prevent us from performing minor restores, but it will make doing a "bare metal" recovery difficult. To ensure the integrity of our backup set we are starting a new one for the OS X clients tonight. This will likely delay all of our colo & dedicated backups on Saturday. We apologize for any delays.

A "zero-day" Windows exploit has been discovered as of yesterday. You can track the discovery and analysis of this on various security and Internet-operations websites. From our observations, this exploit can affect FULLY PATCHED servers, despite claims to the contrary by Microsoft. It was discovered on one of our hosting servers yesterday, which was fully patched (including update KB835732).

There are no current virus scanning signatures for this exploit (on the server side) so it must be manually discovered. We strongly suggest that owners of colocated Microsoft Windows servers take whatever precautions they can, as they become available. We also suggest that server owners read the above referenced web pages to learn how to spot the exploit, and disable it. Take this knowledge and closely search and monitor your systems, both here at digital.forest and on your local networks.

We are updating our intrusion detection systems to monitor for this exploit, but given that it is still unknown how this exploit gets onto servers we'll only be able to spot already infected machines. If we find them, the server owners will be notified.

Thank you for your cooperation and understanding.

We have started a new backup set on our colo-dedicated server backup system. Nightly backups will likely be delayed until early next week as we perform full backups over the weekend.

Please note that Apple has released some security-related updates in the past weeks, including a new OS version (10.3.4) which includes these updates. Experience has shown that staying up-to-date with security-related patches can limit exposure if an exploit is released "into the wild." While we have yet to see a MacOS X worm or similar malware, we still prefer to update when the vendor releases a patch. OS updates are handled a little differently, as they usually involve changes outside the realm of security, and require some testing prior to deployment.

NOTE: If we have administrative access to a client-owned server we almost always install security patches when they are released. The nature of our network, directly connected to very high bandwidth "backbone" connections, means that we have a much greater risk and exposure to newly released malware, especially of the "worm" type, as they spread automatically.

These recent patches from Apple cover a set of vulnerabilities which are classified as "Trojan Horses" which means they require user intervention to activate. However we felt it necessary to apply the patches, if only to set a precedence for our MacOS X using clients with regards to how we handle security-related patches.

If you have a server colocated here running MacOS X, or MacOS X Server that you manage yourself, we strongly suggest that you run Software Update on a regular schedule. Install any security-related patches as soon as you are comfortable. Based on our experience with other platforms, it is better to be patched prior to the release of an actual exploit.

Our methodology for handling unpatched machines if there is a known exploit "in the wild" is to remove them from our network until patched. This is how we were able to survive high-profile issues such as CodeRed, SQLSlammer, etc with minimal downtime and very low infection rates. Our Windows and UNIX using clients already know this, but given the recent widespread publicity about these MacOS X issues, we thought we should make the rest of our clients aware of this policy.

Thank you for your attention to this matter.

The full backup over the weekend encountered some issues toward the end, and left us with a suspect backup set. Rather than trust it going forward, we are re-running a full backup to new media over the next two nights.

Incremental backups will begin again as soon as we have a good full set.

The upgrade went very smoothly, and we have begun the new backup set. We pushed client updates out to all the Windows, Unix & MacOS X clients. We have left the "classic" MacOS clients on the previous client software version, as we found in our internal testing that was the more stable option.

Thanks for your patience, and have a great weekend.

On Friday, April 2nd (one week from today) we will be performing a software upgrade on our Colocated Server Backup System. This will involve pushing a software update out to many of the colocated servers here. We have been testing this upgrade on our own internal and hosting servers over the past month and it brings many speed and media-handling improvements.

At the same time we will begin a new backup set, so expect some delays as we perform the initial backup over the weekend.

The Good News: Further testing has pinpointed the exact issue. The problem does indeed lie with Verio (always the first trouble when dealing with Internet Service Providers: proving where the problem really is).

The Bad News: We have to wait for Verio to fix it.

The Current Status: The Verio connection will remain shut down until we hear from them that it is fixed. Traffic continues to flow via other circuits, so this will remain invisible to our clients.

The Technical Details: Yesterday, some automatic route-filtering scripts on Verio's network decided to change our network size from roughly 8000 IP addresses down to around 4000. From our initial investigation this is based on some very old (circa 2001) Routing Database info. After we get this current issue fixed, we will investigate what caused that to happen. The result of this action is that the "upper" half of our network, where the majority of our colocation subnets lie, was route-filtered by Verio. This lead to intermittent connectivity to those subnets if your traffic passed through Verio's network. Thankfully the time frame for the connectivity issues was in the middle of the night, when traffic is usually pretty light. Our connection with Verio is a Fast Ethernet one, via our OnFiber Gigabit link. We have enough capacity on our other circuits to handle 100% of our traffic so we can safely leave the Verio link shutdown, however we prefer to spread our traffic out as it is more efficient. As soon as Verio confirms that they are advertising our full network in their routing tables we will bring up our link with them again. This could happen at any time today, with the latest possible time being 7pm CST when their automatic route filtering scripts start their nightly run. Of course we are urging them to not wait for that event. We will update this page with news as it happens.

Thanks for your patience.

We started a new backup set on Friday evening. The initial backup usually takes about 48-56 hours to complete. Nightly incremental backups will resume again Monday night.

Thanks for your patience.

Our electrician is installing some new wiring in the datacenter today for our new row of racks. This work will require the brief shutdown of the two rows immediately adjacent in Server Room 2. One row is home to our oldest MacOS-based computers ("BandW G3's and older "beige" units.) The other is home to the servers we acquired last summer from FMPHosting.net. At noon PST we will perform an orderly shutdown/reboot of the Mac servers to accommodate the work. We will delay the FMPHosting.net reboot until after 2PM PST (to match 5 PM EST)

NO OTHER servers will be affected.

Thanks for your patience.

Microsoft today announced a critical vulnerability which affects virtually all versions of their Windows™ operating system. Given the short time frames between vulnerability and actual exploit of similar past announcements, digital.forest strongly urges our colocated server clients running Microsoft Windows™ to patch their servers as soon as possible.

For dedicated and managed colocation clients, we will schedule a patch application within the next 48 hours. If for some reason your server can not be rebooted in that time frame, please contact technical support or your account manager.

If you require assistance or remote-hands for your colocated Microsoft Windows™server, we can provide assistance where required.

As some of you (colocation clients) may have noted our backup service has been running a bit slow this week. We started a new set rotation last weekend, and had to stop it a few times to perform restores for different clients during the week. On Thursday, this new set's catalog was damaged. Rather than repair (which can take quite some time) we chose to begin a new set on Friday.

This set has run now, without interruption, over the weekend and should conclude this evening. Normal backup schedule will resume Monday night.

Thanks for your patience.

Over the next few months, digital.forest will be investigating potential improvements to our data backup services. If you are a co-location or dedicated server customer, please take a moment to stop by our Backup Service Improvement Survey, located at:

http://survey.forest.net.

The survey should take about 5 minutes to complete, and all questions are optional. Your feedback will help us pinpoint areas for improvement in our existing system and develop a better idea of what new features and services our clients would find useful. The survey will be available until November 30th.

Thank you!

Our outgoing mail hub for both hosting and colocation servers, smtp.forest.net had a problem tonight, which slowed outgoing mail for about 4 hours. This has been fixed and mail is flowing again. It appears no mail was lost, as the queues just backed up, so there should be no need to resend.

We have been working with AOL to ensure that mail servers in our netblock are on their whitelist, as they have become very aggressive about mail filtering lately. I don't blame them for this, as all of us have had to take extra measures to protect users from spam.

However, they have been blocking quite a large amount of legitimate mail from servers located at digital.forest. It has taken us some time to find the right people inside AOL to communicate with, but we finally have and are making headway.

The reason I am posting this is if any of you run your own mail servers on our network, and are having trouble delivering mail to "aol.com" addresses, please contact us via our trouble ticket system and let us know. We will do our best to pass the correct information to AOL to fix the problem.

Regards,
Chuck Goolsbee, VP, Technical Operations

We have experienced a significant increase in ICMP traffic, from all corners of the 'Net recently. Today it caused a slowdown and some packet loss on our core switch due to the rapid growth of the forwarding table. We attempted to mitigate this with different switch configurations, but were unable to achieve a proper balance. At 2:30 PM we elected to just block ICMP echo traffic at our boundary router. We realize that this may "break" some monitoring systems, and make some troubleshooting more difficult - but the overall performance of our network is far more critical. We apologize for any inconvenience you may experience.

Should the "worms du jour" that generate this kind of traffic subside, we will enable ICMP echo packets again.

September 17th, 9pm through 12:00 AM

During our normal scheduled maintenance window (Wednesday night,) we will be migrating several hosting & colocation subnets to newer, better ethernet switches in Server Room1. The subnets affected are:

216.168.37.0/24
216.168.47.0/24
216.168.60.0/24

All servers in these subnets will experience very brief outages (less than 30 seconds) as they are patched into the new switches.

No configuration changes will be required.
No servers in Server Room 2 will be affected.

There are a handful machines that we didn't get to during yesterday's update process. If you happen to be one of the lucky people who didn't experience the brief downtime yesterday, you might notice it this morning.

The only servers that remain to be updated are part of the FMPHosing network, so this announcement only applies if you're a former FMPHosting client. No digital.forest hosting servers are included in this morning's update process.

We'll also be updating several core FMPHosting servers, including the iMail server, at some point in the future. Those updates will be done during off-hours, and an annoucement will be made prior to any associated downtime.

Again, if you are a co-location customer with equipment running any variant of Windows NT 4, Windows 2000, Windows XP, or Windows Server 2003, please download and apply the appropriate KB824146 Update.

Customers on Windows-based hosting servers may see a very brief period of downtime this morning while we install several security related updates to correct a severe vulnerability found yesterday in all Windows NT-based operating systems.

We strongly urge all co-location customers running any variant of Windows NT 4, Windows 2000, Windows XP, or Windows Server 2003 to update their systems immediately.

Additional information on this vulnerability is available here:

Buffer Overrun In RPC Interface Could Allow Code Execution (823980)

Critical security vulnerability in Microsoft Operating Systems

During the normal scheduled maintenance window tonight just after midnight (PDT) we will be performing the following procedures:

1. Swapping an Ethernet card on our boundary router.
This serves the 204.142.69.0/23 network that includes our newly acquired clients from Exceedia Inc. Users on this network will experience a brief service interruption, lasting approximately 15 seconds.

2. Installing a new switches in the 216.168.37.0/24, 216.168.58.0/24 & 216.168.62.0/24 networks. Users on these networks will experience a brief service interruption, lasting approximately 30 seconds, as each host is moved from the old switches to the new.

On Wednesday, May 28th, we'll rotate our backup set. This usually introduces a delay in the backups, so it may take more time that usual to complete your server backups. The initial backup will likely stretch out through Friday. Normal nightly backup should resume by the weekend.

Update: 5/31, 17:25 Colocated server backup completed about 30 minutes ago. Normal incremental backups will continue tonight at 22:00.

If you colocate server(s) running Microsoft Windows please take immediate action on this notice.
A trojan (hidden, malicious code) has appeared on the Internet, and has been found on 3 client-owned servers here at digital.forest. It is called "SMSX.EXE" and if your machine is infected, it will most likely show up in your "SYSTEM32" directory. This program is designed to send spam, specifically to mobile phones via text messaging systems. It also can be used to generate "smurf" attacks, and syn & udp floods.

Note: A firewall is NO guarantee that you are protected. Many vulnerabilities are accessible via firewall-allowed ports. Please check your Microsoft Windows systems, even if they are firewall-protected.

We have blocked the network ports SMSX.EXE uses so it can do no more damage, but we do not have login access to many of the colocated servers here. If we detect that your servers are infected, we will disconnect them immediately. We suggest that you all check your server(s) ASAP for this trojan and if found, remove it. We suspect that this is copied to servers using an RPC vulnerability unique to Microsoft Windows. If you have a Microsoft Windows server here, we suggest you check with Microsoft on a weekly (at least, daily at best) for security patches and stay up to date.

Additionally, we can perform security audits and maintenance on your server(s). If you are interested, please contact your account manager.

We have begun a new backup set for dedicated and colocated servers last night (Sunday.) As is always the case with a new media set it takes a while (up to 72 hours) to backup all the servers and begin the incremental backups. At this time we are about 60% through. We should return to our normal nightlies by Tuesday, April 2nd.

At approximately 9:30 PM PST January 24th, digital.forest was affected by the worldwide internet attack known as W32/SQLSlammer. We were alerted within fifteen minutes of the first evidence of this worm as it entered our network. Preventative steps were immediately taken to minimize the impact on our network performance, and prevent damage to network devices.

At this point, all systems are performing normally, with the exception of a handful of client servers that are unfortunately affected by this worm. We are coordinating with clients to have their software patched and brought back online as soon as possible.

Any additional information on this worm can be found at:
http://www.cnn.com/2003/TECH/internet/01/25/internet.attack/index.html
http://www.bayarea.com/mld/mercurynews/news/5030801.htm
http://vil.nai.com/vil/content/v_99992.htm

Thank you for your ongoing understanding and patience.

We are seeing some network issues at our Vancouver datacenter. More news will be posted as it becomes available.

UPDATE: 6:00 PM PST. A technician is onsite at this time. Upstream provider appears to be the target of a DDoS.

UPDATE: 9:15 PM PST. The decision has been made to move the remaining (8) servers in Vancouver to the Seattle datacenter. Downtime should be just over 2 hours.

UPDATE: 11:25 PM PST. The servers are being set up in Seattle right now. New IP's will be mailed to server tech contacts where required.

UPDATE: 12:10 AM PST. The servers are all back online.

Toadstool, our backup server, is currently recovering its catalog file, which has gone corrupt over the past few days. This happens occasionally, and recovery is usually a trivial process. Recovery is, however, very time consuming; the backup machine has to read over two terabytes (2,000 gigabytes) of data to rebuild the catalog set.

We estimate that the backups and restores will be back on track by Thursday afternoon, and we apologize for any inconvenience this delay may cause.

Please be advised that we are required to perform emergency maintenance on Monday, July 8th between 02:00-04:00 PDT. This is required to reload our Core Routers in Vancouver. You may notice up to three 45 second interruptions. These interruptions will occur sometime within the maintenance window. If you have received this email then you may be affected by this maintenance. We apologize for the short notice, and thank you for your understanding in this matter.

The Datacenter in Bothell, WA will be getting a floor makeover on Wednesday, March 27th 2002. The floor will be washed and buffed and waxed. This will inhibit colocation customers from being able to get physical access to their machines, aswell as increased response times for any server reboot. Thanks for your patience in this matter.

Our network engineer will be working to segment the network into smaller subnets which will enhance the efficiency of our network as we move towards the lighting up our new OnFiber connection.

The 47 netblock will be physically separated onto its own VLAN.

Clients hosted on the 47 netblock, ie. your ip address starts with 216.168.47.** will notice an increased amount of latency during the switchover. Network outages should not last more than a couple minutes.

No changes will need to be made on any customer hosting servers.

Our network engineer will be working to segment the network into smaller subnets which will enhance the efficiency of our network as we move towards the lighting up our new OnFiber connection. The first step in this process will be creating two new networks and assigning new subnet masks to specific network infrastructure components.

The new networks will be:
216.168.46.1-127 GW .1 SM 255.255.255.128
216.168.46.129-254 GW .129 Sm 255.255.255.128

Oak will have to have a new subnet mask in it's TCP/IP stack and will have to be reloaded.

Border001 will have to have a new subnet mask applied.

Branch will have to have a new subnet mask applied.

Overall traffic and customers will see a drop in packets for < 30 seconds.

When oak (Primary DNS Server) is rebooting, customers without willow as a secondary will see a loss of traffic until oak comes back up.

No changes will need to be made on any customer hosting servers.