The domain name registrar Register.com has been experiencing a multi-hour outage of their DNS servers today. Clients who rely on Register.com for hosting their DNS may experience issues with anything that relies on DNS, meaning email, web, etc.

No digital.forest services are affected by this outage, but we know that some of our clients use Register.com, and to them it may appear that their data here is offline. That is certainly not the case.

At this time we do not have any more information about the cause, or the expected time of resolution to Regsiter.com's issues. So far Register.com has made no statements and has no notice on their website about the issues. We are aware of them through our constant monitoring of the Internet's operational communications channels. If we hear more definitive information we'll post it here on the support blog.

Update: 4:30 AM PDT At this time it appears Register.com is fully back online.

With a lot of effort from our staff and clients the vast majority of the DNS servers in our datacenters have been patched for the vulnerability announced three weeks ago. However one major server & operating systems vendor has yet to release a patch, namely Apple. Since Apple uses ISC's BIND as the basis for their DNS under MacOS X and MacOS X Server there is no reason why you can not fix this issue yourself on a Macintosh server. With thanks to one of our clients, as well as an old friend of mine who used to work at Apple, we present to you a series of step-by-step instructions for patching BIND on a MacOS X system. These instructions install the update in a location and manner that will allow you to still install Apple's patch, if/when it is finally released.

The following instructions assume you are working locally ON YOUR SERVER HERE at digital.forest. Most will use Apple's "Apple Remote Desktop" or Netopia's "Timbuktu" for remote management. Advanced users can use SSH connections via a terminal and should understand where these instructions below have been simplified for users of a GUI admin tool. If you are unsure about any of these steps, feel free to contact technical support via telephone or trouble ticket.

Resolving BIND insecurity problems on your OS X box at digital.forest:

Option 1: Turn off DNS service. Seriously -- do you really need it? You can feel free to use our recursive resolvers for your server's DNS needs. They are located at 216.168.32.229 and 64.69.73.100. For your own client machines at home or at your office, use your ISP's DNS servers.

Option 2: Updating Your Own BIND

1) If you do not already have the Apple Developer Tools, join the developer program (http://developer.apple.com/ -- it's free), download and install them.

2) At the BIND site (http://www.isc.org/index.pl?/sw/bind/index.php), download "bind-9.4.2-P1.tar.gz".

3) Open your Terminal application, and type the following commands:

cd ~/Downloads
(NOTE: This assumes the standard Safari download location)
tar -zxf bind-9.4.2-P1.tar.gz
cd bind-9.4.2-P1
sudo su
(NOTE: Type your password when prompted)
./configure --prefix=/usr/local
make
make test
make install
cd /usr/sbin
mv named named.hold
ln -s /usr/local/sbin/named named
sync
sleep 10
reboot

4) When Apple releases its patch, before you install it, launch your Terminal app again and type:

sudo su
(NOTE: Type your password when prompted)
cd /usr/sbin
mv named.hold named
sync
sleep 10
reboot

5) Install the Apple Security Update using Software Update

Hopefully Apple will release an official patch soon. Until then however, this is your only recourse to make your server safe from this vulnerability. Again, big thanks go to Glenn Fleishman and Rich Mogull of TidBITs, and Chuq von Rospach for their valuable insight.

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

The DNS Vulnerability we made you aware of recently has been cracked and there are reports of an exploit already "in the wild." If you are running a DNS server, here at digital.forest, or anywhere else we STRONGLY suggest you patch it IMMEDIATELY. The details of this vulnerability were originally scheduled to be announced in early August, giving people time to patch their servers. Unfortunately it appears now we no longer have that time. Let me repeat: If you are running a DNS server, we STRONGLY suggest you patch it IMMEDIATELY.

We addressed this issue with our own servers within 24 hours of the original announcement. Since then we have scanned our network internally and found many DNS servers, and have begun contacting those system owners. We'll have to accelerate that process considerably. If you know you are running a DNS server, please fix it now. If you are uncertain, please check. While we are contacting system owners, it would be better for them to take a proactive approach and not wait for us to call.

Please remember: According to our terms of service we reserve the right to remove your server from our network if it is being attacked or being used to attack others. For everyone's safety and convenience it is critically important that your servers are up-to-date with their security patches. Knowingly running an insecure server puts you uptime and stability at risk.

Thank you for your attention with regard to this critical matter.

--Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

Please note: This event is not connected in any way to our main Seattle facility.

The city of Vancouver, British Columbia, Canada has experienced a widespread electrical outage due to a transformer vault fire as of 9 am today. From what we understand most of the downtown core is without electricity. digital.forest maintains some off-site servers at a facility on Hastings Street in Vancouver, specifically some secondary email and DNS servers. As of this time, those servers are online and operational, on generator power. The local power utility, BC Hydro, has provided an ETA of 9 pm to have power restored. It is unlikely that our servers will go offline, but we know that several of our customers do use these servers for secondary mail, and should be made aware of their status, even if the risk of outage is low.

We are keeping a very close eye on things from 150 miles away, and are in constant contact with people on-site in Vancouver. If anything changes we will update this post.

Again, this event has had no operational impact so far, and is in no way connected to our main Seattle facility.

Today a major vulnerability was announced in the Domain Name Service protocol. You can read the details in the US-CERT announcement here: http://www.kb.cert.org/vuls/id/800113.

We are performing an assessment of our own DNS servers, planning to patch them, and make the configuration changes as required. We will post more on that as needed. The main purpose of this post is to inform our clients who run their own DNS servers inside the digital.forest facilities about this vulnerability. Please note that this is a fundamental vulnerability in the DNS protocol, so it is not vendor specific. This means that virtually EVERY device that can operate as a DNS server is vulnerable. We strongly suggest that you consult with your equipment and software vendors to ascertain your exposure and take appropriate action.

We'll post more information as it becomes available.

The following issue is not directly digital.forest related, but it may effect some of our customers:

As of approximately 7:30am PST this morning, Network Solutions' DNS servers stopped responding to DNS queries.

If your domain is registered through Network Solutions and you are using their DNS servers (*.worldnic.com), your website and email will likely not work until Network Solutions resolves the issue.

Note that if you registered your domain through Network Solutions, but are using digital.forest's DNS servers (oak.forest.net and willow.forest.net), you should not be affected.

Our secondary DNS server and back-up colo mail server, willow.forest.net, has experienced a fatal system malfunction. We don’t expect any interruption from this failure and expect to have a new server deployed and operational by Wednesday. If you are a colo client who uses willow as a back-up to your email server and are experiencing any issues with your primary email server, please notify us immediately so we may assist you.

Yvo V.
digital.forest technical support

We have almost completed server moves to our new Seattle facility. Our DNS and mail servers will be moving over the next few nights. At this rate, we may be able to shut down the Bothell facility as early as next week after the weekend moves complete.

We will post updates concerning the mail servers at least 12 hours before we move them.

All digital.forest customers are strongly urged to read this document, and the pages to which it links, thoroughly and carefully.

A recent change to the domain transfer rules greatly affects the security of your domain name. The Internet Corporation for Assigned Names and Numbers (ICANN) has developed a new transfer policy with the intent of making it simpler and easier to transfer domains between registrars. Unfortunately, in the opinion of many, the changes will also make it easier for people to hijack domains that do not belong to them.

Many of you may be familiar with the sometimes frustrating older process: to transfer a domain, a transfer request was submitted, and a request for approval was then sent to the domain's contact addresses. If one of the contacts submitted an approval, and none of the others declined the transfer, the transfer would then be considered approved and could proceed. If none of the contacts replied within five days, the transfer request was dropped. One result of this process was that if a domain owner moved and changed his or her email address, it could be a very difficult and confusing process to sort out the contact information and get a transfer approved.

The key change in the new transfer policy is that a failure to reply after five days is now considered tacit approval: rather than the transfer request being dropped, the domain will now be transferred. In a perfect world, this wouldn't be a problem; the registrar that is requesting the transfer is still obligated to obtain explicit consent from the domain owner before submitting the transfer request to the original registrar. Unfortunately, this world is not perfect; an unscrupulous registrar could claim it has received approval when it hasn't, or a clever domain hijacker could successfully deceive the requesting registrar. Invalid transfer requests could be made, and under the new rules, they would be approved if the domain owner failed to respond.

You may have a great deal of time, money, and identity invested in your domains. There are two important things you should do to protect them:

First, log into your domain management page at your registrar and make sure your contact information is up-to-date. If you move or change email addresses, be sure to update your domain records to reflect the change. You can't respond to transfer requests if you don't receive them!

Second, all registrars should now permit you to lock your domains. This prevents anyone (including you) from transferring your domains until you explicitly remove the lock. Each registrar will have its own method to allow you to lock your domains, so again, log into your domain management page and make sure you turn domain locking on.

Please follow these links for further reading on this subject:

The announcement:
http://www.icann.org/announcements/announcement-12nov04.htm

The policy:
http://www.icann.org/transfers/policy-12jul04.htm

Commentary:
http://news.netcraft.com/archives/2004/11/09/domain_transfers_and_hijackings_to_become_easier.html

digital.forest strongly encourages you to take these steps immediately to make sure your domain is secure. Thank you!

oak.forest.net, our primary DNS server, will be going down for maintenance at about 5:00pm today. We expect the downtime to last only about fifteen minutes.

We are replacing one of our DNS servers (oak.forest.net) today around noon PDT. The old Oak has reached it's end-of-life and will be retired, the new Oak has actually been running successfully (as "pine.forest.net") for quite some time with several test domains. This change should be completely transparent to our users and network, as it will involve removing the old oak from the network as we change the new oak's IP address. We will be making no DNS zone file changes between 11 AM and 5 PM today.

For those interested in history: This will be our third "oak.forest.net" server. The first was a Sun Microsystems SparcStation 5 that went live in February of 1995. The second is a Sun Microsystems UltraSparc 5 that went live in January of 1999. It is being retired today and replaced with a more modern machine running a current version of the BIND DNS server software. Oak, along with Willow, and occasionally through our history a server named Pine, have been the anchors of our domain name system.

At approximately 9:30 PM PST January 24th, digital.forest was affected by the worldwide internet attack known as W32/SQLSlammer. We were alerted within fifteen minutes of the first evidence of this worm as it entered our network. Preventative steps were immediately taken to minimize the impact on our network performance, and prevent damage to network devices.

At this point, all systems are performing normally, with the exception of a handful of client servers that are unfortunately affected by this worm. We are coordinating with clients to have their software patched and brought back online as soon as possible.

Any additional information on this worm can be found at:
http://www.cnn.com/2003/TECH/internet/01/25/internet.attack/index.html
http://www.bayarea.com/mld/mercurynews/news/5030801.htm
http://vil.nai.com/vil/content/v_99992.htm

Thank you for your ongoing understanding and patience.

One of our upstream providers is currently experiencing some problems which
are in turn affecting one of our DNS servers, willow.forest.tnet 64.69.73.4 Our NOC has called in and trouble ticket with the upstream provider. There is no ETR at this time.