Apple has released an official patch for the DNS vulnerability announced on July 8th. If you haven't already patched your server we suggest you do so immediately. It is available via your Software Update application in Mac OS X and Mac OS X Server.
posted by Chuck G. at 09:27 PM on Thursday, July 31, 2008 Categories:
With a lot of effort from our staff and clients the vast majority of the DNS servers in our datacenters have been patched for the vulnerability announced three weeks ago. However one major server & operating systems vendor has yet to release a patch, namely Apple. Since Apple uses ISC's BIND as the basis for their DNS under MacOS X and MacOS X Server there is no reason why you can not fix this issue yourself on a Macintosh server. With thanks to one of our clients, as well as an old friend of mine who used to work at Apple, we present to you a series of step-by-step instructions for patching BIND on a MacOS X system. These instructions install the update in a location and manner that will allow you to still install Apple's patch, if/when it is finally released.
The following instructions assume you are working locally ON YOUR SERVER HERE at digital.forest. Most will use Apple's "Apple Remote Desktop" or Netopia's "Timbuktu" for remote management. Advanced users can use SSH connections via a terminal and should understand where these instructions below have been simplified for users of a GUI admin tool. If you are unsure about any of these steps, feel free to contact technical support via telephone or trouble ticket.
Resolving BIND insecurity problems on your OS X box at digital.forest:
Option 1: Turn off DNS service. Seriously -- do you really need it? You can feel free to use our recursive resolvers for your server's DNS needs. They are located at 216.168.32.229 and 64.69.73.100. For your own client machines at home or at your office, use your ISP's DNS servers.
Option 2: Updating Your Own BIND
1) If you do not already have the Apple Developer Tools, join the developer program (http://developer.apple.com/ -- it's free), download and install them.
3) Open your Terminal application, and type the following commands:
cd ~/Downloads
(NOTE: This assumes the standard Safari download location) tar -zxf bind-9.4.2-P1.tar.gz cd bind-9.4.2-P1 sudo su
(NOTE: Type your password when prompted) ./configure --prefix=/usr/local make make test make install cd /usr/sbin mv named named.hold ln -s /usr/local/sbin/named named sync sleep 10 reboot
4) When Apple releases its patch, before you install it, launch your Terminal app again and type:
sudo su
(NOTE: Type your password when prompted) cd /usr/sbin mv named.hold named sync sleep 10 reboot
5) Install the Apple Security Update using Software Update
Hopefully Apple will release an official patch soon. Until then however, this is your only recourse to make your server safe from this vulnerability. Again, big thanks go to Glenn Fleishman and Rich Mogull of TidBITs, and Chuq von Rospach for their valuable insight.
--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.
posted by Chuck G. at 03:46 PM on Wednesday, July 30, 2008 Categories:DNS
The DNS Vulnerability we made you aware of recently has been cracked and there are reports of an exploit already "in the wild." If you are running a DNS server, here at digital.forest, or anywhere else we STRONGLY suggest you patch it IMMEDIATELY. The details of this vulnerability were originally scheduled to be announced in early August, giving people time to patch their servers. Unfortunately it appears now we no longer have that time. Let me repeat: If you are running a DNS server, we STRONGLY suggest you patch it IMMEDIATELY.
We addressed this issue with our own servers within 24 hours of the original announcement. Since then we have scanned our network internally and found many DNS servers, and have begun contacting those system owners. We'll have to accelerate that process considerably. If you know you are running a DNS server, please fix it now. If you are uncertain, please check. While we are contacting system owners, it would be better for them to take a proactive approach and not wait for us to call.
Please remember: According to our terms of service we reserve the right to remove your server from our network if it is being attacked or being used to attack others. For everyone's safety and convenience it is critically important that your servers are up-to-date with their security patches. Knowingly running an insecure server puts you uptime and stability at risk.
Thank you for your attention with regard to this critical matter.
--Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.
posted by Chuck G. at 08:49 AM on Wednesday, July 23, 2008 Categories:DNS, Security Alerts
shrubbery.forest.net, the server that processes stats for the older Mac servers and the Windows servers, will be going down later today for an OS upgrade. It will likely be Tuesday or Wednesday before it is fully operational again. We will update the support blog when the work is complete.
On July 27th, 2008 one of our upstream providers will be performing some maintenance on their equipment. They will be performing some BGP session resets and moving our connection to a new switch. There will be several small outages with the entire maintenance window lasting 30 minutes.
During these outage periods our other connections will carry all of our traffic. You may see some latency as routes shift. VPN connections running over this provider will drop and re-connect.
The maintenance will be starting at 1:00 AM PST.
posted by Kyle at 11:28 AM on Monday, July 21, 2008 Categories:Network
During our scheduled maintenance window on Tuesday night, July 22nd, we will be making some changes to our BGP configuration in an effort to better balance our traffic among our upstream connections. This should have minimal to no impact on network uptime, but there is the possibility that persistent connections (such as VPN tunnels) will reset as routes change.
The maintenance window for this action will start at 23:00 PDT on Tuesday, July 22nd, and conclude by 01:00 PDT Wednesday, July 23rd.
posted by Chuck G. at 06:07 PM on Friday, July 18, 2008 Categories:Network
We are currently troublshooting an issue on the Filemaker hosting server "grape.forest.net". One of the steps we've taken is disabling the Alder Database Manager. What this means for you is that if you need to manage your databases on grape, you will need to contact technical support (877-720-0483, option 3, or +1-206-838-1630, option 3) for help in this process. We apologize for any inconvenience this may present to you while we fix some stability issues with this server. This issues ONLY affects grape, and none of the other hosting servers here at digital.forest.
The mail server "palm.forest.net" will require some emergency maintenance tonight around midnight PDT. This will involve a restart of the server, which means mail service will be interrupted for about 5 minutes. No inbound mail will be missed as it will spool on secondary mail servers, but users will not be able to send or read mail during the brief outage. We appreciate your patience while we perform this required update.
We will be taking date.forest.net offline for about 5-10 minuets this evening to install some additional hardware in order to increase performance and increase total server traffic capacity.
This downtime will occur on Wednesday July 16th, starting at 23:50 PDT and ending at about 00:00PDT.
Please note: This event is not connected in any way to our main Seattle facility.
The city of Vancouver, British Columbia, Canada has experienced a widespread electrical outage due to a transformer vault fire as of 9 am today. From what we understand most of the downtown core is without electricity. digital.forest maintains some off-site servers at a facility on Hastings Street in Vancouver, specifically some secondary email and DNS servers. As of this time, those servers are online and operational, on generator power. The local power utility, BC Hydro, has provided an ETA of 9 pm to have power restored. It is unlikely that our servers will go offline, but we know that several of our customers do use these servers for secondary mail, and should be made aware of their status, even if the risk of outage is low.
We are keeping a very close eye on things from 150 miles away, and are in constant contact with people on-site in Vancouver. If anything changes we will update this post.
Again, this event has had no operational impact so far, and is in no way connected to our main Seattle facility.
posted by Chuck G. at 04:24 PM on Monday, July 14, 2008 Categories:DNS, Mail
We will be taking thyme.forest.net offline for about 30 minuets to an hour this weekend to address some if not all of the performance degradation issues that clients on thyme.forest.net have been experiencing over the past few weeks.
This downtime will occur on Sunday July 13th, just after midnight Pacific Daylight Time and will last until around 1:00 AM.
We're seeing large backlogs of outbound mail to yahoo.com mail servers right now. Expect delays in delivery up to 5 hours if you are sending mail to recipients with yahoo.com mail addresses. Mail is getting through (though yahoo is also arbitrarily rejecting some mail as well) just very slowly.
It is pretty common for us to see delays getting mail to Yahoo, but today seems particularly bad for some yet unknown reason. Unfortunately there is not much we can do since the issue is on their end. Please be patient while the queues clear.
posted by Chuck G. at 06:23 PM on Wednesday, July 9, 2008 Categories:Mail
As promised, here is the time lapse video of our roof work from early this morning. Big thanks go out to our contractors, MacDonald-Miller and Ness Crane, and of course our Facilities Manager Kevin Teker for another successful project completed. Great Job Guys!
Stay tuned for more information about datacenter expansion.
posted by Chuck G. at 01:22 AM on Wednesday, July 9, 2008 Categories:Datacenter Expansion
Today a major vulnerability was announced in the Domain Name Service protocol. You can read the details in the US-CERT announcement here: http://www.kb.cert.org/vuls/id/800113.
We are performing an assessment of our own DNS servers, planning to patch them, and make the configuration changes as required. We will post more on that as needed. The main purpose of this post is to inform our clients who run their own DNS servers inside the digital.forest facilities about this vulnerability. Please note that this is a fundamental vulnerability in the DNS protocol, so it is not vendor specific. This means that virtually EVERY device that can operate as a DNS server is vulnerable. We strongly suggest that you consult with your equipment and software vendors to ascertain your exposure and take appropriate action.
We'll post more information as it becomes available.
posted by Chuck G. at 04:22 PM on Tuesday, July 8, 2008 Categories:DNS, Security Alerts
Today is a big day for digital.forest: Our newest datacenter cooling unit arrives on the roof!
Above: Mt. Rainier rises behind the crane.
We were up at the crack of dawn today to accept delivery of our newest Aaon 70 Ton rooftop cooling unit for our latest datacenter expansion. The sun rises around 4:40 AM this time of year so we had most of the operation complete by 8 o'clock, specifically the most difficult part of the process, lifting the components up onto the roof with a crane. The crane arrived at dawn and set up on the east side of our building. Flatbed trucks containing the steel frame for supporting the unit, and the cooling unit itself arrived in turn and their cargoes were lifted successfully.
The first item up was the steel superstructure. Its purpose is to extend the strength and support of the steel frame of the building up through the roof to support the cooling unit's weight. While our Aaon units are very compact and lightweight, extending the building frame to support them allows us to put more of them on the roof, therefore increasing the density and capacity of our datacenter. You can see our two previous units installed and running in the background. This unit brings our total number of cooling units up to four, with a total capacity of 333 tons.
Above: Workers prepare the steel frame and duct work for the arrival of the cooler.
The cooling unit is a "makeup air handler" which means that during the cool nights and all through the winter it operates in an "economizer" mode, meaning it uses cool outside air to chill the datacenter. When it gets warmer outside it can mix mechanical cooling with blended outside air, or on our rare hot days here in Seattle, it can go into full mechanical cooling. These are highly efficient units, so our energy use drops by up to 50% when it is cool outside, which here in Seattle is most of the time.
Above: The cooling unit being prepared for lifting.
Above: Arrival! The unit hovers centimeters above the steel frame.
Above: Kevin Teker, digital.forest Facilities Manager gives the thumbs up, indicating that the unit has landed safely and the crane work is done.
The rest of the morning our contractor, MacDonald-Miller will be securing the unit and preparing it for startup. Stay tuned, as we'll have updates, including a time-lapse video of today's work.
Regards,
Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.
posted by Chuck G. at 11:27 AM on Tuesday, July 8, 2008 Categories:Datacenter Expansion
We will be taking date.forest.net offline for about 30 minuets to an hour this weekend to apply several patches to the Lasso server in order to bring it up to version 8.5.5. This will alleviate some if not all of the performance degradation issues that clients on date.forest.net have been experiencing over the past few weeks. We have beta tested this update and found that it will not adversely effect the sites on this server.
This downtime will occur on Sunday July 13th, just after midnight Pacific Daylight Time and will last until around 1:00 AM.
In observance of Independence Day, our business offices (Sales & Billing) will be closed on Friday, July 4th. Regular hours will resume at 8 am on Monday, July 7th.
Technical Support will remain open all day Friday, and all through the weekend. Colocation customers without card keys requiring access to the datacenter anytime between 6 pm Thursday, July 3rd and 8 am Monday July 7th will need to contact our NOC for access to the building and elevators as they will be secured throughout the holiday weekend.
posted by Chuck G. at 10:40 AM on Wednesday, July 2, 2008 Categories:Holiday Hours
We will be taking the following servers offline for about 10-15 minuets each tonight (Tuesday July 1st, 2008) between the hours of 11:00 and 00:00 to apply an urgent security patch released yesterday for these servers.
