Over two-thirds of our Technical Support staff are out sick at the moment with some sort of mild illness making the rounds here in Seattle. As a result we are a tad shorthanded. Please accept our apologies for any delays is handling incoming calls and support tickets. We appreciate your patience!

Our maintenance could not be completed this morning. We will be doing the work during our scheduled maintenance window this evening between 11:00 pm and midnight.

Our network maintenance originally scheduled for the early tomorrow morning has been pushed back a bit by the vendor. It is now scheduled for the early morning hours of Monday, January 28th.

We will be adding a new BGP Peer over the weekend. The actual cross-connect of the fiber circuit is happening today and BGP turn-up will happen sometime in the wee hours of Friday or Saturday. We'll be adding AS4323, also known as Time-Warner Telecom via a gigabit Ethernet connection. TWTC will be replacing our Fast Ethernet circuit we've had with AS2828, also known as XO Communications.

This should have no impact on service for any of our clients, just a change in routing at our network boundary. If anything we should see an improvement in performance overall. No changes to our other connections is scheduled.

In a few weeks we will be adding another Gigabit Ethernet circuit with AS3356, also known as Level(3). We'll post more news on that as it approaches.

Sharp eyed observers may have noted that when you complete a trouble ticket in our Web Help Desk system it asks if you would like to take a survey. We value feedback from our clients and make use of these surveys all the time to improve our support.

The ticket system is very valuable to us since it ensures that no issue can "fall through the cracks" and our tech support staff can work around the clock on problems, handing them off when shifts change. Trouble tickets allow us to note trends and anticipate issues. The post-ticket surveys allow YOU to provide feedback to us about how satisfied you were with response times, quality of support, and length of time it took to resolve your issue.

We want to encourage our clients to fill out these surveys so we decided to pull names at random from the last two month's of survey responses and award them with a free gift. The names we pulled out were Daniela MacLeod from Rhythm Interactive, and her survey response on October 16th, and Dean Pachl from Photoworks for his survey response on November 11th. We're sending Daniela & Dean an Apple iPod in thanks for there participation in our trouble ticket survey.

You too can win an iPod. Just fill out a survey after your trouble ticket is completed. Provide us with feedback about your support expereience and you'll automatically be entered in our drawing. We appreciate your input and thank you for your time.

--Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

We've had a few phone calls and emails from some of our valued clients around the world asking if we're OK, or under water. Have no worries! All the scenes on TV and pictures in the newspapers of flooding and damage are happening in other parts of the region. Your servers are high and dry, and we're comfy and warm. We do appreciate your concern, and our thoughts are with the people of the region who are in the thick of it, dealing with floodwaters & mudslides.

When we moved in late 2004/early 2005 we relocated to an area south of downtown Seattle and our building is safely up on a hill, far away from any water sources. Additionally our datacenter is on the top floor of the building, six stories above the ground. I do not have my camera today, otherwise I would post a photo of the facility and the environs to show how things look. The sun was even out a little while ago.

Ironically one of the areas hardest hit by the flooding is Bothell, specifically the area around North Creek Parkway, where we operated for seven years between 1997 and 2004. That area is home to many datacenters and other high-tech industries and we hope they are doing OK. In hindsight though we're very happy we relocated here, for many more reasons than just the flood risk.

Thanks again for your concern.

Chuck Goolsbee
VP Tech Ops
digital.forest, Inc.
Seattle, WA

In observance of the Thanksgiving Holiday, digital.forest will be closed Thursday November 22rd and Friday November 23th. We will resume regular business hours at 8am PST Monday November 26th.

Technical Support staff will remain on-site 24 hours a day throughout the holiday period. Please be aware that we'll have limited staff coverage for telephone tech support during the next few days. Please note that our building will be locked throughout the weekend and clients requiring access to the datacenter to work on colocated servers will have to call or email first to be allowed access to the building. We appreciate your patience in this regard.

All of us at digital.forest wish you a happy Thanksgiving and a wonderful holiday season!

Chuck Goolsbee
VP, Tech Ops
digital.forest

digital.forest is proud to announce that it has launched its first FileMaker 9 Advanced server, the new server can handle 125 simultaneous databases and up to 250 simultaneous FileMaker Pro clients.

Our new FileMaker 9 server is an impressive dual 2.66 GHz dual-core Intel Xenon Apple Xserve with 8GB of 667MHz DDR2 RAM and 300 GB of Mirrored SAS 15k RPM data storage.

For more information about FileMaker 9 please check out the FileMaker website at http://www.filemaker.com/

We're pleased to announce that digital.forest has opened a new technical support forum. This new forum is designed to give digital.forest customers the ability to ask questions of each other related to the services that digital.forest provides such as Filemaker databases, Lasso, PHP, etc. These forums will have some limited interaction from the dgital.forest staff however it is mainly a forum in which users can exchange ideas, ask questions, and interact as a community. We know that we have some smart folks among our clients and we hope to have the forum be a great place to exchange knowledge, both between us and our clients, but more importantly from client to client.

You can find this new service at forums.forest.net. There is also a link in the "Tools" menu over there. -->

This will not replace our trouble ticket system, which will remain the primary method of one-to-one technical support for our clients. Nor will it replace this "support blog" as a way for us to announce events, maintenance, and news about digital.forest. Instead we hope the forum will be a great place to share ideas among many people at once. Have a look. Create a login. Start a discussion. Help somebody out, or get some help for yourself. Let us know what you think.

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

That is a tower of power you see above. It is the new benchmark for electrical density here at digital.forest. 528 Watts per square foot'. We know that isn't a world record, but it is a very respectable number for a colocation facility here in the Seattle area. Few can approach it, much less match it.

We can't share the name of the client (shhh... they are a startup in "stealth mode") so we've blurred the pictures a bit and removed any identifying marks. They are however VERY happy that they found digital.forest.

Prior to being here, they were colocated at a competitor. They were NOT allowed to put this many servers into their rack at this competitor. In fact, in order to have this many servers, they had to buy FOUR RACKS.

We think that's nuts. So did our new client. Now they are happily colocated at digital.forest and can do it all in one rack.

At digital.forest we are here to assist our clients to succeed, not restrict them from what they need. It is just one part of the digital.forest difference.

Sharp eyed observers will note that the number changed on our home page today. Yes, today is our birthday. It was 13 years ago today that digital.forest sprouted and began to grow. The number of hosting and colocation companies that were around in July of 1994, and are still around is very small. It has been a crazy 13 years in this business that is for sure.

We know that we owe our success and longevity to YOU, our valued clients. There are several of you that have been with us since that very first year! To that we say "THANK YOU!"

There are also many of you that have only found us relatively recently, such as after our 2005 move to our Seattle facility at Intergate.West. To you we say "We hope to serve you long and well!"

If you call the Support, Sales, or Customer Service staff today, wish us a happy birthday and enjoy the cake & ice cream.

In observance of the Independence Day holiday, digital.forest's offices will be closed Wednesday, July 4th. Business will resume on Thursday morning at 8:00 AM PDT, July 5th.

Technical Operations & Support will remain open through the entire day, as always, and through the holiday as well. Datacenter access remains available, though the building will be locked and clients wishing to access the datacenter will need to call the support phone line (206-838-1630, option3) to gain entry into the building.

We wish our clients and their families an enjoyable holiday.

--Nik Keating
Technical Support Manager
digital.forest, Inc.

In observance of the Memorial Day holiday, digital.forest's offices will be closed Monday, May 28th. Business will resume on Tuesday morning at 8:00 AM PDT, May 29th.

Technical Operations & Support will remain open through the entire weekend, as always, and through the holiday as well. Datacenter access remains available, though the building will be locked and clients wishing to access the datacenter will need to call the support phone line (206-838-1630, option3) to gain entry into the building.

We wish our clients and their families an enjoyable holiday weekend.

--Chuck Goolsbee
VP
digital.forest, Inc.

At 9:50 AM this morning one of our Metropolitan Ethernet providers, OnFiber had an equipment failure here in Seattle. We connect to one of our network peers, NTT/America at The Westin Building via this circuit. This caused us to have have intermittent connectivity over that particular circuit to NTT/America. Some digital.forest clients may have had "slow" or "intermittent" issues reaching servers here for a short period of time while we diagnosed the issue with the NOC's of NTT & OnFiber

We have shut down our BGP connection to NTT/America while OnFiber fixes the problems on their network. At the moment we are running on two of our three network connections. We will update this post when we bring the third circuit back online.

Update: As of 11:02 AM PDT this issue is completely resolved. The OnFiber circuit was manually moved to a different port. After a successful 10-minute testing of the new circuit we turned up our BGP session with NTT/America.

We maintained connectivity to our other BGP network peers through this event, so at no time was our network "down". We do like to keep our clients informed of events here at our datacenter, even if they have no direct impact on your servers. In this case, it was a classic example of Internet Architecture and how it handles outages. The often-used phrase is that it "routes around damage." In this instance when one of our circuits had an issue our traffic just shifted to our other circuits. It is likely that none of our clients even noticed. If they did notice it would have been an intermittent connectivity for a brief period of time. Such is the nature and reason for designing redundant systems. Our fiber optic connectivity to the rest of the Internet flows over multiple physical paths. Those paths do not converge until they are physically inside our datacenter facility. This prevents complete outages through equipment failure or accidental fiber cut. Today's event confirms the built-in redundancies work as designed.

--Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

Yes, Seattle has been hit with a cold snap, but here at digital.forest we're still comfortable and the servers are running cool in the datacenter. The image above was taken just before dawn today. That is Mt. Rainier in the distance with the digital.forest world headquarters under the moon in the foreground.

No, it isn't really "breaking news" but we thought we'd share the view.

The Seattle area experienced a major windstorm last night. This had no effect on our network operations, however, several of our staff are having difficulty reaching the office due to downed trees and power lines. As a result, support calls may take somewhat longer to be answered than is usual.

Thank you for bearing with us,
Digital Forest

The Puget Sound area of western Washington awoke this morning to anywhere from a few inches to over a foot of snow. While this should have no impact at all on our network operations, it likely will have some impact on staffing levels at the office to handle your technical support, customer service, sales, and billing issues. Our region's transportation systems do not cope well with snowfall, so it may take some time, more time than usual, for people to get to work. Some of us may not make it in at all.

The storm is expected to end by tomorrow, so things should be back to normal soon. We appreciate your patience.

--Chuck Goolsbee
VP, Technical Operations
digital.forest
Seattle, WA

In observance of the Thanksgiving Holiday, digital.forest will be closed Thursday November 23rd and Friday November 24th. We will resume regular business hours at 8am PST Monday November 27th.

Technical Support staff will remain on-site 24 hours a day throughout the holiday period. Please be aware that we'll have limited staff coverage for telephone tech support during the next few days. Please note that our building will be locked throughout the weekend and clients requiring access to the datacenter to work on colocated servers will have to call or email first to be allowed access to the building. Additionally we will likely take advantage of the holiday lulls to perform maintenance and upgrades on core equipment. We will post notice of these beforehand. Finally, we are closing the phone support queue for a while on Wednesday, November 22nd in the afternoon for a Tech Department meeting. We appreciate your patience.

All of us at digital.forest wish you a happy Thanksgiving and a wonderful holiday season!

Chuck Goolsbee
VP, Tech Ops
digital.forest

Today, July 12th is our 12th anniversary of operation here at digital.forest.

We enjoyed a brief celebration in the office with some root beer floats to commemorate. I wanted to take this opportunity to thank all of you, our valued clients, for your loyalty and support over the past dozen years. The company heads into our teens with a typical growth-spurt with the completion of our datacenter expansion by the end of this month - stay tuned for more news.

Thanks again for an excellent dozen years, and we're looking forward to the next!

--Chuck Goolsbee
V.P., Technical Operations
digital.forest

The digital.forest offices will be closed Monday July 3rd, and Tuesday July 4th in honor of the Independence Day Holiday. Regular office hours will begin again on Wednesday July 5th at 8 AM PDT.

Technical Operations staff will remain on-site throughout the holiday weekend. Emergency (server down) support & datacenter access for colocation clients, as always, will remain available 7/24 through the holiday. Please note: Since we'll have a minimal number of tech staff on-site, please be patient with non-emergency support issues. Even server-geeks need a day off now and then.

Have a safe & happy 4th.

--Chuck Goolsbee
V.P. Technical Operations
digital.forest

In observance of the Memorial Day Holiday, digital.forest offices will be closed, Monday, May 29th. Sales, Billing, and Technical Support departments will be closed. We will maintain minimal on-site staff, so emergency support will still be available via normal channels. Normal operations will resume Tuesday, May 30th.

Tonight, during our normal scheduled maintenance window, we will be updating the firmware of our shared fibre channel storage array. This will require a reboot of the array. We will also be rebooting the fibre channel switch that attaches the array to the servers. This will happen just after midnight and should take just a few seconds. It may not even be noticeable from attached hosts.

Today I will be applying security updates (Apple Security Update 2006-001 & Security Update 2006-002) during the day today. The downtime for each machine will be less then 5 minutes. Below is a list of servers that will have the security patch installed.

alder.forest.net
banana.forest.net
cactus.forest.net
grapevine.forest.net
newninewire.forest.net
sage.forest.net
slime.forest.net
spruce.forest.net
tangerine.forest.net
date.forest.net
fungus.forest.net
helpdesk.forest.net
honeysuckle.forest.net
jasmine.forest.net
thyme.forest.net

Yvo

It seems that between 4 PM PST and 8 PM PST our support system wouldn't allow the creation of new tickets if you only had one account number with us. We are looking into the problem that caused this but suspect that it was a misconfiguration made by the software developer.

This problem has been fixed and the creation of tickets should once again work. The problem did not effect the tickets already created in the system.

Tonight we will be performing a very minor maintenance on helpdesk.forest.net. The maintenance will begin at 10pm. This maintenance should last about 10-15 minutes. During this time helpdesk.forest.net may be completely unavailable and/or slow.

Services affected:
Our trouble ticketing system - Web Help Desk

Thank you,

Yvo V.
digital.forest technical support

We are closed today, Monday January 2nd. Emergency "server down" support is available via normal channels. Regular tech support coverage will resume at midnight tonight.

In observance of the New Year's Holiday, digital.forest sales and billing will be closed Monday, January 2nd. Normal business operations will resume at 8 AM PST on Tuesday, January 3rd.

Technical Support will remain open through the weekend, but will be closed all day Monday, January 2nd from 6 AM PST until Midnight. Emergency "server down" support will continue to be available to digital.forest clients Monday. Normal technical support coverage will resume at Midnight PST, Tuesday morning January 3rd.

We'd like to extend our wishes for a happy new year and a successful 2006 to all of our valued clients.

Chuck Goolsbee
V.P., Technical Operations

At approximately 1:45 AM PST this morning we experienced a power overload condition on a single electrical circuit, which services half of two racks in our facility. This tripped a breaker in one of our Power Distribution Units. We reset the breaker and used power monitoring equipment to measure the load on that circuit as servers rebooted. With the data collected rerouted some power cables in those two racks to spread the load in a manner which should prevent this from happening again.

Most of the servers affected are digital.forest shared hosting servers, however one of the two racks contained some colocated and one dedicated server. We will be contacting the affected clients during the business day with a follow-up.

Server downtime was limited to about 20 minutes maximum, with most servers being down less that 15 minutes.

We apologize for any inconvenience.

Chuck Goolsbee
VP Technical Operations

We experienced an issue with our telephone system over the holiday weekend where our phones reverted to "normal business hours" mode at some point either Sunday night or Monday morning. This meant that if anyone called the support line instead of the message stating we were closed, they would be put on hold in the support queue. We apologize to anyone that experienced that, especially since the music on hold - with our traditional surf-rock Holiday tunes - was also not working.

The issue is under investigation and we'll make sure it doesn't happen again. Thanks for your understanding.

In celebration of the Christmas holiday, digital.forest sales, billing and support will be closed from Friday, December 23rd @ 7pm through Tuesday, December 27th @ 8am. Emergency "server down" support will remain available via normal channels throughout the weekend.

We extend to all of our valued clients a Happy Christmas and a wonderful holiday season.

Chuck Goolsbee
VP Technical Operations

Just an FYI to our valued clients. The technical support staff at digital.forest is 90% down due to a nasty cold and holiday vacations. Please be patient while the healthy couple of us tackle the full load of support. We appreciate your patience as we may be a bit slow to get to phone calls or work on Web Help Desk requests.

Thanks
Shawn Hammer
Director Technical Operations

Update: 6:30 PM We are, once again, open for business. Thanks for allowing us this brief break for celebrations.

We are shutting down the support phones in a few minutes for our annual digital.forest Christmas party. We're still here and available for server-down emergencies, but due to the impact of Infamous Bill's Evil Eggnog, we're unable to perform more complex support tasks. Normal technical support will resume later this evening.

Thanks for all of your support! 2005 has been an eventful year here at digital.forest with our facility move and acquisitions. We're looking forward to a happy and prosperous 2006 for all of our valued clients.

Merry Christmas, and Happy New Year from all of us at digital.forest!

Chuck Goolsbee, VP Technical Operations

It has been brought to our attention that our toll-free phone number (877-720-0483) is out of service. We have contacted the Telco, and they have assured us that the issue will be resolved shortly.

In the meantime, you can contact us via our regular number: 206-838-1630

Network Solutions, one of the major DNS Registrars, is having trouble with their DNS servers, which is affecting some of our clients.

Your site may be affected if you are using Network Solutions as both your DNS Registrar AND to host the DNS for your site (ie, your domain uses worldnic.com addresses for its nameservers).

From their site (as of 9:55am):
"We are currently experiencing a widespread DNS related outage. If you are attempting to contact us to report a product that is not working, please know that we are aware of the issue, and are doing everything we can to resolve all issues as quickly as possible."

If you are affected by this issue, please keep an eye on Network Solutions' website for updates:
http://www.networksolutions.com

In observance of the Thanksgiving Holiday, digital.forest's business office will be closed Thursday, Nov. 24th and Friday Nov. 25th. Technical support will remain open through the holiday and weekend.

The Web Help Desk system here is down with a software problem. We are working with the vendor to get it working again. In the mean time, feel free to call our technical support group toll-free in North America at 877-720-0483, option 3, or
+1-206-838-1630, option 3 from anywhere Internationally.

We expect the Web Help Desk system to be back online within an hour or two.

UPDATE: Our Web Help Desk is back online.

Just an FYI to our valued clients. The technical support staff at digital.forest is 50% down due to a nasty cold half of us have caught. Please be patient while the healthy half of us tackle the full load of support. We appreciate your patience as we may be a bit slow to get to phone calls or work on Web Help Desk requests.

Thanks
Chuck Goolsbee
VP, Tech Ops.

In observance of the Labor Day holiday, digital.forest will be closed Monday, September 5th. We will resume normal business hours on Tuesday, September 6th.

Emergency "server down" support will be available during the closure.

Update: 5:40 PM PDT: Our phones are working fine once again. Thanks for your patience.

...

Update: 4:15 PM PDT: The source of the issue has been discovered: an error by a Qwest technician two building south of us. We are awaiting an ETA on fully restored service.

...

We have received some reports that some telephone calls into our offices are being met with an 'all circuits are busy' message. We are looking into this with our voice provider. At the moment we do not know if it is a regional telephone issue, or one specific to our number.

We are not experiencing any issues with regards to our Internet connectivity.

Thanks for your patience.

Friday, August 5th, from 3 pm until 5 pm PDT the digital.forest tech support staff will be off-site for a very important meeting. Our wonderful Customer Service staff will be covering the support phones for us while we are out. Emergency support and our trouble ticket system will also be available in our absence. Normal support will resume after 5pm PDT.

In observance of the Independence Day holiday, digital.forest will be closed on July 4th. Emergency "server down" support will be available through normal channels. Regular technical support, sales, and customer service hours will resume at 6 AM PDT July 5th.

Have a safe and happy holiday weekend.

In observance of Memorial Day, digital.forest will be closed Monday, May 30th. We will reopen at 6am PDT Tuesday, May 31st.

Emergency (server down) support will be available via normal channels.

Our scheduled overnight onsite tech could not make it in tonight (Sunday, March 27th) due to a family issue. The on-call techs are still available and answering emergency issues. Facility access may require an approximate 15-20 minute delay. Please be patient if you require facility access. Thanks.

We will be back to our normal 7/24 onsite by 6:00 AM PST Monday morning.

When Apple announced the MacMini at Macworld Expo last January, an immediate discussion began in the technology community about the possibility of using it as a server. Just like when Apple introduced the G4 Cube, and the Xserve, companies even came out of the woodwork, basing their whole business model on MacMini Colocation. While the latter is a bit absurd, the former is worth discussing. While digital.forest has never been a "Mac Only" facility (our very first server in 1994 was a Sun Microsystems SparcStation 5) we have always been very open-minded about platform, and embraced the Mac as one of many we support. The Macintosh community, a fiercely loyal group of customers that any company would be wise to embrace, has shown digital.forest as much loyalty as they have shown Apple. This is how we have become the largest collection of MacOS servers on the Internet, at least outside of Apple's corporate datacenters.

People who run servers are always attracted to small form-factor computers because when you buy colocation, part of what you are buying is space. The more computer you can pack into the least amount of space the better... so the theory goes. It also helps that Apple is selling the MacMini for a very low price. A full-blown FreeBSD UNIX install on a sub-$500 computer.

Given the insane colocation market now the cost-per-rackspace argument is kind of moot. But the appeal of a spending under $500 for a server is hard to argue with... even if it may not be a significant performer. An Xserve is far better suited to serve moderate-to-high traffic sites, but as a low-volume mail server for a small office, or a "personal server" the bang-for-the-buck of the Mini isn't too bad.

So it didn't surprise us today when Phil Herring, a long-time client of digital.forest showed up today to replace his aging "blue & white" G3 mail server with a brand new MacMini.

Above: Phil Herring unpacks the lunchbox-sized container that the Mini ships in. This is the first server that has ever come into our datacenter carried one-handed.

Above: Phil Herring holds up his new server.

Above: This little computer is twice as powerful as the one it sits upon, and 3X more that the one it is replacing. We'll have to find out from Phil how well it performs.

I'm keen to find out myself actually, as we have racks and racks of G3 & G4 computers that carry pretty light loads. Mind you we also have racks and racks of G4 & G5 machines and Xserves that carry pretty high loads... but the combo of Xserves and Minis might be an interesting "Big & Little" option. Personally, I'd like to see an "Xserve lite" more tailored to the realities of the datacenter and Internet serving, (Apple should look at Dell's 1U & 2U servers... very nice) but until that happens, we may be looking at a lot of Minis... who knows.

--chuck goolsbee
vp of technical operations
digital.forest
seattle, wa

Our tech support staff will be in a meeting today from Noon until around 1:30 PM PST. Our tech support phones will be closed during that meeting. You can leave us a message, or better yet, use our online help desk system to log your questions and requests. Sorry for the inconvenience.

Emergency electrical maintence will be done in the datacenter today causing short outages of no more than 5 min. Servers will be gracefully shutdown and powered back up within the 5 min.

Update:
We are halfway done with the work needed in the Datacenter. The second half will begin shortly.

Update: As of 1:10 PM PST, all work has been completed. Sorry for any inconvenience this may have caused.

The power event we experienced at our Bothell facility on Wednesday night was in layman's terms, a "brownout". This explains why some servers were affected, and others were not.

The source of the event was our UPS. This is obviously a concern as one of the two primary functions of a UPS is to prevent spikes or drops in voltage (the other being a buffer between power sources.) Our UPS has been well-maintained with twice a year preventative maintenance since we acquired it in late 2000, so this can not be a maintenance issue. We replaced batteries in it following a very similar event on August 25th, 2004 - triggered by a lightning strike near our facility and the resulting surge damaged our batteries. We have been using the vendor of this UPS for ongoing maintenance and support since digital.forest opened for business in 1994. They have been quite responsive to our needs when dealing with events like this. However, we would prefer to avoid events like this.

We applied lessons learned over the past 10.5 years and have selected a superior UPS system for our new Seattle facility. The entire power system of the new facility is several orders of magnitude better than the one in Bothell. The inbound mains, and backup power systems are much more "industrial" in nature - the power distribution systems are a significant improvement - and the UPS is about 3 times the size and capacity of our Bothell unit. The Seattle UPS also has built-in redundancy in the form of multiple, independent battery cabinets that can be run in serial or parallel depending upon need. The campus of buildings where we are moving to was purpose-built for datacenter facilities with significant power infrastructure. Our new neighbors in Seattle are all industrial datacenter operators, either corporate or telecommunications providers. An example list of them includes Microsoft, Washington Mutual Bank, AboveNet, Savvis, OnFiber, Qwest, ELI, AT&T, Level3, etc. This is a significant change from the "office park" surroundings in Bothell.

The UPS in Bothell will be decomissioned at the end of March when we vacate the old facility and not re-used.

Finally, a question we have heard from many clients is "Why didn't the generator kick in?" The answer is fairly simple. The generator serves as a backup power system should power from the Public Grid fail. The UPS is further down the path of power. Electricity flows in this manner: Public Grid and/or Generator -> Transfer Switch -> UPS -> Servers. That is of course simplified, there are transformers, bypasses, breakers, etc mixed in, but put simply the UPS is "downstream" from the Generator. We have built a redundant power system, but a component failure is always a possibility no matter how it is done. In this case we had a voltage drop that lasted less than 1/10th of a second, but it was enough to cause problems for about 45% of the servers in our datacenter.

We would like to thank you for your patience and understanding, and we appreciate your ongoing business and support. We continually strive to improve our services and facilities, and this is proven by our commitment to our new datacenter.

Preliminary investigations into what happened last night at our Bothell
facility indicate some sort of failure in our UPS or power distribution
systems. This was not a Utility power failure, but some sort of internal
one. We are going over the data we have with our UPS vendor and should have
more answers soon. Please note that the power interruption was less than a
fraction of a second and many servers were unaffected by this event.

We have reacted to this event with the swiftness and thoroughness that
you’ve come to expect with our company. Additional follow up with those
effected in addition to more specific information at to what happened and
how we plan to address it will be forthcoming.

We experienced a power failure this evening at our Bothell facility. It was very brief, and we recovered within a second or two but some systems are having problems coming back up, notably our mail server "treehouse"... we are looking at that now and will update the site with news as soon as we have it.

Update: 6:50 PM: Treehouse's external RAID array is rebuilding. This may take a while. We are also performing a restore from backup onto treehouse's internal array. Given that the latter is from almost 20 hours ago, we would prefer NOT to use it. We will continue updates in a new blog entry.

The Seattle area woke up to several inches of snow Sunday morning, with more forecast. At the moment I am trying to ascertain if the weekend staff have made it to the office or not. I suspect not given the current road conditions. This should have no serious affects on our operations. The sole exception being a server that requires some human interaction. Most things can be handled remotely, but on occasion situations require face-to-monitor time in the datacenter. If you send an emergency email, please be patient with us if it involves direct interaction with the server, as travel time to the datacenter will be longer than usual.

If history is a guide, things could warm up and our usual rains return at any moment, so this is unlikely to be a long-term issue.

Thanks for your patience.
Chuck Goolsbee, VP, Technical Operations.

We will be closing the Tech Support Phones this afternoon (December 14th) at 3 PM PST for our annual company Christmas party. Emergency support and the Web help Desk will still be available of course.

Thanks for your continued support and loyalty to digital.forest. We wish you and yours a happy Holidays and a prosperous New Year. 2005 will be a very exciting one here at digital.forest with the move to our new facility, a great start to our second decade!

Stay tuned.

http://www.forest.net/support/

We experienced a power event this morning at 9:24 AM PDT, possibly related to a lightning strike in the area. During the event our emergency backup power system deployed but was damaged. Our UPS took some sort of surge or spike that damaged the unit. Some servers switched off or lost power. Power was fully restored within one second.

Our UPS vendor (APC) is en route to assess the damage and repair. Our generator is running and will remain our primary power source until the UPS is confirmed to be operational again.

Thanks for your patience and understanding while we deal with this emergency.

We are experiencing some problems with the server that hosts the Helpdesk. As a result, helpdesk.forest.net is unavailable. We are currently working to bring the system back online.

In celebration of the Independence Day holiday, digital.forest will be closed today (Monday, July 5th). We will reopen tomorrow morning.

Emergency support is available via the usual channels.

A life-threatening medical emergency has struck our on-call tech's immediate family. We have staff en route to our facility now, to cover his absence, but there may be delays this morning (Saturday, June 5) answering our pager and emergency emails. Please be patient with us while we adjust our weekend staff schedule on very short notice.

Thank You
Chuck Goolsbee
V.P., Technical Operations

The work on helpdesk.forest.net is complete, and the Help Desk system is now available.

If you use an RSS aggregator/reader, you can now subscribe to digital.forest's support updates via RSS feed. The subscription URL (which will always be available on the support pages) is:

http://www.forest.net/support/index.xml

Simply copy-and-paste that URL into your RSS reader.

Popular RSS readers for Mac OS X include NewsFan, NetNewsWire, and others.
For Windows, you might try FeedDemon, RSS Bandit, or others.

digital.forest will be closed for the Christmas holiday, starting December 24th, at noon. We will re-open again Monday morning, December 29th at 8 AM PST.

Emergency support will still be available via the usual channels.

All of the staff here at digital.forest wishes you a very happy holidays and an excellent new year.

From the "chuckcam"...

Visit the digital.forest web cams.

We're closing early today for our annual Christmas party. Thanks for your continued support of digital.forest. We all wish you a happy holidays and a wonderful new year!

Emergency support will still be available via the usual channels.

From the "chuckcam"...

The digital.forest web cams.

During the coming holidays digital.forest will shorten our technical support telephone hours by 90 minutes. Telephone support will now begin at 6:30 AM PST. Emergency "server down" support will continue to be 24/7.

In order to celebrate the Thanksgiving holiday, digital.forest offices will be closed Thursday & Friday, November 27th & 28th. digital.forest support staff will be on-site as usual up until midnight Wednesday, November 26th, and will return to normal hours Monday, December 1st. Emergency support will remain available through the long weekend for server down situations.

digital.forest staff would like to wish all of our valued clients a happy holiday season.

The Recover-It service is now and running on a new server. We have recovered as much data as possible from the old server, but the RAID failure may have corrupted some data; we strongly urge all Recover-It customers to perform a "Full" or "New" backup as soon as possible to ensure that all their data is backed up.

We do have some data left in the "Lost+Found" directory; if you are missing data, and a new backup is not feasible, please call and we will do our best to recover anything we can find in that directory that belongs to you.

We fully expect the new configuration to eliminate these problems, and we apologize for the inconvenience.

We have decided to move the Recover-It service to a new server, eliminating the troublesome RAID hardware from the configuration. The service will be down overnight; we will be adding users to the new server and moving data over from the old one throughout the day tomorrow. We apologize for the inconvenience.

As passwords were often not recorded by the automatic setup process, we may be contacting Recover-It customers to reset passwords as we configure the accounts on the new server.

The Recover-It service is currently down due to a hardware problem with the RAID. We are working on either repairing the problem, or re-engineering the service to use a different storage method. It may be a day or two before the service is back online; we will post updates here as soon as we have more information.

We are experiencing some inclement weather here today, with very high winds, and have already had several power outages in the past hour. This has not, and should not have any adverse affect on our datacenter, as we have sufficient UPS & Generator backup. However there is a possibility that we could lose our voice communications. If for some reason you have any trouble contacting us via telephone, please use our Web Help Desk. The winds are expected to calm by evening.

We are taking a break from our daily operations for a digital.forest Holiday Party. We have shut down the phones, headed upstairs for nibbles, and we'll be heading out for dinner around 5 PM PT. We do have the pager, so we can be reached if an emergency arises.

Let me take a moment to thank you, and wish all of our clients a happy holiday season and a great new year! We look forward to our continued professional relationship.

Regards,

Chuck Goolsbee,
V.P. Technical Operations

Happy Holidays!

From L-R: Chris, Damian, Nick, Peter, Ryan (kneeling), Jim (hiding), Bill, & Chuck.

phpMyAdmin on crabapple.forest.net will be unavailable for approximately 30 minutes while we upgrade it to the latest version.

Update: phpMyAdmin 2.3.3 has been successfully installed on crabapple.forest.net.

Have a look at our Bothell datacenter, with the video tour!

http://www.forest.net/about/datacenter/videotour.html

You may have been with digital.forest for quite some time, but have you seen it from the inside? We are very proud of the fact that we have our very own datacenter and dont lease space from another provider. This allows us to focus on the services that our customers need and create the best location for all your hosting needs!

We are pleased to announce an upcoming new trouble ticket system! We are confident that this new system will help us to serve you better.

Please bear with us during the transition phase. Until then we have our normal email and phone support available. Please click on the tech support tab above to find this info.

As always, thanks for visiting the digital.forest support site. We appreciate your business!