I'm very pleased to announce that digital.forest is once again expanding our Seattle datacenter. We've been planning and budgeting for quite a while, but now the real work begins. What you see above is the shiny new UPS which arrived last week. It will be going into our UPS room very soon. It will more than double our capacity to deliver continuous, clean power to our customer's servers.

Additional projects slated for this expansion include:

More 3-phase PDUs
This will allow us to deliver more power to both our existing and our newly constructed datacenter spaces.

Two new 70 ton air handling units
These will provide cooling and air handling for our facility far into the future. We'll enjoy greater than N+1 cooling capacity. We'll also be expanding our cold air delivery ducting, as well as adding a hot aisle containment and return system. This will optimize our cooling abilities and allow even greater electrical densities on the datacenter floor.

Expansion into previously unfinished space
We'll be adding approximately 3500sq' of finished datacenter bringing our total to just under 10,000sq'. All of this space will include "cage" collocation, where clients can have private, card-keyed access to their equipment. This space will be built and maintained to the high standard that digital.forest clients have come to expect. With Ecaro25 Fire Suppression systems, static-shedding floor materials, custom HVAC ducting & lighting, custom network configurations, and 24-hour access, we are able to furnish a top-tier datacenter to house your critical IT environment.

It was only last November that we finished our last facility expansion, and that space is close to being almost fully committed! Keep an eye on our support blog for updates and photographs as we begin construction.

Regards,
Chuck Goolsbee
V.P. Technical Operations,
digital.forest, Inc.

We're pleased to announce that digital.forest has opened a new technical support forum. This new forum is designed to give digital.forest customers the ability to ask questions of each other related to the services that digital.forest provides such as Filemaker databases, Lasso, PHP, etc. These forums will have some limited interaction from the dgital.forest staff however it is mainly a forum in which users can exchange ideas, ask questions, and interact as a community. We know that we have some smart folks among our clients and we hope to have the forum be a great place to exchange knowledge, both between us and our clients, but more importantly from client to client.

You can find this new service at forums.forest.net. There is also a link in the "Tools" menu over there. -->

This will not replace our trouble ticket system, which will remain the primary method of one-to-one technical support for our clients. Nor will it replace this "support blog" as a way for us to announce events, maintenance, and news about digital.forest. Instead we hope the forum will be a great place to share ideas among many people at once. Have a look. Create a login. Start a discussion. Help somebody out, or get some help for yourself. Let us know what you think.

--Chuck Goolsbee
VP, Technical Operations
digital.forest, Inc.

Starting tonight at about 10:00 PM PST, Silverpine will be taken down for the next step in its configuration changes. suexec and suPHP will be enabled on the server, and file ownership and permissions will be adjusted for all files on the server to match the requirements of this change.

Several weeks ago we opened tickets for any users we could find who were likely to be affected by this change. If you did not receive such a ticket, then it is unlikely you will experience any problems from this change. If you did receive a ticket, and you have made the changes we described, you should also be in the clear. If you do experience problems with your site after this change, please let us know ASAP so we can investigate and correct the problem.

That is a tower of power you see above. It is the new benchmark for electrical density here at digital.forest. 528 Watts per square foot'. We know that isn't a world record, but it is a very respectable number for a colocation facility here in the Seattle area. Few can approach it, much less match it.

We can't share the name of the client (shhh... they are a startup in "stealth mode") so we've blurred the pictures a bit and removed any identifying marks. They are however VERY happy that they found digital.forest.

Prior to being here, they were colocated at a competitor. They were NOT allowed to put this many servers into their rack at this competitor. In fact, in order to have this many servers, they had to buy FOUR RACKS.

We think that's nuts. So did our new client. Now they are happily colocated at digital.forest and can do it all in one rack.

At digital.forest we are here to assist our clients to succeed, not restrict them from what they need. It is just one part of the digital.forest difference.

Today @ 10:40 am we experienced a processor switchover on both of our core switches.

Both of these switches have redundant processor engines where the standby processor checks the primary processor every 5 to 10 milliseconds. If the standby processor can not communicate with the primary it will take over causing it to become the active processor.

Today this happened on our secondary core switch and then approximately 30 seconds later on our primary core switch. It appears that this was a precautionary measure taken by the on board diagnostics on the switches as both devices are functioning normally at this time.

Since these switches also act as our border routers this had the effect of disconnecting the BGP sessions on both devices. There was less than a minute of downtime while the BGP sessions re-established themselves.

Kyle Murray
Network Manager
digital.forest

On Thursday July 19th we will be updating our Windows hosting servers. There will an approximately 5 minute outage per server as it is rebooted after updates are installed.

The reboots will be occurring between 1:00 pm and 9:00 pm.

Sharp eyed observers will note that the number changed on our home page today. Yes, today is our birthday. It was 13 years ago today that digital.forest sprouted and began to grow. The number of hosting and colocation companies that were around in July of 1994, and are still around is very small. It has been a crazy 13 years in this business that is for sure.

We know that we owe our success and longevity to YOU, our valued clients. There are several of you that have been with us since that very first year! To that we say "THANK YOU!"

There are also many of you that have only found us relatively recently, such as after our 2005 move to our Seattle facility at Intergate.West. To you we say "We hope to serve you long and well!"

If you call the Support, Sales, or Customer Service staff today, wish us a happy birthday and enjoy the cake & ice cream.

The Seattle area experienced a bit of a heat wave yesterday. We realize that the rest of the country has seen temperatures even higher, but it is rare for our cool marine & mountain location to see things get as hot as they did in the past few days. The peak came yesterday afternoon, around 5 pm, when we hit 98°F/37°C outside. I'm happy to say that our datacenter's peak temperature was 74°F/23°C.

Today we have high overcast, and expected highs in the low-80s. Datacenter temps should remain in the mid-high 60s.

We've been making continual improvements to our HVAC system. Adding and making modifications to existing duct work mostly. We've boosted our cooling capacity by about 20% in the past few weeks. As we've filled our facility we have been adapting the generic air diffusers with custom duct work. The aim of which is to direct cool air into specific areas, namely the front of the racks. Servers are generally designed to cool front to back. Visitors to our facility will note that we arrange rows of servers with the backs of servers facing into alternate rows. This creates "hot aisles" and "cold aisles", a standard industry practice in datacenters. Cold air from the HVAC system falls down from just above, is pulled into the servers, and hot exhaust air rises up to the top of the room where it is pulled out by the HVAC system's hot air return system. What we have been doing the past several months in preparation for summer is focussing the cold air. This involves custom duct work.

Above: A shipment of our custom ducting arrives.

Above: Custom ducting awaits installation in the datacenter.

Above: A custom duct installed in a client's cage. Note the room for their planned expansion. Note also the unmodified generic air diffuser to the left. It is from these that we mount the custom ducts.

For our general rackspace datacenter we have taken the generic diffusers and focused the new ducts down and into the cold aisles. In the cage space datacenter, we have extended custom ducts directly to the front of customer's servers. This work allows us to better control airflow and maintain the optimum air temps right where it matters. Later this year we'll be adding similar duct work, this time above the hot aisles, specifically to remove hot air and bring it back to the return of the HVAC systems.

Above: The business end of our auxiliary HVAC system.

We have also added a large custom diffuser with dampers in 14" conduit heads on our auxiliary HVAC system. Throughout most of the year, this unit does not even run. In warmer conditions we can open a subset of the dampers and provide supplementary cooling in a very general fashion to the entire facility. On very hot days like yesterday we open them all and provide the extra HVAC tonnage to keep the facility cool. In emergency situations such as last week we can attach temporary lightweight ducts to the 14" heads and run them through the ladder rack to the specific areas of the datacenter that need it.

Stay tuned for some more exciting news very soon about expanded capabilities and additional layers of redundancy!

Chuck Goolsbee
VP, Technical Operations
digital.forest

UPDATE: The problem port has been disabled

The GigE card replacement that occurred on July 9th has not resolved the problems with the switch. We are still experiencing intermittent outages. To prevent further downtime we will disable the port that is causing problems. There will be a very brief outage as the traffic reverts to the backup uplink.

We will continue to work with our vendor to find a resolution to this problem.

Tonight during our scheduled maintenance window we will be inserting a GigE card in one of our distribution switches. This is being done after the replacement of a fiber that was causing some intermittent connectivity problems.

There will be a very brief outage when the GigE card is re-enabled as it is connected to the primary core switch and traffic will be re-rerouted through this card.

The maintenance will occur between 11:00 pm and midnight tonight.

Last night around 19:30, one of the three primary compressors of one of our two HVAC systems failed. This was noted by NOC personnel who contacted our HVAC contractor who set the controls to bypass the failed unit. We did run on single-stage cooling from one unit for several hours, which lead to both datacenters reaching temperatures in the low-80s F/high-20s C. Once the unit was bypassed 3-stage cooling was achieved and temps dropped to their normal mid-60s F/high-teens C. With outside summertime temperatures in the high-80s F/low-30s C possible we require 4-stage cooling at peak hours.

We will be replacing the failed compressor tonight between 21:00 and 23:00. We have secured portable HVAC units to supplement our secondary HVAC while the primary unit is down for repairs. We will update this website with more information as the repair progresses.

Update: 8:00 AM The new compressor was installed last night. We've resumed normal operations.

Regards,
--Chuck Goolsbee
VP, Tech Ops
digital.forest

In observance of the Independence Day holiday, digital.forest's offices will be closed Wednesday, July 4th. Business will resume on Thursday morning at 8:00 AM PDT, July 5th.

Technical Operations & Support will remain open through the entire day, as always, and through the holiday as well. Datacenter access remains available, though the building will be locked and clients wishing to access the datacenter will need to call the support phone line (206-838-1630, option3) to gain entry into the building.

We wish our clients and their families an enjoyable holiday.

--Nik Keating
Technical Support Manager
digital.forest, Inc.

We are currently performing emergency maintenance on our Souari server (216.168.37.73). We currently have a team of technicians working on the problem to resolve it as quickly as possible and will update you as soon as we have more information. Unfortunately we do not currently have an ETA for when the server will be back up but we will let you know as soon as it is.

Please accept our sincere apologies for this service outage and please let us know if you have any questions or concerns.

Thank You

Update (10:50 PDT): Apache on the souari.forest.net is online and all websites on the server are accessible at this time. We are still working to resolve an issue with the sendmail functionality on the server and our technicians hope to have that service online soon. Unfortunately we do not have and ETA for the repairs to the sendmail service at this time.

Update (11:44 PDT): Currently FTP on the souari.forest.net server is not available for the same reason that the sendmail service is not functioning on the server. We currently have technicians working to resolve these issues and hope to have an ETA for you soon.

Update (12:32 PDT): To correct a problem with bad library files on Souari, there will be another brief downtime, beginning immediately. We will update again when the server is back up and running.

Update (13:52 PDT): While correcting the problem with the bad libraries on souari.forest.net we discovered an additional problem and have been working to resolve that problem. This has resulted in additional emergency maintenance down time. Please know that we are doing everything that we can to resolve these issues as quickly as possible and have assigned out entire technical department to work on this issue.

Update (17:31 PDT): The Emergency Maintenance on the souari.forest.net server has been concluded and the server has returned to normal operating status.

The shrubbery.forest.net server has been taken offline in order to perform some emergency maintenance. We have technicians working on the server and will have it back online as soon as possible.

Please accept our apologies for this service outage and please let us know if you have any questions or concerns.

Digital Forest Technical Support
877-720-0483 option #3

Update (10:55 PDT): We have concluded our emergency maintenance on Shrubbery.forest.net and it is now online and functioning normally.