In observance of the Memorial Day holiday, digital.forest's offices will be closed Monday, May 28th. Business will resume on Tuesday morning at 8:00 AM PDT, May 29th.

Technical Operations & Support will remain open through the entire weekend, as always, and through the holiday as well. Datacenter access remains available, though the building will be locked and clients wishing to access the datacenter will need to call the support phone line (206-838-1630, option3) to gain entry into the building.

We wish our clients and their families an enjoyable holiday weekend.

--Chuck Goolsbee
VP
digital.forest, Inc.

silverpine.forest.net will be taken offline between 8 and 8:30 PM tonight for essential system maintenance. We expect the downtime to be less than 30 minutes.

During our scheduled maintenance window starting at 11:00 PM on Thursday, May 17th we will be resetting our connection with one of our network peers, NTT/America. As part of the post-mortem analysis of the last event on this circuit we discovered a way to improve the failover properties of the ethernet configuration. It will require up to 15 minutes of downtime while we reconfigure the router interfaces.

This should have no impact on uptime or network connectivity as we have other connections with other peers which will take the traffic during the maintenance window.

Springtime brings sunny weather and cleanup tasks! Our backup power generator comes equipped with a 5000 gallon, double walled fuel tank. We can run for almost a week on our own power source with this capacity. While fairly mundane in appearance (beyond of course being HUGE) it actually is a fairly high-tech bit of equipment. It has sensors that provide us feedback concerning fuel quality, quantity, and rate of consumption while the generator is running. It also provides data on water percentage in the fuel. Diesel fuel is basically oil, and oil and water as we all learned in grade school science class, do not mix. Large metal containers condense water from the atmosphere, both inside and out due to normal temperature and humidity changes. Diesel engines all have water separation equipment built into their fuel systems. Rudolf Diesel's invention is a compression engine meaning that unlike a gasoline engine it does not use spark plugs to ignite a fuel/air mixture; instead it compresses the mixture at such a high ratio that it heats up to an ignition point on its own. Unlike a fuel/air mixture, water does not lend itself to compression, so it must be removed from the fuel prior to injection into the compression cylinders.

The fuel monitoring system for our storage tank had an alarm go off last week. It signaled a interstitial leak. This indicates that there could be a possible leak of the main tank into the interior space between it and the outer tank.

Kevin Teker, our Facilities Manager responded by going into the interstitial space with testing equipment to see what was setting off the fluid sensor.

Above: The fluid sensor removed from the fuel tank.

Above: The high-tech method of getting the chemical testing paste to the lower reaches of the tank... a tape ruler.

Using special pastes which change color based on contact with either water or fuel, Kevin was able to determine that the "leak" was not in fact a leak, but instead water that had condensed within the interstitial space between the two tanks. Once satisfied that our primary tank's integrity was sound the next step was to remove the water. How does one drain less than a pint of water from a tank that weighs 50,000 pounds and only has openings at the top?

We solved that problem by lowering absorbent materials attached to that high-tech reach tool pictured above. Once the water was removed, we replaced the sensor and it indicated a dry interstitial space once again.

This is just one small example of the little things we do every day to maintain our facility. Every colocation provider has a facility to maintain, and we know that you choose digital.forest for our excellent service AND our facility. I hope that you see and recognize both every day, in the form of your servers' uptime, and when you have interaction with the Tech Support and Customer Service staff. The facility though is in many ways invisible to most of our clients, so it is nice to highlight things when they present themselves and allow you a glimpse into daily operations.

Regards,
Chuck Goolsbee
VP, Technical Operations
digital.forest

At 9:50 AM this morning one of our Metropolitan Ethernet providers, OnFiber had an equipment failure here in Seattle. We connect to one of our network peers, NTT/America at The Westin Building via this circuit. This caused us to have have intermittent connectivity over that particular circuit to NTT/America. Some digital.forest clients may have had "slow" or "intermittent" issues reaching servers here for a short period of time while we diagnosed the issue with the NOC's of NTT & OnFiber

We have shut down our BGP connection to NTT/America while OnFiber fixes the problems on their network. At the moment we are running on two of our three network connections. We will update this post when we bring the third circuit back online.

Update: As of 11:02 AM PDT this issue is completely resolved. The OnFiber circuit was manually moved to a different port. After a successful 10-minute testing of the new circuit we turned up our BGP session with NTT/America.

We maintained connectivity to our other BGP network peers through this event, so at no time was our network "down". We do like to keep our clients informed of events here at our datacenter, even if they have no direct impact on your servers. In this case, it was a classic example of Internet Architecture and how it handles outages. The often-used phrase is that it "routes around damage." In this instance when one of our circuits had an issue our traffic just shifted to our other circuits. It is likely that none of our clients even noticed. If they did notice it would have been an intermittent connectivity for a brief period of time. Such is the nature and reason for designing redundant systems. Our fiber optic connectivity to the rest of the Internet flows over multiple physical paths. Those paths do not converge until they are physically inside our datacenter facility. This prevents complete outages through equipment failure or accidental fiber cut. Today's event confirms the built-in redundancies work as designed.

--Chuck Goolsbee
VP Technical Operations
digital.forest, Inc.

Tonight at 8:00 pm we will be performing a software update on our helpdesk system. This will involve approximately 5 minutes of downtime. At that time our online trouble ticket system will be unavailable.

All other support options (telephone, emergency pager, etc) will remain online throughout the maintenance window. The trouble ticket system should be back online by 8:05 PM.

We apologize for any inconvenience this may cause.