Tonight during our scheduled maintenance window we will be making a minor configuration change to the distribution switch connections on our primary core switch. This will cause a brief outage to each of our distribution switches. The outage should be less than a second and the connections will be changed one at a time.

The maintenance will occur between 9:00 PM PDT and 10:00 PM PDT.

We will be taking thyme.forest.net offline for about 5-15 minuets starting at 11:45 AM PST today in order to address a performance degradation in the system.

This downtime will occur on Friday August 29th, 2008, starting at 11:45 AM PST and ending at about 12:00 PM PST.

Update: August 29th, 2008 11:55 PST: The server is online again. The server's performance has been greatly increased and the performance degradation that was effecting the server has been addressed.

It has been a while since we posted progress on our facility expansion, so here is an overdue update. Datacenter Three is almost finished, with fully functioning power and cooling. Pictured in the image above is a view of the cold air distribution system above a row of racks. Some clients have already moved in, despite a bit of remaining work; specifically fiber trays, cage walls, and rack row lighting. Those should be wrapped up very soon.

The Seattle area had a brief labor stoppage in late July when a Pipefitters Union went on strike. This delayed just about all related construction work in the region, including our installation of an Ecaro 25 clean agent system. Thankfully the labor issues were sorted out and our project was only delayed about two weeks. The tank arrived recently and is undergoing pipe fitting today. Our fire protection system vendor will be here next week integrating Datacenter Three into our overall fire suppression system.

Stay tuned for more information as we approach the completion of this project.

We are currently experiencing some issues with our Filemaker server savin. We are running maintenance and troubleshooting the server and hope to have it stable again as soon as possible.

We are notified immediately when the server is inaccessible and can correct the problem very shortly afterward, but if you notice that your database on savin is unavailable, please call us at 877-720-0483 and use option 3 to reach support.

During our scheduled maintenance window on Thursday night, August 28th, we will be making some changes to our BGP configuration in an effort to better balance our traffic among our upstream connections. This should have minimal to no impact on network uptime, but there is the possibility that persistent connections (such as VPN tunnels) will reset as routes change.

The maintenance window for this action will start at 23:00 PDT on Thursday, August 28th, and conclude by 01:00 PDT Friday, August 29th.

UPDATE: 3:34 PM PDT The failed switch card has been replaced. The circuit was back up at approximately 1:30pm. After waiting 2 hours to ensure that the circuit is stable we have now turned the link back up and are running on all of our upstreams.

UPDATE: 11:35 AM PDT ETA for repair tech is 12:30pm

At 10:17 am PDT we began to see an issue with one of our upstream providers. The issue is being caused by a problem with our circuit between Digital Forest and the Westin Building. Our upstream is currently investigating, however, they have not been able to give us an ETA for resolution.

In order to minimize the effect of this on our customers we have shut down the link to the upstream affected at 10:39 am PDT. All traffic is currently being routed through our other upstreams.

During our scheduled maintenance window on Thursday night, August 21st, we will be making some changes to our BGP configuration in an effort to better balance our traffic among our upstream connections. This should have minimal to no impact on network uptime, but there is the possibility that persistent connections (such as VPN tunnels) will reset as routes change.

The maintenance window for this action will start at 23:00 PDT on Thursday, August 21st, and conclude by 01:00 PDT Friday, August 22nd.

This past weekend digital.forest experienced a network outage two hours and fifty minutes in duration, from 05:29 until 08:19 AM Saturday morning. Preliminary investigation revealed that the cause of the event was a partially-failed supervisor card in one of our two core network devices. This partial failure first created a loop, then a network storm. The loop occurred in the meshed wiring between devices used to facilitate redundancy. In this case the failure was not complete, so the redundant network path became a loop. The loop caused the network storm, as devices started responding to traffic coming back to themselves, from themselves. Once the source of the loop was discovered and removed from the network things returned to normal in about 20 minutes.

The partially-failed device remains off-line and will be replaced very soon. We are currently performing some forensics upon the failed device to ascertain exactly what lead to its failure and what can be done to prevent a reoccurrence.

Digital.forest has experienced a network level event that has caused portions of our network to be unavailable. Currently our Systems Administration and Network Administration as well as Network Engineering staff are on site working to resolve this issue.

If you are currently experiencing any problems connecting to any services at digital.forest this is the cause.

We will keep this page updated as we get more information on this event and a detailed post will be made when this event has been resolved.

Update 9:30 AM PDT
Connectivity to the majority of our shared hosting services has been restored.

Update 10:55 AM PDT
As of 09:00 PDT most of our network was back online and accepting incoming and outgoing traffic normally. We have confirmed that the connection is stable and we do not expect any further interruptions to service here at digital.forest.

Expect a full writeup on the situation soon.

We have restored the Instant Web Publishing Functionality on Rosemary and would like to thank everyone for their understanding and patience.

The server "rosemary.forest.net" is experiencing some issues at the moment. Specifically FileMaker Pro "Instant Web Publishing" has ceased to function properly. We are making efforts to have it working again as soon as possible. Rosemary may be offline for periods as we address these issues. Thanks for your patience.