This past weekend digital.forest experienced a network outage two hours and fifty minutes in duration, from 05:29 until 08:19 AM Saturday morning. Preliminary investigation revealed that the cause of the event was a partially-failed supervisor card in one of our two core network devices. This partial failure first created a loop, then a network storm. The loop occurred in the meshed wiring between devices used to facilitate redundancy. In this case the failure was not complete, so the redundant network path became a loop. The loop caused the network storm, as devices started responding to traffic coming back to themselves, from themselves. Once the source of the loop was discovered and removed from the network things returned to normal in about 20 minutes.

The partially-failed device remains off-line and will be replaced very soon. We are currently performing some forensics upon the failed device to ascertain exactly what lead to its failure and what can be done to prevent a reoccurrence.