******9/10/09 Network Event******
Between approximately 01:00 HRS PDT and 04:00 HRS PDT on 9/10/09 digtal.forest experienced an intermittent network event that had noticeable impact on specific segments of our network. The event was caused by what our network manager has described as a broadcast storm that was initiated by some client equipment. A broadcast storm can be understood as the result of a malfunction of some computer device(s) (in this case a configuration error or driver failure in some client equipment) sending out a large number of broadcast packets or a continuous stream of broadcast packets. Those packets are then forwarded by every port on every network device on the network effectively amplifying the broadcast by the number of ports. This causes a storm of traffic that continues to grow as long as the malfunctioning equipment is connected to the network. It is normal for all network devices to send broadcast packets and at any given time we will have some amount of broadcast traffic on the network. During this event there were devices sending 200 times the normal amount of broadcast packets.
The duration of the event is a result of the amount of traffic and how that traffic impacts isolating the originating device(s). There is no easy way locate a small number of devices or single device in a broadcast storm because every device on the network is forwarding the broadcast traffic. The particular equipment that impacted the network has been isolated and is currently under evaluation to determine (if possible) the cause of the broadcasts. None of this equipment will be placed back on the network until it we can be certain there is no further risk of malfunction.
During the event our local network continued to pass traffic to the Internet without disruption. At several times though, traffic was passed very slowly from very specific segments of the local network out to the Internet. The reason some parts of the network were impacted and others were not is a result of our having multiple routers at our network core connecting to multiple discrete upstream providers. During this event one of the routers received the bulk of the broadcast traffic and was bogged down processing it while the other equipment continued to operate normally. Traffic departing our network on the normally operating equipment was not impacted by the event.
We at digital.forest apologize for any inconvenience this event may have caused you and greatly appreciate your business and patience.
************
digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PDT at 877-720-0483 Option 2.
posted by Shawn Hammer at 10:33 AM on Friday, September 11, 2009
Categories: Network