We experienced a BGP-related routing issue tonight that reduced our visibility to the outside world for about 5-15 minutes, depending upon where on the Internet you accessed us. It started about 8:15 PM PST, and was corrected just before 8:30 PM PST.

We are investigating and will post more details when they become available.

Update: The issue was the result of a small configuration error in our Seattle router. The slow DNS-like nature of BGP propagation is what caused it to not have immediate impact. Our FastEthernet circuit with NTT/Verio caused us some problems while we worked to migrate it to Seattle overnight, and our Network Management staff spent most of the day today dealing with wrapping up that process. This was completed at exactly 6:30 PM PST this evening. However, at some point during that process an error was introduced into our BGP configuration that caused this issue to arise later in the evening. Thankfully all of our senior network management were still here on site and could address the issue swiftly.

The good news is, that after almost three months of constant configuration changes to all of our routers, we have arrived at a point in our move where no significant network changes will be made for the foreseeable future. All servers, subnets, and connections to the Internet are now fully migrated to the Seattle facility and are here to stay. As I said in yesterday's move update, plenty of work remains to complete the new datacenter, and decommission the Bothell facility, but the server migrations are complete.