November 8th, 2011,

** This is an Informational Notice ** Level 3 transit customers refer to L3 event 5197215

It appears that Level3 ( a major transit provider ) has suffered a global failure. (Appears to be related to a rogue route(s) injected into the BGP table or a juniper bug, according to uncomfirmed reports.. Update: many other ISPs are seeing Juniper issues so the latter appears to be the case. We don't use Juniper and are routing around upstreams that are impacted.)

Customers transiting Level 3 (we use level 3 in many locations in the US and Europe) may have noticed this. We have routed around Level 3 for the time being and will provide more information as available.

Note: While real information is still very sparse, reports from other networks are being posted to twitter under #level3 - https://twitter.com/#!/search/level3

Update: It appears nLayer has been hit with outages related to a juniper bug as well. Most notably impacting our newly soft launched Dallas location which is still waiting on additional peers.

Update: Although Level3 appears to be stable at this point for us after ~40 minutes, there are still many ongoing connectivity issues with individual providers, and widespread reports from our customers of impact. These issues appear to be related to the mentioned Juniper bug, or issues triggered in peers based on the initial L3 issue.

Update: Official Level3 statement expected within a few hours. We are no longer seeing direct impact from this issue.

Update: Level3 reported statement on the outage

"Level3 has issued a statement on the outage. "Shortly after 9 am ET today, Level 3's network experienced several outages across North America and Europe relating to some of the routers on our network," the company said. "Our technicians worked quickly to bring systems back online. At this time, all connection issues have been resolved, and we are working hard with our equipment vendors to determine the exact cause of the outage and ensure all systems are stable."

Update: Level3 Tweets "UPDATE: As many of you know, we experienced a pretty big outage around 9am ET. Apparently due to a router manufacturer software issue…"

Update: Juniper reported statement.

"This morning, Juniper learned of a Border Gateway Protocol edge router issue that affected a small percentage of customers," said Mark Bauhaus, Juniper executive vice president of services, support and operations. "A software fix is available, and we’ve been working with our customers to immediately deploy the fix."

Juniper did not respond to further questions but it’s believed all affected customers are back up. The BGP bug caused the routers dump main memory, reboot and rebuild their BGP routing tables, sources said.

Much <3 to Juniper and JunOS. When the small % of customers impacted route a large % of the global internet traffic, it makes for an interesting Monday morning!

This entry was posted in Uncategorized. Bookmark the permalink.

Comments are closed.