Yesterday, Indosat, one of Indonesia’s largest telecommunications providers, leaked large portions of the global routing table multiple times over a two-hour period. This means that, in effect, Indosat claimed that it “owned” many of the world’s networks. Once someone makes such an assertion, typically via an honest mistake in their routing policy, the only question remaining is how much of the world ends up believing them and hence, what will be the scale of the damage they inflict? Events of this nature, while relatively rare, are certainly not unheard of and can have geopolitical implications, such as when China was involved in a similar incident in 2010.
Keep in mind that this is how the Internet is designed to work, namely, on the honor system. Like Twitter and Facebook, where you can claim to be anyone you want, Internet routing allows you to lay claim to any network you want. There is no authentication or validation. None. But unlike Twitter and Facebook, such false claims propagate through the world in a matter of seconds and decisions, good or bad, are made algorithmically by routers, not humans. This means that innocent errors can have immediate global impacts. In this incident, the impacts were most pronounced on Akamai, one of the world’s largest content delivery networks, which was a very bad thing. Akamai hosts thousands of networks for their customers, including turbotax.com, healthcare.gov, paypal.com and many other high-profile sites.
The trouble with Indosat began at 18:25 UTC yesterday when they leaked over 320,000 routes. Since a full routing table currently contains nearly 500,000 routes, this means that Indosat laid claim to roughly two-thirds of the Internet!
Indosat routing leak involving >320k non-Indonesian BGP routes began at 18:25 UTC (earlier than we previously reported)
— Renesys Corporation (@renesys) April 2, 2014
While many of these routes didn’t travel very far from Indonesia and hence, would not have had much of an impact on Internet traffic, a few hundred were widely accepted, and a large fraction of these belonged to Akamai.
Only 354 prefixes leaked by Indosat (AS4761) this afternoon seen globally (>100 peers). 104 of them were @Akamai
— Renesys Corporation (@renesys) April 2, 2014
Besides disrupting Akamai themselves, this routing leak completely took out Indosat in what amounted to a self-inflicted DDoS attack. Our global latency measurements into this ISP via all of their upstream providers all but stopped during this time period and remained impaired even after the bogus routing announcements were withdrawn.
Surprisingly, for some Akamai prefixes (networks), the Indosat hijack was essentially complete, with most of the world choosing Indonesia as the best place to send this particular Akamai traffic.
For others who were impacted, the hijacking was partial, with some of the world selecting Indosat and others selecting the rightful owners. For example, Chevron in London saw about half our routing sources choosing Indonesia over the UK during much of this 2-hour disruption.
We can assess the probable operational impact on each affected network by examining this split between our peers who selected Indosat (and therefore would have supported traffic misdirection) and those who stuck with the real owner’s routes. Several hundred thousand networks were affected to some degree, but 99.7% of these were minimally affected, with less than 5% of our peerset convinced to take the alternative Indosat origin. We can divide the remaining 0.3% into three tiers:
- Low Impact (0.2% of affected networks): potential traffic redirection affected more than 5%, but not more than 25%, of our peers. Examples:
- PNC Bank NA, US (188.8.131.52/24)
- CreditWest Bank, Turkey (184.108.40.206/24)
- Nova Banka Banjaluka (220.127.116.11/21)
- Starcard Banka Kartlari Merkesi, Turkey (18.104.22.168/24)
- Medium Impact (0.06% of affected networks): potential traffic redirection affected more than 25%, but not more than 50%, of our peers. Examples:
- Halliburton, Norway (22.214.171.124/22)
- Fedex, US (126.96.36.199/24)
- Capstone Financial Advisors, US (188.8.131.52/24)
- Complex Financial Systems Ltd, Russia (184.108.40.206/22)
- Bank-Inform, Ukraine (220.127.116.11/20)
- New People’s Bank, US (18.104.22.168/24)
- Citrix Online, India (22.214.171.124/24, see below)
- High Impact (0.03% of affected networks): more than 50% of our peers routed traffic via Indosat instead of the true owner. Examples (in addition to Akamai-hosted blocks):
- Stan Telecom, Afghanistan (126.96.36.199/24)
- Chevron Corporation, UK (188.8.131.52/24)
- City of Santa Monica, California (184.108.40.206/24, see below)
In the absence of a single world government (for strict authentication) and much greater controls over Internet routing (for strict validation), there is currently no way to completely prevent these types of incidents. In the same way that anyone can set up a fake Facebook account with your name on it, so too can any router in the world claim to be the best way to reach your network. At the very least, enterprises need to be monitoring and managing their own Internet assets, as not all hijacks are necessarily innocent or short-lived.
Enterprises also need to carefully police their own routing policies and understand how the world reaches them. The reason why Chevron was impacted globally was largely of their own making: normally, they heavily prepend their BGP announcements through British Telecom, one of their providers. That is, the AS paths to 220.127.116.11/21 tend to look like … 2856 7862 7862 7862 7862 7862. By this mechanism, Chevron has artificially lengthened its AS path, thereby de-prioritizing the selection of this route. Unfortunately, this approach has also left them open to hijacking, since BGP route selection uses AS path length in its decision-making process. When Indosat starting leaking routes, the heavily prepended (and correct) Chevron routes were some of the first to be misdirected to Indonesia, as AS paths via Indosat were often shorter.
We saw this same behavior back in April of 2010 during China’s routing leak. Some of the worst impacted routes in that incident were from Charlottesville, Virginia. Not because China was targeting this college town of 43,000 in central Virginia, but because those routes were heavily prepended at all times, all but guaranteeing any errant routes from anywhere in the world would be preferred.
In short: route leak events like this one, which happen at least once a year, are a good reminder that BGP routing is fragile and error-prone. There are no easy fixes. That means that every enterprise on the Internet should be monitoring the advertisements of their networks, keep published ASPATHs compact and free from unnecessary prepending, and be prepared to temporarily advertise one or more more-specific routes, if possible, to win back control of inbound traffic. Don’t be part of the 0.03% who suffer serious impacts from large accidental route leaks.
About the Author
Earl leads a peerless team of data scientists who are committed to analyzing Dyn’s vast Internet Performance data resources and applying their expertise to continually improve upon Dyn’s products and services.More Content by Earl Zmijewski