Global Collateral Damage of TMnet leak

June 12, 2015 Doug Madory

The Washington Post recently published a great piece about the development and current weaknesses of the Border Gateway Protocol (BGP, which is used to route all Internet traffic). This morning Telekom Malaysia (a.k.a TMnet) helped to illustrate the points made in the article by leaking almost half of the global routing table via Level3 at 08:44 UTC.

Some of the most affected companies were those peering with Telekom Malaysia.  The following graphics illustrate the impact to routes from Amazon and Cloudflare.

Google’s extensive peering likely insulated it from some of the affects of having some of its routes leaked.  However, it didn’t escape the incident completely unscathed.  Here is an example of a normal traceroute to Google’s data center in Council Bluffs, Iowa from Prague, which goes via Frankfurt and London before crossing the Atlantic Ocean.

trace from Prague to Google, Council Bluffs, IA at 02:45 Jun 11, 2015
1  *
2     16.583
3 22.934
4      Level 3 (Frankfurt, DE)             23.101
5   Google (Frankfurt, DE)              23.796
6   Google (Frankfurt, DE)              24.086
7     Google (London, GB)                 32.709
8   Google (New York City)             103.091
9   Google (Council Bluffs)            133.098
10     Google (Council Bluffs)            133.245
11   Google (Council Bluffs)            133.536
12 *
13   Google (Council Bluffs)            132.643

During the routing leak, traces were redirected to Hong Kong (where Telekom Malaysia gets Level 3 transit) and across the Pacific Ocean for a performance hit of almost 400ms.

trace from Prague to Google, Council Bluffs, IA at 09:04 Jun 12, 2015
1  *
2                     41.213
3  *
4 451.264
5  *
6   Google (Mountain View)                             509.481
7    Google (Mountain View)                             482.303
8   Google (Mountain View)                             459.441
9   Google (Council Bluffs)                            457.846
10     Google (Council Bluffs)                            468.626
11   Google (Council Bluffs)                            456.841
12 *
13   Google (Council Bluffs)                            509.298


One of the most severely impacted entity out there was Free of France.  Free (AS12322) provides service just in France but peers with Telekom Malaysia.  They also massively pre-pend their AS paths for routes transiting Cogent as follows:

… 174 12322 12322 12322 12322

The repeated ‘12322’ in the AS path is a commonly used technique to de-prioritize a particular route — in this case: transit from Cogent. This is because in BGP, all other things being equal, the shortest AS path wins.  The risk of prepending is that by artificially lengthening an AS path, the routes from any leaker will often be shorter and more likely to be chosen, as illustrated in the following path.

… 3549 4788 12322

This example highlights an issue I’ve raised in the past year about the risks of promiscuous peering.

Of course, a routing change like this is bound to have impact on performance.  Note the higher latencies in tan in the graphic below as traffic bound to Free in France is routed through Telekom Malaysia:

In addition to leaking globally routed prefixes, Telekom Malaysia’s leak introduced thousands of new routes from its private peering sessions.  The graphics below show the propagation footprint of two routes (Facebook in Malaysia and Layer 2 Broadband of Australia) over a period of four hours.

This isn’t the first routing leak involving the former Global Crossing network (AS3549) in recent months.  In April, a small provider in Brazil (Tribunal Regional Federal da 1a Regiao, AS61569) that normally announces a single prefix to AS3549 leaked over 12,000 routes from another Brazilian ISP (Global Village Telecom, AS18881), including the routes of several CDNs that peer with AS18881.  As in the case of this morning’s incident, AS3549 clearly hasn’t bothered with a MAXPREF setting as it couldn’t recognize that it shouldn’t be propagating 12,000 routes from AS61569 — or 260,000 prefixes from AS4788.

Small Brazilian provider (AS61569) leaks 12k routes to @Level3 at 13:22UTC including @Google. Similar to

— Dyn Research (@DynResearch) April 7, 2015

Unlike the Indosat leak from last year in which Indosat’s announced routes asserting itself as the origin, this leak preserved the origins of the routes, but placed itself in the AS path.  While the big routing leaks get a lot of attention, be aware that these types of incidents occur on a smaller scale nearly everyday.  Just yesterday Filanco of Russia leaked around 3,000 routes to MTS, briefly impacting networks around the world.  Below is the visualization of the upstreams of an Akamai route over a one hour period — Filanco shouldn’t be here.

With a tool like Dyn’s Internet Intelligence, companies can monitor and help mitigate the impact of routing snafus on their community of users.

And a Happy Friday to you, too!

Goooood morning Malaysia! — Telekom Malaysia (@TMCorp) June 12, 2015


// ]]>

The post Global Collateral Damage of TMnet leak appeared first on Dyn Research.


About the Author

Doug Madory is a Director of Internet Analysis at Dyn where he works on Internet infrastructure analysis projects. Doug has a special interest in mapping the logical Internet to the physical lines that connect it together, with a special interest on submarine cables.

Follow on Twitter More Content by Doug Madory
Previous Article
Iran:  Latest Nation to Host Critical Global Internet Infrastruture
Iran: Latest Nation to Host Critical Global Internet Infrastruture

As crippling economic sanctions are poised to be lifted by the United States, Iran is starting to emerge fr...

Next Article
Earthquake rocks Internet in Nepal
Earthquake rocks Internet in Nepal

Earthquake in Nepal took a toll on the country’s Internet connectivity, which was already one of the least ...