Chinese Routing Errors Redirect Russian Traffic

November 6, 2014 Doug Madory
traceroute-v4

In recent weeks, Russian President Vladimir Putin announced a plan to enact measures to protect the Internet of Russia. In a speech to the Russian National Security Council he said, “we need to greatly improve the security of domestic communications networks and information resources.” Perhaps he should add Internet routing security to his list because, on a number of occasions in the past year, Russian Internet traffic (including domestic traffic) was re-routed out of the country due to routing errors by China Telecom. When international partners carry a country’s domestic traffic out of the country, only to ultimately return it, there are inevitable  security and performance implications.

Last year, Russian mobile provider Vimpelcom and China Telecom signed a network sharing agreement and established a BGP peering relationship. However, as can often happen with these relationships, one party can leak the routes received from the other and effectively insert itself into the path of the other party’s Internet communications. This happened over a dozen times in the past year between these two providers. This is a general phenomenon that occurs with some regularity but isn’t often discussed in BGP security literature. In this blog post, we’ll explore the issue via the lens of this single example. In a follow-on blog, we’ll explore several additional examples.

traceroute-v4

How peering links can go wrong

In the world of BGP parlance, the term peering is overloaded. The same term can refer a simple BGP adjacency between to autonomous systems (AS) as well as a particular type of relationship between two ASes where traffic is exchanged without money changing hands. This is also referred to as a “settlement-free” relationship and is the definition of peering used this blog post.

Peering relationships between ISPs are quite common. By exchanging traffic directly without using a transit provider, each ISP saves on transit costs — although as transit prices fall the overhead of maintaining these peering relationships becomes less attractive. Regardless, from a routing standpoint, routes exchanged between two peering ISPs typically stay within the customer cone of each ISP. In the diagram below, ISP A sends routes from its customers to its peer ISP B. As a result, traffic from ISP B’s customers destined for those routes flows over the peering link to ISP A’s customers.

normal_ops

This arrangement can go wrong in a couple of different ways. ISP B could take the routes learned from ISP A and send them on to its providers and/or peers as illustrated in the graphic below. By doing this, ISP B effectively inserts itself into the path of traffic destined for ISP A from the outside world. I occasionally field questions from customers of Dyn IP Transit Intelligence product (formerly called Renesys Market Intelligence) who want to know why we reclassified one of their peers as a transit provider.

Typically after a little investigation, we determine that the peer is mistakenly passing their company’s routes on to the peer’s transit providers (or its peers) — effectively providing the company with free transit. In fact, one of the popular use cases for IPTI is for ISP operators to monitor how their network’s adjacencies get classified. When there is a difference between what IPTI shows and how the operator thought his network was connected, someone isn’t propagating routes correctly.

leak_scenario1

The other way a peering relationship can go wrong is if ISP B announces routes learned from its providers and/or peers to ISP A. In this scenario (illustrated below), ISP A will then redirect traffic destined for these routes through ISP B, even if these routes are not normally handled by ISP B.

leak_scenario2

Unlike other routing leak scenarios, such as Indosat originating the entire global routing table or VolumeDrive leaking nearly the entire BGP table from one transit provider to another, the leaks described above occur with much greater frequency and with little fanfare. In fact, typically the parties involved are unaware of the glitch and as a result these more limited problems can persist much longer than the larger catastrophic incidents. Nonetheless, the results typically include traffic going where it wasn’t intended with a corresponding immediate hit to performance and the potential for security implications.

The Vimpelcom – China Telecom connection

200px-VimpelCom_logo     505px-China_Telecom_Logo.svg

In September 2013, China Telecom and Vimpelcom (one of Russia’s “big three” mobile operators) announced an agreement to establish a direct network interconnection between their networks. We would have expected to see this relationship to appear in BGP routing, either as a peering relationship or possibly a limited transit relationship to allow Vimpelcom to reach the far east. However, during several occasions in the past year, we observed China Telecom mishandling routes it received from (and sent to) Vimpelcom, it effectively inserting itself into the path of Internet traffic to and from Vimpelcom.

One of those occasions was the 5th of August of this year, when China Telecom (AS4134) announced routes from Vimpelcom (AS3216) to all of its peers for several hours (leak scenario 1 from above) which forced inbound traffic to Vimpelcom to go through China Telecom routers. During this incident, over 7000 routes from Vimpelcom’s customer cone were globally announced by China Telecom in the following form:

   … {1299,3257,1273,3320,6461,174,…} 4134 3216 …

The following charts illustrate the impact of these incidents by looking at how traceroute measurements into Vimpelcom were effected. As the charts show, latencies spike as traffic is dragged out to China and through the molasses of China Telecom’s under-provisioned international links.

NA_3216_vps01.xrs1 NA_3216_vps01.hnl1
NA_3216_vps01.sjc1 NA_3216_vps01.gdl1

The following example traceroute illustrates an errant path traffic was taking during this routing leak. In this example, the traceroute begins in Reston, Virginia where NTT takes it to California and hands it off to China Telecom. China Telecom then takes it to Shanghai, China and then to its routers in Frankfurt, Germany before entering Russia en-route to Omsk.

trace from Reston, VA to Omsk, Russia at 12:00 Aug 05, 2014
1  *
2  129.250.204.153 (NTT America, Ashburn, US)                1.010ms
3  129.250.4.40    (NTT America, Ashburn, US)                0.934ms
4  *
5  129.250.5.13    (NTT America, San Jose, US)              74.649ms
6  129.250.8.6     xe-0.chinanet.snjsca04.us.bb.gin.ntt.net 70.909ms
7  202.97.90.33    (China Telecom, San Jose, US)            71.451ms
8  202.97.58.237   (China Telecom, Shanghai, CN)           341.819ms
9  202.97.58.66    (China Telecom, Frankfurt, DE)          641.424ms
10 118.85.204.58   (China Telecom, Frankfurt, DE)          608.965ms
11 79.104.245.250  pe-r.Omsk.gldn.net                      632.054ms
12 195.239.162.178 (Vimpelcom, Omsk, RU)                   687.536ms
13 89.179.76.25    vpn2-gi0-0.omsk.corbina.net             682.153ms
14 128.73.155.213  128-73-155-213.broadband.corbina.ru     707.525ms

As seen from within China, latencies to Vimpelcom via the direct link from China Telecom got lower and more stable, but did not improve dramatically. In fact, these traces were still getting directed to China Telecom’s routers in Frankfurt, Germany, and given the fact that overland round-trip latencies from Russia to China are around 120ms, the observed latencies suggest that traffic may still go across submarine cables to reach Europe.

CN_3216_vps01.pek1 CN_3216_vps01.pvg1

If this routing arrangement is intended to provide Vimpelcom low-latency access to the far-east, it isn’t working that well. The examples below show latencies increase during routing leak from locations in the far east.

3216_Aug5_vps01.sin3 3216_Aug5_bgp02.tyo1

The August 5th event was one of the times that China Telecom briefly announced nearly a full BGP table (326,622 routes) to Vimpelcom placing itself in the path of outbound traffic from Vimpelcom to the outside world — including Russian routes. (Leak scenario 2 from above.)

Below is a traceroute illustrating the new path to a local ISP here in New Hampshire. Vimpelcom takes the traffic to Frankfurt, hands it over to China Telecom which takes it to Shanghai before handing it over to Cogent in Los Angeles. Cogent takes it across the country to Fairpoint Communications in New Hampshire.

trace from Moscow to Manchester, NH at 12:09 Aug 05, 2014
1  *
2  194.154.89.125 (Vimpelcom, Moscow, RU)                     0.743ms
3  79.104.235.66  mx01.Frankfurt.gldn.net                    40.574ms
4  118.85.204.53  beeline-gw3.china-telecom.net              43.198ms
5  202.97.58.57   (China Telecom, Shanghai, CN)             302.433ms
6  202.97.58.238  (China Telecom, Los Angeles, US)          479.642ms
7  202.97.49.14   (China Telecom, Los Angeles, US)          487.225ms
8  38.104.139.77  te0-7-0-24.ccr21.sjc03.atlas.cogentco.com 380.087ms
9  154.54.6.105   be2000.ccr21.sjc01.atlas.cogentco.com     375.079ms
10 154.54.28.33   be2164.ccr21.sfo01.atlas.cogentco.com     371.727ms
11 154.54.30.54   be2132.ccr21.mci01.atlas.cogentco.com     372.585ms
12 154.54.6.86    be2156.ccr41.ord01.atlas.cogentco.com     370.596ms
13 154.54.44.86   be2351.ccr21.cle04.atlas.cogentco.com     367.498ms
14 154.54.25.89   be2009.ccr21.alb02.atlas.cogentco.com     371.972ms
15 38.104.52.78   (Cogent, Albany, US)                      367.334ms
16 70.109.168.139 burl-lnk.ngn.east.myfairpoint.net         321.980ms
17 64.222.166.166 (Fairpoint Communications, Concord, US)   315.036ms
18 64.223.189.66  static.man.east.myfairpoint.net           321.682ms

But the path from Moscow to New Hampshire was always going to be long. Who’s to say that Los Angeles (or even Shanghai?) might not be a reasonable way-point under certain conditions. Alternatively, we can pick something closer to Moscow to see illustrate how crazy this routing is. The traceroute below shows Vimpelcom taking traffic to Frankfurt, handing it over to China Telecom which takes it to Shanghai before handing it over to Chello Broadband, which peers with China Telecom in Los Angeles. Chello then takes it from New York to Frankfurt (again!) and then into the German countryside.

trace from Moscow, Russia to Marburg an der Lahn, Germany at 14:29 Aug 05, 2014
1  *
2  194.154.89.125   (Vimpelcom, Moscow, RU)                  0.908ms
3  79.104.235.74    mx01.Frankfurt.gldn.net                 40.695ms
4  118.85.204.49    beeline-gw2.china-telecom.net           48.799ms
5  202.97.58.61     (China Telecom, Shanghai, CN)          290.810ms
6  202.97.58.202    (China Telecom, Los Angeles, US)       514.115ms
7  202.97.90.62     (China Telecom, Los Angeles, US)       537.933ms
8  213.46.190.217   213-46-190-217.aorta.net (New York)    414.365ms
9  84.116.135.122   (UPC Austria GmbH, Vienna, AT)         420.591ms
10 213.46.160.205   uk-lon02a-rd1-pos-4-0-0.aorta.net      421.530ms
11 84.116.133.65    (UPC Austria GmbH, Frankfurt, DE)      421.557ms
12 81.210.129.234   7111a-mx960-01-ae1.fra.unity-media.net 420.867ms
13 80.69.107.214    7111a-mx960-02-ae0.fra.unity-media.net 418.427ms
14 80.69.107.185    7211a-mx960-01-ae1.gie.unity-media.net 420.454ms
15 88.152.128.0     (Unitymedia, Marburg an der Lahn, DE)  423.105ms

Even Internet paths from Moscow to other parts of Russia were forced through China Telecom’s routers. In the following example, a traceroute from Moscow is taken by Vimpelcom to Frankfurt, handed over to China Telecom’s routers in Frankfurt and, (mercifully) redirected back into Russia via Megafon without getting directed out to Shanghai.

trace from Moscow, Russia to Yaroslavl, Russia at 13:13 Aug 05, 2014
1  *
2  194.154.89.125  (Vimpelcom, Moscow, RU)             0.542ms
3  79.104.235.74   mx01.Frankfurt.gldn.net            37.006ms
4  118.85.204.57   beeline-gw4.china-telecom.net      39.505ms
5  213.248.84.185  ffm-b10-link.telia.net             41.481ms
6  62.115.137.180  ffm-bb2-link.telia.net             42.227ms
7  80.91.251.159   s-bb4-link.telia.net               42.894ms
8  213.155.133.105 s-b2-link.telia.net                41.528ms
9  80.239.128.234  megafon-ic-151430-s-b2.c.telia.net 42.707ms
10 *
11 78.25.73.42     (MegaFon, Volga,RU)                49.992ms
12 213.187.127.98  ysu1-ccr1036-1.yar.ru              50.301ms
13 213.187.117.230 (NETIS Telecom, Yaroslavl’, RU)    54.769ms

Conclusion

In September, ACM Magazine published a terrific piece by Boston University Professor Sharon Goldberg which asked “Why is it taking so long to secure Internet routing?”. She does a great job covering the issues surrounding routing security and the efforts thus far to address its vulnerabilities. I would add peering leaks like those described in this blog to her list of unresolved issues, problems that are much less well known or even discussed. At present time, the best defense is to monitor how your routes traverse the Internet (such as using Dyn Internet Intelligence) and to carefully filter the routes you receive.  Without both measures, your traffic could be easily misdirected, potentially hurting both the performance and security of your Internet communications.

Investors also weren’t too excited about Putin’s Internet security plan, causing Vimpelcom’s stock to drop to a record low, but maybe they were disappointed that it didn’t address routing security.

The post Chinese Routing Errors Redirect Russian Traffic appeared first on Dyn Research.

Read more...

About the Author

Doug Madory is a Director of Internet Analysis at Dyn where he works on Internet infrastructure analysis projects. Doug has a special interest in mapping the logical Internet to the physical lines that connect it together, with a special interest on submarine cables.

Follow on Twitter More Content by Doug Madory
Previous Article
Use Protection if Peering Promiscuously
Use Protection if Peering Promiscuously

Last week, I wrote a blog post discussing the dangers of BGP routing leaks...

Next Article
Renesys Team Launches Dyn Research
Renesys Team Launches Dyn Research

Welcome to the new Dyn Research Blog!   We’re certainly glad you’re here,...