Pinning Down Latency

October 7, 2011 Doug Madory
Packet latency is a big issue in Internet-based applications (i.e. the stuff in the cloud). In conducting analysis on Internet infrastructure over the years, we have seen many patterns of connectivity. One such pattern that can wreak havoc on latency is “hair-pinning”, a phenomenon where traffic takes an unnecessarily long physical path between two points on the Internet due to suboptimal routing. The increased distance results in increased latency, and the “lag” or “sluggishness” that users experience as a result can hinder latency-sensitive online applications whether they are financial trading applications or MS SharePoint. hairpins.jpg

Geographic Locality does not translate into Internet Locality

A couple of weeks ago (September 23rd to be exact), I noticed a small routing outage (150 globally routed prefixes) in Sakhalinskaya Oblast, the most Eastern part of Russia. We observe these types of outages every day — the Internet is a big messy place behind the scenes.

The obscurity of this location piqued my curiosity and I looked into how this area gets Internet connectivity — almost exclusively through Russian provider, TransTelecom (AS20485). One can see that this province connects to Japan via Hokkaido-Sakhalin Cable System (HSCS) built by NTT in 2008 to provide a shorter path from Japan to Europe over Russian soil. However, despite the proximity to Japan, a traceroute from NTT’s Tokyo Looking Glass to an IP address in Sakhalinskaya looks like this:

Query Results:
Router: Tokyo - JP
Command: traceroute

traceroute to (
1 ( 0.592 ms
2 ( 9.322 ms
3 ( 329.864 ms
4 ( 325.062 ms
5 ( 283.933 ms
6 ( 456.641 ms
7 ( 467.577 ms
8 ( 481.945 ms


Which roughly follows a path like this:


Circuitous routes like this are called “hair-pinning” in the business – this traceroute travels from Japan to London just so it can make its way back to a location just off the coast of Japan. On the Internet, geographic locality does not translate into Internet locality, and the distinction can result in unexpected latency, frustrating users of real-time applications.

Latency in the Middle East

Last week I was in the fascinating city of Muscat, Oman for MENOG 9 giving a talk about latencies we have measured into each Middle Eastern country. While regional traffic in the Middle East often suffers from hair-pinning via European Internet exchanges, latencies to and from Europe appear to be steadily improving.

In the past, when Renesys has analyzed Middle East connectivity it was with regard to complete outages such as the cable cuts in 2008. This year, connectivity into the Middle East has been much more stable (obvious exceptions notwithstanding) and the fact that regional providers can focus energy on reducing latency is a good sign for the region. For example, OmanTel’s strategy (AS8529) to reduce latency to Europe by gaining presence at AMSIX (AS1200) is paying dividends in 2011 by reducing median latency by 41ms or 20% in 2011. This is illustrated in the chart below, which is a plot of overall latencies from London to Oman in 2011 colored by OmanTel’s upstream provider or exchange as observed by traceroute. Higher latency bands (PCCW, AS3491, in Grey and LINX, AS5459, in blue) disappear midway through the year and are replaced by a lower latency path through AMSIX (green).


Until a regional Internet Exchange emerges for the Middle East, the best regional providers can hope for is to reduce latency on the path of the hair-pinning through Europe for regional traffic!

Impact of IMEWE Activation in Lebanon

The most recent (and intriguing) development in the area of latencies to the Middle East is the reported activation of the India-Middle East-Western Europe (IMEWE) cable system at Tripoli, Lebanon. Despite speculation that the activation would be delayed, there is evidence that the cable is now being actively used. Certainly something has happened in Lebanon to reduce latencies in recent days.

Thus far, development of the Internet market in Lebanon has been stifled by the country’s extensive dependence on satellite Internet service — the ultimate form of hair-pinning. In the visualization below, latencies into Lebanon separate into two modes based on delivery medium: satellite (primarily via SatGate, AS30710) and submarine (primarily via Level3, AS3356). Further examination of the two latency modes clearly indicates asymmetric routing. First, latency measurements should be at least 480ms for symmetric round-trip delay across a satellite connection due to the speed of light and altitude of orbit, but we observe latencies clustering around 400ms. Second, the dramatic 70ms drop in median submarine latencies starting September 22 is shadowed by a corresponding effect in the latencies inbound through the satellite provider. Packets are arriving inbound via satellite but returning via submarine cable. Note also the further 15ms reduction on September 29th, which has held through today. (Note: Dates referenced appear at small ticks along x-axis)


Additionally, we see a reconfiguration on August 24th visible as a slight change in this latency distribution. Here there was an address change in the IP-IP hop in Level3’s network (AS3356) that experiences the vast majority of the overall latency to Lebanon. It is a reduction of the latency across this hop that contributes the most to the downward shift in latencies to Lebanon in recent weeks.

In the coming weeks we expect to see a dramatic shift in transit as Lebanese providers move away from expensive and high-latency satellite service to IMEWE-based service. This is likely to resemble the collapsing of satellite Internet markets which we have observed occurring in African countries almost immediately after new cable landings are established nearby. For example, the chart below is the relative mix of Internet transit providers for Lebanese provider TerraNet (AS39010) in 2011. In this stacked chart, a taller band of color indicates greater amount of Internet transit through that provider for a given point in time. Except for Liban (AS42020), the vast majority of their transit is satellite.


This chart is almost certain to look different in a couple of months if the new cable is here to stay. We’ll update this story as it develops. With any luck, latencies to Lebanon may begin to resemble those of other countries of the Eastern Mediterranean and Lebanon’s inexplicable dependence on satellite will be a thing of the past.

What to do about hair-pinning?

There are many causes of latency in network performance, such as congestion and router overutilization, but hair-pinning is often avoidable. Businesses need Internet intelligence to know their high latency is a result of hair-pinning and then press their providers to do something about it! This is particularly important for mobile providers because it adds another potentially severe performance penalty to the Internet service they provide.


The post Pinning Down Latency appeared first on Dyn Research.


About the Author

Doug Madory is a Director of Internet Analysis at Dyn where he works on Internet infrastructure analysis projects. Doug has a special interest in mapping the logical Internet to the physical lines that connect it together, with a special interest on submarine cables.

Follow on Twitter More Content by Doug Madory
Previous Article
Asia on My Mind and on the Move
Asia on My Mind and on the Move

Undersea cables are expensive to install. But if you’re an Asian Internet...

Next Article
Irene Wallops US Internet
Irene Wallops US Internet

Hurricane Irene knocked out power to millions of homes and businesses as it...