2017 Internet Intelligence Roundup

December 21, 2017 David Belson

With 2017 drawing to a close, year-end lookbacks litter media and the blogosphere like so many leaves on the ground. (Or piles of snow, depending on where you are.) Many tend to focus on pop culture, product/movie/music releases, or professional sports. However, given the focus of Oracle Dyn’s Internet Intelligence team on monitoring and measuring the Internet, we’re going to take a look back at significant Internet “events” of the past year, and how they have impacted connectivity for Internet users around the world.

Hurricanes Harvey, Irma, and Maria Cause Internet Disruptions

In late August, and through September, an active Atlantic hurricane season spawned a number of destructive storms that wreaked havoc across the Caribbean, as well as Florida and Texas in the United States. On the Caribbean islands that were hardest hit by the storms, the resulting physical damage was immense, severely impacting last-mile Internet infrastructure across the whole country. This was also the case in Florida and Texas, though on a much more localized basis. On September 25, we looked at the impacts of these hurricanes on Internet connectivity in the affected areas, noting that while some “core” Internet components remained available during these storms thanks to hardened data center infrastructure, backup power generators, and comprehensive disaster planning, local infrastructure – the so-called “last mile” – often didn’t fare as well.

Towards the end of August, Hurricane Harvey forced hundreds of network prefixes in Texas offline, while a few days later, Hurricane Irma caused similar problems in Florida and Puerto Rico. Sint Maarten was also hit extremely hard by Hurricane Irma, causing complete unavailability of network prefixes associated with the island nation.

Internet Impacts from #HurricaneHarvey and #HurricaneIrma in #Texas, #Florida and #PuertoRico pic.twitter.com/01ZWbqgTWz

— InternetIntelligence (@InternetIntel) September 14, 2017

Nearly two weeks later, Hurricane Maria slammed into Puerto Rico, causing problems for local Internet connectivity as it made landfall. The power outages resulting from the storm caused last-mile connectivity to deteriorate, as we observed through a near-complete loss of recursive DNS queries coming from the island. Connectivity continued to struggle a week after Maria, and a recent Internet Intelligence blog post examined the state of Puerto Rico’s post-hurricane Internet connectivity.

Politically Motivated Internet Shutdowns

Nationwide Internet shutdowns for political reasons arguably had their genesis in a January 2011 Internet disruption that occurred in Egypt, which was followed in short order by similar disruptions in Bahrain, Libya, and Syria. These outages took place during what became known as the Arab Spring, highlighting the role that the Internet had come to play in political protest, and heralding the wider use of national Internet shutdowns as a means of control. A November blog post noted that while these shutdowns took place in the Middle East and Northern Africa, they have shifted over the last several years to become more common in sub-Saharan Africa.

Such outages continued to be the case over this past year. In mid-November, Equatorial Guinea’s government ordered a complete Internet blackout ahead of an election that was expected to keep the party of longtime President Teodoro Obiang Nguema in power. This blackout was in addition to blocking of access to opposition Web sites, which started in 2013. In September, the government in Togo blocked access to mobile Internet connectivity amid anti-government protests. Following months of protests, Cameroon’s government ordered an Internet blackout in English-speaking regions of the country starting in mid-January. This outage lasted until April, and Internet connectivity in these regions was again disrupted in early October, apparently in relation to mass protests. As of late November, this latest disruption was still in place.

Multiple Exam-Related Outages in Syria & Iraq

Students around the world have long attempted to get an advantage on standardized tests, by whatever means necessary. Of late, test-related information has been shared via the Internet, leading the governments of Syria and Iraq to sever Internet connectivity within their respective countries in an effort to prevent cheating on such tests. The Iraqi government employed such techniques in 2015 and 2016, while the Syrian government also did so several times in 2016.

In February 2017, the Iraqi government took down the country’s Internet connectivity for multi-hour periods across multiple days. As we noted at the time, the duration of the Internet outages covers the period of time of the physical distribution of the exam materials to testing centers, which typically begins at 5:00 am on exam day. The outages are intended to prevent images of the questions from the exams, along with the answers, from being shared via social media. Similar outages were also observed in Iraq during the first half of June.

Earlier today: 4th internet blackout in #Iraq, 9th in #Syria to combat cheating on exams. https://t.co/LIutGoqjwL pic.twitter.com/8WuH6MMIEe

— InternetIntelligence (@InternetIntel) June 13, 2017

In late May, Syria began a series of nationwide Internet disruptions designed to combat cheating on exams. The outages occurred nine times over the course of two weeks. The Syrian Internet also appeared to go completely offline on July 12, but we don’t believe that outage was related to any academic testing taking place within the country.

Leaked Routes Disrupt Connectivity in Japan and the U.S.

Route leaks occur when a network provider inadvertently announces routes to prefixes other than the ones they are responsible for. Sometimes a provider will announce routes learned from a peer that were not supposed to be shared any further. In other cases, the leaking provider “masquerades” as the origin of the route, while more significant leaks occur when a provider announces a full routing table. Depending on the type of leak and how widely these leaks are propagated across upstream providers, the ultimate impact is that traffic to affected network prefixes is redirected, lost, or intercepted; the severity can range from unnoticed to catastrophic. Blog posts we published in 2015 and 2014 looked at several examples of route leaks and their impacts, while another 2015 post looked at the impact of a routing leak on the availability of Google services.

However, in late August 2017, Google turned the tables, leaking over 160,000 prefixes to Verizon, who accepted the routes and passed them on, severely impacting major Japanese telecommunications providers including KDDI, NTT’s OCN, and IIJ, disrupting Internet connectivity for users across Japan. The leaked routes were “more specifics” of routes already in the global routing table — these “more specific” routes cover smaller ranges of IP addresses, and are preferred to less-specific routes within the BGP route selection process. These “more specific” routes were believed to be used by Google for traffic shaping within their network, but when they were leaked to the world, they were selected by external network providers over existing less specific routes. This ultimately resulted in traffic between the impacted Japanese providers getting routed through Google’s network (in Chicago!), causing much of it to be dropped because of high latency or limited bandwidth.

Upstreams of Google (15169), 25 Aug 2017 through 25 Aug 2017

Just a few months later, a route leak from Level 3 (now CenturyLink) disrupted Internet connectivity for millions of Internet users across the United States and around the world. On November 6, Level 3 began globally announcing thousands of BGP routes that had been learned from customers and peers and that were intended to stay internal to Level 3.  By doing so, Internet traffic to major subscriber networks like Comcast and Bell Canada, as well as major content providers like Netflix, was mistakenly sent through Level 3. Our analysis indicated that other impacted networks included RCN, Giga Provedor de Internet Ltda (Brazil), Cablevision S.A. (Argentina), and even the Weill Cornell Medical College in Qatar. Based on our traceroute measurements, the leak ultimately resulted in increased latencies to reach the affected network prefixes, reportedly causing users to experience delays and problems in reaching some Web sites. A subsequent Tweet from Level 3’s Network Operations Center took responsibility but downplayed the impact, stating “On Nov. 6, our network experienced a disruption affecting some IP customers due to a configuration error. All are restored.”

Attempted Censorship Through BGP Route Hijacking

Authoritarian governments have long attempted to censor content for a variety of reasons, using a number of techniques. As more content (of all types) has moved onto the Internet, governments have often resorted to filtering end user Web and DNS requests, but the effectiveness of doing so has been inconsistent. However, hijacking IP address space belonging to content and/or hosting providers can allow a state telecom to functionally block access to sites served from those IP addresses for users on downstream networks in the country. While the routing announcements that implement the hijack are likely intended to stay within the country’s borders, sometimes they leak out. One example of this was Pakistan’s attempted block of YouTube in 2008.

In January 2017, we observed TIC, the Iranian state telecommunications provider, attempt to do something similar, hijacking IP address space belonging to a provider that hosts numerous Web sites featuring adult content. Unfortunately, these routing announcements made their way to Omantel, which announced them to other network providers, meaning that users outside of Iran may have been unable to access Web sites hosted at the hijacked provider. However, rapid action by Oracle Dyn team members enabled the hosting provider to quickly regain control of their address space. A few days later, TIC announced BGP hijacks of address space belonging to another hosting provider that serves adult content, as well as of 20 individual IP addresses belonging to Apple’s iTunes service.

Iranian state telecom hijacking IP space that is hosting adult websites. Censorship leaking out of Iran? #bgphijack pic.twitter.com/t4XTLnQhIS

— InternetIntelligence (@InternetIntel) January 6, 2017

In May, Ukrainian President Petro Poroshenko enacted a ban on Russia’s four most prominent Internet companies, reportedly in the name of national security.  The ban included the two most widely used social media websites, VKontakte and Odnoklassniki, as well as email service provider Mail.ru and search engine Yandex. In late July, Ukranian service provider UARNet began announcing new BGP routes that were hijacks of the IP address space belonging to these Russian companies, presumably as a means of implementing the previously announced ban. However, similar to what we have observed in the past, these hijacked routes escaped the country’s borders.

Latency Impacts of Submarine Cable Damage and Repair

Submarine cables span the globe like an ever-growing spider web, carrying Internet traffic between continents, and bringing international Internet connectivity to island nations. However, they are also prone to damage from errant ship anchors, as well as intentional sabotage. When cable breaks occur, observed latencies for Internet traffic to/from these countries generally increases as the traffic fails over to higher latency backup satellite connections. Conversely, when a new submarine cable connection is activated, observed latencies for Internet traffic in countries with these new connections generally drops. Over the course of 2017, we saw examples of both.

Starting at the end of December 2016, the Marshall Islands saw a nearly three-week period of reduced connectivity resulting from a submarine cable break — likely the HANTRU1 cable. The break caused lnternet traffic from the islands to transit a backup satellite connection with latency over 2x higher than the submarine cable. In mid-January 2017, damage to the Asia-America Gateway Cable System (AAG) and the Tata TGN-Intra Asia (TGN-IA) cable impacted Internet connectivity in Vietnam, resulting in latencies approximately 50% higher than normal, although the impact lasted just a few days. In late January, the Eastern Africa Submarine System (EASSy) cable was cut, crippling Internet connectivity to Madagascar. Based on measurements to Telecom Malagasy (TELMA), a leading telecommunications company in Madagascar, connectivity was significantly reduced for approximately six hours before a backup link to satellite provider O3b was activated. In late June, the EASSy cable was again cut, significantly impacting connectivity to Somalia. Satellite connectivity through O3b was again used as a fall-back, resulting in latencies approximately one-third higher than normal. The SeaMeWe-3 (SMW3) cable connects a number of countries in Europe, Africa, and Asia, as well as landing in Perth, Australia. In late August, damage to the cable caused latencies to Perth to spike, with repairs estimated at the time to take until mid-October. In November, another cut to the AAG cable again impacted connectivity to Vietnam. However, in this case, we observed that the cable cut caused latencies along some paths to increase as expected, but that latencies along other paths actually dropped because they were now taking a more efficient route instead of “tromboning” through a more distant connection point.

Pacific island nation #Palau activated its 1st submarine cable yesterday. At 20:22 UTC on 21-Nov, PNCC completely switched from MEO satellite to subsea cable-based transit reducing latencies (see graphic). The new SEA-US spans the Pacific Ocean. https://t.co/apadBUFdGe pic.twitter.com/mGJspPH1q2

— InternetIntelligence (@InternetIntel) November 22, 2017

The tiny Pacific island nation of Palau activated its first submarine cable in November. The country previously relied upon an O3b satellite connection for Internet connectivity, and was able to reduce latency by switching to the SEA-US cable.

Cuba & North Korea

Cuba and North Korea have historically been two of the least Internet connected countries in the world. However, during 2017, both saw improvements to their international Internet connectivity. (In-country connectivity for end users is still severely limited in both countries.)

In early January 2017, we observed C&W Networks start to provide transit for ETECSA, marking the first time that a U.S. telecommunications firm provided direct transit to the Cuban telecom provider. C&W joined international providers Tata, Telefonica, and Intelsat in providing transit to ETECSA. Our measurements indicated that the C&W transit is being served from Boca Raton, Florida, with a 35ms round trip time to Havana, making it the lowest-latency link to the United States.

North Korea has historically had a single Internet provider, Star JV, which has relied on China Unicom for international Internet connectivity. However, on October 1, we observed that North Korea had gained a new connection to the global Internet through Russian fixed-line provider Transtelecom (TTK). However, subsequent measurements appeared to indicate that the new transit relationship was somewhat unstable. While it is impossible to tell simply from our Internet measurement data how TTK’s network connects into North Korea, it may be via the Friendship Bridge, a railway crossing over the Tumen River that connects Khasan in Russia with Tumangang in North Korea, as it is the only connection between the two countries.

North Korea activated first internet link via Russia as US steps up campaign against Pyongyang https://t.co/mmuTdilFSq pic.twitter.com/Zz62iMyfFP

— InternetIntelligence (@InternetIntel) October 2, 2017

With just a couple of Internet providers at its international border, North Korea is at severe risk of Internet disconnection. As such, the country is susceptible to complete Internet outages, such as those observed on August 14 and July 31 — the reasons for both are unknown. Cuba also saw a couple of unexplained outages on November 8, though both were brief, as our observations indicated that they lasted for approximately 10 minutes each.

Working Together to Secure the Internet

On August 17th, 2017, multiple Content Delivery Networks (CDNs) and content providers were subject to significant attacks from a botnet dubbed WireX, which was primarily comprised of Android devices running malicious applications and designed to generate DDoS traffic. Researchers from Akamai, Cloudflare, Flashpoint, Google, Oracle Dyn, RiskIQ, Team Cymru, and other organizations cooperated to combat the botnet. Collaborative work across these companies included identification of associated Web traffic and the IP addresses originating the requests, identifying the applications that housed the malware and removing them from app stores, and understanding the underlying code and command & control workflow. While certainly not the first instance of cross-industry collaboration, it is an example of how informal sharing can have a dramatically positive impact for the potential victims and the Internet as a whole.

What Else?

In addition to the events highlighted above, which were shared via blog posts and Tweets from Oracle Dyn’s Internet Intelligence team, 2017 also saw:

  • Hundreds of additional smaller brief network outages and disruptions that we detected, but that weren’t significant enough to share on social media
  • Other submarine cable cuts and activations, as well as an ongoing push by content/infrastructure providers like Google, Facebook, and Amazon to deploy their own cables
  • Many additional route leaks and hijacks

Hopefully the rest of December is quiet, Internet-wise.  But if it isn’t, be sure to follow us on Twitter at @InternetIntel, and on the Internet Intelligence blog at https://internetintelligence.oracle.com/, for the latest information and analysis.

Read more...

Previous Article
Recent Russian Routing Leak was Largely Preventable
Recent Russian Routing Leak was Largely Preventable

Last week, the IP address space belonging to several high-profile companies, including Google, Facebook and...

Next Article
Puerto Rico’s Slow Internet Recovery
Puerto Rico’s Slow Internet Recovery

On 20 September 2017, Hurricane Maria made landfall in Puerto Rico.  Two and a half months later, the islan...