Here we go again. In March we wrote a blog entitled Accidentally Importing Censorship which described how incorrect DNS answers were returned in response to certain queries to the I-root. The problem was tracked down to a single instance of the I-root located in China. Queries to this server for domains blocked in China, such as Facebook, would return seemingly arbitrary answers. As we noted, countries, and even companies, can impose their own standards on the Internet and block anything they want. This story was only noteworthy because those blocks (via bad DNS answers) became visible outside of China. Well, guess what? We are once again seeing the Beijing I-root from outside of China.
Let’s start with a few disclaimers and some background. First and foremost, the sky is not falling. Getting the wrong DNS answer, even when querying the Chinese I-root instance is an extremely rare event. Go back and read our earlier blog to see the exact alignment of the stars that would be necessary. The fact that it is so rare is what kept the problem from being detected for weeks. However, as we noted in that earlier blog, given the broad swath of the Internet potentially querying the Chinese I-root instance, someone was bound to stumble on a bad DNS answer and, as a result, not be able to friend their pals. This is exactly what happened and is what brought the problem to light.
Second, the fine folks at Netnod, who provide the exceptional and free I-root service, vigorously defended their services in China, asserting they provide the same DNS answers regardless of location. We have no reason to think otherwise.
Third, it’s quite easy to see incorrect answers from DNS servers in China yourself, whether or not you happen to live there. This has nothing to do with any of the root name servers. Just pick your favorite DNS server based in China and ask it about Facebook. Here is an example of repeated queries from the Linux command line from a US-based machine to a China Telecom DNS server.
dig @dns1.chinatelecom.com.cn. www.facebook.com. ... www.facebook.com. 11556 IN A 126.96.36.199 www.facebook.com. 24055 IN A 188.8.131.52 www.facebook.com. 38730 IN A 184.108.40.206
None of these IP addresses has anything to do with Facebook. In fact, addresses starting with 37 haven’t even been allocated by IANA as of this writing.
Of course, if you don’t live in China, you probably don’t use a Chinese DNS server directly. The problem is that we all use the root name servers and they are spread throughout the world. Thanks to the vagaries of Internet routing, you may end up querying any of them, regardless of where you live and where they are hosted. Thus, if you live outside of China and just happen to query a root name server hosted in China, your queries will pass through what is known as the The Great Firewall, and hence will be subject to any restrictions it imposes.
While doing some research for next week’s NANOG meeting in San Francisco, we revisited the time line for the March I-root announcements from China and couldn’t help but notice the problem resurfacing on June 3rd. The I-root resolves to 220.127.116.11, which is announced by AS 29216 (which is dedicated to the I-root) as both 18.104.22.168/23 and 22.214.171.124/24. From there, these prefixes travel via Netnod’s AS 8674 and then onto the general Internet. Since Netnod anycasts these prefixes from dozens of locations around the world, we expect to see them via any number of BGP adjacencies to AS 8674 and, in fact, we do. Around 80 different ASes adjacent to Netnod’s AS 8674 see the two I-root prefixes and, in turn, propagate them onward.
What we do not expect to see are mainland Chinese ASes adjacent to AS 8674 propagating these prefixes outside of China. This is what we did see in March 2010 and it implies Internet users outside of China could be directed to the I-root instance inside of China. Unfortunately, this problem has returned. We see AS 8674 passing just 126.96.36.199/24 off to AS 24151 and then AS 7497, both of which are associated with the China Internet Network Information Center. From there, the prefix travels via Pacnet (AS 10026), formerly Asia Netcom, and PCCW (AS 3491) out to the general Internet. This started just before 10:20 UTC on June 3rd and is still ongoing as of the date of this blog.
As we noted last time, to get a bogus DNS response outside of China, you not only have to query the I-root, you have to query the Chinese instance of it. To measure potential impact, we looked at the originating country of all prefixes downstream of any provider selecting the Chinese I-root. We then computed the percentage of these relative to the total number of prefixes in the country. A graph of the top dozen from the March incident is shown below, followed by those from this current (and ongoing) incident.
Not surprisingly, most of the affected countries are in Asia, as before, but there are important differences from the last event. Russia, India and Taiwan all entered the top twelve, while Pakistan, New Zealand and Bangladesh have dropped out. The impact on the countries in both lists is roughly similar, except that US impact went up by a factor of 10. Potentially impacted US states include Florida and California, making up approximately half of the US total. In addition, Singapore increased from 73% to 96%.
Censorship is a fact of life on the Internet today. But unfortunately, given the open, trust-based nature of the network, such censorship can easily spread beyond its intended boundaries. While individuals can do little to avoid such issues, there are actions network and system administrators can take. Filtering root name server announcements with Chinese ASes on the path is one approach. Never querying the I-root is another. Such actions would guard against this particular problem, but probably not the next one — whatever it might be. Ultimately, we are all in this together. We depend on each country or organization not to inadvertently or intentionally interfere with any other. All other paths lead down a very slippery slope.
About the Author
Earl leads a peerless team of data scientists who are committed to analyzing Dyn’s vast Internet Performance data resources and applying their expertise to continually improve upon Dyn’s products and services.More Content by Earl Zmijewski