As in most lovers’ quarrels, it is difficult to objectively evaluate the claims of the combatants. Naturally, we tend to side with the person we know best, as it’s their viewpoint we hear most often and are inclined to be sympathetic towards. Both Cogent and Telia are claiming to be the aggrieved party in their peering dispute and are now making their case in the court of public opinion. We will almost certainly never know the details of their private business relationship, but we can make a few more inferences from the data. Let me state up front that, like many major ISPs, Telia and Cogent are customers of Renesys and we love them both equally. Everything we report in our blogs is based on objective analysis of our global data, independent of our own business relationships.
So let’s recap what we do know at this point.
- At the end of the day on March 13th, the peering link between Cogent and Telia went away, cutting off some of their respective customers (those without other providers) from each other.
- Cogent took responsibility for pulling the plug.
- Despite this, Telia traffic found its way to Cogent via Verizon for 12 hours after the initial event.
- Something then happened to cut off this alternate route.
Why did the Verizon routes disappear?
So what exactly was that “something”? It is difficult to say with certainty, but we have only three choices:
- Cogent stopped accepting routes to Telia via Verizon.
- Telia stopped accepting routes to Cogent via Verizon.
- Verizon stopped transiting traffic between the two estranged parties.
In our last blog, we gave reasons why we thought the first option might be true. After sifting through more data, we will present possible reasons for the other two options, either of which may be more likely than the first. We’ll let you decide, but first we’re going to have to digress a bit and provide some background for those less familiar with the workings of Internet routing.
While it’s an overused term, Tier-1 providers are thought of as those organizations like Sprint and AT&T who are at the top of the Internet food chain. They pay no one for service — people pay them directly or pay their customers. The only reason all customers of Sprint can reach all customers of AT&T, for example, is because both companies agree to exchange traffic with one another without charging. At a routing level, Sprint tells AT&T which networks it is ultimately responsible for so that AT&T will send traffic for those networks to Sprint. AT&T does likewise for Sprint. Such an arrangement is known as peering, and since no money changes hands it is more preciously known as settlement-free peering. There are only about nine Tier-1 providers, and for everything to work, they all need to peer with one another. They also have no economic interest in letting anyone else join the club. But of course that doesn’t stop others from trying to force their way in, and there are certainly marketing advantages to claiming you are at the top of the heap.
Cogent and Telia have long been Tier-1 wannabes. To attain this status, it helps if you have a global network and lots of customers and peers yourself. Then you can start to play the existing Tier-1s off one another for your traffic, and with a lot of effort and good fortune, you might get settlement-free peering with some of them. At worst, you might have to pay for peering from those with whom you don’t have enough leverage. From a routing perspective, paid peering looks exactly like settlement-free peering. At Renesys, our Market Intelligence offering denotes both relationships as simply “peering”, since we have no way of knowing if money is changing hands.
For a long time, we have watched Cogent and Telia get closer and closer to Tier-1 status, at least as far as looking like they have no providers, whether or not they pay for some of their peerings. Cogent is almost there, but is still a customer of NTT (AS 2914) for the sole purpose of reaching AOL (AS 1668). (Interestingly enough, Cogent could become Tier-1 by default before long, as AOL continues to shrink and sell off its network.) As of a few weeks ago, Telia also looked to have one provider (namely Verizon) for reaching certain networks. But we stopped seeing evidence of that on February 27th. That is, after this date, Telia had no known provider to reach any markets anywhere, using only their own network or links routed as peering (paid or not) to reach everyone.
Then Cogent de-peered Telia and suddenly Verizon and others started providing a path between the two and their respective customers. At the time of my last blog, I was still thinking Verizon was a provider for Telia, so I was not surprised. (The other providers we saw were the result of leaks (mistakes) or because of customer dual homing.) But it would make no sense for Verizon to suddenly go away if they were still a transit provider for Telia, hence my conjecture about Cogent stopping this path. As mentioned above, there are two other possibilities, so let’s now examine these.
Suppose Telia really did stop being a customer of Verizon a few weeks ago and got settlement-free peering. Verizon’s routing announcements might not have fully reflected that change in their business relationship. That is, perhaps Verizon was announcing more than their customer routes to Telia, routes that might only show up if Telia lost access to parts of the Internet via other means. And so when the Telia-Cogent link was severed, Verizon inadvertently started providing free transit to Telia to reach Cogent. If true, it would have been in Verizon’s interest to correct this mistake as quickly as possible. They would have no reason to connect Telia to Cogent without getting paid by either party. This scenario seems very unlikely to us as it would involve a substantial misconfiguration on the part of Verizon of a magnitude we have not seen in the past.
Finally, suppose that Telia has paid peering with Verizon and so Verizon still views them as a customer and perhaps is sending them more routes than they would a settlement-free peer. Again, such routes might have only become visible when the Telia-Cogent link went away and Telia started using them instead. If Telia has paid peering with Verizon, then they have to pay according to the amount of traffic they send them. Telia might not be very interested in paying for something (access to Cogent) that was formerly free. Such a situation could have inspired Telia to block this path themselves.
But no matter what happened here at a technical level and what the corresponding motivations were, the Internet has been partitioned for more than one week after the start of this spat. This seems quite odd to people and I keep getting asked “OK, but why isn’t this a bigger deal? Give me just one juicy example, e.g., Norway can’t reach CNN or something like that.” Unfortunately and fortunately, I don’t have any such juicy examples. Unfortunately, since if there were such highly visible examples, there would probably be enough external pressure on the warring parties to bring about a quick peace agreement. Fortunately, since this partitioning is not impacting most of us.
But for those of you who have asked for concrete examples, here is one. Martha Stewart Living is single-homed behind Cogent and announces one network, namely, 220.127.116.11/24. If you go to Telia’s looking glass as of the time of this posting, you cannot get to Martha’s network. As far as Telia is concerned, Martha doesn’t exist. Does this mean that the Swedes are deprived of the pleasure of buying Martha’s wares and sending her email? Not at all. Her web site is hosted by Savvis and a customer service email address points to AOL. But if Martha’s parole officer allows her to visit Scandinavia any time soon, she won’t be able to reach her corporate network. (By the way, I found this network by looking up all the single-homed customers of Cogent in Renesys’ Market Intelligence.) This is not to make light of the situation, but to point out that the disruptions are largely personal in nature. And while this rift in the Internet is seriously impacting a wide range of individuals, it’s unlikely to be resolved anytime soon without a lot more yelling and screaming.
It also points out the dangers of being a Tier-1 provider or a near Tier-1. Sure you can claim you’ve made it to the big time and don’t need to buy from anyone, but you do need cooperation from the rest of the cartel and not one of them is going to do you any favors if you are perceived to be in a position of weakness relative to them. After all, this is all about money, and outside of the US Federal Reserve, no one else seems to be giving that away.
About the Author
Earl leads a peerless team of data scientists who are committed to analyzing Dyn’s vast Internet Performance data resources and applying their expertise to continually improve upon Dyn’s products and services.More Content by Earl Zmijewski