I think Level3's issue is that Comcast is unwilling to increase the capacity of their links without payment from Level3. If Level3 engages in settlement-free peering with Comcast and their network traffic graph looks anything like the 100gbit link in the article, I don't blame them: it's a 20:1 disparity in traffic in vs traffic out.
What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them.
And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?
The big trouble is that we need settlement-free peering, or we'll have a very different sort of internet, or maybe none at all. Though it's hard for some to remember (or to believe), there were big consumer networks prior to the advent of consumer-oriented Internet. Compuserve and AOL were their own networks. And they were abysmal. Peering not only ruined that business model, it managed to co-opt those networks and assimilate them into the Internet proper.
If peering is untenable because it leads to outcomes like the one we have now (Netflix) where it is fundamentally unfair to one party (assuming that is the case) or the other, then we're all screwed. There might not be a fair solution that still manages to resemble the Internet.
In one alternative, the Comcasts and the AT&Ts are smited and no longer exist, and Level 3 goes into the consumer internet business, connecting things end-to-end. That's not something we could trust them with. Same if Comcast replaces Level 3.
In another, Netflix continues to pay the extortion (if it is that) which we have little doubt that Comcast will continue to ratchet up the price for. Or Comcast has to shoulder that burden alone (and can't charge customers extra for doing so, without people screaming "net neutrality!").
Or maybe services like Netflix just can't exist in such an environment. Also a bad outcome.
Or god help us, internet infrastructure is nationalized, and the same people who manage our roads and traffic lights take over.
You could have a situation where the government mandates net neutrality and at the same time compensates the likes of Comcast in some way (recognising their role as public utility and basically sanctioning their monopoly, but without taking over their operations entirely). You could have an anti-monopolistic legal action that splits Comcast but prevents Level3 from taking over / being present in that market. Etc etc...
There are many ways to skin an onion, and full-scale nationalisation is probably the least likely outcome (cash-strapped governments don't need another headache right now). In fact, the real problem here is that one player is leveraging a monopolistic position; remove that position, and the Free Market should start working its magic again.
We tried that once with Bell Telephone. Their profits were dialed down a bit via regulation, and it was an oppressive, horrible, and un-innovative network. You weren't allowed to buy a telephone and plug it into the jack at home... you could only rent them from the company.
Thankfully, that company and environment are gone. Let's not recreate them.
Because all the numbers I see are showing about a 5:1 or 6:1 imbalance.
> ...transit providers are starting to see their industry be squeezed by the big ISPs.
No.
> ...why do transit providers even exist?
Because it's untenable (and inefficient) for Comcast to build a separate fiber network to every service provider (e.g. to start providing a service on your version of the internet, you'd have to build your own network connection to Comcast, and then again to Verizon, and again to AT&T, and again to every other ISP on the planet).
Take another look at the L3 network map from the blog post:
If you think transit providers are purely middlemen for the sake of middlemen then you're basically saying that every service provider should build out a network of that same size in order to reach every ISP.
> Settlement-free peering assumes that bandwidth usage is roughly symmetric
No. It assumes that the benefit to each network is roughly symmetric. Put another way, if Netflix videos only degrade on Comcast's network, Comcast's goodwill with its own customers will increase as a result of increasing transit/peering capacity. To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile.
Comcast empirically does not care about the goodwill of its customers, which is not surprising given its monopoly of local markets. If customers had alternatives to Comcast and it became known that Netflix worked on one ISP and not the other, customers would switch ISPs en masse.
You're right to call me out here. Although I have a reasonable idea what it would take technically, I don't actually know the financial impact to Comcast for increasing capacity to L3 or Cogent (not even ballpark). However, I'm taking the L3 article at face value when it says:
But there are also typically shared costs for networks to
interconnect. Each party pays to augment its own network
to allow for more traffic exchange (the expense to augment
capacity is not significant for either party). And since
we often choose to interconnect in a third party data
center, the networks usually agree to share the cost of
the cross connects by paying for them on an alternating
basis.
I take that to mean the CapEx is insignificant and the recurring expenses are shared equitably. What's notable about that is that Comcast isn't even sitting down at the table to negotiate equitable terms for upgraded capacity. They're out and out refusing to upgrade without direct payment.
Cogent's CEO has already offered to pay outright for the port costs and data-center cross-connects for any upgrades. The point is to prove that the infrastructure cost of the interconnect are not the issue.
Upgrading that interconnect mentioned in the original article seems like just plugging in some $10k-$20k standard hardware and connecting a single extra wire. That kind cost doesn't matter really, it seems that they're hurting their interconnects intentionally for political reasons.
I was going from memory on the 20:1 thing and I was wrong, but 5:1 is still not even close to symmetric.
I also used a bit of hyperbole in the "why do transit providers even exist" part; transit providers do still need to exist to provide access for the long tail. But Level3 and the like do see their market shrinking as the big fish who currently pay them for transit either directly or indirectly move more bandwidth to direct connections with the big ISPs. It is definitely starting to squeeze them.
As far as the benefit to each network goes, that's hogwash. It's not how the contracts are written because you can't quantify "benefit". You can quantify packets sent/received. Netflix needs to make money just as badly as Comcast does, so the fact that their service is crappy is much more their problem than it is Comcast's. Netflix can always purchase transit through another company that's not Comcast.
I can assure you that L3 and Cogent are not seeing the marketplace shrink. 10k quarterly files from each discuss their growths and vulnerabilities.
Historically, eyeball networks have paid for transit from backbone providers. As market consolidation occurred, the cable providers with government granted physical monopolies were able to negotiate for better settlements with their transit providers.
Eventually, the cable companies condensed with the major telephone companies. Some of the new mega companies have backbones and can use their own networks for transit (Verizon, AT&T, ...) others (Comcast) were able to use their size and access to their customers to negotiate largely settlement free exchanges.
However it is important to note that ALL eyeball networks have 1:5 to 1:20 demand ratios. This is the nature of content versus consumption. There is no new news regarding these ratios, and they are not particularly germane unless attempting to engineer the flows.
The real and pertinent issue is that Comcast has not lived up to the 'timely upgrade' clauses in boilerplate (NDA'd) settlement-free peering agreements. Clearly they have an advantage to "defect" from the standard cooperation model. Netflix has chosen to change providers several times, and recently provided data on whom they pay for service.
Why should an end-user ISP ever expect a settlement-free peering bandwidth graph to be symmetric? The important question is whether customers are requesting connections that route through through a particular transit provider. Comcast gets to sell "Internet" access, while the transit provider gets to sell bandwidth. Win win, no symmetry needed.
This "out vs in" comparison in order to determine who pays is a meme that won't die. I've never known agreements to be formed in such a way. Maybe before my day in the dial up era when "out vs in" could serve as rule of thumb of sorts to determin who was an access ISP and who was an "Upstream" or "Tier 1" ISP. Can someone provide a source for this?
Historically, ratios were used for peering between ISPs of the same type. When one backbone peers with another backbone, 1:1 ratio can be a simple proxy for equal value. (ISPs liked simplicity in the old days because it made the Internet cheaper than the complex telco X.25/ATM/SONET networks with their complex tariffs and settlement.) When a content provider peers with a broadband ISP, they can exchange equal value without exchanging equal traffic.
"Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay."
I've never understood why this is - especially in a case like this, where the sending side is sending because (the customers of) the receiving side explicitly asked for it.
Level-3 is actually going around to their peers and trying to create systems to measure the bit-miles traversed on traffic sent to other tier-1 peers; the carriers will then pay for out-of-sync ratios
Because the cost of a packet is borne by the receiver while the sender is the one monetizing it.
The assumption is that if you're sending traffic to a customer, you are being paid for that activity through subscription fees, ads, donations, etc. That packet costs the receiver some minuscule amount to process. Even though it was requested by the ultimate receiver, the ultimate sender is only sending the packet because they make money from doing so. The middlemen only pass on the packet because their costs are covered and they make a small amount of profit. This setup also ensures that if the packets aren't being monetized in some way (i.e. if it's a DDoS) that there's a financial incentive to stop them.
> you are being paid for that activity through subscription fees, ads, donations, etc.
Last time I checked ISPs where heavily monetizing packet receiving to the tune of 5-10x what netflix monetizes from me. For typically slow, shitty service bundled with cable TV or phone I do not want.
It's no less true for cold-potato routing; the source is just closer to the endpoint. And CDNs doing cold-potato routing happily pay for their transit because the service THEY make money off of is providing CDN services to their customers, who theoretically make money off the content they pay the CDNs to distribute.
What it comes down to is Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them. Why should Comcast be treated differently than any other transit provider and not allowed to monetize its transit services? Netflix wasn't paying Comcast, so Comcast had no responsibility to help Netflix make its own service better when there were other ways Netflix could have alleviated the bandwidth situation (e.g. buy transit through another company).
The original L3 article actually touches on this, sort of, by mentioning markets where ISP port congestion is not an issue (i.e. the UK) due to competition between ISPs.
Comcast will get paid by their customers as long as the routes they pass traffic through are 'good enough'. They have a (near) monopoly on the last mile in many markets, so 'good enough' can mean both 'barely working' and 'better than any other option you've got' at the same time.
So, you're right, but so is the parent: Comcast has no financial incentive to provide any more than just enough bandwidth to keep you from calling the support line to complain or cancelling your contract in frustration.
As others have said, Comcast is also being monetarily rewarded here. I think the best argument lies along "the sender is in the best position to take measures to reduce traffic (traded off against their goals)."
> Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay.
Does it assume that?
Since about the time of the web browser, we've had consumer-focused ISPs. Consumer traffic is mostly small requests for large replies. But as the article suggests, Level3 has a lot of settlement-free peering with consumer ISPs. My belief is that being settlement free isn't about equality of packets, but equality of demand. E.g., I have a lot of servers made to serve consumers; you have a lot of consumers wanting access to my servers; let's just split the costs.
In any case, given that those Comcast packets are mostly requesting all those Level3 packets, it seems much fairer to charge Comcast disproportionately, as its their customers who are creating the demand.
Right, there is a disparity — so, explain why it shouldn't instead be Comcast paying Level3?
Even if you want to privileged "sending" as being more charging worthy since you control what you send— in this case it's Comcast customers that have requested those bits, hosts connected to the level3 network are providing them.
Networks like comcast have built out imbalanced networks which likely wouldn't exist without their access to monopoly infrastructure. Practically anyone they peer with except other consumer broadband monopolists is going to be unbalanced.
Indeed: perhaps the answer to the symmetry argument is for Netflix clients to just start uploading scads of random data while the movie is streaming. Then the traffic will be "symmetric", although Comcast won't like it after all the last-mile upload links are saturated and customers can't do anything online.
Netflix hasn't done this (and likely won't) because would make it difficult/impossible to control the quality of their service. P2P connections are a bit of a crap-shoot based on things like router configuration, device type and resources. For example, P2P wouldn't work as well on an iPad where you're only caching a few hundred megs of video (and not the whole video) at any given time.
I don't think the parent meant to imply a P2P implementation of Netflix streaming video, just that Netflix is perfectly capable of making that ratio balance out by uploading crap from the clients they have (partial) control over - which would then be traffic coming from Comcast's network.
Torrenting moves large amounts of traffic into last mile infrastructure which tends to already be congested because its very costly to upgrade.
Due to using many distinct flows it also tends to not be very TCP friendly and takes a fairly unequal share of the bandwidth. And while users will give up and do something else when their interactive service becomes slow, a congested torrenting host keeps torrenting.
There doesn't seem to be any congestion on the last mile infrastructure of providers like Comcast. Their problems are all with interconnection to the real internet. It would actually be better for Comcast if their customers were getting more content from eachother over p2p networks than from external providers like Netflix.
What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them.
And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?