Enhancement To P2P Cuts Network Costs

psycho12345 sends in an article in News.com on a study, sponsored by Verizon and Yale, finding that if P2P software is written more 'intelligently' (by localizing requests), the effect of bandwidth hogging is vastly reduced. According to the study, redoing the P2P into what they call P4P can reduce the number of 'hops' by an average of 400%. With localized P4P, less of the sharing occurs over large distances, instead making requests of nearby clients (geographically). The NYTimes covers the development from the practical standpoint of Verizon's agreement with P2P company Pando Networks, which will be involved in distributing NBC television shows next month. So the network efficiencies will accrue to legal P2P content, not to downloads from The Pirate Bay.

400%?

How do you reduce the number of 'hops' by an average of 400%? Negative number of hops? Also, FP.

Re:400%?

They probably discussed the number so many times that they lost track of how it was referenced. Lets say they cut it down to 25 from 100. If they went from their method, to the old method, then it would be a 400% increase in the hopcount.

Sloppy, but we can understand what they were trying to say.

Re:400%?

The number 400% appears nowhere in the article.

Re:400%?

Well, this being Slashdot, I couldn't exactly read the article, now could I? Instead, I opened the article and searched for "4".

Re:400%?

Just typical market speak. 400% sounds sexier than "a factor of four".

The problem that leaps to my mind is that either you're going to have to collect a huge chunk of routing information so your client can figure out which peers are "close" to you, or a third party is going to have to manage the peering...Neither one of those thrills me, especially since an ISP is pushing the technology, which would make them the obvious third party.

Re:400%?

It gets worse. From RTFA:

"Using the P4P protocol, those same files took an average of 0.89 hops"

How do you possibly get an average of LESS than one hop, unless you're getting the file from yourself?

Re:400%?

How do you possibly get an average of LESS than one hop, unless you're getting the file from yourself?

Easy! They ran it in simulation, using VMware. Have you ever used VMware? It's an amazing tool that makes an excellent platform for simulations and prototypes, especially when you need to know exactly how applications will perform in the real world.

Game developers, for example, routinely use VMware sessions. Especially the hard-core, 3D FPS developers.

No, really!

Re:400%?

Speaking as the guy that ran the test, I should explain the "hop count" decrease observed in the test in more detail than the article. First, I should clarify that the 'hop' is a long-distance link between metro areas, because that is the resource that is scarce - we ignored router hops, because they aren't meaningful, and generally aren't visible inside ISP infrastructures for security reasons. This means that data that moves within a metro area is zero hops, data pulled from a directly connected area is one 'hop', and so on.

So in the field testt we saw data transmission distance drop from an average of 5.5 'hops' to 0.89 'hops'. This happens because P4P provides network mapping information, allowing the p2p network to encourage localized data transfers. Generic p2p moved only 6.27% of data within a metro area, while p4p intelligence resulted in 57.98% same-metro area data transfer. Thus deliveries are both faster and cheaper.

What information are we talking about?

For other ISPs to reap the benefits Verizon did in the test, they too would have to share information about their networks with file-sharing companies, and that they normally keep that information close to their chests.

Excuse my ignorance, but what about their network is secret, other than the prices they're paying?
Network topology isn't & can't be a secret...

Re:What information are we talking about?

You seem so certain.

Your traceroute program doesn't tell you when your traffic is being routed four hops through a tunnel to cut down on visible hops and to save space in the ISP's main routing table. Without the routing tables at hand you don't know the chances of being routed through your usual preferred route and through a backup route kept in case of congestion. Nothing from the customer end shows where companies like Level 3 and Internap have three or four layers of physical switches with VLANs piled on top between any two routers. Nothing tells you when you're in a star build-out of ten mid-sized cities that all go to the same NOC vs. when you're being mesh routed over lowest latency-weight round robin, although you might guess by statistical analysis and mesh routing of commercial ISP traffic outside the main NAPs is getting more and more rare.

There's a lot you can easily deduce, especially if your ISP uses honest and informative PTR records. There's still much that an ISP can do that you'll never, ever know about.

I worked for one ISP where we had 5 Internet connections in four cities to three carriers, but we served 25 cities with them. We had point-to-point lines from our dial-in equipment back to our public-facing NOCs. We had a further 18 or so cities served by having the lines back-hauled from those towns to our dial-in equipment. We had about 12k dialup customers and a few hundred DS1, fractional DS1, frame relay, and DSL customers. Everyone's traffic went through one of two main NOCs on a good day, and their mail, DNS, AAA, and the company's web site traffic never touched the public Internet unless we were routing around trouble. In a couple of places we even put RADIUS slaves and DNS caching servers right in the POP.

I worked for another that served over 40k dial-up and wireless customers by the time they sold. We had what we called "island POPs". Each local calling area we served had dial-in equipment and a public-facing 'Net connection. Authentication, Authorization, and Accounting, DNS, Mail, and the ISP's website traffic all flowed over the public Internet except in the two towns we had actual NOCs. There were tunnels set up between routers that made traffic from the remote sites to the NOCs look like local traffic on traceroute, but that was mainly for our ease of routing and to be able to redirect people to the internal notification site when they needed to pay their late bills. We (I, actually) also set up L2TP so that we could use dial-up pools from companies like CISP who would encapsulate a dial-in session over IP, authenticate it against our RADIUS, and then allow the user to surf from their network. We paid per average used port per month to let someone else handle the customer's net connection while we handled marketing, billing, and support.

The first ISP I worked for had lines to four different carriers in four different NAPs in four different states, lots of point-to-point lines for POPs, and a high-speed wireless (4-7 MBps, depending on weather, flocks of birds, and such) link across a major river to tie together two NOCs in two states. Either NOC could route all of the traffic for all the dozens of small towns in both states as long as one of our four main connections and that wireless stayed up (and all the point-to-point ones did, too). If the wireless went down, the two halves of the network could still talk, but over the public Internet. That one got to about 10k customers before it was sold.

At any of those ISPs, I couldn't tell you exactly who was going to be able to get online or where they were going to be able to get to without my status monitoring systems. On one, all the customers could get online even without the ISP having access to the Internet, but they could only see resources hosted at the ISP. Yet that one might drop five towns from a single cable break. Another one might keep 10k people offline due to a routing issue at a tier-1 NAP, but everyone else was okay. However, if that one's NOC went offline, anyone surfing in other

Is it just that I'm naive ...

... or is it encouraging to see network providers taking a stance other than p2p is bad? This looks good - kind of like "p2p isn't going away, so as long as we have to live with it, let's try to make the best of it"

New math

Reducing hops by 400%, eh? That's a nice trick. Can we reduce bandwidth usage by the same amount? I wouldn't mind some free bandwidth.

I honestly can't figure out where "reduce by 400%" came from. They say the average hops were reduced from 5.5 hops to 0.89 hops, which is either 84% if you're not an idiot or 616% if you are. So I'm really quite confused here. Go figure.

Re:New math

I think I figured out their math, and you aren't going to like it:

5.5 * 0.89 - 0.89 = 4.0050 or 400%

As opposed to:

( 5.5 - 0.89 ) / 5.5 = 84%

Localizing means less anonymity

While I understand what they're saying here, and I understand the surface intent of the message, I get this feeling that there is some sort of devious underlying motive here. Or it could just be that I have my Slashd^H^H^H^Htinfoil hat on a bit too tight.

innumeracy

reduce the number of 'hops' by an average of 400%

This glaring example of innumeracy is from the submitter, as it is nowhere in the article.

On average, Pasko said that regular P2P traffic makes 5.5 hops to get its destination. Using the P4P protocol, those same files took an average of 0.89 hops.

That works out to an average 84% reduction.

Re:innumeracy

On average, Pasko said that regular P2P traffic makes 5.5 hops to get its destination. Using the P4P protocol, those same files took an average of 0.89 hops.

Less than one hop on average? Wow, they must use patented "You downloaded that three months ago, you wanker! Look on your damn file server!" technology.

Alternative NYT article

http://hosted.ap.org/dynamic/stories/P/P2P_VERIZON?SITE=MITRA&SECTION=HOME&TEMPLATE=DEFAULT [ap.org]

Verizon actually doesn't suck

For this reason, Verizon doesn't suck for broadband uses. In my area, I have Verizon DSL (they haven't given us Fios yet, but they ran the fiber cables a few years back) and I don't have any port blocking (that's right folks, I can send email to ANY server), and they don't limit P2P or Bittorrent (My downloads are fast and fresh). And they haven't turned records over to the government (or at least not reportedly, yet). So far, in the category of BIG ISPs Comcast vs Verizon, Verizon is being the underdog. Which is funny, because start arguing cell phone policies and prices, and watch the argument change completely.

Generally, what matters is acceptance

And let's face it, people, the next protocol will have to have a few features to be accepted, and having "local peers" isn't on the top of the list.

What the list includes? Easy:

1. Encryption
2. Onion routing

For very obvious reasons. And neither of them decreases bandwidth used. Quite the opposite.

Geographically isn't what's needed

less of the sharing occurs over large distances, instead making requests of nearby clients (geographically).

How about a BitTorrent client that gives preference to peers on the *same ISP*?

Yeah, less hops and all is great, but if an ISP can keep from having to hand off packets to a backbone, they'll save money and perhaps all the hue and cry over P2P will die down some. I'm sure Comcast would rather contract with UUnet to handle half of the current traffic destined for other ISPs than they do now.

Sort of a 'be nice to the ISPs and they'll be nicer to the users' scenario.

Hey, I've got a study too...

it's called Mbone [wikipedia.org]. It was created 15 years ago by a bunch of people including Van Jacobson, who had already helped create TCP/IP, wrote traceroute, tcpdump and so forth.

It would have made Internet broadcasting much more efficient, but it never took off. Why? Because providers never wanted to turn it on, fearing their tubes would get filled with video. So what happened? People broadcast videos anyhow, they just don't use the more efficient Mbone multicasting method.

Furthermore, when I download a video via Bittorrent, there are usually only a few people, whether they have a complete seed or not, who are sending out data. So how local they are doesn't matter. If there are more people connected, usually most people are sending data out at less than 10K, while there is one (or maybe 2) people sending data out at anywhere from 10K to 200K. So usually I wanted to be hooked to them, no matter where they are - I am getting data from them at many multiples of the average person.

I care about speed, not locality. The whole point of the Internet and World Wide Web is locality doesn't matter. Speed is what matters to me. For Verizon however, they would prefer most traffic goes over their own network - that way they don't have to worry about exchanging traffic with other providers and so forth. Another thing is - there is tons of fiber crisscrossing the country and world, we have plenty of inter-LATA bandwidth, the whole problem is with bandwidth from the home to the local Central Office. In a lot of countries, natural monopolies are controlled by the government - I always hear about how inefficient that would be and how backwards it would be, but here we have the "last mile" controlled by monopolies and they have been giving us decades-old technology for decades. In fact, the little attacks by the government have been rolled back, in a reversal of the Bell breakup, AT&T now owns a lot of last mile in this country. Hey, it's a safe monopoly that the capitalists, I mean, shareholders, I mean, investors can get nice fat dividends from in stead of re-investing in bleeding edge capital equipment, so why give people a fast connection to their homes? Better to spend money on lawyers fighting public wifi and the like, or commissars and think tanks to brag about how efficient capitalism is in the US of A in 2008.

Only work if they open the topology data...

Some of us working in the bleeding edge of p2p have been playing with these ideas for years to improve performance (I'm building open VR/MMO over P2P), here's the basics...

Most true p2p systems use something called a Distributed Hash Table (DHT) [wikipedia.org] to store and search for metadata such as file location and file metadata. Examples are Pastry [wikipedia.org], Chord [wikipedia.org], and (my favorite) Kademlia [wikipedia.org]. These systems index data by ids which are generally a hash (MD5 or SHA1) of the data.

Without going into the details of the algorithms, the search process exploits the topology of the DHT, which becomes something called an "overlay network" [wikipedia.org]. This lets you efficiently search millions of nodes for the IDs you're interested in in seconds, but it doesn't guarantee the nodes you find will be anywhere near you in physical or network topology space.

The trick some of us are playing with is including topology data in our DHT structure and/or search, to weigh the search to nodes which happen to be close in network topology space.

What they are likely doing is something along these lines, since they have the real topology instead of what we can map using tools like tracert.

If they really want to help p2p, then they would expose this topology information to us p2p developers, and let us use it to make all our applications better. What they're likely planning is pushing their own p2p, which will be faster and less stressful on their internal network (by avoiding peering point traversal at all costs, which is when bandwidth actually costs THEM). The problem is their p2p will likely include other less desired features, like RIAA/MPAA friendly logging and DRM, and then they'll have a plausible reason to start degrading other p2p systems which aren't as friendly by their metrics, such as distributing content they don't control or can't monetize... Then again, maybe I'm just a cynic...

Re:P2P - P4P?

Well, strictly speaking, incrementing the number would result in P3P, not P4P. Just as P2P means "Peer to Peer", P4P could be interpreted as "Peer for Peer", justifying the numeral.

Re:P2P - P4P?

Well, strictly speaking, incrementing the number would result in P3P, not P4P. Just as P2P means "Peer to Peer", P4P could be interpreted as "Peer for Peer", justifying the numeral.

Personally I'm waiting for the next binary progression, Peer Ate Peer, or P8P. I'm not sure what it will do, but I'll bring popcorn to watch...

Re:P2P - P4P?

Even better, change the letter too. I'm waiting for B4B: Beer 4 Beer.

mmmm... Beer.