Re: [p2pi] My notes from today's workshop
"Livingood, Jason" <Jason_Livingood@cable.comcast.com> Thu, 29 May 2008 02:05 UTC
Return-Path: <p2pi-bounces@ietf.org>
X-Original-To: p2pi-archive@ietf.org
Delivered-To: ietfarch-p2pi-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E4FE13A6CE6; Wed, 28 May 2008 19:05:16 -0700 (PDT)
X-Original-To: p2pi@core3.amsl.com
Delivered-To: p2pi@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 4A1A128C1AA for <p2pi@core3.amsl.com>; Wed, 28 May 2008 19:05:10 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.214
X-Spam-Level:
X-Spam-Status: No, score=-0.214 tagged_above=-999 required=5 tests=[AWL=0.248, BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368, HTML_MESSAGE=0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2f75HQCZVXqH for <p2pi@core3.amsl.com>; Wed, 28 May 2008 19:04:42 -0700 (PDT)
Received: from paoakoavas10.cable.comcast.com (paoakoavas10.cable.comcast.com [208.17.35.59]) by core3.amsl.com (Postfix) with ESMTP id C04013A6CE0 for <p2pi@ietf.org>; Wed, 28 May 2008 19:04:35 -0700 (PDT)
Received: from ([10.52.116.31]) by paoakoavas10.cable.comcast.com with ESMTP id KP-TDCH7.46235345; Wed, 28 May 2008 22:04:16 -0400
Received: from PACDCEXCMB04.cable.comcast.com ([24.40.15.86]) by PAOAKEXCSMTP02.cable.comcast.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 28 May 2008 22:04:16 -0400
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Date: Wed, 28 May 2008 22:04:12 -0400
Message-ID: <45AEC6EF95942140888406588E1A6602045CB9B9@PACDCEXCMB04.cable.comcast.com>
In-Reply-To: <033801c8c104$9471e020$6401a8c0@china.huawei.com>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: [p2pi] My notes from today's workshop
Thread-Index: AcjBBLJvUaNqje9kTX+yIsp5L3kj8gAKr/mA
References: <033801c8c104$9471e020$6401a8c0@china.huawei.com>
From: "Livingood, Jason" <Jason_Livingood@cable.comcast.com>
To: Spencer Dawkins <spencer@wonderhamster.org>, p2pi@ietf.org
X-OriginalArrivalTime: 29 May 2008 02:04:16.0451 (UTC) FILETIME=[4C67A130:01C8C130]
Subject: Re: [p2pi] My notes from today's workshop
X-BeenThere: p2pi@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: P2P Infrastructure Discussion <p2pi.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/p2pi>, <mailto:p2pi-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/p2pi>
List-Post: <mailto:p2pi@ietf.org>
List-Help: <mailto:p2pi-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/p2pi>, <mailto:p2pi-request@ietf.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1612020863=="
Sender: p2pi-bounces@ietf.org
Errors-To: p2pi-bounces@ietf.org
Hi Spencer - I think it is important to correct that I think you misrecorded what we said regarding Vonage and other R/T traffic. We in fact were NOT managing Vonage or other OTT and R/T traffic in that timeframe, as we were not performing the TCP RSTs at that time. I believe we may have said that some people complaining thought we were in some way degrading Vonage traffic, which we were not. We then noted that after implementation of the NM technique currently in place that the complaints regarding poor Vonage and other R/T traffic began to decline, thus there appeared to be a rough correlation. Thanks Jason ________________________________ From: p2pi-bounces@ietf.org [mailto:p2pi-bounces@ietf.org] On Behalf Of Spencer Dawkins Sent: Wednesday, May 28, 2008 4:51 PM To: p2pi@ietf.org Subject: [p2pi] My notes from today's workshop Here's what I captured... Spencer 3.1 Welcome/Note Well/Intro Slides Goals: * To think about the Comcast/BitTorrent problem. * To think about what the IETF might do to work on the problem * Focus on what's actionable for the IETF, engineering, not research or politics. "Not dealing with illegal traffic" - Cullen asked us to focus on the problems that we still have when all the traffic is legal. 3.2 Service Provider Perspective (Comcast) Presenters are Jason Livingood and Richard Woundy from Comcast (including attributions because they're presenting a sponsor's perspective0. View that this is a two-way street - applications becoming more network-friendly, networks becoming more application-friendly. Comcast has committed to implement a protocol-agnostic network management technique (by yearend). Comcast was managing Vonage customer traffic, and customers were complaining. Not all the complaints reflected this management (Comcast and Vonage both experienced problems with transit providers), but that wasn't obvious to customers. Most ISPs started managing P2P traffic about 2-3 years ago. When Comcast started managing its traffic, they stopped getting complaints from Vonage and from P2P users. The recent concerns have involved a suspicion that Comcast was degrading Vonage VoIP service in an attempt to favor Comcast's own VoIP products. Current DOCSIS can experience congestion independently on upstream and downstream and experience congestion independently across DOCSIS domains. We talked about some DOCSIS details, but remember that we have to solve problems for a broad class of access networks, not just for DOCSIS. Slide point: "How 'TCP flow fairness' applies to the DOCSIS upstream is an open research question". DOCSIS also has constant-bandwidth "service flows" (so you don't need to request-to-send every packet). A Canadian cable company charges extra to set up a second service flow for Vonage, but other cable companies haven't followed this lead because of concerns about "net neutrality". We did note that if you have enough bandwidth, you don't need to prioritize traffic, but this doesn't help when traffic levels increase suddenly ("network capacity increases are not instantaneous"), and capacity increases can be consumed quickly. Not all customers are similar (even at the DOCSIS domain level) - college populations use more P2P, VoIP/streaming video are more diurnal than bulk file distribution. Comcast would like to avoid protocol-specific "cat and mouse" detection and mitigation, and would like to avoid capacity increases that benefit only 5 percent of heavy users. DOCSIS 3.0 adds capacity but doesn't solve the capacity problem. Comcast will deploy DOCSIS 3.0 in 20 percent of network by yearend. This more than doubles bandwidth available per subscriber, but the conflict is between adding capacity and supporting applications that are designed to maximize bulk bandwidth consumption. There are Internet2 techniques for client-to-network signaling that haven't been standardized in IETF yet... Caching content in the network can reduce TCP RTTs and reduce the peak-to-trough differences of individual flows. Network management trials begin in June - management based on traffic characteristics, not on specific protocols. Algorithm - mark all flows as priority and then mark specific flows as "best-effort" when congestion approaches "the Near Congestion State". Comcast hasn't figured out how to notify users that they have been marked as best-effort yet. BT is planning a similar mechanism on their DSL network, but they are planning on varying the bandwidth, not changing QoS markings. Comcast thinks customers are more concerned about changing markings and don't want to penalize high-bandwidth users when the network is NOT congested ("it's a high-bandwidth service, people should use it"). Henning is concerned about mechanisms that have 8 independent tweakable parameters... Comcast does still care about protocol-specific management mechanisms, when the carrier is providing the application, or when the traffic is illegal, etc. 3.3 Application Designer Perspective (BitTorrent) Presenter is Stanislov Shalunov. BitTorrent is willing to make changes based on outcomes at this conference. BitTorrent sees two problems - RTT measured in seconds, inefficient overlay routing. Peer selection is random, except that there is some bias towards established peers. Trackers randomize the peer list, and peers ask the resulting randomized list what peers they know about. Rarest-first ensures best reliability and helps peers have pieces of files to "trade". This also gives peer diversity. The best peers can be farther away - a peer with a big pipe, a long way away, can drive you at your downlink speed, but a peer on your own cable modem subnet can only drive you at its uplink speed. Danny's comment - restated for minutes: "If you have more localized distribution via caching, or location techniques, or whatever, then burst rates (i.e., peak <> trough ratios) may actually be larger and you may increase HFC contention (note: HFC in particular, v. DSL, etc..)" Overlay efficiency - Stanislov is experimenting with trying peers on same AS, or on a set of ASes (but then you need the set), or in a list of IP prefixes (but then you need the list), or based on some cost minimization model (but then you need costs). If you have more than 20 peers in an AS, you can probably trade files happily, and 57 percent of transit traffic could be reduced with this modification. If there aren't enough local peers, make one a cache (and please choose a fast one), or 10, or 100. Stanislov's experience is that they can do 30-percent hit rate with 1 TByte cache, 80-percent with 100 TBytes - and 1 TByte fits into a device. Dave Oran asked what the required bandwidth for an 80-percent hit rate of 1 TByte is - he is concerned that this architecture may reveal bandwidth bottlenecks onto and off the cache device itself. Stanislov hopes the IETF can work in these areas: * Cache discovery protocol * Network information for smart peer selection * Experimental congestion control * BCP on P2P pain Stanislov doesn't think we're ready to do congestion control for P2P yet - do a framework, but don't standardize the mechanism yet. Lars says there's a TSV committee looking at new congestion control mechanisms, but the participants are all from the long-delay community. Current BitTorrent cache discovery protocol is DNS-based, with an SRV record based on the reverse DNS lookup of your IP address. Dave Oran asked for input from DSL and Metro-Ethernet network operators, because he sees the network characteristics being radically different, and the problem he sees on deployed networks is under-provisioned aggregation network links. 3.4 Lightning Talks & General Discussion 3.4.1 Robb Topolski This talk covers about a third of Robb's submitted paper. Robb believes that the common accusations about P2P file-sharing aren't actually true - for example, it's very rare for any applications to rate-limit, so almost all applications will use all available bandwidth. We need to know a lot more about the problems we're facing (including "where congestion actually is") in order to solve problems, instead of simply dealing with effects. 3.4.2 Nick Weaver The problem with P2P filesharing is that it shifts cost, doesn't save costs. It increases aggregate costs, and it's transport-inefficient. Could easily build a BitTorrent cache today - all the components are in place. But it's less efficient than HTTP, and it's lawsuit-bait for pirated content. 3.4.3 Leslie Daigle "The open, decentralized nature must be preserved." The problems aren't about P2P technology itself. There's an assumption that network operators won't/can't adjust the network to meet demand (as P2P supernodes move). What are we trying to do: * Make P2P networks work better? * Make applications and networks more aware of each other? * Solve the general-case of bandwidth-intensive applications? * Deal with in-network issues or impact of cross-network traffic? 3.5 Localization and Caches 3.5.1 Laird Popkin and Haiyong Xie - P4P Average P2P bit travels 5-7 metro hops in Verizon networks - network-oblivious P2P applications may not be network-efficient. Routing information is hidden, TCP rate control is coarse-grained. P4P goal is to design a framework that enables better cooperation between providers and applications. P4P tracker provides static topologies, policies, provider capabilities, virtual cost... Virtual costs can be distributed based on provider preferences, but they are talking to providers about 5-minute refresh windows (for scaling reasons). Costs are provided by the providers (not currently tied to routing, but could be if someone wants to provide a feed from routing). Trials at Verizon and Telefonica showed significant increases in intra-ISP traffic, including increase in "local" traffic based on IP prefixes. This was a large swarm (3500 peers at the peak, 300 after a few days). Possible IETF work - "trackerless P2P" (DNS), lookup mechanisms to find AS-PIDs and trackers for a given ASN. Jon asked for information about what IETF protocols are being modified. Henning said it would be nice to have everyone work together, but IETF is slow... P4P Forum doesn't think they are doing a standard, they think they are validating a concept. 3.5.2 Yu-Shun Wang - P2PI Traffic Localization - Microsoft Lots of reasons to localize - apply QoS policy, throttling, admission control - but localization makes the user experience "non-uniform". Using multi-layer trackers (global, local, anything in between). Want to be "local" - but may have to accommodate traffic conditions, load balancing, policy, etc. Need the right metrics - the policy might be NOT to localize. Most of focus is on topology matching between link layer and application layer. Caches are easy (long-lived peers) but caches have to be in the right place in order to gain benefits. Looking at tracker redirection/delegation. Prefer off-path for reliability and avoiding performance bottlenecks. Eric Burger asked what incentive people have to deploy a mechanism like this. Jon liked this paper/presentation because it seemed easy to chop into engineering-sized parts. Bob raised a concern about ISP liability if they are helping pirates choose optimal paths for particular swarms, but if you provide information for all users in all swarms, this isn't scalable. 3.5.3 Vinay Aggrawa - ISP-aided Neighbor Selection - DT Vinay is talking about something that's similar to P4P - having a provider supply an oracle so that any peer can find its "local neighbors". Henning is concerned about oracles that dump traffic off-net, and beggar their neighbors. Dave Oran thought the oracle would also know who was talking to who now - and this approach doesn't have that information yet. People are thinking that 9000 nodes is a big swarm - for legal content with advertising, it probably isn't. We need to think about much larger swarms. Could just run oracle as web server/UDP server at well-known location (similar to BIND). Open and simple solution, open for all overlays. Oracle service doesn't cache content at ISP, doesn't participate in file sharing. Are ISPs willing to accept approaches like oracles that rely on interaction with clients? They've resisted this in the past. 3.5.4 Cullen - IETF work on localization? (Cullen had a slide of possible work from listening to presentations and questions/answers) Leslie - still need to decide what problem we're trying to solve... EKR - people talked about making clients less aggressive, moving data into caches so it doesn't consume uplink capacity, and choosing better peers. All of Cullen's list was about localization. Doesn't seem that localization would be important to Comcast, and I'm terrified of the legal issues involved in caching. Would be useful for nodes to find their own locations in a topology - find own AS numbers, etc. Henning - providing information that customers can't verify, can lead to undesirable situations (like "hot-potato" in routing). Standardize a protocol to delegate from global trackers to local trackers? Lisa says lots of HTTP servers have their own mirror mapping information distribution mechanisms (proprietary). It's possible this would be reusable, but possible that the two problem spaces are just too different. Lisa also had concerns about layer-nine concerns with inter-ISP localization issues. Why not look at the entire problem, and not just parts by parts? We did IM as an application, and Google is starting to use the standardized versions. Jon thinks that IM came into the IETF in just the way he's thinking of for P2P - what are the parts we need? Lars - we have transport technologies that aren't seeing deployment. Maybe incentives have changed, but something would have to change in order for new transport stuff to be deployed. Localized changes, at the point that benefits from the change, do get deployed. Lisa is concerned that localization is one of the technologies that can work in practice, in a limited environment, but can't be standardized - because of issues about trust, accuracy, etc... 3.6 New Approaches to Congestion 3.6.1 Bob Briscoe Bob is presenting Diffserv weighted scheduling versus weighted congestion points. Diffserv is typically used without an API, so the application itself does nothing about Diffserv. Diffserv isn't designed to work at the granularity of individual users. When a light user sends to a heavy user, the light user is using up Diffserv limits in his own network, only for the traffic to be marked down in the heavy user's network. Bob is concerned that we're going to end up in the same place as ATM - with too many traffic classes. Bob proposes a "congestion harm (cost) metric". Is this anything like Comcast's proposal earlier in the workshop? Important to take what ISPs are doing now into account as we plan years into the future. ECN is helpful but ingress policer can't see congestion on the rest of the path. Bob is asking the IETF/IRTF to set up a longer-term design team for congestion. 3.6.2 Marcin Matusze This presentation is based on conversations with about 20 ISPs. ISP customer acquisition costs are much higher than monthly revenue, and cost structure varies (fixed versus mobile). It's difficult to change contracts, and 5 percent of users generate 75 percent of the traffic. Mobile network operators have more tools (volume-based accounting, fixed pricing up to a limit). Most tools aren't widely used. BT is starting to use deep packet inspection, and Bob says that's 10 times more cost-effective than adding bandwidth. Finnish ISPs were very concerned about P2P traffic in 2004, but much less concerned today. 3.7 Quality of Service 3.7.1 Mary Barnes - QoS-enabled traffic prioritization Many service providers are implementing deep packet inspection, but this isn't viable long-term. Diffserv can be implemented today but doesn't solve all the problems, and provisioning/managing still has to look at the events that cause peak usage. Why hasn't Diffserv been deployed in the past? Maybe the problem is big enough now? Bob says data is showing that the amount of P2P traffic in the world is decreasing... People are using QoS - just not on the Internet. More than 50 percent of Cisco's enterprise users have turned Diffserv on - actually higher than that, for enterprise VoIP or video users. There's a real problem with DSLAMs with very limited processing capabilities - Dave Oran made the point that the problems in layer-two aggregation networks are severe and overlooked. 3.7.2 Henning Schulzrinne - encouraging bandwidth efficiency Average TV consumption in North America for HDTV = 972 GBytes per month - reasonable upper limit. Goal is to minimize OVERALL cost of content delivery. Focusing only on HTTP efficiency misses the point. Transit bandwidth trends changed in 2004 - from a steady decline to very little change since 2004. Cost for sharing content is a step function - moving from the neighborhood to a national distribution requires different infrastructure. FIOS TV architecture is very different from the CATV architecture discussed this morning. We have a general discovery problem (network topology, STUN, HELD, LoST, SIP local network configuration...). We have a variety of solutions (DHCP, DNS, anycast...) If you're volume-based, you need application-visible charging indication to allow people to make rational decisions. Can the IETF help here? Misbehaving applications expose user to risk in a charging situation. 3.8 Conclusions & Wrap Generalized "talking to your local infrastructure" mechanism keeps coming up, over and over. Indication of charging would be machine-parsed for any "rational actor" - person or machine. Eric is concerned that we're identifying so many topics that are application-specific and hard-to-too-hard to generalize into a transport mechanism. Jon thinks P4P "virtual costs" makes sense as something we can extract out. Do we need more than "this will be expensive/this will be cheap"? "I'd be happy to tag low-priority traffic if that made any difference to me" - if ISPs are putting bandwidth caps in place and scavenger-class traffic didn't count, that would make a difference. Congestion control isn't going to fix the cost issue - if we aren't going to grow a network to accommodate increased traffic, we're just deciding what order the packets queue in. The IETF needs to decide what's about economics and what's about technology. Can we stop whining about P2P using too many TCP connections? Is it possible to please all the parties? What if benefits to users and benefits to ISPs are just incompatible? We need to be very cautious about matching up network protocols and economic models - this usually ends badly (micropayments, etc). Focus on technical topics (congestion, etc). Congestion arises when there's a scarcity, and you allocate resources based on cost (which may be money, or might be pain). Congestion isn't different from cost - ISPs know that. Just because cost proposals haven't worked in the past doesn't mean there's no way to address cost. Greg - worth IETF attention, momentum around caching, momentum around optimizing peer selection. Good to think about whether we bring all of the work in at once. On deep packet inspection, as a vendor, unless you want to trust host markings, first-hop routers have to inspect packets and that's not cheap. Cullen - Microsoft API will default-mark voice traffic above router traffic in priority. Unless we get our act together we're headed for a train wreck. Danny - IETF might want to consider guidelines on application interaction with transport issues (multiple ports, etc). Mobile networks have all kinds of differential pricing (nights and weekends), but other providers have no hooks into charging for things like distance (even though it might cost the provider more to retrieve a file from Japan). Lars - a lot of the things we're talking about aren't just P2P - IPTV (for example) might have many of the same issues. What are the other heavy bandwidth usages of the network? Bob - congestion isn't cost until it becomes part of the contract that you have with a provider - and providers can't see congestion outside the provider's own network. Eric - in economics, pricing is a protocol and tells you when you should spend money. We need to remember that P2P is about content, too - networks that were engineered for small-request/large-response applications are now carrying servers from grandmothers. Stanislov - opening multiple TCP connections isn't the problem - delay is the same for P2P and Facebook. (This got responded to: this is only the case when you're the only user on the link - but other applications open multiple links, too. Quoting Dave Oran - "there are non-evil reasons to open multiple TCP connections and you can't tell them apart").
_______________________________________________ p2pi mailing list p2pi@ietf.org https://www.ietf.org/mailman/listinfo/p2pi
- [p2pi] My notes from today's workshop Spencer Dawkins
- Re: [p2pi] My notes from today's workshop Livingood, Jason
- Re: [p2pi] My notes from today's workshop Spencer Dawkins
- Re: [p2pi] My notes from today's workshop Paul Jessop