Re: [p2pi] My notes from today's workshop

"Livingood, Jason" <Jason_Livingood@cable.comcast.com> Thu, 29 May 2008 02:05 UTC

Return-Path: <p2pi-bounces@ietf.org>
X-Original-To: p2pi-archive@ietf.org
Delivered-To: ietfarch-p2pi-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E4FE13A6CE6; Wed, 28 May 2008 19:05:16 -0700 (PDT)
X-Original-To: p2pi@core3.amsl.com
Delivered-To: p2pi@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 4A1A128C1AA for <p2pi@core3.amsl.com>; Wed, 28 May 2008 19:05:10 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.214
X-Spam-Level:
X-Spam-Status: No, score=-0.214 tagged_above=-999 required=5 tests=[AWL=0.248, BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368, HTML_MESSAGE=0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2f75HQCZVXqH for <p2pi@core3.amsl.com>; Wed, 28 May 2008 19:04:42 -0700 (PDT)
Received: from paoakoavas10.cable.comcast.com (paoakoavas10.cable.comcast.com [208.17.35.59]) by core3.amsl.com (Postfix) with ESMTP id C04013A6CE0 for <p2pi@ietf.org>; Wed, 28 May 2008 19:04:35 -0700 (PDT)
Received: from ([10.52.116.31]) by paoakoavas10.cable.comcast.com with ESMTP id KP-TDCH7.46235345; Wed, 28 May 2008 22:04:16 -0400
Received: from PACDCEXCMB04.cable.comcast.com ([24.40.15.86]) by PAOAKEXCSMTP02.cable.comcast.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 28 May 2008 22:04:16 -0400
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Date: Wed, 28 May 2008 22:04:12 -0400
Message-ID: <45AEC6EF95942140888406588E1A6602045CB9B9@PACDCEXCMB04.cable.comcast.com>
In-Reply-To: <033801c8c104$9471e020$6401a8c0@china.huawei.com>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: [p2pi] My notes from today's workshop
Thread-Index: AcjBBLJvUaNqje9kTX+yIsp5L3kj8gAKr/mA
References: <033801c8c104$9471e020$6401a8c0@china.huawei.com>
From: "Livingood, Jason" <Jason_Livingood@cable.comcast.com>
To: Spencer Dawkins <spencer@wonderhamster.org>, p2pi@ietf.org
X-OriginalArrivalTime: 29 May 2008 02:04:16.0451 (UTC) FILETIME=[4C67A130:01C8C130]
Subject: Re: [p2pi] My notes from today's workshop
X-BeenThere: p2pi@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: P2P Infrastructure Discussion <p2pi.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/p2pi>, <mailto:p2pi-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/p2pi>
List-Post: <mailto:p2pi@ietf.org>
List-Help: <mailto:p2pi-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/p2pi>, <mailto:p2pi-request@ietf.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1612020863=="
Sender: p2pi-bounces@ietf.org
Errors-To: p2pi-bounces@ietf.org

Hi Spencer -
 
I think it is important to correct that I think you misrecorded what we
said regarding Vonage and other R/T traffic.  We in fact were NOT
managing Vonage or other OTT and R/T traffic in that timeframe, as we
were not performing the TCP RSTs at that time.  I believe we may have
said that some people complaining thought we were in some way degrading
Vonage traffic, which we were not.  We then noted that after
implementation of the NM technique currently in place that the
complaints regarding poor Vonage and other R/T traffic began to decline,
thus there appeared to be a rough correlation.
 
Thanks 
Jason
 


________________________________

	From: p2pi-bounces@ietf.org [mailto:p2pi-bounces@ietf.org] On
Behalf Of Spencer Dawkins
	Sent: Wednesday, May 28, 2008 4:51 PM
	To: p2pi@ietf.org
	Subject: [p2pi] My notes from today's workshop
	
	
	Here's what I captured...

	Spencer

	

	3.1           Welcome/Note Well/Intro Slides


	Goals:

	*         To think about the Comcast/BitTorrent problem.

	*         To think about what the IETF might do to work on the
problem 

	*         Focus on what's actionable for the IETF, engineering,
not research or politics.

	"Not dealing with illegal traffic" - Cullen asked us to focus on
the problems that we still have when all the traffic is legal.


	3.2           Service Provider Perspective (Comcast)


	Presenters are Jason Livingood and Richard Woundy from Comcast
(including attributions because they're presenting a sponsor's
perspective0.

	View that this is a two-way street - applications becoming more
network-friendly, networks becoming more application-friendly. 

	Comcast has committed to implement a protocol-agnostic network
management technique (by yearend).

	Comcast was managing Vonage customer traffic, and customers were
complaining. Not all the complaints reflected this management (Comcast
and Vonage both experienced problems with transit providers), but that
wasn't obvious to customers.

	Most ISPs started managing P2P traffic about 2-3 years ago.
When Comcast started managing its traffic, they stopped getting
complaints from Vonage and from P2P users. The recent concerns have
involved a suspicion that Comcast was degrading Vonage VoIP service in
an attempt to favor Comcast's own VoIP products.

	Current DOCSIS can experience congestion independently on
upstream and downstream and experience congestion independently across
DOCSIS domains.

	We talked about some DOCSIS details, but remember that we have
to solve problems for a broad class of access networks, not just for
DOCSIS.

	Slide point: "How 'TCP flow fairness' applies to the DOCSIS
upstream is an open research question".

	DOCSIS also has constant-bandwidth "service flows" (so you don't
need to request-to-send every packet). A Canadian cable company charges
extra to set up a second service flow for Vonage, but other cable
companies haven't followed this lead because of concerns about "net
neutrality". 

	We did note that if you have enough bandwidth, you don't need to
prioritize traffic, but this doesn't help when traffic levels increase
suddenly ("network capacity increases are not instantaneous"), and
capacity increases can be consumed quickly.

	Not all customers are similar (even at the DOCSIS domain level)
- college populations use more P2P, VoIP/streaming video are more
diurnal than bulk file distribution.

	Comcast would like to avoid protocol-specific "cat and mouse"
detection and mitigation, and would like to avoid capacity increases
that benefit only 5 percent of heavy users.

	DOCSIS 3.0 adds capacity but doesn't solve the capacity problem.
Comcast will deploy DOCSIS 3.0 in 20 percent of network by yearend. This
more than doubles bandwidth available per subscriber, but the conflict
is between adding capacity and supporting applications that are designed
to maximize bulk bandwidth consumption.

	There are Internet2 techniques for client-to-network signaling
that haven't been standardized in IETF yet...

	Caching content in the network can reduce TCP RTTs and reduce
the peak-to-trough differences of individual flows.

	Network management trials begin in June - management based on
traffic characteristics, not on specific protocols. Algorithm - mark all
flows as priority and then mark specific flows as "best-effort" when
congestion approaches "the Near Congestion State".

	Comcast hasn't figured out how to notify users that they have
been marked as best-effort yet.

	BT is planning a similar mechanism on their DSL network, but
they are planning on varying the bandwidth, not changing QoS markings.
Comcast thinks customers are more concerned about changing markings and
don't want to penalize high-bandwidth users when the network is NOT
congested ("it's a high-bandwidth service, people should use it").

	Henning is concerned about mechanisms that have 8 independent
tweakable parameters...

	Comcast does still care about protocol-specific management
mechanisms, when the carrier is providing the application, or when the
traffic is illegal, etc.


	3.3           Application Designer Perspective (BitTorrent)


	Presenter is Stanislov Shalunov.

	BitTorrent is willing to make changes based on outcomes at this
conference.

	BitTorrent sees two problems - RTT measured in seconds,
inefficient overlay routing.

	Peer selection is random, except that there is some bias towards
established peers. Trackers randomize the peer list, and peers ask the
resulting randomized list what peers they know about. Rarest-first
ensures best reliability and helps peers have pieces of files to
"trade". This also gives peer diversity.

	The best peers can be farther away - a peer with a big pipe, a
long way away, can drive you at your downlink speed, but a peer on your
own cable modem subnet can only drive you at its uplink speed.

	Danny's comment - restated for minutes: "If you have more
localized distribution via caching, or location techniques, or whatever,
then burst rates (i.e., peak <> trough ratios) may actually be larger
and you may increase HFC contention (note: HFC in particular, v. DSL,
etc..)"

	Overlay efficiency - Stanislov  is experimenting with trying
peers on same AS, or on a set of ASes (but then you need the set), or in
a list of IP prefixes (but then you need the list), or based on some
cost minimization model (but then you need costs).

	If you have more than 20 peers in an AS, you can probably trade
files happily, and 57 percent of transit traffic could be reduced with
this modification. If there aren't enough local peers, make one a cache
(and please choose a fast one), or 10, or 100.

	Stanislov's experience is that they can do 30-percent hit rate
with 1 TByte cache, 80-percent with 100 TBytes - and 1 TByte fits into a
device.

	Dave Oran asked what the required bandwidth for an 80-percent
hit rate of 1 TByte is - he is concerned that this architecture may
reveal bandwidth bottlenecks onto and off the cache device itself.

	Stanislov hopes the IETF can work in these areas:

	*         Cache discovery protocol

	*         Network information for smart peer selection

	*         Experimental congestion control

	*         BCP on P2P pain

	Stanislov doesn't think we're ready to do congestion control for
P2P yet - do a framework, but don't standardize the mechanism yet.

	Lars says there's a TSV committee looking at new congestion
control mechanisms, but the participants are all from the long-delay
community.

	Current BitTorrent cache discovery protocol is DNS-based, with
an SRV record based on the reverse DNS lookup of your IP address.

	Dave Oran asked for input from DSL and Metro-Ethernet network
operators, because he sees the network characteristics being radically
different, and the problem he sees on deployed networks is
under-provisioned aggregation network links.


	3.4           Lightning Talks & General Discussion


	3.4.1            Robb Topolski


	This talk covers about a third of Robb's submitted paper.

	Robb believes that the common accusations about P2P file-sharing
aren't actually true - for example, it's very rare for any applications
to rate-limit, so almost all applications will use all available
bandwidth.

	We need to know a lot more about the problems we're facing
(including "where congestion actually is") in order to solve problems,
instead of simply dealing with effects.


	3.4.2            Nick Weaver


	The problem with P2P filesharing is that it shifts cost, doesn't
save costs. It increases aggregate costs, and it's
transport-inefficient.

	Could easily build a BitTorrent cache today - all the components
are in place. But it's less efficient than HTTP, and it's lawsuit-bait
for pirated content. 


	3.4.3            Leslie Daigle


	"The open, decentralized nature must be preserved."

	The problems aren't about P2P technology itself. There's an
assumption that network operators won't/can't adjust the network to meet
demand (as P2P supernodes move).

	What are we trying to do:

	*         Make P2P networks work better?

	*         Make applications and networks more aware of each
other?

	*         Solve the general-case of bandwidth-intensive
applications?

	*         Deal with in-network issues or impact of cross-network
traffic?


	3.5           Localization and Caches


	3.5.1            Laird Popkin and Haiyong Xie - P4P 


	Average P2P bit travels 5-7 metro hops in Verizon networks -
network-oblivious P2P applications may not be network-efficient. Routing
information is hidden, TCP rate control is coarse-grained. 

	P4P goal is to design a framework that enables better
cooperation between providers and applications.

	P4P tracker provides static topologies, policies, provider
capabilities, virtual cost...

	Virtual costs can be distributed based on provider preferences,
but they are talking to providers about 5-minute refresh windows (for
scaling reasons). Costs are provided by the providers (not currently
tied to routing, but could be if someone wants to provide a feed from
routing).

	Trials at Verizon and Telefonica showed significant increases in
intra-ISP traffic, including increase in "local" traffic based on IP
prefixes. This was a large swarm (3500 peers at the peak, 300 after a
few days).

	Possible IETF work - "trackerless P2P" (DNS), lookup mechanisms
to find AS-PIDs and trackers for a given ASN.

	Jon asked for information about what IETF protocols are being
modified.

	Henning said it would be nice to have everyone work together,
but IETF is slow... P4P Forum doesn't think they are doing a standard,
they think they are validating a concept.


	3.5.2            Yu-Shun Wang - P2PI Traffic Localization -
Microsoft


	Lots of reasons to localize - apply QoS policy, throttling,
admission control - but localization makes the user experience
"non-uniform".

	Using multi-layer trackers (global, local, anything in between).

	Want to be "local" - but may have to accommodate traffic
conditions, load balancing, policy, etc. Need the right metrics - the
policy might be NOT to localize.

	Most of focus is on topology matching between link layer and
application layer. Caches are easy (long-lived peers) but caches have to
be in the right place in order to gain benefits.

	Looking at tracker redirection/delegation. Prefer off-path for
reliability and avoiding performance bottlenecks.

	Eric Burger asked what incentive people have to deploy a
mechanism like this.

	Jon liked this paper/presentation because it seemed easy to chop
into engineering-sized parts.

	Bob raised a concern about ISP liability if they are helping
pirates choose optimal paths for particular swarms, but if you provide
information for all users in all swarms, this isn't scalable.


	3.5.3            Vinay Aggrawa - ISP-aided Neighbor Selection -
DT


	Vinay is talking about something that's similar to P4P - having
a provider supply an oracle so that any peer can find its "local
neighbors".

	Henning is concerned about oracles that dump traffic off-net,
and beggar their neighbors.

	Dave Oran thought the oracle would also know who was talking to
who now - and this approach doesn't have that information yet.

	People are thinking that 9000 nodes is a big swarm - for legal
content with advertising, it probably isn't. We need to think about much
larger swarms.

	Could just run oracle as web server/UDP server at well-known
location (similar to BIND). Open and simple solution, open for all
overlays.

	Oracle service doesn't cache content at ISP, doesn't participate
in file sharing.

	Are ISPs willing to accept approaches like oracles that rely on
interaction with clients? They've resisted this in the past.


	3.5.4            Cullen - IETF work on localization?


	(Cullen had a slide of possible work from listening to
presentations and questions/answers)

	Leslie - still need to decide what problem we're trying to
solve...

	EKR - people talked about making clients less aggressive, moving
data into caches so it doesn't consume uplink capacity, and choosing
better peers. All of Cullen's list was about localization. Doesn't seem
that localization would be important to Comcast, and I'm terrified of
the legal issues involved in caching.

	Would be useful for nodes to find their own locations in a
topology - find own AS numbers, etc.

	Henning - providing information that customers can't verify, can
lead to undesirable situations (like "hot-potato" in routing).

	Standardize a protocol to delegate from global trackers to local
trackers? Lisa says lots of HTTP servers have their own mirror mapping
information distribution mechanisms (proprietary). It's possible this
would be reusable, but possible that the two problem spaces are just too
different.

	Lisa also had concerns about layer-nine concerns with inter-ISP
localization issues.

	Why not look at the entire problem, and not just parts by parts?
We did IM as an application, and Google is starting to use the
standardized versions.  Jon thinks that IM came into the IETF in just
the way he's thinking of for P2P - what are the parts we need?

	Lars - we have transport technologies that aren't seeing
deployment. Maybe incentives have changed, but something would have to
change in order for new transport stuff to be deployed. Localized
changes, at the point that benefits from the change, do get deployed.

	Lisa is concerned that localization is one of the technologies
that can work in practice, in a limited environment, but can't be
standardized - because of issues about trust, accuracy, etc...


	3.6           New Approaches to Congestion


	3.6.1            Bob Briscoe


	Bob is presenting Diffserv weighted scheduling versus weighted
congestion points. 

	Diffserv is typically used without an API, so the application
itself does nothing about Diffserv.

	Diffserv isn't designed to work at the granularity of individual
users.

	When a light user sends to a heavy user, the light user is using
up Diffserv limits in his own network, only for the traffic to be marked
down in the heavy user's network.

	Bob is concerned that we're going to end up in the same place as
ATM - with too many traffic classes.

	Bob proposes a "congestion harm (cost) metric". 

	Is this anything like Comcast's proposal earlier in the
workshop? Important to take what ISPs are doing now into account as we
plan years into the future.

	ECN is helpful but ingress policer can't see congestion on the
rest of the path. 

	Bob is asking the IETF/IRTF to set up a longer-term design team
for congestion.


	3.6.2            Marcin Matusze


	This presentation is based on conversations with about 20 ISPs.

	ISP customer acquisition costs are much higher than monthly
revenue, and cost structure varies (fixed versus mobile). It's difficult
to change contracts, and 5 percent of users generate 75 percent of the
traffic.

	Mobile network operators have more tools (volume-based
accounting, fixed pricing up to a limit). Most tools aren't widely used.

	BT is starting to use deep packet inspection, and Bob says
that's 10 times more cost-effective than adding bandwidth.

	Finnish ISPs were very concerned about P2P traffic in 2004, but
much less concerned today.


	3.7           Quality of Service


	3.7.1            Mary Barnes - QoS-enabled traffic
prioritization


	Many service providers are implementing deep packet inspection,
but this isn't viable long-term.

	Diffserv can be implemented today but doesn't solve all the
problems, and provisioning/managing still has to look at the events that
cause peak usage.

	Why hasn't Diffserv been deployed in the past? Maybe the problem
is big enough now? Bob says data is showing that the amount of P2P
traffic in the world is decreasing...

	People are using QoS - just not on the Internet. More than 50
percent of Cisco's enterprise users have turned Diffserv on - actually
higher than that, for enterprise VoIP or video users. There's a real
problem with DSLAMs with very limited processing capabilities - Dave
Oran made the point that the problems in layer-two aggregation networks
are severe and overlooked.


	3.7.2            Henning Schulzrinne - encouraging bandwidth
efficiency


	Average TV consumption in North America for HDTV = 972 GBytes
per month - reasonable upper limit.

	Goal is to minimize OVERALL cost of content delivery.  Focusing
only on HTTP efficiency misses the point. 

	Transit bandwidth trends changed in 2004 - from a steady decline
to very little change since 2004.

	Cost for sharing content is a step function - moving from the
neighborhood to a national distribution requires different
infrastructure.

	FIOS TV architecture is very different from the CATV
architecture discussed this morning. 

	We have a general discovery problem (network topology, STUN,
HELD, LoST, SIP local network configuration...). We have a variety of
solutions (DHCP, DNS, anycast...)

	If you're volume-based, you need application-visible charging
indication to allow people to make rational decisions. Can the IETF help
here?

	Misbehaving applications expose user to risk in a charging
situation.


	3.8           Conclusions & Wrap


	Generalized "talking to your local infrastructure" mechanism
keeps coming up, over and over.

	Indication of charging would be machine-parsed for any "rational
actor" - person or machine.

	Eric is concerned that we're identifying so many topics that are
application-specific and hard-to-too-hard to generalize into a transport
mechanism. 

	Jon thinks P4P "virtual costs" makes sense as something we can
extract out.

	Do we need more than "this will be expensive/this will be
cheap"?

	"I'd be happy to tag low-priority traffic if that made any
difference to me" - if ISPs are putting bandwidth caps in place and
scavenger-class traffic didn't count, that would make a difference.

	Congestion control isn't going to fix the cost issue - if we
aren't going to grow a network to accommodate increased traffic, we're
just deciding what order the packets queue in. The IETF needs to decide
what's about economics and what's about technology.

	Can we stop whining about P2P using too many TCP connections? 

	Is it possible to please all the parties? What if benefits to
users and benefits to ISPs are just incompatible?

	We need to be very cautious about matching up network protocols
and economic models - this usually ends badly (micropayments, etc).
Focus on technical topics (congestion, etc). 

	Congestion arises when there's a scarcity, and you allocate
resources based on cost (which may be money, or might be pain).
Congestion isn't different from cost - ISPs know that. Just because cost
proposals haven't worked in the past doesn't mean there's no way to
address cost.

	Greg - worth IETF attention, momentum around caching, momentum
around optimizing peer selection. Good to think about whether we bring
all of the work in at once. On deep packet inspection, as a vendor,
unless you want to trust host markings, first-hop routers have to
inspect packets and that's not cheap.

	Cullen - Microsoft API will default-mark voice traffic above
router traffic in priority. Unless we get our act together we're headed
for a train wreck.

	Danny - IETF might want to consider guidelines on application
interaction with transport issues (multiple ports, etc). Mobile networks
have all kinds of differential pricing (nights and weekends), but other
providers have no hooks into charging for things like distance (even
though it might cost the provider more to retrieve a file from Japan).

	Lars - a lot of the things we're talking about aren't just P2P -
IPTV (for example) might have many of the same issues. What are the
other heavy bandwidth usages of the network?

	Bob - congestion isn't cost until it becomes part of the
contract that you have with a provider - and providers can't see
congestion outside the provider's own network.

	Eric - in economics, pricing is a protocol and tells you when
you should spend money. We need to remember that P2P is about content,
too - networks that were engineered for small-request/large-response
applications are now carrying servers from grandmothers.

	Stanislov - opening multiple TCP connections isn't the problem -
delay is the same for P2P and Facebook. (This got responded to: this is
only the case when you're the only user on the link - but other
applications open multiple links, too. Quoting Dave Oran - "there are
non-evil reasons to open multiple TCP connections and you can't tell
them apart").

_______________________________________________
p2pi mailing list
p2pi@ietf.org
https://www.ietf.org/mailman/listinfo/p2pi