Subject: Re: [re-ECN] Call for Comments: draft-livingood-woundy-congestion-mgmt-03

From: Matt Mathis <[email-masked]>
Date: Mon, 1 Mar 2010 15:01:17 -0500

Interesting draft.   What are your plans for it?

What do you gain by monitoring the CMTS total load?  If you are not
near congestion I would not expect BE vs PBE marking to make any
difference to the traffic.

I did not notice anything in the draft, other than the terminology,
that seemed to be DOCSIS or cable specific.   The algorithm seems as
though it is completely general, and could recast for any technology.
 Am I correct, or did I overlook something?

Thanks,
--MM--
-------------------------------------------
Matt Mathis      http://staff.psc.edu/mathis
Work:412.268.3319   Home/Cell:412.654.7529
-------------------------------------------
Evil is defined by mortals who think they know
"The Truth" and use force to apply it to others.

On Fri, Feb 12, 2010 at 10:38 AM, Jason Livingood
<[email-masked]> wrote:
> This is a call for comments on an individual draft regarding a
> protocol-agnostic congestion management system, which is related to some of
> the reason that Conex is starting up.  I and my co-authors would like
> feedback on the latest revision of this draft at
> http://tools.ietf.org/html/draft-livingood-woundy-congestion-mgmt-03.
>
> If you have any comments, please send them to me privately for
> consideration.
>
> Thank you,
> Jason
>
> _______________________________________________
> re-ECN mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/re-ecn
>
>

---

From: Jason Livingood <[email-masked]>
Date: Wed, 03 Mar 2010 19:23:29 -0500

Thanks for your response Matt (and for reading the draft of course).  Some
replies inline below.

> Interesting draft.   What are your plans for it?

Heck if I know! ;-)  Just kidding.  We have the system in production for a
little over a year and a quarter now, so soliciting comments on the overall
design continues to be interesting to us and helps improve it.  I'd also
like to see it become an informational RFC at some point.  First off, that
provides a stable and easily consumable reference point for the document and
it can be a reference in related work at the IETF.
 
> What do you gain by monitoring the CMTS total load?

We actually monitor individual upstream and downstream ports on the CMTS
(I'll have to re-read the relevant sections to ensure I'm communicating that
the best I can).  That enables us to detect when a particular segment of the
network is approaching a time of possible congestion.  So we use it as an
early warning system of sorts, at which point we proactively mark some
traffic with a lower best effort QoS so that if congestion does actually
occur we are prepared in advance (as compared to doing such analysis &
action at that time precise of congestion, which is hard to get right).

> If you are not
 
Correct.  It is only at times of actual congestion that BE vs PBE has any
effect whatsoever.
 
>  Am I correct, or did I overlook something?

I think that is a fair conclusion to reach.  We've gotten a few questions
from parties involved in operating or supplying operators of different
network types, and as we've explained the system it is apparent to us that
it is a generalized system (it was not really apparent at first, but we were
thinking only of how to solve our problem initially).  In the next version
of the I-D I will probably add a section describing the applicability of the
system to other access network technologies and indeed to other network
segments other than just the last mile portion.

---

From: Hannes Tschofenig <[email-masked]>
Date: Thu, 04 Mar 2010 11:08:57 +0200

I was a bit surprised to hear that the Comcast approach of dealing with 
congestion is not known to all folks in this group (given that it is 
highly relevant for the work in this group).
This makes me wonder what is known about the currently deployed 
solutions in general.

Unfortunately, many of the solutions being deployed are not published in 
such as nice writeup as the Comcast solution is (if they are public at all).

Maybe some operators with interest in this problem space are willing to 
discuss their approach.

Ciao
Hannes

PS: Jason, if you indeed extend your solution to other access 
technologies then I am wondering whether this shouldn't be a work item 
in a future group working. I believe that it has a lot of properties 
that many operators would be interested to look at. Given the current 
state of deployment in this area I believe that it could for many years 
be probably the most useful output a future group.

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 04 Mar 2010 23:52:00 +0000

Jason,

I had read the draft before, and it will be a useful ref. I would 
advise that you publish soon before things move on, rather than 
trying to add stuff on applicability to non-cable networks (even tho 
that would be interesting - perhaps best as a separate short draft).

I have some hopefully constructive comments below from my reading of 
the latest -03 rev.

1/ Average utilisation
It would be useful for context to state what proportion of their 
provisioned access capacity your customers utilise on average (The 
summary in S.4 would be a good place). If you gave the typical 
figures your customers demand it would allow readers to better 
understand why it doesn't make sense to provision as if everyone uses 
their max bit-rate all the time.

It will also explain why a slight change in the behaviour of many 
customers all at once might easily congest provisioned capacity.

And it would also give context to the stats in S.5.2, where customers 
using 70% of their provisioned capacity are considered excessive when 
aggregate utilisation is about 70%.

2/ Why not increase capacity instead?
Many readers will ask this question. I believe the answer is 
straightforward. It would be nice to be able to point to your 
document for that answer.

We tried to give the answer in a draft we have allowed to expire (but 
we could rejuvenate it if there's demand): See S.3.1 of 
<draft-briscoe-tsvwg-relax-fairness-01.txt> available at: 
<http://www.bobbriscoe.net/pubs.html#relax-fairness>

3/ Port utilisation thresholds & durations
It should be pointed out that inferring congestion from utilisation 
is only a heuristic, so the recommended figures are only applicable 
given current traffic patterns.

For instance, port utilisation thresholds can increase as aggregate 
bit-rates increase over the years (more aggregation leads to more 
smoothing). Conversely, if the traffic mix became more bursty due to 
different popular applications, there would be more congestion at 
lower utilisation.

Also the 15-minute duration was presumably found to be useful because 
shorter episodes of high utilisation tend not to always become 
persistent. It would be worth saying what criteria you used to make 
these judgements, as I suspect figures like this will get quoted as 
gospel by people who don't consider what objective was trying to be achieved.

4/ What is "actually congested"?
At the end of S.5.1 it says, "the instances where the network 
actually becomes congested during the Port Utilization Duration are 
few," How is "actually congested" defined? Is just one loss 
considered "actual congestion"? Presumably not. This is important, as 
many consider mild levels of loss as perfectly healthy, so it's not 
easy to know the situation that Comcast is trying to avoid with this 
congestion mangement scheme. This is the objective of the whole 
congestion management scheme, so it's important to be precise here.

In S.5.3 there's the sentence "in those rare cases where packets may 
be dropped..." But a single TCP flow with RTT 80ms can attain 50Mb/s 
with a loss fraction of 0.0013% (1 in ~74000 packets) so there's no 
need to try to achieve loss figures much lower than this. And indeed, 
if flows aren't bottlenecked elsewhere, TCP will drive the system 
until it gets such loss levels. If, instead, a customer is 
downloading 5x 10Mb/s TCP flows still with 80ms RTT, TCP will drive 
losses up to 1 in ~3000 or 0.03% and any lower loss rates won't be 
able to improve performance.

5/ Straying into "justificatory advertising"?
Generally the doc keeps to technical facts, which is good. In some 
places I detected a little bit of business justification, which was 
OK, because it was mild. However, the para justifying User 
Consumption Thresholds based on usage of VoIP, web surfing, Hulu etc 
etc struck me as too much like justifying Comcast's choices - it felt 
like the doc was speaking to the FCC at this point.

This may well be how Comcast judges where to pitch these thresholds. 
But it doesn't sit well in a doc that says that this technique is 
app-neutral. The same info could be given but in a different way. For 
instance, you could say business success for companies like Comcast 
depends on judging what the maximum demand will be from the large 
majority of households for different applications like VoIP, web 
surfing, Hulu. So Comcast pitch capacity at a level that allows x, y 
& z simultaneously. But of course it is up to each household what mix 
of apps they actually use. Yada yada.

6/ Why hysteresis? Can it fail?
In the para starting "A user's traffic is released from a BE 
state..." in S.5.2, firstly, there's a nit in that 'hysteresis' is 
given as an alternative word for 'QoS oscillation', when hysteresis 
is actually the hi-lo watermark technique you use to prevent QoS oscillation.

Moving to my point: it's not clear why hysteresis is considered 
better than QoS oscillation. You say the approach worked well, which 
presumably means it prevented QoS oscillation. But where's the 
evidence that QoS oscillation is not actually preferred by the 
affected customers?

To regain the right to priority BE, a customer has to drop 
consumption to <50% for ~15mins. Without the low watermark, they 
could regain PBE without having to reduce consumption so much. 
Wouldn't they prefer this? Or are you saying the watermark approach 
is beneficial to all the other customers, rather than to the heavy customers?

It's also not clear to me how the customer's software knows how to 
get out of jail. Does forcing them into BE always force their 
consumption to drop below ~50% for 15mins? So they will automatically 
get out of jail? Do they have to be using a TCP-like transport that 
responds to the drops from BE by taking them below ~50%? If their 
software doesn't automatically take them below 50%, can't they get 
trapped in jail for a long time without knowing the passkey to get 
out (ie. reduce consumption <50%)?

7/ What if no users stand out?
Wrt Fig 2: "What if everyone persistently arrives at Result #2 'No 
action taken'"? This could happen if the distribution of customers 
doesn't include any seriously heavy customers, just a lot of equally 
heavy customers. I guess you consider upgrading capacity at this juncture?

8/ Bytes transferred during congestion not necessarily the problem.
It should be stated somewhere that different transports can respond 
more or less aggressively to others sharing the same capacity, while 
still transferring as many bytes (e.g. LEDBAT-like transports). This 
is one limitation of the approach described: it doesn't reward 
yielding behaviour.

Elsewhere Rich Woundy has compiled a wider list of limitations of 
Comcast's approach. It might be worth stating these, either in this 
doc, or in a companion doc along with the generalisations to other 
technologies that you were discussing with Matt.

9/ All in all...
A very useful contribution to the IETF's knowledge. Thank you very 
much for all the work involved in having produced this.

Nits:
-----

491                                           Even without channel bonding,
492   multiple channels are usually configured to come out each physical
493   port.
s/out each/out of each/

Fig 1: Double vision nightmare!

631   this is usually done through a command sent from the network and
632   without any effect on the subscriber.
s/without any effect on the subscriber./without the subscriber noticing./
[If there's no effect at all, why do it!?]

742                        (Note: Although each cable modem is typically
743   assigned to a particular household, the software does not and cannot
744   actually identify individual users, the number of users sharing a
745   cable modem, or analyze particular users' traffic.)  For purposes of
746   this document, we use "cable modem", "user", and "subscriber"
747   interchangeably to mean a subscriber account or user account and not
748   an individual person.
This note would be better in the terminology section (S.2).

Question #3 under Fig 2: Add: "over a period of 15 minutes."

S.5.3 heading:
s/Users&apos/Users'/

942                                          (It is important to note that
943   congestion can occur in any IP network, and, when it does, packets
944   can be delayed or dropped.  As a result, applications and protocols
945   have been designed to deal with this reality.  Our congestion
946   management system attempts to ensure that, in those rare cases where
947   packets may be dropped, BE packets are dropped before PBE packets are
956   dropped.)
This parenthesis should be upfront somewhere at the start of the doc, 
and not in parentheses.

1019   find a specific complaint that can be traced back to the effected of
1020   this congestion management system.
s/effected/effect/

Cheers

Bob

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: "Woundy, Richard" <[email-masked]>
Date: Thu, 18 Mar 2010 02:31:04 -0400

technologies then I am wondering whether this shouldn't be a work item
in a future group working.

I don't oppose having the IETF work on the Comcast solution.

I do tend to think it is out of scope for Conex, though. As I just said
in another email: "FairShare is a network policing mechanism. Conex is a
network measurement and packet marking mechanism. They can place nicely
together (I believe) but they are still independent mechanisms."

-- Rich

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Hannes Tschofenig
Sent: Thursday, March 04, 2010 4:09 AM
To: Livingood, Jason
Cc: Matt Mathis; [email-masked]
Subject: [re-ECN] Capturing Current Deployments ... was Re: Call for
Comments: draft-livingood-woundy-congestion-mgmt-03

such as nice writeup as the Comcast solution is (if they are public at
all).

Ciao
Hannes