Subject: Re: [iccrg] Requesting a slot in the Prague IETF

From: Michael Welzl <[email-masked]>
Date: Mon, 4 Feb 2019 13:21:41 +0100

Hi Jana, all,

Writing as a co-author of this work, not as ex-chair  ;-)   We sent this request some time ago, no response yet - because the early registration deadline is tonight, we'll just assume that you have a slot for us. Please don't let us down!  :)

Thanks in advance!

Cheers,
Michael

> On 9 Jan 2019, at 13:38, Peyman Teymoori <[email-masked]> wrote:
> 
> Dear Jana,
> 
> Michael and I are planning to present our recent work in the upcoming ICCRG meeting if possible. The title is Estimating an Additive Path Cost with Explicit Congestion Notification. It would be great if we could have a 40-minute slot (including a 10-minute discussion)?!
> 
> Kind regards,
> Peyman

---

From: Christian Huitema <[email-masked]>
Date: Mon, 11 Feb 2019 12:06:30 -1000

I don't have a draft, but I have a discussion topic, which is presented
in this blog:
https://huitema.wordpress.com/2019/01/12/will-transport-innovation-collapse-the-internet/.
I goes to the basic tension between innovation and fairness: the new
stuff tends to always work better than the old stuff, which is somewhat
unfair to the users of the old stuff. In the case of innovation in
congestion control, there is a potential for creating new algorithms
that are "tuned" to be just a little more aggressive than the
competition. If everybody is cautious that may be fine, but if everybody
races to beat their competitors, matters could get out of hand, and we
might even get a return of congestion collapse.

I would like some time to discuss the topic at the ICCRG meeting in
Prague, and hear what researchers have to say on the subject. For
example, could some form of AQM allow innovation while maintaining
fairness and stability? Can they in fact be deployed?

-- Christian Huitema

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Tue, 12 Feb 2019 13:57:41 +0100

Hi Christian,

> https://huitema.wordpress.com/2019/01/12/will-transport-innovation-collapse-the-internet/.

Just two quick comments here w.r.t BBR:
a) BBR's behavior (in v1) depends on the size of the bottleneck buffer.
   If the buffer is smaller than a "BDP", it will be quite aggressive,
   because it is ignoring loss as congestion signal. So this was very
   likely the case in Geoff's experiment. It is, BTW, an intrinsic
   behavior of BBRv1's design to have around 2*BDP in flight
   (for details see sec. 3 of
http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf). In a large
bottleneck buffer, BBRv1 may get a lower
   share than other CCs, because of its 2BDP inflight cap.

b) Since BBRv1 ignores loss as congestion signal an AQM will
   probably not be very helpful, except when combined with FQ.
   However, BBRv2 is going to change that behavior, so it may
   work better with AQMs and even ECN.

> might even get a return of congestion collapse.

What you refer to was already mentioned in Section 4.1 of RFC 2914
as "arms race" (https://tools.ietf.org/html/rfc2914#section-4.1).
We all know that there is no protocol police on the Internet.
The network cannot easily defend itself against misbehaving senders.
There were also BIC and CUBIC introduced in Linux without discussion in
the IETF. That raised similar concerns at the time which eventually
led to https://tools.ietf.org/html/rfc5033
I would have concluded that BBRv1 is not safe to be deployed on
the Internet, whereas Google just deployed it on their servers.
This was probably less critical, because the actual use cases
may be application limited most of the time (but that is pure
speculation). It is quite clear that BBR gives _its_ users better
performance, but I'm not sure that the potentially suppressed
users of other congestion controls at the same bottleneck were
also considered.

Interestingly, we have actually the reverse deployment problem
of what you described: TCP LoLa [1] is a delay-based congestion
control that is not able to achieve a meaningful share if loss-based
congestion controls (e.g., Reno or Cubic) are present at the bottleneck.
So coexistence is also a problem for such new CCs that try to limit the
overall queueing delay [2] and are thus _less_ aggressive than what is
deployed out there. Sure, one can easily detect that and build a
fallback mode in, but that sacrifices the low latency goal.

> fairness and stability? Can they in fact be deployed?

So IMHO AQM is useful as some kind of fallback mechanism in case the
congestion control is not behaving as desired. An additional AQM benefit
is the potential explicit congestion feedback. With congestion controls
such as LoLa there is actually no need for an AQM to limit the queue
size (assuming that all bottleneck flows are using LoLa). AQMs also have
an intrinsic natural trade-off between link utilization and enforced
delay bound (see [3]), e.g., if there are only a few flows present
at the bottleneck, AQMs cannot increase the performance any more: this
is only possible by applying better congestion control algorithms then.
But I think that AQMs can probably contribute to the coexistence problem
indeed. I'm not a big fan of built-in fairness mechanisms though,
because they are hardly flexible and thus limit future innovations.

Regards,
 Roland

[1] TCP LoLa: Congestion Control for Low Latencies and High Throughput
Hock, M.; Neumeister, F.; Zitterbart, M.; Bless, R.
2017. 42nd IEEE Conference on Local Computer Networks, LCN 2017;
Singapore; doi:10.1109/LCN.2017.42
http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf

[2] Toward Coexistence of Different Congestion Control Mechanisms
Hock, M.; Bless, R.; Zitterbart, M., IEEE 41st Conference on Local
Computer Networks (LCN), Dubai, United Arab Emirates, 7–10 November
2016, IEEE. doi:10.1109/LCN.2016.94

[3] Policy-oriented AQM Steering, Bless, R.; Hock, M.; Zitterbart, M.
2018. IFIP Networking Conference (IFIP Networking 2018), Zurich, CH, May
14-16, 2018,
https://doc.tm.kit.edu/2018-kit-aqm-steering-authors-copy.pdf

---

From: Michael Welzl <[email-masked]>
Date: Tue, 12 Feb 2019 16:17:16 +0100

Hi,

> users of other congestion controls at the same bottleneck were

That wouldn't be "_its_ users", but "_its_ single application that is being served by a connection". People often use more than one application; not all of them get traffic from BBR servers. What if I listen to Spotify while I'm accessing something with BBR? And what about other people in my household? I don't believe that it would be in the interest of anyone, including the designers of BBR, to give users this kind of bad performance - various applications not working anymore because of BBR (and this is also why I don't believe in an "arms race" problem).
Personally, I also haven't experienced such issues, and I haven't heard anyone complaining about BBR in practice - it seems to be a matter that only concerns us network researchers looking at plots.

So there's something missing in this picture. What is it?  Is traffic bottlenecked elsewhere?

For example: a little over a decade ago, it often wouldn't matter how aggressive a congestion control mechanism is, because everyone was typically by the rwnd anyway. Then gradually, folks started enabling window scaling, and increasing the scaling value.

Could that be the reason - the window scaling value being too small for the problem to become noticeable?
Or is it that capacities are so large now, that flows rarely ever get to saturate the limit anyway because they're either short or application-limited?  Or is it only used for certain applications? I noticed that BBR presentations often talked about YouTube, which certainly wouldn't be as harmful to competing flows as a bulk file transfer.

Cheers,
Michael

---

From: rrs <[email-masked]>
Date: Tue, 12 Feb 2019 11:02:26 -0500

> For example: a little over a decade ago, it often wouldn't matter how aggressive a congestion control mechanism is, because everyone was typically by the rwnd anyway. Then gradually, folks started…

Michael:

Great questions!!

I would guess a lot of the reason BBR is not a problem, is in respect to 
what applications are using it. A streaming application (e.g. you-tube) 
is not going to be too noticeable to other flows since most if its time 
(assuming you have enough b/w for the stream) is spent waiting for 
either the rwnd to open up or for the next request (depending of course 
on how they are doing the streaming). So usually  you would see 
something like a few 100 milliseconds of data transfer, followed by an 
idle period of 2-4 seconds. Where its a web page you are opening, most 
browsers open a numerous number of connections and download bits and 
pieces of the page. Each one of those connections gets an IW=10 (or 
higher) and then they are done. This is of course not always the case, 
but I think it happens quite often. In such a situation the 
aggressive-ness of BBR would be hidden by all the IW's all pummeling the 
network. No matter if its Cubic or BBR its going to look pretty much the 
same (startup in BBR looks a lot like Cubic startup).

Researches, however typically test with something like iperf over some 
long period. Thus we write nice papers that tell us about unfairness we 
see using a model of application that is probably pretty rare on the 
big-I.. oh yeah there are ftp users out there but the vast majority of 
the traffic I bet does not look like iperf/ftp.

Maybe we need to re-think how we measure fairness i.e. we need to be 
looking at it through the eyes of the applications.. this makes it hard 
of course, since there is a myriad of applications out there each with 
their own requirements/expectations...

It is an interesting problem though :)

R

> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg
>

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Tue, 12 Feb 2019 17:04:48 +0100

Hi Michael,

> That wouldn't be "_its_ users", but "_its_ single application that is being served by a connection". People often use more than one application; not all of them get traffic from BBR servers. What i…

Yes, it was also shown that BBRv1 sometimes suppresses other BBR flows.
Especially, multiple BBR flows behave differently than a single BBR
flow. So you are right, using BBR for one flow/application may degrade
the performance of other flows at the bottleneck, even if they are the
same user. The danger is indeed that if a user is just running a single
transfer and turns BBR on, that user will probably see a huge
performance boost. What this user then probably neglects or cannot
detect is what happens to the flows of the other users at the
bottleneck. That was basically my point. If the user is suppressing
other flows from herself, she will probably notice.

> So there's something missing in this picture. What is it?  Is traffic bottlenecked elsewhere?

I think that I tried to answer that already:
Web transfers and Youtube streams are very likely application-limited,
so they mainly benefit from ignoring a loss as congestion signal. The
harm to other competing flows is limited then.

> Or is it that capacities are so large now, that flows rarely ever get to saturate the limit anyway because they're either short or application-limited?  Or is it only used for certain applications?…

I think either short or application-limited, but as I said: the size of
the bottleneck buffer also plays an important role. The suppression
by BBR happens mainly with small buffers at the bottleneck and if
at least two BBR flows are used simultaneously. So I don't know
who will be measuring the effect of using multiple BBR streams
together with other traffic in a six person household if their
Internet access is saturated.

Regards
 Roland

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 12 Feb 2019 23:58:54 +0000

Michael, Roland, all,

It's short-sighted to purely judge the performance of new congestion 
controls by their effect on today's dominant applications. The reason we 
agree on design principles is not just for existing applications, it's 
also so that new, more demanding applications can emerge.

Bob

PS. To be clear, I have argued repeatedly that flow-rate 'fairness' 
should not be a design principle - because it is unsubstantiated dogma. 
However, avoiding starvation of other flows (at any instant) is an 
extremely important design principle. So, as loss levels increase, 
instantaneous flow-rate 'fairness' becomes increasingly important; but 
as loss levels reduce, there should be, and can be, much more room to 
expand beyond the narrow confines of this rule. This is the sort of 
controlled relaxation of the rules that will enable new 
ultra-high-throughput interactive applications to emerge - by not being 
expected to yield half the link rate as soon as any other flow appears.

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Michael Welzl <[email-masked]>
Date: Wed, 13 Feb 2019 01:28:15 +0100

Hi all,

Thanks for your answers - and thanks to Christian for kicking this off! This is getting quite interesting.

I agree with many of the things there were said - I was also guessing the traffic type plays an important role; and yes, Bob has rightly made the point in the past that “flow-rate fairness” is a strange dogma (I was in fact about to point at his work in my next email, but he beat me to it  :)  ).

More importantly, I fully agree with Bob’s point that it would be short-sighted to judge the performance of a congestion control solely on such a narrow basis. Back in the days when everyone focused on making TCP work in large-BDP environments, it was standard practice for authors of new mechanisms to present a comparison against a reasonable number of SoA competitors, along with evaluations covering a wealth of scenarios (certainly going beyond 1 or 2 flows or so). It seems that in recent years, the onus of doing and presenting such evaluation work has shifted, and I don’t think that this is a good development.

But my question wasn’t about congestion control design - I was just curious why we’re not seeing a big problem in operational practice today.

Roland made an interesting point, at the end of his email:
"the size of the bottleneck buffer also plays an important role. The suppression
by BBR happens mainly with small buffers at the bottleneck and if
at least two BBR flows are used simultaneously. So I don't know
who will be measuring the effect of using multiple BBR streams
together with other traffic in a six person household if their
Internet access is saturated.”

… so we’d need such a situation - a small enough bottleneck capacity and buffer, multiple BBR flows competing with others, AND the BBR flows would need to be bulk transfers, to see the problem?

Is that really it?   E.g., I was serious about window scaling, I do wonder if the problem could also be masked by the standard choice of the value there in today’s OSes.

Cheers,
Michael

> On Feb 13, 2019, at 12:58 AM, Bob Briscoe <[email-masked]> wrote:
> 
> Michael, Roland, all,
> 
> It's short-sighted to purely judge the performance of new congestion controls by their effect on today's dominant applications. The reason we agree on design principles is not just for existing applications, it's also so that new, more demanding applications can emerge.
> 
> 
> Bob
> 
> PS. To be clear, I have argued repeatedly that flow-rate 'fairness' should not be a design principle - because it is unsubstantiated dogma. However, avoiding starvation of other flows (at any instant) is an extremely important design principle. So, as loss levels increase, instantaneous flow-rate 'fairness' becomes increasingly important; but as loss levels reduce, there should be, and can be, much more room to expand beyond the narrow confines of this rule. This is the sort of controlled relaxation of the rules that will enable new ultra-high-throughput interactive applications to emerge - by not being expected to yield half the link rate as soon as any other flow appears.
> 
> 
>

---

From: Christian Huitema <[email-masked]>
Date: Tue, 12 Feb 2019 18:46:05 -1000

To be clear, I am not overly concerned about BBR. My real worry is that
the move to process-level transports removes the traditional
"gatekeepers", the code reviews and the discussions that enforce
compliance with standards. It is a lot easier to tweak a transport that
ships in your own application than to get a new congestion control
installed in Windows, IOS or a big distribution of Linux. Google is
somewhat of an exception there, because shipping in Chrome probably
entails the same kind of verification as shipping in the big
distributions. Or maybe a little less, but not much less. On the other
hand, the smaller the app gets, the fewer controls one can expect.
That's where I fear the arms race.

What I would like to see is some kind of traffic management at the
bottleneck that isolate the competing flows. If that isolation was
effective, we could innovate safely. Different applications may use
different algorithms, so they get the trade-off between bandwidth and
latency that best suits their needs. But as Roland Bess pointed out, the
current crop of AQM don't quite give you that, except maybe FQ-Codel.
Even if your application wants low latency, you cannot really deploy
something that TCP Lola  because you will lose the competition for
resource at the bottleneck. So after many iterations you will end up
deploying some "Lola++" hybrid that gets low latency when not competing,
but gets really aggressive when sharing a bottleneck. And we are back to
the arms race.

Hence my question. Can we design and deploy traffic management
mechanisms that provide this kind of flow isolation?

-- Christian Huitema

---

From: Loganaden Velvindron <[email-masked]>
Date: Wed, 13 Feb 2019 10:28:48 +0400

> current crop of AQM don't quite give you that, except maybe FQ-Codel.

and CAKE too I suppose :)

---

From: Mario Hock <[email-masked]>
Date: Wed, 13 Feb 2019 11:35:09 +0100

This is a fair point.

We deliberately decided for LoLa not to have such a hybrid mechanism for 
just the reasons you mentioned. This decision also implied that TCP LoLa 
became mainly a research prototype not meant for immediate deployment in 
the current Internet. Therefore, before we even started to develop LoLa, 
we posed the same question as you did: "What do we need from the network 
to enable less-aggressive congestion controls?"

[1] is a very early work of ours on this subject. Here, we took a look 
into how we could separate different classes of flows in an OpenFlow 
based SDN network. One important point here is that we did not used flow 
isolation. We separated classes of flows (in our case, flows with 
traditional loss based congestion control and flows with low delay 
congestion control). Since LoLa did not exist then, we used TCP Vegas as 
a proof-of-concept. However, this work focuses only on mechanisms in the 
switches not on the (also very important) signaling part.

Another interesting approach we want to look into is how AQM (not 
FQ-schedulers) change the game here. We can envision a combined 
approach: AQM at the bottleneck and a special hybrid mode of LoLa. The 
AQM makes sure no one can excessively fill the queue. The hybrid mode 
makes sure that LoLa flows co-exist nicely with e.g. Cubic flows when an 
AQM is present. Still, we would get high throughput, low delay and a 
reasonable fairness. If only LoLa flows (or the like) are present, the 
AQM is never triggered.

That said, I agree that it is also interesting to pursue FQ-approaches. 
The open issue I see here is to decide on which level fairness should be 
enforced: per flow, per sender, per sender-receiver pair, etc.

Best, Mario (working with Roland on TCP LoLa)

[1] M. Hock, R. Bless and M. Zitterbart, "Toward Coexistence of 
Different Congestion Control Mechanisms," 2016 IEEE 41st Conference on 
Local Computer Networks (LCN), Dubai, 2016, pp. 567-570.
doi: 10.1109/LCN.2016.94

---

From: Mario Hock <[email-masked]>
Date: Wed, 13 Feb 2019 11:45:34 +0100

> … so we’d need such a situation - a small enough bottleneck capacity and buffer, multiple BBR flows competing with others, AND the BBR flows would need to be bulk transfers, to see the problem?

I would say, yes, that's about it.

> Is that really it?   E.g., I was serious about window scaling, I do wonder if the problem could also be masked by the standard choice of the value there in today’s OSes.

For our experiments in 10Gbit/s and 100Gbit/s networks we, indeed, have 
to tune the flow control limits manually. But, for a regular home 
Internet access, let's say 100Mbit/s and 20ms RTT with nearby CDNs, 
flows should not be limited by the flow control. However, this is just 
spoken from our experiences with Linux default configurations (e.g. 
Ubuntu server). I have no insights how a CDN manages its flow control 
limits.

In the context of BBR it was actually mentioned that Google started with 
tuning the flow control window of their Cubic flows to fit the BDP (or 
even stay below it, just to not trigger the congestion control!), before 
they developed BBR.

---

From: Keith Winstein <[email-masked]>
Date: Wed, 13 Feb 2019 04:11:31 -0800

Christian, I don't disagree with these sentiments (either below or in your
blog post), but it seems to me that the battle was lost a long time ago.

Most of the traffic on the Internet comes from a small number of senders
and isn't subject to the OS gatekeepers, much less the IETF. The CDN
operators have long deployed whatever initial window and congestion-control
behavior they fancied. Google's and Microsoft's (and probably Akamai's and
Facebook's and Netflix's and Tencent's and Alibaba's) public-facing servers
have shown little shame in violating the Host Requirements RFCs on IW and
CC behavior since perhaps the 1990s.

Furthermore, flow initiation isn't subject to a congestion-control
mechanism, and there has been an "arm's race" in connections-per-host
between the browser vendors dating back to the 1990s, and between the
authors of sharded websites, and with the rise of applications like
BitTorrent when it ran over TCP. The most perfect per-flow FQ/CC mechanism
can't prevent problems when some applications start hundreds of flows each.
(BitTorrent switched mostly to UDP and user-space congestion control to try
to be *less* aggressive than the OS TCP's default CC scheme.)

And, for that matter, the presence of large in-network buffers means that a
substantial percentage of TCP connections never exit slow-start -- which
renders a lot of the details of a congestion-control scheme irrelevant.

The operators and implementers who send 90% of the datagrams and much of
the standards and research communities have contrary views on fundamental
questions (like, "Is 'TCP-friendly' a well-defined concept?", or "Is it
possible to determine if an algorithm is safe to deploy without just
deploying it over the real Internet and slowly ramping it up?" or "Do we
really understand the congestion collapses of the 1980s?") and often talk
past one other. Or, they don't talk, because the operators deployed X at
world-scale years ago (where X=CUBIC or IW64 or WebRTC congestion control
or BBR or QUIC) and didn't need to wait for standardization or inclusion in
mainline Linux/BSD or a peer-reviewed paper.

Per-user or per-application traffic management at the bottlenecks would be
a great thing, but I'd be surprised if this is a problem where the solution
hinges on more research or standards. Does the community have dramatically
new ideas in this space that haven't basically been covered by "bottleneck
links should have a queue for each flow/each user/each application" and
schemes like Conex or L4S? The problem may be more in persuading the people
who control the bottleneck queues to deploy something different. Despite
having a lot of evidence that it would be a good idea, the bufferbloat
folks have not persuaded the makers of cellular uplinks (e.g. Qualcomm) and
the makers and operators of eNodeBs, or most makers of Wi-Fi access points,
to deploy *smaller* queues, or AQM, or ECN marking. These would still be an
easier task than getting them to implement *more* queues and a way of
separating traffic that discourages arms races (e.g. per-user or
per-application).

To the extent the eNodeB operators recognize there is a problem at all,
it's the kind of problem they like to solve by having the traffic source
give them $100 million to do some joint R&D that ends up deploying a
classifier and differential treatment for one company's traffic. (This is
why I'm not sure about your conclusion that it's the *small* apps we should
be worried about...) So to your last question ("can we design and deploy
traffic management mechanisms"), I suspect for "design" the answer is "yes,
good-enough mechanisms already exist" and to "deploy" the answer seems to
be "apparently not, unless you have a lot of money to bribe the operators."

My honest view is that congestion control is not a natural subject for the
IETF process -- it doesn't really affect interoperability, practice
diverged from the RFCs relatively early in the Internet's history and
hasn't returned, we lack a real understanding and common language about
some of the fundamental concepts and hard parts, and there is an economic
incentive for the traffic sources to cheat a bit (perhaps in increasing the
IW, or perhaps in rolling out BBR, or CUBIC, or more
connections-per-application) and not try *too* hard to discover and be
concerned about how that might be harming their competitors. If there is a
silver lining, maybe it's one of Mutually Assured Destruction -- in that
Google, Netflix, and other major traffic sources do have a substantial
economic inventive in the Internet's continued health.

-Keith

---

From: Szilveszter Nadas <[email-masked]>
Date: Fri, 15 Feb 2019 15:51:27 +0000

Christian, our paper "Towards a Congestion Control-Independent Core-Stateless AQM" targets to be such a solution (some kind of traffic management at the bottleneck that isolate the competing flows). An option is to have the packet marker in the OS, and in that case the OS could act as a watchdog. This would allow much higher freedom in CC design. (The paper and other related papers are at http://ppv.elte.hu/).

CSAQM does not solve all coexistence issues, especially when it comes to congestion indication. My understanding is that many solutions (including BBR and TCP LoLa) started using jitter as congestion indication (instead of ECN CE or packet loss), mainly because proper AQM is not available, and buffers are large. In my mind-map that jitter detection is a replacement of a proper AQM, and assumes a "pretty large buffer". The other trend I am seeing is that buffer memory is becoming a significant part of the equipment cost, so vendors might start to look into solutions to minimize it (in ms), especially as bottleneck speed increases. In summary I am not sure how these delay based CC-s can be integrated into e.g. CSAMQ, but I am also not 100% sure that it is necessary. 

Loganaden, while FQ-Codel and CAKE are quite good for end-hosts, my understanding is that it does not scale well with the number of flows to be isolated. One of the advantages of Core-Stateless AQM is that it also works with very high number of flows.

Keith, I agree that getting fairness to the Internet has issues like this. But the consequence is that we are still at the mercy of the good old TCP-friendly behavior when it comes to resource sharing. Even though IETF cannot mandate Congestion Control behavior, most companies still take into account IETF's view to some extent. 

Szilveszter

------------------------
On Tue, Feb 12, 2019 at 8:46 PM Christian Huitema <mailto:[email-masked]> wrote:

-- Christian Huitema

mailto:[email-masked]

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Fri, 15 Feb 2019 19:15:42 +0100

Szilveszter Nadas <[email-masked]> writes:

> Loganaden, while FQ-Codel and CAKE are quite good for end-hosts, my
> understanding is that it does not scale well with the number of flows
> to be isolated.

Scaling is not that much of an issue, actually. See
https://www.researchgate.net/publication/247045479_On_the_Scalability_of_Fair_Queueing

-Toke

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 15 Feb 2019 12:07:07 -0800

On Wed, Feb 13, 2019 at 4:12 AM Keith Winstein <[email-masked]> wrote:
>
> Christian, I don't disagree with these sentiments (either below or in your blog post), but it seems to me that the battle was lost a long time ago.
>
> Most of the traffic on the Internet comes from a small number of senders and isn't subject to the OS gatekeepers, much less the IETF. The CDN operators have long deployed whatever initial window and congestion-control behavior they fancied. Google's and Microsoft's (and probably Akamai's and Facebook's and Netflix's and Tencent's and Alibaba's) public-facing servers have shown little shame in violating the Host Requirements RFCs on IW and CC behavior since perhaps the 1990s.
>
> Furthermore, flow initiation isn't subject to a congestion-control mechanism, and there has been an "arm's race" in connections-per-host between the browser vendors dating back to the 1990s, and between the authors of sharded websites, and with the rise of applications like BitTorrent when it ran over TCP. The most perfect per-flow FQ/CC mechanism can't prevent problems when some applications start hundreds of flows each. (BitTorrent switched mostly to UDP and user-space congestion control to try to be *less* aggressive than the OS TCP's default CC scheme.)
>
> And, for that matter, the presence of large in-network buffers means that a substantial percentage of TCP connections never exit slow-start -- which renders a lot of the details of a congestion-control scheme irrelevant.
+1

I have an email draft expressing the same. People have acted on bad
congestion behaviors: before HTTP2, the browsers and websites are
creating massive connections (consider a new C.C. with Nx IW & slow
start) and buffer bloat. After HTTP2 the number of connections have
reduced dramatically and the bytes per connection has consequently
increased for TCP per-flow C.C. to do a better job in handling congestion.

And QUIC really changes the C.C. game: we finally are able to change
the both client and server sides quickly, without worrying too much
about middle-boxes messing around.

>
> The operators and implementers who send 90% of the datagrams and much of the standards and research communities have contrary views on fundamental questions (like, "Is 'TCP-friendly' a well-defined concept?", or "Is it possible to determine if an algorithm is safe to deploy without just deploying it over the real Internet and slowly ramping it up?" or "Do we really understand the congestion collapses of the 1980s?") and often talk past one other. Or, they don't talk, because the operators deployed X at world-scale years ago (where X=CUBIC or IW64 or WebRTC congestion control or BBR or QUIC) and didn't need to wait for standardization or inclusion in mainline Linux/BSD or a peer-reviewed paper.
>
> Per-user or per-application traffic management at the bottlenecks would be a great thing, but I'd be surprised if this is a problem where the solution hinges on more research or standards. Does the community have dramatically new ideas in this space that haven't basically been covered by "bottleneck links should have a queue for each flow/each user/each application" and schemes like Conex or L4S? The problem may be more in persuading the people who control the bottleneck queues to deploy something different. Despite having a lot of evidence that it would be a good idea, the bufferbloat folks have not persuaded the makers of cellular uplinks (e.g. Qualcomm) and the makers and operators of eNodeBs, or most makers of Wi-Fi access points, to deploy *smaller* queues, or AQM, or ECN marking. These would still be an easier task than getting them to implement *more* queues and a way of separating traffic that discourages arms races (e.g. per-user or per-application).
>
> To the extent the eNodeB operators recognize there is a problem at all, it's the kind of problem they like to solve by having the traffic source give them $100 million to do some joint R&D that ends up deploying a classifier and differential treatment for one company's traffic. (This is why I'm not sure about your conclusion that it's the *small* apps we should be worried about...) So to your last question ("can we design and deploy traffic management mechanisms"), I suspect for "design" the answer is "yes, good-enough mechanisms already exist" and to "deploy" the answer seems to be "apparently not, unless you have a lot of money to bribe the operators."
>
> My honest view is that congestion control is not a natural subject for the IETF process -- it doesn't really affect interoperability, practice diverged from the RFCs relatively early in the Internet's history and hasn't returned, we lack a real understanding and common language about some of the fundamental concepts and hard parts, and there is an economic incentive for the traffic sources to cheat a bit (perhaps in increasing the IW, or perhaps in rolling out BBR, or CUBIC, or more connections-per-application) and not try *too* hard to discover and be concerned about how that might be harming their competitors. If there is a silver lining, maybe it's one of Mutually Assured Destruction -- in that Google, Netflix, and other major traffic sources do have a substantial economic inventive in the Internet's continued health.
>
> -Keith
>
> On Tue, Feb 12, 2019 at 8:46 PM Christian Huitema <[email-masked]> wrote:
>>

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Mon, 18 Feb 2019 11:36:49 +0100

Hi Szilveszter,

> CSAQM does not solve all coexistence issues, especially when it comes to congestion indication. My understanding is that many solutions (including BBR and TCP LoLa) started using jitter as congesti…

Let me correct some points w.r.t. TCP LoLa.
The motivation behind LoLa is to study new approaches for getting an
end-to-end congestion control that achieves high throughput and fairness
while keeping the overall queuing delay bounded. LoLa does not require
an AQM and also does not require a large buffer. The interesting part
is that LoLa uses the presence of a larger standing queue as congestion
indication (i.e., a small standing queue up to a configurable limit is
ok). However, explicit network feedback that AQMs could provide may
indeed help LoLa to perform better, because it depends mainly on RTT
measurements otherwise. Typically, if the delay bounds of an AQM and
LoLa match well, LoLa flows will ideally never trigger the AQM.
So as I already said: for good performance you also need to work on
the end-to-end CC solution, an AQM alone will not always be able to
achieve the best performance.

Regards
 Roland

---

From: Szilveszter Nadas <[email-masked]>
Date: Wed, 20 Feb 2019 15:25:06 +0000

Getting all the policies to the bottleneck and identifying flow groups/aggregates right is still hard to scale. Getting the same share per 5-tuple is probably not a policy one wants to achieve (i.e. one flow per 5-tuple and equal sharing for those flows).

Szilveszter

-----Original Message-----
From: Toke Høiland-Jørgensen <[email-masked]> 
Sent: Friday, February 15, 2019 19:16
To: Szilveszter Nadas <[email-masked]>; Keith Winstein <[email-masked]>; Christian Huitema <[email-masked]>
Cc: Sándor Laki <[email-masked]>; Bless, Roland (TM) <[email-masked]>; Bob Briscoe <[email-masked]>; Michael Welzl <[email-masked]>; [email-masked]
Subject: Re: [iccrg] Encrypted Message

> understanding is that it does not scale well with the number of flows 

Scaling is not that much of an issue, actually. See https://www.researchgate.net/publication/247045479_On_the_Scalability_of_Fair_Queueing

-Toke

---

From: Szilveszter Nadas <[email-masked]>
Date: Wed, 20 Feb 2019 15:34:25 +0000

Hi Roland,

I do agree with "for good performance you also need to work on the end-to-end CC solution, an AQM alone will not always be able to achieve the best performance.".

A question to solve is to have reasonable fairness among the CCs, old ones and new ones. One way is to have separate queues for "compatible CCs". For TCP-fairness at least it is the AQM making sure that flows have reasonable fairness. In absence of a synchronous congestion signal (at least probabilistically)  what can be the synchronizer of CCs? Core-Stateless AQM can be one of those synchronizers, though it might still be incompatible with some solutions, where delay or delay variation is used as congestion signal.

Regards,
Szilveszter

-----Original Message-----
From: Bless, Roland (TM) <[email-masked]> 
Sent: Monday, February 18, 2019 11:37
To: Szilveszter Nadas <[email-masked]>
Cc: [email-masked]; Sándor Laki <[email-masked]>
Subject: Re: [iccrg] Encrypted Message

Hi Szilveszter,

Let me correct some points w.r.t. TCP LoLa.
The motivation behind LoLa is to study new approaches for getting an end-to-end congestion control that achieves high throughput and fairness while keeping the overall queuing delay bounded. LoLa does not require an AQM and also does not require a large buffer. The interesting part is that LoLa uses the presence of a larger standing queue as congestion indication (i.e., a small standing queue up to a configurable limit is ok). However, explicit network feedback that AQMs could provide may indeed help LoLa to perform better, because it depends mainly on RTT measurements otherwise. Typically, if the delay bounds of an AQM and LoLa match well, LoLa flows will ideally never trigger the AQM.
So as I already said: for good performance you also need to work on the end-to-end CC solution, an AQM alone will not always be able to achieve the best performance.

Regards
 Roland