Hi Karen,

Good point. We definitely need to specify coupled L4S/Classic ECN marking schemes for shapers/policers as well. On the other hand, we should have a look if some BBR mechanisms could be used in TCP-Prague as well. We have pacing already in the requirements, we could investigate if (non-L4S supporting) policer detection would be useful too.

I'm not yet sure (until verified) if the current BBR version will be safe/fair to use on the Internet and how it would interact with DualQ. If it pushes away Classic traffic, it will be regarded as non-responsive traffic and also push away L4S traffic (due to the coupling). The PI2 classic AQM might be seen as a policer to BBR, and set a limit to the queue that BBR might create.

In theory L4S/ect(1) support should be possible to add on top of any classic congestion control, so also BBR. TCP-Prague CC should be able to take over control if ECN marked packets are received, and fall back to its classic behavior if not.

Koen.


> -----Original Message-----
> From: iccrg [mailto:iccrg-bounces@irtf.org] On Behalf Of Karen Elisabeth
> Egede Nielsen
> Sent: donderdag 17 november 2016 13:36
> To: in@bobbriscoe.net; Michael Welzl <michawe@ifi.uio.no>
> Cc: iccrg@irtf.org; Yuchung Cheng <ycheng@google.com>; tsvwg@ietf.org
> Subject: [iccrg] Policing and L4S WAS RE: BBR and other congestion control
> mechanisms
> 
> HI Bob, All,
> 
> Very interesting from my perspective. Thanks a lot for this presentation and
> mail discussion.
> 
> With the point of view of acc ECN/l4s then I wonder if it was worth to
> include some text about token bucket policing and
> L4S in the L4S Architecture draft, draft-briscoe-tsvwg-l4s-arch-00.
> 
> There is some considerations there already in section 5.2 and section 8.1,
> but would it be relevant to make it more explicit that the following
> statement (section 5.2)
> 
>       Nonetheless, if there are
>       Diffserv classes for important traffic, the L4S approach can
>       provide low latency for _all_ traffic within each Diffserv class
>       (including the case where there is only one Diffserv class).
> 
> Is only really truly valid if ingress diff serv
> classification/policing/shaping  it self - e.g., a BW policier - does not
> implement drop and in case of a shaper (and queuing) then ECN must be
> implemented by the shaper as well.
>  I suppose the latter somehow falls into ECN recommendations for AQM, so
> perhaps
>  these considerations (end the differences to a policier that do drop) are
> already totally clear to everybody.
> 
> ECN sort of is the alternative to let the CC be latency increase aware, but
> as pointed out here then
>  with bold drop token bucket policing we really have neither.
> 
> Thanks
> 
> BR, Karen
> 
> > -----Original Message-----
> > From: iccrg [mailto:iccrg-bounces@irtf.org] On Behalf Of Bob Briscoe
> > Sent: 17. november 2016 08:35
> > To: Michael Welzl <michawe@ifi.uio.no>
> > Cc: iccrg@irtf.org; Yuchung Cheng <ycheng@google.com>
> > Subject: Re: [iccrg] BBR and other congestion control mechanisms
> >
> > Michael,
> >
> > On Thu, November 17, 2016 1:49 am, Michael Welzl wrote:
> > > Hi,
> > >
> > >
> > > An add-on to my previous email below:
> > >
> > >
> > > Mechanisms like Veno and CTCP, if I get this right, only decide that a
> > > loss is due to congestion when it also comes with delay growth. I just
> > > witnessed your presentation in MAPRG about policing, and how BBR
> > > interacts with it:
> > > https://www.ietf.org/proceedings/97/slides/slides-97-maprg-traffic-
> polici
> > > ng-in-the-internet-yuchung-cheng-and-neal-cardwell-00.pdf
> > > <https://www.ietf.org/proceedings/97/slides/slides-97-maprg-traffic-
> polic
> > > ing-in-the-internet-yuchung-cheng-and-neal-cardwell-00.pdf> ( Thanks
> for
> > > giving this presentation, this was very interesting. )
> > >
> > > So this would mean that Veno, CTCP etc. would get it wrong in the face
> > > of
> > > a token bucket - there would be loss without delay growth, yet it IS due
> > > to a rate limit. Given the prevalence of policing that you have shown,
> > > and
> > > how BBR nicely interacts with it, I find this very interesting (and
> > > good); I’m not sure if this has been considered in the design of any
> > > congestion control before?  Does anyone know of any other examples?
> >
> > 1/ I would be interested if there is enough data in the BBR dataset to
> > zoom in on clients in specific ISPs. I know at least one traffic
> > management vendor uses a congestion policer that limits the rate of heavy
> > users, but the rate at which it limits varies all the time, because it is
> > the outcome congestion limit, not a rate limit. More precisely, it
> > enforces a "congestion rate token bucket" limit on the per-user
> > contribution to shared congestion.
> >
> > This would look similar to the varying capacity of a scheduler, but
> > distinguished from 3G/LTE or WiFi by only tiny additional delay, because
> > the measurement of shared congestion in the policer is based on an AQM
> on
> > a very high bandwidth link (typically 10G or 40Gb/s), so queuing delay is
> > very low before loss appears.
> >
> >
> > 2/ HULL (High throughput, utra-low latency) uses DCTCP at the sender. In
> > the network it marks ECN on the real packets based on the length of a
> > 'phantom' queue, which is a number incremented by the size of every
> > arriving packet, and decremented slightly more slowly than the drain rate
> > of the real queue. This keeps the real queue extremely short.
> >
> > This is an example where there is zero delay build-up before the intended
> > operating point. Because of the lack of delay, it also uses hardware
> > pacing on the sender NIC, because there is no queuing delay to space out
> > TCP's ACK clock otherwise.
> >
> > I don't know whether any DC operators are using HULL (does anyone?).
> >
> > You would not want to put Not-ECT packets (like current BBR) into such a
> > queue structure.
> >
> >
> > Bob
> >
> > >
> > > Cheers,
> > > Michael
> > >
> > >
> > >
> > >
> > >> On Nov 17, 2016, at 9:57 AM, Michael Welzl <michawe@ifi.uio.no>
> wrote:
> > >>
> > >>
> > >> Hi Yuchung,
> > >>
> > >>
> > >>
> > >>> On Nov 17, 2016, at 8:22 AM, Yuchung Cheng <ycheng@google.com
> > >>> <mailto:ycheng@google.com>> wrote:
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Nov 16, 2016 at 5:14 PM, Michael Welzl <michawe@ifi.uio.no
> > >>> <mailto:michawe@ifi.uio.no>> wrote:
> > >>>
> > >>>>
> > >>>> Dear all,
> > >>>>
> > >>>>
> > >>>> Thanks very much to the authors of BBR for an interesting
> > >>>> presentation yesterday! (see
> > >>>> https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-cong
> > >>>> estion-control-01.pdf
> > >>>> <https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-con
> > >>>> gestion-control-01.pdf> if you missed it).
> > >>>>
> > >>>> I’d be interested to know how BBR compares to some other TCP
> > >>>> variants. For example, TIMELY:
> > >>>>
> http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p537.pdf
> > >>>>
> > <http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p537.pdf>
> > >>>>
> > >>>>
> > >>>> Any idea, anyone?
> > >>>>
> > >>>>
> > >>>> ( I tried, honestly, I did - but I just can’t manage to hold back a
> > >>>> tiny bit of sarcasm here:  ...is BBR also 133 times faster than
> > >>>> TIMELY?  :-)  )
> > >>>>
> > >>>
> > >>> I'd like to explain further about 133x result of BBR vs Cubic first.
> > >>>
> > >>>
> > >>> This is a spot-test. It is not a general quantification of BBR's
> > >>> improvement on our B4 network (we'll publish that data in a separate
> > >>> report). This is a 8MB RPC every 30s on a warmed connection on
> > >>> east-US <-> west-EU using the lowest QoS (best effort). That
> > >>> particular path has quite a bit of burst induced losses. In this
> > >>> case, Cubic just died due to 1/sqrt(p). Notice the network path is
> > >>> fairly different from the typical Internet access: it's +10G from
> > >>> start to end, with little buffers (compared to BDP) on the way to
> > >>> build persistent large queues. The losses are caused by burst traffic
> > >>> (of same or higher QoS). What the number 133x highlighted was when
> > >>> loss does not indicate a persistent queue, using it as a sole
> > >>> congestion signal is brittle. It is not to say loss signal is
> > >>> completely useless, but it does not always indicate self-induced
> > >>> queues.
> > >>
> > >> Thanks for this information!
> > >>
> > >>
> > >>
> > >>> Comparing TIMELY and BBR are at one level comparing apples to
> > >>> oranges. TIMELY is designed for a very specific hw/transport
> > >>> (RDMA/kernel-bypass) and networks (intra-DC) with the goal to
> reduce
> > >>> tail latency. Hence it uses specific NIC features like HW timestamps
> > >>> to perform its control.
> > >>
> > >> Well … I understand it was made for intra-DC communication, and it uses
> > >> these features to get the necessary timer granularity. I can’t see why
> > >> this wouldn’t make it applicable to networks with larger RTTs?
> > >>
> > >>
> > >>> BBR is designed to be a generic good transport to achieve high tput
> > >>> with adequate queue. It faces an unknown network and sole feedback
> is
> > >>> TCP-acks hence it must first perform a exponential search to get a
> > >>> basic sense of the network. It also needs to deal with all sorts of
> > >>> adverse issues in the wild Internet as we'v described.
> > >>
> > >> Right… E.g. I guess Timely wouldn’t work so well when facing losses.
> > >>
> > >>
> > >>
> > >>> A more interesting comparison would be vs other Internet-CC like
> > >>> Vegas (which we are working on).
> > >>>
> > >>
> > >> No, I think that’s a terribly uninteresting comparison to do. I’d even
> > >> call it ridiculous, as many many MANY mechanisms work better than
> > >> Vegas, after all it's now 16 years old.
> > >> May I propose comparing against the state of the art, not the state of
> > >> 2000?
> > >>
> > >>
> > >> Some of the direct follow-ups on Vegas for example, like FAST? Or
> > >> delay-based algorithms that also use a bit more information than just
> > >> delay?
> > >>
> > >> E.g. Westwood comes to mind, as a mechanism that also looks at the
> rate
> > >> at which ACKs arrive. AFAIK this one is readily available in Linux for
> > >> comparison.
> > >>
> > >> Also, some TCP variants try to distinguish between random packet loss
> > >> and congestion-induced packet loss - so similar to BBR, you wouldn’t
> > >> overreact to random losses with them. Veno is an example, and CTCP
> > >> (Coded TCP, not Compound TCP) too.
> > >>
> > >>
> > >> Cheers,
> > >> Michael
> > >>
> > >>
> > >> _______________________________________________
> > >> iccrg mailing list iccrg@irtf.org
> > >> https://www.irtf.org/mailman/listinfo/iccrg
> > >>
> > >
> > > _______________________________________________
> > > iccrg mailing list iccrg@irtf.org
> > > https://www.irtf.org/mailman/listinfo/iccrg
> > >
> > >
> >
> >
> > _______________________________________________
> > iccrg mailing list
> > iccrg@irtf.org
> > https://www.irtf.org/mailman/listinfo/iccrg
> 
> _______________________________________________
> iccrg mailing list
> iccrg@irtf.org
> https://www.irtf.org/mailman/listinfo/iccrg