Re: [tcpm] Adoption of draft-zimmermann-tcpm-cubic?

Neal Cardwell <ncardwell@google.com> Thu, 19 March 2015 01:21 UTC

MIME-Version: 1.0
In-Reply-To: <2235E611-C0A1-42A9-9B00-D584FF3C8B3B@netapp.com>
References: <ae97de90f83c460f8cfd0273f47611dd@hioexcmbx05-prd.hq.netapp.com> <655C07320163294895BBADA28372AF5D16C60F4D@FR712WXCHMBA15.zeu.alcatel-lucent.com> <2235E611-C0A1-42A9-9B00-D584FF3C8B3B@netapp.com>
Date: Wed, 18 Mar 2015 21:20:54 -0400
Message-ID: <CADVnQyk-hqAf_2VDTx841yTRO3SaLJNZpi3XmnQcyKgcQ4nAPA@mail.gmail.com>
From: Neal Cardwell <ncardwell@google.com>
To: "Zimmermann, Alexander" <Alexander.Zimmermann@netapp.com>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <http://mailarchive.ietf.org/arch/msg/tcpm/wZicL4ps9Rk_jHccHQwyYK5qedQ>
Cc: "tcpm-chairs@tools.ietf.org" <tcpm-chairs@tools.ietf.org>, Eric Dumazet <edumazet@google.com>, "tcpm@ietf.org" <tcpm@ietf.org>
Subject: Re: [tcpm] Adoption of draft-zimmermann-tcpm-cubic?
Precedence: list

On Wed, Mar 18, 2015 at 6:26 AM, Zimmermann, Alexander
<Alexander.Zimmermann@netapp.com> wrote:
> Hi Michael,
>
>> Am 18.03.2015 um 11:18 schrieb Scharf, Michael (Michael) <michael.scharf@alcatel-lucent.com>:
>>
>> My understanding is that TCPM recharting is still pending and IESG waits for potential feedback until March 23 [1].
>>
>> The chairs are well aware of the strong support for draft-zimmermann-tcpm-cubic during IETF 91.
>> We plan to run a formal adoption call once our charter explicitly allows adoption in TCPM.

Regarding draft-zimmermann-tcpm-cubic-00:

  https://tools.ietf.org/id/draft-zimmermann-tcpm-cubic-00.txt

I support this draft.

But I did want to note that our experience with CUBIC at Google shows
that (1) it's important to consider stretch ACKs, and (2) it's critical to
carefully consider the maximum rate of cwnd increase for CUBIC
in congestion avoidance.

I would suggest the following small diffs to the draft:

Where the draft says (in sections "3.3. Concave region" and "3.4.
Convex region"):

  In this region, cwnd MUST be incremented by (W(t+RTT) - cwnd)/cwnd
  for each received ACK.

I'd suggest the following new text in these two spots:

  In this region, cwnd MUST be incremented by
     min((W(t+RTT) - cwnd)/cwnd, 1/2)
  for each newly-acknowledged packet.

Detailed rationale:

(1) First is the issue of the treatment of "stretch ACKs" covering
more than 2 packets.  The CUBIC paper and the Linux code through v3.18
cap the de facto maximum rate of cwnd increase (in congestion
avoidance) at 1 packet for every alternate ACK. In Google's experience
with high-BDP paths with receiver hosts using receive
offload/aggregation mechanisms (LRO and GRO in Linux terminology), the
ACKs for up to 40 or more packets at a time caused this cap of "1
packet for every alternate ACK" to result in cwnd increases that were
too slow, leading to underutilization. So our team at Google
contributed some changes in v3.19 that changed the rate of increase to
be expressed in terms of packets ACKed, rather than number of ACKs.
This approach allows full utilization even with receiver offload
mechanisms.

(2) Second is the issue of the maximum rate of increase of cwnd. As it
stands, this document says in section "3.4. Convex region" that:

   In this region, cwnd MUST be incremented by (W(t+RTT) - cwnd)/cwnd
   for each received ACK.

There is similar language in section "3.3. Concave region".

Since the rate of increase of a cubic function can be quite large, if
we take this language literally then in steep sections of the curve
this can require that the sender "MUST" increase cwnd by quite
a large amount.

I am not sure what other CUBIC implementations do, but this does not
match the CUBIC paper or the Linux implementation. As mentioned above,
the CUBIC paper and the Linux code through v3.18 actually cap the de
facto maximum rate of cwnd increase (in congestion avoidance) at 1
packet for every alternate ACK.

In our recent experiments with using CUBIC for YouTube video traffic
using the revised, stretch-ACK-savvy CUBIC logic in Linux v3.19 we saw
retransmit rates double due to v3.19 allowing cwnd to increase by 1
packet for every packet ACKed.

So in Linux v4.0 we set a limit of increasing cwnd by at most 1 packet
for every 2 packets ACKed, which restored the retransmit rates to
their previous low levels.

For reference:

(1) Commits in Linux v3.19 to make CUBIC stretch-ACK-savvy:

  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=e73ebb0881ea5534ce606c1d71b4ac44db5c6930
  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=814d488c61260521b1b3cc97063700a5a6667c8f
  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=9cd981dcf174d26805a032aefa791436da709bee
  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=d6b1a8a92a1417f8859a6937d2e6ffe2dfab4e6d

(2) Commits slated for Linux v4.0 to fix excessive cwnd growth in CUBIC:

  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=9949afa42be0b76f5832db112ce51bb6b35b2abb
  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=d578e18ce93f5d33a7120fd57c453e22a4c0fc37

neal

Re: [tcpm] Adoption of draft-zimmermann-tcpm-cubi… Zimmermann, Alexander
Re: [tcpm] Adoption of draft-zimmermann-tcpm-cubi… Scharf, Michael (Michael)
[tcpm] Adoption of draft-zimmermann-tcpm-cubic? Scheffenegger, Richard
Re: [tcpm] Adoption of draft-zimmermann-tcpm-cubi… Neal Cardwell
Re: [tcpm] Adoption of draft-zimmermann-tcpm-cubi… Zimmermann, Alexander