Network Working Group M. Bagnulo Internet-Draft A. Garcia-Martinez Intended status: Informational UC3M Expires: 12 August 2023 8 February 2023 LEDBAT++ BBR interoperability issues draft-bagnulo-iccrg-iccrg-ledbat-bbr-interop-00 Abstract This document specifies describes some interoperability issues identified between LEDBAT++ and BBR, resulting in unexpected behaviour. Specifically, that under a set of common conditions, LEDBAT++ fails to yield in front of both BBRv1 and BBRv2(instead of the opposite expected behaviour). Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 12 August 2023. Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Bagnulo & Garcia-MartinezExpires 12 August 2023 [Page 1] Internet-Draft LEDBAT++ BBR February 2023 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. BBR LEDBAT++ Interoperability issues . . . . . . . . . . . . 3 3. Proposed solution . . . . . . . . . . . . . . . . . . . . . . 5 4. Experiment data . . . . . . . . . . . . . . . . . . . . . . . 6 5. Security Considerations . . . . . . . . . . . . . . . . . . . 7 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7 8. Informative References . . . . . . . . . . . . . . . . . . . 7 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7 1. Introduction BBR (Bottleneck Bandwidth and Round-trip propagation time) [I-D.cardwell-iccrg-bbr-congestion-control] is a model-based congestion control algorithm that attempts to improve the performance of Internet communications by reducing the delay (when bottleneck buffers are large) and increase the throughput (when bottleneck buffers are small). BBR estimates the queueing delay by monitoring the round-trip-time (RTT) and it detects congestion onset when the queue in the bottleneck's buffer starts to build up. BBR then adjusts its rate to send as fast as possible while avoiding the formation of queues in the bottleneck, operating closer to the optimal operation point. Two versions of BBR exist. BBRv1 is included in Linux since version 4.9, and it is currently widely used in the Internet. BBRv2 is being developed and it aims to overcome some limitations identified for BBRv1. LEDBAT++ (Low Extra Delay Background Transport ++)[I-D.irtf-iccrg-ledbat-plus-plus] is a congestion-control algorithm that implements a less-than-best-effort (LBE) traffic class. When LEDBAT++ traffic shares a bottleneck with one or more TCP connections using Cubic or other loss-based congestion control algorithms, it reduces its sending rate earlier and more aggressively than competing flows, allowing Cubic traffic to use more of the available capacity. This effectively implements an LBE traffic class that has less priority than Cubic-TCP/best-effort traffic. LEDBAT++ reacts both to packet loss and to variations in delay. Regarding packet loss, LEDBAT++ reacts with a multiplicative decrease, similar to most TCP congestion controllers. Regarding delay, LEDBAT++ aims for a target queueing delay T (i.e., on top of the base RTT), set to 60 ms. Similarly to BBR, LEDBAT++ estimates the queueing delay by monitoring the RTT. When the measured queueing delay is below the target T, LEDBAT++ additively increases the sending rate and when the delay is Bagnulo & Garcia-MartinezExpires 12 August 2023 [Page 2] Internet-Draft LEDBAT++ BBR February 2023 above the target T, it multiplicatively reduces the sending rate. LEDBAT++ is the evolution of the original LEDBAT algorithm [RFC6817] that overcomes multiple limitations identified in the original specification. Since BBR aims to provide a best effort service i.e. a traffic class with similar priority than Cubic and New Reno and LEDBAT++ is designed to provide a less-than-best-effort traffic class that yields in front of best effort traffic, it is expected that LEDBAT yields in front of BBR. On other other hand, BBR aims to operate closer to the point where there is no queue while LEDBAT++ is designed to operate with a queueing delay equal to thew target T, which may hint that LEDBAT++ is more aggressive than BBR (as it is willing to endure a longer queue). In the following sections, we report the results of some experiments that confirm that indeed in certain conditions, LEDBAT++ is more aggressive than BBR and because of that, LEDBAT++ fails to yield in front of BBR traffic, unfulfilling the design goals of both protocols. We detail the conditions when this happens and propose a possible solution to the identified interoperability problem. 2. BBR LEDBAT++ Interoperability issues We performed a number of experiments to understand LEDBAT++ BBR interaction. We provide details about the experimental setup in section XX below. We found the following results: Experiment 1: Two TCP connections with the same RTT, one of them using BBRv1 and the other one using LEDBAT++, compete for the capacity of the bottleneck link that is using a buffer that is large enough to generate a queueing delay of at least LEDBAT++'s target. We observe that: * For base RTTs larger than T, LEDBAT++ behaves as a scavenger transport and yields in front of BBRv1. * For base RTTs smaller than T, LEDBAT++ does not yield and seizes a significant share of the capacity (about half of the available capacity, depending on the specific RTT). * Uncoordinated slowdowns of BBRv1 and LEDBAT++ flows are not enough to enable both flows to have visibility of the base RTT. BBRv1 is unable to seize all the available capacity when the RTT is smaller than T because of its flightsize cap. Indeed, BBRv1 defines a flightsize cap of twice the bandwidth-delay product. This implies that the maximum queueing delay that a BBRv1 flow can generate is one (additional) RTT. LEDBAT++ on the other hand, can tolerate a queuing Bagnulo & Garcia-MartinezExpires 12 August 2023 [Page 3] Internet-Draft LEDBAT++ BBR February 2023 delay of T. So, when the RTT is smaller than T, LEDBAT++ tolerates more queueing delay than BBRv1, which implies that the LEDBAT++ flow will bot back off when competing against a BBRv1 flow, explaining the observed behaviour. Experiment 2: Two TCP connections with the same RTT, one of them using BBRv1 and the other one using LEDBAT++, compete for the capacity of the bottleneck link that is using a buffer that is capable of generating a maximum queueing delay of B seconds, B being smaller than LEDBAT++'s target T (i.e. same as Experiment 1 but with B smaller than T). We observe that: * For base RTTs larger than B, LEDBAT++ behaves as a scavenger transport and yields in front of BBRv1. * For base RTTs smaller than B, LEDBAT++ does not yield and seizes a significant share of the capacity (about half of the available capacity, depending on the specific RTT). In this case, we observe a similar behaviour than in experiment 1, only that the tipping point when LEDBAT++ starts to yield in front of BBRv1 is determined by the buffer size. For RTTs smaller than the buffer size, the same justification used for the Experiment 1 results apply to explain why BBRv1 yields. For RTTs larger than B, then the BBRv1 flow is able to generate a queue that is alrge enough to create losses, which forces LEDBAT++ into packet loss mode, behaving less aggressively than Cubic and thus yielding in front of BBRv1 Experiment 3: Two TCP connections with the same RTT, one of them using BBRv2 and the other one using LEDBAT++, compete for the capacity of the bottleneck link that is using a buffer that is large enough to generate a queueing delay of at least LEDBAT++'s target. We observe that: * For base RTTs larger than T, LEDBAT++ behaves as a scavenger transport and yields in front of BBRv2. * For base RTTs smaller than T, BBRv2 yields in front of LEDBAT++, behaving exactly the opposite than expected. Bagnulo & Garcia-MartinezExpires 12 August 2023 [Page 4] Internet-Draft LEDBAT++ BBR February 2023 Most modifications introduced in BBRv2 are intended to make it less aggressive, which is likely to explain the experiment's results. In particular, in BBRv2 the flightsize is not limited only by the flightsize cap, but also for the inflight model parameter that is aiming for a small queue, which seem to be at odds with LEDBAT++ actions to bloat the queue up to the target T. Also, BBRv2 aims to leave a headroom in the link to enable other flows to enter. It may the case that this headroom is filled by LEDBAT++ traffic, enabling LEDBAT++ to seize more capacity. Experiment 4: Two TCP connections with the same RTT, one of them using BBRv2 and the other one using LEDBAT++, compete for the capacity of the bottleneck link that is using a buffer that is capable of generating a maximum queueing delay of B seconds, B being smaller than LEDBAT++'s target T (i.e. same as Experiment 3 but with B smaller than T). We observe that: * For base RTTs smaller than B, BBRv2 yields in front of LEDBAT++. * For base RTTs larger than B, BBRv2 gradually seizes a larger share of the capacity, but it is not until much larger RTTs that is able to seize all the capacity. BBRv2 incorporates new mechanisms to react to losses. When using buffers smaller that the target T, loss is the main congestion signal and both BBRv2 and LEDBAT++ respond to it ad the new BBRv2 mechanism seem to make it more aggressive, leaving some room for LEDBAT++ to seize. 3. Proposed solution In our previous analysis, we posit that a BBR flow are unable to seize all the available capacity when competing against a LEDBAT++ flow due to its self-imposed limitation on the flightsize. This limitation is set to one additional BDP, and it is in place to limit the size of the resulting queue. When the RTT is smaller than the target T, the queue generated by the LEDBAT++ flow is larger than the one BBR is willing to generate, hence BBR yields. In order to address this issue, we propose to limit the queue generated by LEDBAT++ to the minimum of the target T and the base RTT. This should solve the problem since this would result in LEDBAT++ refraining from generating queues larger than one BDP, so, it would consequently yield in from of BBR. We believe this would also positive for LEDBAT++ itself, as it would prevent it from bloating the RTT in relative terms compared to the base RTT for small base RTTs. Bagnulo & Garcia-MartinezExpires 12 August 2023 [Page 5] Internet-Draft LEDBAT++ BBR February 2023 We have implemented the proposed solution and we have verified that it addresses the identified issue i.e. with the proposed solution, LEDBAT++ always yields in front of BBR. 4. Experiment data In our experiments, we use the virtualised setup depicted in Figure 1, which features a dumbbell topology, that allows us to generate both LEDBAT++ and BBR flows that compete for the capacity of a bottleneck link. We configure the characteristics of the bottleneck link, including the size of the buffer, the capacity and the delay. We use this topology for our experiments, because any path, no matter how complex it is, can be accurately modelled from the transport layer perspective as a single link with the RTT of the overall path and the capacity of the path's bottleneck link, which is exactly what this simple topology represents. +-----+ +--| S1 | +-----+ +-----+ +-----+ | +-----+ | C1 |------------| R1 |----------| R2 |-------------+ +-----+ | | | |-------------+ +-----+ +-----+ | +-----+ +--| S2 | +-----+ Figure 1: The rLEDBAT architecture. C1, R1, R2 and S1 are Linux systems while S2 is a Windows 2019 Server with LEDBAT++ capability. S1 has BBR enabled. BBRv1 is already available in the Linux kernel and BBRv2 is installed. Traffic is generated in S1 using the nc tool and in S2 using the ctsTraffic tool (i.e., bulk transfer type of traffic in both cases). The client in C1 uses nc. The link connecting R2 with R1 is the bottleneck link of the communications between S1 (S2) and C1. We set its capacity to different values using the tbf queueing discipline for the tc traffic control tool. A drop-tail buffer is configured in the R2-to-R1 link, with a size that we vary on different experiments, to represent different network setups. The links between S1 (S2) and R2 and the ones between C1 and R1 are configured with (much) larger capacities than the bottleneck link. During the experiments, we set the RTT of the path between S1 (S2) and C1 using tc netem. Bagnulo & Garcia-MartinezExpires 12 August 2023 [Page 6] Internet-Draft LEDBAT++ BBR February 2023 In all the experiments, C1 connects to S1 and S2 nodes to perform downloads. Each flow is greedy, in the sense that it aims to transmit as much data as possible. Data is transferred using TCP, using BBR between C1 and S1 and LEDBAT++ between C1 and S2. TCP flow control never limits the communication rate, as we manually configure a large receiver window. To compute the rates for each flow, we start a tcpdump capture in C1. The MSS used is 1,390 bytes while the MTU is 1,456 bytes. Additional data about the experiments, including graphs with the measurement results can be found at [COMNET] 5. Security Considerations 6. IANA Considerations 7. Acknowledgements This work was supported by the EU through the StandICT CEL6 project. 8. Informative References [COMNET] Bagnulo, M.B. and A.G. Garcia-Martinez, "When less is more: BBR versus LEDBAT++", Computer Networks Volume 219, 2022. [I-D.cardwell-iccrg-bbr-congestion-control] Cardwell, N., Cheng, Y., Yeganeh, S. H., Swett, I., and V. Jacobson, "BBR Congestion Control", Work in Progress, Internet-Draft, draft-cardwell-iccrg-bbr-congestion- control-02, 7 March 2022, . [I-D.irtf-iccrg-ledbat-plus-plus] Balasubramanian, P., Ertugay, O., and D. Havey, "LEDBAT++: Congestion Control for Background Traffic", Work in Progress, Internet-Draft, draft-irtf-iccrg-ledbat-plus- plus-01, 25 August 2020, . [RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind, "Low Extra Delay Background Transport (LEDBAT)", RFC 6817, DOI 10.17487/RFC6817, December 2012, . Authors' Addresses Bagnulo & Garcia-MartinezExpires 12 August 2023 [Page 7] Internet-Draft LEDBAT++ BBR February 2023 Marcelo Bagnulo UC3M Email: marcelo@it.uc3m.es Alberto Garcia-Martinez UC3M Email: alberto@it.uc3m.es Bagnulo & Garcia-MartinezExpires 12 August 2023 [Page 8]