idnits 2.17.1 draft-irtf-iccrg-rledbat-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 179: '...nders. In particular, the sender MUST...' RFC 2119 keyword, line 180: '... implement [I-D.ietf-tcpm-rfc793bis] and it also MUST implement the...' RFC 2119 keyword, line 181: '... Time Stamp Option as defined in [RFC7323]. Also, the sender SHOULD...' RFC 2119 keyword, line 187: '... The rLEDBAT receiver MUST use an LBE...' RFC 2119 keyword, line 197: '...rLEDBAT receiver SHOULD use the LEDBAT...' (11 more instances...) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 25, 2020) is 1330 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Outdated reference: A later version (-28) exists of draft-ietf-tcpm-rfc793bis-18 -- Obsolete informational reference (is this intentional?): RFC 8312 (Obsoleted by RFC 9438) Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group M. Bagnulo 3 Internet-Draft A. Garcia-Martinez 4 Intended status: Experimental UC3M 5 Expires: February 26, 2021 G. Montenegro 6 P. Balasubramanian 7 Microsoft 8 August 25, 2020 10 rLEDBAT: receiver-driven Low Extra Delay Background Transport for TCP 11 draft-irtf-iccrg-rledbat-01.txt 13 Abstract 15 This document specifies the rLEDBAT, a set of mechanisms that enable 16 the execution of a less-than-best-effort congestion control algorithm 17 for TCP at the receiver end. 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at https://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on February 26, 2021. 36 Copyright Notice 38 Copyright (c) 2020 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents 43 (https://trustee.ietf.org/license-info) in effect on the date of 44 publication of this document. Please review these documents 45 carefully, as they describe your rights and restrictions with respect 46 to this document. Code Components extracted from this document must 47 include Simplified BSD License text as described in Section 4.e of 48 the Trust Legal Provisions and are provided without warranty as 49 described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 2. Motivations for rLEDBAT . . . . . . . . . . . . . . . . . . . 3 55 3. rLEDBAT mechanisms . . . . . . . . . . . . . . . . . . . . . 4 56 3.1. Controlling the receive window . . . . . . . . . . . . . 5 57 3.1.1. Avoiding window shrinking . . . . . . . . . . . . . . 6 58 3.1.2. Window Scale Option . . . . . . . . . . . . . . . . . 6 59 3.2. Measuring delays . . . . . . . . . . . . . . . . . . . . 7 60 3.2.1. Measuring the RTT to estimate the queueing delay . . 7 61 3.2.2. Measuring one way delay to estimate the queueing 62 delay . . . . . . . . . . . . . . . . . . . . . . . . 10 63 3.3. Detecting packet losses and retransmissions . . . . . . . 12 64 4. Security Considerations . . . . . . . . . . . . . . . . . . . 12 65 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 66 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 67 7. Informative References . . . . . . . . . . . . . . . . . . . 12 68 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 13 70 1. Introduction 72 LEDBAT (Low Extra Delay Background Transport) [RFC6817] is a 73 congestion-control algorithm that implements a less-than-best-effort 74 (LBE) traffic class. 76 When LEDBAT traffic shares a bottleneck with one or more TCP 77 connections using standard congestion control algorithms such as 78 Cubic [RFC8312] (hereafter standard-TCP for short), it reduces its 79 sending rate earlier and more aggressively than standard-TCP 80 congestion control, allowing standard-TCP traffic to use more of the 81 available capacity. In the absence of competing standard-TCP 82 traffic, LEDBAT aims to make an efficient use of the available 83 capacity, while keeping the queuing delay within predefined bounds. 85 LEDBAT reacts both to packet loss and to variations in delay. 86 Regarding to packet loss, LEDBAT reacts with a multiplicative 87 decrease, similar to most TCP congestion controllers. Regarding 88 delay, LEDBAT aims for a target queueing delay. When the measured 89 current queueing delay is below the target, LEDBAT increases the 90 sending rate and when the delay is above the target, it reduces the 91 sending rate. LEDBAT estimates the queuing delay by subtracting the 92 measured current one-way delay from the estimated base one-way delay 93 (i.e. the one-way delay in the absence of queues). 95 The LEDBAT specification [RFC6817] defines the LEDBAT congestion- 96 control algorithm, implemented in the sender to control its sending 97 rate. LEDBAT is specified in a protocol and layer agnostic manner. 99 LEDBAT++ [I-D.balasubramanian-iccrg-ledbatplusplus] is also an LBE 100 congestion control algorithm which is inspired in LEDBAT while 101 addressing several problems identified with the original LEDBAT 102 specification. In particular the differences between LEDBAT and 103 LEDBAT++ include: i) LEDBAT++ uses the round-trip-time (RTT) (as 104 opposed to the one way delay used in LEDBAT) to estimate the queuing 105 delay; ii) LEDBAT++ uses an Additive Increase/Multiplicative Decrease 106 algorithm to achieve inter-LEDBAT++ fairness and avoid the late-comer 107 advantage observed in LEDBAT; iii) LEDBAT++ performs periodic 108 slowdowns to improve the measurement of the base delay; iv) LEDBAT++ 109 is defined for TCP. 111 In this note, we describe rLEDBAT, a set of mechanisms that enable 112 the execution of an LBE delay-based congestion control algorithm such 113 as LEDBAT or LEDBAT++ in the receiver end of a TCP connection. 115 2. Motivations for rLEDBAT 117 rLEDBAT enables new use cases and new deployment models, fostering 118 the use of LBE traffic and benefitting the global Internet by 119 improving overall allocation of resources. The following scenarios 120 are enabled by rLEDBAT: 122 Content Delivery Networks and more sophisticated file distribution 123 scenarios: Consider the case where the source of a file to be 124 distributed (e.g., a software developer that wishes to distribute 125 a software update) would prefer to use LBE and it enables LEDBAT/ 126 LEDBAT++ in the servers containing the source file. However, 127 because the file is being distributed through a CDN which 128 surrogates do not support LBE congestion control, the result is 129 that the file transfers, originated from CDN surrogates will not 130 be using LBE. Interestingly enough, in the case of the software 131 update, the developer may also control the software performing the 132 download in the client, the receiver of the file, but because 133 current LEDBAT/LEDBAT++ are sender-based algorithms, controlling 134 the client is not enough to enable LBE congestion control in the 135 communication. rLEDBAT would enable the use of LBE traffic class 136 for file distribution in this setup. 138 Interference from proxies and other middleboxes: Proxies and other 139 middleboxes are a commonplace in the Internet. For instance, in 140 the case of mobile networks, proxies are frequently used. In the 141 case of enterprise networks, it is common to deploy corporate 142 proxies for filtering and firewalling. In the case of satellite 143 links, Performance Enhancement Proxies (PEPs) are deployed to 144 mitigate the effect of the long delay in TCP connection. These 145 proxies terminate the TCP connection on both ends and prevent the 146 use of LBE congestion control in the segment between the proxy and 147 the sink of the content, the client. By enabling rLEDBAT, clients 148 would be able to enable LBE traffic between them and the proxy. 150 Receiver-defined preferences. It is frequent that the bottleneck 151 of the communication is the access link. This is particularly 152 true in the case of mobile devices. It is then especially 153 relevant for mobile devices to properly manage the capacity of the 154 access link. With current technologies, it is possible for the 155 mobile device to use different congestion control algorithms 156 expressing different preferences for the traffic. For instance, a 157 device can choose to use standard-TCP for some traffic and to use 158 LEDBAT/LEDBAT++ for other traffic. However, this would only 159 affect the outgoing traffic since both standard-TCP and LEDBAT/ 160 LEDBAT++ are sender-driven. The mobile device has no means to 161 manage the traffic in the down-link, which is in most cases, the 162 communication bottleneck for a typical eye-ball end-user. rLEDBAT 163 enables the mobile device to selectively use LBE traffic class for 164 some of the incoming traffic. For instance, by using rLEDBAT, a 165 user can use regular standard-TCP/UDP for video stream (e.g., 166 Youtube) and use rLEDBAT for other background file download. 168 3. rLEDBAT mechanisms 170 rLEDBAT provides the mechanisms to implement an LBE congestion 171 control algorithm at the receiver-end of a TCP connection. The 172 rLEDBAT receiver controls the sender's rate through the Receive 173 Window announced to the receiver in the TCP header. 175 rLEDBAT assumes that the sender is a standard TCP sender. rLEDBAT 176 does not require any rLEDBAT-specific modifications to the TCP 177 sender. The envisioned deployment model for rLEDBAT is that the 178 clients implement rLEDBAT and this enable rLEDBAT in communications 179 with existent standard TCP senders. In particular, the sender MUST 180 implement [I-D.ietf-tcpm-rfc793bis] and it also MUST implement the 181 Time Stamp Option as defined in [RFC7323]. Also, the sender SHOULD 182 implement some of the standard congestion control mechanisms, such as 183 Cubic [RFC8312] or New Reno [RFC5681]. 185 rLEDBAT does not defines a new congestion control algorithm. The LBE 186 congestion control algorithm executed in the rLEDBAT receiver is 187 defined in other documents. The rLEDBAT receiver MUST use an LBE 188 congestion control algorithm. Because rLEDBAT assumes a standard TCP 189 sender, the sender will be using a "best effort" congestion control 190 algorithm (such as Cubic or New Reno). Since rLEDBAT uses the 191 Receive Window to control the sender's rate and the sender calculates 192 the sender's window as the minimum of the Receive window and the 193 congestion window, rLEDBAT will only be effective as long as the 194 congestion control algorithm executed in the receiver yields a 195 smaller window than the one calculated by the sender. This is 196 normally the case when the receiver is using an LBE congestion 197 control algorithm. The rLEDBAT receiver SHOULD use the LEDBAT 198 congestion control algorithm [RFC6817] or the LEDBAT++ congestion 199 control algorithm [I-D.balasubramanian-iccrg-ledbatplusplus]. The 200 rLEDBAT MAY use other LBE congestion control algorithms defined 201 elsewhere. Irrespectively of which congestion control algorithm is 202 executed in the receiver, an rLEDBAT connection will never be more 203 aggressive than standard TCP since it is always bounded by the 204 congestion control algorithm executed at the sender. 206 rLEDBAT is essentially composed of three types of mechanisms, namely, 207 those that provide the means to measure the packet delay (either the 208 round trip time or the one way delay, depending on the selected 209 algorithm), mechanisms to detect packet loss and the means to 210 manipulate the Receive Window to control the sender's rate. We 211 describe them next. 213 3.1. Controlling the receive window 215 rLEDBAT uses the Receive Window (RCV.WND) of TCP to enable the 216 receiver to control the sender's rate. [I-D.ietf-tcpm-rfc793bis] 217 defines that the RCV.WND is used to announce the available receive 218 buffer to the sender for flow control purposes. In order to avoid 219 confusion, we will call fc.WND the value that a standard RFC793bis 220 TCP receiver calculates to set in the receive window for flow control 221 purposes. We call rl.WND the window value calculated by rLEDBAT 222 algorithm and we call RCV.WND the value actually included in the 223 Receive Window field of the TCP header. For a RFC793bis receiver, 224 RCV.WND == fc.WND. 226 In the case of rLEDBAT receiver, the rLEDBAT receiver MUST NOT set 227 the RCV.WND to a value larger than fc.WND and it SHOULD set the 228 RCV.WND to the minimum of rl.WND and fc.WND, honoring both. 230 When using rLEDBAT, two congestion controllers are in action in the 231 flow of data from the sender to the receiver, namely, the congestion 232 control algorithm of TCP in the sender side and the LBE congestion 233 control algorithm executed in the receiver and conveyed to the sender 234 through the RCV.WND. In the normal TCP operation, the sender uses 235 the minimum of the congestion window cwnd and the receiver window 236 RCV.WND to calculate the sender's window SND.WND. This is also true 237 for rLEDBAT, as the sender is a regular TCP sender. This guarantees 238 that the rLEDBAT flow will never transmit more aggressively than a 239 TCP flow, as the sender's congestion window limits the sending rate. 240 Moreover, because a LBE congestion control algorithm such as LEDBAT/ 241 LEDBAT++ is designed to react earlier and more aggressively to 242 congestion than regular TCP congestion control, the rl.WND contained 243 in the RCV.WND field of TCP will be in general smaller than the 244 congestion window calculated by the TCP sender, implying that the 245 rLEDBAT congestion control algorithm will be effectively controlling 246 the sender's window. 248 In summary, the sender's window is: SND.WND = min(cwnd, rl.WND, 249 fc.WND) 251 3.1.1. Avoiding window shrinking 253 The LEDBAT/LEDBAT++ algorithm executed in a rLEDBAT receiver 254 increases or decreases the rl.WND according to congestion signals 255 (variations on the estimations of the queueing delay and packet 256 loss). If the new congestion window is smaller than the current one 257 then directly announcing it in the RCV.WND may result in shrinking 258 the window, i.e., moving the right window edge to the left. 259 Shrinking the window is discouraged as per [I-D.ietf-tcpm-rfc793bis], 260 as it may cause unnecessary packet loss and performance penalty. To 261 be consistent with [I-D.ietf-tcpm-rfc793bis], the rLEDBAT receiver 262 SHOULD NOT shrink the receive window. 264 In order to avoid window shrinking, upon the reception of a data 265 packet, the announced window can be reduced in the number of bytes 266 contained in the packet at most. This may fall short to honor the 267 new calculated value of the rl.WND. So, in order to reduce the 268 window as dictated by the rLEDBAT algorithm, the receiver will 269 progressively reduce the advertised RCV.WND, always honoring that the 270 reduction is less or equal than the received bytes, until the target 271 window determined by the rLEDBAT algorithm is reached. This implies 272 that it may take up to one RTT for the rLEDBAT receiver to drain 273 enough in-flight bytes to completely close its receive window without 274 shrinking it. This is more than sufficient to honor the window 275 output from the LEDBAT/LEDBAT++ algorithms since they only allows to 276 perform at most one multiplicative decrease per RTT. 278 3.1.2. Window Scale Option 280 The Window Scale (WS) option [RFC7323] is a mean to increase the 281 maximum window size permitted by the Receive Window. The use of the 282 WS option implies that the changes in the window are expressed in the 283 units resulting of the WS option used in the TCP connection. This 284 means that the rLEDBAT client will have to accumulate the increases 285 resulting from the different received packets, and only convey a 286 change in the window when the accumulated sum of increases is equal 287 or higher than one unit used to express the receive window according 288 to the WS option in place for the TCP connection. 290 Changes in the receive window that are smaller than 1 MSS are 291 unlikely to have any immediate impact on the sender's rate, as usual 292 TCP segmentation practice results in sending full segments (i.e., 293 segments of size equal to the MSS). So, accumulating changes in the 294 receive window until completing a full MSS in the sender or in the 295 receiver makes little difference. 297 Current WS option specification [RFC7323] defines that allowed values 298 for the WS option are between 0 and 14. Assuming a MSS around 1500 299 bytes, WS option values between 0 and 11 result in the receive window 300 being expressed in units that are about 1 MSS or smaller. So, WS 301 option values between 0 and 11 have no impact in rLEDBAT. 303 WS option values higher than 11 can affect the dynamics of rLEDBAT, 304 since control may become too coarse (e.g., with WS of 14, a change in 305 one unit of the receive window implies a change of 10 MSS in the 306 effective window). 308 For the above reasons, the rLEDBAT client SHOULD set WS option values 309 lower than 12. Additional experimentation is required to explore the 310 impact of larger WS values in rLEDBAT dynamics. 312 Note that the recommendation for rLEDBAT to set the WS option value 313 to lower values does not precludes the communication with servers 314 that set the WS option values to larger values, since the WS option 315 value used is set independently for each direction of the TCP 316 connection. 318 3.2. Measuring delays 320 Both LEDBAT and LEDBAT++ measure base and current delays to estimate 321 the queueing delay. LEDBAT uses the one way delay while LEDBAT++ 322 uses the round trip time. In the next sections we describe how 323 rLEDBAt mechanisms enable the receiver to measure the one way delay 324 or the round trip time, whatever needed depending on the congestion 325 control algorithm used. 327 3.2.1. Measuring the RTT to estimate the queueing delay 329 LEDBAT++ uses the round trip time (RTT) to estimate the queueing 330 delay. In order to estimate the queueing delay using the RTT, the 331 rLEDBAT receiver estimates the base RTT (i.e., the constant 332 components of the RTT) and also measures the current RTT. By 333 subtracting these two values, we obtain the queuing delay to be used 334 by the rLEDBAT controller. 336 LEDBAT++ discovers the base RTT (RTTb) by taking the minimum value of 337 the measured RTTs over a period of time. The current RTT (RTTc) is 338 estimated using a number of recent samples and applying a filter, 339 such as the minimum (or the mean) of the last k samples. Using the 340 RTT to estimate the queueing delay has a number of shortcomings and 341 difficulties that we discuss next. 343 The queuing delay measured using the RTT includes also the queueing 344 delay experienced by the return packets in the direction from the 345 rLEDBAT receiver to the sender. This is a fundamental limitation of 346 this approach. The impact of this error is that the rLEDBAT 347 controller will also react to congestion in the reverse path 348 direction which results in an even more conservative mechanism. 350 In order to measure the RTT, the rLEDBAT client MUST enable the Time 351 Stamp (TS) option [RFC7323]. By matching the TSVal value carried in 352 outgoing packets with the TSecr value observed in incoming packets, 353 it is possible to measure the RTT. This allows the rLEDBAT receiver 354 to measure the RTT even if it is acting as a pure receiver. In a 355 pure receiver there is no data flowing from the rLEDBAT receiver to 356 the sender, making impossible to match data packets with 357 acknowledgements packets to measure the RTT, as it is usually done in 358 TCP for other purposes. 360 Depending on the frequency of the local clock used to generate the 361 values included in the TS option, several packets may carry the same 362 TSVal value. If that happens, the rLEDBAT receiver will be unable to 363 match the different outgoing packets carrying the same TSVal value 364 with the different incoming packets carrying also the same TSecr 365 value. However, it is not necessary for rLEDBAT to use all packets 366 to estimate the RTT and sampling a subset of in-flight packets per 367 RTT is enough to properly assess the queueing delay. The RTT MUST 368 then be calculated as the time since the first packet with a given 369 TSVal was sent and the first packet that was received with the same 370 value contained in the TSecr. Other packets with repeated TS values 371 SHOULD NOT be used for the RTT calculation. 373 Several issues must be addressed in order to avoid an artificial 374 increase of the observed RTT. Different issues emerge depending 375 whether the rLEDBAT capable host is sending data packets or pure ACKs 376 to measure the RTT. We next consider the issues separately. 378 3.2.1.1. Measuring RTT sending pure ACKs 380 In this scenario, the rLEDBAT node (node A) sends a pure ACK to the 381 other endpoint of the TCP connection (node B), including the TS 382 option. Upon the reception of the TS Option, host B will copy the 383 value of the TSVal into the TSecr field of the TS option and include 384 that option into the next data packet towards host A. However, there 385 are two reasons why B may not send a packet immediately back to A, 386 artificially increasing the measured RTT. The first reason is when A 387 has no data to send. The second is when A has no available window to 388 put more packets in-flight. We describe next how each of these cases 389 is addressed. 391 The case where the host B has no data to send when it receives the 392 pure Acknowledgement is expected to be rare in the rLEDBAT use cases. 393 rLEDBAT will be used mostly for background file transfers so the 394 expected common case is that the sender will have data to send 395 throughout the lifetime of the communication. However, if, for 396 example, the file is structured in blocks of data, it may be the case 397 that seldom, the sender will have to wait until the next block is 398 available to proceed with the data transfer and momentarily lack of 399 data to send. To address this situation, the filter used by the 400 congestion control algorithm executed in the receiver SHOULD discard 401 the larger samples (e.g. a min filter would achieve this) when 402 measuring the RTT using pure ACK packets. 404 The limitation of available sender's window to send more packets can 405 come either from the TCP congestion window in host B or from the 406 announced receive window from the rLEDBAT in host A. Normally, the 407 receive window will be the one to limit the sender's transmission 408 rate, since the LBE congestion control algorithm used by the rLEDBAT 409 node is designed to be more restrictive on the sender's rate than 410 standard-TCP. If the limiting factor is the congestion window in the 411 sender, it is less relevant if rLEDBAT further reduces the receive 412 window due to a bloated RTT measurement, since the rLEDBAT is not 413 actively controlling the sender's rate. Nevertheless, the proposed 414 approach to discard larger samples would also address this issue. 416 To address the case in which the limiting factor is the receive 417 window announced by rLEDBAT, the congestion control algorithm at the 418 receiver SHOULD discard the RTT measurements done using pure ACK 419 packets while reducing the window and avoid including bloated samples 420 in the queueing delay estimation. The rLEDBAT receiver is aware 421 whether a given TSVal value was sent in a pure ACK packet where the 422 window was reduced, and if so, it can discard the corresponding RTT 423 measurement. 425 3.2.1.2. Measuring the RTT sending data packets 427 In the case that the rLEDBAT node is sending data packets and 428 matching them with pure ACKs to measure the RTT, a factor that can 429 artificially increase the RTT measured is the presence of delayed 430 Acknowledgements. According to the TS option generation rules 431 [RFC7323], the value included in the TSecr for a delayed ACK is the 432 one in the TSVal field of the earliest unacknowledged segment. This 433 may artificially increase the measured RTT. 435 If both endpoints of the connection are sending data packets, 436 Acknowledgments are piggybacked into the data packets and they are 437 not delayed. Delayed ACKs only increase the RTT measurement in the 438 case that the sender has no data to send. Since the expected use 439 case for rLEDBAT is that the sender will be sending background 440 traffic to the rLEDBAT receiver, the cases where delayed ACKs 441 increase the measured RTT are expected to be rare. 443 Nevertheless, for those measurements done using data packets sent by 444 the rLEDBAT node matching pure ACKs sent from the other endpoint of 445 the connection, they will result in an increased RTT. The additional 446 increase in the measured RTT will range between the transmission 447 delay of on packet and 500 ms. The reason for this is that delayed 448 ACKs are generated every second data packet received and not delayed 449 more than 500 ms according to [I-D.ietf-tcpm-rfc793bis]. The rLEDBAT 450 receiver MAY discard the RTT measurements done using data packets 451 from the rLEBDAT receiver and matching pure ACKs, especially if it 452 has recent measurements done using other packet combinations.Also, 453 applying a filter that discard larger samples would also address this 454 issue (e.g. a min filter). 456 3.2.2. Measuring one way delay to estimate the queueing delay 458 The LEDBAT algorithm uses the one-way delay of packets as input. A 459 TCP receiver can measure the delay of incoming packets directly (as 460 opposed to the sender-based LEDBAT, where the receiver measures the 461 one-way delay and needs to convey it to the sender). 463 In the case of TCP, the receiver can use the Time Stamp option to 464 measure the one way delay by subtracting the time stamp contained in 465 the incoming packet from the local time at which the packet has 466 arrived. As noted in [RFC6817] the clock offset between the clock of 467 the sender and the clock in the receiver does not affect the LEDBAT 468 operation, since LEDBAT uses the difference between the base one way 469 delay and the current one way delay to estimate the queuing delay, 470 effectively canceling the clock offset error in the queueing delay 471 estimation. There are however two other issues that the rLEDBAT 472 receiver needs to take into account in order to properly estimate the 473 one way delay, namely, the units in which the received timestamps are 474 expressed and the clock skew. We address them next. 476 In order to measure the one way delay using TCP timestamps, the 477 rLEDBAT receiver needs to discover the units in which the values of 478 the TS option are expressed and second, to account for the skew 479 between the two clocks of the endpoints of the TCP connection. Note 480 that a mismatch of 100 ppm (parts per million) in the estimation at 481 the receiver of the clock rate of the sender accounts for 6 ms of 482 variation per minute in the measured delay for a communication, just 483 one order of magnitude below the target set for controlling the rate 484 by rLEDBAT. Typical skew for untrained clocks is reported to be 485 around 100-200 ppm [RFC6817]. 487 In order to learn both the TS units and the clock skew, the rLEDBAT 488 receiver compares how much local time has elapsed between the sender 489 has issued two packets with different TS values. By comparing the 490 local time difference and the TS value difference, the receiver can 491 assess the TS units and relative clock skews. In order for this to 492 be accurate, the packets carrying the different TS values should 493 experience equal (or at least similar delay) when traveling from the 494 sender to the receiver, as any difference in the experienced delays 495 would introduce error in the unit/skew estimation. One possible 496 approach is to select packets that experienced the minimum delay 497 (i.e. close to zero queueing delay) to make the estimations. 499 An additional difficulty regarding the estimation of the TS units and 500 clock skew in the context of (r)LEDBAT is that the LEDBAT congestion 501 controller actions directly affect the (queueing) delay experienced 502 by packets. In particular, if there is an error in the estimation of 503 the TS units/skew, the LEDBAT controller will attempt to compensate 504 it by reducing/increasing the load. The result is that the LEDBAT 505 operation interferes with the TS units/clock skew measurements. 506 Because of this, measurements are more accurate when there is no 507 traffic in the connection (in addition to the packets used for the 508 measurements). The problem is that the receiver is unaware if the 509 sender is injecting traffic at any point in time, and 510 opportunistically seize quiet intervals to preform measurements. The 511 receiver can however, force periodic slowdowns, reducing the 512 announced receive window to a few packets and perform the 513 measurements then. 515 It is possible for the rLEDBAT receiver to perform multiple 516 measurements to assess both the TS units and the relative clock skew 517 during the lifetime of the connection, in order to obtain more 518 accurate results. Clock skew measurements are more accurate if the 519 time period used to discover the skew is larger, as the impact of the 520 skew becomes more apparent. Due to the same logic, accurately 521 learning the clock skew is more pressing as the time separating the 522 two delays to compare increases. It is a reasonable approach for the 523 rLEDBAT receiver to perform an early discovery of the TS units (and 524 the clock skew) using the first few packets of the TCP connection and 525 then improve the accuracy of the TS units/clock skew estimation using 526 periodic measurements later in the lifetime of the connection. 528 3.3. Detecting packet losses and retransmissions 530 The rLEDBAT receiver is capable of detecting retransmitted packets in 531 the following way. We call RCV.HGH the highest sequence number 532 correspondent to a received byte of data (not assuming that all bytes 533 with smaller sequence numbers have been received already, there may 534 be holes) and we call TSV.HGH the TSVal value corresponding to the 535 segment in which that byte was carried. SEG.SEQ stands for the 536 sequence number of a newly received segment and we call TSV.SEQ the 537 TSVal value of the newly received segment. 539 If SEG.SEQ < RCV.HGH and TSV.SEQ > TSV.HGH then the newly received 540 segment is a retransmission. This is so because the newly received 541 segment was generated later than another already received segment 542 which contained data with a larger sequence number. This means that 543 this segment was lost and was retransmitted. 545 The proposed mechanism to detect retransmissions at the receiver 546 fails when there are window tail drops. If all packets in the tail 547 of the window are lost, the receiver will not be able to detect a 548 mismatch between the sequence numbers of the packets and the order of 549 the timestamps. In this case, rLEDBAT will not react to losses but 550 the TCP congestion controller at the sender will, most likely 551 reducing its window to 1MSS and take over the control of the sending 552 rate, until slow start ramps up and catches the current value of the 553 rLEDBAT window. 555 4. Security Considerations 557 5. IANA Considerations 559 6. Acknowledgements 561 This work was supported by the EU through the H2020 5G-RANGE project 562 and by the Spanish Ministry of Economy and Competitiveness through 563 the 5G-City project (TEC2016-76795-C6-3-R). 565 7. Informative References 567 [I-D.balasubramanian-iccrg-ledbatplusplus] 568 Balasubramanian, P., Ertugay, O., and D. Havey, "LEDBAT++: 569 Congestion Control for Background Traffic", draft- 570 balasubramanian-iccrg-ledbatplusplus-01 (work in 571 progress), November 2019. 573 [I-D.ietf-tcpm-rfc793bis] 574 Eddy, W., "Transmission Control Protocol Specification", 575 draft-ietf-tcpm-rfc793bis-18 (work in progress), August 576 2020. 578 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 579 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 580 . 582 [RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind, 583 "Low Extra Delay Background Transport (LEDBAT)", RFC 6817, 584 DOI 10.17487/RFC6817, December 2012, 585 . 587 [RFC7323] Borman, D., Braden, B., Jacobson, V., and R. 588 Scheffenegger, Ed., "TCP Extensions for High Performance", 589 RFC 7323, DOI 10.17487/RFC7323, September 2014, 590 . 592 [RFC8312] Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and 593 R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", 594 RFC 8312, DOI 10.17487/RFC8312, February 2018, 595 . 597 Authors' Addresses 599 Marcelo Bagnulo 600 UC3M 602 Email: marcelo@it.uc3m.es 604 Alberto Garcia-Martinez 605 UC3M 607 Email: alberto@it.uc3m.es 609 Gabriel Montenegro 610 Microsoft 612 Email: Gabriel.Montenegro@microsoft.com 614 Praveen Balasubramanian 615 Microsoft 617 Email: pravb@microsoft.com