idnits 2.17.1 draft-fairhurst-quic-ack-scaling-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 15, 2021) is 1135 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-06) exists of draft-gomez-tcpm-ack-rate-request-01 Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force G. Fairhurst 3 Internet-Draft A. Custura 4 Intended status: Informational T. Jones 5 Expires: September 16, 2021 University of Aberdeen 6 March 15, 2021 8 Changing the Default QUIC ACK Policy 9 draft-fairhurst-quic-ack-scaling-04 11 Abstract 13 Acknowledgement packets (ACKs) are used by transport protocols to 14 confirm the delivery of packets, and their reception is used in a 15 variety of other ways (to measure path round trip time, to gauge path 16 congestion, etc). However, the transmission of ACKs also consumes 17 resources at the receiver, forwarding resource in the network and 18 processing resources at the sender. 20 On network paths with significant path asymmetry, transmission of 21 ACKs can limit the available throughput or can reduce the efficient 22 use of network capacity. This occurs when the return capacity is 23 significantly more constrained than the forward capacity, and/or the 24 cost of transmission per packet is a significant component of the 25 total transmission cost. In these cases, reducing the ratio of ACK 26 packets to data packets can improve link utilisation and reduce link 27 transmission costs. It can also reduce processing overhead at the 28 sender and receiver. 30 This document proposes a change to the default acknowledgement policy 31 of the QUIC transport protocol to improve performance over paths with 32 appreciable asymmetry. 34 Status of This Memo 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF). Note that other groups may also distribute 41 working documents as Internet-Drafts. The list of current Internet- 42 Drafts is at https://datatracker.ietf.org/drafts/current/. 44 Internet-Drafts are draft documents valid for a maximum of six months 45 and may be updated, replaced, or obsoleted by other documents at any 46 time. It is inappropriate to use Internet-Drafts as reference 47 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on September 16, 2021. 50 Copyright Notice 52 Copyright (c) 2021 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (https://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 68 2. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 3 69 2.1. ACK Ratio Impact on Asymmetric Paths . . . . . . . . . . 3 70 2.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 71 2.3. Adapting the ACK Ratio in Current Transport 72 Specifications . . . . . . . . . . . . . . . . . . . . . 4 73 2.4. Adapting the TCP ACK Ratio in the Network . . . . . . . . 5 74 2.5. QUIC ACKs . . . . . . . . . . . . . . . . . . . . . . . . 5 75 3. Updating the Default ACK Ratio for QUIC . . . . . . . . . . . 6 76 3.1. Considerations Updating the Default QUIC ACK Policy . . . 6 77 3.2. Mitigating the Impact of ACKs during Slow Start . . . . . 7 78 3.3. ACKs after Slow Start . . . . . . . . . . . . . . . . . . 8 79 3.4. ACKs after re-ordering is Detected . . . . . . . . . . . 8 80 3.5. Further Tuning of the ACK Policy . . . . . . . . . . . . 8 81 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 82 5. Security Considerations . . . . . . . . . . . . . . . . . . . 9 83 6. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 84 6.1. Normative References . . . . . . . . . . . . . . . . . . 9 85 6.2. Informative References . . . . . . . . . . . . . . . . . 9 86 Appendix A. Summary of Recommended ACK Policy for TCP . . . . . 11 87 Appendix B. Understanding ACKs in Slow Start . . . . . . . . . . 13 88 Appendix C. Experiments Exploring an ACK Ratio of 1:10 . . . . . 16 89 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 18 91 1. Introduction 93 Acknowledgement packets (ACKs) are used by transport protocols to 94 confirm the delivery of packets, and their reception is used in a 95 variety of other ways (to measure path round trip time, to gauge path 96 congestion, etc) [Cust20]. However, the transmission of ACKs also 97 consumes resources at the receiver, forwarding resource in the 98 network and processing resources at the sender. 100 The current design of QUIC [I-D.ietf-quic-transport] currently 101 proposes a default acknowledgement (ACK) ratio of 1:2 (at least one 102 ACK for every 2 ack-eliciting packets) inspired by current 103 recommendations for TCP, see Appendix A. 105 This document motivates an increase in the ratio of ACK packets to 106 data packets from 1:2 to 1:10 for QUIC flows. 108 2. Motivation 110 When the characteristics of the forward and return paths are not 111 symmetric [RFC3449], the transmission of ACKs can adversely impact 112 either transport performance and/or the cost of sending data across a 113 link. 115 2.1. ACK Ratio Impact on Asymmetric Paths 117 TCP Performance Implications of Network Path Asymmetry [RFC3449] 118 describes a series of problems and mitigations when transports use an 119 asymmetric path. Performance problems arise in several access 120 networks, including bandwidth-asymmetric networks (such as broadband 121 satellite access, DOCSIS cable networks, cellular mobile, WiFi, etc) 122 [Cust20]. 124 Where the ACK rate is limited by the capacity of the return path, 125 this constrains the maximum throughput for the forward path. ACK 126 traffic also competes for capacity and/or transmission opportunities 127 with other traffic that shares a constrained return path. This 128 motivates a need to reduce the volume of ACK traffic (increase the 129 number of segments/packets that are acknowledged by each ACK). 131 Capacity is not the only asymmetric path constraint. Sending ACKs 132 can consume significant transmission resources and the cost of 133 transmitting ACKs can become a significant part of the cost of 134 transmission when using a network segment. In many wireless 135 technologies, there is appreciable overhead for the transmission of 136 each packet burst (data and ACK). There can also be associated costs 137 (e.g., in radio resource management and transmission scheduling) that 138 are often different for the forward and return paths because they use 139 different technologies or configurations. 141 These provides an incentive to reduce the rate of ACK traffic 142 generated by a transport protocol. 144 2.2. Terminology 146 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 147 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 148 "OPTIONAL" in this document are to be interpreted as described in BCP 149 14 [RFC2119] [RFC8174] when, and only when, they appear in all 150 capitals, as shown here. 152 2.3. Adapting the ACK Ratio in Current Transport Specifications 154 We define the ACK Ratio as the ratio between the number of packets 155 that are received by a transport endpoint (on the forward path) and 156 the number of ACK packets that are returned (on the return path). A 157 simple protocol may send an ACK for each data packet, resulting in an 158 ACK Ratio of 1:1. One that acknowledges alternate data packets has a 159 ratio of 1:2. Most IETF-defined protocols specify a default ACK 160 Ratio. Notes on the use of ACKs in TCP and the resulting ACK Ratio 161 are provided in Appendix A. 163 Various methods have been proposed to modify the ACK Ratio used by 164 transport protocols. Two examples follow, based on a receiver 165 understanding whether the return path has become congested: 167 Some proposed methods allow TCP to dynamically adjust the AR (e.g., 168 [Lan08]). ACK-CC [RFC5690] proposed a method to control the rate of 169 ACKs to avoid the return path from becoming congested, but this did 170 not achieve wide-scale deployment. 172 The Datagram Congestion Control Protocol (DCCP) [RFC4340] has methods 173 to control the ACK ratio at the receiver. DCCP specifies a TCP- 174 friendly congestion control [RFC4341], which includes the ability to 175 use signalling that allows this sender to adjust the receiver ACK 176 ratio within certain parameters. This also was not widely used. 178 A proposed TCP option [I-D.gomez-tcpm-ack-rate-request] would allow a 179 sender to set (or request) a change in the ACK policy use by the 180 remote TCP receiver, enabling the ACK ratio to be adapted from the 181 default. 183 These methods are suggested to help when there is congestion on the 184 return path. However, end-to-end transport methods do not have a way 185 to detect and account for the cost of ACK transmission on a link 186 along the return path, and are difficult to tune to adapt to delays 187 introduced by links (e.g., the variations produced by radio resource 188 management). Transport adaptation therefore can only provide a 189 partial mitigation to the impacts when sending ACKs over an 190 asymmetric paths. 192 2.4. Adapting the TCP ACK Ratio in the Network 194 An alternative approach has been deployed for TCP that uses a 195 middlebox in the network to "thin" the rate of ACKs. This method is 196 used with paths that exhibit significant ACK asymmetry to improve 197 performance of TCP. They can modify the ACK Ratio experienced by a 198 network link from the default TCP ACK Ratio of 1:2 [RFC3449]. 199 Performance Enhancing Proxies (PEPs) implement these functions when 200 they detect/know an upstream link is (or is likely to be) filled with 201 TCP ACKs, or on cross-layer information provided by the physical 202 layer. 204 Removing redundant TCP ACKs (also known as "ACK Thinning") leads to 205 stretch ACKs (where a single ACK acknowledges more than two TCP 206 segments). Stretch ACKs have been observed on Internet paths [All98] 207 and are are now common [Din15] for various reasons. 209 The introduction of TCP Appropriate Byte Counting (ABC) [RFC3465] 210 mostly mitigates the impact of stretch ACKs, and also recommends 211 burst mitigation techniques at a TCP sender. 213 2.5. QUIC ACKs 215 QUIC, like other transports, generates ACK information and sends this 216 in QUIC packets on the return path. 218 ACK Thinning methods can only be used when ACKs can be observed by a 219 network device on the path. In contrast, QUIC uses an encrypted 220 feedback packet to communicate an ACK. This use of encryption 221 intentionally prevents such in-network optimisations. 223 A typical QUIC ACK is around 25% larger than a corresponding TCP ACK, 224 and can still be larger when there is loss/reordering. This means 225 additional processing overhead and link usage for all Internet paths, 226 with a significant impact on asymmetric links, where this can also 227 limit throughput: 229 o The smallest IPv4 TCP ACK is 20 bytes, resulting in a 40 byte IPv4 230 TCP ACK packet, although this could be up to 40 bytes larger when 231 TCP options are present. 233 o The TCP ACK size often increases following a reordering event due 234 to the presence of SACK option blocks. The maximum size for IPv4 235 is max 60 bytes. 237 o The smallest IPv4 QUIC ACK packet is 51 bytes. The ACK frame is 238 only 4 bytes, but is sent using a QUIC packet. For IPv4 this is: 239 20 (IP) + 8 (UDP) + 3 (1+1+1 QUIC header ... Assuming a 1 B CID) + 240 4 (minimum ACK frame size) + 16 (crypto overhead). Other types of 241 frames may also be sent in the same packet, increasing this size. 242 The QUIC ACK packet size also becomes larger for very long 243 connections due to the varint encoding. 245 o The packet size for an IPv4 QUIC ACK increases after a reordering 246 event due to the inclusion of ACK range information. The maximum 247 size of this information is limited to the size of a QUIC packet. 248 Implementation may further limit the size (often observed as 100's 249 of bytes depending on the pattern of loss/reordering). 251 Compared to TCP, the performance of QUIC is therefore disadvantaged 252 when QUIC uses an ACK Ratio of 1:2. 254 3. Updating the Default ACK Ratio for QUIC 256 Any ACK policy that changes the ACK ratio from 1:2 needs to consider 257 three issues: 259 o A reduced frequency of feedback can increase the time to detect 260 loss, impacting the loss recovery algorithm, and potentially 261 leading to cases of spurious retransmission. QUIC mitigates this 262 by using timer-based retransmission (using the PTO). 264 o A reduced frequency of feedback can increase the time to detect 265 and react to congestion, impacting the congestion control 266 algorithm. The speed of loss detection can be impacted by delayed 267 acknowledgements. Timer-based methods (e.g., using the QUIC Probe 268 Timeout (PTO), see section 5.1 of [I-D.ietf-quic-recovery]) can 269 reduce the reliance on ACKs to detect loss. 271 o A reduced ACK rate can lead to bursts of acknowledged packets, and 272 introduces a need for burst mitigation at the sender. 274 3.1. Considerations Updating the Default QUIC ACK Policy 276 A QUIC receiver can generate one ACK frame for every received ack- 277 eliciting packet, but normally aggregates the ACK information into a 278 cumulative ACK. The QUIC transport protocol currently recommends a 279 default ACK Ratio of 1:2 [I-D.ietf-quic-transport]. Additionally, 280 QUIC relies on pacing, rather than ACK-Clocking as a burst 281 mitigation. Hence reception of larger cumulative ACKs does not 282 normally have a significant impact on the sender's traffic. 284 Since the introduction of the specification to allow a larger TCP 285 Initial Window (IW) [RFC6928], there has been deployment experience 286 using TCP with an IW of 10 segments at startup. QUIC continues this 287 practice, which in part motivates the need for a QUIC transport 288 protocol to operate when a burst of acknowledgements for up to ten 289 packets is received. QUIC transport recommends the use of pacing to 290 mitigate packet bursts generated by a sender (see section 6.8 of 291 [I-D.ietf-quic-recovery]) 293 This document proposes changing the QUIC behaviour to send an ACK for 294 at least every 10 received packets. Further background to the 295 proposed method is detailed in the Annexe (Appendix C). 297 The QUIC transport protocol also currently specifies a maximum ACK 298 Delay, which is communicated by the sender at connection setup to 299 indicate the maximum time an endpoint will delay sending 300 acknowledgements. A default of 25 milliseconds is recommended 301 [I-D.ietf-quic-transport]. The ACK Delay timer ensures ACKs are not 302 unduly delayed (e.g., when data packets are spaced in time, or at the 303 end of a packet burst). The effect of a large delay could be 304 significant when a stretch ACK acknowledges more QUIC packets. 306 When the ACK Ratio is increased, an appropriate choice of the maximum 307 ACK Delay becomes more important - because it could otherwise 308 withhold ACK information for a time long enough to impact protocol 309 performance or operation. The proposed therefore method also 310 triggers ACK feedback at least four times per RTT (this additional 311 rule is less important when the RTT is greater than 100ms and the ACK 312 Delay forms a small part of the total RTT). 314 3.2. Mitigating the Impact of ACKs during Slow Start 316 A sender during slow start is often cwnd-limited, so any additional 317 delay in returning an ACK has the effect of increasing the duration 318 of the slow-start phase (see Appendix B). The requested change is: 320 The receiver can provide a higher rate of acknowledge for the 321 first 100 ACK-eliciting packets, where it acknowledges at least 322 every second received ACK-eliciting packet. A suitable method 323 might send an ACK frame for every two received ack-eliciting 324 packets for the first 100 received packets if max_ack_delay time 325 has passed since the oldest unacknowledged data was received. 327 NOTE: The original design of TCP used each ACK as a trigger to 328 increase the congestion window, motivating an initial ACK Ratio of 329 1:1 (as in DAAS Appendix A). 331 NOTE: A receiver typically has no understanding of the sender's 332 congestion control state, so the number 100 reflects a trade-off, 333 corresponding to an appreciable opening of the sender's congestion 334 window. 336 NOTE: A congestion controller typically re-enters slow start after 337 congestion is detected, if the congestion window is collapsed to a 338 small value, this could motivate again using this method. However, 339 the receiver is typically unaware of this event, and we do not 340 propose this is considered. 342 3.3. ACKs after Slow Start 344 We recommend an ACK frame should be generated for at least every 345 tenth ack-eliciting packet. The requested change is: 347 The receiver SHOULD send an ACK frame if a period more than 348 MIN(max_ack_delay,min_rtt/4) has passed since receiving the oldest 349 unacknowledged data or when the received has accumulated at most 350 10 unacknowledged ack-eliciting packets. 352 NOTE: The maximum of receiving not more than 10 ack-eliciting packets 353 is derived from the recommended TCP Initial Window [RFC6928]. 355 NOTE: This utilises the receiver's estimated min_rtt, when this is 356 known. 358 3.4. ACKs after re-ordering is Detected 360 Prompt and reliable communication of ACK ranges after loss is 361 important for efficient loss recovery. This suggests that additional 362 ACKs may be needed after a reordering event to protect the sender 363 from potential ACK loss in the return direction. However, such ACKs 364 also consume return path capacity and the number of additional 365 packets needs to be considered. ( [I-D.gomez-tcpm-ack-rate-request] 366 proposes that the sender also controls parameters that detect 367 reordering.) 369 Following detected loss or congestion, a receiver sends ACKs 370 according to section 13.2.1 of QUIC transport 371 [I-D.ietf-quic-transport]. 373 NOTE: A future recommendation could recommend reducing the ACK 374 frequency after loss/re-ordering. 376 3.5. Further Tuning of the ACK Policy 378 In situations where the default method is not sufficient, the ACK 379 Ratio might be further tuned by server, as described in 380 [I-D.iyengar-quic-delayed-ack]. This could permit the ACK method to 381 be adapted to match the behaviour of a new congestion control 382 algorithm. Reducing the rate of ACKs can also lower the 383 computational effort required to process ACKs at the sender and 384 receiver, important for some high speed connections. For instance, 385 this could reduce the workload for high speed network interfaces by 386 reducing the rate of cache ejection for Generic Receiver Offload 387 (GRO). 389 The change to the default motivated in this document is not related 390 to such further tuning of the ACK policy. 392 4. IANA Considerations 394 This memo includes no request to IANA. 396 5. Security Considerations 398 The security considerations for the QUIC transport protocol are 399 described in [I-D.ietf-quic-transport]. 401 6. References 403 6.1. Normative References 405 [I-D.ietf-quic-recovery] 406 Iyengar, J. and I. Swett, "QUIC Loss Detection and 407 Congestion Control", draft-ietf-quic-recovery-34 (work in 408 progress), January 2021. 410 [I-D.ietf-quic-transport] 411 Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed 412 and Secure Transport", draft-ietf-quic-transport-34 (work 413 in progress), January 2021. 415 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 416 Requirement Levels", BCP 14, RFC 2119, 417 DOI 10.17487/RFC2119, March 1997, 418 . 420 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 421 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 422 May 2017, . 424 6.2. Informative References 426 [All98] Allman, M., "On the Generation and use of TCP 427 Acknowledgments; ACM SIGCOMM CCR, 28 (5) p4-21.", October 428 1998. 430 [Cust20] Custura, A., Jones, T., and G. Fairhurst, "Rethinking ACKs 431 at the Transport Layer; FIT Workshop, IEEE IFIP 432 Networking", June 2020. 434 [Din15] Ding, H., "TCP Stretch Acknowledgements and Timestamps, 435 Findings and Implications for Passive RTT Measurement; ACM 436 SIGCOMM CCR, 45 (3) p20-27.", July 2015. 438 [I-D.gomez-tcpm-ack-rate-request] 439 Gomez, C. and J. Crowcroft, "TCP ACK Rate Request Option", 440 draft-gomez-tcpm-ack-rate-request-01 (work in progress), 441 November 2020. 443 [I-D.iyengar-quic-delayed-ack] 444 Iyengar, J. and I. Swett, "Sender Control of 445 Acknowledgement Delays in QUIC", draft-iyengar-quic- 446 delayed-ack-02 (work in progress), November 2020. 448 [Lan08] Landstroem, S., "TCP/IP Technology for Modern Network 449 Environments; PhD Thesis", 2008. 451 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 452 Communication Layers", STD 3, RFC 1122, 453 DOI 10.17487/RFC1122, October 1989, 454 . 456 [RFC2525] Paxson, V., Allman, M., Dawson, S., Fenner, W., Griner, 457 J., Heavens, I., Lahey, K., Semke, J., and B. Volz, "Known 458 TCP Implementation Problems", RFC 2525, 459 DOI 10.17487/RFC2525, March 1999, 460 . 462 [RFC3449] Balakrishnan, H., Padmanabhan, V., Fairhurst, G., and M. 463 Sooriyabandara, "TCP Performance Implications of Network 464 Path Asymmetry", BCP 69, RFC 3449, DOI 10.17487/RFC3449, 465 December 2002, . 467 [RFC3465] Allman, M., "TCP Congestion Control with Appropriate Byte 468 Counting (ABC)", RFC 3465, DOI 10.17487/RFC3465, February 469 2003, . 471 [RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram 472 Congestion Control Protocol (DCCP)", RFC 4340, 473 DOI 10.17487/RFC4340, March 2006, 474 . 476 [RFC4341] Floyd, S. and E. Kohler, "Profile for Datagram Congestion 477 Control Protocol (DCCP) Congestion Control ID 2: TCP-like 478 Congestion Control", RFC 4341, DOI 10.17487/RFC4341, March 479 2006, . 481 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 482 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 483 . 485 [RFC5690] Floyd, S., Arcia, A., Ros, D., and J. Iyengar, "Adding 486 Acknowledgement Congestion Control to TCP", RFC 5690, 487 DOI 10.17487/RFC5690, February 2010, 488 . 490 [RFC6928] Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis, 491 "Increasing TCP's Initial Window", RFC 6928, 492 DOI 10.17487/RFC6928, April 2013, 493 . 495 Appendix A. Summary of Recommended ACK Policy for TCP 497 A TCP sender regards reception of a TCP ACK as a positive indication 498 that data has been received across the path. The congestion control 499 algorithm uses this to increase the size of the congestion window, 500 cwnd RFC 5681 [RFC5681]. 502 To reduce the ACK Rate, a receiver can delay sending an ACK for a 503 period of time called the ACK Delay. This can increase network 504 efficiency. 506 RFC 5681 [RFC5681] clarifies the TCP ACK frequency described in 507 RFC 1122 [RFC1122]. This recommends that a receiver generates an ACK 508 corresponding to every second maximum segment size, MSS, of received 509 data, Section 4.2 of RFC 5681 [RFC5681], specifies the ACK policy for 510 a TCP receiver: "an ACK SHOULD be generated for at least every second 511 full-sized segment, and MUST be generated within 500 ms of the 512 arrival of the first unacknowledged packet". However, RFC 3449 513 [RFC3449] also notes the need for, and deployment of, methods to 514 further reduce the number of TCP ACKs in networks with asymmetric 515 paths. 517 When a receiver decides to delay ACKs, this can reduce the rate of 518 growth of the cwnd. 520 Some congestion controllers can benefit from frequent feedback during 521 an initial slow start period, where the sender is probing for 522 available path capacity. When a TCP sender uses each ACK to increase 523 the cwnd, this directly impacts the cwnd growth. TCP implementations 524 often use heuristics such as DAAS (Delayed ACK after Slow Start) to 525 mitigate this. This allows the receiver to estimate when the sender 526 could be in the slow start phase of cwnd growth, and for a period of 527 time sends an ACK for each received segment/packet (i.e., an ACK 528 Ratio of 1:1). In contrast, a TCP congestion controller can increase 529 its congestion window based on the number of bytes acknowledged, not 530 the number of ACK packets. (QUIC chooses this approach). 532 A delayed ACK can also increase the time to complete slow start, by 533 introducing an additional delay when returning ACKs. This effect is 534 different to a direct change to the cwnd, but never-the-less can 535 impact the performance, especially over paths where the ACK Delay 536 period is comparable to the RTT. 538 RFC 3465 [RFC3465] further discusses the issue of bursts that may be 539 caused by the interaction between ACK processing and congestion 540 control. This motivates a need to deal with bursts within TCP. TCP 541 senders can mitigate bursts by using Appropriate Byte Counting (ABC), 542 which increases the congestion window in proportion to the amount of 543 data sent into the network, rather than upon the arrival of each ACK. 544 (This also defends against ACK Splitting, where multiple ACKs are 545 received for parts of the same segment/packet). 547 ACKs can be lost on the return path (either through packet loss, or 548 by intentional thinning of the ACK stream). If a sender does not 549 receive an ACK for every second segment, a stretch ACK has occurred, 550 similar to using a larger ACK Ratio. RFC2525 [RFC2525] describes the 551 significance of stretch ACK violations: 553 "This behavior will cause TCP senders to generate burstier 554 traffic, which can degrade performance in congested environments. 555 In addition, generating fewer ACKs increases the amount of time 556 needed by the slow start algorithm to open the congestion window 557 to an appropriate point, which diminishes performance in 558 environments with large bandwidth-delay products. Finally, 559 generating fewer ACKs may cause needless retransmission timeouts 560 in lossy environments, as it increases the possibility that an 561 entire window of ACKs is lost, forcing a retransmission timeout." 563 There are other scenarios where a change to the TCP ACK policy could 564 have improved performance. However, the design of TCP, and 565 ossification of the protocol has made it hard for new mechanisms to 566 be deployed. QUIC does not suffer from these design constraints. 568 Appendix B. Understanding ACKs in Slow Start 570 This section provides diagrams to help explain the way ACKs are using 571 during slow start. This assumes the sender uses volume-based methods 572 to increase cwnd, as in QUIC. 574 Some congestion controllers can benefit from frequent feedback during 575 an initial slow start period, where the sender is probing for 576 available path capacity (see Appendix A). Since a QUIC CC is 577 expected to work in terms of the volume of data acknowledged, rather 578 than the number of ACKs received, this is not expected to be an 579 issue. 581 Consider a burst of 10 packets sent over a forward path. If the path 582 RTT is shorter than the ACK Delay value, then all packets are 583 cumulatively acknowledged by a single ACK. This is therefore the ACK 584 pattern with an ACK Ratio of 1:10, where 10 packets are sent as a 585 burst: 1 2 ... 10. 587 1 2 ... 10 588 \ \ \ \ / 589 \ \ \ \ / 590 \ \ \ \ / 591 \ \ \ \ / 592 \ \ \ \ / 593 \ \ \ \ / 594 \ \ \ \ / 595 \ \ \ \ / 596 \ \ \ \ / 597 \ \ \ \ / 598 \ \ \ \ / 599 \ \ \ X 600 ACK 602 Figure B1: Burst; ACK Delay 25ms; RTT 10ms; ACK Ratio 1:10. 604 The single ACK both cumulatively acknowledges all the data, and 605 increases the cwnd corresponding to reception of the 10 packets. An 606 ACK policy that generates more frequent ACKs (e.g., a lower ACK Delay 607 or ACK Ratio 1:1 or 1:2, would send multiple ACKs when receiving the 608 10 packets, the duration of the ACK burst is the same as the duration 609 of the transmission burst: 1 2 ... 10. 611 1 2 ... 10 612 \ \ \ \ / ..... / 613 \ \ \ \ / / 614 \ \ \ \ / / 615 \ \ \ \ / / 616 \ \ \ \ / / 617 \ \ \ \ / / 618 \ \ \ \ / / 619 \ \ \ X / 620 \ \ \ / \ / 621 \ \ X \ / 622 \ \ / \ \ / 623 \ X \ X 624 ACK ACK 626 Figure : Burst; ACK Delay 25ms; RTT 10ms; ACK Ratio 1:2. 628 The result of using this ACK policy is that it reduces the time to 629 the first ACK by about the burst size. This is not usually a 630 significant benefit, because often the RTT is greater than the burst 631 size. A similar result occurs when ACKs are generated by either the 632 ACK Ratio or the ACK Delay timer. This also has an advantage that it 633 generates more than one ACK, preventing progress being a hostage to 634 fortune on a single ACK. QUIC recommends pacing. If the sender were 635 to pace the 10 packets over the RTT, and the receiver ACK Ratio was 636 1:1, then this is the ACK pattern for the 10 packets paced over the 637 RTT: 639 1 .............. 9 10 640 \ \ \ \ \ \ \ / \/ \/ \/ / / / / / 641 \ \ \ \ \ \ X /\ /\ /\ / / / / / 642 \ \ \ \ \ \ / \ / \/ \/ X / / / / 643 \ \ \ \ \ X X /\ /\ / \ / / / / 644 \ \ \ \ \/ \ / \ / \/ X X / / / 645 \ \ \ \ /\ X X /\ / \ / \ / / / 646 \ \ \ \/ \/ \ / \ / X X X / / 647 \ \ \ /\ /\ X X / \ / \ / \ / / 648 \ \ \/ \/ \/ \ / \/ X X \/ / 649 \ \ /\ /\ /\ X /\ / \ / \ /\ / 650 \ \/ \/ \/ \/ \/ \/ X \/ X / 651 \ /\ /\ /\ /\ /\ /\ / \ /\ / \/ 652 ACK ACK ACK ..................... ACK 654 Figure B2: Pacing; ACK Delay not relevant; RTT 10ms; ACK Ratio 1:1. 656 The time to receive the first ACK is similar to that without pacing, 657 and all later ACKs arrive paced. For the same path (RTT shorter than 658 the ACK delay), but with the ACK Ratio set to 1:10, then the pattern 659 for 10 packets paced over an RTT is: 661 1 2 3 4 5 6 7 8 9 10 11 12 13 662 \ \ \ \ \ \ \ \ \ \ X \ \ 663 \ \ \ \ \ \ \ \ \ \ / \ \ \ 664 \ \ \ \ \ \ \ \ \ \ / \ \ \ 665 \ \ \ \ \ \ \ \ \ \ / 666 \ \ \ \ \ \ \ \ \ \ / 667 \ \ \ \ \ \ \ \ \ \ / 668 \ \ \ \ \ \ \ \ \ \ / 669 \ \ \ \ \ \ \ \ \ \ / 670 \ \ \ \ \ \ \ \ \ \ / 671 \ \ \ \ \ \ \ \ \ \ / 672 \ \ \ \ \ \ \ \ \ X 673 ACK 675 Figure B3: Pacing; ACK Delay 25ms; RTT 25ms; ACK Ratio 1;10. 677 The effect is similar to a very large ACK Delay. The time to receive 678 the first ACK is increased by an RTT (because that was the pacing 679 target). This causes the sender to wait an additional RTT (during 680 which the sender becomes idle) before it receives an ACK for the 681 series of 10 packets, and resumes pacing new data (at the new higher 682 rate, following the ACK). Because this increases the time for 683 receiver to send the first ACK, it therefore slows the window growth. 684 If the path is much longer than the ACK delay, this pathology does 685 not occur. The patterns of ACKs is primarily the result of the ACK 686 Delay setting. The first ACK is returned after waiting for the ACK 687 Delay, and arrives shortly after one RTT: 689 1 .............. 9 10 690 \ \ \ \ \ \ \ / \/ \/ \/ / / / / / 691 \ \ \ \ \ \ X /\ /\ /\ / / / / / 692 \ \ \ \ \ \ / \ / \/ \/ X / / / / 693 \ \ \ \ \ X X /\ /\ / \ / / / / 694 \ \ \ \ \/ \ / \ / \/ X X / / / 695 \ \ \ \ /\ X X /\ / \ / \ / / / 696 \ \ \ \/ \/ \ / \ / X X X / / 697 \ \ \ /\ /\ X X / \ / \ / \ / / 698 \ \ \/ \/ \/ \ / \/ X X \/ / 699 \ \ /\ /\ /\ X /\ / \ / \ /\ / 700 \ \/ \/ \/ \/ \/ \/ X \/ X / 701 \ /\ /\ /\ /\ /\ /\ / \ /\ / \/ 702 >ACK >ACK >ACK ................. >ACK 704 Figure B4: Pacing; ACK Delay 25ms; RTT 650ms; ACK Ratio - not 705 relevant. 707 The window growth is slowed only by the additional time of the ACK 708 delay period. In this case the role of the ACK Ratio becomes 709 apparent only as the rate increases so that multiple ACKs are 710 received with the ACK Delay interval. 712 Although QUIC uses cumulative ACKs to inflate the cwnd, and does not 713 rely upon ACK-clocking to time the transmission of new packets, it is 714 still influenced by the rate of ACKs in the time it is building the 715 congestion window. This could be mitigated by choosing an 716 appropriate ACK Delay, relative to the RTT, but the RTT is often 717 unknown at the time when ACK Delay is negotiated. Another way to 718 mitigate this is to ensure sufficient ACKs are sent for the first "n" 719 packets. 721 For example, setting the ACK Ratio as 1:2 for the first 100 received 722 packets. A value of 1:2 is expected to be a reasonable compromise 723 between reducing delay and conserving return path capacity, since 724 QUIC uses volume-based accounting to increase cwnd, it does not need 725 to ACK each received packet as in DAAS. 727 Appendix C. Experiments Exploring an ACK Ratio of 1:10 729 We used an experimental approach to examine a change to QUIC's ACK 730 Policy . This 731 section provides a few of the many results. These experiments were 732 performed in January 2020 based on available implementations at that 733 time. 735 Our tests show that the proposed change to the ACK Ratio did not 736 negatively impact the protocol. It reduced the amount of IP, UDP and 737 ACK overhead by a factor of approximately 5. The implemented 738 congestion control, was also not negatively impacted. Unlike TCP, 739 QUIC sends other types of data frames in addition to ACK frames, 740 increasing the total overhead on the return path. On asymmetrical 741 paths an ACK Ratio of 1:10 may still reduce the ACK traffic, helping 742 to avoid return path capacity limits impacting the ability to use the 743 forward path capacity. 745 Figure 1 presents a table with a set of asymmetric scenarios. The 746 columns present the rate of ACK traffic required (in kbps) to fill 747 each of the forward paths. The table shows the results for TCP 748 (without a PEP), both for lossless communication. It considers the 749 period after loss, when ACKs communicate the loss information. It 750 also shows the impact of using an ACK Ratio of 1:10 with QUIC. 752 An ACK Ratio of 1:10 reduces the utilisation of the return path. 753 Scenarios where the ACK traffic exceeds the return link capacity 754 (i.e. where this limits the forward path capacity that can be used) 755 are marked with a star. Note that the QUIC figure does not include 756 the encryption overhead, which would be dependent on the ciphers 757 chosen. This would add several additional bytes for every QUIC 758 packet. 760 +-------------------+-------------+---------------+-----------------+ 761 | | 10/2 Mbps | 50/10 Mbps | 250/3 Mbps | 762 +-------------------+-------------+---------------+-----------------+ 763 | TCP no loss | 133 - 346 | 650 -1,730 | 3250 - 8,650* | 764 | TCP loss | 346 - 560 | 1,730 - 2,800 | 8,650 - 14,000* | 765 | QUIC 1:2 no loss | 144 - 438 | 720 - 2,190 | 3,600 - 10,950* | 766 | QUIC 1:2 loss | 290 - * | 1450 - * | 7,250 - * | 767 | QUIC 1:10 no loss | 28.8 - 87.6 | 144 - 438 | 720 - 2,190 | 768 +-------------------+-------------+---------------+-----------------+ 770 Figure 1: ACK traffic required to fill the forward path in different 771 loss and asymmetry scenarios. The QUIC figures do not include 772 encryption overhead. 774 Figure 2 presents a table with the numbers of packets sent by two 775 QUIC implementations using a 1:2 and a 1:10 ACK Ratio. This shows 776 between a four and five time reduction in the number of packets sent. 778 +-------------------------------+----------------+----------------+ 779 | | Chromium | Quicly | 780 | | Draft 23 | Draft 23 | 781 +-------------------------------+----------------+----------------+ 782 | Packets sent | 77,419 | 83,238 | 783 | Packets on return path (1:2) | 39,089 (50.4%) | 41,108 (49.3%) | 784 | Packets on return path (1:10) | 10,409 (13.4%) | 9650 (11.5%) | 785 +-------------------------------+----------------+----------------+ 787 Figure 2: Number of packets sent and received during a 100MB QUIC 788 transfer using different ACK ratios, for two implementations 790 Figure 3 presents a table with the number of bytes sent by one of the 791 QUIC implementations using a 1:2 and a 1:10 ACK Ratio. This shows a 792 reduction in the number of bytes on the return path from 2.7 to 0.7% 793 of the total bytes sent. In these scenarios, this is sufficient to 794 take full advantage of the forward path capacity. 796 +---------------------------------+----------------+ 797 | | Chromium | 798 | | Draft 23 | 799 +---------------------------------+----------------+ 800 | Bytes sent | 110M | 801 | Bytes on return path (1:2 AR) | 3,056KB (2.7%) | 802 | Bytes on return path (1:10 AR) | 810KB (0.7%) | 803 +---------------------------------+----------------+ 805 Figure 3: Number of bytes sent and received during a 100MB Chromium 806 QUIC transfer using different ACK ratios 808 The number of bytes and packets reduces, as expected, when using an 809 ACK Ratio of 1:10, without any increase in loss, on a high-latency 810 path with an asymmetry of 5:1. This offers a clear benefit for paths 811 that are capacity-constrained, as well as paths which would benefit 812 from a reduction in the ACK Rate. 814 Additional data and analysis is provided in [Cust20]. 816 Authors' Addresses 818 Godred Fairhurst 819 University of Aberdeen 820 School of Engineering 821 Fraser Noble Building 822 Aberdeen AB24 3UE 823 UK 825 Email: gorry@erg.abdn.ac.uk 827 Ana Custura 828 University of Aberdeen 829 School of Engineering 830 Fraser Noble Building 831 Aberdeen AB24 3UE 832 UK 834 Email: ana@erg.abdn.ac.uk 835 Tom Jones 836 University of Aberdeen 837 School of Engineering 838 Fraser Noble Building 839 Aberdeen AB24 3UE 840 UK 842 Email: tom@erg.abdn.ac.uk