idnits 2.17.1 draft-corre-quic-throughput-testing-00.txt: -(798): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 2 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([RFC6349], [RFC9000]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: QUIC includes precise RTT statistics and MAY be directly used to gather RTT statistics if the QUIC implementation exposes them. As QUIC packets are mostly encrypted, packet capture tools do not allow to measure QUIC RTT and retransmissions and SHOULD not be used for this methodology. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: In addition to the solutions proposed in [RFC6349] for measuring RTT, short QUIC sessions MAY be employed to baseline RTT values off-peak hours and outside of test sessions. This solution requires that the QUIC implementation expose RTT measurements [RFC9002]. latest_rtt SHOULD be used to sample baseline RTT using the minimum observed latest_RTT during off-peak hours and outside of test sessions. instead of latest_rtt, min_rtt MAY be used to sample baseline RTT during off-peak hours and outside of test sessions. smoothed_rtt MUST not be used to baseline RTT. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: In practice, a QUIC Throughput test probably implements a control channel to control the test and one or more throughput channels to send data. The control channel would serve to authenticate the TTD, request a test session through some throughput channels, and exchange test results. As 0-RTT packets are vulnerable to replay attacks [RFC9001], 0-RTT packets MUST not be used by a TTD client initiating a control channel to a TTD server. -- The document date (17 September 2021) is 951 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC6582' is defined on line 785, but no explicit reference was found in the text ** Obsolete normative reference: RFC 793 (Obsoleted by RFC 9293) Summary: 2 errors (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group K. Corre 3 Internet-Draft EXFO 4 Intended status: Informational 17 September 2021 5 Expires: 21 March 2022 7 Framework for QUIC Throughput Testing 8 draft-corre-quic-throughput-testing-00 10 Abstract 12 This document adapts the [RFC6349] Framework for TCP Throughput 13 Testing to the [RFC9000] QUIC protocol. The adapted framework 14 describes a practical methodology for measuring end-to-end QUIC 15 Throughput in a managed IP network. The goal of the methodology is 16 to provide a better indication in regard to user experience. In this 17 framework, QUIC, UDP, and IP parameters are specified to optimize 18 QUIC Throughput. 20 Discussion Venues 22 This note is to be removed before publishing as an RFC. 24 Discussion of this document takes place on the QUIC Working Group 25 mailing list (quic@ietf.org), which is archived at 26 https://mailarchive.ietf.org/arch/browse/quic/. 28 Source for this draft and an issue tracker can be found at 29 https://github.com/Sparika/draft-corre-quic-throughput-testing. 31 Status of This Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF). Note that other groups may also distribute 38 working documents as Internet-Drafts. The list of current Internet- 39 Drafts is at https://datatracker.ietf.org/drafts/current/. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 This Internet-Draft will expire on 21 March 2022. 48 Copyright Notice 50 Copyright (c) 2021 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 55 license-info) in effect on the date of publication of this document. 56 Please review these documents carefully, as they describe your rights 57 and restrictions with respect to this document. Code Components 58 extracted from this document must include Simplified BSD License text 59 as described in Section 4.e of the Trust Legal Provisions and are 60 provided without warranty as described in the Simplified BSD License. 62 Table of Contents 64 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 65 1.1. Impacting changes between TCP and QUIC . . . . . . . . . 3 66 2. Conventions and Definitions . . . . . . . . . . . . . . . . . 5 67 3. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 5 68 3.1. Path MTU . . . . . . . . . . . . . . . . . . . . . . . . 6 69 3.2. Round-Trip Time (RTT) and Bottleneck Bandwidth (BB) . . . 6 70 3.3. Measuring RTT . . . . . . . . . . . . . . . . . . . . . . 6 71 3.4. Measuring BB . . . . . . . . . . . . . . . . . . . . . . 6 72 3.5. Measuring QUIC Throughput . . . . . . . . . . . . . . . . 6 73 3.6. Minimum QUIC Congestion Control Credit . . . . . . . . . 7 74 4. QUIC Metrics . . . . . . . . . . . . . . . . . . . . . . . . 8 75 4.1. Transfert Time Ratio . . . . . . . . . . . . . . . . . . 8 76 4.1.1. Maximum Achievable QUIC Throughput Calculation . . . 8 77 4.1.2. QUIC Transfer Time and Transfer Time Ratio 78 Calculation . . . . . . . . . . . . . . . . . . . . . 11 79 4.2. QUIC Efficiency . . . . . . . . . . . . . . . . . . . . . 12 80 4.2.1. QUIC Efficiency Percentage Calculation . . . . . . . 13 81 4.3. Buffer Delay . . . . . . . . . . . . . . . . . . . . . . 13 82 5. Conducting QUIC Throughput Tests . . . . . . . . . . . . . . 14 83 5.1. Using Multiple QUIC Streams . . . . . . . . . . . . . . . 14 84 5.2. 1-RTT and 0-RTT . . . . . . . . . . . . . . . . . . . . . 15 85 5.3. Results Interpretation . . . . . . . . . . . . . . . . . 15 86 6. Security Considerations . . . . . . . . . . . . . . . . . . . 16 87 6.1. 0-RTT attack {#0RTTattack} . . . . . . . . . . . . . . . 17 88 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 89 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 17 90 References . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 91 Normative References . . . . . . . . . . . . . . . . . . . . . 17 92 Informative References . . . . . . . . . . . . . . . . . . . . 18 93 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 18 95 1. Introduction 97 This document adapts the [RFC6349] Framework for TCP Throughput 98 Testing to the QUIC protocol [RFC9000]. RFC6349 defines a 99 methodology for testing sustained TCP Layer performance. Like TCP 100 [RFC793], QUIC is a connection-oriented transport protocol. However, 101 there are multiple differences between both protocols and some of 102 these differences impact the throughput testing methodology. 104 For easier reference, this document follows the same organization as 105 [RFC6349] and each section describes changes relevant to the 106 equivalent [RFC6349] section. The scope and goals section is omitted 107 as the objective of this document is precisely to remain within the 108 same scope and goals of [RFC6349]. Section 3 presents changes to the 109 individual steps of the [RFC6349] methodology. Section 4 describes 110 the three main metrics used to measure QUIC Throughput. Section 5 111 covers conducting QUIC Throughput testing. In particular, the 112 section presents QUIC streams usage in the context of QUIC Throughput 113 testing and discusses possible results interpretation. Finally, 114 Section 6 presents additional security considerations. 116 1.1. Impacting changes between TCP and QUIC 118 This methodology proposes QUIC Throughput performance testing 119 focusing on: 121 * Bottleneck Bandwidth (BB) 123 * Round-Trip Time (RTT) 125 * Send and Receive Socket Buffers 127 * Available Control-Flow Credits and QUIC CWND 129 * Path Maximum Transmission Unit (MTU) 131 There are multiple changes between TCP and QUIC impacting the 132 throughput testing methodology. Firstly, the multiple QUIC headers 133 and frames result in overheads variable in size which in turn impact 134 the computation for the maximum achievable QUIC throughput, in 135 particular the Transfert Time Ration metric presented in Section 4.1. 136 Secondly, QUIC provides streams that can be used for throughput 137 testing but which may also result in variable overheads. 139 TODO: 141 Thirdly, QUIC provides congestion control genericity and receiver- 142 controlled control flow credits instead of the TCP sliding receiver 143 window. While QUIC Loss Detection and Congestion Control [RFC9002] 144 exmplifies with a congestion controller similar to TCP 145 NewReno~[RFC6582], the signals QUIC provides for congestion control 146 are generic and are designed to support different sender-side 147 algorithms. In this document, the achievable QUIC Throughput is the 148 amount of data per unit of time that QUIC transports when in the QUIC 149 Equilibrium state. Derived from Round-Trip Time (RTT) and network 150 Bottleneck Bandwidth (BB), the Bandwidth-Delay Product (BDP) 151 determines the Send and Received Socket buffer sizes required to 152 achieve the maximum QUIC Throughput. Throughout this document, 153 "maximum achievable throughput" refers to the theoretical achievable 154 throughput when QUIC is in the Equilibrium state. In this document, 155 we assume that QUIC uses a transmitting side congestion controller 156 with a congestion window (QUIC CWND) and congestion controller states 157 similar to TCP NewReno. 159 New path or +------------+ 160 persistent congestion | Slow | 161 (O)---------------------->| Start | 162 +------------+ 163 | 164 Loss or | 165 ECN-CE increase | 166 v 167 +------------+ Loss or +------------+ 168 | Congestion | ECN-CE increase | Recovery | 169 | Avoidance |------------------>| Period | 170 +------------+ +------------+ 171 ^ | 172 | | 173 +----------------------------+ 174 Acknowledgment of packet 175 sent during recovery 177 Figure 1: QUIC NewReno Congestion Control States and Transitions 179 On the receiving side, QUIC does not use a sliding receiver window 180 and instead uses a control flow credits mechanism to inform the 181 transmitting end on the maximum total amount of data the receiver is 182 willing to receive for each stream and connection. The amount of 183 available control flow credits does not directly control the QUIC 184 Throughput but insufficient credits may lead to a stream or a whole 185 connection being rate-limited. This is discussed in Section 3.6. 187 2. Conventions and Definitions 189 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 190 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 191 "OPTIONAL" in this document are to be interpreted as described in 192 BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all 193 capitals, as shown here. 195 3. Methodology 197 Overall, the testing methodology remains the same as the one 198 described in [RFC6349] with a few minor differences. The following 199 represents the sequential order of steps for this testing 200 methodology: 202 1. Identify the Path MTU. As QUIC relies on UDP, Packetization 203 Layer Path MTU Discovery for Datagram Transports (Datagram PMTUD) 204 [RFC8899] SHOULD be conducted instead of Packetization Layer Path 205 MTU Discovery (PLPMTUD). It is important to identify the path 206 MTU so that the QUIC Throughput Test Device (TTD) is configured 207 properly to avoid fragmentation. 209 2. Baseline Round-Trip Time and Bandwidth. This step establishes 210 the inherent, non-congested Round-Trip Time (RTT) and the 211 Bottleneck Bandwidth (BB) of the end-to-end network path. These 212 measurements are used to provide estimates of the QUIC minimum 213 congestion control credit and Send Socket Buffer sizes that 214 SHOULD be used during subsequent test steps. 216 3. QUIC Connection Throughput Tests. With baseline measurements of 217 Round-Trip Time and Bottleneck Bandwidth, single- and multiple- 218 QUIC-connection throughput tests SHOULD be conducted to baseline 219 network performance. 221 These three (3) steps are detailed in Section 3.1 to Section 3.5. 223 Regarding, the QUIC TTD, the same key characteristics and criteria 224 SHOULD be considered than with the [RFC6349] TCP TTD. The test host 225 MAY be a standard computer or a dedicated communications test 226 instrument and in both cases, it MUST be capable of emulating both a 227 client and a server. One major difference is that contrary to TCP, 228 QUIC may not be provided by OSs usually providing transport protocol 229 implementations and may instead be sourced by an application from a 230 third-party library. These different implementations may present "a 231 large behavioural heterogeneity" [marx2020same] potentially impacting 232 the QUIC throughput measurements. Thus, in addition to the OS 233 version (e.g. a specific LINUX OS kernel version), the QUIC 234 implementation used by the test hosts MUST be considered, including 235 used congestion control algorithms available and supported QUIC 236 options. The QUIC test implementation and hosts MUST be capable of 237 generating and receiving stateful QUIC test traffic at the full BB of 238 the NUT. 240 QUIC includes precise RTT statistics and MAY be directly used to 241 gather RTT statistics if the QUIC implementation exposes them. As 242 QUIC packets are mostly encrypted, packet capture tools do not allow 243 to measure QUIC RTT and retransmissions and SHOULD not be used for 244 this methodology. 246 3.1. Path MTU 248 QUIC implementations SHOULD use Datagram Path MTU Discovery 249 techniques (Datagram PMTUD). 251 3.2. Round-Trip Time (RTT) and Bottleneck Bandwidth (BB) 253 Before stateful QUIC testing can begin, it is important to determine 254 the baseline RTT (i.e. non-congested inherent delay) and BB of the 255 end-to-end network to be tested. These measurements are used to 256 calculate the BDP and to provide estimates for the congestion control 257 credit and Send Socket Buffer sizes that SHOULD be used in subsequent 258 test steps. 260 3.3. Measuring RTT 262 In addition to the solutions proposed in [RFC6349] for measuring RTT, 263 short QUIC sessions MAY be employed to baseline RTT values off-peak 264 hours and outside of test sessions. This solution requires that the 265 QUIC implementation expose RTT measurements [RFC9002]. latest_rtt 266 SHOULD be used to sample baseline RTT using the minimum observed 267 latest_RTT during off-peak hours and outside of test sessions. 268 instead of latest_rtt, min_rtt MAY be used to sample baseline RTT 269 during off-peak hours and outside of test sessions. smoothed_rtt MUST 270 not be used to baseline RTT. 272 3.4. Measuring BB 274 This section is unchanged. 276 3.5. Measuring QUIC Throughput 278 This methodology specifically defines QUIC Throughput measurement 279 techniques to verify maximum achievable QUIC performance in a managed 280 business-class IP network. 282 With baseline measurements of RTT and BB from Section 3.2, a series 283 of single- and/or multiple-QUIC-connection throughput tests SHOULD be 284 conducted. 286 Like with TCP, the number of trials and the choice between single or 287 multiple QUIC connections will be based on the intention of the test. 288 In all circumstances, it is RECOMMENDED to run the tests in each 289 direction independently first and then to run them in both directions 290 simultaneously. It is also RECOMMENDED to run the tests at different 291 times of the day. 293 The metrics measured are the QUIC Transfer Time Ratio, the QUIC 294 Efficiency Percentage, and the Buffer Delay Percentage. These 3 295 metrics are defined in Section 4 and MUST be measured in each 296 direction. 298 3.6. Minimum QUIC Congestion Control Credit 300 As for [RFC6349] TCP throughput testing, the QUIC throughput test 301 MUST ensure that the QUIC performance is never rate-limited. Whereas 302 TCP uses a sliding receiver window (TCP RWND), QUIC relies on 303 congestion control credits that are not automatically sliding. 304 Instead, available control-flow credits are increased by sending 305 MAX_DATA and MAX_STREAM_DATA frames. The algorithm used to send 306 these frames depends on the QUIC implementation or may be left for 307 the application to control at each receiving side. 309 In addition to setting Send Socket Buffer size higher than the BDP, 310 the QUIC TTD MUST ensure that connections and streams always have 311 Control-Flow Credits (CFC) in excess of the BDP so that any QUIC 312 stream or connection is never restricted below the BDP before 313 MAX_DATA and MAX_STREAM_DATA frames have a chance to arrive. If a 314 QUIC connection or stream has CFC below the minimum required CFC, 315 then there is a risk of the CFC reaching 0 before the limits can be 316 increased. If a QUIC stream has CFC at 0, then that stream is rate- 317 limited and there is a risk of the QUIC Throughput to not be optimal. 318 Note that other streams may still manage to fill the BDP of the NUT. 319 If a QUIC connection has CFC at 0, then that connection is rate- 320 limited and the QUIC Throughput cannot be optimal. The QUIC TTD MUST 321 report every events and events' duration where a QUIC stream or 322 connection is rate-limited. 324 A QUIC implementation could implement a sliding receive window 325 similar to TCP RWND by sending MAX_DATA and MAX_STREAM_DATA frames 326 with each ACK frame. In this case, the minimum CFC (i.e. initial 327 data limit) is similar to the minimum TCP RWND and example numbers 328 proposed in [RFC6349]. However, such a solution imposes a heavy 329 overhead and it is more likely that sending of MAX_DATA and 330 MAX_STREAM_DATA frames are delayed by the QUIC implementation. Thus, 331 the minimum CFC should be set sufficiently large so that the QUIC 332 implementation can send MAX_DATA and MAX_STREAM_DATA frames with new 333 limits before control-flow credits are exhausted. 335 4. QUIC Metrics 337 The proposed metrics for measuring QUIC Throughput remain the same as 338 for measuring TCP Throughput with some minor differences. 340 4.1. Transfert Time Ratio 342 The first metric is the QUIC Transfer Time Ratio, which is the ratio 343 between the Actual QUIC Transfer Time versus the Ideal QUIC Transfer 344 Time. The Actual QUIC Transfer Time is simply the time it takes to 345 transfer a block of data across QUIC connection(s). The Ideal QUIC 346 Transfer Time is the predicted time for which a block of data SHOULD 347 transfer across QUIC connection(s), considering the BB of the NUT. 349 QUIC Actual QUIC Transfer Time 350 Transfer Time = --------------------------- 351 Ratio Ideal QUIC Transfer Time 353 The Ideal QUIC Transfer Time is derived from the Maximum Achievable 354 QUIC Throughput, which is related to the BB and Layer 1/2/3/4 355 overheads associated with the network path. The following sections 356 provide derivations for the Maximum Achievable QUIC Throughput and 357 example calculations for the QUIC Transfer Time Ratio. 359 4.1.1. Maximum Achievable QUIC Throughput Calculation 361 This section provides formulas to calculate the Maximum Achievable 362 QUIC Throughput, with examples for T3 (44.21 Mbps) and Ethernet. 364 On the contrary to TCP, the QUIC overhead is variable in size, in 365 particular, due to variable-length fields and streams multiplexing. 366 Furthermore, QUIC is a versioned protocol and the overheads may 367 change from version to version. Other underlying protocol changes 368 such as IP version may also increase overheads or make them variable 369 in size. The following calculations are based on IP version 4 with 370 IP headers of 20 Bytes, UDP headers of 8 Bytes, QUIC version 1, and 371 within an MTU of 1500 Bytes. 373 We are interested in estimating the maximal QUIC Throughput at 374 equilibrium, so we first simplify the problem by considering an 375 optimistic scenario with data exchanged over 1-RTT packets with only 376 a single stream frame per packet. For the content of the STREAM 377 frame, we consider that the offset field is used but not the length 378 field since there is only one frame per packet. 380 Here are the 1-RTT packet and STREAM frame headers as defined in 381 [RFC9000]. 383 1-RTT Packet { 384 Header Form (1) = 0, 385 Fixed Bit (1) = 1, 386 Spin Bit (1), 387 Reserved Bits (2), 388 Key Phase (1), 389 Packet Number Length (2), 390 Destination Connection ID (0..160), 391 Packet Number (8..32), 392 Packet Payload (8..), 393 } 395 STREAM Frame { 396 Type (i) = 0x08..0x0f, 397 Stream ID (i), 398 [Offset (i)], 399 [Length (i)], 400 Stream Data (..), 401 } 403 Supposing all variable-length fields are encoded as the minimum size 404 possible, the overhead would be 3 Bytes for QUIC 1-RTT packet and 2 405 Bytes for QUIC STREAM frame. This gives us a minimal IP/UDP/QUIC 406 overhead of 20 + 8 + 3 + 2 = 33 Bytes. Note that alternatively, 407 considering all variable length fields encoded as the largest size 408 possible, the overhead would be 25 Bytes for QUIC 1-RTT packet and 17 409 Bytes for QUIC STREAM frame. In this worst-case scenario, the 410 overhead could mount up to 20 + 8 + 25 + 17 = 70 Bytes for one STREAM 411 frame per packet. 413 To estimate the Maximum Achievable QUIC Throughput, we first need to 414 determine the maximum quantity of Frames Per Second (FPS) permitted 415 by the actual physical layer (Layer 1) speed. 417 For a T3 link, the calculation formula is: 419 FPS = T3 Physical Speed / 420 ((MTU + PPP + Flags + CRC16) X 8) 422 FPS = (44.21 Mbps / 423 ((1500 Bytes + 4 Bytes 424 + 2 Bytes + 2 Bytes) X 8 ))) 425 FPS = (44.21 Mbps / (1508 Bytes X 8)) 426 FPS = 44.21 Mbps / 12064 bits 427 FPS = 3664 429 Then, to obtain the Maximum Achievable QUIC Throughput (Layer 4), we 430 simply use: 432 (MTU - 33) in Bytes X 8 bits X max FPS 434 For a T3, the maximum QUIC Throughput = 436 1467 Bytes X 8 bits X 3664 FPS 438 Max QUIC Throughput = 11736 bits X 3664 FPS 439 Max QUIC Throughput = 43.0 Mbps 441 On Ethernet, the maximum achievable Layer 2 throughput is limited by 442 the maximum Frames Per Second permitted by the IEEE802.3 standard. 444 The maximum FPS for 100-Mbps Ethernet is 8127, and the calculation 445 formula is: 447 FPS = (100 Mbps / (1538 Bytes X 8 bits)) 449 The maximum FPS for GigE is 81274, and the calculation formula is: 451 FPS = (1 Gbps / (1538 Bytes X 8 bits)) 453 The maximum FPS for 10GigE is 812743, and the calculation formula is: 455 FPS = (10 Gbps / (1538 Bytes X 8 bits)) 457 The 1538 Bytes equates to: 459 MTU + Ethernet + CRC32 + IFG + Preamble + SFD 460 IFG = Inter-Frame Gap 461 SFD = Start of Frame Delimiter 463 where MTU is 1500 Bytes, Ethernet is 14 Bytes, CRC32 is 4 Bytes, IFG 464 is 12 Bytes, Preamble is 7 Bytes, and SFD is 1 Byte. 466 Then, to obtain the Maximum Achievable QUIC Throughput (Layer 4), we 467 simply use: 469 (MTU - 33) in Bytes X 8 bits X max FPS 471 For 100-Mbps Ethernet, the maximum QUIC Throughput = 473 1467 Bytes X 8 bits X 8127 FPS 475 Max QUIC Throughput = 11736 bits X 8127 FPS 476 Max QUIC Throughput = 95.4 Mbps 478 4.1.2. QUIC Transfer Time and Transfer Time Ratio Calculation 480 The following table illustrates the Ideal QUIC Transfer Time of a 481 single QUIC connection when its Send Socket Buffer size equals or 482 exceeds the BDP and the sender is never control-flow limited. 484 +============+=======+=======+================+====================+ 485 | Link Speed | RTT | BDP | Max. Ach. QUIC | Ideal QUIC | 486 | (Mbps) | (ms) | (KBs) | Thr. (Mbps) | Transfer Time (s)* | 487 +============+=======+=======+================+====================+ 488 | 1.536 | 50.00 | 9.6 | 1.4 | 571.0 | 489 +------------+-------+-------+----------------+--------------------+ 490 | 44.210 | 25.00 | 138.2 | 43.0 | 18.6 | 491 +------------+-------+-------+----------------+--------------------+ 492 | 100.000 | 2.00 | 25.0 | 95.4 | 8.4 | 493 +------------+-------+-------+----------------+--------------------+ 494 | 1,000.000 | 1.00 | 125.0 | 953.7 | 0.8 | 495 +------------+-------+-------+----------------+--------------------+ 496 | 10,000.000 | 0.05 | 62.5 | 9,537.5 | 0.1 | 497 +------------+-------+-------+----------------+--------------------+ 499 Table 1: Link Speed, RTT, BDP, Maximum QUIC Throughput, and 500 Ideal QUIC Transfer Time for a 100-MB File 502 | Note (*): Transfer times are rounded for simplicity. 504 For a 100-MB file (100 X 8 = 800 Mbits), the Ideal QUIC Transfer Time 505 is derived as follows: 507 800 Mbits 508 Ideal QUIC = ------------------------- 509 Transfer Time Maximum Achievable 510 QUIC Throughput 512 To illustrate the QUIC Transfer Time Ratio, an example would be the 513 bulk transfer of 100 MB over 5 simultaneous QUIC connections (each 514 connection transferring 100 MB). In this example, the Ethernet 515 service provides a Committed Access Rate (CAR) of 500 Mbps. Each 516 connection may achieve different throughputs during a test, and the 517 overall throughput rate is not always easy to determine (especially 518 as the number of connections increases). 520 The Ideal QUIC Transfer Time would be ~8.4 seconds, but in this 521 example, the Actual QUIC Transfer Time was 12 seconds. The QUIC 522 Transfer Time Ratio would then be 12/8.4 = 1.43, which indicates that 523 the transfer across all connections took 1.43 times longer than the 524 ideal. Note that our estimation of the Maximum Achievable QUIC 525 Throughput is overestimated due to the optimistic scenario initially 526 described. The actual QUIC Transfer Time is thus expected to be 527 higher than the Ideal QUIC Transfer Time due to eventual network 528 inefficiencies but also by overheads caused by the selected QUIC 529 implementation as well as the number of concurrent streams being used 530 for the test. 532 4.2. QUIC Efficiency 534 In [RFC6349], the TCP efficiency metric represents the percentage of 535 Bytes that were not retransmitted. However, when packet loss occurs 536 in QUIC, Bytes are not retransmitted per se, instead, new packets are 537 generated. As a result, a packet may contain retransmitted frames as 538 well as new frames, and measuring the amount of retransmitted Bytes 539 is not straightforward. 541 Instead, the QUIC efficiency MUST be measured by calculating the 542 ratio between QUIC Payload Bytes and UDP Bytes. QUIC Payload Bytes 543 are the total number of payload Bytes sent or received over QUIC 544 while UDP Bytes are the total number of bytes sent or received on the 545 UDP socket, i.e. including QUIC protocol overheads. This method 546 allows to capture the impact of retransmission as retransmission of 547 frames will impact the UDP Bytes without impacting the QUIC Bytes. 548 Retransmitted data will effectively lower the measured QUIC 549 Efficiency. 551 QUIC payload Bytes 552 QUIC = ----------------------- X 100 553 Efficiency UDP Bytes 555 In comparison with [RFC6349], the sender side QUIC Efficiency metric 556 also measures the inherent overheads of QUIC caused by headers, 557 frames, multiple streams, and implementation choices. As the 558 receiver side UDP Bytes also measures these overheads but does not 559 measure retransmission overheads, the receiver side QUIC Efficiency 560 MAY be used to normalize the sender side QUIC Efficiency. 562 Normalized Sender QUIC Efficiency 563 QUIC = -------------------------- 564 Efficiency Receiver QUIC Efficiency 566 4.2.1. QUIC Efficiency Percentage Calculation 568 As an example, considering a scenario where the MTU is 1500 bits and 569 the QUIC overhead is 50bits per packet on average. An application 570 sending 100,000 Bytes over QUIC would emit approximately 552 UDP 571 packets of 1500 bits resulting in a total of 103,448 UDP Bytes. The 572 QUIC Efficiency Percentage would be calculated as: 574 100,000 575 QUIC Efficiency % = --------- X 100 = 96,6% 576 103,448 578 Considering a similar example with 1% of packet loss, the UDP Bytes 579 emitted would be approximately 104,500 Bytes. The resulting QUIC 580 Efficiency Percentage would thus be calculated as: 582 100,000 583 QUIC Efficiency % = --------- X 100 = 95,6% 584 104,500 586 On the receiver side, the measured UDP Bytes would be approximately 587 103,448 UDP Bytes in both scenarios since no loss overheads would be 588 measured resulting in Receiver QUIC Efficiency = 96,6%. Normalizing 589 the previously measured QUIC Efficiency metrics (sender side) would 590 thus result in Normalised QUIC Efficiency = 100% in the first example 591 and Normalised QUIC Efficiency = 99% in the example with 1% of packet 592 loss. 594 4.3. Buffer Delay 596 The third metric is the Buffer Delay Percentage, which represents the 597 increase in RTT during a QUIC Throughput test versus the inherent or 598 baseline RTT. This metric remains identical to the one described in 599 [RFC6349]. 601 Average RTT Total RTTs during transfer 602 during transfer = ---------------------------- 603 Transfer duration in seconds 605 Average RTT 606 during transfer 607 - Baseline RTT 608 Buffer Delay % = --------------------- X 100 609 Baseline RTT 611 Note that while [RFC9002] specifies QUIC RTT measurements in the form 612 of latest_rtt, smoothed_rtt, and min_rtt, the average RTT used in the 613 Buffer Delay Percentage metric is derived from the total of all 614 measured RTTs during the actual test at every second divided by the 615 test duration in seconds. RTT metrics smoothed_rtt and min_rtt 616 described in [RFC9002] MUST NOT be used to compute the Average RTT 617 during transfer. RTT metric latest_rtt MAY be used for computing 618 Average RTT during transfer by sampling it at every second. 620 5. Conducting QUIC Throughput Tests 622 The methodology for QUIC Throughput testing follows the same 623 considerations as described in [RFC6349]. 625 In the end, a QUIC Throughput Test Device (QUIC TTD) SHOULD generate 626 a report with the calculated BDP and QUIC Throughput results. The 627 report SHOULD also include the results for the 3 metrics defined in 628 Section 4. As QUIC does not use a receive window, a QUIC Throughput 629 test is constituted of multiple Send and Receive Socket Buffer 630 experiments. The report SHOULD include QUIC Throughput results for 631 each Socket Buffer size tested. The goal is to provide achievable 632 versus actual QUIC Throughput results for the Socket Buffer size 633 tested when no fragmentation occurs and to provide a clear 634 relationship between these 3 metrics and user experience. As an 635 example, for the same Transfer Time Ratio, a better QUIC Efficiency 636 could be obtained at the cost of higher Buffer Delays. 638 5.1. Using Multiple QUIC Streams 640 In addition to opening multiple connections, QUIC allows data to be 641 exchanged over fully multiplexed streams. The primary goal of the 642 methodology relates to ensuring the capacity of a managed network to 643 deliver a particular SLA using QUIC. As the NUT cannot observe the 644 number of streams opened for a QUIC connection, we do not expect the 645 NUT to apply specific policies based on the number of streams. In 646 the basic test scenario, A QUIC TTD MAY use two streams over a single 647 connection where one stream is a test stream filling the BDP of the 648 NUT and the other is a control stream used for controlling the test 649 and exchanging test results. 651 Implementers of a QUIC Throughput test may however want to use 652 multiple streams in parallel. On one hand, using multiple streams 653 over a single QUIC connection may result in a variable number of 654 STREAM frames per packet, with the actual numbers depending on the 655 number of streams, the amount of data per stream, and the QUIC 656 implementation [marx2020same]. This overhead variability may lower 657 the QUIC Efficiency, increase the QUIC Transfer Time Ratio, and more 658 generally reduce the reproducibility of the test with respect to the 659 capacity of the NUT to provide an SLA. 661 On the other hand, using multiple streams over a single connection is 662 a typical scenario for using QUIC and one of the selling-point of the 663 protocol compared to TCP. Unstable networks in particular those with 664 a high packet loss are expected to benefit from stream multiplexing. 665 Implementing more complex scenarios in a QUIC Throughput test to 666 better represent the use-cases of the NUT is possible. A QUIC TTD 667 MAY use multiple test streams to fill the BDP of the NUT. In such a 668 case, the QUIC TTD SHOULD also run the test using a single test 669 stream to facilitate root-cause analysis. The specific parameters of 670 a scenario using multiple test streams depend however on the use-case 671 being considered. For instance, emulating multiple HTTP/3 browsing 672 sessions may involve a large number of short-lived small streams, 673 while emulating real-time conversational streaming sessions may 674 involve long-lived streams with large chunks of data. Details for 675 such scenarios and relevant QoE influence factors are out-of-scope of 676 the proposed methodology. 678 5.2. 1-RTT and 0-RTT 680 Another new feature of the QUIC protocol is the reduced number of RTT 681 required to establish a connection enabling so-called 1-RTT 682 connection establishment and 0-RTT connection re-establishment. It 683 may be interesting to get a measure of how these QUIC features result 684 in gain over a particular network. However, the methodology 685 described in this document is interested in throughput measurements 686 at an equilibrium state. While the present methodology baselines 687 minimum RTT and measures average RTT during throughput testing, 688 measuring connection re-establishment speed is out-of-scope for the 689 proposed methodology. 691 5.3. Results Interpretation 693 For cases where the test results are not equal to the ideal values, 694 some possible causes are as follows: 696 * Network congestion causing packet loss, which may be inferred from 697 a poor Normalised QUIC Efficiency % (i.e. higher Normalised QUIC 698 Efficiency % = less packet loss). 700 * Too many QUIC streams in parallel or an inefficient QUIC 701 implementation with regards to protocol overheads may be inferred 702 from a poor QUIC Efficiency at both the sender and receiver sides 703 (i.e. low QUIC Efficiency but high Normalised QUIC Efficiency). 705 * Rate limiting by traffic policing may cause packet loss (i.e. low 706 Normalised QUIC Efficiency). 708 * Network congestion causing an increase in RTT, which may be 709 inferred from the Buffer Delay Percentage (i.e. 0% = no increase 710 in RTT over baseline). 712 * Rate limiting by traffic shaping may also cause an increase in 713 RTT. 715 * Maximum UDP Buffer Space. All operating systems have a global 716 mechanism to limit the quantity of system memory to be used by UDP 717 sockets. On some systems, each socket is subject to a memory 718 limit that is applied to the total memory used for input data, 719 output data, and controls. On other systems, there are separate 720 limits for input and output buffer spaces per socket. Client/ 721 server IP hosts might be configured with Maximum UDP Buffer Space 722 limits that are far too small for high-performance networks. 723 These socket buffers MUST be large enough to hold a full BDP of 724 UDP Bytes plus some overhead. 726 * Path MTU. The client/server IP host system SHOULD use the largest 727 possible MTU for the path. This may require enabling Path MTU 728 Discovery [RFC1191] and [RFC4821]. Since [RFC1191] is flawed, 729 Path MTU Discovery is sometimes not enabled by default and may 730 need to be explicitly enabled by the system administrator. 731 [RFC4821] describes a new, more robust algorithm for MTU discovery 732 and ICMP black hole recovery. 734 6. Security Considerations 736 Measuring QUIC network performance raises security concerns. Metrics 737 produced within this framework may create security issues. 739 Security considerations mentionned in [RFC6349] remain valid for QUIC 740 Throughput testing. In particular, as QUIC encrypts traffic over 741 UDP, QUIC Throughput testing may appear as a denial-of-service 742 attack. Cooperation between the end-user customer and the network 743 provider is thus required. 745 6.1. 0-RTT attack {#0RTTattack} 747 In practice, a QUIC Throughput test probably implements a control 748 channel to control the test and one or more throughput channels to 749 send data. The control channel would serve to authenticate the TTD, 750 request a test session through some throughput channels, and exchange 751 test results. As 0-RTT packets are vulnerable to replay attacks 752 [RFC9001], 0-RTT packets MUST not be used by a TTD client initiating 753 a control channel to a TTD server. 755 7. IANA Considerations 757 This document has no IANA actions. 759 Acknowledgments 761 Thanks to Sylvain Nadeau for his valuable inputs. 763 References 765 Normative References 767 [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191, 768 DOI 10.17487/RFC1191, November 1990, 769 . 771 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 772 Requirement Levels", BCP 14, RFC 2119, 773 DOI 10.17487/RFC2119, March 1997, 774 . 776 [RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU 777 Discovery", RFC 4821, DOI 10.17487/RFC4821, March 2007, 778 . 780 [RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage, 781 "Framework for TCP Throughput Testing", RFC 6349, 782 DOI 10.17487/RFC6349, August 2011, 783 . 785 [RFC6582] Henderson, T., Floyd, S., Gurtov, A., and Y. Nishida, "The 786 NewReno Modification to TCP's Fast Recovery Algorithm", 787 RFC 6582, DOI 10.17487/RFC6582, April 2012, 788 . 790 [RFC793] Postel, J., "Transmission Control Protocol", STD 7, 791 RFC 793, DOI 10.17487/RFC0793, September 1981, 792 . 794 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 795 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 796 May 2017, . 798 [RFC8899] Fairhurst, G., Jones, T., Tüxen, M., Rüngeler, I., and T. 799 Völker, "Packetization Layer Path MTU Discovery for 800 Datagram Transports", RFC 8899, DOI 10.17487/RFC8899, 801 September 2020, . 803 [RFC9000] Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based 804 Multiplexed and Secure Transport", RFC 9000, 805 DOI 10.17487/RFC9000, May 2021, 806 . 808 [RFC9001] Thomson, M., Ed. and S. Turner, Ed., "Using TLS to Secure 809 QUIC", RFC 9001, DOI 10.17487/RFC9001, May 2021, 810 . 812 [RFC9002] Iyengar, J., Ed. and I. Swett, Ed., "QUIC Loss Detection 813 and Congestion Control", RFC 9002, DOI 10.17487/RFC9002, 814 May 2021, . 816 Informative References 818 [marx2020same] 819 Marx, R., Herbots, J., Lamotte, W., and P. Quax, "Same 820 standards, different decisions: A study of quic and http/3 821 implementation diversity", Proceedings of the Workshop on 822 the Evolution, Performance, and Interoperability of QUIC, 823 2020. 825 Author's Address 827 Kevin Corre 828 EXFO 830 Email: kevin.corre@exfo.com