idnits 2.17.1 draft-kuhn-aqm-eval-guidelines-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 3) being 60 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 325 has weird spacing: '...metrics and t...' == Line 503 has weird spacing: '...ts data betwe...' == Line 591 has weird spacing: '...schemes may i...' == Line 792 has weird spacing: '...fferent conge...' -- The document date (August 11, 2014) is 3546 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-11) exists of draft-ietf-aqm-recommendation-07 -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) Summary: 0 errors (**), 0 flaws (~~), 7 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force N. Kuhn, Ed. 3 Internet-Draft Telecom Bretagne 4 Intended status: Informational P. Natarajan, Ed. 5 Expires: February 10, 2015 Cisco Systems 6 D. Ros 7 Simula Research Laboratory AS 8 N. Khademi 9 University of Oslo 10 August 11, 2014 12 AQM Characterization Guidelines 13 draft-kuhn-aqm-eval-guidelines-02 15 Abstract 17 Unmanaged large buffers in today's networks have given rise to a slew 18 of performance issues. These performance issues can be addressed by 19 some form of Active Queue Management (AQM), optionally in combination 20 with a packet scheduling scheme such as fair queuing. The IETF AQM 21 and packet scheduling working group was formed to standardize AQM 22 schemes that are robust, easily implemented, and successfully 23 deployed in today's networks. This document describes various 24 criteria for performing precautionary characterizations of AQM 25 proposals. This document also helps in ascertaining whether any 26 given AQM proposal should be taken up for standardization by the AQM 27 WG. 29 Status of this Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at http://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on February 10, 2015. 46 Copyright Notice 48 Copyright (c) 2014 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents (http://trustee.ietf.org/ 53 license-info) in effect on the date of publication of this document. 54 Please review these documents carefully, as they describe your rights 55 and restrictions with respect to this document. Code Components 56 extracted from this document must include Simplified BSD License text 57 as described in Section 4.e of the Trust Legal Provisions and are 58 provided without warranty as described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 63 1.1. Guidelines for AQM designers . . . . . . . . . . . . . . . 4 64 1.2. Reducing the latency and maximizing the goodput . . . . . 5 65 1.3. Glossary . . . . . . . . . . . . . . . . . . . . . . . . . 5 66 1.4. Requirements Language . . . . . . . . . . . . . . . . . . 5 67 2. End-to-end metrics . . . . . . . . . . . . . . . . . . . . . . 5 68 2.1. Flow Completion time . . . . . . . . . . . . . . . . . . . 6 69 2.2. Packet loss . . . . . . . . . . . . . . . . . . . . . . . 6 70 2.3. Packet loss synchronization . . . . . . . . . . . . . . . 6 71 2.4. Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 6 72 2.5. Latency and jitter . . . . . . . . . . . . . . . . . . . . 7 73 2.6. Discussion on the trade-off between latency and goodput . 7 74 3. Generic set up for evaluations . . . . . . . . . . . . . . . . 7 75 3.1. Topology and notations . . . . . . . . . . . . . . . . . . 8 76 3.2. Buffer size . . . . . . . . . . . . . . . . . . . . . . . 9 77 3.3. Congestion controls . . . . . . . . . . . . . . . . . . . 9 78 4. Various TCP variants . . . . . . . . . . . . . . . . . . . . . 9 79 4.1. TCP-friendly Sender . . . . . . . . . . . . . . . . . . . 10 80 4.2. Aggressive Transport Sender . . . . . . . . . . . . . . . 10 81 4.3. Unresponsive Transport Sender . . . . . . . . . . . . . . 10 82 4.4. TCP initial congestion window . . . . . . . . . . . . . . 10 83 4.5. Traffic Mix . . . . . . . . . . . . . . . . . . . . . . . 11 84 5. RTT fairness . . . . . . . . . . . . . . . . . . . . . . . . . 12 85 5.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . 12 86 5.2. Required tests . . . . . . . . . . . . . . . . . . . . . . 12 87 5.3. Metrics to evaluate the RTT fairness . . . . . . . . . . . 13 88 6. Burst absorption . . . . . . . . . . . . . . . . . . . . . . . 13 89 6.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . 13 90 6.2. Required tests . . . . . . . . . . . . . . . . . . . . . . 14 91 6.3. Metrics to evaluate the burst absorption capacity . . . . 14 92 7. Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 14 93 7.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . 14 94 7.2. Required tests . . . . . . . . . . . . . . . . . . . . . . 15 95 7.2.1. Mild Congestion . . . . . . . . . . . . . . . . . . . 15 96 7.2.2. Medium Congestion . . . . . . . . . . . . . . . . . . 15 97 7.2.3. Heavy Congestion . . . . . . . . . . . . . . . . . . . 15 98 7.2.4. Varying Available Bandwidth . . . . . . . . . . . . . 15 99 7.3. Parameter sensitivity and stability analysis . . . . . . . 16 100 8. Implementation cost . . . . . . . . . . . . . . . . . . . . . 16 101 8.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . 16 102 8.2. Required discussion . . . . . . . . . . . . . . . . . . . 17 103 9. Operator control knobs and auto-tuning . . . . . . . . . . . . 17 104 10. Interaction with ECN . . . . . . . . . . . . . . . . . . . . . 18 105 10.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 18 106 10.2. Required discussion . . . . . . . . . . . . . . . . . . . 18 107 11. Interaction with scheduling . . . . . . . . . . . . . . . . . 18 108 11.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 18 109 11.2. Required discussion . . . . . . . . . . . . . . . . . . . 18 110 12. Discussion on methodology, metrics, AQM comparisons and packet 111 sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 112 12.1. Methodology . . . . . . . . . . . . . . . . . . . . . . . 18 113 12.2. Comments on metrics measurement . . . . . . . . . . . . . 19 114 12.3. Comparing AQM schemes . . . . . . . . . . . . . . . . . . 19 115 12.3.1. Performance comparison . . . . . . . . . . . . . . . 19 116 12.3.2. Deployment comparison . . . . . . . . . . . . . . . . 20 117 12.4. Packet sizes and congestion notification . . . . . . . . 20 118 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 20 119 14. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 20 120 15. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 121 16. Security Considerations . . . . . . . . . . . . . . . . . . . 21 122 17. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 123 17.1. Normative References . . . . . . . . . . . . . . . . . . 21 124 17.2. Informative References . . . . . . . . . . . . . . . . . 21 125 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 22 127 1. Introduction 129 Active Queue Management (AQM) addresses the concerns arising from 130 using unnecessarily large and unmanaged buffers in order to improve 131 network and application performance. Several AQM algorithms have 132 been proposed in the past years, most notable being Random Early 133 Detection (RED), BLUE, and Proportional Integral controller (PI), and 134 more recently CoDel [CODEL] and PIE [PIE]. In general, these 135 algorithms actively interact with the Transmission Control Protocol 136 (TCP) and any other transport protocol that deploys a congestion 137 control scheme to manage the amount of data they keep in the network. 138 The available buffer space in the routers and switches is large 139 enough to accommodate the short-term buffering requirements. AQM 140 schemes aim at reducing mean buffer occupancy, and therefore both 141 end-to-end delay and jitter. Some of these algorithms, notably RED, 142 have also been widely implemented in some network devices. However, 143 any potential benefits of the RED AQM scheme have not been realized 144 since RED is reported to be usually turned off. The main reason of 145 this reluctance to use RED in today's deployments is its sensitivity 146 to the operating conditions in the network and the difficulty of 147 tuning its parameters. 149 A buffer is a physical volume of memory in which a queue or set of 150 queues are stored. In real implementations of switches, a global 151 memory is shared between the available devices: the size of the 152 buffer for a given communication does not make sense, as its 153 dedicated memory may vary over the time and real world buffering 154 architectures are complex. For the sake of simplicity, when speaking 155 of a specific queue in this document, "buffer size" refers to the 156 maximum amount of data the buffer may store, which may be measured in 157 bytes or packets. The rest of this memo therefore refers to the 158 maximum queue depth as the size of the buffer for a given 159 communication. 161 In order to meet mostly throughput-based SLA requirements and to 162 avoid packet drops, many home gateway manufacturers resort to 163 increasing the available memory beyond "reasonable values". This 164 increase is also referred to as Bufferbloat [BB2011]. Deploying 165 large unmanaged buffers on the Internet has lead to the increase in 166 end-to-end delay, resulting in poor performance for latency sensitive 167 applications such as real-time multimedia (e.g., voice, video, 168 gaming, etc.). The degree to which this affects modern networking 169 equipment, especially consumer-grade equipment, produces problems 170 even with commonly used web services. Active queue management is 171 thus essential to control queuing delay and decrease network latency. 173 The AQM and Packet Scheduling working group was recently formed 174 within the TSV area to address the problems with large unmanaged 175 buffers in the Internet. Specifically, the AQM WG is tasked with 176 standardizing AQM schemes that not only address concerns with such 177 buffers, but also that are robust under a wide variety of operating 178 conditions. In order to ascertain whether the WG should undertake 179 standardizing an AQM proposal, the WG requires guidelines for 180 assessing AQM proposals. This document provides the necessary 181 characterization guidelines. 183 1.1. Guidelines for AQM designers 185 One of the key objectives behind formulating the guidelines is to 186 help ascertain whether a specific AQM is not only better than drop- 187 tail but also safe to deploy. The guidelines help to quantify AQM 188 schemes' performance in terms of latency reduction, goodput 189 maximization and the trade-off between the two. The guidelines also 190 help to discuss AQM's safe deployment, including self adaptation, 191 stability analysis, fairness, design/implementation complexity and 192 robustness to different operating conditions. 194 This memo details generic characterization scenarios that any AQM 195 proposal MUST be evaluated against. Irrespective of whether or not 196 an AQM is standardized by the WG, we recommend the relevant scenarios 197 and metrics discussed in this document to be considered. This 198 document presents central aspects of an AQM algorithm that MUST be 199 considered whatever the context is, such as burst absorption 200 capacity, RTT fairness or resilience to fluctuating network 201 conditions. These guidelines could not cover every possible aspect 202 of a particular algorithm. In addition, it is worth noting that the 203 proposed criteria are not bound to a particular evaluation toolset. 204 These guidelines do not present context dependent scenarios (such as 205 Wi-Fi, data-centers or rural broadband). 207 This document details how an AQM designer can rate the feasibility of 208 their proposal in different types of network devices (switches, 209 routers, firewalls, hosts, drivers, etc.) where an AQM may be 210 implemented. 212 1.2. Reducing the latency and maximizing the goodput 214 The trade-off between reducing the latency and maximizing the goodput 215 is intrinsically linked to each AQM scheme and is key to evaluating 216 its performance. This trade-off MUST be considered in various 217 scenarios to ensure the safety of an AQM deployment. Whenever 218 possible, solutions should aim at both maximizing goodput and 219 minimizing latency. This document proposes guidelines that enable 220 the reader to quantify (1) reduction of latency, (2) maximization of 221 goodput and (3) the trade-off between the two. 223 Testers SHOULD discuss in a reference document the performance of 224 their proposal in terms of performance and deployment in regards with 225 those of drop-tail: basically, these guidelines provide the tools to 226 understand the deployment costs versus the potential gain in 227 performance of the introduction of the proposed scheme. 229 1.3. Glossary 231 o AQM: there may be confusion whether a scheduling scheme is added 232 to an AQM or is a part of the AQM. The rest of this memo refers to 233 AQM as a dropping policy that does not feature a scheduling 234 scheme. 236 o buffer: a physical volume of memory in which a queue or set of 237 queues are stored. 239 o buffer size: the maximum amount of data that may be stored in a 240 buffer, measured in bytes or packets. 242 1.4. Requirements Language 244 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 245 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 246 document are to be interpreted as described in RFC 2119 [RFC2119]. 248 2. End-to-end metrics 249 End-to-end delay is the result of propagation delay, serialization 250 delay, service delay in a switch, medium-access delay and queuing 251 delay, summed over the network elements in the path. AQM algorithms 252 may reduce the queuing delay by providing signals to the sender on 253 the emergence of congestion, but any impact on the goodput must be 254 carefully considered. This section presents the metrics that SHOULD 255 be used to better quantify (1) the reduction of latency, (2) 256 maximization of goodput and (3) the trade-off between the two. These 257 metrics SHOULD be considered to better assess the performance of an 258 AQM scheme. 260 2.1. Flow Completion time 262 The flow completion time is an important performance metric for the 263 end user. Considering the fact that an AQM scheme may drop packets, 264 the flow completion time is directly linked to the dropping policy of 265 the AQM scheme. This metric helps to better assess the performance 266 of an AQM depending on the flow size. 268 2.2. Packet loss 270 Packet losses, that may occur in a queue, impact on the end-to-end 271 performance at the receiver's side. 273 The tester MUST evaluate, at the receiver: 275 o the long term packet loss probability; 277 o the interval between consecutive losses; 279 o the packet loss pattern. 281 2.3. Packet loss synchronization 283 One goal of an AQM algorithm should be to help with avoiding global 284 synchronization of flows going through the bottleneck buffer on which 285 the AQM operates ([RFC2309]). It is therefore important to assess the 286 "degree" of packet-loss synchronization between flows, with and 287 without the AQM under consideration. 289 As discussed e.g. in [LOSS-SYNCH-MET-08], loss synchronization among 290 flows may be quantified by several, slightly different, metrics that 291 capture different aspects of the same issue. However, in real-world 292 measurements the choice of metric may be imposed by practical 293 considerations (e.g., is there fine-grained information on packet 294 losses in the bottleneck available or not). For the purpose of AQM 295 characterization, a good candidate metric is the global 296 synchronization ratio, measuring the proportion of flows losing 297 packets during a loss event. [YU06] used this metric in real-world 298 experiments to characterize synchronization along arbitrary Internet 299 paths; the full methodology is described in [YU06]. 301 2.4. Goodput 302 Measuring the goodput enables an end-to-end appreciation of how well 303 the AQM improves transport and application performance. The measured 304 end-to-end goodput is linked to the AQM scheme's dropping policy -- 305 the smaller the packet drops, the fewer packets need retransmission, 306 minimizing AQM's impact on transport and application performance. 307 Additionally, an AQM scheme may resort to Explicit Congestion 308 Notification (ECN) marking as an initial means to control delay. 309 Again, marking packets instead of dropping them reduces number of 310 packet retransmissions and increases goodput. Overall, end-to-end 311 goodput values help evaluate the AQM scheme's effectiveness in 312 minimizing packet drops that impact application performance and 313 estimate how well the AQM scheme works with ECN. 315 2.5. Latency and jitter 317 The end-to-end latency differs from the queuing delay: it is linked 318 to the network topology and the path characteristics. Moreover, the 319 jitter strongly depends on the traffic and the topology as well. The 320 introduction of an AQM scheme would impact on these metrics and the 321 end-to-end evaluation of performance SHOULD consider them to better 322 assess the AQM schemes. 324 The guidelines advice that the tester SHOULD determine the minimum, 325 average and maximum measurements for these metrics and the 326 coefficient of variation for their average values as well. 328 2.6. Discussion on the trade-off between latency and goodput 330 The metrics presented in this section MAY be considered, in order to 331 discuss and quantify the trade-off between latency and goodput. 333 This trade-off can also be illustrated with figures following the 334 recommendations of the section 5 of [TCPEVAL2013]. 336 The end-to-end trade-off MUST be considered: 338 o end-to-end delay vs. goodput: the x-axis shows the average end- 339 to-end delay and the y-axis the average goodput; 341 o drop rate vs. end-to-end delay: the x-axis shows the end-to-end 342 delay and the y-axis the drop rate. 344 This pair of graphs provide part of a better understanding (1) of the 345 delay/goodput/drop-rate trade-off for a given congestion control 346 mechanism, and (2) of how the goodput and average queue size vary as 347 a function of the traffic load. 349 3. Generic set up for evaluations 351 This section presents the topology that can be used for each of the 352 following scenarios, the corresponding notations and discuss various 353 assumptions that have been made in the document. 355 3.1. Topology and notations 357 +---------+ +-----------+ 358 |senders A| |receivers B| 359 +---------+ +-----------+ 361 +--------------+ +--------------+ 362 |traffic class1| |traffic class1| 363 |--------------| |--------------| 364 | SEN.Flow1.1 +---------+ +-----------+ REC.Flow1.1 | 365 | + | | | | + | 366 | | | | | | | | 367 | + | | | | + | 368 | SEN.Flow1.X +-----+ | | +--------+ REC.Flow1.X | 369 +--------------+ | | | | +--------------+ 370 + +-+---+---+ +--+--+---+ + 371 | |Router L | |Router R | | 372 | |---------| |---------| | 373 | | AQM | | | | 374 | | BuffSize| | | | 375 | | (Bsize) +-----+ | | 376 | +-----+--++ ++-+------+ | 377 + | | | | + 378 +--------------+ | | | | +--------------+ 379 |traffic classN| | | | | |traffic classN| 380 |--------------| | | | | |--------------| 381 | SEN.FlowN.1 +---------+ | | +-----------+ REC.FlowN.1 | 382 | + | | | | + | 383 | | | | | | | | 384 | + | | | | + | 385 | SEN.FlowN.Y +------------+ +-------------+ REC.FlowN.Y | 386 +--------------+ +--------------+ 388 Figure 1 is a generic topology where: 390 o various classes of traffic can be introduced; 392 o the timing of each flow (i.e., when does each flow start and stop) 393 may be different; 395 o each class of traffic can consider various number of flows; 397 o each link is characterized by a couple (RTT,capacity); 399 o Flows are generated between A and B, sharing a bottleneck (Routers 400 L and R); 402 o The links are supposed to be asymmetric in terms of bandwidth: the 403 capacity from senders to receivers is higher than the one from 404 receivers to senders. 406 This topology may not perfectly reflect actual topologies, however, 407 this simple topology is commonly used in the world of simulations and 408 small testbeds. This topology can be considered as adequate to 409 evaluate AQM proposals, such as proposed in [TCPEVAL2013]. The 410 tester should pay attention to the topology that has been used to 411 evaluate the AQM scheme against which he compares his proposal. 413 3.2. Buffer size 415 The size of the buffers MAY be carefully set considering the 416 bandwidth-delay product. However, if the context or the application 417 requires a specific buffer size, the tester MUST justify and detail 418 the way the maximum queue depth is set while presenting the results 419 of its evaluation. Indeed, the size of the buffer may impact on the 420 AQM performance and is a dimensioning parameter that will be 421 considered for a fair comparison between AQM proposals. 423 3.3. Congestion controls 425 This memo features three kind of congestion controls: 427 o TCP-friendly congestion controls: a base-line congestion control 428 for this category is TCP New Reno, as explained in [RFC5681]. 430 o Aggressive congestion controls: a base-line congestion control for 431 this category is TCP Cubic. 433 o Less-than Best Effort (LBE) congestion controls: an LBE congestion 434 control 'results in smaller bandwidth and/or delay impact on 435 standard TCP than standard TCP itself, when sharing a bottleneck 436 with it.' [RFC6297] 438 Recent transport layer protocols are not mentioned in the following 439 sections, for the sake of simplicity. 441 4. Various TCP variants 443 Network and end devices need to be configured with a reasonable 444 amount of buffers in order to absorb transient bursts. In some 445 situations, network providers configure devices with large buffers to 446 avoid packet drops and increase goodput. Transmission Control 447 Protocol (TCP) fills up these unmanaged buffers until the TCP sender 448 receives a signal (packet drop) to cut down the sending rate. The 449 larger the buffer, the higher the buffer occupancy, and therefore the 450 queuing delay. On the other hand, an efficient AQM scheme sends out 451 early congestion signals to TCP senders so that the queuing delay is 452 brought under control. 454 Not all applications run over the same flavor of TCP. Variety of 455 senders generate different classes of traffic which may not react to 456 congestion signals (aka unresponsive flows) or may not cut down their 457 sending rate as expected (aka aggressive flows): AQM schemes aim at 458 maintaining the queuing delay under control, which is challenged if 459 blasting traffics are present. 461 This section provides guidelines to assess the performance of an AQM 462 proposal based on various metrics presented in Section 2 irrespective 463 of traffic profiles involved -- different senders (TCP variants, 464 unresponsive, aggressive), traffic mix with different applications, 465 etc. 467 4.1. TCP-friendly Sender 469 This scenario helps to evaluate how an AQM scheme reacts to a TCP- 470 friendly transport sender. A single long-lived, non application 471 limited, TCP New Reno flow transmits data between sender A and 472 receiver B. Other TCP friendly congestion control schemes such as 473 TCP-friendly rate control [RFC5348] etc MAY also be considered. 475 For each TCP-friendly transport considered, the graphs described in 476 Section 2.6 MUST be generated. 478 4.2. Aggressive Transport Sender 480 This scenario helps to evaluate how an AQM scheme reacts to a 481 transport sender whose sending rate is more aggressive than a single 482 TCP-friendly sender. A single long-lived, non application limited, 483 TCP Cubic flow transmits data between sender A and receiver B. Other 484 aggressive congestion control schemes MAY also be considered. 486 For each flavor of aggressive transport, the graphs described in 487 Section 2.6 MUST be generated. 489 4.3. Unresponsive Transport Sender 491 This scenario helps evaluate how an AQM scheme reacts to a transport 492 sender who is not responsive to congestion signals (ECN marks and/or 493 packet drops) from the AQM scheme. In order to create a test 494 environment that results in queue build up, we consider unresponsive 495 flow(s) whose sending rate is greater than the bottleneck link 496 capacity between routers L and R. Note that faulty transport 497 implementations on end hosts and/or faulty network elements en-route 498 that "hide" congestion signals in packet headers [I-D.ietf-aqm- 499 recommendation] may also lead to a similar situation, such that the 500 AQM scheme needs to adapt to unresponsive traffic. 502 This scenario consists of long-lived non application limited UDP flow 503 transmits data between sender A and receiver B. Graphs described in 504 Section 2.6 MUST be generated. 506 4.4. TCP initial congestion window 507 This scenario helps evaluate how an AQM scheme adapts to a traffic 508 mix consisting of different variants of TCP for various values of the 509 initial congestion window (IW): 511 o TCP: Cubic and/or New Reno; 513 o IW: 3 or 10 packets. 515 Figure 2 presents the various cases for the traffic that MUST be 516 generated between sender A and receiver B. 518 +----+-----------------------------------------------+ 519 |Case| Number of flows | 520 + +-----------+------------+----------+-----------+ 521 | |Cubic (IW3)|Cubic (IW10)|Reno (IW3)|Reno (IW10)| 522 +----+-----------+------------+----------+-----------+ 523 |I | 1 | 1 | 0 | 0 | 524 | | | | | | 525 |II | 0 | 0 | 1 | 1 | 526 | | | | | | 527 |III | 1 | 0 | 1 | 0 | 528 | | | | | | 529 |IV | 0 | 1 | 0 | 1 | 530 +----+-----------+------------+----------+-----------+ 532 For each of these scenarios, the graphs described in Section 2.6 MUST 533 be generated for each class of traffic. 535 4.5. Traffic Mix 537 This scenario helps to evaluate how an AQM scheme reacts to a traffic 538 mix consisting of different applications such as bulk transfer, web, 539 voice, video traffic. These testing cases presented in this 540 subsection have been inspired by the table 2 of [DOCSIS2013]: 542 o Bulk TCP transfer 544 o Web traffic 546 o VoIP 548 o Constant bit rate UDP traffic 550 o Adaptive video streaming 552 Figure 3 presents the various cases for the traffic that MUST be 553 generated between sender A and receiver B. 555 +----+-----------------------------+ 556 |Case| Number of flows | 557 + +----+----+----+---------+----+ 558 | |VoIP|Webs|CBR |AdaptVid |FTP | 559 +----+----+----+----+---------+----+ 560 |I | 1 | 1 | 0 | 0 | 0 | 561 | | | | | | | 562 |II | 1 | 1 | 0 | 0 | 1 | 563 | | | | | | | 564 |III | 1 | 1 | 0 | 0 | 5 | 565 | | | | | | | 566 |IV | 1 | 1 | 1 | 0 | 5 | 567 | | | | | | | 568 |V | 1 | 1 | 0 | 1 | 5 | 569 | | | | | | | 570 +----+----+----+----+---------+----+ 572 For each of these scenarios, the graphs described in Section 2.6 MUST 573 be generated for each class of traffic. In addition, other metrics 574 such as end-to-end latency, jitter and flow completion time MUST be 575 generated. 577 5. RTT fairness 579 5.1. Motivation 581 The capability of AQM schemes to control the queuing delay highly 582 depends on the way end-to-end protocols react to congestion signals. 583 When the RTT varies, the behaviour of congestion controls is impacted 584 and so the capability of AQM schemes to control the queue. It is 585 therefore important to assess the AQM schemes against a set of RTTs 586 (e.g., from 5 ms to 200 ms). 588 Also, asymmetry in terms of RTT between various paths SHOULD be 589 considered so that the fairness between the flows can be discussed as 590 one may react faster to congestion than another. The introduction of 591 AQM schemes may improve this fairness. 593 Moreover, introducing an AQM scheme may result in the absence of 594 fairness between the flows, even when the RTTs are identical. This 595 potential lack of fairness SHOULD be evaluated. 597 5.2. Required tests 599 The topology that SHOULD be used is detailed in Figure 1: 601 o to evaluate the inter-RTT fairness, for each run, ten flows 602 divided into two categories. Category I (Flow1.1, ..., Flow1.5) 603 which RTT between sender A and Router L SHOULD be 5ms. Category 604 II (Flow2.1, ..., Flow 2.5) which RTT between sender A and Router 605 L SHOULD be in [5ms;200ms]. 607 o to evaluate the impact of the RTT value on the AQM performance and 608 the intra-protocol fairness, for each run, ten flows (Flow1.1, 609 ..., Flow1.5 and Flow2.1, ..., Flow2.5) SHOULD be introduced. For 610 each experiment, the set of RTT SHOULD be the same for all the 611 flows and in [5ms;200ms]. 613 These flows MUST use the same congestion control algorithm. 615 5.3. Metrics to evaluate the RTT fairness 617 The output that MUST be measured is: 619 o for the inter-RTT fairness: (1) the cumulated average goodput of 620 the flows from Category I, goodput_Cat_I (Section 2.4); (2) the 621 cumulated average goodput of the flows from Category II, 622 goodput_Cat_II (Section 2.4); (3) the ratio goodput_Cat_II/ 623 goodput_Cat_I; (4) the average packet drop rate for each category 624 (Section 2.2). 626 o for the intra-protocol RTT fairness: (1) the cumulated averga 627 goodput of the ten flows (Section 2.4); (2) the average packet 628 drop rate for the ten flows(Section 2.2). 630 6. Burst absorption 632 6.1. Motivation 634 Packet arrivals can be bursty due to various reasons. Dropping one 635 or more packets from a burst may result in performance penalties for 636 the corresponding flows since the dropped packets have to be 637 retransmitted. Performance penalties may turn into unmet SLAs and be 638 disincentives to AQM adoption. Therefore, an AQM scheme SHOULD be 639 designed to accommodate transient bursts. AQM schemes do not present 640 the same tolerance to bursts of packets arriving in the buffer: this 641 tolerance MUST be quantified. 643 Note that accommodating bursts translates to higher queue length and 644 queuing delay. Naturally, it is important that the AQM scheme brings 645 bursty traffic under control quickly. On the other hand, spiking 646 packet drops in order to bring packet bursts quickly under control 647 could result in multiple drops per flow and severely impact transport 648 and application performance. Therefore, an AQM scheme SHOULD bring 649 bursts under control by balancing both aspects -- (1) queuing delay 650 spikes are minimized and (2) performance penalties for ongoing flows 651 in terms of packet drops are minimized. 653 An AQM scheme maintains short queues to allow the remaining space in 654 the queue for bursts of packets. The tolerance to bursts of packets 655 depends on the number of packets in the queue, which is directly 656 linked to the AQM algorithm. Moreover, one AQM scheme may implement 657 a feature controlling the maximum size of accepted bursts, that may 658 depend on the buffer occupancy or the currently estimated queuing 659 delay. Also, the impact of the buffer size on the burst allowance 660 MAY be evaluated. 662 6.2. Required tests 664 For this scenario, the following traffic MUST be generated from 665 sender A to receiver B: 667 o IW10: TCP transfer with initial congestion window set to 10 of 668 5MB; 670 o Bursty video frames; 672 o Web traffic; 674 o Constant bit rate UDP traffic. 676 Figure 4 presents the various cases for the traffic that MUST be 677 generated between sender A and receiver B. 679 +-----------------------------------------+ 680 |Case| Number of traffic | 681 | +-----+----+----+--------------------+ 682 | |Video|Webs| CBR| Bulk Traffic (IW10)| 683 +----|-----|----|----|--------------------| 684 |I | 0 | 1 | 1 | 0 | 685 |----|-----|----|----|--------------------| 686 |II | 0 | 1 | 1 | 1 | 687 |----|-----|----|----|--------------------| 688 |III | 1 | 1 | 1 | 0 | 689 +----|-----|----|----|--------------------| 690 |IV | 1 | 1 | 1 | 0 | 691 +----|-----|----|----|--------------------| 692 |V | 1 | 1 | 1 | 1 | 693 +----+-----+----+----+--------------------+ 695 6.3. Metrics to evaluate the burst absorption capacity 697 For each of these scenarios, the graphs described in Section 2.6 MUST 698 be generated. In addition, other metrics such as end-to-end latency, 699 jitter, flow completion time MUST be generated. 701 7. Stability 703 7.1. Motivation 704 Network devices experience varying operating conditions depending on 705 factors such as time of day, deployment scenario etc. For example: 707 o Traffic and congestion levels are higher during peak hours than 708 off-peak hours. 710 o In the presence of scheduler, a queue's draining rate may vary 711 depending on other queues: a low load on a high priority queue 712 implies higher draining rate for lower priority queues. 714 o The available capacity on the physical layer may vary over time 715 such as in the context of lossy channels. 717 Whether the target context is a not stable environment, the 718 capability of an AQM scheme to actually maintain its control on the 719 queuing delay and buffer occupancy is challenged. This document 720 propose guidelines to assess the behaviour of AQM schemes under 721 varying congestion levels and varying draining rates. 723 7.2. Required tests 725 7.2.1. Mild Congestion 727 This scenario helps to evaluate how an AQM scheme reacts to a light 728 load of incoming traffic resulting in mild congestion -- packet drop 729 rates less than 1%. Each single-lived non application limited TCP 730 flow transfers data. 732 For this scenario, the graphs described in Section 2.6 MUST be 733 generated. 735 7.2.2. Medium Congestion 737 This scenario helps to evaluate how an AQM scheme reacts to incoming 738 traffic resulting in medium congestion -- packet drop rates between 739 1%-3%. Each single-lived non application limited TCP flow transfers 740 data. 742 For this scenario, the graphs described in Section 2.6 MUST be 743 generated. 745 7.2.3. Heavy Congestion 747 This scenario helps to evaluate how an AQM scheme reacts to incoming 748 traffic resulting in heavy congestion -- packet drop rates between 749 5%-10%. Each single lived non application limited TCP flow transfers 750 data. 752 For this scenario, the graphs described in Section 2.6 MUST be 753 generated. 755 7.2.4. Varying Available Bandwidth 756 This scenario helps evaluate how an AQM scheme adapts to varying 757 available bandwidth on the outgoing link. 759 To simulate varying draining rates, the bottleneck bandwidth between 760 nodes 'Router L' and 'Router R' varies over the course of the 761 experiment as follows: 763 o Experiment 1: the capacity varies between two values according to 764 a large time scale. As an example, the following phases may be 765 considered: phase I - 100Mbps during 0-5s; phase II - 10Mbps 766 during 5-10s: phase I again, ... and so on. 768 o Experiment 2: the capacity varies between two values according to 769 a short time scale. As an example, the following phases may be 770 considered: phase I - 100Mbps during 100ms; phase II - 10Mbps 771 during 100ms; phase I again during 100ms, ... and so on. 773 More realistic fluctuating bandwidth patterns MAY be considered. 775 The scenario consists of TCP New Reno flows between sender A and 776 receiver B. In order to better assess the impact of draining rates on 777 the AQM behavior, the tester MUST compare its performance with those 778 of drop-tail. 780 For this scenario, the graphs described in Section 2.6 MUST be 781 generated. Moreover, one graph SHOULD be generated for each of the 782 phases previously detailed. 784 7.3. Parameter sensitivity and stability analysis 786 An AQM scheme's control law is the primary means by which the AQM 787 controls queuing delay. Hence understanding the AQM control law is 788 critical to understanding AQM behavior. The AQM's control law may 789 include several input parameters whose values affect the AQM output 790 behavior and stability. Additionally, AQM schemes may auto-tune 791 parameter values in order to maintain stability under different 792 network conditions (such as different congestion levels, draining 793 rates or network environments). The stability of these auto-tuning 794 techniques is also important to understand. 796 AQM proposals SHOULD provide background material showing control 797 theoretic analysis of the AQM control law and the input parameter 798 space within which the control law operates as expected; or could use 799 other ways to discuss its stability. For parameters that are auto- 800 tuned, the material SHOULD include stability analysis of the auto- 801 tuning mechanism(s) as well. Such analysis helps to understand an 802 AQM's control law better and the network conditions/deployments 803 under which the AQM is stable. 805 8. Implementation cost 807 8.1. Motivation 808 An AQM's successful deployment is directly related to its ease of 809 implementation. Network devices may need hardware or software 810 implementations of the AQM. Depending on a device's capabilities and 811 limitations, the device may or may not be able to implement some or 812 all parts of the AQM logic. 814 AQM proposals SHOULD provide pseudo-code for the complete AQM scheme, 815 highlighting generic implementation-specific aspects of the scheme 816 such as "drop-tail" vs. "drop-head", inputs (current queuing delay, 817 queue length), computations involved, need for timers etc. This 818 helps identify costs associated with implementing the AQM on a 819 particular hardware or software device. Also, it helps the WG 820 understand which kind of devices can easily support the AQM and 821 which cannot. 823 8.2. Required discussion 825 AQM proposals SHOULD highlight parts of AQM logic that are device 826 dependent and discuss if and how AQM behavior could be impacted by 827 the device. For example, a queue-delay based AQM scheme requires 828 current queuing delay as input from the device. If the device 829 already maintains this value, then it is trivial to implement the AQM 830 logic on the device. On the other hand, if the device provides 831 indirect means to estimate queuing delay (for example: timestamps, 832 dequeing rate etc.), then the AQM behavior is sensitive to how good 833 the queuing delay estimate turns out on that device. Highlighting 834 the AQM's sensitivity to queuing delay estimate helps implementers 835 identify optimal means of implementing the AQM on a device. 837 9. Operator control knobs and auto-tuning 839 One of the biggest hurdles for RED deployment was/is its parameter 840 sensitivity to operating conditions -- how difficult it is to tune 841 important RED parameters for a deployment in order to get maximum 842 benefit from the RED implementation. Fluctuating congestion levels 843 and network conditions add to the complexity. Incorrect parameter 844 values lead to poor performance. This is one reason why RED is 845 reported to be usually turned off. 847 Any AQM scheme is likely to have parameters whose values affect the 848 AQM's control law and behavior. Exposing all these parameters as 849 control knobs to a network operator (or user) can easily result in an 850 unsafe AQM deployment. Unexpected AQM behavior ensues when parameter 851 values are not set properly. A minimal number of control knobs 852 minimizes the number of ways a, possible naive, user can break the 853 AQM system. Fewer control knobs make the AQM scheme more user- 854 friendly and easier to deploy and debug. 856 We recommend that an AQM scheme SHOULD minimize the number of control 857 knobs exposed for operator tuning. An AQM scheme SHOULD expose only 858 those knobs that control the macroscopic AQM behavior such as queue 859 delay threshold, queue length threshold, etc. 861 Additionally, an AQM scheme's safety is directly related to its 862 stability under varying operating conditions such as varying traffic 863 profiles and fluctuating network conditions, as described in Section 864 7. Operating conditions vary often and hence it is necessary that the 865 AQM MUST remain stable under these conditions without the need for 866 additional external tuning. If AQM parameters require tuning under 867 these conditions, then the AQM MUST self-adapt necessary parameter 868 values by employing auto-tuning techniques. 870 10. Interaction with ECN 872 10.1. Motivation 874 Apart from packet drops, Explicit Congestion Notification (ECN) is an 875 alternative means to signal data senders about network congestion. 876 The AQM recommendation document [I-D.ietf-aqm-recommendation] 877 describes some of the benefits of using ECN with AQM. 879 10.2. Required discussion 881 An AQM scheme MAY support ECN, in which case testers MUST discuss and 882 describe the support of ECN. 884 11. Interaction with scheduling 886 11.1. Motivation 888 Coupled with an AQM scheme, a router may schedule the transmission of 889 packets in a specific manner by introducing a scheduling scheme. 890 This algorithm may create sub-queues and integrate a dropping policy 891 on each of these sub-queues. Another scheduling policy may modify 892 the way packets are sequenced, modifying the timestamp of each 893 packet. 895 11.2. Required discussion 897 The scheduling and the AQM conjointly impact on the end-to-end 898 performance. During the characterization process of a dropping 899 policy, the tester MAY discuss the feasibility to add scheduling on 900 top of its algorithm. This discussion MAY detail if the dropping 901 policy is applied while packets are enqueued or dequeued. 903 12. Discussion on methodology, metrics, AQM comparisons and packet 904 sizes 906 12.1. Methodology 908 A sufficiently detailed description of the test setup SHOULD be 909 provided. Indeed, that would allow other to replicate the tests if 910 needed. This test setup MAY include software and hardware versions. 911 The tester MAY make its data available. 913 The proposals SHOULD be experimented on real systems, or they MAY be 914 evaluated with event-driven simulations (such as NS-2, NS-3, OMNET, 915 etc.). The proposed scenarios are not bound to a particular 916 evaluation toolset. 918 12.2. Comments on metrics measurement 920 In this document, we present the end-to-end metrics that SHOULD be 921 evaluated to evaluate the trade-off between latency and goodput. The 922 queue-related metrics enable a better understanding of the AQM 923 behavior under tests and the impact of its internal parameters. 924 Whenever it is possible, these guidelines advice to consider queue- 925 related metrics, such as link utilization, queuing delay, queue size 926 or packet loss. 928 These guidelines could hardly detail the way the metrics can be 929 measured depends highly on the evaluation toolset. 931 12.3. Comparing AQM schemes 933 This memo recognizes that the guidelines mentioned above may be used 934 for comparing AQM schemes. This memo recommends that AQM schemes 935 MUST be compared against both performance and deployment categories. 936 In addition, this section details how best to achieve a fair 937 comparison of AQM schemes by avoiding certain pitfalls. 939 12.3.1. Performance comparison 941 AQM schemes MUST be compared against all the generic scenarios 942 presented in this memo. AQM schemes MAY be compared for specific 943 network environments such as data center, home networks etc. If an 944 AQM scheme's parameter(s) were externally tuned for optimization or 945 other purposes, these values MUST be disclosed. 947 Note that AQM schemes belong to different varieties such as queue- 948 length based scheme (ex: RED) or queue-delay based scheme (ex: CoDel, 949 PIE). Also, AQM schemes expose different control knobs associated 950 with different semantics. For example, while both PIE and CoDel are 951 queue-delay based schemes and each expose a knob to control the 952 queueing delay -- PIE's "queueing delay reference" vs. CoDel's 953 "queueing delay target", the two schemes' knobs have different 954 semantics resulting in different control points. Such differences in 955 AQM schemes can be easily overlooked while making comparisons. 957 This document recommends the following procedures for a fair 958 performance comparison of two AQM schemes: 960 1. comparable control parameters and comparable input values: 961 carefully identify the set of parameters that control similar 962 behavior between the two AQM schemes and ensure these parameters 963 have comparable input values. For example, while comparing how 964 well a queue-length based AQM X controls queueing delay vs. 965 queue-delay based AQM Y, identify the two schemes' parameters 966 that control queue delay and ensure that their input values are 967 comparable. Similarly, to compare two AQM schemes on how well 968 they accommodate bursts, identify burst-related control 969 parameters and ensure they are configured with similar values. 971 2. compare over a range of input configurations: there could be 972 situations when the set of control parameters that affect a 973 specific behavior have different semantics between the two AQM 974 schemes. As mentioned above, PIE's knob to control queue delay 975 has different semantics from CoDel's. In such situations, the 976 schemes MUST be compared over a range of input configurations. 977 For example, compare PIE vs. CoDel over the range of delay input 978 configurations -- 5ms, 10ms, 15ms etc. 980 12.3.2. Deployment comparison 982 AQM schemes MUST be compared against deployment criteria such as the 983 parameter sensitivity (Section 7.3), the auto-tuning (Section 9) or 984 the implementation cost (Section 8). 986 12.4. Packet sizes and congestion notification 988 An AQM scheme may be considering packet sizes while generating 989 congestion signals. [RFC7141] discusses the motivations behind the 990 same. For example, control packets such as DNS requests/responses, 991 TCP SYNs/ACKs are small, and their loss can severely impact 992 application performance. An AQM scheme may therefore be biased 993 towards small packets by dropping them with smaller probability 994 compared to larger packets. However, such an AQM scheme is unfair to 995 data senders generating larger packets. Data senders, malicious or 996 otherwise, are motivated to take advantage of the AQM scheme by 997 transmitting smaller packets, and could result in unsafe deployments 998 and unhealthy transport and/or application designs. 1000 An AQM scheme SHOULD adhere to recommendations outlined in [RFC7141], 1001 and SHOULD NOT provide undue advantage to flows with smaller packets. 1003 13. Acknowledgements 1005 This work has been partially supported by the European Community 1006 under its Seventh Framework Programme through the Reducing Internet 1007 Transport Latency (RITE) project (ICT-317700). 1009 14. Contributors 1010 Many thanks to S. Akhtar, A.B. Bagayoko, F. Baker, D. Collier-Brown, 1011 G. Fairhurst, T. Hoiland-Jorgensen, C. Kulatunga, R. Pan, D. Taht and 1012 M. Welzl for detailed and wise feedback on this document. 1014 15. IANA Considerations 1016 This memo includes no request to IANA. 1018 16. Security Considerations 1020 This document, by itself, presents no new privacy nor security 1021 issues. 1023 17. References 1025 17.1. Normative References 1027 [I-D.ietf-aqm-recommendation] 1028 Baker, F. and G. Fairhurst, "IETF Recommendations 1029 Regarding Active Queue Management", Internet-Draft draft- 1030 ietf-aqm-recommendation-07, August 2014. 1032 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1033 Requirement Levels", RFC 2119, 1997. 1035 [RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion 1036 Notification", RFC 7141, 2014. 1038 17.2. Informative References 1040 [BB2011] "BufferBloat: what's wrong with the internet?", ACM Queue 1041 vol. 9, 2011. 1043 [CODEL] Nichols, K. and V. Jacobson, "Controlling Queue Delay", 1044 ACM Queue , 2012. 1046 [DOCSIS2013] 1047 White, G. and D. Rice, "Active Queue Management Algorithms 1048 for DOCSIS 3.0", Technical report - Cable Television 1049 Laboratories , 2013. 1051 [LOSS-SYNCH-MET-08] 1052 Hassayoun, S. and D. Ros, "Loss Synchronization and Router 1053 Buffer Sizing with High-Speed Versions of TCP", IEEE 1054 INFOCOM Workshops , 2008. 1056 [PIE] Pan, R., Natarajan, P., Piglione, C., Prabhu, MS., 1057 Subramanian, V., Baker, F. and B. VerSteeg, "PIE: A 1058 lightweight control scheme to address the bufferbloat 1059 problem", IEEE HPSR , 2013. 1061 [RFC2309] Braden, B., Clark, D.D., Crowcroft, J., Davie, B., 1062 Deering, S., Estrin, D., Floyd, S., Jacobson, V., 1063 Minshall, G., Partridge, C., Peterson, L., Ramakrishnan, 1064 K.K., Shenker, S., Wroclawski, J. and L. Zhang, 1065 "Recommendations on Queue Management and Congestion 1066 Avoidance in the Internet", RFC 2309, April 1998. 1068 [RFC5348] Floyd, S., Handley, M., Padhye, J. and J. Widmer, "TCP 1069 Friendly Rate Control (TFRC): Protocol Specification", RFC 1070 5348, September 2008. 1072 [RFC5681] Allman, M., Paxson, V. and E. Blanton, "TCP Congestion 1073 Control", RFC 5681, September 2009. 1075 [RFC6297] Welzl, M. and D. Ros, "A Survey of Lower-than-Best-Effort 1076 Transport Protocols", RFC 6297, June 2011. 1078 [TCPEVAL2013] 1079 Hayes, D., Ros, D., Andrew, L.L.H. and S. Floyd, "Common 1080 TCP Evaluation Suite", IRTF ICCRG , 2013. 1082 [YU06] Jay, P., Fu, Q. and G. Armitage, "A preliminary analysis 1083 of loss synchronisation between concurrent TCP flows", 1084 Australian Telecommunication Networks and Application 1085 Conference (ATNAC) , 2006. 1087 Authors' Addresses 1089 Nicolas Kuhn, editor 1090 Telecom Bretagne 1091 2 rue de la Chataigneraie 1092 Cesson-Sevigne, 35510 1093 France 1095 Phone: +33 2 99 12 70 46 1096 Email: nicolas.kuhn@telecom-bretagne.eu 1098 Preethi Natarajan, editor 1099 Cisco Systems 1100 510 McCarthy Blvd 1101 Milpitas, California 1102 United States 1104 Email: prenatar@cisco.com 1105 David Ros 1106 Simula Research Laboratory AS 1107 P.O. Box 134 1108 Lysaker, 1325, 1109 Norway 1111 Phone: +33 299 25 21 21 1112 Email: dros@simula.no 1114 Naeem Khademi 1115 University of Oslo 1116 Department of Informatics, PO Box 1080 Blindern 1117 N-0316 Oslo, 1118 Norway 1120 Phone: +47 2285 24 93 1121 Email: naeemk@ifi.uio.no