idnits 2.17.1 draft-ietf-aqm-eval-guidelines-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 21, 2015) is 3262 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'B' is mentioned on line 291, but not defined == Missing Reference: 'Mbps' is mentioned on line 291, but not defined -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) -- Obsolete informational reference (is this intentional?): RFC 2679 (Obsoleted by RFC 7679) -- Obsolete informational reference (is this intentional?): RFC 2680 (Obsoleted by RFC 7680) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force N. Kuhn, Ed. 3 Internet-Draft Telecom Bretagne 4 Intended status: Informational N. Khademi, Ed. 5 Expires: November 22, 2015 University of Oslo 6 P. Natarajan, Ed. 7 Cisco Systems 8 D. Ros 9 Simula Research Laboratory AS 10 May 21, 2015 12 AQM Characterization Guidelines 13 draft-ietf-aqm-eval-guidelines-03 15 Abstract 17 Unmanaged large buffers in today's networks have given rise to a slew 18 of performance issues. These performance issues can be addressed by 19 some form of Active Queue Management (AQM) mechanism, optionally in 20 combination with a packet scheduling scheme such as fair queuing. 21 The IETF Active Queue Management and Packet Scheduling working group 22 was formed to standardize AQM schemes that are robust, easily 23 implementable, and successfully deployable in today's networks. This 24 document describes various criteria for performing precautionary 25 characterizations of AQM proposals. This document also helps in 26 ascertaining whether any given AQM proposal should be taken up for 27 standardization by the AQM WG. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at http://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on November 22, 2015. 46 Copyright Notice 48 Copyright (c) 2015 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (http://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with respect 56 to this document. Code Components extracted from this document must 57 include Simplified BSD License text as described in Section 4.e of 58 the Trust Legal Provisions and are provided without warranty as 59 described in the Simplified BSD License. 61 Table of Contents 63 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 64 1.1. Reducing the latency and maximizing the goodput . . . . . 5 65 1.2. Guidelines for AQM evaluation . . . . . . . . . . . . . . 5 66 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 6 67 1.4. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 6 68 2. End-to-end metrics . . . . . . . . . . . . . . . . . . . . . 6 69 2.1. Flow completion time . . . . . . . . . . . . . . . . . . 7 70 2.2. Packet loss . . . . . . . . . . . . . . . . . . . . . . . 7 71 2.3. Packet loss synchronization . . . . . . . . . . . . . . . 8 72 2.4. Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 8 73 2.5. Latency and jitter . . . . . . . . . . . . . . . . . . . 9 74 2.6. Discussion on the trade-off between latency and goodput . 9 75 3. Generic set up for evaluations . . . . . . . . . . . . . . . 10 76 3.1. Topology and notations . . . . . . . . . . . . . . . . . 10 77 3.2. Buffer size . . . . . . . . . . . . . . . . . . . . . . . 12 78 3.3. Congestion controls . . . . . . . . . . . . . . . . . . . 12 79 4. Transport Protocols . . . . . . . . . . . . . . . . . . . . . 13 80 4.1. TCP-friendly sender . . . . . . . . . . . . . . . . . . . 13 81 4.1.1. TCP-friendly sender with the same initial congestion 82 window . . . . . . . . . . . . . . . . . . . . . . . 13 83 4.1.2. TCP-friendly sender with different initial congestion 84 windows . . . . . . . . . . . . . . . . . . . . . . . 14 85 4.2. Aggressive transport sender . . . . . . . . . . . . . . . 14 86 4.3. Unresponsive transport sender . . . . . . . . . . . . . . 15 87 4.4. Less-than Best Effort transport sender . . . . . . . . . 15 88 5. Round Trip Time Fairness . . . . . . . . . . . . . . . . . . 16 89 5.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 16 90 5.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 16 91 5.3. Metrics to evaluate the RTT fairness . . . . . . . . . . 17 92 6. Burst Absorption . . . . . . . . . . . . . . . . . . . . . . 17 93 6.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 17 94 6.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 18 95 7. Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 18 96 7.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 18 97 7.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 19 98 7.2.1. Definition of the congestion Level . . . . . . . . . 19 99 7.2.2. Mild congestion . . . . . . . . . . . . . . . . . . . 20 100 7.2.3. Medium congestion . . . . . . . . . . . . . . . . . . 20 101 7.2.4. Heavy congestion . . . . . . . . . . . . . . . . . . 20 102 7.2.5. Varying congestion levels . . . . . . . . . . . . . . 20 103 7.2.6. Varying available capacity . . . . . . . . . . . . . 20 104 7.3. Parameter sensitivity and stability analysis . . . . . . 21 105 8. Various Traffic Profiles . . . . . . . . . . . . . . . . . . 22 106 8.1. Traffic mix . . . . . . . . . . . . . . . . . . . . . . . 22 107 8.2. Bi-directional traffic . . . . . . . . . . . . . . . . . 22 108 9. Multi-AQM Scenario . . . . . . . . . . . . . . . . . . . . . 23 109 9.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 23 110 9.2. Details on the evaluation scenario . . . . . . . . . . . 23 111 10. Implementation cost . . . . . . . . . . . . . . . . . . . . . 24 112 10.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 24 113 10.2. Recommended discussion . . . . . . . . . . . . . . . . . 24 114 11. Operator Control and Auto-tuning . . . . . . . . . . . . . . 24 115 11.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 24 116 11.2. Required discussion . . . . . . . . . . . . . . . . . . 25 117 12. Interaction with ECN . . . . . . . . . . . . . . . . . . . . 25 118 12.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 25 119 12.2. Recommended discussion . . . . . . . . . . . . . . . . . 25 120 13. Interaction with Scheduling . . . . . . . . . . . . . . . . . 26 121 13.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 26 122 13.2. Recommended discussion . . . . . . . . . . . . . . . . . 26 123 14. Discussion on Methodology, Metrics, AQM Comparisons and 124 Packet Sizes . . . . . . . . . . . . . . . . . . . . . . . . 26 125 14.1. Methodology . . . . . . . . . . . . . . . . . . . . . . 26 126 14.2. Comments on metrics measurement . . . . . . . . . . . . 27 127 14.3. Comparing AQM schemes . . . . . . . . . . . . . . . . . 27 128 14.3.1. Performance comparison . . . . . . . . . . . . . . . 27 129 14.3.2. Deployment comparison . . . . . . . . . . . . . . . 28 130 14.4. Packet sizes and congestion notification . . . . . . . . 28 131 15. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 29 132 16. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 29 133 17. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 30 134 18. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 135 19. Security Considerations . . . . . . . . . . . . . . . . . . . 30 136 20. References . . . . . . . . . . . . . . . . . . . . . . . . . 30 137 20.1. Normative References . . . . . . . . . . . . . . . . . . 30 138 20.2. Informative References . . . . . . . . . . . . . . . . . 30 139 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32 141 1. Introduction 143 Active Queue Management (AQM) [I-D.ietf-aqm-recommendation] addresses 144 the concerns arising from using unnecessarily large and unmanaged 145 buffers to improve network and application performance. Several AQM 146 algorithms have been proposed in the past years, most notably Random 147 Early Detection (RED), BLUE, and Proportional Integral controller 148 (PI), and more recently CoDel [CODEL] and PIE [PIE]. In general, 149 these algorithms actively interact with the Transmission Control 150 Protocol (TCP) and any other transport protocol that deploys a 151 congestion control scheme to manage the amount of data they keep in 152 the network. The available buffer space in the routers and switches 153 should be large enough to accommodate the short-term buffering 154 requirements. AQM schemes aim at reducing mean buffer occupancy, and 155 therefore both end-to-end delay and jitter. Some of these 156 algorithms, notably RED, have also been widely implemented in some 157 network devices. However, the potential benefits of the RED scheme 158 have not been realized since RED is reported to be usually turned 159 off. The main reason of this reluctance to use RED in today's 160 deployments comes from its sensitivity to the operating conditions in 161 the network and the difficulty of tuning its parameters. 163 A buffer is a physical volume of memory in which a queue or set of 164 queues are stored. In real implementations of switches, a global 165 memory is shared between the available devices: the size of the 166 buffer for a given communication does not make sense, as its 167 dedicated memory may vary over the time and real-world buffering 168 architectures are complex. For the sake of simplicity, when speaking 169 of a specific queue in this document, "buffer size" refers to the 170 maximum amount of data the buffer may store, which can be measured in 171 bytes or packets. The rest of this memo therefore refers to the 172 maximum queue depth as the size of the buffer for a given 173 communication. 175 In order to meet mostly throughput-based Service-Level Agreement 176 (SLA) requirements and to avoid packet drops, many home gateway 177 manufacturers resort to increasing the available memory beyond 178 "reasonable values". This increase is also referred to as 179 Bufferbloat [BB2011]. Deploying large unmanaged buffers on the 180 Internet has lead to an increase in end-to-end delay, resulting in 181 poor performance for latency-sensitive applications such as real-time 182 multimedia (e.g., voice, video, gaming, etc). The degree to which 183 this affects modern networking equipment, especially consumer-grade 184 equipment's, produces problems even with commonly used web services. 185 Active queue management is thus essential to control queuing delay 186 and decrease network latency. 188 The Active Queue Management and Packet Scheduling Working Group (AQM 189 WG) was recently formed within the TSV area to address the problems 190 with large unmanaged buffers in the Internet. Specifically, the AQM 191 WG is tasked with standardizing AQM schemes that not only address 192 concerns with such buffers, but also are robust under a wide variety 193 of operating conditions. In order to ascertain whether the WG should 194 undertake standardizing an AQM proposal, the WG requires guidelines 195 for assessing AQM proposals. This document provides the necessary 196 characterization guidelines. 198 1.1. Reducing the latency and maximizing the goodput 200 The trade-off between reducing the latency and maximizing the goodput 201 is intrinsically linked to each AQM scheme and is key to evaluating 202 its performance. This trade-off MUST be considered in various 203 scenarios to ensure the safety of an AQM deployment. Whenever 204 possible, solutions ought to aim at both maximizing goodput and 205 minimizing latency. This document provides guidelines that enable 206 the reader to quantify (1) reduction of latency, (2) maximization of 207 goodput and (3) the trade-off between the two. 209 These guidelines provide the tools to understand the deployment costs 210 versus the potential gain in performance from the introduction of the 211 proposed scheme. 213 1.2. Guidelines for AQM evaluation 215 The guidelines help to quantify performance of AQM schemes in terms 216 of latency reduction, goodput maximization and the trade-off between 217 these two. The guidelines also help to discuss safe deployment of 218 AQM, including self-adaptation, stability analysis, fairness, design 219 and implementation complexity and robustness to different operating 220 conditions. 222 This memo details generic characterization scenarios against which 223 any AQM proposal must be evaluated, irrespective of whether or not an 224 AQM is standardized by the IETF. This documents recommends the 225 relevant scenarios and metrics to be considered. 227 The document presents central aspects of an AQM algorithm that must 228 be considered whatever the context, such as burst absorption 229 capacity, RTT fairness or resilience to fluctuating network 230 conditions. These guidelines do not cover every possible aspect of a 231 particular algorithm. In addition, it is worth noting that the 232 proposed criteria are not bound to a particular evaluation toolset. 234 This document details how an AQM designer can rate the feasibility of 235 their proposal in different types of network devices (switches, 236 routers, firewalls, hosts, drivers, etc) where an AQM may be 237 implemented. However, these guidelines do not present context- 238 dependent scenarios (such as 802.11 WLANs, data-centers or rural 239 broadband networks). 241 1.3. Requirements Language 243 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 244 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 245 document are to be interpreted as described in RFC 2119 [RFC2119]. 247 1.4. Glossary 249 o AQM: there may be a debate on whether a scheduling scheme is 250 additional to an AQM algorithm or is a part of an AQM algorithm. 251 The rest of this memo refers to AQM as a dropping/marking policy 252 that does not feature a scheduling scheme. 254 o buffer: a physical volume of memory in which a queue or set of 255 queues are stored. 257 o buffer size: the maximum amount of data that may be stored in a 258 buffer, measured in bytes or packets. 260 2. End-to-end metrics 262 End-to-end delay is the result of propagation delay, serialization 263 delay, service delay in a switch, medium-access delay and queuing 264 delay, summed over the network elements along the path. AQM schemes 265 may reduce the queuing delay by providing signals to the sender on 266 the emergence of congestion, but any impact on the goodput must be 267 carefully considered. This section presents the metrics that could 268 be used to better quantify (1) the reduction of latency, (2) 269 maximization of goodput and (3) the trade-off between these two. 270 This section provides normative requirements for metrics that can be 271 used to assess the performance of an AQM scheme. 273 Some metrics listed in this section are not suited to every type of 274 traffic detailed in the rest of this document. It is therefore not 275 necessary to measure all of the following metrics: the chosen metric 276 may not be relevant to the context of the evaluation scenario (e.g. 277 latency vs. goodput trade-off in application-limited traffic 278 scenarios). Guidance is provided for each metric. 280 2.1. Flow completion time 282 The flow completion time is an important performance metric for the 283 end-user when the flow size is finite. Considering the fact that an 284 AQM scheme may drop/mark packets, the flow completion time is 285 directly linked to the dropping/marking policy of the AQM scheme. 286 This metric helps to better assess the performance of an AQM 287 depending on the flow size. The Flow Completion Time (FCT) is 288 related to the flow size (Fs) and the goodput for the flow (G) as 289 follows: 291 FCT [s] = Fs [B] / ( G [Mbps] / 8 ) 293 2.2. Packet loss 295 Packet loss can occur within a network device, this can impact the 296 end-to-end performance measured at receiver. 298 The tester SHOULD evaluate loss experienced at the receiver using one 299 of the two metrics: 301 o the packet loss probability: this metric is to be frequently 302 measured during the experiment. The long-term loss probability is 303 of interest for steady-state scenarios only; 305 o the interval between consecutive losses: the time between two 306 losses is to be measured. 308 The packet loss probability can be assessed by simply evaluating the 309 loss ratio as a function of the number of lost packets and the total 310 number of packets sent. This might not be easily done in laboratory 311 testing, for which these guidelines advice the tester: 313 o to check that for every packet, a corresponding packet was 314 received within a reasonable time, as explained in [RFC2680]. 316 o to keep a count of all packets sent, and a count of the non- 317 duplicate packets received, as explained in the section 10 of 318 [RFC2544]. 320 The interval between consecutive losses, which is also called a gap, 321 is a metric of interest for VoIP traffic and, as a result, has been 322 further specified in [RFC3611]. 324 2.3. Packet loss synchronization 326 One goal of an AQM algorithm ought be to help to avoid global 327 synchronization of flows sharing a bottleneck buffer on which the AQM 328 operates ([RFC2309],[I-D.ietf-aqm-recommendation]). The "degree" of 329 packet-loss synchronization between flows SHOULD be assessed, with 330 and without the AQM under consideration. 332 As discussed e.g. in [LOSS-SYNCH-MET-08], loss synchronization among 333 flows may be quantified by several slightly different metrics that 334 capture different aspects of the same issue. However, in real-world 335 measurements the choice of metric could be imposed by practical 336 considerations -- e.g. whether fine-grained information on packet 337 losses in the bottleneck available or not. For the purpose of AQM 338 characterization, a good candidate metric is the global 339 synchronization ratio, measuring the proportion of flows losing 340 packets during a loss event. [YU06] used this metric in real-world 341 experiments to characterize synchronization along arbitrary Internet 342 paths; the full methodology is described in [YU06]. 344 If an AQM scheme is evaluated using real-life network environments, 345 it is worth pointing out that some network events, such as failed 346 link restoration may cause synchronized losses between active flows 347 and thus confuse the meaning of this metric. 349 2.4. Goodput 351 The goodput has been defined in the section 3.17 of [RFC2647] as the 352 number of bits per unit of time forwarded to the correct destination 353 interface of the Device Under Test (DUT) or the System Under Test 354 (SUT), minus any bits lost or retransmitted. This definition induces 355 that the test setup needs to be qualified to assure that it is not 356 generating losses on its own. 358 Measuring the end-to-end goodput provides an appreciation of how well 359 an AQM scheme improves transport and application performance. The 360 measured end-to-end goodput is linked to the dropping/marking policy 361 of the AQM scheme -- e.g. the fewer the number of packet drops, the 362 fewer packets need retransmission, minimizing the impact of AQM on 363 transport and application performance. Additionally, an AQM scheme 364 may resort to Explicit Congestion Notification (ECN) marking as an 365 initial means to control delay. Again, marking packets instead of 366 dropping them reduces the number of packet retransmissions and 367 increases goodput. End-to-end goodput values help to evaluate the 368 AQM scheme's effectiveness of an AQM scheme in minimizing packet 369 drops that impact application performance and to estimate how well 370 the AQM scheme works with ECN. 372 The measurement of the goodput allows the tester evaluate to which 373 extent an AQM is able to maintain a high bottleneck utilization. 374 This metric should be also obtained frequently during an experiment 375 as the long-term goodput is relevant for steady-state scenarios only 376 and may not necessarily reflect how the introduction of an AQM 377 actually impacts the link utilization during at a certain period of 378 time. Fluctuations in the values obtained from these measurements 379 may depend on other factors than the introduction of an AQM, such as 380 link layer losses due to external noise or corruption, fluctuating 381 bandwidths (802.11 WLANs), heavy congestion levels or transport 382 layer's rate reduction by congestion control mechanism. 384 2.5. Latency and jitter 386 The latency, or the one-way delay metric, is discussed in [RFC2679]. 387 There is a consensus on a adequate metric for the jitter, that 388 represents the one-way delay variations for packets from the same 389 flow: the Packet Delay Variation (PDV), detailed in [RFC5481], serves 390 well all use cases. 392 The end-to-end latency differs from the queuing delay: it is linked 393 to the network topology and the path characteristics. Moreover, the 394 jitter also strongly depends on the traffic pattern and the topology. 395 The introduction of an AQM scheme would impact these metrics and 396 therefore they should be considered in the end-to-end evaluation of 397 performance. 399 2.6. Discussion on the trade-off between latency and goodput 401 The metrics presented in this section may be considered as explained 402 in the rest of this document, in order to discuss and quantify the 403 trade-off between latency and goodput. 405 This trade-off can also be illustrated with figures following the 406 recommendations of section 5 of [TCPEVAL2013]. Each of the end-to- 407 end delay and the goodput SHOULD be measured frequently for every 408 fixed time interval. 410 With regards to the goodput, and in addition to the long-term 411 stationary goodput value, it is RECOMMENDED to take measurements 412 every multiple of RTTs. We suggest a minimum value of 10 x RTT (to 413 smooth out the fluctuations) but higher values are encouraged 414 whenever appropriate for the presentation depending on the network's 415 path characteristics. The measurement period MUST be disclosed for 416 each experiment and when results/values are compared across different 417 AQM schemes, the comparisons SHOULD use exactly the same measurement 418 periods. 420 With regards to latency, it is highly RECOMMENDED to take the samples 421 on per-packet basis whenever possible depending on the features 422 provided by hardware/software and the impact of sampling itself on 423 the hardware performance. It is generally RECOMMENDED to provide at 424 least 10 samples per RTT. 426 From each of these sets of measurements, the 10th and 90th 427 percentiles and the median value SHOULD be computed. For each 428 scenario, a graph can be generated, with the x-axis showing the end- 429 to-end delay and the y-axis the goodput. This graph provides part of 430 a better understanding of (1) the delay/goodput trade-off for a given 431 congestion control mechanism, and (2) how the goodput and average 432 queue size vary as a function of the traffic load. 434 3. Generic set up for evaluations 436 This section presents the topology that can be used for each of the 437 following scenarios, the corresponding notations and discusses 438 various assumptions that have been made in the document. 440 3.1. Topology and notations 441 +---------+ +-----------+ 442 |senders A| |receivers B| 443 +---------+ +-----------+ 445 +--------------+ +--------------+ 446 |traffic class1| |traffic class1| 447 |--------------| |--------------| 448 | SEN.Flow1.1 +---------+ +-----------+ REC.Flow1.1 | 449 | + | | | | + | 450 | | | | | | | | 451 | + | | | | + | 452 | SEN.Flow1.X +-----+ | | +--------+ REC.Flow1.X | 453 +--------------+ | | | | +--------------+ 454 + +-+---+---+ +--+--+---+ + 455 | |Router L | |Router R | | 456 | |---------| |---------| | 457 | | AQM | | | | 458 | | BuffSize| | | | 459 | | (Bsize) +-----+ | | 460 | +-----+--++ ++-+------+ | 461 + | | | | + 462 +--------------+ | | | | +--------------+ 463 |traffic classN| | | | | |traffic classN| 464 |--------------| | | | | |--------------| 465 | SEN.FlowN.1 +---------+ | | +-----------+ REC.FlowN.1 | 466 | + | | | | + | 467 | | | | | | | | 468 | + | | | | + | 469 | SEN.FlowN.Y +------------+ +-------------+ REC.FlowN.Y | 470 +--------------+ +--------------+ 472 Figure 1: Topology and notations 474 Figure 1 is a generic topology where: 476 o various classes of traffic can be introduced; 478 o the timing of each flow could be different (i.e., when does each 479 flow start and stop); 481 o each class of traffic can comprise various number of flows; 483 o each link is characterized by a couple (RTT,capacity); 485 o flows are generated between A and B, sharing a bottleneck (Routers 486 L and R); 488 o the tester SHOULD consider both scenarios of asymmetric and 489 symmetric bottleneck links in terms of bandwidth. In case of 490 asymmetric link, the capacity from senders to receivers is higher 491 than the one from receivers to senders; the symmetric link 492 scenario provides a basic understanding of the operation of the 493 AQM mechanism whereas the asymmetric link scenario evaluates an 494 AQM mechanism in a more realistic setup; 496 o in asymmetric link scenarios, the tester SHOULD study the bi- 497 directional traffic between A and B (downlink and uplink) with the 498 AQM mechanism deployed on one direction only. The tester MAY 499 additionally consider a scenario with AQM mechanism being deployed 500 on both directions. In each scenario, the tester SHOULD 501 investigate the impact of drop policy of the AQM on TCP ACK 502 packets and its impact on the performance. 504 Although this topology may not perfectly reflect actual topologies, 505 the simple topology is commonly used in the world of simulations and 506 small testbeds. It can be considered as adequate to evaluate AQM 507 proposals, similarly to the topology proposed in [TCPEVAL2013]. 508 Testers ought to pay attention to the topology that has been used to 509 evaluate an AQM scheme when comparing this scheme with a new proposed 510 AQM scheme. 512 3.2. Buffer size 514 The size of the buffers should be carefully chosen, and is to be set 515 to the bandwidth-delay product. The size of the buffer can impact on 516 the AQM performance and is a dimensioning parameter that will be 517 considered when comparing AQM proposals. 519 If the context or the application requires a specific buffer size, 520 the tester MUST justify and detail the way the maximum queue size is 521 set. Indeed, the maximum size of the buffer may affect the AQM's 522 performance and its choice SHOULD be elaborated for a fair comparison 523 between AQM proposals. While comparing AQM schemes the buffer size 524 SHOULD remain the same across the tests. 526 3.3. Congestion controls 528 This memo features three kind of congestion controls: 530 o Standard TCP congestion control: the base-line congestion control 531 is TCP NewReno with SACK, as explained in [RFC5681]. 533 o Aggressive congestion controls: a base-line congestion control for 534 this category is TCP Cubic. 536 o Less-than Best Effort (LBE) congestion controls: an LBE congestion 537 control 'results in smaller bandwidth and/or delay impact on 538 standard TCP than standard TCP itself, when sharing a bottleneck 539 with it.' [RFC6297] 541 Other transport congestion controls can OPTIONALLY be evaluatied in 542 addition. Recent transport layer protocols are not mentioned in the 543 following sections, for the sake of simplicity. 545 4. Transport Protocols 547 Network and end-devices need to be configured with a reasonable 548 amount of buffer space to absorb transient bursts. In some 549 situations, network providers tend to configure devices with large 550 buffers to avoid packet drops triggered by a full buffer and to 551 maximize the link utilization for standard loss-based TCP traffic. 553 AQM algorithms are often evaluated by considering Transmission 554 Control Protocol (TCP) [RFC0793] with a limited number of 555 applications. TCP is a widely deployed transport. It fills up 556 unmanaged buffers until the TCP sender receives a signal (packet 557 drop) that reduces the sending rate. The larger the buffer, the 558 higher the buffer occupancy, and therefore the queuing delay. An 559 efficient AQM scheme sends out early congestion signals to TCP to 560 bring the queuing delay under control. 562 Not all applications using TCP use the same flavor of TCP. Variety 563 of senders generate different classes of traffic which may not react 564 to congestion signals (aka non-responsive flows 565 [I-D.ietf-aqm-recommendation]) or may not reduce their sending rate 566 as expected (aka Transport Flows that are less responsive than 567 TCP[I-D.ietf-aqm-recommendation], also called "aggressive flows"). 568 In these cases, AQM schemes seek to control the queuing delay. 570 This section provides guidelines to assess the performance of an AQM 571 proposal for various traffic profiles -- different types of senders 572 (with different TCP congestion control variants, unresponsive, 573 aggressive). 575 4.1. TCP-friendly sender 577 4.1.1. TCP-friendly sender with the same initial congestion window 579 This scenario helps to evaluate how an AQM scheme reacts to a TCP- 580 friendly transport sender. A single long-lived, non application- 581 limited, TCP NewReno flow, with an Initial congestion Window (IW) set 582 to 3 packets, transfers data between sender A and receiver B. Other 583 TCP friendly congestion control schemes such as TCP-friendly rate 584 control [RFC5348] etc MAY also be considered. 586 For each TCP-friendly transport considered, the graph described in 587 Section 2.6 could be generated. 589 4.1.2. TCP-friendly sender with different initial congestion windows 591 This scenario can be used to evaluate how an AQM scheme adapts to a 592 traffic mix consisting of TCP flows with different values of the IW. 594 For this scenario, two types of flows MUST be generated between 595 sender A and receiver B: 597 o A single long-lived non application-limited TCP NewReno flow; 599 o A single long-lived application-limited TCP NewReno flow, with an 600 IW set to 3 or 10 packets. The size of the data transferred must 601 be strictly higher than 10 packets and should be lower than 100 602 packets. 604 The transmission of the non application-limited flow must start 605 before the transmission of the application-limited flow and only 606 after the steady state has been reached by non application-limited 607 flow. 609 For each of these scenarios, the graph described in Section 2.6 could 610 be generated for each class of traffic (application-limited and non 611 application-limited). The completion time of the application-limited 612 TCP flow could be measured. 614 4.2. Aggressive transport sender 616 This scenario helps testers to evaluate how an AQM scheme reacts to a 617 transport sender that is more aggressive than a single TCP-friendly 618 sender. We define 'aggressiveness' as a higher increase factor than 619 standard upon a successful transmission and/or a lower than standard 620 decrease factor upon a unsuccessful transmission (e.g. in case of 621 congestion controls with Additive-Increase Multiplicative-Decrease 622 (AIMD) principle, a larger AI and/or MD factors). A single long- 623 lived, non application-limited, TCP Cubic flow transfers data between 624 sender A and receiver B. Other aggressive congestion control schemes 625 MAY also be considered. 627 For each flavor of aggressive transports, the graph described in 628 Section 2.6 could be generated. 630 4.3. Unresponsive transport sender 632 This scenario helps testers to evaluate how an AQM scheme reacts to a 633 transport sender that is less responsive than TCP. Note that faulty 634 transport implementations on an end host and/or faulty network 635 elements en-route that "hide" congestion signals in packet headers 636 [I-D.ietf-aqm-recommendation] may also lead to a similar situation, 637 such that the AQM scheme needs to adapt to unresponsive traffic. To 638 this end, these guidelines propose the two following scenarios. 640 The first scenario can be used to evaluate queue build up. It 641 considers unresponsive flow(s) whose sending rate is greater than the 642 bottleneck link capacity between routers L and R. This scenario 643 consists of a long-lived non application limited UDP flow transmits 644 data between sender A and receiver B. Graphs described in 645 Section 2.6 could be generated. 647 The second scenario can be used to evaluate if the AQM scheme is able 648 to keep responsive fraction under control. This scenario considers a 649 mixture of TCP-friendly and unresponsive traffics. It consists of a 650 long-lived non application-limited UDP flow and a single long-lived, 651 non-application-limited, TCP New Reno flow that transmit data between 652 sender A and receiver B. As opposed to the first scenario, the rate 653 of the UDP traffic should not be greater than the bottleneck 654 capacity, and should not be higher than half of the bottleneck 655 capacity. For each type of traffic, the graph described in 656 Section 2.6 could be generated. 658 4.4. Less-than Best Effort transport sender 660 This scenario helps to evaluate how an AQM scheme reacts to LBE 661 congestion controls that 'results in smaller bandwidth and/or delay 662 impact on standard TCP than standard TCP itself, when sharing a 663 bottleneck with it.' [RFC6297]. The potential fateful interaction 664 when AQM and LBE techniques are combined has been shown in [LBE-AQM]; 665 this scenario helps to evaluate whether the coexistence of the 666 proposed AQM and LBE techniques may be possible. 668 Single long-lived non application-limited TCP NewReno flows transfer 669 data between sender A and receiver B. Other TCP-friendly congestion 670 control schemes MAY also be considered. Single long-lived non 671 application-limited LEDBAT [RFC6817] flows transfer data between 672 sender A and receiver B. We recommend to set the target delay and 673 gain values of LEDBAT respectively to 5 ms and 10 [LEDBAT-PARAM]. 674 Other LBE congestion control schemes, any of those listed in 675 [RFC6297], MAY also be considered. 677 For each of the TCP-friendly and LBE transports, the graph described 678 in Section 2.6 could be generated. 680 5. Round Trip Time Fairness 682 5.1. Motivation 684 The ability of AQM schemes to control the queuing delay highly 685 depends on the way end-to-end protocols react to congestion signals. 686 When the RTT varies, the behaviour of a congestion control is 687 impacted and this impacts the ability of an AQM scheme to control the 688 queue. It is therefore important to assess the AQM schemes for a set 689 of RTTs (e.g., from 5 ms to 200 ms). 691 The asymmetry in terms of difference in intrinsic RTT between various 692 paths sharing the same bottleneck SHOULD be considered so that the 693 fairness between the flows can be discussed since in this scenario, a 694 flow traversing on shorter RTT path may react faster to congestion 695 and recover faster from it compared to another flow on a longer RTT 696 path. The introduction of AQM schemes may potentially improve this 697 type of fairness. 699 Introducing an AQM scheme may cause the unfairness between the flows, 700 even if the RTTs are identical. This potential unfairness SHOULD be 701 investigated as well. 703 5.2. Recommended tests 705 The RECOMMENDED topology is detailed in Figure 1: 707 o To evaluate the inter-RTT fairness, for each run, two flows 708 divided into two categories. Category I which RTT between sender 709 A and Router L SHOULD be 100ms. Category II which RTT between 710 sender A and Router L should be in [5ms;560ms]. The maximum value 711 for the RTT represents the RTT of a satellite link that, according 712 to the section 2 of [RFC2488] should be at least 558ms. 714 o To evaluate the impact of the RTT value on the AQM performance and 715 the intra-protocol fairness (the fairness for the flows using the 716 same paths/congestion control), for each run, two flows (Flow1 and 717 Flow2) should be introduced. For each experiment, the set of RTT 718 SHOULD be the same for the two flows and in [5ms;560ms]. 720 A set of evaluated flows MUST use the same congestion control 721 algorithm. 723 5.3. Metrics to evaluate the RTT fairness 725 The outputs that MUST be measured are: 727 o for the inter-RTT fairness: (1) the cumulative average goodput of 728 the flow from Category I, goodput_Cat_I (Section 2.4); (2) the 729 cumulative average goodput of the flow from Category II, 730 goodput_Cat_II (Section 2.4); (3) the ratio goodput_Cat_II/ 731 goodput_Cat_I; (4) the average packet drop rate for each category 732 (Section 2.2). 734 o for the intra-protocol RTT fairness: (1) the cumulative average 735 goodput of the two flows (Section 2.4); (2) the average packet 736 drop rate for the two flows (Section 2.2). 738 6. Burst Absorption 740 "AQM mechanisms need to control the overall queue sizes, to ensure 741 that arriving bursts can be accommodated without dropping packets" 742 [I-D.ietf-aqm-recommendation] 744 6.1. Motivation 746 An AQM scheme can result in bursts of packet arrivals due to various 747 reasons. Dropping one or more packets from a burst can result in 748 performance penalties for the corresponding flows, since dropped 749 packets have to be retransmitted. Performance penalties can result 750 in failing to meet SLAs and be a disincentive to AQM adoption. 752 The ability to accommodate bursts translates to larger queue length 753 and hence more queuing delay. On the one hand, it is important that 754 an AQM scheme quickly brings bursty traffic under control. On the 755 other hand, a peak in the packet drop rates to bring a packet burst 756 quickly under control could result in multiple drops per flow and 757 severely impact transport and application performance. Therefore, an 758 AQM scheme ought to bring bursts under control by balancing both 759 aspects -- (1) queuing delay spikes are minimized and (2) performance 760 penalties for ongoing flows in terms of packet drops are minimized. 762 An AQM scheme that maintains short queues allows some remaining space 763 in the queue for bursts of arriving packets. The tolerance to bursts 764 of packets depends upon the number of packets in the queue, which is 765 directly linked to the AQM algorithm. Moreover, one AQM scheme may 766 implement a feature controlling the maximum size of accepted bursts, 767 that can depend on the buffer occupancy or the currently estimated 768 queuing delay. The impact of the buffer size on the burst allowance 769 may be evaluated. 771 6.2. Recommended tests 773 For this scenario, tester MUST evaluate how the AQM performs with the 774 following traffic generated from sender A to receiver B: 776 o Web traffic with IW10: Web transfer of 100 packets with initial 777 congestion window set to 10; 779 o Bursty video frames; 781 o Constant bit rate UDP traffic. 783 o A single bulk TCP flow as background traffic. 785 Figure 2 presents the various cases for the traffic that MUST be 786 generated between sender A and receiver B. 788 +-------------------------------------------------+ 789 |Case| Traffic Type | 790 | +-----+------------+----+--------------------+ 791 | |Video|Webs (IW 10)| CBR| Bulk TCP Traffic | 792 +----|-----|------------|----|--------------------| 793 |I | 0 | 1 | 1 | 0 | 794 +----|-----|------------|----|--------------------| 795 |II | 0 | 1 | 1 | 1 | 796 |----|-----|------------|----|--------------------| 797 |III | 1 | 1 | 1 | 0 | 798 +----|-----|------------|----|--------------------| 799 |IV | 1 | 1 | 1 | 1 | 800 +----+-----+------------+----+--------------------+ 802 Figure 2: Bursty traffic scenarios 804 For each of these scenarios, the graph described in Section 2.6 could 805 be generated. Metrics such as end-to-end latency, jitter, flow 806 completion time MAY be generated. For the cases of frame generation 807 of bursty video traffic as well as the choice of web traffic pattern, 808 we leave these details and their presentation to the testers. 810 7. Stability 812 7.1. Motivation 814 Network devices can experience varying operating conditions depending 815 on factors such as time of the day, deployment scenario, etc. For 816 example: 818 o Traffic and congestion levels are higher during peak hours than 819 off-peak hours. 821 o In the presence of a scheduler, the draining rate of a queue can 822 vary depending on the occupancy of other queues: a low load on a 823 high priority queue implies a higher draining rate for the lower 824 priority queues. 826 o The available capacity at the physical layer can vary over time 827 (e.g., a lossy channel, a link supporting traffic in a higher 828 diffserv class). 830 Whether the target context is a not stable environment, the ability 831 of an AQM scheme to maintain its control over the queuing delay and 832 buffer occupancy can be challenged. This document proposes 833 guidelines to assess the behavior of AQM schemes under varying 834 congestion levels and varying draining rates. 836 7.2. Recommended tests 838 Note that the traffic profiles explained below comprises non 839 application-limited TCP flows. For each of the below scenarios, the 840 results described in Section 2.6 SHOULD be generated. For 841 Section 7.2.5 and Section 7.2.6 they SHOULD incorporate the results 842 in per-phase basis as well. 844 Wherever the notion of time has explicitly mentioned in this 845 subsection, time 0 starts from the moment all TCP flows have already 846 reached their congestion avoidance phase. 848 7.2.1. Definition of the congestion Level 850 In these guidelines, the congestion levels are represented by the 851 projected packet drop rate, had a drop-tail queue was chosen instead 852 of an AQM scheme. When the bottleneck is shared among non- 853 application-limited TCP flows. l_r, the loss rate projection can be 854 expressed as a function of N, the number of bulk TCP flows, and S, 855 the sum of the bandwidth-delay product and the maximum buffer size, 856 both expressed in packets, based on Eq. 3 of [SCL-TCP]: 858 l_r = 0.76 * N^2 / S^2 860 N = S * sqrt(1/0.76) * sqrt (l_r) 862 7.2.2. Mild congestion 864 This scenario can be used to evaluate how an AQM scheme reacts to a 865 light load of incoming traffic resulting in mild congestion -- packet 866 drop rates around 0.1%. The number of bulk flows required to achieve 867 this congestion level, N_mild, is then: 869 N_mild = round(0.036*S) 871 7.2.3. Medium congestion 873 This scenario can be used to evaluate how an AQM scheme reacts to 874 incoming traffic resulting in medium congestion -- packet drop rates 875 around 0.5%. The number of bulk flows required to achieve this 876 congestion level, N_med, is then: 878 N_med = round (0.081*S) 880 7.2.4. Heavy congestion 882 This scenario can be used to evaluate how an AQM scheme reacts to 883 incoming traffic resulting in heavy congestion -- packet drop rates 884 around 1%. The number of bulk flows required to achieve this 885 congestion level, N_heavy, is then: 887 N_heavy = round (0.114*S) 889 7.2.5. Varying congestion levels 891 This scenario can be used to evaluate how an AQM scheme reacts to 892 incoming traffic resulting in various level of congestions during the 893 experiment. In this scenario, the congestion level varies within a 894 large time-scale. The following phases may be considered: phase I - 895 mild congestion during 0-20s; phase II - medium congestion during 896 20-40s; phase III - heavy congestion during 40-60s; phase I again, 897 and so on. 899 7.2.6. Varying available capacity 901 This scenario can be used to evaluate how an AQM scheme adapts to 902 varying available capacity on the outgoing link. 904 To emulate varying draining rates, the bottleneck capacity between 905 nodes 'Router L' and 'Router R' varies over the course of the 906 experiment as follows: 908 o Experiment 1: the capacity varies between two values within a 909 large time-scale. As an example, the following phases may be 910 considered: phase I - 100Mbps during 0-20s; phase II - 10Mbps 911 during 20-40s; phase I again, and so on. 913 o Experiment 2: the capacity varies between two values within a 914 short time-scale. As an example, the following phases may be 915 considered: phase I - 100Mbps during 0-100ms; phase II - 10Mbps 916 during 100-200ms; phase I again, and so on. 918 The tester MAY choose a phase time-interval value different than what 919 is stated above, if the network's path conditions (such as bandwidth- 920 delay product) necessitate. In this case the choice of such time- 921 interval value SHOULD be stated and elaborated. 923 The tester MAY additionally evaluate the two mentioned scenarios 924 (short-term and long-term capacity variations), during and/or 925 including TCP slow-start phase. 927 More realistic fluctuating capacity patterns MAY be considered. The 928 tester MAY choose to incorporate realistic scenarios with regards to 929 common fluctuation of bandwidth in state-of-the-art technologies. 931 The scenario consists of TCP NewReno flows between sender A and 932 receiver B. To better assess the impact of draining rates on the AQM 933 behavior, the tester MUST compare its performance with those of drop- 934 tail and SHOULD provide a reference document for their proposal 935 discussing performance and deployment compared to those of drop-tail. 937 7.3. Parameter sensitivity and stability analysis 939 The control law used by an AQM is the primary means by which the 940 queuing delay is controlled. Hence understanding the control law is 941 critical to understanding the behavior of the AQM scheme. The 942 control law could include several input parameters whose values 943 affect the AQM scheme's output behavior and its stability. 944 Additionally, AQM schemes may auto-tune parameter values in order to 945 maintain stability under different network conditions (such as 946 different congestion levels, draining rates or network environments). 947 The stability of these auto-tuning techniques is also important to 948 understand. 950 Transports operating under the control of AQM experience the effect 951 of multiple control loops that react over different timescales. It 952 is therefore important that proposed AQM schemes are seen to be 953 stable when they are deployed at multiple points of potential 954 congestion along an Internet path. The pattern of congestion signals 955 (loss or ECN-marking) arising from AQM methods also need to not 956 adversely interact with the dynamics of the transport protocols that 957 they control. 959 AQM proposals SHOULD provide background material showing control 960 theoretic analysis of the AQM control law and the input parameter 961 space within which the control law operates as expected; or could use 962 another way to discuss the stability of the control law. For 963 parameters that are auto-tuned, the material SHOULD include stability 964 analysis of the auto-tuning mechanism(s) as well. Such analysis 965 helps to understand an AQM control law better and the network 966 conditions/deployments under which the AQM is stable. 968 8. Various Traffic Profiles 970 This section provides guidelines to assess the performance of an AQM 971 proposal for various traffic profiles such as traffic with different 972 applications or bi-directional traffic. 974 8.1. Traffic mix 976 This scenario can be used to evaluate how an AQM scheme reacts to a 977 traffic mix consisting of different applications such as: 979 o Bulk TCP transfer 981 o Web traffic 983 o VoIP 985 o Constant Bit Rate (CBR) UDP traffic 987 o Adaptive video streaming 989 Various traffic mixes can be considered. These guidelines RECOMMEND 990 to examine at least the following example: 1 bi-directional VoIP; 6 991 Webs; 1 CBR; 1 Adaptive Video; 5 bulk TCP. Any other combinations 992 could be considered and should be carefully documented. 994 For each scenario, the graph described in Section 2.6 could be 995 generated for each class of traffic. Metrics such as end-to-end 996 latency, jitter and flow completion time MAY be reported. 998 8.2. Bi-directional traffic 1000 Control packets such as DNS requests/responses, TCP SYNs/ACKs are 1001 small, but their loss can severely impact the application 1002 performance. The scenario proposed in this section will help in 1003 assessing whether the introduction of an AQM scheme increases the 1004 loss probability of these important packets. 1006 For this scenario, traffic MUST be generated in both downlink and 1007 uplink, such as defined in Section 3.1. These guidelines RECOMMEND 1008 to consider a mild congestion level and the traffic presented in 1009 Section 7.2.2 in both directions. In this case, the metrics reported 1010 MUST be the same as in Section 7.2 for each direction. 1012 The traffic mix presented in Section 8.1 MAY also be generated in 1013 both directions. 1015 9. Multi-AQM Scenario 1017 9.1. Motivation 1019 Transports operating under the control of AQM experience the effect 1020 of multiple control loops that react over different timescales. It 1021 is therefore important that proposed AQM schemes are seen to be 1022 stable when they are deployed at multiple points of potential 1023 congestion along an Internet path. The pattern of congestion signals 1024 (loss or ECN-marking) arising from AQM methods also need to not 1025 adversely interact with the dynamics of the transport protocols that 1026 they control. 1028 9.2. Details on the evaluation scenario 1030 +---------+ +-----------+ 1031 |senders A|---+ +---|receivers A| 1032 +---------+ | | +-----------+ 1033 +-----+---+ +---------+ +--+-----+ 1034 |Router L |--|Router M |--|Router R| 1035 |AQM | |AQM | |No AQM | 1036 +---------+ +--+------+ +--+-----+ 1037 +---------+ | | +-----------+ 1038 |senders B|-------------+ +---|receivers B| 1039 +---------+ +-----------+ 1041 Figure 3: Topology for the Multi-AQM scenario 1043 This scenario can be used to evaluate how having AQM schemes in 1044 sequence impact the induced latency reduction, the induced goodput 1045 maximization and the trade-off between these two. The topology 1046 presented in Figure 3 could be used. We recommend that the AQM 1047 schemes introduced in Router L and Router M should be the same; any 1048 other configurations could be considered. For this scenario, we 1049 recommend to consider a mild congestion level, the number of flows 1050 specified in Section 7.2.2 being equally shared among senders A and 1051 B. Any other relevant combination of congestion levels could be 1052 considered. We recommend to measure the metrics presented in 1053 Section 7.2. 1055 10. Implementation cost 1057 10.1. Motivation 1059 Successful deployment of AQM is directly related to its cost of 1060 implementation. Network devices can need hardware or software 1061 implementations of the AQM mechanism. Depending on a device's 1062 capabilities and limitations, the device may or may not be able to 1063 implement some or all parts of the AQM logic. 1065 AQM proposals SHOULD provide pseudo-code for the complete AQM scheme, 1066 highlighting generic implementation-specific aspects of the scheme 1067 such as "drop-tail" vs. "drop-head", inputs (e.g. current queuing 1068 delay, queue length), computations involved, need for timers, etc. 1069 This helps to identify costs associated with implementing the AQM 1070 scheme on a particular hardware or software device. This also helps 1071 the WG understand which kind of devices can easily support the AQM 1072 and which cannot. 1074 10.2. Recommended discussion 1076 AQM proposals SHOULD highlight parts of AQM logic that are device 1077 dependent and discuss if and how AQM behavior could be impacted by 1078 the device. For example, a queueing-delay based AQM scheme requires 1079 current queuing delay as input from the device. If the device 1080 already maintains this value, then it can be trivial to implement the 1081 AQM logic on the device. If the device provides indirect means to 1082 estimate the queuing delay (for example: timestamps, dequeuing rate), 1083 then the AQM behavior is sensitive to the precision of the queuing 1084 delay estimations are for that device. Highlighting the sensitivity 1085 of an AQM scheme to queuing delay estimations helps implementers to 1086 identify appropriate means of implementing the mechanism on a device. 1088 11. Operator Control and Auto-tuning 1090 11.1. Motivation 1092 One of the biggest hurdles of RED deployment was/is its parameter 1093 sensitivity to operating conditions -- how difficult it is to tune 1094 RED parameters for a deployment to achieve acceptable benefit from 1095 using RED. Fluctuating congestion levels and network conditions add 1096 to the complexity. Incorrect parameter values lead to poor 1097 performance. 1099 Any AQM scheme is likely to have parameters whose values affect the 1100 control law and behaviour of an AQM. Exposing all these parameters 1101 as control parameters to a network operator (or user) can easily 1102 result in a unsafe AQM deployment. Unexpected AQM behavior ensues 1103 when parameter values are set improperly. A minimal number of 1104 control parameters minimizes the number of ways a possibly naive user 1105 can break a system where an AQM scheme is deployed at. Fewer control 1106 parameters make the AQM scheme more user-friendly and easier to 1107 deploy and debug. 1109 [I-D.ietf-aqm-recommendation] states "AQM algorithms SHOULD NOT 1110 require tuning of initial or configuration parameters in common use 1111 cases." A scheme ought to expose only those parameters that control 1112 the macroscopic AQM behavior such as queue delay threshold, queue 1113 length threshold, etc. 1115 Additionally, the safety of an AQM scheme is directly related to its 1116 stability under varying operating conditions such as varying traffic 1117 profiles and fluctuating network conditions, as described in 1118 Section 7. Operating conditions vary often and hence the AQM needs 1119 to remain stable under these conditions without the need for 1120 additional external tuning. If AQM parameters require tuning under 1121 these conditions, then the AQM must self-adapt necessary parameter 1122 values by employing auto-tuning techniques. 1124 11.2. Required discussion 1126 AQM proposals SHOULD describe the parameters that control the 1127 macroscopic AQM behavior, and identify any parameters that require 1128 require tuning to operational conditions. If an AQM scheme is 1129 evaluated with parameter(s) that were externally tuned for 1130 optimization or other purposes, these values MUST be disclosed. 1132 12. Interaction with ECN 1134 Deployed AQM algorithms SHOULD support Explicit Congestion 1135 Notification (ECN) as well as loss to signal congestion to 1136 endpoints"[I-D.ietf-aqm-recommendation]. The benefits of providing 1137 ECN support for an AQM scheme are descibed in [ECN-Benefit]. 1139 12.1. Motivation 1141 (ECN) [RFC3168] is an alternative that allows AQM schemes to signal 1142 receivers about network congestion that does not use packet drop. 1144 12.2. Recommended discussion 1146 An AQM scheme can support ECN [I-D.ietf-aqm-recommendation], in which 1147 case testers MUST discuss and describe the support of ECN. 1149 13. Interaction with Scheduling 1151 A network device may use per-flow or per-class queuing with a 1152 scheduling algorithm to either prioritize certain applications or 1153 classes of traffic, limit the rate of transmission, or to provide 1154 isolation between different traffic flows within a common class 1155 [I-D.ietf-aqm-recommendation]. 1157 13.1. Motivation 1159 Coupled with an AQM scheme, a router may schedule the transmission of 1160 packets in a specific manner by introducing a scheduling scheme. 1161 This algorithm may create sub-queues and integrate a dropping policy 1162 on each of these sub-queues. Another scheduling policy may modify 1163 the way packets are sequenced, modifying the timestamp of each 1164 packet. 1166 13.2. Recommended discussion 1168 The scheduling and the AQM conjointly impact on the end-to-end 1169 performance. During the characterization process of a dropping 1170 policy, the tester MUST discuss the feasibility to add scheduling 1171 combined with the AQM algorithm. This discussion as an instance, MAY 1172 explain whether the dropping policy is applied when packets are being 1173 enqueued or dequeued. 1175 14. Discussion on Methodology, Metrics, AQM Comparisons and Packet 1176 Sizes 1178 14.1. Methodology 1180 One key objective behind formulating the guidelines is to help 1181 ascertain whether a specific AQM is not only better than drop-tail 1182 but also safe to deploy. Testers therefore need to provide a 1183 reference document for their proposal discussing performance and 1184 deployment compared to those of drop-tail. 1186 A description of each test setup SHOULD be detailed to allow this 1187 test to be compared with other tests. This also allows others to 1188 replicate the tests if needed. This test setup SHOULD detail 1189 software and hardware versions. The tester could make its data 1190 available. 1192 The proposals SHOULD be evaluated on real-life systems, or they MAY 1193 be evaluated with event-driven simulations (such as ns-2, ns-3, 1194 OMNET, etc). The proposed scenarios are not bound to a particular 1195 evaluation toolset. 1197 The tester is encouraged to make the detailed test setup and the 1198 results publicly available. 1200 14.2. Comments on metrics measurement 1202 The document presents the end-to-end metrics that ought to be used to 1203 evaluate the trade-off between latency and goodput in Section 2. In 1204 addition to the end-to-end metrics, the queue-level metrics (normally 1205 collected at the device operating the AQM) provide a better 1206 understanding of the AQM behavior under study and the impact of its 1207 internal parameters. Whenever it is possible (e.g. depending on the 1208 features provided by the hardware/software), these guidelines advice 1209 to consider queue-level metrics, such as link utilization, queuing 1210 delay, queue size or packet drop/mark statistics in addition to the 1211 AQM-specific parameters. However, the evaluation MUST be primarily 1212 based on externally observed end-to-end metrics. 1214 These guidelines do not aim to detail on the way these metrics can be 1215 measured, since the way these metrics are measured is expected to 1216 depend on the evaluation toolset. 1218 14.3. Comparing AQM schemes 1220 This document recognizes that the guidelines mentioned above may be 1221 used for comparing AQM schemes. 1223 AQM schemes need to be compared against both performance and 1224 deployment categories. In addition, this section details how best to 1225 achieve a fair comparison of AQM schemes by avoiding certain 1226 pitfalls. 1228 14.3.1. Performance comparison 1230 AQM schemes MUST be compared against all the generic scenarios 1231 presented in this memo. AQM schemes MAY be compared for specific 1232 network environments such as data centers, home networks, etc. If an 1233 AQM scheme has parameter(s) that were externally tuned for 1234 optimization or other purposes, these values MUST be disclosed. 1236 AQM schemes belong to different varieties such as queue-length based 1237 schemes (ex. RED) or queueing-delay based scheme (ex. CoDel, PIE). 1238 AQM schemes expose different control knobs associated with different 1239 semantics. For example, while both PIE and CoDel are queueing-delay 1240 based schemes and each expose a knob to control the queueing delay -- 1241 PIE's "queueing delay reference" vs. CoDel's "queueing delay target", 1242 the two tuning paramerters of the two schemes have different 1243 semantics, resulting in different control points. Such differences 1244 in AQM schemes can be easily overlooked while making comparisons. 1246 This document RECOMMENDS the following procedures for a fair 1247 performance comparison between the AQM schemes: 1249 1. comparable control parameters and comparable input values: 1250 carefully identify the set of parameters that control similar 1251 behavior between the two AQM schemes and ensure these parameters 1252 have comparable input values. For example, to compare how well a 1253 queue-length based AQM scheme controls queueing delay vs. a 1254 queueing-delay based AQM scheme, a tester can identify the 1255 parameters of the schemes that control queue delay and ensure 1256 that their input values are comparable. Similarly, to compare 1257 how well two AQM schemes accommodate packet bursts, the tester 1258 can identify burst-related control parameters and ensure they are 1259 configured with similar values. 1261 2. compare over a range of input configurations: there could be 1262 situations when the set of control parameters that affect a 1263 specific behavior have different semantics between the two AQM 1264 schemes. As mentioned above, PIE has tuning parameters to 1265 control queue delay that has a different semantics from those 1266 used in CoDel. In such situations, these schemes need to be 1267 compared over a range of input configurations. For example, 1268 compare PIE vs. CoDel over the range of target delay input 1269 configurations. 1271 14.3.2. Deployment comparison 1273 AQM schemes MUST be compared against deployment criteria such as the 1274 parameter sensitivity (Section 7.3), auto-tuning (Section 11) or 1275 implementation cost (Section 10). 1277 14.4. Packet sizes and congestion notification 1279 An AQM scheme may be considering packet sizes while generating 1280 congestion signals. [RFC7141] discusses the motivations behind this. 1281 For example, control packets such as DNS requests/responses, TCP 1282 SYNs/ACKs are small, but their loss can severely impact the 1283 application performance. An AQM scheme may therefore be biased 1284 towards small packets by dropping them with smaller probability 1285 compared to larger packets. However, such an AQM scheme is unfair to 1286 data senders generating larger packets. Data senders, malicious or 1287 otherwise, are motivated to take advantage of such AQM scheme by 1288 transmitting smaller packets, and could result in unsafe deployments 1289 and unhealthy transport and/or application designs. 1291 An AQM scheme SHOULD adhere to the recommendations outlined in 1292 [RFC7141], and SHOULD NOT provide undue advantage to flows with 1293 smaller packets [I-D.ietf-aqm-recommendation]. 1295 15. Conclusion 1297 Figure 4 lists the scenarios and their requirements. 1299 +------------------------------------------------------------------+ 1300 |Scenario |Sec. |Requirement | 1301 +------------------------------------------------------------------+ 1302 +------------------------------------------------------------------+ 1303 |Transport Protocols |4. | | 1304 | TCP-friendly sender | 4.1 |Scenario MUST be considered | 1305 | Aggressive sender | 4.2 |Scenario MUST be considered | 1306 | Unresponsive sender | 4.3 |Scenario MUST be considered | 1307 | LBE sender | 4.4 |Scenario MAY be considered | 1308 +------------------------------------------------------------------+ 1309 |Round Trip Time Fairness | 5.2 |Scenario MUST be considered | 1310 +------------------------------------------------------------------+ 1311 |Burst Absorption | 6.2 |Scenario MUST be considered | 1312 +------------------------------------------------------------------+ 1313 |Stability |7. | | 1314 | Varying congestion levels | 7.2.5|Scenario MUST be considered | 1315 | Varying available capacity| 7.2.6|Scenario MUST be considered | 1316 | Parameters and stability | 7.3 |This SHOULD be discussed | 1317 +------------------------------------------------------------------+ 1318 |Various Traffic Profiles |8. | | 1319 | Traffic mix | 8.1 |Scenario is RECOMMENDED | 1320 | Bi-directional traffic | 8.2 |Scenario MAY be considered | 1321 +------------------------------------------------------------------+ 1322 |Multi-AQM | 9.2 |Scenario MAY be considered | 1323 +------------------------------------------------------------------+ 1324 |Implementation Cost | 10.2 |Pseudo-code SHOULD be provided | 1325 +------------------------------------------------------------------+ 1326 |Operator Control | 11.2 |Tuning SHOULD NOT be required | 1327 +------------------------------------------------------------------+ 1328 |Interaction with ECN | 12.2 |MUST be discussed if supported | 1329 +------------------------------------------------------------------+ 1330 |Interaction with Scheduling| 13.2 |Feasibility MUST be discussed | 1331 +------------------------------------------------------------------+ 1333 Figure 4: Summary of the scenarios and their requirements 1335 16. Acknowledgements 1337 This work has been partially supported by the European Community 1338 under its Seventh Framework Programme through the Reducing Internet 1339 Transport Latency (RITE) project (ICT-317700). 1341 17. Contributors 1343 Many thanks to S. Akhtar, A.B. Bagayoko, F. Baker, D. Collier- 1344 Brown, G. Fairhurst, J. Gettys, T. Hoiland-Jorgensen, C. 1345 Kulatunga, W. Lautenschlager, A.C. Morton, R. Pan, D. Taht and M. 1346 Welzl for detailed and wise feedback on this document. 1348 18. IANA Considerations 1350 This memo includes no request to IANA. 1352 19. Security Considerations 1354 Some security considerations for AQM are identified in 1355 [I-D.ietf-aqm-recommendation].This document, by itself, presents no 1356 new privacy nor security issues. 1358 20. References 1360 20.1. Normative References 1362 [I-D.ietf-aqm-recommendation] 1363 Baker, F. and G. Fairhurst, "IETF Recommendations 1364 Regarding Active Queue Management", draft-ietf-aqm- 1365 recommendation-11 (work in progress), February 2015. 1367 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1368 Requirement Levels", RFC 2119, 1997. 1370 [RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion 1371 Notification", RFC 7141, 2014. 1373 20.2. Informative References 1375 [BB2011] "BufferBloat: what's wrong with the internet?", ACM Queue 1376 vol. 9, 2011. 1378 [CODEL] Nichols, K. and V. Jacobson, "Controlling Queue Delay", 1379 ACM Queue , 2012. 1381 [ECN-Benefit] 1382 Welzl, M. and G. Fairhurst, "The Benefits to Applications 1383 of using Explicit Congestion Notification (ECN)", IETF 1384 (Work-in-Progress) , February 2014. 1386 [LBE-AQM] Gong, Y., Rossi, D., Testa, C., Valenti, S., and D. Taht, 1387 "Fighting the bufferbloat: on the coexistence of AQM and 1388 low priority congestion control", Computer Networks, 1389 Elsevier, 2014, 60, pp.115 - 128 , 2014. 1391 [LEDBAT-PARAM] 1392 Trang, S., Kuhn, N., Lochin, E., Baudoin, C., Dubois, E., 1393 and P. Gelard, "On The Existence Of Optimal LEDBAT 1394 Parameters", IEEE ICC 2014 - Communication QoS, 1395 Reliability and Modeling Symposium , 2014. 1397 [LOSS-SYNCH-MET-08] 1398 Hassayoun, S. and D. Ros, "Loss Synchronization and Router 1399 Buffer Sizing with High-Speed Versions of TCP", IEEE 1400 INFOCOM Workshops , 2008. 1402 [PIE] Pan, R., Natarajan, P., Piglione, C., Prabhu, MS., 1403 Subramanian, V., Baker, F., and B. VerSteeg, "PIE: A 1404 lightweight control scheme to address the bufferbloat 1405 problem", IEEE HPSR , 2013. 1407 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC 1408 793, September 1981. 1410 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 1411 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 1412 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 1413 S., Wroclawski, J., and L. Zhang, "Recommendations on 1414 Queue Management and Congestion Avoidance in the 1415 Internet", RFC 2309, April 1998. 1417 [RFC2488] Allman, M., Glover, D., and L. Sanchez, "Enhancing TCP 1418 Over Satellite Channels using Standard Mechanisms", BCP 1419 28, RFC 2488, January 1999. 1421 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1422 Network Interconnect Devices", RFC 2544, March 1999. 1424 [RFC2647] Newman, D., "Benchmarking Terminology for Firewall 1425 Performance", RFC 2647, August 1999. 1427 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1428 Delay Metric for IPPM", RFC 2679, September 1999. 1430 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1431 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1433 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1434 of Explicit Congestion Notification (ECN) to IP", RFC 1435 3168, September 2001. 1437 [RFC3611] Friedman, T., Caceres, R., and A. Clark, "RTP Control 1438 Protocol Extended Reports (RTCP XR)", RFC 3611, November 1439 2003. 1441 [RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP 1442 Friendly Rate Control (TFRC): Protocol Specification", RFC 1443 5348, September 2008. 1445 [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation 1446 Applicability Statement", RFC 5481, March 2009. 1448 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1449 Control", RFC 5681, September 2009. 1451 [RFC6297] Welzl, M. and D. Ros, "A Survey of Lower-than-Best-Effort 1452 Transport Protocols", RFC 6297, June 2011. 1454 [RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind, 1455 "Low Extra Delay Background Transport (LEDBAT)", RFC 6817, 1456 December 2012. 1458 [SCL-TCP] Morris, R., "Scalable TCP congestion control", IEEE 1459 INFOCOM , 2000. 1461 [TCPEVAL2013] 1462 Hayes, D., Ros, D., Andrew, L., and S. Floyd, "Common TCP 1463 Evaluation Suite", IRTF ICCRG , 2013. 1465 [YU06] Jay, P., Fu, Q., and G. Armitage, "A preliminary analysis 1466 of loss synchronisation between concurrent TCP flows", 1467 Australian Telecommunication Networks and Application 1468 Conference (ATNAC) , 2006. 1470 Authors' Addresses 1472 Nicolas Kuhn (editor) 1473 Telecom Bretagne 1474 2 rue de la Chataigneraie 1475 Cesson-Sevigne 35510 1476 France 1478 Phone: +33 2 99 12 70 46 1479 Email: nicolas.kuhn@telecom-bretagne.eu 1480 Naeem Khademi (editor) 1481 University of Oslo 1482 Department of Informatics, PO Box 1080 Blindern 1483 N-0316 Oslo 1484 Norway 1486 Phone: +47 2285 24 93 1487 Email: naeemk@ifi.uio.no 1489 Preethi Natarajan (editor) 1490 Cisco Systems 1491 510 McCarthy Blvd 1492 Milpitas, California 1493 United States 1495 Email: prenatar@cisco.com 1497 David Ros 1498 Simula Research Laboratory AS 1499 P.O. Box 134 1500 Lysaker, 1325 1501 Norway 1503 Phone: +33 299 25 21 21 1504 Email: dros@simula.no