idnits 2.17.1 draft-ietf-aqm-eval-guidelines-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 6, 2015) is 3210 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'B' is mentioned on line 303, but not defined == Missing Reference: 'Mbps' is mentioned on line 303, but not defined -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) -- Obsolete informational reference (is this intentional?): RFC 2679 (Obsoleted by RFC 7679) -- Obsolete informational reference (is this intentional?): RFC 2680 (Obsoleted by RFC 7680) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force N. Kuhn, Ed. 3 Internet-Draft Telecom Bretagne 4 Intended status: Informational P. Natarajan, Ed. 5 Expires: January 7, 2016 Cisco Systems 6 N. Khademi, Ed. 7 University of Oslo 8 D. Ros 9 Simula Research Laboratory AS 10 July 6, 2015 12 AQM Characterization Guidelines 13 draft-ietf-aqm-eval-guidelines-07 15 Abstract 17 Unmanaged large buffers in today's networks have given rise to a slew 18 of performance issues. These performance issues can be addressed by 19 some form of Active Queue Management (AQM) mechanism, optionally in 20 combination with a packet scheduling scheme such as fair queuing. 21 The IETF Active Queue Management and Packet Scheduling working group 22 was formed to standardize AQM schemes that are robust, easily 23 implementable, and successfully deployable in today's networks. This 24 document describes various criteria for performing precautionary 25 characterizations of AQM proposals. This document also helps in 26 ascertaining whether any given AQM proposal should be taken up for 27 standardization by the AQM WG. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at http://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on January 7, 2016. 46 Copyright Notice 48 Copyright (c) 2015 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (http://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with respect 56 to this document. Code Components extracted from this document must 57 include Simplified BSD License text as described in Section 4.e of 58 the Trust Legal Provisions and are provided without warranty as 59 described in the Simplified BSD License. 61 Table of Contents 63 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 64 1.1. Reducing the latency and maximizing the goodput . . . . . 5 65 1.2. Guidelines for AQM evaluation . . . . . . . . . . . . . . 5 66 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 6 67 1.4. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 6 68 2. End-to-end metrics . . . . . . . . . . . . . . . . . . . . . 6 69 2.1. Flow completion time . . . . . . . . . . . . . . . . . . 7 70 2.2. Flow start up time . . . . . . . . . . . . . . . . . . . 7 71 2.3. Packet loss . . . . . . . . . . . . . . . . . . . . . . . 7 72 2.4. Packet loss synchronization . . . . . . . . . . . . . . . 8 73 2.5. Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 9 74 2.6. Latency and jitter . . . . . . . . . . . . . . . . . . . 9 75 2.7. Discussion on the trade-off between latency and goodput . 10 76 3. Generic setup for evaluations . . . . . . . . . . . . . . . . 10 77 3.1. Topology and notations . . . . . . . . . . . . . . . . . 11 78 3.2. Buffer size . . . . . . . . . . . . . . . . . . . . . . . 12 79 3.3. Congestion controls . . . . . . . . . . . . . . . . . . . 12 80 4. Transport Protocols . . . . . . . . . . . . . . . . . . . . . 13 81 4.1. TCP-friendly sender . . . . . . . . . . . . . . . . . . . 14 82 4.1.1. TCP-friendly sender with the same initial congestion 83 window . . . . . . . . . . . . . . . . . . . . . . . 14 84 4.1.2. TCP-friendly sender with different initial congestion 85 windows . . . . . . . . . . . . . . . . . . . . . . . 14 86 4.2. Aggressive transport sender . . . . . . . . . . . . . . . 14 87 4.3. Unresponsive transport sender . . . . . . . . . . . . . . 15 88 4.4. Less-than Best Effort transport sender . . . . . . . . . 15 89 5. Round Trip Time Fairness . . . . . . . . . . . . . . . . . . 16 90 5.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 16 91 5.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 16 92 5.3. Metrics to evaluate the RTT fairness . . . . . . . . . . 17 93 6. Burst Absorption . . . . . . . . . . . . . . . . . . . . . . 17 94 6.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 17 95 6.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 18 96 7. Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 19 97 7.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 19 98 7.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 19 99 7.2.1. Definition of the congestion Level . . . . . . . . . 19 100 7.2.2. Mild congestion . . . . . . . . . . . . . . . . . . . 20 101 7.2.3. Medium congestion . . . . . . . . . . . . . . . . . . 20 102 7.2.4. Heavy congestion . . . . . . . . . . . . . . . . . . 20 103 7.2.5. Varying the congestion level . . . . . . . . . . . . 20 104 7.2.6. Varying available capacity . . . . . . . . . . . . . 21 105 7.3. Parameter sensitivity and stability analysis . . . . . . 22 106 8. Various Traffic Profiles . . . . . . . . . . . . . . . . . . 22 107 8.1. Traffic mix . . . . . . . . . . . . . . . . . . . . . . . 22 108 8.2. Bi-directional traffic . . . . . . . . . . . . . . . . . 23 109 9. Multi-AQM Scenario . . . . . . . . . . . . . . . . . . . . . 23 110 9.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 23 111 9.2. Details on the evaluation scenario . . . . . . . . . . . 24 112 10. Implementation cost . . . . . . . . . . . . . . . . . . . . . 24 113 10.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 24 114 10.2. Recommended discussion . . . . . . . . . . . . . . . . . 25 115 11. Operator Control and Auto-tuning . . . . . . . . . . . . . . 25 116 11.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 25 117 11.2. Required discussion . . . . . . . . . . . . . . . . . . 26 118 12. Interaction with ECN . . . . . . . . . . . . . . . . . . . . 26 119 12.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 26 120 12.2. Recommended discussion . . . . . . . . . . . . . . . . . 26 121 13. Interaction with Scheduling . . . . . . . . . . . . . . . . . 26 122 13.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 27 123 13.2. Recommended discussion . . . . . . . . . . . . . . . . . 27 124 13.3. Assessing the interaction between AQM and scheduling . . 27 125 14. Discussion on Methodology, Metrics, AQM Comparisons and 126 Packet Sizes . . . . . . . . . . . . . . . . . . . . . . . . 27 127 14.1. Methodology . . . . . . . . . . . . . . . . . . . . . . 27 128 14.2. Comments on metrics measurement . . . . . . . . . . . . 28 129 14.3. Comparing AQM schemes . . . . . . . . . . . . . . . . . 28 130 14.3.1. Performance comparison . . . . . . . . . . . . . . . 28 131 14.3.2. Deployment comparison . . . . . . . . . . . . . . . 29 132 14.4. Packet sizes and congestion notification . . . . . . . . 29 133 15. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 30 134 16. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 31 135 17. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 31 136 18. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 137 19. Security Considerations . . . . . . . . . . . . . . . . . . . 32 138 20. References . . . . . . . . . . . . . . . . . . . . . . . . . 32 139 20.1. Normative References . . . . . . . . . . . . . . . . . . 32 140 20.2. Informative References . . . . . . . . . . . . . . . . . 32 141 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35 143 1. Introduction 145 Active Queue Management (AQM) [I-D.ietf-aqm-recommendation] addresses 146 the concerns arising from using unnecessarily large and unmanaged 147 buffers to improve network and application performance. Several AQM 148 algorithms have been proposed in the past years, most notably Random 149 Early Detection (RED), BLUE, and Proportional Integral controller 150 (PI), and more recently CoDel [NICH2012] and PIE [PAN2013]. In 151 general, these algorithms actively interact with the Transmission 152 Control Protocol (TCP) and any other transport protocol that deploys 153 a congestion control scheme to manage the amount of data they keep in 154 the network. The available buffer space in the routers and switches 155 should be large enough to accommodate the short-term buffering 156 requirements. AQM schemes aim at reducing mean buffer occupancy, and 157 therefore both end-to-end delay and jitter. Some of these 158 algorithms, notably RED, have also been widely implemented in some 159 network devices. However, the potential benefits of the RED scheme 160 have not been realized since RED is reported to be usually turned 161 off. The main reason of this reluctance to use RED in today's 162 deployments comes from its sensitivity to the operating conditions in 163 the network and the difficulty of tuning its parameters. 165 A buffer is a physical volume of memory in which a queue or set of 166 queues are stored. In real implementations of switches, a global 167 memory is shared between the available devices: the size of the 168 buffer for a given communication does not make sense, as its 169 dedicated memory may vary over the time and real-world buffering 170 architectures are complex. For the sake of simplicity, when speaking 171 of a specific queue in this document, "buffer size" refers to the 172 maximum amount of data the buffer may store, which can be measured in 173 bytes or packets. The rest of this memo therefore refers to the 174 maximum queue depth as the size of the buffer for a given 175 communication. 177 Bufferbloat [BB2011] is the consequence of deploying large unmanaged 178 buffers on the Internet, which has lead to an increase in end-to-end 179 delay: the buffering has often been measured to be ten times or 180 hundred times larger than needed. This results in poor performance 181 for latency-sensitive applications such as real-time multimedia 182 (e.g., voice, video, gaming, etc). The degree to which this affects 183 modern networking equipment, especially consumer-grade equipment's, 184 produces problems even with commonly used web services. Active queue 185 management is thus essential to control queuing delay and decrease 186 network latency. 188 The Active Queue Management and Packet Scheduling Working Group (AQM 189 WG) was recently formed within the TSV area to address the problems 190 with large unmanaged buffers in the Internet. Specifically, the AQM 191 WG is tasked with standardizing AQM schemes that not only address 192 concerns with such buffers, but also are robust under a wide variety 193 of operating conditions. 195 In order to ascertain whether the WG should undertake standardizing 196 an AQM proposal, the WG requires guidelines for assessing AQM 197 proposals. This document provides the necessary characterization 198 guidelines. [I-D.ietf-aqm-recommendation] separately describes the 199 AQM algorithm implemented in a router from the scheduling of packets 200 sent by the router. The rest of this memo refers to the AQM as a 201 dropping/marking policy as a separate feature to any interface 202 scheduling scheme. This document may be complemented with another 203 one on guidelines for assessing combination of packet scheduling and 204 AQM. We note that such a document will inherit all the guidelines 205 from this document plus any additional scenarios relevant for packet 206 scheduling such as flow starvation evaluation or impact of the number 207 of hash buckets. 209 1.1. Reducing the latency and maximizing the goodput 211 The trade-off between reducing the latency and maximizing the goodput 212 is intrinsically linked to each AQM scheme and is key to evaluating 213 its performance. This trade-off MUST be considered in a variety of 214 scenarios to ensure the safety of an AQM deployment. Whenever 215 possible, solutions ought to aim at both maximizing goodput and 216 minimizing latency. This document provides guidelines that enable 217 the reader to quantify (1) reduction of latency, (2) maximization of 218 goodput and (3) the trade-off between the two. 220 These guidelines provide the tools to understand the deployment costs 221 versus the potential gain in performance from the introduction of the 222 proposed scheme. 224 1.2. Guidelines for AQM evaluation 226 The guidelines help to quantify performance of AQM schemes in terms 227 of latency reduction, goodput maximization and the trade-off between 228 these two. The guidelines also help to discuss safe deployment of 229 AQM, including self-adaptation, stability analysis, fairness, design 230 and implementation complexity and robustness to different operating 231 conditions. 233 This memo details generic characterization scenarios against which 234 any AQM proposal needs to be evaluated, irrespective of whether or 235 not an AQM is standardized by the IETF. This documents recommends 236 the relevant scenarios and metrics to be considered. 238 The document presents central aspects of an AQM algorithm that must 239 be considered whatever the context, such as burst absorption 240 capacity, RTT fairness or resilience to fluctuating network 241 conditions. These guidelines do not cover every possible aspect of a 242 particular algorithm. In addition, it is worth noting that the 243 proposed criteria are not bound to a particular evaluation toolset. 245 This document details how an AQM designer can rate the feasibility of 246 their proposal in different types of network devices (switches, 247 routers, firewalls, hosts, drivers, etc) where an AQM may be 248 implemented. However, these guidelines do not present context- 249 dependent scenarios (such as 802.11 WLANs, data-centers or rural 250 broadband networks). 252 1.3. Requirements Language 254 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 255 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 256 document are to be interpreted as described in RFC 2119 [RFC2119]. 258 1.4. Glossary 260 o AQM: [I-D.ietf-aqm-recommendation] separately describes the AQM 261 algorithm implemented in a router from the scheduling of packets 262 sent by the router. The rest of this memo refers to the AQM as a 263 dropping/marking policy as a separate feature to any interface 264 scheduling scheme. 266 o buffer: a physical volume of memory in which a queue or set of 267 queues are stored. 269 o buffer size: the maximum amount of data that may be stored in a 270 buffer, measured in bytes or packets. 272 2. End-to-end metrics 274 End-to-end delay is the result of propagation delay, serialization 275 delay, service delay in a switch, medium-access delay and queuing 276 delay, summed over the network elements along the path. AQM schemes 277 may reduce the queuing delay by providing signals to the sender on 278 the emergence of congestion, but any impact on the goodput must be 279 carefully considered. This section presents the metrics that could 280 be used to better quantify (1) the reduction of latency, (2) 281 maximization of goodput and (3) the trade-off between these two. 282 This section provides normative requirements for metrics that can be 283 used to assess the performance of an AQM scheme. 285 Some metrics listed in this section are not suited to every type of 286 traffic detailed in the rest of this document. It is therefore not 287 necessary to measure all of the following metrics: the chosen metric 288 may not be relevant to the context of the evaluation scenario (e.g., 289 latency vs. goodput trade-off in application-limited traffic 290 scenarios). Guidance is provided for each metric. 292 2.1. Flow completion time 294 The flow completion time is an important performance metric for the 295 end-user when the flow size is finite. Considering the fact that an 296 AQM scheme may drop/mark packets, the flow completion time is 297 directly linked to the dropping/marking policy of the AQM scheme. 298 This metric helps to better assess the performance of an AQM 299 depending on the flow size. The Flow Completion Time (FCT) is 300 related to the flow size (Fs) and the goodput for the flow (G) as 301 follows: 303 FCT [s] = Fs [B] / ( G [Mbps] / 8 ) 305 If this metric is used to evaluate the performance of web transfers, 306 we propose to rather consider the time needed to download all the 307 objects that compose the web page, as this makes more sense in terms 308 of user experience than assessing the time needed to download each 309 object. 311 2.2. Flow start up time 313 The flow start up time is the time between the request has been sent 314 from the client and the server starts to transmit data. The amount 315 of packets dropped by an AQM may seriously affect the waiting period 316 during which the data transfer has not started. This metric would 317 specifically focus on the operations such as DNS lookups, TCP opens 318 of SSL handshakes. 320 2.3. Packet loss 322 Packet loss can occur within a network device, this can impact the 323 end-to-end performance measured at receiver. 325 The tester SHOULD evaluate loss experienced at the receiver using one 326 of the two metrics: 328 o the packet loss probability: this metric is to be frequently 329 measured during the experiment. The long-term loss probability is 330 of interest for steady-state scenarios only; 332 o the interval between consecutive losses: the time between two 333 losses is to be measured. 335 The packet loss probability can be assessed by simply evaluating the 336 loss ratio as a function of the number of lost packets and the total 337 number of packets sent. This might not be easily done in laboratory 338 testing, for which these guidelines advice the tester: 340 o to check that for every packet, a corresponding packet was 341 received within a reasonable time, as explained in [RFC2680]. 343 o to keep a count of all packets sent, and a count of the non- 344 duplicate packets received, as explained in the section 10 of 345 [RFC2544]. 347 The interval between consecutive losses, which is also called a gap, 348 is a metric of interest for VoIP traffic and, as a result, has been 349 further specified in [RFC3611]. 351 2.4. Packet loss synchronization 353 One goal of an AQM algorithm ought be to help to avoid global 354 synchronization of flows sharing a bottleneck buffer on which the AQM 355 operates ([RFC2309],[I-D.ietf-aqm-recommendation]). The "degree" of 356 packet-loss synchronization between flows SHOULD be assessed, with 357 and without the AQM under consideration. 359 As discussed e.g., in [HASS2008], loss synchronization among flows 360 may be quantified by several slightly different metrics that capture 361 different aspects of the same issue. However, in real-world 362 measurements the choice of metric could be imposed by practical 363 considerations -- e.g., whether fine-grained information on packet 364 losses in the bottleneck available or not. For the purpose of AQM 365 characterization, a good candidate metric is the global 366 synchronization ratio, measuring the proportion of flows losing 367 packets during a loss event. [JAY2006] used this metric in real- 368 world experiments to characterize synchronization along arbitrary 369 Internet paths; the full methodology is described in [JAY2006]. 371 If an AQM scheme is evaluated using real-life network environments, 372 it is worth pointing out that some network events, such as failed 373 link restoration may cause synchronized losses between active flows 374 and thus confuse the meaning of this metric. 376 2.5. Goodput 378 The goodput has been defined in the section 3.17 of [RFC2647] as the 379 number of bits per unit of time forwarded to the correct destination 380 interface of the Device Under Test or the System Under Test, minus 381 any bits lost or retransmitted. This definition induces that the 382 test setup needs to be qualified to assure that it is not generating 383 losses on its own. 385 Measuring the end-to-end goodput provides an appreciation of how well 386 an AQM scheme improves transport and application performance. The 387 measured end-to-end goodput is linked to the dropping/marking policy 388 of the AQM scheme -- e.g., the fewer the number of packet drops, the 389 fewer packets need retransmission, minimizing the impact of AQM on 390 transport and application performance. Additionally, an AQM scheme 391 may resort to Explicit Congestion Notification (ECN) marking as an 392 initial means to control delay. Again, marking packets instead of 393 dropping them reduces the number of packet retransmissions and 394 increases goodput. End-to-end goodput values help to evaluate the 395 AQM scheme's effectiveness of an AQM scheme in minimizing packet 396 drops that impact application performance and to estimate how well 397 the AQM scheme works with ECN. 399 The measurement of the goodput allows the tester evaluate to which 400 extent an AQM is able to maintain a high bottleneck utilization. 401 This metric should be also obtained frequently during an experiment 402 as the long-term goodput is relevant for steady-state scenarios only 403 and may not necessarily reflect how the introduction of an AQM 404 actually impacts the link utilization during at a certain period of 405 time. Fluctuations in the values obtained from these measurements 406 may depend on other factors than the introduction of an AQM, such as 407 link layer losses due to external noise or corruption, fluctuating 408 bandwidths (802.11 WLANs), heavy congestion levels or transport 409 layer's rate reduction by congestion control mechanism. 411 2.6. Latency and jitter 413 The latency, or the one-way delay metric, is discussed in [RFC2679]. 414 There is a consensus on a adequate metric for the jitter, that 415 represents the one-way delay variations for packets from the same 416 flow: the Packet Delay Variation (PDV), detailed in [RFC5481], serves 417 well all use cases. 419 The end-to-end latency differs from the queuing delay: it is linked 420 to the network topology and the path characteristics. Moreover, the 421 jitter also strongly depends on the traffic pattern and the topology. 422 The introduction of an AQM scheme would impact these metrics and 423 therefore they should be considered in the end-to-end evaluation of 424 performance. 426 2.7. Discussion on the trade-off between latency and goodput 428 The metrics presented in this section may be considered as explained 429 in the rest of this document, in order to discuss and quantify the 430 trade-off between latency and goodput. 432 This trade-off can also be illustrated with figures following the 433 recommendations of section 5 of [HAYE2013]. Each of the end-to-end 434 delay and the goodput SHOULD be measured frequently for every fixed 435 time interval. 437 With regards to the goodput, and in addition to the long-term 438 stationary goodput value, it is RECOMMENDED to take measurements 439 every multiple of RTTs. We suggest a minimum value of 10 x RTT (to 440 smooth out the fluctuations) but higher values are encouraged 441 whenever appropriate for the presentation depending on the network's 442 path characteristics. The measurement period MUST be disclosed for 443 each experiment and when results/values are compared across different 444 AQM schemes, the comparisons SHOULD use exactly the same measurement 445 periods. 447 With regards to latency, it is RECOMMENDED to take the samples on 448 per-packet basis whenever possible depending on the features provided 449 by hardware/software and the impact of sampling itself on the 450 hardware performance. It is generally RECOMMENDED to provide at 451 least 10 samples per RTT. 453 From each of these sets of measurements, the CDF of the considered 454 metrics SHOULD be computed. For each scenario, the following graph 455 may be generated: the x-axis shows the end-to-end delay, the y-axis 456 the goodput and ellipses are computed such as detailed in [WINS2014]. 457 This graph provides part of a better understanding of (1) the delay/ 458 goodput trade-off for a given congestion control mechanism, and (2) 459 how the goodput and average queue size vary as a function of the 460 traffic load. 462 3. Generic setup for evaluations 464 This section presents the topology that can be used for each of the 465 following scenarios, the corresponding notations and discusses 466 various assumptions that have been made in the document. 468 3.1. Topology and notations 470 +---------+ +-----------+ 471 |senders A| |receivers B| 472 +---------+ +-----------+ 474 +--------------+ +--------------+ 475 |traffic class1| |traffic class1| 476 |--------------| |--------------| 477 | SEN.Flow1.1 +---------+ +-----------+ REC.Flow1.1 | 478 | + | | | | + | 479 | | | | | | | | 480 | + | | | | + | 481 | SEN.Flow1.X +-----+ | | +--------+ REC.Flow1.X | 482 +--------------+ | | | | +--------------+ 483 + +-+---+---+ +--+--+---+ + 484 | |Router L | |Router R | | 485 | |---------| |---------| | 486 | | AQM | | | | 487 | | BuffSize| | | | 488 | | (Bsize) +-----+ | | 489 | +-----+--++ ++-+------+ | 490 + | | | | + 491 +--------------+ | | | | +--------------+ 492 |traffic classN| | | | | |traffic classN| 493 |--------------| | | | | |--------------| 494 | SEN.FlowN.1 +---------+ | | +-----------+ REC.FlowN.1 | 495 | + | | | | + | 496 | | | | | | | | 497 | + | | | | + | 498 | SEN.FlowN.Y +------------+ +-------------+ REC.FlowN.Y | 499 +--------------+ +--------------+ 501 Figure 1: Topology and notations 503 Figure 1 is a generic topology where: 505 o various classes of traffic can be introduced; 507 o the timing of each flow could be different (i.e., when does each 508 flow start and stop); 510 o each class of traffic can comprise various number of flows; 512 o each link is characterized by a couple (RTT,capacity); 513 o flows are generated between A and B, sharing a bottleneck (Routers 514 L and R); 516 o the tester SHOULD consider both scenarios of asymmetric and 517 symmetric bottleneck links in terms of bandwidth. In case of 518 asymmetric link, the capacity from senders to receivers is higher 519 than the one from receivers to senders; the symmetric link 520 scenario provides a basic understanding of the operation of the 521 AQM mechanism whereas the asymmetric link scenario evaluates an 522 AQM mechanism in a more realistic setup; 524 o in asymmetric link scenarios, the tester SHOULD study the bi- 525 directional traffic between A and B (downlink and uplink) with the 526 AQM mechanism deployed on one direction only. The tester MAY 527 additionally consider a scenario with AQM mechanism being deployed 528 on both directions. In each scenario, the tester SHOULD 529 investigate the impact of drop policy of the AQM on TCP ACK 530 packets and its impact on the performance. 532 Although this topology may not perfectly reflect actual topologies, 533 the simple topology is commonly used in the world of simulations and 534 small testbeds. It can be considered as adequate to evaluate AQM 535 proposals, similarly to the topology proposed in [HAYE2013]. Testers 536 ought to pay attention to the topology that has been used to evaluate 537 an AQM scheme when comparing this scheme with a new proposed AQM 538 scheme. 540 3.2. Buffer size 542 The size of the buffers should be carefully chosen, and is to be set 543 to the bandwidth-delay product; the bandwidth being the bottleneck 544 capacity and the delay the larger RTT in the considered network. The 545 size of the buffer can impact on the AQM performance and is a 546 dimensioning parameter that will be considered when comparing AQM 547 proposals. 549 If the context or the application requires a specific buffer size, 550 the tester MUST justify and detail the way the maximum queue size is 551 set. Indeed, the maximum size of the buffer may affect the AQM's 552 performance and its choice SHOULD be elaborated for a fair comparison 553 between AQM proposals. While comparing AQM schemes the buffer size 554 SHOULD remain the same across the tests. 556 3.3. Congestion controls 558 This memo features three kind of congestion controls: 560 o Standard TCP congestion control: the base-line congestion control 561 is TCP NewReno with SACK, as explained in [RFC5681]. 563 o Aggressive congestion controls: a base-line congestion control for 564 this category is TCP Cubic. 566 o Less-than Best Effort (LBE) congestion controls: an LBE congestion 567 control 'results in smaller bandwidth and/or delay impact on 568 standard TCP than standard TCP itself, when sharing a bottleneck 569 with it.' [RFC6297] 571 Other transport congestion controls can OPTIONALLY be evaluated in 572 addition. Recent transport layer protocols are not mentioned in the 573 following sections, for the sake of simplicity. 575 4. Transport Protocols 577 Network and end-devices need to be configured with a reasonable 578 amount of buffer space to absorb transient bursts. In some 579 situations, network providers tend to configure devices with large 580 buffers to avoid packet drops triggered by a full buffer and to 581 maximize the link utilization for standard loss-based TCP traffic. 583 AQM algorithms are often evaluated by considering Transmission 584 Control Protocol (TCP) [RFC0793] with a limited number of 585 applications. TCP is a widely deployed transport. It fills up 586 unmanaged buffers until a sender transfering a bulk flow with TCP 587 receives a signal (packet drop) that reduces the sending rate. The 588 larger the buffer, the higher the buffer occupancy, and therefore the 589 queuing delay. An efficient AQM scheme sends out early congestion 590 signals to TCP to bring the queuing delay under control. 592 Not all endpoints (or applications) using TCP use the same flavor of 593 TCP. Variety of senders generate different classes of traffic which 594 may not react to congestion signals (aka non-responsive flows 595 [I-D.ietf-aqm-recommendation]) or may not reduce their sending rate 596 as expected (aka Transport Flows that are less responsive than 597 TCP[I-D.ietf-aqm-recommendation], also called "aggressive flows"). 598 In these cases, AQM schemes seek to control the queuing delay. 600 This section provides guidelines to assess the performance of an AQM 601 proposal for various traffic profiles -- different types of senders 602 (with different TCP congestion control variants, unresponsive, 603 aggressive). 605 4.1. TCP-friendly sender 607 4.1.1. TCP-friendly sender with the same initial congestion window 609 This scenario helps to evaluate how an AQM scheme reacts to a TCP- 610 friendly transport sender. A single long-lived, non application- 611 limited, TCP NewReno flow, with an Initial congestion Window (IW) set 612 to 3 packets, transfers data between sender A and receiver B. Other 613 TCP friendly congestion control schemes such as TCP-friendly rate 614 control [RFC5348] etc MAY also be considered. 616 For each TCP-friendly transport considered, the graph described in 617 Section 2.7 could be generated. 619 4.1.2. TCP-friendly sender with different initial congestion windows 621 This scenario can be used to evaluate how an AQM scheme adapts to a 622 traffic mix consisting of TCP flows with different values of the IW. 624 For this scenario, two types of flows MUST be generated between 625 sender A and receiver B: 627 o A single long-lived non application-limited TCP NewReno flow; 629 o A single long-lived application-limited TCP NewReno flow, with an 630 IW set to 3 or 10 packets. The size of the data transferred must 631 be strictly higher than 10 packets and should be lower than 100 632 packets. 634 The transmission of the non application-limited flow must start 635 before the transmission of the application-limited flow and only 636 after the steady state has been reached by non application-limited 637 flow. 639 For each of these scenarios, the graph described in Section 2.7 could 640 be generated for each class of traffic (application-limited and non 641 application-limited). The completion time of the application-limited 642 TCP flow could be measured. 644 4.2. Aggressive transport sender 646 This scenario helps testers to evaluate how an AQM scheme reacts to a 647 transport sender that is more aggressive than a single TCP-friendly 648 sender. We define 'aggressiveness' as a higher increase factor than 649 standard upon a successful transmission and/or a lower than standard 650 decrease factor upon a unsuccessful transmission (e.g., in case of 651 congestion controls with Additive-Increase Multiplicative-Decrease 652 (AIMD) principle, a larger AI and/or MD factors). A single long- 653 lived, non application-limited, TCP Cubic flow transfers data between 654 sender A and receiver B. Other aggressive congestion control schemes 655 MAY also be considered. 657 For each flavor of aggressive transports, the graph described in 658 Section 2.7 could be generated. 660 4.3. Unresponsive transport sender 662 This scenario helps testers to evaluate how an AQM scheme reacts to a 663 transport sender that is less responsive than TCP. Note that faulty 664 transport implementations on an end host and/or faulty network 665 elements en-route that "hide" congestion signals in packet headers 666 [I-D.ietf-aqm-recommendation] may also lead to a similar situation, 667 such that the AQM scheme needs to adapt to unresponsive traffic. To 668 this end, these guidelines propose the two following scenarios. 670 The first scenario can be used to evaluate queue build up. It 671 considers unresponsive flow(s) whose sending rate is greater than the 672 bottleneck link capacity between routers L and R. This scenario 673 consists of a long-lived non application limited UDP flow transmits 674 data between sender A and receiver B. Graphs described in 675 Section 2.7 could be generated. 677 The second scenario can be used to evaluate if the AQM scheme is able 678 to keep responsive fraction under control. This scenario considers a 679 mixture of TCP-friendly and unresponsive traffics. It consists of a 680 long-lived non application-limited UDP flow and a single long-lived, 681 non-application-limited, TCP New Reno flow that transmit data between 682 sender A and receiver B. As opposed to the first scenario, the rate 683 of the UDP traffic should not be greater than the bottleneck 684 capacity, and should not be higher than half of the bottleneck 685 capacity. For each type of traffic, the graph described in 686 Section 2.7 could be generated. 688 4.4. Less-than Best Effort transport sender 690 This scenario helps to evaluate how an AQM scheme reacts to LBE 691 congestion controls that 'results in smaller bandwidth and/or delay 692 impact on standard TCP than standard TCP itself, when sharing a 693 bottleneck with it.' [RFC6297]. The potential fateful interaction 694 when AQM and LBE techniques are combined has been shown in 695 [GONG2014]; this scenario helps to evaluate whether the coexistence 696 of the proposed AQM and LBE techniques may be possible. 698 Single long-lived non application-limited TCP NewReno flows transfer 699 data between sender A and receiver B. Other TCP-friendly congestion 700 control schemes MAY also be considered. Single long-lived non 701 application-limited LEDBAT [RFC6817] flows transfer data between 702 sender A and receiver B. We recommend to set the target delay and 703 gain values of LEDBAT respectively to 5 ms and 10 [TRAN2014]. Other 704 LBE congestion control schemes, any of those listed in [RFC6297], MAY 705 also be considered. 707 For each of the TCP-friendly and LBE transports, the graph described 708 in Section 2.7 could be generated. 710 5. Round Trip Time Fairness 712 5.1. Motivation 714 The ability of AQM schemes to control the queuing delay highly 715 depends on the way end-to-end protocols react to congestion signals. 716 When the RTT varies, the behaviour of a congestion control is 717 impacted and this impacts the ability of an AQM scheme to control the 718 queue. It is therefore important to assess the AQM schemes for a set 719 of RTTs (e.g., from 5 ms to 200 ms). 721 The asymmetry in terms of difference in intrinsic RTT between various 722 paths sharing the same bottleneck SHOULD be considered so that the 723 fairness between the flows can be discussed since in this scenario, a 724 flow traversing on shorter RTT path may react faster to congestion 725 and recover faster from it compared to another flow on a longer RTT 726 path. The introduction of AQM schemes may potentially improve this 727 type of fairness. 729 Introducing an AQM scheme may cause the unfairness between the flows, 730 even if the RTTs are identical. This potential unfairness SHOULD be 731 investigated as well. 733 5.2. Recommended tests 735 The RECOMMENDED topology is detailed in Figure 1: 737 o To evaluate the inter-RTT fairness, for each run, two flows 738 divided into two categories. Category I which RTT between sender 739 A and Router L SHOULD be 100ms. Category II which RTT between 740 sender A and Router L should be in [5ms;560ms]. The maximum value 741 for the RTT represents the RTT of a satellite link that, according 742 to section 2 of [RFC2488] should be at least 558ms. 744 o To evaluate the impact of the RTT value on the AQM performance and 745 the intra-protocol fairness (the fairness for the flows using the 746 same paths/congestion control), for each run, two flows (Flow1 and 747 Flow2) should be introduced. For each experiment, the set of RTT 748 SHOULD be the same for the two flows and in [5ms;560ms]. 750 A set of evaluated flows MUST use the same congestion control 751 algorithm. 753 5.3. Metrics to evaluate the RTT fairness 755 The outputs that MUST be measured are: 757 o for the inter-RTT fairness: (1) the cumulative average goodput of 758 the flow from Category I, goodput_Cat_I (Section 2.5); (2) the 759 cumulative average goodput of the flow from Category II, 760 goodput_Cat_II (Section 2.5); (3) the ratio goodput_Cat_II/ 761 goodput_Cat_I; (4) the average packet drop rate for each category 762 (Section 2.3). 764 o for the intra-protocol RTT fairness: (1) the cumulative average 765 goodput of the two flows (Section 2.5); (2) the average packet 766 drop rate for the two flows (Section 2.3). 768 6. Burst Absorption 770 "AQM mechanisms need to control the overall queue sizes, to ensure 771 that arriving bursts can be accommodated without dropping packets" 772 [I-D.ietf-aqm-recommendation]. 774 6.1. Motivation 776 An AQM scheme can result in bursts of packet arrivals due to various 777 reasons. Dropping one or more packets from a burst can result in 778 performance penalties for the corresponding flows, since dropped 779 packets have to be retransmitted. Performance penalties can result 780 in failing to meet SLAs and be a disincentive to AQM adoption. 782 The ability to accommodate bursts translates to larger queue length 783 and hence more queuing delay. On the one hand, it is important that 784 an AQM scheme quickly brings bursty traffic under control. On the 785 other hand, a peak in the packet drop rates to bring a packet burst 786 quickly under control could result in multiple drops per flow and 787 severely impact transport and application performance. Therefore, an 788 AQM scheme ought to bring bursts under control by balancing both 789 aspects -- (1) queuing delay spikes are minimized and (2) performance 790 penalties for ongoing flows in terms of packet drops are minimized. 792 An AQM scheme that maintains short queues allows some remaining space 793 in the queue for bursts of arriving packets. The tolerance to bursts 794 of packets depends upon the number of packets in the queue, which is 795 directly linked to the AQM algorithm. Moreover, one AQM scheme may 796 implement a feature controlling the maximum size of accepted bursts, 797 that can depend on the buffer occupancy or the currently estimated 798 queuing delay. The impact of the buffer size on the burst allowance 799 may be evaluated. 801 6.2. Recommended tests 803 For this scenario, tester MUST evaluate how the AQM performs with the 804 following traffic generated from sender A to receiver B: 806 o Web traffic with IW10; 808 o Bursty video frames; 810 o Constant bit rate UDP traffic. 812 o A single bulk TCP flow as background traffic. 814 Figure 2 presents the various cases for the traffic that MUST be 815 generated between sender A and receiver B. 817 +-------------------------------------------------+ 818 |Case| Traffic Type | 819 | +-----+------------+----+--------------------+ 820 | |Video|Webs (IW 10)| CBR| Bulk TCP Traffic | 821 +----|-----|------------|----|--------------------| 822 |I | 0 | 1 | 1 | 0 | 823 +----|-----|------------|----|--------------------| 824 |II | 0 | 1 | 1 | 1 | 825 |----|-----|------------|----|--------------------| 826 |III | 1 | 1 | 1 | 0 | 827 +----|-----|------------|----|--------------------| 828 |IV | 1 | 1 | 1 | 1 | 829 +----+-----+------------+----+--------------------+ 831 Figure 2: Bursty traffic scenarios 833 A new web page download could start after the previous web page 834 download is finished. Each web page could be composed by at least 50 835 objects and the size of each object should be at least 1kB. 6 TCP 836 parallel connections SHOULD be generated to download the objects, 837 each parallel connections having an initial congestion window set to 838 10 packets. 840 For each of these scenarios, the graph described in Section 2.7 could 841 be generated. Metrics such as end-to-end latency, jitter, flow 842 completion time MAY be generated. For the cases of frame generation 843 of bursty video traffic as well as the choice of web traffic pattern, 844 we leave these details and their presentation to the testers. 846 7. Stability 848 7.1. Motivation 850 Network devices can experience varying operating conditions depending 851 on factors such as time of the day, deployment scenario, etc. For 852 example: 854 o Traffic and congestion levels are higher during peak hours than 855 off-peak hours. 857 o In the presence of a scheduler, the draining rate of a queue can 858 vary depending on the occupancy of other queues: a low load on a 859 high priority queue implies a higher draining rate for the lower 860 priority queues. 862 o The capacity available can vary over time (e.g., a lossy channel, 863 a link supporting traffic in a higher diffserv class). 865 Whether the target context is a not stable environment, the ability 866 of an AQM scheme to maintain its control over the queuing delay and 867 buffer occupancy can be challenged. This document proposes 868 guidelines to assess the behavior of AQM schemes under varying 869 congestion levels and varying draining rates. 871 7.2. Recommended tests 873 Note that the traffic profiles explained below comprises non 874 application-limited TCP flows. For each of the below scenarios, the 875 results described in Section 2.7 SHOULD be generated. For 876 Section 7.2.5 and Section 7.2.6 they SHOULD incorporate the results 877 in per-phase basis as well. 879 Wherever the notion of time has explicitly mentioned in this 880 subsection, time 0 starts from the moment all TCP flows have already 881 reached their congestion avoidance phase. 883 7.2.1. Definition of the congestion Level 885 In these guidelines, the congestion levels are represented by the 886 projected packet drop rate, had a drop-tail queue was chosen instead 887 of an AQM scheme. When the bottleneck is shared among non- 888 application-limited TCP flows. l_r, the loss rate projection can be 889 expressed as a function of N, the number of bulk TCP flows, and S, 890 the sum of the bandwidth-delay product and the maximum buffer size, 891 both expressed in packets, based on Eq. 3 of [MORR2000]: 893 l_r = 0.76 * N^2 / S^2 894 N = S * sqrt(1/0.76) * sqrt (l_r) 896 These guidelines use the loss rate to define the different congestion 897 levels, but they do not stipulate that in other circumstances, 898 measuring the congestion level gives you an accurate estimation of 899 the loss rate or vice-versa. 901 7.2.2. Mild congestion 903 This scenario can be used to evaluate how an AQM scheme reacts to a 904 light load of incoming traffic resulting in mild congestion -- packet 905 drop rates around 0.1%. The number of bulk flows required to achieve 906 this congestion level, N_mild, is then: 908 N_mild = round(0.036*S) 910 7.2.3. Medium congestion 912 This scenario can be used to evaluate how an AQM scheme reacts to 913 incoming traffic resulting in medium congestion -- packet drop rates 914 around 0.5%. The number of bulk flows required to achieve this 915 congestion level, N_med, is then: 917 N_med = round (0.081*S) 919 7.2.4. Heavy congestion 921 This scenario can be used to evaluate how an AQM scheme reacts to 922 incoming traffic resulting in heavy congestion -- packet drop rates 923 around 1%. The number of bulk flows required to achieve this 924 congestion level, N_heavy, is then: 926 N_heavy = round (0.114*S) 928 7.2.5. Varying the congestion level 930 This scenario can be used to evaluate how an AQM scheme reacts to 931 incoming traffic resulting in various levels of congestion during the 932 experiment. In this scenario, the congestion level varies within a 933 large time-scale. The following phases may be considered: phase I - 934 mild congestion during 0-20s; phase II - medium congestion during 935 20-40s; phase III - heavy congestion during 40-60s; phase I again, 936 and so on. 938 7.2.6. Varying available capacity 940 This scenario can be used to help characterize how the AQM behaves 941 and adapts to bandwidth changes. The experiments are not meant to 942 reflect the exact conditions of Wi-Fi environments since its hard to 943 design repetitive experiments or accurate simulations for such 944 scenarios. 946 To emulate varying draining rates, the bottleneck capacity between 947 nodes 'Router L' and 'Router R' varies over the course of the 948 experiment as follows: 950 o Experiment 1: the capacity varies between two values within a 951 large time-scale. As an example, the following phases may be 952 considered: phase I - 100Mbps during 0-20s; phase II - 10Mbps 953 during 20-40s; phase I again, and so on. 955 o Experiment 2: the capacity varies between two values within a 956 short time-scale. As an example, the following phases may be 957 considered: phase I - 100Mbps during 0-100ms; phase II - 10Mbps 958 during 100-200ms; phase I again, and so on. 960 The tester MAY choose a phase time-interval value different than what 961 is stated above, if the network's path conditions (such as bandwidth- 962 delay product) necessitate. In this case the choice of such time- 963 interval value SHOULD be stated and elaborated. 965 The tester MAY additionally evaluate the two mentioned scenarios 966 (short-term and long-term capacity variations), during and/or 967 including TCP slow-start phase. 969 More realistic fluctuating capacity patterns MAY be considered. The 970 tester MAY choose to incorporate realistic scenarios with regards to 971 common fluctuation of bandwidth in state-of-the-art technologies. 973 The scenario consist of TCP NewReno flows between sender A and 974 receiver B. To better assess the impact of draining rates on the AQM 975 behavior, the tester MUST compare its performance with those of drop- 976 tail and SHOULD provide a reference document for their proposal 977 discussing performance and deployment compared to those of drop-tail. 978 Burst traffic, such as presented in Section 6.2, could also be 979 considered to assess the impact of varying available capacity on the 980 burst absorption of the AQM. 982 7.3. Parameter sensitivity and stability analysis 984 The control law used by an AQM is the primary means by which the 985 queuing delay is controlled. Hence understanding the control law is 986 critical to understanding the behavior of the AQM scheme. The 987 control law could include several input parameters whose values 988 affect the AQM scheme's output behavior and its stability. 989 Additionally, AQM schemes may auto-tune parameter values in order to 990 maintain stability under different network conditions (such as 991 different congestion levels, draining rates or network environments). 992 The stability of these auto-tuning techniques is also important to 993 understand. 995 Transports operating under the control of AQM experience the effect 996 of multiple control loops that react over different timescales. It 997 is therefore important that proposed AQM schemes are seen to be 998 stable when they are deployed at multiple points of potential 999 congestion along an Internet path. The pattern of congestion signals 1000 (loss or ECN-marking) arising from AQM methods also need to not 1001 adversely interact with the dynamics of the transport protocols that 1002 they control. 1004 AQM proposals SHOULD provide background material showing control 1005 theoretic analysis of the AQM control law and the input parameter 1006 space within which the control law operates as expected; or could use 1007 another way to discuss the stability of the control law. For 1008 parameters that are auto-tuned, the material SHOULD include stability 1009 analysis of the auto-tuning mechanism(s) as well. Such analysis 1010 helps to understand an AQM control law better and the network 1011 conditions/deployments under which the AQM is stable. 1013 8. Various Traffic Profiles 1015 This section provides guidelines to assess the performance of an AQM 1016 proposal for various traffic profiles such as traffic with different 1017 applications or bi-directional traffic. 1019 8.1. Traffic mix 1021 This scenario can be used to evaluate how an AQM scheme reacts to a 1022 traffic mix consisting of different applications such as: 1024 o Bulk TCP transfer 1026 o Web traffic 1028 o VoIP 1029 o Constant Bit Rate (CBR) UDP traffic 1031 o Adaptive video streaming 1033 Various traffic mixes can be considered. These guidelines RECOMMEND 1034 to examine at least the following example: 1 bi-directional VoIP; 6 1035 Webs pages download (such as detailed in Section 6.2); 1 CBR; 1 1036 Adaptive Video; 5 bulk TCP. Any other combinations could be 1037 considered and should be carefully documented. 1039 For each scenario, the graph described in Section 2.7 could be 1040 generated for each class of traffic. Metrics such as end-to-end 1041 latency, jitter and flow completion time MAY be reported. 1043 8.2. Bi-directional traffic 1045 Control packets such as DNS requests/responses, TCP SYNs/ACKs are 1046 small, but their loss can severely impact the application 1047 performance. The scenario proposed in this section will help in 1048 assessing whether the introduction of an AQM scheme increases the 1049 loss probability of these important packets. 1051 For this scenario, traffic MUST be generated in both downlink and 1052 uplink, such as defined in Section 3.1. These guidelines RECOMMEND 1053 to consider a mild congestion level and the traffic presented in 1054 Section 7.2.2 in both directions. In this case, the metrics reported 1055 MUST be the same as in Section 7.2 for each direction. 1057 The traffic mix presented in Section 8.1 MAY also be generated in 1058 both directions. 1060 9. Multi-AQM Scenario 1062 9.1. Motivation 1064 Transports operating under the control of AQM experience the effect 1065 of multiple control loops that react over different timescales. It 1066 is therefore important that proposed AQM schemes are seen to be 1067 stable when they are deployed at multiple points of potential 1068 congestion along an Internet path. The pattern of congestion signals 1069 (loss or ECN-marking) arising from AQM methods also need to not 1070 adversely interact with the dynamics of the transport protocols that 1071 they control. 1073 9.2. Details on the evaluation scenario 1075 +---------+ +-----------+ 1076 |senders A|---+ +---|receivers A| 1077 +---------+ | | +-----------+ 1078 +-----+---+ +---------+ +--+-----+ 1079 |Router L |--|Router M |--|Router R| 1080 |AQM | |AQM | |No AQM | 1081 +---------+ +--+------+ +--+-----+ 1082 +---------+ | | +-----------+ 1083 |senders B|-------------+ +---|receivers B| 1084 +---------+ +-----------+ 1086 Figure 3: Topology for the Multi-AQM scenario 1088 This scenario can be used to evaluate how having AQM schemes in 1089 sequence impact the induced latency reduction, the induced goodput 1090 maximization and the trade-off between these two. The topology 1091 presented in Figure 3 could be used. We recommend that the AQM 1092 schemes introduced in Router L and Router M should be the same; any 1093 other configurations could be considered. For this scenario, we 1094 recommend to consider a mild congestion level, the number of flows 1095 specified in Section 7.2.2 being equally shared among senders A and 1096 B. Any other relevant combination of congestion levels could be 1097 considered. We recommend to measure the metrics presented in 1098 Section 7.2. 1100 10. Implementation cost 1102 10.1. Motivation 1104 Successful deployment of AQM is directly related to its cost of 1105 implementation. Network devices can need hardware or software 1106 implementations of the AQM mechanism. Depending on a device's 1107 capabilities and limitations, the device may or may not be able to 1108 implement some or all parts of their AQM logic. 1110 AQM proposals SHOULD provide pseudo-code for the complete AQM scheme, 1111 highlighting generic implementation-specific aspects of the scheme 1112 such as "drop-tail" vs. "drop-head", inputs (e.g., current queuing 1113 delay, queue length), computations involved, need for timers, etc. 1114 This helps to identify costs associated with implementing the AQM 1115 scheme on a particular hardware or software device. This also helps 1116 the WG understand which kind of devices can easily support the AQM 1117 and which cannot. 1119 10.2. Recommended discussion 1121 AQM proposals SHOULD highlight parts of their AQM logic that are 1122 device dependent and discuss if and how AQM behavior could be 1123 impacted by the device. For example, a queueing-delay based AQM 1124 scheme requires current queuing delay as input from the device. If 1125 the device already maintains this value, then it can be trivial to 1126 implement the their AQM logic on the device. If the device provides 1127 indirect means to estimate the queuing delay (for example: 1128 timestamps, dequeuing rate), then the AQM behavior is sensitive to 1129 the precision of the queuing delay estimations are for that device. 1130 Highlighting the sensitivity of an AQM scheme to queuing delay 1131 estimations helps implementers to identify appropriate means of 1132 implementing the mechanism on a device. 1134 11. Operator Control and Auto-tuning 1136 11.1. Motivation 1138 One of the biggest hurdles of RED deployment was/is its parameter 1139 sensitivity to operating conditions -- how difficult it is to tune 1140 RED parameters for a deployment to achieve acceptable benefit from 1141 using RED. Fluctuating congestion levels and network conditions add 1142 to the complexity. Incorrect parameter values lead to poor 1143 performance. 1145 Any AQM scheme is likely to have parameters whose values affect the 1146 control law and behaviour of an AQM. Exposing all these parameters 1147 as control parameters to a network operator (or user) can easily 1148 result in a unsafe AQM deployment. Unexpected AQM behavior ensues 1149 when parameter values are set improperly. A minimal number of 1150 control parameters minimizes the number of ways a possibly naive user 1151 can break a system where an AQM scheme is deployed at. Fewer control 1152 parameters make the AQM scheme more user-friendly and easier to 1153 deploy and debug. 1155 [I-D.ietf-aqm-recommendation] states "AQM algorithms SHOULD NOT 1156 require tuning of initial or configuration parameters in common use 1157 cases." A scheme ought to expose only those parameters that control 1158 the macroscopic AQM behavior such as queue delay threshold, queue 1159 length threshold, etc. 1161 Additionally, the safety of an AQM scheme is directly related to its 1162 stability under varying operating conditions such as varying traffic 1163 profiles and fluctuating network conditions, as described in 1164 Section 7. Operating conditions vary often and hence the AQM needs 1165 to remain stable under these conditions without the need for 1166 additional external tuning. If AQM parameters require tuning under 1167 these conditions, then the AQM must self-adapt necessary parameter 1168 values by employing auto-tuning techniques. 1170 11.2. Required discussion 1172 AQM proposals SHOULD describe the parameters that control the 1173 macroscopic AQM behavior, and identify any parameters that require 1174 require tuning to operational conditions. It could be interesting to 1175 also discuss that even if an AQM scheme may not adequately auto-tune 1176 its parameters, the resulting performance may not be optimal, but 1177 close to something reasonable. 1179 If there are any fixed parameters within the AQM, their setting 1180 SHOULD be discussed and justified. 1182 If an AQM scheme is evaluated with parameter(s) that were externally 1183 tuned for optimization or other purposes, these values MUST be 1184 disclosed. 1186 12. Interaction with ECN 1188 Deployed AQM algorithms SHOULD implement Explicit Congestion 1189 Notification (ECN) as well as loss to signal congestion to endpoints 1190 [I-D.ietf-aqm-recommendation]. The benefits of providing ECN support 1191 for an AQM scheme are described in [WELZ2015]. Section 3 of 1192 [WELZ2015] describes expected operation of routers enabling ECN. AQM 1193 schemes SHOULD NOT drop or remark packets solely because the ECT(0) 1194 or ECT(1) codepoints are used, and when ECN-capable SHOULD set a CE- 1195 mark on ECN-capable packets in the presence of incipient congestion. 1197 12.1. Motivation 1199 ECN [RFC3168] is an alternative that allows AQM schemes to signal 1200 receivers about network congestion that does not use packet drop. 1202 12.2. Recommended discussion 1204 An AQM scheme can support ECN [I-D.ietf-aqm-recommendation], in which 1205 case testers MUST discuss and describe the support of ECN. 1207 13. Interaction with Scheduling 1209 A network device may use per-flow or per-class queuing with a 1210 scheduling algorithm to either prioritize certain applications or 1211 classes of traffic, limit the rate of transmission, or to provide 1212 isolation between different traffic flows within a common class 1213 [I-D.ietf-aqm-recommendation]. 1215 13.1. Motivation 1217 Coupled with an AQM scheme, a router may schedule the transmission of 1218 packets in a specific manner by introducing a scheduling scheme. 1219 This algorithm may create sub-queues and integrate a dropping policy 1220 on each of these sub-queues. Another scheduling policy may modify 1221 the way packets are sequenced, modifying the timestamp of each 1222 packet. 1224 13.2. Recommended discussion 1226 The scheduling and the AQM conjointly impact on the end-to-end 1227 performance. During the characterization process of a dropping 1228 policy, the tester MUST discuss the feasibility to add scheduling 1229 combined with the AQM algorithm. This discussion as an instance, MAY 1230 explain whether the dropping policy is applied when packets are being 1231 enqueued or dequeued. 1233 13.3. Assessing the interaction between AQM and scheduling 1235 These guidelines do not propose guidelines to assess the performance 1236 of scheduling algorithms. Indeed, as opposed to characterizing AQM 1237 schemes that is related to their capacity to control the queuing 1238 delay in a queue, characterizing scheduling schemes is related to the 1239 scheduling itself and its interaction with the AQM scheme. As one 1240 example, the scheduler may create sub-queues and the AQM scheme may 1241 be applied on each of the sub-queues, and/or the AQM could be applied 1242 on the whole queue. Also, schedulers might, such as FQ-CoDel 1243 [HOEI2015] or FavorQueue [ANEL2014], introduce flow prioritization. 1244 In these cases, specific scenarios should be proposed to ascertain 1245 that these scheduler schemes not only helps in tackling the 1246 bufferbloat, but also are robust under a wide variety of operating 1247 conditions. This is out of the scope of this document that focus on 1248 dropping and/or marking AQM schemes. 1250 14. Discussion on Methodology, Metrics, AQM Comparisons and Packet 1251 Sizes 1253 14.1. Methodology 1255 One key objective behind formulating the guidelines is to help 1256 ascertain whether a specific AQM is not only better than drop-tail 1257 but also safe to deploy. Testers therefore need to provide a 1258 reference document for their proposal discussing performance and 1259 deployment compared to those of drop-tail. 1261 A description of each test setup SHOULD be detailed to allow this 1262 test to be compared with other tests. This also allows others to 1263 replicate the tests if needed. This test setup SHOULD detail 1264 software and hardware versions. The tester could make its data 1265 available. 1267 The proposals SHOULD be evaluated on real-life systems, or they MAY 1268 be evaluated with event-driven simulations (such as ns-2, ns-3, 1269 OMNET, etc). The proposed scenarios are not bound to a particular 1270 evaluation toolset. 1272 The tester is encouraged to make the detailed test setup and the 1273 results publicly available. 1275 14.2. Comments on metrics measurement 1277 The document presents the end-to-end metrics that ought to be used to 1278 evaluate the trade-off between latency and goodput in Section 2. In 1279 addition to the end-to-end metrics, the queue-level metrics (normally 1280 collected at the device operating the AQM) provide a better 1281 understanding of the AQM behavior under study and the impact of its 1282 internal parameters. Whenever it is possible (e.g., depending on the 1283 features provided by the hardware/software), these guidelines advice 1284 to consider queue-level metrics, such as link utilization, queuing 1285 delay, queue size or packet drop/mark statistics in addition to the 1286 AQM-specific parameters. However, the evaluation MUST be primarily 1287 based on externally observed end-to-end metrics. 1289 These guidelines do not aim to detail on the way these metrics can be 1290 measured, since the way these metrics are measured is expected to 1291 depend on the evaluation toolset. 1293 14.3. Comparing AQM schemes 1295 This document recognizes that the guidelines mentioned above may be 1296 used for comparing AQM schemes. 1298 AQM schemes need to be compared against both performance and 1299 deployment categories. In addition, this section details how best to 1300 achieve a fair comparison of AQM schemes by avoiding certain 1301 pitfalls. 1303 14.3.1. Performance comparison 1305 AQM schemes MUST be compared against all the generic scenarios 1306 presented in this memo. AQM schemes MAY be compared for specific 1307 network environments such as data centers, home networks, etc. If an 1308 AQM scheme has parameter(s) that were externally tuned for 1309 optimization or other purposes, these values MUST be disclosed. 1311 AQM schemes belong to different varieties such as queue-length based 1312 schemes (ex. RED) or queueing-delay based scheme (ex. CoDel, PIE). 1313 AQM schemes expose different control knobs associated with different 1314 semantics. For example, while both PIE and CoDel are queueing-delay 1315 based schemes and each expose a knob to control the queueing delay -- 1316 PIE's "queueing delay reference" vs. CoDel's "queueing delay target", 1317 the two tuning parameters of the two schemes have different 1318 semantics, resulting in different control points. Such differences 1319 in AQM schemes can be easily overlooked while making comparisons. 1321 This document RECOMMENDS the following procedures for a fair 1322 performance comparison between the AQM schemes: 1324 1. comparable control parameters and comparable input values: 1325 carefully identify the set of parameters that control similar 1326 behavior between the two AQM schemes and ensure these parameters 1327 have comparable input values. For example, to compare how well a 1328 queue-length based AQM scheme controls queueing delay vs. a 1329 queueing-delay based AQM scheme, a tester can identify the 1330 parameters of the schemes that control queue delay and ensure 1331 that their input values are comparable. Similarly, to compare 1332 how well two AQM schemes accommodate packet bursts, the tester 1333 can identify burst-related control parameters and ensure they are 1334 configured with similar values. 1336 2. compare over a range of input configurations: there could be 1337 situations when the set of control parameters that affect a 1338 specific behavior have different semantics between the two AQM 1339 schemes. As mentioned above, PIE has tuning parameters to 1340 control queue delay that has a different semantics from those 1341 used in CoDel. In such situations, these schemes need to be 1342 compared over a range of input configurations. For example, 1343 compare PIE vs. CoDel over the range of target delay input 1344 configurations. 1346 14.3.2. Deployment comparison 1348 AQM schemes MUST be compared against deployment criteria such as the 1349 parameter sensitivity (Section 7.3), auto-tuning (Section 11) or 1350 implementation cost (Section 10). 1352 14.4. Packet sizes and congestion notification 1354 An AQM scheme may be considering packet sizes while generating 1355 congestion signals. [RFC7141] discusses the motivations behind this. 1356 For example, control packets such as DNS requests/responses, TCP 1357 SYNs/ACKs are small, but their loss can severely impact the 1358 application performance. An AQM scheme may therefore be biased 1359 towards small packets by dropping them with smaller probability 1360 compared to larger packets. However, such an AQM scheme is unfair to 1361 data senders generating larger packets. Data senders, malicious or 1362 otherwise, are motivated to take advantage of such AQM scheme by 1363 transmitting smaller packets, and could result in unsafe deployments 1364 and unhealthy transport and/or application designs. 1366 An AQM scheme SHOULD adhere to the recommendations outlined in 1367 [RFC7141], and SHOULD NOT provide undue advantage to flows with 1368 smaller packets [I-D.ietf-aqm-recommendation]. 1370 15. Conclusion 1372 Figure 4 lists the scenarios and their requirements. 1374 +------------------------------------------------------------------+ 1375 |Scenario |Sec. |Requirement | 1376 +------------------------------------------------------------------+ 1377 +------------------------------------------------------------------+ 1378 |Transport Protocols |4. | | 1379 | TCP-friendly sender | 4.1 |Scenario MUST be considered | 1380 | Aggressive sender | 4.2 |Scenario MUST be considered | 1381 | Unresponsive sender | 4.3 |Scenario MUST be considered | 1382 | LBE sender | 4.4 |Scenario MAY be considered | 1383 +------------------------------------------------------------------+ 1384 |Round Trip Time Fairness | 5.2 |Scenario MUST be considered | 1385 +------------------------------------------------------------------+ 1386 |Burst Absorption | 6.2 |Scenario MUST be considered | 1387 +------------------------------------------------------------------+ 1388 |Stability |7. | | 1389 | Varying congestion levels | 7.2.5|Scenario MUST be considered | 1390 | Varying available capacity| 7.2.6|Scenario MUST be considered | 1391 | Parameters and stability | 7.3 |This SHOULD be discussed | 1392 +------------------------------------------------------------------+ 1393 |Various Traffic Profiles |8. | | 1394 | Traffic mix | 8.1 |Scenario is RECOMMENDED | 1395 | Bi-directional traffic | 8.2 |Scenario MAY be considered | 1396 +------------------------------------------------------------------+ 1397 |Multi-AQM | 9.2 |Scenario MAY be considered | 1398 +------------------------------------------------------------------+ 1399 |Implementation Cost | 10.2 |Pseudo-code SHOULD be provided | 1400 +------------------------------------------------------------------+ 1401 |Operator Control | 11.2 |Tuning SHOULD NOT be required | 1402 +------------------------------------------------------------------+ 1403 |Interaction with ECN | 12.2 |MUST be discussed if supported | 1404 +------------------------------------------------------------------+ 1405 |Interaction with Scheduling| 13.2 |Feasibility MUST be discussed | 1406 +------------------------------------------------------------------+ 1408 Figure 4: Summary of the scenarios and their requirements 1410 16. Acknowledgements 1412 This work has been partially supported by the European Community 1413 under its Seventh Framework Programme through the Reducing Internet 1414 Transport Latency (RITE) project (ICT-317700). 1416 17. Contributors 1418 Many thanks to S. Akhtar, A.B. Bagayoko, F. Baker, D. Collier- 1419 Brown, G. Fairhurst, J. Gettys, T. Hoiland-Jorgensen, C. 1421 Kulatunga, W. Lautenschlager, A.C. Morton, R. Pan, D. Taht and M. 1422 Welzl for detailed and wise feedback on this document. 1424 18. IANA Considerations 1426 This memo includes no request to IANA. 1428 19. Security Considerations 1430 Some security considerations for AQM are identified in 1431 [I-D.ietf-aqm-recommendation].This document, by itself, presents no 1432 new privacy nor security issues. 1434 20. References 1436 20.1. Normative References 1438 [I-D.ietf-aqm-recommendation] 1439 Baker, F. and G. Fairhurst, "IETF Recommendations 1440 Regarding Active Queue Management", draft-ietf-aqm- 1441 recommendation-11 (work in progress), February 2015. 1443 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1444 Requirement Levels", RFC 2119, 1997. 1446 [RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion 1447 Notification", RFC 7141, 2014. 1449 20.2. Informative References 1451 [ANEL2014] 1452 Anelli, P., Diana, R., and E. Lochin, "FavorQueue: a 1453 Parameterless Active Queue Management to Improve TCP 1454 Traffic Performance", Computer Networks vol. 60, 2014. 1456 [BB2011] "BufferBloat: what's wrong with the internet?", ACM Queue 1457 vol. 9, 2011. 1459 [GONG2014] 1460 Gong, Y., Rossi, D., Testa, C., Valenti, S., and D. Taht, 1461 "Fighting the bufferbloat: on the coexistence of AQM and 1462 low priority congestion control", Computer Networks, 1463 Elsevier, 2014, 60, pp.115 - 128 , 2014. 1465 [HASS2008] 1466 Hassayoun, S. and D. Ros, "Loss Synchronization and Router 1467 Buffer Sizing with High-Speed Versions of TCP", IEEE 1468 INFOCOM Workshops , 2008. 1470 [HAYE2013] 1471 Hayes, D., Ros, D., Andrew, L., and S. Floyd, "Common TCP 1472 Evaluation Suite", IRTF (Work-in-Progress) , 2013. 1474 [HOEI2015] 1475 Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, 1476 J., and E. Dumazet, "FlowQueue-Codel", IETF (Work-in- 1477 Progress) , January 2015. 1479 [JAY2006] Jay, P., Fu, Q., and G. Armitage, "A preliminary analysis 1480 of loss synchronisation between concurrent TCP flows", 1481 Australian Telecommunication Networks and Application 1482 Conference (ATNAC) , 2006. 1484 [MORR2000] 1485 Morris, R., "Scalable TCP congestion control", IEEE 1486 INFOCOM , 2000. 1488 [NICH2012] 1489 Nichols, K. and V. Jacobson, "Controlling Queue Delay", 1490 ACM Queue , 2012. 1492 [PAN2013] Pan, R., Natarajan, P., Piglione, C., Prabhu, MS., 1493 Subramanian, V., Baker, F., and B. VerSteeg, "PIE: A 1494 lightweight control scheme to address the bufferbloat 1495 problem", IEEE HPSR , 2013. 1497 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC 1498 793, September 1981. 1500 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 1501 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 1502 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 1503 S., Wroclawski, J., and L. Zhang, "Recommendations on 1504 Queue Management and Congestion Avoidance in the 1505 Internet", RFC 2309, April 1998. 1507 [RFC2488] Allman, M., Glover, D., and L. Sanchez, "Enhancing TCP 1508 Over Satellite Channels using Standard Mechanisms", BCP 1509 28, RFC 2488, January 1999. 1511 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1512 Network Interconnect Devices", RFC 2544, March 1999. 1514 [RFC2647] Newman, D., "Benchmarking Terminology for Firewall 1515 Performance", RFC 2647, August 1999. 1517 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1518 Delay Metric for IPPM", RFC 2679, September 1999. 1520 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1521 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1523 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1524 of Explicit Congestion Notification (ECN) to IP", RFC 1525 3168, September 2001. 1527 [RFC3611] Friedman, T., Caceres, R., and A. Clark, "RTP Control 1528 Protocol Extended Reports (RTCP XR)", RFC 3611, November 1529 2003. 1531 [RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP 1532 Friendly Rate Control (TFRC): Protocol Specification", RFC 1533 5348, September 2008. 1535 [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation 1536 Applicability Statement", RFC 5481, March 2009. 1538 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1539 Control", RFC 5681, September 2009. 1541 [RFC6297] Welzl, M. and D. Ros, "A Survey of Lower-than-Best-Effort 1542 Transport Protocols", RFC 6297, June 2011. 1544 [RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind, 1545 "Low Extra Delay Background Transport (LEDBAT)", RFC 6817, 1546 December 2012. 1548 [TRAN2014] 1549 Trang, S., Kuhn, N., Lochin, E., Baudoin, C., Dubois, E., 1550 and P. Gelard, "On The Existence Of Optimal LEDBAT 1551 Parameters", IEEE ICC 2014 - Communication QoS, 1552 Reliability and Modeling Symposium , 2014. 1554 [WELZ2015] 1555 Welzl, M. and G. Fairhurst, "The Benefits to Applications 1556 of using Explicit Congestion Notification (ECN)", IETF 1557 (Work-in-Progress) , June 2015. 1559 [WINS2014] 1560 Winstein, K., "Transport Architectures for an Evolving 1561 Internet", PhD thesis, Massachusetts Institute of 1562 Technology , 2014. 1564 Authors' Addresses 1566 Nicolas Kuhn (editor) 1567 Telecom Bretagne 1568 2 rue de la Chataigneraie 1569 Cesson-Sevigne 35510 1570 France 1572 Phone: +33 2 99 12 70 46 1573 Email: nicolas.kuhn@telecom-bretagne.eu 1575 Preethi Natarajan (editor) 1576 Cisco Systems 1577 510 McCarthy Blvd 1578 Milpitas, California 1579 United States 1581 Email: prenatar@cisco.com 1583 Naeem Khademi (editor) 1584 University of Oslo 1585 Department of Informatics, PO Box 1080 Blindern 1586 N-0316 Oslo 1587 Norway 1589 Phone: +47 2285 24 93 1590 Email: naeemk@ifi.uio.no 1592 David Ros 1593 Simula Research Laboratory AS 1594 P.O. Box 134 1595 Lysaker, 1325 1596 Norway 1598 Phone: +33 299 25 21 21 1599 Email: dros@simula.no