idnits 2.17.1 draft-ietf-aqm-eval-guidelines-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 29, 2015) is 3223 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'B' is mentioned on line 302, but not defined == Missing Reference: 'Mbps' is mentioned on line 302, but not defined -- Obsolete informational reference (is this intentional?): RFC 793 (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) -- Obsolete informational reference (is this intentional?): RFC 2679 (Obsoleted by RFC 7679) -- Obsolete informational reference (is this intentional?): RFC 2680 (Obsoleted by RFC 7680) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force N. Kuhn, Ed. 3 Internet-Draft Telecom Bretagne 4 Intended status: Informational P. Natarajan, Ed. 5 Expires: December 31, 2015 Cisco Systems 6 N. Khademi, Ed. 7 University of Oslo 8 D. Ros 9 Simula Research Laboratory AS 10 June 29, 2015 12 AQM Characterization Guidelines 13 draft-ietf-aqm-eval-guidelines-05 15 Abstract 17 Unmanaged large buffers in today's networks have given rise to a slew 18 of performance issues. These performance issues can be addressed by 19 some form of Active Queue Management (AQM) mechanism, optionally in 20 combination with a packet scheduling scheme such as fair queuing. 21 The IETF Active Queue Management and Packet Scheduling working group 22 was formed to standardize AQM schemes that are robust, easily 23 implementable, and successfully deployable in today's networks. This 24 document describes various criteria for performing precautionary 25 characterizations of AQM proposals. This document also helps in 26 ascertaining whether any given AQM proposal should be taken up for 27 standardization by the AQM WG. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at http://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on December 31, 2015. 46 Copyright Notice 48 Copyright (c) 2015 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (http://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with respect 56 to this document. Code Components extracted from this document must 57 include Simplified BSD License text as described in Section 4.e of 58 the Trust Legal Provisions and are provided without warranty as 59 described in the Simplified BSD License. 61 Table of Contents 63 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 64 1.1. Reducing the latency and maximizing the goodput . . . . . 5 65 1.2. Guidelines for AQM evaluation . . . . . . . . . . . . . . 5 66 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 6 67 1.4. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 6 68 2. End-to-end metrics . . . . . . . . . . . . . . . . . . . . . 6 69 2.1. Flow completion time . . . . . . . . . . . . . . . . . . 7 70 2.2. Flow start up time . . . . . . . . . . . . . . . . . . . 7 71 2.3. Packet loss . . . . . . . . . . . . . . . . . . . . . . . 7 72 2.4. Packet loss synchronization . . . . . . . . . . . . . . . 8 73 2.5. Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 8 74 2.6. Latency and jitter . . . . . . . . . . . . . . . . . . . 9 75 2.7. Discussion on the trade-off between latency and goodput . 9 76 3. Generic set up for evaluations . . . . . . . . . . . . . . . 10 77 3.1. Topology and notations . . . . . . . . . . . . . . . . . 10 78 3.2. Buffer size . . . . . . . . . . . . . . . . . . . . . . . 12 79 3.3. Congestion controls . . . . . . . . . . . . . . . . . . . 12 80 4. Transport Protocols . . . . . . . . . . . . . . . . . . . . . 13 81 4.1. TCP-friendly sender . . . . . . . . . . . . . . . . . . . 13 82 4.1.1. TCP-friendly sender with the same initial congestion 83 window . . . . . . . . . . . . . . . . . . . . . . . 14 84 4.1.2. TCP-friendly sender with different initial congestion 85 windows . . . . . . . . . . . . . . . . . . . . . . . 14 86 4.2. Aggressive transport sender . . . . . . . . . . . . . . . 14 87 4.3. Unresponsive transport sender . . . . . . . . . . . . . . 15 88 4.4. Less-than Best Effort transport sender . . . . . . . . . 15 89 5. Round Trip Time Fairness . . . . . . . . . . . . . . . . . . 16 90 5.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 16 91 5.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 16 92 5.3. Metrics to evaluate the RTT fairness . . . . . . . . . . 17 93 6. Burst Absorption . . . . . . . . . . . . . . . . . . . . . . 17 94 6.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 17 95 6.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 18 96 7. Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 19 97 7.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 19 98 7.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 19 99 7.2.1. Definition of the congestion Level . . . . . . . . . 19 100 7.2.2. Mild congestion . . . . . . . . . . . . . . . . . . . 20 101 7.2.3. Medium congestion . . . . . . . . . . . . . . . . . . 20 102 7.2.4. Heavy congestion . . . . . . . . . . . . . . . . . . 20 103 7.2.5. Varying the congestion level . . . . . . . . . . . . 20 104 7.2.6. Varying available capacity . . . . . . . . . . . . . 21 105 7.3. Parameter sensitivity and stability analysis . . . . . . 22 106 8. Various Traffic Profiles . . . . . . . . . . . . . . . . . . 22 107 8.1. Traffic mix . . . . . . . . . . . . . . . . . . . . . . . 22 108 8.2. Bi-directional traffic . . . . . . . . . . . . . . . . . 23 109 9. Multi-AQM Scenario . . . . . . . . . . . . . . . . . . . . . 23 110 9.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 23 111 9.2. Details on the evaluation scenario . . . . . . . . . . . 24 112 10. Implementation cost . . . . . . . . . . . . . . . . . . . . . 24 113 10.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 24 114 10.2. Recommended discussion . . . . . . . . . . . . . . . . . 25 115 11. Operator Control and Auto-tuning . . . . . . . . . . . . . . 25 116 11.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 25 117 11.2. Required discussion . . . . . . . . . . . . . . . . . . 26 118 12. Interaction with ECN . . . . . . . . . . . . . . . . . . . . 26 119 12.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 26 120 12.2. Recommended discussion . . . . . . . . . . . . . . . . . 26 121 13. Interaction with Scheduling . . . . . . . . . . . . . . . . . 26 122 13.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 27 123 13.2. Recommended discussion . . . . . . . . . . . . . . . . . 27 124 13.3. Assessing the interaction between AQM and scheduling . . 27 125 14. Discussion on Methodology, Metrics, AQM Comparisons and 126 Packet Sizes . . . . . . . . . . . . . . . . . . . . . . . . 27 127 14.1. Methodology . . . . . . . . . . . . . . . . . . . . . . 27 128 14.2. Comments on metrics measurement . . . . . . . . . . . . 28 129 14.3. Comparing AQM schemes . . . . . . . . . . . . . . . . . 28 130 14.3.1. Performance comparison . . . . . . . . . . . . . . . 28 131 14.3.2. Deployment comparison . . . . . . . . . . . . . . . 29 132 14.4. Packet sizes and congestion notification . . . . . . . . 29 133 15. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 30 134 16. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 31 135 17. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 31 136 18. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 137 19. Security Considerations . . . . . . . . . . . . . . . . . . . 32 138 20. References . . . . . . . . . . . . . . . . . . . . . . . . . 32 139 20.1. Normative References . . . . . . . . . . . . . . . . . . 32 140 20.2. Informative References . . . . . . . . . . . . . . . . . 32 141 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 34 143 1. Introduction 145 Active Queue Management (AQM) [I-D.ietf-aqm-recommendation] addresses 146 the concerns arising from using unnecessarily large and unmanaged 147 buffers to improve network and application performance. Several AQM 148 algorithms have been proposed in the past years, most notably Random 149 Early Detection (RED), BLUE, and Proportional Integral controller 150 (PI), and more recently CoDel [CODEL] and PIE [PIE]. In general, 151 these algorithms actively interact with the Transmission Control 152 Protocol (TCP) and any other transport protocol that deploys a 153 congestion control scheme to manage the amount of data they keep in 154 the network. The available buffer space in the routers and switches 155 should be large enough to accommodate the short-term buffering 156 requirements. AQM schemes aim at reducing mean buffer occupancy, and 157 therefore both end-to-end delay and jitter. Some of these 158 algorithms, notably RED, have also been widely implemented in some 159 network devices. However, the potential benefits of the RED scheme 160 have not been realized since RED is reported to be usually turned 161 off. The main reason of this reluctance to use RED in today's 162 deployments comes from its sensitivity to the operating conditions in 163 the network and the difficulty of tuning its parameters. 165 A buffer is a physical volume of memory in which a queue or set of 166 queues are stored. In real implementations of switches, a global 167 memory is shared between the available devices: the size of the 168 buffer for a given communication does not make sense, as its 169 dedicated memory may vary over the time and real-world buffering 170 architectures are complex. For the sake of simplicity, when speaking 171 of a specific queue in this document, "buffer size" refers to the 172 maximum amount of data the buffer may store, which can be measured in 173 bytes or packets. The rest of this memo therefore refers to the 174 maximum queue depth as the size of the buffer for a given 175 communication. 177 Bufferbloat [BB2011] is the consequence of deploying large unmanaged 178 buffers on the Internet, which has lead to an increase in end-to-end 179 delay: the buffering has often been measured to be ten times or 180 hundred times larger than needed. This results in poor performance 181 for latency-sensitive applications such as real-time multimedia 182 (e.g., voice, video, gaming, etc). The degree to which this affects 183 modern networking equipment, especially consumer-grade equipment's, 184 produces problems even with commonly used web services. Active queue 185 management is thus essential to control queuing delay and decrease 186 network latency. 188 The Active Queue Management and Packet Scheduling Working Group (AQM 189 WG) was recently formed within the TSV area to address the problems 190 with large unmanaged buffers in the Internet. Specifically, the AQM 191 WG is tasked with standardizing AQM schemes that not only address 192 concerns with such buffers, but also are robust under a wide variety 193 of operating conditions. 195 In order to ascertain whether the WG should undertake standardizing 196 an AQM proposal, the WG requires guidelines for assessing AQM 197 proposals. This document provides the necessary characterization 198 guidelines. There may be a debate on whether a scheduling scheme is 199 additional to an AQM algorithm or is a part of an AQM algorithm. The 200 rest of this memo refers to AQM as a dropping/marking policy that 201 does not feature a scheduling scheme. This document may be 202 complemented with another one on guidelines for assessing combination 203 of packet scheduling and AQM. We note that such a document will 204 inherit all the guidelines from this document plus any additional 205 scenarios relevant for packet scheduling such as flow starvation 206 evaluation or impact of the number of hash buckets. 208 1.1. Reducing the latency and maximizing the goodput 210 The trade-off between reducing the latency and maximizing the goodput 211 is intrinsically linked to each AQM scheme and is key to evaluating 212 its performance. This trade-off MUST be considered in various 213 scenarios to ensure the safety of an AQM deployment. Whenever 214 possible, solutions ought to aim at both maximizing goodput and 215 minimizing latency. This document provides guidelines that enable 216 the reader to quantify (1) reduction of latency, (2) maximization of 217 goodput and (3) the trade-off between the two. 219 These guidelines provide the tools to understand the deployment costs 220 versus the potential gain in performance from the introduction of the 221 proposed scheme. 223 1.2. Guidelines for AQM evaluation 225 The guidelines help to quantify performance of AQM schemes in terms 226 of latency reduction, goodput maximization and the trade-off between 227 these two. The guidelines also help to discuss safe deployment of 228 AQM, including self-adaptation, stability analysis, fairness, design 229 and implementation complexity and robustness to different operating 230 conditions. 232 This memo details generic characterization scenarios against which 233 any AQM proposal must be evaluated, irrespective of whether or not an 234 AQM is standardized by the IETF. This documents recommends the 235 relevant scenarios and metrics to be considered. 237 The document presents central aspects of an AQM algorithm that must 238 be considered whatever the context, such as burst absorption 239 capacity, RTT fairness or resilience to fluctuating network 240 conditions. These guidelines do not cover every possible aspect of a 241 particular algorithm. In addition, it is worth noting that the 242 proposed criteria are not bound to a particular evaluation toolset. 244 This document details how an AQM designer can rate the feasibility of 245 their proposal in different types of network devices (switches, 246 routers, firewalls, hosts, drivers, etc) where an AQM may be 247 implemented. However, these guidelines do not present context- 248 dependent scenarios (such as 802.11 WLANs, data-centers or rural 249 broadband networks). 251 1.3. Requirements Language 253 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 254 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 255 document are to be interpreted as described in RFC 2119 [RFC2119]. 257 1.4. Glossary 259 o AQM: there may be a debate on whether a scheduling scheme is 260 additional to an AQM algorithm or is a part of an AQM algorithm. 261 The rest of this memo refers to AQM as a dropping/marking policy 262 that does not feature a scheduling scheme. 264 o buffer: a physical volume of memory in which a queue or set of 265 queues are stored. 267 o buffer size: the maximum amount of data that may be stored in a 268 buffer, measured in bytes or packets. 270 2. End-to-end metrics 272 End-to-end delay is the result of propagation delay, serialization 273 delay, service delay in a switch, medium-access delay and queuing 274 delay, summed over the network elements along the path. AQM schemes 275 may reduce the queuing delay by providing signals to the sender on 276 the emergence of congestion, but any impact on the goodput must be 277 carefully considered. This section presents the metrics that could 278 be used to better quantify (1) the reduction of latency, (2) 279 maximization of goodput and (3) the trade-off between these two. 280 This section provides normative requirements for metrics that can be 281 used to assess the performance of an AQM scheme. 283 Some metrics listed in this section are not suited to every type of 284 traffic detailed in the rest of this document. It is therefore not 285 necessary to measure all of the following metrics: the chosen metric 286 may not be relevant to the context of the evaluation scenario (e.g. 288 latency vs. goodput trade-off in application-limited traffic 289 scenarios). Guidance is provided for each metric. 291 2.1. Flow completion time 293 The flow completion time is an important performance metric for the 294 end-user when the flow size is finite. Considering the fact that an 295 AQM scheme may drop/mark packets, the flow completion time is 296 directly linked to the dropping/marking policy of the AQM scheme. 297 This metric helps to better assess the performance of an AQM 298 depending on the flow size. The Flow Completion Time (FCT) is 299 related to the flow size (Fs) and the goodput for the flow (G) as 300 follows: 302 FCT [s] = Fs [B] / ( G [Mbps] / 8 ) 304 If this metric is used to evaluate the performance of web transfers, 305 we propose to rather consider the time needed to download all the 306 objects that compose the web page, as this makes more sense in terms 307 of user experience than assessing the time needed to download each 308 object. 310 2.2. Flow start up time 312 The flow start up time is the time between the request has been sent 313 from the client and the server starts to transmit data. The amount 314 of packets dropped by an AQM may seriously affect the waiting period 315 during which the data transfer has not started. This metric would 316 specifically focus on the operations such as DNS lookups, TCP opens 317 of SSL handshakes. 319 2.3. Packet loss 321 Packet loss can occur within a network device, this can impact the 322 end-to-end performance measured at receiver. 324 The tester SHOULD evaluate loss experienced at the receiver using one 325 of the two metrics: 327 o the packet loss probability: this metric is to be frequently 328 measured during the experiment. The long-term loss probability is 329 of interest for steady-state scenarios only; 331 o the interval between consecutive losses: the time between two 332 losses is to be measured. 334 The packet loss probability can be assessed by simply evaluating the 335 loss ratio as a function of the number of lost packets and the total 336 number of packets sent. This might not be easily done in laboratory 337 testing, for which these guidelines advice the tester: 339 o to check that for every packet, a corresponding packet was 340 received within a reasonable time, as explained in [RFC2680]. 342 o to keep a count of all packets sent, and a count of the non- 343 duplicate packets received, as explained in the section 10 of 344 [RFC2544]. 346 The interval between consecutive losses, which is also called a gap, 347 is a metric of interest for VoIP traffic and, as a result, has been 348 further specified in [RFC3611]. 350 2.4. Packet loss synchronization 352 One goal of an AQM algorithm ought be to help to avoid global 353 synchronization of flows sharing a bottleneck buffer on which the AQM 354 operates ([RFC2309],[I-D.ietf-aqm-recommendation]). The "degree" of 355 packet-loss synchronization between flows SHOULD be assessed, with 356 and without the AQM under consideration. 358 As discussed e.g. in [LOSS-SYNCH-MET-08], loss synchronization among 359 flows may be quantified by several slightly different metrics that 360 capture different aspects of the same issue. However, in real-world 361 measurements the choice of metric could be imposed by practical 362 considerations -- e.g. whether fine-grained information on packet 363 losses in the bottleneck available or not. For the purpose of AQM 364 characterization, a good candidate metric is the global 365 synchronization ratio, measuring the proportion of flows losing 366 packets during a loss event. [YU06] used this metric in real-world 367 experiments to characterize synchronization along arbitrary Internet 368 paths; the full methodology is described in [YU06]. 370 If an AQM scheme is evaluated using real-life network environments, 371 it is worth pointing out that some network events, such as failed 372 link restoration may cause synchronized losses between active flows 373 and thus confuse the meaning of this metric. 375 2.5. Goodput 377 The goodput has been defined in the section 3.17 of [RFC2647] as the 378 number of bits per unit of time forwarded to the correct destination 379 interface of the Device Under Test (DUT) or the System Under Test 380 (SUT), minus any bits lost or retransmitted. This definition induces 381 that the test setup needs to be qualified to assure that it is not 382 generating losses on its own. 384 Measuring the end-to-end goodput provides an appreciation of how well 385 an AQM scheme improves transport and application performance. The 386 measured end-to-end goodput is linked to the dropping/marking policy 387 of the AQM scheme -- e.g. the fewer the number of packet drops, the 388 fewer packets need retransmission, minimizing the impact of AQM on 389 transport and application performance. Additionally, an AQM scheme 390 may resort to Explicit Congestion Notification (ECN) marking as an 391 initial means to control delay. Again, marking packets instead of 392 dropping them reduces the number of packet retransmissions and 393 increases goodput. End-to-end goodput values help to evaluate the 394 AQM scheme's effectiveness of an AQM scheme in minimizing packet 395 drops that impact application performance and to estimate how well 396 the AQM scheme works with ECN. 398 The measurement of the goodput allows the tester evaluate to which 399 extent an AQM is able to maintain a high bottleneck utilization. 400 This metric should be also obtained frequently during an experiment 401 as the long-term goodput is relevant for steady-state scenarios only 402 and may not necessarily reflect how the introduction of an AQM 403 actually impacts the link utilization during at a certain period of 404 time. Fluctuations in the values obtained from these measurements 405 may depend on other factors than the introduction of an AQM, such as 406 link layer losses due to external noise or corruption, fluctuating 407 bandwidths (802.11 WLANs), heavy congestion levels or transport 408 layer's rate reduction by congestion control mechanism. 410 2.6. Latency and jitter 412 The latency, or the one-way delay metric, is discussed in [RFC2679]. 413 There is a consensus on a adequate metric for the jitter, that 414 represents the one-way delay variations for packets from the same 415 flow: the Packet Delay Variation (PDV), detailed in [RFC5481], serves 416 well all use cases. 418 The end-to-end latency differs from the queuing delay: it is linked 419 to the network topology and the path characteristics. Moreover, the 420 jitter also strongly depends on the traffic pattern and the topology. 421 The introduction of an AQM scheme would impact these metrics and 422 therefore they should be considered in the end-to-end evaluation of 423 performance. 425 2.7. Discussion on the trade-off between latency and goodput 427 The metrics presented in this section may be considered as explained 428 in the rest of this document, in order to discuss and quantify the 429 trade-off between latency and goodput. 431 This trade-off can also be illustrated with figures following the 432 recommendations of section 5 of [TCPEVAL2013]. Each of the end-to- 433 end delay and the goodput SHOULD be measured frequently for every 434 fixed time interval. 436 With regards to the goodput, and in addition to the long-term 437 stationary goodput value, it is RECOMMENDED to take measurements 438 every multiple of RTTs. We suggest a minimum value of 10 x RTT (to 439 smooth out the fluctuations) but higher values are encouraged 440 whenever appropriate for the presentation depending on the network's 441 path characteristics. The measurement period MUST be disclosed for 442 each experiment and when results/values are compared across different 443 AQM schemes, the comparisons SHOULD use exactly the same measurement 444 periods. 446 With regards to latency, it is highly RECOMMENDED to take the samples 447 on per-packet basis whenever possible depending on the features 448 provided by hardware/software and the impact of sampling itself on 449 the hardware performance. It is generally RECOMMENDED to provide at 450 least 10 samples per RTT. 452 From each of these sets of measurements, the 10th and 90th 453 percentiles and the median value SHOULD be computed. For each 454 scenario, a graph can be generated, with the x-axis showing the end- 455 to-end delay and the y-axis the goodput. This graph provides part of 456 a better understanding of (1) the delay/goodput trade-off for a given 457 congestion control mechanism, and (2) how the goodput and average 458 queue size vary as a function of the traffic load. 460 3. Generic set up for evaluations 462 This section presents the topology that can be used for each of the 463 following scenarios, the corresponding notations and discusses 464 various assumptions that have been made in the document. 466 3.1. Topology and notations 467 +---------+ +-----------+ 468 |senders A| |receivers B| 469 +---------+ +-----------+ 471 +--------------+ +--------------+ 472 |traffic class1| |traffic class1| 473 |--------------| |--------------| 474 | SEN.Flow1.1 +---------+ +-----------+ REC.Flow1.1 | 475 | + | | | | + | 476 | | | | | | | | 477 | + | | | | + | 478 | SEN.Flow1.X +-----+ | | +--------+ REC.Flow1.X | 479 +--------------+ | | | | +--------------+ 480 + +-+---+---+ +--+--+---+ + 481 | |Router L | |Router R | | 482 | |---------| |---------| | 483 | | AQM | | | | 484 | | BuffSize| | | | 485 | | (Bsize) +-----+ | | 486 | +-----+--++ ++-+------+ | 487 + | | | | + 488 +--------------+ | | | | +--------------+ 489 |traffic classN| | | | | |traffic classN| 490 |--------------| | | | | |--------------| 491 | SEN.FlowN.1 +---------+ | | +-----------+ REC.FlowN.1 | 492 | + | | | | + | 493 | | | | | | | | 494 | + | | | | + | 495 | SEN.FlowN.Y +------------+ +-------------+ REC.FlowN.Y | 496 +--------------+ +--------------+ 498 Figure 1: Topology and notations 500 Figure 1 is a generic topology where: 502 o various classes of traffic can be introduced; 504 o the timing of each flow could be different (i.e., when does each 505 flow start and stop); 507 o each class of traffic can comprise various number of flows; 509 o each link is characterized by a couple (RTT,capacity); 511 o flows are generated between A and B, sharing a bottleneck (Routers 512 L and R); 514 o the tester SHOULD consider both scenarios of asymmetric and 515 symmetric bottleneck links in terms of bandwidth. In case of 516 asymmetric link, the capacity from senders to receivers is higher 517 than the one from receivers to senders; the symmetric link 518 scenario provides a basic understanding of the operation of the 519 AQM mechanism whereas the asymmetric link scenario evaluates an 520 AQM mechanism in a more realistic setup; 522 o in asymmetric link scenarios, the tester SHOULD study the bi- 523 directional traffic between A and B (downlink and uplink) with the 524 AQM mechanism deployed on one direction only. The tester MAY 525 additionally consider a scenario with AQM mechanism being deployed 526 on both directions. In each scenario, the tester SHOULD 527 investigate the impact of drop policy of the AQM on TCP ACK 528 packets and its impact on the performance. 530 Although this topology may not perfectly reflect actual topologies, 531 the simple topology is commonly used in the world of simulations and 532 small testbeds. It can be considered as adequate to evaluate AQM 533 proposals, similarly to the topology proposed in [TCPEVAL2013]. 534 Testers ought to pay attention to the topology that has been used to 535 evaluate an AQM scheme when comparing this scheme with a new proposed 536 AQM scheme. 538 3.2. Buffer size 540 The size of the buffers should be carefully chosen, and is to be set 541 to the bandwidth-delay product; the bandwidth being the bottleneck 542 capacity and the delay the larger RTT in the considered network. The 543 size of the buffer can impact on the AQM performance and is a 544 dimensioning parameter that will be considered when comparing AQM 545 proposals. 547 If the context or the application requires a specific buffer size, 548 the tester MUST justify and detail the way the maximum queue size is 549 set. Indeed, the maximum size of the buffer may affect the AQM's 550 performance and its choice SHOULD be elaborated for a fair comparison 551 between AQM proposals. While comparing AQM schemes the buffer size 552 SHOULD remain the same across the tests. 554 3.3. Congestion controls 556 This memo features three kind of congestion controls: 558 o Standard TCP congestion control: the base-line congestion control 559 is TCP NewReno with SACK, as explained in [RFC5681]. 561 o Aggressive congestion controls: a base-line congestion control for 562 this category is TCP Cubic. 564 o Less-than Best Effort (LBE) congestion controls: an LBE congestion 565 control 'results in smaller bandwidth and/or delay impact on 566 standard TCP than standard TCP itself, when sharing a bottleneck 567 with it.' [RFC6297] 569 Other transport congestion controls can OPTIONALLY be evaluated in 570 addition. Recent transport layer protocols are not mentioned in the 571 following sections, for the sake of simplicity. 573 4. Transport Protocols 575 Network and end-devices need to be configured with a reasonable 576 amount of buffer space to absorb transient bursts. In some 577 situations, network providers tend to configure devices with large 578 buffers to avoid packet drops triggered by a full buffer and to 579 maximize the link utilization for standard loss-based TCP traffic. 581 AQM algorithms are often evaluated by considering Transmission 582 Control Protocol (TCP) [RFC0793] with a limited number of 583 applications. TCP is a widely deployed transport. It fills up 584 unmanaged buffers until the TCP sender receives a signal (packet 585 drop) that reduces the sending rate. The larger the buffer, the 586 higher the buffer occupancy, and therefore the queuing delay. An 587 efficient AQM scheme sends out early congestion signals to TCP to 588 bring the queuing delay under control. 590 Not all applications using TCP use the same flavor of TCP. Variety 591 of senders generate different classes of traffic which may not react 592 to congestion signals (aka non-responsive flows 593 [I-D.ietf-aqm-recommendation]) or may not reduce their sending rate 594 as expected (aka Transport Flows that are less responsive than 595 TCP[I-D.ietf-aqm-recommendation], also called "aggressive flows"). 596 In these cases, AQM schemes seek to control the queuing delay. 598 This section provides guidelines to assess the performance of an AQM 599 proposal for various traffic profiles -- different types of senders 600 (with different TCP congestion control variants, unresponsive, 601 aggressive). 603 4.1. TCP-friendly sender 604 4.1.1. TCP-friendly sender with the same initial congestion window 606 This scenario helps to evaluate how an AQM scheme reacts to a TCP- 607 friendly transport sender. A single long-lived, non application- 608 limited, TCP NewReno flow, with an Initial congestion Window (IW) set 609 to 3 packets, transfers data between sender A and receiver B. Other 610 TCP friendly congestion control schemes such as TCP-friendly rate 611 control [RFC5348] etc MAY also be considered. 613 For each TCP-friendly transport considered, the graph described in 614 Section 2.7 could be generated. 616 4.1.2. TCP-friendly sender with different initial congestion windows 618 This scenario can be used to evaluate how an AQM scheme adapts to a 619 traffic mix consisting of TCP flows with different values of the IW. 621 For this scenario, two types of flows MUST be generated between 622 sender A and receiver B: 624 o A single long-lived non application-limited TCP NewReno flow; 626 o A single long-lived application-limited TCP NewReno flow, with an 627 IW set to 3 or 10 packets. The size of the data transferred must 628 be strictly higher than 10 packets and should be lower than 100 629 packets. 631 The transmission of the non application-limited flow must start 632 before the transmission of the application-limited flow and only 633 after the steady state has been reached by non application-limited 634 flow. 636 For each of these scenarios, the graph described in Section 2.7 could 637 be generated for each class of traffic (application-limited and non 638 application-limited). The completion time of the application-limited 639 TCP flow could be measured. 641 4.2. Aggressive transport sender 643 This scenario helps testers to evaluate how an AQM scheme reacts to a 644 transport sender that is more aggressive than a single TCP-friendly 645 sender. We define 'aggressiveness' as a higher increase factor than 646 standard upon a successful transmission and/or a lower than standard 647 decrease factor upon a unsuccessful transmission (e.g. in case of 648 congestion controls with Additive-Increase Multiplicative-Decrease 649 (AIMD) principle, a larger AI and/or MD factors). A single long- 650 lived, non application-limited, TCP Cubic flow transfers data between 651 sender A and receiver B. Other aggressive congestion control schemes 652 MAY also be considered. 654 For each flavor of aggressive transports, the graph described in 655 Section 2.7 could be generated. 657 4.3. Unresponsive transport sender 659 This scenario helps testers to evaluate how an AQM scheme reacts to a 660 transport sender that is less responsive than TCP. Note that faulty 661 transport implementations on an end host and/or faulty network 662 elements en-route that "hide" congestion signals in packet headers 663 [I-D.ietf-aqm-recommendation] may also lead to a similar situation, 664 such that the AQM scheme needs to adapt to unresponsive traffic. To 665 this end, these guidelines propose the two following scenarios. 667 The first scenario can be used to evaluate queue build up. It 668 considers unresponsive flow(s) whose sending rate is greater than the 669 bottleneck link capacity between routers L and R. This scenario 670 consists of a long-lived non application limited UDP flow transmits 671 data between sender A and receiver B. Graphs described in 672 Section 2.7 could be generated. 674 The second scenario can be used to evaluate if the AQM scheme is able 675 to keep responsive fraction under control. This scenario considers a 676 mixture of TCP-friendly and unresponsive traffics. It consists of a 677 long-lived non application-limited UDP flow and a single long-lived, 678 non-application-limited, TCP New Reno flow that transmit data between 679 sender A and receiver B. As opposed to the first scenario, the rate 680 of the UDP traffic should not be greater than the bottleneck 681 capacity, and should not be higher than half of the bottleneck 682 capacity. For each type of traffic, the graph described in 683 Section 2.7 could be generated. 685 4.4. Less-than Best Effort transport sender 687 This scenario helps to evaluate how an AQM scheme reacts to LBE 688 congestion controls that 'results in smaller bandwidth and/or delay 689 impact on standard TCP than standard TCP itself, when sharing a 690 bottleneck with it.' [RFC6297]. The potential fateful interaction 691 when AQM and LBE techniques are combined has been shown in [LBE-AQM]; 692 this scenario helps to evaluate whether the coexistence of the 693 proposed AQM and LBE techniques may be possible. 695 Single long-lived non application-limited TCP NewReno flows transfer 696 data between sender A and receiver B. Other TCP-friendly congestion 697 control schemes MAY also be considered. Single long-lived non 698 application-limited LEDBAT [RFC6817] flows transfer data between 699 sender A and receiver B. We recommend to set the target delay and 700 gain values of LEDBAT respectively to 5 ms and 10 [LEDBAT-PARAM]. 701 Other LBE congestion control schemes, any of those listed in 702 [RFC6297], MAY also be considered. 704 For each of the TCP-friendly and LBE transports, the graph described 705 in Section 2.7 could be generated. 707 5. Round Trip Time Fairness 709 5.1. Motivation 711 The ability of AQM schemes to control the queuing delay highly 712 depends on the way end-to-end protocols react to congestion signals. 713 When the RTT varies, the behaviour of a congestion control is 714 impacted and this impacts the ability of an AQM scheme to control the 715 queue. It is therefore important to assess the AQM schemes for a set 716 of RTTs (e.g., from 5 ms to 200 ms). 718 The asymmetry in terms of difference in intrinsic RTT between various 719 paths sharing the same bottleneck SHOULD be considered so that the 720 fairness between the flows can be discussed since in this scenario, a 721 flow traversing on shorter RTT path may react faster to congestion 722 and recover faster from it compared to another flow on a longer RTT 723 path. The introduction of AQM schemes may potentially improve this 724 type of fairness. 726 Introducing an AQM scheme may cause the unfairness between the flows, 727 even if the RTTs are identical. This potential unfairness SHOULD be 728 investigated as well. 730 5.2. Recommended tests 732 The RECOMMENDED topology is detailed in Figure 1: 734 o To evaluate the inter-RTT fairness, for each run, two flows 735 divided into two categories. Category I which RTT between sender 736 A and Router L SHOULD be 100ms. Category II which RTT between 737 sender A and Router L should be in [5ms;560ms]. The maximum value 738 for the RTT represents the RTT of a satellite link that, according 739 to the section 2 of [RFC2488] should be at least 558ms. 741 o To evaluate the impact of the RTT value on the AQM performance and 742 the intra-protocol fairness (the fairness for the flows using the 743 same paths/congestion control), for each run, two flows (Flow1 and 744 Flow2) should be introduced. For each experiment, the set of RTT 745 SHOULD be the same for the two flows and in [5ms;560ms]. 747 A set of evaluated flows MUST use the same congestion control 748 algorithm. 750 5.3. Metrics to evaluate the RTT fairness 752 The outputs that MUST be measured are: 754 o for the inter-RTT fairness: (1) the cumulative average goodput of 755 the flow from Category I, goodput_Cat_I (Section 2.5); (2) the 756 cumulative average goodput of the flow from Category II, 757 goodput_Cat_II (Section 2.5); (3) the ratio goodput_Cat_II/ 758 goodput_Cat_I; (4) the average packet drop rate for each category 759 (Section 2.3). 761 o for the intra-protocol RTT fairness: (1) the cumulative average 762 goodput of the two flows (Section 2.5); (2) the average packet 763 drop rate for the two flows (Section 2.3). 765 6. Burst Absorption 767 "AQM mechanisms need to control the overall queue sizes, to ensure 768 that arriving bursts can be accommodated without dropping packets" 769 [I-D.ietf-aqm-recommendation] 771 6.1. Motivation 773 An AQM scheme can result in bursts of packet arrivals due to various 774 reasons. Dropping one or more packets from a burst can result in 775 performance penalties for the corresponding flows, since dropped 776 packets have to be retransmitted. Performance penalties can result 777 in failing to meet SLAs and be a disincentive to AQM adoption. 779 The ability to accommodate bursts translates to larger queue length 780 and hence more queuing delay. On the one hand, it is important that 781 an AQM scheme quickly brings bursty traffic under control. On the 782 other hand, a peak in the packet drop rates to bring a packet burst 783 quickly under control could result in multiple drops per flow and 784 severely impact transport and application performance. Therefore, an 785 AQM scheme ought to bring bursts under control by balancing both 786 aspects -- (1) queuing delay spikes are minimized and (2) performance 787 penalties for ongoing flows in terms of packet drops are minimized. 789 An AQM scheme that maintains short queues allows some remaining space 790 in the queue for bursts of arriving packets. The tolerance to bursts 791 of packets depends upon the number of packets in the queue, which is 792 directly linked to the AQM algorithm. Moreover, one AQM scheme may 793 implement a feature controlling the maximum size of accepted bursts, 794 that can depend on the buffer occupancy or the currently estimated 795 queuing delay. The impact of the buffer size on the burst allowance 796 may be evaluated. 798 6.2. Recommended tests 800 For this scenario, tester MUST evaluate how the AQM performs with the 801 following traffic generated from sender A to receiver B: 803 o Web traffic with IW10; 805 o Bursty video frames; 807 o Constant bit rate UDP traffic. 809 o A single bulk TCP flow as background traffic. 811 Figure 2 presents the various cases for the traffic that MUST be 812 generated between sender A and receiver B. 814 +-------------------------------------------------+ 815 |Case| Traffic Type | 816 | +-----+------------+----+--------------------+ 817 | |Video|Webs (IW 10)| CBR| Bulk TCP Traffic | 818 +----|-----|------------|----|--------------------| 819 |I | 0 | 1 | 1 | 0 | 820 +----|-----|------------|----|--------------------| 821 |II | 0 | 1 | 1 | 1 | 822 |----|-----|------------|----|--------------------| 823 |III | 1 | 1 | 1 | 0 | 824 +----|-----|------------|----|--------------------| 825 |IV | 1 | 1 | 1 | 1 | 826 +----+-----+------------+----+--------------------+ 828 Figure 2: Bursty traffic scenarios 830 A new web page download could start after the previous web page 831 download is finished. Each web page could be composed by at least 50 832 objects and the size of each object should be at least 1kB. 6 TCP 833 parallel connections SHOULD be generated to download the objects, 834 each parallel connections having an initial congestion window set to 835 10 packets. 837 For each of these scenarios, the graph described in Section 2.7 could 838 be generated. Metrics such as end-to-end latency, jitter, flow 839 completion time MAY be generated. For the cases of frame generation 840 of bursty video traffic as well as the choice of web traffic pattern, 841 we leave these details and their presentation to the testers. 843 7. Stability 845 7.1. Motivation 847 Network devices can experience varying operating conditions depending 848 on factors such as time of the day, deployment scenario, etc. For 849 example: 851 o Traffic and congestion levels are higher during peak hours than 852 off-peak hours. 854 o In the presence of a scheduler, the draining rate of a queue can 855 vary depending on the occupancy of other queues: a low load on a 856 high priority queue implies a higher draining rate for the lower 857 priority queues. 859 o The available capacity at the physical layer can vary over time 860 (e.g., a lossy channel, a link supporting traffic in a higher 861 diffserv class). 863 Whether the target context is a not stable environment, the ability 864 of an AQM scheme to maintain its control over the queuing delay and 865 buffer occupancy can be challenged. This document proposes 866 guidelines to assess the behavior of AQM schemes under varying 867 congestion levels and varying draining rates. 869 7.2. Recommended tests 871 Note that the traffic profiles explained below comprises non 872 application-limited TCP flows. For each of the below scenarios, the 873 results described in Section 2.7 SHOULD be generated. For 874 Section 7.2.5 and Section 7.2.6 they SHOULD incorporate the results 875 in per-phase basis as well. 877 Wherever the notion of time has explicitly mentioned in this 878 subsection, time 0 starts from the moment all TCP flows have already 879 reached their congestion avoidance phase. 881 7.2.1. Definition of the congestion Level 883 In these guidelines, the congestion levels are represented by the 884 projected packet drop rate, had a drop-tail queue was chosen instead 885 of an AQM scheme. When the bottleneck is shared among non- 886 application-limited TCP flows. l_r, the loss rate projection can be 887 expressed as a function of N, the number of bulk TCP flows, and S, 888 the sum of the bandwidth-delay product and the maximum buffer size, 889 both expressed in packets, based on Eq. 3 of [SCL-TCP]: 891 l_r = 0.76 * N^2 / S^2 893 N = S * sqrt(1/0.76) * sqrt (l_r) 895 These guidelines use the loss rate to define the different congestion 896 levels, but they do not stipulate that in other circumstances, 897 measuring the congestion level gives you an accurate estimation of 898 the loss rate or vice-versa. 900 7.2.2. Mild congestion 902 This scenario can be used to evaluate how an AQM scheme reacts to a 903 light load of incoming traffic resulting in mild congestion -- packet 904 drop rates around 0.1%. The number of bulk flows required to achieve 905 this congestion level, N_mild, is then: 907 N_mild = round(0.036*S) 909 7.2.3. Medium congestion 911 This scenario can be used to evaluate how an AQM scheme reacts to 912 incoming traffic resulting in medium congestion -- packet drop rates 913 around 0.5%. The number of bulk flows required to achieve this 914 congestion level, N_med, is then: 916 N_med = round (0.081*S) 918 7.2.4. Heavy congestion 920 This scenario can be used to evaluate how an AQM scheme reacts to 921 incoming traffic resulting in heavy congestion -- packet drop rates 922 around 1%. The number of bulk flows required to achieve this 923 congestion level, N_heavy, is then: 925 N_heavy = round (0.114*S) 927 7.2.5. Varying the congestion level 929 This scenario can be used to evaluate how an AQM scheme reacts to 930 incoming traffic resulting in various levels of congestion during the 931 experiment. In this scenario, the congestion level varies within a 932 large time-scale. The following phases may be considered: phase I - 933 mild congestion during 0-20s; phase II - medium congestion during 934 20-40s; phase III - heavy congestion during 40-60s; phase I again, 935 and so on. 937 7.2.6. Varying available capacity 939 This scenario can be used to help characterize how the AQM behaves 940 and adapts to bandwidth changes. The experiments are not meant to 941 reflect the exact conditions of Wi-Fi environments since its hard to 942 design repetitive experiments or accurate simulations for such 943 scenarios. 945 To emulate varying draining rates, the bottleneck capacity between 946 nodes 'Router L' and 'Router R' varies over the course of the 947 experiment as follows: 949 o Experiment 1: the capacity varies between two values within a 950 large time-scale. As an example, the following phases may be 951 considered: phase I - 100Mbps during 0-20s; phase II - 10Mbps 952 during 20-40s; phase I again, and so on. 954 o Experiment 2: the capacity varies between two values within a 955 short time-scale. As an example, the following phases may be 956 considered: phase I - 100Mbps during 0-100ms; phase II - 10Mbps 957 during 100-200ms; phase I again, and so on. 959 The tester MAY choose a phase time-interval value different than what 960 is stated above, if the network's path conditions (such as bandwidth- 961 delay product) necessitate. In this case the choice of such time- 962 interval value SHOULD be stated and elaborated. 964 The tester MAY additionally evaluate the two mentioned scenarios 965 (short-term and long-term capacity variations), during and/or 966 including TCP slow-start phase. 968 More realistic fluctuating capacity patterns MAY be considered. The 969 tester MAY choose to incorporate realistic scenarios with regards to 970 common fluctuation of bandwidth in state-of-the-art technologies. 972 The scenario MAY consist of TCP NewReno flows between sender A and 973 receiver B. To better assess the impact of draining rates on the AQM 974 behavior, the tester MUST compare its performance with those of drop- 975 tail and SHOULD provide a reference document for their proposal 976 discussing performance and deployment compared to those of drop-tail. 977 Burst traffic, such as presented in Section 6.2, could also be 978 considered to assess the impact of varying available capacity on the 979 burst absorption of the AQM. 981 7.3. Parameter sensitivity and stability analysis 983 The control law used by an AQM is the primary means by which the 984 queuing delay is controlled. Hence understanding the control law is 985 critical to understanding the behavior of the AQM scheme. The 986 control law could include several input parameters whose values 987 affect the AQM scheme's output behavior and its stability. 988 Additionally, AQM schemes may auto-tune parameter values in order to 989 maintain stability under different network conditions (such as 990 different congestion levels, draining rates or network environments). 991 The stability of these auto-tuning techniques is also important to 992 understand. 994 Transports operating under the control of AQM experience the effect 995 of multiple control loops that react over different timescales. It 996 is therefore important that proposed AQM schemes are seen to be 997 stable when they are deployed at multiple points of potential 998 congestion along an Internet path. The pattern of congestion signals 999 (loss or ECN-marking) arising from AQM methods also need to not 1000 adversely interact with the dynamics of the transport protocols that 1001 they control. 1003 AQM proposals SHOULD provide background material showing control 1004 theoretic analysis of the AQM control law and the input parameter 1005 space within which the control law operates as expected; or could use 1006 another way to discuss the stability of the control law. For 1007 parameters that are auto-tuned, the material SHOULD include stability 1008 analysis of the auto-tuning mechanism(s) as well. Such analysis 1009 helps to understand an AQM control law better and the network 1010 conditions/deployments under which the AQM is stable. 1012 8. Various Traffic Profiles 1014 This section provides guidelines to assess the performance of an AQM 1015 proposal for various traffic profiles such as traffic with different 1016 applications or bi-directional traffic. 1018 8.1. Traffic mix 1020 This scenario can be used to evaluate how an AQM scheme reacts to a 1021 traffic mix consisting of different applications such as: 1023 o Bulk TCP transfer 1025 o Web traffic 1027 o VoIP 1028 o Constant Bit Rate (CBR) UDP traffic 1030 o Adaptive video streaming 1032 Various traffic mixes can be considered. These guidelines RECOMMEND 1033 to examine at least the following example: 1 bi-directional VoIP; 6 1034 Webs pages download (such as detailed in Section 6.2); 1 CBR; 1 1035 Adaptive Video; 5 bulk TCP. Any other combinations could be 1036 considered and should be carefully documented. 1038 For each scenario, the graph described in Section 2.7 could be 1039 generated for each class of traffic. Metrics such as end-to-end 1040 latency, jitter and flow completion time MAY be reported. 1042 8.2. Bi-directional traffic 1044 Control packets such as DNS requests/responses, TCP SYNs/ACKs are 1045 small, but their loss can severely impact the application 1046 performance. The scenario proposed in this section will help in 1047 assessing whether the introduction of an AQM scheme increases the 1048 loss probability of these important packets. 1050 For this scenario, traffic MUST be generated in both downlink and 1051 uplink, such as defined in Section 3.1. These guidelines RECOMMEND 1052 to consider a mild congestion level and the traffic presented in 1053 Section 7.2.2 in both directions. In this case, the metrics reported 1054 MUST be the same as in Section 7.2 for each direction. 1056 The traffic mix presented in Section 8.1 MAY also be generated in 1057 both directions. 1059 9. Multi-AQM Scenario 1061 9.1. Motivation 1063 Transports operating under the control of AQM experience the effect 1064 of multiple control loops that react over different timescales. It 1065 is therefore important that proposed AQM schemes are seen to be 1066 stable when they are deployed at multiple points of potential 1067 congestion along an Internet path. The pattern of congestion signals 1068 (loss or ECN-marking) arising from AQM methods also need to not 1069 adversely interact with the dynamics of the transport protocols that 1070 they control. 1072 9.2. Details on the evaluation scenario 1074 +---------+ +-----------+ 1075 |senders A|---+ +---|receivers A| 1076 +---------+ | | +-----------+ 1077 +-----+---+ +---------+ +--+-----+ 1078 |Router L |--|Router M |--|Router R| 1079 |AQM | |AQM | |No AQM | 1080 +---------+ +--+------+ +--+-----+ 1081 +---------+ | | +-----------+ 1082 |senders B|-------------+ +---|receivers B| 1083 +---------+ +-----------+ 1085 Figure 3: Topology for the Multi-AQM scenario 1087 This scenario can be used to evaluate how having AQM schemes in 1088 sequence impact the induced latency reduction, the induced goodput 1089 maximization and the trade-off between these two. The topology 1090 presented in Figure 3 could be used. We recommend that the AQM 1091 schemes introduced in Router L and Router M should be the same; any 1092 other configurations could be considered. For this scenario, we 1093 recommend to consider a mild congestion level, the number of flows 1094 specified in Section 7.2.2 being equally shared among senders A and 1095 B. Any other relevant combination of congestion levels could be 1096 considered. We recommend to measure the metrics presented in 1097 Section 7.2. 1099 10. Implementation cost 1101 10.1. Motivation 1103 Successful deployment of AQM is directly related to its cost of 1104 implementation. Network devices can need hardware or software 1105 implementations of the AQM mechanism. Depending on a device's 1106 capabilities and limitations, the device may or may not be able to 1107 implement some or all parts of the AQM logic. 1109 AQM proposals SHOULD provide pseudo-code for the complete AQM scheme, 1110 highlighting generic implementation-specific aspects of the scheme 1111 such as "drop-tail" vs. "drop-head", inputs (e.g. current queuing 1112 delay, queue length), computations involved, need for timers, etc. 1113 This helps to identify costs associated with implementing the AQM 1114 scheme on a particular hardware or software device. This also helps 1115 the WG understand which kind of devices can easily support the AQM 1116 and which cannot. 1118 10.2. Recommended discussion 1120 AQM proposals SHOULD highlight parts of AQM logic that are device 1121 dependent and discuss if and how AQM behavior could be impacted by 1122 the device. For example, a queueing-delay based AQM scheme requires 1123 current queuing delay as input from the device. If the device 1124 already maintains this value, then it can be trivial to implement the 1125 AQM logic on the device. If the device provides indirect means to 1126 estimate the queuing delay (for example: timestamps, dequeuing rate), 1127 then the AQM behavior is sensitive to the precision of the queuing 1128 delay estimations are for that device. Highlighting the sensitivity 1129 of an AQM scheme to queuing delay estimations helps implementers to 1130 identify appropriate means of implementing the mechanism on a device. 1132 11. Operator Control and Auto-tuning 1134 11.1. Motivation 1136 One of the biggest hurdles of RED deployment was/is its parameter 1137 sensitivity to operating conditions -- how difficult it is to tune 1138 RED parameters for a deployment to achieve acceptable benefit from 1139 using RED. Fluctuating congestion levels and network conditions add 1140 to the complexity. Incorrect parameter values lead to poor 1141 performance. 1143 Any AQM scheme is likely to have parameters whose values affect the 1144 control law and behaviour of an AQM. Exposing all these parameters 1145 as control parameters to a network operator (or user) can easily 1146 result in a unsafe AQM deployment. Unexpected AQM behavior ensues 1147 when parameter values are set improperly. A minimal number of 1148 control parameters minimizes the number of ways a possibly naive user 1149 can break a system where an AQM scheme is deployed at. Fewer control 1150 parameters make the AQM scheme more user-friendly and easier to 1151 deploy and debug. 1153 [I-D.ietf-aqm-recommendation] states "AQM algorithms SHOULD NOT 1154 require tuning of initial or configuration parameters in common use 1155 cases." A scheme ought to expose only those parameters that control 1156 the macroscopic AQM behavior such as queue delay threshold, queue 1157 length threshold, etc. 1159 Additionally, the safety of an AQM scheme is directly related to its 1160 stability under varying operating conditions such as varying traffic 1161 profiles and fluctuating network conditions, as described in 1162 Section 7. Operating conditions vary often and hence the AQM needs 1163 to remain stable under these conditions without the need for 1164 additional external tuning. If AQM parameters require tuning under 1165 these conditions, then the AQM must self-adapt necessary parameter 1166 values by employing auto-tuning techniques. 1168 11.2. Required discussion 1170 AQM proposals SHOULD describe the parameters that control the 1171 macroscopic AQM behavior, and identify any parameters that require 1172 require tuning to operational conditions. It could be interesting to 1173 also discuss that even if an AQM scheme may not adequately auto-tune 1174 its parameters, the resulting performance may not be optimal, but 1175 close to something reasonable. 1177 If there are any fixed parameters within the AQM, their setting 1178 SHOULD be discussed and justified. 1180 If an AQM scheme is evaluated with parameter(s) that were externally 1181 tuned for optimization or other purposes, these values MUST be 1182 disclosed. 1184 12. Interaction with ECN 1186 Deployed AQM algorithms SHOULD support Explicit Congestion 1187 Notification (ECN) as well as loss to signal congestion to 1188 endpoints"[I-D.ietf-aqm-recommendation]. The benefits of providing 1189 ECN support for an AQM scheme are described in [ECN-Benefit]. 1191 12.1. Motivation 1193 (ECN) [RFC3168] is an alternative that allows AQM schemes to signal 1194 receivers about network congestion that does not use packet drop. 1196 12.2. Recommended discussion 1198 An AQM scheme can support ECN [I-D.ietf-aqm-recommendation], in which 1199 case testers MUST discuss and describe the support of ECN. 1201 13. Interaction with Scheduling 1203 A network device may use per-flow or per-class queuing with a 1204 scheduling algorithm to either prioritize certain applications or 1205 classes of traffic, limit the rate of transmission, or to provide 1206 isolation between different traffic flows within a common class 1207 [I-D.ietf-aqm-recommendation]. 1209 13.1. Motivation 1211 Coupled with an AQM scheme, a router may schedule the transmission of 1212 packets in a specific manner by introducing a scheduling scheme. 1213 This algorithm may create sub-queues and integrate a dropping policy 1214 on each of these sub-queues. Another scheduling policy may modify 1215 the way packets are sequenced, modifying the timestamp of each 1216 packet. 1218 13.2. Recommended discussion 1220 The scheduling and the AQM conjointly impact on the end-to-end 1221 performance. During the characterization process of a dropping 1222 policy, the tester MUST discuss the feasibility to add scheduling 1223 combined with the AQM algorithm. This discussion as an instance, MAY 1224 explain whether the dropping policy is applied when packets are being 1225 enqueued or dequeued. 1227 13.3. Assessing the interaction between AQM and scheduling 1229 These guidelines do not propose guidelines to assess the performance 1230 of scheduling algorithms. Indeed, as opposed to characterizing AQM 1231 schemes that is related to their capacity to control the queuing 1232 delay in a queue, characterizing scheduling schemes is related to the 1233 scheduling itself and its interaction with the AQM scheme. As one 1234 example, the scheduler may create sub-queues and the AQM scheme may 1235 be applied on each of the sub-queues, and/or the AQM could be applied 1236 on the whole queue. Also, schedulers might, such as FQ-CoDel 1237 [FQ-CoDel] or FavorQueue [FAVOUR], introduce flow prioritization. In 1238 these cases, specific scenarios should be proposed to ascertain that 1239 these scheduler schemes not only helps in tackling the bufferbloat, 1240 but also are robust under a wide variety of operating conditions. 1241 This is out of the scope of this document that focus on dropping and/ 1242 or marking AQM schemes. 1244 14. Discussion on Methodology, Metrics, AQM Comparisons and Packet 1245 Sizes 1247 14.1. Methodology 1249 One key objective behind formulating the guidelines is to help 1250 ascertain whether a specific AQM is not only better than drop-tail 1251 but also safe to deploy. Testers therefore need to provide a 1252 reference document for their proposal discussing performance and 1253 deployment compared to those of drop-tail. 1255 A description of each test setup SHOULD be detailed to allow this 1256 test to be compared with other tests. This also allows others to 1257 replicate the tests if needed. This test setup SHOULD detail 1258 software and hardware versions. The tester could make its data 1259 available. 1261 The proposals SHOULD be evaluated on real-life systems, or they MAY 1262 be evaluated with event-driven simulations (such as ns-2, ns-3, 1263 OMNET, etc). The proposed scenarios are not bound to a particular 1264 evaluation toolset. 1266 The tester is encouraged to make the detailed test setup and the 1267 results publicly available. 1269 14.2. Comments on metrics measurement 1271 The document presents the end-to-end metrics that ought to be used to 1272 evaluate the trade-off between latency and goodput in Section 2. In 1273 addition to the end-to-end metrics, the queue-level metrics (normally 1274 collected at the device operating the AQM) provide a better 1275 understanding of the AQM behavior under study and the impact of its 1276 internal parameters. Whenever it is possible (e.g. depending on the 1277 features provided by the hardware/software), these guidelines advice 1278 to consider queue-level metrics, such as link utilization, queuing 1279 delay, queue size or packet drop/mark statistics in addition to the 1280 AQM-specific parameters. However, the evaluation MUST be primarily 1281 based on externally observed end-to-end metrics. 1283 These guidelines do not aim to detail on the way these metrics can be 1284 measured, since the way these metrics are measured is expected to 1285 depend on the evaluation toolset. 1287 14.3. Comparing AQM schemes 1289 This document recognizes that the guidelines mentioned above may be 1290 used for comparing AQM schemes. 1292 AQM schemes need to be compared against both performance and 1293 deployment categories. In addition, this section details how best to 1294 achieve a fair comparison of AQM schemes by avoiding certain 1295 pitfalls. 1297 14.3.1. Performance comparison 1299 AQM schemes MUST be compared against all the generic scenarios 1300 presented in this memo. AQM schemes MAY be compared for specific 1301 network environments such as data centers, home networks, etc. If an 1302 AQM scheme has parameter(s) that were externally tuned for 1303 optimization or other purposes, these values MUST be disclosed. 1305 AQM schemes belong to different varieties such as queue-length based 1306 schemes (ex. RED) or queueing-delay based scheme (ex. CoDel, PIE). 1307 AQM schemes expose different control knobs associated with different 1308 semantics. For example, while both PIE and CoDel are queueing-delay 1309 based schemes and each expose a knob to control the queueing delay -- 1310 PIE's "queueing delay reference" vs. CoDel's "queueing delay target", 1311 the two tuning parameters of the two schemes have different 1312 semantics, resulting in different control points. Such differences 1313 in AQM schemes can be easily overlooked while making comparisons. 1315 This document RECOMMENDS the following procedures for a fair 1316 performance comparison between the AQM schemes: 1318 1. comparable control parameters and comparable input values: 1319 carefully identify the set of parameters that control similar 1320 behavior between the two AQM schemes and ensure these parameters 1321 have comparable input values. For example, to compare how well a 1322 queue-length based AQM scheme controls queueing delay vs. a 1323 queueing-delay based AQM scheme, a tester can identify the 1324 parameters of the schemes that control queue delay and ensure 1325 that their input values are comparable. Similarly, to compare 1326 how well two AQM schemes accommodate packet bursts, the tester 1327 can identify burst-related control parameters and ensure they are 1328 configured with similar values. 1330 2. compare over a range of input configurations: there could be 1331 situations when the set of control parameters that affect a 1332 specific behavior have different semantics between the two AQM 1333 schemes. As mentioned above, PIE has tuning parameters to 1334 control queue delay that has a different semantics from those 1335 used in CoDel. In such situations, these schemes need to be 1336 compared over a range of input configurations. For example, 1337 compare PIE vs. CoDel over the range of target delay input 1338 configurations. 1340 14.3.2. Deployment comparison 1342 AQM schemes MUST be compared against deployment criteria such as the 1343 parameter sensitivity (Section 7.3), auto-tuning (Section 11) or 1344 implementation cost (Section 10). 1346 14.4. Packet sizes and congestion notification 1348 An AQM scheme may be considering packet sizes while generating 1349 congestion signals. [RFC7141] discusses the motivations behind this. 1350 For example, control packets such as DNS requests/responses, TCP 1351 SYNs/ACKs are small, but their loss can severely impact the 1352 application performance. An AQM scheme may therefore be biased 1353 towards small packets by dropping them with smaller probability 1354 compared to larger packets. However, such an AQM scheme is unfair to 1355 data senders generating larger packets. Data senders, malicious or 1356 otherwise, are motivated to take advantage of such AQM scheme by 1357 transmitting smaller packets, and could result in unsafe deployments 1358 and unhealthy transport and/or application designs. 1360 An AQM scheme SHOULD adhere to the recommendations outlined in 1361 [RFC7141], and SHOULD NOT provide undue advantage to flows with 1362 smaller packets [I-D.ietf-aqm-recommendation]. 1364 15. Conclusion 1366 Figure 4 lists the scenarios and their requirements. 1368 +------------------------------------------------------------------+ 1369 |Scenario |Sec. |Requirement | 1370 +------------------------------------------------------------------+ 1371 +------------------------------------------------------------------+ 1372 |Transport Protocols |4. | | 1373 | TCP-friendly sender | 4.1 |Scenario MUST be considered | 1374 | Aggressive sender | 4.2 |Scenario MUST be considered | 1375 | Unresponsive sender | 4.3 |Scenario MUST be considered | 1376 | LBE sender | 4.4 |Scenario MAY be considered | 1377 +------------------------------------------------------------------+ 1378 |Round Trip Time Fairness | 5.2 |Scenario MUST be considered | 1379 +------------------------------------------------------------------+ 1380 |Burst Absorption | 6.2 |Scenario MUST be considered | 1381 +------------------------------------------------------------------+ 1382 |Stability |7. | | 1383 | Varying congestion levels | 7.2.5|Scenario MUST be considered | 1384 | Varying available capacity| 7.2.6|Scenario MUST be considered | 1385 | Parameters and stability | 7.3 |This SHOULD be discussed | 1386 +------------------------------------------------------------------+ 1387 |Various Traffic Profiles |8. | | 1388 | Traffic mix | 8.1 |Scenario is RECOMMENDED | 1389 | Bi-directional traffic | 8.2 |Scenario MAY be considered | 1390 +------------------------------------------------------------------+ 1391 |Multi-AQM | 9.2 |Scenario MAY be considered | 1392 +------------------------------------------------------------------+ 1393 |Implementation Cost | 10.2 |Pseudo-code SHOULD be provided | 1394 +------------------------------------------------------------------+ 1395 |Operator Control | 11.2 |Tuning SHOULD NOT be required | 1396 +------------------------------------------------------------------+ 1397 |Interaction with ECN | 12.2 |MUST be discussed if supported | 1398 +------------------------------------------------------------------+ 1399 |Interaction with Scheduling| 13.2 |Feasibility MUST be discussed | 1400 +------------------------------------------------------------------+ 1402 Figure 4: Summary of the scenarios and their requirements 1404 16. Acknowledgements 1406 This work has been partially supported by the European Community 1407 under its Seventh Framework Programme through the Reducing Internet 1408 Transport Latency (RITE) project (ICT-317700). 1410 17. Contributors 1412 Many thanks to S. Akhtar, A.B. Bagayoko, F. Baker, D. Collier- 1413 Brown, G. Fairhurst, J. Gettys, T. Hoiland-Jorgensen, C. 1415 Kulatunga, W. Lautenschlager, A.C. Morton, R. Pan, D. Taht and M. 1416 Welzl for detailed and wise feedback on this document. 1418 18. IANA Considerations 1420 This memo includes no request to IANA. 1422 19. Security Considerations 1424 Some security considerations for AQM are identified in 1425 [I-D.ietf-aqm-recommendation].This document, by itself, presents no 1426 new privacy nor security issues. 1428 20. References 1430 20.1. Normative References 1432 [I-D.ietf-aqm-recommendation] 1433 Baker, F. and G. Fairhurst, "IETF Recommendations 1434 Regarding Active Queue Management", draft-ietf-aqm- 1435 recommendation-11 (work in progress), February 2015. 1437 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1438 Requirement Levels", RFC 2119, 1997. 1440 [RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion 1441 Notification", RFC 7141, 2014. 1443 20.2. Informative References 1445 [BB2011] "BufferBloat: what's wrong with the internet?", ACM Queue 1446 vol. 9, 2011. 1448 [CODEL] Nichols, K. and V. Jacobson, "Controlling Queue Delay", 1449 ACM Queue , 2012. 1451 [ECN-Benefit] 1452 Welzl, M. and G. Fairhurst, "The Benefits to Applications 1453 of using Explicit Congestion Notification (ECN)", IETF 1454 (Work-in-Progress) , February 2014. 1456 [FAVOUR] Anelli, P., Diana, R., and E. Lochin, "FavorQueue: a 1457 Parameterless Active Queue Management to Improve TCP 1458 Traffic Performance", Computer Networks vol. 60, 2014. 1460 [FQ-CoDel] 1461 Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, 1462 J., and E. Dumazet, "FlowQueue-Codel", IETF (Work-in- 1463 Progress) , January 2015. 1465 [LBE-AQM] Gong, Y., Rossi, D., Testa, C., Valenti, S., and D. Taht, 1466 "Fighting the bufferbloat: on the coexistence of AQM and 1467 low priority congestion control", Computer Networks, 1468 Elsevier, 2014, 60, pp.115 - 128 , 2014. 1470 [LEDBAT-PARAM] 1471 Trang, S., Kuhn, N., Lochin, E., Baudoin, C., Dubois, E., 1472 and P. Gelard, "On The Existence Of Optimal LEDBAT 1473 Parameters", IEEE ICC 2014 - Communication QoS, 1474 Reliability and Modeling Symposium , 2014. 1476 [LOSS-SYNCH-MET-08] 1477 Hassayoun, S. and D. Ros, "Loss Synchronization and Router 1478 Buffer Sizing with High-Speed Versions of TCP", IEEE 1479 INFOCOM Workshops , 2008. 1481 [PIE] Pan, R., Natarajan, P., Piglione, C., Prabhu, MS., 1482 Subramanian, V., Baker, F., and B. VerSteeg, "PIE: A 1483 lightweight control scheme to address the bufferbloat 1484 problem", IEEE HPSR , 2013. 1486 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC 1487 793, September 1981. 1489 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 1490 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 1491 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 1492 S., Wroclawski, J., and L. Zhang, "Recommendations on 1493 Queue Management and Congestion Avoidance in the 1494 Internet", RFC 2309, April 1998. 1496 [RFC2488] Allman, M., Glover, D., and L. Sanchez, "Enhancing TCP 1497 Over Satellite Channels using Standard Mechanisms", BCP 1498 28, RFC 2488, January 1999. 1500 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1501 Network Interconnect Devices", RFC 2544, March 1999. 1503 [RFC2647] Newman, D., "Benchmarking Terminology for Firewall 1504 Performance", RFC 2647, August 1999. 1506 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1507 Delay Metric for IPPM", RFC 2679, September 1999. 1509 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1510 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1512 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1513 of Explicit Congestion Notification (ECN) to IP", RFC 1514 3168, September 2001. 1516 [RFC3611] Friedman, T., Caceres, R., and A. Clark, "RTP Control 1517 Protocol Extended Reports (RTCP XR)", RFC 3611, November 1518 2003. 1520 [RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP 1521 Friendly Rate Control (TFRC): Protocol Specification", RFC 1522 5348, September 2008. 1524 [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation 1525 Applicability Statement", RFC 5481, March 2009. 1527 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1528 Control", RFC 5681, September 2009. 1530 [RFC6297] Welzl, M. and D. Ros, "A Survey of Lower-than-Best-Effort 1531 Transport Protocols", RFC 6297, June 2011. 1533 [RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind, 1534 "Low Extra Delay Background Transport (LEDBAT)", RFC 6817, 1535 December 2012. 1537 [SCL-TCP] Morris, R., "Scalable TCP congestion control", IEEE 1538 INFOCOM , 2000. 1540 [TCPEVAL2013] 1541 Hayes, D., Ros, D., Andrew, L., and S. Floyd, "Common TCP 1542 Evaluation Suite", IRTF ICCRG , 2013. 1544 [YU06] Jay, P., Fu, Q., and G. Armitage, "A preliminary analysis 1545 of loss synchronisation between concurrent TCP flows", 1546 Australian Telecommunication Networks and Application 1547 Conference (ATNAC) , 2006. 1549 Authors' Addresses 1550 Nicolas Kuhn (editor) 1551 Telecom Bretagne 1552 2 rue de la Chataigneraie 1553 Cesson-Sevigne 35510 1554 France 1556 Phone: +33 2 99 12 70 46 1557 Email: nicolas.kuhn@telecom-bretagne.eu 1559 Preethi Natarajan (editor) 1560 Cisco Systems 1561 510 McCarthy Blvd 1562 Milpitas, California 1563 United States 1565 Email: prenatar@cisco.com 1567 Naeem Khademi (editor) 1568 University of Oslo 1569 Department of Informatics, PO Box 1080 Blindern 1570 N-0316 Oslo 1571 Norway 1573 Phone: +47 2285 24 93 1574 Email: naeemk@ifi.uio.no 1576 David Ros 1577 Simula Research Laboratory AS 1578 P.O. Box 134 1579 Lysaker, 1325 1580 Norway 1582 Phone: +33 299 25 21 21 1583 Email: dros@simula.no