idnits 2.17.1 draft-ietf-aqm-eval-guidelines-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 15, 2016) is 2986 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'Byte' is mentioned on line 313, but not defined == Outdated reference: A later version (-07) exists of draft-ietf-tcpm-cubic-01 ** Obsolete normative reference: RFC 793 (Obsoleted by RFC 9293) ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) Summary: 3 errors (**), 0 flaws (~~), 3 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force N. Kuhn, Ed. 3 Internet-Draft CNES, Telecom Bretagne 4 Intended status: Informational P. Natarajan, Ed. 5 Expires: August 18, 2016 Cisco Systems 6 N. Khademi, Ed. 7 University of Oslo 8 D. Ros 9 Simula Research Laboratory AS 10 February 15, 2016 12 AQM Characterization Guidelines 13 draft-ietf-aqm-eval-guidelines-11 15 Abstract 17 Unmanaged large buffers in today's networks have given rise to a slew 18 of performance issues. These performance issues can be addressed by 19 some form of Active Queue Management (AQM) mechanism, optionally in 20 combination with a packet scheduling scheme such as fair queuing. 21 This document describes various criteria for performing precautionary 22 characterizations of AQM schemes. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on August 18, 2016. 41 Copyright Notice 43 Copyright (c) 2016 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 1.1. Goals of this document . . . . . . . . . . . . . . . . . 5 60 1.2. Requirements Language . . . . . . . . . . . . . . . . . . 6 61 1.3. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 6 62 2. End-to-end metrics . . . . . . . . . . . . . . . . . . . . . 6 63 2.1. Flow completion time . . . . . . . . . . . . . . . . . . 7 64 2.2. Flow start up time . . . . . . . . . . . . . . . . . . . 7 65 2.3. Packet loss . . . . . . . . . . . . . . . . . . . . . . . 7 66 2.4. Packet loss synchronization . . . . . . . . . . . . . . . 8 67 2.5. Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 9 68 2.6. Latency and jitter . . . . . . . . . . . . . . . . . . . 9 69 2.7. Discussion on the trade-off between latency and goodput . 10 70 3. Generic setup for evaluations . . . . . . . . . . . . . . . . 10 71 3.1. Topology and notations . . . . . . . . . . . . . . . . . 11 72 3.2. Buffer size . . . . . . . . . . . . . . . . . . . . . . . 12 73 3.3. Congestion controls . . . . . . . . . . . . . . . . . . . 12 74 4. Methodology, Metrics, AQM Comparisons, Packet Sizes, 75 Scheduling and ECN . . . . . . . . . . . . . . . . . . . . . 13 76 4.1. Methodology . . . . . . . . . . . . . . . . . . . . . . . 13 77 4.2. Comments on metrics measurement . . . . . . . . . . . . . 13 78 4.3. Comparing AQM schemes . . . . . . . . . . . . . . . . . . 14 79 4.3.1. Performance comparison . . . . . . . . . . . . . . . 14 80 4.3.2. Deployment comparison . . . . . . . . . . . . . . . . 15 81 4.4. Packet sizes and congestion notification . . . . . . . . 15 82 4.5. Interaction with ECN . . . . . . . . . . . . . . . . . . 15 83 4.6. Interaction with Scheduling . . . . . . . . . . . . . . . 16 84 5. Transport Protocols . . . . . . . . . . . . . . . . . . . . . 16 85 5.1. TCP-friendly sender . . . . . . . . . . . . . . . . . . . 17 86 5.1.1. TCP-friendly sender with the same initial congestion 87 window . . . . . . . . . . . . . . . . . . . . . . . 17 88 5.1.2. TCP-friendly sender with different initial congestion 89 windows . . . . . . . . . . . . . . . . . . . . . . . 17 90 5.2. Aggressive transport sender . . . . . . . . . . . . . . . 18 91 5.3. Unresponsive transport sender . . . . . . . . . . . . . . 18 92 5.4. Less-than Best Effort transport sender . . . . . . . . . 19 93 6. Round Trip Time Fairness . . . . . . . . . . . . . . . . . . 19 94 6.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 19 95 6.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 20 96 6.3. Metrics to evaluate the RTT fairness . . . . . . . . . . 20 98 7. Burst Absorption . . . . . . . . . . . . . . . . . . . . . . 20 99 7.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 20 100 7.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 21 101 8. Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 22 102 8.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 22 103 8.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 23 104 8.2.1. Definition of the congestion Level . . . . . . . . . 23 105 8.2.2. Mild congestion . . . . . . . . . . . . . . . . . . . 23 106 8.2.3. Medium congestion . . . . . . . . . . . . . . . . . . 23 107 8.2.4. Heavy congestion . . . . . . . . . . . . . . . . . . 24 108 8.2.5. Varying the congestion level . . . . . . . . . . . . 24 109 8.2.6. Varying available capacity . . . . . . . . . . . . . 24 110 8.3. Parameter sensitivity and stability analysis . . . . . . 25 111 9. Various Traffic Profiles . . . . . . . . . . . . . . . . . . 26 112 9.1. Traffic mix . . . . . . . . . . . . . . . . . . . . . . . 26 113 9.2. Bi-directional traffic . . . . . . . . . . . . . . . . . 26 114 10. Multi-AQM Scenario . . . . . . . . . . . . . . . . . . . . . 27 115 10.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 27 116 10.2. Details on the evaluation scenario . . . . . . . . . . . 27 117 11. Implementation cost . . . . . . . . . . . . . . . . . . . . . 27 118 11.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 27 119 11.2. Recommended discussion . . . . . . . . . . . . . . . . . 28 120 12. Operator Control and Auto-tuning . . . . . . . . . . . . . . 28 121 12.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 28 122 12.2. Recommended discussion . . . . . . . . . . . . . . . . . 29 123 13. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 29 124 14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 125 15. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 30 126 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 127 17. Security Considerations . . . . . . . . . . . . . . . . . . . 31 128 18. References . . . . . . . . . . . . . . . . . . . . . . . . . 31 129 18.1. Normative References . . . . . . . . . . . . . . . . . . 31 130 18.2. Informative References . . . . . . . . . . . . . . . . . 33 131 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 34 133 1. Introduction 135 Active Queue Management (AQM) [RFC7567] addresses the concerns 136 arising from using unnecessarily large and unmanaged buffers to 137 improve network and application performance. Several AQM algorithms 138 have been proposed in the past years, most notably Random Early 139 Detection (RED), BLUE, and Proportional Integral controller (PI), and 140 more recently CoDel [NICH2012] and PIE [PAN2013]. In general, these 141 algorithms actively interact with the Transmission Control Protocol 142 (TCP) and any other transport protocol that deploys a congestion 143 control scheme to manage the amount of data they keep in the network. 144 The available buffer space in the routers and switches should be 145 large enough to accommodate the short-term buffering requirements. 147 AQM schemes aim at reducing buffer occupancy, and therefore the end- 148 to-end delay. Some of these algorithms, notably RED, have also been 149 widely implemented in some network devices. However, the potential 150 benefits of the RED scheme have not been realized since RED is 151 reported to be usually turned off. The main reason of this 152 reluctance to use RED in today's deployments comes from its 153 sensitivity to the operating conditions in the network and the 154 difficulty of tuning its parameters. 156 A buffer is a physical volume of memory in which a queue or set of 157 queues are stored. When speaking of a specific queue in this 158 document, "buffer occupancy" refers to the amount of data (measured 159 in bytes or packets) that are in the queue, and the "maximum buffer 160 size" refers to the maximum buffer occupancy. In real 161 implementations of switches, a global memory is often shared between 162 the available devices, and thus, the maximum buffer size may vary 163 over the time. 165 Bufferbloat [BB2011] is the consequence of deploying large unmanaged 166 buffers on the Internet -- the buffering has often been measured to 167 be ten times or hundred times larger than needed. Large buffer sizes 168 in combination with TCP and/or unresponsive flows increases end-to- 169 end delay. This results in poor performance for latency-sensitive 170 applications such as real-time multimedia (e.g., voice, video, 171 gaming, etc). The degree to which this affects modern networking 172 equipment, especially consumer-grade equipment's, produces problems 173 even with commonly used web services. Active queue management is 174 thus essential to control queuing delay and decrease network latency. 176 The Active Queue Management and Packet Scheduling Working Group (AQM 177 WG) was chartered to address the problems with large unmanaged 178 buffers in the Internet. Specifically, the AQM WG is tasked with 179 standardizing AQM schemes that not only address concerns with such 180 buffers, but also are robust under a wide variety of operating 181 conditions. This document provides characterization guidelines that 182 can be used to assess the deployability of an AQM, whether it is 183 candidate for standardization at IETF or not. 185 [RFC7567] separately describes the AQM algorithm implemented in a 186 router from the scheduling of packets sent by the router. The rest 187 of this memo refers to the AQM as a dropping/marking policy as a 188 separate feature to any interface scheduling scheme. This document 189 may be complemented with another one on guidelines for assessing 190 combination of packet scheduling and AQM. We note that such a 191 document will inherit all the guidelines from this document plus any 192 additional scenarios relevant for packet scheduling such as flow 193 starvation evaluation or impact of the number of hash buckets. 195 1.1. Goals of this document 197 The trade-off between reducing the latency and maximizing the goodput 198 is intrinsically linked to each AQM scheme and is key to evaluating 199 its performance. Whenever possible, solutions ought to aim at both 200 maximizing goodput and minimizing latency. Moreover, to ensure the 201 safety deployment of an AQM, its behaviour should be assessed in a 202 variety of scenarios. 204 This document recommends a generic list of scenarios against which an 205 AQM proposal should be evaluated, considering both potential 206 performance gain and safety of deployment. The guidelines help to 207 quantify performance of AQM schemes in terms of latency reduction, 208 goodput maximization and the trade-off between these two. The 209 document presents central aspects of an AQM algorithm that should be 210 considered whatever the context, such as burst absorption capacity, 211 RTT fairness or resilience to fluctuating network conditions. The 212 guidelines also discuss methods to understand the various aspects 213 associated with safely deploying and operating the AQM scheme. Thus, 214 one of the key objectives behind formulating the guidelines is to 215 help ascertain whether a specific AQM is not only better than drop- 216 tail (i.e. without AQM and with a BDP-sized buffer) but also safe to 217 deploy: the guidelines can be used to compare several AQM proposals 218 with each other, and should be used to compare a proposal with drop- 219 tail. 221 These guidelines do not define and are not bound to a particular 222 environment or evaluation toolset. Instead the guidelines can be 223 used to assert the potential gain of introducing an AQM for the 224 particular environment, which is of interest to the testers. These 225 guidelines do not cover every possible aspect of a particular 226 algorithm. These guidelines do not present context-dependent 227 scenarios (such as 802.11 WLANs, data-centers or rural broadband 228 networks). To keep the guidelines generic, a number of potential 229 router components and algorithms (such as DiffServ) are omitted. 231 The goals of this document can thus be summarized as follows: 233 o The present characterization guidelines provide a non-exhaustive 234 list of scenarios to help ascertain whether an AQM is not only 235 better than drop-tail (with a BDP-sized buffer), but also safe to 236 deploy; the guidelines can also be used to compare several AQM 237 proposals with each other. 239 o The present characterization guidelines (1) are not bound to a 240 particular evaluation toolset and (2) can be used for various 241 deployment contexts; testers are free to select a toolset that is 242 best suited for the environment in which their proposal will be 243 deployed. 245 o The present characterization guidelines are intended to provide 246 guidance for better selecting an AQM for a specific environment; 247 it is not required that an AQM proposal is evaluated following 248 these guidelines for its standardization. 250 1.2. Requirements Language 252 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 253 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 254 document are to be interpreted as described in RFC 2119 [RFC2119]. 256 1.3. Glossary 258 o AQM: [RFC7567] separately describes the Active Queue Managment 259 (AQM) algorithm implemented in a router from the scheduling of 260 packets sent by the router. The rest of this memo refers to the 261 AQM as a dropping/marking policy as a separate feature to any 262 interface scheduling scheme. 264 o buffer: a physical volume of memory in which a queue or set of 265 queues are stored. 267 o buffer occupancy: amount of data that are stored in a buffer, 268 measured in bytes or packets. 270 o buffer size: maximum buffer occupancy, that is the maximum amount 271 of data that may be stored in a buffer, measured in bytes or 272 packets. 274 o goodput: goodput is defined as the number of bits per unit of time 275 forwarded to the correct destination minus any bits lost or 276 retransmitted [RFC2647]. 278 o SQRT: the square root function. 280 o ROUND: the round function. 282 2. End-to-end metrics 284 End-to-end delay is the result of propagation delay, serialization 285 delay, service delay in a switch, medium-access delay and queuing 286 delay, summed over the network elements along the path. AQM schemes 287 may reduce the queuing delay by providing signals to the sender on 288 the emergence of congestion, but any impact on the goodput must be 289 carefully considered. This section presents the metrics that could 290 be used to better quantify (1) the reduction of latency, (2) 291 maximization of goodput and (3) the trade-off between these two. 292 This section provides normative requirements for metrics that can be 293 used to assess the performance of an AQM scheme. 295 Some metrics listed in this section are not suited to every type of 296 traffic detailed in the rest of this document. It is therefore not 297 necessary to measure all of the following metrics: the chosen metric 298 may not be relevant to the context of the evaluation scenario (e.g., 299 latency vs. goodput trade-off in application-limited traffic 300 scenarios). Guidance is provided for each metric. 302 2.1. Flow completion time 304 The flow completion time is an important performance metric for the 305 end-user when the flow size is finite. Considering the fact that an 306 AQM scheme may drop/mark packets, the flow completion time is 307 directly linked to the dropping/marking policy of the AQM scheme. 308 This metric helps to better assess the performance of an AQM 309 depending on the flow size. The Flow Completion Time (FCT) is 310 related to the flow size (Fs) and the goodput for the flow (G) as 311 follows: 313 FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] ) 315 If this metric is used to evaluate the performance of web transfers, 316 it is suggested to rather consider the time needed to download all 317 the objects that compose the web page, as this makes more sense in 318 terms of user experience than assessing the time needed to download 319 each object. 321 2.2. Flow start up time 323 The flow start up time is the time between the request has been sent 324 from the client and the server starts to transmit data. The amount 325 of packets dropped by an AQM may seriously affect the waiting period 326 during which the data transfer has not started. This metric would 327 specifically focus on the operations such as DNS lookups, TCP opens 328 of SSL handshakes. 330 2.3. Packet loss 332 Packet loss can occur en-route, this can impact the end-to-end 333 performance measured at receiver. 335 The tester SHOULD evaluate loss experienced at the receiver using one 336 of the two metrics: 338 o the packet loss ratio: this metric is to be frequently measured 339 during the experiment. The long-term loss ratio is of interest 340 for steady-state scenarios only; 342 o the interval between consecutive losses: the time between two 343 losses is to be measured. 345 The packet loss ratio can be assessed by simply evaluating the loss 346 ratio as a function of the number of lost packets and the total 347 number of packets sent. This might not be easily done in laboratory 348 testing, for which these guidelines advice the tester: 350 o to check that for every packet, a corresponding packet was 351 received within a reasonable time, as explained in [RFC2680]. 353 o to keep a count of all packets sent, and a count of the non- 354 duplicate packets received, as explained in the section 10 of 355 [RFC2544]. 357 The interval between consecutive losses, which is also called a gap, 358 is a metric of interest for VoIP traffic and, as a result, has been 359 further specified in [RFC3611]. 361 2.4. Packet loss synchronization 363 One goal of an AQM algorithm is to help to avoid global 364 synchronization of flows sharing a bottleneck buffer on which the AQM 365 operates ([RFC2309],[RFC7567]). The "degree" of packet-loss 366 synchronization between flows SHOULD be assessed, with and without 367 the AQM under consideration. 369 As discussed e.g., in [HASS2008], loss synchronization among flows 370 may be quantified by several slightly different metrics that capture 371 different aspects of the same issue. However, in real-world 372 measurements the choice of metric could be imposed by practical 373 considerations -- e.g., whether fine-grained information on packet 374 losses in the bottleneck available or not. For the purpose of AQM 375 characterization, a good candidate metric is the global 376 synchronization ratio, measuring the proportion of flows losing 377 packets during a loss event. [JAY2006] used this metric in real- 378 world experiments to characterize synchronization along arbitrary 379 Internet paths; the full methodology is described in [JAY2006]. 381 If an AQM scheme is evaluated using real-life network environments, 382 it is worth pointing out that some network events, such as failed 383 link restoration may cause synchronized losses between active flows 384 and thus confuse the meaning of this metric. 386 2.5. Goodput 388 The goodput has been defined in section 3.17 of [RFC2647] as the 389 number of bits per unit of time forwarded to the correct destination 390 interface, minus any bits lost or retransmitted. This definition 391 induces that the test setup needs to be qualified to assure that it 392 is not generating losses on its own. 394 Measuring the end-to-end goodput provides an appreciation of how well 395 an AQM scheme improves transport and application performance. The 396 measured end-to-end goodput is linked to the dropping/marking policy 397 of the AQM scheme -- e.g., the fewer the number of packet drops, the 398 fewer packets need retransmission, minimizing the impact of AQM on 399 transport and application performance. Additionally, an AQM scheme 400 may resort to Explicit Congestion Notification (ECN) marking as an 401 initial means to control delay. Again, marking packets instead of 402 dropping them reduces the number of packet retransmissions and 403 increases goodput. End-to-end goodput values help to evaluate the 404 AQM scheme's effectiveness of an AQM scheme in minimizing packet 405 drops that impact application performance and to estimate how well 406 the AQM scheme works with ECN. 408 The measurement of the goodput allows the tester evaluate to which 409 extent an AQM is able to maintain a high bottleneck utilization. 410 This metric should be also obtained frequently during an experiment 411 as the long-term goodput is relevant for steady-state scenarios only 412 and may not necessarily reflect how the introduction of an AQM 413 actually impacts the link utilization during at a certain period of 414 time. Fluctuations in the values obtained from these measurements 415 may depend on other factors than the introduction of an AQM, such as 416 link layer losses due to external noise or corruption, fluctuating 417 bandwidths (802.11 WLANs), heavy congestion levels or transport 418 layer's rate reduction by congestion control mechanism. 420 2.6. Latency and jitter 422 The latency, or the one-way delay metric, is discussed in [RFC2679]. 423 There is a consensus on an adequate metric for the jitter, that 424 represents the one-way delay variations for packets from the same 425 flow: the Packet Delay Variation (PDV), detailed in [RFC5481], serves 426 well all use cases. 428 The end-to-end latency includes components other than just the 429 queuing delay, such as the signal processing delay, transmission 430 delay and the processing delay. Moreover, the jitter is caused by 431 variations in queuing and processing delay (e.g., scheduling 432 effects). The introduction of an AQM scheme would impact these 433 metrics (end-to-end latency and jitter) and therefore they should be 434 considered in the end-to-end evaluation of performance. 436 2.7. Discussion on the trade-off between latency and goodput 438 The metrics presented in this section may be considered as explained 439 in the rest of this document, in order to discuss and quantify the 440 trade-off between latency and goodput. 442 With regards to the goodput, and in addition to the long-term 443 stationary goodput value, it is RECOMMENDED to take measurements 444 every multiple of the minimum RTT (minRTT) between A and B. It is 445 suggested to take measurements at least every K x minRTT (to smooth 446 out the fluctuations), with K=10. Higher values for K are encouraged 447 whenever it is more appropriate for the presentation of the results. 448 The value for K may depend on the network's path characteristics. 449 The measurement period MUST be disclosed for each experiment and when 450 results/values are compared across different AQM schemes, the 451 comparisons SHOULD use exactly the same measurement periods. With 452 regards to latency, it is RECOMMENDED to take the samples on per- 453 packet basis whenever possible depending on the features provided by 454 hardware/software and the impact of sampling itself on the hardware 455 performance. It is generally RECOMMENDED to provide at least 10 456 samples per RTT. 458 From each of these sets of measurements, the cumulative density 459 function (CDF) of the considered metrics SHOULD be computed. If the 460 considered scenario introduces dynamically varying parameters, 461 temporal evolution of the metrics could also be generated. For each 462 scenario, the following graph may be generated: the x-axis shows 463 queuing delay (that is the average per-packet delay in excess of 464 minimum RTT), the y-axis the goodput. Ellipses are computed such as 465 detailed in [WINS2014]: "We take each individual [...] run [...] as 466 one point, and then compute the 1-epsilon elliptic contour of the 467 maximum-likelihood 2D Gaussian distribution that explains the points. 468 [...] we plot the median per-sender throughput and queueing delay as 469 a circle. [...] The orientation of an ellipse represents the 470 covariance between the throughput and delay measured for the 471 protocol." This graph provides part of a better understanding of (1) 472 the delay/goodput trade-off for a given congestion control mechanism 473 Section 5, and (2) how the goodput and average queue delay vary as a 474 function of the traffic load Section 8.2. 476 3. Generic setup for evaluations 478 This section presents the topology that can be used for each of the 479 following scenarios, the corresponding notations and discusses 480 various assumptions that have been made in the document. 482 3.1. Topology and notations 484 +---------+ +-----------+ 485 |senders A| |receivers B| 486 +---------+ +-----------+ 488 +--------------+ +--------------+ 489 |traffic class1| |traffic class1| 490 |--------------| |--------------| 491 | SEN.Flow1.1 +---------+ +-----------+ REC.Flow1.1 | 492 | + | | | | + | 493 | | | | | | | | 494 | + | | | | + | 495 | SEN.Flow1.X +-----+ | | +--------+ REC.Flow1.X | 496 +--------------+ | | | | +--------------+ 497 + +-+---+---+ +--+--+---+ + 498 | |Router L | |Router R | | 499 | |---------| |---------| | 500 | | AQM | | | | 501 | | BuffSize| | BuffSize| | 502 | | (Bsize) +-----+ (Bsize) | | 503 | +-----+--++ ++-+------+ | 504 + | | | | + 505 +--------------+ | | | | +--------------+ 506 |traffic classN| | | | | |traffic classN| 507 |--------------| | | | | |--------------| 508 | SEN.FlowN.1 +---------+ | | +-----------+ REC.FlowN.1 | 509 | + | | | | + | 510 | | | | | | | | 511 | + | | | | + | 512 | SEN.FlowN.Y +------------+ +-------------+ REC.FlowN.Y | 513 +--------------+ +--------------+ 515 Figure 1: Topology and notations 517 Figure 1 is a generic topology where: 519 o sender with different traffic characteristics (i.e., traffic 520 profiles) can be introduced; 522 o the timing of each flow could be different (i.e., when does each 523 flow start and stop); 525 o each traffic profile can comprise various number of flows; 527 o each link is characterized by a couple (one-way delay, capacity); 528 o flows are generated at A and sent to B, sharing a bottleneck (the 529 link between routers L and R); 531 o the tester SHOULD consider both scenarios of asymmetric and 532 symmetric bottleneck links in terms of bandwidth. In case of 533 asymmetric link, the capacity from senders to receivers is higher 534 than the one from receivers to senders; the symmetric link 535 scenario provides a basic understanding of the operation of the 536 AQM mechanism whereas the asymmetric link scenario evaluates an 537 AQM mechanism in a more realistic setup; 539 o in asymmetric link scenarios, the tester SHOULD study the bi- 540 directional traffic between A and B (downlink and uplink) with the 541 AQM mechanism deployed on one direction only. The tester MAY 542 additionally consider a scenario with AQM mechanism being deployed 543 on both directions. In each scenario, the tester SHOULD 544 investigate the impact of drop policy of the AQM on TCP ACK 545 packets and its impact on the performance. 547 Although this topology may not perfectly reflect actual topologies, 548 the simple topology is commonly used in the world of simulations and 549 small testbeds. It can be considered as adequate to evaluate AQM 550 proposals, similarly to the topology proposed in 551 [I-D.irtf-iccrg-tcpeval]. Testers ought to pay attention to the 552 topology that has been used to evaluate an AQM scheme when comparing 553 this scheme with a newly proposed AQM scheme. 555 3.2. Buffer size 557 The size of the buffers should be carefully chosen, and MAY be set to 558 the bandwidth-delay product; the bandwidth being the bottleneck 559 capacity and the delay the largest RTT in the considered network. 560 The size of the buffer can impact the AQM performance and is a 561 dimensioning parameter that will be considered when comparing AQM 562 proposals. 564 If a specific buffer size is required, the tester MUST justify and 565 detail the way the maximum queue size is set. Indeed, the maximum 566 size of the buffer may affect the AQM's performance and its choice 567 SHOULD be elaborated for a fair comparison between AQM proposals. 568 While comparing AQM schemes the buffer size SHOULD remain the same 569 across the tests. 571 3.3. Congestion controls 573 This document considers running three different congestion control 574 algorithms between A and B 575 o Standard TCP congestion control: the base-line congestion control 576 is TCP NewReno with SACK, as explained in [RFC5681]. 578 o Aggressive congestion controls: a base-line congestion control for 579 this category is TCP Cubic [I-D.ietf-tcpm-cubic]. 581 o Less-than Best Effort (LBE) congestion controls: an LBE congestion 582 control 'results in smaller bandwidth and/or delay impact on 583 standard TCP than standard TCP itself, when sharing a bottleneck 584 with it.' [RFC6297] 586 Other transport congestion controls can OPTIONALLY be evaluated in 587 addition. Recent transport layer protocols are not mentioned in the 588 following sections, for the sake of simplicity. 590 4. Methodology, Metrics, AQM Comparisons, Packet Sizes, Scheduling and 591 ECN 593 4.1. Methodology 595 A description of each test setup SHOULD be detailed to allow this 596 test to be compared with other tests. This also allows others to 597 replicate the tests if needed. This test setup SHOULD detail 598 software and hardware versions. The tester could make its data 599 available. 601 The proposals SHOULD be evaluated on real-life systems, or they MAY 602 be evaluated with event-driven simulations (such as ns-2, ns-3, 603 OMNET, etc). The proposed scenarios are not bound to a particular 604 evaluation toolset. 606 The tester is encouraged to make the detailed test setup and the 607 results publicly available. 609 4.2. Comments on metrics measurement 611 The document presents the end-to-end metrics that ought to be used to 612 evaluate the trade-off between latency and goodput in Section 2. In 613 addition to the end-to-end metrics, the queue-level metrics (normally 614 collected at the device operating the AQM) provide a better 615 understanding of the AQM behavior under study and the impact of its 616 internal parameters. Whenever it is possible (e.g., depending on the 617 features provided by the hardware/software), these guidelines advice 618 to consider queue-level metrics, such as link utilization, queuing 619 delay, queue size or packet drop/mark statistics in addition to the 620 AQM-specific parameters. However, the evaluation MUST be primarily 621 based on externally observed end-to-end metrics. 623 These guidelines do not aim to detail on the way these metrics can be 624 measured, since the way these metrics are measured is expected to 625 depend on the evaluation toolset. 627 4.3. Comparing AQM schemes 629 This document recognizes that these guidelines may be used for 630 comparing AQM schemes. 632 AQM schemes need to be compared against both performance and 633 deployment categories. In addition, this section details how best to 634 achieve a fair comparison of AQM schemes by avoiding certain 635 pitfalls. 637 4.3.1. Performance comparison 639 AQM schemes should be compared against the generic scenarios that are 640 summarized in Section 13. AQM schemes MAY be compared for specific 641 network environments such as data centers, home networks, etc. If an 642 AQM scheme has parameter(s) that were externally tuned for 643 optimization or other purposes, these values MUST be disclosed. 645 AQM schemes belong to different varieties such as queue-length based 646 schemes (ex. RED) or queueing-delay based scheme (ex. CoDel, PIE). 647 AQM schemes expose different control knobs associated with different 648 semantics. For example, while both PIE and CoDel are queueing-delay 649 based schemes and each expose a knob to control the queueing delay -- 650 PIE's "queueing delay reference" vs. CoDel's "queueing delay target", 651 the two tuning parameters of the two schemes have different 652 semantics, resulting in different control points. Such differences 653 in AQM schemes can be easily overlooked while making comparisons. 655 This document RECOMMENDS the following procedures for a fair 656 performance comparison between the AQM schemes: 658 1. comparable control parameters and comparable input values: 659 carefully identify the set of parameters that control similar 660 behavior between the two AQM schemes and ensure these parameters 661 have comparable input values. For example, to compare how well a 662 queue-length based AQM scheme controls queueing delay vs. a 663 queueing-delay based AQM scheme, a tester can identify the 664 parameters of the schemes that control queue delay and ensure 665 that their input values are comparable. Similarly, to compare 666 how well two AQM schemes accommodate packet bursts, the tester 667 can identify burst-related control parameters and ensure they are 668 configured with similar values. Additionally, it would be 669 preferable if an AQM proposal listed such parameters and 670 discussed how each relates to network characteristics such as 671 capacity, average RTT etc. 673 2. compare over a range of input configurations: there could be 674 situations when the set of control parameters that affect a 675 specific behavior have different semantics between the two AQM 676 schemes. As mentioned above, PIE has tuning parameters to 677 control queue delay that has a different semantics from those 678 used in CoDel. In such situations, these schemes need to be 679 compared over a range of input configurations. For example, 680 compare PIE vs. CoDel over the range of target delay input 681 configurations. 683 4.3.2. Deployment comparison 685 AQM schemes MUST be compared against deployment criteria such as the 686 parameter sensitivity (Section 8.3), auto-tuning (Section 12) or 687 implementation cost (Section 11). 689 4.4. Packet sizes and congestion notification 691 An AQM scheme may be considering packet sizes while generating 692 congestion signals. [RFC7141] discusses the motivations behind this. 693 For example, control packets such as DNS requests/responses, TCP 694 SYNs/ACKs are small, but their loss can severely impact the 695 application performance. An AQM scheme may therefore be biased 696 towards small packets by dropping them with smaller probability 697 compared to larger packets. However, such an AQM scheme is unfair to 698 data senders generating larger packets. Data senders, malicious or 699 otherwise, are motivated to take advantage of such AQM scheme by 700 transmitting smaller packets, and could result in unsafe deployments 701 and unhealthy transport and/or application designs. 703 An AQM scheme SHOULD adhere to the recommendations outlined in 704 [RFC7141], and SHOULD NOT provide undue advantage to flows with 705 smaller packets [RFC7567]. 707 4.5. Interaction with ECN 709 Deployed AQM algorithms SHOULD implement Explicit Congestion 710 Notification (ECN) as well as loss to signal congestion to endpoints 711 [RFC7567]. ECN [RFC3168] is an alternative that allows AQM schemes 712 to signal receivers about network congestion that does not use packet 713 drop. The benefits of providing ECN support for an AQM scheme are 714 described in [WELZ2015]. Section 3 of [WELZ2015] describes expected 715 operation of routers enabling ECN. AQM schemes SHOULD NOT drop or 716 remark packets solely because the ECT(0) or ECT(1) codepoints are 717 used, and when ECN-capable SHOULD set a CE-mark on ECN-capable 718 packets in the presence of incipient congestion. 720 If the tested AQM scheme can support ECN [RFC7567], the testers MUST 721 discuss and describe the support of ECN. Since these guidelines can 722 be used to evaluate the performance of the tested AQM with and 723 without ECN markings, they could also be used to quantify the 724 interest of enabling ECN. 726 4.6. Interaction with Scheduling 728 A network device may use per-flow or per-class queuing with a 729 scheduling algorithm to either prioritize certain applications or 730 classes of traffic, limit the rate of transmission, or to provide 731 isolation between different traffic flows within a common class 732 [RFC7567]. 734 The scheduling and the AQM conjointly impact on the end-to-end 735 performance. Therefore, the AQM proposal MUST discuss the 736 feasibility to add scheduling combined with the AQM algorithm. This 737 discussion as an instance, MAY explain whether the dropping policy is 738 applied when packets are being enqueued or dequeued. 740 These guidelines do not propose guidelines to assess the performance 741 of scheduling algorithms. Indeed, as opposed to characterizing AQM 742 schemes that is related to their capacity to control the queuing 743 delay in a queue, characterizing scheduling schemes is related to the 744 scheduling itself and its interaction with the AQM scheme. As one 745 example, the scheduler may create sub-queues and the AQM scheme may 746 be applied on each of the sub-queues, and/or the AQM could be applied 747 on the whole queue. Also, schedulers might, such as FQ-CoDel 748 [HOEI2015] or FavorQueue [ANEL2014], introduce flow prioritization. 749 In these cases, specific scenarios should be proposed to ascertain 750 that these scheduler schemes not only helps in tackling the 751 bufferbloat, but also are robust under a wide variety of operating 752 conditions. This is out of the scope of this document that focus on 753 dropping and/or marking AQM schemes. 755 5. Transport Protocols 757 Network and end-devices need to be configured with a reasonable 758 amount of buffer space to absorb transient bursts. In some 759 situations, network providers tend to configure devices with large 760 buffers to avoid packet drops triggered by a full buffer and to 761 maximize the link utilization for standard loss-based TCP traffic. 763 AQM algorithms are often evaluated by considering Transmission 764 Control Protocol (TCP) [RFC0793] with a limited number of 765 applications. TCP is a widely deployed transport. It fills up 766 available buffers until a sender transfering a bulk flow with TCP 767 receives a signal (packet drop) that reduces the sending rate. The 768 larger the buffer, the higher the buffer occupancy, and therefore the 769 queuing delay. An efficient AQM scheme sends out early congestion 770 signals to TCP to bring the queuing delay under control. 772 Not all endpoints (or applications) using TCP use the same flavor of 773 TCP. Variety of senders generate different classes of traffic which 774 may not react to congestion signals (aka non-responsive flows 775 [RFC7567]) or may not reduce their sending rate as expected (aka 776 Transport Flows that are less responsive than TCP[RFC7567], also 777 called "aggressive flows"). In these cases, AQM schemes seek to 778 control the queuing delay. 780 This section provides guidelines to assess the performance of an AQM 781 proposal for various traffic profiles -- different types of senders 782 (with different TCP congestion control variants, unresponsive, 783 aggressive). 785 5.1. TCP-friendly sender 787 5.1.1. TCP-friendly sender with the same initial congestion window 789 This scenario helps to evaluate how an AQM scheme reacts to a TCP- 790 friendly transport sender. A single long-lived, non application- 791 limited, TCP NewReno flow, with an Initial congestion Window (IW) set 792 to 3 packets, transfers data between sender A and receiver B. Other 793 TCP friendly congestion control schemes such as TCP-friendly rate 794 control [RFC5348] etc MAY also be considered. 796 For each TCP-friendly transport considered, the graph described in 797 Section 2.7 could be generated. 799 5.1.2. TCP-friendly sender with different initial congestion windows 801 This scenario can be used to evaluate how an AQM scheme adapts to a 802 traffic mix consisting of TCP flows with different values of the IW. 804 For this scenario, two types of flows MUST be generated between 805 sender A and receiver B: 807 o A single long-lived non application-limited TCP NewReno flow; 809 o A single application-limited TCP NewReno flow, with an IW set to 3 810 or 10 packets. The size of the data transferred must be strictly 811 higher than 10 packets and should be lower than 100 packets. 813 The transmission of the non application-limited flow must start 814 before the transmission of the application-limited flow and only 815 after the steady state has been reached by non application-limited 816 flow. 818 For each of these scenarios, the graph described in Section 2.7 could 819 be generated for each class of traffic (application-limited and non 820 application-limited). The completion time of the application-limited 821 TCP flow could be measured. 823 5.2. Aggressive transport sender 825 This scenario helps testers to evaluate how an AQM scheme reacts to a 826 transport sender that is more aggressive than a single TCP-friendly 827 sender. We define 'aggressiveness' as a higher increase factor than 828 standard upon a successful transmission and/or a lower than standard 829 decrease factor upon a unsuccessful transmission (e.g., in case of 830 congestion controls with Additive-Increase Multiplicative-Decrease 831 (AIMD) principle, a larger AI and/or MD factors). A single long- 832 lived, non application-limited, TCP Cubic flow transfers data between 833 sender A and receiver B. Other aggressive congestion control schemes 834 MAY also be considered. 836 For each flavor of aggressive transports, the graph described in 837 Section 2.7 could be generated. 839 5.3. Unresponsive transport sender 841 This scenario helps testers to evaluate how an AQM scheme reacts to a 842 transport sender that is less responsive than TCP. Note that faulty 843 transport implementations on an end host and/or faulty network 844 elements en-route that "hide" congestion signals in packet headers 845 [RFC7567] may also lead to a similar situation, such that the AQM 846 scheme needs to adapt to unresponsive traffic. To this end, these 847 guidelines propose the two following scenarios. 849 The first scenario can be used to evaluate queue build up. It 850 considers unresponsive flow(s) whose sending rate is greater than the 851 bottleneck link capacity between routers L and R. This scenario 852 consists of a long-lived non application limited UDP flow transmits 853 data between sender A and receiver B. Graphs described in 854 Section 2.7 could be generated. 856 The second scenario can be used to evaluate if the AQM scheme is able 857 to keep the responsive fraction under control. This scenario 858 considers a mixture of TCP-friendly and unresponsive traffics. It 859 consists of a long-lived UDP flow from unresponsive application and a 860 single long-lived, non application-limited (unlimited data available 861 to the transport sender from application layer), TCP New Reno flow 862 that transmit data between sender A and receiver B. As opposed to 863 the first scenario, the rate of the UDP traffic should not be greater 864 than the bottleneck capacity, and should be higher than half of the 865 bottleneck capacity. For each type of traffic, the graph described 866 in Section 2.7 could be generated. 868 5.4. Less-than Best Effort transport sender 870 This scenario helps to evaluate how an AQM scheme reacts to LBE 871 congestion controls that 'results in smaller bandwidth and/or delay 872 impact on standard TCP than standard TCP itself, when sharing a 873 bottleneck with it.' [RFC6297]. The potential fateful interaction 874 when AQM and LBE techniques are combined has been shown in 875 [GONG2014]; this scenario helps to evaluate whether the coexistence 876 of the proposed AQM and LBE techniques may be possible. 878 A single long-lived non application-limited TCP NewReno flow 879 transfers data between sender A and receiver B. Other TCP-friendly 880 congestion control schemes MAY also be considered. Single long-lived 881 non application-limited LEDBAT [RFC6817] flows transfer data between 882 sender A and receiver B. We recommend to set the target delay and 883 gain values of LEDBAT respectively to 5 ms and 10 [TRAN2014]. Other 884 LBE congestion control schemes, any of those listed in [RFC6297], MAY 885 also be considered. 887 For each of the TCP-friendly and LBE transports, the graph described 888 in Section 2.7 could be generated. 890 6. Round Trip Time Fairness 892 6.1. Motivation 894 An AQM scheme's congestion signals (via drops or ECN marks) must 895 reach the transport sender so that a responsive sender can initiate 896 its congestion control mechanism and adjust the sending rate. This 897 procedure is thus dependent on the end-to-end path RTT. When the RTT 898 varies, the onset of congestion control is impacted, and in turn 899 impacts the ability of an AQM scheme to control the queue. It is 900 therefore important to assess the AQM schemes for a set of RTTs 901 between A and B (e.g., from 5 ms to 200 ms). 903 The asymmetry in terms of difference in intrinsic RTT between various 904 paths sharing the same bottleneck SHOULD be considered so that the 905 fairness between the flows can be discussed since in this scenario, a 906 flow traversing on shorter RTT path may react faster to congestion 907 and recover faster from it compared to another flow on a longer RTT 908 path. The introduction of AQM schemes may potentially improve this 909 type of fairness. 911 Introducing an AQM scheme may cause the unfairness between the flows, 912 even if the RTTs are identical. This potential unfairness SHOULD be 913 investigated as well. 915 6.2. Recommended tests 917 The RECOMMENDED topology is detailed in Figure 1. 919 To evaluate the RTT fairness, for each run, two flows divided into 920 two categories. Category I whose RTT between sender A and receiver B 921 SHOULD be 100ms. Category II which RTT between sender A and receiver 922 B should be in the range [5ms;560ms] inclusive. The maximum value 923 for the RTT represents the RTT of a satellite link that, according to 924 section 2 of [RFC2488] should be at least 558ms. 926 A set of evaluated flows MUST use the same congestion control 927 algorithm: all the generated flows could be single long-lived non 928 application-limited TCP NewReno flows. 930 6.3. Metrics to evaluate the RTT fairness 932 The outputs that MUST be measured are: (1) the cumulative average 933 goodput of the flow from Category I, goodput_Cat_I (Section 2.5); (2) 934 the cumulative average goodput of the flow from Category II, 935 goodput_Cat_II (Section 2.5); (3) the ratio goodput_Cat_II/ 936 goodput_Cat_I; (4) the average packet drop rate for each category 937 (Section 2.3). 939 7. Burst Absorption 941 "AQM mechanisms need to control the overall queue sizes, to ensure 942 that arriving bursts can be accommodated without dropping packets" 943 [RFC7567]. 945 7.1. Motivation 947 An AQM scheme can face bursts of packet arrivals due to various 948 reasons. Dropping one or more packets from a burst can result in 949 performance penalties for the corresponding flows, since dropped 950 packets have to be retransmitted. Performance penalties can result 951 in failing to meet SLAs and be a disincentive to AQM adoption. 953 The ability to accommodate bursts translates to larger queue length 954 and hence more queuing delay. On the one hand, it is important that 955 an AQM scheme quickly brings bursty traffic under control. On the 956 other hand, a peak in the packet drop rates to bring a packet burst 957 quickly under control could result in multiple drops per flow and 958 severely impact transport and application performance. Therefore, an 959 AQM scheme ought to bring bursts under control by balancing both 960 aspects -- (1) queuing delay spikes are minimized and (2) performance 961 penalties for ongoing flows in terms of packet drops are minimized. 963 An AQM scheme that maintains short queues allows some remaining space 964 in the buffer for bursts of arriving packets. The tolerance to 965 bursts of packets depends upon the number of packets in the queue, 966 which is directly linked to the AQM algorithm. Moreover, an AQM 967 scheme may implement a feature controlling the maximum size of 968 accepted bursts, that can depend on the buffer occupancy or the 969 currently estimated queuing delay. The impact of the buffer size on 970 the burst allowance may be evaluated. 972 7.2. Recommended tests 974 For this scenario, tester MUST evaluate how the AQM performs with the 975 following traffic generated from sender A to receiver B: 977 o Web traffic with IW10; 979 o Bursty video frames; 981 o Constant Bit Rate (CBR) UDP traffic. 983 o A single non application-limited bulk TCP flow as background 984 traffic. 986 Figure 2 presents the various cases for the traffic that MUST be 987 generated between sender A and receiver B. 989 +-------------------------------------------------+ 990 |Case| Traffic Type | 991 | +-----+------------+----+--------------------+ 992 | |Video|Web (IW 10)| CBR| Bulk TCP Traffic | 993 +----|-----|------------|----|--------------------| 994 |I | 0 | 1 | 1 | 0 | 995 +----|-----|------------|----|--------------------| 996 |II | 0 | 1 | 1 | 1 | 997 |----|-----|------------|----|--------------------| 998 |III | 1 | 1 | 1 | 0 | 999 +----|-----|------------|----|--------------------| 1000 |IV | 1 | 1 | 1 | 1 | 1001 +----+-----+------------+----+--------------------+ 1003 Figure 2: Bursty traffic scenarios 1005 A new web page download could start after the previous web page 1006 download is finished. Each web page could be composed by at least 50 1007 objects and the size of each object should be at least 1kB. 6 TCP 1008 parallel connections SHOULD be generated to download the objects, 1009 each parallel connections having an initial congestion window set to 1010 10 packets. 1012 For each of these scenarios, the graph described in Section 2.7 could 1013 be generated for each application. Metrics such as end-to-end 1014 latency, jitter, flow completion time MAY be generated. For the 1015 cases of frame generation of bursty video traffic as well as the 1016 choice of web traffic pattern, these details and their presentation 1017 are left to the testers. 1019 8. Stability 1021 8.1. Motivation 1023 The safety of an AQM scheme is directly related to its stability 1024 under varying operating conditions such as varying traffic profiles 1025 and fluctuating network conditions. Since operating conditions can 1026 vary often the AQM needs to remain stable under these conditions 1027 without the need for additional external tuning. 1029 Network devices can experience varying operating conditions depending 1030 on factors such as time of the day, deployment scenario, etc. For 1031 example: 1033 o Traffic and congestion levels are higher during peak hours than 1034 off-peak hours. 1036 o In the presence of a scheduler, the draining rate of a queue can 1037 vary depending on the occupancy of other queues: a low load on a 1038 high priority queue implies a higher draining rate for the lower 1039 priority queues. 1041 o The capacity available can vary over time (e.g., a lossy channel, 1042 a link supporting traffic in a higher diffserv class). 1044 Whether the target context is a not stable environment, the ability 1045 of an AQM scheme to maintain its control over the queuing delay and 1046 buffer occupancy can be challenged. This document proposes 1047 guidelines to assess the behavior of AQM schemes under varying 1048 congestion levels and varying draining rates. 1050 8.2. Recommended tests 1052 Note that the traffic profiles explained below comprises non 1053 application-limited TCP flows. For each of the below scenarios, the 1054 graphs described in Section 2.7 SHOULD be generated, and the goodput 1055 of the various flows should be cumulated. For Section 8.2.5 and 1056 Section 8.2.6 they SHOULD incorporate the results in per-phase basis 1057 as well. 1059 Wherever the notion of time has explicitly mentioned in this 1060 subsection, time 0 starts from the moment all TCP flows have already 1061 reached their congestion avoidance phase. 1063 8.2.1. Definition of the congestion Level 1065 In these guidelines, the congestion levels are represented by the 1066 projected packet drop rate, had a drop-tail queue was chosen instead 1067 of an AQM scheme. When the bottleneck is shared among non 1068 application-limited TCP flows. l_r, the loss rate projection can be 1069 expressed as a function of N, the number of bulk TCP flows, and S, 1070 the sum of the bandwidth-delay product and the maximum buffer size, 1071 both expressed in packets, based on Eq. 3 of [MORR2000]: 1073 l_r = 0.76 * N^2 / S^2 1075 N = S * SQRT(1/0.76) * SQRT (l_r) 1077 These guidelines use the loss rate to define the different congestion 1078 levels, but they do not stipulate that in other circumstances, 1079 measuring the congestion level gives you an accurate estimation of 1080 the loss rate or vice-versa. 1082 8.2.2. Mild congestion 1084 This scenario can be used to evaluate how an AQM scheme reacts to a 1085 light load of incoming traffic resulting in mild congestion -- packet 1086 drop rates around 0.1%. The number of bulk flows required to achieve 1087 this congestion level, N_mild, is then: 1089 N_mild = ROUND (0.036*S) 1091 8.2.3. Medium congestion 1093 This scenario can be used to evaluate how an AQM scheme reacts to 1094 incoming traffic resulting in medium congestion -- packet drop rates 1095 around 0.5%. The number of bulk flows required to achieve this 1096 congestion level, N_med, is then: 1098 N_med = ROUND (0.081*S) 1100 8.2.4. Heavy congestion 1102 This scenario can be used to evaluate how an AQM scheme reacts to 1103 incoming traffic resulting in heavy congestion -- packet drop rates 1104 around 1%. The number of bulk flows required to achieve this 1105 congestion level, N_heavy, is then: 1107 N_heavy = ROUND (0.114*S) 1109 8.2.5. Varying the congestion level 1111 This scenario can be used to evaluate how an AQM scheme reacts to 1112 incoming traffic resulting in various levels of congestion during the 1113 experiment. In this scenario, the congestion level varies within a 1114 large time-scale. The following phases may be considered: phase I - 1115 mild congestion during 0-20s; phase II - medium congestion during 1116 20-40s; phase III - heavy congestion during 40-60s; phase I again, 1117 and so on. 1119 8.2.6. Varying available capacity 1121 This scenario can be used to help characterize how the AQM behaves 1122 and adapts to bandwidth changes. The experiments are not meant to 1123 reflect the exact conditions of Wi-Fi environments since it is hard 1124 to design repetitive experiments or accurate simulations for such 1125 scenarios. 1127 To emulate varying draining rates, the bottleneck capacity between 1128 nodes 'Router L' and 'Router R' varies over the course of the 1129 experiment as follows: 1131 o Experiment 1: the capacity varies between two values within a 1132 large time-scale. As an example, the following phases may be 1133 considered: phase I - 100Mbps during 0-20s; phase II - 10Mbps 1134 during 20-40s; phase I again, and so on. 1136 o Experiment 2: the capacity varies between two values within a 1137 short time-scale. As an example, the following phases may be 1138 considered: phase I - 100Mbps during 0-100ms; phase II - 10Mbps 1139 during 100-200ms; phase I again, and so on. 1141 The tester MAY choose a phase time-interval value different than what 1142 is stated above, if the network's path conditions (such as bandwidth- 1143 delay product) necessitate. In this case the choice of such time- 1144 interval value SHOULD be stated and elaborated. 1146 The tester MAY additionally evaluate the two mentioned scenarios 1147 (short-term and long-term capacity variations), during and/or 1148 including TCP slow-start phase. 1150 More realistic fluctuating capacity patterns MAY be considered. The 1151 tester MAY choose to incorporate realistic scenarios with regards to 1152 common fluctuation of bandwidth in state-of-the-art technologies. 1154 The scenario consists of TCP NewReno flows between sender A and 1155 receiver B. To better assess the impact of draining rates on the AQM 1156 behavior, the tester MUST compare its performance with those of drop- 1157 tail and SHOULD provide a reference document for their proposal 1158 discussing performance and deployment compared to those of drop-tail. 1159 Burst traffic, such as presented in Section 7.2, could also be 1160 considered to assess the impact of varying available capacity on the 1161 burst absorption of the AQM. 1163 8.3. Parameter sensitivity and stability analysis 1165 The control law used by an AQM is the primary means by which the 1166 queuing delay is controlled. Hence understanding the control law is 1167 critical to understanding the behavior of the AQM scheme. The 1168 control law could include several input parameters whose values 1169 affect the AQM scheme's output behavior and its stability. 1170 Additionally, AQM schemes may auto-tune parameter values in order to 1171 maintain stability under different network conditions (such as 1172 different congestion levels, draining rates or network environments). 1173 The stability of these auto-tuning techniques is also important to 1174 understand. 1176 Transports operating under the control of AQM experience the effect 1177 of multiple control loops that react over different timescales. It 1178 is therefore important that proposed AQM schemes are seen to be 1179 stable when they are deployed at multiple points of potential 1180 congestion along an Internet path. The pattern of congestion signals 1181 (loss or ECN-marking) arising from AQM methods also need to not 1182 adversely interact with the dynamics of the transport protocols that 1183 they control. 1185 AQM proposals SHOULD provide background material showing control 1186 theoretic analysis of the AQM control law and the input parameter 1187 space within which the control law operates as expected; or could use 1188 another way to discuss the stability of the control law. For 1189 parameters that are auto-tuned, the material SHOULD include stability 1190 analysis of the auto-tuning mechanism(s) as well. Such analysis 1191 helps to understand an AQM control law better and the network 1192 conditions/deployments under which the AQM is stable. 1194 9. Various Traffic Profiles 1196 This section provides guidelines to assess the performance of an AQM 1197 proposal for various traffic profiles such as traffic with different 1198 applications or bi-directional traffic. 1200 9.1. Traffic mix 1202 This scenario can be used to evaluate how an AQM scheme reacts to a 1203 traffic mix consisting of different applications such as: 1205 o Bulk TCP transfer 1207 o Web traffic 1209 o VoIP 1211 o Constant Bit Rate (CBR) UDP traffic 1213 o Adaptive video streaming 1215 Various traffic mixes can be considered. These guidelines RECOMMEND 1216 to examine at least the following example: 1 bi-directional VoIP; 6 1217 Web pages download (such as detailed in Section 7.2); 1 CBR; 1 1218 Adaptive Video; 5 bulk TCP. Any other combinations could be 1219 considered and should be carefully documented. 1221 For each scenario, the graph described in Section 2.7 could be 1222 generated for each class of traffic. Metrics such as end-to-end 1223 latency, jitter and flow completion time MAY be reported. 1225 9.2. Bi-directional traffic 1227 Control packets such as DNS requests/responses, TCP SYNs/ACKs are 1228 small, but their loss can severely impact the application 1229 performance. The scenario proposed in this section will help in 1230 assessing whether the introduction of an AQM scheme increases the 1231 loss probability of these important packets. 1233 For this scenario, traffic MUST be generated in both downlink and 1234 uplink, such as defined in Section 3.1. These guidelines RECOMMEND 1235 to consider a mild congestion level and the traffic presented in 1236 Section 8.2.2 in both directions. In this case, the metrics reported 1237 MUST be the same as in Section 8.2 for each direction. 1239 The traffic mix presented in Section 9.1 MAY also be generated in 1240 both directions. 1242 10. Multi-AQM Scenario 1244 10.1. Motivation 1246 Transports operating under the control of AQM experience the effect 1247 of multiple control loops that react over different timescales. It 1248 is therefore important that proposed AQM schemes are seen to be 1249 stable when they are deployed at multiple points of potential 1250 congestion along an Internet path. The pattern of congestion signals 1251 (loss or ECN-marking) arising from AQM methods also need to not 1252 adversely interact with the dynamics of the transport protocols that 1253 they control. 1255 10.2. Details on the evaluation scenario 1257 +---------+ +-----------+ 1258 |senders A|---+ +---|receivers A| 1259 +---------+ | | +-----------+ 1260 +-----+---+ +---------+ +--+-----+ 1261 |Router L |--|Router M |--|Router R| 1262 |AQM | |AQM | |No AQM | 1263 +---------+ +--+------+ +--+-----+ 1264 +---------+ | | +-----------+ 1265 |senders B|-------------+ +---|receivers B| 1266 +---------+ +-----------+ 1268 Figure 3: Topology for the Multi-AQM scenario 1270 This scenario can be used to evaluate how having AQM schemes in 1271 sequence impact the induced latency reduction, the induced goodput 1272 maximization and the trade-off between these two. The topology 1273 presented in Figure 3 could be used. AQM schemes introduced in 1274 Router L and Router M should be the same; any other configurations 1275 could be considered. For this scenario, it is recommended to 1276 consider a mild congestion level, the number of flows specified in 1277 Section 8.2.2 being equally shared among senders A and B. Any other 1278 relevant combination of congestion levels could be considered. We 1279 recommend to measure the metrics presented in Section 8.2. 1281 11. Implementation cost 1283 11.1. Motivation 1285 Successful deployment of AQM is directly related to its cost of 1286 implementation. Network devices can need hardware or software 1287 implementations of the AQM mechanism. Depending on a device's 1288 capabilities and limitations, the device may or may not be able to 1289 implement some or all parts of their AQM logic. 1291 AQM proposals SHOULD provide pseudo-code for the complete AQM scheme, 1292 highlighting generic implementation-specific aspects of the scheme 1293 such as "drop-tail" vs. "drop-head", inputs (e.g., current queuing 1294 delay, queue length), computations involved, need for timers, etc. 1295 This helps to identify costs associated with implementing the AQM 1296 scheme on a particular hardware or software device. This also 1297 facilitates discsusions around which kind of devices can easily 1298 support the AQM and which cannot. 1300 11.2. Recommended discussion 1302 AQM proposals SHOULD highlight parts of their AQM logic that are 1303 device dependent and discuss if and how AQM behavior could be 1304 impacted by the device. For example, a queueing-delay based AQM 1305 scheme requires current queuing delay as input from the device. If 1306 the device already maintains this value, then it can be trivial to 1307 implement the their AQM logic on the device. If the device provides 1308 indirect means to estimate the queuing delay (for example: 1309 timestamps, dequeuing rate), then the AQM behavior is sensitive to 1310 the precision of the queuing delay estimations are for that device. 1311 Highlighting the sensitivity of an AQM scheme to queuing delay 1312 estimations helps implementers to identify appropriate means of 1313 implementing the mechanism on a device. 1315 12. Operator Control and Auto-tuning 1317 12.1. Motivation 1319 One of the biggest hurdles of RED deployment was/is its parameter 1320 sensitivity to operating conditions -- how difficult it is to tune 1321 RED parameters for a deployment to achieve acceptable benefit from 1322 using RED. Fluctuating congestion levels and network conditions add 1323 to the complexity. Incorrect parameter values lead to poor 1324 performance. 1326 Any AQM scheme is likely to have parameters whose values affect the 1327 control law and behaviour of an AQM. Exposing all these parameters 1328 as control parameters to a network operator (or user) can easily 1329 result in a unsafe AQM deployment. Unexpected AQM behavior ensues 1330 when parameter values are set improperly. A minimal number of 1331 control parameters minimizes the number of ways a user can break a 1332 system where an AQM scheme is deployed at. Fewer control parameters 1333 make the AQM scheme more user-friendly and easier to deploy and 1334 debug. 1336 [RFC7567] states "AQM algorithms SHOULD NOT require tuning of initial 1337 or configuration parameters in common use cases." A scheme ought to 1338 expose only those parameters that control the macroscopic AQM 1339 behavior such as queue delay threshold, queue length threshold, etc. 1341 Additionally, the safety of an AQM scheme is directly related to its 1342 stability under varying operating conditions such as varying traffic 1343 profiles and fluctuating network conditions, as described in 1344 Section 8. Operating conditions vary often and hence the AQM needs 1345 to remain stable under these conditions without the need for 1346 additional external tuning. If AQM parameters require tuning under 1347 these conditions, then the AQM must self-adapt necessary parameter 1348 values by employing auto-tuning techniques. 1350 12.2. Recommended discussion 1352 In order to understand an AQM's deployment considerations and 1353 performance under a specific environment, AQM proposals SHOULD 1354 describe the parameters that control the macroscopic AQM behavior, 1355 and identify any parameters that require tuning to operational 1356 conditions. It could be interesting to also discuss that even if an 1357 AQM scheme may not adequately auto-tune its parameters, the resulting 1358 performance may not be optimal, but close to something reasonable. 1360 If there are any fixed parameters within the AQM, their setting 1361 SHOULD be discussed and justified, to help understand whether a fixed 1362 parameter value is applicable for a particular environment. 1364 If an AQM scheme is evaluated with parameter(s) that were externally 1365 tuned for optimization or other purposes, these values MUST be 1366 disclosed. 1368 13. Conclusion 1370 Figure 4 lists the scenarios and their requirements. 1372 +------------------------------------------------------------------+ 1373 |Scenario |Sec. |Requirement | 1374 +------------------------------------------------------------------+ 1375 +------------------------------------------------------------------+ 1376 |Interaction with ECN | 4.5 |MUST be discussed if supported | 1377 +------------------------------------------------------------------+ 1378 |Interaction with Scheduling| 4.6 |Feasibility MUST be discussed | 1379 +------------------------------------------------------------------+ 1380 |Transport Protocols |5. | | 1381 | TCP-friendly sender | 5.1 |Scenario MUST be considered | 1382 | Aggressive sender | 5.2 |Scenario MUST be considered | 1383 | Unresponsive sender | 5.3 |Scenario MUST be considered | 1384 | LBE sender | 5.4 |Scenario MAY be considered | 1385 +------------------------------------------------------------------+ 1386 |Round Trip Time Fairness | 6.2 |Scenario MUST be considered | 1387 +------------------------------------------------------------------+ 1388 |Burst Absorption | 7.2 |Scenario MUST be considered | 1389 +------------------------------------------------------------------+ 1390 |Stability |8. | | 1391 | Varying congestion levels | 8.2.5|Scenario MUST be considered | 1392 | Varying available capacity| 8.2.6|Scenario MUST be considered | 1393 | Parameters and stability | 8.3 |This SHOULD be discussed | 1394 +------------------------------------------------------------------+ 1395 |Various Traffic Profiles |9. | | 1396 | Traffic mix | 9.1 |Scenario is RECOMMENDED | 1397 | Bi-directional traffic | 9.2 |Scenario MAY be considered | 1398 +------------------------------------------------------------------+ 1399 |Multi-AQM | 10.2 |Scenario MAY be considered | 1400 +------------------------------------------------------------------+ 1401 |Implementation Cost | 11.2 |Pseudo-code SHOULD be provided | 1402 +------------------------------------------------------------------+ 1403 |Operator Control | 12.2 |Tuning SHOULD NOT be required | 1404 +------------------------------------------------------------------+ 1406 Figure 4: Summary of the scenarios and their requirements 1408 14. Acknowledgements 1410 This work has been partially supported by the European Community 1411 under its Seventh Framework Programme through the Reducing Internet 1412 Transport Latency (RITE) project (ICT-317700). 1414 15. Contributors 1416 Many thanks to S. Akhtar, A.B. Bagayoko, F. Baker, R. Bless, D. 1417 Collier-Brown, G. Fairhurst, J. Gettys, P. Goltsman, T. Hoiland- 1418 Jorgensen, K. Kilkki, C. Kulatunga, W. Lautenschlager, A.C. 1420 Morton, R. Pan, G. Skinner, D. Taht and M. Welzl for detailed and 1421 wise feedback on this document. 1423 16. IANA Considerations 1425 This memo includes no request to IANA. 1427 17. Security Considerations 1429 Some security considerations for AQM are identified in [RFC7567].This 1430 document, by itself, presents no new privacy nor security issues. 1432 18. References 1434 18.1. Normative References 1436 [I-D.ietf-tcpm-cubic] 1437 Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and 1438 R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", 1439 draft-ietf-tcpm-cubic-01 (work in progress), January 2016. 1441 [I-D.irtf-iccrg-tcpeval] 1442 Hayes, D., Ros, D., Andrew, L., and S. Floyd, "Common TCP 1443 Evaluation Suite", draft-irtf-iccrg-tcpeval-01 (work in 1444 progress), July 2014. 1446 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 1447 RFC 793, DOI 10.17487/RFC0793, September 1981, 1448 . 1450 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1451 Requirement Levels", RFC 2119, 1997. 1453 [RFC2488] Allman, M., Glover, D., and L. Sanchez, "Enhancing TCP 1454 Over Satellite Channels using Standard Mechanisms", 1455 BCP 28, RFC 2488, DOI 10.17487/RFC2488, January 1999, 1456 . 1458 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1459 Network Interconnect Devices", RFC 2544, 1460 DOI 10.17487/RFC2544, March 1999, 1461 . 1463 [RFC2647] Newman, D., "Benchmarking Terminology for Firewall 1464 Performance", RFC 2647, DOI 10.17487/RFC2647, August 1999, 1465 . 1467 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1468 Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679, 1469 September 1999, . 1471 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1472 Packet Loss Metric for IPPM", RFC 2680, 1473 DOI 10.17487/RFC2680, September 1999, 1474 . 1476 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1477 of Explicit Congestion Notification (ECN) to IP", 1478 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1479 . 1481 [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., 1482 "RTP Control Protocol Extended Reports (RTCP XR)", 1483 RFC 3611, DOI 10.17487/RFC3611, November 2003, 1484 . 1486 [RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP 1487 Friendly Rate Control (TFRC): Protocol Specification", 1488 RFC 5348, DOI 10.17487/RFC5348, September 2008, 1489 . 1491 [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation 1492 Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, 1493 March 2009, . 1495 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1496 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1497 . 1499 [RFC6297] Welzl, M. and D. Ros, "A Survey of Lower-than-Best-Effort 1500 Transport Protocols", RFC 6297, DOI 10.17487/RFC6297, June 1501 2011, . 1503 [RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind, 1504 "Low Extra Delay Background Transport (LEDBAT)", RFC 6817, 1505 DOI 10.17487/RFC6817, December 2012, 1506 . 1508 [RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion 1509 Notification", RFC 7141, 2014. 1511 [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF 1512 Recommendations Regarding Active Queue Management", 1513 BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, 1514 . 1516 18.2. Informative References 1518 [ANEL2014] 1519 Anelli, P., Diana, R., and E. Lochin, "FavorQueue: a 1520 Parameterless Active Queue Management to Improve TCP 1521 Traffic Performance", Computer Networks vol. 60, 2014. 1523 [BB2011] "BufferBloat: what's wrong with the internet?", ACM 1524 Queue vol. 9, 2011. 1526 [GONG2014] 1527 Gong, Y., Rossi, D., Testa, C., Valenti, S., and D. Taht, 1528 "Fighting the bufferbloat: on the coexistence of AQM and 1529 low priority congestion control", Computer Networks, 1530 Elsevier, 2014, 60, pp.115 - 128 , 2014. 1532 [HASS2008] 1533 Hassayoun, S. and D. Ros, "Loss Synchronization and Router 1534 Buffer Sizing with High-Speed Versions of TCP", IEEE 1535 INFOCOM Workshops , 2008. 1537 [HOEI2015] 1538 Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, 1539 J., and E. Dumazet, "FlowQueue-Codel", IETF (Work-in- 1540 Progress) , January 2015. 1542 [JAY2006] Jay, P., Fu, Q., and G. Armitage, "A preliminary analysis 1543 of loss synchronisation between concurrent TCP flows", 1544 Australian Telecommunication Networks and Application 1545 Conference (ATNAC) , 2006. 1547 [MORR2000] 1548 Morris, R., "Scalable TCP congestion control", IEEE 1549 INFOCOM , 2000. 1551 [NICH2012] 1552 Nichols, K. and V. Jacobson, "Controlling Queue Delay", 1553 ACM Queue , 2012. 1555 [PAN2013] Pan, R., Natarajan, P., Piglione, C., Prabhu, MS., 1556 Subramanian, V., Baker, F., and B. VerSteeg, "PIE: A 1557 lightweight control scheme to address the bufferbloat 1558 problem", IEEE HPSR , 2013. 1560 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 1561 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 1562 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 1563 S., Wroclawski, J., and L. Zhang, "Recommendations on 1564 Queue Management and Congestion Avoidance in the 1565 Internet", RFC 2309, April 1998. 1567 [TRAN2014] 1568 Trang, S., Kuhn, N., Lochin, E., Baudoin, C., Dubois, E., 1569 and P. Gelard, "On The Existence Of Optimal LEDBAT 1570 Parameters", IEEE ICC 2014 - Communication QoS, 1571 Reliability and Modeling Symposium , 2014. 1573 [WELZ2015] 1574 Welzl, M. and G. Fairhurst, "The Benefits to Applications 1575 of using Explicit Congestion Notification (ECN)", IETF 1576 (Work-in-Progress) , June 2015. 1578 [WINS2014] 1579 Winstein, K., "Transport Architectures for an Evolving 1580 Internet", PhD thesis, Massachusetts Institute of 1581 Technology , 2014. 1583 Authors' Addresses 1585 Nicolas Kuhn (editor) 1586 CNES, Telecom Bretagne 1587 18 avenue Edouard Belin 1588 Toulouse 31400 1589 France 1591 Phone: +33 5 61 27 32 13 1592 Email: nicolas.kuhn@cnes.fr 1594 Preethi Natarajan (editor) 1595 Cisco Systems 1596 510 McCarthy Blvd 1597 Milpitas, California 1598 United States 1600 Email: prenatar@cisco.com 1601 Naeem Khademi (editor) 1602 University of Oslo 1603 Department of Informatics, PO Box 1080 Blindern 1604 N-0316 Oslo 1605 Norway 1607 Phone: +47 2285 24 93 1608 Email: naeemk@ifi.uio.no 1610 David Ros 1611 Simula Research Laboratory AS 1612 P.O. Box 134 1613 Lysaker, 1325 1614 Norway 1616 Phone: +33 299 25 21 21 1617 Email: dros@simula.no