idnits 2.17.1 draft-briscoe-tsvwg-l4s-diffserv-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (November 1, 2018) is 2003 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-25) exists of draft-ietf-tsvwg-aqm-dualq-coupled-07 == Outdated reference: A later version (-29) exists of draft-ietf-tsvwg-ecn-l4s-id-03 == Outdated reference: A later version (-20) exists of draft-ietf-tsvwg-l4s-arch-03 == Outdated reference: A later version (-10) exists of draft-ietf-tsvwg-le-phb-06 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Transport Area Working Group B. Briscoe 3 Internet-Draft CableLabs 4 Intended status: Informational November 1, 2018 5 Expires: May 5, 2019 7 Interactions between Low Latency, Low Loss, Scalable Throughput (L4S) 8 and Differentiated Services 9 draft-briscoe-tsvwg-l4s-diffserv-02 11 Abstract 13 L4S and Diffserv offer somewhat overlapping services (low latency and 14 low loss), but bandwidth allocation is out of scope for L4S. 15 Therefore there is scope for the two approaches to complement each 16 other, but also to conflict. This informational document explains 17 how the two approaches interact, how they can be arranged to 18 complement each other and in which cases one can stand alone without 19 needing the other. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on May 5, 2019. 38 Copyright Notice 40 Copyright (c) 2018 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (https://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 56 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 57 1.2. Document Roadmap . . . . . . . . . . . . . . . . . . . . 4 58 2. Architectural Comparison of L4S and Diffserv . . . . . . . . 4 59 2.1. Overlaps between L4S and Diffserv . . . . . . . . . . . . 4 60 2.2. Differences between L4S and Diffserv . . . . . . . . . . 4 61 3. Low Latency Diffserv Classes within a DualQ Bandwidth Pool . 5 62 4. DualQ Bandwidth Pool within a Hierarchy of (Diffserv) 63 Bandwidth Queues . . . . . . . . . . . . . . . . . . . . . . 9 64 4.1. DualQ Complemented by an Assured Bandwidth Service . . . 10 65 4.2. DualQ Complemented by a Guaranteed Low Latency Service . 12 66 4.3. DualQ Complemented by a Scavenger Service . . . . . . . . 13 67 5. Coupling More than Two AQMs within a Bandwidth Pool . . . . . 14 68 6. Applicability of Coupled AQM to Global Diffserv PHBs . . . . 14 69 7. Best Practice for Classification and Marking . . . . . . . . 16 70 7.1. Never Re-Mark a DSCP . . . . . . . . . . . . . . . . . . 16 71 7.2. Classification Order . . . . . . . . . . . . . . . . . . 16 72 7.2.1. Classification Order: Problem . . . . . . . . . . . . 17 73 7.2.2. Classification Order: Solutions . . . . . . . . . . . 17 74 8. Policing and Traffic Conditioning . . . . . . . . . . . . . . 17 75 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 76 10. Security Considerations . . . . . . . . . . . . . . . . . . . 18 77 11. Comments Solicited . . . . . . . . . . . . . . . . . . . . . 18 78 12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 18 79 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 80 13.1. Normative References . . . . . . . . . . . . . . . . . . 18 81 13.2. Informative References . . . . . . . . . . . . . . . . . 18 82 Appendix A. Open Issues . . . . . . . . . . . . . . . . . . . . 20 83 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 20 85 1. Introduction 87 The Low Latency Low Loss Scalable throughput (L4S) Internet service 88 [I-D.ietf-tsvwg-l4s-arch] provides a new Internet service that could 89 eventually replace best efforts, but with ultra-low queuing delay and 90 loss. A structure called the Dual-Queue Coupled AQM manages to 91 provide the L4S service alongside a second queue for Classic Internet 92 traffic, but without prejudging the bandwidth allocations between 93 them. L4S is orthogonal to allocation of bandwidth, so it can be 94 complemented by various bandwidth allocation approaches without 95 prejudging which one. 97 The Differentiated Services (Diffserv) architecture [RFC2475] 98 provides for various service classes, some defined globally, others 99 defined locally per network domain. Certain of these service classes 100 offer low latency and low loss, as well as differentiated allocation 101 of bandwidth. 103 Thus, L4S and Diffserv offer somewhat overlapping services (low 104 latency and low loss), but bandwidth allocation is out of scope for 105 L4S. Therefore there is scope for the two approaches to complement 106 each other, but also to conflict. This informational document 107 explains how the two approaches interact, how they can be arranged to 108 complement each other and in which cases one can stand alone without 109 needing the other. 111 1.1. Terminology 113 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 114 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 115 document are to be interpreted as described in [RFC2119]. In this 116 document, these words will appear with that interpretation only when 117 in ALL CAPS. Lower case uses of these words are not to be 118 interpreted as carrying RFC-2119 significance. 120 Classic service: The 'Classic' service is intended for all the 121 congestion control behaviours that currently co-exist with TCP 122 Reno [RFC5681] (e.g. TCP Cubic, Compound, SCTP, etc). 124 Low-Latency, Low-Loss and Scalable (L4S) service: The 'L4S' service 125 is intended for traffic from scalable congestion control 126 algorithms such as Data Centre TCP [RFC8257]. But it is also more 127 general--it will allow a set of congestion controls with similar 128 scaling properties to DCTCP to evolve. 130 Both Classic and L4S services can cope with a proportion of 131 unresponsive or less-responsive traffic as well (e.g. DNS, VoIP, 132 etc). 134 Pure L4S: L4S without unresponsive traffic. 136 Scalable Congestion Control: See [I-D.ietf-tsvwg-l4s-arch] for 137 definition. 139 Classic Congestion Control: See [I-D.ietf-tsvwg-l4s-arch] for 140 definition. 142 DualQ: Abbreviation for Dual-Queue Coupled AQM 143 [I-D.ietf-tsvwg-aqm-dualq-coupled], which is not a specific AQM, 144 but a framework for coupling two AQMs in order to provide L4S 145 service while doing no harm to 'Classic' traffic from traditional 146 sources. 148 ECN field: The Explicit Congestion Notification field [RFC3168] in 149 the IP header (v4 or v6). [RFC8311] has relaxed some of the 150 restrictions that RFC 3168 placed on the use of ECN, in order to 151 enable experiments like L4S, among others. 153 Site: A home, mobile device, small enterprise or campus, where the 154 network bottleneck is typically the access link to the site. Not 155 all network arrangements fit this model but it is a useful, widely 156 applicable generalisation. 158 1.2. Document Roadmap 160 {ToDo} 162 2. Architectural Comparison of L4S and Diffserv 164 This section compares the L4S architecture [I-D.ietf-tsvwg-l4s-arch] 165 with the Diffserv architecture [RFC2475]. 167 L4S uses an identifier [I-D.ietf-tsvwg-ecn-l4s-id] in the ECN field 168 in IP packet headers that is orthogonal to the Diffserv field 169 [RFC2474]. This is because the two approaches can either overlap or 170 complement each other, as outlined in the following two subsections. 172 2.1. Overlaps between L4S and Diffserv 174 L4S provides a low queuing latency, low loss Internet Service. 175 Specific Diffserv service classes also provide low latency and low 176 loss. 178 This means that it is possible to mix traffic from certain Diffserv 179 classes in the same queue as L4S traffic (see Section 3). 181 2.2. Differences between L4S and Diffserv 183 Bandwidth allocation: L4S is orthogonal to allocation of bandwidth, 184 so it can be complemented by various bandwidth allocation 185 approaches without prejudging which one. In contrast, with 186 Diffserv it was never possible to completely separate control of 187 latency and loss from allocation of bandwidth. The only 188 bandwidth-related aspect of L4S is that it ensures that the 189 capacity seeking behaviour of end-systems can scale with 190 increasing flow rate. 192 Differentiation vs. General improvement: Diffserv concerns give and 193 take of bandwidth, latency and loss between traffic classes. In 194 contrast, the separation of L4S from Classic traffic in separate 195 queues concerns incremental deployment of a general improvement in 196 latency and loss, without taking from the other queue. 198 Open vs. closed loop control: The Diffserv architecture requires the 199 source to keep traffic within a contract and, failing that, it has 200 mechanisms to enforce the contract. In this respect, Diffserv is 201 an open-loop control system that is primarily concerned with 202 keeping traffic within capacity limits. Nonetheless, there is an 203 element of closed-loop control in Diffserv. The weighted AQM 204 (e.g. WRED) used for Assured Forwarding [RFC2597] expects traffic 205 to seek to fill capacity and exploits the response to feedback of 206 congestion controllers at traffic sources (closed-loop). 207 Nonetheless, the Diffserv architecture still provides for traffic 208 conditioners that tag traffic that is outside the bandwidth 209 contract for each AF class (open-loop). Then out-of-contract 210 traffic can be discarded if it would otherwise lead to congestion. 212 L4S uses a similar closed-loop mechanism to the weighted AQM used 213 in Diffserv AF in order to ensure roughly equal per-flow 214 throughput between the L4S and Classic queues. That is, L4S 215 relies on the source's closed-loop response to feedback, not any 216 open-loop obligation of each source to keep within a traffic 217 contract. With L4S, any enforcement of per-flow throughput 218 (whether open-loop or closed) is set aside as a separate issue 219 that may or may not be addressed by separate mechanisms, dependent 220 on policy. 222 Per bottleneck vs. per domain: L4S can be independently and 223 incrementally deployed at certain bottlenecks. In contrast a 224 Diffserv system is domain-based consisting of the per-hop 225 behaviour of interior nodes and the traffic conditioning behaviour 226 of boundary nodes, which have to be deployed as a coordinated 227 whole. 229 Degree of multiplexing: Diffserv components such as traffic 230 conditioning are less applicable in access networks where 231 statistical multiplexing is low, whereas L4S was initially 232 designed for access networks, but is also applicable at larger 233 pinch-points (e.g. public peerings). 235 3. Low Latency Diffserv Classes within a DualQ Bandwidth Pool 237 The experimental Dual-Queue Coupled AQM 238 [I-D.ietf-tsvwg-aqm-dualq-coupled] consists of a pair of queues. One 239 provides a low latency low loss service but both have full access to 240 the same pool of bandwidth. When Diffserv was defined no mechanism 241 like this was available that could provide low latency without also 242 requiring bandwidth controls. All Diffserv's mechanisms for low 243 latency and low loss use some form of priority over bandwidth, then 244 apply a bandwidth constraint to prevent the lower priority traffic 245 from being starved. 247 This Diffserv bandwidth constraint has a flip side - it can also 248 provide a bandwidth assurance. However, in turn, bandwidth assurance 249 has both positive and negative aspects. It certainly prevents other 250 traffic encroaching on the bandwidth of the low latency class, but it 251 also carves off a partition within which low latency sessions are 252 more prone to encroach on each other. 254 The DualQ offers an alternative where low latency traffic can access 255 the whole pool of bandwidth (in effect, the largest possible 256 bandwidth constraint). This is expected to be preferred by many 257 network operators and users who would rather not set a bandwidth 258 limit for their low latency traffic - particularly at links in access 259 networks where the very low level of flow multiplexing makes the 260 bandwidth shares of different traffic classes nearly impossible to 261 predict. Nonetheless, if a bandwidth partition is required for 262 bandwidth assurance purposes, it can still be provided separately 263 (see Section 4). 265 The DualQ classifies packets with the ECN field set to ECT(1) or CE 266 into the low latency low loss (L) queue. The L queue maintains a low 267 latency low loss service primarily because an L4S source paces its 268 packets and is linearly responsive to ECN markings, which earns it 269 the right to set the ECT(1) codepoint [I-D.ietf-tsvwg-ecn-l4s-id] 270 [RFC8311]. 272 Nonetheless, a low level of non-L4S traffic can share the L queue 273 without compromising the low latency and low loss of the service. 274 Certain existing Diffserv classes are already intended as low latency 275 and low loss services. An operator could use the DualQ instead of 276 traditional Diffserv queues to give a few of these classes the 277 benefit of low latency and access to the whole pool of bandwidth. 279 However, that would only be safe for those Diffserv service classes 280 that would not risk ruining the low latency of the service. 281 Therefore, an operator must take care to only classify a Diffserv 282 traffic class into the L queue if it is expected to send smoothly 283 without multi-packet bursts. Below we give examples of classes that 284 should (and should not) be safe to mix into the L queue. 286 Table 1 lists the Diffserv service classes that have been allocated 287 global use Diffserv codepoints (DSCPs) from Pool 1. They are 288 described in RFC 4594 [RFC4594] and its updates ([RFC5865] and 289 [I-D.ietf-tsvwg-le-phb] so far). An operator that only deploys a 290 DualQ [I-D.ietf-tsvwg-aqm-dualq-coupled] but not the relevant 291 Diffserv PHBs could classify those with an 'L' in the 'Coupled Queue' 292 column (or local use DSCPs with similar characteristics) into its L 293 queue, irrespective of the setting of the ECN field. 295 +--------------------+-------------+--------+-------+---------------+ 296 | Service Class Name | DSCP Name | DSCP | AQM? | Coupled Queue | 297 +--------------------+-------------+--------+-------+---------------+ 298 | Network Control{1} | CS7 | 111000 | Y & N | L if L4S | 299 | Network Control | CS6 | 110000 | Y & N | L if L4S | 300 | OAM | CS2 | 010000 | Y & N | L if L4S | 301 | Signalling | CS5 | 101000 | N | L if L4S{2} | 302 | Telephony | EF | 101110 | N | L | 303 | RFC 5865 | Voice-Admit | 101100 | N | L{3} | 304 | R-T Interactive | CS4 | 100000 | N | L if L4S{4} | 305 | MM Conferencing | AF4n | 100nn0 | Y | L if L4S | 306 | Broadcast Video | CS3 | 011000 | N | L if L4S{4} | 307 | MM Streaming | AF3n | 011nn0 | Y | L if L4S | 308 | Low Latency Data | AF2n | 010nn0 | Y | L if L4S | 309 | High Thru'put Data | AF1n | 001nn0 | Y | L if L4S{5} | 310 | Standard | BE/DF/CS0 | 000000 | Y | L if L4S | 311 | Low Priority Data | LE{6} | 000001 | Y | L if L4S{7} | 312 +--------------------+-------------+--------+-------+---------------+ 314 Some service class names have been abbreviated to fit. Abbreviations 315 are expanded in RFC 4594 or its updates. For the assured forwarding 316 (AF) DSCP names, the digit 'n' represents 1, 2 or 3 and the 317 corresponding binary digits 'nn' in the DSCP value represent 01,10 or 318 11. The 'Coupled Queue' column is explained in the text. 320 Table 1: Mapping of RFC4594 Diffserv Service Classes in a Coupled AQM 322 Notes for Table 1: 324 {1}: Reserved by RFC 2474 [RFC2474]. 326 {2}: Superficially, CS5 is a candidate for classification into the 327 L queue irrespective of its ECN field, given application 328 signalling is bursty but usually lightweight. However, at 329 least one major equipment vendor uses CS5 by default to 330 indicate unresponsive broadcast video traffic (to which RFC 331 4594 allocates CS3). 333 {3}: Voice-Admit [RFC5865] could be given priority over Expedited 334 Forwarding (EF) [RFC3246]. 336 {4}: The Real-Time Interactive and Broadcast Video service classes 337 (or any equivalent local-use classes) are intended for 338 inelastic traffic. Therefore they would not be expected to 339 mark themselves as ECN-capable. If they did they would be 340 claiming to be elastic and therefore eligible for 341 classification into the L queue (subject to any policing). 342 These classes should not be classified into the L queue on the 343 basis of DSCP alone, because high bandwidth unresponsive 344 traffic with potentially variable rate is not compatible with 345 the L4S service. 347 {5}: High Throughput Data (or any equivalent local-use class) might 348 use the L4S service because of its support for scalable 349 congestion control. 351 {6}: [I-D.ietf-tsvwg-le-phb] updates RFC 4594 to deprecate using 352 CS1 for Lower Effort (LE). 354 {7}: If a packet is marked LE and ECT(1) and the operator has 355 solely provided a DualQ, this recommends that the packet is 356 classified into the L queue. This could result in LE traffic 357 competing for bandwidth with other classes of traffic in the L 358 queue, but at least it should not harm the latency of other 359 traffic. This is because the ECT(1) marking means the source 360 "MUST" use a scalable congestion control 361 [I-D.ietf-tsvwg-ecn-l4s-id], but the LE marking only means it 362 "SHOULD" use an LBE congestion control 363 [I-D.ietf-tsvwg-le-phb]. 365 Those classes with an 'L' in the 'DualQ-Coupled' column would not be 366 expected to have the ECT(1) codepoint set because they are generally 367 unresponsive to congestion. Nonetheless, they could coexist in the 368 same queue as L4S traffic because traffic in all of these classes is 369 expected to arrive smoothly, not in bursts of more than a few 370 packets. Therefore an operator could configure a DualQ Coupled AQM 371 to classify such packets into the L queue solely based on their DSCP, 372 irrespective of their ECN codepoint [I-D.ietf-tsvwg-ecn-l4s-id]. 374 Otherwise, [I-D.ietf-tsvwg-ecn-l4s-id] requires that any other DSCP 375 has no effect on classification into the L queue. Thus a packet of 376 any other DSCP will not be classified into the L queue unless it 377 carries an ECT(1) or CE codepoint in the ECN field. This is shown as 378 'L if L4S' in in the 'DualQ-Coupled' column of Table 1. 380 4. DualQ Bandwidth Pool within a Hierarchy of (Diffserv) Bandwidth 381 Queues 383 The DualQ Coupled AQM offers an L queue that provides low latency low 384 loss service but it pools bandwidth with the Classic (C) service as 385 if they shared a single FIFO. As explained earlier, unlike previous 386 Diffserv low latency mechanisms, the L queue can offer low latency 387 without needing to limit its bandwidth. 389 Typically the DualQ will be able to use all the bandwidth available 390 to a customer site, e.g. a household, a campus or a mobile node, as a 391 single pool. However, this section considers scenarios where the 392 network operator might want to carve off a fraction of a site's 393 bandwidth for other purposes, for instance: 395 1. to ensure that a particularly demanding application (e.g. a 396 virtual reality session) survives even if excess traffic 397 overloads the remainder of the site's bandwidth; 399 2. to give guaranteed low latency to a particular application (e.g. 400 industrial process control), if the statistically assured low 401 latency of the L queue is insufficiently stable; 403 3. to provide a bandwidth scavenger service that will have no effect 404 on any other applications at the site, but will scavenge any 405 unused bandwidth, for instance to transfer backups or large data 406 sets. 408 In all cases, it is assumed that the DualQ has to be able to borrow 409 back any of the carved off bandwidth that is unused by the other 410 service. 412 The following three subsections present solutions for each of the 413 above scenarios. Depending on the reader's viewpoint, each scenario 414 can be seen as: 416 o either taking a queue within an existing Diffserv hierarchy and 417 splitting it into L4S and Classic queues; 419 o or building a queuing hierarchy around a pre-existing dual L4S/ 420 Classic queue. 422 In each case, the DualQ remains as an indivisible 'atomic' component 423 as if it were a single queue with a single pool of bandwidth (but 424 that can either be used for low latency or classic service). 426 The three examples represent the three main ways that this queue-like 427 'atom' can be included in a hierarchy of other queues. Without loss 428 of generality only one other queue complements the DualQ in each 429 case, but it would be straightforward to extend the examples with 430 more queues. 432 Although these examples are framed in the context of IP and Diffserv, 433 similar queuing hierarchies could be constructed at a lower layer, as 434 long as it supported a similar capability to ECN and a similar 435 Traffic Class identifier to Diffserv. 437 4.1. DualQ Complemented by an Assured Bandwidth Service 439 Figure 1 shows a DualQ complemented by an additional queue to add a 440 bandwidth assured service. It is assumed that the operator 441 classifies certain packets into the assured bandwidth queue, perhaps 442 by class of service, source address or 5-tuple flow ID. 444 ---------+--+ 445 Assured b/w | |-----------. 446 ---------+--+ \ Weighted 447 w\.-.scheduler 448 , -----------++ ( )---> 449 | L .->||---. /`-' 450 DualQ | -------/---++ c\.-. / 451 b/w < (Coupling ( )--' 452 pool | ----+--\----+ /`-'Conditional 453 | C | \ |---' priority 454 ` ----+-------+ scheduler 456 Figure 1: How to Complement a DualQ with an Assured Bandwidth Service 458 The DualQ is used as if it were an indivisible 'atomic' component, 459 unchanged from its original description in 460 [I-D.ietf-tsvwg-aqm-dualq-coupled]: 462 o The outputs of the AQMs in the two queues (L and C) are coupled 463 together so that L4S sources leave enough space for C packets so 464 that all 'standard' flows get roughly equal throughput; 466 o A scheduler recombines the outputs of the two queues, giving 467 conditional priority to L packets (the condition prevents 468 starvation of the C queue if any L traffic misbehaves). 470 A weighted scheduler, e.g. weighted round robin (WRR), is used to 471 combine the outputs of the assured bandwidth queue and the DualQ. It 472 is configured with weight w for the assured bandwidth queue. Then, 473 packets requesting assured bandwidth will have priority access to 474 fraction w of the link capacity. However, whenever the assured 475 bandwidth queue is idle or under-utilized, the DualQ can borrow the 476 balance of the bandwidth. Likewise the assured bandwidth queue can 477 borrow more than fraction w if the DualQ under-utilizes its remaining 478 share. 480 Note that a weighted scheduler such as WRR can be used to implement 481 the conditional priority scheduler between the L and C queues. 482 However, the system will not work as intended if the two weighted 483 schedulers in series are replaced by a single three-input weighted 484 scheduler. This is because, whenever one queue under-uses its 485 weighted share, a weighted scheduler allows the other queue to borrow 486 unused capacity. Whenever traffic is present in the C queue, the 487 coupling ensures that L traffic makes space for it by underutiliizing 488 its share of the first scheduler. If the assured bandwidth queue was 489 also served by the same scheduler, the assured bandwidth service 490 would continually borrow the spare capacity left by the L queue that 491 was intended for the C queue. 493 The assured bandwidth service could itself also support applications 494 using low latency low loss and scalable throughput (L4S). This would 495 be done by serving assured bandwidth traffic with a DualQ (Figure 2) 496 and, as usual, confining legacy queue-building traffic to the C 497 queue. 499 , -----------++ Conditional 500 | L .->||---. priority 501 Assured | -------/---++ c\.-.scheduler 502 b/w < (Coupling ( )--. 503 | ----+--\----+ /`-' \ 504 | C | \ |---' \ Weighted 505 ` ----+-------+ w\.-.scheduler 506 ( )---> 507 , -----------++ /`-' 508 | L .->||---. / 509 DualQ | -------/---++ c\.-. / 510 b/w < (Coupling ( )--' 511 pool | ----+--\----+ /`-'Conditional 512 | C | \ |---' priority 513 ` ----+-------+ scheduler 515 Figure 2: How to Complement a DualQ with an Assured Bandwidth Service 516 that also Supports L4S 518 The symmetry of Figure 2 reveals that both DualQs actually have 519 assured bandwidth. Nonetheless, the label 'Assured bandwidth' is 520 only really meaningful from a per-application perspective if the 521 traffic classified into that DualQ is limited to a small number of 522 application sessions at any one time. 524 4.2. DualQ Complemented by a Guaranteed Low Latency Service 526 Figure 3 shows a DualQ complemented by an additional queue to add a 527 guaranteed latency service. It is assumed that the operator 528 classifies certain packets into the guaranteed latency queue, perhaps 529 by class of service, source address or 5-tuple flow ID. 531 o Token bucket 532 | o |rate/burst limiter 533 |___| 534 |___| -----------++ 535 Guaranteed low latency||-----------. 536 -----------++ \ Priority 537 1\.-.scheduler 538 , -----------++ ( )---> 539 | L .->||---. /`-' 540 DualQ | -------/---++ c\.-. / 541 b/w < (Coupling ( )--' 542 pool | ----+--\----+ /`-'Conditional 543 | C | \ |---' priority 544 ` ----+-------+ scheduler 546 Figure 3: How to Complement a DualQ with a Guaranteed Low Latency 547 Service 549 As in all the previous example, the DualQ is used as if it were an 550 indivisible 'atomic' component. 552 A strict priority scheduler is used to combine the outputs of the 553 guaranteed latency queue and the DualQ. Guaranteed low latency 554 traffic is shown as subject to a token bucket that limits rate and 555 tightly limits burst size, which ensures that: 557 o Excessive guaranteed latency traffic cannot abuse its priority and 558 cause the DualQ to starve; 560 o Guaranteed latency traffic cannot ruin its own latency guarantees 561 - it has to keep to a the traffic contract enforced by the token 562 bucket. 564 In a traditional Diffserv architecture, the token bucket would be 565 deployed at the ingress network edge, to limit traffic at each entry 566 point. Alternatively, the token bucket could be deployed directly in 567 front of the queue, where it would only limit the total traffic from 568 all entry points to the network. For an access link into a network, 569 these two alternative would amount to the same thing. 571 Whenever the guaranteed latency queue is idle or under-utilized, the 572 DualQ can borrow the balance of the bandwidth. However, the 573 guaranteed latency queue cannot borrow more than the token bucket 574 allows, even if the DualQ under-utilizes its remaining share. 576 4.3. DualQ Complemented by a Scavenger Service 578 Figure 3 shows a DualQ complemented by an additional queue to add a 579 bandwidth scavenger service. It is assumed that the operator 580 classifies certain packets into the scavenger queue, probably by 581 class of service, e.g. the global-use Lower Effort (LE) Diffserv 582 codepoint [I-D.ietf-tsvwg-le-phb]. 584 , -----------++ Conditional 585 | L .->||---. priority 586 DualQ | -------/---++ c\.-.scheduler 587 b/w < (Coupling ( )--. 588 pool | ----+--\----+ /`-' \ Priority 589 | C | \ |---' 1\.-.scheduler 590 ` ----+-------+ ( )---> 591 /`-' 592 -+-----------+ / 593 Bandwidth|scavenger |-----------' 594 -+-----------+ 596 Figure 4: How to Complement a DualQ with a Bandwidth Scavenger 597 Service 599 As in all the previous example, the DualQ is used as if it were an 600 indivisible 'atomic' component. 602 A strict priority scheduler is used to combine the outputs of the 603 DualQ and the scavenger service. Section 2 of 604 [I-D.ietf-tsvwg-le-phb] suggests alternative mechanisms. 606 Whenever the DualQ is idle or under-utilized, the scavenger service 607 can borrow the balance of the bandwidth. In contrast to the previous 608 guaranteed latency example, no rate limiter is needed on the DualQ 609 because, by definition, the scavenger service is expected to starve 610 if the higher priority service is using all the capacity. 612 5. Coupling More than Two AQMs within a Bandwidth Pool 614 The Diffserv Assured Forwarding (AF) classes of service [RFC2597] use 615 an AQM with differently weighted outputs, e.g. WRED, to provide 616 weighted congestion feedback to the transport layer. Flows 617 classified to use a higher weight AQM each take more of the available 618 capacity, because the weighted AQM has fooled their congestion 619 controller into detecting that the bottleneck is more lightly loaded. 621 A similar mechanism can be used to add throughput differentiation to 622 either or both of the queues within a DualQ. Figure 5 illustrates an 623 example with an AQM offering three weights within the L queue, where 624 L1 gets the highest throughput per flow. It would be a matter of 625 operator policy to choose which of the three L4S AQMs the Classic AQM 626 would couple to. If it were coupled to L3, then C and L3 flows would 627 get roughly equal throughput, while L2 and L1 flows would get more. 629 , -----------++ 630 | L1 || 631 | L2 ||--. 632 | L3 .-> || \ 633 DualQ | -----/-----++ c\.-. 634 b/w < ( Coupling ( )---> 635 pool | ----+\------+ /`-'Conditional 636 | C | \ |---' priority 637 ` ----+-------+ scheduler 639 Figure 5: Coupling the Classic AQM to Multiple L4S AQMs 641 Note: this structure seems straightforward to implement, but the 642 authors are not aware of any implementation or evaluation of AQMs 643 that are both weighted and coupled to other AQMs. 645 6. Applicability of Coupled AQM to Global Diffserv PHBs 647 As has been explained, Diffserv always divides up bandwidth and 648 divides up latency along the same lines as a consequence, whereas the 649 DualQ Coupled AQM solely provides latency separation without 650 bandwidth separation (the idea being that bandwidth separation can be 651 added if needed, using Diffserv mechanisms). 653 In this draft so far, various queuing structures have been described 654 in terms of the way they separate bandwidth and latency. Operators 655 with existing Diffserv deployments may put the question the other way 656 round and ask whether the DualQ Coupled AQM can be used to isolate 657 low latency traffic within the bandwidth allocated to one of the 658 standardized Diffserv PHBs. For instance: 660 o Bandwidth has been allocated to Network Control traffic, but some 661 BGP speakers have been upgraded to a low latency Scalable TCP 662 while others still use Classic TCP. However it's not possible to 663 predict how much bandwidth one or the other needs at any one time. 664 So it would be useful to isolate the low latency BGP and all the 665 control signalling from the delay caused by the legacy BGP 666 speaker, without having to decide how to carve up the Network 667 Control bandwidth. 669 o Bandwidth has been allocated to Assured Forwarding (AF) traffic 670 but it all shares the same WRED queue and therefore all suffers 671 the same delay. So it would be useful to isolate the AF traffic 672 that supports low latency congestion control from the rest. 673 However, again, it is not possible to predict how many flows of 674 each type there will be at any one time. 676 Table 2 lists all the PHBs with standardized global-use DSCPs from 677 [RFC4594] and the right-hand 'Latency Separation?' column identifies 678 all those that could benefit from an unknowable and variable fraction 679 of their traffic being separated between ultra-low and regular delay 680 using a DualQ Coupled AQM. There is no implication that it is 681 sensible to do this in any of the cases; just that it is possible. 683 For convenience, the 'Mechanism' column also answers the question 684 "How do PHBs for the global-use DSCPs map to the scenarios in this 685 draft?" 686 +---------------+-----------+------+---------------+----------------+ 687 | Service Class | DSCP Name | AQM? | Mechanism | Latency | 688 | Name | | | | Separation? | 689 +---------------+-----------+------+---------------+----------------+ 690 | Network | CS7 | Y&N | Figure 1 or | Y | 691 | Control | | | Figure 2 | | 692 | Network | CS6 | Y&N | Figure 1 or | Y | 693 | Control | | | Figure 2 | | 694 | OAM | CS2 | Y&N | Figure 1 or | Y | 695 | | | | Figure 2 | | 696 | Signalling | CS5 | N | Figure 1 | N | 697 | Telephony | EF | N | Section 4.2 | N | 698 | RFC 5865 | VA | N | Section 4.2 | N | 699 | R-T | CS4 | N | Figure 2 | Y | 700 | Interactive | | | | | 701 | MM | AF4n | Y | Section 5 | Y | 702 | Conferencing | | | | | 703 | Broadcast | CS3 | N | Figure 2 | N | 704 | Video | | | | | 705 | MM Streaming | AF3n | Y | Section 5 | Y | 706 | Low Latency | AF2n | Y | Section 5 | Y | 707 | Data | | | | | 708 | High Thru'put | AF1n | Y | Section 5 | Y | 709 | Data | | | | | 710 | Standard | BE/DF/CS0 | Y | Section 3 | Y | 711 | Low Priority | LE | Y | Section 4.3 | n/a | 712 | Data | | | | | 713 +---------------+-----------+------+---------------+----------------+ 715 Table 2: Applicability of a Coupled AQM to RFC4594 Diffserv PHBs 717 7. Best Practice for Classification and Marking 719 7.1. Never Re-Mark a DSCP 721 It is not a DualQ's job to alter Diffserv codepoints to attempt to 722 make other downstream AQMs classify selected packets in certain ways. 723 Each DualQ Coupled AQM is independently (but hopefully consistently) 724 configured to select certain DSCPs for classification into the L 725 queue. It never alters the DSCP nor the ECN codepoint (except 726 setting CE to indicate that congestion was experienced) 727 [I-D.ietf-tsvwg-aqm-dualq-coupled]. 729 7.2. Classification Order 730 7.2.1. Classification Order: Problem 732 The above wide range of possible structures raises the question of 733 which order it would be more efficient for classifier rules to take: 734 DSCP before ECN, ECN before DSCP or some hybrid. 736 On the one hand, for a structure like that in Figure 1 it would make 737 sense to classify on DSCP first, then ECN. Otherwise, if packets 738 were classified on ECN first, an extra merge stage would be required 739 because the assured bandwidth queue handles all ECN codepoints for a 740 particular DSCP. 742 On the other hand, for a structure like that in Figure 5 it would 743 make sense to classify on ECN first, then DSCP. Otherwise, again an 744 extra merge stage would be needed, because the C queue handles all 745 DSCPs but only some ECN codepoints. 747 A hybrid of these two scenarios would be possible, for instance where 748 the L queue in Figure 1 was further broken down into three weighted 749 AQMs, as in Figure 5. In this case, the ideal matching order would 750 be DSCP, ECN, DSCP. 752 7.2.2. Classification Order: Solutions 754 Probably the most straightforward solution would be to classify in a 755 single stage over all 8 octets of the IPv6 Traffic Class field or the 756 former IPv4 TOS octet, irrespective of the boundary between the 6-bit 757 DS field and the 2-bit ECN field [RFC3260]. As long as hardware 758 supports this, it will be possible because all the inputs to the 759 queues are at the same level of hierarchy, even though the outputs 760 form a multi-level hierarchy of schedulers in some cases. 762 Pre-existing classifier hardware might consider the 6-bit and 2-bit 763 fields as separate. Then it would seem most efficient for the order 764 of the classifiers to depend on the structure of the queues being 765 classified (given the structure has to have been designed before the 766 classifiers are designed). 768 8. Policing and Traffic Conditioning 770 {ToDo: L4S latency policing is discussed in the Security 771 Considerations section of [I-D.ietf-tsvwg-l4s-arch]. This section 772 will compare Diffserv traffic conditioning with L4S latency 773 policing.} 775 9. IANA Considerations 777 This specification contains no IANA considerations. 779 10. Security Considerations 781 {ToDo} 783 11. Comments Solicited 785 Comments and questions are encouraged and very welcome. They can be 786 addressed to the IETF Transport Area working group mailing list 787 , and/or to the authors. 789 12. Acknowledgements 791 Thanks to Greg White, David Black, Wes Eddy and Gorry Fairhurst for 792 their useful discussions prior to this -00 draft. 794 13. References 796 13.1. Normative References 798 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 799 Requirement Levels", BCP 14, RFC 2119, 800 DOI 10.17487/RFC2119, March 1997, 801 . 803 13.2. Informative References 805 [I-D.ietf-tsvwg-aqm-dualq-coupled] 806 Schepper, K., Briscoe, B., Bondarenko, O., and I. Tsang, 807 "DualQ Coupled AQMs for Low Latency, Low Loss and Scalable 808 Throughput (L4S)", draft-ietf-tsvwg-aqm-dualq-coupled-07 809 (work in progress), October 2018. 811 [I-D.ietf-tsvwg-ecn-l4s-id] 812 Schepper, K. and B. Briscoe, "Identifying Modified 813 Explicit Congestion Notification (ECN) Semantics for 814 Ultra-Low Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s- 815 id-03 (work in progress), July 2018. 817 [I-D.ietf-tsvwg-l4s-arch] 818 Briscoe, B., Schepper, K., and M. Bagnulo, "Low Latency, 819 Low Loss, Scalable Throughput (L4S) Internet Service: 820 Architecture", draft-ietf-tsvwg-l4s-arch-03 (work in 821 progress), October 2018. 823 [I-D.ietf-tsvwg-le-phb] 824 Bless, R., "A Lower Effort Per-Hop Behavior (LE PHB)", 825 draft-ietf-tsvwg-le-phb-06 (work in progress), October 826 2018. 828 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 829 "Definition of the Differentiated Services Field (DS 830 Field) in the IPv4 and IPv6 Headers", RFC 2474, 831 DOI 10.17487/RFC2474, December 1998, 832 . 834 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 835 and W. Weiss, "An Architecture for Differentiated 836 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 837 . 839 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 840 "Assured Forwarding PHB Group", RFC 2597, 841 DOI 10.17487/RFC2597, June 1999, 842 . 844 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 845 of Explicit Congestion Notification (ECN) to IP", 846 RFC 3168, DOI 10.17487/RFC3168, September 2001, 847 . 849 [RFC3246] Davie, B., Charny, A., Bennet, J., Benson, K., Le Boudec, 850 J., Courtney, W., Davari, S., Firoiu, V., and D. 851 Stiliadis, "An Expedited Forwarding PHB (Per-Hop 852 Behavior)", RFC 3246, DOI 10.17487/RFC3246, March 2002, 853 . 855 [RFC3260] Grossman, D., "New Terminology and Clarifications for 856 Diffserv", RFC 3260, DOI 10.17487/RFC3260, April 2002, 857 . 859 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 860 Guidelines for DiffServ Service Classes", RFC 4594, 861 DOI 10.17487/RFC4594, August 2006, 862 . 864 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 865 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 866 . 868 [RFC5865] Baker, F., Polk, J., and M. Dolly, "A Differentiated 869 Services Code Point (DSCP) for Capacity-Admitted Traffic", 870 RFC 5865, DOI 10.17487/RFC5865, May 2010, 871 . 873 [RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L., 874 and G. Judd, "Data Center TCP (DCTCP): TCP Congestion 875 Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257, 876 October 2017, . 878 [RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion 879 Notification (ECN) Experimentation", RFC 8311, 880 DOI 10.17487/RFC8311, January 2018, 881 . 883 Appendix A. Open Issues 885 o The Abstract promises "in which cases one can stand alone without 886 needing the other". That's in the text, but not highlighted as 887 such. 889 o Document Roadmap TBA 891 o Mapping to 802.11 user priorities (or LTE QCIs)? Not strictly 892 within the scope, but perhaps desirable to add, or at least to 893 mention how L4S (experimental) would affect RFC8325 which gives 894 (standards track) mappings between Diffserv and 802.11. 896 o Identify L4S-friendly rate policers 898 o Comparison between L4S policing and Diffserv traffic conditioning 899 is TBA 901 o Security Considerations are TBA (largely depends on the previous 902 bullet) 904 Author's Address 906 Bob Briscoe 907 CableLabs 908 UK 910 Email: ietf@bobbriscoe.net 911 URI: http://bobbriscoe.net/