idnits 2.17.1 draft-white-tsvwg-l4sops-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 30, 2020) is 1365 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'I-D.ietf-tsvwg-aqm-dualq-coupled' is defined on line 293, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-tsvwg-ecn-l4s-id' is defined on line 299, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-tsvwg-l4s-arch' is defined on line 305, but no explicit reference was found in the text == Unused Reference: 'RFC2119' is defined on line 311, but no explicit reference was found in the text == Unused Reference: 'RFC8174' is defined on line 316, but no explicit reference was found in the text == Unused Reference: 'RFC8290' is defined on line 320, but no explicit reference was found in the text == Outdated reference: A later version (-25) exists of draft-ietf-tsvwg-aqm-dualq-coupled-12 == Outdated reference: A later version (-29) exists of draft-ietf-tsvwg-ecn-l4s-id-10 == Outdated reference: A later version (-20) exists of draft-ietf-tsvwg-l4s-arch-06 Summary: 0 errors (**), 0 flaws (~~), 10 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Transport Area Working Group G. White, Ed. 3 Internet-Draft CableLabs 4 Intended status: Informational July 30, 2020 5 Expires: January 31, 2021 7 Operational Guidance for Deployment of L4S in the Internet 8 draft-white-tsvwg-l4sops-00 10 Abstract 12 This is an early, work-in-progress draft - a start at getting some of 13 the ideas from the mailing list and email exchanges on paper. 15 This draft is intended to provide guidance to operators of end- 16 systems, operators of networks, and researchers in order to ensure 17 reasonable fairness between L4S and Classic flows sharing a single- 18 queue RFC3168 bottleneck link. This draft identifies opportunites to 19 prevent and/or detect and resolve fairness problems in such networks. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on January 31, 2021. 38 Copyright Notice 40 Copyright (c) 2020 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (https://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 56 2. Per-Flow Fairness . . . . . . . . . . . . . . . . . . . . . . 3 57 3. Operator of an L4S host . . . . . . . . . . . . . . . . . . . 3 58 3.1. CDN Servers . . . . . . . . . . . . . . . . . . . . . . . 3 59 3.2. Other hosts . . . . . . . . . . . . . . . . . . . . . . . 4 60 4. Operator of a Network . . . . . . . . . . . . . . . . . . . . 4 61 4.1. Configure AQM to treat ECT1 as NotECT . . . . . . . . . . 4 62 4.2. Configure Non-Coupled Dual Queue . . . . . . . . . . . . 5 63 4.3. WRED with ECT1 Differentation . . . . . . . . . . . . . . 5 64 4.4. ECT1 Tunnel Bypass . . . . . . . . . . . . . . . . . . . 6 65 4.5. Disable RFC3168 ECN Marking . . . . . . . . . . . . . . . 6 66 4.6. Re-mark ECT1 to NotECT Prior to AQM . . . . . . . . . . . 6 67 5. Researchers . . . . . . . . . . . . . . . . . . . . . . . . . 6 68 5.1. Detection of Classic ECN FIFO Bottlenecks . . . . . . . . 6 69 5.2. End-to-end measurement of L4S vs. Classic performance . . 6 70 6. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 6 71 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 6 72 8. Security Considerations . . . . . . . . . . . . . . . . . . . 7 73 9. Informative References . . . . . . . . . . . . . . . . . . . 7 74 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 7 76 1. Introduction 78 In the majority of network paths, including paths where the 79 bottleneck link utilizes packet drops (either due to buffer overrun 80 or active queue management) in response to congestion, as well as 81 paths that implement a 'flow-queuing' scheduler such as fq_codel or 82 Cobalt, and those that implement dual-Q-coupled AQM, L4S traffic 83 coexists well with classic congestion controlled traffic. 85 On network paths where the bottleneck link implements a shared-queue 86 (FIFO) with an Active Queue Management algorithm that provides 87 Explicit Congestion Notification signaling according to RFC3168, it 88 has been demonstrated that when a set of long-running flows 89 comprising both "Classic" congestion controlled flows and L4S- 90 compliant congestion controlled flows compete for bandwidth, the 91 classic congestion controlled flows may achieve lower throughput when 92 compared to the L4S congestion controlled flows. This 'unfairness' 93 between the two classes appears to be more pronounced on longer RTT 94 paths (e.g. 50ms and above) and/or at higher link rates (e.g. 50 Mbps 95 and above). 97 The root cause of this unfairness is that RFC3168 does not 98 differentiate between packets marked ECT0 (used by classic senders) 99 and those marked ECT1 (used by L4S senders), and provides an 100 identical congestion signal (CE marks) to both classes, whereas the 101 two classes respond differently to that congestion signal. The 102 classic senders expect that CE marks are sent very rarely (e.g. 103 approximately 1 CE mark every 200 round trips on a 50 Mbps x 50ms 104 path) while the L4S senders expect very frequent CE marking (e.g. 105 approximately 2 CE marks per round trip). The result is that the 106 classic senders respond to the CE marks provided by the bottleneck by 107 yielding capacity to the L4S flows. While this has not been 108 demonstrated to cause starvation of the classic flows, the resulting 109 rate imbalance can be a cause of concern. 111 2. Per-Flow Fairness 113 There are a number of factors that influence the relative rates 114 achieved by a set of congestion controlled flows sharing a queue in a 115 bottleneck link. 117 TODO: discuss startup & convergence times, short flows, RTT- 118 unfairness, differences in deployed CC algorithms, etc. 120 TODO: also mention that flow sharding is commonplace, so per-flow 121 fairness does not imply per-application fairness 123 3. Operator of an L4S host 125 Support for L4S involves both endpoints: ECT1 marking & L4S- 126 compatible congestion control on the sender, and ECN feedback on the 127 receiver. Between these two entities, it is incumbent upon the 128 sender to evaluate the potential for unfairness and make decisions 129 whether or not to use L4S congestion control. The receiver is not 130 expected to perform any testing or monitoring for unfairness, and is 131 also not expected to invoke any active response in the case that 132 unfairness occurs. 134 3.1. CDN Servers 136 Some hosts (such as CDN leaf nodes and servers internal to an ISP) 137 are deployed in environments in which they serve content to a 138 constrained set of networks or clients. The operator of such hosts 139 may be able to determine whether there is the possibility of RFC3168 140 FIFO bottlenecks being present, and utilize this information to make 141 decisions on selectively deploying L4S. 143 o Prior to deploying L4S on servers: 145 * Consult with network operators on presence of RFC3168 FIFO 146 bottlenecks 148 * Perform downstream tests per access network 150 + Tests (TBD) to detect absence of RFC 3168 152 + Enable AccECN feedback, but enable/disable L4S per access 153 network 155 o In-band RFC3168 detection and monitoring: 157 * Real-time response (fallback) 159 * Non-real-time response (disable for future connections) 161 3.2. Other hosts 163 o In-band RFC3168 detection (and possibly fallback) 165 o Per-dst path test: 167 * For a connection capable of L4S feedback 169 * If CE feedback, perform active test (TBD) for RFC3168 presence 171 * Could cache result per-dst 173 o Query a TBD public whitelist of domains that are participating in 174 L4S experiment 176 4. Operator of a Network 178 While it is, of course, preferred for networks to deploy L4S-capable 179 high fidelity congestion signaling, a network operator who has 180 deployed equipment in a likely bottleneck link location (i.e. a link 181 that is expected to be fully saturated) that is configured with an 182 RFC3168 FIFO AQM can take certain steps in order to improve rate 183 fairness between classic traffic and L4S traffic. 185 4.1. Configure AQM to treat ECT1 as NotECT 187 If equipment is configurable in such as way as to only supply CE 188 marks to ECT0 packets, and treat ECT1 packets identically to NotECT, 189 or is upgradable to support this capability, doing so will eliminate 190 the risk of unfairness. 192 4.2. Configure Non-Coupled Dual Queue 194 Equipment supporting RFC3168 may be configurable to enable two 195 parallel queues for the same traffic class, with classification done 196 based on the ECN field. 198 Option 1: 200 o Configure 2 queues, both with ECN; 50:50 WRR scheduler 202 o Queue #1: ECT1 & CE packets - Shallow immediate AQM target 204 o Queue #2: ECT0 & NotECT packets - Classic AQM target 206 o Outcome 208 * n L4S flows and m long-running Classic flows 210 * if m & n are non-zero, get 1/2n and 1/2m of the capacity, 211 otherwise 1/n or 1/m 213 * never < 1/2 each flow's rate if all had been Classic 215 Option 2: 217 o Configure 2 queues, both with AQM; 50:50 WRR scheduler 219 o Queue #1: ECT1 & NotECT packets - ECN disabled 221 o Queue #2: ECT0 & CE packets - ECN enabled 223 o Outcome 225 * ECT1 treated as NotECT 227 * Flow balance for the 2 queues the same as in option 1 229 4.3. WRED with ECT1 Differentation 231 This configuration is similar to Option 2 in the previous section, 232 but uses a single queue with WRED functionality. 234 o Configure the queue with two WRED classes 236 o Class #1: ECT1 & NotECT packets - ECN disabled 238 o Class #2: ECT0 & CE packets - ECN enabled 240 4.4. ECT1 Tunnel Bypass 242 Using an RFC6040 compatibility mode tunnel, tunnel ECT1 traffic 243 through the RFC3168 bottleneck with the outer header indicating Not- 244 ECT. 246 Two variants 248 1. per-domain: tunnel ECT1 pkts to domain edge towards dst 250 2. per-dst: tunnel ECT1 pkts to dst 252 4.5. Disable RFC3168 ECN Marking 254 While not a recommended alternative, disabling RFC3168 ECN marking 255 eliminates the fairness issue. Clearly a downside to this approach 256 is that classic senders will no longer get the benefits of Explict 257 Congestion Notification. 259 4.6. Re-mark ECT1 to NotECT Prior to AQM 261 While not a recommended alternative, remarking ECT1 packets as NotECT 262 ensures that they are treated identically to classic NotECT senders. 263 However, this also eliminates the possibility of downstream L4S 264 bottlenecks providing high fidelity congestion signals. 266 5. Researchers 268 5.1. Detection of Classic ECN FIFO Bottlenecks 270 TODO: Describe active testing methods, in-band or out-of-band, that 271 can distinguish FIFO from FQ. 273 5.2. End-to-end measurement of L4S vs. Classic performance 275 TBD 277 6. Contributors 279 Thanks to Bob Briscoe, Jake Holland, Koen De Schepper, Olivier 280 Tilmans, Tom Henderson, Asad Ahmed, and members of the TSVWG mailing 281 list for their contributions to this document. 283 7. IANA Considerations 285 None. 287 8. Security Considerations 289 None. 291 9. Informative References 293 [I-D.ietf-tsvwg-aqm-dualq-coupled] 294 Schepper, K., Briscoe, B., and G. White, "DualQ Coupled 295 AQMs for Low Latency, Low Loss and Scalable Throughput 296 (L4S)", draft-ietf-tsvwg-aqm-dualq-coupled-12 (work in 297 progress), July 2020. 299 [I-D.ietf-tsvwg-ecn-l4s-id] 300 Schepper, K. and B. Briscoe, "Identifying Modified 301 Explicit Congestion Notification (ECN) Semantics for 302 Ultra-Low Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s- 303 id-10 (work in progress), March 2020. 305 [I-D.ietf-tsvwg-l4s-arch] 306 Briscoe, B., Schepper, K., Bagnulo, M., and G. White, "Low 307 Latency, Low Loss, Scalable Throughput (L4S) Internet 308 Service: Architecture", draft-ietf-tsvwg-l4s-arch-06 (work 309 in progress), March 2020. 311 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 312 Requirement Levels", BCP 14, RFC 2119, 313 DOI 10.17487/RFC2119, March 1997, 314 . 316 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 317 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 318 May 2017, . 320 [RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, 321 J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler 322 and Active Queue Management Algorithm", RFC 8290, 323 DOI 10.17487/RFC8290, January 2018, 324 . 326 Author's Address 328 Greg White (editor) 329 CableLabs 331 Email: g.white@cablelabs.com