idnits 2.17.1 draft-ietf-tsvwg-ecn-l4s-id-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1804 has weird spacing: '...initial even...' -- The document date (July 8, 2019) is 1754 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Outdated reference: A later version (-09) exists of draft-ietf-avtcore-cc-feedback-message-03 == Outdated reference: A later version (-34) exists of draft-ietf-quic-transport-20 == Outdated reference: A later version (-28) exists of draft-ietf-tcpm-accurate-ecn-08 == Outdated reference: A later version (-15) exists of draft-ietf-tcpm-generalized-ecn-03 == Outdated reference: A later version (-15) exists of draft-ietf-tcpm-rack-05 == Outdated reference: A later version (-25) exists of draft-ietf-tsvwg-aqm-dualq-coupled-09 == Outdated reference: A later version (-22) exists of draft-ietf-tsvwg-ecn-encap-guidelines-13 == Outdated reference: A later version (-20) exists of draft-ietf-tsvwg-l4s-arch-03 == Outdated reference: A later version (-07) exists of draft-stewart-tsvwg-sctpecn-05 -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) -- Obsolete informational reference (is this intentional?): RFC 4960 (Obsoleted by RFC 9260) -- Obsolete informational reference (is this intentional?): RFC 8312 (Obsoleted by RFC 9438) Summary: 0 errors (**), 0 flaws (~~), 11 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Transport Services (tsv) K. De Schepper 3 Internet-Draft Nokia Bell Labs 4 Intended status: Experimental B. Briscoe, Ed. 5 Expires: January 9, 2020 CableLabs 6 July 8, 2019 8 Identifying Modified Explicit Congestion Notification (ECN) Semantics 9 for Ultra-Low Queuing Delay (L4S) 10 draft-ietf-tsvwg-ecn-l4s-id-07 12 Abstract 14 This specification defines the identifier to be used on IP packets 15 for a new network service called low latency, low loss and scalable 16 throughput (L4S). It is similar to the original (or 'Classic') 17 Explicit Congestion Notification (ECN). 'Classic' ECN marking was 18 required to be equivalent to a drop, both when applied in the network 19 and when responded to by a transport. Unlike 'Classic' ECN marking, 20 for packets carrying the L4S identifier, the network applies marking 21 more immediately and more aggressively than drop, and the transport 22 response to each mark is reduced and smoothed relative to that for 23 drop. The two changes counterbalance each other so that the 24 throughput of an L4S flow will be roughly the same as a 'Classic' 25 flow under the same conditions. However, the much more frequent 26 control signals and the finer responses to them result in ultra-low 27 queuing delay without compromising link utilization, and low delay is 28 maintained during high load. Examples of new active queue management 29 (AQM) marking algorithms and examples of new transports (whether TCP- 30 like or real-time) are specified separately. The new L4S identifier 31 is the key piece that enables them to interwork and distinguishes 32 them from 'Classic' traffic. It gives an incremental migration path 33 so that existing 'Classic' TCP traffic will be no worse off, but it 34 can be prevented from degrading the ultra-low delay and loss of the 35 new scalable transports. 37 Status of This Memo 39 This Internet-Draft is submitted in full conformance with the 40 provisions of BCP 78 and BCP 79. 42 Internet-Drafts are working documents of the Internet Engineering 43 Task Force (IETF). Note that other groups may also distribute 44 working documents as Internet-Drafts. The list of current Internet- 45 Drafts is at https://datatracker.ietf.org/drafts/current/. 47 Internet-Drafts are draft documents valid for a maximum of six months 48 and may be updated, replaced, or obsoleted by other documents at any 49 time. It is inappropriate to use Internet-Drafts as reference 50 material or to cite them other than as "work in progress." 52 This Internet-Draft will expire on January 9, 2020. 54 Copyright Notice 56 Copyright (c) 2019 IETF Trust and the persons identified as the 57 document authors. All rights reserved. 59 This document is subject to BCP 78 and the IETF Trust's Legal 60 Provisions Relating to IETF Documents 61 (https://trustee.ietf.org/license-info) in effect on the date of 62 publication of this document. Please review these documents 63 carefully, as they describe your rights and restrictions with respect 64 to this document. Code Components extracted from this document must 65 include Simplified BSD License text as described in Section 4.e of 66 the Trust Legal Provisions and are provided without warranty as 67 described in the Simplified BSD License. 69 Table of Contents 71 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 72 1.1. Problem . . . . . . . . . . . . . . . . . . . . . . . . . 4 73 1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 6 74 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 75 2. Consensus Choice of L4S Packet Identifier: Requirements . . . 7 76 3. L4S Packet Identification at Run-Time . . . . . . . . . . . . 8 77 4. Prerequisite Transport Layer Behaviour . . . . . . . . . . . 8 78 4.1. Prerequisite Codepoint Setting . . . . . . . . . . . . . 8 79 4.2. Prerequisite Transport Feedback . . . . . . . . . . . . . 8 80 4.3. Prerequisite Congestion Response . . . . . . . . . . . . 9 81 5. Prerequisite Network Node Behaviour . . . . . . . . . . . . . 11 82 5.1. Prerequisite Classification and Re-Marking Behaviour . . 11 83 5.2. The Meaning of L4S CE Relative to Drop . . . . . . . . . 11 84 5.3. Exception for L4S Packet Identification by Network Nodes 85 with Transport-Layer Awareness . . . . . . . . . . . . . 12 86 5.4. Interaction of the L4S Identifier with other Identifiers 13 87 5.4.1. Examples of Other Identifiers Complementing L4S 88 Identifiers . . . . . . . . . . . . . . . . . . . . . 13 89 5.4.1.1. Inclusion of Additional Traffic with L4S . . . . 13 90 5.4.1.2. Exclusion of Traffic From L4S Treatment . . . . . 14 91 5.4.2. Generalized Combination of L4S and Other Identifiers 15 92 6. L4S Experiments . . . . . . . . . . . . . . . . . . . . . . . 16 93 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 94 8. Security Considerations . . . . . . . . . . . . . . . . . . . 16 95 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 96 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 17 97 10.1. Normative References . . . . . . . . . . . . . . . . . . 17 98 10.2. Informative References . . . . . . . . . . . . . . . . . 18 99 Appendix A. The 'Prague L4S Requirements' . . . . . . . . . . . 23 100 A.1. Requirements for Scalable Transport Protocols . . . . . . 24 101 A.1.1. Use of L4S Packet Identifier . . . . . . . . . . . . 24 102 A.1.2. Accurate ECN Feedback . . . . . . . . . . . . . . . . 24 103 A.1.3. Fall back to Reno-friendly congestion control on 104 packet loss . . . . . . . . . . . . . . . . . . . . . 25 105 A.1.4. Fall back to Reno-friendly congestion control on 106 classic ECN bottlenecks . . . . . . . . . . . . . . . 25 107 A.1.5. Reduce RTT dependence . . . . . . . . . . . . . . . . 26 108 A.1.6. Scaling down to fractional congestion windows . . . . 26 109 A.1.7. Measuring Reordering Tolerance in Time Units . . . . 27 110 A.2. Scalable Transport Protocol Optimizations . . . . . . . . 29 111 A.2.1. Setting ECT in TCP Control Packets and 112 Retransmissions . . . . . . . . . . . . . . . . . . . 29 113 A.2.2. Faster than Additive Increase . . . . . . . . . . . . 30 114 A.2.3. Faster Convergence at Flow Start . . . . . . . . . . 30 115 Appendix B. Alternative Identifiers . . . . . . . . . . . . . . 31 116 B.1. ECT(1) and CE codepoints . . . . . . . . . . . . . . . . 31 117 B.2. ECN Plus a Diffserv Codepoint (DSCP) . . . . . . . . . . 34 118 B.3. ECN capability alone . . . . . . . . . . . . . . . . . . 36 119 B.4. Protocol ID . . . . . . . . . . . . . . . . . . . . . . . 38 120 B.5. Source or destination addressing . . . . . . . . . . . . 38 121 B.6. Summary: Merits of Alternative Identifiers . . . . . . . 38 122 Appendix C. Potential Competing Uses for the ECT(1) Codepoint . 39 123 C.1. Integrity of Congestion Feedback . . . . . . . . . . . . 39 124 C.2. Notification of Less Severe Congestion than CE . . . . . 40 125 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 41 127 1. Introduction 129 This specification defines the identifier to be used on IP packets 130 for a new network service called low latency, low loss and scalable 131 throughput (L4S). It is similar to the original (or 'Classic') 132 Explicit Congestion Notification (ECN [RFC3168]). 'Classic' ECN 133 marking was required to be equivalent to a drop, both when applied in 134 the network and when responded to by a transport. Unlike 'Classic' 135 ECN marking, the network applies L4S marking more immediately and 136 more aggressively than drop, and the transport response to each mark 137 is reduced and smoothed relative to that for drop. The two changes 138 counterbalance each other so that the throughput of an L4S flow will 139 be roughly the same as a 'Classic' flow under the same conditions. 140 Nonetheless, the much more frequent control signals and the finer 141 responses to them result in ultra-low queuing delay without 142 compromising link utilization, and low delay is maintained during 143 high load. 145 An example of a scalable congestion control that would enable the L4S 146 service is Data Center TCP (DCTCP), which until now has been 147 applicable solely to controlled environments like data centres 148 [RFC8257], because it is too aggressive to co-exist with existing 149 TCP. The DualQ Coupled AQM, which is defined in a complementary 150 experimental specification [I-D.ietf-tsvwg-aqm-dualq-coupled], is an 151 AQM framework that enables scalable congestion controls like DCTCP to 152 co-exist with existing traffic, each getting roughly the same flow 153 rate when they compete under similar conditions. Note that a 154 transport such as DCTCP is still not safe to deploy on the Internet 155 unless it satisfies the requirements listed in Section 4. Also note 156 that L4S is not only for elastic TCP-like traffic - there are 157 scalable congestion controls for real-time media, such as the L4S 158 variant of the SCReAM [RFC8298] real-time media congestion avoidance 159 technique (RMCAT). 161 The new L4S identifier is the key piece that enables L4S hosts and 162 L4S network nodes to interwork and distinguishes their traffic from 163 'Classic' traffic. It gives an incremental migration path so that 164 existing 'Classic' TCP traffic will be no worse off, but it can be 165 prevented from degrading the ultra-low delay and loss of the new 166 scalable congestion controls. The performance improvement is so 167 great that it is motivating initial deployment of the separate parts 168 of this system. 170 1.1. Problem 172 Latency is becoming the critical performance factor for many (most?) 173 applications on the public Internet, e.g. interactive Web, Web 174 services, voice, conversational video, interactive video, interactive 175 remote presence, instant messaging, online gaming, remote desktop, 176 cloud-based applications, and video-assisted remote control of 177 machinery and industrial processes. In the developed world, further 178 increases in access network bit-rate offer diminishing returns, 179 whereas latency is still a multi-faceted problem. In the last decade 180 or so, much has been done to reduce propagation time by placing 181 caches or servers closer to users. However, queuing remains a major 182 intermittent component of latency. 184 The Diffserv architecture provides Expedited Forwarding [RFC3246], so 185 that low latency traffic can jump the queue of other traffic. 186 However, on access links dedicated to individual sites (homes, small 187 enterprises or mobile devices), often all traffic at any one time 188 will be latency-sensitive. Then Diffserv is of little use. Instead, 189 we need to remove the causes of any unnecessary delay. 191 The bufferbloat project has shown that excessively-large buffering 192 ('bufferbloat') has been introducing significantly more delay than 193 the underlying propagation time. These delays appear only 194 intermittently--only when a capacity-seeking (e.g. TCP) flow is long 195 enough for the queue to fill the buffer, making every packet in other 196 flows sharing the buffer sit through the queue. 198 Active queue management (AQM) was originally developed to solve this 199 problem (and others). Unlike Diffserv, which gives low latency to 200 some traffic at the expense of others, AQM controls latency for _all_ 201 traffic in a class. In general, AQMs introduce an increasing level 202 of discard from the buffer the longer the queue persists above a 203 shallow threshold. This gives sufficient signals to capacity-seeking 204 (aka. greedy) flows to keep the buffer empty for its intended 205 purpose: absorbing bursts. However, RED [RFC2309] and other 206 algorithms from the 1990s were sensitive to their configuration and 207 hard to set correctly. So, AQM was not widely deployed. 209 More recent state-of-the-art AQMs, e.g. fq_CoDel [RFC8290], 210 PIE [RFC8033], Adaptive RED [ARED01], are easier to configure, 211 because they define the queuing threshold in time not bytes, so it is 212 invariant for different link rates. However, no matter how good the 213 AQM, the sawtoothing rate of TCP will either cause queuing delay to 214 vary or cause the link to be under-utilized. Even with a perfectly 215 tuned AQM, the additional queuing delay will be of the same order as 216 the underlying speed-of-light delay across the network. Flow-queuing 217 can isolate one flow from another, but it cannot isolate a TCP flow 218 from the delay variations it inflicts on itself, and it has other 219 problems - it overrides the flow rate decisions of variable rate 220 video applications, it does not recognise the flows within IPSec VPN 221 tunnels and it is relatively expensive to implement. 223 Latency is not our only concern: It was known when TCP was first 224 developed that it would not scale to high bandwidth-delay products 225 [TCP-CA]. Given regular broadband bit-rates over WAN distances are 226 already [RFC3649] beyond the scaling range of 'Classic' TCP Reno, 227 'less unscalable' Cubic [RFC8312] and 228 Compound [I-D.sridharan-tcpm-ctcp] variants of TCP have been 229 successfully deployed. However, these are now approaching their 230 scaling limits. Unfortunately, fully scalable congestion controls 231 such as DCTCP [RFC8257] cause 'Classic' TCP to starve itself, which 232 is why they have been confined to private data centres or research 233 testbeds (until now). 235 It turns out that a TCP algorithm like DCTCP that solves the latency 236 problem also solves TCP's scalability problem. The finer sawteeth 237 have low amplitude, so they cause very little queuing delay variation 238 and the number of sawteeth per round trip remains invariant, which 239 maintains constant tight control as flow-rate scales. A supporting 240 paper [DCttH15] gives the full explanation of why the design solves 241 both the latency and the scaling problems, both in plain English and 242 in more precise mathematical form. The explanation is summarised 243 without the maths in the L4S architecture document 244 [I-D.ietf-tsvwg-l4s-arch]. 246 1.2. Terminology 248 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 249 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 250 "OPTIONAL" in this document are to be interpreted as described in 251 [RFC2119]. In this document, these words will appear with that 252 interpretation only when in ALL CAPS. Lower case uses of these words 253 are not to be interpreted as carrying RFC-2119 significance. 255 Classic service: The 'Classic' service is intended for all the 256 behaviours that currently co-exist with TCP Reno (e.g. TCP Cubic, 257 Compound, SCTP, etc). 259 Low-Latency, Low-Loss and Scalable (L4S) service: The 'L4S' service 260 is intended for traffic from scalable congestion control 261 algorithms such as Data Center TCP. But it is also more general-- 262 it allows the set of congestion controls with similar scaling 263 properties to DCTCP to evolve (e.g. Relentless TCP [Mathis09] and 264 the L4S variant of SCREAM for real-time media [RFC8298]. 266 Both Classic and L4S services can cope with a proportion of 267 unresponsive or less-responsive traffic as well, as long as it 268 does not build a queue (e.g. DNS, VoIP, game sync datagrams, 269 etc). 271 Classic ECN: The original Explicit Congestion Notification (ECN) 272 protocol [RFC3168]. 274 1.3. Scope 276 The new L4S identifier defined in this specification is applicable 277 for IPv4 and IPv6 packets (as for classic ECN [RFC3168]). It is 278 applicable for the unicast, multicast and anycast forwarding modes. 280 The L4S identifier is an orthogonal packet classification to the 281 Differentiated Services Code Point (DSCP [RFC2474]). Section 5.4 282 explains what this means in practice. 284 This document is intended for experimental status, so it does not 285 update any standards track RFCs. Therefore it depends on [RFC8311], 286 which is a standards track specification that: 288 o updates the ECN proposed standard [RFC3168] to allow experimental 289 track RFCs to relax the requirement that an ECN mark must be 290 equivalent to a drop, both when applied by the network, and when 291 responded to by the sender; 293 o changes the status of the experimental ECN nonce [RFC3540] to 294 historic; 296 o makes consequent updates to the following additional proposed 297 standard RFCs to reflect the above two bullets: 299 * ECN for RTP [RFC6679]; 301 * the congestion control specifications of various DCCP 302 congestion control identifier (CCID) profiles [RFC4341], 303 [RFC4342], [RFC5622]. 305 2. Consensus Choice of L4S Packet Identifier: Requirements 307 This subsection briefly records the process that led to a consensus 308 choice of L4S identifier, selected from all the alternatives in 309 Appendix B. 311 Ideally, the identifier for packets using the Low Latency, Low Loss, 312 Scalable throughput (L4S) service ought to meet the following 313 requirements: 315 o it SHOULD survive end-to-end between source and destination 316 applications: across the boundary between host and network, 317 between interconnected networks, and through middleboxes; 319 o it SHOULD be common to IPv4 and IPv6 and transport-agnostic; 321 o it SHOULD be incrementally deployable; 323 o it SHOULD enable an AQM to classify packets encapsulated by outer 324 IP or lower-layer headers; 326 o it SHOULD consume minimal extra codepoints; 328 o it SHOULD be consistent on all the packets of a transport layer 329 flow, so that some packets of a flow are not served by a different 330 queue to others. 332 Whether the identifier would be recoverable if the experiment failed 333 is a factor that could be taken into account. However, this has not 334 been made a requirement, because that would favour schemes that would 335 be easier to fail, rather than those more likely to succeed. 337 It is recognised that the chosen identifier is unlikely to satisfy 338 all these requirements, particularly given the limited space left in 339 the IP header. Therefore a compromise will be necessary, which is 340 why all the requirements are expressed with the word 'SHOULD' not 341 'MUST'. Appendix B discusses the pros and cons of the compromises 342 made in various competing identification schemes against the above 343 requirements. 345 On the basis of this analysis, "ECT(1) and CE codepoints" is the best 346 compromise. Therefore this scheme is defined in detail in the 347 following sections, while Appendix B records the rationale for this 348 decision. 350 3. L4S Packet Identification at Run-Time 352 The L4S treatment is an experimental track alternative packet marking 353 treatment [RFC4774] to the classic ECN treatment [RFC3168], which has 354 been updated by [RFC8311] to allow this experiment (amongst others). 355 Like classic ECN, L4S ECN identifies both network and host behaviour: 356 it identifies the marking treatment that network nodes are expected 357 to apply to L4S packets, and it identifies packets that have been 358 sent from hosts that are expected to comply with a broad type of 359 sending behaviour. 361 For a packet to receive L4S treatment as it is forwarded, the sender 362 sets the ECN field in the IP header to the ECT(1) codepoint. See 363 Section 4 for full transport layer behaviour requirements, including 364 feedback and congestion response. 366 A network node that implements the L4S service normally classifies 367 arriving ECT(1) and CE packets for L4S treatment. See Section 5 for 368 full network element behaviour requirements, including 369 classification, ECN-marking and interaction of the L4S identifier 370 with other identifiers and per-hop behaviours. 372 4. Prerequisite Transport Layer Behaviour 374 4.1. Prerequisite Codepoint Setting 376 A sender that wishes a packet to receive L4S treatment as it is 377 forwarded, MUST set the ECN field in the IP header (v4 or v6) to the 378 ECT(1) codepoint. 380 4.2. Prerequisite Transport Feedback 382 In general, a scalable congestion control needs feedback of the 383 extent of CE marking on the forward path. When ECN was added to TCP 384 [RFC3168], the feedback method reported no more than one CE mark per 385 round trip. Some transport protocols derived from TCP mimic this 386 behaviour while others report the accurate extent of TCP marking. 387 This means that some transport protocols will need to be updated as a 388 prerequisite for scalable congestion control. The position for a few 389 well-known transport protocols is given below. 391 TCP: Support for accurate ECN feedback (AccECN 392 [I-D.ietf-tcpm-accurate-ecn]) by both ends is a prerequisite for 393 scalable congestion control. Therefore, the presence of ECT(1) in 394 the IP headers even in one direction of a TCP connection will 395 imply that both ends support AccECN. However, the converse does 396 not apply. So even if both ends support AccECN, either of the two 397 ends can choose not to use a scalable congestion control, whatever 398 the other end's choice. 400 SCTP: A suitable ECN feedback mechanism for SCTP could add a chunk 401 to report the number of received CE marks (e.g. 402 [I-D.stewart-tsvwg-sctpecn]), and update the ECN feedback protocol 403 sketched out in Appendix A of the standards track specification of 404 SCTP [RFC4960]. 406 RTP over UDP: A prerequisite for scalable congestion control is for 407 both (all) ends of one media-level hop to signal ECN support 408 [RFC6679] and use the new generic RTCP feedback format of 409 [I-D.ietf-avtcore-cc-feedback-message]. The presence of ECT(1) 410 implies that both (all) ends of that hop support ECN. However, 411 the converse does not apply, so each end of a media-level hop can 412 independently choose not to use a scalable congestion control, 413 even if both ends support ECN. 415 QUIC: Support for sufficiently fine-grained ECN feedback is provided 416 by the first IETF QUIC transport [I-D.ietf-quic-transport]. 418 DCCP: The ACK vector in DCCP [RFC4340] is already sufficient to 419 report the extent of CE marking as needed by a scalable congestion 420 control. 422 4.3. Prerequisite Congestion Response 424 As a condition for a host to send packets with the L4S identifier 425 (ECT(1)), it SHOULD implement a congestion control behaviour that 426 ensures the flow rate is inversely proportional to the proportion of 427 bytes in packets marked with the CE codepoint. This is termed a 428 scalable congestion control, because the average number of control 429 signals (ECN marks) per round trip remains roughly constant for any 430 flow rate. As with all transport behaviours, a detailed 431 specification will need to be defined for each type of transport or 432 application, including the timescale over which the proportionality 433 is averaged, and control of burstiness. The inverse proportionality 434 requirement above is worded as a 'SHOULD' rather than a 'MUST' to 435 allow reasonable flexibility when defining these specifications. 437 Data Center TCP (DCTCP [RFC8257]) and the L4S variant of SCReAM 438 [RFC8298] are examples of a scalable congestion controls. 440 Each sender in a session can use a scalable congestion control 441 independently of the congestion control used by the receiver(s) when 442 they send data. Therefore there might be ECT(1) packets in one 443 direction and ECT(0) or Not-ECT in the other. 445 In order to coexist safely with other Internet traffic, a scalable 446 congestion control MUST NOT tag its packets with the ECT(1) codepoint 447 unless it complies with the following bulleted requirements. The 448 specification of a particular scalable congestion control MUST 449 describe in detail how it satisfies each requirement: 451 o A scalable congestion control MUST react to packet loss in a way 452 that will coexist safely with a TCP Reno congestion control 453 [RFC5681] (see Appendix A.1.3 for rationale). 455 o A scalable congestion control MUST react to ECN marking from a 456 non-L4S but ECN-capable bottleneck in a way that will coexist with 457 a TCP Reno congestion control [RFC5681] (see Appendix A.1.4 for 458 rationale). 460 Note that a scalable congestion control is not expected to change 461 to setting ECT(0) while it temporarily falls back to coexist with 462 Reno . However an implementer who believes this would be 463 beneficial if fall-back persists, can choose to do so, 465 o A scalable congestion control MUST reduce or eliminate RTT bias 466 over as wide a range of RTTs as possible, or at least over the 467 typical range of RTTs that will interact in the intended 468 deployment scenario (see Appendix A.1.5 for rationale). 470 o A scalable congestion control MUST remain responsive to congestion 471 when the RTT is significantly smaller than in the current public 472 Internet (see Appendix A.1.6 for rationale). 474 o A scalable congestion control MUST detect loss by counting in 475 time-based units, which is scalable, as opposed to counting in 476 units of packets (as in the 3 DupACK rule of traditional TCP), 477 which is not scalable (see Appendix A.1.7 for rationale). 479 As well as traffic controlled by a scalable congestion control, a 480 reasonable level of smooth unresponsive traffic at a low rate 481 relative to typical broadband capacities is likely to be acceptable 482 (see "'Safe' Unresponsive Traffic" in Section 5.4.1.1.1). 484 5. Prerequisite Network Node Behaviour 486 5.1. Prerequisite Classification and Re-Marking Behaviour 488 A network node that implements the L4S service MUST classify arriving 489 ECT(1) packets for L4S treatment and, other than in the exceptional 490 case referred to next, it MUST classify arriving CE packets for L4S 491 treatment as well. CE packets might have originated as ECT(1) or 492 ECT(0), but the above rule to classify them as if they originated as 493 ECT(1) is the safe choice (see Appendix B.1 for rationale). The 494 exception is where some flow-aware in-network mechanism happens to be 495 available for distinguishing CE packets that originated as ECT(0), as 496 described in Section 5.3, but there is no implication that such a 497 mechanism is necessary. 499 An L4S AQM treatment follows similar codepoint transition rules to 500 those in RFC 3168. Specifically, the ECT(1) codepoint MUST NOT be 501 changed to any other codepoint than CE, and CE MUST NOT be changed to 502 any other codepoint. An ECT(1) packet is classified as ECN-capable 503 and, if congestion increases, an L4S AQM algorithm will increasingly 504 mark the ECN field as CE, otherwise forwarding packets unchanged as 505 ECT(1). Necessary conditions for an L4S marking treatment are 506 defined in Section 5.2. Under persistent overload an L4S marking 507 treatment SHOULD turn off ECN marking, using drop as a congestion 508 signal until the overload episode has subsided, as recommended for 509 all AQMs in [RFC7567] (Section 4.2.1), which follows the similar 510 advice in RFC 3168 (Section 7). 512 For backward compatibility in uncontrolled environments, a network 513 node that implements the L4S treatment MUST also implement a classic 514 AQM treatment. It MUST classify arriving ECT(0) and Not-ECT packets 515 for treatment by the Classic AQM (see the discussion of the 516 classifier for the dual-queue coupled AQM in 517 [I-D.ietf-tsvwg-aqm-dualq-coupled]). Classic treatment means that 518 the AQM will mark ECT(0) packets under the same conditions as it 519 would drop Not-ECT packets [RFC3168]. 521 5.2. The Meaning of L4S CE Relative to Drop 523 The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST 524 be roughly proportional to the square of the likelihood that it would 525 have marked it if it had been an L4S packet (p_L). That is 527 p_C ~= (p_L / k)^2 529 The constant of proportionality (k) does not have to be standardised 530 for interoperability, but a value of 2 is RECOMMENDED. 532 This formula ensures that Scalable and Classic flows will converge to 533 roughly equal congestion windows. This is because the congestion 534 windows of Scalable and Classic congestion controls are inversely 535 proportional to p_L and sqrt(p_C) respectively. So squaring p_C in 536 the above formula counterbalances the square root that characterizes 537 all TCP-friendly flows. 539 [I-D.ietf-tsvwg-aqm-dualq-coupled] specifies the essential aspects of 540 an L4S AQM, as well as recommending other aspects. It gives example 541 implementations in appendices. 543 The term 'likelihood' is used above to allow for marking and dropping 544 to be either probabilistic or deterministic. The example AQMs in 545 [I-D.ietf-tsvwg-aqm-dualq-coupled] drop and mark probabilistically, 546 so the drop probability is arranged to be the square of the marking 547 probability. Nonetheless, an alternative AQM that dropped and marked 548 deterministically would be valid, as long as the dropping frequency 549 was proportional to the square of the marking frequency. 551 Note that, contrary to RFC 3168, a Dual AQM implementing the L4S and 552 Classic treatments does not mark an ECT(1) packet under the same 553 conditions that it would have dropped a Not-ECT packet, as allowed by 554 [RFC8311], which updates RFC 3168. However, it does mark an ECT(0) 555 packet under the same conditions that it would have dropped a Not-ECT 556 packet. 558 5.3. Exception for L4S Packet Identification by Network Nodes with 559 Transport-Layer Awareness 561 To implement the L4S treatment, a network node does not need to 562 identify transport-layer flows. Nonetheless, if an implementer is 563 willing to identify transport-layer flows at a network node, and if 564 the most recent ECT packet in the same flow was ECT(0), the node MAY 565 classify CE packets for classic ECN [RFC3168] treatment. In all 566 other cases, a network node MUST classify all CE packets for L4S 567 treatment. Examples of such other cases are: i) if no ECT packets 568 have yet been identified in a flow; ii) if it is not desirable for a 569 network node to identify transport-layer flows; or iii) if the most 570 recent ECT packet in a flow was ECT(1). 572 If an implementer uses flow-awareness to classify CE packets, to 573 determine whether the flow is using ECT(0) or ECT(1) it only uses the 574 most recent ECT packet of a flow (this advice will need to be 575 verified as part of L4S experiments). This is because a sender might 576 switch from sending ECT(1) (L4S) packets to sending ECT(0) (Classic) 577 packets, or back again, in the middle of a transport-layer flow (e.g. 578 it might manually switch its congestion control module mid- 579 connection, or it might be deliberately attempting to confuse the 580 network). 582 5.4. Interaction of the L4S Identifier with other Identifiers 584 5.4.1. Examples of Other Identifiers Complementing L4S Identifiers 586 5.4.1.1. Inclusion of Additional Traffic with L4S 588 In a typical case for the public Internet a network element that 589 implements L4S might want to classify some low-rate but unresponsive 590 traffic (e.g. DNS, LDAP, NTP, voice, game sync packets) into the low 591 latency queue to mix with L4S traffic. Such non-ECN-based packet 592 types MUST be safe to mix with L4S traffic without harming the low 593 latency service, where 'safe' is explained in Section 5.4.1.1.1 594 below. 596 In this case it would not be appropriate to call the queue an L4S 597 queue, because it is shared by L4S and non-L4S traffic. Instead it 598 will be called the low latency or L queue. The L queue then offers 599 two different treatments: 601 o The L4S treatment, which is a combination of the L4S AQM treatment 602 and a priority scheduling treatment; 604 o The low latency treatment, which is solely the priority scheduling 605 treatment, without ECN-marking by the AQM. 607 To identify packets for just the scheduling treatment, it would be 608 inappropriate to use the L4S ECT(1) identifier, because such traffic 609 is unresponsive to ECN marking. Therefore, a network element that 610 implements L4S MAY classify additional packets into the L queue if 611 they carry certain non-ECN identifiers. For instance: 613 o addresses of specific applications or hosts configured to be safe 614 (but for example cannot set the ECN field for some temporary 615 reason); 617 o certain protocols that are usually lightweight (e.g. ARP, DNS); 619 o specific Diffserv codepoints that indicate traffic with limited 620 burstiness such as the EF (Expedited Forwarding [RFC3246]), Voice- 621 Admit [RFC5865] or proposed NQB (Non-Queue-Building 622 [I-D.white-tsvwg-nqb]) service classes or equivalent local-use 623 DSCPs (see [I-D.briscoe-tsvwg-l4s-diffserv]). 625 Of course, a packet that carried both the ECT(1) codepoint and a 626 relevant non-ECN identifier would also be classified into the L 627 queue. 629 For clarity, non-ECN identifiers, such as the examples itemized 630 above, might be used by some network operators who believe they 631 identify non-L4S traffic that would be safe to mix with L4S traffic. 632 They are not alternative ways for a host to indicate that it is 633 sending L4S packets. Only the ECT(1) ECN codepoint indicates to a 634 network element that a host is sending L4S packets (and CE indicates 635 that it could be). Specifically ECT(1) indicates that the host 636 claims its behaviour satisfies the per-requisite transport 637 requirements in Section 4. 639 To include additional traffic with L4S, a network element only reads 640 identifiers such as those itemized above. It MUST NOT alter these 641 non-ECN identifiers. 643 5.4.1.1.1. 'Safe' Unresponsive Traffic 645 The above section requires unresponsive traffic to be 'safe' to mix 646 with L4S traffic. Ideally this means that the sender never sends any 647 sequence of packets at a data rate that exceeds the available 648 capacity of the bottleneck link. However, typically an unresponsive 649 transport does not even know the bottleneck capacity of the path, let 650 alone its available capacity. Nonetheless, an application can be 651 considered safe enough if it paces packets out (not necessarily 652 completely regularly) such that its maximum instantaneous data rate 653 from packet to packet stays well below a typical broadband access 654 rate. 656 This is a vague but useful definition, because it encompasses many 657 low latency applications of interest, such as DNS, voice, game sync 658 packets, RPC, ACKs, keep-alives, etc. 660 5.4.1.2. Exclusion of Traffic From L4S Treatment 662 To extend the above example, an operator might want to exclude some 663 traffic from the L4S treatment for policy reason, e.g. security 664 (traffic from malicious sources) or commercial (e.g. initially the 665 operator may wish to confine the benefits of L4S to business 666 customers). 668 In this exclusion case, the operator MUST classify on the relevant 669 locally-used identifiers (e.g. source addresses) before classifying 670 the non-matching traffic on the end-to-end L4S ECN identifier. 672 The operator MUST NOT alter the end-to-end L4S ECN identifier, 673 because its decision to exclude certain traffic from L4S treatment is 674 local-only. The end-to-end L4S identifier then survives for other 675 operators to use, or indeed, they can apply their own policy, 676 independently based on their own choice of locally-used identifiers. 677 This approach also allows any operator to remove its locally-applied 678 exclusions in future, e.g. if it wishes to widen the benefit of the 679 L4S treatment to all its customers. 681 5.4.2. Generalized Combination of L4S and Other Identifiers 683 L4S concerns low latency, which it can provide for all traffic 684 without differentiation and without affecting bandwidth allocation. 685 Diffserv provides for differentiation of both bandwidth and low 686 latency, but its control of latency depends on its control of 687 bandwidth. The two can be combined if a network operator wants to 688 control bandwidth allocation but it also wants to provide low latency 689 - for any amount of traffic within one of these allocations of 690 bandwidth (rather than only providing low latency by limiting 691 bandwidth) [I-D.briscoe-tsvwg-l4s-diffserv]. 693 The examples above were framed in the context of providing the 694 default Best Efforts Per-Hop Behaviour (PHB) using two queues - a Low 695 Latency (L) queue and a Classic (C) Queue. This single DualQ 696 structure is expected to be by far the most common and useful 697 arrangement. But, more generally, an operator might choose to 698 control bandwidth allocation through a hierarchy of Diffserv PHBs at 699 a node, and to offer one (or more) of these PHBs with a low latency 700 and a classic variant. 702 In the first case, if we assume that there are no other PHBs except 703 the DualQ, if a packet carries ECT(1) or CE, a network element would 704 classify it for the L4S treatment irrespective of its DSCP. And, if 705 a packet carried (say) the EF DSCP, the network element could 706 classify it into the L queue irrespective of its ECN codepoint. 707 However, where the DualQ is in a hierarchy of other PHBs, the 708 classifier would classify some traffic into other PHBs based on DSCP 709 before classifying between the latency and classic queues (based on 710 ECT(1), CE and perhaps also the EF DSCP or other identifiers as in 711 the above example). [I-D.briscoe-tsvwg-l4s-diffserv] gives a number 712 of examples of such arrangements to address various requirements. 714 [I-D.briscoe-tsvwg-l4s-diffserv] describes how an operator might use 715 L4S to offer low latency for all L4S traffic as well as using 716 Diffserv for bandwidth differentiation. It identifies two main types 717 of approach, which can be combined: the operator might split certain 718 Diffserv PHBs between L4S and a corresponding Classic service. Or it 719 might split the L4S and/or the Classic service into multiple Diffserv 720 PHBs. In any of these cases, a packet would have to be classified on 721 its Diffserv and ECN codepoints. 723 In summary, there are numerous ways in which the L4S ECN identifier 724 (ECT(1) and CE) could be combined with other identifiers to achieve 725 particular objectives. The following categorization articulates 726 those that are valid, but it is not necessarily exhaustive. Those 727 tagged 'Recommended-standard-use' could be set by the sending host or 728 a network. Those tagged 'Local-use' would only be set by a network: 730 1. Identifiers Complementing the L4S Identifier 732 A. Including More Traffic in the L Queue 733 (Recommended-standard-use or Local-use) 735 B. Excluding Certain Traffic from the L Queue 736 (Local-use only) 738 2. Identifiers to place L4S classification in a PHB Hierarchy 739 (Recommended-standard-use or Local-use) 741 A. PHBs Before L4S ECN Classification 743 B. PHBs After L4S ECN Classification 745 6. L4S Experiments 747 [I-D.ietf-tsvwg-aqm-dualq-coupled] sets operational and management 748 requirements for experiments with DualQ Coupled AQMs. General 749 operational and management requirements for experiments with L4S 750 congestion controls are given in Section 4 and Section 5 above, e.g. 751 co-existence and scaling requirements, incremental deployment 752 arrangements. The specification of each scalable congestion control 753 will need to include protocol-specific requirements for configuration 754 and monitoring performance during experiments. Appendix A of 755 [RFC5706] provides a helpful checklist. 757 7. IANA Considerations 759 This specification contains no IANA considerations. 761 8. Security Considerations 763 Approaches to assure the integrity of signals using the new identifer 764 are introduced in Appendix C.1. 766 The requirement to detect loss in time units prevents the ACK- 767 splitting attacks described in [Savage-TCP]. 769 9. Acknowledgements 771 Thanks to Richard Scheffenegger, John Leslie, David Taeht, Jonathan 772 Morton, Gorry Fairhurst, Michael Welzl, Mikael Abrahamsson and Andrew 773 McGregor for the discussions that led to this specification. Ing-jyh 774 (Inton) Tsang was a contributor to the early drafts of this document. 775 And thanks to Mikael Abrahamsson, Lloyd Wood, Nicolas Kuhn, Greg 776 White, Tom Henderson, David Black, Gorry Fairhurst and Brian 777 Carpenter for providing help and reviewing this draft and to Ingemar 778 Johansson for reviewing and providing substantial text. Appendix A 779 listing the Prague L4S Requirements is based on text authored by 780 Marcelo Bagnulo Braun that was originally an appendix to 781 [I-D.ietf-tsvwg-l4s-arch]. That text was in turn based on the 782 collective output of the attendees listed in the minutes of a 'bar 783 BoF' on DCTCP Evolution during IETF-94 [TCPPrague]. 785 The authors' contributions were part-funded by the European Community 786 under its Seventh Framework Programme through the Reducing Internet 787 Transport Latency (RITE) project (ICT-317700). Bob Briscoe was also 788 part-funded by the Research Council of Norway through the TimeIn 789 project. The views expressed here are solely those of the authors. 791 10. References 793 10.1. Normative References 795 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 796 Requirement Levels", BCP 14, RFC 2119, 797 DOI 10.17487/RFC2119, March 1997, 798 . 800 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 801 of Explicit Congestion Notification (ECN) to IP", 802 RFC 3168, DOI 10.17487/RFC3168, September 2001, 803 . 805 [RFC4774] Floyd, S., "Specifying Alternate Semantics for the 806 Explicit Congestion Notification (ECN) Field", BCP 124, 807 RFC 4774, DOI 10.17487/RFC4774, November 2006, 808 . 810 [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., 811 and K. Carlberg, "Explicit Congestion Notification (ECN) 812 for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 813 2012, . 815 10.2. Informative References 817 [Alizadeh-stability] 818 Alizadeh, M., Javanmard, A., and B. Prabhakar, "Analysis 819 of DCTCP: Stability, Convergence, and Fairness", ACM 820 SIGMETRICS 2011 , June 2011. 822 [ARED01] Floyd, S., Gummadi, R., and S. Shenker, "Adaptive RED: An 823 Algorithm for Increasing the Robustness of RED's Active 824 Queue Management", ACIRI Technical Report , August 2001, 825 . 827 [DCttH15] De Schepper, K., Bondarenko, O., Briscoe, B., and I. 828 Tsang, "'Data Centre to the Home': Ultra-Low Latency for 829 All", RITE Project Technical Report , 2015, 830 . 832 [I-D.briscoe-tsvwg-l4s-diffserv] 833 Briscoe, B., "Interactions between Low Latency, Low Loss, 834 Scalable Throughput (L4S) and Differentiated Services", 835 draft-briscoe-tsvwg-l4s-diffserv-02 (work in progress), 836 November 2018. 838 [I-D.ietf-avtcore-cc-feedback-message] 839 Sarker, Z., Perkins, C., Singh, V., and M. Ramalho, "RTP 840 Control Protocol (RTCP) Feedback for Congestion Control", 841 draft-ietf-avtcore-cc-feedback-message-03 (work in 842 progress), December 2018. 844 [I-D.ietf-quic-transport] 845 Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed 846 and Secure Transport", draft-ietf-quic-transport-20 (work 847 in progress), April 2019. 849 [I-D.ietf-tcpm-accurate-ecn] 850 Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More 851 Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate- 852 ecn-08 (work in progress), March 2019. 854 [I-D.ietf-tcpm-generalized-ecn] 855 Bagnulo, M. and B. Briscoe, "ECN++: Adding Explicit 856 Congestion Notification (ECN) to TCP Control Packets", 857 draft-ietf-tcpm-generalized-ecn-03 (work in progress), 858 October 2018. 860 [I-D.ietf-tcpm-rack] 861 Cheng, Y., Cardwell, N., Dukkipati, N., and P. Jha, "RACK: 862 a time-based fast loss detection algorithm for TCP", 863 draft-ietf-tcpm-rack-05 (work in progress), April 2019. 865 [I-D.ietf-tsvwg-aqm-dualq-coupled] 866 Schepper, K., Briscoe, B., and G. White, "DualQ Coupled 867 AQMs for Low Latency, Low Loss and Scalable Throughput 868 (L4S)", draft-ietf-tsvwg-aqm-dualq-coupled-09 (work in 869 progress), July 2019. 871 [I-D.ietf-tsvwg-ecn-encap-guidelines] 872 Briscoe, B., Kaippallimalil, J., and P. Thaler, 873 "Guidelines for Adding Congestion Notification to 874 Protocols that Encapsulate IP", draft-ietf-tsvwg-ecn- 875 encap-guidelines-13 (work in progress), May 2019. 877 [I-D.ietf-tsvwg-l4s-arch] 878 Briscoe, B., Schepper, K., and M. Bagnulo, "Low Latency, 879 Low Loss, Scalable Throughput (L4S) Internet Service: 880 Architecture", draft-ietf-tsvwg-l4s-arch-03 (work in 881 progress), October 2018. 883 [I-D.sridharan-tcpm-ctcp] 884 Sridharan, M., Tan, K., Bansal, D., and D. Thaler, 885 "Compound TCP: A New TCP Congestion Control for High-Speed 886 and Long Distance Networks", draft-sridharan-tcpm-ctcp-02 887 (work in progress), November 2008. 889 [I-D.stewart-tsvwg-sctpecn] 890 Stewart, R., Tuexen, M., and X. Dong, "ECN for Stream 891 Control Transmission Protocol (SCTP)", draft-stewart- 892 tsvwg-sctpecn-05 (work in progress), January 2014. 894 [I-D.white-tsvwg-nqb] 895 White, G. and T. Fossati, "Identifying and Handling Non 896 Queue Building Flows in a Bottleneck Link", draft-white- 897 tsvwg-nqb-02 (work in progress), June 2019. 899 [Mathis09] 900 Mathis, M., "Relentless Congestion Control", PFLDNeT'09 , 901 May 2009, . 904 [Paced-Chirping] 905 Misund, J., "Rapid Acceleration in TCP Prague", Masters 906 Thesis , May 2018, 907 . 910 [QV] Briscoe, B. and P. Hurtig, "Up to Speed with Queue View", 911 RITE Technical Report D2.3; Appendix C.2, August 2015, 912 . 915 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 916 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 917 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 918 S., Wroclawski, J., and L. Zhang, "Recommendations on 919 Queue Management and Congestion Avoidance in the 920 Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998, 921 . 923 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 924 "Definition of the Differentiated Services Field (DS 925 Field) in the IPv4 and IPv6 Headers", RFC 2474, 926 DOI 10.17487/RFC2474, December 1998, 927 . 929 [RFC2983] Black, D., "Differentiated Services and Tunnels", 930 RFC 2983, DOI 10.17487/RFC2983, October 2000, 931 . 933 [RFC3246] Davie, B., Charny, A., Bennet, J., Benson, K., Le Boudec, 934 J., Courtney, W., Davari, S., Firoiu, V., and D. 935 Stiliadis, "An Expedited Forwarding PHB (Per-Hop 936 Behavior)", RFC 3246, DOI 10.17487/RFC3246, March 2002, 937 . 939 [RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit 940 Congestion Notification (ECN) Signaling with Nonces", 941 RFC 3540, DOI 10.17487/RFC3540, June 2003, 942 . 944 [RFC3649] Floyd, S., "HighSpeed TCP for Large Congestion Windows", 945 RFC 3649, DOI 10.17487/RFC3649, December 2003, 946 . 948 [RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram 949 Congestion Control Protocol (DCCP)", RFC 4340, 950 DOI 10.17487/RFC4340, March 2006, 951 . 953 [RFC4341] Floyd, S. and E. Kohler, "Profile for Datagram Congestion 954 Control Protocol (DCCP) Congestion Control ID 2: TCP-like 955 Congestion Control", RFC 4341, DOI 10.17487/RFC4341, March 956 2006, . 958 [RFC4342] Floyd, S., Kohler, E., and J. Padhye, "Profile for 959 Datagram Congestion Control Protocol (DCCP) Congestion 960 Control ID 3: TCP-Friendly Rate Control (TFRC)", RFC 4342, 961 DOI 10.17487/RFC4342, March 2006, 962 . 964 [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", 965 RFC 4960, DOI 10.17487/RFC4960, September 2007, 966 . 968 [RFC5562] Kuzmanovic, A., Mondal, A., Floyd, S., and K. 969 Ramakrishnan, "Adding Explicit Congestion Notification 970 (ECN) Capability to TCP's SYN/ACK Packets", RFC 5562, 971 DOI 10.17487/RFC5562, June 2009, 972 . 974 [RFC5622] Floyd, S. and E. Kohler, "Profile for Datagram Congestion 975 Control Protocol (DCCP) Congestion ID 4: TCP-Friendly Rate 976 Control for Small Packets (TFRC-SP)", RFC 5622, 977 DOI 10.17487/RFC5622, August 2009, 978 . 980 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 981 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 982 . 984 [RFC5706] Harrington, D., "Guidelines for Considering Operations and 985 Management of New Protocols and Protocol Extensions", 986 RFC 5706, DOI 10.17487/RFC5706, November 2009, 987 . 989 [RFC5865] Baker, F., Polk, J., and M. Dolly, "A Differentiated 990 Services Code Point (DSCP) for Capacity-Admitted Traffic", 991 RFC 5865, DOI 10.17487/RFC5865, May 2010, 992 . 994 [RFC5925] Touch, J., Mankin, A., and R. Bonica, "The TCP 995 Authentication Option", RFC 5925, DOI 10.17487/RFC5925, 996 June 2010, . 998 [RFC6077] Papadimitriou, D., Ed., Welzl, M., Scharf, M., and B. 999 Briscoe, "Open Research Issues in Internet Congestion 1000 Control", RFC 6077, DOI 10.17487/RFC6077, February 2011, 1001 . 1003 [RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three 1004 Pre-Congestion Notification (PCN) States in the IP Header 1005 Using a Single Diffserv Codepoint (DSCP)", RFC 6660, 1006 DOI 10.17487/RFC6660, July 2012, 1007 . 1009 [RFC7560] Kuehlewind, M., Ed., Scheffenegger, R., and B. Briscoe, 1010 "Problem Statement and Requirements for Increased Accuracy 1011 in Explicit Congestion Notification (ECN) Feedback", 1012 RFC 7560, DOI 10.17487/RFC7560, August 2015, 1013 . 1015 [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF 1016 Recommendations Regarding Active Queue Management", 1017 BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, 1018 . 1020 [RFC7713] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) 1021 Concepts, Abstract Mechanism, and Requirements", RFC 7713, 1022 DOI 10.17487/RFC7713, December 2015, 1023 . 1025 [RFC8033] Pan, R., Natarajan, P., Baker, F., and G. White, 1026 "Proportional Integral Controller Enhanced (PIE): A 1027 Lightweight Control Scheme to Address the Bufferbloat 1028 Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017, 1029 . 1031 [RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L., 1032 and G. Judd, "Data Center TCP (DCTCP): TCP Congestion 1033 Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257, 1034 October 2017, . 1036 [RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, 1037 J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler 1038 and Active Queue Management Algorithm", RFC 8290, 1039 DOI 10.17487/RFC8290, January 2018, 1040 . 1042 [RFC8298] Johansson, I. and Z. Sarker, "Self-Clocked Rate Adaptation 1043 for Multimedia", RFC 8298, DOI 10.17487/RFC8298, December 1044 2017, . 1046 [RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion 1047 Notification (ECN) Experimentation", RFC 8311, 1048 DOI 10.17487/RFC8311, January 2018, 1049 . 1051 [RFC8312] Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and 1052 R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", 1053 RFC 8312, DOI 10.17487/RFC8312, February 2018, 1054 . 1056 [Savage-TCP] 1057 Savage, S., Cardwell, N., Wetherall, D., and T. Anderson, 1058 "TCP Congestion Control with a Misbehaving Receiver", ACM 1059 SIGCOMM Computer Communication Review 29(5):71--78, 1060 October 1999. 1062 [TCP-CA] Jacobson, V. and M. Karels, "Congestion Avoidance and 1063 Control", Laurence Berkeley Labs Technical Report , 1064 November 1988, . 1066 [TCP-sub-mss-w] 1067 Briscoe, B. and K. De Schepper, "Scaling TCP's Congestion 1068 Window for Small Round Trip Times", BT Technical Report 1069 TR-TUB8-2015-002, May 2015, 1070 . 1073 [TCPPrague] 1074 Briscoe, B., "Notes: DCTCP evolution 'bar BoF': Tue 21 Jul 1075 2015, 17:40, Prague", tcpprague mailing list archive , 1076 July 2015, . 1079 [VCP] Xia, Y., Subramanian, L., Stoica, I., and S. Kalyanaraman, 1080 "One more bit is enough", Proc. SIGCOMM'05, ACM CCR 1081 35(4)37--48, 2005, 1082 . 1084 Appendix A. The 'Prague L4S Requirements' 1086 This appendix is informative, not normative. It gives a list of 1087 modifications to current scalable congestion controls so that they 1088 can be deployed over the public Internet and coexist safely with 1089 existing traffic. The list complements the normative requirements in 1090 Section 4 that a sender has to comply with before it can set the L4S 1091 identifier in packets it sends into the Internet. As well as 1092 necessary safety improvements (requirements) this appendix also 1093 includes preferable performance improvements (optimizations). 1095 These recommendations have become know as the Prague L4S 1096 Requirements, because they were originally identified at an ad hoc 1097 meeting during IETF-94 in Prague [TCPPrague]. The wording has been 1098 generalized to apply to all scalable congestion controls, not just 1099 TCP congestion control specifically. They were originally called the 1100 'TCP Prague Requirements', but they are not solely applicable to TCP, 1101 so the name has been generalized, and TCP Prague is now used for a 1102 specific implementation of the requirements. 1104 DCTCP [RFC8257] is currently the most widely used scalable transport 1105 protocol. In its current form, DCTCP is specified to be deployable 1106 only in controlled environments. Deploying it in the public Internet 1107 would lead to a number of issues, both from the safety and the 1108 performance perspective. The modifications and additional mechanisms 1109 listed in this section will be necessary for its deployment over the 1110 global Internet. Where an example is needed, DCTCP is used as a 1111 base, but it is likely that most of these requirements equally apply 1112 to other scalable congestion controls. 1114 A.1. Requirements for Scalable Transport Protocols 1116 A.1.1. Use of L4S Packet Identifier 1118 Description: A scalable congestion control needs to distinguish the 1119 packets it sends from those sent by classic congestion controls. 1121 Motivation: It needs to be possible for a network node to classify 1122 L4S packets without flow state into a queue that applies an L4S ECN 1123 marking behaviour and isolates L4S packets from the queuing delay of 1124 classic packets. 1126 A.1.2. Accurate ECN Feedback 1128 Description: The transport protocol for a scalable congestion control 1129 needs to provide timely, accurate feedback about the extent of ECN 1130 marking experienced by all packets. 1132 Motivation: Classic congestion controls only need feedback about the 1133 existence of a congestion episode within a round trip, not precisely 1134 how many packets were marked with ECN or dropped. Therefore, in 1135 2001, when ECN feedback was added to TCP [RFC3168], it could not 1136 inform the sender of more than one ECN mark per RTT. Since then, 1137 requirements for more accurate ECN feedback in TCP have been defined 1138 in [RFC7560] and [I-D.ietf-tcpm-accurate-ecn] specifies an 1139 experimental change to the TCP wire protocol to satisfy these 1140 requirements. Most other transport protocols already satisfy this 1141 requirement. 1143 A.1.3. Fall back to Reno-friendly congestion control on packet loss 1145 Description: A scalable congestion control needs to react to packet 1146 loss in a way that will coexist safely with a TCP Reno congestion 1147 control [RFC5681]. 1149 Motivation: Part of the safety conditions for deploying a scalable 1150 congestion control on the public Internet is to make sure that it 1151 behaves properly when it builds a queue at a network bottleneck that 1152 has not been upgraded to support L4S. Packet loss can have many 1153 causes, but it usually has to be conservatively assumed that it is a 1154 sign of congestion. Therefore, on detecting packet loss, a scalable 1155 congestion control will need to fall back to classic congestion 1156 control behaviour. If it does not comply with this requirement it 1157 could starve classic traffic. 1159 A scalable congestion control can be used for different types of 1160 transport, e.g. for real-time media or for reliable bulk transport 1161 like TCP. Therefore, the particular classic congestion control 1162 behaviour to fall back on will need to be part of the congestion 1163 control specification of the relevant transport. In the particular 1164 case of DCTCP, the current DCTCP specification states that "It is 1165 RECOMMENDED that an implementation deal with loss episodes in the 1166 same way as conventional TCP." For safe deployment of a scalable 1167 congestion control in the public Internet, the above requirement 1168 would need to be defined as a "MUST". 1170 Packet loss might (rarely) occur in the case that the bottleneck is 1171 L4S capable. In this case, the sender may receive a high number of 1172 packets marked with the CE bit set and also experience a loss. 1173 Current DCTCP implementations react differently to this situation. 1174 At least one implementation reacts only to the drop signal (e.g. by 1175 halving the CWND) and at least another DCTCP implementation reacts to 1176 both signals (e.g. by halving the CWND due to the drop and also 1177 further reducing the CWND based on the proportion of marked packet). 1178 We believe that further experimentation is needed to understand what 1179 is the best behaviour for the public Internet, which may or not be 1180 one of these existing approaches. 1182 A.1.4. Fall back to Reno-friendly congestion control on classic ECN 1183 bottlenecks 1185 Description: A scalable congestion control needs to react to ECN 1186 marking from a non-L4S but ECN-capable bottleneck in a way that will 1187 coexist with a TCP Reno congestion control [RFC5681]. 1189 Motivation: Similarly to the requirement in Appendix A.1.3, this 1190 requirement is a safety condition to ensure a scalable congestion 1191 control behaves properly when it builds a queue at a network 1192 bottleneck that has not been upgraded to support L4S. On detecting 1193 classic ECN marking (see below), a scalable congestion control will 1194 need to fall back to classic congestion control behaviour. If it 1195 does not comply with this requirement it could starve classic 1196 traffic. 1198 It would take time for endpoints to distinguish classic and L4S ECN 1199 marking. An increase in queuing delay or in delay variation would be 1200 a tell-tale sign, but it is not yet clear where a line would be drawn 1201 between the two behaviours. It might be possible to cache what was 1202 learned about the path to help subsequent attempts to detect the type 1203 of marking. 1205 A.1.5. Reduce RTT dependence 1207 Description: A scalable congestion control needs to reduce or 1208 eliminate RTT bias over as wide a range of RTTs as possible, or at 1209 least over the typical range of RTTs that will interact in the 1210 intended deployment scenario. 1212 Motivation: Classic TCP's throughput is known to be inversely 1213 proportional to RTT, so one would expect flows over very low RTT 1214 paths to nearly starve flows over larger RTTs. However, Classic TCP 1215 has never allowed a very low RTT path to exist because it induces a 1216 large queue. For instance, consider two paths with base RTT 1ms and 1217 100ms. If Classic TCP induces a 100ms queue, it turns these RTTs 1218 into 101ms and 200ms leading to a throughput ratio of about 2:1. 1219 Whereas if a Scalable TCP induces only a 1ms queue, the ratio is 1220 2:101, leading to a throughput ratio of about 50:1. 1222 Therefore, with very small queues, long RTT flows will essentially 1223 starve, unless scalable congestion controls comply with this 1224 requirement. 1226 A.1.6. Scaling down to fractional congestion windows 1228 Description: A scalable congestion control needs to remain responsive 1229 to congestion when RTTs are significantly smaller than in the current 1230 public Internet. 1232 Motivation: As currently specified, the minimum required congestion 1233 window of TCP (and its derivatives) is set to 2 maximum segment sizes 1234 (MSS) (see equation (4) in [RFC5681]). Once the congestion window 1235 reaches this minimum, all current TCP algorithms become unresponsive 1236 to congestion signals. No matter how much drop or ECN marking, the 1237 congestion window no longer reduces. Instead, TCP forces the queue 1238 to grow, overriding any AQM and increasing queuing delay. 1240 L4S mechanisms significantly reduce queueing delay so, over the same 1241 path, the RTT becomes lower. Then this problem becomes surprisingly 1242 common [TCP-sub-mss-w]. This is because, for the same link capacity, 1243 smaller RTT implies a smaller window. For instance, consider a 1244 residential setting with an upstream broadband Internet access of 8 1245 Mb/s, assuming a max segment size of 1500 B. Two upstream flows will 1246 each have the minimum window of 2 MSS if the RTT is 6ms or less, 1247 which is quite common when accessing a nearby data centre. So, any 1248 more than two such parallel TCP flows will become unresponsive and 1249 increase queuing delay. 1251 Unless scalable congestion controls are required to comply with this 1252 requirement from the start, they will frequently become unresponsive, 1253 negating the low latency benefit of L4S, for themselves and for 1254 others. One possible sub-MSS window mechanism is described in 1255 [TCP-sub-mss-w], and other approaches are likely to be feasible. 1257 A.1.7. Measuring Reordering Tolerance in Time Units 1259 Description: A scalable congestion control needs to detect loss by 1260 counting in time-based units, which is scalable, rather than counting 1261 in units of packets, which is not. 1263 Motivation: If it is known that all L4S senders using a link obey 1264 this rule, then link technologies that support L4S can remove the 1265 head-of-line blocking delay they have to introduce while trying to 1266 keep packets in tight order to avoid triggering loss detection based 1267 on counting packets. 1269 End-systems cannot know whether a missing packet is due to loss or 1270 reordering, except in hindsight - if it appears later. If senders 1271 deem that loss has occurred by counting reordered packets (e.g. the 3 1272 Duplicate ACK rule of Classic TCP), the time over which the network 1273 has to keep packets in order scales down as packet rates scale up 1274 over the years. In contrast, if senders allow a reordering window in 1275 time-based units before they deem there has been a loss, the time 1276 over which the network has to keep packets in order stays constant. 1278 The potential benefit for links comes in two parts: i) switching the 1279 unit from packet count to time-based; ii) potentially relaxing the 1280 amount of time available for re-ordering. The initial switch to 1281 time-based units offers no immediate benefit, but as the years 1282 progress it stops the reordering requirement becoming tighter. The 1283 secondary relaxation might be possible where some transport protocols 1284 find they can tolerate more re-ordering (e.g. more than the 3 DupACK 1285 rule, perhaps because it was reasonable when packet rates were low, 1286 but it is now far too tight). 1288 Tolerance of reordering over a small duration could allow parallel 1289 (e.g. bonded-channel) link technologies to relax their need to 1290 deliver packets strictly in order. Such links typically give 1291 arriving packets a link-level sequence number and introduce delay 1292 while buffering packets at the receiving end until they can be 1293 delivered in the same order. For radio links, this delay usually 1294 includes the time allowed for link-layer retransmissions. 1296 Note, however, that a link technology will only be able to relax its 1297 ordering requirement if it is certain that it will not degrade the 1298 transport /most/ sensitive to reordering that might use the link. 1299 Also note that in some controlled environments no reordering is 1300 tolerated by some transports (e.g. RoCEv2 used for RDMA), therefore 1301 a switch to time-based units could not be exploited to relax 1302 reordering. 1304 For receivers that need their packets in order, it would seem that 1305 relaxing network ordering would simply shift this reordering delay 1306 from the network to the receiver. However, that is not true in the 1307 general case because links generally do not recognize transport layer 1308 flows and often cannot even see application layer streams within the 1309 flows (as in SCTP, HTTP/2 or QUIC). So a link will often be holding 1310 back packets from one flow or stream while waiting for those from 1311 another. Relaxing strict ordering in the network will remove this 1312 head-of-line blocking delay. {ToDo: this is being quantified 1313 experimentally - will need to add the figures here.} 1315 Classic TCP implementations are switching over to the time-based 1316 approach of RACK (Recent ACKnowledgements [I-D.ietf-tcpm-rack]). 1317 However, it will be many years (decades?) before networks no longer 1318 have to allow for the presence of traditional TCP senders still using 1319 the 3 DupACK rule. This specification (Section 4.3) says that 1320 senders are not entitled to identify packets as L4S in the IP/ECN 1321 field unless they use the time-based approach. Then networks that 1322 identify L4S traffic separately (e.g. using 1323 [I-D.ietf-tsvwg-aqm-dualq-coupled]) can know for certain that all L4S 1324 traffic is using the scalable time-based approach. 1326 This will allow networks to remove the head-of-line blocking delay of 1327 resequencing straight away, but only for L4S traffic. Classic 1328 traffic will have to wait for many years until incremental deployment 1329 of RACK has become near-universal. Nonetheless, experience with RACK 1330 will determine how much reordering tolerance networks will be 1331 reasonable for L4S traffic. 1333 Performance Optimization as well as Safety Improvement: The delay 1334 benefit would be lost if any L4S sender did not follow the time-based 1335 approach. Therefore, the time-based approach is made a normative 1336 requirement (a necessary safety improvement). Nonetheless, the time- 1337 based approach also enables a throughput benefit that a flow can 1338 enjoy independently of others (a performance optimization), explained 1339 next. 1341 Given the requirement for a scalable congestion control to fall-back 1342 to Reno or Cubic on a loss (see Appendix A.1.3), it is important that 1343 a scalable congestion control does not deem that a loss has occurred 1344 too soon. If, later within the same round trip, an out-of-order 1345 acknowledgement fills the gap, the sender would have halved its rate 1346 spuriously (as well as retransmitting spuriously). With a RACK-like 1347 approach, allowing longer before a loss is deemed to have occurred 1348 maintains higher throughput in the presence of reordering {ToDo: 1349 Quantify this statement}. 1351 On the other hand, it is also important not to wait too long before 1352 deeming that a gap is due to a loss (termed a long reordering 1353 window), otherwise loss recovery would be slow. 1355 The speed of loss recovery is much more significant for short flows 1356 than long, therefore a good compromise would adapt the reordering 1357 window; from a small fraction of the RTT at the start of a flow, to a 1358 larger fraction of the RTT for flows that continue for many round 1359 trips. This is the approach adopted by TCP RACK (Recent 1360 ACKnowledgements) [I-D.ietf-tcpm-rack] and recommended for all L4S 1361 senders, whether using TCP or another transport protocol. 1363 The requirement to detect loss in time units also prevents the ACK- 1364 splitting attacks described in [Savage-TCP]. 1366 A.2. Scalable Transport Protocol Optimizations 1368 A.2.1. Setting ECT in TCP Control Packets and Retransmissions 1370 Description: This item only concerns TCP and its derivatives (e.g. 1371 SCTP), because the original specification of ECN for TCP precluded 1372 the use of ECN on control packets and retransmissions. To improve 1373 performance, scalable transport protocols ought to enable ECN at the 1374 IP layer in TCP control packets (SYN, SYN-ACK, pure ACKs, etc.) and 1375 in retransmitted packets. The same is true for derivatives of TCP, 1376 e.g. SCTP. 1378 Motivation: RFC 3168 prohibits the use of ECN on these types of TCP 1379 packet, based on a number of arguments. This means these packets are 1380 not protected from congestion loss by ECN, which considerably harms 1381 performance, particularly for short flows. 1382 [I-D.ietf-tcpm-generalized-ecn] counters each argument in RFC 3168 in 1383 turn, showing it was over-cautious. Instead it proposes experimental 1384 use of ECN on all types of TCP packet as long as AccECN feedback 1385 [I-D.ietf-tcpm-accurate-ecn] is available (which is itself a 1386 prerequisite for using a scalable congestion control). 1388 A.2.2. Faster than Additive Increase 1390 Description: It would improve performance if scalable congestion 1391 controls did not limit their congestion window increase to the 1392 traditional additive increase of 1 MSS per round trip [RFC5681] 1393 during congestion avoidance. The same is true for derivatives of TCP 1394 congestion control, including similar approaches used for real-time 1395 media. 1397 Motivation: As currently defined, DCTCP uses the traditional TCP Reno 1398 additive increase in congestion avoidance phase. When the available 1399 capacity suddenly increases (e.g. when another flow finishes, or if 1400 radio capacity increases) it can take very many round trips to take 1401 advantage of the new capacity. In the steady state, DCTCP induces 1402 about 2 ECN marks per round trip, so it should be possible to quickly 1403 detect when these signals have disappeared and seek available 1404 capacity more rapidly. It will of course be necessary to minimize 1405 the impact on other flows (classic and scalable). 1407 TCP Cubic was designed to solve this problem, but as flow rates have 1408 continued to increase, the delay accelerating into available capacity 1409 has become prohibitive. For instance, with RTT=20 ms, to increase 1410 flow rate from 100Mb/s to 200Mb/s Cubic takes between 50 and 100 1411 round trips. Every 8x increase in flow rate leads to 2x more 1412 acceleration delay. 1414 A.2.3. Faster Convergence at Flow Start 1416 Description: Particularly when a flow starts, scalable congestion 1417 controls need to converge (reach their steady-state share of the 1418 capacity) at least as fast as classic TCP and preferably faster. 1419 This does not just affect TCP Prague, but also the flow start 1420 behaviour of any L4S congestion control derived from a Classic 1421 transport that uses TCP slow start, including those for real-time 1422 media. 1424 Motivation: As an example, a new DCTCP flow takes longer than classic 1425 TCP to obtain its share of the capacity of the bottleneck when there 1426 are already ongoing flows using the bottleneck capacity. In a data 1427 centre environment DCTCP takes about a factor of 1.5 to 2 longer to 1428 converge due to the much higher typical level of ECN marking that 1429 DCTCP background traffic induces, which causes new flows to exit slow 1430 start early [Alizadeh-stability]. In testing for use over the public 1431 Internet the convergence time of DCTCP relative to regular TCP is 1432 even less favourable [Paced-Chirping]). It is exacerbated by the 1433 typically greater mismatch between the link rate of the sending host 1434 and typical Internet access bottlenecks, in combination with the 1435 shallow ECN marking threshold needed for L4S. This problem is 1436 detrimental in general, but would particularly harm the performance 1437 of short flows relative to classic TCP. 1439 Appendix B. Alternative Identifiers 1441 This appendix is informative, not normative. It records the pros and 1442 cons of various alternative ways to identify L4S packets to record 1443 the rationale for the choice of ECT(1) (Appendix B.1) as the L4S 1444 identifier. At the end, Appendix B.6 summarises the distinguishing 1445 features of the leading alternatives. It is intended to supplement, 1446 not replace the detailed text. 1448 The leading solutions all use the ECN field, sometimes in combination 1449 with the Diffserv field. Both the ECN and Diffserv fields have the 1450 additional advantage that they are no different in either IPv4 or 1451 IPv6. A couple of alternatives that use other fields are mentioned 1452 at the end, but it is quickly explained why they are not serious 1453 contenders. 1455 B.1. ECT(1) and CE codepoints 1457 Definition: 1459 Packets with ECT(1) and conditionally packets with CE would 1460 signify L4S semantics as an alternative to the semantics of 1461 classic ECN [RFC3168], specifically: 1463 * The ECT(1) codepoint would signify that the packet was sent by 1464 an L4S-capable sender; 1466 * Given shortage of codepoints, both L4S and classic ECN sides of 1467 an AQM would have to use the same CE codepoint to indicate that 1468 a packet had experienced congestion. If a packet that had 1469 already been marked CE in an upstream buffer arrived at a 1470 subsequent AQM, this AQM would then have to guess whether to 1471 classify CE packets as L4S or classic ECN. Choosing the L4S 1472 treatment would be a safer choice, because then a few classic 1473 packets might arrive early, rather than a few L4S packets 1474 arriving late; 1476 * Additional information might be available if the classifier 1477 were transport-aware. Then it could classify a CE packet for 1478 classic ECN treatment if the most recent ECT packet in the same 1479 flow had been marked ECT(0). However, the L4S service ought 1480 not to need tranport-layer awareness; 1482 Cons: 1484 Consumes the last ECN codepoint: The L4S service is intended to 1485 supersede the service provided by classic ECN, therefore using 1486 ECT(1) to identify L4S packets could ultimately mean that the 1487 ECT(0) codepoint was 'wasted' purely to distinguish one form of 1488 ECN from its successor; 1490 ECN hard in some lower layers: It is not always possible to support 1491 ECN in an AQM acting in a buffer below the IP layer 1492 [I-D.ietf-tsvwg-ecn-encap-guidelines]. In such cases, the L4S 1493 service would have to drop rather than mark frames even though 1494 they might contain an ECN-capable packet. However, such cases 1495 would be unusual. 1497 Risk of reordering classic CE packets: Classifying all CE packets 1498 into the L4S queue risks any CE packets that were originally 1499 ECT(0) being incorrectly classified as L4S. If there were delay 1500 in the Classic queue, these incorrectly classified CE packets 1501 would arrive early, which is a form of reordering. Reordering can 1502 cause TCP senders (and senders of similar transports) to 1503 retransmit spuriously. However, the risk of spurious 1504 retransmissions would be extremely low for the following reasons: 1506 1. It is quite unusual to experience queuing at more than one 1507 bottleneck on the same path (the available capacities have to 1508 be identical). 1510 2. In only a subset of these unusual cases would the first 1511 bottleneck support classic ECN marking while the second 1512 supported L4S ECN marking, which would be the only scenario 1513 where some ECT(0) packets could be CE marked by a non-L4S AQM 1514 then the remainder experienced further delay through the 1515 Classic side of a subsequent L4S DualQ AQM. 1517 3. Even then, when a few packets are delivered early, it takes 1518 very unusual conditions to cause a spurious retransmission, in 1519 contrast to when some packets are delivered late. The first 1520 bottleneck has to apply CE-marks to at least N contiguous 1521 packets and the second bottleneck has to inject an 1522 uninterrupted sequence of at least N of these packets between 1523 two packets earlier in the stream (where N is the reordering 1524 window that the transport protocol allows before it considers 1525 a packet is lost). 1527 For example consider N=3, and consider the sequence of 1528 packets 100, 101, 102, 103,... and imagine that packets 1529 150,151,152 from later in the flow are injected as follows: 1530 100, 150, 151, 101, 152, 102, 103... If this were late 1531 reordering, even one packet arriving 50 out of sequence 1532 would trigger a spurious retransmission, but there is no 1533 spurious retransmission here, because packet 101 moves the 1534 cumulative ACK counter forward before 3 packets have 1535 arrived out of order. Later, when packets 148, 149, 153... 1536 arrive, even though there is a 3-packet hole, there will be 1537 no problem, because the packets to fill the hole are 1538 already in the receive buffer. 1540 4. Even with the current recommended TCP (N=3) spurious 1541 retransmissions will be unlikely for all the above reasons. 1542 As RACK [I-D.ietf-tcpm-rack] is becoming widely deployed, it 1543 tends to adapt its reordering window to a larger value of N, 1544 which will make the chance of a contiguous sequence of N early 1545 arrivals vanishingly small. 1547 5. Even a run of 2 CE marks within a classic ECN flow is 1548 unlikely, given FQ-CoDel is the only known widely deployed AQM 1549 that supports classic ECN marking and it takes great care to 1550 separate out flows and to space any markings evenly along each 1551 flow. 1553 It is extremely unlikely that the above set of 5 eventualities 1554 that are each unusual in themselves would all happen 1555 simultaneously. But, even if they did, the consequences would 1556 hardly be dire: the odd spurious fast retransmission. Admittedly 1557 TCP reduces its congestion window when it deems there has been a 1558 loss, but even this can be recovered once the sender detects that 1559 the retransmission was spurious. 1561 Non-L4S service for control packets: The classic ECN RFCs [RFC3168] 1562 and [RFC5562] require a sender to clear the ECN field to Not-ECT 1563 for retransmissions and certain control packets specifically pure 1564 ACKs, window probes and SYNs. When L4S packets are classified by 1565 the ECN field alone, these control packets would not be classified 1566 into an L4S queue, and could therefore be delayed relative to the 1567 other packets in the flow. This would not cause re-ordering 1568 (because retransmissions are already out of order, and the control 1569 packets carry no data). However, it would make critical control 1570 packets more vulnerable to loss and delay. To address this 1571 problem, [I-D.ietf-tcpm-generalized-ecn] proposes an experiment in 1572 which all TCP control packets and retransmissions are ECN-capable 1573 as long as ECN feedback is available. 1575 Pros: 1577 Should work e2e: The ECN field generally works end-to-end across the 1578 Internet. Unlike the DSCP, the setting of the ECN field is at 1579 least forwarded unchanged by networks that do not support ECN, and 1580 networks rarely clear it to zero; 1582 Should work in tunnels: Unlike Diffserv, ECN is defined to always 1583 work across tunnels. However, tunnels do not always implement ECN 1584 processing as they should do, particularly because IPsec tunnels 1585 were defined differently for a few years. 1587 Could migrate to one codepoint: If all classic ECN senders 1588 eventually evolve to use the L4S service, the ECT(0) codepoint 1589 could be reused for some future purpose, but only once use of 1590 ECT(0) packets had reduced to zero, or near-zero, which might 1591 never happen. 1593 B.2. ECN Plus a Diffserv Codepoint (DSCP) 1595 Definition: 1597 For packets with a defined DSCP, all codepoints of the ECN field 1598 (except Not-ECT) would signify alternative L4S semantics to those 1599 for classic ECN [RFC3168], specifically: 1601 * The L4S DSCP would signifiy that the packet came from an L4S- 1602 capable sender; 1604 * ECT(0) and ECT(1) would both signify that the packet was 1605 travelling between transport endpoints that were both ECN- 1606 capable; 1608 * CE would signify that the packet had been marked by an AQM 1609 implementing the L4S service. 1611 Use of a DSCP is the only approach for alternative ECN semantics 1612 given as an example in [RFC4774]. However, it was perhaps considered 1613 more for controlled environments than new end-to-end services; 1615 Cons: 1617 Consumes DSCP pairs: A DSCP is obviously not orthogonal to Diffserv. 1618 Therefore, wherever the L4S service is applied to multiple 1619 Diffserv scheduling behaviours, it would be necessary to replace 1620 each DSCP with a pair of DSCPs. 1622 Uses critical lower-layer header space: The resulting increased 1623 number of DSCPs might be hard to support for some lower layer 1624 technologies, e.g. 802.1p and MPLS both offer only 3-bits for a 1625 maximum of 8 traffic class identifiers. Although L4S should 1626 reduce and possibly remove the need for some DSCPs intended for 1627 differentiated queuing delay, it will not remove the need for 1628 Diffserv entirely, because Diffserv is also used to allocate 1629 bandwidth, e.g. by prioritising some classes of traffic over 1630 others when traffic exceeds available capacity. 1632 Not end-to-end (host-network): Very few networks honour a DSCP set 1633 by a host. Typically a network will zero (bleach) the Diffserv 1634 field from all hosts. Sometimes networks will attempt to identify 1635 applications by some form of packet inspection and, based on 1636 network policy, they will set the DSCP considered appropriate for 1637 the identified application. Network-based application 1638 identification might use some combination of protocol ID, port 1639 numbers(s), application layer protocol headers, IP address(es), 1640 VLAN ID(s) and even packet timing. 1642 Not end-to-end (network-network): Very few networks honour a DSCP 1643 received from a neighbouring network. Typically a network will 1644 zero (bleach) the Diffserv field from all neighbouring networks at 1645 an interconnection point. Sometimes bilateral arrangements are 1646 made between networks, such that the receiving network remarks 1647 some DSCPs to those it uses for roughly equivalent services. The 1648 likelihood that a DSCP will be bleached or ignored depends on the 1649 type of DSCP: 1651 Local-use DSCP: These tend to be used to implement application- 1652 specific network policies, but a bilateral arrangement to 1653 remark certain DSCPs is often applied to DSCPs in the local-use 1654 range simply because it is easier not to change all of a 1655 network's internal configurations when a new arrangement is 1656 made with a neighbour; 1658 Recommended standard DSCP: These do not tend to be honoured 1659 across network interconnections more than local-use DSCPs. 1660 However, if two networks decide to honour certain of each 1661 other's DSCPs, the reconfiguration is a little easier if both 1662 of their globally recognised services are already represented 1663 by the relevant recommended standard DSCPs. 1665 Note that today a recommended standard DSCP gives little more 1666 assurance of end-to-end service than a local-use DSCP. In 1667 future the range recommended as standard might give more 1668 assurance of end-to-end service than local-use, but it is 1669 unlikely that either assurance will be high, particularly given 1670 the hosts are included in the end-to-end path. 1672 Not all tunnels: Diffserv codepoints are often not propagated to the 1673 outer header when a packet is encapsulated by a tunnel header. 1674 DSCPs are propagated to the outer of uniform mode tunnels, but not 1675 pipe mode [RFC2983], and pipe mode is fairly common. 1677 ECN hard in some lower layers:: Because this approach uses both the 1678 Diffserv and ECN fields, an AQM wil only work at a lower layer if 1679 both can be supported. If individual network operators wished to 1680 deploy an AQM at a lower layer, they would usually propagate an IP 1681 Diffserv codepoint to the lower layer, using for example IEEE 1682 802.1p. However, the ECN capability is harder to propagate down 1683 to lower layers because few lower layers support it. 1685 Pros: 1687 Could migrate to e2e: If all usage of classic ECN migrates to usage 1688 of L4S, the DSCP would become redundant, and the ECN capability 1689 alone could eventually identify L4S packets without the 1690 interconnection problems of Diffserv detailed above, and without 1691 having permanently consumed more than one codepoint in the IP 1692 header. Although the DSCP does not generally function as an end- 1693 to-end identifier (see above), it could be used initially by 1694 individual ISPs to introduce the L4S service for their own locally 1695 generated traffic; 1697 B.3. ECN capability alone 1699 Definition: 1701 This approach uses ECN capability alone as the L4S identifier. It 1702 is only feasible if classic ECN is not widely deployed. The 1703 specific definition of codepoints would be: 1705 * Any ECN codepoint other than Not-ECT would signify an L4S- 1706 capable sender; 1708 * ECN codepoints would not be used for classic [RFC3168] ECN, and 1709 the classic network service would only be used for Not-ECT 1710 packets. 1712 This approach would only be feasible if 1714 A. it was generally agreed that there was little chance of any 1715 classic [RFC3168] ECN deployment in any network nodes; 1717 B. it was generally agreed that there was little chance of any 1718 client devices being deployed with classic [RFC3168] TCP-ECN 1719 on by default (note that classic TCP-ECN is already on-by- 1720 default on many servers); 1722 C. for TCP connections, developers of client OSs would all have 1723 to agree not to encourage further deployment of classic ECN. 1724 Specifically, at the start of a TCP connection classic ECN 1725 could be disabled during negotation of the ECN capability: 1727 + an L4S-capable host would have to disable ECN if the 1728 corresponding host did not support accurate ECN feedback 1729 [RFC7560], which is a prerequisite for the L4S service; 1731 + developers of operating systems for user devices would only 1732 enable ECN by default for TCP once the stack implemented 1733 L4S and accurate ECN feedback [RFC7560] including 1734 requesting accurate ECN feedback by default. 1736 Cons: 1738 Near-infeasible deployment constraints: The constraints for 1739 deployment above represent a highly unlikely, but not completely 1740 impossible, set of circumstances. If, despite the above measures, 1741 a pair of hosts did negotiate to use classic ECN, their packets 1742 would be classified into the same queue as L4S traffic, and if 1743 they had to compete with a long-running L4S flow they would get a 1744 very small capacity share; 1746 ECN hard in some lower layers: See the same issue with "ECT(1) and 1747 CE codepoints" (Appendix B.1); 1749 Non-L4S service for control packets: See the same issue with "ECT(1) 1750 and CE codepoints" (Appendix B.1). 1752 Pros: 1754 Consumes no additional codepoints: The ECT(1) codepoint and all 1755 spare Diffserv codepoints would remain available for future use; 1757 Should work e2e: As with "ECT(1) and CE codepoints" (Appendix B.1); 1759 Should work in tunnels: As with "ECT(1) and CE codepoints" 1760 (Appendix B.1). 1762 B.4. Protocol ID 1764 It has been suggested that a new ID in the IPv4 Protocol field or the 1765 IPv6 Next Header field could identify L4S packets. However this 1766 approach is ruled out by numerous problems: 1768 o A new protocol ID would need to be paired with the old one for 1769 each transport (TCP, SCTP, UDP, etc.); 1771 o In IPv6, there can be a sequence of Next Header fields, and it 1772 would not be obvious which one would be expected to identify a 1773 network service like L4S; 1775 o A new protocol ID would rarely provide an end-to-end service, 1776 because It is well-known that new protocol IDs are often blocked 1777 by numerous types of middlebox; 1779 o The approach is not a solution for AQMs below the IP layer; 1781 B.5. Source or destination addressing 1783 Locally, a network operator could arrange for L4S service to be 1784 applied based on source or destination addressing, e.g. packets from 1785 its own data centre and/or CDN hosts, packets to its business 1786 customers, etc. It could use addressing at any layer, e.g. IP 1787 addresses, MAC addresses, VLAN IDs, etc. Although addressing might 1788 be a useful tactical approach for a single ISP, it would not be a 1789 feasible approach to identify an end-to-end service like L4S. Even 1790 for a single ISP, it would require packet classifiers in buffers to 1791 be dependent on changing topology and address allocation decisions 1792 elsewhere in the network. Therefore this approach is not a feasible 1793 solution. 1795 B.6. Summary: Merits of Alternative Identifiers 1797 Table 1 provides a very high level summary of the pros and cons 1798 detailed against the schemes described respectively in Appendix B.2, 1799 Appendix B.3 and Appendix B.1, for six issues that set them apart. 1801 +--------------+--------------------+---------+--------------------+ 1802 | Issue | DSCP + ECN | ECN | ECT(1) + CE | 1803 +--------------+--------------------+---------+--------------------+ 1804 | | initial eventual | initial | initial eventual | 1805 | | | | | 1806 | end-to-end | N . . . ? . | . . Y | . . Y . . Y | 1807 | tunnels | . O . . O . | . . ? | . . ? . . Y | 1808 | lower layers | N . . . ? . | . O . | . O . . . ? | 1809 | codepoints | N . . . . ? | . . Y | N . . . . ? | 1810 | reordering | . . Y . . Y | . . Y | . O . . . ? | 1811 | ctrl pkts | . . Y . . Y | . O . | . O . . . ? | 1812 | | | | | 1813 | | | Note 1 | | 1814 +--------------+--------------------+---------+--------------------+ 1816 Note 1: Only feasible if classic ECN is obsolete. 1818 Table 1: Comparison of the Merits of Three Alternative Identifiers 1820 The schemes are scored based on both their capabilities now 1821 ('initial') and in the long term ('eventual'). The 'ECN' scheme 1822 shares the 'eventual' scores of the 'ECT(1) + CE' scheme. The scores 1823 are one of 'N, O, Y', meaning 'Poor', 'Ordinary', 'Good' 1824 respectively. The same scores are aligned vertically to aid the eye. 1825 A score of "?" in one of the positions means that this approach might 1826 optimisitically become this good, given sufficient effort. The table 1827 summarises the text and is not meant to be understandable without 1828 having read the text. 1830 Appendix C. Potential Competing Uses for the ECT(1) Codepoint 1832 The ECT(1) codepoint of the ECN field has already been assigned once 1833 for the ECN nonce [RFC3540], which has now been categorized as 1834 historic [RFC8311]. ECN is probably the only remaining field in the 1835 Internet Protocol that is common to IPv4 and IPv6 and still has 1836 potential to work end-to-end, with tunnels and with lower layers. 1837 Therefore, ECT(1) should not be reassigned to a different 1838 experimental use (L4S) without carefully assessing competing 1839 potential uses. These fall into the following categories: 1841 C.1. Integrity of Congestion Feedback 1843 Receiving hosts can fool a sender into downloading faster by 1844 suppressing feedback of ECN marks (or of losses if retransmissions 1845 are not necessary or available otherwise). 1847 The historic ECN nonce protocol [RFC3540] proposed that a TCP sender 1848 could set either of ECT(0) or ECT(1) in each packet of a flow and 1849 remember the sequence it had set. If any packet was lost or 1850 congestion marked, the receiver would miss that bit of the sequence. 1851 An ECN Nonce receiver had to feed back the least significant bit of 1852 the sum, so it could not suppress feedback of a loss or mark without 1853 a 50-50 chance of guessing the sum incorrectly. 1855 It is highly unlikely that ECT(1) will be needed for integrity 1856 protection in future. The ECN Nonce RFC [RFC3540] as been 1857 reclassified as historic, partly because other ways have been 1858 developed to protect TCP feedback integrity [RFC8311] that do not 1859 consume a codepoint in the IP header. For instance: 1861 o the sender can test the integrity of the receiver's feedback by 1862 occasionally setting the IP-ECN field to a value normally only set 1863 by the network. Then it can test whether the receiver's feedback 1864 faithfully reports what it expects (see para 2 of Section 20.2 of 1865 [RFC3168]. This works for loss and it will work for the accurate 1866 ECN feedback [RFC7560] intended for L4S; 1868 o A network can enforce a congestion response to its ECN markings 1869 (or packet losses) by auditing congestion exposure (ConEx) 1870 [RFC7713]. Whether the receiver or a downstream network is 1871 suppressing congestion feedback or the sender is unresponsive to 1872 the feedback, or both, ConEx audit can neutralise any advantage 1873 that any of these three parties would otherwise gain. 1875 o The TCP authentication option (TCP-AO [RFC5925]) can be used to 1876 detect any tampering with TCP congestion feedback (whether 1877 malicious or accidental). TCP's congestion feedback fields are 1878 immutable end-to-end, so they are amenable to TCP-AO protection, 1879 which covers the main TCP header and TCP options by default. 1880 However, TCP-AO is often too brittle to use on many end-to-end 1881 paths, where middleboxes can make verification fail in their 1882 attempts to improve performance or security, e.g. by 1883 resegmentation or shifting the sequence space. 1885 C.2. Notification of Less Severe Congestion than CE 1887 Various researchers have proposed to use ECT(1) as a less severe 1888 congestion notification than CE, particularly to enable flows to fill 1889 available capacity more quickly after an idle period, when another 1890 flow departs or when a flow starts, e.g. VCP [VCP], Queue View (QV) 1891 [QV]. 1893 Before assigning ECT(1) as an identifer for L4S, we must carefully 1894 consider whether it might be better to hold ECT(1) in reserve for 1895 future standardisation of rapid flow acceleration, which is an 1896 important and enduring problem [RFC6077]. 1898 Pre-Congestion Notification (PCN) is another scheme that assigns 1899 alternative semantics to the ECN field. It uses ECT(1) to signify a 1900 less severe level of pre-congestion notification than CE [RFC6660]. 1901 However, the ECN field only takes on the PCN semantics if packets 1902 carry a Diffserv codepoint defined to indicate PCN marking within a 1903 controlled environment. PCN is required to be applied solely to the 1904 outer header of a tunnel across the controlled region in order not to 1905 interfere with any end-to-end use of the ECN field. Therefore a PCN 1906 region on the path would not interfere with any of the L4S service 1907 identifiers proposed in Appendix B. 1909 Authors' Addresses 1911 Koen De Schepper 1912 Nokia Bell Labs 1913 Antwerp 1914 Belgium 1916 Email: koen.de_schepper@nokia.com 1917 URI: https://www.bell-labs.com/usr/koen.de_schepper 1919 Bob Briscoe (editor) 1920 CableLabs 1921 UK 1923 Email: ietf@bobbriscoe.net 1924 URI: http://bobbriscoe.net/