idnits 2.17.1 draft-ietf-tsvwg-ecn-l4s-id-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1380 has weird spacing: '...initial even...' == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: o it SHOULD not lead to some packets of a transport-layer flow being served by a different queue from others. -- The document date (March 22, 2018) is 2227 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Outdated reference: A later version (-02) exists of draft-briscoe-tsvwg-l4s-diffserv-00 == Outdated reference: A later version (-28) exists of draft-ietf-tcpm-accurate-ecn-06 == Outdated reference: A later version (-15) exists of draft-ietf-tcpm-generalized-ecn-02 == Outdated reference: A later version (-25) exists of draft-ietf-tsvwg-aqm-dualq-coupled-04 == Outdated reference: A later version (-22) exists of draft-ietf-tsvwg-ecn-encap-guidelines-10 == Outdated reference: A later version (-20) exists of draft-ietf-tsvwg-l4s-arch-01 == Outdated reference: A later version (-07) exists of draft-stewart-tsvwg-sctpecn-05 -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) -- Obsolete informational reference (is this intentional?): RFC 4960 (Obsoleted by RFC 9260) Summary: 0 errors (**), 0 flaws (~~), 10 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Transport Services (tsv) K. De Schepper 3 Internet-Draft Nokia Bell Labs 4 Intended status: Experimental B. Briscoe, Ed. 5 Expires: September 23, 2018 CableLabs 6 March 22, 2018 8 Identifying Modified Explicit Congestion Notification (ECN) Semantics 9 for Ultra-Low Queuing Delay (L4S) 10 draft-ietf-tsvwg-ecn-l4s-id-02 12 Abstract 14 This specification defines the identifier to be used on IP packets 15 for a new network service called low latency, low loss and scalable 16 throughput (L4S). It is similar to the original (or 'Classic') 17 Explicit Congestion Notification (ECN). 'Classic' ECN marking was 18 required to be equivalent to a drop, both when applied in the network 19 and when responded to by a transport. Unlike 'Classic' ECN marking, 20 for packets carrying the L4S identifier, the network applies marking 21 more immediately and more aggressively than drop, and the transport 22 response to each mark is reduced and smoothed relative to that for 23 drop. The two changes counterbalance each other so that the 24 throughput of an L4S flow will be roughly the same as a 'Classic' 25 flow under the same conditions. However, the much more frequent 26 control signals and the finer responses to them result in ultra-low 27 queuing delay without compromising link utilization, even during high 28 load. Examples of new active queue management (AQM) marking 29 algorithms and examples of new transports (whether TCP-like or real- 30 time) are specified separately. The new L4S identifier is the key 31 piece that enables them to interwork and distinguishes them from 32 'Classic' traffic. It gives an incremental migration path so that 33 existing 'Classic' TCP traffic will be no worse off, but it can be 34 prevented from degrading the ultra-low delay and loss of the new 35 scalable transports. 37 Status of This Memo 39 This Internet-Draft is submitted in full conformance with the 40 provisions of BCP 78 and BCP 79. 42 Internet-Drafts are working documents of the Internet Engineering 43 Task Force (IETF). Note that other groups may also distribute 44 working documents as Internet-Drafts. The list of current Internet- 45 Drafts is at https://datatracker.ietf.org/drafts/current/. 47 Internet-Drafts are draft documents valid for a maximum of six months 48 and may be updated, replaced, or obsoleted by other documents at any 49 time. It is inappropriate to use Internet-Drafts as reference 50 material or to cite them other than as "work in progress." 52 This Internet-Draft will expire on September 23, 2018. 54 Copyright Notice 56 Copyright (c) 2018 IETF Trust and the persons identified as the 57 document authors. All rights reserved. 59 This document is subject to BCP 78 and the IETF Trust's Legal 60 Provisions Relating to IETF Documents 61 (https://trustee.ietf.org/license-info) in effect on the date of 62 publication of this document. Please review these documents 63 carefully, as they describe your rights and restrictions with respect 64 to this document. Code Components extracted from this document must 65 include Simplified BSD License text as described in Section 4.e of 66 the Trust Legal Provisions and are provided without warranty as 67 described in the Simplified BSD License. 69 Table of Contents 71 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 72 1.1. Problem . . . . . . . . . . . . . . . . . . . . . . . . . 4 73 1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 74 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 75 2. L4S Packet Identifier . . . . . . . . . . . . . . . . . . . . 7 76 2.1. Consensus Choice of L4S Packet Identifier: Requirements . 7 77 2.2. L4S Packet Identification at Run-Time . . . . . . . . . . 8 78 2.3. Interaction of the L4S Identifier with other Identifiers 8 79 2.4. Pre-Requisite Transport Layer Behaviour . . . . . . . . . 9 80 2.4.1. Pre-Requisite Congestion Response . . . . . . . . . . 10 81 2.4.2. Pre-Requisite Transport Feedback . . . . . . . . . . 11 82 2.5. Exception for L4S Packet Identification by Network Nodes 83 with Transport-Layer Awareness . . . . . . . . . . . . . 11 84 2.6. The Meaning of L4S CE Relative to Drop . . . . . . . . . 12 85 3. L4S Experiments . . . . . . . . . . . . . . . . . . . . . . . 13 86 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 87 5. Security Considerations . . . . . . . . . . . . . . . . . . . 13 88 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 13 89 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 90 7.1. Normative References . . . . . . . . . . . . . . . . . . 13 91 7.2. Informative References . . . . . . . . . . . . . . . . . 14 92 Appendix A. The 'TCP Prague Requirements' . . . . . . . . . . . 18 93 A.1. Requirements for Scalable Transport Protocols . . . . . . 19 94 A.1.1. Use of L4S Packet Identifier . . . . . . . . . . . . 19 95 A.1.2. Accurate ECN Feedback . . . . . . . . . . . . . . . . 19 96 A.1.3. Fall back to Reno/Cubic congestion control on packet 97 loss . . . . . . . . . . . . . . . . . . . . . . . . 19 98 A.1.4. Fall back to Reno/Cubic congestion control on classic 99 ECN bottlenecks . . . . . . . . . . . . . . . . . . . 20 100 A.1.5. Reduce RTT dependence . . . . . . . . . . . . . . . . 21 101 A.1.6. Scaling down the congestion window . . . . . . . . . 21 102 A.2. Scalable Transport Protocol Optimizations . . . . . . . . 22 103 A.2.1. Setting ECT in TCP Control Packets and 104 Retransmissions . . . . . . . . . . . . . . . . . . . 22 105 A.2.2. Faster than Additive Increase . . . . . . . . . . . . 22 106 A.2.3. Faster Convergence at Flow Start . . . . . . . . . . 23 107 Appendix B. Alternative Identifiers . . . . . . . . . . . . . . 23 108 B.1. ECT(1) and CE codepoints . . . . . . . . . . . . . . . . 23 109 B.2. ECN Plus a Diffserv Codepoint (DSCP) . . . . . . . . . . 25 110 B.3. ECN capability alone . . . . . . . . . . . . . . . . . . 27 111 B.4. Protocol ID . . . . . . . . . . . . . . . . . . . . . . . 29 112 B.5. Source or destination addressing . . . . . . . . . . . . 29 113 B.6. Summary: Merits of Alternative Identifiers . . . . . . . 29 114 Appendix C. Potential Competing Uses for the ECT(1) Codepoint . 30 115 C.1. Integrity of Congestion Feedback . . . . . . . . . . . . 30 116 C.2. Notification of Less Severe Congestion than CE . . . . . 31 117 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 119 1. Introduction 121 This specification defines the identifier to be used on IP packets 122 for a new network service called low latency, low loss and scalable 123 throughput (L4S). It is similar to the original (or 'Classic') 124 Explicit Congestion Notification (ECN). 'Classic' ECN marking was 125 required to be equivalent to a drop, both when applied in the network 126 and when responded to by a transport. Unlike 'Classic' ECN marking, 127 the network applies L4S marking more immediately and more 128 aggressively than drop, and the transport response to each mark is 129 reduced and smoothed relative to that for drop. The two changes 130 counterbalance each other so that the bit-rate of an L4S flow will be 131 roughly the same as a 'Classic' flow under the same conditions. 132 Nonetheless, the much more frequent control signals and the finer 133 responses to them result in ultra-low queuing delay without 134 compromising link utilization, even during high load. 136 An example of an active queue management (AQM) marking algorithm that 137 enables the L4S service is the DualQ Coupled AQM defined in a 138 complementary specification [I-D.ietf-tsvwg-aqm-dualq-coupled]. An 139 example of a scalable transport that would enable the L4S service is 140 Data Centre TCP (DCTCP), which until now has been applicable solely 141 to controlled environments like data centres [RFC8257], because it is 142 too aggressive to co-exist with existing TCP. However, AQMs like 143 DualQ Coupled enable scalable transports like DCTCP to co-exist with 144 existing traffic, each getting roughly the same flow rate when they 145 compete under similar conditions. Note that DCTCP will still not be 146 safe to deploy on the Internet until it satisfies the requirements 147 listed in Section 2.4. 149 The new L4S identifier is the key piece that enables these two parts 150 to interwork and distinguishes them from 'Classic' traffic. It gives 151 an incremental migration path so that existing 'Classic' TCP traffic 152 will be no worse off, but it can be prevented from degrading the 153 ultra-low delay and loss of the new scalable transports. The 154 performance improvement is so great that it is hoped it will motivate 155 initial deployment of the separate parts of this system. 157 1.1. Problem 159 Latency is becoming the critical performance factor for many (most?) 160 applications on the public Internet, e.g. Web, voice, conversational 161 video, gaming, finance apps, remote desktop and cloud-based 162 applications. In the developed world, further increases in access 163 network bit-rate offer diminishing returns, whereas latency is still 164 a multi-faceted problem. In the last decade or so, much has been 165 done to reduce propagation time by placing caches or servers closer 166 to users. However, queuing remains a major component of latency. 168 The Diffserv architecture provides Expedited Forwarding [RFC3246], so 169 that low latency traffic can jump the queue of other traffic. 170 However, on access links dedicated to individual sites (homes, small 171 enterprises or mobile devices), often all traffic at any one time 172 will be latency-sensitive. Then Diffserv is of little use. Instead, 173 we need to remove the causes of any unnecessary delay. 175 The bufferbloat project has shown that excessively-large buffering 176 ('bufferbloat') has been introducing significantly more delay than 177 the underlying propagation time. These delays appear only 178 intermittently--only when a capacity-seeking (e.g. TCP) flow is long 179 enough for the queue to fill the buffer, making every packet in other 180 flows sharing the buffer sit through the queue. 182 Active queue management (AQM) was originally developed to solve this 183 problem (and others). Unlike Diffserv, which gives low latency to 184 some traffic at the expense of others, AQM controls latency for _all_ 185 traffic in a class. In general, AQMs introduce an increasing level 186 of discard from the buffer the longer the queue persists above a 187 shallow threshold. This gives sufficient signals to capacity-seeking 188 (aka. greedy) flows to keep the buffer empty for its intended 189 purpose: absorbing bursts. However, RED [RFC2309] and other 190 algorithms from the 1990s were sensitive to their configuration and 191 hard to set correctly. So, AQM was not widely deployed. 193 More recent state-of-the-art AQMs, e.g. 194 fq_CoDel [I-D.ietf-aqm-fq-codel], PIE [I-D.ietf-aqm-pie], Adaptive 195 RED [ARED01], are easier to configure, because they define the 196 queuing threshold in time not bytes, so it is invariant for different 197 link rates. However, no matter how good the AQM, the sawtoothing 198 rate of TCP will either cause queuing delay to vary or cause the link 199 to be under-utilized. Even with a perfectly tuned AQM, the 200 additional queuing delay will be of the same order as the underlying 201 speed-of-light delay across the network. Flow-queuing can isolate 202 one flow from another, but it cannot isolate a TCP flow from the 203 delay variations it inflicts on itself, and it has other problems - 204 it overrides the flow rate decisions of variable rate video 205 applications, it does not recognise the flows within IPSec VPN 206 tunnels and it is relatively expensive to implement. 208 Latency is not our only concern: It was known when TCP was first 209 developed that it would not scale to high bandwidth-delay products. 210 Given regular broadband bit-rates over WAN distances are 211 already [RFC3649] beyond the scaling range of 'Classic' TCP Reno, 212 'less unscalable' Cubic [I-D.ietf-tcpm-cubic] and 213 Compound [I-D.sridharan-tcpm-ctcp] variants of TCP have been 214 successfully deployed. However, these are now approaching their 215 scaling limits. Unfortunately, fully scalable TCPs such as DCTCP 216 [RFC8257] cause 'Classic' TCP to starve itself, which is why they 217 have been confined to private data centres or research testbeds 218 (until now). 220 It turns out that a TCP algorithm like DCTCP that solves TCP's 221 scalability problem also solves the latency problem, because the 222 finer sawteeth cause very little queuing delay. A supporting paper 223 [DCttH15] gives the full explanation of why the design solves both 224 the latency and the scaling problems, both in plain English and in 225 more precise mathematical form. The explanation is summarised 226 without the maths in the L4S architecture document 227 [I-D.ietf-tsvwg-l4s-arch]. 229 1.2. Terminology 231 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 232 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 233 "OPTIONAL" in this document are to be interpreted as described in 234 [RFC2119]. In this document, these words will appear with that 235 interpretation only when in ALL CAPS. Lower case uses of these words 236 are not to be interpreted as carrying RFC-2119 significance. 238 Classic service: The 'Classic' service is intended for all the 239 behaviours that currently co-exist with TCP Reno (e.g. TCP Cubic, 240 Compound, SCTP, etc). 242 Low-Latency, Low-Loss and Scalable (L4S) service: The 'L4S' service 243 is intended for traffic from scalable TCP algorithms such as Data 244 Centre TCP. But it is also more general--it will allow a set of 245 congestion controls with similar scaling properties to DCTCP (e.g. 246 Relentless [Mathis09]) to evolve. 248 Both Classic and L4S services can cope with a proportion of 249 unresponsive or less-responsive traffic as well (e.g. DNS, VoIP, 250 etc). 252 Classic ECN: The original Explicit Congestion Notification (ECN) 253 protocol [RFC3168]. 255 1.3. Scope 257 The new L4S identifier defined in this specification is applicable 258 for IPv4 and IPv6 packets (as for classic ECN [RFC3168]). It is 259 applicable for the unicast, multicast and anycast forwarding modes. 261 The L4S identifier is an orthogonal packet classification to the 262 Differentiated Services Code Point (DSCP [RFC2474]). Section 2.3 263 explains what this means in practice. 265 This document is intended for experimental status, so it does not 266 update any standards track RFCs. Therefore it depends on [RFC8311], 267 which: 269 o updates the ECN proposed standard [RFC3168] (in certain specified 270 cases including the present document) to relax the requirement 271 that an ECN mark must be equivalent to a drop, both when applied 272 by the network, and when responded to by the sender; 274 o changes the status of the experimental ECN nonce [RFC3540] to 275 historic; 277 o makes consequent updates to the following proposed standard RFCs 278 to reflect the above two bullets: 280 * ECN for RTP [RFC6679]; 282 * the congestion control specifications of various DCCP 283 congestion control identifier (CCID) profiles [RFC4341], 284 [RFC4342], [RFC5622]. 286 2. L4S Packet Identifier 288 2.1. Consensus Choice of L4S Packet Identifier: Requirements 290 This subsection briefly records the process that led to a consensus 291 choice of L4S identifier, selected from all the alternatives in 292 Appendix B. 294 Ideally, the identifier for packets using the Low Latency, Low Loss, 295 Scalable throughput (L4S) service ought to meet the following 296 requirements: 298 o it SHOULD survive end-to-end between source and destination 299 applications: across the boundary between host and network, 300 between interconnected networks, and through middleboxes; 302 o it SHOULD be common to IPv4 and IPv6 and transport agnostic; 304 o it SHOULD be incrementally deployable; 306 o it SHOULD enable an AQM to classify packets encapsulated by outer 307 IP or lower-layer headers; 309 o it SHOULD consume minimal extra codepoints; 311 o it SHOULD not lead to some packets of a transport-layer flow being 312 served by a different queue from others. 314 Whether the identifier would be recoverable if the experiment failed 315 is a factor that could be taken into account. However, this has not 316 been made a requirement, because that would favour schemes that would 317 be easier to fail, rather than those more likely to succeed. 319 It is recognised that the chosen identifier is unlikely to satisfy 320 all these requirements, particularly given the limited space left in 321 the IP header. Therefore a compromise will be necessary, which is 322 why all the requirements are expressed with the word 'SHOULD' not 323 'MUST'. Appendix B discusses the pros and cons of the compromises 324 made in various competing identification schemes against the above 325 requirements. 327 On the basis of this analysis, "ECT(1) and CE codepoints" is the best 328 compromise. Therefore this scheme is defined in detail in the 329 following sections, while Appendix B records the rationale for this 330 decision. 332 2.2. L4S Packet Identification at Run-Time 334 The L4S treatment is an alternative packet marking treatment 335 [RFC4774] to the classic ECN treatment [RFC3168]. Like classic ECN, 336 it identifies both network and host behaviour: it identifies the 337 marking treatment that network nodes are expected to apply to L4S 338 packets, and it identifies packets that have been sent from hosts 339 that are expected to comply with a broad type of behaviour. 341 For a packet to receive L4S treatment as it is forwarded, the sender 342 MUST set the ECN field in the IP header (v4 or v6) to the ECT(1) 343 codepoint. 345 A network node that implements the L4S service MUST classify arriving 346 ECT(1) packets for L4S treatment and it SHOULD classify arriving CE 347 packets for L4S treatment as well. Section 2.5 describes a possible 348 exception to this latter rule. 350 An L4S AQM treatment follows similar codepoint transition rules to 351 those in RFC 3168. Specifically, the ECT(1) codepoint MUST NOT be 352 changed to any other codepoint than CE, and CE MUST NOT be changed to 353 any other codepoint. An ECT(1) packet is classified as ECN-capable 354 and, if congestion increases, an L4S AQM algorithm will mark the ECN 355 field as CE for an increasing proportion of packets, otherwise 356 forwarding packets unchanged as ECT(1). Necessary conditions for an 357 L4S marking treatment are defined in Section 2.6. Under persistent 358 overload an L4S marking treatment SHOULD turn off ECN marking, using 359 drop as a congestion signal until the overload episode has subsided, 360 as recommended for all AQMs [RFC7567] (Section 4.2.1), which follows 361 similar advice in RFC 3168 (Section 7). 363 For backward compatibility in uncontrolled environments, a network 364 node that implements the L4S treatment MUST also implement a classic 365 AQM treatment. It MUST classify arriving ECT(0) and Not-ECT packets 366 for treatment by the Classic AQM (see the discussion of the 367 classifier for the dual-queue coupled AQM in 368 [I-D.ietf-tsvwg-aqm-dualq-coupled]). Classic treatment means that 369 the AQM will mark ECT(0) packets under the same conditions as it 370 would drop Not-ECT packets [RFC3168]. 372 2.3. Interaction of the L4S Identifier with other Identifiers 374 In a typical case for the public Internet with just the default Best 375 Efforts Per-Hop Behaviour (PHB), a network element that implements 376 L4S MAY classify packets into the L4S treatment if they carry certain 377 operator-defined non-ECN identifiers. Such non-ECN-based packets 378 types MUST be safe to mix with L4S traffic without harming the low 379 latency service. For instance: 381 o addresses of specific applications or hosts configured to be safe 382 (but for example cannot set the ECN field for some temporary 383 reason); 385 o certain protocols that are usually lightweight (e.g. ARP, DNS); 387 o specific Diffserv codepoints that indicate traffic with limited 388 burstiness such as the EF (Expedited Forwarding), Voice-Admit and 389 CS5 (Application Signalling) service classes or equivalent local- 390 use DSCPs (see [I-D.briscoe-tsvwg-l4s-diffserv]). 392 For clarity, non-ECN identifiers, such as the examples itemized 393 above, might be used by some network operators who believe they 394 identify traffic that would be safe to mix with L4S traffic. They 395 are not alternative ways for a host to indicate that it is sending 396 L4S packets. Only the ECT(1) and CE ECN codepoints indicate that a 397 host is sending L4S packets - specifically that the host claims its 398 behaviour satisfies the pre-requisite transport requirements in 399 Section 2.4. 401 [I-D.briscoe-tsvwg-l4s-diffserv] gives detailed discussion on the 402 interactions between L4S and Diffserv. In summary, Diffserv provides 403 for differentiation of both bandwidth and low latency, but its 404 control of latency depends on its control of bandwidth. In contrast, 405 L4S concerns low latency, which it can provide for all traffic 406 without differentiation and without affecting bandwidth allocation. 408 The way a network element would classify on the DSCP and ECN fields 409 depends on which of the following cases applies: 411 o A common case is where a network element only offers Default 412 Forwarding (DF a.k.a. Best Efforts). In this case, if a packet 413 carries ECT(1) or CE, an L4S AQM would classify it as L4S 414 irrespective of its DSCP. Whereas, if a packet matched any of the 415 DSCPs in the bullet above, the AQM could classify it as L4S 416 irrespective of its ECN codepoint. 418 o A network operator might offer additional Diffserv PHBs. It might 419 split one or some of them between L4S and a corresponding Classic 420 service. Or it might split the L4S and/or the Classic service 421 into multiple Diffserv PHBs. In any of these cases, a packet 422 would have to be classified on its Diffserv and ECN codepoints. 424 2.4. Pre-Requisite Transport Layer Behaviour 425 2.4.1. Pre-Requisite Congestion Response 427 For a host to send packets with the L4S identifier (ECT(1)), it 428 SHOULD implement a congestion control behaviour that ensures the flow 429 rate is inversely proportional to the proportion of bytes in packets 430 marked with the CE codepoint. This is termed a scalable congestion 431 control, because the number of control signals (ECN marks) per round 432 trip remains roughly constant for any flow rate. As with all 433 transport behaviours, a detailed specification will need to be 434 defined for each type of transport or application, including the 435 timescale over which the proportionality is averaged, and control of 436 burstiness. The inverse proportionality requirement above is worded 437 as a 'SHOULD' rather than a 'MUST' to allow reasonable flexibility 438 when defining these specifications. 440 Data Center TCP (DCTCP [RFC8257]) is an example of a scalable 441 congestion control. 443 Each sender in a session can use a scalable congestion control 444 independently of the congestion control used by the receiver(s) when 445 they send data. Therefore there might be ECT(1) packets in one 446 direction and ECT(0) in the other. 448 In order to coexist safely with other Internet traffic, a scalable 449 congestion control MUST NOT identify its packets with the ECT(1) 450 codepoint unless it complies with the following bulleted 451 requirements. The specification of a particular scalable congestion 452 control MUST describe in detail how it satisfies each requirement: 454 o A scalable congestion control MUST react to packet loss in a way 455 that will coexist safely with a TCP Reno congestion control 456 [RFC5681] (see Appendix A.1.3 for rationale). 458 o A scalable congestion control MUST react to ECN marking from a 459 non-L4S but ECN-capable bottleneck in a way that will coexist with 460 a TCP Reno congestion control [RFC5681] (see Appendix A.1.4 for 461 rationale). 463 o A scalable congestion control MUST reduce or eliminate RTT bias 464 over as wide a range of RTTs as possible, or at least over the 465 typical range of RTTs that will interact in the intended 466 deployment scenario (see Appendix A.1.5 for rationale). 468 o A scalable congestion control MUST remain responsive to congestion 469 when the RTT is significantly smaller than in the current public 470 Internet (see Appendix A.1.6 for rationale). 472 2.4.2. Pre-Requisite Transport Feedback 474 In general, a scalable congestion control needs feedback of the 475 extent of CE marking on the forward path. Due to the history of TCP 476 development, when ECN was added it reported no more than one CE mark 477 per round trip. Some transport protocols derived from TCP mimic this 478 behaviour while others report the accurate extent of TCP marking. 479 This means that some transport protocols will need to be updated as a 480 prerequisite for scalable congestion control. The position for a few 481 well-known transport protocols is given below. 483 TCP: Support for accurate ECN feedback (AccECN 484 [I-D.ietf-tcpm-accurate-ecn]) by both ends is a prerequisite for 485 scalable congestion control. Therefore, the presence of ECT(1) in 486 the IP headers even in one direction of a TCP connection will 487 imply that both ends support AccECN. However, the converse does 488 not apply. So even if both ends support AccECN, either of the two 489 ends can choose not to use a scalable congestion control, whatever 490 the other end's choice. 492 SCTP: An ECN feedback protocol such as that specified in 493 [I-D.stewart-tsvwg-sctpecn] would be a prerequisite for scalable 494 congestion control. That draft would update the ECN feedback 495 protocol sketched out in Appendix A of the standards track 496 specification of SCTP [RFC4960] by adding a field to report the 497 number of CE marks. 499 RTP over UDP: A prerequisite for scalable congestion control is for 500 both (all) ends of one media-level hop to signal ECN support using 501 the ecn-capable-rtp attribute [RFC6679]. Therefore, the presence 502 of ECT(1) implies that both (all) ends of that hop support ECN. 503 However, the converse does not apply, so each end of a media-level 504 hop can independently choose not to use a scalable congestion 505 control, even if both ends support ECN. 507 DCCP: The ACK vector in DCCP [RFC4340] is already sufficient to 508 report the extent of CE marking as needed by a scalable congestion 509 control. 511 2.5. Exception for L4S Packet Identification by Network Nodes with 512 Transport-Layer Awareness 514 To implement the L4S treatment, a network node does not need to 515 identify transport-layer flows. Nonetheless, if an implementer is 516 willing to identify transport-layer flows at a network node, and if 517 the most recent ECT packet in the same flow was ECT(0), the node MAY 518 classify CE packets for classic ECN [RFC3168] treatment. In all 519 other cases, a network node MUST classify CE packets for L4S 520 treatment. Examples of such other cases are: i) if no ECT packets 521 have yet been identified in a flow; ii) if it is not desirable for a 522 network node to identify transport-layer flows; or iii) if the most 523 recent ECT packet in a flow was ECT(1). 525 If an implementer uses flow-awareness to classify CE packets, to 526 determine whether the flow is using ECT(0) or ECT(1) it only uses the 527 most recent ECT packet of a flow {ToDo: this advice will need to be 528 verified experimentally}. This is because a sender might have to 529 switch from sending ECT(1) (L4S) packets to sending ECT(0) (Classic) 530 packets, or back again, in the middle of a transport-layer flow. 531 Such a switch-over is likely to be very rare, but It could be 532 necessary if the path bottleneck moves from a network node that 533 supports L4S to one that only supports Classic ECN. A host ought to 534 be able to detect such a change from a change in RTT variation. 536 2.6. The Meaning of L4S CE Relative to Drop 538 The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST 539 be roughly proportional to the square of the likelihood that it would 540 have marked it if it had been an L4S packet (p_L). That is 542 p_C ~= (p_L / k)^2 544 The constant of proportionality (k) does not have to be standardised 545 for interoperability, but a value of 2 is RECOMMENDED. 547 [I-D.ietf-tsvwg-aqm-dualq-coupled] specifies the essential aspects of 548 an L4S AQM, as well as recommending other aspects. It gives example 549 implementations in appendices. 551 The term 'likelihood' is used above to allow for marking and dropping 552 to be either probabilistic or deterministic. The example AQMs in 553 [I-D.ietf-tsvwg-aqm-dualq-coupled] drop and mark probabilistically, 554 so the drop probability is arranged to be the square of the marking 555 probability. Nonetheless, an alternative AQM that dropped and marked 556 deterministically would be valid, as long as the dropping frequency 557 was proportional to the square of the marking frequency. 559 Note that, contrary to RFC 3168, an AQM implementing the L4S and 560 Classic treatments does not mark an ECT(1) packet under the same 561 conditions that it would have dropped a Not-ECT packet. However, it 562 does mark an ECT(0) packet under the same conditions that it would 563 have dropped a Not-ECT packet. 565 3. L4S Experiments 567 {ToDo: See https://tools.ietf.org/html/draft-ietf-tsvwg-ecn- 568 experimentation-07#section-2.3 which refers to the checklist in: 569 https://tools.ietf.org/html/rfc5706#appendix-A } 571 4. IANA Considerations 573 This specification contains no IANA considerations. 575 5. Security Considerations 577 Approaches to assure the integrity of signals using the new identifer 578 are introduced in Appendix C.1. 580 6. Acknowledgements 582 Thanks to Richard Scheffenegger, John Leslie, David Taeht, Jonathan 583 Morton, Gorry Fairhurst, Michael Welzl, Mikael Abrahamsson and Andrew 584 McGregor for the discussions that led to this specification. Ing-jyh 585 (Inton) Tsang was a contributor to the early drafts of this document. 586 Appendix A listing the TCP Prague Requirements is based on text 587 authored by Marcelo Bagnulo Braun that was originally an appendix to 588 [I-D.ietf-tsvwg-l4s-arch]. That text was in turn based on the 589 collective output of the attendees listed in the minutes of a 'bar 590 BoF' on DCTCP Evolution during IETF-94 [TCPPrague]. 592 The authors' contributions were part-funded by the European Community 593 under its Seventh Framework Programme through the Reducing Internet 594 Transport Latency (RITE) project (ICT-317700). Bob Briscoe was also 595 part-funded by the Research Council of Norway through the TimeIn 596 project. The views expressed here are solely those of the authors. 598 7. References 600 7.1. Normative References 602 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 603 Requirement Levels", BCP 14, RFC 2119, 604 DOI 10.17487/RFC2119, March 1997, 605 . 607 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 608 of Explicit Congestion Notification (ECN) to IP", 609 RFC 3168, DOI 10.17487/RFC3168, September 2001, 610 . 612 [RFC4774] Floyd, S., "Specifying Alternate Semantics for the 613 Explicit Congestion Notification (ECN) Field", BCP 124, 614 RFC 4774, DOI 10.17487/RFC4774, November 2006, 615 . 617 [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., 618 and K. Carlberg, "Explicit Congestion Notification (ECN) 619 for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 620 2012, . 622 7.2. Informative References 624 [Alizadeh-stability] 625 Alizadeh, M., Javanmard, A., and B. Prabhakar, "Analysis 626 of DCTCP: Stability, Convergence, and Fairness", ACM 627 SIGMETRICS 2011 , June 2011. 629 [ARED01] Floyd, S., Gummadi, R., and S. Shenker, "Adaptive RED: An 630 Algorithm for Increasing the Robustness of RED's Active 631 Queue Management", ACIRI Technical Report , August 2001, 632 . 634 [DCttH15] De Schepper, K., Bondarenko, O., Briscoe, B., and I. 635 Tsang, "'Data Centre to the Home': Ultra-Low Latency for 636 All", 2015, . 639 (Under submission) 641 [I-D.briscoe-tsvwg-l4s-diffserv] 642 Briscoe, B., "Interactions between Low Latency, Low Loss, 643 Scalable Throughput (L4S) and Differentiated Services", 644 draft-briscoe-tsvwg-l4s-diffserv-00 (work in progress), 645 March 2018. 647 [I-D.ietf-aqm-fq-codel] 648 Hoeiland-Joergensen, T., McKenney, P., 649 dave.taht@gmail.com, d., Gettys, J., and E. Dumazet, "The 650 FlowQueue-CoDel Packet Scheduler and Active Queue 651 Management Algorithm", draft-ietf-aqm-fq-codel-06 (work in 652 progress), March 2016. 654 [I-D.ietf-aqm-pie] 655 Pan, R., Natarajan, P., Baker, F., and G. White, "PIE: A 656 Lightweight Control Scheme To Address the Bufferbloat 657 Problem", draft-ietf-aqm-pie-10 (work in progress), 658 September 2016. 660 [I-D.ietf-tcpm-accurate-ecn] 661 Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More 662 Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate- 663 ecn-06 (work in progress), March 2018. 665 [I-D.ietf-tcpm-cubic] 666 Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and 667 R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", 668 draft-ietf-tcpm-cubic-07 (work in progress), November 669 2017. 671 [I-D.ietf-tcpm-generalized-ecn] 672 Bagnulo, M. and B. Briscoe, "ECN++: Adding Explicit 673 Congestion Notification (ECN) to TCP Control Packets", 674 draft-ietf-tcpm-generalized-ecn-02 (work in progress), 675 October 2017. 677 [I-D.ietf-tsvwg-aqm-dualq-coupled] 678 Schepper, K., Briscoe, B., Bondarenko, O., and I. Tsang, 679 "DualQ Coupled AQMs for Low Latency, Low Loss and Scalable 680 Throughput (L4S)", draft-ietf-tsvwg-aqm-dualq-coupled-04 681 (work in progress), March 2018. 683 [I-D.ietf-tsvwg-ecn-encap-guidelines] 684 Briscoe, B., Kaippallimalil, J., and P. Thaler, 685 "Guidelines for Adding Congestion Notification to 686 Protocols that Encapsulate IP", draft-ietf-tsvwg-ecn- 687 encap-guidelines-10 (work in progress), March 2018. 689 [I-D.ietf-tsvwg-l4s-arch] 690 Briscoe, B., Schepper, K., and M. Bagnulo, "Low Latency, 691 Low Loss, Scalable Throughput (L4S) Internet Service: 692 Architecture", draft-ietf-tsvwg-l4s-arch-01 (work in 693 progress), October 2017. 695 [I-D.sridharan-tcpm-ctcp] 696 Sridharan, M., Tan, K., Bansal, D., and D. Thaler, 697 "Compound TCP: A New TCP Congestion Control for High-Speed 698 and Long Distance Networks", draft-sridharan-tcpm-ctcp-02 699 (work in progress), November 2008. 701 [I-D.stewart-tsvwg-sctpecn] 702 Stewart, R., Tuexen, M., and X. Dong, "ECN for Stream 703 Control Transmission Protocol (SCTP)", draft-stewart- 704 tsvwg-sctpecn-05 (work in progress), January 2014. 706 [Mathis09] 707 Mathis, M., "Relentless Congestion Control", PFLDNeT'09 , 708 May 2009, . 711 [QV] Briscoe, B. and P. Hurtig, "Up to Speed with Queue View", 712 RITE Technical Report D2.3; Appendix C.2, August 2015, 713 . 716 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 717 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 718 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 719 S., Wroclawski, J., and L. Zhang, "Recommendations on 720 Queue Management and Congestion Avoidance in the 721 Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998, 722 . 724 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 725 "Definition of the Differentiated Services Field (DS 726 Field) in the IPv4 and IPv6 Headers", RFC 2474, 727 DOI 10.17487/RFC2474, December 1998, 728 . 730 [RFC2983] Black, D., "Differentiated Services and Tunnels", 731 RFC 2983, DOI 10.17487/RFC2983, October 2000, 732 . 734 [RFC3246] Davie, B., Charny, A., Bennet, J., Benson, K., Le Boudec, 735 J., Courtney, W., Davari, S., Firoiu, V., and D. 736 Stiliadis, "An Expedited Forwarding PHB (Per-Hop 737 Behavior)", RFC 3246, DOI 10.17487/RFC3246, March 2002, 738 . 740 [RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit 741 Congestion Notification (ECN) Signaling with Nonces", 742 RFC 3540, DOI 10.17487/RFC3540, June 2003, 743 . 745 [RFC3649] Floyd, S., "HighSpeed TCP for Large Congestion Windows", 746 RFC 3649, DOI 10.17487/RFC3649, December 2003, 747 . 749 [RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram 750 Congestion Control Protocol (DCCP)", RFC 4340, 751 DOI 10.17487/RFC4340, March 2006, 752 . 754 [RFC4341] Floyd, S. and E. Kohler, "Profile for Datagram Congestion 755 Control Protocol (DCCP) Congestion Control ID 2: TCP-like 756 Congestion Control", RFC 4341, DOI 10.17487/RFC4341, March 757 2006, . 759 [RFC4342] Floyd, S., Kohler, E., and J. Padhye, "Profile for 760 Datagram Congestion Control Protocol (DCCP) Congestion 761 Control ID 3: TCP-Friendly Rate Control (TFRC)", RFC 4342, 762 DOI 10.17487/RFC4342, March 2006, 763 . 765 [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", 766 RFC 4960, DOI 10.17487/RFC4960, September 2007, 767 . 769 [RFC5562] Kuzmanovic, A., Mondal, A., Floyd, S., and K. 770 Ramakrishnan, "Adding Explicit Congestion Notification 771 (ECN) Capability to TCP's SYN/ACK Packets", RFC 5562, 772 DOI 10.17487/RFC5562, June 2009, 773 . 775 [RFC5622] Floyd, S. and E. Kohler, "Profile for Datagram Congestion 776 Control Protocol (DCCP) Congestion ID 4: TCP-Friendly Rate 777 Control for Small Packets (TFRC-SP)", RFC 5622, 778 DOI 10.17487/RFC5622, August 2009, 779 . 781 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 782 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 783 . 785 [RFC6077] Papadimitriou, D., Ed., Welzl, M., Scharf, M., and B. 786 Briscoe, "Open Research Issues in Internet Congestion 787 Control", RFC 6077, DOI 10.17487/RFC6077, February 2011, 788 . 790 [RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three 791 Pre-Congestion Notification (PCN) States in the IP Header 792 Using a Single Diffserv Codepoint (DSCP)", RFC 6660, 793 DOI 10.17487/RFC6660, July 2012, 794 . 796 [RFC7560] Kuehlewind, M., Ed., Scheffenegger, R., and B. Briscoe, 797 "Problem Statement and Requirements for Increased Accuracy 798 in Explicit Congestion Notification (ECN) Feedback", 799 RFC 7560, DOI 10.17487/RFC7560, August 2015, 800 . 802 [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF 803 Recommendations Regarding Active Queue Management", 804 BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, 805 . 807 [RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L., 808 and G. Judd, "Data Center TCP (DCTCP): TCP Congestion 809 Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257, 810 October 2017, . 812 [RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion 813 Notification (ECN) Experimentation", RFC 8311, 814 DOI 10.17487/RFC8311, January 2018, 815 . 817 [TCP-sub-mss-w] 818 Briscoe, B. and K. De Schepper, "Scaling TCP's Congestion 819 Window for Small Round Trip Times", BT Technical Report 820 TR-TUB8-2015-002, May 2015, 821 . 824 [TCPPrague] 825 Briscoe, B., "Notes: DCTCP evolution 'bar BoF': Tue 21 Jul 826 2015, 17:40, Prague", tcpprague mailing list archive , 827 July 2015, . 830 [VCP] Xia, Y., Subramanian, L., Stoica, I., and S. Kalyanaraman, 831 "One more bit is enough", Proc. SIGCOMM'05, ACM CCR 832 35(4)37--48, 2005, 833 . 835 Appendix A. The 'TCP Prague Requirements' 837 This appendix is informative, not normative. It gives a list of 838 modifications to current scalable transport protocols so that they 839 can be deployed over the public Internet and coexist safely with 840 existing traffic. The list complements the normative requirements in 841 Section 2.4 that a sender has to comply with before it can set the 842 L4S identifier in packets it sends into the Internet. As well as 843 necessary safety improvements (requirements) this appendix also 844 includes preferable performance improvements (optimizations). 846 These recommendations have become know as the TCP Prague 847 Requirements, because they were originally identified at an ad hoc 848 meeting during IETF-94 in Prague [TCPPrague]. The wording has been 849 generalized to apply to all scalable congestion controls, not just 850 TCP congestion control specifically. 852 DCTCP [RFC8257] is currently the most widely used scalable transport 853 protocol. In its current form, DCTCP is specified to be deployable 854 only in controlled environments. Deploying it in the public Internet 855 would lead to a number of issues, both from the safety and the 856 performance perspective. The modifications and additional mechanisms 857 listed in this section will be necessary for its deployment over the 858 global Internet. Where an example is needed, DCTCP is used as a 859 base, but it is likely that most of these requirements equally apply 860 to other scalable transport protocols. 862 A.1. Requirements for Scalable Transport Protocols 864 A.1.1. Use of L4S Packet Identifier 866 Description: A scalable congestion control needs to distinguish the 867 packets it sends from those sent by classic congestion controls. 869 Motivation: It needs to be possible for a network node to classify 870 L4S packets without flow state into a queue that applies an L4S ECN 871 marking behaviour and isolates L4S packets from the queuing delay of 872 classic packets. 874 A.1.2. Accurate ECN Feedback 876 Description: A scalable transport protocol needs to provide timely, 877 accurate feedback about the extent of ECN marking experienced by all 878 packets. 880 Motivation: Classic congestion controls only need feedback about the 881 existence of a congestion episode within a round trip, not precisely 882 how many packets were marked with ECN or dropped. Therefore, in 883 2001, when ECN feedback was added to TCP [RFC3168], it could not 884 inform the sender of more than one ECN mark per RTT. Since then, 885 requirements for more accurate ECN feedback in TCP have been defined 886 in [RFC7560] and [I-D.ietf-tcpm-accurate-ecn] specifies an 887 experimental change to the TCP wire protocol to satisfy these 888 requirements. Most other transport protocols already satisfy this 889 requirement. 891 A.1.3. Fall back to Reno/Cubic congestion control on packet loss 893 Description: A scalable congestion control needs to react to packet 894 loss in a way that will coexist safely with a TCP Reno congestion 895 control [RFC5681]. 897 Motivation: Part of the safety conditions for deploying a scalable 898 congestion control on the public Internet is to make sure that it 899 behaves properly when it builds a queue at a network bottleneck that 900 has not been upgraded to support L4S. Packet loss can have many 901 causes, but it usually has to be conservatively assumed that it is a 902 sign of congestion. Therefore, on detecting packet loss, a scalable 903 congestion control will need to fall back to classic congestion 904 control behaviour. If it does not comply with this requirement it 905 could starve classic traffic. 907 A scalable congestion control can be used for different types of 908 transport, e.g. for real-time media or for reliable bulk transport 909 like TCP. Therefore, the particular classic congestion control 910 behaviour to fall back on will need to be part of the congestion 911 control specification of the relevant transport. In the particular 912 case of DCTCP, the current DCTCP specification states that "It is 913 RECOMMENDED that an implementation deal with loss episodes in the 914 same way as conventional TCP." For safe deployment of a scalable 915 transport in the public Internet, the above requirement would need to 916 be defined as a "MUST". 918 Packet loss might (rarely) occur in the case that the bottleneck is 919 L4S capable. In this case, the sender may receive a high number of 920 packets marked with the CE bit set and also experience a loss. 921 Current DCTCP implementations react differently to this situation. 922 At least one implementation reacts only to the drop signal (e.g. by 923 halving the CWND) and at least another DCTCP implementation reacts to 924 both signals (e.g. by halving the CWND due to the drop and also 925 further reducing the CWND based on the proportion of marked packet). 926 We believe that further experimentation is needed to understand what 927 is the best behaviour for the public Internet, which may or not be 928 one of these existing approaches. 930 A.1.4. Fall back to Reno/Cubic congestion control on classic ECN 931 bottlenecks 933 Description: A scalable congestion control needs to react to ECN 934 marking from a non-L4S but ECN-capable bottleneck in a way that will 935 coexist with a TCP Reno congestion control [RFC5681]. 937 Motivation: Similarly to the requirement in Appendix A.1.3, this 938 requirement is a safety condition to ensure a scalable congestion 939 control behaves properly when it builds a queue at a network 940 bottleneck that has not been upgraded to support L4S. On detecting 941 classic ECN marking (see below), a scalable congestion control will 942 need to fall back to classic congestion control behaviour. If it 943 does not comply with this requirement it could starve classic 944 traffic. 946 It would take time for endpoints to distinguish classic and L4S ECN 947 marking. An increase in queuing delay or in delay variation would be 948 a tell-tale sign, but it is not yet clear where a line would be drawn 949 between the two behaviours. It might be possible to cache what was 950 learned about the path to help subsequent attempts to detect the type 951 of marking. 953 A.1.5. Reduce RTT dependence 955 Description: A scalable congestion control needs to reduce or 956 eliminate RTT bias over as wide a range of RTTs as possible, or at 957 least over the typical range of RTTs that will interact in the 958 intended deployment scenario. 960 Motivation: Classic TCP's throughput is known to be inversely 961 proportional to RTT, so one would expect flows over very low RTT 962 paths to nearly starve flows over larger RTTs. However, Classic TCP 963 has never allowed a very low RTT path to exist because it induces a 964 large queue. For instance, consider two paths with base RTT 1ms and 965 100ms. If Classic TCP induces a 100ms queue, it turns these RTTs 966 into 101ms and 200ms leading to a throughput ratio of about 2:1. 967 Whereas if a Scalable TCP induces only a 1ms queue, the ratio is 968 2:101, leading to a throughput ratio of about 50:1. 970 Therefore, with very small queues, long RTT flows will essentially 971 starve, unless scalable congestion controls comply with this 972 requirement. 974 A.1.6. Scaling down the congestion window 976 Description: A scalable congestion control needs to remain responsive 977 to congestion when RTTs are significantly smaller than in the current 978 public Internet. 980 Motivation: As currently specified, the minimum required congestion 981 window of TCP (and its derivatives) is set to 2 maximum segment sizes 982 (MSS) (see equation (4) in [RFC5681]). Once the congestion window 983 reaches this minimum, all current TCP algorithms become unresponsive 984 to congestion signals. No matter how much drop or ECN marking, the 985 congestion window no longer reduces. Instead, TCP forces the queue 986 to grow, overriding any AQM and increasing queuing delay. 988 L4S mechanisms significantly reduce queueing delay so, over the same 989 path, the RTT becomes lower. Then this problem becomes surprisingly 990 common [TCP-sub-mss-w]. This is because, for the same link capacity, 991 smaller RTT implies a smaller window. For instance, consider a 992 residential setting with an upstream broadband Internet access of 8 993 Mb/s, assuming a max segment size of 1500 B. Two upstream flows will 994 each have the minimum window of 2 MSS if the RTT is 6ms or less, 995 which is quite common when accessing a nearby data centre. So, any 996 more than two such parallel TCP flows will become unresponsive and 997 increase queuing delay. 999 Unless scalable congestion controls are required to comply with this 1000 requirement from the start, they will frequently become unresponsive, 1001 negating the low latency benefit of L4S, for themselves and for 1002 others. One possible sub-MSS window mechanism is described in 1003 [TCP-sub-mss-w], and other approaches are likely to be feasible. 1005 A.2. Scalable Transport Protocol Optimizations 1007 A.2.1. Setting ECT in TCP Control Packets and Retransmissions 1009 Description: To improve performance, scalable transport protocols 1010 ought to enable ECN at the IP layer in TCP control packets (SYN, SYN- 1011 ACK, pure ACKs, etc.) and in retransmitted packets. The same is true 1012 for derivatives of TCP, e.g. SCTP. 1014 Motivation: RFC 3168 prohibits the use of ECN on these types of TCP 1015 packet, based on a number of arguments. This means these packets are 1016 not protected from congestion loss by ECN, which considerably harms 1017 performance, particularly for short flows. 1018 [I-D.ietf-tcpm-generalized-ecn] counters each argument in RFC 3168 in 1019 turn, showing it was over-cautious. Instead it proposes experimental 1020 use of ECN on all types of TCP packet. 1022 A.2.2. Faster than Additive Increase 1024 Description: It would improve performance if scalable congestion 1025 controls did not limit their congestion window increase to the 1026 traditional additive increase of 1 MSS per round trip [RFC5681] 1027 during congestion avoidance. The same is true for derivatives of TCP 1028 congestion control. 1030 Motivation: As currently defined, DCTCP uses the traditional TCP Reno 1031 additive increase in congestion avoidance phase. When the available 1032 capacity suddenly increases (e.g. when another flow finishes, or if 1033 radio capacity increases) it can take very many round trips to take 1034 advantage of the new capacity. In the steady state, DCTCP induces 1035 about 2 ECN marks per round trip, so it should be possible to quickly 1036 detect when these signals have disappeared and seek available 1037 capacity more rapidly. It will of course be necessary to minimize 1038 the impact on other flows (classic and scalable). 1040 TCP Cubic was designed to solve this problem, but as flow rates have 1041 continued to increase, the delay accelerating into available capacity 1042 has become prohibitive. For instance, with RTT=20 ms, to increase 1043 flow rate from 100Mb/s to 200Mb/s Cubic takes between 50 and 100 1044 round trips. Every 8x increase in flow rate leads to 2x more 1045 acceleration delay. 1047 A.2.3. Faster Convergence at Flow Start 1049 Description: Particularly when a flow starts, scalable congestion 1050 controls need to converge (reach their steady-state share of the 1051 capacity) at least as fast as classic TCP and preferably faster. 1052 This affects the flow start behaviour of all transports that are 1053 based on TCP slow start. 1055 Motivation: As an example, a new DCTCP flow takes longer than classic 1056 TCP to obtain its share of the capacity of the bottleneck when there 1057 are already ongoing flows using the bottleneck capacity (about a 1058 factor of 1.5 and 2 longer according to [Alizadeh-stability]). This 1059 is detrimental in general, but particularly harms the performance of 1060 short flows relative to classic TCP. 1062 Appendix B. Alternative Identifiers 1064 This appendix is informative, not normative. It records the pros and 1065 cons of various alternative ways to identify L4S packets to record 1066 the rationale for the choice of ECT(1) (Appendix B.1) as the L4S 1067 identifier. At the end, Appendix B.6 summarises the distinguishing 1068 features of the leading alternatives. It is intended to supplement, 1069 not replace the detailed text. 1071 The leading solutions all use the ECN field, sometimes in combination 1072 with the Diffserv field. Both the ECN and Diffserv fields have the 1073 additional advantage that they are no different in either IPv4 or 1074 IPv6. A couple of alternatives that use other fields are mentioned 1075 at the end, but it is quickly explained why they are not serious 1076 contenders. 1078 B.1. ECT(1) and CE codepoints 1080 Definition: 1082 Packets with ECT(1) and conditionally packets with CE would 1083 signify L4S semantics as an alternative to the semantics of 1084 classic ECN [RFC3168], specifically: 1086 * The ECT(1) codepoint would signify that the packet was sent by 1087 an L4S-capable sender; 1089 * Given shortage of codepoints, both L4S and classic ECN sides of 1090 an AQM would have to use the same CE codepoint to indicate that 1091 a packet had experienced congestion. If a packet that had 1092 already been marked CE in an upstream buffer arrived at a 1093 subsequent AQM, this AQM would then have to guess whether to 1094 classify CE packets as L4S or classic ECN. Choosing the L4S 1095 treatment would be a safer choice, because then a few classic 1096 packets might arrive early, rather than a few L4S packets 1097 arriving late; 1099 * Additional information might be available if the classifier 1100 were transport-aware. Then it could classify a CE packet for 1101 classic ECN treatment if the most recent ECT packet in the same 1102 flow had been marked ECT(0). However, the L4S service ought 1103 not to need tranport-layer awareness; 1105 Cons: 1107 Consumes the last ECN codepoint: The L4S service is intended to 1108 supersede the service provided by classic ECN, therefore using 1109 ECT(1) to identify L4S packets could ultimately mean that the 1110 ECT(0) codepoint was 'wasted' purely to distinguish one form of 1111 ECN from its successor; 1113 ECN hard in some lower layers: It is not always possible to support 1114 ECN in an AQM acting in a buffer below the IP layer 1115 [I-D.ietf-tsvwg-ecn-encap-guidelines]. In such cases, the L4S 1116 service would have to drop rather than mark frames even though 1117 they might contain an ECN-capable packet. However, such cases 1118 would be unusual. 1120 Risk of reordering classic CE packets: Having to classify all CE 1121 packets as L4S risks some classic CE packets arriving early, which 1122 is a form of reordering. Reordering can cause the TCP sender to 1123 retransmit spuriously. However, one or two packets delivered 1124 early does not cause any spurious retransmissions because the 1125 subsequent packets continue to move the cumulative acknowledgement 1126 boundary forwards. Anyway, the risk of reordering would be low, 1127 because: i) it is quite unusual to experience more than one 1128 bottleneck queue on a path; ii) even then, reordering would only 1129 occur if there was simultaneous mixing of classic and L4S traffic, 1130 which would be more unlikely in an access link, which is where 1131 most bottlenecks are located; iii) even then, spurious 1132 retransmissions would only occur if a contiguous sequence of three 1133 or more classic CE packets from one bottleneck arrived at the 1134 next, which should in itself happen very rarely with a good AQM. 1135 The risk would be completely eliminated in AQMs that were 1136 transport-aware (but they should not need to be); 1138 Non-L4S service for control packets: The classic ECN RFCs [RFC3168] 1139 and [RFC5562] require a sender to clear the ECN field to Not-ECT 1140 for retransmissions and certain control packets specifically pure 1141 ACKs, window probes and SYNs. When L4S packets are classified by 1142 the ECN field alone, these control packets would not be classified 1143 into an L4S queue, and could therefore be delayed relative to the 1144 other packets in the flow. This would not cause re-ordering 1145 (because retransmissions are already out of order, and the control 1146 packets carry no data). However, it would make critical control 1147 packets more vulnerable to loss and delay. To address this 1148 problem, [I-D.ietf-tcpm-generalized-ecn] proposes an experiment in 1149 which all TCP control packets and retransmissions are ECN-capable. 1151 Pros: 1153 Should work e2e: The ECN field generally works end-to-end across the 1154 Internet. Unlike the DSCP, the setting of the ECN field is at 1155 least forwarded unchanged by networks that do not support ECN, and 1156 networks rarely clear it to zero; 1158 Should work in tunnels: Unlike Diffserv, ECN is defined to always 1159 work across tunnels. However, tunnels do not always implement ECN 1160 processing as they should do, particularly because IPsec tunnels 1161 were defined differently for a few years. 1163 Could migrate to one codepoint: If all classic ECN senders 1164 eventually evolve to use the L4S service, the ECT(0) codepoint 1165 could be reused for some future purpose, but only once use of 1166 ECT(0) packets had reduced to zero, or near-zero, which might 1167 never happen. 1169 B.2. ECN Plus a Diffserv Codepoint (DSCP) 1171 Definition: 1173 For packets with a defined DSCP, all codepoints of the ECN field 1174 (except Not-ECT) would signify alternative L4S semantics to those 1175 for classic ECN [RFC3168], specifically: 1177 * The L4S DSCP would signifiy that the packet came from an L4S- 1178 capable sender; 1180 * ECT(0) and ECT(1) would both signify that the packet was 1181 travelling between transport endpoints that were both ECN- 1182 capable; 1184 * CE would signify that the packet had been marked by an AQM 1185 implementing the L4S service. 1187 Use of a DSCP is the only approach for alternative ECN semantics 1188 given as an example in [RFC4774]. However, it was perhaps considered 1189 more for controlled environments than new end-to-end services; 1191 Cons: 1193 Consumes DSCP pairs: A DSCP is obviously not orthogonal to Diffserv. 1194 Therefore, wherever the L4S service is applied to multiple 1195 Diffserv scheduling behaviours, it would be necessary to replace 1196 each DSCP with a pair of DSCPs. 1198 Uses critical lower-layer header space: The resulting increased 1199 number of DSCPs might be hard to support for some lower layer 1200 technologies, e.g. 802.1p and MPLS both offer only 3-bits for a 1201 maximum of 8 traffic class identifiers. Although L4S should 1202 reduce and possibly remove the need for some DSCPs intended for 1203 differentiated queuing delay, it will not remove the need for 1204 Diffserv entirely, because Diffserv is also used to allocate 1205 bandwidth, e.g. by prioritising some classes of traffic over 1206 others when traffic exceeds available capacity. 1208 Not end-to-end (host-network): Very few networks honour a DSCP set 1209 by a host. Typically a network will zero (bleach) the Diffserv 1210 field from all hosts. Sometimes networks will attempt to identify 1211 applications by some form of packet inspection and, based on 1212 network policy, they will set the DSCP considered appropriate for 1213 the identified application. Network-based application 1214 identification might use some combination of protocol ID, port 1215 numbers(s), application layer protocol headers, IP address(es), 1216 VLAN ID(s) and even packet timing. 1218 Not end-to-end (network-network): Very few networks honour a DSCP 1219 received from a neighbouring network. Typically a network will 1220 zero (bleach) the Diffserv field from all neighbouring networks at 1221 an interconnection point. Sometimes bilateral arrangements are 1222 made between networks, such that the receiving network remarks 1223 some DSCPs to those it uses for roughly equivalent services. The 1224 likelihood that a DSCP will be bleached or ignored depends on the 1225 type of DSCP: 1227 Local-use DSCP: These tend to be used to implement application- 1228 specific network policies, but a bilateral arrangement to 1229 remark certain DSCPs is often applied to DSCPs in the local-use 1230 range simply because it is easier not to change all of a 1231 network's internal configurations when a new arrangement is 1232 made with a neighbour; 1234 Global-use DSCP: These do not tend to be honoured across network 1235 interconnections more than local-use DSCPs. However, if two 1236 networks decide to honour certain of each other's DSCPs, the 1237 reconfiguration is a little easier if both of their globally 1238 recognised services are already represented by the relevant 1239 global-use DSCPs. 1241 Note that today a global-use DSCP gives little more assurance 1242 of end-to-end service than a local-use DSCP. In future the 1243 global-use range might give more assurance of end-to-end 1244 service than local-use, but it is unlikely that either 1245 assurance will be high, particularly given the hosts are 1246 included in the end-to-end path. 1248 Not all tunnels: Diffserv codepoints are often not propagated to the 1249 outer header when a packet is encapsulated by a tunnel header. 1250 DSCPs are propagated to the outer of uniform mode tunnels, but not 1251 pipe mode [RFC2983], and pipe mode is fairly common. 1253 ECN hard in some lower layers:: Because this approach uses both the 1254 Diffserv and ECN fields, an AQM wil only work at a lower layer if 1255 both can be supported. If individual network operators wished to 1256 deploy an AQM at a lower layer, they would usually propagate an IP 1257 Diffserv codepoint to the lower layer, using for example IEEE 1258 802.1p. However, the ECN capability is harder to propagate down 1259 to lower layers because few lower layers support it. 1261 Pros: 1263 Could migrate to e2e: If all usage of classic ECN migrates to usage 1264 of L4S, the DSCP would become redundant, and the ECN capability 1265 alone could eventually identify L4S packets without the 1266 interconnection problems of Diffserv detailed above, and without 1267 having permanently consumed more than one codepoint in the IP 1268 header. Although the DSCP does not generally function as an end- 1269 to-end identifier (see above), it could be used initially by 1270 individual ISPs to introduce the L4S service for their own locally 1271 generated traffic; 1273 B.3. ECN capability alone 1275 Definition: 1277 This approach uses ECN capability alone as the L4S identifier. It 1278 is only feasible if classic ECN is not widely deployed. The 1279 specific definition of codepoints would be: 1281 * Any ECN codepoint other than Not-ECT would signify an L4S- 1282 capable sender; 1284 * ECN codepoints would not be used for classic [RFC3168] ECN, and 1285 the classic network service would only be used for Not-ECT 1286 packets. 1288 This approach would only be feasible if 1290 A. it was generally agreed that there was little chance of any 1291 classic [RFC3168] ECN deployment in any network nodes; 1293 B. it was generally agreed that there was little chance of any 1294 client devices being deployed with classic [RFC3168] TCP-ECN 1295 on by default (note that classic TCP-ECN is already on-by- 1296 default on many servers); 1298 C. for TCP connections, developers of client OSs would all have 1299 to agree not to encourage further deployment of classic ECN. 1300 Specifically, at the start of a TCP connection classic ECN 1301 could be disabled during negotation of the ECN capability: 1303 + an L4S-capable host would have to disable ECN if the 1304 corresponding host did not support accurate ECN feedback 1305 [RFC7560], which is a prerequisite for the L4S service; 1307 + developers of operating systems for user devices would only 1308 enable ECN by default for TCP once the stack implemented 1309 L4S and accurate ECN feedback [RFC7560] including 1310 requesting accurate ECN feedback by default. 1312 Cons: 1314 Near-infeasible deployment constraints: The constraints for 1315 deployment above represent a highly unlikely, but not completely 1316 impossible, set of circumstances. If, despite the above measures, 1317 a pair of hosts did negotiate to use classic ECN, their packets 1318 would be classified into the same queue as L4S traffic, and if 1319 they had to compete with a long-running L4S flow they would get a 1320 very small capacity share; 1322 ECN hard in some lower layers: See the same issue with "ECT(1) and 1323 CE codepoints" (Appendix B.1); 1325 Non-L4S service for control packets: See the same issue with "ECT(1) 1326 and CE codepoints" (Appendix B.1). 1328 Pros: 1330 Consumes no additional codepoints: The ECT(1) codepoint and all 1331 spare Diffserv codepoints would remain available for future use; 1333 Should work e2e: As with "ECT(1) and CE codepoints" (Appendix B.1); 1335 Should work in tunnels: As with "ECT(1) and CE codepoints" 1336 (Appendix B.1). 1338 B.4. Protocol ID 1340 It has been suggested that a new ID in the IPv4 Protocol field or the 1341 IPv6 Next Header field could identify L4S packets. However this 1342 approach is ruled out by numerous problems: 1344 o A new protocol ID would need to be paired with the old one for 1345 each transport (TCP, SCTP, UDP, etc.); 1347 o In IPv6, there can be a sequence of Next Header fields, and it 1348 would not be obvious which one would be expected to identify a 1349 network service like L4S; 1351 o A new protocol ID would rarely provide an end-to-end service, 1352 because It is well-known that new protocol IDs are often blocked 1353 by numerous types of middlebox; 1355 o The approach is not a solution for AQMs below the IP layer; 1357 B.5. Source or destination addressing 1359 Locally, a network operator could arrange for L4S service to be 1360 applied based on source or destination addressing, e.g. packets from 1361 its own data centre and/or CDN hosts, packets to its business 1362 customers, etc. It could use addressing at any layer, e.g. IP 1363 addresses, MAC addresses, VLAN IDs, etc. Although addressing might 1364 be a useful tactical approach for a single ISP, it would not be a 1365 feasible approach to identify an end-to-end service like L4S. Even 1366 for a single ISP, it would require packet classifiers in buffers to 1367 be dependent on changing topology and address allocation decisions 1368 elsewhere in the network. Therefore this approach is not a feasible 1369 solution. 1371 B.6. Summary: Merits of Alternative Identifiers 1373 Table 1 provides a very high level summary of the pros and cons 1374 detailed against the schemes described respectively in Appendix B.2, 1375 Appendix B.3 and Appendix B.1, for six issues that set them apart. 1377 +--------------+--------------------+---------+--------------------+ 1378 | Issue | DSCP + ECN | ECN | ECT(1) + CE | 1379 +--------------+--------------------+---------+--------------------+ 1380 | | initial eventual | initial | initial eventual | 1381 | | | | | 1382 | end-to-end | N . . . ? . | . . Y | . . Y . . Y | 1383 | tunnels | . O . . O . | . . ? | . . ? . . Y | 1384 | lower layers | N . . . ? . | . O . | . O . . . ? | 1385 | codepoints | N . . . . ? | . . Y | N . . . . ? | 1386 | reordering | . . Y . . Y | . . Y | . O . . . ? | 1387 | ctrl pkts | . . Y . . Y | . O . | . O . . . ? | 1388 | | | | | 1389 | | | Note 1 | | 1390 +--------------+--------------------+---------+--------------------+ 1392 Note 1: Only feasible if classic ECN is obsolete. 1394 Table 1: Comparison of the Merits of Three Alternative Identifiers 1396 The schemes are scored based on both their capabilities now 1397 ('initial') and in the long term ('eventual'). The 'ECN' scheme 1398 shares the 'eventual' scores of the 'ECT(1) + CE' scheme. The scores 1399 are one of 'N, O, Y', meaning 'Poor', 'Ordinary', 'Good' 1400 respectively. The same scores are aligned vertically to aid the eye. 1401 A score of "?" in one of the positions means that this approach might 1402 optimisitically become this good, given sufficient effort. The table 1403 summarises the text and is not meant to be understandable without 1404 having read the text. 1406 Appendix C. Potential Competing Uses for the ECT(1) Codepoint 1408 The ECT(1) codepoint of the ECN field has already been assigned once 1409 for the ECN nonce [RFC3540], which has now been categorized as 1410 historic [RFC8311]. ECN is probably the only remaining field in the 1411 Internet Protocol that is common to IPv4 and IPv6 and still has 1412 potential to work end-to-end, with tunnels and with lower layers. 1413 Therefore, ECT(1) should not be reassigned to a different 1414 experimental use (L4S) without carefully assessing competing 1415 potential uses. These fall into the following categories: 1417 C.1. Integrity of Congestion Feedback 1419 Receiving hosts can fool a sender into downloading faster by 1420 suppressing feedback of ECN marks (or of losses if retransmissions 1421 are not necessary or available otherwise). 1423 The historic ECN nonce protocol [RFC3540] proposed that a TCP sender 1424 could set either of ECT(0) or ECT(1) in each packet of a flow and 1425 remember the sequence it had set. If any packet was lost or 1426 congestion marked, the receiver would miss that bit of the sequence. 1427 An ECN Nonce receiver had to feed back the least significant bit of 1428 the sum, so it could not suppress feedback of a loss or mark without 1429 a 50-50 chance of guessing the sum incorrectly. 1431 The ECN Nonce RFC [RFC3540] as been reclassified as historic, partly 1432 because other ways have been developed to protect TCP feedback 1433 integrity [RFC8311] that do not consume a codepoint in the IP header. 1434 So it is highly unlikely that ECT(1) will be needed for integrity 1435 protection in future. 1437 C.2. Notification of Less Severe Congestion than CE 1439 Various researchers have proposed to use ECT(1) as a less severe 1440 congestion notification than CE, particularly to enable flows to fill 1441 available capacity more quickly after an idle period, when another 1442 flow departs or when a flow starts, e.g. VCP [VCP], Queue View (QV) 1443 [QV]. 1445 Before assigning ECT(1) as an identifer for L4S, we must carefully 1446 consider whether it might be better to hold ECT(1) in reserve for 1447 future standardisation of rapid flow acceleration, which is an 1448 important and enduring problem [RFC6077]. 1450 Pre-Congestion Notification (PCN) is another scheme that assigns 1451 alternative semantics to the ECN field. It uses ECT(1) to signify a 1452 less severe level of pre-congestion notification than CE [RFC6660]. 1453 However, the ECN field only takes on the PCN semantics if packets 1454 carry a Diffserv codepoint defined to indicate PCN marking within a 1455 controlled environment. PCN is required to be applied solely to the 1456 outer header of a tunnel across the controlled region in order not to 1457 interfere with any end-to-end use of the ECN field. Therefore a PCN 1458 region on the path would not interfere with any of the L4S service 1459 identifiers proposed in Appendix B. 1461 Authors' Addresses 1463 Koen De Schepper 1464 Nokia Bell Labs 1465 Antwerp 1466 Belgium 1468 Email: koen.de_schepper@nokia.com 1469 URI: https://www.bell-labs.com/usr/koen.de_schepper 1470 Bob Briscoe (editor) 1471 CableLabs 1472 UK 1474 Email: ietf@bobbriscoe.net 1475 URI: http://bobbriscoe.net/