idnits 2.17.1 draft-ietf-tsvwg-ecn-l4s-id-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1337 has weird spacing: '...initial even...' == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: o it SHOULD not lead to some packets of a transport-layer flow being served by a different queue from others. -- The document date (May 5, 2017) is 2548 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Outdated reference: A later version (-04) exists of draft-bagnulo-tcpm-generalized-ecn-03 == Outdated reference: A later version (-28) exists of draft-ietf-tcpm-accurate-ecn-02 == Outdated reference: A later version (-07) exists of draft-ietf-tcpm-cubic-04 == Outdated reference: A later version (-10) exists of draft-ietf-tcpm-dctcp-05 == Outdated reference: A later version (-25) exists of draft-ietf-tsvwg-aqm-dualq-coupled-00 == Outdated reference: A later version (-22) exists of draft-ietf-tsvwg-ecn-encap-guidelines-08 == Outdated reference: A later version (-08) exists of draft-ietf-tsvwg-ecn-experimentation-00 == Outdated reference: A later version (-20) exists of draft-ietf-tsvwg-l4s-arch-00 == Outdated reference: A later version (-07) exists of draft-stewart-tsvwg-sctpecn-05 -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) -- Obsolete informational reference (is this intentional?): RFC 4960 (Obsoleted by RFC 9260) Summary: 0 errors (**), 0 flaws (~~), 12 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Transport Services (tsv) K. De Schepper 3 Internet-Draft Nokia Bell Labs 4 Intended status: Experimental B. Briscoe, Ed. 5 Expires: November 6, 2017 Simula Research Lab 6 May 5, 2017 8 Identifying Modified Explicit Congestion Notification (ECN) Semantics 9 for Ultra-Low Queuing Delay 10 draft-ietf-tsvwg-ecn-l4s-id-00 12 Abstract 14 This specification defines the identifier to be used on IP packets 15 for a new network service called low latency, low loss and scalable 16 throughput (L4S). It is similar to the original (or 'Classic') 17 Explicit Congestion Notification (ECN). 'Classic' ECN marking was 18 required to be equivalent to a drop, both when applied in the network 19 and when responded to by a transport. Unlike 'Classic' ECN marking, 20 for packets carrying the L4S identifier, the network applies marking 21 more immediately and more aggressively than drop, and the transport 22 response to each mark is reduced and smoothed relative to that for 23 drop. The two changes counterbalance each other so that the 24 throughput of an L4S flow will be roughly the same as a 'Classic' 25 flow under the same conditions. However, the much more frequent 26 control signals and the finer responses to them result in ultra-low 27 queuing delay without compromising link utilization, even during high 28 load. Examples of new active queue management (AQM) marking 29 algorithms and examples of new transports (whether TCP-like or real- 30 time) are specified separately. The new L4S identifier is the key 31 piece that enables them to interwork and distinguishes them from 32 'Classic' traffic. It gives an incremental migration path so that 33 existing 'Classic' TCP traffic will be no worse off, but it can be 34 prevented from degrading the ultra-low delay and loss of the new 35 scalable transports. 37 Status of This Memo 39 This Internet-Draft is submitted in full conformance with the 40 provisions of BCP 78 and BCP 79. 42 Internet-Drafts are working documents of the Internet Engineering 43 Task Force (IETF). Note that other groups may also distribute 44 working documents as Internet-Drafts. The list of current Internet- 45 Drafts is at http://datatracker.ietf.org/drafts/current/. 47 Internet-Drafts are draft documents valid for a maximum of six months 48 and may be updated, replaced, or obsoleted by other documents at any 49 time. It is inappropriate to use Internet-Drafts as reference 50 material or to cite them other than as "work in progress." 52 This Internet-Draft will expire on November 6, 2017. 54 Copyright Notice 56 Copyright (c) 2017 IETF Trust and the persons identified as the 57 document authors. All rights reserved. 59 This document is subject to BCP 78 and the IETF Trust's Legal 60 Provisions Relating to IETF Documents 61 (http://trustee.ietf.org/license-info) in effect on the date of 62 publication of this document. Please review these documents 63 carefully, as they describe your rights and restrictions with respect 64 to this document. Code Components extracted from this document must 65 include Simplified BSD License text as described in Section 4.e of 66 the Trust Legal Provisions and are provided without warranty as 67 described in the Simplified BSD License. 69 Table of Contents 71 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 72 1.1. Problem . . . . . . . . . . . . . . . . . . . . . . . . . 4 73 1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 74 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 75 2. L4S Packet Identifier . . . . . . . . . . . . . . . . . . . . 7 76 2.1. Consensus Choice of L4S Packet Identifier: Requirements . 7 77 2.2. L4S Packet Identification at Run-Time . . . . . . . . . . 8 78 2.3. Pre-Requisite Transport Layer Behaviour . . . . . . . . . 8 79 2.3.1. Pre-Requisite Congestion Response . . . . . . . . . . 8 80 2.3.2. Pre-Requisite Transport Feedback . . . . . . . . . . 9 81 2.4. Exception for L4S Packet Identification by Network Nodes 82 with Transport-Layer Awareness . . . . . . . . . . . . . 10 83 2.5. The Meaning of L4S CE Relative to Drop . . . . . . . . . 11 84 3. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 85 4. Security Considerations . . . . . . . . . . . . . . . . . . . 12 86 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 87 6. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 88 6.1. Normative References . . . . . . . . . . . . . . . . . . 12 89 6.2. Informative References . . . . . . . . . . . . . . . . . 13 90 Appendix A. The 'TCP Prague Requirements' . . . . . . . . . . . 17 91 A.1. Requirements for Scalable Transport Protocols . . . . . . 18 92 A.1.1. Use of L4S Packet Identifier . . . . . . . . . . . . 18 93 A.1.2. Accurate ECN Feedback . . . . . . . . . . . . . . . . 18 94 A.1.3. Fall back to Reno/Cubic congestion control on packet 95 loss . . . . . . . . . . . . . . . . . . . . . . . . 19 96 A.1.4. Fall back to Reno/Cubic congestion control on classic 97 ECN bottlenecks . . . . . . . . . . . . . . . . . . . 19 98 A.1.5. Reduce RTT dependence . . . . . . . . . . . . . . . . 20 99 A.1.6. Scaling down the congestion window . . . . . . . . . 20 100 A.2. Scalable Transport Protocol Optimizations . . . . . . . . 21 101 A.2.1. Setting ECT in TCP Control Packets and 102 Retransmissions . . . . . . . . . . . . . . . . . . . 21 103 A.2.2. Faster than Additive Increase . . . . . . . . . . . . 21 104 A.2.3. Faster Convergence at Flow Start . . . . . . . . . . 22 105 Appendix B. Alternative Identifiers . . . . . . . . . . . . . . 22 106 B.1. ECT(1) and CE codepoints . . . . . . . . . . . . . . . . 22 107 B.2. ECN Plus a Diffserv Codepoint (DSCP) . . . . . . . . . . 24 108 B.3. ECN capability alone . . . . . . . . . . . . . . . . . . 27 109 B.4. Protocol ID . . . . . . . . . . . . . . . . . . . . . . . 28 110 B.5. Source or destination addressing . . . . . . . . . . . . 28 111 B.6. Summary: Merits of Alternative Identifiers . . . . . . . 29 112 Appendix C. Potential Competing Uses for the ECT(1) Codepoint . 29 113 C.1. Integrity of Congestion Feedback . . . . . . . . . . . . 30 114 C.2. Notification of Less Severe Congestion than CE . . . . . 31 115 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 117 1. Introduction 119 This specification defines the identifier to be used on IP packets 120 for a new network service called low latency, low loss and scalable 121 throughput (L4S). It is similar to the original (or 'Classic') 122 Explicit Congestion Notification (ECN). 'Classic' ECN marking was 123 required to be equivalent to a drop, both when applied in the network 124 and when responded to by a transport. Unlike 'Classic' ECN marking, 125 the network applies L4S marking more immediately and more 126 aggressively than drop, and the transport response to each mark is 127 reduced and smoothed relative to that for drop. The two changes 128 counterbalance each other so that the bit-rate of an L4S flow will be 129 roughly the same as a 'Classic' flow under the same conditions. 130 Nonetheless, the much more frequent control signals and the finer 131 responses to them result in ultra-low queuing delay without 132 compromising link utilization, even during high load. 134 An example of an active queue management (AQM) marking algorithm that 135 enables the L4S service is the DualQ Coupled AQM defined in a 136 complementary specification [I-D.ietf-tsvwg-aqm-dualq-coupled]. An 137 example of a scalable transport that would enable the L4S service is 138 Data Centre TCP (DCTCP), which until now has been applicable solely 139 to controlled environments like data centres [I-D.ietf-tcpm-dctcp], 140 because it is too aggressive to co-exist with existing TCP. However, 141 AQMs like DualQ Coupled enable scalable transports like DCTCP to co- 142 exist with existing traffic, each getting roughly the same flow rate 143 when they compete under similar conditions. Note that DCTCP will 144 still not be safe to deploy on the Internet until it satisfies the 145 requirements listed in Section 2.3. 147 The new L4S identifier is the key piece that enables these two parts 148 to interwork and distinguishes them from 'Classic' traffic. It gives 149 an incremental migration path so that existing 'Classic' TCP traffic 150 will be no worse off, but it can be prevented from degrading the 151 ultra-low delay and loss of the new scalable transports. The 152 performance improvement is so great that it is hoped it will motivate 153 initial deployment of the separate parts of this system. 155 1.1. Problem 157 Latency is becoming the critical performance factor for many (most?) 158 applications on the public Internet, e.g. Web, voice, conversational 159 video, gaming, finance apps, remote desktop and cloud-based 160 applications. In the developed world, further increases in access 161 network bit-rate offer diminishing returns, whereas latency is still 162 a multi-faceted problem. In the last decade or so, much has been 163 done to reduce propagation time by placing caches or servers closer 164 to users. However, queuing remains a major component of latency. 166 The Diffserv architecture provides Expedited Forwarding [RFC3246], so 167 that low latency traffic can jump the queue of other traffic. 168 However, on access links dedicated to individual sites (homes, small 169 enterprises or mobile devices), often all traffic at any one time 170 will be latency-sensitive. Then Diffserv is of little use. Instead, 171 we need to remove the causes of any unnecessary delay. 173 The bufferbloat project has shown that excessively-large buffering 174 ('bufferbloat') has been introducing significantly more delay than 175 the underlying propagation time. These delays appear only 176 intermittently--only when a capacity-seeking (e.g. TCP) flow is long 177 enough for the queue to fill the buffer, making every packet in other 178 flows sharing the buffer sit through the queue. 180 Active queue management (AQM) was originally developed to solve this 181 problem (and others). Unlike Diffserv, which gives low latency to 182 some traffic at the expense of others, AQM controls latency for _all_ 183 traffic in a class. In general, AQMs introduce an increasing level 184 of discard from the buffer the longer the queue persists above a 185 shallow threshold. This gives sufficient signals to capacity-seeking 186 (aka. greedy) flows to keep the buffer empty for its intended 187 purpose: absorbing bursts. However, RED [RFC2309] and other 188 algorithms from the 1990s were sensitive to their configuration and 189 hard to set correctly. So, AQM was not widely deployed. 191 More recent state-of-the-art AQMs, e.g. 192 fq_CoDel [I-D.ietf-aqm-fq-codel], PIE [I-D.ietf-aqm-pie], Adaptive 193 RED [ARED01], are easier to configure, because they define the 194 queuing threshold in time not bytes, so it is invariant for different 195 link rates. However, no matter how good the AQM, the sawtoothing 196 rate of TCP will either cause queuing delay to vary or cause the link 197 to be under-utilized. Even with a perfectly tuned AQM, the 198 additional queuing delay will be of the same order as the underlying 199 speed-of-light delay across the network. Flow-queuing can isolate 200 one flow from another, but it cannot isolate a TCP flow from the 201 delay variations it inflicts on itself, and it has other problems - 202 it overrides the flow rate decisions of variable rate video 203 applications, it does not recognise the flows within IPSec VPN 204 tunnels and it is relatively expensive to implement. 206 Latency is not our only concern: It was known when TCP was first 207 developed that it would not scale to high bandwidth-delay products. 208 Given regular broadband bit-rates over WAN distances are 209 already [RFC3649] beyond the scaling range of 'Classic' TCP Reno, 210 'less unscalable' Cubic [I-D.ietf-tcpm-cubic] and 211 Compound [I-D.sridharan-tcpm-ctcp] variants of TCP have been 212 successfully deployed. However, these are now approaching their 213 scaling limits. Unfortunately, fully scalable TCPs such as DCTCP 214 [I-D.ietf-tcpm-dctcp] cause 'Classic' TCP to starve itself, which is 215 why they have been confined to private data centres or research 216 testbeds (until now). 218 It turns out that a TCP algorithm like DCTCP that solves TCP's 219 scalability problem also solves the latency problem, because the 220 finer sawteeth cause very little queuing delay. A supporting paper 221 [DCttH15] gives the full explanation of why the design solves both 222 the latency and the scaling problems, both in plain English and in 223 more precise mathematical form. The explanation is summarised 224 without the maths in the L4S architecture document 225 [I-D.ietf-tsvwg-l4s-arch]. 227 1.2. Terminology 229 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 230 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 231 "OPTIONAL" in this document are to be interpreted as described in 232 [RFC2119]. In this document, these words will appear with that 233 interpretation only when in ALL CAPS. Lower case uses of these words 234 are not to be interpreted as carrying RFC-2119 significance. 236 Classic service: The 'Classic' service is intended for all the 237 behaviours that currently co-exist with TCP Reno (e.g. TCP Cubic, 238 Compound, SCTP, etc). 240 Low-Latency, Low-Loss and Scalable (L4S) service: The 'L4S' service 241 is intended for traffic from scalable TCP algorithms such as Data 242 Centre TCP. But it is also more general--it will allow a set of 243 congestion controls with similar scaling properties to DCTCP (e.g. 244 Relentless [Mathis09]) to evolve. 246 Both Classic and L4S services can cope with a proportion of 247 unresponsive or less-responsive traffic as well (e.g. DNS, VoIP, 248 etc). 250 Classic ECN: The original Explicit Congestion Notification (ECN) 251 protocol [RFC3168]. 253 1.3. Scope 255 The new L4S identifier defined in this specification is applicable 256 for IPv4 and IPv6 packets (as for classic ECN [RFC3168]). It is 257 applicable for the unicast, multicast and anycast forwarding modes. 258 It is an orthogonal packet classification to Differentiated Services 259 (Diffserv [RFC2474]), therefore it can be applied to any packet in 260 any Diffserv traffic class. However, as with classic ECN, any 261 particular forwarding node might not implement an active queue 262 management algorithm in all its Diffserv queues. 264 This document is intended for experimental status, so it does not 265 update any standards track RFCs. Therefore it depends on 266 [I-D.ietf-tsvwg-ecn-experimentation], which: 268 o updates the ECN proposed standard [RFC3168] (in certain specified 269 cases including the present document) to relax the requirement 270 that an ECN mark must be equivalent to a drop, both when applied 271 by the network, and when responded to by the sender; 273 o makes the case for changing the status of the experimental ECN 274 nonce [RFC3540] to historic (see Appendix C.1 for rationale); 276 o makes consequent updates to the following proposed standard RFCs 277 to reflect the above two bullets: 279 * ECN for RTP [RFC6679]; 281 * the congestion control specifications of various DCCP 282 congestion control identifier (CCID) profiles [RFC4341], 283 [RFC4342], [RFC5622]. 285 2. L4S Packet Identifier 287 2.1. Consensus Choice of L4S Packet Identifier: Requirements 289 This subsection briefly records the process that led to a consensus 290 choice of L4S identifier, selected from all the alternatives in 291 Appendix B. 293 Ideally, the identifier for packets using the Low Latency, Low Loss, 294 Scalable throughput (L4S) service ought to meet the following 295 requirements: 297 o it SHOULD survive end-to-end between source and destination 298 applications: across the boundary between host and network, 299 between interconnected networks, and through middleboxes; 301 o it SHOULD be common to IPv4 and IPv6 and transport agnostic; 303 o it SHOULD be incrementally deployable; 305 o it SHOULD enable an AQM to classify packets encapsulated by outer 306 IP or lower-layer headers; 308 o it SHOULD consume minimal extra codepoints; 310 o it SHOULD not lead to some packets of a transport-layer flow being 311 served by a different queue from others. 313 Whether the identifier would be recoverable if the experiment failed 314 is a factor that could be taken into account. However, this has not 315 been made a requirement, because that would favour schemes that would 316 be easier to fail, rather than those more likely to succeed. 318 It is recognised that the chosen identifier is unlikely to satisfy 319 all these requirements, particularly given the limited space left in 320 the IP header. Therefore a compromise will be necessary, which is 321 why all the requirements are expressed with the word 'SHOULD' not 322 'MUST'. Appendix B discusses the pros and cons of the compromises 323 made in various competing identification schemes against the above 324 requirements. 326 On the basis of this analysis, "ECT(1) and CE codepoints" is the best 327 compromise. Therefore this scheme is defined in detail in the 328 following sections, while Appendix B records the rationale for this 329 decision. 331 2.2. L4S Packet Identification at Run-Time 333 The L4S treatment is an alternative packet marking treatment 334 [RFC4774] to the classic ECN treatment [RFC3168]. Like classic ECN, 335 it identifies both network and host behaviour: it identifies the 336 marking treatment that network nodes are expected to apply to L4S 337 packets, and it identifies packets that have been sent from hosts 338 that are expected to comply with a broad type of behaviour. 340 For a packet to receive L4S treatment as it is forwarded, the sender 341 MUST set the ECN field in the IP header (v4 or v6) to the ECT(1) 342 codepoint. 344 A network node that implements the L4S service MUST classify arriving 345 ECT(1) packets for L4S treatment and it SHOULD classify arriving CE 346 packets for L4S treatment as well. Section 2.4 describes a possible 347 exception to this latter rule. 349 An L4S AQM treatment follows similar codepoint transition rules to 350 those in RFC 3168. Specifically, the ECT(1) codepoint MUST NOT be 351 changed to any other codepoint than CE, and CE MUST NOT be changed to 352 any other codepoint. An ECT(1) packet is classified as ECN-capable 353 and, if congestion increases, an L4S AQM algorithm will mark the ECN 354 field as CE for an increasing proportion of packets, otherwise 355 forwarding packets unchanged as ECT(1). Necessary conditions for an 356 L4S marking treatment are defined in Section 2.5. Under persistent 357 overload an L4S marking treatment SHOULD turn off ECN marking, using 358 drop as a congestion signal until the overload episode has subsided, 359 as recommended for all AQMs [RFC7567] (Section 4.2.1), which follows 360 similar advice in RFC 3168 (Section 7). 362 The L4S treatment is the default for ECT(1) packets in all Diffserv 363 Classes [RFC4774]. 365 For backward compatibility in uncontrolled environments, a network 366 node that implements the L4S treatment MUST also implement a classic 367 AQM treatment. It MUST classify arriving ECT(0) and Not-ECT packets 368 for treatment by the Classic AQM. Classic treatment means that the 369 AQM will mark ECT(0) packets under the same conditions as it would 370 drop Not-ECT packets [RFC3168]. 372 2.3. Pre-Requisite Transport Layer Behaviour 374 2.3.1. Pre-Requisite Congestion Response 376 For a host to send packets with the L4S identifier (ECT(1)), it 377 SHOULD implement a congestion control behaviour that ensures the flow 378 rate is inversely proportional to the proportion of bytes in packets 379 marked with the CE codepoint. This is termed a scalable congestion 380 control, because the number of control signals (ECN marks) per round 381 trip remains roughly constant for any flow rate. As with all 382 transport behaviours, a detailed specification will need to be 383 defined for each type of transport or application, including the 384 timescale over which the proportionality is averaged, and control of 385 burstiness. The inverse proportionality requirement above is worded 386 as a 'SHOULD' rather than a 'MUST' to allow reasonable flexibility 387 when defining these specifications. 389 Data Center TCP (DCTCP [I-D.ietf-tcpm-dctcp]) is an example of a 390 scalable congestion control. 392 Each sender in a session can use a scalable congestion control 393 independently of the congestion control used by the receiver(s) when 394 they send data. Therefore there might be ECT(1) packets in one 395 direction and ECT(0) in the other. 397 In order to coexist safely with other Internet traffic, a scalable 398 congestion control MUST NOT identify its packets with the ECT(1) 399 codepoint unless it complies with the following bulleted 400 requirements. The specification of a particular scalable congestion 401 control MUST describe in detail how it satisfies each requirement: 403 o A scalable congestion control MUST react to packet loss in a way 404 that will coexist safely with a TCP Reno congestion control 405 [RFC5681] (see Appendix A.1.3 for rationale). 407 o A scalable congestion control MUST react to ECN marking from a 408 non-L4S but ECN-capable bottleneck in a way that will coexist with 409 a TCP Reno congestion control [RFC5681] (see Appendix A.1.4 for 410 rationale). 412 o A scalable congestion control MUST reduce or eliminate RTT bias 413 over as wide a range of RTTs as possible, or at least over the 414 typical range of RTTs that will interact in the intended 415 deployment scenario (see Appendix A.1.5 for rationale). 417 o A scalable congestion control MUST remain responsive to congestion 418 when the RTT is significantly smaller than in the current public 419 Internet (see Appendix A.1.6 for rationale). 421 2.3.2. Pre-Requisite Transport Feedback 423 In general, a scalable congestion control needs feedback of the 424 extent of CE marking on the forward path. Due to the history of TCP 425 development, when ECN was added it reported no more than one CE mark 426 per round trip. Some transport protocols derived from TCP mimic this 427 behaviour while others report the accurate extent of TCP marking. 428 This means that some transport protocols will need to be updated as a 429 prerequisite for scalable congestion control. The position for a few 430 well-known transport protocols is given below. 432 TCP: Support for accurate ECN feedback (AccECN 433 [I-D.ietf-tcpm-accurate-ecn]) by both ends is a prerequisite for 434 scalable congestion control. Therefore, the presence of ECT(1) in 435 the IP headers even in one direction of a TCP connection will 436 imply that both ends support AccECN. However, the converse does 437 not apply. So even if both ends support AccECN, either of the two 438 ends can choose not to use a scalable congestion control, whatever 439 the other end's choice. 441 SCTP: An ECN feedback protocol such as that specified in 442 [I-D.stewart-tsvwg-sctpecn] would be a prerequisite for scalable 443 congestion control. That draft would update the ECN feedback 444 protocol sketched out in Appendix A of the standards track 445 specification of SCTP [RFC4960] by adding a field to report the 446 number of CE marks. 448 RTP over UDP: A prerequisite for scalable congestion control is for 449 both (all) ends of one media-level hop to signal ECN support using 450 the ecn-capable-rtp attribute [RFC6679]. Therefore, the presence 451 of ECT(1) implies that both (all) ends of that hop support ECN. 452 However, the converse does not apply, so each end of a media-level 453 hop can independently choose not to use a scalable congestion 454 control, even if both ends support ECN. 456 DCCP: The ACK vector in DCCP [RFC4340] is already sufficient to 457 report the extent of CE marking as needed by a scalable congestion 458 control. 460 2.4. Exception for L4S Packet Identification by Network Nodes with 461 Transport-Layer Awareness 463 To implement the L4S treatment, a network node does not need to 464 identify transport-layer flows. Nonetheless, if an implementer is 465 willing to identify transport-layer flows at a network node, and if 466 the most recent ECT packet in the same flow was ECT(0), the node MAY 467 classify CE packets for classic ECN [RFC3168] treatment. In all 468 other cases, a network node MUST classify CE packets for L4S 469 treatment. Examples of such other cases are: i) if no ECT packets 470 have yet been identified in a flow; ii) if it is not desirable for a 471 network node to identify transport-layer flows; or iii) if the most 472 recent ECT packet in a flow was ECT(1). 474 If an implementer uses flow-awareness to classify CE packets, to 475 determine whether the flow is using ECT(0) or ECT(1) it only uses the 476 most recent ECT packet of a flow {ToDo: this advice will need to be 477 verified experimentally}. This is because a sender might have to 478 switch from sending ECT(1) (L4S) packets to sending ECT(0) (Classic) 479 packets, or back again, in the middle of a transport-layer flow. 480 Such a switch-over is likely to be very rare, but It could be 481 necessary if the path bottleneck moves from a network node that 482 supports L4S to one that only supports Classic ECN. A host ought to 483 be able to detect such a change from a change in RTT variation. 485 2.5. The Meaning of L4S CE Relative to Drop 487 The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST 488 be roughly proportional to the square of the likelihood that it would 489 have marked it if it had been an L4S packet (p_L). That is 491 p_C ~= (p_L / k)^2 493 The constant of proportionality (k) does not have to be standardised 494 for interoperability, but a value of 2 is RECOMMENDED. 496 [I-D.ietf-tsvwg-aqm-dualq-coupled] specifies the essential aspects of 497 an L4S AQM, as well as recommending other aspects. It gives example 498 implementations in appendices. 500 The term 'likelihood' is used above to allow for marking and dropping 501 to be either probabilistic or deterministic. The example AQMs in 502 [I-D.ietf-tsvwg-aqm-dualq-coupled] drop and mark probabilistically, 503 so the drop probability is arranged to be the square of the marking 504 probability. Nonetheless, an alternative AQM that dropped and marked 505 deterministically would be valid, as long as the dropping frequency 506 was proportional to the square of the marking frequency. 508 Note that, contrary to RFC 3168, an AQM implementing the L4S and 509 Classic treatments does not mark an ECT(1) packet under the same 510 conditions that it would have dropped a Not-ECT packet. However, it 511 does mark an ECT(0) packet under the same conditions that it would 512 have dropped a Not-ECT packet. 514 3. IANA Considerations 516 This specification contains no IANA considerations. 518 4. Security Considerations 520 Approaches to assure the integrity of signals using the new identifer 521 are introduced in Appendix C.1. 523 5. Acknowledgements 525 Thanks to Richard Scheffenegger, John Leslie, David Taeht, Jonathan 526 Morton, Gorry Fairhurst, Michael Welzl, Mikael Abrahamsson and Andrew 527 McGregor for the discussions that led to this specification. Ing-jyh 528 (Inton) Tsang was a contributor to the early drafts of this document. 529 inAppendix A listing the TCP Prague Requirements is based on text 530 authored by Marcelo Bagnulo Braun that was originally an appendix to 531 [I-D.ietf-tsvwg-l4s-arch]. That text was in turn based on the 532 collective output of the attendees listed in the minutes of a 'bar 533 BoF' on DCTCP Evolution during IETF-94 [TCPPrague]. 535 The authors' contributions were part-funded by the European Community 536 under its Seventh Framework Programme through the Reducing Internet 537 Transport Latency (RITE) project (ICT-317700). The views expressed 538 here are solely those of the authors. 540 6. References 542 6.1. Normative References 544 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 545 Requirement Levels", BCP 14, RFC 2119, 546 DOI 10.17487/RFC2119, March 1997, 547 . 549 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 550 of Explicit Congestion Notification (ECN) to IP", 551 RFC 3168, DOI 10.17487/RFC3168, September 2001, 552 . 554 [RFC4774] Floyd, S., "Specifying Alternate Semantics for the 555 Explicit Congestion Notification (ECN) Field", BCP 124, 556 RFC 4774, DOI 10.17487/RFC4774, November 2006, 557 . 559 [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., 560 and K. Carlberg, "Explicit Congestion Notification (ECN) 561 for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 562 2012, . 564 6.2. Informative References 566 [Alizadeh-stability] 567 Alizadeh, M., Javanmard, A., and B. Prabhakar, "Analysis 568 of DCTCP: Stability, Convergence, and Fairness", ACM 569 SIGMETRICS 2011 , June 2011. 571 [ARED01] Floyd, S., Gummadi, R., and S. Shenker, "Adaptive RED: An 572 Algorithm for Increasing the Robustness of RED's Active 573 Queue Management", ACIRI Technical Report , August 2001, 574 . 576 [DCttH15] De Schepper, K., Bondarenko, O., Briscoe, B., and I. 577 Tsang, "'Data Centre to the Home': Ultra-Low Latency for 578 All", 2015, . 581 (Under submission) 583 [I-D.bagnulo-tcpm-generalized-ecn] 584 Bagnulo, M. and B. Briscoe, "Adding Explicit Congestion 585 Notification (ECN) to TCP control packets and TCP 586 retransmissions", draft-bagnulo-tcpm-generalized-ecn-03 587 (work in progress), April 2017. 589 [I-D.bagnulo-tswg-generalized-ecn] 590 Bagnulo, M. and B. Briscoe, "Adding Explicit Congestion 591 Notification (ECN) to TCP control packets", draft-bagnulo- 592 tswg-generalized-ecn-00 (work in progress), July 2016. 594 [I-D.ietf-aqm-fq-codel] 595 Hoeiland-Joergensen, T., McKenney, P., 596 dave.taht@gmail.com, d., Gettys, J., and E. Dumazet, "The 597 FlowQueue-CoDel Packet Scheduler and Active Queue 598 Management Algorithm", draft-ietf-aqm-fq-codel-06 (work in 599 progress), March 2016. 601 [I-D.ietf-aqm-pie] 602 Pan, R., Natarajan, P., Baker, F., and G. White, "PIE: A 603 Lightweight Control Scheme To Address the Bufferbloat 604 Problem", draft-ietf-aqm-pie-10 (work in progress), 605 September 2016. 607 [I-D.ietf-tcpm-accurate-ecn] 608 Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More 609 Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate- 610 ecn-02 (work in progress), October 2016. 612 [I-D.ietf-tcpm-cubic] 613 Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and 614 R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", 615 draft-ietf-tcpm-cubic-04 (work in progress), February 616 2017. 618 [I-D.ietf-tcpm-dctcp] 619 Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L., 620 and G. Judd, "Datacenter TCP (DCTCP): TCP Congestion 621 Control for Datacenters", draft-ietf-tcpm-dctcp-05 (work 622 in progress), March 2017. 624 [I-D.ietf-tsvwg-aqm-dualq-coupled] 625 Schepper, K., Briscoe, B., Bondarenko, O., and I. Tsang, 626 "DualQ Coupled AQM for Low Latency, Low Loss and Scalable 627 Throughput", draft-ietf-tsvwg-aqm-dualq-coupled-00 (work 628 in progress), November 2016. 630 [I-D.ietf-tsvwg-ecn-encap-guidelines] 631 Briscoe, B., Kaippallimalil, J., and P. Thaler, 632 "Guidelines for Adding Congestion Notification to 633 Protocols that Encapsulate IP", draft-ietf-tsvwg-ecn- 634 encap-guidelines-08 (work in progress), March 2017. 636 [I-D.ietf-tsvwg-ecn-experimentation] 637 Black, D., "Explicit Congestion Notification (ECN) 638 Experimentation", draft-ietf-tsvwg-ecn-experimentation-00 639 (work in progress), November 2016. 641 [I-D.ietf-tsvwg-l4s-arch] 642 Briscoe, B., Schepper, K., and M. Bagnulo, "Low Latency, 643 Low Loss, Scalable Throughput (L4S) Internet Service: 644 Architecture", draft-ietf-tsvwg-l4s-arch-00 (work in 645 progress), November 2016. 647 [I-D.moncaster-tcpm-rcv-cheat] 648 Moncaster, T., Briscoe, B., and A. Jacquet, "A TCP Test to 649 Allow Senders to Identify Receiver Non-Compliance", draft- 650 moncaster-tcpm-rcv-cheat-03 (work in progress), July 2014. 652 [I-D.sridharan-tcpm-ctcp] 653 Sridharan, M., Tan, K., Bansal, D., and D. Thaler, 654 "Compound TCP: A New TCP Congestion Control for High-Speed 655 and Long Distance Networks", draft-sridharan-tcpm-ctcp-02 656 (work in progress), November 2008. 658 [I-D.stewart-tsvwg-sctpecn] 659 Stewart, R., Tuexen, M., and X. Dong, "ECN for Stream 660 Control Transmission Protocol (SCTP)", draft-stewart- 661 tsvwg-sctpecn-05 (work in progress), January 2014. 663 [Mathis09] 664 Mathis, M., "Relentless Congestion Control", PFLDNeT'09 , 665 May 2009, . 668 [QV] Briscoe, B. and P. Hurtig, "Up to Speed with Queue View", 669 RITE Technical Report D2.3; Appendix C.2, August 2015, 670 . 673 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 674 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 675 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 676 S., Wroclawski, J., and L. Zhang, "Recommendations on 677 Queue Management and Congestion Avoidance in the 678 Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998, 679 . 681 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 682 "Definition of the Differentiated Services Field (DS 683 Field) in the IPv4 and IPv6 Headers", RFC 2474, 684 DOI 10.17487/RFC2474, December 1998, 685 . 687 [RFC2983] Black, D., "Differentiated Services and Tunnels", 688 RFC 2983, DOI 10.17487/RFC2983, October 2000, 689 . 691 [RFC3246] Davie, B., Charny, A., Bennet, J., Benson, K., Le Boudec, 692 J., Courtney, W., Davari, S., Firoiu, V., and D. 693 Stiliadis, "An Expedited Forwarding PHB (Per-Hop 694 Behavior)", RFC 3246, DOI 10.17487/RFC3246, March 2002, 695 . 697 [RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit 698 Congestion Notification (ECN) Signaling with Nonces", 699 RFC 3540, DOI 10.17487/RFC3540, June 2003, 700 . 702 [RFC3649] Floyd, S., "HighSpeed TCP for Large Congestion Windows", 703 RFC 3649, DOI 10.17487/RFC3649, December 2003, 704 . 706 [RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram 707 Congestion Control Protocol (DCCP)", RFC 4340, 708 DOI 10.17487/RFC4340, March 2006, 709 . 711 [RFC4341] Floyd, S. and E. Kohler, "Profile for Datagram Congestion 712 Control Protocol (DCCP) Congestion Control ID 2: TCP-like 713 Congestion Control", RFC 4341, DOI 10.17487/RFC4341, March 714 2006, . 716 [RFC4342] Floyd, S., Kohler, E., and J. Padhye, "Profile for 717 Datagram Congestion Control Protocol (DCCP) Congestion 718 Control ID 3: TCP-Friendly Rate Control (TFRC)", RFC 4342, 719 DOI 10.17487/RFC4342, March 2006, 720 . 722 [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", 723 RFC 4960, DOI 10.17487/RFC4960, September 2007, 724 . 726 [RFC5562] Kuzmanovic, A., Mondal, A., Floyd, S., and K. 727 Ramakrishnan, "Adding Explicit Congestion Notification 728 (ECN) Capability to TCP's SYN/ACK Packets", RFC 5562, 729 DOI 10.17487/RFC5562, June 2009, 730 . 732 [RFC5622] Floyd, S. and E. Kohler, "Profile for Datagram Congestion 733 Control Protocol (DCCP) Congestion ID 4: TCP-Friendly Rate 734 Control for Small Packets (TFRC-SP)", RFC 5622, 735 DOI 10.17487/RFC5622, August 2009, 736 . 738 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 739 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 740 . 742 [RFC5925] Touch, J., Mankin, A., and R. Bonica, "The TCP 743 Authentication Option", RFC 5925, DOI 10.17487/RFC5925, 744 June 2010, . 746 [RFC6077] Papadimitriou, D., Ed., Welzl, M., Scharf, M., and B. 747 Briscoe, "Open Research Issues in Internet Congestion 748 Control", RFC 6077, DOI 10.17487/RFC6077, February 2011, 749 . 751 [RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three 752 Pre-Congestion Notification (PCN) States in the IP Header 753 Using a Single Diffserv Codepoint (DSCP)", RFC 6660, 754 DOI 10.17487/RFC6660, July 2012, 755 . 757 [RFC7560] Kuehlewind, M., Ed., Scheffenegger, R., and B. Briscoe, 758 "Problem Statement and Requirements for Increased Accuracy 759 in Explicit Congestion Notification (ECN) Feedback", 760 RFC 7560, DOI 10.17487/RFC7560, August 2015, 761 . 763 [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF 764 Recommendations Regarding Active Queue Management", 765 BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, 766 . 768 [RFC7713] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) 769 Concepts, Abstract Mechanism, and Requirements", RFC 7713, 770 DOI 10.17487/RFC7713, December 2015, 771 . 773 [TCP-sub-mss-w] 774 Briscoe, B. and K. De Schepper, "Scaling TCP's Congestion 775 Window for Small Round Trip Times", BT Technical Report 776 TR-TUB8-2015-002, May 2015, 777 . 780 [TCPPrague] 781 Briscoe, B., "Notes: DCTCP evolution 'bar BoF': Tue 21 Jul 782 2015, 17:40, Prague", tcpprague mailing list archive , 783 July 2015, . 786 [VCP] Xia, Y., Subramanian, L., Stoica, I., and S. Kalyanaraman, 787 "One more bit is enough", Proc. SIGCOMM'05, ACM CCR 788 35(4)37--48, 2005, 789 . 791 Appendix A. The 'TCP Prague Requirements' 793 This appendix is informative, not normative. It gives a list of 794 modifications to current scalable transport protocols so that they 795 can be deployed over the public Internet and coexist safely with 796 existing traffic. The list complements the normative requirements in 797 Section 2.3 that a sender has to comply with before it can set the 798 L4S identifier in packets it sends into the Internet. As well as 799 necessary safety improvements (requirements) this appendix also 800 includes preferable performance improvements (optimizations). 802 These recommendations have become know as the TCP Prague 803 Requirements, because they were originally identified at an ad hoc 804 meeting during IETF-94 in Prague [TCPPrague]. The wording has been 805 generalized to apply to all scalable congestion controls, not just 806 TCP congestion control specifically. 808 DCTCP [I-D.ietf-tcpm-dctcp] is currently the most widely used 809 scalable transport protocol. In its current form, DCTCP is specified 810 to be deployable only in controlled environments. Deploying it in 811 the public Internet would lead to a number of issues, both from the 812 safety and the performance perspective. The modifications and 813 additional mechanisms listed in this section will be necessary for 814 its deployment over the global Internet. Where an example is needed, 815 DCTCP is used as a base, but it is likely that most of these 816 requirements equally apply to other scalable transport protocols. 818 A.1. Requirements for Scalable Transport Protocols 820 A.1.1. Use of L4S Packet Identifier 822 Description: A scalable congestion control needs to distinguish the 823 packets it sends from those sent by classic congestion controls. 825 Motivation: It needs to be possible for a network node to classify 826 L4S packets without flow state into a queue that applies an L4S ECN 827 marking behaviour and isolates L4S packets from the queuing delay of 828 classic packets. 830 A.1.2. Accurate ECN Feedback 832 Description: A scalable transport protocol needs to provide timely, 833 accurate feedback about the extent of ECN marking experienced by all 834 packets. 836 Motivation: Classic congestion controls only need feedback about the 837 existence of a congestion episode within a round trip, not precisely 838 how many packets were marked with ECN or dropped. Therefore, in 839 2001, when ECN feedback was added to TCP [RFC3168], it could not 840 inform the sender of more than one ECN mark per RTT. Since then, 841 requirements for more accurate ECN feedback in TCP have been defined 842 in [RFC7560] and [I-D.ietf-tcpm-accurate-ecn] specifies an 843 experimental change to the TCP wire protocol to satisfy these 844 requirements. Most other transport protocols already satisfy this 845 requirement. 847 A.1.3. Fall back to Reno/Cubic congestion control on packet loss 849 Description: A scalable congestion control needs to react to packet 850 loss in a way that will coexist safely with a TCP Reno congestion 851 control [RFC5681]. 853 Motivation: Part of the safety conditions for deploying a scalable 854 congestion control on the public Internet is to make sure that it 855 behaves properly when it builds a queue at a network bottleneck that 856 has not been upgraded to support L4S. Packet loss can have many 857 causes, but it usually has to be conservatively assumed that it is a 858 sign of congestion. Therefore, on detecting packet loss, a scalable 859 congestion control will need to fall back to classic congestion 860 control behaviour. If it does not comply with this requirement it 861 could starve classic traffic. 863 A scalable congestion control can be used for different types of 864 transport, e.g. for real-time media or for reliable bulk transport 865 like TCP. Therefore, the particular classic congestion control 866 behaviour to fall back on will need to be part of the congestion 867 control specification of the relevant transport. In the particular 868 case of DCTCP, the current DCTCP specification states that "It is 869 RECOMMENDED that an implementation deal with loss episodes in the 870 same way as conventional TCP." For safe deployment of a scalable 871 transport in the public Internet, the above requirement would need to 872 be defined as a "MUST". 874 Packet loss might (rarely) occur in the case that the bottleneck is 875 L4S capable. In this case, the sender may receive a high number of 876 packets marked with the CE bit set and also experience a loss. 877 Current DCTCP implementations react differently to this situation. 878 At least one implementation reacts only to the drop signal (e.g. by 879 halving the CWND) and at least another DCTCP implementation reacts to 880 both signals (e.g. by halving the CWND due to the drop and also 881 further reducing the CWND based on the proportion of marked packet). 882 We believe that further experimentation is needed to understand what 883 is the best behaviour for the public Internet, which may or not be 884 one of these existing approaches. 886 A.1.4. Fall back to Reno/Cubic congestion control on classic ECN 887 bottlenecks 889 Description: A scalable congestion control needs to react to ECN 890 marking from a non-L4S but ECN-capable bottleneck in a way that will 891 coexist with a TCP Reno congestion control [RFC5681]. 893 Motivation: Similarly to the requirement in Appendix A.1.3, this 894 requirement is a safety condition to ensure a scalable congestion 895 control behaves properly when it builds a queue at a network 896 bottleneck that has not been upgraded to support L4S. On detecting 897 classic ECN marking (see below), a scalable congestion control will 898 need to fall back to classic congestion control behaviour. If it 899 does not comply with this requirement it could starve classic 900 traffic. 902 It would take time for endpoints to distinguish classic and L4S ECN 903 marking. An increase in queuing delay or in delay variation would be 904 a tell-tale sign, but it is not yet clear where a line would be drawn 905 between the two behaviours. It might be possible to cache what was 906 learned about the path to help subsequent attempts to detect the type 907 of marking. 909 A.1.5. Reduce RTT dependence 911 Description: A scalable congestion control needs to reduce or 912 eliminate RTT bias over as wide a range of RTTs as possible, or at 913 least over the typical range of RTTs that will interact in the 914 intended deployment scenario. 916 Motivation: Classic TCP's throughput is known to be inversely 917 proportional to RTT, so one would expect flows over very low RTT 918 paths to nearly starve flows over larger RTTs. However, Classic TCP 919 has never allowed a very low RTT path to exist because it induces a 920 large queue. For instance, consider two paths with base RTT 1ms and 921 100ms. If Classic TCP induces a 100ms queue, it turns these RTTs 922 into 101ms and 200ms leading to a throughput ratio of about 2:1. 923 Whereas if a Scalable TCP induces only a 1ms queue, the ratio is 924 2:101, leading to a throughput ratio of about 50:1. 926 Therefore, with very small queues, long RTT flows will essentially 927 starve, unless scalable congestion controls comply with this 928 requirement. 930 A.1.6. Scaling down the congestion window 932 Description: A scalable congestion control needs to remain responsive 933 to congestion when RTTs are significantly smaller than in the current 934 public Internet. 936 Motivation: As currently specified, the minimum required congestion 937 window of TCP (and its derivatives) is set to 2 maximum segment sizes 938 (MSS) (see equation (4) in [RFC5681]). Once the congestion window 939 reaches this minimum, all current TCP algorithms become unresponsive 940 to congestion signals. No matter how much drop or ECN marking, the 941 congestion window no longer reduces. Instead, TCP forces the queue 942 to grow, overriding any AQM and increasing queuing delay. 944 L4S mechanisms significantly reduce queueing delay so, over the same 945 path, the RTT becomes lower. Then this problem becomes surprisingly 946 common [TCP-sub-mss-w]. This is because, for the same link capacity, 947 smaller RTT implies a smaller window. For instance, consider a 948 residential setting with an upstream broadband Internet access of 8 949 Mb/s, assuming a max segment size of 1500 B. Two upstream flows will 950 each have the minimum window of 2 MSS if the RTT is 6ms or less, 951 which is quite common when accessing a nearby data centre. So, any 952 more than two such parallel TCP flows will become unresponsive and 953 increase queuing delay. 955 Unless scalable congestion controls are required to comply with this 956 requirement from the start, they will frequently become unresponsive, 957 negating the low latency benefit of L4S, for themselves and for 958 others. One possible sub-MSS window mechanism is described in 959 [TCP-sub-mss-w], and other approaches are likely to be feasible. 961 A.2. Scalable Transport Protocol Optimizations 963 A.2.1. Setting ECT in TCP Control Packets and Retransmissions 965 Description: To improve performance, scalable transport protocols 966 ought to enable ECN at the IP layer in TCP control packets (SYN, SYN- 967 ACK, pure ACKs, etc.) and in retransmitted packets. The same is true 968 for derivatives of TCP, e.g. SCTP. 970 Motivation: RFC 3168 prohibits the use of ECN on these types of TCP 971 packet, based on a number of arguments. This means these packets are 972 not protected from congestion loss by ECN, which considerably harms 973 performance, particularly for short flows. 974 [I-D.bagnulo-tcpm-generalized-ecn] counters each argument in RFC 3168 975 in turn, showing it was over-cautious. Instead it proposes 976 experimental use of ECN on all types of TCP packet. 978 A.2.2. Faster than Additive Increase 980 Description: It would improve performance if scalable congestion 981 controls did not limit their congestion window increase to the 982 traditional additive increase of 1 MSS per round trip [RFC5681] 983 during congestion avoidance. The same is true for derivatives of TCP 984 congestion control. 986 Motivation: As currently defined, DCTCP uses the traditional TCP Reno 987 additive increase in congestion avoidance phase. When the available 988 capacity suddenly increases (e.g. when another flow finishes, or if 989 radio capacity increases) it can take very many round trips to take 990 advantage of the new capacity. In the steady state, DCTCP induces 991 about 2 ECN marks per round trip, so it should be possible to quickly 992 detect when these signals have disappeared and seek available 993 capacity more rapidly. It will of course be necessary to minimize 994 the impact on other flows (classic and scalable). 996 TCP Cubic was designed to solve this problem, but as flow rates have 997 continued to increase, the delay accelerating into available capacity 998 has become prohibitive. For instance, with RTT=20 ms, to increase 999 flow rate from 100Mb/s to 200Mb/s Cubic takes between 50 and 100 1000 round trips. Every 8x increase in flow rate leads to 2x more 1001 acceleration delay. 1003 A.2.3. Faster Convergence at Flow Start 1005 Description: Particularly when a flow starts, scalable congestion 1006 controls need to converge (reach their steady-state share of the 1007 capacity) at least as fast as classic TCP and preferably faster. 1008 This affects the flow start behaviour of all transports that are 1009 based on TCP slow start. 1011 Motivation: As an example, a new DCTCP flow takes longer than classic 1012 TCP to obtain its share of the capacity of the bottleneck when there 1013 are already ongoing flows using the bottleneck capacity (about a 1014 factor of 1.5 and 2 longer according to [Alizadeh-stability]). This 1015 is detrimental in general, but particularly harms the performance of 1016 short flows relative to classic TCP. 1018 Appendix B. Alternative Identifiers 1020 This appendix is informative, not normative. It records the pros and 1021 cons of various alternative ways to identify L4S packets to record 1022 the rationale for the choice of ECT(1) (Appendix B.1) as the L4S 1023 identifier. At the end, Appendix B.6 summarises the distinguishing 1024 features of the leading alternatives. It is intended to supplement, 1025 not replace the detailed text. 1027 The leading solutions all use the ECN field, sometimes in combination 1028 with the Diffserv field. Both the ECN and Diffserv fields have the 1029 additional advantage that they are no different in either IPv4 or 1030 IPv6. A couple of alternatives that use other fields are mentioned 1031 at the end, but it is quickly explained why they are not serious 1032 contenders. 1034 B.1. ECT(1) and CE codepoints 1036 Definition: 1038 Packets with ECT(1) and conditionally packets with CE would 1039 signify L4S semantics as an alternative to the semantics of 1040 classic ECN [RFC3168], specifically: 1042 * The ECT(1) codepoint would signify that the packet was sent by 1043 an L4S-capable sender; 1045 * Given shortage of codepoints, both L4S and classic ECN sides of 1046 an AQM would have to use the same CE codepoint to indicate that 1047 a packet had experienced congestion. If a packet that had 1048 already been marked CE in an upstream buffer arrived at a 1049 subsequent AQM, this AQM would then have to guess whether to 1050 classify CE packets as L4S or classic ECN. Choosing the L4S 1051 treatment would be a safer choice, because then a few classic 1052 packets might arrive early, rather than a few L4S packets 1053 arriving late; 1055 * Additional information might be available if the classifier 1056 were transport-aware. Then it could classify a CE packet for 1057 classic ECN treatment if the most recent ECT packet in the same 1058 flow had been marked ECT(0). However, the L4S service ought 1059 not to need tranport-layer awareness; 1061 Cons: 1063 Consumes the last ECN codepoint: The L4S service is intended to 1064 supersede the service provided by classic ECN, therefore using 1065 ECT(1) to identify L4S packets could ultimately mean that the 1066 ECT(0) codepoint was 'wasted' purely to distinguish one form of 1067 ECN from its successor; 1069 ECN hard in some lower layers: It is not always possible to support 1070 ECN in an AQM acting in a buffer below the IP layer 1071 [I-D.ietf-tsvwg-ecn-encap-guidelines]. In such cases, the L4S 1072 service would have to drop rather than mark frames even though 1073 they might contain an ECN-capable packet. However, such cases 1074 would be unusual. 1076 Risk of reordering classic CE packets: Having to classify all CE 1077 packets as L4S risks some classic CE packets arriving early, which 1078 is a form of reordering. Reordering can cause the TCP sender to 1079 retransmit spuriously. However, one or two packets delivered 1080 early does not cause any spurious retransmissions because the 1081 subsequent packets continue to move the cumulative acknowledgement 1082 boundary forwards. Anyway, the risk of reordering would be low, 1083 because: i) it is quite unusual to experience more than one 1084 bottleneck queue on a path; ii) even then, reordering would only 1085 occur if there was simultaneous mixing of classic and L4S traffic, 1086 which would be more unlikely in an access link, which is where 1087 most bottlenecks are located; iii) even then, spurious 1088 retransmissions would only occur if a contiguous sequence of three 1089 or more classic CE packets from one bottleneck arrived at the 1090 next, which should in itself happen very rarely with a good AQM. 1091 The risk would be completely eliminated in AQMs that were 1092 transport-aware (but they should not need to be); 1094 Non-L4S service for control packets: The classic ECN RFCs [RFC3168] 1095 and [RFC5562] require a sender to clear the ECN field to Not-ECT 1096 for retransmissions and certain control packets specifically pure 1097 ACKs, window probes and SYNs. When L4S packets are classified by 1098 the ECN field alone, these control packets would not be classified 1099 into an L4S queue, and could therefore be delayed relative to the 1100 other packets in the flow. This would not cause re-ordering 1101 (because retransmissions are already out of order, and the control 1102 packets carry no data). However, it would make critical control 1103 packets more vulnerable to loss and delay. To address this 1104 problem, [I-D.bagnulo-tswg-generalized-ecn] proposes an experiment 1105 in which all TCP control packets and retransmissions are ECN- 1106 capable. 1108 Pros: 1110 Should work e2e: The ECN field generally works end-to-end across the 1111 Internet. Unlike the DSCP, the setting of the ECN field is at 1112 least forwarded unchanged by networks that do not support ECN, and 1113 networks rarely clear it to zero; 1115 Should work in tunnels: Unlike Diffserv, ECN is defined to always 1116 work across tunnels. However, tunnels do not always implement ECN 1117 processing as they should do, particularly because IPsec tunnels 1118 were defined differently for a few years. 1120 Could migrate to one codepoint: If all classic ECN senders 1121 eventually evolve to use the L4S service, the ECT(0) codepoint 1122 could be reused for some future purpose, but only once use of 1123 ECT(0) packets had reduced to zero, or near-zero, which might 1124 never happen. 1126 B.2. ECN Plus a Diffserv Codepoint (DSCP) 1128 Definition: 1130 For packets with a defined DSCP, all codepoints of the ECN field 1131 (except Not-ECT) would signify alternative L4S semantics to those 1132 for classic ECN [RFC3168], specifically: 1134 * The L4S DSCP would signifiy that the packet came from an L4S- 1135 capable sender; 1137 * ECT(0) and ECT(1) would both signify that the packet was 1138 travelling between transport endpoints that were both ECN- 1139 capable; 1141 * CE would signify that the packet had been marked by an AQM 1142 implementing the L4S service. 1144 Use of a DSCP is the only approach for alternative ECN semantics 1145 given as an example in [RFC4774]. However, it was perhaps considered 1146 more for controlled environments than new end-to-end services; 1148 Cons: 1150 Consumes DSCP pairs: A DSCP is obviously not orthogonal to Diffserv. 1151 Therefore, wherever the L4S service is applied to multiple 1152 Diffserv scheduling behaviours, it would be necessary to replace 1153 each DSCP with a pair of DSCPs. 1155 Uses critical lower-layer header space: The resulting increased 1156 number of DSCPs might be hard to support for some lower layer 1157 technologies, e.g. 802.1p and MPLS both offer only 3-bits for a 1158 maximum of 8 traffic class identifiers. Although L4S should 1159 reduce and possibly remove the need for some DSCPs intended for 1160 differentiated queuing delay, it will not remove the need for 1161 Diffserv entirely, because Diffserv is also used to allocate 1162 bandwidth, e.g. by prioritising some classes of traffic over 1163 others when traffic exceeds available capacity. 1165 Not end-to-end (host-network): Very few networks honour a DSCP set 1166 by a host. Typically a network will zero (bleach) the Diffserv 1167 field from all hosts. Sometimes networks will attempt to identify 1168 applications by some form of packet inspection and, based on 1169 network policy, they will set the DSCP considered appropriate for 1170 the identified application. Network-based application 1171 identification might use some combination of protocol ID, port 1172 numbers(s), application layer protocol headers, IP address(es), 1173 VLAN ID(s) and even packet timing. 1175 Not end-to-end (network-network): Very few networks honour a DSCP 1176 received from a neighbouring network. Typically a network will 1177 zero (bleach) the Diffserv field from all neighbouring networks at 1178 an interconnection point. Sometimes bilateral arrangements are 1179 made between networks, such that the receiving network remarks 1180 some DSCPs to those it uses for roughly equivalent services. The 1181 likelihood that a DSCP will be bleached or ignored depends on the 1182 type of DSCP: 1184 Local-use DSCP: These tend to be used to implement application- 1185 specific network policies, but a bilateral arrangement to 1186 remark certain DSCPs is often applied to DSCPs in the local-use 1187 range simply because it is easier not to change all of a 1188 network's internal configurations when a new arrangement is 1189 made with a neighbour; 1191 Global-use DSCP: These do not tend to be honoured across network 1192 interconnections more than local-use DSCPs. However, if two 1193 networks decide to honour certain of each other's DSCPs, the 1194 reconfiguration is a little easier if both of their globally 1195 recognised services are already represented by the relevant 1196 global-use DSCPs. 1198 Note that today a global-use DSCP gives little more assurance 1199 of end-to-end service than a local-use DSCP. In future the 1200 global-use range might give more assurance of end-to-end 1201 service than local-use, but it is unlikely that either 1202 assurance will be high, particularly given the hosts are 1203 included in the end-to-end path. 1205 Not all tunnels: Diffserv codepoints are often not propagated to the 1206 outer header when a packet is encapsulated by a tunnel header. 1207 DSCPs are propagated to the outer of uniform mode tunnels, but not 1208 pipe mode [RFC2983], and pipe mode is fairly common. 1210 ECN hard in some lower layers:: Because this approach uses both the 1211 Diffserv and ECN fields, an AQM wil only work at a lower layer if 1212 both can be supported. If individual network operators wished to 1213 deploy an AQM at a lower layer, they would usually propagate an IP 1214 Diffserv codepoint to the lower layer, using for example IEEE 1215 802.1p. However, the ECN capability is harder to propagate down 1216 to lower layers because few lower layers support it. 1218 Pros: 1220 Could migrate to e2e: If all usage of classic ECN migrates to usage 1221 of L4S, the DSCP would become redundant, and the ECN capability 1222 alone could eventually identify L4S packets without the 1223 interconnection problems of Diffserv detailed above, and without 1224 having permanently consumed more than one codepoint in the IP 1225 header. Although the DSCP does not generally function as an end- 1226 to-end identifier (see above), it could be used initially by 1227 individual ISPs to introduce the L4S service for their own locally 1228 generated traffic; 1230 B.3. ECN capability alone 1232 Definition: 1234 This approach uses ECN capability alone as the L4S identifier. It 1235 is only feasible if classic ECN is not widely deployed. The 1236 specific definition of codepoints would be: 1238 * Any ECN codepoint other than Not-ECT would signify an L4S- 1239 capable sender; 1241 * ECN codepoints would not be used for classic [RFC3168] ECN, and 1242 the classic network service would only be used for Not-ECT 1243 packets. 1245 This approach would only be feasible if 1247 A. it was generally agreed that there was little chance of any 1248 classic [RFC3168] ECN deployment in any network nodes; 1250 B. it was generally agreed that there was little chance of any 1251 client devices being deployed with classic [RFC3168] TCP-ECN 1252 on by default (note that classic TCP-ECN is already on-by- 1253 default on many servers); 1255 C. for TCP connections, developers of client OSs would all have 1256 to agree not to encourage further deployment of classic ECN. 1257 Specifically, at the start of a TCP connection classic ECN 1258 could be disabled during negotation of the ECN capability: 1260 + an L4S-capable host would have to disable ECN if the 1261 corresponding host did not support accurate ECN feedback 1262 [RFC7560], which is a prerequisite for the L4S service; 1264 + developers of operating systems for user devices would only 1265 enable ECN by default for TCP once the stack implemented 1266 L4S and accurate ECN feedback [RFC7560] including 1267 requesting accurate ECN feedback by default. 1269 Cons: 1271 Near-infeasible deployment constraints: The constraints for 1272 deployment above represent a highly unlikely, but not completely 1273 impossible, set of circumstances. If, despite the above measures, 1274 a pair of hosts did negotiate to use classic ECN, their packets 1275 would be classified into the same queue as L4S traffic, and if 1276 they had to compete with a long-running L4S flow they would get a 1277 very small capacity share; 1279 ECN hard in some lower layers: See the same issue with "ECT(1) and 1280 CE codepoints" (Appendix B.1); 1282 Non-L4S service for control packets: See the same issue with "ECT(1) 1283 and CE codepoints" (Appendix B.1). 1285 Pros: 1287 Consumes no additional codepoints: The ECT(1) codepoint and all 1288 spare Diffserv codepoints would remain available for future use; 1290 Should work e2e: As with "ECT(1) and CE codepoints" (Appendix B.1); 1292 Should work in tunnels: As with "ECT(1) and CE codepoints" 1293 (Appendix B.1). 1295 B.4. Protocol ID 1297 It has been suggested that a new ID in the IPv4 Protocol field or the 1298 IPv6 Next Header field could identify L4S packets. However this 1299 approach is ruled out by numerous problems: 1301 o A new protocol ID would need to be paired with the old one for 1302 each transport (TCP, SCTP, UDP, etc.); 1304 o In IPv6, there can be a sequence of Next Header fields, and it 1305 would not be obvious which one would be expected to identify a 1306 network service like L4S; 1308 o A new protocol ID would rarely provide an end-to-end service, 1309 because It is well-known that new protocol IDs are often blocked 1310 by numerous types of middlebox; 1312 o The approach is not a solution for AQMs below the IP layer; 1314 B.5. Source or destination addressing 1316 Locally, a network operator could arrange for L4S service to be 1317 applied based on source or destination addressing, e.g. packets from 1318 its own data centre and/or CDN hosts, packets to its business 1319 customers, etc. It could use addressing at any layer, e.g. IP 1320 addresses, MAC addresses, VLAN IDs, etc. Although addressing might 1321 be a useful tactical approach for a single ISP, it would not be a 1322 feasible approach to identify an end-to-end service like L4S. Even 1323 for a single ISP, it would require packet classifiers in buffers to 1324 be dependent on changing topology and address allocation decisions 1325 elsewhere in the network. Therefore this approach is not a feasible 1326 solution. 1328 B.6. Summary: Merits of Alternative Identifiers 1330 Table 1 provides a very high level summary of the pros and cons 1331 detailed against the schemes described respectively in Appendix B.2, 1332 Appendix B.3 and Appendix B.1, for six issues that set them apart. 1334 +--------------+--------------------+---------+--------------------+ 1335 | Issue | DSCP + ECN | ECN | ECT(1) + CE | 1336 +--------------+--------------------+---------+--------------------+ 1337 | | initial eventual | initial | initial eventual | 1338 | | | | | 1339 | end-to-end | N . . . ? . | . . Y | . . Y . . Y | 1340 | tunnels | . O . . O . | . . ? | . . ? . . Y | 1341 | lower layers | N . . . ? . | . O . | . O . . . ? | 1342 | codepoints | N . . . . ? | . . Y | N . . . . ? | 1343 | reordering | . . Y . . Y | . . Y | . O . . . ? | 1344 | ctrl pkts | . . Y . . Y | . O . | . O . . . ? | 1345 | | | | | 1346 | | | Note 1 | | 1347 +--------------+--------------------+---------+--------------------+ 1349 Note 1: Only feasible if classic ECN is obsolete. 1351 Table 1: Comparison of the Merits of Three Alternative Identifiers 1353 The schemes are scored based on both their capabilities now 1354 ('initial') and in the long term ('eventual'). The 'ECN' scheme 1355 shares the 'eventual' scores of the 'ECT(1) + CE' scheme. The scores 1356 are one of 'N, O, Y', meaning 'Poor', 'Ordinary', 'Good' 1357 respectively. The same scores are aligned vertically to aid the eye. 1358 A score of "?" in one of the positions means that this approach might 1359 optimisitically become this good, given sufficient effort. The table 1360 summarises the text and is not meant to be understandable without 1361 having read the text. 1363 Appendix C. Potential Competing Uses for the ECT(1) Codepoint 1365 The ECT(1) codepoint of the ECN field has already been assigned once 1366 for experimental use as the ECN nonce [RFC3540]. ECN is probably the 1367 only remaining field in the Internet Protocol that is common to IPv4 1368 and IPv6 and still has potential to work end-to-end, with tunnels and 1369 with lower layers. Therefore, ECT(1) should not be reassigned to a 1370 different experimental use (L4S) without carefully assessing 1371 competing potential uses. These fall into the following categories: 1373 C.1. Integrity of Congestion Feedback 1375 Receiving hosts can fool a sender into downloading faster by 1376 suppressing feedback of ECN marks (or of losses if retransmissions 1377 are not necessary or available otherwise). 1379 [RFC3540] proposes that a TCP sender could set either of ECT(0) or 1380 ECT(1) in each packet of a flow and remember the sequence it had set, 1381 termed the ECN nonce. If any packet is lost or congestion marked, 1382 the receiver will miss that bit of the sequence. An ECN Nonce 1383 receiver has to feed back the least significant bit of the sum, so it 1384 cannot suppress feedback of a loss or mark without a 50-50 chance of 1385 guessing the sum incorrectly. 1387 The ECN Nonce RFC [RFC3540] is in the process of being reclassified 1388 as historic. As far as is known, it has never been deployed, and it 1389 was only implemented for a couple of testbed evaluations. It would 1390 be nearly impossible to deploy now, because any misbehaving receiver 1391 can simply opt-out, which would be unremarkable given all receivers 1392 currently opt-out. 1394 Other ways to protect TCP feedback integrity have since been 1395 developed that do not consume any extra codepoints in the base IP 1396 header. For instance: 1398 o the sender can test the integrity of the receiver's feedback by 1399 occasionally setting the IP-ECN field to a value normally only set 1400 by the network. Then it can test whether the receiver's feedback 1401 faithfully reports what it expects [I-D.moncaster-tcpm-rcv-cheat]. 1402 This works for loss and it will work for the accurate ECN feedback 1403 [RFC7560] intended for L4S; 1405 o A network can enforce a congestion response to its ECN markings 1406 (or packet losses) by auditing congestion exposure (ConEx) 1407 [RFC7713]. Whether the receiver or a downstream network is 1408 suppressing congestion feedback or the sender is unresponsive to 1409 the feedback, or both, ConEx audit can neutralise any advantage 1410 that any of these three parties would otherwise gain. 1412 o The TCP authentication option (TCP-AO [RFC5925]) can be used to 1413 detect any tampering with TCP congestion feedback (whether 1414 malicious or accidental). TCP's congestion feedback fields are 1415 immutable end-to-end, so they are amenable to TCP-AO protection, 1416 which covers the main TCP header and TCP options by default. 1417 However, TCP-AO is often too brittle to use on many end-to-end 1418 paths, where middleboxes can make verification fail in their 1419 attempts to improve performance or security, e.g. by 1420 resegmentation or shifting the sequence space. 1422 ECN in RTP [RFC6679] is defined so that the receiver can ask the 1423 sender to send all ECT(0); all ECT(1); or both randomly. It 1424 recommends that the receiver asks for ECT(0), which is the default. 1425 The sender can choose to ignore the receiver's request. A rather 1426 complex but optional nonce mechanism was included in early drafts of 1427 RFC 6679, but it was replaced with a statement that a nonce mechanism 1428 is not specified, explaining that misbehaving receivers could opt-out 1429 anyway. RFC 6679 as published gives no rationale for why ECT(1) or 1430 'random' might be needed, but it warns that 'random' would make 1431 header compression highly inefficient. The possibility of using 1432 ECT(1) may have been left in the RFC to allow a nonce mechanism to be 1433 added later. 1435 Therefore, it seems unlikely that anyone has implemented the optional 1436 use of ECT(1) for RTP. Even if they have, it seems even less likely 1437 that any deployment actually uses it. However these assumptions will 1438 need to be verified. 1440 C.2. Notification of Less Severe Congestion than CE 1442 Various researchers have proposed to use ECT(1) as a less severe 1443 congestion notification than CE, particularly to enable flows to fill 1444 available capacity more quickly after an idle period, when another 1445 flow departs or when a flow starts, e.g. VCP [VCP], Queue View (QV) 1446 [QV]. 1448 Before assigning ECT(1) as an identifer for L4S, we must carefully 1449 consider whether it might be better to hold ECT(1) in reserve for 1450 future standardisation of rapid flow acceleration, which is an 1451 important and enduring problem [RFC6077]. 1453 Pre-Congestion Notification (PCN) is another scheme that assigns 1454 alternative semantics to the ECN field. It uses ECT(1) to signify a 1455 less severe level of pre-congestion notification than CE [RFC6660]. 1456 However, the ECN field only takes on the PCN semantics if packets 1457 carry a Diffserv codepoint defined to indicate PCN marking within a 1458 controlled environment. PCN is required to be applied solely to the 1459 outer header of a tunnel across the controlled region in order not to 1460 interfere with any end-to-end use of the ECN field. Therefore a PCN 1461 region on the path would not interfere with any of the L4S service 1462 identifiers proposed in Appendix B. 1464 Authors' Addresses 1465 Koen De Schepper 1466 Nokia Bell Labs 1467 Antwerp 1468 Belgium 1470 Email: koen.de_schepper@nokia.com 1471 URI: https://www.bell-labs.com/usr/koen.de_schepper 1473 Bob Briscoe (editor) 1474 Simula Research Lab 1476 Email: ietf@bobbriscoe.net 1477 URI: http://bobbriscoe.net/