idnits 2.17.1 draft-ietf-tsvwg-ecn-encap-guidelines-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- -- The draft header indicates that this document updates RFC3819, but the abstract doesn't seem to mention this, which it should. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year (Using the creation date from RFC3819, updated by this document, for RFC5378 checks: 1999-10-14) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (November 12, 2018) is 1985 days in the past. Is this intentional? Checking references for intended status: Best Current Practice ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-09) exists of draft-ietf-intarea-gue-06 == Outdated reference: A later version (-16) exists of draft-ietf-nvo3-geneve-08 == Outdated reference: A later version (-29) exists of draft-ietf-tsvwg-ecn-l4s-id-05 == Outdated reference: A later version (-23) exists of draft-ietf-tsvwg-rfc6040update-shim-06 -- Obsolete informational reference (is this intentional?): RFC 1323 (Obsoleted by RFC 7323) -- Obsolete informational reference (is this intentional?): RFC 6830 (Obsoleted by RFC 9300, RFC 9301) Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Transport Area Working Group B. Briscoe 3 Internet-Draft Independent 4 Updates: 3819 (if approved) J. Kaippallimalil 5 Intended status: Best Current Practice Huawei 6 Expires: May 16, 2019 P. Thaler 7 Broadcom Corporation 8 November 12, 2018 10 Guidelines for Adding Congestion Notification to Protocols that 11 Encapsulate IP 12 draft-ietf-tsvwg-ecn-encap-guidelines-11 14 Abstract 16 The purpose of this document is to guide the design of congestion 17 notification in any lower layer or tunnelling protocol that 18 encapsulates IP. The aim is for explicit congestion signals to 19 propagate consistently from lower layer protocols into IP. Then the 20 IP internetwork layer can act as a portability layer to carry 21 congestion notification from non-IP-aware congested nodes up to the 22 transport layer (L4). Following these guidelines should assure 23 interworking between new lower layer congestion notification 24 mechanisms, whether specified by the IETF or other standards bodies. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at https://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on May 16, 2019. 43 Copyright Notice 45 Copyright (c) 2018 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (https://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 1.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 5 62 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 7 63 3. Modes of Operation . . . . . . . . . . . . . . . . . . . . . 8 64 3.1. Feed-Forward-and-Up Mode . . . . . . . . . . . . . . . . 9 65 3.2. Feed-Up-and-Forward Mode . . . . . . . . . . . . . . . . 10 66 3.3. Feed-Backward Mode . . . . . . . . . . . . . . . . . . . 11 67 3.4. Null Mode . . . . . . . . . . . . . . . . . . . . . . . . 13 68 4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 69 Notification . . . . . . . . . . . . . . . . . . . . . . . . 13 70 4.1. IP-in-IP Tunnels with Shim Headers . . . . . . . . . . . 14 71 4.2. Wire Protocol Design: Indication of ECN Support . . . . . 15 72 4.3. Encapsulation Guidelines . . . . . . . . . . . . . . . . 17 73 4.4. Decapsulation Guidelines . . . . . . . . . . . . . . . . 19 74 4.5. Sequences of Similar Tunnels or Subnets . . . . . . . . . 20 75 4.6. Reframing and Congestion Markings . . . . . . . . . . . . 21 76 5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 77 Notification . . . . . . . . . . . . . . . . . . . . . . . . 21 78 6. Feed-Backward Mode: Guidelines for Adding Congestion 79 Notification . . . . . . . . . . . . . . . . . . . . . . . . 23 80 7. IANA Considerations (to be removed by RFC Editor) . . . . . . 24 81 8. Security Considerations . . . . . . . . . . . . . . . . . . . 24 82 9. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 25 83 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 25 84 11. Comments Solicited . . . . . . . . . . . . . . . . . . . . . 26 85 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 26 86 12.1. Normative References . . . . . . . . . . . . . . . . . . 26 87 12.2. Informative References . . . . . . . . . . . . . . . . . 26 88 Appendix A. Changes in This Version (to be removed by RFC 89 Editor) . . . . . . . . . . . . . . . . . . . . . . 32 90 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 36 92 1. Introduction 94 The benefits of Explicit Congestion Notification (ECN) described in 95 [RFC8087] and summarized below can only be fully realised if support 96 for ECN is added to the relevant subnetwork technology, as well as to 97 IP. When a lower layer buffer drops a packet obviously it does not 98 just drop at that layer; the packet disappears from all layers. In 99 contrast, when a lower layer marks a packet with ECN, the marking 100 needs to be explicitly propagated up the layers. The same is true if 101 a buffer marks the outer header of a packet that encapsulates inner 102 tunnelled headers. Forwarding ECN is not as straightforward as other 103 headers because it has to be assumed ECN may be only partially 104 deployed. If an egress at any layer is not ECN-aware, or if the 105 ultimate receiver or sender is not ECN-aware, congestion needs to be 106 indicated by dropping a packet, not marking it. 108 The purpose of this document is to guide the addition of congestion 109 notification to any subnet technology or tunnelling protocol, so that 110 lower layer equipment can signal congestion explicitly and it will 111 propagate consistently into encapsulated (higher layer) headers, 112 otherwise the signals will not reach their ultimate destination. 114 ECN is defined in the IP header (v4 and v6) [RFC3168] to allow a 115 resource to notify the onset of queue build-up without having to drop 116 packets, by explicitly marking a proportion of packets with the 117 congestion experienced (CE) codepoint. 119 Given a suitable marking scheme, ECN removes nearly all congestion 120 loss and it cuts delays for two main reasons: 122 o It avoids the delay when recovering from congestion losses, which 123 particularly benefits small flows or real-time flows, making their 124 delivery time predictably short [RFC2884]; 126 o As ECN is used more widely by end-systems, it will gradually 127 remove the need to configure a degree of delay into buffers before 128 they start to notify congestion (the cause of bufferbloat). This 129 is because drop involves a trade-off between sending a timely 130 signal and trying to avoid impairment, whereas ECN is solely a 131 signal not an impairment, so there is no harm triggering it 132 earlier. 134 Some lower layer technologies (e.g. MPLS, Ethernet) are used to form 135 subnetworks with IP-aware nodes only at the edges. These networks 136 are often sized so that it is rare for interior queues to overflow. 137 However, until recently this was more due to the inability of TCP to 138 saturate the links. For many years, fixes such as window scaling 139 [RFC1323] (now [RFC7323]) proved hard to deploy. And the Reno 140 variant of TCP has remained in widespread use despite its inability 141 to scale to high flow rates. However, now that modern operating 142 systems are finally capable of saturating interior links, even the 143 buffers of well-provisioned interior switches will need to signal 144 episodes of queuing. 146 Propagation of ECN is defined for MPLS [RFC5129], and is being 147 defined for TRILL [RFC7780], [I-D.ietf-trill-ecn-support], but it 148 remains to be defined for a number of other subnetwork technologies. 150 Similarly, ECN propagation is yet to be defined for many tunnelling 151 protocols. [RFC6040] defines how ECN should be propagated for IP-in- 152 IPv4 [RFC2003], IP-in-IPv6 [RFC2473] and IPsec [RFC4301] tunnels, but 153 there are numerous other tunnelling protocols with a shim and/or a 154 layer 2 header between two IP headers (v4 or v6). Some address ECN 155 propagation between the IP headers, but many do not. This document 156 gives guidance on how to address ECN propagation for future 157 tunnelling protocols, and a companion standards track specification 158 [I-D.ietf-tsvwg-rfc6040update-shim] updates those existing IP-shim- 159 (L2)-IP protocols that are under IETF change control and still widely 160 used. 162 Incremental deployment is the most delicate aspect when adding 163 support for ECN. The original ECN protocol in IP [RFC3168] was 164 carefully designed so that a congested buffer would not mark a packet 165 (rather than drop it) unless both source and destination hosts were 166 ECN-capable. Otherwise its congestion markings would never be 167 detected and congestion would just build up further. However, to 168 support congestion marking below the IP layer or within tunnels, it 169 is not sufficient to only check that the two layer 4 transport end- 170 points support ECN; correct operation also depends on the 171 decapsulator at each subnet or tunnel egress faithfully propagating 172 congestion notifications to the higher layer. Otherwise, a legacy 173 decapsulator might silently fail to propagate any ECN signals from 174 the outer to the forwarded header. Then the lost signals would never 175 be detected and again congestion would build up further. The 176 guidelines given later require protocol designers to carefully 177 consider incremental deployment, and suggest various safe approaches 178 for different circumstances. 180 Of course, the IETF does not have standards authority over every link 181 layer protocol. So this document gives guidelines for designing 182 propagation of congestion notification across the interface between 183 IP and protocols that may encapsulate IP (i.e. that can be layered 184 beneath IP). Each lower layer technology will exhibit different 185 issues and compromises, so the IETF or the relevant standards body 186 must be free to define the specifics of each lower layer congestion 187 notification scheme. Nonetheless, if the guidelines are followed, 188 congestion notification should interwork between different 189 technologies, using IP in its role as a 'portability layer'. 191 Therefore, the capitalised terms 'SHOULD' or 'SHOULD NOT' are often 192 used in preference to 'MUST' or 'MUST NOT', because it is difficult 193 to know the compromises that will be necessary in each protocol 194 design. If a particular protocol design chooses to contradict a 195 'SHOULD (NOT)' given in the advice below, it MUST include a sound 196 justification. 198 It has not been possible to give common guidelines for all lower 199 layer technologies, because they do not all fit a common pattern. 200 Instead they have been divided into a few distinct modes of 201 operation: feed-forward-and-upward; feed-upward-and-forward; feed- 202 backward; and null mode. These modes are described in Section 3, 203 then in the subsequent sections separate guidelines are given for 204 each mode. 206 This document updates the advice to subnetwork designers about ECN in 207 Section 13 of [RFC3819]. 209 1.1. Scope 211 This document only concerns wire protocol processing of explicit 212 notification of congestion. It makes no changes or recommendations 213 concerning algorithms for congestion marking or for congestion 214 response, because algorithm issues should be independent of the layer 215 the algorithm operates in. 217 The default ECN semantics are described in [RFC3168] and updated by 218 [RFC8311]. Also the guidelines for AQM designers [RFC7567] clarify 219 the semantics of both drop and ECN signals from AQM algorithms. 220 [RFC4774] is the appropriate best current practice specification of 221 how algorithms with alternative semantics for the ECN field can be 222 partitioned from Internet traffic that uses the default ECN 223 semantics, for example: 225 o by using the ECN field in combination with a Diffserv codepoint 226 such as in PCN [RFC6660], Voice over 3G [UTRAN] or Voice over LTE 227 (VoLTE) [LTE-RA]; 229 o or by using the ECT(1) codepoint of the ECN field to indicate 230 alternative semantics such as for the experimental Low Latency Low 231 Loss Scalable throughput (L4S) service [RFC8311] 232 [I-D.ietf-tsvwg-ecn-l4s-id]). 234 The aim is that the default rules for encapsulating and decapsulating 235 the ECN field are sufficiently generic that tunnels and subnets will 236 encapsulate and decapsulate packets without regard to how algorithms 237 elsewhere are setting or interpreting the semantics of the ECN field. 238 [RFC6040] updates RFC 4774 to allow alternative encapsulation and 239 decapsulation behaviours to be defined for alternative ECN semantics. 240 However it reinforces the same point - that it is far preferable to 241 try to fit within the common ECN encapsulation and decapsulation 242 behaviours, because expecting all lower layer technologies and 243 tunnels to be updated is likely to be completely impractical. 245 Alternative semantics for the ECN field can be defined to depend on 246 the traffic class indicated by the DSCP. Therefore correct 247 propagation of congestion signals could depend on correct propagation 248 of the DSCP between the layers. For instance, if the meaning of the 249 ECN field depends on the DSCP (as in PCN or VoLTE) and if the outer 250 DSCP is stripped on descapsulation, as in the pipe model of 251 [RFC2983], the special semantics of the ECN field will be lost. This 252 is an important implication of the localized scope of most Diffserv 253 arrangements. In this document, correct propagation of traffic class 254 information is assumed, while what 'correct' means and how it is 255 achieved is covered elsewhere (e.g. RFC 2983) and is outside the 256 scope of the present document. 258 The guidelines in this document do ensure that common encapsulation 259 and decapsulation rules are sufficiently generic to cover cases where 260 ECT(1) is used instead of ECT(0) to identify alternative ECN 261 semantics (as in L4S [I-D.ietf-tsvwg-ecn-l4s-id]) and where ECN 262 marking algorithms use ECT(1) to encode 3 severity levels into the 263 ECN field (e.g. PCN [RFC6660]) rather than the default of 2. All 264 these different semantics for the ECN field work because it has been 265 possible to define common default decapsulation rules that allow for 266 all cases. 268 Note that the guidelines in this document do not necessarily require 269 the subnet wire protocol to be changed to add support for congestion 270 notification. For instance, the Feed-Up-and-Forward Mode 271 (Section 3.2) and the Null Mode (Section 3.4) do not. Another way to 272 add congestion notification without consuming header space in the 273 subnet protocol might be to use a parallel control plane protocol. 275 This document focuses on the congestion notification interface 276 between IP and lower layer or tunnel protocols that can encapsulate 277 IP, where the term 'IP' includes v4 or v6, unicast, multicast or 278 anycast. However, it is likely that the guidelines will also be 279 useful when a lower layer protocol or tunnel encapsulates itself 280 (e.g. Ethernet MAC in MAC [IEEE802.1Qah]) or when it encapsulates 281 other protocols. In the feed-backward mode, propagation of 282 congestion signals for multicast and anycast packets is out-of-scope 283 (because it would be so complicated that it is hoped no-one would 284 attempt such an abomination). 286 2. Terminology 288 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 289 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 290 document are to be interpreted as described in RFC 2119 [RFC2119] 291 when, and only when, they appear in all capitals, as shown here. 293 Further terminology used within this document: 295 Protocol data unit (PDU): Information that is delivered as a unit 296 among peer entities of a layered network consisting of protocol 297 control information (typically a header) and possibly user data 298 (payload) of that layer. The scope of this document includes 299 layer 2 and layer 3 networks, where the PDU is respectively termed 300 a frame or a packet (or a cell in ATM). PDU is a general term for 301 any of these. This definition also includes a payload with a shim 302 header lying somewhere between layer 2 and 3. 304 Transport: The end-to-end transmission control function, 305 conventionally considered at layer-4 in the OSI reference model. 306 Given the audience for this document will often use the word 307 transport to mean low level bit carriage, whenever the term is 308 used it will be qualified, e.g. 'L4 transport'. 310 Encapsulator: The link or tunnel endpoint function that adds an 311 outer header to a PDU (also termed the 'link ingress', the 'subnet 312 ingress', the 'ingress tunnel endpoint' or just the 'ingress' 313 where the context is clear). 315 Decapsulator: The link or tunnel endpoint function that removes an 316 outer header from a PDU (also termed the 'link egress', the 317 'subnet egress', the 'egress tunnel endpoint' or just the 'egress' 318 where the context is clear). 320 Incoming header: The header of an arriving PDU before encapsulation. 322 Outer header: The header added to encapsulate a PDU. 324 Inner header: The header encapsulated by the outer header. 326 Outgoing header: The header forwarded by the decapsulator. 328 CE: Congestion Experienced [RFC3168] 330 ECT: ECN-Capable (L4) Transport [RFC3168] 331 Not-ECT: Not ECN-Capable (L4) Transport [RFC3168] 333 Load Regulator: For each flow of PDUs, the transport function that 334 is capable of controlling the data rate. Typically located at the 335 data source, but in-path nodes can regulate load in some 336 congestion control arrangements (e.g. admission control, policing 337 nodes or transport circuit-breakers [RFC8084]). Note the term "a 338 function capable of controlling the load" deliberately includes a 339 transport that doesn't actually control the load responsively but 340 ideally it ought to (e.g. a sending application without 341 congestion control that uses UDP). 343 ECN-PDU: A PDU that is part of a feedback loop within which all the 344 nodes that need to propagate explicit congestion notifications 345 back to the Load Regulator are ECN-capable. An IP packet with a 346 non-zero ECN field implies that the endpoints are ECN-capable, so 347 this would be an ECN-PDU. However, ECN-PDU is intended to be a 348 general term for a PDU at any layer, not just IP. 350 Not-ECN-PDU: A PDU that is part of a feedback-loop within which some 351 nodes necessary to propagate explicit congestion notifications 352 back to the load regulator are not ECN-capable. 354 Congestion Baseline: The location of the function on the path that 355 initialised the values of all congestion notification fields in a 356 sequence of packets, before any are set to the congestion 357 experienced (CE) codepoint if they experience congestion further 358 downstream. Typically the original data source at layer-4. 360 3. Modes of Operation 362 This section sets down the different modes by which congestion 363 information is passed between the lower layer and the higher one. It 364 acts as a reference framework for the following sections, which give 365 normative guidelines for designers of explicit congestion 366 notification protocols, taking each mode in turn: 368 Feed-Forward-and-Up: Nodes feed forward congestion notification 369 towards the egress within the lower layer then up and along the 370 layers towards the end-to-end destination at the transport layer. 371 The following local optimisation is possible: 373 Feed-Up-and-Forward: A lower layer switch feeds-up congestion 374 notification directly into the ECN field in the higher layer 375 (e.g. IP) header, irrespective of whether the node is at the 376 egress of a subnet. 378 Feed-Backward: Nodes feed back congestion signals towards the 379 ingress of the lower layer and (optionally) attempt to control 380 congestion within their own layer. 382 Null: Nodes cannot experience congestion at the lower layer except 383 at ingress nodes (which are IP-aware or equivalently higher-layer- 384 aware). 386 3.1. Feed-Forward-and-Up Mode 388 Like IP and MPLS, many subnet technologies are based on self- 389 contained protocol data units (PDUs) or frames sent unreliably. They 390 provide no feedback channel at the subnetwork layer, instead relying 391 on higher layers (e.g. TCP) to feed back loss signals. 393 In these cases, ECN may best be supported by standardising explicit 394 notification of congestion into the lower layer protocol that carries 395 the data forwards. It will then also be necessary to define how the 396 egress of the lower layer subnet propagates this explicit signal into 397 the forwarded upper layer (IP) header. It can then continue forwards 398 until it finally reaches the destination transport (at L4). Then 399 typically the destination will feed this congestion notification back 400 to the source transport using an end-to-end protocol (e.g. TCP). 401 This is the arrangement that has already been used to add ECN to IP- 402 in-IP tunnels [RFC6040], IP-in-MPLS and MPLS-in-MPLS [RFC5129]. 404 This mode is illustrated in Figure 1. Along the middle of the 405 figure, layers 2, 3 and 4 of the protocol stack are shown, and one 406 packet is shown along the bottom as it progresses across the network 407 from source to destination, crossing two subnets connected by a 408 router, and crossing two switches on the path across each subnet. 409 Congestion at the output of the first switch (shown as *) leads to a 410 congestion marking in the L2 header (shown as C in the illustration 411 of the packet). The chevrons show the progress of the resulting 412 congestion indication. It is propagated from link to link across the 413 subnet in the L2 header, then when the router removes the marked L2 414 header, it propagates the marking up into the L3 (IP) header. The 415 router forwards the marked L3 header into subnet 2, and when it adds 416 a new L2 header it copies the L3 marking into the L2 header as well, 417 as shown by the 'C's in both layers (assuming the technology of 418 subnet 2 also supports explicit congestion marking). 420 Note that there is no implication that each 'C' marking is encoded 421 the same; a different encoding might be used for the 'C' marking in 422 each protocol. 424 Finally, for completeness, we show the L3 marking arriving at the 425 destination, where the host transport protocol (e.g. TCP) feeds it 426 back to the source in the L4 acknowledgement (the 'C' at L4 in the 427 packet at the top of the diagram). 429 _ _ _ 430 /_______ | | |C| ACK Packet (V) 431 \ |_|_|_| 432 +---+ layer: 2 3 4 header +---+ 433 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 434 | | +---+ | ^ | 435 | | . . . . . . Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3 436 | | +---+ +---+ | ^ | +---+ +---+ | | 437 | | | *|>>>>>|>>>|>>>>>|>^ | | | | | | |L2 438 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 439 source subnet A router subnet B dest 440 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ 441 | | | | | | | | |C| | | |C| | | |C|C| Data________\ 442 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) / 443 layer: 4 3 2A 4 3 2A 4 3 4 3 2B 444 header 446 Figure 1: Feed-Forward-and-Up Mode 448 Of course, modern networks are rarely as simple as this text-book 449 example, often involving multiple nested layers. For example, a 3GPP 450 mobile network may have two IP-in-IP (GTP) tunnels in series and an 451 MPLS backhaul between the base station and the first router. 452 Nonetheless, the example illustrates the general idea of feeding 453 congestion notification forward then upward whenever a header is 454 removed at the egress of a subnet. 456 Note that the FECN (forward ECN) bit in Frame Relay and the explicit 457 forward congestion indication (EFCI [ITU-T.I.371]) bit in ATM user 458 data cells follow a feed-forward pattern. However, in ATM, this 459 arrangement is only part of a feed-forward-and-backward pattern at 460 the lower layer, not feed-forward-and-up out of the lower layer--the 461 intention was never to interface to IP ECN at the subnet egress. To 462 our knowledge, Frame Relay FECN is solely used to detect where more 463 capacity should be provisioned [Buck00]. 465 3.2. Feed-Up-and-Forward Mode 467 Ethernet is particularly difficult to extend incrementally to support 468 explicit congestion notification. One way to support ECN in such 469 cases has been to use so called 'layer-3 switches'. These are 470 Ethernet switches that bury into the Ethernet payload to find an IP 471 header and manipulate or act on certain IP fields (specifically 472 Diffserv & ECN). For instance, in Data Center TCP [RFC8257], layer-3 473 switches are configured to mark the ECN field of the IP header within 474 the Ethernet payload when their output buffer becomes congested. 475 With respect to switching, a layer-3 switch acts solely on the 476 addresses in the Ethernet header; it doesn't use IP addresses, and it 477 doesn't decrement the TTL field in the IP header. 479 _ _ _ 480 /_______ | | |C| ACK packet (V) 481 \ |_|_|_| 482 +---+ layer: 2 3 4 header +---+ 483 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 484 | | +---+ | ^ | 485 | | . . . >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3 486 | | +--^+ +---+ | | +---+ +---+ | | 487 | | | *| | | | | | | | | | |L2 488 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 489 source subnet E router subnet F dest 490 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ 491 | | | | | | | |C| | | | |C| | | |C|C| data________\ 492 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / 493 layer: 4 3 2 4 3 2 4 3 4 3 2 494 header 496 Figure 2: Feed-Up-and-Forward Mode 498 By comparing Figure 2 with Figure 1, it can be seen that subnet E 499 (perhaps a subnet of layer-3 Ethernet switches) works in feed-up-and- 500 forward mode by notifying congestion directly into L3 at the point of 501 congestion, even though the congested switch does not otherwise act 502 at L3. In this example, the technology in subnet F (e.g. MPLS) does 503 support ECN natively, so when the router adds the layer-2 header it 504 copies the ECN marking from L3 to L2 as well. 506 3.3. Feed-Backward Mode 508 In some layer 2 technologies, explicit congestion notification has 509 been defined for use internally within the subnet with its own 510 feedback and load regulation, but typically the interface with IP for 511 ECN has not been defined. 513 For instance, for the available bit-rate (ABR) service in ATM, the 514 relative rate mechanism was one of the more popular mechanisms for 515 managing traffic, tending to supersede earlier designs. In this 516 approach ATM switches send special resource management (RM) cells in 517 both the forward and backward directions to control the ingress rate 518 of user data into a virtual circuit. If a switch buffer is 519 approaching congestion or is congested it sends an RM cell back 520 towards the ingress with respectively the No Increase (NI) or 521 Congestion Indication (CI) bit set in its message type field 523 [ATM-TM-ABR]. The ingress then holds or decreases its sending bit- 524 rate accordingly. 526 _ _ _ 527 /_______ | | |C| ACK packet (X) 528 \ |_|_|_| 529 +---+ layer: 2 3 4 header +---+ 530 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4 531 | | +---+ | ^ | 532 | | | *|>>> Packet W >>>>>>>>>>>>|>^ |L3 533 | | +---+ +---+ | | +---+ +---+ | | 534 | | | | | | | <|<<<<<|<<<|<(V)<|<<<| | |L2 535 | | . . | . |Packet U | . . | . | . . | . | . . | .*| . . | |L2 536 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 537 source subnet G router subnet H dest 538 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ later 539 | | | | | | | | | | | | | | | | |C| | data________\ 540 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (W) / 541 4 3 2 4 3 2 4 3 4 3 2 542 _ 543 /__ |C| Feedback control 544 \ |_| cell/frame (V) 545 2 546 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ earlier 547 | | | | | | | | | | | | | | | | | | | data________\ 548 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / 549 layer: 4 3 2 4 3 2 4 3 4 3 2 550 header 552 Figure 3: Feed-Backward Mode 554 ATM's feed-backward approach doesn't fit well when layered beneath 555 IP's feed-forward approach--unless the initial data source is the 556 same node as the ATM ingress. Figure 3 shows the feed-backward 557 approach being used in subnet H. If the final switch on the path is 558 congested (*), it doesn't feed-forward any congestion indications on 559 packet (U). Instead it sends a control cell (V) back to the router 560 at the ATM ingress. 562 However, the backward feedback doesn't reach the original data source 563 directly because IP doesn't support backward feedback (and subnet G 564 is independent of subnet H). Instead, the router in the middle 565 throttles down its sending rate but the original data sources don't 566 reduce their rates. The resulting rate mismatch causes the middle 567 router's buffer at layer 3 to back up until it becomes congested, 568 which it signals forwards on later data packets at layer 3 (e.g. 569 packet W). Note that the forward signal from the middle router is 570 not triggered directly by the backward signal. Rather, it is 571 triggered by congestion resulting from the middle router's mismatched 572 rate response to the backward signal. 574 In response to this later forward signalling, end-to-end feedback at 575 layer-4 finally completes the tortuous path of congestion indications 576 back to the origin data source, as before. 578 3.4. Null Mode 580 Often link and physical layer resources are 'non-blocking' by design. 581 In these cases congestion notification may be implemented but it does 582 not need to be deployed at the lower layer; ECN in IP would be 583 sufficient. 585 A degenerate example is a point-to-point Ethernet link. Excess 586 loading of the link merely causes the queue from the higher layer to 587 back up, while the lower layer remains immune to congestion. Even a 588 whole meshed subnetwork can be made immune to interior congestion by 589 limiting ingress capacity and sufficient sizing of interior links, 590 e.g. a non-blocking fat-tree network. An alternative to fat links 591 near the root is numerous thin links with multi-path routing to 592 ensure even worst-case patterns of load cannot congest any link, e.g. 593 a Clos network. 595 4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 596 Notification 598 Feed-forward-and-up is the mode already used for signalling ECN up 599 the layers through MPLS into IP [RFC5129] and through IP-in-IP 600 tunnels [RFC6040], whether encapsulating with IPv4 [RFC2003], IPv6 601 [RFC2473] or IPsec [RFC4301]. These RFCs take a consistent approach 602 and the following guidelines are designed to ensure this consistency 603 continues as ECN support is added to other protocols that encapsulate 604 IP. The guidelines are also designed to ensure compliance with the 605 more general best current practice for the design of alternate ECN 606 schemes given in [RFC4774]. 608 The rest of this section is structured as follows: 610 o Section 4.1 addresses the most straightforward cases, where 611 [RFC6040] can be applied directly to add ECN to tunnels that are 612 effectively IP-in-IP tunnels, but with shim header(s) between the 613 IP headers. 615 o The subsequent sections give guidelines for adding ECN to a subnet 616 technology that uses feed-forward-and-up mode like IP, but it is 617 not so similar to IP that [RFC6040] rules can be applied directly. 618 Specifically: 620 * Sections 4.2, 4.3 and 4.4 respectively address how to add ECN 621 support to the wire protocol and to the encapsulators and 622 decapsulators at the ingress and egress of the subnet. 624 * Section 4.5 deals with the special, but common, case of 625 sequences of tunnels or subnets that all use the same 626 technology 628 * Section 4.6 deals with the question of reframing when IP 629 packets do not map 1:1 into lower layer frames. 631 4.1. IP-in-IP Tunnels with Shim Headers 633 A common pattern for many tunnelling protocols is to encapsulate an 634 inner IP header with shim header(s) then an outer IP header. A shim 635 header is defined as one that is not sufficient alone to forward the 636 packet as an outer header. Another common pattern is for a shim to 637 encapsulate a layer 2 (L2) header, which in turn encapsulates (or 638 might encapsulate) an IP header. [I-D.ietf-tsvwg-rfc6040update-shim] 639 clarifies that RFC 6040 is just as applicable when there are shim(s) 640 and possibly a L2 header between two IP headers. 642 However, it is not always feasible or necessary to propagate ECN 643 between IP headers when separated by a shim. For instance, it might 644 be too costly to dig to arbitrary depths to find an inner IP header, 645 there may be little or no congestion within the tunnel by design (see 646 null mode in Section 3.4 above), or a legacy implementation might not 647 support ECN. In cases where a tunnel does not support ECN, it is 648 important that the ingress does not copy the ECN field from an inner 649 IP header to an outer. Therefore section 4 of 650 [I-D.ietf-tsvwg-rfc6040update-shim] requires network operators to 651 configure the ingress of a non-ECN tunnel so that it zeros the ECN 652 field in the outer IP header. 654 Nonetheless, in many cases it is feasible to propagate the ECN field 655 between IP headers separated by shim header(s) and/or a L2 header. 656 Particularly in the typical case when the outer IP header and the 657 shim(s) are added (or removed) as part of the same procedure. Even 658 if the shim(s) encapsulate a L2 header, it is often possible to find 659 an inner IP header within the L2 header and propagate ECN between 660 that and the outer IP header. This can be thought of as a special 661 case of the feed-up-and-forward mode (Section 3.2), so the guidelines 662 for this mode apply (Section 5). 664 Numerous shim protocols have been defined for IP tunnelling. More 665 recent ones e.g. Generic UDP Encapsulation (GUE) 666 [I-D.ietf-intarea-gue] and Geneve [I-D.ietf-nvo3-geneve] cite and 667 follow RFC 6040. And some earlier ones, e.g. CAPWAP [RFC5415] and 668 LISP [RFC6830], cite RFC 3168, which is compatible with RFC 6040. 670 However, as Section 9.3 of RFC 3168 pointed out, ECN support needs to 671 be defined for many earlier shim-based tunnelling protocols, e.g. 672 L2TPv2 [RFC2661], L2TPv3 [RFC3931], GRE [RFC2784], PPTP [RFC2637], 673 GTP [GTPv1], [GTPv1-U], [GTPv2-C] and Teredo [RFC4380] as well as 674 some recent ones, e.g. VXLAN [RFC7348], NVGRE [RFC7637] and NSH 675 [RFC8300]. 677 All these IP-based encapsulations can be updated in one shot by 678 simple reference to RFC 6040. However, it would not be appropriate 679 to update all these protocols from within the present guidance 680 document. Instead a companion specification 681 [I-D.ietf-tsvwg-rfc6040update-shim] has been prepared that has 682 sufficient standards track status to update standards track 683 protocols. For those that are not under IETF change control 684 [I-D.ietf-tsvwg-rfc6040update-shim] can only recommend that the 685 relevant body updates them. 687 4.2. Wire Protocol Design: Indication of ECN Support 689 This section is intended to guide the redesign of any lower layer 690 protocol that encapsulate IP to add native ECN support at the lower 691 layer. It reflects the approaches used in [RFC6040] and in 692 [RFC5129]. Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS 693 encapsulations that already comply with [RFC6040] or [RFC5129] will 694 already satisfy this guidance. 696 A lower layer (or subnet) congestion notification system: 698 1. SHOULD NOT apply explicit congestion notifications to PDUs that 699 are destined for legacy layer-4 transport implementations that 700 will not understand ECN, and 702 2. SHOULD NOT apply explicit congestion notifications to PDUs if the 703 egress of the subnet might not propagate congestion notifications 704 onward into the higher layer. 706 We use the term ECN-PDUs for a PDU on a feedback loop that will 707 propagate congestion notification properly because it meets both 708 the above criteria. And a Not-ECN-PDU is a PDU on a feedback 709 loop that does not meet both criteria, and will therefore not 710 propagate congestion notification properly. A corollary of the 711 above is that a lower layer congestion notification protocol: 713 3. SHOULD be able to distinguish ECN-PDUs from Not-ECN-PDUs. 715 Note that there is no need for all interior nodes within a subnet to 716 be able to mark congestion explicitly. A mix of ECN and drop signals 717 from different nodes is fine. However, if _any_ interior nodes might 718 generate ECN markings, guideline 2 above says that all relevant 719 egress node(s) SHOULD be able to propagate those markings up to the 720 higher layer. 722 In IP, if the ECN field in each PDU is cleared to the Not-ECT (not 723 ECN-capable transport) codepoint, it indicates that the L4 transport 724 will not understand congestion markings. A congested buffer must not 725 mark these Not-ECT PDUs, and therefore drops them instead. 727 The mechanism a lower layer uses to distinguish the ECN-capability of 728 PDUs need not mimic that of IP. The above guidelines merely say that 729 the lower layer system, as a whole, should achieve the same outcome. 730 For instance, ECN-capable feedback loops might use PDUs that are 731 identified by a particular set of labels or tags. Alternatively, 732 logical link protocols that use flow state might determine whether a 733 PDU can be congestion marked by checking for ECN-support in the flow 734 state. Other protocols might depend on out-of-band control signals. 736 The per-domain checking of ECN support in MPLS [RFC5129] is a good 737 example of a way to avoid sending congestion markings to L4 738 transports that will not understand them, without using any header 739 space in the subnet protocol. 741 In MPLS, header space is extremely limited, therefore RFC5129 does 742 not provide a field in the MPLS header to indicate whether the PDU is 743 an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are 744 allowed to set explicit congestion indications without checking 745 whether the PDU is destined for a L4 transport that will understand 746 them. Nonetheless, this is made safe by requiring that the network 747 operator upgrades all decapsulating edges of a whole domain at once, 748 as soon as even one switch within the domain is configured to mark 749 rather than drop during congestion. Therefore, any edge node that 750 might decapsulate a packet will be capable of checking whether the 751 higher layer transport is ECN-capable. When decapsulating a CE- 752 marked packet, if the decapsulator discovers that the higher layer 753 (inner header) indicates the transport is not ECN-capable, it drops 754 the packet--effectively on behalf of the earlier congested node (see 755 Decapsulation Guideline 1 in Section 4.4). 757 It was only appropriate to define such an incremental deployment 758 strategy because MPLS is targeted solely at professional operators, 759 who can be expected to ensure that a whole subnetwork is consistently 760 configured. This strategy might not be appropriate for other link 761 technologies targeted at zero-configuration deployment or deployment 762 by the general public (e.g. Ethernet). For such 'plug-and-play' 763 environments it will be necessary to invent a failsafe approach that 764 ensures congestion markings will never fall into black holes, no 765 matter how inconsistently a system is put together. Alternatively, 766 congestion notification relying on correct system configuration could 767 be confined to flavours of Ethernet intended only for professional 768 network operators, such as IEEE 802.1ah Provider Backbone Bridges 769 (PBB). 771 ECN support in TRILL [I-D.ietf-trill-ecn-support] provides a good 772 example of how to add ECN to a lower layer protocol without relying 773 on careful and consistent operator configuration. TRILL provides an 774 extension header word with space for flags of different categories 775 depending on whether logic to understand the extension is critical. 776 The congestion experienced marking has been defined as a 'critical 777 ingress-to-egress' flag. So if a transit RBridge sets this flag and 778 an egress RBridge does not have any logic to process it, it will drop 779 it; which is the desired default action anyway. Therefore TRILL 780 RBridges can be updated with support for ECN in no particular order 781 and, at the egress of the TRILL campus, congestion notification will 782 be propagated to IP as ECN whenever ECN logic has been implemented, 783 or as drop otherwise. 785 QCN [IEEE802.1Qau] is not intended to extend beyond a single subnet, 786 or to interoperate with ECN. Nonetheless, the way QCN indicates to 787 lower layer devices that the end-points will not understand QCN 788 provides another example that a lower layer protocol designer might 789 be able to mimic for their scenario. An operator can define certain 790 802.1p classes of service to indicate non-QCN frames and an ingress 791 bridge is required to map arriving not-QCN-capable IP packets to one 792 of these non-QCN 802.1p classes. 794 4.3. Encapsulation Guidelines 796 This section is intended to guide the redesign of any node that 797 encapsulates IP with a lower layer header when adding native ECN 798 support to the lower layer protocol. It reflects the approaches used 799 in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or IP-in- 800 MPLS or MPLS-in-MPLS encapsulations that already comply with 801 [RFC6040] or [RFC5129] will already satisfy this guidance. 803 1. Egress Capability Check: A subnet ingress needs to be sure that 804 the corresponding egress of a subnet will propagate any 805 congestion notification added to the outer header across the 806 subnet. This is necessary in addition to checking that an 807 incoming PDU indicates an ECN-capable (L4) transport. Examples 808 of how this guarantee might be provided include: 810 * by configuration (e.g. if any label switches in a domain 811 support ECN marking, [RFC5129] requires all egress nodes to 812 have been configured to propagate ECN) 814 * by the ingress explicitly checking that the egress propagates 815 ECN (e.g. TRILL uses IS-IS to check path capabilities before 816 using critical options [RFC7780]) 818 * by inherent design of the protocol (e.g. by encoding ECN 819 marking on the outer header in such a way that a legacy egress 820 that does not understand ECN will consider the PDU corrupt and 821 discard it, thus at least propagating a form of congestion 822 signal). 824 2. Egress Fails Capability Check: If the ingress cannot guarantee 825 that the egress will propagate congestion notification, the 826 ingress SHOULD disable ECN when it forwards the PDU at the lower 827 layer. An example of how the ingress might disable ECN at the 828 lower layer would be by setting the outer header of the PDU to 829 identify it as a Not-ECN-PDU, assuming the subnet technology 830 supports such a concept. 832 3. Standard Congestion Monitoring Baseline: Once the ingress to a 833 subnet has established that the egress will correctly propagate 834 ECN, on encapsulation it SHOULD encode the same level of 835 congestion in outer headers as is arriving in incoming headers. 836 For example it might copy any incoming congestion notification 837 into the outer header of the lower layer protocol. 839 This ensures that all outer headers reflect congestion 840 accumulated along the whole upstream path since the Load 841 Regulator, not just since the ingress of the subnet. A node that 842 is not the Load Regulator SHOULD NOT re-initialise the level of 843 CE markings in the outer to zero. 845 This guideline is intended to ensure that any bulk congestion 846 monitoring of outer headers (e.g. by a network management node 847 monitoring ECN in passing frames) is most meaningful. For 848 instance, if an operator measures CE in 0.4% of passing outer 849 headers, this information is only useful if the operator knows 850 where the proportion of CE markings was last initialised to 0% 851 (the Congestion Baseline). Such monitoring information will not 852 be useful if some subnet ingress nodes reset all outer CE 853 markings while others copy incoming CE markings into the outer. 855 Most information can be extracted if the Congestion Baseline is 856 standardised at the node that is regulating the load (the Load 857 Regulator--typically the data source). Then the operator can 858 measure both congestion since the Load Regulator, and congestion 859 since the subnet ingress. The latter might be measurable by 860 subtracting the level of CE markings on inner headers from that 861 on outer headers (see Appendix C of [RFC6040]). 863 4.4. Decapsulation Guidelines 865 This section is intended to guide the redesign of any node that 866 decapsulates IP from within a lower layer header when adding native 867 ECN support to the lower layer protocol. It reflects the approaches 868 used in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or 869 IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with 870 [RFC6040] or [RFC5129] will already satisfy this guidance. 872 A subnet egress SHOULD NOT simply copy congestion notification from 873 outer headers to the forwarded header. It SHOULD calculate the 874 outgoing congestion notification field from the inner and outer 875 headers using the following guidelines. If there is any conflict, 876 rules earlier in the list take precedence over rules later in the 877 list: 879 1. If the arriving inner header is a Not-ECN-PDU it implies the L4 880 transport will not understand explicit congestion markings. 881 Then: 883 * If the outer header carries an explicit congestion marking, 884 drop is the only indication of congestion that the L4 885 transport will understand. If the congestion marking is the 886 most severe possible, the packet MUST be dropped. However, if 887 congestion can be marked with multiple levels severity and the 888 packet's marking is not the most severe, the packet MAY be 889 forwarded, but it SHOULD be dropped. 891 * If the outer is an ECN-PDU that carries no indication of 892 congestion or a Not-ECN-PDU the PDU SHOULD be forwarded, but 893 still as a Not-ECN-PDU. 895 2. If the outer header does not support explicit congestion 896 notification (a Not-ECN-PDU), but the inner header does (an ECN- 897 PDU), the inner header SHOULD be forwarded unchanged. 899 3. In some lower layer protocols congestion may be signalled as a 900 numerical level, such as in the control frames of quantised 901 congestion notification [IEEE802.1Qau]. If such a multi-bit 902 encoding encapsulates an ECN-capable IP data packet, a function 903 will be needed to convert the quantised congestion level into the 904 frequency of congestion markings in outgoing IP packets. 906 4. Congestion indications might be encoded by a severity level. For 907 instance increasing levels of congestion might be encoded by 908 numerically increasing indications, e.g. pre-congestion 909 notification (PCN) can be encoded in each PDU at three severity 910 levels in IP or MPLS [RFC6660] and the default encapsulation and 911 decapsulation rules [RFC6040] are compatible with this 912 interpretation of the ECN field. 914 If the arriving inner header is an ECN-PDU, where the inner and 915 outer headers carry indications of congestion of different 916 severity, the more severe indication SHOULD be forwarded in 917 preference to the less severe. 919 5. The inner and outer headers might carry a combination of 920 congestion notification fields that should not be possible given 921 any currently used protocol transitions. For instance, if 922 Encapsulation Guideline 3 in Section 4.3 had been followed, it 923 should not be possible to have a less severe indication of 924 congestion in the outer than in the inner. It MAY be appropriate 925 to log unexpected combinations of headers and possibly raise an 926 alarm. 928 If a safe outgoing codepoint can be defined for such a PDU, the 929 PDU SHOULD be forwarded rather than dropped. Some implementers 930 discard PDUs with currently unused combinations of headers just 931 in case they represent an attack. However, an approach using 932 alarms and policy-mediated drop is preferable to hard-coded drop, 933 so that operators can keep track of possible attacks but 934 currently unused combinations are not precluded from future use 935 through new standards actions. 937 4.5. Sequences of Similar Tunnels or Subnets 939 In some deployments, particularly in 3GPP networks, an IP packet may 940 traverse two or more IP-in-IP tunnels in sequence that all use 941 identical technology (e.g. GTP). 943 In such cases, it would be sufficient for every encapsulation and 944 decapsulation in the chain to comply with RFC 6040. Alternatively, 945 as an optimisation, a node that decapsulates a packet and immediately 946 re-encapsulates it for the next tunnel MAY copy the incoming outer 947 ECN field directly to the outgoing outer and the incoming inner ECN 948 field directly to the outgoing inner. Then the overall behavior 949 across the sequence of tunnel segments would still be consistent with 950 RFC 6040. 952 Appendix C of RFC6040 describes how a tunnel egress can monitor how 953 much congestion has been introduced within a tunnel. A network 954 operator might want to monitor how much congestion had been 955 introduced within a whole sequence of tunnels. Using the technique 956 in Appendix C of RFC6040 at the final egress, the operator could 957 monitor the whole sequence of tunnels, but only if the above 958 optimisation were used consistently along the sequence of tunnels, in 959 order to make it appear as a single tunnel. Therefore, tunnel 960 endpoint implementations SHOULD allow the operator to configure 961 whether this optimisation is enabled. 963 When ECN support is added to a subnet technology, consideration 964 SHOULD be given to a similar optimisation between subnets in sequence 965 if they all use the same technology. 967 4.6. Reframing and Congestion Markings 969 The guidance in this section is worded in terms of framing 970 boundaries, but it applies equally whether the protocol data units 971 are frames, cells or packets. 973 Where framing boundaries are different between two layers, congestion 974 indications SHOULD be propagated on the basis that a congestion 975 indication on a PDU applies to all the octets in the PDU. On 976 average, an encapsulator or decapsulator SHOULD approximately 977 preserve the number of marked octets arriving and leaving (counting 978 the size of inner headers, but not added encapsulating headers). 980 The next departing frame SHOULD be immediately marked even if only 981 enough incoming marked octets have arrived for part of the departing 982 frame. This ensures that any outstanding congestion marked octets 983 are propagated immediately, rather than held back waiting for a frame 984 no bigger than the outstanding marked octets--which might involve a 985 long wait. 987 For instance, an algorithm for marking departing frames could 988 maintain a counter representing the balance of arriving marked octets 989 minus departing marked octets. It adds the size of every marked 990 frame that arrives and if the counter is positive it marks the next 991 frame to depart and subtracts its size from the counter. This will 992 often leave a negative remainder in the counter, which is deliberate. 994 5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 995 Notification 997 The guidance in this section is applicable, for example, when IP 998 packets: 1000 o are encapsulated in Ethernet headers, which have no support for 1001 ECN; 1003 o are forwarded by the eNode-B (base station) of a 3GPP radio access 1004 network, which is required to apply ECN marking during congestion, 1005 [LTE-RA], [UTRAN], but the Packet Data Convergence Protocol (PDCP) 1006 that encapsulates the IP header over the radio access has no 1007 support for ECN. 1009 This guidance also generalises to encapsulation by other subnet 1010 technologies with no native support for explicit congestion 1011 notification at the lower layer, but with support for finding and 1012 processing an IP header. It is unlikely to be applicable or 1013 necessary for IP-in-IP encapsulation, where feed-forward-and-up mode 1014 based on [RFC6040] would be more appropriate. 1016 Marking the IP header while switching at layer-2 (by using a layer-3 1017 switch) or while forwarding in a radio access network seems to 1018 represent a layering violation. However, it can be considered as a 1019 benign optimisation if the guidelines below are followed. Feed-up- 1020 and-forward is certainly not a general alternative to implementing 1021 feed-forward congestion notification in the lower layer, because: 1023 o IPv4 and IPv6 are not the only layer-3 protocols that might be 1024 encapsulated by lower layer protocols 1026 o Link-layer encryption might be in use, making the layer-2 payload 1027 inaccessible 1029 o Many Ethernet switches do not have 'layer-3 switch' capabilities 1030 so they cannot read or modify an IP payload 1032 o It might be costly to find an IP header (v4 or v6) when it may be 1033 encapsulated by more than one lower layer header, e.g. Ethernet 1034 MAC in MAC [IEEE802.1Qah]. 1036 Nonetheless, configuring lower layer equipment to look for an ECN 1037 field in an encapsulated IP header is a useful optimisation. If the 1038 implementation follows the guidelines below, this optimisation does 1039 not have to be confined to a controlled environment such as within a 1040 data centre; it could usefully be applied on any network--even if the 1041 operator is not sure whether the above issues will never apply: 1043 1. If a native lower-layer congestion notification mechanism exists 1044 for a subnet technology, it is safe to mix feed-up-and-forward 1045 with feed-forward-and-up on other switches in the same subnet. 1046 However, it will generally be more efficient to use the native 1047 mechanism. 1049 2. The depth of the search for an IP header SHOULD be limited. If 1050 an IP header is not found soon enough, or an unrecognised or 1051 unreadable header is encountered, the switch SHOULD resort to an 1052 alternative means of signalling congestion (e.g. drop, or the 1053 native lower layer mechanism if available). 1055 3. It is sufficient to use the first IP header found in the stack; 1056 the egress of the relevant tunnel can propagate congestion 1057 notification upwards to any more deeply encapsulated IP headers 1058 later. 1060 6. Feed-Backward Mode: Guidelines for Adding Congestion Notification 1062 It can be seen from Section 3.3 that congestion notification in a 1063 subnet using feed-backward mode has generally not been designed to be 1064 directly coupled with IP layer congestion notification. The subnet 1065 attempts to minimise congestion internally, and if the incoming load 1066 at the ingress exceeds the capacity somewhere through the subnet, the 1067 layer 3 buffer into the ingress backs up. Thus, a feed-backward mode 1068 subnet is in some sense similar to a null mode subnet, in that there 1069 is no need for any direct interaction between the subnet and higher 1070 layer congestion notification. Therefore no detailed protocol design 1071 guidelines are appropriate. Nonetheless, a more general guideline is 1072 appropriate: 1074 A subnetwork technology intended to eventually interface to IP 1075 SHOULD NOT be designed using only the feed-backward mode, which is 1076 certainly best for a stand-alone subnet, but would need to be 1077 modified to work efficiently as part of the wider Internet, 1078 because IP uses feed-forward-and-up mode. 1080 The feed-backward approach at least works beneath IP, where the term 1081 'works' is used only in a narrow functional sense because feed- 1082 backward can result in very inefficient and sluggish congestion 1083 control--except if it is confined to the subnet directly connected to 1084 the original data source, when it is faster than feed-forward. It 1085 would be valid to design a protocol that could work in feed-backward 1086 mode for paths that only cross one subnet, and in feed-forward-and-up 1087 mode for paths that cross subnets. 1089 In the early days of TCP/IP, a similar feed-backward approach was 1090 tried for explicit congestion signalling, using source-quench (SQ) 1091 ICMP control packets. However, SQ fell out of favour and is now 1092 formally deprecated [RFC6633]. The main problem was that it is hard 1093 for a data source to tell the difference between a spoofed SQ message 1094 and a quench request from a genuine buffer on the path. It is also 1095 hard for a lower layer buffer to address an SQ message to the 1096 original source port number, which may be buried within many layers 1097 of headers, and possibly encrypted. 1099 Quantised congestion notification (QCN--also known as backward 1100 congestion notification or BCN) [IEEE802.1Qau] uses a feed-backward 1101 mode structurally similar to ATM's relative rate mechanism. However, 1102 QCN confines its applicability to scenarios such as some data centres 1103 where all endpoints are directly attached by the same Ethernet 1104 technology. If a QCN subnet were later connected into a wider IP- 1105 based internetwork (e.g. when attempting to interconnect multiple 1106 data centres) it would suffer the inefficiency shown in Figure 3. 1108 7. IANA Considerations (to be removed by RFC Editor) 1110 This memo includes no request to IANA. 1112 8. Security Considerations 1114 If a lower layer wire protocol is redesigned to include explicit 1115 congestion signalling in-band in the protocol header, care SHOULD be 1116 take to ensure that the field used is specified as mutable during 1117 transit. Otherwise interior nodes signalling congestion would 1118 invalidate any authentication protocol applied to the lower layer 1119 header--by altering a header field that had been assumed as 1120 immutable. 1122 The redesign of protocols that encapsulate IP in order to propagate 1123 congestion signals between layers raises potential signal integrity 1124 concerns. Experimental or proposed approaches exist for assuring the 1125 end-to-end integrity of in-band congestion signals, e.g.: 1127 o Congestion exposure (ConEx ) for networks to audit that their 1128 congestion signals are not being suppressed by other networks or 1129 by receivers, and for networks to police that senders are 1130 responding sufficiently to the signals, irrespective of the L4 1131 transport protocol used [RFC7713]. 1133 o A test for a sender to detect whether a network or the receiver is 1134 suppressing congestion signals (for example see 2nd para of 1135 Section 20.2 of [RFC3168]). 1137 Given these end-to-end approaches are already being specified, it 1138 would make little sense to attempt to design hop-by-hop congestion 1139 signal integrity into a new lower layer protocol, because end-to-end 1140 integrity inherently achieves hop-by-hop integrity. 1142 Section 6 gives vulnerability to spoofing as one of the reasons for 1143 deprecating feed-backward mode. 1145 9. Conclusions 1147 Following the guidance in the document enables ECN support to be 1148 extended to numerous protocols that encapsulate IP (v4 & v6) in a 1149 consistent way, so that IP continues to fulfil its role as an end-to- 1150 end interoperability layer. This includes: 1152 o A wide range of tunnelling protocols including those with various 1153 forms of shim header between two IP headers, possibly also 1154 separated by a L2 header; 1156 o A wide range of subnet technologies, particularly those that work 1157 in the same 'feed-forward-and-up' mode that is used to support ECN 1158 in IP and MPLS. 1160 Guidelines have been defined for supporting propagation of ECN 1161 between Ethernet and IP on so-called Layer-3 Ethernet switches, using 1162 a 'feed-up-an-forward' mode. This approach could enable other subnet 1163 technologies to pass ECN signals into the IP layer, even if they do 1164 not support ECN natively. 1166 Finally, attempting to add ECN to a subnet technology in feed- 1167 backward mode is deprecated except in special cases, due to its 1168 likely sluggish response to congestion. 1170 10. Acknowledgements 1172 Thanks to Gorry Fairhurst for extensive reviews. Thanks also to the 1173 following reviewers: Richard Scheffenegger, Ingemar Johansson, Piers 1174 O'Hanlon and Michael Welzl, who pointed out that lower layer 1175 congestion notification signals may have different semantics to those 1176 in IP. Thanks are also due to the tsvwg chairs, TSV ADs and IETF 1177 liaison people such as Eric Gray, Dan Romascanu and Gonzalo Camarillo 1178 for helping with the liaisons with the IEEE and 3GPP. And thanks to 1179 Georg Mayer and particularly to Erik Guttman for the extensive search 1180 and categorisation of any 3GPP specifications that cite ECN 1181 specifications. 1183 Bob Briscoe was part-funded by the European Community under its 1184 Seventh Framework Programme through the Trilogy project (ICT-216372) 1185 for initial drafts and through the Reducing Internet Transport 1186 Latency (RITE) project (ICT-317700) subsequently. The views 1187 expressed here are solely those of the authors. 1189 11. Comments Solicited 1191 Comments and questions are encouraged and very welcome. They can be 1192 addressed to the IETF Transport Area working group mailing list 1193 , and/or to the authors. 1195 12. References 1197 12.1. Normative References 1199 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1200 Requirement Levels", BCP 14, RFC 2119, 1201 DOI 10.17487/RFC2119, March 1997, 1202 . 1204 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1205 of Explicit Congestion Notification (ECN) to IP", 1206 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1207 . 1209 [RFC3819] Karn, P., Ed., Bormann, C., Fairhurst, G., Grossman, D., 1210 Ludwig, R., Mahdavi, J., Montenegro, G., Touch, J., and L. 1211 Wood, "Advice for Internet Subnetwork Designers", BCP 89, 1212 RFC 3819, DOI 10.17487/RFC3819, July 2004, 1213 . 1215 [RFC4774] Floyd, S., "Specifying Alternate Semantics for the 1216 Explicit Congestion Notification (ECN) Field", BCP 124, 1217 RFC 4774, DOI 10.17487/RFC4774, November 2006, 1218 . 1220 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 1221 Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 1222 2008, . 1224 [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion 1225 Notification", RFC 6040, DOI 10.17487/RFC6040, November 1226 2010, . 1228 12.2. Informative References 1230 [ATM-TM-ABR] 1231 Cisco, "Understanding the Available Bit Rate (ABR) Service 1232 Category for ATM VCs", Design Technote 10415, June 2005. 1234 [Buck00] Buckwalter, J., "Frame Relay: Technology and Practice", 1235 Pub. Addison Wesley ISBN-13: 978-0201485240, 2000. 1237 [GTPv1] 3GPP, "GPRS Tunnelling Protocol (GTP) across the Gn and Gp 1238 interface", Technical Specification TS 29.060. 1240 [GTPv1-U] 3GPP, "General Packet Radio System (GPRS) Tunnelling 1241 Protocol User Plane (GTPv1-U)", Technical Specification TS 1242 29.281. 1244 [GTPv2-C] 3GPP, "Evolved General Packet Radio Service (GPRS) 1245 Tunnelling Protocol for Control plane (GTPv2-C)", 1246 Technical Specification TS 29.274. 1248 [I-D.ietf-intarea-gue] 1249 Herbert, T. and O. Zia, "Generic UDP Encapsulation", 1250 draft-ietf-intarea-gue-06 (work in progress), August 2018. 1252 [I-D.ietf-nvo3-geneve] 1253 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 1254 Network Virtualization Encapsulation", draft-ietf- 1255 nvo3-geneve-08 (work in progress), October 2018. 1257 [I-D.ietf-trill-ecn-support] 1258 Eastlake, D. and B. Briscoe, "TRILL (TRansparent 1259 Interconnection of Lots of Links): ECN (Explicit 1260 Congestion Notification) Support", draft-ietf-trill-ecn- 1261 support-07 (work in progress), February 2018. 1263 [I-D.ietf-tsvwg-ecn-l4s-id] 1264 Schepper, K. and B. Briscoe, "Identifying Modified 1265 Explicit Congestion Notification (ECN) Semantics for 1266 Ultra-Low Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s- 1267 id-05 (work in progress), November 2018. 1269 [I-D.ietf-tsvwg-rfc6040update-shim] 1270 Briscoe, B., "Propagating Explicit Congestion Notification 1271 Across IP Tunnel Headers Separated by a Shim", draft-ietf- 1272 tsvwg-rfc6040update-shim-06 (work in progress), March 1273 2018. 1275 [IEEE802.1Qah] 1276 IEEE, "IEEE Standard for Local and Metropolitan Area 1277 Networks--Virtual Bridged Local Area Networks--Amendment 1278 6: Provider Backbone Bridges", IEEE Std 802.1Qah-2008, 1279 August 2008, 1280 . 1282 (Access Controlled link within page) 1284 [IEEE802.1Qau] 1285 Finn, N., Ed., "IEEE Standard for Local and Metropolitan 1286 Area Networks--Virtual Bridged Local Area Networks - 1287 Amendment 13: Congestion Notification", IEEE Std 802.1Qau- 1288 2010, March 2010, . 1291 (Access Controlled link within page) 1293 [ITU-T.I.371] 1294 ITU-T, "Traffic Control and Congestion Control in B-ISDN", 1295 ITU-T Rec. I.371 (03/04), March 2004, 1296 . 1299 [LTE-RA] 3GPP, "Evolved Universal Terrestrial Radio Access (E-UTRA) 1300 and Evolved Universal Terrestrial Radio Access Network 1301 (E-UTRAN); Overall description; Stage 2", Technical 1302 Specification TS 36.300. 1304 [RFC1323] Jacobson, V., Braden, R., and D. Borman, "TCP Extensions 1305 for High Performance", RFC 1323, DOI 10.17487/RFC1323, May 1306 1992, . 1308 [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, 1309 DOI 10.17487/RFC2003, October 1996, 1310 . 1312 [RFC2473] Conta, A. and S. Deering, "Generic Packet Tunneling in 1313 IPv6 Specification", RFC 2473, DOI 10.17487/RFC2473, 1314 December 1998, . 1316 [RFC2637] Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little, 1317 W., and G. Zorn, "Point-to-Point Tunneling Protocol 1318 (PPTP)", RFC 2637, DOI 10.17487/RFC2637, July 1999, 1319 . 1321 [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, 1322 G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", 1323 RFC 2661, DOI 10.17487/RFC2661, August 1999, 1324 . 1326 [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. 1327 Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, 1328 DOI 10.17487/RFC2784, March 2000, 1329 . 1331 [RFC2884] Hadi Salim, J. and U. Ahmed, "Performance Evaluation of 1332 Explicit Congestion Notification (ECN) in IP Networks", 1333 RFC 2884, DOI 10.17487/RFC2884, July 2000, 1334 . 1336 [RFC2983] Black, D., "Differentiated Services and Tunnels", 1337 RFC 2983, DOI 10.17487/RFC2983, October 2000, 1338 . 1340 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1341 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 1342 RFC 3931, DOI 10.17487/RFC3931, March 2005, 1343 . 1345 [RFC4301] Kent, S. and K. Seo, "Security Architecture for the 1346 Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, 1347 December 2005, . 1349 [RFC4380] Huitema, C., "Teredo: Tunneling IPv6 over UDP through 1350 Network Address Translations (NATs)", RFC 4380, 1351 DOI 10.17487/RFC4380, February 2006, 1352 . 1354 [RFC5415] Calhoun, P., Ed., Montemurro, M., Ed., and D. Stanley, 1355 Ed., "Control And Provisioning of Wireless Access Points 1356 (CAPWAP) Protocol Specification", RFC 5415, 1357 DOI 10.17487/RFC5415, March 2009, 1358 . 1360 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 1361 RFC 6633, DOI 10.17487/RFC6633, May 2012, 1362 . 1364 [RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three 1365 Pre-Congestion Notification (PCN) States in the IP Header 1366 Using a Single Diffserv Codepoint (DSCP)", RFC 6660, 1367 DOI 10.17487/RFC6660, July 2012, 1368 . 1370 [RFC6830] Farinacci, D., Fuller, V., Meyer, D., and D. Lewis, "The 1371 Locator/ID Separation Protocol (LISP)", RFC 6830, 1372 DOI 10.17487/RFC6830, January 2013, 1373 . 1375 [RFC7323] Borman, D., Braden, B., Jacobson, V., and R. 1376 Scheffenegger, Ed., "TCP Extensions for High Performance", 1377 RFC 7323, DOI 10.17487/RFC7323, September 2014, 1378 . 1380 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 1381 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 1382 eXtensible Local Area Network (VXLAN): A Framework for 1383 Overlaying Virtualized Layer 2 Networks over Layer 3 1384 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, 1385 . 1387 [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF 1388 Recommendations Regarding Active Queue Management", 1389 BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, 1390 . 1392 [RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network 1393 Virtualization Using Generic Routing Encapsulation", 1394 RFC 7637, DOI 10.17487/RFC7637, September 2015, 1395 . 1397 [RFC7713] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) 1398 Concepts, Abstract Mechanism, and Requirements", RFC 7713, 1399 DOI 10.17487/RFC7713, December 2015, 1400 . 1402 [RFC7780] Eastlake 3rd, D., Zhang, M., Perlman, R., Banerjee, A., 1403 Ghanwani, A., and S. Gupta, "Transparent Interconnection 1404 of Lots of Links (TRILL): Clarifications, Corrections, and 1405 Updates", RFC 7780, DOI 10.17487/RFC7780, February 2016, 1406 . 1408 [RFC8084] Fairhurst, G., "Network Transport Circuit Breakers", 1409 BCP 208, RFC 8084, DOI 10.17487/RFC8084, March 2017, 1410 . 1412 [RFC8087] Fairhurst, G. and M. Welzl, "The Benefits of Using 1413 Explicit Congestion Notification (ECN)", RFC 8087, 1414 DOI 10.17487/RFC8087, March 2017, 1415 . 1417 [RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L., 1418 and G. Judd, "Data Center TCP (DCTCP): TCP Congestion 1419 Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257, 1420 October 2017, . 1422 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 1423 "Network Service Header (NSH)", RFC 8300, 1424 DOI 10.17487/RFC8300, January 2018, 1425 . 1427 [RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion 1428 Notification (ECN) Experimentation", RFC 8311, 1429 DOI 10.17487/RFC8311, January 2018, 1430 . 1432 [UTRAN] 3GPP, "UTRAN Overall Description", Technical 1433 Specification TS 25.401. 1435 Appendix A. Changes in This Version (to be removed by RFC Editor) 1437 From ietf-10 to ietf-11 1439 * Removed short section (was 3) 'Guidelines for All Cases' 1440 because it was out of scope, being covered by RFC 4774. 1441 Expanded the Scope section (1.2) to explain all this. 1442 Explained that the default encap/decap rules already support 1443 certain alternative semantics, particularly all three of the 1444 alternative semantics for ECT(1): equivalent to ECT(0) , higher 1445 severity than ECT(0), and unmarked but implying different 1446 marking semantics from ECT(0). 1448 * Clarified why the QCN example was being given even though not 1449 about increment deployment of ECN 1451 * Pointed to the spoofing issue with feed-backward mode from the 1452 Security Considerations section, to aid security review. 1454 * Removed any ambiguity in the word 'transport' throughout 1456 From ietf-09 to ietf-10 1458 * Updated section 5.1 on "IP-in-IP tunnels with Shim Headers" to 1459 be consistent with updates to draft-ietf-tsvwg-rfc6040update- 1460 shim. 1462 * Removed reference to the ECN nonce, which has been made 1463 historic by RFC 8311 1465 * Removed "Open Issues" Appendix, given all have been addressed. 1467 From ietf-08 to ietf-09 1469 * Updated para in Intro that listed all the IP-in-IP tunnelling 1470 protocols, to instead refer to draft-ietf-tsvwg-rfc6040update- 1471 shim 1473 * Updated section 5.1 on "IP-in-IP tunnels with Shim Headers" to 1474 summarize guidance that has evolved as rfc6040update-shim has 1475 developed. 1477 From ietf-07 to ietf-08: Refreshed to avoid expiry. Updated 1478 references. 1480 From ietf-06 to ietf-07: 1482 * Added the people involved in liaisons to the acknowledgements. 1484 From ietf-05 to ietf-06: 1486 * Introduction: Added GUE and Geneve as examples of tightly 1487 coupled shims between IP headers that cite RFC 6040. And added 1488 VXLAN to list of those that do not. 1490 * Replaced normative text about tightly coupled shims between IP 1491 headers, with reference to new draft-ietf-tsvwg-rfc6040update- 1492 shim 1494 * Wire Protocol Design: Indication of ECN Support: Added TRILL as 1495 an example of a well-design protocol that does not need an 1496 indication of ECN support in the wire protocol. 1498 * Encapsulation Guidelines: In the case of a Not-ECN-PDU with a 1499 CE outer, replaced SHOULD be dropped, with explanations of when 1500 SHOULD or MUST are appropriate. 1502 * Feed-Up-and-Forward Mode: Explained examples more carefully, 1503 referred to PDCP and cited UTRAN spec as well as E-UTRAN. 1505 * Updated references. 1507 * Marked open issues as resolved, but did not delete Open Issues 1508 Appendix (yet). 1510 From ietf-04 to ietf-05: 1512 * Explained why tightly coupled shim headers only "SHOULD" comply 1513 with RFC 6040, not "MUST". 1515 * Updated references 1517 From ietf-03 to ietf-04: 1519 * Addressed Richard Scheffenegger's review comments: primarily 1520 editorial corrections, and addition of examples for clarity. 1522 From ietf-02 to ietf-03: 1524 * Updated references, ad cited RFC4774. 1526 From ietf-01 to ietf-02: 1528 * Added Section for guidelines that are applicable in all cases. 1530 * Updated references. 1532 From ietf-00 to ietf-01: Updated references. 1534 From briscoe-04 to ietf-00: Changed filename following tsvwg 1535 adoption. 1537 From briscoe-03 to 04: 1539 * Re-arranged the introduction to describe the purpose of the 1540 document first before introducing ECN in more depth. And 1541 clarified the introduction throughout. 1543 * Added applicability to 3GPP TS 36.300. 1545 From briscoe-02 to 03: 1547 * Scope section: 1549 + Added dependence on correct propagation of traffic class 1550 information 1552 + For the feed-backward mode, deemed multicast and anycast out 1553 of scope 1555 * Ensured all guidelines referring to subnet technologies also 1556 refer to tunnels and vice versa by adding applicability 1557 sentences at the start of sections 4.1, 4.2, 4.3, 4.4, 4.6 and 1558 5. 1560 * Added Security Considerations on ensuring congestion signal 1561 fields are classed as immutable and on using end-to-end 1562 congestion signal integrity technologies rather than hop-by- 1563 hop. 1565 From briscoe-01 to 02: 1567 * Added authors: JK & PT 1569 * Added 1571 + Section 4.1 "IP-in-IP Tunnels with Tightly Coupled Shim 1572 Headers" 1574 + Section 4.5 "Sequences of Similar Tunnels or Subnets" 1576 + roadmap at the start of Section 4, given the subsections 1577 have become quite fragmented. 1579 + Section 9 "Conclusions" 1581 * Clarified why transports are starting to be able to saturate 1582 interior links 1584 * Under Section 1.1, addressed the question of alternative signal 1585 semantics and included multicast & anycast. 1587 * Under Section 3.1, included a 3GPP example. 1589 * Section 4.2. "Wire Protocol Design": 1591 + Altered guideline 2. to make it clear that it only applies 1592 to the immediate subnet egress, not later ones 1594 + Added a reminder that it is only necessary to check that ECN 1595 propagates at the egress, not whether interior nodes mark 1596 ECN 1598 + Added example of how QCN uses 802.1p to indicate support for 1599 QCN. 1601 * Added references to Appendix C of RFC6040, about monitoring the 1602 amount of congestion signals introduced within a tunnel 1604 * Appendix A: Added more issues to be addressed, including plan 1605 to produce a standards track update to IP-in-IP tunnel 1606 protocols. 1608 * Updated acks and references 1610 From briscoe-00 to 01: 1612 * Intended status: BCP (was Informational) & updates 3819 added. 1614 * Briefer Introduction: Introductory para justifying benefits of 1615 ECN. Moved all but a brief enumeration of modes of operation 1616 to their own new section (from both Intro & Scope). Introduced 1617 incr. deployment as most tricky part. 1619 * Tightened & added to terminology section 1621 * Structured with Modes of Operation, then Guidelines section for 1622 each mode. 1624 * Tightened up guideline text to remove vagueness / passive voice 1625 / ambiguity and highlight main guidelines as numbered items. 1627 * Added Outstanding Document Issues Appendix 1628 * Updated references 1630 Authors' Addresses 1632 Bob Briscoe 1633 Independent 1634 UK 1636 EMail: ietf@bobbriscoe.net 1637 URI: http://bobbriscoe.net/ 1639 John Kaippallimalil 1640 Huawei 1641 5340 Legacy Drive, Suite 175 1642 Plano, Texas 75024 1643 USA 1645 EMail: john.kaippallimalil@huawei.com 1647 Pat Thaler 1648 Broadcom Corporation 1649 5025 Keane Drive 1650 Carmichael, CA 95608 1651 USA 1653 EMail: pthaler@broadcom.com