idnits 2.17.1 draft-ietf-tsvwg-ecn-encap-guidelines-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- -- The draft header indicates that this document updates RFC3819, but the abstract doesn't seem to mention this, which it should. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year (Using the creation date from RFC3819, updated by this document, for RFC5378 checks: 1999-10-14) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (March 07, 2014) is 3695 days in the past. Is this intentional? Checking references for intended status: Best Current Practice ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'GF' is mentioned on line 1222, but not defined == Outdated reference: A later version (-13) exists of draft-ietf-conex-abstract-mech-08 == Outdated reference: A later version (-03) exists of draft-moncaster-tcpm-rcv-cheat-01 -- Obsolete informational reference (is this intentional?): RFC 1323 (Obsoleted by RFC 7323) Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Transport Area Working Group B. Briscoe 3 Internet-Draft BT 4 Updates: 3819 (if approved) J. Kaippallimalil 5 Intended status: Best Current Practice Huawei 6 Expires: September 8, 2014 P. Thaler 7 Broadcom Corporation 8 March 07, 2014 10 Guidelines for Adding Congestion Notification to Protocols that 11 Encapsulate IP 12 draft-ietf-tsvwg-ecn-encap-guidelines-00 14 Abstract 16 The purpose of this document is to guide the design of congestion 17 notification in any lower layer or tunnelling protocol that 18 encapsulates IP. The aim is for explicit congestion signals to 19 propagate consistently from lower layer protocols into IP. Then the 20 IP internetwork layer can act as a portability layer to carry 21 congestion notification from non-IP-aware congested nodes up to the 22 transport layer (L4). Following these guidelines should assure 23 interworking between new lower layer congestion notification 24 mechanisms, whether specified by the IETF or other standards bodies. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on September 8, 2014. 43 Copyright Notice 45 Copyright (c) 2014 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 1.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 5 62 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 63 3. Modes of Operation . . . . . . . . . . . . . . . . . . . . . 7 64 3.1. Feed-Forward-and-Up Mode . . . . . . . . . . . . . . . . 8 65 3.2. Feed-Up-and-Forward Mode . . . . . . . . . . . . . . . . 9 66 3.3. Feed-Backward Mode . . . . . . . . . . . . . . . . . . . 10 67 3.4. Null Mode . . . . . . . . . . . . . . . . . . . . . . . . 12 68 4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 69 Notification . . . . . . . . . . . . . . . . . . . . . . . . 12 70 4.1. IP-in-IP Tunnels with Tightly Coupled Shim Headers . . . 13 71 4.2. Wire Protocol Design: Indication of ECN Support . . . . . 13 72 4.3. Encapsulation Guidelines . . . . . . . . . . . . . . . . 15 73 4.4. Decapsulation Guidelines . . . . . . . . . . . . . . . . 17 74 4.5. Sequences of Similar Tunnels or Subnets . . . . . . . . . 18 75 4.6. Reframing and Congestion Markings . . . . . . . . . . . . 19 76 5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 77 Notification . . . . . . . . . . . . . . . . . . . . . . . . 19 78 6. Feed-Backward Mode: Guidelines for Adding Congestion 79 Notification . . . . . . . . . . . . . . . . . . . . . . . . 21 80 7. IANA Considerations (to be removed by RFC Editor) . . . . . . 22 81 8. Security Considerations . . . . . . . . . . . . . . . . . . . 22 82 9. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 22 83 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 23 84 11. Comments Solicited . . . . . . . . . . . . . . . . . . . . . 23 85 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 86 12.1. Normative References . . . . . . . . . . . . . . . . . . 23 87 12.2. Informative References . . . . . . . . . . . . . . . . . 24 88 Appendix A. Outstanding Document Issues . . . . . . . . . . . . 27 89 Appendix B. Changes in This Version (to be removed by RFC 90 Editor) . . . . . . . . . . . . . . . . . . . . . . 27 92 1. Introduction 94 The benefits of Explicit Congestion Notification (ECN) described 95 below can only be fully realised if support for ECN is added to the 96 relevant subnetwork technology, as well as to IP. When a lower layer 97 buffer drops a packet obviously it does not just drop at that layer; 98 the packet disappears from all layers. In contrast, when a lower 99 layer marks a packet with ECN, the marking needs to be explicitly 100 propagated up the layers. The same is true if a buffer marks the 101 outer header of a packet that encapsulates inner tunnelled headers. 102 Forwarding ECN is not as straightforward as other headers because it 103 has to be assumed ECN may be only partially deployed. If an egress 104 at any layer is not ECN-aware, or if the ultimate receiver or sender 105 is not ECN-aware, congestion needs to be indicated by dropping a 106 packet, not marking it. 108 The purpose of this document is to guide the addition of congestion 109 notification to any subnet technology or tunnelling protocol, so that 110 lower layer equipment can signal congestion explicitly and it will 111 propagate consistently into encapsulated (higher layer) headers, 112 otherwise the signals will not reach their ultimate destination. 114 ECN is defined in the IP header (v4 & v6) [RFC3168] to allow a 115 resource to notify the onset of queue build-up without having to drop 116 packets, by explicitly marking a proportion of packets with the 117 congestion experienced (CE) codepoint. 119 Given a suitable marking scheme, ECN removes nearly all congestion 120 loss and it cuts delays for two main reasons: 122 o It avoids the delay when recovering from congestion losses, which 123 particularly benefits small flows or real-time flows, making their 124 delivery time predictably short [RFC2884]; 126 o As ECN is used more widely by end-systems, it will gradually 127 remove the need to configure a degree of delay into buffers before 128 they start to notify congestion (the cause of bufferbloat). This 129 is because drop involves a trade-off between sending a timely 130 signal and trying to avoid impairment, whereas ECN is solely a 131 signal not an impairment, so there is no harm triggering it 132 earlier. 134 Some lower layer technologies (e.g. MPLS, Ethernet) are used to form 135 subnetworks with IP-aware nodes only at the edges. These networks 136 are often sized so that it is rare for interior queues to overflow. 137 However, this has often be more due to the inability of the original 138 TCP protocol to saturate the links. For many years, fixes such as 139 window scaling [RFC1323] proved hard to deploy. But now that modern 140 operating systems are finally capable of saturating interior links, 141 even the buffers of well-provisioned interior switches will need to 142 signal episodes of queuing. 144 Propagation of ECN is defined for MPLS [RFC5129], and is being 145 defined for TRILL [trill-rbridge-options], but it remains to be 146 defined for a number of other subnetwork technologies. 148 Similarly, ECN propagation is yet to be defined for many tunnelling 149 protocols. [RFC6040] defines how ECN should be propagated for IP-in- 150 IP [RFC2003] and IPsec [RFC4301] tunnels. However, as Section 9.3 of 151 RFC3168 pointed out, ECN support will need to be defined for other 152 tunnelling protocols, e.g. L2TP [RFC2661], GRE [RFC1701], [RFC2784], 153 PPTP [RFC2637] and GTP [GTPv1], [GTPv1-U], [GTPv2-C]. 155 Incremental deployment is the most tricky aspect when adding support 156 for ECN. The original ECN protocol in IP [RFC3168] was carefully 157 designed so that a congested buffer would not mark a packet (rather 158 than drop it) unless both source and destination hosts were ECN- 159 capable. Otherwise its congestion markings would never be detected 160 and congestion would just deteriorate further. However, to support 161 congestion marking below the IP layer, it is not sufficient to only 162 check that the two end-points support ECN; correct operation also 163 depends on the decapsulator at each subnet egress faithfully 164 propagating congestion notifications to the higher layer. Otherwise, 165 a legacy decapsulator might silently fail to propagate any ECN 166 signals from the outer to the forwarded header. Then the lost 167 signals would never be detected and again congestion would 168 deteriorate further. The guidelines given later require protocol 169 designers to carefully consider incremental deployment, and suggest 170 various safe approaches for different circumstances. 172 Of course, the IETF does not have standards authority over every link 173 layer protocol. So this document gives guidelines for designing 174 propagation of congestion notification across the interface between 175 IP and protocols that may encapsulate IP (i.e. that can be layered 176 beneath IP). Each lower layer technology will exhibit different 177 issues and compromises, so the IETF or the relevant standards body 178 must be free to define the specifics of each lower layer congestion 179 notification scheme. Nonetheless, if the guidelines are followed, 180 congestion notification should interwork between different 181 technologies, using IP in its role as a 'portability layer'. 183 Therefore, the capitalised term 'SHOULD' or 'SHOULD NOT' are often 184 used in preference to 'MUST' or 'MUST NOT', because it is difficult 185 to know the compromises that will be necessary in each protocol 186 design. If a particular protocol design chooses to contradict a 187 'SHOULD (NOT)' given in the advice below, it MUST include a sound 188 justification. 190 It has not been possible to give common guidelines for all lower 191 layer technologies, because they do not all fit a common pattern. 192 Instead they have been divided into a few distinct modes of 193 operation: feed-forward-and-upward; feed-upward-and-forward; feed- 194 backward; and null mode. These modes are described in Section 3, 195 then in the following sections separate guidelines are given for each 196 mode. 198 This document updates the advice to subnetwork designers about ECN in 199 Section 13 of [RFC3819]. 201 1.1. Scope 203 This document only concerns wire protocol processing of explicit 204 notification of congestion and makes no changes or recommendations 205 concerning algorithms for congestion marking or for congestion 206 response (algorithm issues should be independent of the layer the 207 algorithm operates in). 209 The question of congestion notification signals with different 210 semantics to those of ECN in IP is touched on in a couple of specific 211 cases (e.g. QCN [IEEE802.1Qau]) and with schemes with multiple 212 severity levels such as PCN [RFC6660]). However, no attempt is made 213 to give guidelines about schemes with different semantics that are 214 yet to be invented. 216 The semantics of congestion signals can be relative to the traffic 217 class. Therefore correct propagation of congestion signals could 218 depend on correct propagation of any traffic class field between the 219 layers. In this document, correct propagation of traffic class 220 information is assumed, while what 'correct' means and how it is 221 achieved is covered elsewhere (e.g. [RFC2983]) and is outside the 222 scope of the present document. 224 Note that these guidelines do not require the subnet wire protocol to 225 be changed to accommodate congestion notification. Another way to 226 add congestion notification without consuming header space in the 227 subnet protocol might be to use a parallel control plane protocol. 229 This document focuses on the congestion notification interface 230 between IP and lower layer protocols that can encapsulate IP, where 231 the term 'IP' includes v4 or v6, unicast, multicast or anycast. 232 However, it is likely that the guidelines will also be useful when a 233 lower layer protocol or tunnel encapsulates itself (e.g. Ethernet MAC 234 in MAC [IEEE802.1Qah]) or when it encapsulates other protocols. In 235 the feed-backward mode, propagation of congestion signals for 236 multicast and anycast packets is out-of-scope (because it would be so 237 complicated that it is hoped no-one would attempt such an 238 abomination). 240 2. Terminology 242 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 243 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 244 document are to be interpreted as described in RFC 2119 [RFC2119]. 246 Further terminology used within this document: 248 Protocol data unit (PDU): Information that is delivered as a unit 249 among peer entities of a layered network consisting of protocol 250 control information (typically a header) and possibly user data 251 (payload) of that layer. The scope of this document includes 252 layer 2 and layer 3 networks, where the PDU is respectively termed 253 a frame or a packet (or a cell in ATM). PDU is a general term for 254 any of these. This definition also includes a payload with a shim 255 header lying somewhere between layer 2 & 3. 257 Transport: The end-to-end transmission control function, 258 conventionally considered at layer-4 in the OSI reference model. 259 Given the audience for this document will often use the word 260 transport to mean low level bit carriage, whenever the term is 261 used it will be qualified, e.g. 'L4 transport'. 263 Encapsulator: The link or tunnel endpoint function that adds an 264 outer header to a PDU (also termed the 'link ingress', the 'subnet 265 ingress', the 'ingress tunnel endpoint' or just the 'ingress' 266 where the context is clear). 268 Decapsulator: The link or tunnel endpoint function that removes an 269 outer header from a PDU (also termed the 'link egress', the 270 'subnet egress', the 'egress tunnel endpoint' or just the 'egress' 271 where the context is clear). 273 Incoming header: The header of an arriving PDU before encapsulation. 275 Outer header: The header added to encapsulate a PDU. 277 Inner header: The header encapsulated by the outer header. 279 Outgoing header: The header forwarded by the decapsulator. 281 CE: Congestion Experienced [RFC3168] 282 ECT: ECN-Capable Transport [RFC3168] 284 Not-ECT: Not ECN-Capable Transport [RFC3168] 286 ECN-PDU: A PDU that is part of a feedback loop within which all the 287 nodes that need to propagate explicit congestion notifications 288 back to the Load Regulator are ECN-capable. An IP packet with a 289 non-zero ECN field implies that the endpoints are ECN-capable, so 290 this would be an ECN-PDU. However, ECN-PDU is intended to be a 291 general term for a PDU at any layer, not just IP. 293 Not-ECN-PDU: A PDU that is part of a feedback-loop within which some 294 nodes necessary to propagate explicit congestion notifications 295 back to the load regulator are not ECN-capable. 297 Load Regulator: For each flow of PDUs, the transport function that 298 is capable of controlling the data rate. Typically located at the 299 data source, but in-path nodes can regulate load in some 300 congestion control arrangements (e.g. admission control or 301 policing nodes). Note the term "a function capable of controlling 302 the load" deliberately includes a transport that doesn't actually 303 control the load but ideally it ought to (e.g. a sending 304 application without congestion control that uses UDP). 306 Congestion Baseline: The location of the function on the path that 307 initialised the values of all congestion notification fields in a 308 sequence of packets, before any are set to the congestion 309 experienced (CE) codepoint if they experience congestion further 310 downstream. Typically the original data source at layer-4. 312 3. Modes of Operation 314 This section sets down the different modes by which congestion 315 information is passed between the lower layer and the higher one. It 316 acts as a reference framework for the following sections, which give 317 normative guidelines for designers of explicit congestion 318 notification protocols, taking each mode in turn: 320 Feed-Forward-and-Up: Nodes feed forward congestion notification 321 towards the egress within the lower layer then up and along the 322 layers towards the end-to-end destination at the transport layer. 323 The following local optimisation is possible: 325 Feed-Up-and-Forward: A lower layer switch feeds-up congestion 326 notification directly into the ECN field in the higher layer 327 (e.g. IP) header, irrespective of whether the node is at the 328 egress of a subnet. 330 Feed-Backward: Nodes feed back congestion signals towards the 331 ingress of the lower layer and (optionally) attempt to control 332 congestion within their own layer. 334 Null: Nodes cannot experience congestion at the lower layer except 335 at ingress nodes (which are IP-aware or equivalently higher-layer- 336 aware). 338 3.1. Feed-Forward-and-Up Mode 340 Like IP and MPLS, many subnet technologies are based on self- 341 contained protocol data units (PDUs) or frames sent unreliably. They 342 provide no feedback channel at the subnetwork layer, instead relying 343 on higher layers (e.g. TCP) to feed back loss signals. 345 In these cases, ECN may best be supported by standardising explicit 346 notification of congestion into the lower layer protocol that carries 347 the data forwards. It will then also be necessary to define how the 348 egress of the lower layer subnet propagates this explicit signal into 349 the forwarded upper layer (IP) header. It can then continue forwards 350 until it finally reaches the destination transport (at L4). Then 351 typically the destination will feed this congestion notification back 352 to the source transport using an end-to-end protocol (e.g. TCP). 353 This is the arrangement that has already been used to add ECN to IP- 354 in-IP tunnels [RFC6040], IP-in-MPLS and MPLS-in-MPLS [RFC5129]. 356 This mode is illustrated in Figure 1. Along the middle of the 357 figure, layers 2, 3 & 4 of the protocol stack are shown, and one 358 packet is shown along the bottom as it progresses across the network 359 from source to destination, crossing two subnets connected by a 360 router, and crossing two switches on the path across each subnet. 361 Congestion at the output of the first switch (shown as *) leads to a 362 congestion marking in the L2 header (shown as C in the illustration 363 of the packet). The chevrons show the progress of the resulting 364 congestion indication. It is propagated from link to link across the 365 subnet in the L2 header, then when the router removes the marked L2 366 header, it propagates the marking up into the L3 (IP) header. The 367 router forwards the marked L3 header into subnet 2, and when it adds 368 a new L2 header it copies the L3 marking into the L2 header as well, 369 as shown by the 'C's in both layers (assuming the technology of 370 subnet 2 also supports explicit congestion marking). 372 Note that there is no implication that each 'C' marking is encoded 373 the same; a different encoding might be used for the 'C' marking in 374 each protocol. 376 Finally, for completeness, we show the L3 marking arriving at the 377 destination, where the host transport protocol (e.g. TCP) feeds it 378 back to the source in the L4 acknowledgement (the 'C' at L4 in the 379 packet at the top of the diagram). 381 _ _ _ 382 /_______ | | |C| ACK Packet (V) 383 \ |_|_|_| 384 +---+ layer: 2 3 4 header +---+ 385 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 386 | | +---+ | ^ | 387 | | . . . . . . Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3 388 | | +---+ +---+ | ^ | +---+ +---+ | | 389 | | | *|>>>>>|>>>|>>>>>|>^ | | | | | | |L2 390 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 391 source subnet A router subnet B dest 392 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ 393 | | | | | | | | |C| | | |C| | | |C|C| Data________\ 394 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) / 395 layer: 4 3 2A 4 3 2A 4 3 4 3 2B 396 header 398 Figure 1: Feed-Forward-and-Up Mode 400 Of course, modern networks are rarely as simple as this text-book 401 example, often involving multiple nested layers. For example, a 3GPP 402 mobile network may have two IP-in-IP (GTP) tunnels in series and an 403 MPLS backhaul between the base station and the first router. 404 Nonetheless, the example illustrates the general idea of feeding 405 congestion notification forward then upward whenever a header is 406 removed at the egress of a subnet. 408 Note that the FECN (forward ECN) bit in Frame Relay and the explicit 409 forward congestion indication (EFCI [ITU-T.I.371]) bit in ATM user 410 data cells follow a feed-forward pattern. However, in ATM, this is 411 only as part of a feed-forward-and-backward pattern at the lower 412 layer, not feed-forward-and-up out of the lower layer--the intention 413 was never to interface to IP ECN at the subnet egress. To our 414 knowledge, Frame Relay FECN is solely used to detect where more 415 capacity should be provisioned [Buck00]. 417 3.2. Feed-Up-and-Forward Mode 419 Ethernet is particularly difficult to extend incrementally to support 420 explicit congestion notification. One way to support ECN in such 421 cases has been to use so called 'layer-3 switches'. These are 422 Ethernet switches that bury into the Ethernet payload to find an IP 423 header and manipulate or act on certain IP fields (specifically 424 Diffserv & ECN). For instance, in Data Center TCP [DCTCP], layer-3 425 switches are configured to mark the ECN field of the IP header within 426 the Ethernet payload when their output buffer becomes congested. 427 With respect to switching, a layer-3 switch acts solely on the 428 addresses in the Ethernet header; it doesn't use IP addresses, and it 429 doesn't decrement the TTL field in the IP header. 431 _ _ _ 432 /_______ | | |C| ACK packet (V) 433 \ |_|_|_| 434 +---+ layer: 2 3 4 header +---+ 435 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 436 | | +---+ | ^ | 437 | | . . . >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3 438 | | +--^+ +---+ | | +---+ +---+ | | 439 | | | *| | | | | | | | | | |L2 440 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 441 source subnet E router subnet F dest 442 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ 443 | | | | | | | |C| | | | |C| | | |C|C| data________\ 444 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / 445 layer: 4 3 2 4 3 2 4 3 4 3 2 446 header 448 Figure 2: Feed-Up-and-Forward Mode 450 By comparing Figure 2 with Figure 1, it can be seen that subnet E 451 (perhaps a subnet of layer-3 Ethernet switches) works in feed-up-and- 452 forward mode by notifying congestion directly into L3 at the point of 453 congestion, even though the congested switch does not otherwise act 454 at L3. In this example, the technology in subnet F (e.g. MPLS) does 455 support ECN natively, so when the router adds the layer-2 header it 456 copies the ECN marking from L3 to L2 as well. 458 3.3. Feed-Backward Mode 460 In some layer 2 technologies, explicit congestion notification has 461 been defined for use internally within the subnet with its own 462 feedback and load regulation, but typically the interface with IP for 463 ECN has not been defined. 465 For instance, for the available bit-rate (ABR) service in ATM, the 466 relative rate mechanism was one of the more popular mechanisms for 467 managing traffic, tending to supersede earlier designs. In this 468 approach ATM switches send special resource management (RM) cells in 469 both the forward and backward directions to control the ingress rate 470 of user data into a virtual circuit. If a switch buffer is 471 approaching congestion or congested it sends an RM cell back towards 472 the ingress with respectively the No Increase (NI) or Congestion 473 Indication (CI) bit set in its message type field [ATM-TM-ABR]. The 474 ingress then holds or decreases its sending bit-rate accordingly. 476 _ _ _ 477 /_______ | | |C| ACK packet (X) 478 \ |_|_|_| 479 +---+ layer: 2 3 4 header +---+ 480 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4 481 | | +---+ | ^ | 482 | | | *|>>> Packet W >>>>>>>>>>>>|>^ |L3 483 | | +---+ +---+ | | +---+ +---+ | | 484 | | | | | | | <|<<<<<|<<<|<(V)<|<<<| | |L2 485 | | . . | . |Packet U | . . | . | . . | . | . . | .*| . . | |L2 486 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 487 source subnet G router subnet H dest 488 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ later 489 | | | | | | | | | | | | | | | | |C| | data________\ 490 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (W) / 491 4 3 2 4 3 2 4 3 4 3 2 492 _ 493 /__ |C| Feedback control 494 \ |_| cell/frame (V) 495 2 496 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ earlier 497 | | | | | | | | | | | | | | | | | | | data________\ 498 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / 499 layer: 4 3 2 4 3 2 4 3 4 3 2 500 header 502 Figure 3: Feed-Backward Mode 504 ATM's feed-backward approach doesn't fit well when layered beneath 505 IP's feed-forward approach--unless the initial data source is the 506 same node as the ATM ingress. Figure 3 shows the feed-backward 507 approach being used in subnet H. If the final switch on the path is 508 congested (*), it doesn't feed-forward any congestion indications on 509 packet (U). Instead it sends a control cell (V) back to the router 510 at the ATM ingress. 512 However, the backward feedback doesn't reach the original data source 513 directly because IP doesn't support backward feedback (and subnet G 514 is independent of subnet H). Instead, the router in the middle 515 throttles down its sending rate but the original data sources don't 516 reduce their rates. The resulting rate mismatch causes the middle 517 router's buffer at layer 3 to back up until it becomes congested, 518 which it signals forwards on later data packets at layer 3 (e.g. 519 packet W). Note that the forward signal from the middle router is 520 not triggered directly by the backward signal. Rather, it is 521 triggered by congestion resulting from the middle router's mismatched 522 rate response to the backward signal. 524 In response to this later forward signalling, end-to-end feedback at 525 layer-4 finally completes the tortuous path of congestion indications 526 back to the origin data source, as before. 528 3.4. Null Mode 530 Often link and physical layer resources are 'non-blocking' by design. 531 In these cases congestion notification may be implemented but it does 532 not need to be deployed at the lower layer; ECN in IP would be 533 sufficient. 535 A degenerate example is a point-to-point Ethernet link. Excess 536 loading of the link merely causes the queue from the higher layer to 537 back up, while the lower layer remains immune to congestion. Even a 538 whole meshed subnetwork can be made immune to interior congestion by 539 limiting ingress capacity and careful sizing of links, particularly 540 if multi-path routing is used to ensure even worst-case patterns of 541 load cannot congest any link. 543 4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 544 Notification 546 Feed-forward-and-up is the mode already used for signalling ECN up 547 the layers through MPLS into IP [RFC5129] and through IP-in-IP 548 tunnels [RFC6040]. These RFCs take a consistent approach and the 549 following guidelines are designed to ensure this consistency 550 continues as ECN support is added to other protocols that encapsulate 551 IP. The guidelines are also designed to ensure compliance with the 552 more general best current practice for the design of alternate ECN 553 schemes given in [RFC4774]. 555 The rest of this section is structured as follows: 557 o Section 4.1 addresses the most straightforward cases, where 558 [RFC6040] can be applied directly to add ECN to tunnels that are 559 effectively the same as IP-in-IP tunnels. 561 o The subsequent sections give guidelines for adding ECN to a subnet 562 technology that uses feed-forward-and-up mode like IP, but it is 563 not so similar to IP that [RFC6040] rules can be applied directly. 564 Specifically: 566 * Sections 4.2, 4.3 and 4.4 respectively address how to add ECN 567 support to the wire protocol and to the encapsulators and 568 decapsulators at the ingress and egress of the subnet. 570 * Section 4.5 deals with the special, but common, case of 571 sequences of tunnels or subnets that all use the same 572 technology 574 * Section 4.6 deals with the question of reframing when IP 575 packets do not map 1:1 into lower layer frames. 577 4.1. IP-in-IP Tunnels with Tightly Coupled Shim Headers 579 A common pattern for many tunnelling protocols is to encapsulate an 580 inner IP header with shim header(s) then an outer IP header. In many 581 cases the shim header(s) always have to be tightly coupled to the 582 outer IP header because they are not sufficient as outer headers in 583 their own right. In such cases the shim header(s) and the outer IP 584 header are always added (or removed) in the same operation. 585 Therefore, in all such tightly coupled IP-in-IP tunnelling protocols, 586 the rules in [RFC6040] for propagating the ECN field between the two 587 IP headers SHOULD be applied directly. 589 Examples of tightly coupled IP-in-IP tunnelling protocols where 590 [RFC6040] can be applied directly are: 592 o L2TP [RFC2661] 594 o GRE [RFC1701], [RFC2784] 596 o PPTP [RFC2637] 598 o GTP [GTPv1], [GTPv1-U], [GTPv2-C] 600 o VXLAN [vxlan]. 602 4.2. Wire Protocol Design: Indication of ECN Support 604 This section is intended to guide the redesign of any lower layer 605 protocol that encapsulate IP to add native ECN support at the lower 606 layer. It reflects the approaches used in [RFC6040] and in 607 [RFC5129]. Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS 608 encapsulations that already comply with [RFC6040] or [RFC5129] will 609 already satisfy this guidance. 611 A lower layer (or subnet) congestion notification system: 613 1. SHOULD NOT apply explicit congestion notifications to PDUs that 614 are destined for legacy layer-4 transport implementations that 615 will not understand ECN, and 617 2. SHOULD NOT apply explicit congestion notifications to PDUs if the 618 egress of the subnet might not propagate congestion notifications 619 onward into the higher layer. 621 We use the term ECN-PDUs for a PDU on a feedback loop that will 622 propagate congestion notification properly because it meets both 623 the above criteria. And a Not-ECN-PDU is a PDU on a feedback 624 loop that does not meet both criteria, and will therefore not 625 propagate congestion notification properly. A corollary of the 626 above is that a lower layer congestion notification protocol: 628 3. SHOULD be able to distinguish ECN-PDUs from Not-ECN-PDUs. 630 Note that there is no need for all interior nodes within a subnet to 631 be able to mark congestion explicitly. A mix of ECN and drop signals 632 from different nodes is fine. However, if _any_ interior nodes might 633 generate ECN markings, guideline 2 above says that all relevant 634 egress node(s) SHOULD be able to propagate those markings up to the 635 higher layer. 637 In IP, if the ECN field in each PDU is cleared to the Not-ECT (not 638 ECN-capable transport) codepoint, it indicates that the L4 transport 639 will not understand congestion markings. A congested buffer must not 640 mark these Not-ECT PDUs, and therefore drops them instead. 642 The mechanism a lower layer uses to distinguish the ECN-capability of 643 PDUs need not mimic that of IP. All the above guidelines say is that 644 the lower layer system, as a whole, should achieve the same outcome. 645 For instance, ECN-capable feedback loops might use PDUs that are 646 identified by a particular set of labels or tags. Alternatively, 647 logical link protocols that use flow state might determine whether a 648 PDU can be congestion marked by checking for ECN-support in the flow 649 state. Other protocols might depend on out-of-band control signals. 651 The per-domain checking of ECN support in MPLS [RFC5129] is a good 652 example of a way to avoid sending congestion markings to transports 653 that will not understand them, without using any header space in the 654 subnet protocol. 656 In MPLS, header space is extremely limited, therefore RFC5129 does 657 not provide a field in the MPLS header to indicate whether the PDU is 658 an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are 659 allowed to set explicit congestion indications without checking 660 whether the PDU is destined for a transport that will understand 661 them. Nonetheless, this is made safe by requiring that the network 662 operator upgrades all decapsulating edges of a whole domain at once, 663 as soon as even one switch within the domain is configured to mark 664 rather than drop during congestion. Therefore, any edge node that 665 might decapsulate a packet will be capable of checking whether the 666 higher layer transport is ECN-capable. When decapsulating a CE- 667 marked packet, if the decapsulator discovers that the higher layer 668 (inner header) indicates the transport is not ECN-capable, it drops 669 the packet--effectively on behalf of the earlier congested node (see 670 Decapsulation Guideline 1 in Section 4.4). 672 It was only appropriate to define such an incremental deployment 673 strategy because MPLS is targeted solely at professional operators, 674 who can be expected to ensure that a whole subnetwork is consistently 675 configured. This strategy might not be appropriate for other link 676 technologies targeted at zero-configuration deployment or deployment 677 by the general public (e.g. Ethernet). For such 'plug-and-play' 678 environments it will be necessary to invent a failsafe approach that 679 ensures congestion markings will never fall into black holes, no 680 matter how inconsistently a system is put together. Alternatively, 681 congestion notification relying on correct system configuration could 682 be confined to flavours of Ethernet intended only for professional 683 network operators, such as IEEE 802.1ah Provider Backbone Bridges 684 (PBB). 686 QCN [IEEE802.1Qau] provides another example of how to indicate to 687 lower layer devices that the end-points will not understand ECN. An 688 operator can define certain 802.1p classes of service to indicate 689 non-QCN frames and an ingress bridge is required to map arriving not- 690 QCN-capable IP packets to one of these non-QCN 802.1p classes. 692 4.3. Encapsulation Guidelines 694 This section is intended to guide the redesign of any node that 695 encapsulates IP with a lower layer header when adding native ECN 696 support to the lower layer protocol. It reflects the approaches used 697 in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or IP-in- 698 MPLS or MPLS-in-MPLS encapsulations that already comply with 699 [RFC6040] or [RFC5129] will already satisfy this guidance. 701 1. Egress Capability Check: A subnet ingress needs to be sure that 702 the corresponding egress of a subnet will propagate any 703 congestion notification added to the outer header across the 704 subnet. This is necessary in addition to checking that an 705 incoming PDU indicates an ECN-capable (L4) transport. Examples 706 of how this guarantee might be provided include: 708 * by configuration (e.g. if any label switches in a domain 709 support ECN marking, [RFC5129] requires all egress nodes to 710 have been configured to propagate ECN) 712 * by the ingress explicitly checking that the egress propagates 713 ECN (e.g. TRILL uses IS-IS to check path capabilities before 714 using critical options [trill-rbridge-options]) 716 * by inherent design of the protocol (e.g. by encoding ECN 717 marking on the outer header in such a way that a legacy egress 718 that does not understand ECN will consider the PDU corrupt and 719 discard it, thus at least propagating a form of congestion 720 signal). 722 2. Egress Fails Capability Check: If the ingress cannot guarantee 723 that the egress will propagate congestion notification, the 724 ingress SHOULD disable ECN when it forwards the PDU at the lower 725 layer. An example of how the ingress might disable ECN at the 726 lower layer would be by setting the outer header of the PDU to 727 identify it as a Not-ECN-PDU, assuming the subnet technology 728 supports such a concept. 730 3. Standard Congestion Monitoring Baseline: Once the ingress to a 731 subnet has established that the egress will correctly propagate 732 ECN, on encapsulation it SHOULD encode the same level of 733 congestion in outer headers as is arriving in incoming headers. 734 For example it might copy any incoming congestion notification 735 into the outer header of the lower layer protocol. 737 This ensures that all outer headers reflect congestion 738 accumulated along the whole upstream path since the Load 739 Regulator, not just since the ingress of the subnet. A node that 740 is not the Load Regulator SHOULD NOT re-initialise the level of 741 CE markings in the outer to zero. 743 This guideline is intended to ensure that any bulk congestion 744 monitoring of outer headers (e.g. by a network management node 745 monitoring ECN in passing frames) is most meaningful. For 746 instance, if an operator measures CE in 0.4% of passing outer 747 headers, this information is only useful if the operator knows 748 where the proportion of CE markings was last initialised to 0% 749 (the Congestion Baseline). Such monitoring information will not 750 be useful if some subnet ingress nodes reset all outer CE 751 markings while others copy incoming CE markings into the outer. 753 Most information can be extracted if the Congestion Baseline is 754 standardised at the node that is regulating the load (the Load 755 Regulator--typically the data source). Then the operator can 756 measure both congestion since the Load Regulator, and congestion 757 since the subnet ingress. The latter might be measurable by 758 subtracting the level of CE markings on inner headers from that 759 on outer headers (see Appendix C of [RFC6040]). 761 4.4. Decapsulation Guidelines 763 This section is intended to guide the redesign of any node that 764 decapsulates IP from within a lower layer header when adding native 765 ECN support to the lower layer protocol. It reflects the approaches 766 used in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or 767 IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with 768 [RFC6040] or [RFC5129] will already satisfy this guidance. 770 A subnet egress SHOULD NOT simply copy congestion notification from 771 outer headers to the forwarded header. It SHOULD calculate the 772 outgoing congestion notification field from the inner and outer 773 headers using the following guidelines. If there is any conflict, 774 rules earlier in the list take precedence over rules later in the 775 list: 777 1. If the arriving inner header is a Not-ECN-PDU it implies the L4 778 transport will not understand explicit congestion markings. 779 Then: 781 * If the outer header carries an explicit congestion marking, 782 the packet SHOULD be dropped--the only indication of 783 congestion that the L4 transport will understand. 785 * If the outer is an ECN-PDU that carries no indication of 786 congestion or a Not-ECN-PDU the PDU SHOULD be forwarded, but 787 still as a Not-ECN-PDU. 789 2. If the outer header does not support explicit congestion 790 notification (a Not-ECN-PDU), but the inner header does (an ECN- 791 PDU), the inner header SHOULD be forwarded unchanged. 793 3. In some lower layer protocols congestion may be signalled as a 794 numerical level, such as in the control frames of quantised 795 congestion notification [IEEE802.1Qau]. If such a multi-bit 796 encoding encapsulates an ECN-capable IP data packet, a function 797 will be needed to convert the quantised congestion level into the 798 frequency of congestion markings in outgoing IP packets. 800 4. Congestion indications may be encoded by a severity level. For 801 instance increasing levels of congestion might be encoded by 802 numerically increasing indications, e.g. pre-congestion 803 notification (PCN) can be encoded in each PDU at three severity 804 levels in IP or MPLS [RFC6660]. 806 If the arriving inner header is an ECN-PDU, where the inner and 807 outer headers carry indications of congestion of different 808 severity, the more severe indication SHOULD be forwarded in 809 preference to the less severe. 811 5. The inner and outer headers might carry a combination of 812 congestion notification fields that should not be possible given 813 any currently used protocol transitions. For instance, if 814 Encapsulation Guideline 3 in Section 4.3 had been followed, it 815 should not be possible to have a less severe indication of 816 congestion in the outer than in the inner. It MAY be appropriate 817 to log unexpected combinations of headers and possibly raise an 818 alarm. 820 If a safe outgoing codepoint can be defined for such a PDU, the 821 PDU SHOULD be forwarded rather than dropped. Some implementers 822 discard PDUs with currently unused combinations of headers just 823 in case they represent an attack. However, an approach using 824 alarms and policy-mediated drop is preferable to hard-coded drop, 825 so that operators can keep track of possible attacks but 826 currently unused combinations are not precluded from future use 827 through new standards actions. 829 4.5. Sequences of Similar Tunnels or Subnets 831 In some deployments, particularly in 3GPP networks, an IP packet may 832 traverse two or more IP-in-IP tunnels in sequence that all use 833 identical technology (e.g. GTP). 835 In such cases, it would be sufficient for every encapsulation and 836 decapsulation in the chain to comply with RFC 6040. Alternatively, 837 as an optimisation, a node that decapsulates a packet and immediately 838 re-encapsulates it for the next tunnel MAY copy the incoming outer 839 ECN field directly to the outgoing outer and the incoming inner ECN 840 field directly to the outgoing inner. Then the overall behavior 841 across the sequence of tunnel segments would still be consistent with 842 RFC 6040. 844 Appendix C of RFC6040 describes how a tunnel egress can monitor how 845 much congestion has been introduced within a tunnel. A network 846 operator might want to monitor how much congestion had been 847 introduced within a whole sequence of tunnels. Using the technique 848 in Appendix C of RFC6040 at the final egress, the operator could 849 monitor the whole sequence of tunnels, but only if the above 850 optimisation were used consistently along the sequence of tunnels, in 851 order to make it appear as a single tunnel. Therefore, tunnel 852 endpoint implementations SHOULD allow the operator to configure 853 whether this optimisation is enabled. 855 When ECN support is added to a subnet technology, consideration 856 SHOULD be given to a similar optimisation between subnets in sequence 857 if they all use the same technology. 859 4.6. Reframing and Congestion Markings 861 The guidance in this section is worded in terms of framing 862 boundaries, but it applies equally whether the protocol data units 863 are frames, cells or packets. 865 Where framing boundaries are different between two layers, congestion 866 indications SHOULD be propagated on the basis that a congestion 867 indication on a PDU applies to all the octets in the PDU. On 868 average, an encapsulator or decapsulator SHOULD approximately 869 preserve the number of marked octets arriving and leaving (counting 870 the size of inner headers, but not added encapsulating headers). 872 The next departing frame SHOULD be immediately marked even if only 873 enough incoming marked octets have arrived for part of the departing 874 frame. This ensures that any outstanding congestion marked octets 875 are propagated immediately, rather than held back waiting for a frame 876 no bigger than the outstanding marked octets--which might involve a 877 long wait. 879 For instance, an algorithm for marking departing frames could 880 maintain a counter representing the balance of arriving marked octets 881 minus departing marked octets. It adds the size of every marked 882 frame that arrives and if the counter is positive it marks the next 883 frame to depart and subtracts its size from the counter. This will 884 often leave a negative remainder in the counter, which is deliberate. 886 5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 887 Notification 889 The guidance in this section is applicable when IP packets: 891 o are encapsulated in Ethernet headers; 893 o are forwarded by the eNode-B (base station) of a 3GPP radio access 894 network, which is required to apply ECN marking during congestion 895 [LTE-RA]. 897 This guidance also generalises to encapsulation by other subnet 898 technologies with no native support for explicit congestion 899 notification at the lower layer, but with support for finding and 900 processing an IP header. It is unlikely to be applicable or 901 necessary for IP-in-IP encapsulation, where feed-forward-and-up mode 902 based on [RFC6040] would be more appropriate. 904 Marking the IP header while switching at layer-2 (by using a layer-3 905 switch) or while forwarding in a radio access network seems to 906 represent a layering violation. However, it can be considered as a 907 benign optimisation if the guidelines below are followed. Feed-up- 908 and-forward is certainly not a general alternative to implementing 909 feed-forward congestion notification in the lower layer, because: 911 o IPv4 and IPv6 are not the only layer-3 protocols that might be 912 encapsulated by lower layer protocols 914 o Link-layer encryption might be in use, making the layer-2 payload 915 inaccessible 917 o Many Ethernet switches do not have 'layer-3 switch' capabilities 918 so they cannot read or modify an IP payload 920 o It might be costly to find an IP header (v4 or v6) when it may be 921 encapsulated by more than one lower layer header, e.g. Ethernet 922 MAC in MAC [IEEE802.1Qah]. 924 Nonetheless, configuring lower layer equipment to look for an ECN 925 field in an encapsulated IP header is a useful optimisation. If the 926 implementation follows the guidelines below, this optimisation does 927 not have to be confined to a controlled environment such as within a 928 data centre; it could usefully be applied on any network--even if the 929 operator is not sure whether the above issues will never apply: 931 1. If a native lower-layer congestion notification mechanism exists 932 for a subnet technology, it is safe to mix feed-up-and-forward 933 with feed-forward-and-up on other switches in the same subnet. 934 However, it will generally be more efficient to use the native 935 mechanism. 937 2. The depth of the search for an IP header SHOULD be limited. If 938 an IP header is not found soon enough, or an unrecognised or 939 unreadable header is encountered, the switch SHOULD resort to an 940 alternative means of signalling congestion (e.g. drop, or the 941 native lower layer mechanism if available). 943 3. It is sufficient to use the first IP header found in the stack; 944 the egress of the relevant tunnel can propagate congestion 945 notification upwards to any more deeply encapsulated IP headers 946 later. 948 6. Feed-Backward Mode: Guidelines for Adding Congestion Notification 950 It can be seen from Section 3.3 that congestion notification in a 951 subnet using feed-backward mode has generally not been designed to be 952 directly coupled with IP layer congestion notification. The subnet 953 attempts to minimise congestion internally, and if the incoming load 954 at the ingress exceeds the capacity somewhere through the subnet, the 955 layer 3 buffer into the ingress backs up. Thus, a feed-backward mode 956 subnet is in some sense similar to a null mode subnet, in that there 957 is no need for any direct interaction between the subnet and higher 958 layer congestion notification. Therefore no detailed protocol design 959 guidelines are appropriate. Nonetheless, a more general guideline is 960 appropriate: 962 1. A subnetwork technology intended to eventually interface to IP 963 SHOULD NOT be designed using only the feed-backward mode, which 964 is certainly best for a stand-alone subnet, but would need to be 965 modified to work efficiently as part of the wider Internet, 966 because IP uses feed-forward-and-up mode. 968 The feed-backward approach at least works beneath IP, where the term 969 'works' is used only in a narrow functional sense because feed- 970 backward can result in very inefficient and sluggish congestion 971 control--except if it is confined to the subnet directly connected to 972 the original data source, when it is faster than feed-forward. It 973 would be valid to design a protocol that could work in feed-backward 974 mode for paths that only cross one subnet, and in feed-forward-and-up 975 mode for paths that cross subnets. 977 In the early days of TCP/IP, a similar feed-backward approach was 978 tried for explicit congestion signalling, using source-quench (SQ) 979 ICMP control packets. However, SQ fell out of favour and is now 980 formally deprecated [RFC6633]. The main problem was that it is hard 981 for a data source to tell the difference between a spoofed SQ message 982 and a quench request from a genuine buffer on the path. It is also 983 hard for a lower layer buffer to address an SQ message to the 984 original source port number, which may be buried within many layers 985 of headers, and possibly encrypted. 987 Quantised congestion notification (QCN--also known as backward 988 congestion notification or BCN) [IEEE802.1Qau] uses a feed-backward 989 mode structurally similar to ATM's relative rate mechanism. However, 990 QCN confines its applicability to scenarios such as some data centres 991 where all endpoints are directly attached by the same Ethernet 992 technology. If a QCN subnet were later connected into a wider IP- 993 based internetwork (e.g. when attempting to interconnect multiple 994 data centres) it would suffer the inefficiency shown Figure 3. 996 7. IANA Considerations (to be removed by RFC Editor) 998 This memo includes no request to IANA. 1000 8. Security Considerations 1002 If a lower layer wire protocol is redesigned to include explicit 1003 congestion signalling in-band in the protocol header, care SHOULD be 1004 take to ensure that the field used is specified as mutable during 1005 transit. Otherwise interior nodes signalling congestion would 1006 invalidate any authentication protocol applied to the lower layer 1007 header--by altering a header field that had been assumed as 1008 immutable. 1010 The redesign of protocols that encapsulate IP in order to propagate 1011 congestion signals between layers raises potential signal integrity 1012 concerns. Experimental or proposed approaches exist for assuring the 1013 end-to-end integrity of in-band congestion signals, e.g.: 1015 o Congestion exposure (ConEx ) for networks to audit that their 1016 congestion signals are not being suppressed by other networks or 1017 by receivers, and for networks to police that senders are 1018 responding sufficiently to the signals, irrespective of the 1019 transport protocol used [I-D.ietf-conex-abstract-mech]. 1021 o The ECN nonce [RFC3540] for a TCP sender to detect whether a 1022 network or the receiver is suppressing congestion signals. 1024 o A test with the same goals as the ECN nonce, but without the need 1025 for the receiver to co-operate with the protocol 1026 [I-D.moncaster-tcpm-rcv-cheat]. 1028 Given these end-to-end approaches are already being specified, it 1029 would make little sense to attempt to design hop-by-hop congestion 1030 signal integrity into a new lower layer protocol, because end-to-end 1031 integrity inherently achieves hop-by-hop integrity. 1033 9. Conclusions 1035 Following the guidance in the document enables ECN support to be 1036 extended to numerous protocols that encapsulate IP (v4 & v6) in a 1037 consistent way, so that IP continues to fulfil its role as an end-to- 1038 end interoperability layer. This includes: 1040 o A wide range of tunnelling protocols with various forms of shim 1041 header between two IP headers; 1043 o A wide range of subnet technologies, particularly those that work 1044 in the same 'feed-forward-and-up' mode that is used to support ECN 1045 in IP and MPLS. 1047 Guidelines have been defined for supporting propagation of ECN 1048 between Ethernet and IP on so-called Layer-3 Ethernet switches, using 1049 a 'feed-up-an-forward' mode. This approach could enable other subnet 1050 technologies to pass ECN signals into the IP layer, even if they do 1051 not support ECN natively. 1053 Finally, attempting to add ECN to a subnet technology in feed- 1054 backward mode is deprecated except in special cases, due to its 1055 likely sluggish response to congestion. 1057 10. Acknowledgements 1059 Thanks to Gorry Fairhurst for extensive reviews. Thanks also to the 1060 following reviewers: Ingemar Johansson and Piers O'Hanlon and Michael 1061 Welzl, who pointed out that lower layer congestion notification 1062 signals may have different semantics to those in IP. 1064 Bob Briscoe was part-funded by the European Community under its 1065 Seventh Framework Programme through the Trilogy project (ICT-216372) 1066 for initial drafts and through the Reducing Internet Transport 1067 Latency (RITE) project (ICT-317700) subsequently. The views 1068 expressed here are solely those of the authors. 1070 11. Comments Solicited 1072 Comments and questions are encouraged and very welcome. They can be 1073 addressed to the IETF Transport Area working group mailing list 1074 , and/or to the authors. 1076 12. References 1078 12.1. Normative References 1080 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1081 Requirement Levels", BCP 14, RFC 2119, March 1997. 1083 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1084 of Explicit Congestion Notification (ECN) to IP", RFC 1085 3168, September 2001. 1087 [RFC3819] Karn, P., Bormann, C., Fairhurst, G., Grossman, D., 1088 Ludwig, R., Mahdavi, J., Montenegro, G., Touch, J., and L. 1089 Wood, "Advice for Internet Subnetwork Designers", BCP 89, 1090 RFC 3819, July 2004. 1092 [RFC4774] Floyd, S., "Specifying Alternate Semantics for the 1093 Explicit Congestion Notification (ECN) Field", BCP 124, 1094 RFC 4774, November 2006. 1096 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 1097 Marking in MPLS", RFC 5129, January 2008. 1099 [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion 1100 Notification", RFC 6040, November 2010. 1102 12.2. Informative References 1104 [ATM-TM-ABR] 1105 Cisco, "Understanding the Available Bit Rate (ABR) Service 1106 Category for ATM VCs", Design Technote 10415, June 2005. 1108 [Buck00] Buckwalter, J., "Frame Relay: Technology and Practice", 1109 Pub. Addison Wesley ISBN-13: 978-0201485240, 2000. 1111 [DCTCP] Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel, 1112 P., Prabhakar, B., Sengupta, S., and M. Sridharan, "Data 1113 Center TCP (DCTCP)", ACM SIGCOMM CCR 40(4)63--74, October 1114 2010, . 1116 [GTPv1-U] 3GPP, "General Packet Radio System (GPRS) Tunnelling 1117 Protocol User Plane (GTPv1-U)", Technical Specification TS 1118 29.281, . 1120 [GTPv1] 3GPP, "GPRS Tunnelling Protocol (GTP) across the Gn and Gp 1121 interface", Technical Specification TS 29.060, . 1123 [GTPv2-C] 3GPP, "Evolved General Packet Radio Service (GPRS) 1124 Tunnelling Protocol for Control plane (GTPv2-C)", 1125 Technical Specification TS 29.274, . 1127 [I-D.ietf-conex-abstract-mech] 1128 Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) 1129 Concepts and Abstract Mechanism", draft-ietf-conex- 1130 abstract-mech-08 (work in progress), October 2013. 1132 [I-D.moncaster-tcpm-rcv-cheat] 1133 Moncaster, T., "A TCP Test to Allow Senders to Identify 1134 Receiver Non-Compliance", draft-moncaster-tcpm-rcv- 1135 cheat-01 (work in progress), June 2007. 1137 [IEEE802.1Qah] 1138 IEEE, "IEEE Standard for Local and Metropolitan Area 1139 Networks--Virtual Bridged Local Area Networks--Amendment 1140 6: Provider Backbone Bridges", IEEE Std 802.1Qah-2008, 1141 August 2008, 1142 . 1144 (Access Controlled link within page) 1146 [IEEE802.1Qau] 1147 Finn, N., Ed., "IEEE Standard for Local and Metropolitan 1148 Area Networks--Virtual Bridged Local Area Networks - 1149 Amendment 13: Congestion Notification", IEEE Std 1150 802.1Qau-2010, March 2010, . 1153 (Access Controlled link within page) 1155 [ITU-T.I.371] 1156 ITU-T, "Traffic Control and Congestion Control in B-ISDN", 1157 ITU-T Rec. I.371 (03/04), March 2004, 1158 . 1161 [LTE-RA] 3GPP, "Evolved Universal Terrestrial Radio Access (E-UTRA) 1162 and Evolved Universal Terrestrial Radio Access Network 1163 (E-UTRAN); Overall description; Stage 2", Technical 1164 Specification TS 36.300, . 1166 [RFC1323] Jacobson, V., Braden, B., and D. Borman, "TCP Extensions 1167 for High Performance", RFC 1323, May 1992. 1169 [RFC1701] Hanks, S., Li, T., Farinacci, D., and P. Traina, "Generic 1170 Routing Encapsulation (GRE)", RFC 1701, October 1994. 1172 [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, 1173 October 1996. 1175 [RFC2637] Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little, 1176 W., and G. Zorn, "Point-to-Point Tunneling Protocol", RFC 1177 2637, July 1999. 1179 [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, 1180 G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", 1181 RFC 2661, August 1999. 1183 [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. 1184 Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, 1185 March 2000. 1187 [RFC2884] Hadi Salim, J. and U. Ahmed, "Performance Evaluation of 1188 Explicit Congestion Notification (ECN) in IP Networks", 1189 RFC 2884, July 2000. 1191 [RFC2983] Black, D., "Differentiated Services and Tunnels", RFC 1192 2983, October 2000. 1194 [RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit 1195 Congestion Notification (ECN) Signaling with Nonces", RFC 1196 3540, June 2003. 1198 [RFC4301] Kent, S. and K. Seo, "Security Architecture for the 1199 Internet Protocol", RFC 4301, December 2005. 1201 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 1202 RFC 6633, May 2012. 1204 [RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three 1205 Pre-Congestion Notification (PCN) States in the IP Header 1206 Using a Single Diffserv Codepoint (DSCP)", RFC 6660, July 1207 2012. 1209 [trill-rbridge-options] 1210 Eastlake, D., Ghanwani, A., Manral, V., and C. Bestler, 1211 "RBridges: Further TRILL Header Extensions", draft-ietf- 1212 trill-rbridge-options-07 (work in progress), June 2012. 1214 [vxlan] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 1215 L., Sridhar, T., Bursell, M., and C. Wright, "VXLAN: A 1216 Framework for Overlaying Virtualized Layer 2 Networks over 1217 Layer 3 Networks", draft-mahalingam-dutt-dcops-vxlan-08 1218 (work in progress), February 2014. 1220 Appendix A. Outstanding Document Issues 1222 1. [GF] Concern that certain guidelines warrant a MUST (NOT) rather 1223 than a SHOULD (NOT). Given the guidelines say that if any SHOULD 1224 (NOT)s are not followed, a strong justification will be needed, 1225 they have been left as SHOULD (NOT) pending further list 1226 discussion. In particular: 1228 * If inner is a Not-ECN-PDU and Outer is CE (or highest severity 1229 congestion level), MUST (not SHOULD) drop? 1231 2. Consider whether an IETF Standard Track doc will be needed to 1232 Update the IP-in-IP protocols listed in Section 4.1--at least 1233 those that the IET 1235 Appendix B. Changes in This Version (to be removed by RFC Editor) 1237 From briscoe-04 to ietf-00: Changed filename following tsvwg 1238 adoption. 1240 From briscoe-03 to 04: 1242 * Re-arranged the introduction to describe the purpose of the 1243 document first before introducing ECN in more depth. And 1244 clarified the introduction throughout. 1246 * Added applicability to 3GPP TS 36.300. 1248 From briscoe-02 to 03: 1250 * Scope section: 1252 + Added dependence on correct propagation of traffic class 1253 information 1255 + For the feed-backward mode, deemed multicast and anycast out 1256 of scope 1258 * Ensured all guidelines referring to subnet technologies also 1259 refer to tunnels and vice versa by adding applicability 1260 sentences at the start of sections 4.1, 4.2, 4.3, 4.4, 4.6 and 1261 5. 1263 * Added Security Considerations on ensuring congestion signal 1264 fields are classed as immutable and on using end-to-end 1265 congestion signal integrity technologies rather than hop-by- 1266 hop. 1268 From briscoe-01 to 02: 1270 * Added authors: JK & PT 1272 * Added 1274 + Section 4.1 "IP-in-IP Tunnels with Tightly Coupled Shim 1275 Headers" 1277 + Section 4.5 "Sequences of Similar Tunnels or Subnets" 1279 + roadmap at the start of Section 4, given the subsections 1280 have become quite fragmented. 1282 + Section 9 "Conclusions" 1284 * Clarified why transports are starting to be able to saturate 1285 interior links 1287 * Under Section 1.1, addressed the question of alternative signal 1288 semantics and included multicast & anycast. 1290 * Under Section 3.1, included a 3GPP example. 1292 * Section 4.2. "Wire Protocol Design": 1294 + Altered guideline 2. to make it clear that it only applies 1295 to the immediate subnet egress, not later ones 1297 + Added a reminder that it is only necessary to check that ECN 1298 propagates at the egress, not whether interior nodes mark 1299 ECN 1301 + Added example of how QCN uses 802.1p to indicate support for 1302 QCN. 1304 * Added references to Appendix C of RFC6040, about monitoring the 1305 amount of congestion signals introduced within a tunnel 1307 * Appendix A: Added more issues to be addressed, including plan 1308 to produce a standards track update to IP-in-IP tunnel 1309 protocols. 1311 * Updated acks and references 1313 From briscoe-00 to 01: 1315 * Intended status: BCP (was Informational) & updates 3819 added. 1317 * Briefer Introduction: Introductory para justifying benefits of 1318 ECN. Moved all but a brief enumeration of modes of operation 1319 to their own new section (from both Intro & Scope). Introduced 1320 incr. deployment as most tricky part. 1322 * Tightened & added to terminology section 1324 * Structured with Modes of Operation, then Guidelines section for 1325 each mode. 1327 * Tightened up guideline text to remove vagueness / passive voice 1328 / ambiguity and highlight main guidelines as numbered items. 1330 * Added Outstanding Document Issues Appendix 1332 * Updated references 1334 Authors' Addresses 1336 Bob Briscoe 1337 BT 1338 B54/77, Adastral Park 1339 Martlesham Heath 1340 Ipswich IP5 3RE 1341 UK 1343 Phone: +44 1473 645196 1344 EMail: bob.briscoe@bt.com 1345 URI: http://bobbriscoe.net/ 1347 John Kaippallimalil 1348 Huawei 1349 5340 Legacy Drive, Suite 175 1350 Plano, Texas 75024 1351 USA 1353 EMail: john.kaippallimalil@huawei.com 1355 Pat Thaler 1356 Broadcom Corporation 1357 5025 Keane Drive 1358 Carmichael, CA 95608 1359 USA 1361 EMail: pthaler@broadcom.com