idnits 2.17.1 draft-ietf-tsvwg-ecn-encap-guidelines-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- -- The draft header indicates that this document updates RFC3819, but the abstract doesn't seem to mention this, which it should. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year (Using the creation date from RFC3819, updated by this document, for RFC5378 checks: 1999-10-14) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (October 8, 2015) is 3117 days in the past. Is this intentional? Checking references for intended status: Best Current Practice ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'GF' is mentioned on line 1275, but not defined == Outdated reference: A later version (-07) exists of draft-ietf-trill-rfc7180bis-06 == Outdated reference: A later version (-15) exists of draft-ietf-tsvwg-circuit-breaker-05 -- Obsolete informational reference (is this intentional?): RFC 1323 (Obsoleted by RFC 7323) Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Transport Area Working Group B. Briscoe 3 Internet-Draft Simula Research Laboratory 4 Updates: 3819 (if approved) J. Kaippallimalil 5 Intended status: Best Current Practice Huawei 6 Expires: April 10, 2016 P. Thaler 7 Broadcom Corporation 8 October 8, 2015 10 Guidelines for Adding Congestion Notification to Protocols that 11 Encapsulate IP 12 draft-ietf-tsvwg-ecn-encap-guidelines-04 14 Abstract 16 The purpose of this document is to guide the design of congestion 17 notification in any lower layer or tunnelling protocol that 18 encapsulates IP. The aim is for explicit congestion signals to 19 propagate consistently from lower layer protocols into IP. Then the 20 IP internetwork layer can act as a portability layer to carry 21 congestion notification from non-IP-aware congested nodes up to the 22 transport layer (L4). Following these guidelines should assure 23 interworking between new lower layer congestion notification 24 mechanisms, whether specified by the IETF or other standards bodies. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on April 10, 2016. 43 Copyright Notice 45 Copyright (c) 2015 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 1.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 5 62 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 63 3. Guidelines in All Cases . . . . . . . . . . . . . . . . . . . 7 64 4. Modes of Operation . . . . . . . . . . . . . . . . . . . . . 7 65 4.1. Feed-Forward-and-Up Mode . . . . . . . . . . . . . . . . 8 66 4.2. Feed-Up-and-Forward Mode . . . . . . . . . . . . . . . . 10 67 4.3. Feed-Backward Mode . . . . . . . . . . . . . . . . . . . 10 68 4.4. Null Mode . . . . . . . . . . . . . . . . . . . . . . . . 12 69 5. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 70 Notification . . . . . . . . . . . . . . . . . . . . . . . . 12 71 5.1. IP-in-IP Tunnels with Tightly Coupled Shim Headers . . . 13 72 5.2. Wire Protocol Design: Indication of ECN Support . . . . . 14 73 5.3. Encapsulation Guidelines . . . . . . . . . . . . . . . . 15 74 5.4. Decapsulation Guidelines . . . . . . . . . . . . . . . . 17 75 5.5. Sequences of Similar Tunnels or Subnets . . . . . . . . . 18 76 5.6. Reframing and Congestion Markings . . . . . . . . . . . . 19 77 6. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 78 Notification . . . . . . . . . . . . . . . . . . . . . . . . 20 79 7. Feed-Backward Mode: Guidelines for Adding Congestion 80 Notification . . . . . . . . . . . . . . . . . . . . . . . . 21 81 8. IANA Considerations (to be removed by RFC Editor) . . . . . . 22 82 9. Security Considerations . . . . . . . . . . . . . . . . . . . 22 83 10. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 23 84 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 23 85 12. Comments Solicited . . . . . . . . . . . . . . . . . . . . . 23 86 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 24 87 13.1. Normative References . . . . . . . . . . . . . . . . . . 24 88 13.2. Informative References . . . . . . . . . . . . . . . . . 24 89 Appendix A. Outstanding Document Issues . . . . . . . . . . . . 28 90 Appendix B. Changes in This Version (to be removed by RFC 91 Editor) . . . . . . . . . . . . . . . . . . . . . . 28 92 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 30 94 1. Introduction 96 The benefits of Explicit Congestion Notification (ECN) described 97 below can only be fully realised if support for ECN is added to the 98 relevant subnetwork technology, as well as to IP. When a lower layer 99 buffer drops a packet obviously it does not just drop at that layer; 100 the packet disappears from all layers. In contrast, when a lower 101 layer marks a packet with ECN, the marking needs to be explicitly 102 propagated up the layers. The same is true if a buffer marks the 103 outer header of a packet that encapsulates inner tunnelled headers. 104 Forwarding ECN is not as straightforward as other headers because it 105 has to be assumed ECN may be only partially deployed. If an egress 106 at any layer is not ECN-aware, or if the ultimate receiver or sender 107 is not ECN-aware, congestion needs to be indicated by dropping a 108 packet, not marking it. 110 The purpose of this document is to guide the addition of congestion 111 notification to any subnet technology or tunnelling protocol, so that 112 lower layer equipment can signal congestion explicitly and it will 113 propagate consistently into encapsulated (higher layer) headers, 114 otherwise the signals will not reach their ultimate destination. 116 ECN is defined in the IP header (v4 and v6) [RFC3168] to allow a 117 resource to notify the onset of queue build-up without having to drop 118 packets, by explicitly marking a proportion of packets with the 119 congestion experienced (CE) codepoint. 121 Given a suitable marking scheme, ECN removes nearly all congestion 122 loss and it cuts delays for two main reasons: 124 o It avoids the delay when recovering from congestion losses, which 125 particularly benefits small flows or real-time flows, making their 126 delivery time predictably short [RFC2884]; 128 o As ECN is used more widely by end-systems, it will gradually 129 remove the need to configure a degree of delay into buffers before 130 they start to notify congestion (the cause of bufferbloat). This 131 is because drop involves a trade-off between sending a timely 132 signal and trying to avoid impairment, whereas ECN is solely a 133 signal not an impairment, so there is no harm triggering it 134 earlier. 136 Some lower layer technologies (e.g. MPLS, Ethernet) are used to form 137 subnetworks with IP-aware nodes only at the edges. These networks 138 are often sized so that it is rare for interior queues to overflow. 139 However, until recently this was more due to the inability of TCP to 140 saturate the links. For many years, fixes such as window scaling 141 [RFC1323] proved hard to deploy. And the New Reno variant of TCP has 142 remained in widespread use despite its inability to scale to high 143 flow rates. However, now that modern operating systems are finally 144 capable of saturating interior links, even the buffers of well- 145 provisioned interior switches will need to signal episodes of 146 queuing. 148 Propagation of ECN is defined for MPLS [RFC5129], and is being 149 defined for TRILL [I-D.ietf-trill-rfc7180bis], but it remains to be 150 defined for a number of other subnetwork technologies. 152 Similarly, ECN propagation is yet to be defined for many tunnelling 153 protocols. [RFC6040] defines how ECN should be propagated for IP-in- 154 IP [RFC2003] and IPsec [RFC4301] tunnels. However, as Section 9.3 of 155 RFC3168 pointed out, ECN support will need to be defined for other 156 tunnelling protocols, e.g. L2TP [RFC2661], GRE [RFC1701], [RFC2784], 157 PPTP [RFC2637] and GTP [GTPv1], [GTPv1-U], [GTPv2-C]. 159 Incremental deployment is the most delicate aspect when adding 160 support for ECN. The original ECN protocol in IP [RFC3168] was 161 carefully designed so that a congested buffer would not mark a packet 162 (rather than drop it) unless both source and destination hosts were 163 ECN-capable. Otherwise its congestion markings would never be 164 detected and congestion would just build up further. However, to 165 support congestion marking below the IP layer, it is not sufficient 166 to only check that the two end-points support ECN; correct operation 167 also depends on the decapsulator at each subnet egress faithfully 168 propagating congestion notifications to the higher layer. Otherwise, 169 a legacy decapsulator might silently fail to propagate any ECN 170 signals from the outer to the forwarded header. Then the lost 171 signals would never be detected and again congestion would build up 172 further. The guidelines given later require protocol designers to 173 carefully consider incremental deployment, and suggest various safe 174 approaches for different circumstances. 176 Of course, the IETF does not have standards authority over every link 177 layer protocol. So this document gives guidelines for designing 178 propagation of congestion notification across the interface between 179 IP and protocols that may encapsulate IP (i.e. that can be layered 180 beneath IP). Each lower layer technology will exhibit different 181 issues and compromises, so the IETF or the relevant standards body 182 must be free to define the specifics of each lower layer congestion 183 notification scheme. Nonetheless, if the guidelines are followed, 184 congestion notification should interwork between different 185 technologies, using IP in its role as a 'portability layer'. 187 Therefore, the capitalised term 'SHOULD' or 'SHOULD NOT' are often 188 used in preference to 'MUST' or 'MUST NOT', because it is difficult 189 to know the compromises that will be necessary in each protocol 190 design. If a particular protocol design chooses to contradict a 191 'SHOULD (NOT)' given in the advice below, it MUST include a sound 192 justification. 194 It has not been possible to give common guidelines for all lower 195 layer technologies, because they do not all fit a common pattern. 196 Instead they have been divided into a few distinct modes of 197 operation: feed-forward-and-upward; feed-upward-and-forward; feed- 198 backward; and null mode. These modes are described in Section 4, 199 then in the following sections separate guidelines are given for each 200 mode. 202 This document updates the advice to subnetwork designers about ECN in 203 Section 13 of [RFC3819]. 205 1.1. Scope 207 This document only concerns wire protocol processing of explicit 208 notification of congestion and makes no changes or recommendations 209 concerning algorithms for congestion marking or for congestion 210 response (algorithm issues should be independent of the layer the 211 algorithm operates in). 213 The question of congestion notification signals with different 214 semantics to those of ECN in IP is touched on in a couple of specific 215 cases (e.g. QCN [IEEE802.1Qau]) and with schemes with multiple 216 severity levels such as PCN [RFC6660]). However, no attempt is made 217 to give guidelines about schemes with different semantics that are 218 yet to be invented. 220 The semantics of congestion signals can be relative to the traffic 221 class. Therefore correct propagation of congestion signals could 222 depend on correct propagation of any traffic class field between the 223 layers. In this document, correct propagation of traffic class 224 information is assumed, while what 'correct' means and how it is 225 achieved is covered elsewhere (e.g. [RFC2983]) and is outside the 226 scope of the present document. 228 Note that these guidelines do not require the subnet wire protocol to 229 be changed to accommodate congestion notification. Another way to 230 add congestion notification without consuming header space in the 231 subnet protocol might be to use a parallel control plane protocol. 233 This document focuses on the congestion notification interface 234 between IP and lower layer protocols that can encapsulate IP, where 235 the term 'IP' includes v4 or v6, unicast, multicast or anycast. 236 However, it is likely that the guidelines will also be useful when a 237 lower layer protocol or tunnel encapsulates itself (e.g. Ethernet 238 MAC in MAC [IEEE802.1Qah]) or when it encapsulates other protocols. 239 In the feed-backward mode, propagation of congestion signals for 240 multicast and anycast packets is out-of-scope (because it would be so 241 complicated that it is hoped no-one would attempt such an 242 abomination). 244 2. Terminology 246 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 247 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 248 document are to be interpreted as described in RFC 2119 [RFC2119]. 250 Further terminology used within this document: 252 Protocol data unit (PDU): Information that is delivered as a unit 253 among peer entities of a layered network consisting of protocol 254 control information (typically a header) and possibly user data 255 (payload) of that layer. The scope of this document includes 256 layer 2 and layer 3 networks, where the PDU is respectively termed 257 a frame or a packet (or a cell in ATM). PDU is a general term for 258 any of these. This definition also includes a payload with a shim 259 header lying somewhere between layer 2 and 3. 261 Transport: The end-to-end transmission control function, 262 conventionally considered at layer-4 in the OSI reference model. 263 Given the audience for this document will often use the word 264 transport to mean low level bit carriage, whenever the term is 265 used it will be qualified, e.g. 'L4 transport'. 267 Encapsulator: The link or tunnel endpoint function that adds an 268 outer header to a PDU (also termed the 'link ingress', the 'subnet 269 ingress', the 'ingress tunnel endpoint' or just the 'ingress' 270 where the context is clear). 272 Decapsulator: The link or tunnel endpoint function that removes an 273 outer header from a PDU (also termed the 'link egress', the 274 'subnet egress', the 'egress tunnel endpoint' or just the 'egress' 275 where the context is clear). 277 Incoming header: The header of an arriving PDU before encapsulation. 279 Outer header: The header added to encapsulate a PDU. 281 Inner header: The header encapsulated by the outer header. 283 Outgoing header: The header forwarded by the decapsulator. 285 CE: Congestion Experienced [RFC3168] 286 ECT: ECN-Capable Transport [RFC3168] 288 Not-ECT: Not ECN-Capable Transport [RFC3168] 290 Load Regulator: For each flow of PDUs, the transport function that 291 is capable of controlling the data rate. Typically located at the 292 data source, but in-path nodes can regulate load in some 293 congestion control arrangements (e.g. admission control, policing 294 nodes or transport circuit-breakers 295 [I-D.ietf-tsvwg-circuit-breaker]). Note the term "a function 296 capable of controlling the load" deliberately includes a transport 297 that doesn't actually control the load responsively but ideally it 298 ought to (e.g. a sending application without congestion control 299 that uses UDP). 301 ECN-PDU: A PDU that is part of a feedback loop within which all the 302 nodes that need to propagate explicit congestion notifications 303 back to the Load Regulator are ECN-capable. An IP packet with a 304 non-zero ECN field implies that the endpoints are ECN-capable, so 305 this would be an ECN-PDU. However, ECN-PDU is intended to be a 306 general term for a PDU at any layer, not just IP. 308 Not-ECN-PDU: A PDU that is part of a feedback-loop within which some 309 nodes necessary to propagate explicit congestion notifications 310 back to the load regulator are not ECN-capable. 312 Congestion Baseline: The location of the function on the path that 313 initialised the values of all congestion notification fields in a 314 sequence of packets, before any are set to the congestion 315 experienced (CE) codepoint if they experience congestion further 316 downstream. Typically the original data source at layer-4. 318 3. Guidelines in All Cases 320 RFC 3168 specifies that the ECN field in the IP header is intended to 321 be marked by active queue management algorithms. Any congestion 322 notification from an algorithm that does not conform to the 323 recommendations in [I-D.ietf-aqm-recommendation] MUST NOT be 324 propagated from a lower layer into the ECN field in IP (see also 325 [RFC4774] on alternate uses of the ECN field). 327 4. Modes of Operation 329 This section sets down the different modes by which congestion 330 information is passed between the lower layer and the higher one. It 331 acts as a reference framework for the following sections, which give 332 normative guidelines for designers of explicit congestion 333 notification protocols, taking each mode in turn: 335 Feed-Forward-and-Up: Nodes feed forward congestion notification 336 towards the egress within the lower layer then up and along the 337 layers towards the end-to-end destination at the transport layer. 338 The following local optimisation is possible: 340 Feed-Up-and-Forward: A lower layer switch feeds-up congestion 341 notification directly into the ECN field in the higher layer 342 (e.g. IP) header, irrespective of whether the node is at the 343 egress of a subnet. 345 Feed-Backward: Nodes feed back congestion signals towards the 346 ingress of the lower layer and (optionally) attempt to control 347 congestion within their own layer. 349 Null: Nodes cannot experience congestion at the lower layer except 350 at ingress nodes (which are IP-aware or equivalently higher-layer- 351 aware). 353 4.1. Feed-Forward-and-Up Mode 355 Like IP and MPLS, many subnet technologies are based on self- 356 contained protocol data units (PDUs) or frames sent unreliably. They 357 provide no feedback channel at the subnetwork layer, instead relying 358 on higher layers (e.g. TCP) to feed back loss signals. 360 In these cases, ECN may best be supported by standardising explicit 361 notification of congestion into the lower layer protocol that carries 362 the data forwards. It will then also be necessary to define how the 363 egress of the lower layer subnet propagates this explicit signal into 364 the forwarded upper layer (IP) header. It can then continue forwards 365 until it finally reaches the destination transport (at L4). Then 366 typically the destination will feed this congestion notification back 367 to the source transport using an end-to-end protocol (e.g. TCP). 368 This is the arrangement that has already been used to add ECN to IP- 369 in-IP tunnels [RFC6040], IP-in-MPLS and MPLS-in-MPLS [RFC5129]. 371 This mode is illustrated in Figure 1. Along the middle of the 372 figure, layers 2, 3 and 4 of the protocol stack are shown, and one 373 packet is shown along the bottom as it progresses across the network 374 from source to destination, crossing two subnets connected by a 375 router, and crossing two switches on the path across each subnet. 376 Congestion at the output of the first switch (shown as *) leads to a 377 congestion marking in the L2 header (shown as C in the illustration 378 of the packet). The chevrons show the progress of the resulting 379 congestion indication. It is propagated from link to link across the 380 subnet in the L2 header, then when the router removes the marked L2 381 header, it propagates the marking up into the L3 (IP) header. The 382 router forwards the marked L3 header into subnet 2, and when it adds 383 a new L2 header it copies the L3 marking into the L2 header as well, 384 as shown by the 'C's in both layers (assuming the technology of 385 subnet 2 also supports explicit congestion marking). 387 Note that there is no implication that each 'C' marking is encoded 388 the same; a different encoding might be used for the 'C' marking in 389 each protocol. 391 Finally, for completeness, we show the L3 marking arriving at the 392 destination, where the host transport protocol (e.g. TCP) feeds it 393 back to the source in the L4 acknowledgement (the 'C' at L4 in the 394 packet at the top of the diagram). 396 _ _ _ 397 /_______ | | |C| ACK Packet (V) 398 \ |_|_|_| 399 +---+ layer: 2 3 4 header +---+ 400 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 401 | | +---+ | ^ | 402 | | . . . . . . Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3 403 | | +---+ +---+ | ^ | +---+ +---+ | | 404 | | | *|>>>>>|>>>|>>>>>|>^ | | | | | | |L2 405 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 406 source subnet A router subnet B dest 407 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ 408 | | | | | | | | |C| | | |C| | | |C|C| Data________\ 409 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) / 410 layer: 4 3 2A 4 3 2A 4 3 4 3 2B 411 header 413 Figure 1: Feed-Forward-and-Up Mode 415 Of course, modern networks are rarely as simple as this text-book 416 example, often involving multiple nested layers. For example, a 3GPP 417 mobile network may have two IP-in-IP (GTP) tunnels in series and an 418 MPLS backhaul between the base station and the first router. 419 Nonetheless, the example illustrates the general idea of feeding 420 congestion notification forward then upward whenever a header is 421 removed at the egress of a subnet. 423 Note that the FECN (forward ECN) bit in Frame Relay and the explicit 424 forward congestion indication (EFCI [ITU-T.I.371]) bit in ATM user 425 data cells follow a feed-forward pattern. However, in ATM, this 426 arrangement is only part of a feed-forward-and-backward pattern at 427 the lower layer, not feed-forward-and-up out of the lower layer--the 428 intention was never to interface to IP ECN at the subnet egress. To 429 our knowledge, Frame Relay FECN is solely used to detect where more 430 capacity should be provisioned [Buck00]. 432 4.2. Feed-Up-and-Forward Mode 434 Ethernet is particularly difficult to extend incrementally to support 435 explicit congestion notification. One way to support ECN in such 436 cases has been to use so called 'layer-3 switches'. These are 437 Ethernet switches that bury into the Ethernet payload to find an IP 438 header and manipulate or act on certain IP fields (specifically 439 Diffserv & ECN). For instance, in Data Center TCP [DCTCP], layer-3 440 switches are configured to mark the ECN field of the IP header within 441 the Ethernet payload when their output buffer becomes congested. 442 With respect to switching, a layer-3 switch acts solely on the 443 addresses in the Ethernet header; it doesn't use IP addresses, and it 444 doesn't decrement the TTL field in the IP header. 446 _ _ _ 447 /_______ | | |C| ACK packet (V) 448 \ |_|_|_| 449 +---+ layer: 2 3 4 header +---+ 450 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 451 | | +---+ | ^ | 452 | | . . . >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3 453 | | +--^+ +---+ | | +---+ +---+ | | 454 | | | *| | | | | | | | | | |L2 455 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 456 source subnet E router subnet F dest 457 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ 458 | | | | | | | |C| | | | |C| | | |C|C| data________\ 459 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / 460 layer: 4 3 2 4 3 2 4 3 4 3 2 461 header 463 Figure 2: Feed-Up-and-Forward Mode 465 By comparing Figure 2 with Figure 1, it can be seen that subnet E 466 (perhaps a subnet of layer-3 Ethernet switches) works in feed-up-and- 467 forward mode by notifying congestion directly into L3 at the point of 468 congestion, even though the congested switch does not otherwise act 469 at L3. In this example, the technology in subnet F (e.g. MPLS) does 470 support ECN natively, so when the router adds the layer-2 header it 471 copies the ECN marking from L3 to L2 as well. 473 4.3. Feed-Backward Mode 475 In some layer 2 technologies, explicit congestion notification has 476 been defined for use internally within the subnet with its own 477 feedback and load regulation, but typically the interface with IP for 478 ECN has not been defined. 480 For instance, for the available bit-rate (ABR) service in ATM, the 481 relative rate mechanism was one of the more popular mechanisms for 482 managing traffic, tending to supersede earlier designs. In this 483 approach ATM switches send special resource management (RM) cells in 484 both the forward and backward directions to control the ingress rate 485 of user data into a virtual circuit. If a switch buffer is 486 approaching congestion or is congested it sends an RM cell back 487 towards the ingress with respectively the No Increase (NI) or 488 Congestion Indication (CI) bit set in its message type field 489 [ATM-TM-ABR]. The ingress then holds or decreases its sending bit- 490 rate accordingly. 492 _ _ _ 493 /_______ | | |C| ACK packet (X) 494 \ |_|_|_| 495 +---+ layer: 2 3 4 header +---+ 496 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4 497 | | +---+ | ^ | 498 | | | *|>>> Packet W >>>>>>>>>>>>|>^ |L3 499 | | +---+ +---+ | | +---+ +---+ | | 500 | | | | | | | <|<<<<<|<<<|<(V)<|<<<| | |L2 501 | | . . | . |Packet U | . . | . | . . | . | . . | .*| . . | |L2 502 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 503 source subnet G router subnet H dest 504 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ later 505 | | | | | | | | | | | | | | | | |C| | data________\ 506 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (W) / 507 4 3 2 4 3 2 4 3 4 3 2 508 _ 509 /__ |C| Feedback control 510 \ |_| cell/frame (V) 511 2 512 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ earlier 513 | | | | | | | | | | | | | | | | | | | data________\ 514 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / 515 layer: 4 3 2 4 3 2 4 3 4 3 2 516 header 518 Figure 3: Feed-Backward Mode 520 ATM's feed-backward approach doesn't fit well when layered beneath 521 IP's feed-forward approach--unless the initial data source is the 522 same node as the ATM ingress. Figure 3 shows the feed-backward 523 approach being used in subnet H. If the final switch on the path is 524 congested (*), it doesn't feed-forward any congestion indications on 525 packet (U). Instead it sends a control cell (V) back to the router 526 at the ATM ingress. 528 However, the backward feedback doesn't reach the original data source 529 directly because IP doesn't support backward feedback (and subnet G 530 is independent of subnet H). Instead, the router in the middle 531 throttles down its sending rate but the original data sources don't 532 reduce their rates. The resulting rate mismatch causes the middle 533 router's buffer at layer 3 to back up until it becomes congested, 534 which it signals forwards on later data packets at layer 3 (e.g. 535 packet W). Note that the forward signal from the middle router is 536 not triggered directly by the backward signal. Rather, it is 537 triggered by congestion resulting from the middle router's mismatched 538 rate response to the backward signal. 540 In response to this later forward signalling, end-to-end feedback at 541 layer-4 finally completes the tortuous path of congestion indications 542 back to the origin data source, as before. 544 4.4. Null Mode 546 Often link and physical layer resources are 'non-blocking' by design. 547 In these cases congestion notification may be implemented but it does 548 not need to be deployed at the lower layer; ECN in IP would be 549 sufficient. 551 A degenerate example is a point-to-point Ethernet link. Excess 552 loading of the link merely causes the queue from the higher layer to 553 back up, while the lower layer remains immune to congestion. Even a 554 whole meshed subnetwork can be made immune to interior congestion by 555 limiting ingress capacity and sufficient sizing of interior links, 556 e.g. a non-blocking fat-tree network. An alternative to fat links 557 near the root is numerous thin links with multi-path routing to 558 ensure even worst-case patterns of load cannot congest any link, e.g. 559 a Clos network. 561 5. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 562 Notification 564 Feed-forward-and-up is the mode already used for signalling ECN up 565 the layers through MPLS into IP [RFC5129] and through IP-in-IP 566 tunnels [RFC6040]. These RFCs take a consistent approach and the 567 following guidelines are designed to ensure this consistency 568 continues as ECN support is added to other protocols that encapsulate 569 IP. The guidelines are also designed to ensure compliance with the 570 more general best current practice for the design of alternate ECN 571 schemes given in [RFC4774]. 573 The rest of this section is structured as follows: 575 o Section 5.1 addresses the most straightforward cases, where 576 [RFC6040] can be applied directly to add ECN to tunnels that are 577 effectively the same as IP-in-IP tunnels. 579 o The subsequent sections give guidelines for adding ECN to a subnet 580 technology that uses feed-forward-and-up mode like IP, but it is 581 not so similar to IP that [RFC6040] rules can be applied directly. 582 Specifically: 584 * Sections 5.2, 5.3 and 5.4 respectively address how to add ECN 585 support to the wire protocol and to the encapsulators and 586 decapsulators at the ingress and egress of the subnet. 588 * Section 5.5 deals with the special, but common, case of 589 sequences of tunnels or subnets that all use the same 590 technology 592 * Section 5.6 deals with the question of reframing when IP 593 packets do not map 1:1 into lower layer frames. 595 5.1. IP-in-IP Tunnels with Tightly Coupled Shim Headers 597 A common pattern for many tunnelling protocols is to encapsulate an 598 inner IP header with shim header(s) then an outer IP header. In many 599 cases the shim header(s) always have to be tightly coupled to the 600 outer IP header because they are not sufficient as outer headers in 601 their own right. In such cases the shim header(s) and the outer IP 602 header are always added (or removed) in the same operation. 603 Therefore, in all such tightly coupled IP-in-IP tunnelling protocols, 604 the rules in [RFC6040] for propagating the ECN field between the two 605 IP headers SHOULD be applied directly. 607 Examples of tightly coupled IP-in-IP tunnelling protocols where 608 [RFC6040] can be applied directly are: 610 o L2TP [RFC2661] 612 o GRE [RFC1701], [RFC2784] 614 o PPTP [RFC2637] 616 o GTP [GTPv1], [GTPv1-U], [GTPv2-C] 618 o VXLAN [RFC7348]. 620 5.2. Wire Protocol Design: Indication of ECN Support 622 This section is intended to guide the redesign of any lower layer 623 protocol that encapsulate IP to add native ECN support at the lower 624 layer. It reflects the approaches used in [RFC6040] and in 625 [RFC5129]. Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS 626 encapsulations that already comply with [RFC6040] or [RFC5129] will 627 already satisfy this guidance. 629 A lower layer (or subnet) congestion notification system: 631 1. SHOULD NOT apply explicit congestion notifications to PDUs that 632 are destined for legacy layer-4 transport implementations that 633 will not understand ECN, and 635 2. SHOULD NOT apply explicit congestion notifications to PDUs if the 636 egress of the subnet might not propagate congestion notifications 637 onward into the higher layer. 639 We use the term ECN-PDUs for a PDU on a feedback loop that will 640 propagate congestion notification properly because it meets both 641 the above criteria. And a Not-ECN-PDU is a PDU on a feedback 642 loop that does not meet both criteria, and will therefore not 643 propagate congestion notification properly. A corollary of the 644 above is that a lower layer congestion notification protocol: 646 3. SHOULD be able to distinguish ECN-PDUs from Not-ECN-PDUs. 648 Note that there is no need for all interior nodes within a subnet to 649 be able to mark congestion explicitly. A mix of ECN and drop signals 650 from different nodes is fine. However, if _any_ interior nodes might 651 generate ECN markings, guideline 2 above says that all relevant 652 egress node(s) SHOULD be able to propagate those markings up to the 653 higher layer. 655 In IP, if the ECN field in each PDU is cleared to the Not-ECT (not 656 ECN-capable transport) codepoint, it indicates that the L4 transport 657 will not understand congestion markings. A congested buffer must not 658 mark these Not-ECT PDUs, and therefore drops them instead. 660 The mechanism a lower layer uses to distinguish the ECN-capability of 661 PDUs need not mimic that of IP. All the above guidelines say is that 662 the lower layer system, as a whole, should achieve the same outcome. 663 For instance, ECN-capable feedback loops might use PDUs that are 664 identified by a particular set of labels or tags. Alternatively, 665 logical link protocols that use flow state might determine whether a 666 PDU can be congestion marked by checking for ECN-support in the flow 667 state. Other protocols might depend on out-of-band control signals. 669 The per-domain checking of ECN support in MPLS [RFC5129] is a good 670 example of a way to avoid sending congestion markings to transports 671 that will not understand them, without using any header space in the 672 subnet protocol. 674 In MPLS, header space is extremely limited, therefore RFC5129 does 675 not provide a field in the MPLS header to indicate whether the PDU is 676 an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are 677 allowed to set explicit congestion indications without checking 678 whether the PDU is destined for a transport that will understand 679 them. Nonetheless, this is made safe by requiring that the network 680 operator upgrades all decapsulating edges of a whole domain at once, 681 as soon as even one switch within the domain is configured to mark 682 rather than drop during congestion. Therefore, any edge node that 683 might decapsulate a packet will be capable of checking whether the 684 higher layer transport is ECN-capable. When decapsulating a CE- 685 marked packet, if the decapsulator discovers that the higher layer 686 (inner header) indicates the transport is not ECN-capable, it drops 687 the packet--effectively on behalf of the earlier congested node (see 688 Decapsulation Guideline 1 in Section 5.4). 690 It was only appropriate to define such an incremental deployment 691 strategy because MPLS is targeted solely at professional operators, 692 who can be expected to ensure that a whole subnetwork is consistently 693 configured. This strategy might not be appropriate for other link 694 technologies targeted at zero-configuration deployment or deployment 695 by the general public (e.g. Ethernet). For such 'plug-and-play' 696 environments it will be necessary to invent a failsafe approach that 697 ensures congestion markings will never fall into black holes, no 698 matter how inconsistently a system is put together. Alternatively, 699 congestion notification relying on correct system configuration could 700 be confined to flavours of Ethernet intended only for professional 701 network operators, such as IEEE 802.1ah Provider Backbone Bridges 702 (PBB). 704 QCN [IEEE802.1Qau] provides another example of how to indicate to 705 lower layer devices that the end-points will not understand ECN. An 706 operator can define certain 802.1p classes of service to indicate 707 non-QCN frames and an ingress bridge is required to map arriving not- 708 QCN-capable IP packets to one of these non-QCN 802.1p classes. 710 5.3. Encapsulation Guidelines 712 This section is intended to guide the redesign of any node that 713 encapsulates IP with a lower layer header when adding native ECN 714 support to the lower layer protocol. It reflects the approaches used 715 in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or IP-in- 716 MPLS or MPLS-in-MPLS encapsulations that already comply with 717 [RFC6040] or [RFC5129] will already satisfy this guidance. 719 1. Egress Capability Check: A subnet ingress needs to be sure that 720 the corresponding egress of a subnet will propagate any 721 congestion notification added to the outer header across the 722 subnet. This is necessary in addition to checking that an 723 incoming PDU indicates an ECN-capable (L4) transport. Examples 724 of how this guarantee might be provided include: 726 * by configuration (e.g. if any label switches in a domain 727 support ECN marking, [RFC5129] requires all egress nodes to 728 have been configured to propagate ECN) 730 * by the ingress explicitly checking that the egress propagates 731 ECN (e.g. TRILL uses IS-IS to check path capabilities before 732 using critical options [I-D.ietf-trill-rfc7180bis]) 734 * by inherent design of the protocol (e.g. by encoding ECN 735 marking on the outer header in such a way that a legacy egress 736 that does not understand ECN will consider the PDU corrupt and 737 discard it, thus at least propagating a form of congestion 738 signal). 740 2. Egress Fails Capability Check: If the ingress cannot guarantee 741 that the egress will propagate congestion notification, the 742 ingress SHOULD disable ECN when it forwards the PDU at the lower 743 layer. An example of how the ingress might disable ECN at the 744 lower layer would be by setting the outer header of the PDU to 745 identify it as a Not-ECN-PDU, assuming the subnet technology 746 supports such a concept. 748 3. Standard Congestion Monitoring Baseline: Once the ingress to a 749 subnet has established that the egress will correctly propagate 750 ECN, on encapsulation it SHOULD encode the same level of 751 congestion in outer headers as is arriving in incoming headers. 752 For example it might copy any incoming congestion notification 753 into the outer header of the lower layer protocol. 755 This ensures that all outer headers reflect congestion 756 accumulated along the whole upstream path since the Load 757 Regulator, not just since the ingress of the subnet. A node that 758 is not the Load Regulator SHOULD NOT re-initialise the level of 759 CE markings in the outer to zero. 761 This guideline is intended to ensure that any bulk congestion 762 monitoring of outer headers (e.g. by a network management node 763 monitoring ECN in passing frames) is most meaningful. For 764 instance, if an operator measures CE in 0.4% of passing outer 765 headers, this information is only useful if the operator knows 766 where the proportion of CE markings was last initialised to 0% 767 (the Congestion Baseline). Such monitoring information will not 768 be useful if some subnet ingress nodes reset all outer CE 769 markings while others copy incoming CE markings into the outer. 771 Most information can be extracted if the Congestion Baseline is 772 standardised at the node that is regulating the load (the Load 773 Regulator--typically the data source). Then the operator can 774 measure both congestion since the Load Regulator, and congestion 775 since the subnet ingress. The latter might be measurable by 776 subtracting the level of CE markings on inner headers from that 777 on outer headers (see Appendix C of [RFC6040]). 779 5.4. Decapsulation Guidelines 781 This section is intended to guide the redesign of any node that 782 decapsulates IP from within a lower layer header when adding native 783 ECN support to the lower layer protocol. It reflects the approaches 784 used in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or 785 IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with 786 [RFC6040] or [RFC5129] will already satisfy this guidance. 788 A subnet egress SHOULD NOT simply copy congestion notification from 789 outer headers to the forwarded header. It SHOULD calculate the 790 outgoing congestion notification field from the inner and outer 791 headers using the following guidelines. If there is any conflict, 792 rules earlier in the list take precedence over rules later in the 793 list: 795 1. If the arriving inner header is a Not-ECN-PDU it implies the L4 796 transport will not understand explicit congestion markings. 797 Then: 799 * If the outer header carries an explicit congestion marking, 800 the packet SHOULD be dropped--the only indication of 801 congestion that the L4 transport will understand. 803 * If the outer is an ECN-PDU that carries no indication of 804 congestion or a Not-ECN-PDU the PDU SHOULD be forwarded, but 805 still as a Not-ECN-PDU. 807 2. If the outer header does not support explicit congestion 808 notification (a Not-ECN-PDU), but the inner header does (an ECN- 809 PDU), the inner header SHOULD be forwarded unchanged. 811 3. In some lower layer protocols congestion may be signalled as a 812 numerical level, such as in the control frames of quantised 813 congestion notification [IEEE802.1Qau]. If such a multi-bit 814 encoding encapsulates an ECN-capable IP data packet, a function 815 will be needed to convert the quantised congestion level into the 816 frequency of congestion markings in outgoing IP packets. 818 4. Congestion indications may be encoded by a severity level. For 819 instance increasing levels of congestion might be encoded by 820 numerically increasing indications, e.g. pre-congestion 821 notification (PCN) can be encoded in each PDU at three severity 822 levels in IP or MPLS [RFC6660]. 824 If the arriving inner header is an ECN-PDU, where the inner and 825 outer headers carry indications of congestion of different 826 severity, the more severe indication SHOULD be forwarded in 827 preference to the less severe. 829 5. The inner and outer headers might carry a combination of 830 congestion notification fields that should not be possible given 831 any currently used protocol transitions. For instance, if 832 Encapsulation Guideline 3 in Section 5.3 had been followed, it 833 should not be possible to have a less severe indication of 834 congestion in the outer than in the inner. It MAY be appropriate 835 to log unexpected combinations of headers and possibly raise an 836 alarm. 838 If a safe outgoing codepoint can be defined for such a PDU, the 839 PDU SHOULD be forwarded rather than dropped. Some implementers 840 discard PDUs with currently unused combinations of headers just 841 in case they represent an attack. However, an approach using 842 alarms and policy-mediated drop is preferable to hard-coded drop, 843 so that operators can keep track of possible attacks but 844 currently unused combinations are not precluded from future use 845 through new standards actions. 847 5.5. Sequences of Similar Tunnels or Subnets 849 In some deployments, particularly in 3GPP networks, an IP packet may 850 traverse two or more IP-in-IP tunnels in sequence that all use 851 identical technology (e.g. GTP). 853 In such cases, it would be sufficient for every encapsulation and 854 decapsulation in the chain to comply with RFC 6040. Alternatively, 855 as an optimisation, a node that decapsulates a packet and immediately 856 re-encapsulates it for the next tunnel MAY copy the incoming outer 857 ECN field directly to the outgoing outer and the incoming inner ECN 858 field directly to the outgoing inner. Then the overall behavior 859 across the sequence of tunnel segments would still be consistent with 860 RFC 6040. 862 Appendix C of RFC6040 describes how a tunnel egress can monitor how 863 much congestion has been introduced within a tunnel. A network 864 operator might want to monitor how much congestion had been 865 introduced within a whole sequence of tunnels. Using the technique 866 in Appendix C of RFC6040 at the final egress, the operator could 867 monitor the whole sequence of tunnels, but only if the above 868 optimisation were used consistently along the sequence of tunnels, in 869 order to make it appear as a single tunnel. Therefore, tunnel 870 endpoint implementations SHOULD allow the operator to configure 871 whether this optimisation is enabled. 873 When ECN support is added to a subnet technology, consideration 874 SHOULD be given to a similar optimisation between subnets in sequence 875 if they all use the same technology. 877 5.6. Reframing and Congestion Markings 879 The guidance in this section is worded in terms of framing 880 boundaries, but it applies equally whether the protocol data units 881 are frames, cells or packets. 883 Where framing boundaries are different between two layers, congestion 884 indications SHOULD be propagated on the basis that a congestion 885 indication on a PDU applies to all the octets in the PDU. On 886 average, an encapsulator or decapsulator SHOULD approximately 887 preserve the number of marked octets arriving and leaving (counting 888 the size of inner headers, but not added encapsulating headers). 890 The next departing frame SHOULD be immediately marked even if only 891 enough incoming marked octets have arrived for part of the departing 892 frame. This ensures that any outstanding congestion marked octets 893 are propagated immediately, rather than held back waiting for a frame 894 no bigger than the outstanding marked octets--which might involve a 895 long wait. 897 For instance, an algorithm for marking departing frames could 898 maintain a counter representing the balance of arriving marked octets 899 minus departing marked octets. It adds the size of every marked 900 frame that arrives and if the counter is positive it marks the next 901 frame to depart and subtracts its size from the counter. This will 902 often leave a negative remainder in the counter, which is deliberate. 904 6. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 905 Notification 907 The guidance in this section is applicable when IP packets: 909 o are encapsulated in Ethernet headers; 911 o are forwarded by the eNode-B (base station) of a 3GPP radio access 912 network, which is required to apply ECN marking during congestion 913 [LTE-RA]. 915 This guidance also generalises to encapsulation by other subnet 916 technologies with no native support for explicit congestion 917 notification at the lower layer, but with support for finding and 918 processing an IP header. It is unlikely to be applicable or 919 necessary for IP-in-IP encapsulation, where feed-forward-and-up mode 920 based on [RFC6040] would be more appropriate. 922 Marking the IP header while switching at layer-2 (by using a layer-3 923 switch) or while forwarding in a radio access network seems to 924 represent a layering violation. However, it can be considered as a 925 benign optimisation if the guidelines below are followed. Feed-up- 926 and-forward is certainly not a general alternative to implementing 927 feed-forward congestion notification in the lower layer, because: 929 o IPv4 and IPv6 are not the only layer-3 protocols that might be 930 encapsulated by lower layer protocols 932 o Link-layer encryption might be in use, making the layer-2 payload 933 inaccessible 935 o Many Ethernet switches do not have 'layer-3 switch' capabilities 936 so they cannot read or modify an IP payload 938 o It might be costly to find an IP header (v4 or v6) when it may be 939 encapsulated by more than one lower layer header, e.g. Ethernet 940 MAC in MAC [IEEE802.1Qah]. 942 Nonetheless, configuring lower layer equipment to look for an ECN 943 field in an encapsulated IP header is a useful optimisation. If the 944 implementation follows the guidelines below, this optimisation does 945 not have to be confined to a controlled environment such as within a 946 data centre; it could usefully be applied on any network--even if the 947 operator is not sure whether the above issues will never apply: 949 1. If a native lower-layer congestion notification mechanism exists 950 for a subnet technology, it is safe to mix feed-up-and-forward 951 with feed-forward-and-up on other switches in the same subnet. 953 However, it will generally be more efficient to use the native 954 mechanism. 956 2. The depth of the search for an IP header SHOULD be limited. If 957 an IP header is not found soon enough, or an unrecognised or 958 unreadable header is encountered, the switch SHOULD resort to an 959 alternative means of signalling congestion (e.g. drop, or the 960 native lower layer mechanism if available). 962 3. It is sufficient to use the first IP header found in the stack; 963 the egress of the relevant tunnel can propagate congestion 964 notification upwards to any more deeply encapsulated IP headers 965 later. 967 7. Feed-Backward Mode: Guidelines for Adding Congestion Notification 969 It can be seen from Section 4.3 that congestion notification in a 970 subnet using feed-backward mode has generally not been designed to be 971 directly coupled with IP layer congestion notification. The subnet 972 attempts to minimise congestion internally, and if the incoming load 973 at the ingress exceeds the capacity somewhere through the subnet, the 974 layer 3 buffer into the ingress backs up. Thus, a feed-backward mode 975 subnet is in some sense similar to a null mode subnet, in that there 976 is no need for any direct interaction between the subnet and higher 977 layer congestion notification. Therefore no detailed protocol design 978 guidelines are appropriate. Nonetheless, a more general guideline is 979 appropriate: 981 A subnetwork technology intended to eventually interface to IP 982 SHOULD NOT be designed using only the feed-backward mode, which is 983 certainly best for a stand-alone subnet, but would need to be 984 modified to work efficiently as part of the wider Internet, 985 because IP uses feed-forward-and-up mode. 987 The feed-backward approach at least works beneath IP, where the term 988 'works' is used only in a narrow functional sense because feed- 989 backward can result in very inefficient and sluggish congestion 990 control--except if it is confined to the subnet directly connected to 991 the original data source, when it is faster than feed-forward. It 992 would be valid to design a protocol that could work in feed-backward 993 mode for paths that only cross one subnet, and in feed-forward-and-up 994 mode for paths that cross subnets. 996 In the early days of TCP/IP, a similar feed-backward approach was 997 tried for explicit congestion signalling, using source-quench (SQ) 998 ICMP control packets. However, SQ fell out of favour and is now 999 formally deprecated [RFC6633]. The main problem was that it is hard 1000 for a data source to tell the difference between a spoofed SQ message 1001 and a quench request from a genuine buffer on the path. It is also 1002 hard for a lower layer buffer to address an SQ message to the 1003 original source port number, which may be buried within many layers 1004 of headers, and possibly encrypted. 1006 Quantised congestion notification (QCN--also known as backward 1007 congestion notification or BCN) [IEEE802.1Qau] uses a feed-backward 1008 mode structurally similar to ATM's relative rate mechanism. However, 1009 QCN confines its applicability to scenarios such as some data centres 1010 where all endpoints are directly attached by the same Ethernet 1011 technology. If a QCN subnet were later connected into a wider IP- 1012 based internetwork (e.g. when attempting to interconnect multiple 1013 data centres) it would suffer the inefficiency shown Figure 3. 1015 8. IANA Considerations (to be removed by RFC Editor) 1017 This memo includes no request to IANA. 1019 9. Security Considerations 1021 If a lower layer wire protocol is redesigned to include explicit 1022 congestion signalling in-band in the protocol header, care SHOULD be 1023 take to ensure that the field used is specified as mutable during 1024 transit. Otherwise interior nodes signalling congestion would 1025 invalidate any authentication protocol applied to the lower layer 1026 header--by altering a header field that had been assumed as 1027 immutable. 1029 The redesign of protocols that encapsulate IP in order to propagate 1030 congestion signals between layers raises potential signal integrity 1031 concerns. Experimental or proposed approaches exist for assuring the 1032 end-to-end integrity of in-band congestion signals, e.g.: 1034 o Congestion exposure (ConEx ) for networks to audit that their 1035 congestion signals are not being suppressed by other networks or 1036 by receivers, and for networks to police that senders are 1037 responding sufficiently to the signals, irrespective of the 1038 transport protocol used [I-D.ietf-conex-abstract-mech]. 1040 o The ECN nonce [RFC3540] for a TCP sender to detect whether a 1041 network or the receiver is suppressing congestion signals. 1043 o A test with the same goals as the ECN nonce, but without the need 1044 for the receiver to co-operate with the protocol 1045 [I-D.moncaster-tcpm-rcv-cheat]. 1047 Given these end-to-end approaches are already being specified, it 1048 would make little sense to attempt to design hop-by-hop congestion 1049 signal integrity into a new lower layer protocol, because end-to-end 1050 integrity inherently achieves hop-by-hop integrity. 1052 10. Conclusions 1054 Following the guidance in the document enables ECN support to be 1055 extended to numerous protocols that encapsulate IP (v4 & v6) in a 1056 consistent way, so that IP continues to fulfil its role as an end-to- 1057 end interoperability layer. This includes: 1059 o A wide range of tunnelling protocols with various forms of shim 1060 header between two IP headers; 1062 o A wide range of subnet technologies, particularly those that work 1063 in the same 'feed-forward-and-up' mode that is used to support ECN 1064 in IP and MPLS. 1066 Guidelines have been defined for supporting propagation of ECN 1067 between Ethernet and IP on so-called Layer-3 Ethernet switches, using 1068 a 'feed-up-an-forward' mode. This approach could enable other subnet 1069 technologies to pass ECN signals into the IP layer, even if they do 1070 not support ECN natively. 1072 Finally, attempting to add ECN to a subnet technology in feed- 1073 backward mode is deprecated except in special cases, due to its 1074 likely sluggish response to congestion. 1076 11. Acknowledgements 1078 Thanks to Gorry Fairhurst for extensive reviews. Thanks also to the 1079 following reviewers: Richard Scheffenegger, Ingemar Johansson, Piers 1080 O'Hanlon and Michael Welzl, who pointed out that lower layer 1081 congestion notification signals may have different semantics to those 1082 in IP. 1084 Bob Briscoe was part-funded by the European Community under its 1085 Seventh Framework Programme through the Trilogy project (ICT-216372) 1086 for initial drafts and through the Reducing Internet Transport 1087 Latency (RITE) project (ICT-317700) subsequently. The views 1088 expressed here are solely those of the authors. 1090 12. Comments Solicited 1092 Comments and questions are encouraged and very welcome. They can be 1093 addressed to the IETF Transport Area working group mailing list 1094 , and/or to the authors. 1096 13. References 1098 13.1. Normative References 1100 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1101 Requirement Levels", BCP 14, RFC 2119, 1102 DOI 10.17487/RFC2119, March 1997, 1103 . 1105 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1106 of Explicit Congestion Notification (ECN) to IP", 1107 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1108 . 1110 [RFC3819] Karn, P., Ed., Bormann, C., Fairhurst, G., Grossman, D., 1111 Ludwig, R., Mahdavi, J., Montenegro, G., Touch, J., and L. 1112 Wood, "Advice for Internet Subnetwork Designers", BCP 89, 1113 RFC 3819, DOI 10.17487/RFC3819, July 2004, 1114 . 1116 [RFC4774] Floyd, S., "Specifying Alternate Semantics for the 1117 Explicit Congestion Notification (ECN) Field", BCP 124, 1118 RFC 4774, DOI 10.17487/RFC4774, November 2006, 1119 . 1121 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 1122 Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 1123 2008, . 1125 [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion 1126 Notification", RFC 6040, DOI 10.17487/RFC6040, November 1127 2010, . 1129 13.2. Informative References 1131 [ATM-TM-ABR] 1132 Cisco, "Understanding the Available Bit Rate (ABR) Service 1133 Category for ATM VCs", Design Technote 10415, June 2005. 1135 [Buck00] Buckwalter, J., "Frame Relay: Technology and Practice", 1136 Pub. Addison Wesley ISBN-13: 978-0201485240, 2000. 1138 [DCTCP] Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel, 1139 P., Prabhakar, B., Sengupta, S., and M. Sridharan, "Data 1140 Center TCP (DCTCP)", ACM SIGCOMM CCR 40(4)63--74, October 1141 2010, . 1143 [GTPv1] 3GPP, "GPRS Tunnelling Protocol (GTP) across the Gn and Gp 1144 interface", Technical Specification TS 29.060. 1146 [GTPv1-U] 3GPP, "General Packet Radio System (GPRS) Tunnelling 1147 Protocol User Plane (GTPv1-U)", Technical Specification TS 1148 29.281. 1150 [GTPv2-C] 3GPP, "Evolved General Packet Radio Service (GPRS) 1151 Tunnelling Protocol for Control plane (GTPv2-C)", 1152 Technical Specification TS 29.274. 1154 [I-D.ietf-aqm-recommendation] 1155 Baker, F. and G. Fairhurst, "IETF Recommendations 1156 Regarding Active Queue Management", draft-ietf-aqm- 1157 recommendation-11 (work in progress), February 2015. 1159 [I-D.ietf-conex-abstract-mech] 1160 Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) 1161 Concepts, Abstract Mechanism and Requirements", draft- 1162 ietf-conex-abstract-mech-13 (work in progress), October 1163 2014. 1165 [I-D.ietf-trill-rfc7180bis] 1166 Eastlake, D., Zhang, M., Perlman, R., Banerjee, A., 1167 Ghanwani, A., and S. Gupta, "TRILL: Clarifications, 1168 Corrections, and Updates", draft-ietf-trill-rfc7180bis-06 1169 (work in progress), October 2015. 1171 [I-D.ietf-tsvwg-circuit-breaker] 1172 Fairhurst, G., "Network Transport Circuit Breakers", 1173 draft-ietf-tsvwg-circuit-breaker-05 (work in progress), 1174 October 2015. 1176 [I-D.moncaster-tcpm-rcv-cheat] 1177 Moncaster, T., Briscoe, B., and A. Jacquet, "A TCP Test to 1178 Allow Senders to Identify Receiver Non-Compliance", draft- 1179 moncaster-tcpm-rcv-cheat-03 (work in progress), July 2014. 1181 [IEEE802.1Qah] 1182 IEEE, "IEEE Standard for Local and Metropolitan Area 1183 Networks--Virtual Bridged Local Area Networks--Amendment 1184 6: Provider Backbone Bridges", IEEE Std 802.1Qah-2008, 1185 August 2008, 1186 . 1188 (Access Controlled link within page) 1190 [IEEE802.1Qau] 1191 Finn, N., Ed., "IEEE Standard for Local and Metropolitan 1192 Area Networks--Virtual Bridged Local Area Networks - 1193 Amendment 13: Congestion Notification", IEEE Std 802.1Qau- 1194 2010, March 2010, . 1197 (Access Controlled link within page) 1199 [ITU-T.I.371] 1200 ITU-T, "Traffic Control and Congestion Control in B-ISDN", 1201 ITU-T Rec. I.371 (03/04), March 2004, 1202 . 1205 [LTE-RA] 3GPP, "Evolved Universal Terrestrial Radio Access (E-UTRA) 1206 and Evolved Universal Terrestrial Radio Access Network 1207 (E-UTRAN); Overall description; Stage 2", Technical 1208 Specification TS 36.300. 1210 [RFC1323] Jacobson, V., Braden, R., and D. Borman, "TCP Extensions 1211 for High Performance", RFC 1323, DOI 10.17487/RFC1323, May 1212 1992, . 1214 [RFC1701] Hanks, S., Li, T., Farinacci, D., and P. Traina, "Generic 1215 Routing Encapsulation (GRE)", RFC 1701, 1216 DOI 10.17487/RFC1701, October 1994, 1217 . 1219 [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, 1220 DOI 10.17487/RFC2003, October 1996, 1221 . 1223 [RFC2637] Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little, 1224 W., and G. Zorn, "Point-to-Point Tunneling Protocol 1225 (PPTP)", RFC 2637, DOI 10.17487/RFC2637, July 1999, 1226 . 1228 [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, 1229 G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", 1230 RFC 2661, DOI 10.17487/RFC2661, August 1999, 1231 . 1233 [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. 1234 Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, 1235 DOI 10.17487/RFC2784, March 2000, 1236 . 1238 [RFC2884] Hadi Salim, J. and U. Ahmed, "Performance Evaluation of 1239 Explicit Congestion Notification (ECN) in IP Networks", 1240 RFC 2884, DOI 10.17487/RFC2884, July 2000, 1241 . 1243 [RFC2983] Black, D., "Differentiated Services and Tunnels", 1244 RFC 2983, DOI 10.17487/RFC2983, October 2000, 1245 . 1247 [RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit 1248 Congestion Notification (ECN) Signaling with Nonces", 1249 RFC 3540, DOI 10.17487/RFC3540, June 2003, 1250 . 1252 [RFC4301] Kent, S. and K. Seo, "Security Architecture for the 1253 Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, 1254 December 2005, . 1256 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 1257 RFC 6633, DOI 10.17487/RFC6633, May 2012, 1258 . 1260 [RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three 1261 Pre-Congestion Notification (PCN) States in the IP Header 1262 Using a Single Diffserv Codepoint (DSCP)", RFC 6660, 1263 DOI 10.17487/RFC6660, July 2012, 1264 . 1266 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 1267 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 1268 eXtensible Local Area Network (VXLAN): A Framework for 1269 Overlaying Virtualized Layer 2 Networks over Layer 3 1270 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, 1271 . 1273 Appendix A. Outstanding Document Issues 1275 1. [GF] Concern that certain guidelines warrant a MUST (NOT) rather 1276 than a SHOULD (NOT). Given the guidelines say that if any SHOULD 1277 (NOT)s are not followed, a strong justification will be needed, 1278 they have been left as SHOULD (NOT) pending further list 1279 discussion. In particular: 1281 * If inner is a Not-ECN-PDU and Outer is CE (or highest severity 1282 congestion level), MUST (not SHOULD) drop? 1284 2. Consider whether an IETF Standard Track doc will be needed to 1285 Update the IP-in-IP protocols listed in Section 5.1--at least 1286 those that the IETF controls--and which Area it should sit under. 1288 Appendix B. Changes in This Version (to be removed by RFC Editor) 1290 From ietf-03 to ietf-04: 1292 * Addressed Richard Scheffenegger's review comments: primarily 1293 editorial corrections, and addition of examples for clarity. 1295 From ietf-02 to ietf-03: 1297 * Updated references, ad cited RFC4774. 1299 From ietf-01 to ietf-02: 1301 * Added Section for guidelines that are applicable in all cases. 1303 * Updated references. 1305 From ietf-00 to ietf-01: Updated references. 1307 From briscoe-04 to ietf-00: Changed filename following tsvwg 1308 adoption. 1310 From briscoe-03 to 04: 1312 * Re-arranged the introduction to describe the purpose of the 1313 document first before introducing ECN in more depth. And 1314 clarified the introduction throughout. 1316 * Added applicability to 3GPP TS 36.300. 1318 From briscoe-02 to 03: 1320 * Scope section: 1322 + Added dependence on correct propagation of traffic class 1323 information 1325 + For the feed-backward mode, deemed multicast and anycast out 1326 of scope 1328 * Ensured all guidelines referring to subnet technologies also 1329 refer to tunnels and vice versa by adding applicability 1330 sentences at the start of sections 4.1, 4.2, 4.3, 4.4, 4.6 and 1331 5. 1333 * Added Security Considerations on ensuring congestion signal 1334 fields are classed as immutable and on using end-to-end 1335 congestion signal integrity technologies rather than hop-by- 1336 hop. 1338 From briscoe-01 to 02: 1340 * Added authors: JK & PT 1342 * Added 1344 + Section 4.1 "IP-in-IP Tunnels with Tightly Coupled Shim 1345 Headers" 1347 + Section 4.5 "Sequences of Similar Tunnels or Subnets" 1349 + roadmap at the start of Section 4, given the subsections 1350 have become quite fragmented. 1352 + Section 9 "Conclusions" 1354 * Clarified why transports are starting to be able to saturate 1355 interior links 1357 * Under Section 1.1, addressed the question of alternative signal 1358 semantics and included multicast & anycast. 1360 * Under Section 3.1, included a 3GPP example. 1362 * Section 4.2. "Wire Protocol Design": 1364 + Altered guideline 2. to make it clear that it only applies 1365 to the immediate subnet egress, not later ones 1367 + Added a reminder that it is only necessary to check that ECN 1368 propagates at the egress, not whether interior nodes mark 1369 ECN 1371 + Added example of how QCN uses 802.1p to indicate support for 1372 QCN. 1374 * Added references to Appendix C of RFC6040, about monitoring the 1375 amount of congestion signals introduced within a tunnel 1377 * Appendix A: Added more issues to be addressed, including plan 1378 to produce a standards track update to IP-in-IP tunnel 1379 protocols. 1381 * Updated acks and references 1383 From briscoe-00 to 01: 1385 * Intended status: BCP (was Informational) & updates 3819 added. 1387 * Briefer Introduction: Introductory para justifying benefits of 1388 ECN. Moved all but a brief enumeration of modes of operation 1389 to their own new section (from both Intro & Scope). Introduced 1390 incr. deployment as most tricky part. 1392 * Tightened & added to terminology section 1394 * Structured with Modes of Operation, then Guidelines section for 1395 each mode. 1397 * Tightened up guideline text to remove vagueness / passive voice 1398 / ambiguity and highlight main guidelines as numbered items. 1400 * Added Outstanding Document Issues Appendix 1402 * Updated references 1404 Authors' Addresses 1406 Bob Briscoe 1407 Simula Research Laboratory 1408 UK 1410 EMail: ietf@bobbriscoe.net 1411 URI: http://bobbriscoe.net/ 1412 John Kaippallimalil 1413 Huawei 1414 5340 Legacy Drive, Suite 175 1415 Plano, Texas 75024 1416 USA 1418 EMail: john.kaippallimalil@huawei.com 1420 Pat Thaler 1421 Broadcom Corporation 1422 5025 Keane Drive 1423 Carmichael, CA 95608 1424 USA 1426 EMail: pthaler@broadcom.com