idnits 2.17.1 draft-ietf-tsvwg-ecn-encap-guidelines-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- -- The draft header indicates that this document updates RFC3819, but the abstract doesn't seem to mention this, which it should. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year (Using the creation date from RFC3819, updated by this document, for RFC5378 checks: 1999-10-14) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (July 20, 2017) is 2466 days in the past. Is this intentional? Checking references for intended status: Best Current Practice ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'GF' is mentioned on line 1386, but not defined == Outdated reference: A later version (-16) exists of draft-ietf-nvo3-geneve-04 == Outdated reference: A later version (-07) exists of draft-ietf-trill-ecn-support-03 == Outdated reference: A later version (-23) exists of draft-ietf-tsvwg-rfc6040update-shim-04 -- Obsolete informational reference (is this intentional?): RFC 1323 (Obsoleted by RFC 7323) -- Obsolete informational reference (is this intentional?): RFC 6830 (Obsoleted by RFC 9300, RFC 9301) Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Transport Area Working Group B. Briscoe 3 Internet-Draft Simula Research Laboratory 4 Updates: 3819 (if approved) J. Kaippallimalil 5 Intended status: Best Current Practice Huawei 6 Expires: January 21, 2018 P. Thaler 7 Broadcom Corporation 8 July 20, 2017 10 Guidelines for Adding Congestion Notification to Protocols that 11 Encapsulate IP 12 draft-ietf-tsvwg-ecn-encap-guidelines-09 14 Abstract 16 The purpose of this document is to guide the design of congestion 17 notification in any lower layer or tunnelling protocol that 18 encapsulates IP. The aim is for explicit congestion signals to 19 propagate consistently from lower layer protocols into IP. Then the 20 IP internetwork layer can act as a portability layer to carry 21 congestion notification from non-IP-aware congested nodes up to the 22 transport layer (L4). Following these guidelines should assure 23 interworking between new lower layer congestion notification 24 mechanisms, whether specified by the IETF or other standards bodies. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on January 21, 2018. 43 Copyright Notice 45 Copyright (c) 2017 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 1.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 5 62 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 63 3. Guidelines in All Cases . . . . . . . . . . . . . . . . . . . 7 64 4. Modes of Operation . . . . . . . . . . . . . . . . . . . . . 8 65 4.1. Feed-Forward-and-Up Mode . . . . . . . . . . . . . . . . 8 66 4.2. Feed-Up-and-Forward Mode . . . . . . . . . . . . . . . . 10 67 4.3. Feed-Backward Mode . . . . . . . . . . . . . . . . . . . 11 68 4.4. Null Mode . . . . . . . . . . . . . . . . . . . . . . . . 13 69 5. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 70 Notification . . . . . . . . . . . . . . . . . . . . . . . . 13 71 5.1. IP-in-IP Tunnels with Shim Headers . . . . . . . . . . . 14 72 5.2. Wire Protocol Design: Indication of ECN Support . . . . . 15 73 5.3. Encapsulation Guidelines . . . . . . . . . . . . . . . . 17 74 5.4. Decapsulation Guidelines . . . . . . . . . . . . . . . . 19 75 5.5. Sequences of Similar Tunnels or Subnets . . . . . . . . . 20 76 5.6. Reframing and Congestion Markings . . . . . . . . . . . . 21 77 6. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 78 Notification . . . . . . . . . . . . . . . . . . . . . . . . 21 79 7. Feed-Backward Mode: Guidelines for Adding Congestion 80 Notification . . . . . . . . . . . . . . . . . . . . . . . . 23 81 8. IANA Considerations (to be removed by RFC Editor) . . . . . . 24 82 9. Security Considerations . . . . . . . . . . . . . . . . . . . 24 83 10. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 24 84 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 25 85 12. Comments Solicited . . . . . . . . . . . . . . . . . . . . . 25 86 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 25 87 13.1. Normative References . . . . . . . . . . . . . . . . . . 25 88 13.2. Informative References . . . . . . . . . . . . . . . . . 26 89 Appendix A. Outstanding Document Issues . . . . . . . . . . . . 31 90 Appendix B. Changes in This Version (to be removed by RFC 91 Editor) . . . . . . . . . . . . . . . . . . . . . . 31 92 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35 94 1. Introduction 96 The benefits of Explicit Congestion Notification (ECN) described 97 below can only be fully realised if support for ECN is added to the 98 relevant subnetwork technology, as well as to IP. When a lower layer 99 buffer drops a packet obviously it does not just drop at that layer; 100 the packet disappears from all layers. In contrast, when a lower 101 layer marks a packet with ECN, the marking needs to be explicitly 102 propagated up the layers. The same is true if a buffer marks the 103 outer header of a packet that encapsulates inner tunnelled headers. 104 Forwarding ECN is not as straightforward as other headers because it 105 has to be assumed ECN may be only partially deployed. If an egress 106 at any layer is not ECN-aware, or if the ultimate receiver or sender 107 is not ECN-aware, congestion needs to be indicated by dropping a 108 packet, not marking it. 110 The purpose of this document is to guide the addition of congestion 111 notification to any subnet technology or tunnelling protocol, so that 112 lower layer equipment can signal congestion explicitly and it will 113 propagate consistently into encapsulated (higher layer) headers, 114 otherwise the signals will not reach their ultimate destination. 116 ECN is defined in the IP header (v4 and v6) [RFC3168] to allow a 117 resource to notify the onset of queue build-up without having to drop 118 packets, by explicitly marking a proportion of packets with the 119 congestion experienced (CE) codepoint. 121 Given a suitable marking scheme, ECN removes nearly all congestion 122 loss and it cuts delays for two main reasons: 124 o It avoids the delay when recovering from congestion losses, which 125 particularly benefits small flows or real-time flows, making their 126 delivery time predictably short [RFC2884]; 128 o As ECN is used more widely by end-systems, it will gradually 129 remove the need to configure a degree of delay into buffers before 130 they start to notify congestion (the cause of bufferbloat). This 131 is because drop involves a trade-off between sending a timely 132 signal and trying to avoid impairment, whereas ECN is solely a 133 signal not an impairment, so there is no harm triggering it 134 earlier. 136 Some lower layer technologies (e.g. MPLS, Ethernet) are used to form 137 subnetworks with IP-aware nodes only at the edges. These networks 138 are often sized so that it is rare for interior queues to overflow. 139 However, until recently this was more due to the inability of TCP to 140 saturate the links. For many years, fixes such as window scaling 141 [RFC1323] proved hard to deploy. And the Reno variant of TCP has 142 remained in widespread use despite its inability to scale to high 143 flow rates. However, now that modern operating systems are finally 144 capable of saturating interior links, even the buffers of well- 145 provisioned interior switches will need to signal episodes of 146 queuing. 148 Propagation of ECN is defined for MPLS [RFC5129], and is being 149 defined for TRILL [RFC7780], [I-D.ietf-trill-ecn-support], but it 150 remains to be defined for a number of other subnetwork technologies. 152 Similarly, ECN propagation is yet to be defined for many tunnelling 153 protocols. [RFC6040] defines how ECN should be propagated for IP-in- 154 IPv4 [RFC2003], IP-in-IPv6 [RFC2473] and IPsec [RFC4301] tunnels, but 155 there are numerous other tunnelling protocols with a shim and/or a 156 layer 2 header between two IP headers (v4 or v6). Some address ECN 157 propagation between the IP headers, but many do not. This document 158 gives guidance on how to address ECN propagation for future 159 tunnelling protocols, and a companion standards track specification 160 [I-D.ietf-tsvwg-rfc6040update-shim] updates those existing IP-shim- 161 (L2)-IP protocols that are under IETF change control and still widely 162 used. 164 Incremental deployment is the most delicate aspect when adding 165 support for ECN. The original ECN protocol in IP [RFC3168] was 166 carefully designed so that a congested buffer would not mark a packet 167 (rather than drop it) unless both source and destination hosts were 168 ECN-capable. Otherwise its congestion markings would never be 169 detected and congestion would just build up further. However, to 170 support congestion marking below the IP layer, it is not sufficient 171 to only check that the two transport layer end-points support ECN; 172 correct operation also depends on the decapsulator at each subnet 173 egress faithfully propagating congestion notifications to the higher 174 layer. Otherwise, a legacy decapsulator might silently fail to 175 propagate any ECN signals from the outer to the forwarded header. 176 Then the lost signals would never be detected and again congestion 177 would build up further. The guidelines given later require protocol 178 designers to carefully consider incremental deployment, and suggest 179 various safe approaches for different circumstances. 181 Of course, the IETF does not have standards authority over every link 182 layer protocol. So this document gives guidelines for designing 183 propagation of congestion notification across the interface between 184 IP and protocols that may encapsulate IP (i.e. that can be layered 185 beneath IP). Each lower layer technology will exhibit different 186 issues and compromises, so the IETF or the relevant standards body 187 must be free to define the specifics of each lower layer congestion 188 notification scheme. Nonetheless, if the guidelines are followed, 189 congestion notification should interwork between different 190 technologies, using IP in its role as a 'portability layer'. 192 Therefore, the capitalised term 'SHOULD' or 'SHOULD NOT' are often 193 used in preference to 'MUST' or 'MUST NOT', because it is difficult 194 to know the compromises that will be necessary in each protocol 195 design. If a particular protocol design chooses to contradict a 196 'SHOULD (NOT)' given in the advice below, it MUST include a sound 197 justification. 199 It has not been possible to give common guidelines for all lower 200 layer technologies, because they do not all fit a common pattern. 201 Instead they have been divided into a few distinct modes of 202 operation: feed-forward-and-upward; feed-upward-and-forward; feed- 203 backward; and null mode. These modes are described in Section 4, 204 then in the following sections separate guidelines are given for each 205 mode. 207 This document updates the advice to subnetwork designers about ECN in 208 Section 13 of [RFC3819]. 210 1.1. Scope 212 This document only concerns wire protocol processing of explicit 213 notification of congestion and makes no changes or recommendations 214 concerning algorithms for congestion marking or for congestion 215 response (algorithm issues should be independent of the layer the 216 algorithm operates in). 218 The question of congestion notification signals with different 219 semantics to those of ECN in IP is touched on in a couple of specific 220 cases (e.g. QCN [IEEE802.1Qau]) and with schemes with multiple 221 severity levels such as PCN [RFC6660]). However, no attempt is made 222 to give guidelines about schemes with different semantics that are 223 yet to be invented. 225 The semantics of congestion signals can be relative to the traffic 226 class. Therefore correct propagation of congestion signals could 227 depend on correct propagation of any traffic class field between the 228 layers. In this document, correct propagation of traffic class 229 information is assumed, while what 'correct' means and how it is 230 achieved is covered elsewhere (e.g. [RFC2983]) and is outside the 231 scope of the present document. 233 Note that these guidelines do not require the subnet wire protocol to 234 be changed to accommodate congestion notification. For instance, the 235 Feed-Up-and-Forward Mode (Section 4.2) and the Null Mode 236 (Section 4.4) do not. Another way to add congestion notification 237 without consuming header space in the subnet protocol might be to use 238 a parallel control plane protocol. 240 This document focuses on the congestion notification interface 241 between IP and lower layer protocols that can encapsulate IP, where 242 the term 'IP' includes v4 or v6, unicast, multicast or anycast. 243 However, it is likely that the guidelines will also be useful when a 244 lower layer protocol or tunnel encapsulates itself (e.g. Ethernet 245 MAC in MAC [IEEE802.1Qah]) or when it encapsulates other protocols. 246 In the feed-backward mode, propagation of congestion signals for 247 multicast and anycast packets is out-of-scope (because it would be so 248 complicated that it is hoped no-one would attempt such an 249 abomination). 251 2. Terminology 253 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 254 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 255 document are to be interpreted as described in RFC 2119 [RFC2119]. 257 Further terminology used within this document: 259 Protocol data unit (PDU): Information that is delivered as a unit 260 among peer entities of a layered network consisting of protocol 261 control information (typically a header) and possibly user data 262 (payload) of that layer. The scope of this document includes 263 layer 2 and layer 3 networks, where the PDU is respectively termed 264 a frame or a packet (or a cell in ATM). PDU is a general term for 265 any of these. This definition also includes a payload with a shim 266 header lying somewhere between layer 2 and 3. 268 Transport: The end-to-end transmission control function, 269 conventionally considered at layer-4 in the OSI reference model. 270 Given the audience for this document will often use the word 271 transport to mean low level bit carriage, whenever the term is 272 used it will be qualified, e.g. 'L4 transport'. 274 Encapsulator: The link or tunnel endpoint function that adds an 275 outer header to a PDU (also termed the 'link ingress', the 'subnet 276 ingress', the 'ingress tunnel endpoint' or just the 'ingress' 277 where the context is clear). 279 Decapsulator: The link or tunnel endpoint function that removes an 280 outer header from a PDU (also termed the 'link egress', the 281 'subnet egress', the 'egress tunnel endpoint' or just the 'egress' 282 where the context is clear). 284 Incoming header: The header of an arriving PDU before encapsulation. 286 Outer header: The header added to encapsulate a PDU. 288 Inner header: The header encapsulated by the outer header. 290 Outgoing header: The header forwarded by the decapsulator. 292 CE: Congestion Experienced [RFC3168] 294 ECT: ECN-Capable Transport [RFC3168] 296 Not-ECT: Not ECN-Capable Transport [RFC3168] 298 Load Regulator: For each flow of PDUs, the transport function that 299 is capable of controlling the data rate. Typically located at the 300 data source, but in-path nodes can regulate load in some 301 congestion control arrangements (e.g. admission control, policing 302 nodes or transport circuit-breakers [RFC8084]). Note the term "a 303 function capable of controlling the load" deliberately includes a 304 transport that doesn't actually control the load responsively but 305 ideally it ought to (e.g. a sending application without 306 congestion control that uses UDP). 308 ECN-PDU: A PDU that is part of a feedback loop within which all the 309 nodes that need to propagate explicit congestion notifications 310 back to the Load Regulator are ECN-capable. An IP packet with a 311 non-zero ECN field implies that the endpoints are ECN-capable, so 312 this would be an ECN-PDU. However, ECN-PDU is intended to be a 313 general term for a PDU at any layer, not just IP. 315 Not-ECN-PDU: A PDU that is part of a feedback-loop within which some 316 nodes necessary to propagate explicit congestion notifications 317 back to the load regulator are not ECN-capable. 319 Congestion Baseline: The location of the function on the path that 320 initialised the values of all congestion notification fields in a 321 sequence of packets, before any are set to the congestion 322 experienced (CE) codepoint if they experience congestion further 323 downstream. Typically the original data source at layer-4. 325 3. Guidelines in All Cases 327 RFC 3168 specifies that the ECN field in the IP header is intended to 328 be marked by active queue management algorithms. Any congestion 329 notification from an algorithm that does not conform to the 330 recommendations in [RFC7567] MUST NOT be propagated from a lower 331 layer into the ECN field in IP (see also [RFC4774] on alternate uses 332 of the ECN field). 334 4. Modes of Operation 336 This section sets down the different modes by which congestion 337 information is passed between the lower layer and the higher one. It 338 acts as a reference framework for the following sections, which give 339 normative guidelines for designers of explicit congestion 340 notification protocols, taking each mode in turn: 342 Feed-Forward-and-Up: Nodes feed forward congestion notification 343 towards the egress within the lower layer then up and along the 344 layers towards the end-to-end destination at the transport layer. 345 The following local optimisation is possible: 347 Feed-Up-and-Forward: A lower layer switch feeds-up congestion 348 notification directly into the ECN field in the higher layer 349 (e.g. IP) header, irrespective of whether the node is at the 350 egress of a subnet. 352 Feed-Backward: Nodes feed back congestion signals towards the 353 ingress of the lower layer and (optionally) attempt to control 354 congestion within their own layer. 356 Null: Nodes cannot experience congestion at the lower layer except 357 at ingress nodes (which are IP-aware or equivalently higher-layer- 358 aware). 360 4.1. Feed-Forward-and-Up Mode 362 Like IP and MPLS, many subnet technologies are based on self- 363 contained protocol data units (PDUs) or frames sent unreliably. They 364 provide no feedback channel at the subnetwork layer, instead relying 365 on higher layers (e.g. TCP) to feed back loss signals. 367 In these cases, ECN may best be supported by standardising explicit 368 notification of congestion into the lower layer protocol that carries 369 the data forwards. It will then also be necessary to define how the 370 egress of the lower layer subnet propagates this explicit signal into 371 the forwarded upper layer (IP) header. It can then continue forwards 372 until it finally reaches the destination transport (at L4). Then 373 typically the destination will feed this congestion notification back 374 to the source transport using an end-to-end protocol (e.g. TCP). 375 This is the arrangement that has already been used to add ECN to IP- 376 in-IP tunnels [RFC6040], IP-in-MPLS and MPLS-in-MPLS [RFC5129]. 378 This mode is illustrated in Figure 1. Along the middle of the 379 figure, layers 2, 3 and 4 of the protocol stack are shown, and one 380 packet is shown along the bottom as it progresses across the network 381 from source to destination, crossing two subnets connected by a 382 router, and crossing two switches on the path across each subnet. 383 Congestion at the output of the first switch (shown as *) leads to a 384 congestion marking in the L2 header (shown as C in the illustration 385 of the packet). The chevrons show the progress of the resulting 386 congestion indication. It is propagated from link to link across the 387 subnet in the L2 header, then when the router removes the marked L2 388 header, it propagates the marking up into the L3 (IP) header. The 389 router forwards the marked L3 header into subnet 2, and when it adds 390 a new L2 header it copies the L3 marking into the L2 header as well, 391 as shown by the 'C's in both layers (assuming the technology of 392 subnet 2 also supports explicit congestion marking). 394 Note that there is no implication that each 'C' marking is encoded 395 the same; a different encoding might be used for the 'C' marking in 396 each protocol. 398 Finally, for completeness, we show the L3 marking arriving at the 399 destination, where the host transport protocol (e.g. TCP) feeds it 400 back to the source in the L4 acknowledgement (the 'C' at L4 in the 401 packet at the top of the diagram). 403 _ _ _ 404 /_______ | | |C| ACK Packet (V) 405 \ |_|_|_| 406 +---+ layer: 2 3 4 header +---+ 407 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 408 | | +---+ | ^ | 409 | | . . . . . . Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3 410 | | +---+ +---+ | ^ | +---+ +---+ | | 411 | | | *|>>>>>|>>>|>>>>>|>^ | | | | | | |L2 412 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 413 source subnet A router subnet B dest 414 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ 415 | | | | | | | | |C| | | |C| | | |C|C| Data________\ 416 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) / 417 layer: 4 3 2A 4 3 2A 4 3 4 3 2B 418 header 420 Figure 1: Feed-Forward-and-Up Mode 422 Of course, modern networks are rarely as simple as this text-book 423 example, often involving multiple nested layers. For example, a 3GPP 424 mobile network may have two IP-in-IP (GTP) tunnels in series and an 425 MPLS backhaul between the base station and the first router. 426 Nonetheless, the example illustrates the general idea of feeding 427 congestion notification forward then upward whenever a header is 428 removed at the egress of a subnet. 430 Note that the FECN (forward ECN) bit in Frame Relay and the explicit 431 forward congestion indication (EFCI [ITU-T.I.371]) bit in ATM user 432 data cells follow a feed-forward pattern. However, in ATM, this 433 arrangement is only part of a feed-forward-and-backward pattern at 434 the lower layer, not feed-forward-and-up out of the lower layer--the 435 intention was never to interface to IP ECN at the subnet egress. To 436 our knowledge, Frame Relay FECN is solely used to detect where more 437 capacity should be provisioned [Buck00]. 439 4.2. Feed-Up-and-Forward Mode 441 Ethernet is particularly difficult to extend incrementally to support 442 explicit congestion notification. One way to support ECN in such 443 cases has been to use so called 'layer-3 switches'. These are 444 Ethernet switches that bury into the Ethernet payload to find an IP 445 header and manipulate or act on certain IP fields (specifically 446 Diffserv & ECN). For instance, in Data Center TCP [DCTCP], layer-3 447 switches are configured to mark the ECN field of the IP header within 448 the Ethernet payload when their output buffer becomes congested. 449 With respect to switching, a layer-3 switch acts solely on the 450 addresses in the Ethernet header; it doesn't use IP addresses, and it 451 doesn't decrement the TTL field in the IP header. 453 _ _ _ 454 /_______ | | |C| ACK packet (V) 455 \ |_|_|_| 456 +---+ layer: 2 3 4 header +---+ 457 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 458 | | +---+ | ^ | 459 | | . . . >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3 460 | | +--^+ +---+ | | +---+ +---+ | | 461 | | | *| | | | | | | | | | |L2 462 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 463 source subnet E router subnet F dest 464 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ 465 | | | | | | | |C| | | | |C| | | |C|C| data________\ 466 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / 467 layer: 4 3 2 4 3 2 4 3 4 3 2 468 header 470 Figure 2: Feed-Up-and-Forward Mode 472 By comparing Figure 2 with Figure 1, it can be seen that subnet E 473 (perhaps a subnet of layer-3 Ethernet switches) works in feed-up-and- 474 forward mode by notifying congestion directly into L3 at the point of 475 congestion, even though the congested switch does not otherwise act 476 at L3. In this example, the technology in subnet F (e.g. MPLS) does 477 support ECN natively, so when the router adds the layer-2 header it 478 copies the ECN marking from L3 to L2 as well. 480 4.3. Feed-Backward Mode 482 In some layer 2 technologies, explicit congestion notification has 483 been defined for use internally within the subnet with its own 484 feedback and load regulation, but typically the interface with IP for 485 ECN has not been defined. 487 For instance, for the available bit-rate (ABR) service in ATM, the 488 relative rate mechanism was one of the more popular mechanisms for 489 managing traffic, tending to supersede earlier designs. In this 490 approach ATM switches send special resource management (RM) cells in 491 both the forward and backward directions to control the ingress rate 492 of user data into a virtual circuit. If a switch buffer is 493 approaching congestion or is congested it sends an RM cell back 494 towards the ingress with respectively the No Increase (NI) or 495 Congestion Indication (CI) bit set in its message type field 496 [ATM-TM-ABR]. The ingress then holds or decreases its sending bit- 497 rate accordingly. 499 _ _ _ 500 /_______ | | |C| ACK packet (X) 501 \ |_|_|_| 502 +---+ layer: 2 3 4 header +---+ 503 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4 504 | | +---+ | ^ | 505 | | | *|>>> Packet W >>>>>>>>>>>>|>^ |L3 506 | | +---+ +---+ | | +---+ +---+ | | 507 | | | | | | | <|<<<<<|<<<|<(V)<|<<<| | |L2 508 | | . . | . |Packet U | . . | . | . . | . | . . | .*| . . | |L2 509 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| 510 source subnet G router subnet H dest 511 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ later 512 | | | | | | | | | | | | | | | | |C| | data________\ 513 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (W) / 514 4 3 2 4 3 2 4 3 4 3 2 515 _ 516 /__ |C| Feedback control 517 \ |_| cell/frame (V) 518 2 519 __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ earlier 520 | | | | | | | | | | | | | | | | | | | data________\ 521 |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / 522 layer: 4 3 2 4 3 2 4 3 4 3 2 523 header 525 Figure 3: Feed-Backward Mode 527 ATM's feed-backward approach doesn't fit well when layered beneath 528 IP's feed-forward approach--unless the initial data source is the 529 same node as the ATM ingress. Figure 3 shows the feed-backward 530 approach being used in subnet H. If the final switch on the path is 531 congested (*), it doesn't feed-forward any congestion indications on 532 packet (U). Instead it sends a control cell (V) back to the router 533 at the ATM ingress. 535 However, the backward feedback doesn't reach the original data source 536 directly because IP doesn't support backward feedback (and subnet G 537 is independent of subnet H). Instead, the router in the middle 538 throttles down its sending rate but the original data sources don't 539 reduce their rates. The resulting rate mismatch causes the middle 540 router's buffer at layer 3 to back up until it becomes congested, 541 which it signals forwards on later data packets at layer 3 (e.g. 542 packet W). Note that the forward signal from the middle router is 543 not triggered directly by the backward signal. Rather, it is 544 triggered by congestion resulting from the middle router's mismatched 545 rate response to the backward signal. 547 In response to this later forward signalling, end-to-end feedback at 548 layer-4 finally completes the tortuous path of congestion indications 549 back to the origin data source, as before. 551 4.4. Null Mode 553 Often link and physical layer resources are 'non-blocking' by design. 554 In these cases congestion notification may be implemented but it does 555 not need to be deployed at the lower layer; ECN in IP would be 556 sufficient. 558 A degenerate example is a point-to-point Ethernet link. Excess 559 loading of the link merely causes the queue from the higher layer to 560 back up, while the lower layer remains immune to congestion. Even a 561 whole meshed subnetwork can be made immune to interior congestion by 562 limiting ingress capacity and sufficient sizing of interior links, 563 e.g. a non-blocking fat-tree network. An alternative to fat links 564 near the root is numerous thin links with multi-path routing to 565 ensure even worst-case patterns of load cannot congest any link, e.g. 566 a Clos network. 568 5. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 569 Notification 571 Feed-forward-and-up is the mode already used for signalling ECN up 572 the layers through MPLS into IP [RFC5129] and through IP-in-IP 573 tunnels [RFC6040], whether encapsulating with IPv4 [RFC2003], IPv6 574 [RFC2473] or IPsec [RFC4301]. These RFCs take a consistent approach 575 and the following guidelines are designed to ensure this consistency 576 continues as ECN support is added to other protocols that encapsulate 577 IP. The guidelines are also designed to ensure compliance with the 578 more general best current practice for the design of alternate ECN 579 schemes given in [RFC4774]. 581 The rest of this section is structured as follows: 583 o Section 5.1 addresses the most straightforward cases, where 584 [RFC6040] can be applied directly to add ECN to tunnels that are 585 effectively IP-in-IP tunnels, but with shim header(s) between the 586 IP headers. 588 o The subsequent sections give guidelines for adding ECN to a subnet 589 technology that uses feed-forward-and-up mode like IP, but it is 590 not so similar to IP that [RFC6040] rules can be applied directly. 591 Specifically: 593 * Sections 5.2, 5.3 and 5.4 respectively address how to add ECN 594 support to the wire protocol and to the encapsulators and 595 decapsulators at the ingress and egress of the subnet. 597 * Section 5.5 deals with the special, but common, case of 598 sequences of tunnels or subnets that all use the same 599 technology 601 * Section 5.6 deals with the question of reframing when IP 602 packets do not map 1:1 into lower layer frames. 604 5.1. IP-in-IP Tunnels with Shim Headers 606 A common pattern for many tunnelling protocols is to encapsulate an 607 inner IP header with shim header(s) then an outer IP header. A shim 608 header is defined as one that is not sufficient alone to forward the 609 packet as an outer header. Another common pattern is for a shim to 610 encapsulate a layer 2 (L2) header, which in turn encapsulates (or 611 might encapsulate) an IP header. [I-D.ietf-tsvwg-rfc6040update-shim] 612 clarifies that RFC 6040 is just as applicable when there are shim(s) 613 and possibly a L2 header between two IP headers. 615 However, it is not always feasible or necessary to propagate ECN 616 between IP headers when separated by a shim. For instance, it might 617 be too costly to dig to arbitrary depths to find an inner IP header, 618 there may be little or no congestion within the tunnel by design (see 619 null mode in Section 4.4 above), or a legacy implementation might not 620 support ECN. In cases where a tunnel does not support ECN, it is 621 important that the ingress does not copy the ECN field from an inner 622 IP header to an outer. Therefore section 4 of 623 [I-D.ietf-tsvwg-rfc6040update-shim] requires network operators to 624 configure the ingress of a non-ECN tunnel so that it zeros the ECN 625 field in the outer IP header. 627 Nonetheless, in many cases it is feasible to propagate the ECN field 628 between IP headers separated by shim header(s) and/or a L2 header. 629 Particularly in the typical case when the outer IP header and the 630 shim(s) are added (or removed) as part of the same procedure. Even 631 if the shim(s) encapsulate a L2 header, it is often possible to find 632 an inner IP header within the L2 header and propagate ECN between 633 that and the outer IP header. This can be thought of as a special 634 case of the feed-up-and-forward mode (Section 4.2), so the guidelines 635 for this mode apply (Section 6). 637 Numerous shim protocols have been defined for IP tunnelling. More 638 recent ones e.g. Generic UDP Encapsulation (GUE) [I-D.ietf-nvo3-gue] 639 and Geneve [I-D.ietf-nvo3-geneve] cite RFC 6040. And some earlier 640 ones, e.g. CAPWAP [RFC5415] and LISP [RFC6830], cite RFC 3168, which 641 is compatible with RFC 6040. 643 However, as Section 9.3 of RFC3168 pointed out, ECN support needs to 644 be defined for many earlier shim-based tunnelling protocols, e.g. 645 L2TPv2 [RFC2661], L2TPv3 [RFC3931], GRE [RFC2784], PPTP [RFC2637], 646 GTP [GTPv1], [GTPv1-U], [GTPv2-C] and Teredo [RFC4380] as well as 647 some recent ones, e.g. VXLAN [RFC7348] and NVGRE [RFC7637]. 649 All these IP-based encapsulations can be updated in one shot by 650 simple reference to RFC 6040. However, it would not be appropriate 651 to update all these protocols from within the present guidance 652 document. Instead a companion specification 653 [I-D.ietf-tsvwg-rfc6040update-shim] has been prepared that has 654 sufficient standards track status to update standards track 655 protocols. For those that are not under IETF change control 656 [I-D.ietf-tsvwg-rfc6040update-shim] can only recommend that the 657 relevant body updates them. 659 5.2. Wire Protocol Design: Indication of ECN Support 661 This section is intended to guide the redesign of any lower layer 662 protocol that encapsulate IP to add native ECN support at the lower 663 layer. It reflects the approaches used in [RFC6040] and in 664 [RFC5129]. Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS 665 encapsulations that already comply with [RFC6040] or [RFC5129] will 666 already satisfy this guidance. 668 A lower layer (or subnet) congestion notification system: 670 1. SHOULD NOT apply explicit congestion notifications to PDUs that 671 are destined for legacy layer-4 transport implementations that 672 will not understand ECN, and 674 2. SHOULD NOT apply explicit congestion notifications to PDUs if the 675 egress of the subnet might not propagate congestion notifications 676 onward into the higher layer. 678 We use the term ECN-PDUs for a PDU on a feedback loop that will 679 propagate congestion notification properly because it meets both 680 the above criteria. And a Not-ECN-PDU is a PDU on a feedback 681 loop that does not meet both criteria, and will therefore not 682 propagate congestion notification properly. A corollary of the 683 above is that a lower layer congestion notification protocol: 685 3. SHOULD be able to distinguish ECN-PDUs from Not-ECN-PDUs. 687 Note that there is no need for all interior nodes within a subnet to 688 be able to mark congestion explicitly. A mix of ECN and drop signals 689 from different nodes is fine. However, if _any_ interior nodes might 690 generate ECN markings, guideline 2 above says that all relevant 691 egress node(s) SHOULD be able to propagate those markings up to the 692 higher layer. 694 In IP, if the ECN field in each PDU is cleared to the Not-ECT (not 695 ECN-capable transport) codepoint, it indicates that the L4 transport 696 will not understand congestion markings. A congested buffer must not 697 mark these Not-ECT PDUs, and therefore drops them instead. 699 The mechanism a lower layer uses to distinguish the ECN-capability of 700 PDUs need not mimic that of IP. The above guidelines merely say that 701 the lower layer system, as a whole, should achieve the same outcome. 702 For instance, ECN-capable feedback loops might use PDUs that are 703 identified by a particular set of labels or tags. Alternatively, 704 logical link protocols that use flow state might determine whether a 705 PDU can be congestion marked by checking for ECN-support in the flow 706 state. Other protocols might depend on out-of-band control signals. 708 The per-domain checking of ECN support in MPLS [RFC5129] is a good 709 example of a way to avoid sending congestion markings to transports 710 that will not understand them, without using any header space in the 711 subnet protocol. 713 In MPLS, header space is extremely limited, therefore RFC5129 does 714 not provide a field in the MPLS header to indicate whether the PDU is 715 an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are 716 allowed to set explicit congestion indications without checking 717 whether the PDU is destined for a transport that will understand 718 them. Nonetheless, this is made safe by requiring that the network 719 operator upgrades all decapsulating edges of a whole domain at once, 720 as soon as even one switch within the domain is configured to mark 721 rather than drop during congestion. Therefore, any edge node that 722 might decapsulate a packet will be capable of checking whether the 723 higher layer transport is ECN-capable. When decapsulating a CE- 724 marked packet, if the decapsulator discovers that the higher layer 725 (inner header) indicates the transport is not ECN-capable, it drops 726 the packet--effectively on behalf of the earlier congested node (see 727 Decapsulation Guideline 1 in Section 5.4). 729 It was only appropriate to define such an incremental deployment 730 strategy because MPLS is targeted solely at professional operators, 731 who can be expected to ensure that a whole subnetwork is consistently 732 configured. This strategy might not be appropriate for other link 733 technologies targeted at zero-configuration deployment or deployment 734 by the general public (e.g. Ethernet). For such 'plug-and-play' 735 environments it will be necessary to invent a failsafe approach that 736 ensures congestion markings will never fall into black holes, no 737 matter how inconsistently a system is put together. Alternatively, 738 congestion notification relying on correct system configuration could 739 be confined to flavours of Ethernet intended only for professional 740 network operators, such as IEEE 802.1ah Provider Backbone Bridges 741 (PBB). 743 ECN support in TRILL [I-D.ietf-trill-ecn-support] provides a good 744 example of how to add ECN to a lower layer protocol without relying 745 on careful and consistent operator configuration. TRILL provides an 746 extension header word with space for flags of different categories 747 depending on whether logic to understand the extension is critical. 748 The congestion experienced marking has been defined as a 'critical 749 ingress-to-egress' flag. So if a transit RBridge sets this flag and 750 an egress RBridge does not have any logic to process it, it will drop 751 it; which is the desired default action anyway. Therefore TRILL 752 RBridges can be updated with support for ECN in no particular order 753 and, at the egress of the TRILL campus, congestion notification will 754 be propagated to IP as ECN whenever ECN logic has been implemented, 755 and as drop otherwise. 757 QCN [IEEE802.1Qau] provides another example of how to indicate to 758 lower layer devices that the end-points will not understand ECN. An 759 operator can define certain 802.1p classes of service to indicate 760 non-QCN frames and an ingress bridge is required to map arriving not- 761 QCN-capable IP packets to one of these non-QCN 802.1p classes. 763 5.3. Encapsulation Guidelines 765 This section is intended to guide the redesign of any node that 766 encapsulates IP with a lower layer header when adding native ECN 767 support to the lower layer protocol. It reflects the approaches used 768 in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or IP-in- 769 MPLS or MPLS-in-MPLS encapsulations that already comply with 770 [RFC6040] or [RFC5129] will already satisfy this guidance. 772 1. Egress Capability Check: A subnet ingress needs to be sure that 773 the corresponding egress of a subnet will propagate any 774 congestion notification added to the outer header across the 775 subnet. This is necessary in addition to checking that an 776 incoming PDU indicates an ECN-capable (L4) transport. Examples 777 of how this guarantee might be provided include: 779 * by configuration (e.g. if any label switches in a domain 780 support ECN marking, [RFC5129] requires all egress nodes to 781 have been configured to propagate ECN) 783 * by the ingress explicitly checking that the egress propagates 784 ECN (e.g. TRILL uses IS-IS to check path capabilities before 785 using critical options [RFC7780]) 787 * by inherent design of the protocol (e.g. by encoding ECN 788 marking on the outer header in such a way that a legacy egress 789 that does not understand ECN will consider the PDU corrupt and 790 discard it, thus at least propagating a form of congestion 791 signal). 793 2. Egress Fails Capability Check: If the ingress cannot guarantee 794 that the egress will propagate congestion notification, the 795 ingress SHOULD disable ECN when it forwards the PDU at the lower 796 layer. An example of how the ingress might disable ECN at the 797 lower layer would be by setting the outer header of the PDU to 798 identify it as a Not-ECN-PDU, assuming the subnet technology 799 supports such a concept. 801 3. Standard Congestion Monitoring Baseline: Once the ingress to a 802 subnet has established that the egress will correctly propagate 803 ECN, on encapsulation it SHOULD encode the same level of 804 congestion in outer headers as is arriving in incoming headers. 805 For example it might copy any incoming congestion notification 806 into the outer header of the lower layer protocol. 808 This ensures that all outer headers reflect congestion 809 accumulated along the whole upstream path since the Load 810 Regulator, not just since the ingress of the subnet. A node that 811 is not the Load Regulator SHOULD NOT re-initialise the level of 812 CE markings in the outer to zero. 814 This guideline is intended to ensure that any bulk congestion 815 monitoring of outer headers (e.g. by a network management node 816 monitoring ECN in passing frames) is most meaningful. For 817 instance, if an operator measures CE in 0.4% of passing outer 818 headers, this information is only useful if the operator knows 819 where the proportion of CE markings was last initialised to 0% 820 (the Congestion Baseline). Such monitoring information will not 821 be useful if some subnet ingress nodes reset all outer CE 822 markings while others copy incoming CE markings into the outer. 824 Most information can be extracted if the Congestion Baseline is 825 standardised at the node that is regulating the load (the Load 826 Regulator--typically the data source). Then the operator can 827 measure both congestion since the Load Regulator, and congestion 828 since the subnet ingress. The latter might be measurable by 829 subtracting the level of CE markings on inner headers from that 830 on outer headers (see Appendix C of [RFC6040]). 832 5.4. Decapsulation Guidelines 834 This section is intended to guide the redesign of any node that 835 decapsulates IP from within a lower layer header when adding native 836 ECN support to the lower layer protocol. It reflects the approaches 837 used in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or 838 IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with 839 [RFC6040] or [RFC5129] will already satisfy this guidance. 841 A subnet egress SHOULD NOT simply copy congestion notification from 842 outer headers to the forwarded header. It SHOULD calculate the 843 outgoing congestion notification field from the inner and outer 844 headers using the following guidelines. If there is any conflict, 845 rules earlier in the list take precedence over rules later in the 846 list: 848 1. If the arriving inner header is a Not-ECN-PDU it implies the L4 849 transport will not understand explicit congestion markings. 850 Then: 852 * If the outer header carries an explicit congestion marking, 853 drop is the only indication of congestion that the L4 854 transport will understand. If the congestion marking is the 855 most severe possible, the packet MUST be dropped. However, if 856 congestion can be marked with multiple levels severity and the 857 packet's marking is not the most severe, the packet MAY be 858 forwarded, but it SHOULD be dropped. 860 * If the outer is an ECN-PDU that carries no indication of 861 congestion or a Not-ECN-PDU the PDU SHOULD be forwarded, but 862 still as a Not-ECN-PDU. 864 2. If the outer header does not support explicit congestion 865 notification (a Not-ECN-PDU), but the inner header does (an ECN- 866 PDU), the inner header SHOULD be forwarded unchanged. 868 3. In some lower layer protocols congestion may be signalled as a 869 numerical level, such as in the control frames of quantised 870 congestion notification [IEEE802.1Qau]. If such a multi-bit 871 encoding encapsulates an ECN-capable IP data packet, a function 872 will be needed to convert the quantised congestion level into the 873 frequency of congestion markings in outgoing IP packets. 875 4. Congestion indications may be encoded by a severity level. For 876 instance increasing levels of congestion might be encoded by 877 numerically increasing indications, e.g. pre-congestion 878 notification (PCN) can be encoded in each PDU at three severity 879 levels in IP or MPLS [RFC6660]. 881 If the arriving inner header is an ECN-PDU, where the inner and 882 outer headers carry indications of congestion of different 883 severity, the more severe indication SHOULD be forwarded in 884 preference to the less severe. 886 5. The inner and outer headers might carry a combination of 887 congestion notification fields that should not be possible given 888 any currently used protocol transitions. For instance, if 889 Encapsulation Guideline 3 in Section 5.3 had been followed, it 890 should not be possible to have a less severe indication of 891 congestion in the outer than in the inner. It MAY be appropriate 892 to log unexpected combinations of headers and possibly raise an 893 alarm. 895 If a safe outgoing codepoint can be defined for such a PDU, the 896 PDU SHOULD be forwarded rather than dropped. Some implementers 897 discard PDUs with currently unused combinations of headers just 898 in case they represent an attack. However, an approach using 899 alarms and policy-mediated drop is preferable to hard-coded drop, 900 so that operators can keep track of possible attacks but 901 currently unused combinations are not precluded from future use 902 through new standards actions. 904 5.5. Sequences of Similar Tunnels or Subnets 906 In some deployments, particularly in 3GPP networks, an IP packet may 907 traverse two or more IP-in-IP tunnels in sequence that all use 908 identical technology (e.g. GTP). 910 In such cases, it would be sufficient for every encapsulation and 911 decapsulation in the chain to comply with RFC 6040. Alternatively, 912 as an optimisation, a node that decapsulates a packet and immediately 913 re-encapsulates it for the next tunnel MAY copy the incoming outer 914 ECN field directly to the outgoing outer and the incoming inner ECN 915 field directly to the outgoing inner. Then the overall behavior 916 across the sequence of tunnel segments would still be consistent with 917 RFC 6040. 919 Appendix C of RFC6040 describes how a tunnel egress can monitor how 920 much congestion has been introduced within a tunnel. A network 921 operator might want to monitor how much congestion had been 922 introduced within a whole sequence of tunnels. Using the technique 923 in Appendix C of RFC6040 at the final egress, the operator could 924 monitor the whole sequence of tunnels, but only if the above 925 optimisation were used consistently along the sequence of tunnels, in 926 order to make it appear as a single tunnel. Therefore, tunnel 927 endpoint implementations SHOULD allow the operator to configure 928 whether this optimisation is enabled. 930 When ECN support is added to a subnet technology, consideration 931 SHOULD be given to a similar optimisation between subnets in sequence 932 if they all use the same technology. 934 5.6. Reframing and Congestion Markings 936 The guidance in this section is worded in terms of framing 937 boundaries, but it applies equally whether the protocol data units 938 are frames, cells or packets. 940 Where framing boundaries are different between two layers, congestion 941 indications SHOULD be propagated on the basis that a congestion 942 indication on a PDU applies to all the octets in the PDU. On 943 average, an encapsulator or decapsulator SHOULD approximately 944 preserve the number of marked octets arriving and leaving (counting 945 the size of inner headers, but not added encapsulating headers). 947 The next departing frame SHOULD be immediately marked even if only 948 enough incoming marked octets have arrived for part of the departing 949 frame. This ensures that any outstanding congestion marked octets 950 are propagated immediately, rather than held back waiting for a frame 951 no bigger than the outstanding marked octets--which might involve a 952 long wait. 954 For instance, an algorithm for marking departing frames could 955 maintain a counter representing the balance of arriving marked octets 956 minus departing marked octets. It adds the size of every marked 957 frame that arrives and if the counter is positive it marks the next 958 frame to depart and subtracts its size from the counter. This will 959 often leave a negative remainder in the counter, which is deliberate. 961 6. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 962 Notification 964 The guidance in this section is applicable, for example, when IP 965 packets: 967 o are encapsulated in Ethernet headers, which have no support for 968 ECN; 970 o are forwarded by the eNode-B (base station) of a 3GPP radio access 971 network, which is required to apply ECN marking during congestion, 972 [LTE-RA], [UTRAN], but the Packet Data Convergence Protocol (PDCP) 973 that encapsulates the IP header over the radio access has no 974 support for ECN. 976 This guidance also generalises to encapsulation by other subnet 977 technologies with no native support for explicit congestion 978 notification at the lower layer, but with support for finding and 979 processing an IP header. It is unlikely to be applicable or 980 necessary for IP-in-IP encapsulation, where feed-forward-and-up mode 981 based on [RFC6040] would be more appropriate. 983 Marking the IP header while switching at layer-2 (by using a layer-3 984 switch) or while forwarding in a radio access network seems to 985 represent a layering violation. However, it can be considered as a 986 benign optimisation if the guidelines below are followed. Feed-up- 987 and-forward is certainly not a general alternative to implementing 988 feed-forward congestion notification in the lower layer, because: 990 o IPv4 and IPv6 are not the only layer-3 protocols that might be 991 encapsulated by lower layer protocols 993 o Link-layer encryption might be in use, making the layer-2 payload 994 inaccessible 996 o Many Ethernet switches do not have 'layer-3 switch' capabilities 997 so they cannot read or modify an IP payload 999 o It might be costly to find an IP header (v4 or v6) when it may be 1000 encapsulated by more than one lower layer header, e.g. Ethernet 1001 MAC in MAC [IEEE802.1Qah]. 1003 Nonetheless, configuring lower layer equipment to look for an ECN 1004 field in an encapsulated IP header is a useful optimisation. If the 1005 implementation follows the guidelines below, this optimisation does 1006 not have to be confined to a controlled environment such as within a 1007 data centre; it could usefully be applied on any network--even if the 1008 operator is not sure whether the above issues will never apply: 1010 1. If a native lower-layer congestion notification mechanism exists 1011 for a subnet technology, it is safe to mix feed-up-and-forward 1012 with feed-forward-and-up on other switches in the same subnet. 1013 However, it will generally be more efficient to use the native 1014 mechanism. 1016 2. The depth of the search for an IP header SHOULD be limited. If 1017 an IP header is not found soon enough, or an unrecognised or 1018 unreadable header is encountered, the switch SHOULD resort to an 1019 alternative means of signalling congestion (e.g. drop, or the 1020 native lower layer mechanism if available). 1022 3. It is sufficient to use the first IP header found in the stack; 1023 the egress of the relevant tunnel can propagate congestion 1024 notification upwards to any more deeply encapsulated IP headers 1025 later. 1027 7. Feed-Backward Mode: Guidelines for Adding Congestion Notification 1029 It can be seen from Section 4.3 that congestion notification in a 1030 subnet using feed-backward mode has generally not been designed to be 1031 directly coupled with IP layer congestion notification. The subnet 1032 attempts to minimise congestion internally, and if the incoming load 1033 at the ingress exceeds the capacity somewhere through the subnet, the 1034 layer 3 buffer into the ingress backs up. Thus, a feed-backward mode 1035 subnet is in some sense similar to a null mode subnet, in that there 1036 is no need for any direct interaction between the subnet and higher 1037 layer congestion notification. Therefore no detailed protocol design 1038 guidelines are appropriate. Nonetheless, a more general guideline is 1039 appropriate: 1041 A subnetwork technology intended to eventually interface to IP 1042 SHOULD NOT be designed using only the feed-backward mode, which is 1043 certainly best for a stand-alone subnet, but would need to be 1044 modified to work efficiently as part of the wider Internet, 1045 because IP uses feed-forward-and-up mode. 1047 The feed-backward approach at least works beneath IP, where the term 1048 'works' is used only in a narrow functional sense because feed- 1049 backward can result in very inefficient and sluggish congestion 1050 control--except if it is confined to the subnet directly connected to 1051 the original data source, when it is faster than feed-forward. It 1052 would be valid to design a protocol that could work in feed-backward 1053 mode for paths that only cross one subnet, and in feed-forward-and-up 1054 mode for paths that cross subnets. 1056 In the early days of TCP/IP, a similar feed-backward approach was 1057 tried for explicit congestion signalling, using source-quench (SQ) 1058 ICMP control packets. However, SQ fell out of favour and is now 1059 formally deprecated [RFC6633]. The main problem was that it is hard 1060 for a data source to tell the difference between a spoofed SQ message 1061 and a quench request from a genuine buffer on the path. It is also 1062 hard for a lower layer buffer to address an SQ message to the 1063 original source port number, which may be buried within many layers 1064 of headers, and possibly encrypted. 1066 Quantised congestion notification (QCN--also known as backward 1067 congestion notification or BCN) [IEEE802.1Qau] uses a feed-backward 1068 mode structurally similar to ATM's relative rate mechanism. However, 1069 QCN confines its applicability to scenarios such as some data centres 1070 where all endpoints are directly attached by the same Ethernet 1071 technology. If a QCN subnet were later connected into a wider IP- 1072 based internetwork (e.g. when attempting to interconnect multiple 1073 data centres) it would suffer the inefficiency shown Figure 3. 1075 8. IANA Considerations (to be removed by RFC Editor) 1077 This memo includes no request to IANA. 1079 9. Security Considerations 1081 If a lower layer wire protocol is redesigned to include explicit 1082 congestion signalling in-band in the protocol header, care SHOULD be 1083 take to ensure that the field used is specified as mutable during 1084 transit. Otherwise interior nodes signalling congestion would 1085 invalidate any authentication protocol applied to the lower layer 1086 header--by altering a header field that had been assumed as 1087 immutable. 1089 The redesign of protocols that encapsulate IP in order to propagate 1090 congestion signals between layers raises potential signal integrity 1091 concerns. Experimental or proposed approaches exist for assuring the 1092 end-to-end integrity of in-band congestion signals, e.g.: 1094 o Congestion exposure (ConEx ) for networks to audit that their 1095 congestion signals are not being suppressed by other networks or 1096 by receivers, and for networks to police that senders are 1097 responding sufficiently to the signals, irrespective of the 1098 transport protocol used [RFC7713]. 1100 o The ECN nonce [RFC3540] for a TCP sender to detect whether a 1101 network or the receiver is suppressing congestion signals. 1103 o A test with the same goals as the ECN nonce, but without the need 1104 for the receiver to co-operate with the protocol 1105 [I-D.moncaster-tcpm-rcv-cheat]. 1107 Given these end-to-end approaches are already being specified, it 1108 would make little sense to attempt to design hop-by-hop congestion 1109 signal integrity into a new lower layer protocol, because end-to-end 1110 integrity inherently achieves hop-by-hop integrity. 1112 10. Conclusions 1114 Following the guidance in the document enables ECN support to be 1115 extended to numerous protocols that encapsulate IP (v4 & v6) in a 1116 consistent way, so that IP continues to fulfil its role as an end-to- 1117 end interoperability layer. This includes: 1119 o A wide range of tunnelling protocols with various forms of shim 1120 header between two IP headers; 1122 o A wide range of subnet technologies, particularly those that work 1123 in the same 'feed-forward-and-up' mode that is used to support ECN 1124 in IP and MPLS. 1126 Guidelines have been defined for supporting propagation of ECN 1127 between Ethernet and IP on so-called Layer-3 Ethernet switches, using 1128 a 'feed-up-an-forward' mode. This approach could enable other subnet 1129 technologies to pass ECN signals into the IP layer, even if they do 1130 not support ECN natively. 1132 Finally, attempting to add ECN to a subnet technology in feed- 1133 backward mode is deprecated except in special cases, due to its 1134 likely sluggish response to congestion. 1136 11. Acknowledgements 1138 Thanks to Gorry Fairhurst for extensive reviews. Thanks also to the 1139 following reviewers: Richard Scheffenegger, Ingemar Johansson, Piers 1140 O'Hanlon and Michael Welzl, who pointed out that lower layer 1141 congestion notification signals may have different semantics to those 1142 in IP. Thanks are also due to the tsvwg chairs, TSV ADs and IETF 1143 liaison people such as Eric Gray, Dan Romascanu and Gonzalo Camarillo 1144 for helping with the liaisons with the IEEE and 3GPP. And thanks to 1145 Georg Mayer and particularly to Erik Guttman for the extensive search 1146 and categorisation of any 3GPP specifications that cite ECN 1147 specifications. 1149 Bob Briscoe was part-funded by the European Community under its 1150 Seventh Framework Programme through the Trilogy project (ICT-216372) 1151 for initial drafts and through the Reducing Internet Transport 1152 Latency (RITE) project (ICT-317700) subsequently. The views 1153 expressed here are solely those of the authors. 1155 12. Comments Solicited 1157 Comments and questions are encouraged and very welcome. They can be 1158 addressed to the IETF Transport Area working group mailing list 1159 , and/or to the authors. 1161 13. References 1163 13.1. Normative References 1165 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1166 Requirement Levels", BCP 14, RFC 2119, 1167 DOI 10.17487/RFC2119, March 1997, 1168 . 1170 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1171 of Explicit Congestion Notification (ECN) to IP", 1172 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1173 . 1175 [RFC3819] Karn, P., Ed., Bormann, C., Fairhurst, G., Grossman, D., 1176 Ludwig, R., Mahdavi, J., Montenegro, G., Touch, J., and L. 1177 Wood, "Advice for Internet Subnetwork Designers", BCP 89, 1178 RFC 3819, DOI 10.17487/RFC3819, July 2004, 1179 . 1181 [RFC4774] Floyd, S., "Specifying Alternate Semantics for the 1182 Explicit Congestion Notification (ECN) Field", BCP 124, 1183 RFC 4774, DOI 10.17487/RFC4774, November 2006, 1184 . 1186 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 1187 Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 1188 2008, . 1190 [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion 1191 Notification", RFC 6040, DOI 10.17487/RFC6040, November 1192 2010, . 1194 13.2. Informative References 1196 [ATM-TM-ABR] 1197 Cisco, "Understanding the Available Bit Rate (ABR) Service 1198 Category for ATM VCs", Design Technote 10415, June 2005. 1200 [Buck00] Buckwalter, J., "Frame Relay: Technology and Practice", 1201 Pub. Addison Wesley ISBN-13: 978-0201485240, 2000. 1203 [DCTCP] Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel, 1204 P., Prabhakar, B., Sengupta, S., and M. Sridharan, "Data 1205 Center TCP (DCTCP)", ACM SIGCOMM CCR 40(4)63--74, October 1206 2010, . 1208 [GTPv1] 3GPP, "GPRS Tunnelling Protocol (GTP) across the Gn and Gp 1209 interface", Technical Specification TS 29.060. 1211 [GTPv1-U] 3GPP, "General Packet Radio System (GPRS) Tunnelling 1212 Protocol User Plane (GTPv1-U)", Technical Specification TS 1213 29.281. 1215 [GTPv2-C] 3GPP, "Evolved General Packet Radio Service (GPRS) 1216 Tunnelling Protocol for Control plane (GTPv2-C)", 1217 Technical Specification TS 29.274. 1219 [I-D.ietf-nvo3-geneve] 1220 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 1221 Network Virtualization Encapsulation", draft-ietf- 1222 nvo3-geneve-04 (work in progress), March 2017. 1224 [I-D.ietf-nvo3-gue] 1225 Herbert, T., Yong, L., and O. Zia, "Generic UDP 1226 Encapsulation", draft-ietf-nvo3-gue-05 (work in progress), 1227 October 2016. 1229 [I-D.ietf-trill-ecn-support] 1230 Eastlake, D. and B. Briscoe, "TRILL: ECN (Explicit 1231 Congestion Notification) Support", draft-ietf-trill-ecn- 1232 support-03 (work in progress), May 2017. 1234 [I-D.ietf-tsvwg-rfc6040update-shim] 1235 Briscoe, B., "Propagating Explicit Congestion Notification 1236 Across IP Tunnel Headers Separated by a Shim", draft-ietf- 1237 tsvwg-rfc6040update-shim-04 (work in progress), July 2017. 1239 [I-D.moncaster-tcpm-rcv-cheat] 1240 Moncaster, T., Briscoe, B., and A. Jacquet, "A TCP Test to 1241 Allow Senders to Identify Receiver Non-Compliance", draft- 1242 moncaster-tcpm-rcv-cheat-03 (work in progress), July 2014. 1244 [IEEE802.1Qah] 1245 IEEE, "IEEE Standard for Local and Metropolitan Area 1246 Networks--Virtual Bridged Local Area Networks--Amendment 1247 6: Provider Backbone Bridges", IEEE Std 802.1Qah-2008, 1248 August 2008, 1249 . 1251 (Access Controlled link within page) 1253 [IEEE802.1Qau] 1254 Finn, N., Ed., "IEEE Standard for Local and Metropolitan 1255 Area Networks--Virtual Bridged Local Area Networks - 1256 Amendment 13: Congestion Notification", IEEE Std 802.1Qau- 1257 2010, March 2010, . 1260 (Access Controlled link within page) 1262 [ITU-T.I.371] 1263 ITU-T, "Traffic Control and Congestion Control in B-ISDN", 1264 ITU-T Rec. I.371 (03/04), March 2004, 1265 . 1268 [LTE-RA] 3GPP, "Evolved Universal Terrestrial Radio Access (E-UTRA) 1269 and Evolved Universal Terrestrial Radio Access Network 1270 (E-UTRAN); Overall description; Stage 2", Technical 1271 Specification TS 36.300. 1273 [RFC1323] Jacobson, V., Braden, R., and D. Borman, "TCP Extensions 1274 for High Performance", RFC 1323, DOI 10.17487/RFC1323, May 1275 1992, . 1277 [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, 1278 DOI 10.17487/RFC2003, October 1996, 1279 . 1281 [RFC2473] Conta, A. and S. Deering, "Generic Packet Tunneling in 1282 IPv6 Specification", RFC 2473, DOI 10.17487/RFC2473, 1283 December 1998, . 1285 [RFC2637] Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little, 1286 W., and G. Zorn, "Point-to-Point Tunneling Protocol 1287 (PPTP)", RFC 2637, DOI 10.17487/RFC2637, July 1999, 1288 . 1290 [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, 1291 G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", 1292 RFC 2661, DOI 10.17487/RFC2661, August 1999, 1293 . 1295 [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. 1296 Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, 1297 DOI 10.17487/RFC2784, March 2000, 1298 . 1300 [RFC2884] Hadi Salim, J. and U. Ahmed, "Performance Evaluation of 1301 Explicit Congestion Notification (ECN) in IP Networks", 1302 RFC 2884, DOI 10.17487/RFC2884, July 2000, 1303 . 1305 [RFC2983] Black, D., "Differentiated Services and Tunnels", 1306 RFC 2983, DOI 10.17487/RFC2983, October 2000, 1307 . 1309 [RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit 1310 Congestion Notification (ECN) Signaling with Nonces", 1311 RFC 3540, DOI 10.17487/RFC3540, June 2003, 1312 . 1314 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1315 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 1316 RFC 3931, DOI 10.17487/RFC3931, March 2005, 1317 . 1319 [RFC4301] Kent, S. and K. Seo, "Security Architecture for the 1320 Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, 1321 December 2005, . 1323 [RFC4380] Huitema, C., "Teredo: Tunneling IPv6 over UDP through 1324 Network Address Translations (NATs)", RFC 4380, 1325 DOI 10.17487/RFC4380, February 2006, 1326 . 1328 [RFC5415] Calhoun, P., Ed., Montemurro, M., Ed., and D. Stanley, 1329 Ed., "Control And Provisioning of Wireless Access Points 1330 (CAPWAP) Protocol Specification", RFC 5415, 1331 DOI 10.17487/RFC5415, March 2009, 1332 . 1334 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 1335 RFC 6633, DOI 10.17487/RFC6633, May 2012, 1336 . 1338 [RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three 1339 Pre-Congestion Notification (PCN) States in the IP Header 1340 Using a Single Diffserv Codepoint (DSCP)", RFC 6660, 1341 DOI 10.17487/RFC6660, July 2012, 1342 . 1344 [RFC6830] Farinacci, D., Fuller, V., Meyer, D., and D. Lewis, "The 1345 Locator/ID Separation Protocol (LISP)", RFC 6830, 1346 DOI 10.17487/RFC6830, January 2013, 1347 . 1349 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 1350 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 1351 eXtensible Local Area Network (VXLAN): A Framework for 1352 Overlaying Virtualized Layer 2 Networks over Layer 3 1353 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, 1354 . 1356 [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF 1357 Recommendations Regarding Active Queue Management", 1358 BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, 1359 . 1361 [RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network 1362 Virtualization Using Generic Routing Encapsulation", 1363 RFC 7637, DOI 10.17487/RFC7637, September 2015, 1364 . 1366 [RFC7713] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) 1367 Concepts, Abstract Mechanism, and Requirements", RFC 7713, 1368 DOI 10.17487/RFC7713, December 2015, 1369 . 1371 [RFC7780] Eastlake 3rd, D., Zhang, M., Perlman, R., Banerjee, A., 1372 Ghanwani, A., and S. Gupta, "Transparent Interconnection 1373 of Lots of Links (TRILL): Clarifications, Corrections, and 1374 Updates", RFC 7780, DOI 10.17487/RFC7780, February 2016, 1375 . 1377 [RFC8084] Fairhurst, G., "Network Transport Circuit Breakers", 1378 BCP 208, RFC 8084, DOI 10.17487/RFC8084, March 2017, 1379 . 1381 [UTRAN] 3GPP, "UTRAN Overall Description", Technical 1382 Specification TS 25.401. 1384 Appendix A. Outstanding Document Issues 1386 1. [GF] Concern that certain guidelines warrant a MUST (NOT) rather 1387 than a SHOULD (NOT). Given the guidelines say that if any SHOULD 1388 (NOT)s are not followed, a strong justification will be needed, 1389 they have been left as SHOULD (NOT) pending further list 1390 discussion. In particular: 1392 * If inner is a Not-ECN-PDU and Outer is CE (or highest severity 1393 congestion level), MUST (not SHOULD) drop? 1395 This issue has been addressed by explaining when SHOULD or 1396 MUST is appropriate. 1398 2. Consider whether an IETF Standard Track doc will be needed to 1399 Update the IP-in-IP protocols listed in Section 5.1--at least 1400 those that the IETF controls--and which Area it should sit under. 1402 This issue has been addressed by the production of 1403 [I-D.ietf-tsvwg-rfc6040update-shim], but this text is left 1404 outstanding until that draft is adopted. 1406 Appendix B. Changes in This Version (to be removed by RFC Editor) 1408 From ietf-08 to ietf-09 1410 * Updated para in Intro that listed all the IP-in-IP tunnelling 1411 protocols, to instead refer to draft-ietf-tsvwg-rfc6040update- 1412 shim 1414 * Updated section 5.1 on "IP-in-IP tunnels with Shim Headers" to 1415 summarize guidance that has evolved as rfc6040update-shim has 1416 developed. 1418 From ietf-07 to ietf-08: Refreshed to avoid expiry. Updated 1419 references. 1421 From ietf-06 to ietf-07: 1423 * Added the people involved in liaisons to the acknowledgements. 1425 From ietf-05 to ietf-06: 1427 * Introduction: Added GUE and Geneve as examples of tightly 1428 coupled shims between IP headers that cite RFC 6040. And added 1429 VXLAN to list of those that do not. 1431 * Replaced normative text about tightly coupled shims between IP 1432 headers, with reference to new draft-ietf-tsvwg-rfc6040update- 1433 shim 1435 * Wire Protocol Design: Indication of ECN Support: Added TRILL as 1436 an example of a well-design protocol that does not need an 1437 indication of ECN support in the wire protocol. 1439 * Encapsulation Guidelines: In the case of a Not-ECN-PDU with a 1440 CE outer, replaced SHOULD be dropped, with explanations of when 1441 SHOULD or MUST are appropriate. 1443 * Feed-Up-and-Forward Mode: Explained examples more carefully, 1444 referred to PDCP and cited UTRAN spec as well as E-UTRAN. 1446 * Updated references. 1448 * Marked open issues as resolved, but did not delete Open Issues 1449 Appendix (yet). 1451 From ietf-04 to ietf-05: 1453 * Explained why tightly coupled shim headers only "SHOULD" comply 1454 with RFC 6040, not "MUST". 1456 * Updated references 1458 From ietf-03 to ietf-04: 1460 * Addressed Richard Scheffenegger's review comments: primarily 1461 editorial corrections, and addition of examples for clarity. 1463 From ietf-02 to ietf-03: 1465 * Updated references, ad cited RFC4774. 1467 From ietf-01 to ietf-02: 1469 * Added Section for guidelines that are applicable in all cases. 1471 * Updated references. 1473 From ietf-00 to ietf-01: Updated references. 1475 From briscoe-04 to ietf-00: Changed filename following tsvwg 1476 adoption. 1478 From briscoe-03 to 04: 1480 * Re-arranged the introduction to describe the purpose of the 1481 document first before introducing ECN in more depth. And 1482 clarified the introduction throughout. 1484 * Added applicability to 3GPP TS 36.300. 1486 From briscoe-02 to 03: 1488 * Scope section: 1490 + Added dependence on correct propagation of traffic class 1491 information 1493 + For the feed-backward mode, deemed multicast and anycast out 1494 of scope 1496 * Ensured all guidelines referring to subnet technologies also 1497 refer to tunnels and vice versa by adding applicability 1498 sentences at the start of sections 4.1, 4.2, 4.3, 4.4, 4.6 and 1499 5. 1501 * Added Security Considerations on ensuring congestion signal 1502 fields are classed as immutable and on using end-to-end 1503 congestion signal integrity technologies rather than hop-by- 1504 hop. 1506 From briscoe-01 to 02: 1508 * Added authors: JK & PT 1510 * Added 1512 + Section 4.1 "IP-in-IP Tunnels with Tightly Coupled Shim 1513 Headers" 1515 + Section 4.5 "Sequences of Similar Tunnels or Subnets" 1517 + roadmap at the start of Section 4, given the subsections 1518 have become quite fragmented. 1520 + Section 9 "Conclusions" 1522 * Clarified why transports are starting to be able to saturate 1523 interior links 1525 * Under Section 1.1, addressed the question of alternative signal 1526 semantics and included multicast & anycast. 1528 * Under Section 3.1, included a 3GPP example. 1530 * Section 4.2. "Wire Protocol Design": 1532 + Altered guideline 2. to make it clear that it only applies 1533 to the immediate subnet egress, not later ones 1535 + Added a reminder that it is only necessary to check that ECN 1536 propagates at the egress, not whether interior nodes mark 1537 ECN 1539 + Added example of how QCN uses 802.1p to indicate support for 1540 QCN. 1542 * Added references to Appendix C of RFC6040, about monitoring the 1543 amount of congestion signals introduced within a tunnel 1545 * Appendix A: Added more issues to be addressed, including plan 1546 to produce a standards track update to IP-in-IP tunnel 1547 protocols. 1549 * Updated acks and references 1551 From briscoe-00 to 01: 1553 * Intended status: BCP (was Informational) & updates 3819 added. 1555 * Briefer Introduction: Introductory para justifying benefits of 1556 ECN. Moved all but a brief enumeration of modes of operation 1557 to their own new section (from both Intro & Scope). Introduced 1558 incr. deployment as most tricky part. 1560 * Tightened & added to terminology section 1562 * Structured with Modes of Operation, then Guidelines section for 1563 each mode. 1565 * Tightened up guideline text to remove vagueness / passive voice 1566 / ambiguity and highlight main guidelines as numbered items. 1568 * Added Outstanding Document Issues Appendix 1570 * Updated references 1572 Authors' Addresses 1574 Bob Briscoe 1575 Simula Research Laboratory 1576 UK 1578 EMail: ietf@bobbriscoe.net 1579 URI: http://bobbriscoe.net/ 1581 John Kaippallimalil 1582 Huawei 1583 5340 Legacy Drive, Suite 175 1584 Plano, Texas 75024 1585 USA 1587 EMail: john.kaippallimalil@huawei.com 1589 Pat Thaler 1590 Broadcom Corporation 1591 5025 Keane Drive 1592 Carmichael, CA 95608 1593 USA 1595 EMail: pthaler@broadcom.com