idnits 2.17.1 draft-rtg-dt-encap-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 278: '...italized keyword MUST is used as defin...' RFC 2119 keyword, line 351: '...lancing procedure MUST choose the same...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 9, 2015) is 3308 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'I-D.xu-bier-encapsulation' is defined on line 1612, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2460 (Obsoleted by RFC 8200) ** Obsolete normative reference: RFC 5405 (Obsoleted by RFC 8085) ** Obsolete normative reference: RFC 6830 (Obsoleted by RFC 9300, RFC 9301) == Outdated reference: A later version (-05) exists of draft-farrelll-mpls-opportunistic-encrypt-04 == Outdated reference: A later version (-02) exists of draft-herbert-remotecsumoffload-01 == Outdated reference: A later version (-08) exists of draft-ietf-nvo3-arch-02 == Outdated reference: A later version (-11) exists of draft-ietf-sfc-architecture-07 == Outdated reference: A later version (-15) exists of draft-ietf-tsvwg-circuit-breaker-00 == Outdated reference: A later version (-19) exists of draft-ietf-tsvwg-gre-in-udp-encap-05 == Outdated reference: A later version (-11) exists of draft-ietf-tsvwg-port-use-07 == Outdated reference: A later version (-12) exists of draft-saldana-tsvwg-simplemux-02 == Outdated reference: A later version (-08) exists of draft-sridharan-virtualization-nvgre-07 == Outdated reference: A later version (-04) exists of draft-wei-tsvwg-tunnel-congestion-feedback-03 == Outdated reference: A later version (-06) exists of draft-xu-bier-encapsulation-02 Summary: 5 errors (**), 0 flaws (~~), 13 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RTGWG E. Nordmark (ed) 3 Internet-Draft Arista Networks 4 Intended status: Informational A. Tian 5 Expires: September 10, 2015 Ericsson Inc. 6 J. Gross 7 VMware 8 J. Hudson 9 Brocade Communications Systems, 10 Inc. 11 L. Kreeger 12 Cisco Systems, Inc. 13 P. Garg 14 Microsoft 15 P. Thaler 16 Broadcom Corporation 17 T. Herbert 18 Google 19 March 9, 2015 21 Encapsulation Considerations 22 draft-rtg-dt-encap-01 24 Abstract 26 The IETF Routing Area director has chartered a design team to look at 27 common issues for the different data plane encapsulations being 28 discussed in the NVO3 and SFC working groups and also in the BIER 29 BoF, and also to look at the relationship between such encapsulations 30 in the case that they might be used at the same time. The purpose of 31 this design team is to discover, discuss and document considerations 32 across the different encapsulations in the different WGs/BoFs so that 33 we can reduce the number of wheels that need to be reinvented in the 34 future. 36 Status of this Memo 38 This Internet-Draft is submitted in full conformance with the 39 provisions of BCP 78 and BCP 79. 41 Internet-Drafts are working documents of the Internet Engineering 42 Task Force (IETF). Note that other groups may also distribute 43 working documents as Internet-Drafts. The list of current Internet- 44 Drafts is at http://datatracker.ietf.org/drafts/current/. 46 Internet-Drafts are draft documents valid for a maximum of six months 47 and may be updated, replaced, or obsoleted by other documents at any 48 time. It is inappropriate to use Internet-Drafts as reference 49 material or to cite them other than as "work in progress." 51 This Internet-Draft will expire on September 10, 2015. 53 Copyright Notice 55 Copyright (c) 2015 IETF Trust and the persons identified as the 56 document authors. All rights reserved. 58 This document is subject to BCP 78 and the IETF Trust's Legal 59 Provisions Relating to IETF Documents 60 (http://trustee.ietf.org/license-info) in effect on the date of 61 publication of this document. Please review these documents 62 carefully, as they describe your rights and restrictions with respect 63 to this document. Code Components extracted from this document must 64 include Simplified BSD License text as described in Section 4.e of 65 the Trust Legal Provisions and are provided without warranty as 66 described in the Simplified BSD License. 68 Table of Contents 70 1. Design Team Charter . . . . . . . . . . . . . . . . . . . . . 4 71 2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 72 3. Common Issues . . . . . . . . . . . . . . . . . . . . . . . . 6 73 4. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 74 5. Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . 7 75 6. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 7 76 7. Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 77 8. Next-protocol indication . . . . . . . . . . . . . . . . . . . 9 78 9. MTU and Fragmentation . . . . . . . . . . . . . . . . . . . . 10 79 10. OAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 80 11. Security Considerations . . . . . . . . . . . . . . . . . . . 12 81 11.1. Encapsulation-specific considerations . . . . . . . . . . 12 82 11.2. Virtual network isolation . . . . . . . . . . . . . . . . 14 83 11.3. Packet level security . . . . . . . . . . . . . . . . . . 15 84 12. QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 85 13. Congestion Considerations . . . . . . . . . . . . . . . . . . 16 86 14. Header Protection . . . . . . . . . . . . . . . . . . . . . . 18 87 15. Extensibility Considerations . . . . . . . . . . . . . . . . . 19 88 15.1. Next-protocol . . . . . . . . . . . . . . . . . . . . . . 22 89 16. Layering Considerations . . . . . . . . . . . . . . . . . . . 22 90 17. Service model . . . . . . . . . . . . . . . . . . . . . . . . 23 91 18. Hardware Friendly . . . . . . . . . . . . . . . . . . . . . . 24 92 18.1. Considerations for NIC offload . . . . . . . . . . . . . 26 93 19. Middlebox Considerations . . . . . . . . . . . . . . . . . . . 29 94 20. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . 30 95 21. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 31 96 22. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 31 97 23. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32 98 23.1. Normative References . . . . . . . . . . . . . . . . . . 32 99 23.2. Informative References . . . . . . . . . . . . . . . . . 33 100 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 36 102 1. Design Team Charter 104 There have been multiple efforts over the years that have resulted in 105 new or modified data plane behaviors involving encapsulations. That 106 includes IETF efforts like MPLS, LISP, and TRILL but also industry 107 efforts like VXLAN and NVGRE. These collectively can be seen as a 108 source of insight into the properties that data planes need to meet. 109 The IETF is currently working on potentially new encapsulations in 110 NVO3 and SFC and considering working on BIER. In addition there is 111 work on tunneling in the INT area. 113 This is a short term design team chartered to collect and construct 114 useful advice to parties working on new or modified data plane 115 behaviors that include additional encapsulations. The goal is for 116 the group to document useful advice gathered from interacting with 117 ongoing efforts. An Internet Draft will be produced for IETF92 to 118 capture that advice, which will be discussed in RTGWG. 120 Data plane encapsulations face a set of common issues such as: 121 o How to provide entropy for ECMP 122 o Issues around packet size and fragmentation/reassembly 123 o OAM - what support is needed in an encapsulation format? 124 o Security and privacy. 125 o QoS 126 o Congestion Considerations 127 o IPv6 header protection (zero UDP checksum over IPv6 issue) 128 o Extensibility - e.g., for evolving OAM, security, and/or 129 congestion control 130 o Layering of multiple encapsulations e.g., SFC over NVO3 over BIER 131 The design team will provide advice on those issues. The intention 132 is that even where we have different encapsulations for different 133 purposes carrying different information, each such encapsulation 134 doesn't have to reinvent the wheel for the above common issues. 136 The design team will look across the routing area in particular at 137 SFC, NVO3 and BIER. It will not be involved in comparing or 138 analyzing any particular encapsulation formats proposed in those WGs 139 and BoFs but instead focus on common advice. 141 2. Overview 143 The references provide background information on NVO3, SFC, and BIER. 144 In particular, NVO3 is introduced in [RFC7364], [RFC7365], and 145 [I-D.ietf-nvo3-arch]. SFC is introduced in 146 [I-D.ietf-sfc-architecture] and [I-D.ietf-sfc-problem-statement]. 147 Finally, the information on BIER is in 148 [I-D.shepherd-bier-problem-statement], 150 [I-D.wijnands-bier-architecture], and 151 [I-D.wijnands-mpls-bier-encapsulation]. We assume the reader has 152 some basic familiarity with those proposed encapsulations. The 153 Related Work section points at some prior work that relates to the 154 encapsulation considerations in this document. 156 Encapsulation protocols typically have some unique information that 157 they need to carry. In some cases that information might be modified 158 along the path and in other cases it is constant. The in-flight 159 modifications has impacts on what it means to provide security for 160 the encapsulation headers. 161 o NVO3 carries a VNI Identifier edge to edge which is not modified. 162 There has been OAM discussions in the WG and it isn't clear 163 whether some of the OAM information might be modified in flight. 164 o SFC carries service meta-data which might be modified or 165 unmodified as the packets follow the service path. SFC talks of 166 some loop avoidance mechanism which is likely to result in 167 modifications for for each hop in the service chain even if the 168 meta-data is unmodified. 169 o BIER carries a bitmap of egress ports to which a packet should be 170 delivered, and as the packet is forwarded down different paths 171 different bits are cleared in that bitmap. 173 Even if information isn't modified in flight there might be devices 174 that wish to inspect that information. For instance, one can 175 envision future NVO3 security devices which filter based on the 176 virtual network identifier. 178 The need for extensibility is different across the protocols 179 o NVO3 might need some extensions for OAM and security. 180 o SFC is all about carrying service meta-data along a path, and 181 different services might need different types and amount of meta- 182 data. 183 o BIER might need variable number of bits in their bitmaps, or other 184 future schemes to scale up to larger network. 185 The extensibility needs and constraints might be different when 186 considering hardware vs. software implementations of the 187 encapsulation headers. NIC hardware might have different constraints 188 than switch hardware. 190 As the IETF designs these encapsulations the different WGs solve the 191 issues for their own encapsulation. But there are likely to be 192 future cases when the different encapsulations are combined in the 193 same header. For instance, NVO3 might be a "transport" used to carry 194 SFC between the different hops in the service chain. 196 Most of the issues discussed in this document are not new. The IETF 197 and industry as specified and deployed many different encapsulation 198 or tunneling protocols over time, ranging from simple IP-in-IP and 199 GRE encapsulation, IPsec, pseudo-wires, session-based approached like 200 L2TP, and the use of MPLS control and data planes. IEEE 802 has also 201 defined layered encapsulation for Provider Backbone Bridges (PBB) and 202 IEEE 802.1Qbp (ECMP). This document tries to leverage what we 203 collectively have learned from that experience and summarize what 204 would be relevant for new encapsulations like NVO3, SFC, and BIER. 206 3. Common Issues 208 [This section is mostly a repeat of the charter but with a few 209 modifications and additions.] 211 Any new encapsulation protocol would need to address a large set of 212 issues that are not central to the new information that this protocol 213 intends to carry. The common issues explored in this document are: 214 o How to provide entropy for Equal Cost MultiPath (ECMP) routing 215 o Issues around packet size and fragmentation/reassembly 216 o Next header indication - each encapsulation might be able to carry 217 different payloads 218 o OAM - what support is needed in an encapsulation format? 219 o Security and privacy 220 o QoS 221 o Congestion Considerations 222 o Header protection 223 o Extensibility - e.g., for evolving OAM, security, and/or 224 congestion control 225 o Layering of multiple encapsulations e.g., SFC over NVO3 over BIER 226 o Importance of being friendly to hardware and software 227 implementations 229 4. Scope 231 It is important to keep in mind what we are trying to cover and not 232 cover in this document and effort. This is 233 o A look across the three new encapsulations, while taking lots of 234 previous work into account 235 o Focus on the class of encapsulations that would run over IP/UDP. 236 That was done to avoid being distracted by the data-plane and 237 control-plane interaction, which is more significant for protocols 238 that are designed to run over "transports" that maintain session 239 or path state. 240 o We later expanded the scope somewhat to consider how the 241 encapsulations would play with MPLS "transport", which is 242 important because SFC and BIER seem to target being independent of 243 the underlying "transport" 245 However, this document and effort is NOT intended to: 246 o Design some new encapsulation header to rule them all 247 o Design yet another new NVO3 encapsulation header 248 o Try to select the best encapsulation header 249 o Evaluate any existing and proposed encapsulations 251 5. Assumptions 253 The design center for the new encapsulations is a well-managed 254 network. That network can be a datacenter network (plus datacenter 255 interconnect) or a service provider network. Based on the existing 256 and proposed encapsulations in those environment it is reasonable to 257 make these assumptions: 258 o The MTU is carefully managed and configured. Hence an 259 encapsulation protocol can make the packets bigger without 260 resulting in a requirement for fragmentation and reassembly 261 between ingress and egress. (However, it might be useful to 262 detecting MTU misconfigurations.) 263 o In general an encapsulation needs some approach for congestion 264 management. But the assumptions are different than for arbitrary 265 Internet paths in that the underlay might be well-provisioned and 266 better policed at the edge, and due to multi-tenancy, the 267 congestion control in the endpoints might be even less trusted 268 than on the Internet at large. 270 The goal is to implement these encapsulations in hardware and 271 software hence we can't assume that the needs of either 272 implementation approach can trump the needs of the other. In 273 particular, around extensibility the needs and constraints might be 274 quite different. 276 6. Terminology 278 The capitalized keyword MUST is used as defined in 279 http://en.wikipedia.org/wiki/Julmust 281 TBD: Refer to existing documents for at least NVO3 and SFC 282 terminology. We use at least the VNI ID in this document. 284 7. Entropy 286 In many cases the encapsulation format needs to enable ECMP in 287 unmodified routers. Those routers might use different fields in TCP/ 288 UDP packets to do ECMP without a risk of reordering a flow. 290 The common way to do ECMP-enabled encapsulation over IP today is to 291 add a UDP header and to use UDP with the UDP source port carrying 292 entropy from the inner/original packet headers as in LISP [RFC6830]. 293 The total entropy consists of 14 bits in the UDP source port (using 294 the ephemeral port range) plus the outer IP addresses which seems to 295 be sufficient for entropy; using outer IPv6 headers would give the 296 option for more entropy should it be needed in the future. 298 In some environments it might be fine to use all 16 bits of the port 299 range. However, middleboxes might make assumptions about the system 300 ports or user ports. But they should not make any assumptions about 301 the ports in the Dynamic and/or Private Port range, which have the 302 two MSBs set to 11b. 304 The UDP source port might change over the lifetime of an encapsulated 305 flow, for instance for DoS mitigation or re-balancing load across 306 ECMP. 308 There is some interaction between entropy and OAM and extensibility 309 mechanism. It is desirable to be able to send OAM packets to follow 310 the same path as network packets. Hence OAM packets should use the 311 same entropy mechanism as data packets. While routers might use 312 information in addition the entropy field and outer IP header, they 313 can not use arbitrary parts of the encapsulation header since that 314 might result in OAM frames taking a different path. Likewise if 315 routers look past the encapsulation header they need to be aware of 316 the extensibility mechanism(s) in the encapsulation format to be able 317 to find the inner headers in the presence of extensions; OAM frames 318 might use some extensions e.g. for timestamps. 320 Architecturally the entropy and the next header field are really part 321 of enclosing delivery header. UDP with entropy goes hand-in-hand 322 with the outer IP header. Thus the UDP entropy is present for the 323 underlay IP routers the same way that an MPLS entropy label is 324 present for LSRs. The entropy above is all about providing entropy 325 for the outer delivery of the encapsulated packets. 327 It has been suggested that when IPv6 is used it would not be 328 necessary to add a UDP header for entropy, since the IPv6 flow label 329 can be used for entropy. (This assumes that there is an IP protocol 330 number for the encaps in addition to a UDP destination port number 331 since UDP would be used with IPv4 underlay. And any use of UDP 332 checksums would need to be replaced by an encaps-specific checksum or 333 secure hash.) While such an approach would save 8 bytes of headers 334 when the underlay is IPv6, it does assume that the underlay routers 335 use the flow label for ECMP, and it also would make the IPv6 approach 336 different than the IPv4 approach. Currently the leaning is towards 337 recommending using the UDP encaps for both IPv4 and IPv6 underlay. 339 The IPv6 flow label can be used for additional entropy if need be. 341 Note that in the proposed BIER encapsulation 342 [I-D.wijnands-mpls-bier-encapsulation], there is an an 8-bit field 343 which specifies an entropy value that can be used for load balancing 344 purposes. This entropy is for the BIER forwarding decisions, which 345 is independent of any outer delivery ECMP between BIER routers. Thus 346 it is not part of the delivery ECMP discussed in this section. 347 [Note: For any given bit in BIER (that identifies an exit from the 348 BIER domain) there might be multiple immediate next hops. The 349 BIER entropy field is used to select that next hop as part of BIER 350 processing. The BIER forwarding process may do equal cost load 351 balancing, but the load balancing procedure MUST choose the same 352 path for any two packets have the same entropy value.] 354 8. Next-protocol indication 356 The transport delivery mechanism for the encapsulations we discuss in 357 this document need some way to indicate which encapsulation header 358 (or other payload) comes next in the packet. Some encapsulations 359 might be identified by a UDP port; others might be identified by an 360 Ethernet type or IP protocol number. Which approach is used is a 361 function of the preceding header the same was as IPv4 being 362 identified by both an Ethernet type and an IP protocol number (for 363 IP-in-IP). In some cases the header type is implicit in some session 364 (L2TP) or path (MPLS) setup. But this is largely beyond the control 365 of the encapsulation protocol. For instance, if there is a 366 requirement to carry the encapsulation after an Ethernet header, then 367 an Ethernet type is needed. If required to be carried after an IP/ 368 UDP header, then a UDP port number is needed. 370 The encapsulation needs to indicate the type of its payload, which is 371 in scope for the design of the encapsulation. We have existing 372 protocols which use Ethernet types (such as GRE). Here each 373 encapsulation header can potentially makes its own choices between: 374 o Reuse Ethernet types - makes it easy to carry existing L2 and L3 375 protocols 376 o Reuse IP protocol numbers - makes it easy to carry e.g., ESP but 377 brings in all existing protocol numbers many of which would never 378 be used directly on top of the encapsulation protocol. 379 o Define their own next-protocol number space, which can use fewer 380 bits than an Ethernet type and give more flexibility, but at the 381 cost of administering that numbering space. 383 If the IETF ends up defining multiple encapsulations at about the 384 same time, and there is some chance that multiple such encapsulations 385 can be combined in the same packet, there is a question whether it 386 makes sense to use a common approach and numbering space for the 387 encapsulation across the different protocols. A common approach 388 might not be beneficial as long as there is only one way to indicate 389 e.g., SFC inside NVO3. 391 9. MTU and Fragmentation 393 A common approach today is to assume that the underlay have 394 sufficient MTU to carry the encapsulated packets without any 395 fragmentation and reassembly at the tunnel endpoints. That is 396 sufficient when the operator of the ingress and egress have full 397 control of the paths between those endpoints. And it makes for 398 simpler (hardware) implementations if fragmentation and reassembly 399 can be avoided. 401 However, even under that assumption it would be beneficial to be able 402 to detect when there is some misconfiguration causing packets to be 403 dropped due to MTU issues. One way to do this is to have the 404 encapsulator set the don't-fragment (DF) flag in the outer IPv4 405 header and receive and log any received ICMP "packet too big" (PTB) 406 errors. Note that no flag needs to be set in an outer IPv6 header 407 [RFC2460]. 409 Encapsulations could also define an optional tunnel fragmentation and 410 reassembly mechanism which would be useful in the case when the 411 operator doesn't have full control of the path. Such a mechanism 412 would be required if the underlay might have a path MTU which makes 413 it impossible to carry at least 1518 bytes (if offering Ethernet 414 service), or at least 1280 (if offering IPv6 service). The use of 415 such a protocol mechanism could be triggered by receiving a PTB. But 416 such a mechanism might not be implemented by all encaps and decaps 417 nodes. [Aerolink is one example of such a protocol.] 419 Depending on the payload carried by the encapsulation there are some 420 additional possibilities: 421 o If payload is IPv4/6 then the underlay path MTU could be used to 422 report end-to-end path MTU. 423 o If the payload service is Ethernet/L2, then there is no such per 424 destination reporting mechanism. However, there is a LLDP TLV for 425 reporting max frame size; might be useful to report minimum to end 426 stations, but unmodified end stations would do nothing with that 427 TLV since they assume that the MTU is at least 1518. 429 10. OAM 431 The OAM area is seeing active development in the IETF with 432 discussions (at least) in NVO3 and SFC working groups, plus the new 433 LIME WG looking at architecture and YANG models. 435 The design team has take a narrow view of OAM to explore the 436 potential OAM implications on the encapsulation format. 438 In terms of what we have heard from the various working groups there 439 seem to be needs to: 440 o Be able to send out-of-band OAM messages - that potentially should 441 follow the same path through the network as some flow of data 442 packets. 443 * Such OAM messages should not accidentally be decapsulated and 444 forwarded to the end stations. 445 * Be able to add OAM information to data packets that are 446 encapsulated. Discussions have been around 447 * Using a bit in the OAM to synchronize sampling of counters 448 between the encapsulator and decapsulator. 449 * Optional timestamps, sequence numbers, etc for more detailed 450 measurements between encapsulator and decapsulator. 451 o Usable for both proactive monitoring (akin to BFD) and reactive 452 checks (akin to traceroute to pin-point a failure) 454 To ensure that the OAM messages can follow the same path the OAM 455 messages need to get the same ECMP (and LAG hashing) results as a 456 given data flow. An encaps can choose between one of: 457 o Limit ECMP hashing to not look past the UDP header i.e. the 458 entropy needs to be in the source/destination IP and UDP ports 459 o Make OAM packets look the same as data packets i.e. the initial 460 part of the OAM payload has the inner Ethernet, IP, TCP/UDP 461 headers as a payload. (This approach was taken in TRILL out of 462 necessity since there is no UDP header.) OAM bit in encaps must 463 in any case be excluded from the entropy. 465 There can be several ways to prevent OAM packets from accidentally 466 being forwarded to hosts using: 467 o A bit in the frame (as in TRILL) indicating OAM 468 o A next protocol indication with a designated value for "none" or 469 "oam". 470 This assumes that the bit or next protocol, respectively, would not 471 affect entropy/ECMP in the underlay. 473 There has been suggestions that one (or more) marker bits in the 474 encaps header would be useful in order to delineate measurement 475 epochs on the encapsulator and decapsulator and use that to compare 476 counters to determine packet loss. 478 A result of the above is that OAM is likely to evolve and needs some 479 degree of extensibility from the encapsulation format; a bit or two 480 plus the ability to define additional larger extensions. 482 An open question is how to handle error messages or other reports 483 relating to OAM. One can think if such reporting as being associated 484 with the encaps the same way ICMP is associated with IP. Would it 485 make sense for the IETF to develop a common Encaps Error Reporting 486 Protocol as part of OAM, which can be used for different 487 encapsulations? And if so, what are the technical challenges. For 488 instance, how to avoid it being filtered as ICMP often is? 490 A potential additional consideration for OAM is the possible future 491 existence of gateways that "stitch" together different dataplane 492 encapsulations and might want to carry OAM end-to-end across the 493 different encapsulations. 495 11. Security Considerations 497 Different encapsulation use cases will have different requirements 498 around security. For instance, when encapsulation is used to build 499 overlay networks for network virtualization, isolation between 500 virtual networks may be paramount. BIER support of multicast may 501 entail different security requirements than encapsulation for 502 unicast. 504 In real deployment, the security of the underlying network may be 505 considered for determining the level of security needed in the 506 encapsulation layer. However for the purposes of this discussion, we 507 assume that network security is out of scope and that the underlying 508 network does not itself provide adequate or as least uniform security 509 mechanisms for encapsulation. 511 There are at least three considerations for security: 512 o Anti-spoofing/virtual network isolation 513 o Interaction with packet level security such as IPsec 514 o Privacy (e.g., VNI ID confidentially for NVO3) 516 This section uses a VNI ID in NVO3 as an example. A SFC or BIER 517 encapsulation is likely to have fields with similar security and 518 privacy requirements. 520 11.1. Encapsulation-specific considerations 522 Some of these considerations appear for a new encapsulation, and 523 others are more specific to network virtualization in datacenters. 524 o New attack vectors: 526 * DDOS on specific queued/paths by attempting to reproduce the 527 5-tuple hash for targeted connections. 528 * Entropy in outer 5-tuple may be too little or predictable. 529 * Leakage of identifying information in the encapsulation header 530 for an encrypted payload. 531 * Vulnerabilities of using global values in fields like VNI ID. 532 o Trusted versus untrusted tenants in network virtualization: 533 * The criticality of virtual network isolation depends on whether 534 tenants are trusted or untrusted. In the most extreme cases, 535 tenants might not only be untrusted but may be considered 536 hostile. 537 * For a trusted set of users (e.g. a private cloud) it may be 538 sufficient to have just a virtual network identifier to provide 539 isolation. Packets inadvertently crossing virtual networks 540 should be dropped similar to a TCP packet with a corrupted port 541 being received on the wrong connection. 542 * In the presence of untrusted users (e.g. a public cloud) the 543 virtual network identifier must be adequately protected against 544 corruption and verified for integrity. This case may warrant 545 keyed integrity. 546 o Different forms of isolation: 547 * Isolation could be blocking all traffic between tenants (or 548 except as allowed by some firewall) 549 * Could also be about performance isolation i.e. one tenant can 550 overload the network in a way that affects other tenants 551 * Physical isolation of traffic for different tenants in network 552 may be required, as well as required restrictions that tenants 553 may have on where their packets may be routed. 554 o New attack vectors from untrusted tenants: 555 * Third party VMs with untrusted tenants allows internally borne 556 attacks within data centers 557 * Hostile VMs inside the system may exist (e.g. public cloud) 558 * Internally launched DDOS 559 * Passive snooping for mis-delivered packets 560 * Mitigate damage and detection in event that a VM is able to 561 circumvent isolation mechanisms 562 o Tenant-provider relationship: 563 * Tenant might not trust provider, hypervisors, network 564 * Provider likely will need to provide SLA or a least a statement 565 on security 566 * Tenant may implement their own additional layers of security 567 * Regulation and certification consuderations 568 o Trend towards tighter security: 569 * Tenants' data in network increases in volume and value, attacks 570 become more sophisticated 571 * Large DCs already encrypt everything on disk 572 * DCs likely to encrypt inter-DC traffic at this point, use TLS 573 to Internet. 574 * Encryption within DC is becoming more commonplace, becomes 575 ubiquitous when cost is low enough. 576 * Cost/performance considerations. Cost of support for strong 577 security has made strong network security in DCs prohibitive. 578 * Are there lessons from MacSec? 580 11.2. Virtual network isolation 582 The first requirement is isolation between virtual networks. Packets 583 sent in one virtual network should never be illegitimately received 584 by a node in another virtual network. Isolation should be protected 585 in the presence of malicious attacks or inadvertent packet 586 corruption. 588 The second requirement is sender authentication. Sender identity is 589 authenticated to prevent anti-spoofing. Even if an attacker has 590 access to the packets in the network, they cannot send packets into a 591 virtual network. This may have two possibilities: 592 o Pairwise sender authentication. Any two communicating hosts 593 negotiate a shared key. 594 o Group authentication. A group of hosts share a key (this may be 595 more appropriate for multicast of encapsulation). 597 Possible security solutions: 598 o Security cookie: This is similar to L2TP cookie mechanism 599 [RFC3931]. A shared plain text cookie is shared between 600 encapsulator and decapsulator. A receiver validates a packet by 601 evaluating if the cookie is correct for the virtual network and 602 address of a sender. Validation function is F(cookie, VNI ID, 603 source addr). If cookie matches, accept packet, else drop. Since 604 cookie is plain text this method does not protect against an 605 eavesdropping. Cookies are set and may be rotated out of band. 606 o Secure hash: This is a stronger mechanism than simple cookies that 607 borrows from IPsec and PPP authentication methods. In this model 608 security field contains a secure hash of some fields in the packet 609 using a shared key. Hash function may be something like H(key, 610 VNI ID, addrs, salt). The salt ensures the hash is not the same 611 for every packet, and if it includes a sequence number may also 612 protect against replay attacks. 614 In any use of a shared key, periodic re-keying should be allowed. 615 This could include use of techniques like generation numbers, key 616 windows, etc. See [I-D.farrelll-mpls-opportunistic-encrypt] for an 617 example application. 619 We might see firewalls that are aware of the encaps and can provide 620 some defense in depth combined with the above example anti-spoofing 621 approaches. An example would be an NVO3-aware firewall being able to 622 check the VNI ID. 624 Separately and in addition to such filtering, there might be a desire 625 to completely block an encapsulation protocol at certain places in 626 the network, e.g., at the edge of a datacenter. Using a fixed 627 standard UDP destination port number for each encapsulation protocol 628 would facilitate such blocking. 630 11.3. Packet level security 632 An encapsulated packet may itself be encapsulated in IPsec (e.g. 633 ESP). This should be straightforward and in fact is what would 634 happen today in security gateways. In this case, there is no special 635 consideration for the fact that packet is encapsulated, however since 636 the encapsulation layer headers are included (part of encrypted data 637 for instance) we lose visibility in the network of the encapsulation. 639 The more interesting case is when security is applied to the 640 encapsulation payload. This will keep the encapsulation headers in 641 the outer header and visible to the network (for instance in nvo3 we 642 may want to firewall based on VNI ID even if packet is encrypted). 643 In this model protocol stack may be something like 644 IP|UDP|Encap|ESP|IP in tunnel mode, but there's nothing that prevents 645 using transport mode (this looks a lot like ESP/UDP [RFC3948]). The 646 encapsulation and security are probably done together at encapsulator 647 and resolved at decapsulator. Since the encapsulation header is 648 outside of the security coverage, this may itself require security 649 also (like described above). 651 In both of the above the security associations (SAs) may be between 652 physical hosts, so for instance in nvo3 we can have packets of 653 different virtual networks using the same SA-- this should not be an 654 issue since it is the VNI ID that ensures isolation (which needs to 655 be secured also). In this case of security applied to encap payload, 656 this does present a bit of protocol layer inversion in the header 657 (encapsulation refers to overlay, but ESP operates on underlay), but 658 this should be okay as long as semantics are clear and processing is 659 deterministic. 661 12. QoS 663 In the Internet architecture we support QoS using the Differentiated 664 Services Code Points (DSCP) in the formerly named Type-of-Service 665 field in the IPv4 header, and in the Traffic-Class field in the IPv6 666 header. The ToS and TC fields also contain the two ECN bits. 668 We have existing specifications how to process those bits. See 669 [RFC2983] for diffserv handling, which specifies how the received 670 DSCP value is used to set the DSCP value in an outer IP header when 671 encapsulating. (There are also existing specifications how DSCP can 672 be mapped to layer2 priorities.) 674 Those specifications apply whether or not there is some intervening 675 headers (e.g., for NVO3 or SFC) between the inner and outer IP 676 headers. Thus the encapsulation considerations in this area are 677 mainly about applying the framework in [RFC2983]. 679 There are some other considerations specific to doing OAM for 680 encapsulations. If OAM messages are used to measure latency, it 681 would make sense to treat them the same as data payloads. Thus they 682 need to have the same outer DSCP value as the data packets which they 683 wish to measure. 685 Due to OAM there are constraints on middleboxes in general. If 686 middleboxes inspect the packet past the outer IP+UDP and encaps 687 header and look for inner IP and TCP/UDP headers, that might violate 688 the assumption that OAM packets will be handled the same as regular 689 data packets. That issue is broader than just QoS - applies to 690 firewall filters etc. 692 13. Congestion Considerations 694 Additional encapsulation headers does not introduce anything new for 695 Explicit Congestion Notification. It is just like IP-in-IP and IPsec 696 tunnels which is specified in [RFC6040] in terms of how the ECN bits 697 in the inner and outer header are handled when encapsulating and 698 decapsulating packets. Thus new encapsulations can more or less 699 include that by reference. 701 There are additional considerations around carrying non-congestion 702 controlled traffic. These details have been worked out in 703 [I-D.ietf-mpls-in-udp]. As specified in [RFC5405]: "IP-based traffic 704 is generally assumed to be congestion-controlled, i.e., it is assumed 705 that the transport protocols generating IP-based traffic at the 706 sender already employ mechanisms that are sufficient to address 707 congestion on the path Consequently, a tunnel carrying IP-based 708 traffic should already interact appropriately with other traffic 709 sharing the path, and specific congestion control mechanisms for the 710 tunnel are not necessary". 712 For this reason, where an encaps is used to carry IP traffic that is 713 known to be congestion controlled, the UDP tunnels does not create an 714 additional need for congestion control. Internet IP traffic is 715 generally assumed to be congestion-controlled. Similarly, in general 716 Layer 3 VPNs are carrying IP traffic that is similarly assumed to be 717 congestion controlled. 719 However, some of the encapsulations (at least NVO3) will be able to 720 carry arbitrary Layer 2 packets to provide an L2 service, in which 721 case one can not assume that the traffic is congestion controlled. 723 One could handle this by adding some congestion control support to 724 the encapsulation header (one instance of which would end up looking 725 like DCCP). However, if the underlay is well-provisioned and managed 726 as opposed to being arbitrary Internet path, it might be sufficient 727 to have a slower reaction to congestion induced by that traffic. 728 There is work underway on a notion of "circuit breakers" for this 729 purpose. See See [I-D.ietf-tsvwg-circuit-breaker]. Encapsulations 730 which carry arbitrary Layer 2 packets want to consider that ongoing 731 work. 733 If the underlay is provisioned in such a way that it can guarantee 734 sufficient capacity for non-congestion controlled Layer 2 traffic, 735 then such circuit breakers might not be needed. 737 Two other considerations appear in the context of these 738 encapsulations as applied to overlay networks: 739 o Protect against malicious end stations 740 o Ensure fairness and/or measure resource usage across multiple 741 tenants 742 Those issues are really orthogonal to the encapsulation, in that they 743 are present even when no new encapsulation header is in use. 744 However, the application of the new encapsulations are likely to be 745 in environments where those issues are becoming more important. 746 Hence it makes sense to consider them. 748 One could make the encapsulation header be extensible to that it can 749 carry sufficient information to be able to measure resource usage, 750 delays, and congestion. The suggestions in the OAM section about a 751 single bit for counter synchronization, and optional timestamps 752 and/or sequence numbers, could be part of such an approach. There 753 might also be additional congestion-control extensions to be carried 754 in the encapsulation. Overall this results in a consideration to be 755 able to have sufficient extensibility in the encapsulation to be 756 handle to handle potential future developments in this space. 758 Coarse measurements are likely to suffice, at least for circuit- 759 breaker-like purposes, see [I-D.wei-tsvwg-tunnel-congestion-feedback] 760 and [I-D.briscoe-conex-data-centre] for examples on active work in 761 this area via use of ECN. [RFC6040] Appendix C is also relevant. 762 The outer ECN bits seem sufficient (at least when everything uses 763 ECN) to do this course measurements. Needs some more study for the 764 case when there are also drops; might need to exchange counters 765 between ingress and egress to handle drops. 767 Circuit breakers are not sufficient to make a network with different 768 congestion control when the goal is to provide a predictable service 769 to different tenants. The fallback would be to rate limit different 770 traffic. 772 14. Header Protection 774 Many UDP based encapsulations such as VXLAN [RFC7348] either 775 discourage or explicitly disallow the use of UDP checksums. The 776 reason is that the UDP checksum covers the entire payload of the 777 packet and switching ASICs are typically optimized to look at only a 778 small set of headers as the packet passes through the switch. In 779 these case, computing a checksum over the packet is very expensive. 780 (Software endpoints and the NICs used with them generally do not have 781 the same issue as they need to look at the entire packet anyways.) 783 The lack a header checksum creates the possibility that bit errors 784 can be introduced into any information carried by the new headers. 785 Specifically, in the case of IPv6, the assumption is that a transport 786 layer checksum - UDP in this case - will protect the IP addresses 787 through the inclusion of a pseudoheader in the calculation. This is 788 different from IPv4 on which many of these encapsulation protocols 789 are initially deployed which contains its own header checksum. In 790 addition to IP addresses, the encapsulation header often contains its 791 own information which is used for addressing packets or other high 792 value network functions. Without a checksum, this information is 793 potentially vulnerable - an issue regardless of whether the packet is 794 carried over IPv4 or IPv6. 796 Several protocols cite [RFC6935] and [RFC6936] as an exemption to the 797 IPv6 checksum requirements. However, these are intended to be 798 tailored to a fairly narrow set of circumstances - primarily relying 799 on sparseness of the address space to detect invalid values and well 800 managed networks - and are not a one size fits all solution. In 801 these cases, an analysis should be performed of the intended 802 environment, including the probability of errors being introduced and 803 the use of ECC memory in routing equipment. 805 Conceptually, the ideal solution to this problem is a checksum that 806 covers only the newly added headers of interest. There is little 807 value in the portion of the UDP checksum that covers the encapsulated 808 packet because that would generally be protected by other checksums 809 and this is the expensive portion to compute. In fact, this solution 810 already exists in the form of UDP-Lite and UDP based encapsulations 811 could be easily ported to run on top of it. Unfortunately, the main 812 value in using UDP as part of the encapsulation header is that it is 813 recognized by already deployed equipment for the purposes of ECMP, 814 RSS, and middlebox operations. As UDP-Lite uses a different protocol 815 number than UDP and it is not widely implemented in middleboxes, this 816 value is lost. A possible solution is to incorporate the same 817 partial-checksum concept as UDP-Lite or other header checksum 818 protection into the encapsulation header and continue using UDP as 819 the outer protocol. One potential challenge with this approach is 820 the use of NAT or other form of translation on the outer header will 821 result in an invalid checksum as the translator will not know to 822 update the encapsulation header. 824 The method chosen to protect headers is often related to the security 825 needs of the encapsulation mechanism. On one hand, the impact of a 826 poorly protected header is not limited to only data corruption but 827 can also introduce a security vulnerability in the form of 828 misdirected packets to an unauthorized recipient. Conversely, high 829 security protocols that already include a secure hash over the 830 valuable portion of the header (such as by encrypting the entire IP 831 packet using IPsec, or some secure hash of the encap header) do not 832 require additional checksum protection as the hash provides stronger 833 assurance than a simple checksum. 835 15. Extensibility Considerations 837 Protocol extensibility is the concept that a networking protocol may 838 be extended to include new use cases or functionality that were not 839 part of the original protocol specification. Extensibility may be 840 used to add security, control, management, or performance features to 841 a protocol. A solution may allow private extensions for 842 customization or experimentation. 844 Extending a protocol often implies that a protocol header must carry 845 new information. There are two usual methods to accomplish this: 846 1. Define or redefine the meaning of existing fields in a protocol 847 header. 848 2. Add new (optional) fields to the protocol header. 849 It is also possible to create a new protocol version, but this is 850 more associated with defining a protocol than extending it (IPv6 851 being a successor to IPv4 is an example of protocol versioning). 853 Many protocol definitions include some number of reserved fields or 854 bits which can be used for future extension. VXLAN is an example of 855 a protocol that includes reserved bits which are subsequently being 856 allocated for new purposes. Another technique employed is to 857 repurpose existing header fields with new meanings. A classic 858 example of this is the definition of DSCP code point which redefines 859 the ToS field originally specified in IPv4. When a field is 860 redefined, some mechanism may be needed to ensure that all interested 861 parties agree on the meaning of the field. The techniques of 862 defining meaning for reserved bits or redefining existing fields have 863 the advantage that a protocol header can be kept a fixed length. The 864 disadvantage is that the extensibility is limited. For instance, the 865 number reserved bits in a fixed protocol header is limited. For 866 standard protocols the decision to commit to a definition for a field 867 can be wrenching since it is difficult to retract later. Also, it is 868 difficult to predict a priori how many reserved fields or bits to put 869 into a protocol header to satisfy the extensions create over the 870 lifetime of the protocol. 872 Extending a protocol header with new fields can be done in several 873 ways. 874 o TLVs are a very popular method used in such protocols as IP and 875 TCP. Depending on the type field size and structure, TLVs can 876 offer a virtually unlimited range of extensions. A disadvantage 877 of TLVs is that processing them can be verbose, quite complicated, 878 several validations must often be done for each TLV, and there is 879 no deterministic ordering for a list of TLVs. TCP serves as an 880 example of a protocol where TLVs have been successfully used (i.e. 881 required for protocol operation). IP is an example of a protocol 882 that allows TLVs but are rarely used in practice (router fast 883 paths usually that assume no IP options). Note that TCP TLVs are 884 implemented in software as well as (NIC) hardware handling various 885 forms of TCP offload. 886 o Extension headers are closely related to TLVs. These also carry 887 type/value information, but instead of being a list of TLVs within 888 a single protocol header, each one is in its own protocol header. 889 IPv6 extension headers and SFC NSH are examples of this technique. 890 Similar to TLVs these offer a wide range of extensibility, but 891 have similarly complex processing. Another difference with TLVs 892 is that each extension header is idempotent. This is beneficial 893 in cases where a protocol implements a push/pop model for header 894 elements like service chaining, but makes it more difficult group 895 correlated information within one protocol header. 896 o A particular form of extension headers are the tags used by IEEE 897 802 protocols. Those are similar to e.g., IPv6 extension headers 898 but with the key difference that each tag is a fixed length header 899 where the length is implicit in the tag value. Thus as long as a 900 receiver can be programmed with a tag value to length map, it can 901 skip those new tags. 902 o Flag-fields are a non-TLV like method of extending a protocol 903 header. The basic idea is that the header contains a set of 904 flags, where each set flags corresponds to optional field that is 905 present in the header. GRE is an example of a protocol that 906 employs this mechanism. The fields are present in the header in 907 the order of the flags, and the length of each field is fixed. 908 Flag-fields are simpler to process compared to TLVs, having fewer 909 validations and the order of the optional fields is deterministic. 910 A disadvantage is that range of possible extensions with flag- 911 fields is smaller than TLVs. 913 The requirements for receiving unknown or unimplemented extensible 914 elements in an encapsulation protocol (flags, TLVs, optional fields) 915 need to be specified. There are two parties to consider, middle 916 boxes and terminal endpoints of encapsulation (at the decapsulator). 918 A protocol may allow or expect nodes in a path to modify fields in an 919 encapsulation (example use of this is BIER). In this case, the 920 middleboxes should follow the same requirements as nodes terminating 921 the encapsulation. In the case that middle boxes do not modify the 922 encapsulation, we can assume that they may still inspect any fields 923 of the encapsulation. Missing or unknown fields should be accepted 924 per protocol specification, however it is permissible for a site to 925 implement a local policy otherwise (e.g. a firewall may drop packets 926 with unknown options). 928 For handling unknown options at terminal nodes, there are two 929 possibilities: drop packet or accept while ignoring the unknown 930 options. Many Internet protocols specify that reserved flags must be 931 set to zero on transmission and ignored on reception. L2TP is 932 example data protocol that has such flags. GRE is a notable 933 exception to this rule, reserved flag bits 1-5 cannot be ignored 934 [RFC2890]. For TCP and IPv4, implementations must ignore optional 935 TLVs with unknown type; however in IPv6 if a packet contains an 936 unknown extension header (unrecognized next header type) the packet 937 must be dropped with an ICMP error message returned. The IPv6 938 options themselves (encoded inside the destinations options or hop- 939 by-hop options extension header) have more flexibility. There bits 940 in the option code are used to instruct the receiver whether to 941 ignore, silently drop, or drop and send error if the option is 942 unknown. Some protocols define a "mandatory bit" that can is set 943 with TLVs to indicate that an option must not be ignored. 944 Conceptually, optional data elements can only be ignored if they are 945 idempotent and do not alter how the rest of the packet is parsed or 946 processed. 948 Depending on what type of protocol evolution one can predict, it 949 might make sense to have an way for a sender to express that the 950 packet should be dropped by a terminal node which does not understand 951 the new information. In other cases it would make sense to have the 952 receiver silently ignore the new info. The former can be expressed 953 by having a version field in the encapsulation, or a notion of 954 "mandatory bit" as discussed above. 956 A security mechanism which use some form secure hash over the encaps 957 header would need to be able to know which extensions can be changed 958 in flight. 960 15.1. Next-protocol 962 [Note that there is editorial duplication between this section and 963 the Next Protocol Indication section earlier in the document] 965 Choices: 966 o Payload type implied (by UDP port for instance). ESP, L2TP, MPLS, 967 over UDP are example, Linux Foo-Over-UDP is example 968 implementation. This model is simple, however allocating a port 969 for every possible protocol might be difficult given the trend 970 towards port conservation as described in 971 [I-D.ietf-tsvwg-port-use]. 972 o Encapsulation contains a next header field. Possibilities: 973 * EtherType: GRE protocol field is example. Allows encapsulation 974 of any EtherType (including IPv4, IPv6, end Ethernet). 975 Disadvantages are that it is 16 bits for less than 100 needed 976 values, and the number space is controlled by the IEEE 802 RAC. 977 * IP protocol: IPv6 extension headers, ESP are examples. Allows 978 encapsulation of any IP protocol including Ethernet via 979 ETHERIP, IPv4, IPv6, IPsec protocols, UDP (transport mode 980 considerations needed). IANA managed eight bit values, 981 presumably more difficult to get an assigned number than to get 982 a transport port assignment. 983 * Protocol specific number space. Example PPP. Could be 8 or 16 984 bits and would be IANA controlled. Primary advantage is that 985 it might be easier to define protocols for encapsulation that 986 are not defined in other number spaces (802.11 for instance). 987 Disadvantage is that it represents yet another number space to 988 be managed and doesn't leverage existing ones. 990 16. Layering Considerations 992 One can envision that SFC might use NVO3 as a delivery/transport 993 mechanism. With more imagination that in turn might be delivered 994 using BIER. Thus it is useful to think about what things look like 995 when we have BIER+NVO3+SFC+payload. Also, if NVO3 is widely deployed 996 there might be cases of NVO3 nesting where a customer uses NVO3 to 997 provide network virtualization e.g., across departments. That 998 customer uses a service provider which happens to use NVO3 to provide 999 transport for their customers.Thus NVO3 in NVO3 might happen. 1001 A key question we set out to answer is what the packets might look 1002 like in such a case, and in particular whether we would end up with 1003 multiple UDP headers for entropy. 1005 Based on the discussion in the Entropy section, the entropy is 1006 associated with the outer delivery IP header. Thus if there are 1007 multiple IP headers there would be a UDP header for each one of the 1008 IP headers. But SFC does not require its own IP header. So a case 1009 of NVO3+SFC would be IP+UDP+NVO3+SFC. A nested NVO3 encapsulation 1010 would have independent IP+UDP headers. 1012 The layering also has some implications for middleboxes. 1013 o A device on the path between the ingress and egress is allowed to 1014 transparently inspect all layers of the protocol stack and drop or 1015 forward, but not transparently modify anything but the layer in 1016 which they operate. What this means is that an IP router is 1017 allowed modify the outer IP ttl and ECN bits, but not the encaps 1018 header or inner headers and payload. And a BIER router is allowed 1019 to modify the BIER header. 1020 o Alternatively such a device can become visible at a higher layer. 1021 E.g., a middlebox could become an decaps + function + encaps which 1022 means it will generate a new encaps header. 1024 The design team asked itself some additional questions: 1025 o Would it make sense to have a common encaps base header (for OAM, 1026 security?, etc) and then followed by the specific information for 1027 NVO3, SFC, BIER? Given that there are separate proposals and the 1028 set of information needing to be carried differs, and the 1029 extensibility needs might be different, it would be difficult and 1030 not that useful to have a common base header. 1031 o With a base header in place, one could view the different 1032 functions (NVO3, SFC, and BIER) as different extensions to that 1033 base header resulting in encodings which are more space optimal by 1034 not repeating the same base header. The base header would only be 1035 repeated when there is an additional IP (and hence UDP) header. 1036 That could mean a single length field (to skip to get to the 1037 payload after all the encaps headers). That might be technically 1038 feasible, but it would create a lot of dependencies between 1039 different WGs making it harder to make progress. Compare with the 1040 potential savings in packet size. 1042 17. Service model 1044 The IP service is lossy and subject to reordering. In order to avoid 1045 a performance impact on transports like TCP the handling of packets 1046 is designed to avoid reordering packets that are in the same 1047 transport flow (which is typically identified by the 5-tuple). But 1048 across such flows the receiver can see different ordering for a given 1049 sender. That is the case for a unicast vs. a multicast flow from the 1050 same sender. 1052 There is a general tussle between the desire for high capacity 1053 utilization across a multipath network and the import on packet 1054 ordering within the same flow (which results in lower transport 1055 protocol performance). That isn't affected by the introduction of an 1056 encapsulation. However, the encapsulation comes with some entropy, 1057 and there might be cases where folks want to change that in response 1058 to overload or failures. For instance, might want to change UDP 1059 source port to try different ECMP route. Such changes can result in 1060 packet reordering within a flow, hence would need to be done 1061 infrequently and with care e.g., by identifying packet trains. 1063 There might be some applications/services which are not able to 1064 handle reordering across flows. The IETF has defined pseudo-wires 1065 [RFC3985] which provides the ability to ensure ordering (implemented 1066 using sequence numbers and/or timestamps). 1068 Architectural such services would make sense, but as a separate layer 1069 on top of an encapsulation protocol. They could be deployed between 1070 ingress and egress of a tunnel which uses some encaps. Potentially 1071 the tunnel control points in the form of an ingress and egress will 1072 become a platform for fixing suboptimal behavior elsewhere in the 1073 network. For example, this document suggests that some congestion 1074 handling might be needed to handle non-congestion controlled traffic 1075 that gets tunneled, and also that fairness/QoS policing can be 1076 deployed on those devices. Others have suggested that tunnels is one 1077 way to deploy ECN without having to add ECN support in the endpoints 1078 [I-D.briscoe-conex-data-centre]. 1080 But the tunnels could potentially do more like increase reliability 1081 (retransmissions, FEC) or load spreading using e.g. MP-TCP between 1082 ingress and egress. 1084 18. Hardware Friendly 1086 Hosts, switches and routers often leverage capabilities in the 1087 hardware to accelerate packet encapsulation, decapsulation and 1088 forwarding. 1090 Some design considerations in encapsulation that leverage these 1091 hardware capabilities may result in more efficiently packet 1092 processing and higher overall protocol throughput. 1094 While "hardware friendliness" can be viewed as unnecessary 1095 considerations for a design, part of the motivation for considering 1096 this is ease of deployment; being able to leverage existing NIC and 1097 switch chips for at least a useful subset of the functionality that 1098 the new encapsulation provides. The other part is the ease of 1099 implementing new NICs and switch/router chips that support the 1100 encapsulation at ever increasing line rates. 1102 [disclaimer] There are many different types of hardware in any given 1103 network, each maybe better at some tasks while worse at others. We 1104 would still recommend protocol designers to examine the specific 1105 hardware that are likely to be used in their networks and make 1106 decisions on a case by case basis. 1108 Some considerations are: 1109 o Keep the encap header small. Switches and routers usually only 1110 read the first small number of bytes into the fast memory for 1111 quick processing and easy manipulation. The bulk of the packets 1112 are usually stored in slow memory. A big encap header may not fit 1113 and additional read from the slow memory will hurt the overall 1114 performance and throughput. 1115 o Put important information at the beginning of the encapsulation 1116 header. The reasoning is similar as explained in the previous 1117 point. If important information are located at the beginning of 1118 the encapsulation header, the packet may be processed with smaller 1119 number of bytes to be read into the fast memory and improve 1120 performance. 1121 o Separation of NVO3 header from SFC header such that an encap can 1122 also be processed by forwarding hardware (who can only process 1123 network virtualization and pass the service chaining function to 1124 another device specialized in service offering) 1125 o Avoid full packet checksums in the encapsulation if possible. 1126 Most of the switch/router hardware make switching/forwarding 1127 decisions by reading and examining only the first certain number 1128 of bytes in the packet. Most of the body of the packet do not 1129 need to be processed normally. if we are concerned of preventing 1130 packet to be misdelivered due to memory errors, consider only 1131 perform header checksums. Note that NIC chips can typically 1132 already do full packet checksums for TCP/UDP, while adding a 1133 header checksum might require adding some hardware support. 1134 o Place important information at fixed offset in the encapsulation 1135 header. Packet processing hardware may be capable of parallel 1136 processing. If important information can be found at fixed 1137 offset, different part of the encapsulation header may be 1138 processed by different hardware units in parallel (for example 1139 multiple table lookups may be launched in parallel). Hardware can 1140 handle optional information as long as when the information is 1141 present it is found in one and only one place in the header. 1142 Typical TLV encoding of options does not have that property since 1143 the order of TLVs is unconstrained. 1144 o Limit the number of header combinations. In many cases the 1145 hardware can explore different combinations of headers in 1146 parallel, however there is some added cost for this. 1148 18.1. Considerations for NIC offload 1150 This section provides guidelines to provide support of common 1151 offloads for encapsulation in Network Interface Cards (NICs). 1152 Offload mechanisms are techniques that are implemented separately 1153 from the normal protocol implementation of a host networking stack 1154 and are intended to optimize or speed up protocol processing. 1155 Hardware offload is performed within a NIC device on behalf of a 1156 host. 1158 There are three basic offload techniques of interest: 1159 o Receive multi queue 1160 o Checksum offload 1161 o Segmentation offload 1163 18.1.1. Receive multi-queue 1165 Contemporary NICs support multiple receive descriptor queues (multi- 1166 queue). Multi-queue enables load balancing of network processing for 1167 a NIC across multiple CPUs. On packet reception, a NIC must select 1168 the appropriate queue for host processing. Receive Side Scaling 1169 (RSS) is a common method which uses the flow hash for a packet to 1170 index an indirection table where each entry stores a queue number. 1172 UDP encapsulation, where the source port is used for entropy, should 1173 be compatible with multi-queue NICs that support five-tuple hash 1174 calculation for UDP/IP packets as input to RSS. The source port 1175 ensures classification of the encapsulated flow even in the case that 1176 the outer source and destination addresses are the same for all flows 1177 (e.g. all flows are going over a single tunnel). 1179 18.1.2. Checksum offload 1181 Many NICs provide capabilities to calculate standard ones complement 1182 payload checksum for packets in transmit or receive. When using 1183 encapsulation over UDP there are at least two checksums that may be 1184 of interest: the encapsulated packet's transport checksum, and the 1185 UDP checksum in the outer header. 1187 18.1.2.1. Transmit checksum offload 1189 NICs may provide a protocol agnostic method to offload transmit 1190 checksum (NETIF_F_HW_CSUM in Linux parlance) that can be used with 1191 UDP encapsulation. In this method the host provides checksum related 1192 parameters in a transmit descriptor for a packet. These parameters 1193 include the starting offset of data to checksum, the length of data 1194 to checksum, and the offset in the packet where the computed checksum 1195 is to be written. The host initializes the checksum field to pseudo 1196 header checksum. In the case of encapsulated packet, the checksum 1197 for an encapsulated transport layer packet, a TCP packet for 1198 instance, can be offloaded by setting the appropriate checksum 1199 parameters. 1201 NICs typically can offload only one transmit checksum per packet, so 1202 simultaneously offloading both an inner transport packet's checksum 1203 and the outer UDP checksum is likely not possible. In this case 1204 setting UDP checksum to zero (per above discussion) and offloading 1205 the inner transport packet checksum might be acceptable. 1207 There is a proposal in [I-D.herbert-remotecsumoffload] to leverage 1208 NIC checksum offload when an encapsulator is co-resident with a host. 1210 18.1.2.2. Receive checksum offload 1212 Protocol encapsulation is compatible with NICs that perform a 1213 protocol agnostic receive checksum (CHECKSUM_COMPLETE in Linux 1214 parlance). In this technique, a NIC computes a ones complement 1215 checksum over all (or some predefined portion) of a packet. The 1216 computed value is provided to the host stack in the packet's receive 1217 descriptor. The host driver can use this checksum to "patch up" and 1218 validate any inner packet transport checksum, as well as the outer 1219 UDP checksum if it is non-zero. 1221 Many legacy NICs don't provide checksum-complete but instead provide 1222 an indication that a checksum has been verified (CHECKSUM_UNNECESSARY 1223 in Linux). Usually, such validation is only done for simple TCP/IP 1224 or UDP/IP packets. If a NIC indicates that a UDP checksum is valid, 1225 the checksum-complete value for the UDP packet is the "not" of the 1226 pseudo header checksum. In this way, checksum-unnecessary can be 1227 converted to checksum-complete. So if the NIC provides checksum- 1228 unnecessary for the outer UDP header in an encapsulation, checksum 1229 conversion can be done so that the checksum-complete value is derived 1230 and can be used by the stack to validate an checksums in the 1231 encapsulated packet. 1233 18.1.3. Segmentation offload 1235 Segmentation offload refers to techniques that attempt to reduce CPU 1236 utilization on hosts by having the transport layers of the stack 1237 operate on large packets. In transmit segmentation offload, a 1238 transport layer creates large packets greater than MTU size (Maximum 1239 Transmission Unit). It is only at much lower point in the stack, or 1240 possibly the NIC, that these large packets are broken up into MTU 1241 sized packet for transmission on the wire. Similarly, in receive 1242 segmentation offload, small packets are coalesced into large, greater 1243 than MTU size packets at a point low in the stack receive path or 1244 possibly in a device. The effect of segmentation offload is that the 1245 number of packets that need to be processed in various layers of the 1246 stack is reduced, and hence CPU utilization is reduced. 1248 18.1.3.1. Transmit Segmentation Offload 1250 Transmit Segmentation Offload (TSO) is a NIC feature where a host 1251 provides a large (larger than MTU size) TCP packet to the NIC, which 1252 in turn splits the packet into separate segments and transmits each 1253 one. This is useful to reduce CPU load on the host. 1255 The process of TSO can be generalized as: 1256 o Split the TCP payload into segments which allow packets with size 1257 less than or equal to MTU. 1258 o For each created segment: 1259 1. Replicate the TCP header and all preceding headers of the 1260 original packet. 1261 2. Set payload length fields in any headers to reflect the length 1262 of the segment. 1263 3. Set TCP sequence number to correctly reflect the offset of the 1264 TCP data in the stream. 1265 4. Recompute and set any checksums that either cover the payload 1266 of the packet or cover header which was changed by setting a 1267 payload length. 1269 Following this general process, TSO can be extended to support TCP 1270 encapsulation UDP. For each segment the Ethernet, outer IP, UDP 1271 header, encapsulation header, inner IP header if tunneling, and TCP 1272 headers are replicated. Any packet length header fields need to be 1273 set properly (including the length in the outer UDP header), and 1274 checksums need to be set correctly (including the outer UDP checksum 1275 if being used). 1277 To facilitate TSO with encapsulation it is recommended that optional 1278 fields should not contain values that must be updated on a per 1279 segment basis-- for example an encapsulation header should not 1280 include checksums, lengths, or sequence numbers that refer to the 1281 payload. If the encapsulation header does not contain such fields 1282 then the TSO engine only needs to copy the bits in the encapsulation 1283 header when creating each segment and does not need to parse the 1284 encapsulation header. 1286 18.1.3.2. Large Receive Offload 1288 Large Receive Offload (LRO) is a NIC feature where packets of a TCP 1289 connection are reassembled, or coalesced, in the NIC and delivered to 1290 the host as one large packet. This feature can reduce CPU 1291 utilization in the host. 1293 LRO requires significant protocol awareness to be implemented 1294 correctly and is difficult to generalize. Packets in the same flow 1295 need to be unambiguously identified. In the presence of tunnels or 1296 network virtualization, this may require more than a five-tuple match 1297 (for instance packets for flows in two different virtual networks may 1298 have identical five-tuples). Additionally, a NIC needs to perform 1299 validation over packets that are being coalesced, and needs to 1300 fabricate a single meaningful header from all the coalesced packets. 1302 The conservative approach to supporting LRO for encapsulation would 1303 be to assign packets to the same flow only if they have identical 1304 five-tuple and were encapsulated the same way. That is the outer IP 1305 addresses, the outer UDP ports, encapsulated protocol, encapsulation 1306 headers, and inner five tuple are all identical. 1308 19. Middlebox Considerations 1310 This document has touched upon middleboxes in different section. The 1311 reason for this is as encapsulations get widely deployed one would 1312 expect different forms of middleboxes might become aware of the 1313 encapsulation protocol just as middleboxes have been made aware of 1314 other protocols where there are business and deployment 1315 opportunities. Such middleboxes are likely to do more than just drop 1316 packets based on the UDP port number used by an encapsulation 1317 protocol. 1319 We note that various forms of encapsulation gateways that stitch one 1320 encapsulation protocol together with another form of protocol could 1321 have similar effects. 1323 An example of a middlebox that could see some use would be an NVO3- 1324 aware firewall that would filter on the VNI IDs to provide some 1325 defense in depth inside or across NVO3 datacenters. 1327 A question for the IETF is whether we should document what to do or 1328 what not to do in such middleboxes. This document touches on areas 1329 of OAM and ECMP as it relates to middleboxes and it might make sense 1330 to document how encaps-aware middleboxes should avoid unintended 1331 consequences in those (and perhaps other) areas. 1333 20. Related Work 1335 The IETF and industry has defined encapsulations for a long time, 1336 with examples like GRE [RFC2890], VXLAN [RFC7348], and NVGRE 1337 [I-D.sridharan-virtualization-nvgre] being able to carry arbitrary 1338 Ethernet payloads, and various forms of IP-in-IP and IPsec 1339 encapsulations that can carry IP packets. As part of NVO3 there has 1340 been additional proposals like Geneve [I-D.gross-geneve] and GUE 1341 [I-D.herbert-gue] which look at more extensibility. NSH 1342 [I-D.quinn-sfc-nsh] is an example of an encapsulation that tries to 1343 provide extensibility mechanisms which target both hardware and 1344 software implementations. 1346 There is also a large body of work around MPLS encapsulations 1347 [RFC3032]. The MPLS-in-UDP work [I-D.ietf-mpls-in-udp] and GRE over 1348 UDP [I-D.ietf-tsvwg-gre-in-udp-encap] have worked on some of the 1349 common issues around checksum and congestion control. MPLS also 1350 introduced a entropy label [RFC6790]. There is also a proposal for 1351 MPLS encryption [I-D.farrelll-mpls-opportunistic-encrypt]. 1353 The idea to use a UDP encapsulation with a UDP source port for 1354 entropy for the underlay routers' ECMP dates back to LISP [RFC6830]. 1356 The pseudo-wire work [RFC3985] is interesting in the notion of 1357 layering additional services/characteristics such as ordered delivery 1358 or timely deliver on top of an encapsulation. That layering approach 1359 might be useful for the new encapsulations as well. For instance, 1360 the control word [RFC4385]. 1362 Both MPLS and L2TP [RFC3931] rely on some control or signaling to 1363 establish state (for the path/labels in the case of MPLS, and for the 1364 session in the case of L2TP). The NVO3, SFC, and BIER encapsulations 1365 will also have some separation between the data plane and control 1366 plane, but the type of separation appears to be different. 1368 IEEE 802.1 has defined encapsulations for L2 over L2, in the form of 1369 Provider backbone Bridging (PBB) [IEEE802.1Q-2014] and Equal Cost 1370 Multipath (ECMP) [IEEE802.1Q-2014]. The latter includes something 1371 very similar to the way the UDP source port is used as entropy: "The 1372 flow hash, carried in an F-TAG, serves to distinguish frames 1373 belonging to different flows and can be used in the forwarding 1374 process to distribute frames over equal cost paths" 1376 TRILL, which is also a L2 over L2 encapsulation, took a different 1377 approach to entropy but preserved the ability for OAM frames 1378 [RFC7174] to use the same entropy hence ECMP path as data frames. In 1379 [I-D.ietf-trill-oam-fm] there 96 bytes of headers for entropy in the 1380 OAM frames, followed by the actual OAM content. This ensures that 1381 any headers, which fit in those 96 bytes except the OAM bit in the 1382 TRILL header, can be used for ECMP hashing. 1384 As encapsulations evolve there might be a desire to fit multiple 1385 inner packets into one outer packet. The work in 1386 [I-D.saldana-tsvwg-simplemux] might be interesting for that purpose. 1388 21. Acknowledgements 1390 The authors acknowledge the comments from David Black, Andy Malis, 1391 and Radia Perlman. 1393 22. Open Issues 1395 o Middleboxes: 1396 * Due to OAM there are constraints on middleboxes in general. If 1397 middleboxes inspect the packet past the outer IP+UDP and encaps 1398 header and look for inner IP and TCP/UDP headers, that might 1399 violate the assumption that OAM packets will be handled the 1400 same as regular data packets. That issue is broader than just 1401 QoS - applies to firewall filters etc. 1402 * Firewalls looking at inner payload? How does that work for OAM 1403 frames? Even if it only drops ... TRILL approach might be an 1404 option? Would that encourage more middleboxes making the 1405 network more fragile? 1406 * Editorially perhaps we should pull the above two into a 1407 separate section about middlebox considerations? 1408 o Next protocol indication - should it be common across different 1409 encapsulation headers? We will have different ways to indicate 1410 the presence of the first encapsulation header in a packet (could 1411 be a UDP destination port, an Ethernet type, etc depending on the 1412 outer delivery header). But for the next protocol past an 1413 encapsulation header one could envision creating or adoption a 1414 common scheme. Such a would also need to be able to identify 1415 following headers like Ethernet, IPv4/IPv6, ESP, etc. 1416 o Common OAM error reporting protocol? 1417 o There is discussion about timestamps, sequence numbers, etc in 1418 three different parts of the document. OAM, Congestion 1419 Considerations, and Service Model, where the latter argues that a 1420 pseudo-wire service should really be layered on top of the encaps 1421 using its own header. Those recommendations seem to be at odds 1422 with each other. Do we envision sequence numbers, timestamps, etc 1423 as potential extensions for OAM and CC? If so, those extensions 1424 could be used to provide a service which doesn't reorder packets. 1426 23. References 1428 23.1. Normative References 1430 [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1431 (IPv6) Specification", RFC 2460, December 1998. 1433 [RFC2890] Dommety, G., "Key and Sequence Number Extensions to GRE", 1434 RFC 2890, September 2000. 1436 [RFC2983] Black, D., "Differentiated Services and Tunnels", 1437 RFC 2983, October 2000. 1439 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 1440 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 1441 Encoding", RFC 3032, January 2001. 1443 [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling 1444 Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 1446 [RFC3948] Huttunen, A., Swander, B., Volpe, V., DiBurro, L., and M. 1447 Stenberg, "UDP Encapsulation of IPsec ESP Packets", 1448 RFC 3948, January 2005. 1450 [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- 1451 Edge (PWE3) Architecture", RFC 3985, March 2005. 1453 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 1454 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 1455 Use over an MPLS PSN", RFC 4385, February 2006. 1457 [RFC5405] Eggert, L. and G. Fairhurst, "Unicast UDP Usage Guidelines 1458 for Application Designers", BCP 145, RFC 5405, 1459 November 2008. 1461 [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion 1462 Notification", RFC 6040, November 2010. 1464 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 1465 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 1466 RFC 6790, November 2012. 1468 [RFC6830] Farinacci, D., Fuller, V., Meyer, D., and D. Lewis, "The 1469 Locator/ID Separation Protocol (LISP)", RFC 6830, 1470 January 2013. 1472 [RFC6935] Eubanks, M., Chimento, P., and M. Westerlund, "IPv6 and 1473 UDP Checksums for Tunneled Packets", RFC 6935, April 2013. 1475 [RFC6936] Fairhurst, G. and M. Westerlund, "Applicability Statement 1476 for the Use of IPv6 UDP Datagrams with Zero Checksums", 1477 RFC 6936, April 2013. 1479 [RFC7174] Salam, S., Senevirathne, T., Aldrin, S., and D. Eastlake, 1480 "Transparent Interconnection of Lots of Links (TRILL) 1481 Operations, Administration, and Maintenance (OAM) 1482 Framework", RFC 7174, May 2014. 1484 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 1485 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 1486 eXtensible Local Area Network (VXLAN): A Framework for 1487 Overlaying Virtualized Layer 2 Networks over Layer 3 1488 Networks", RFC 7348, August 2014. 1490 [RFC7364] Narten, T., Gray, E., Black, D., Fang, L., Kreeger, L., 1491 and M. Napierala, "Problem Statement: Overlays for Network 1492 Virtualization", RFC 7364, October 2014. 1494 [RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. 1495 Rekhter, "Framework for Data Center (DC) Network 1496 Virtualization", RFC 7365, October 2014. 1498 23.2. Informative References 1500 [I-D.briscoe-conex-data-centre] 1501 Briscoe, B. and M. Sridharan, "Network Performance 1502 Isolation in Data Centres using Congestion Policing", 1503 draft-briscoe-conex-data-centre-02 (work in progress), 1504 February 2014. 1506 [I-D.farrelll-mpls-opportunistic-encrypt] 1507 Farrel, A. and S. Farrell, "Opportunistic Security in MPLS 1508 Networks", draft-farrelll-mpls-opportunistic-encrypt-04 1509 (work in progress), January 2015. 1511 [I-D.gross-geneve] 1512 Gross, J., Sridhar, T., Garg, P., Wright, C., Ganga, I., 1513 Agarwal, P., Duda, K., Dutt, D., and J. Hudson, "Geneve: 1514 Generic Network Virtualization Encapsulation", 1515 draft-gross-geneve-02 (work in progress), October 2014. 1517 [I-D.herbert-gue] 1518 Herbert, T., Yong, L., and O. Zia, "Generic UDP 1519 Encapsulation", draft-herbert-gue-03 (work in progress), 1520 March 2015. 1522 [I-D.herbert-remotecsumoffload] 1523 Herbert, T., "Remote checksum offload for encapsulation", 1524 draft-herbert-remotecsumoffload-01 (work in progress), 1525 November 2014. 1527 [I-D.ietf-mpls-in-udp] 1528 Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black, 1529 "Encapsulating MPLS in UDP", draft-ietf-mpls-in-udp-11 1530 (work in progress), January 2015. 1532 [I-D.ietf-nvo3-arch] 1533 Black, D., Hudson, J., Kreeger, L., Lasserre, M., and T. 1534 Narten, "An Architecture for Overlay Networks (NVO3)", 1535 draft-ietf-nvo3-arch-02 (work in progress), October 2014. 1537 [I-D.ietf-sfc-architecture] 1538 Halpern, J. and C. Pignataro, "Service Function Chaining 1539 (SFC) Architecture", draft-ietf-sfc-architecture-07 (work 1540 in progress), March 2015. 1542 [I-D.ietf-sfc-problem-statement] 1543 Quinn, P. and T. Nadeau, "Service Function Chaining 1544 Problem Statement", draft-ietf-sfc-problem-statement-13 1545 (work in progress), February 2015. 1547 [I-D.ietf-trill-oam-fm] 1548 Senevirathne, T., Finn, N., Salam, S., Kumar, D., 1549 Eastlake, D., Aldrin, S., and L. Yizhou, "TRILL Fault 1550 Management", draft-ietf-trill-oam-fm-11 (work in 1551 progress), October 2014. 1553 [I-D.ietf-tsvwg-circuit-breaker] 1554 Fairhurst, G., "Network Transport Circuit Breakers", 1555 draft-ietf-tsvwg-circuit-breaker-00 (work in progress), 1556 September 2014. 1558 [I-D.ietf-tsvwg-gre-in-udp-encap] 1559 Crabbe, E., Yong, L., Xu, X., and T. Herbert, "GRE-in-UDP 1560 Encapsulation", draft-ietf-tsvwg-gre-in-udp-encap-05 (work 1561 in progress), March 2015. 1563 [I-D.ietf-tsvwg-port-use] 1564 Touch, J., "Recommendations for Transport Port Number 1565 Uses", draft-ietf-tsvwg-port-use-07 (work in progress), 1566 January 2015. 1568 [I-D.quinn-sfc-nsh] 1569 Quinn, P., Guichard, J., Surendra, S., Smith, M., 1570 Henderickx, W., Nadeau, T., Agarwal, P., Manur, R., 1571 Chauhan, A., Halpern, J., Majee, S., Elzur, U., Melman, 1572 D., Garg, P., McConnell, B., Wright, C., and K. Kevin, 1573 "Network Service Header", draft-quinn-sfc-nsh-07 (work in 1574 progress), February 2015. 1576 [I-D.saldana-tsvwg-simplemux] 1577 Saldana, J., "Simplemux. A generic multiplexing protocol", 1578 draft-saldana-tsvwg-simplemux-02 (work in progress), 1579 January 2015. 1581 [I-D.shepherd-bier-problem-statement] 1582 Shepherd, G., Dolganow, A., and a. 1583 arkadiy.gulko@thomsonreuters.com, "Bit Indexed Explicit 1584 Replication (BIER) Problem Statement", 1585 draft-shepherd-bier-problem-statement-02 (work in 1586 progress), February 2015. 1588 [I-D.sridharan-virtualization-nvgre] 1589 Garg, P. and Y. Wang, "NVGRE: Network Virtualization using 1590 Generic Routing Encapsulation", 1591 draft-sridharan-virtualization-nvgre-07 (work in 1592 progress), November 2014. 1594 [I-D.wei-tsvwg-tunnel-congestion-feedback] 1595 Wei, X., Zhu, L., and L. Deng, "Tunnel Congestion 1596 Feedback", draft-wei-tsvwg-tunnel-congestion-feedback-03 1597 (work in progress), October 2014. 1599 [I-D.wijnands-bier-architecture] 1600 Wijnands, I., Rosen, E., Dolganow, A., Przygienda, T., and 1601 S. Aldrin, "Multicast using Bit Index Explicit 1602 Replication", draft-wijnands-bier-architecture-05 (work in 1603 progress), March 2015. 1605 [I-D.wijnands-mpls-bier-encapsulation] 1606 Wijnands, I., Rosen, E., Dolganow, A., Tantsura, J., and 1607 S. Aldrin, "Encapsulation for Bit Index Explicit 1608 Replication in MPLS Networks", 1609 draft-wijnands-mpls-bier-encapsulation-02 (work in 1610 progress), December 2014. 1612 [I-D.xu-bier-encapsulation] 1613 Xu, X., Somasundaram, S., Jacquenet, C., and R. Raszuk, 1614 "BIER Encapsulation", draft-xu-bier-encapsulation-02 (work 1615 in progress), February 2015. 1617 [IEEE802.1Q-2014] 1618 IEEE, "IEEE Standard for Local and metropolitan area 1619 networks--Bridges and Bridged Networks", IEEE Std 802.1Q- 1620 2014, 2014, 1621 . 1623 (Access Controlled link within page) 1625 Authors' Addresses 1627 Erik Nordmark 1628 Arista Networks 1629 5453 Great America Parkway 1630 Santa Clara, CA 95054 1631 USA 1633 Email: nordmark@arista.com 1635 Albert Tian 1636 Ericsson Inc. 1637 300 Holger Way 1638 San Jose, California 95134 1639 USA 1641 Email: albert.tian@ericsson.com 1643 Jesse Gross 1644 VMware 1645 3401 Hillview Ave. 1646 Palo Alto, CA 94304 1647 USA 1649 Email: jgross@vmware.com 1651 Jon Hudson 1652 Brocade Communications Systems, Inc. 1653 130 Holger Way 1654 San Jose, CA 95134 1655 USA 1657 Email: jon.hudson@gmail.com 1658 Lawrence Kreeger 1659 Cisco Systems, Inc. 1660 170 W. Tasman Avenue 1661 San Jose, CA 95134 1662 USA 1664 Email: kreeger@cisco.com 1666 Pankaj Garg 1667 Microsoft 1668 1 Microsoft Way 1669 Redmond, WA 98052 1670 USA 1672 Email: pankajg@microsoft.com 1674 Patricia Thaler 1675 Broadcom Corporation 1676 3151 Zanker Road 1677 San Jose, CA 95134 1678 USA 1680 Email: pthaler@broadcom.com 1682 Tom Herbert 1683 Google 1684 1600 Amphitheatre Parkway 1685 Mountain View, CA 1686 USA 1688 Email: therbert@google.com