idnits 2.17.1 draft-gross-geneve-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 25, 2014) is 3464 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) == Outdated reference: A later version (-08) exists of draft-davie-stt-06 == Outdated reference: A later version (-08) exists of draft-sridharan-virtualization-nvgre-06 -- Obsolete informational reference (is this intentional?): RFC 1981 (Obsoleted by RFC 8201) Summary: 1 error (**), 0 flaws (~~), 3 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group J. Gross 3 Internet-Draft T. Sridhar 4 Intended status: Standards Track VMware 5 Expires: April 28, 2015 P. Garg 6 Microsoft 7 C. Wright 8 Red Hat 9 I. Ganga 10 Intel 11 P. Agarwal 12 Broadcom 13 K. Duda 14 Arista 15 D. Dutt 16 Cumulus 17 J. Hudson 18 Brocade 19 October 25, 2014 21 Geneve: Generic Network Virtualization Encapsulation 22 draft-gross-geneve-02 24 Abstract 26 Network virtualization involves the cooperation of devices with a 27 wide variety of capabilities such as software and hardware tunnel 28 endpoints, transit fabrics, and centralized control clusters. As a 29 result of their role in tying together different elements in the 30 system, the requirements on tunnels are influenced by all of these 31 components. Flexibility is therefore the most important aspect of a 32 tunnel protocol if it is to keep pace with the evolution of the 33 system. This draft describes Geneve, a protocol designed to 34 recognize and accommodate these changing capabilities and needs. 36 Status of This Memo 38 This Internet-Draft is submitted in full conformance with the 39 provisions of BCP 78 and BCP 79. 41 Internet-Drafts are working documents of the Internet Engineering 42 Task Force (IETF). Note that other groups may also distribute 43 working documents as Internet-Drafts. The list of current Internet- 44 Drafts is at http://datatracker.ietf.org/drafts/current/. 46 Internet-Drafts are draft documents valid for a maximum of six months 47 and may be updated, replaced, or obsoleted by other documents at any 48 time. It is inappropriate to use Internet-Drafts as reference 49 material or to cite them other than as "work in progress." 51 This Internet-Draft will expire on April 28, 2015. 53 Copyright Notice 55 Copyright (c) 2014 IETF Trust and the persons identified as the 56 document authors. All rights reserved. 58 This document is subject to BCP 78 and the IETF Trust's Legal 59 Provisions Relating to IETF Documents 60 (http://trustee.ietf.org/license-info) in effect on the date of 61 publication of this document. Please review these documents 62 carefully, as they describe your rights and restrictions with respect 63 to this document. Code Components extracted from this document must 64 include Simplified BSD License text as described in Section 4.e of 65 the Trust Legal Provisions and are provided without warranty as 66 described in the Simplified BSD License. 68 Table of Contents 70 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 71 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 72 1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 73 2. Design Requirements . . . . . . . . . . . . . . . . . . . . . 5 74 2.1. Control Plane Independence . . . . . . . . . . . . . . . 6 75 2.2. Data Plane Extensibility . . . . . . . . . . . . . . . . 7 76 2.2.1. Efficient Implementation . . . . . . . . . . . . . . 7 77 2.3. Use of Standard IP Fabrics . . . . . . . . . . . . . . . 8 78 3. Geneve Encapsulation Details . . . . . . . . . . . . . . . . 9 79 3.1. Geneve Frame Format Over IPv4 . . . . . . . . . . . . . . 9 80 3.2. Geneve Frame Format Over IPv6 . . . . . . . . . . . . . . 10 81 3.3. UDP Header . . . . . . . . . . . . . . . . . . . . . . . 12 82 3.4. Tunnel Header Fields . . . . . . . . . . . . . . . . . . 13 83 3.5. Tunnel Options . . . . . . . . . . . . . . . . . . . . . 14 84 3.5.1. Options Processing . . . . . . . . . . . . . . . . . 16 85 4. Implementation and Deployment Considerations . . . . . . . . 16 86 4.1. Encapsulation of Geneve in IP . . . . . . . . . . . . . . 16 87 4.1.1. IP Fragmentation . . . . . . . . . . . . . . . . . . 16 88 4.1.2. DSCP and ECN . . . . . . . . . . . . . . . . . . . . 17 89 4.1.3. Broadcast and Multicast . . . . . . . . . . . . . . . 17 90 4.2. NIC Offloads . . . . . . . . . . . . . . . . . . . . . . 18 91 4.3. Inner VLAN Handling . . . . . . . . . . . . . . . . . . . 18 92 5. Interoperability Issues . . . . . . . . . . . . . . . . . . . 19 93 6. Security Considerations . . . . . . . . . . . . . . . . . . . 19 94 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 95 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 96 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 97 9.1. Normative References . . . . . . . . . . . . . . . . . . 20 98 9.2. Informative References . . . . . . . . . . . . . . . . . 21 99 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 101 1. Introduction 103 Networking has long featured a variety of tunneling, tagging, and 104 other encapsulation mechanisms. However, the advent of network 105 virtualization has caused a surge of renewed interest and a 106 corresponding increase in the introduction of new protocols. The 107 large number of protocols in this space, ranging all the way from 108 VLANs [IEEE.802.1Q-2011] and MPLS [RFC3031] through the more recent 109 VXLAN [RFC7348], NVGRE [I-D.sridharan-virtualization-nvgre], and STT 110 [I-D.davie-stt], often leads to questions about the need for new 111 encapsulation formats and what it is about network virtualization in 112 particular that leads to their proliferation. 114 While many encapsulation protocols seek to simply partition the 115 underlay network or bridge between two domains, network 116 virtualization views the transit network as providing connectivity 117 between multiple components of an integrated system. In many ways 118 this system is similar to a chassis switch with the IP underlay 119 network playing the role of the backplane and tunnel endpoints on the 120 edge as line cards. When viewed in this light, the requirements 121 placed on the tunnel protocol are significantly different in terms of 122 the quantity of metadata necessary and the role of transit nodes. 124 Current work such as [VL2] and the NVO3 working group 125 [I-D.ietf-nvo3-dataplane-requirements] have described some of the 126 properties that the data plane must have to support network 127 virtualization. However, one additional defining requirement is the 128 need to carry system state along with the packet data. The use of 129 some metadata is certainly not a foreign concept - nearly all 130 protocols used for virtualization have at least 24 bits of identifier 131 space as a way to partition between tenants. This is often described 132 as overcoming the limits of 12-bit VLANs, and when seen in that 133 context, or any context where it is a true tenant identifier, 16 134 million possible entries is a large number. However, the reality is 135 that the metadata is not exclusively used to identify tenants and 136 encoding other information quickly starts to crowd the space. In 137 fact, when compared to the tags used to exchange metadata between 138 line cards on a chassis switch, 24-bit identifiers start to look 139 quite small. There are nearly endless uses for this metadata, 140 ranging from storing input ports for simple security policies to 141 service based context for interposing advanced middleboxes. 143 Existing tunnel protocols have each attempted to solve different 144 aspects of these new requirements, only to be quickly rendered out of 145 date by changing control plane implementations and advancements. 146 Furthermore, software and hardware components and controllers all 147 have different advantages and rates of evolution - a fact that should 148 be viewed as a benefit, not a liability or limitation. This draft 149 describes Geneve, a protocol which seeks to avoid these problems by 150 providing a framework for tunneling for network virtualization rather 151 than being prescriptive about the entire system. 153 1.1. Requirements Language 155 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 156 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 157 document are to be interpreted as described in [RFC2119]. 159 In this document, these words will appear with that interpretation 160 only when in ALL CAPS. Lower case uses of these words are not to be 161 interpreted as carrying RFC-2119 significance. 163 1.2. Terminology 165 The NVO3 framework [RFC7365] defines many of the concepts commonly 166 used in network virtualization. In addition, the following terms are 167 specifically meaningful in this document: 169 Checksum offload. An optimization implemented by many NICs which 170 enables computation and verification of upper layer protocol 171 checksums in hardware on transmit and receive, respectively. This 172 typically includes IP and TCP/UDP checksums which would otherwise be 173 computed by the protocol stack in software. 175 Clos network. A technique for composing network fabrics larger than 176 a single switch while maintaining non-blocking bandwidth across 177 connection points. ECMP is used to divide traffic across the 178 multiple links and switches that constitute the fabric. Sometimes 179 termed "leaf and spine" or "fat tree" topologies. 181 ECMP. Equal Cost Multipath. A routing mechanism for selecting from 182 among multiple best next hop paths by hashing packet headers in order 183 to better utilize network bandwidth while avoiding reordering a 184 single stream. 186 Geneve. Generic Network Virtualization Encapsulation. The tunnel 187 protocol described in this draft. 189 LRO. Large Receive Offload. The receive-side equivalent function of 190 LSO, in which multiple protocol segments (primarily TCP) are 191 coalesced into larger data units. 193 NIC. Network Interface Card. A NIC could be part of a tunnel 194 endpoint or transit device and can either process Geneve packets or 195 aid in the processing of Geneve packets. 197 OAM. Operations, Administration, and Management. A suite of tools 198 used to monitor and troubleshoot network problems. 200 Transit device. A forwarding element along the path of the tunnel 201 making up part of the Underlay Network. A transit device MAY be 202 capable of understanding the Geneve frame format but does not 203 originate or terminate Geneve packets. 205 LSO. Large Segmentation Offload. A function provided by many 206 commercial NICs that allows data units larger than the MTU to be 207 passed to the NIC to improve performance, the NIC being responsible 208 for creating smaller segments with correct protocol headers. When 209 referring specifically to TCP/IP, this feature is often known as TSO 210 (TCP Segmentation Offload). 212 Tunnel endpoint. A component encapsulating packets, such as Ethernet 213 frames or IP datagrams, in Geneve headers and vice versa. As the 214 ultimate consumer of any tunnel metadata, endpoints have the highest 215 level of requirements for parsing and interpreting tunnel headers. 216 Tunnel endpoints may consist of either software or hardware 217 implementations or a combination of the two. Endpoints are 218 frequently a component of an NVE but may also be found in middleboxes 219 or other elements making up an NVO3 Network. 221 VM. Virtual Machine. 223 2. Design Requirements 225 Geneve is designed to support network virtualization use cases, where 226 tunnels are typically established to act as a backplane between the 227 virtual switches residing in hypervisors, physical switches, or 228 middleboxes or other appliances. An arbitrary IP network can be used 229 as an underlay although Clos networks composed using ECMP links are a 230 common choice to provide consistent bisectional bandwidth across all 231 connection points. Figure 1 shows an example of a hypervisor, top of 232 rack switch for connectivity to physical servers, and a WAN uplink 233 connected using Geneve tunnels over a simplified Clos network. These 234 tunnels are used to encapsulate and forward frames from the attached 235 components such as VMs or physical links. 237 +---------------------+ +-------+ +------+ 238 | +--+ +-------+---+ | |Transit|--|Top of|==Physical 239 | |VM|--| | | | +------+ /|Router | | Rack |==Servers 240 | +--+ |Virtual|NIC|---|Top of|/ +-------+\/+------+ 241 | +--+ |Switch | | | | Rack |\ +-------+/\+------+ 242 | |VM|--| | | | +------+ \|Transit| |Uplink| WAN 243 | +--+ +-------+---+ | |Router |--| |=========> 244 +---------------------+ +-------+ +------+ 245 Hypervisor 247 ()===================================() 248 Switch-Switch Geneve Tunnels 250 Figure 1: Sample Geneve Deployment 252 To support the needs of network virtualization, the tunnel protocol 253 should be able to take advantage of the differing (and evolving) 254 capabilities of each type of device in both the underlay and overlay 255 networks. This results in the following requirements being placed on 256 the data plane tunneling protocol: 258 o The data plane is generic and extensible enough to support current 259 and future control planes. 261 o Tunnel components are efficiently implementable in both hardware 262 and software without restricting capabilities to the lowest common 263 denominator. 265 o High performance over existing IP fabrics. 267 These requirements are described further in the following 268 subsections. 270 2.1. Control Plane Independence 272 Although some protocols for network virtualization have included a 273 control plane as part of the tunnel format specification (most 274 notably, the original VXLAN spec prescribed a multicast learning- 275 based control plane), these specifications have largely been treated 276 as describing only the data format. The VXLAN frame format has 277 actually seen a wide variety of control planes built on top of it. 279 There is a clear advantage in settling on a data format: most of the 280 protocols are only superficially different and there is little 281 advantage in duplicating effort. However, the same cannot be said of 282 control planes, which are diverse in very fundamental ways. The case 283 for standardization is also less clear given the wide variety in 284 requirements, goals, and deployment scenarios. 286 As a result of this reality, Geneve aims to be a pure tunnel format 287 specification that is capable of fulfilling the needs of many control 288 planes by explicitly not selecting any one of them. This 289 simultaneously promotes a shared data format and increases the 290 chances that it will not be obsoleted by future control plane 291 enhancements. 293 2.2. Data Plane Extensibility 295 Achieving the level of flexibility needed to support current and 296 future control planes effectively requires an options infrastructure 297 to allow new metadata types to be defined, deployed, and either 298 finalized or retired. Options also allow for differentiation of 299 products by encouraging independent development in each vendor's core 300 specialty, leading to an overall faster pace of advancement. By far 301 the most common mechanism for implementing options is Type-Length- 302 Value (TLV) format. 304 It should be noted that while options can be used to support non- 305 wirespeed control frames, they are equally important on data frames 306 as well to segregate and direct forwarding (for instance, the 307 examples given before of input port based security policies and 308 service interposition both require tags to be placed on data 309 packets). Therefore, while it would be desirable to limit the 310 extensibility to only control frames for the purposes of simplifying 311 the datapath, that would not satisfy the design requirements. 313 2.2.1. Efficient Implementation 315 There is often a conflict between software flexibility and hardware 316 performance that is difficult to resolve. For a given set of 317 functionality, it is obviously desirable to maximize performance. 318 However, that does not mean new features that cannot be run at that 319 speed today should be disallowed. Therefore, for a protocol to be 320 efficiently implementable means that a set of common capabilities can 321 be reasonably handled across platforms along with a graceful 322 mechanism to handle more advanced features in the appropriate 323 situations. 325 The use of a variable length header and options in a protocol often 326 raises questions about whether it is truly efficiently implementable 327 in hardware. To answer this question in the context of Geneve, it is 328 important to first divide "hardware" into two categories: tunnel 329 endpoints and transit devices. 331 Endpoints must be able to parse the variable header, including any 332 options, and take action. Since these devices are actively 333 participating in the protocol, they are the most affected by Geneve. 335 However, as endpoints are the ultimate consumers of the data, 336 transmitters can tailor their output to the capabilities of the 337 recipient. As new functionality becomes sufficiently well defined to 338 add to endpoints, supporting options can be designed using ordering 339 restrictions and other techniques to ease parsing. 341 Transit devices MAY be able to interpret the options and participate 342 in Geneve packet processing. However, as non-terminating devices, 343 they do not originate or terminate the Geneve packet. The 344 participation of transit devices in Geneve packet processing is 345 OPTIONAL. 347 Further, either tunnel endpoints or transit devices MAY use offload 348 capabilities of NICs such as checksum offload to improve the 349 performance of Geneve packet processing. The presence of a Geneve 350 variable length header SHOULD NOT prevent the tunnel endpoints and 351 transit devices from using such offload capabilities. 353 2.3. Use of Standard IP Fabrics 355 IP has clearly cemented its place as the dominant transport mechanism 356 and many techniques have evolved over time to make it robust, 357 efficient, and inexpensive. As a result, it is natural to use IP 358 fabrics as a transit network for Geneve. Fortunately, the use of IP 359 encapsulation and addressing is enough to achieve the primary goal of 360 delivering packets to the correct point in the network through 361 standard switching and routing. 363 In addition, nearly all underlay fabrics are designed to exploit 364 parallelism in traffic to spread load across multiple links without 365 introducing reordering in individual flows. These equal cost 366 multipathing (ECMP) techniques typically involve parsing and hashing 367 the addresses and port numbers from the packet to select an outgoing 368 link. However, the use of tunnels often results in poor ECMP 369 performance without additional knowledge of the protocol as the 370 encapsulated traffic is hidden from the fabric by design and only 371 endpoint addresses are available for hashing. 373 Since it is desirable for Geneve to perform well on these existing 374 fabrics, it is necessary for entropy from encapsulated packets to be 375 exposed in the tunnel header. The most common technique for this is 376 to use the UDP source port, which is discussed further in 377 Section 3.3. 379 3. Geneve Encapsulation Details 381 The Geneve frame format consists of a compact tunnel header 382 encapsulated in UDP over either IPv4 or IPv6. A small fixed tunnel 383 header provides control information plus a base level of 384 functionality and interoperability with a focus on simplicity. This 385 header is then followed by a set of variable options to allow for 386 future innovation. Finally, the payload consists of a protocol data 387 unit of the indicated type, such as an Ethernet frame. The following 388 subsections provide examples of Geneve frames transported (for 389 example) over Ethernet along with an Ethernet payload. 391 3.1. Geneve Frame Format Over IPv4 393 0 1 2 3 394 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 396 Outer Ethernet Header: 397 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 398 | Outer Destination MAC Address | 399 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 400 | Outer Destination MAC Address | Outer Source MAC Address | 401 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 402 | Outer Source MAC Address | 403 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 404 |Optional Ethertype=C-Tag 802.1Q| Outer VLAN Tag Information | 405 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 406 | Ethertype=0x0800 | 407 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 409 Outer IPv4 Header: 410 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 411 |Version| IHL |Type of Service| Total Length | 412 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 413 | Identification |Flags| Fragment Offset | 414 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 415 | Time to Live |Protocol=17 UDP| Header Checksum | 416 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 417 | Outer Source IPv4 Address | 418 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 419 | Outer Destination IPv4 Address | 420 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 422 Outer UDP Header: 423 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 424 | Source Port = xxxx | Dest Port = 6081 | 425 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 426 | UDP Length | UDP Checksum | 427 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 429 Geneve Header: 430 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 431 |Ver| Opt Len |O|C| Rsvd. | Protocol Type | 432 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 433 | Virtual Network Identifier (VNI) | Reserved | 434 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 435 | Variable Length Options | 436 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 438 Inner Ethernet Header (example payload): 439 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 440 | Inner Destination MAC Address | 441 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 442 | Inner Destination MAC Address | Inner Source MAC Address | 443 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 444 | Inner Source MAC Address | 445 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 446 |Optional Ethertype=C-Tag 802.1Q| Inner VLAN Tag Information | 447 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 449 Payload: 450 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 451 | Ethertype of Original Payload | | 452 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 453 | Original Ethernet Payload | 454 | | 455 | (Note that the original Ethernet Frame's FCS is not included) | 456 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 458 Frame Check Sequence: 459 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 460 | New FCS (Frame Check Sequence) for Outer Ethernet Frame | 461 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 463 3.2. Geneve Frame Format Over IPv6 464 0 1 2 3 465 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 467 Outer Ethernet Header: 468 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 469 | Outer Destination MAC Address | 470 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 471 | Outer Destination MAC Address | Outer Source MAC Address | 472 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 473 | Outer Source MAC Address | 474 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 475 |Optional Ethertype=C-Tag 802.1Q| Outer VLAN Tag Information | 476 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 477 | Ethertype=0x86DD | 478 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 480 Outer IPv6 Header: 481 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 482 |Version| Traffic Class | Flow Label | 483 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 484 | Payload Length | NxtHdr=17 UDP | Hop Limit | 485 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 486 | | 487 + + 488 | | 489 + Outer Source IPv6 Address + 490 | | 491 + + 492 | | 493 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 494 | | 495 + + 496 | | 497 + Outer Destination IPv6 Address + 498 | | 499 + + 500 | | 501 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 503 Outer UDP Header: 504 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 505 | Source Port = xxxx | Dest Port = 6081 | 506 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 507 | UDP Length | UDP Checksum | 508 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 510 Geneve Header: 511 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 512 |Ver| Opt Len |O|C| Rsvd. | Protocol Type | 513 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 514 | Virtual Network Identifier (VNI) | Reserved | 515 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 516 | Variable Length Options | 517 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 519 Inner Ethernet Header (example payload): 520 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 521 | Inner Destination MAC Address | 522 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 523 | Inner Destination MAC Address | Inner Source MAC Address | 524 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 525 | Inner Source MAC Address | 526 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 527 |Optional Ethertype=C-Tag 802.1Q| Inner VLAN Tag Information | 528 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 530 Payload: 531 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 532 | Ethertype of Original Payload | | 533 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 534 | Original Ethernet Payload | 535 | | 536 | (Note that the original Ethernet Frame's FCS is not included) | 537 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 539 Frame Check Sequence: 540 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 541 | New FCS (Frame Check Sequence) for Outer Ethernet Frame | 542 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 544 3.3. UDP Header 546 The use of an encapsulating UDP [RFC0768] header follows the 547 connectionless semantics of Ethernet and IP in addition to providing 548 entropy to routers performing ECMP. The header fields are therefore 549 interpreted as follows: 551 Source port: A source port selected by the ingress tunnel endpoint. 552 This source port SHOULD be the same for all packets belonging to a 553 single encapsulated flow to prevent reordering due to the use of 554 different paths. To encourage an even distribution of flows 555 across multiple links, the source port SHOULD be calculated using 556 a hash of the encapsulated packet headers using, for example, a 557 traditional 5-tuple. Since the port represents a flow identifier 558 rather than a true UDP connection, the entire 16-bit range MAY be 559 used to maximize entropy. 561 Dest port: IANA has assigned port 6081 as the fixed well-known 562 destination port for Geneve. This port MUST be used in both 563 directions of a flow. Although the well-known value should be 564 used by default, it is RECOMMENDED that implementations make this 565 configurable. 567 UDP length: The length of the UDP packet including the UDP header. 569 UDP checksum: The checksum MAY be set to zero on transmit for 570 packets encapsulated in both IPv4 and IPv6 [RFC6935]. When a 571 packet is received with a UDP checksum of zero it MUST be accepted 572 and decapsulated. If the ingress tunnel endpoint optionally 573 encapsulates a packet with a non-zero checksum, it MUST be a 574 correctly computed UDP checksum. Upon receiving such a packet, 575 the egress endpoint MUST validate the checksum. If the checksum 576 is not correct, the packet MUST be dropped, otherwise the packet 577 MUST be accepted for decapsulation. It is RECOMMENDED that the 578 UDP checksum be computed to protect the Geneve header and options 579 in situations where the network reliability is not high and the 580 packet is not protected by another checksum or CRC. 582 3.4. Tunnel Header Fields 584 Ver (2 bits): The current version number is 0. Packets received by 585 an endpoint with an unknown version MUST be dropped. Non- 586 terminating devices processing Geneve packets with an unknown 587 version number MUST treat them as UDP packets with an unknown 588 payload. 590 Opt Len (6 bits): The length of the options fields, expressed in 591 four byte multiples, not including the eight byte fixed tunnel 592 header. This results in a minimum total Geneve header size of 8 593 bytes and a maximum of 260 bytes. The start of the payload 594 headers can be found using this offset from the end of the base 595 Geneve header. 597 O (1 bit): OAM frame. This packet contains a control message 598 instead of a data payload. Endpoints MUST NOT forward the payload 599 and transit devices MUST NOT attempt to interpret or process it. 600 Since these are infrequent control messages, it is RECOMMENDED 601 that endpoints direct these packets to a high priority control 602 queue (for example, to direct the packet to a general purpose CPU 603 from a forwarding ASIC or to separate out control traffic on a 604 NIC). Transit devices MUST NOT alter forwarding behavior on the 605 basis of this bit, such as ECMP link selection. 607 C (1 bit): Critical options present. One or more options has the 608 critical bit set (see Section 3.5). If this bit is set then 609 tunnel endpoints MUST parse the options list to interpret any 610 critical options. On devices where option parsing is not 611 supported the frame MUST be dropped on the basis of the 'C' bit in 612 the base header. If the bit is not set tunnel endpoints MAY strip 613 all options using 'Opt Len' and forward the decapsulated frame. 614 Transit devices MUST NOT drop or modify packets on the basis of 615 this bit. 617 Rsvd. (6 bits): Reserved field which MUST be zero on transmission 618 and ignored on receipt. 620 Protocol Type (16 bits): The type of the protocol data unit 621 appearing after the Geneve header. This follows the EtherType 622 [ETYPES] convention with Ethernet itself being represented by the 623 value 0x6558. 625 Virtual Network Identifier (VNI) (24 bits): An identifier for a 626 unique element of a virtual network. In many situations this may 627 represent an L2 segment, however, the control plane defines the 628 forwarding semantics of decapsulated packets. The VNI MAY be used 629 as part of ECMP forwarding decisions or MAY be used as a mechanism 630 to distinguish between overlapping address spaces contained in the 631 encapsulated packet when load balancing across CPUs. 633 Reserved (8 bits): Reserved field which MUST be zero on transmission 634 and ignored on receipt. 636 Transit devices MUST maintain consistent forwarding behavior 637 irrespective of the value of 'Opt Len', including ECMP link 638 selection. These devices SHOULD be able to forward packets 639 containing options without resorting to a slow path. 641 3.5. Tunnel Options 643 0 1 2 3 644 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 645 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 646 | Option Class | Type |R|R|R| Length | 647 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 648 | Variable Option Data | 649 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 651 Geneve Option 653 The base Geneve header is followed by zero or more options in Type- 654 Length-Value format. Each option consists of a four byte option 655 header and a variable amount of option data interpreted according to 656 the type. 658 Option Class (16 bits): Namespace for the 'Type' field. IANA will 659 be requested to create a "Geneve Option Class" registry to 660 allocate identifiers for organizations, technologies, and vendors 661 that have an interest in creating types for options. Each 662 organization may allocate types independently to allow 663 experimentation and rapid innovation. It is expected that over 664 time certain options will become well known and a given 665 implementation may use option types from a variety of sources. In 666 addition, IANA will be requested to reserve specific ranges for 667 standardized and experimental options. 669 Type (8 bits): Type indicating the format of the data contained in 670 this option. Options are primarily designed to encourage future 671 extensibility and innovation and so standardized forms of these 672 options will be defined in a separate document. 674 The high order bit of the option type indicates that this is a 675 critical option. If the receiving endpoint does not recognize 676 this option and this bit is set then the frame MUST be dropped. 677 If the critical bit is set in any option then the 'C' bit in the 678 Geneve base header MUST also be set. Transit devices MUST NOT 679 drop packets on the basis of this bit. The following figure shows 680 the location of the 'C' bit in the 'Type' field: 682 0 1 2 3 4 5 6 7 8 683 +-+-+-+-+-+-+-+-+ 684 |C| Type | 685 +-+-+-+-+-+-+-+-+ 687 The requirement to drop a packet with an unknown critical option 688 applies to the entire tunnel endpoint system and not a particular 689 component of the implementation. For example, in a system 690 comprised of a forwarding ASIC and a general purpose CPU, this 691 does not mean that the packet must be dropped in the ASIC. An 692 implementation may send the packet to the CPU using a rate-limited 693 control channel for slow-path exception handling. 695 R (3 bits): Option control flags reserved for future use. MUST be 696 zero on transmission and ignored on receipt. 698 Length (5 bits): Length of the option, expressed in four byte 699 multiples excluding the option header. The total length of each 700 option may be between 4 and 128 bytes. Packets in which the total 701 length of all options is not equal to the 'Opt Len' in the base 702 header are invalid and MUST be silently dropped if received by an 703 endpoint. 705 Variable Option Data: Option data interpreted according to 'Type'. 707 3.5.1. Options Processing 709 Geneve options are intended to be originated and processed by tunnel 710 endpoints. Options MAY be processed by transit devices along the 711 tunnel path as well. This document only details the handling of 712 options by tunnel endpoints. A future version of this document will 713 provide details of options processing by transit devices. Transit 714 devices not processing Geneve options SHOULD process Geneve frame as 715 any other UDP frame and maintain consistent forwarding behavior. 717 In tunnel endpoints, the generation and interpretation of options is 718 determined by the control plane, which is out of the scope of this 719 document. However, to ensure interoperability between heterogeneous 720 devices two requirements are imposed on endpoint devices: 722 o Receiving endpoints MUST drop packets containing unknown options 723 with the 'C' bit set in the option type. 725 o Sending endpoints MUST NOT assume that options will be processed 726 sequentially by the receiver in the order they were transmitted. 728 4. Implementation and Deployment Considerations 730 4.1. Encapsulation of Geneve in IP 732 As an IP-based tunnel protocol, Geneve shares many properties and 733 techniques with existing protocols. The application of some of these 734 are described in further detail, although in general most concepts 735 applicable to the IP layer or to IP tunnels generally also function 736 in the context of Geneve. 738 4.1.1. IP Fragmentation 740 To prevent fragmentation and maximize performance, the best practice 741 when using Geneve is to ensure that the MTU of the physical network 742 is greater than or equal to the MTU of the encapsulated network plus 743 tunnel headers. Manual or upper layer (such as TCP MSS clamping) 744 configuration can be used to ensure that fragmentation never takes 745 place, however, in some situations this may not be feasible. 747 It is strongly RECOMMENDED that Path MTU Discovery ([RFC1191], 748 [RFC1981]) be used by setting the DF bit in the IP header when Geneve 749 packets are transmitted over IPv4 (this is the default with IPv6). 751 The use of Path MTU Discovery on the transit network provides the 752 encapsulating endpoint with soft-state about the link that it may use 753 to prevent or minimize fragmentation depending on its role in the 754 virtualized network. 756 Note that some implementations may not be capable of supporting 757 fragmentation or other less common features of the IP header, such as 758 options and extension headers. 760 4.1.2. DSCP and ECN 762 When encapsulating IP (including over Ethernet) frames in Geneve, 763 there are several options for propagating DSCP and ECN bits from the 764 inner header to the tunnel on transmission and the reverse on 765 reception. 767 [RFC2983] lists considerations for mapping DSCP between inner and 768 outer IP headers. Network virtualization is typically more closely 769 aligned with the Pipe model described, where the DSCP value on the 770 tunnel header is set based on a policy (which may be a fixed value, 771 one based on the inner traffic class, or some other mechanism for 772 grouping traffic). Aspects of the Uniform model (which treats the 773 inner and outer DSCP value as a single field by copying on ingress 774 and egress) may also apply, such as the ability to remark the inner 775 header on tunnel egress based on transit marking. However, the 776 Uniform model is not conceptually consistent with network 777 virtualization, which seeks to provide strong isolation between 778 encapsulated traffic and the physical network. 780 [RFC6040] describes the mechanism for exposing ECN capabilities on IP 781 tunnels and propagating congestion markers to the inner packets. 782 This behavior SHOULD be followed for IP packets encapsulated in 783 Geneve. 785 4.1.3. Broadcast and Multicast 787 Geneve tunnels may either be point-to-point unicast between two 788 endpoints or may utilize broadcast or multicast addressing. It is 789 not required that inner and outer addressing match in this respect. 790 For example, in physical networks that do not support multicast, 791 encapsulated multicast traffic may be replicated into multiple 792 unicast tunnels or forwarded by policy to a unicast location 793 (possibly to be replicated there). 795 With physical networks that do support multicast it may be desirable 796 to use this capability to take advantage of hardware replication for 797 encapsulated packets. In this case, multicast addresses may be 798 allocated in the physical network corresponding to tenants, 799 encapsulated multicast groups, or some other factor. The allocation 800 of these groups is a component of the control plane and therefore 801 outside of the scope of this document. When physical multicast is in 802 use, the 'C' bit in the Geneve header may be used with groups of 803 devices with heterogeneous capabilities as each device can interpret 804 only the options that are significant to it if they are not critical. 806 4.2. NIC Offloads 808 Modern NICs currently provide a variety of offloads to enable the 809 efficient processing of packets. The implementation of many of these 810 offloads requires only that the encapsulated packet be easily parsed 811 (for example, checksum offload). However, optimizations such as LSO 812 and LRO involve some processing of the options themselves since they 813 must be replicated/merged across multiple packets. In these 814 situations, it is desirable to not require changes to the offload 815 logic to handle the introduction of new options. To enable this, 816 some constraints are placed on the definitions of options to allow 817 for simple processing rules: 819 o When performing LSO, a NIC MUST replicate the entire Geneve header 820 and all options, including those unknown to the device, onto each 821 resulting segment. However, a given option definition may 822 override this rule and specify different behavior in supporting 823 devices. Conversely, when performing LRO, a NIC MAY assume that a 824 binary comparison of the options (including unknown options) is 825 sufficient to ensure equality and MAY merge packets with equal 826 Geneve headers. 828 o Option ordering is not significant and packets with the same 829 options in a different order MAY be processed alike. 831 o NICs performing offloads MUST NOT drop packets with unknown 832 options, including those marked as critical. 834 There is no requirement that a given implementation of Geneve employ 835 the offloads listed as examples above. However, as these offloads 836 are currently widely deployed in commercially available NICs, the 837 rules described here are intended to enable efficient handling of 838 current and future options across a variety of devices. 840 4.3. Inner VLAN Handling 842 Geneve is capable of encapsulating a wide range of protocols and 843 therefore a given implementation is likely to support only a small 844 subset of the possibilities. However, as Ethernet is expected to be 845 widely deployed, it is useful to describe the behavior of VLANs 846 inside encapsulated Ethernet frames. 848 As with any protocol, support for inner VLAN headers is OPTIONAL. In 849 many cases, the use of encapsulated VLANs may be disallowed due to 850 security or implementation considerations. However, in other cases 851 trunking of VLAN frames across a Geneve tunnel can prove useful. As 852 a result, the processing of inner VLAN tags upon ingress or egress 853 from a tunnel endpoint is based upon the configuration of the 854 endpoint and/or control plane and not explicitly defined as part of 855 the data format. 857 5. Interoperability Issues 859 Viewed exclusively from the data plane, Geneve does not introduce any 860 interoperability issues as it appears to most devices as UDP frames. 861 However, as there are already a number of tunnel protocols deployed 862 in network virtualization environments, there is a practical question 863 of transition and coexistence. 865 Since Geneve is a superset of the functionality of the three most 866 common protocols used for network virtualization (VXLAN, NVGRE, and 867 STT) it should be straightforward to port an existing control plane 868 to run on top of it with minimal effort. With both the old and new 869 frame formats supporting the same set of capabilities, there is no 870 need for a hard transition - endpoints directly communicating with 871 each other use any common protocol, which may be different even 872 within a single overall system. As transit devices are primarily 873 forwarding frames on the basis of the IP header, all protocols appear 874 similar and these devices do not introduce additional 875 interoperability concerns. 877 To assist with this transition, it is strongly suggested that 878 implementations support simultaneous operation of both Geneve and 879 existing tunnel protocols as it is expected to be common for a single 880 node to communicate with a mixture of other nodes. Eventually, older 881 protocols may be phased out as they are no longer in use. 883 6. Security Considerations 885 As UDP/IP packets, Geneve does not have any inherent security 886 mechanisms. As a result, an attacker with access to the underlay 887 network transporting the IP frames has the ability to snoop or inject 888 packets. Legitimate but malicious tunnel endpoints may also spoof 889 identifiers in the tunnel header to gain access to networks owned by 890 other tenants. 892 Within a particular security domain, such as a data center operated 893 by a single provider, the most common and highest performing security 894 mechanism is isolation of trusted components. Tunnel traffic can be 895 carried over a separate VLAN and filtered at any untrusted 896 boundaries. In addition, tunnel endpoints should only be operated in 897 environments controlled by the service provider, such as the 898 hypervisor itself rather than within a customer VM. 900 When crossing an untrusted link, such as the public Internet, IPsec 901 [RFC4301] may be used to provide authentication and/or encryption of 902 the IP packets. If the remote tunnel endpoint is not completely 903 trusted, for example it resides on a customer premises, then it may 904 also be necessary to sanitize any tunnel metadata to prevent tenant- 905 hopping attacks. 907 7. IANA Considerations 909 IANA has allocated UDP port 6081 as the well-known destination port 910 for Geneve. Upon publication, the registry should be updated to cite 911 this document. The original request was: 913 Service Name: geneve 914 Transport Protocol(s): UDP 915 Assignee: Jesse Gross 916 Contact: Jesse Gross 917 Description: Generic Network Virtualization Encapsulation (Geneve) 918 Reference: This document 919 Port Number: 6081 921 In addition, IANA is requested to create a "Geneve Option Class" 922 registry to allocate Option Classes. This shall be a registry of 923 16-bit hexadecimal values along with descriptive strings. The 924 identifiers 0x0-0xFF are to be reserved for standardized options for 925 allocation by IETF Review [RFC5226] and 0xFFFF for Experimental Use. 926 Otherwise, identifiers are to be assigned to any organization with an 927 interest in creating Geneve options on a First Come First Served 928 basis. There are no initial registry assignments. 930 8. Acknowledgements 932 The authors wish to thank Martin Casado, Bruce Davie and Dave Thaler 933 for their input, feedback, and helpful suggestions. 935 9. References 937 9.1. Normative References 939 [RFC0768] Postel, J., "User Datagram Protocol", STD 6, RFC 768, 940 August 1980. 942 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 943 Requirement Levels", BCP 14, RFC 2119, March 1997. 945 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 946 IANA Considerations Section in RFCs", BCP 26, RFC 5226, 947 May 2008. 949 9.2. Informative References 951 [ETYPES] The IEEE Registration Authority, "IEEE 802 Numbers", 2013, 952 . 955 [I-D.davie-stt] 956 Davie, B. and J. Gross, "A Stateless Transport Tunneling 957 Protocol for Network Virtualization (STT)", draft-davie- 958 stt-06 (work in progress), April 2014. 960 [I-D.ietf-nvo3-dataplane-requirements] 961 Bitar, N., Lasserre, M., Balus, F., Morin, T., Jin, L., 962 and B. Khasnabish, "NVO3 Data Plane Requirements", draft- 963 ietf-nvo3-dataplane-requirements-03 (work in progress), 964 April 2014. 966 [I-D.sridharan-virtualization-nvgre] 967 Sridharan, M., Greenberg, A., Wang, Y., Garg, P., 968 Venkataramiah, N., Duda, K., Ganga, I., Lin, G., Pearson, 969 M., Thaler, P., and C. Tumuluri, "NVGRE: Network 970 Virtualization using Generic Routing Encapsulation", 971 draft-sridharan-virtualization-nvgre-06 (work in 972 progress), October 2014. 974 [IEEE.802.1Q-2011] 975 IEEE, "IEEE Standard for Local and metropolitan area 976 networks -- Media Access Control (MAC) Bridges and Virtual 977 Bridged Local Area Networks", IEEE Std 802.1Q, 2011. 979 [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191, 980 November 1990. 982 [RFC1981] McCann, J., Deering, S., and J. Mogul, "Path MTU Discovery 983 for IP version 6", RFC 1981, August 1996. 985 [RFC2983] Black, D., "Differentiated Services and Tunnels", RFC 986 2983, October 2000. 988 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 989 Label Switching Architecture", RFC 3031, January 2001. 991 [RFC4301] Kent, S. and K. Seo, "Security Architecture for the 992 Internet Protocol", RFC 4301, December 2005. 994 [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion 995 Notification", RFC 6040, November 2010. 997 [RFC6935] Eubanks, M., Chimento, P., and M. Westerlund, "IPv6 and 998 UDP Checksums for Tunneled Packets", RFC 6935, April 2013. 1000 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 1001 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 1002 eXtensible Local Area Network (VXLAN): A Framework for 1003 Overlaying Virtualized Layer 2 Networks over Layer 3 1004 Networks", RFC 7348, August 2014. 1006 [RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. 1007 Rekhter, "Framework for Data Center (DC) Network 1008 Virtualization", RFC 7365, October 2014. 1010 [VL2] Greenberg et al, , "VL2: A Scalable and Flexible Data 1011 Center Network", 2009. 1013 Proc. ACM SIGCOMM 2009 1015 Authors' Addresses 1017 Jesse Gross 1018 VMware, Inc. 1019 3401 Hillview Ave. 1020 Palo Alto, CA 94304 1021 USA 1023 Email: jgross@vmware.com 1025 T. Sridhar 1026 VMware, Inc. 1027 3401 Hillview Ave. 1028 Palo Alto, CA 94304 1029 USA 1031 Email: tsridhar@vmware.com 1033 Pankaj Garg 1034 Microsoft Corporation 1035 1 Microsoft Way 1036 Redmond, WA 98052 1037 USA 1039 Email: pankajg@microsoft.com 1040 Chris Wright 1041 Red Hat Inc. 1042 1801 Varsity Drive 1043 Raleigh, NC 27606 1044 USA 1046 Email: chrisw@redhat.com 1048 Ilango Ganga 1049 Intel Corporation 1050 2200 Mission College Blvd. 1051 Santa Clara, CA 95054 1052 USA 1054 Email: ilango.s.ganga@intel.com 1056 Puneet Agarwal 1057 Broadcom Corporation 1058 3151 Zanker Road 1059 San Jose, CA 95134 1060 USA 1062 Email: pagarwal@broadcom.com 1064 Kenneth Duda 1065 Arista Networks 1066 5453 Great America Parkway 1067 Santa Clara, CA 95054 1068 USA 1070 Email: kduda@arista.com 1072 Dinesh G. Dutt 1073 Cumulus Networks 1074 140C S. Whisman Road 1075 Mountain View, CA 94041 1076 USA 1078 Email: ddutt@cumulusnetworks.com 1079 Jon Hudson 1080 Brocade Communications Systems, Inc. 1081 130 Holger Way 1082 San Jose, CA 95134 1083 USA 1085 Email: jon.hudson@gmail.com