idnits 2.17.1 draft-gross-geneve-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 11, 2014) is 3538 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-08) exists of draft-davie-stt-06 == Outdated reference: A later version (-08) exists of draft-sridharan-virtualization-nvgre-05 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group J. Gross 3 Internet-Draft T. Sridhar 4 Intended status: Standards Track VMware 5 Expires: February 12, 2015 P. Garg 6 Microsoft 7 C. Wright 8 Red Hat 9 I. Ganga 10 Intel 11 August 11, 2014 13 Geneve: Generic Network Virtualization Encapsulation 14 draft-gross-geneve-01 16 Abstract 18 Network virtualization involves the cooperation of devices with a 19 wide variety of capabilities such as software and hardware tunnel 20 endpoints, transit fabrics, and centralized control clusters. As a 21 result of their role in tying together different elements in the 22 system, the requirements on tunnels are influenced by all of these 23 components. Flexibility is therefore the most important aspect of a 24 tunnel protocol if it is keep pace with the evolution of the system. 25 This draft describes Geneve, a protocol designed to recognize and 26 accommodate these changing capabilities and needs. 28 Status of This Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at http://datatracker.ietf.org/drafts/current/. 38 Internet-Drafts are draft documents valid for a maximum of six months 39 and may be updated, replaced, or obsoleted by other documents at any 40 time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress." 43 This Internet-Draft will expire on February 12, 2015. 45 Copyright Notice 47 Copyright (c) 2014 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (http://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with respect 55 to this document. Code Components extracted from this document must 56 include Simplified BSD License text as described in Section 4.e of 57 the Trust Legal Provisions and are provided without warranty as 58 described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 63 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 64 1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 65 2. Design Requirements . . . . . . . . . . . . . . . . . . . . . 5 66 2.1. Control Plane Independence . . . . . . . . . . . . . . . 6 67 2.2. Efficient Implementation . . . . . . . . . . . . . . . . 7 68 2.3. Use of Standard IP Fabrics . . . . . . . . . . . . . . . 8 69 3. Geneve Encapsulation Details . . . . . . . . . . . . . . . . 8 70 3.1. Geneve Frame Format Over IPv4 . . . . . . . . . . . . . . 9 71 3.2. Geneve Frame Format Over IPv6 . . . . . . . . . . . . . . 10 72 3.3. UDP Header . . . . . . . . . . . . . . . . . . . . . . . 12 73 3.4. Tunnel Header Fields . . . . . . . . . . . . . . . . . . 13 74 3.5. Tunnel Options . . . . . . . . . . . . . . . . . . . . . 14 75 3.5.1. Options Processing . . . . . . . . . . . . . . . . . 16 76 4. Implementation and Deployment Considerations . . . . . . . . 16 77 4.1. Encapsulation of Geneve in IP . . . . . . . . . . . . . . 16 78 4.1.1. IP Fragmentation . . . . . . . . . . . . . . . . . . 16 79 4.1.2. DSCP and ECN . . . . . . . . . . . . . . . . . . . . 17 80 4.1.3. Broadcast and Multicast . . . . . . . . . . . . . . . 17 81 4.2. NIC Offloads . . . . . . . . . . . . . . . . . . . . . . 18 82 4.3. Inner VLAN Handling . . . . . . . . . . . . . . . . . . . 19 83 5. Interoperability Issues . . . . . . . . . . . . . . . . . . . 19 84 6. Security Considerations . . . . . . . . . . . . . . . . . . . 20 85 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 86 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21 87 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 88 9.1. Normative References . . . . . . . . . . . . . . . . . . 21 89 9.2. Informative References . . . . . . . . . . . . . . . . . 21 90 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 92 1. Introduction 94 Networking has long featured a variety of tunneling, tagging, and 95 other encapsulation mechanisms. However, the advent of network 96 virtualization has caused a surge of renewed interest and a 97 corresponding increase in the introduction of new protocols. The 98 large number of protocols in this space, ranging all the way from 99 VLANs [IEEE.802.1Q-2011] and MPLS [RFC3031] through the more recent 100 VXLAN [I-D.mahalingam-dutt-dcops-vxlan], NVGRE 101 [I-D.sridharan-virtualization-nvgre], and STT [I-D.davie-stt], often 102 leads to questions about the need for new encapsulation formats and 103 what it is about network virtualization in particular that leads to 104 their proliferation. 106 While many encapsulation protocols seek to simply partition the 107 underlay network or bridge between two domains, network 108 virtualization views the transit network as providing connectivity 109 between multiple components of an integrated system. In many ways 110 this system is similar to a chassis switch with the IP underlay 111 network playing the role of the backplane and tunnel endpoints on the 112 edge as line cards. When viewed in this light, the requirements 113 placed on the tunnel protocol are significantly different in terms of 114 the quantity of metadata necessary and the role of transit nodes. 116 Current work such as [VL2] and the NVO3 working group 117 [I-D.ietf-nvo3-dataplane-requirements] have described some of the 118 properties that the data plane must have to support network 119 virtualization. However, one additional defining requirement is the 120 need to carry system state along with the packet data. The use of 121 some metadata is certainly not a foreign concept - nearly all 122 protocols used for virtualization have at least 24 bits of identifier 123 space as a way to partition between tenants. This is often described 124 as overcoming the limits of 12-bit VLANs, and when seen in that 125 context, or any context where it is a true tenant identifier, 16 126 million possible entries is a large number. However, the reality is 127 that the metadata is not exclusively used to identify tenants and 128 encoding other information quickly starts to crowd the space. In 129 fact, when compared to the tags used to exchange metadata between 130 line cards on a chassis switch, 24-bit identifiers start to look 131 quite small. There are nearly endless uses for this metadata, 132 ranging from storing input ports for simple security policies to 133 service based context for interposing advanced middleboxes. 135 Existing tunnel protocols have each attempted to solve different 136 aspects of these new requirements, only to be quickly rendered out of 137 date by changing control plane implementations and advancements. 138 Furthermore, software and hardware components and controllers all 139 have different advantages and rates of evolution - a fact that should 140 be viewed as a benefit, not a liability or limitation. This draft 141 describes Geneve, a protocol which seeks to avoid these problems by 142 providing a framework for tunneling rather than being prescriptive 143 about the entire system. 145 1.1. Requirements Language 147 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 148 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 149 document are to be interpreted as described in RFC 2119 [RFC2119]. 151 In this document, these words will appear with that interpretation 152 only when in ALL CAPS. Lower case uses of these words are not to be 153 interpreted as carrying RFC-2119 significance. 155 1.2. Terminology 157 The following terms are used in this document: 159 Checksum offload. An optimization implemented by many NICs which 160 enables computation and verification of upper layer protocol 161 checksums in hardware on transmit and receive, respectively. This 162 typically includes IP and TCP/UDP checksums which would otherwise be 163 computed by the protocol stack in software. 165 Clos network. A technique for composing network fabrics larger than 166 a single switch while maintaining non-blocking bandwidth across 167 connection points. ECMP is used to divide traffic across the 168 multiple links and switches that constitute the fabric. Sometimes 169 termed "leaf and spine" or "fat tree" topologies. 171 ECMP. Equal Cost Multipath. A routing mechanism for selecting from 172 among multiple best next hop paths by hashing packet headers in order 173 to better utilize network bandwidth while avoiding reordering a 174 single stream. 176 Geneve. Generic Network Virtualization Encapsulation. The tunnel 177 protocol described in this draft. 179 LRO. Large Receive Offload. The receive-side equivalent function of 180 LSO, in which multiple protocol segments (primarily TCP) are 181 coalesced into larger data units. 183 NIC. Network Interface Card. A NIC could be part of a tunnel 184 endpoint or transit device and can either process Geneve packets or 185 aid in the processing of Geneve packets. 187 OAM. Operations, Administration, and Management. A suite of tools 188 used to monitor and troubleshoot network problems. 190 Transit device. A forwarding element along the path of the tunnel. 191 A transit device MAY be capable of understanding the Geneve frame 192 format but does not originate or terminate Geneve packets. 194 LSO. Large Segmentation Offload. A function provided by many 195 commercial NICs that allows data units larger than the MTU to be 196 passed to the NIC to improve performance, the NIC being responsible 197 for creating smaller segments with correct protocol headers. When 198 referring specifically to TCP/IP, this feature is often known as TSO 199 (TCP Segmentation Offload). 201 Tunnel endpoint. A component encapsulating Ethernet frames in Geneve 202 headers and vice versa. As the ultimate consumer of any tunnel 203 metadata, endpoints have the highest level of requirements for 204 parsing and interpreting tunnel headers. Tunnel endpoints may 205 consist of either software or hardware implementations or a 206 combination of the two. 208 VM. Virtual Machine. 210 2. Design Requirements 212 Geneve is designed to support network virtualization use cases, where 213 tunnels are typically established to act as a backplane between the 214 virtual switches residing in hypervisors, physical switches, or 215 middleboxes or other appliances. An arbitrary IP network can be used 216 as an underlay although Clos networks composed using ECMP links are a 217 common choice to provide consistent bisectional bandwidth across all 218 connection points. Figure 1 shows an example of a hypervisor, top of 219 rack switch for connectivity to physical servers, and a WAN uplink 220 connected using Geneve tunnels over a simplified Clos network. These 221 tunnels are used to encapsulate and forward frames from the attached 222 components such as VMs or physical links. 224 +---------------------+ +-------+ +------+ 225 | +--+ +-------+---+ | |Transit|--|Top of|==Physical 226 | |VM|--| | | | +------+ /|Router | | Rack |==Servers 227 | +--+ |Virtual|NIC|---|Top of|/ +-------+\/+------+ 228 | +--+ |Switch | | | | Rack |\ +-------+/\+------+ 229 | |VM|--| | | | +------+ \|Transit| |Uplink| WAN 230 | +--+ +-------+---+ | |Router |--| |=========> 231 +---------------------+ +-------+ +------+ 232 Hypervisor 234 ()===================================() 235 Switch-Switch Geneve Tunnels 237 Figure 1: Sample Geneve Deployment 239 In order to support the needs of network virtualization, the tunnel 240 protocol should be able to take advantage of the differing (and 241 evolving) capabilities of each type of device in both the underlay 242 and overlay networks. This results in the following requirements 243 being placed on the data plane tunneling protocol: 245 o The data plane is generic and extensible enough to support current 246 and future control planes. 248 o Tunnel components are efficiently implementable in both hardware 249 and software without restricting capabilities to the lowest common 250 denominator. 252 o High performance over existing IP fabrics. 254 These requirements are described further in the following 255 subsections. 257 2.1. Control Plane Independence 259 Although some protocols for network virtualization have included a 260 control plane as part of the tunnel format specification (most 261 notably, the original VXLAN spec prescribed a multicast learning- 262 based control plane), these specifications have largely been treated 263 as describing only the data format. The VXLAN frame format has 264 actually seen a wide variety of control planes built on top of it. 266 There is a clear advantage in settling on a data format: most of the 267 protocols are only superficially different and there is little 268 advantage in duplicating effort. However, the same cannot be said of 269 control planes, which are diverse in very fundamental ways. The case 270 for standardization is also less clear given the wide variety in 271 requirements, goals, and deployment scenarios. 273 As a result of this reality, Geneve aims to be a pure tunnel format 274 specification that is capable of fulfilling the needs of many control 275 planes by explicitly not selecting any one of them. This 276 simultaneously promotes a shared data format and increases the 277 chances that it will not be obsoleted by future control plane 278 enhancements. 280 Achieving this level of flexibility effectively requires an options 281 infrastructure to allow new metadata types to be defined, deployed, 282 and either finalized or retired. Options also allow for 283 differentiation of products by encouraging independent development in 284 each vendor's core specialty, leading to an overall faster pace of 285 advancement. By far the most common mechanism for implementing 286 options is Type-Length-Value (TLV) format. 288 It should be noted that while options can be used to support non- 289 wirespeed control frames, they are equally important on data frames 290 as well to segregate and direct forwarding (for instance, the 291 examples given before of input port based security policies and 292 service interposition both require tags to be placed on data 293 packets). Therefore, while it would be desirable to limit the 294 extensibility to only control frames for the purposes of simplifying 295 the datapath, that would not satisfy the design requirements. 297 2.2. Efficient Implementation 299 There is often a conflict between software flexibility and hardware 300 performance that is difficult to resolve. For a given set of 301 functionality, it is obviously desirable to maximize performance. 302 However, that does not mean new features that cannot be run at that 303 speed today should be disallowed. Therefore, for a protocol to be 304 efficiently implementable means that a set of common capabilities can 305 be reasonably handled across platforms along with a graceful 306 mechanism to handle more advanced features in the appropriate 307 situations. 309 The use of a variable length header and options in a protocol often 310 raises questions about whether it is truly efficiently implementable 311 in hardware. To answer this question in the context of Geneve, it is 312 important to first divide "hardware" into two categories: tunnel 313 endpoints and transit devices. 315 Endpoints must be able to parse the variable header, including any 316 options, and take action. Since these devices are actively 317 participating in the protocol, they are the most affected by Geneve. 318 However, as endpoints are the ultimate consumers of the data, 319 transmitters can tailor their output to the capabilities of the 320 recipient. As new functionality becomes sufficiently well defined to 321 add to endpoints, supporting options can be designed using ordering 322 restrictions and other techniques to ease parsing. 324 Transit devices MAY be able to interpret the options and participate 325 in Geneve packet processing. However, as non-terminating devices, 326 they do not originate or terminate the Geneve packet. The 327 participation of transit devices in Geneve packet processing is 328 OPTIONAL. 330 Further, either tunnel endpoints or transit devices MAY use offload 331 capabilities of NICs such as checksum offload to improve the 332 performance of Geneve packet processing. The presence of a Geneve 333 variable length header SHOULD NOT prevent the tunnel endpoints and 334 transit devices from using such offload capabilities. 336 2.3. Use of Standard IP Fabrics 338 IP has clearly cemented its place as the dominant transport mechanism 339 and many techniques have evolved over time to make it robust, 340 efficient, and inexpensive. As a result, it is natural to use IP 341 fabrics as a transit network for Geneve. Fortunately, the use of IP 342 encapsulation and addressing is enough to achieve the primary goal of 343 delivering packets to the correct point in the network through 344 standard switching and routing. 346 In addition, nearly all underlay fabrics are designed to exploit 347 parallelism in traffic to spread load across multiple links without 348 introducing reordering in individual flows. These equal cost 349 multipathing (ECMP) techniques typically involve parsing and hashing 350 the addresses and port numbers from the packet to select an outgoing 351 link. However, the use of tunnels often results in poor ECMP 352 performance without additional knowledge of the protocol as the 353 encapsulated traffic is hidden from the fabric by design and only 354 endpoint addresses are available for hashing. 356 Since it is desirable for Geneve to perform well on these existing 357 fabrics, it is necessary for entropy from encapsulated packets to be 358 exposed in the tunnel header. The most common technique for this is 359 to use the UDP source port, which is discussed further in 360 Section 3.3. 362 3. Geneve Encapsulation Details 364 The Geneve frame format consists of a compact tunnel header 365 encapsulated in UDP over either IPv4 or IPv6. A small fixed tunnel 366 header provides control information plus a base level of 367 functionality and interoperability with a focus on simplicity. This 368 header is then followed by a set of variable options to allow for 369 future innovation. Finally, the payload consists of a protocol data 370 unit of the indicated type, such as an Ethernet frame. The following 371 subsections provide examples of Geneve frames transported (for 372 example) over Ethernet along with an Ethernet payload. 374 3.1. Geneve Frame Format Over IPv4 376 0 1 2 3 377 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 379 Outer Ethernet Header: 380 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 381 | Outer Destination MAC Address | 382 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 383 | Outer Destination MAC Address | Outer Source MAC Address | 384 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 385 | Outer Source MAC Address | 386 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 387 |Optional Ethertype=C-Tag 802.1Q| Outer VLAN Tag Information | 388 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 389 | Ethertype=0x0800 | 390 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 392 Outer IPv4 Header: 393 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 394 |Version| IHL |Type of Service| Total Length | 395 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 396 | Identification |Flags| Fragment Offset | 397 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 398 | Time to Live |Protocol=17 UDP| Header Checksum | 399 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 400 | Outer Source IPv4 Address | 401 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 402 | Outer Destination IPv4 Address | 403 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 405 Outer UDP Header: 406 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 407 | Source Port = xxxx | Dest Port = 6081 | 408 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 409 | UDP Length | UDP Checksum | 410 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 412 Geneve Header: 413 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 414 |Ver| Opt Len |O|C| Rsvd. | Protocol Type | 415 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 416 | Virtual Network Identifier (VNI) | Reserved | 417 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 418 | Variable Length Options | 419 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 421 Inner Ethernet Header: 422 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 423 | Inner Destination MAC Address | 424 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 425 | Inner Destination MAC Address | Inner Source MAC Address | 426 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 427 | Inner Source MAC Address | 428 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 429 |Optional Ethertype=C-Tag 802.1Q| Inner VLAN Tag Information | 430 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 432 Payload: 433 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 434 | Ethertype of Original Payload | | 435 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 436 | Original Ethernet Payload | 437 | | 438 | (Note that the original Ethernet Frame's FCS is not included) | 439 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 441 Frame Check Sequence: 442 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 443 | New FCS (Frame Check Sequence) for Outer Ethernet Frame | 444 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 446 3.2. Geneve Frame Format Over IPv6 447 0 1 2 3 448 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 450 Outer Ethernet Header: 451 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 452 | Outer Destination MAC Address | 453 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 454 | Outer Destination MAC Address | Outer Source MAC Address | 455 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 456 | Outer Source MAC Address | 457 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 458 |Optional Ethertype=C-Tag 802.1Q| Outer VLAN Tag Information | 459 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 460 | Ethertype=0x86DD | 461 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 463 Outer IPv6 Header: 464 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 465 |Version| Traffic Class | Flow Label | 466 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 467 | Payload Length | NxtHdr=17 UDP | Hop Limit | 468 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 469 | | 470 + + 471 | | 472 + Outer Source IPv6 Address + 473 | | 474 + + 475 | | 476 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 477 | | 478 + + 479 | | 480 + Outer Destination IPv6 Address + 481 | | 482 + + 483 | | 484 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 486 Outer UDP Header: 487 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 488 | Source Port = xxxx | Dest Port = 6081 | 489 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 490 | UDP Length | UDP Checksum | 491 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 493 Geneve Header: 494 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 495 |Ver| Opt Len |O|C| Rsvd. | Protocol Type | 496 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 497 | Virtual Network Identifier (VNI) | Reserved | 498 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 499 | Variable Length Options | 500 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 502 Inner Ethernet Header: 503 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 504 | Inner Destination MAC Address | 505 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 506 | Inner Destination MAC Address | Inner Source MAC Address | 507 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 508 | Inner Source MAC Address | 509 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 510 |Optional Ethertype=C-Tag 802.1Q| Inner VLAN Tag Information | 511 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 513 Payload: 514 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 515 | Ethertype of Original Payload | | 516 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 517 | Original Ethernet Payload | 518 | | 519 | (Note that the original Ethernet Frame's FCS is not included) | 520 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 522 Frame Check Sequence: 523 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 524 | New FCS (Frame Check Sequence) for Outer Ethernet Frame | 525 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 527 3.3. UDP Header 529 The use of an encapsulating UDP [RFC0768] header follows the 530 connectionless semantics of Ethernet and IP in addition to providing 531 entropy to routers performing ECMP. The header fields are therefore 532 interpreted as follows: 534 Source port: A source port selected by the ingress tunnel endpoint. 535 This source port SHOULD be the same for all packets belonging to a 536 single encapsulated flow to prevent reordering due to the use of 537 different paths. To encourage an even distribution of flows 538 across multiple links, the source port MAY be calculated using a 539 hash of the encapsulated packet headers using, for example, a 540 traditional 5-tuple. Since the port represents a flow identifier 541 rather than a true UDP connection, the entire 16-bit range MAY be 542 used to maximize entropy. 544 Dest port: IANA has assigned port 6081 as the fixed well-known 545 destination port for Geneve. This port MUST be used in both 546 directions of a flow. Although the well-known value should be 547 used by default, it is RECOMMENDED that implementations make this 548 configurable. 550 UDP length: The length of the UDP packet including the UDP header. 552 UDP checksum: The checksum MAY be set to zero on transmit for 553 packets encapsulated in both IPv4 and IPv6 [RFC6935]. When a 554 packet is received with a UDP checksum of zero it MUST be accepted 555 and decapsulated. If the ingress tunnel endpoint optionally 556 encapsulates a packet with a non-zero checksum, it MUST be a 557 correctly computed UDP checksum. Upon receiving such a packet, 558 the egress endpoint MUST validate the checksum. If the checksum 559 is not correct, the packet MUST be dropped, otherwise the packet 560 MUST be accepted for decapsulation. It is RECOMMENDED that the 561 UDP checksum be computed to protect the Geneve header and options 562 in situations where the network reliability is not high and the 563 packet is not protected by another checksum or CRC. 565 3.4. Tunnel Header Fields 567 Ver (2 bits): The current version number is 0. Packets received by 568 an endpoint with an unknown version MUST be dropped. Non- 569 terminating devices processing Geneve packets with an unknown 570 version number MUST treat them as UDP packets with an unknown 571 payload. 573 Opt Len (6 bits): The length of the options fields, expressed in 574 four byte multiples, not including the eight byte fixed tunnel 575 header. This results in a minimum total Geneve header size of 8 576 bytes and a maximum of 260 bytes. The start of the payload 577 headers can be found using this offset from the end of the base 578 Geneve header. 580 O (1 bit): OAM frame. This packet contains a control message 581 instead of a data payload. Endpoints MUST NOT forward the payload 582 and transit devices MUST NOT attempt to interpret or process it. 583 Since these are infrequent control messages, it is RECOMMENDED 584 that endpoints direct these packets to a high priority control 585 queue (for example, to direct the packet to a general purpose CPU 586 from a forwarding ASIC or to separate out control traffic on a 587 NIC). Transit devices MUST NOT alter forwarding behavior on the 588 basis of this bit, such as ECMP link selection. 590 C (1 bit): Critical options present. One or more options has the 591 critical bit set (see Section 3.5). If this bit is set then 592 tunnel endpoints MUST parse the options list to interpret any 593 critical options. If no option types are supported then endpoints 594 MAY silently drop the frame on the basis of the 'C' bit (including 595 invalid combinations such as 'C' bit set and 'Opt Len' is zero or 596 no options with a corresponding 'C' bit). If the bit is not set 597 tunnel endpoints MAY strip all options using 'Opt Len' and forward 598 the decapsulated frame. Transit devices MUST NOT drop or modify 599 packets on the basis of this bit. 601 Rsvd. (6 bits): Reserved field which MUST be zero on transmission 602 and ignored on receipt. 604 Protocol Type (16 bits): The type of the protocol data unit 605 appearing after the Geneve header. This follows the EtherType 606 [ETYPES] convention with Ethernet itself being represented by the 607 value 0x6558. 609 Virtual Network Identifier (VNI) (24 bits): An identifier for a 610 unique element of a virtual network. In many situations this may 611 represent an L2 segment, however, the control plane defines the 612 forwarding semantics of decapsulated packets. The VNI MAY be used 613 as part of ECMP forwarding decisions or MAY be used as a mechanism 614 to distinguish between overlapping address spaces contained in the 615 encapsulated packet when load balancing across CPUs. 617 Reserved (8 bits): Reserved field which MUST be zero on transmission 618 and ignored on receipt. 620 Transit devices MUST maintain consistent forwarding behavior 621 irrespective of the value of 'Opt Len', including ECMP link 622 selection. These devices SHOULD be able to forward packets 623 containing options without resorting to a slow path. 625 3.5. Tunnel Options 627 0 1 2 3 628 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 629 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 630 | Option Class | Type |R|R|R| Length | 631 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 632 | Variable Option Data | 633 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 635 Geneve Option 637 The base Geneve header is followed by zero or more options in Type- 638 Length-Value format. Each option consists of a four byte option 639 header and a variable amount of option data interpreted according to 640 the type. 642 Option Class (16 bits): Namespace for the 'Type' field. IANA will 643 be requested to create a "Geneve Option Class" registry to 644 allocate identifiers for organizations, technologies, and vendors 645 that have an interest in creating types for options. Each 646 organization may allocate types independently to allow 647 experimentation and rapid innovation. It is expected that over 648 time certain options will become well known and a given 649 implementation may use option types from a variety of sources. In 650 addition, IANA will be requested to reserve specific ranges for 651 standardized and experimental options. 653 Type (8 bits): Type indicating the format of the data contained in 654 this option. Options are primarily designed to encourage future 655 extensibility and innovation and so standardized forms of these 656 options will be defined in a separate document. 658 The high order bit of the option type indicates that this is a 659 critical option. If the receiving endpoint does not recognize 660 this option and this bit is set then the frame MUST be dropped. 661 If the critical bit is set in any option then the 'C' bit in the 662 Geneve base header MUST also be set. Transit devices MUST NOT 663 drop packets on the basis of this bit. 665 The requirement to drop a packet with an unknown critical option 666 applies to the entire tunnel endpoint system and not a particular 667 component of the implementation. For example, in a system 668 comprised of a forwarding ASIC and a general purpose CPU, this 669 does not mean that the packet must be dropped in the ASIC. An 670 implementation may send the packet to the CPU using a rate-limited 671 control channel for slow-path exception handling. 673 R (1 bit): Option control flag reserved for future use. MUST be 674 zero on transmission and ignored on receipt. 676 Length (5 bits): Length of the option, expressed in four byte 677 multiples excluding the option header. The total length of each 678 option may be between 4 and 128 bytes. Packets in which the total 679 length of all options is not equal to the 'Opt Len' in the base 680 header are invalid and MUST be silently dropped if received by an 681 endpoint. 683 Variable Option Data: Option data interpreted according to 'Type'. 685 3.5.1. Options Processing 687 Geneve options are intended to be originated and processed by tunnel 688 endpoints. Options MAY be processed by transit devices along the 689 tunnel path as well. This document only details the handling of 690 options by tunnel endpoints. A future version of this document will 691 provide details of options processing by transit devices. Transit 692 devices not processing Geneve options SHOULD process Geneve frame as 693 any other UDP frame and maintain consistent forwarding behavior. 695 In tunnel endpoints, the generation and interpretation of options is 696 determined by the control plane, which is out of the scope of this 697 document. However, to ensure interoperability between heterogeneous 698 devices two requirements are imposed on endpoint devices: 700 o Receiving endpoints MUST drop packets containing unknown options 701 with the 'C' bit set in the option type. 703 o Sending endpoints MUST NOT assume that options will be processed 704 sequentially by the receiver in the order they were transmitted. 706 4. Implementation and Deployment Considerations 708 4.1. Encapsulation of Geneve in IP 710 As an IP-based tunnel protocol, Geneve shares many properties and 711 techniques with existing protocols. The application of some of these 712 are described in further detail, although in general most concepts 713 applicable to the IP layer or to IP tunnels generally also function 714 in the context of Geneve. 716 4.1.1. IP Fragmentation 718 In order to prevent fragmentation and maximize performance, the best 719 practice when using Geneve is to ensure that the MTU of the physical 720 network is greater than or equal to the MTU of the encapsulated 721 network plus tunnel headers. Manual or upper layer (such as TCP MSS 722 clamping) configuration can be used to ensure that fragmentation 723 never takes place, however, in some situations this may not be 724 feasible. 726 It is RECOMMENDED that Path MTU Discovery be used by setting the DF 727 bit in the IP header when Geneve packets are transmitted over IPv4 728 (this is the default with IPv6). The use of Path MTU Discovery on 729 the transit network provides the encapsulating endpoint with soft- 730 state about the link that it may use to prevent or minimize 731 fragmentation depending on its role in the virtualized network. 733 If necessary, it is RECOMMENDED that fragmentation be performed 734 preferentially on the encapsulated payload. This may be possible if 735 the encapsulating endpoint is also acting as an L3 node in the 736 virtualized network, in which case the endpoint might use the derived 737 transit MTU and the tunnel header length to either implement Path MTU 738 Discovery or fragment the inner packet to the correct size. 740 In many cases it may not be possible or desirable for the tunnel 741 endpoint to interact with the payload, such as when implementing a 742 completely transparent L2 bridge. In these situations, fragmentation 743 of the transit IP header MAY be performed to ensure connectivity. If 744 a packet is fragmented endpoints SHOULD use the path MTU of the 745 transit link to ensure a size is chosen such that fragmentation is 746 only required once between endpoints. Note that some implementations 747 may not be capable of supporting fragmentation or other less common 748 features of the IP header, such as options and extension headers. 750 4.1.2. DSCP and ECN 752 When encapsulating IP (including over Ethernet) frames in Geneve, 753 there are several options for propagating DSCP and ECN bits from the 754 inner header to the tunnel on transmission and the reverse on 755 reception. 757 [RFC2983] lists considerations for mapping DSCP between inner and 758 outer IP headers. Network virtualization is typically more closely 759 aligned with the Pipe model described, where the DSCP value on the 760 tunnel header is set based on a policy (which may be a fixed value, 761 one based on the inner traffic class, or some other mechanism for 762 grouping traffic). Aspects of the Uniform model (which treats the 763 inner and outer DSCP value as a single field by copying on ingress 764 and egress) may also apply, such as the ability to remark the inner 765 header on tunnel egress based on transit marking. However, the 766 Uniform model is not conceptually consistent with network 767 virtualization, which seeks to provide strong isolation between 768 encapsulated traffic and the physical network. 770 [RFC6040] describes the mechanism for exposing ECN capabilities on IP 771 tunnels and propagating congestion markers to the inner packets. 772 This behavior SHOULD be followed for IP packets encapsulated in 773 Geneve. 775 4.1.3. Broadcast and Multicast 777 Geneve tunnels may either be point-to-point unicast between two 778 endpoints or may utilize broadcast or multicast addressing. It is 779 not required that inner and outer addressing match in this respect. 780 For example, in physical networks that do not support multicast, 781 encapsulated multicast traffic may be replicated into multiple 782 unicast tunnels or forwarded by policy to a unicast location 783 (possibly to be replicated there). 785 With physical networks that do support multicast it may be desirable 786 to use this capability to take advantage of hardware replication for 787 encapsulated packets. In this case, multicast addresses may be 788 allocated in the physical network corresponding to tenants, 789 encapsulated multicast groups, or some other factor. The allocation 790 of these groups is a component of the control plane and therefore 791 outside of the scope of this document. When physical multicast is in 792 use, the 'C' bit in the Geneve header may be used with groups of 793 devices with heterogeneous capabilities as each device can interpret 794 only the options that are significant to it if they are not critical. 796 4.2. NIC Offloads 798 Modern NICs currently provide a variety of offloads to enable the 799 efficient processing of packets. The implementation of many of these 800 offloads requires only that the encapsulated packet be easily parsed 801 (for example, checksum offload). However, optimizations such as LSO 802 and LRO involve some processing of the options themselves since they 803 must be replicated/merged across multiple packets. In these 804 situations, it is desirable to not require changes to the offload 805 logic to handle the introduction of new options. To enable this, 806 some constraints are placed on the definitions of options to allow 807 for simple processing rules: 809 o When performing LSO, a NIC MUST replicate the entire Geneve header 810 and all options, including those unknown to the device, onto each 811 resulting segment. However, a given option definition may 812 override this rule and specify different behavior in supporting 813 devices. Conversely, when performing LRO, a NIC MAY assume that a 814 binary comparison of the options (including unknown options) is 815 sufficient to ensure equality and MAY merge packets with equal 816 Geneve headers. 818 o Option ordering is not significant and packets with the same 819 options in a different order MAY be processed alike. 821 o NICs performing offloads MUST NOT drop packets with unknown 822 options, including those marked as critical. 824 There is no requirement that a given implementation of Geneve employ 825 the offloads listed as examples above. However, as these offloads 826 are currently widely deployed in commercially available NICs, the 827 rules described here are intended to enable efficient handling of 828 current and future options across a variety of devices. 830 4.3. Inner VLAN Handling 832 Geneve is capable of encapsulating a wide range of protocols and 833 therefore a given implementation is likely to support only a small 834 subset of the possibilities. However, as Ethernet is expected to be 835 widely deployed, it is useful to describe the behavior of VLANs 836 inside encapsulated Ethernet frames. 838 As with any protocol, support for inner VLAN headers is OPTIONAL. In 839 many cases, the use of encapsulated VLANs may be disallowed due to 840 security or implementation considerations. However, in other cases 841 trunking of VLAN frames across a Geneve tunnel can prove useful. As 842 a result, the processing of inner VLAN tags upon ingress or egress 843 from a tunnel endpoint is based upon the configuration of the 844 endpoint and/or control plane and not explicitly defined as part of 845 the data format. 847 5. Interoperability Issues 849 Viewed exclusively from the data plane, Geneve does not introduce any 850 interoperability issues as it appears to most devices as UDP frames. 851 However, as there are already a number of tunnel protocols deployed 852 in network virtualization environments, there is a practical question 853 of transition and coexistence. 855 Since Geneve is a superset of the functionality of the three most 856 common protocols used for network virtualization (VXLAN, NVGRE, and 857 STT) it should be straightforward to port an existing control plane 858 to run on top of it with minimal effort. With both the old and new 859 frame formats supporting the same set of capabilities, there is no 860 need for a hard transition - endpoints directly communicating with 861 each other use any common protocol, which may be different even 862 within a single overall system. As transit devices are primarily 863 forwarding frames on the basis of the IP header, all protocols appear 864 similar and these devices do not introduce additional 865 interoperability concerns. 867 In order to assist with this transition, it is strongly suggested 868 that implementations support simultaneous operation of both Geneve 869 and existing tunnel protocols as it is expected to be common for a 870 single node to communicate with a mixture of other nodes. 871 Eventually, older protocols may be phased out as they are no longer 872 in use. 874 6. Security Considerations 876 As UDP/IP packets, Geneve does not have any inherent security 877 mechanisms. As a result, an attacker with access to the underlay 878 network transporting the IP frames has the ability to snoop or inject 879 packets. Legitimate but malicious tunnel endpoints may also spoof 880 identifiers in the tunnel header to gain access to networks owned by 881 other tenants. 883 Within a particular security domain, such as a data center operated 884 by a single provider, the most common and highest performing security 885 mechanism is isolation of trusted components. Tunnel traffic can be 886 carried over a separate VLAN and filtered at any untrusted 887 boundaries. In addition, tunnel endpoints should only be operated in 888 environments controlled by the service provider, such as the 889 hypervisor itself rather than within a customer VM. 891 When crossing an untrusted link, such as the public Internet, IPsec 892 [RFC4301] may be used to provide authentication and/or encryption of 893 the IP packets. If the remote tunnel endpoint is not completely 894 trusted, for example it resides on a customer premises, then it may 895 also be necessary to sanitize any tunnel metadata to prevent tenant- 896 hopping attacks. 898 7. IANA Considerations 900 IANA has allocated UDP port 6081 as the well-known destination port 901 for Geneve. Upon publication, the registry should be updated to cite 902 this document. The original request was: 904 Service Name: geneve 905 Transport Protocol(s): UDP 906 Assignee: Jesse Gross 907 Contact: Jesse Gross 908 Description: Generic Network Virtualization Encapsulation (Geneve) 909 Reference: This document 910 Port Number: 6081 912 In addition, IANA will be requested to create a "Geneve Option Class" 913 registry to allocate Option Classes. This shall be a registry of 914 16-bit hexadecimal values along with descriptive strings. The 915 identifiers 0x0-0xFF are to be reserved for standardized options for 916 allocation by an IETF working group document or RFC and 0xFFFF for 917 experimental use. Otherwise, identifiers are to be assigned to any 918 organization with an interest in creating Geneve options on a first 919 come first serve basis. 921 8. Acknowledgements 923 The authors wish to thank Martin Casado, Bruce Davie and Dave Thaler 924 for their input, feedback, and helpful suggestions. 926 9. References 928 9.1. Normative References 930 [RFC0768] Postel, J., "User Datagram Protocol", STD 6, RFC 768, 931 August 1980. 933 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 934 Requirement Levels", BCP 14, RFC 2119, March 1997. 936 9.2. Informative References 938 [ETYPES] The IEEE Registration Authority, "IEEE 802 Numbers", 2013, 939 . 942 [I-D.davie-stt] 943 Davie, B. and J. Gross, "A Stateless Transport Tunneling 944 Protocol for Network Virtualization (STT)", draft-davie- 945 stt-06 (work in progress), April 2014. 947 [I-D.ietf-nvo3-dataplane-requirements] 948 Bitar, N., Lasserre, M., Balus, F., Morin, T., Jin, L., 949 and B. Khasnabish, "NVO3 Data Plane Requirements", draft- 950 ietf-nvo3-dataplane-requirements-03 (work in progress), 951 April 2014. 953 [I-D.mahalingam-dutt-dcops-vxlan] 954 Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 955 L., Sridhar, T., Bursell, M., and C. Wright, "VXLAN: A 956 Framework for Overlaying Virtualized Layer 2 Networks over 957 Layer 3 Networks", draft-mahalingam-dutt-dcops-vxlan-09 958 (work in progress), April 2014. 960 [I-D.sridharan-virtualization-nvgre] 961 Sridharan, M., Greenberg, A., Wang, Y., Garg, P., 962 Venkataramiah, N., Duda, K., Ganga, I., Lin, G., Pearson, 963 M., Thaler, P., and C. Tumuluri, "NVGRE: Network 964 Virtualization using Generic Routing Encapsulation", 965 draft-sridharan-virtualization-nvgre-05 (work in 966 progress), July 2014. 968 [IEEE.802.1Q-2011] 969 IEEE, "IEEE Standard for Local and metropolitan area 970 networks -- Media Access Control (MAC) Bridges and Virtual 971 Bridged Local Area Networks", IEEE Std 802.1Q, 2011. 973 [RFC2983] Black, D., "Differentiated Services and Tunnels", RFC 974 2983, October 2000. 976 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 977 Label Switching Architecture", RFC 3031, January 2001. 979 [RFC4301] Kent, S. and K. Seo, "Security Architecture for the 980 Internet Protocol", RFC 4301, December 2005. 982 [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion 983 Notification", RFC 6040, November 2010. 985 [RFC6935] Eubanks, M., Chimento, P., and M. Westerlund, "IPv6 and 986 UDP Checksums for Tunneled Packets", RFC 6935, April 2013. 988 [VL2] Greenberg et al, , "VL2: A Scalable and Flexible Data 989 Center Network", 2009. 991 Proc. ACM SIGCOMM 2009 993 Authors' Addresses 995 Jesse Gross 996 VMware, Inc. 997 3401 Hillview Ave. 998 Palo Alto, CA 94304 999 USA 1001 Email: jgross@vmware.com 1003 T. Sridhar 1004 VMware, Inc. 1005 3401 Hillview Ave. 1006 Palo Alto, CA 94304 1007 USA 1009 Email: tsridhar@vmware.com 1010 Pankaj Garg 1011 Microsoft Corporation 1012 1 Microsoft Way 1013 Redmond, WA 98052 1014 USA 1016 Email: pankajg@microsoft.com 1018 Chris Wright 1019 Red Hat Inc. 1020 1801 Varsity Drive 1021 Raleigh, NC 27606 1022 USA 1024 Email: chrisw@redhat.com 1026 Ilango Ganga 1027 Intel Corporation 1028 2200 Mission College Blvd. 1029 Santa Clara, CA 95054 1030 USA 1032 Email: ilango.s.ganga@intel.com