idnits 2.17.1 draft-ietf-nvo3-framework-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 4, 2014) is 3582 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Obsolete informational reference (is this intentional?): RFC 1981 (Obsoleted by RFC 8201) -- Obsolete informational reference (is this intentional?): RFC 2679 (Obsoleted by RFC 7679) -- Obsolete informational reference (is this intentional?): RFC 2680 (Obsoleted by RFC 7680) Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force Marc Lasserre 2 Internet Draft Florin Balus 3 Intended status: Informational Alcatel-Lucent 4 Expires: Jan 2015 5 Thomas Morin 6 France Telecom Orange 8 Nabil Bitar 9 Verizon 11 Yakov Rekhter 12 Juniper 14 July 4, 2014 16 Framework for DC Network Virtualization 17 draft-ietf-nvo3-framework-09.txt 19 Abstract 21 This document provides a framework for Data Center (DC) Network 22 Virtualization Overlays (NVO3) and it defines a reference model 23 along with logical components required to design a solution. 25 Status of this Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at http://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six 36 months and may be updated, replaced, or obsoleted by other documents 37 at any time. It is inappropriate to use Internet-Drafts as 38 reference material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on Jan 4, 2015. 42 Copyright Notice 44 Copyright (c) 2014 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with 52 respect to this document. Code Components extracted from this 53 document must include Simplified BSD License text as described in 54 Section 4.e of the Trust Legal Provisions and are provided without 55 warranty as described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction..................................................3 60 1.1. General terminology......................................3 61 1.2. DC network architecture..................................6 62 2. Reference Models..............................................8 63 2.1. Generic Reference Model..................................8 64 2.2. NVE Reference Model.....................................10 65 2.3. NVE Service Types.......................................10 66 2.3.1. L2 NVE providing Ethernet LAN-like service.........11 67 2.3.2. L3 NVE providing IP/VRF-like service...............11 68 2.4. Operational Management Considerations...................11 69 3. Functional components........................................12 70 3.1. Service Virtualization Components.......................12 71 3.1.1. Virtual Access Points (VAPs).......................12 72 3.1.2. Virtual Network Instance (VNI).....................12 73 3.1.3. Overlay Modules and VN Context.....................12 74 3.1.4. Tunnel Overlays and Encapsulation options..........13 75 3.1.5. Control Plane Components...........................14 76 3.1.5.1. Distributed vs Centralized Control Plane.........14 77 3.1.5.2. Auto-provisioning/Service discovery..............14 78 3.1.5.3. Address advertisement and tunnel mapping.........15 79 3.1.5.4. Overlay Tunneling................................15 80 3.2. Multi-homing............................................16 81 3.3. VM Mobility.............................................17 82 4. Key aspects of overlay networks..............................17 83 4.1. Pros & Cons.............................................17 84 4.2. Overlay issues to consider..............................19 85 4.2.1. Data plane vs Control plane driven.................19 86 4.2.2. Coordination between data plane and control plane..19 87 4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) 88 traffic...................................................19 89 4.2.4. Path MTU...........................................20 90 4.2.5. NVE location trade-offs............................21 91 4.2.6. Interaction between network overlays and underlays.22 92 5. Security Considerations......................................22 93 6. IANA Considerations..........................................23 94 7. References...................................................23 95 7.1. Informative References..................................23 96 8. Acknowledgments..............................................25 98 1. Introduction 100 This document provides a framework for Data Center (DC) Network 101 Virtualization over Layer3 (L3) tunnels. This framework is intended 102 to aid in standardizing protocols and mechanisms to support large- 103 scale network virtualization for data centers. 105 [NVOPS] defines the rationale for using overlay networks in order to 106 build large multi-tenant data center networks. Compute, storage and 107 network virtualization are often used in these large data centers to 108 support a large number of communication domains and end systems. 110 This document provides reference models and functional components of 111 data center overlay networks as well as a discussion of technical 112 issues that have to be addressed. 114 1.1. General terminology 116 This document uses the following terminology: 118 NVO3 Network: An overlay network that provides a Layer2 (L2) or 119 Layer3 (L3) service to Tenant Systems over an L3 underlay network 120 using the architecture and protocols as defined by the NVO3 Working 121 Group. 123 Network Virtualization Edge (NVE). An NVE is the network entity that 124 sits at the edge of an underlay network and implements L2 and/or L3 125 network virtualization functions. The network-facing side of the NVE 126 uses the underlying L3 network to tunnel tenant frames to and from 127 other NVEs. The tenant-facing side of the NVE sends and receives 128 Ethernet frames to and from individual Tenant Systems. An NVE could 129 be implemented as part of a virtual switch within a hypervisor, a 130 physical switch or router, a Network Service Appliance, or be split 131 across multiple devices. 133 Virtual Network (VN): A VN is a logical abstraction of a physical 134 network that provides L2 or L3 network services to a set of Tenant 135 Systems. A VN is also known as a Closed User Group (CUG). 137 Virtual Network Instance (VNI): A specific instance of a VN from the 138 perspective of an NVE. 140 Virtual Network Context (VN Context) Identifier: Field in overlay 141 encapsulation header that identifies the specific VN the packet 142 belongs to. The egress NVE uses the VN Context identifier to deliver 143 the packet to the correct Tenant System. The VN Context identifier 144 can be a locally significant identifier or a globally unique 145 identifier. 147 Underlay or Underlying Network: The network that provides the 148 connectivity among NVEs and over which NVO3 packets are tunneled, 149 where an NVO3 packet carries an NVO3 overlay header followed by a 150 tenant packet. The Underlay Network does not need to be aware that 151 it is carrying NVO3 packets. Addresses on the Underlay Network 152 appear as "outer addresses" in encapsulated NVO3 packets. In 153 general, the Underlay Network can use a completely different 154 protocol (and address family) from that of the overlay. In the case 155 of NVO3, the underlay network is IP. 157 Data Center (DC): A physical complex housing physical servers, 158 network switches and routers, network service appliances and 159 networked storage. The purpose of a Data Center is to provide 160 application, compute and/or storage services. One such service is 161 virtualized infrastructure data center services, also known as 162 Infrastructure as a Service. 164 Virtual Data Center (Virtual DC): A container for virtualized 165 compute, storage and network services. A Virtual DC is associated 166 with a single tenant, and can contain multiple VNs and Tenant 167 Systems connected to one or more of these VNs. 169 Virtual machine (VM): A software implementation of a physical 170 machine that runs programs as if they were executing on a physical, 171 non-virtualized machine. Applications (generally) do not know they 172 are running on a VM as opposed to running on a "bare metal" host or 173 server, though some systems provide a para-virtualization 174 environment that allows an operating system or application to be 175 aware of the presence of virtualization for optimization purposes. 177 Hypervisor: Software running on a server that allows multiple VMs to 178 run on the same physical server. The hypervisor manages and provides 179 shared compute/memory/storage and network connectivity to the VMs 180 that it hosts. Hypervisors often embed a Virtual Switch (see below). 182 Server: A physical end host machine that runs user applications. A 183 standalone (or "bare metal") server runs a conventional operating 184 system hosting a single-tenant application. A virtualized server 185 runs a hypervisor supporting one or more VMs. 187 Virtual Switch (vSwitch): A function within a Hypervisor (typically 188 implemented in software) that provides similar forwarding services 189 to a physical Ethernet switch. A vSwitch forwards Ethernet frames 190 between VMs running on the same server, or between a VM and a 191 physical NIC card connecting the server to a physical Ethernet 192 switch or router. A vSwitch also enforces network isolation between 193 VMs that by policy are not permitted to communicate with each other 194 (e.g., by honoring VLANs). A vSwitch may be bypassed when an NVE is 195 enabled on the host server. 197 Tenant: The customer using a virtual network and any associated 198 resources (e.g., compute, storage and network). A tenant could be 199 an enterprise, or a department/organization within an enterprise. 201 Tenant System: A physical or virtual system that can play the role 202 of a host, or a forwarding element such as a router, switch, 203 firewall, etc. It belongs to a single tenant and connects to one or 204 more VNs of that tenant. 206 Tenant Separation: Tenant Separation refers to isolating traffic of 207 different tenants such that traffic from one tenant is not visible 208 to or delivered to another tenant, except when allowed by policy. 209 Tenant Separation also refers to address space separation, whereby 210 different tenants can use the same address space without conflict. 212 Virtual Access Points (VAPs): A logical connection point on the NVE 213 for connecting a Tenant System to a virtual network. Tenant Systems 214 connect to VNIs at an NVE through VAPs. VAPs can be physical ports 215 or virtual ports identified through logical interface identifiers 216 (e.g., VLAN ID, internal vSwitch Interface ID connected to a VM). 218 End Device: A physical device that connects directly to the DC 219 Underlay Network. This is in contrast to a Tenant System, which 220 connects to a corresponding tenant VN. An End Device is administered 221 by the DC operator rather than a tenant, and is part of the DC 222 infrastructure. An End Device may implement NVO3 technology in 223 support of NVO3 functions. Examples of an End Device include hosts 224 (e.g., server or server blade), storage systems (e.g., file servers, 225 iSCSI storage systems), and network devices (e.g., firewall, load- 226 balancer, IPSec gateway). 228 Network Virtualization Authority (NVA): Entity that provides 229 reachability and forwarding information to NVEs. 231 1.2. DC network architecture 233 A generic architecture for Data Centers is depicted in Figure 1: 235 ,---------. 236 ,' `. 237 ( IP/MPLS WAN ) 238 `. ,' 239 `-+------+' 240 \ / 241 +--------+ +--------+ 242 | DC |+-+| DC | 243 |gateway |+-+|gateway | 244 +--------+ +--------+ 245 | / 246 .--. .--. 247 ( ' '.--. 248 .-.' Intra-DC ' 249 ( network ) 250 ( .'-' 251 '--'._.'. )\ \ 252 / / '--' \ \ 253 / / | | \ \ 254 +--------+ +--------+ +--------+ 255 | access | | access | | access | 256 | switch | | switch | | switch | 257 +--------+ +--------+ +--------+ 258 / \ / \ / \ 259 __/_ \ / \ /_ _\__ 260 '--------' '--------' '--------' '--------' 261 : End : : End : : End : : End : 262 : Device : : Device : : Device : : Device : 263 '--------' '--------' '--------' '--------' 265 Figure 1 : A Generic Architecture for Data Centers 267 An example of multi-tier DC network architecture is presented in 268 Figure 1. It provides a view of physical components inside a DC. 270 A DC network is usually composed of intra-DC networks and network 271 services, and inter-DC network and network connectivity services. 273 DC networking elements can act as strict L2 switches and/or provide 274 IP routing capabilities, including network service virtualization. 276 In some DC architectures, some tier layers could provide L2 and/or 277 L3 services. In addition, some tier layers may be collapsed, and 278 Internet connectivity, inter-DC connectivity and VPN support may be 279 handled by a smaller number of nodes. Nevertheless, one can assume 280 that the network functional blocks in a DC fit in the architecture 281 depicted in Figure 1. 283 The following components can be present in a DC: 285 - Access switch: Hardware-based Ethernet switch aggregating all 286 Ethernet links from the End Devices in a rack representing the 287 entry point in the physical DC network for the hosts. It may also 288 provide routing functionality, virtual IP network connectivity, or 289 Layer2 tunneling over IP for instance. Access switches are usually 290 multi-homed to aggregation switches in the Intra-DC network. A 291 typical example of an access switch is a Top of Rack (ToR) switch. 292 Other deployment scenarios may use an intermediate Blade Switch 293 before the ToR, or an EoR (End of Row) switch, to provide similar 294 functions to a ToR. 296 - Intra-DC Network: Network composed of high capacity core nodes 297 (Ethernet switches/routers). Core nodes may provide virtual 298 Ethernet bridging and/or IP routing services. 300 - DC Gateway (DC GW): Gateway to the outside world providing DC 301 Interconnect and connectivity to Internet and VPN customers. In 302 the current DC network model, this may be simply a router 303 connected to the Internet and/or an IP Virtual Private Network 304 (VPN)/L2VPN PE. Some network implementations may dedicate DC GWs 305 for different connectivity types (e.g., a DC GW for Internet, and 306 another for VPN). 308 Note that End Devices may be single or multi-homed to access 309 switches. 311 2. Reference Models 313 2.1. Generic Reference Model 315 Figure 2 depicts a DC reference model for network virtualization 316 overlay where NVEs provide a logical interconnect between Tenant 317 Systems that belong to a specific VN. 319 +--------+ +--------+ 320 | Tenant +--+ +----| Tenant | 321 | System | | (') | System | 322 +--------+ | ................. ( ) +--------+ 323 | +---+ +---+ (_) 324 +--|NVE|---+ +---|NVE|-----+ 325 +---+ | | +---+ 326 / . +-----+ . 327 / . +--| NVA |--+ . 328 / . | +-----+ \ . 329 | . | \ . 330 | . | Overlay +--+--++--------+ 331 +--------+ | . | Network | NVE || Tenant | 332 | Tenant +--+ . | | || System | 333 | System | . \ +---+ +--+--++--------+ 334 +--------+ .....|NVE|......... 335 +---+ 336 | 337 | 338 ===================== 339 | | 340 +--------+ +--------+ 341 | Tenant | | Tenant | 342 | System | | System | 343 +--------+ +--------+ 345 Figure 2 : Generic reference model for DC network virtualization 346 overlay 348 In order to obtain reachability information, NVEs may exchange 349 information directly between themselves via a control plane 350 protocol. In this case, a control plane module resides in every NVE. 352 It is also possible for NVEs to communicate with an external Network 353 Virtualization Authority (NVA) to obtain reachability and forwarding 354 information. In this case, a protocol is used between NVEs and 355 NVA(s) to exchange information. 357 It should be noted that NVAs may be organized in clusters for 358 redundancy and scalability and can appear as one logically 359 centralized controller. In this case, inter-NVA communication is 360 necessary to synchronize state among nodes within a cluster or share 361 information across clusters. The information exchanged between NVAs 362 of the same cluster could be different from the information 363 exchanged across clusters. 365 A Tenant System can be attached to an NVE in several ways: 367 - locally, by being co-located in the same End Device 369 - remotely, via a point-to-point connection or a switched network 371 When an NVE is co-located with a Tenant System, the state of the 372 Tenant System can be determined without protocol assistance. For 373 instance, the operational status of a VM can be communicated via a 374 local API. When an NVE is remotely connected to a Tenant System, the 375 state of the Tenant System or NVE needs to be exchanged directly or 376 via a management entity, using a control plane protocol or API, or 377 directly via a dataplane protocol. 379 The functional components in Figure 2 do not necessarily map 380 directly to the physical components described in Figure 1. For 381 example, an End Device can be a server blade with VMs and a virtual 382 switch. A VM can be a Tenant System and the NVE functions may be 383 performed by the host server. In this case, the Tenant System and 384 NVE function are co-located. Another example is the case where the 385 End Device is the Tenant System, and the NVE function can be 386 implemented by the connected ToR. In this case, the Tenant System 387 and NVE function are not co-located. 389 Underlay nodes utilize L3 technologies to interconnect NVE nodes. 390 These nodes perform forwarding based on outer L3 header information, 391 and generally do not maintain per tenant-service state albeit some 392 applications (e.g., multicast) may require control plane or 393 forwarding plane information that pertain to a tenant, group of 394 tenants, tenant service or a set of services that belong to one or 395 more tenants. Mechanisms to control the amount of state maintained 396 in the underlay may be needed. 398 2.2. NVE Reference Model 400 Figure 3 depicts the NVE reference model. One or more VNIs can be 401 instantiated on an NVE. A Tenant System interfaces with a 402 corresponding VNI via a VAP. An overlay module provides tunneling 403 overlay functions (e.g., encapsulation and decapsulation of tenant 404 traffic, tenant identification and mapping, etc.). 406 +-------- L3 Network -------+ 407 | | 408 | Tunnel Overlay | 409 +------------+---------+ +---------+------------+ 410 | +----------+-------+ | | +---------+--------+ | 411 | | Overlay Module | | | | Overlay Module | | 412 | +---------+--------+ | | +---------+--------+ | 413 | |VN context| | VN context| | 414 | | | | | | 415 | +--------+-------+ | | +--------+-------+ | 416 | | |VNI| . |VNI| | | | |VNI| . |VNI| | 417 NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2 418 | | VAPs | | | | VAPs | | 419 +----+------------+----+ +----+-----------+-----+ 420 | | | | 421 | | | | 422 Tenant Systems Tenant Systems 424 Figure 3 : Generic NVE reference model 426 Note that some NVE functions (e.g., data plane and control plane 427 functions) may reside in one device or may be implemented separately 428 in different devices. 430 2.3. NVE Service Types 432 An NVE provides different types of virtualized network services to 433 multiple tenants, i.e. an L2 service or an L3 service. Note that an 434 NVE may be capable of providing both L2 and L3 services for a 435 tenant. This section defines the service types and associated 436 attributes. 438 2.3.1. L2 NVE providing Ethernet LAN-like service 440 An L2 NVE implements Ethernet LAN emulation, an Ethernet based 441 multipoint service similar to an IETF VPLS [RFC4761][RFC4762] or 442 EVPN [EVPN] service, where the Tenant Systems appear to be 443 interconnected by a LAN environment over an L3 overlay. As such, an 444 L2 NVE provides per-tenant virtual switching instance (L2 VNI), and 445 L3 (IP/MPLS) tunneling encapsulation of tenant MAC frames across the 446 underlay. Note that the control plane for an L2 NVE could be 447 implemented locally on the NVE or in a separate control entity. 449 2.3.2. L3 NVE providing IP/VRF-like service 451 An L3 NVE provides Virtualized IP forwarding service, similar to 452 IETF IP VPN (e.g., BGP/MPLS IPVPN [RFC4364]) from a service 453 definition perspective. That is, an L3 NVE provides per-tenant 454 forwarding and routing instance (L3 VNI), and L3 (IP/MPLS) tunneling 455 encapsulation of tenant IP packets across the underlay. Note that 456 routing could be performed locally on the NVE or in a separate 457 control entity. 459 2.4. Operational Management Considerations 461 NVO3 services are overlay services over an IP underlay. 463 As far as the IP underlay is concerned, existing IP OAM facilities 464 are used. 466 With regards to the NVO3 overlay, both L2 and L3 services can be 467 offered. it is expected that existing fault and performance OAM 468 facilities will be used. Sections 4.1. and 4.2.6. below provide 469 further discussion of additional fault and performance management 470 issues to consider. 472 As far as configuration is concerned, the DC environment is driven 473 by the need to bring new services up rapidly and is typically very 474 dynamic specifically in the context of virtualized services. It is 475 therefore critical to automate the configuration of NVO3 services. 477 3. Functional components 479 This section decomposes the Network Virtualization architecture into 480 functional components described in Figure 3 to make it easier to 481 discuss solution options for these components. 483 3.1. Service Virtualization Components 485 3.1.1. Virtual Access Points (VAPs) 487 Tenant Systems are connected to VNIs through Virtual Access Points 488 (VAPs). 490 VAPs can be physical ports or virtual ports identified through 491 logical interface identifiers (e.g., VLAN ID, internal vSwitch 492 Interface ID connected to a VM). 494 3.1.2. Virtual Network Instance (VNI) 496 A VNI is a specific VN instance on an NVE. Each VNI defines a 497 forwarding context that contains reachability information and 498 policies. 500 3.1.3. Overlay Modules and VN Context 502 Mechanisms for identifying each tenant service are required to allow 503 the simultaneous overlay of multiple tenant services over the same 504 underlay L3 network topology. In the data plane, each NVE, upon 505 sending a tenant packet, must be able to encode the VN Context for 506 the destination NVE in addition to the L3 tunneling information 507 (e.g., source IP address identifying the source NVE and the 508 destination IP address identifying the destination NVE, or MPLS 509 label). This allows the destination NVE to identify the tenant 510 service instance and therefore appropriately process and forward the 511 tenant packet. 513 The Overlay module provides tunneling overlay functions: tunnel 514 initiation/termination as in the case of stateful tunnels (see 515 Section 3.1.4), and/or simply encapsulation/decapsulation of frames 516 from VAPs/L3 underlay. 518 In a multi-tenant context, tunneling aggregates frames from/to 519 different VNIs. Tenant identification and traffic demultiplexing are 520 based on the VN Context identifier. 522 The following approaches can be considered: 524 - VN Context identifier per Tenant: Globally unique (on a per-DC 525 administrative domain) VN identifier used to identify the 526 corresponding VNI. Examples of such identifiers in existing 527 technologies are IEEE VLAN IDs and ISID tags that identify virtual 528 L2 domains when using IEEE 802.1aq and IEEE 802.1ah, respectively. 529 Note that multiple VN identifiers can belong to a tenant. 531 - One VN Context identifier per VNI: Each VNI value is automatically 532 generated by the egress NVE, or a control plane associated with 533 that NVE, and usually distributed by a control plane protocol to 534 all the related NVEs. An example of this approach is the use of 535 per VRF MPLS labels in IP VPN [RFC4364]. The VNI value is 536 therefore locally significant to the egress NVE. 538 - One VN Context identifier per VAP: A value locally significant to 539 an NVE is assigned and usually distributed by a control plane 540 protocol to identify a VAP. An example of this approach is the use 541 of per CE-PE MPLS labels in IP VPN [RFC4364]. 543 Note that when using one VN Context per VNI or per VAP, an 544 additional global identifier (e.g., a VN identifier or name) may be 545 used by the control plane to identify the Tenant context. 547 3.1.4. Tunnel Overlays and Encapsulation options 549 Once the VN context identifier is added to the frame, an L3 Tunnel 550 encapsulation is used to transport the frame to the destination NVE. 552 Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS 553 tunneling can be used. Tunneling could be stateless or stateful. 554 Stateless tunneling simply entails the encapsulation of a tenant 555 packet with another header necessary for forwarding the packet 556 across the underlay (e.g., IP tunneling over an IP underlay). 557 Stateful tunneling on the other hand entails maintaining tunneling 558 state at the tunnel endpoints (i.e., NVEs). Tenant packets on an 559 ingress NVE can then be transmitted over such tunnels to a 560 destination (egress) NVE by encapsulating the packets with a 561 corresponding tunneling header. The tunneling state at the endpoints 562 may be configured or dynamically established. Solutions should 563 specify the tunneling technology used, whether it is stateful or 564 stateless. In this document, however, tunneling and tunneling 565 encapsulation are used interchangeably to simply mean the 566 encapsulation of a tenant packet with a tunneling header necessary 567 to carry the packet between an ingress NVE and an egress NVE across 568 the underlay. It should be noted that stateful tunneling, especially 569 when configuration is involved, does impose management overhead and 570 scale constraints. When confidentiality is required, the use of 571 opportunistic security [OPPSEC] can be used as a stateless tunneling 572 solution. 574 3.1.5. Control Plane Components 576 3.1.5.1. Distributed vs Centralized Control Plane 578 A control/management plane entity can be centralized or distributed. 579 Both approaches have been used extensively in the past. The routing 580 model of the Internet is a good example of a distributed approach. 581 Transport networks have usually used a centralized approach to 582 manage transport paths. 584 It is also possible to combine the two approaches, i.e., using a 585 hybrid model. A global view of network state can have many benefits 586 but it does not preclude the use of distributed protocols within the 587 network. Centralized models provide a facility to maintain global 588 state, and distribute that state to the network. When used in 589 combination with distributed protocols, greater network 590 efficiencies, improved reliability and robustness can be achieved. 591 Domain and/or deployment specific constraints define the balance 592 between centralized and distributed approaches. 594 3.1.5.2. Auto-provisioning/Service discovery 596 NVEs must be able to identify the appropriate VNI for each Tenant 597 System. This is based on state information that is often provided by 598 external entities. For example, in an environment where a VM is a 599 Tenant System, this information is provided by VM orchestration 600 systems, since these are the only entities that have visibility of 601 which VM belongs to which tenant. 603 A mechanism for communicating this information to the NVE is 604 required. VAPs have to be created and mapped to the appropriate VNI. 605 Depending upon the implementation, this control interface can be 606 implemented using an auto-discovery protocol between Tenant Systems 607 and their local NVE or through management entities. In either case, 608 appropriate security and authentication mechanisms to verify that 609 Tenant System information is not spoofed or altered are required. 610 This is one critical aspect for providing integrity and tenant 611 isolation in the system. 613 NVEs may learn reachability information to VNIs on other NVEs via a 614 control protocol that exchanges such information among NVEs, or via 615 a management control entity. 617 3.1.5.3. Address advertisement and tunnel mapping 619 As traffic reaches an ingress NVE on a VAP, a lookup is performed to 620 determine which NVE or local VAP the packet needs to be sent to. If 621 the packet is to be sent to another NVE, the packet is encapsulated 622 with a tunnel header containing the destination information 623 (destination IP address or MPLS label) of the egress NVE. 624 Intermediate nodes (between the ingress and egress NVEs) switch or 625 route traffic based upon the tunnel destination information. 627 A key step in the above process consists of identifying the 628 destination NVE the packet is to be tunneled to. NVEs are 629 responsible for maintaining a set of forwarding or mapping tables 630 that hold the bindings between destination VM and egress NVE 631 addresses. Several ways of populating these tables are possible: 632 control plane driven, management plane driven, or data plane driven. 634 When a control plane protocol is used to distribute address 635 reachability and tunneling information, the auto- 636 provisioning/Service discovery could be accomplished by the same 637 protocol. In this scenario, the auto-provisioning/Service discovery 638 could be combined with (be inferred from) the address advertisement 639 and associated tunnel mapping. Furthermore, a control plane protocol 640 that carries both MAC and IP addresses eliminates the need for ARP, 641 and hence addresses one of the issues with explosive ARP handling as 642 discussed in [RFC6820]. 644 3.1.5.4. Overlay Tunneling 646 For overlay tunneling, and dependent upon the tunneling technology 647 used for encapsulating the Tenant System packets, it may be 648 sufficient to have one or more local NVE addresses assigned and used 649 in the source and destination fields of a tunneling encapsulation 650 header. Other information that is part of the 651 tunneling encapsulation header may also need to be configured. In 652 certain cases, local NVE configuration may be sufficient while in 653 other cases, some tunneling related information may need to 654 be shared among NVEs. The information that needs to be shared will 655 be technology dependent. For instance, potential information could 656 include tunnel identity, encapsulation type, and/or tunnel 657 resources. In certain cases, such as when using IP multicast in the 658 underlay, tunnels which interconnect NVEs may need to be 659 established. When tunneling information needs to be exchanged or 660 shared among NVEs, a control plane protocol may be required. For 661 instance, it may be necessary to provide active/standby status 662 information between NVEs, up/down status information, 663 pruning/grafting information for multicast tunnels, etc. 665 In addition, a control plane may be required to setup the tunnel 666 path for some tunneling technologies. This applies to both unicast 667 and multicast tunneling. 669 3.2. Multi-homing 671 Multi-homing techniques can be used to increase the reliability of 672 an NVO3 network. It is also important to ensure that physical 673 diversity in an NVO3 network is taken into account to avoid single 674 points of failure. 676 Multi-homing can be enabled in various nodes, from Tenant Systems 677 into ToRs, ToRs into core switches/routers, and core nodes into DC 678 GWs. 680 The NVO3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP 681 routing as the means to re-route traffic upon failures techniques or 682 on MPLS re-rerouting capabilities. 684 When a Tenant System is co-located with the NVE, the Tenant System 685 is effectively single homed to the NVE via a virtual port. When the 686 Tenant System and the NVE are separated, the Tenant System is 687 connected to the NVE via a logical Layer2 (L2) construct such as a 688 VLAN and it can be multi-homed to various NVEs. An NVE may provide 689 an L2 service to the end system or an l3 service. An NVE may be 690 multi-homed to a next layer in the DC at Layer2 (L2) or Layer3 691 (L3). When an NVE provides an L2 service and is not co-located with 692 the end system, loop avoidance techniques must be used. Similarly, 693 when the NVE provides L3 service, similar dual-homing techniques can 694 be used. When the NVE provides a L3 service to the end system, it is 695 possible that no dynamic routing protocol is enabled between the end 696 system and the NVE. The end system can be multi-homed to 697 multiple physically-separated L3 NVEs over multiple interfaces. When 698 one of the links connected to an NVE fails, the other interfaces can 699 be used to reach the end system. 701 External connectivity from a DC can be handled by two or more DC 702 gateways. Each gateway provides access to external networks such as 703 VPNs or the Internet. A gateway may be connected to two or more edge 704 nodes in the external network for redundancy. When a connection to 705 an upstream node is lost, the alternative connection is used and the 706 failed route withdrawn. 708 3.3. VM Mobility 710 In DC environments utilizing VM technologies, an important feature 711 is that VMs can move from one server to another server in the same 712 or different L2 physical domains (within or across DCs) in a 713 seamless manner. 715 A VM can be moved from one server to another in stopped or suspended 716 state ("cold" VM mobility) or in running/active state ("hot" VM 717 mobility). With "hot" mobility, VM L2 and L3 addresses need to be 718 preserved. With "cold" mobility, it may be desired to preserve at 719 least VM L3 addresses. 721 Solutions to maintain connectivity while a VM is moved are necessary 722 in the case of "hot" mobility. This implies that connectivity among 723 VMs is preserved. For instance, for L2 VNs, ARP caches are updated 724 accordingly. 726 Upon VM mobility, NVE policies that define connectivity among VMs 727 must be maintained. 729 During VM mobility, it is expected that the path to the VM's default 730 gateway assures adequate QoS to VM applications, i.e. QoS that 731 matches the expected service level agreement for these applications. 733 4. Key aspects of overlay networks 735 The intent of this section is to highlight specific issues that 736 proposed overlay solutions need to address. 738 4.1. Pros & Cons 740 An overlay network is a layer of virtual network topology on top of 741 the physical network. 743 Overlay networks offer the following key advantages: 745 - Unicast tunneling state management and association of Tenant 746 Systems reachability are handled at the edge of the network (at 747 the NVE). Intermediate transport nodes are unaware of such 748 state. Note that when multicast is enabled in the underlay 749 network to build multicast trees for tenant VNs, there would be 750 more state related to tenants in the underlay core network. 752 - Tunneling is used to aggregate traffic and hide tenant 753 addresses from the underlay network, and hence offer the 754 advantage of minimizing the amount of forwarding state required 755 within the underlay network 757 - Decoupling of the overlay addresses (MAC and IP) used by VMs 758 from the underlay network for tenant separation and separation 759 of the tenant address spaces from the underlay address space. 761 - Support of a large number of virtual network identifiers 763 Overlay networks also create several challenges: 765 - Overlay networks have typically no control of underlay networks 766 and lack underlay network information (e.g. underlay 767 utilization): 769 - Overlay networks and/or their associated management entities 770 typically probe the network to measure link or path 771 properties, such as available bandwidth or packet loss rate. 772 It is difficult to accurately evaluate network properties. It 773 might be preferable for the underlay network to expose usage 774 and performance information. 775 - Miscommunication or lack of coordination between overlay and 776 underlay networks can lead to an inefficient usage of network 777 resources. 778 - When multiple overlays co-exist on top of a common underlay 779 network, the lack of coordination between overlays can lead 780 to performance issues and/or resource usage inefficiencies. 782 - Traffic carried over an overlay might fail to traverse 783 firewalls and NAT devices. 785 - Multicast service scalability: Multicast support may be 786 required in the underlay network to address tenant flood 787 containment or efficient multicast handling. The underlay may 788 also be required to maintain multicast state on a per-tenant 789 basis, or even on a per-individual multicast flow of a given 790 tenant. Ingress replication at the NVE eliminates that 791 additional multicast state in the underlay core, but depending 792 on the multicast traffic volume, it may cause inefficient use 793 of bandwidth. 795 4.2. Overlay issues to consider 797 4.2.1. Data plane vs Control plane driven 799 In the case of an L2 NVE, it is possible to dynamically learn MAC 800 addresses against VAPs. It is also possible that such addresses be 801 known and controlled via management or a control protocol for both 802 L2 NVEs and L3 NVEs. Dynamic data plane learning implies that 803 flooding of unknown destinations be supported and hence implies that 804 broadcast and/or multicast be supported or that ingress replication 805 be used as described in section 4.2.3. Multicasting in the underlay 806 network for dynamic learning may lead to significant scalability 807 limitations. Specific forwarding rules must be enforced to prevent 808 loops from happening. This can be achieved using a spanning tree, a 809 shortest path tree, or a split-horizon mesh. 811 It should be noted that the amount of state to be distributed is 812 dependent upon network topology and the number of virtual machines. 813 Different forms of caching can also be utilized to minimize state 814 distribution between the various elements. The control plane should 815 not require an NVE to maintain the locations of all the Tenant 816 Systems whose VNs are not present on the NVE. The use of a control 817 plane does not imply that the data plane on NVEs has to maintain all 818 the forwarding state in the control plane. 820 4.2.2. Coordination between data plane and control plane 822 For an L2 NVE, the NVE needs to be able to determine MAC addresses 823 of the Tenant Systems connected via a VAP. This can be achieved via 824 dataplane learning or a control plane. For an L3 NVE, the NVE needs 825 to be able to determine IP addresses of the Tenant Systems connected 826 via a VAP. 828 In both cases, coordination with the NVE control protocol is needed 829 such that when the NVE determines that the set of addresses behind a 830 VAP has changed, it triggers the NVE control plane to distribute 831 this information to its peers. 833 4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic 835 There are several options to support packet replication needed for 836 broadcast, unknown unicast and multicast. Typical methods include: 838 - Ingress replication 839 - Use of underlay multicast trees 841 There is a bandwidth vs state trade-off between the two approaches. 842 Depending upon the degree of replication required (i.e. the number 843 of hosts per group) and the amount of multicast state to maintain, 844 trading bandwidth for state should be considered. 846 When the number of hosts per group is large, the use of underlay 847 multicast trees may be more appropriate. When the number of hosts is 848 small (e.g. 2-3) and/or the amount of multicast traffic is small, 849 ingress replication may not be an issue. 851 Depending upon the size of the data center network and hence the 852 number of (S,G) entries, and also the duration of multicast flows, 853 the use of underlay multicast trees can be a challenge. 855 When flows are well known, it is possible to pre-provision such 856 multicast trees. However, it is often difficult to predict 857 application flows ahead of time, and hence programming of (S,G) 858 entries for short-lived flows could be impractical. 860 A possible trade-off is to use in the underlay shared multicast 861 trees as opposed to dedicated multicast trees. 863 4.2.4. Path MTU 865 When using overlay tunneling, an outer header is added to the 866 original frame. This can cause the MTU of the path to the egress 867 tunnel endpoint to be exceeded. 869 It is usually not desirable to rely on IP fragmentation for 870 performance reasons. Ideally, the interface MTU as seen by a Tenant 871 System is adjusted such that no fragmentation is needed. 873 It is possible for the MTU to be configured manually or to be 874 discovered dynamically. Various Path MTU discovery techniques exist 875 in order to determine the proper MTU size to use: 877 - Classical ICMP-based MTU Path Discovery [RFC1191] [RFC1981] 879 - Tenant Systems rely on ICMP messages to discover the MTU of the 880 end-to-end path to its destination. This method is not always 881 possible, such as when traversing middle boxes (e.g. firewalls) 882 which disable ICMP for security reasons 884 - Extended MTU Path Discovery techniques such as defined in 885 [RFC4821] 887 - Tenant Systems send probe packets of different sizes, and rely 888 on confirmation of receipt or lack thereof from receivers to 889 allow a sender to discover the MTU of the end-to-end paths. 891 While it could also be possible to rely on the NVE to perform 892 segmentation and reassembly operations without relying on the Tenant 893 Systems to know about the end-to-end MTU, this would lead to 894 undesired performance and congestion issues as well as significantly 895 increase the complexity of hardware NVEs required for buffering and 896 reassembly logic. 898 Preferably, the underlay network should be designed in such a way 899 that the MTU can accommodate the extra tunneling and possibly 900 additional NVO3 header encapsulation overhead. 902 4.2.5. NVE location trade-offs 904 In the case of DC traffic, traffic originated from a VM is native 905 Ethernet traffic. This traffic can be switched by a local virtual 906 switch or ToR switch and then by a DC gateway. The NVE function can 907 be embedded within any of these elements. 909 There are several criteria to consider when deciding where the NVE 910 function should happen: 912 - Processing and memory requirements 914 - Datapath (e.g. lookups, filtering, encapsulation/decapsulation) 916 - Control plane processing (e.g. routing, signaling, OAM) and 917 where specific control plane functions should be enabled 919 - FIB/RIB size 921 - Multicast support 923 - Routing/signaling protocols 925 - Packet replication capability 927 - Multicast FIB 929 - Fragmentation support 930 - QoS support (e.g. marking, policing, queuing) 932 - Resiliency 934 4.2.6. Interaction between network overlays and underlays 936 When multiple overlays co-exist on top of a common underlay network, 937 resources (e.g., bandwidth) should be provisioned to ensure that 938 traffic from overlays can be accommodated and QoS objectives can be 939 met. Overlays can have partially overlapping paths (nodes and 940 links). 942 Each overlay is selfish by nature. It sends traffic so as to 943 optimize its own performance without considering the impact on other 944 overlays, unless the underlay paths are traffic engineered on a per 945 overlay basis to avoid congestion of underlay resources. 947 Better visibility between overlays and underlays, or generally 948 coordination in placing overlay demand on an underlay network, may 949 be achieved by providing mechanisms to exchange performance and 950 liveliness information between the underlay and overlay(s) or the 951 use of such information by a coordination system. Such information 952 may include: 954 - Performance metrics (throughput, delay, loss, jitter) such as 955 defined in [RFC3148], [RFC2679], [RFC2680], and [RFC3393]. 957 - Cost metrics 959 5. Security Considerations 961 There are three points-of-view when considering security for NVO3. 962 First, the service offered by a service provider via NVO3 technology 963 to a tenant must meet the mutually agreed security requirements. 964 Second, a network implementing NVO3 must be able to trust the 965 virtual network identity associated with packets received from a 966 tenant. Third, an NVO3 network must consider the security associated 967 with running as an overlay across the underlaying network. 969 To meet a tenant's security requirements, the NVO3 service must 970 deliver packets from the tenant to the indicated destination(s) in 971 the overlay network and external networks. The NVO3 service 972 provides data confidentiality through data separation. The use of 973 both VNIs and tunneling of tenant traffic by NVEs ensures that NVO3 974 data is kept in a separate context and thus separated from other 975 tenant traffic. The infrastructure supporting an NVO3 service (e.g. 977 management systems, NVEs, NVAs, and intermediate underlay networks) 978 should be limited to authorized access so that data integrity can be 979 expected. If a tenant requires that its data be confidential, then 980 the tenant system may choose to encrypt its data before transmission 981 into the NVO3 service. 983 An NVO3 service must be able to verify the VNI received on a packet 984 from the tenant. To ensure this, not only tenant data but also NVO3 985 control data must be secured (e.g. control traffic between NVAs and 986 NVEs, between NVAs and between NVEs). Since NVEs and NVAs play a 987 central role in NVO3, it is critical that a secure access to NVEs 988 and NVAs be ensured such that no unauthorized access is possible. As 989 discussed in section 3.1.5.2. , Tenant Systems identification is 990 based upon state that is often provided by management systems (e.g. 991 a VM orchestration system in a virtualized environment). Secure 992 access to such management systems must also be ensured. When an NVE 993 receives data from a Tenant System, the tenant identity needs to be 994 verified in order to guarantee that it is authorized to access the 995 corresponding VN. This can be achieved by identifying incoming 996 packets against specific VAPs in some cases. In other circumstances, 997 authentication may be necessary. Once this verification is done, 998 the packet is allowed into the NVO3 overlay and no integrity 999 protection is provided on the overlay packet encapsulation (e.g. the 1000 VNI, destination VNE, etc.). 1002 Since an NVO3 service can run across diverse underlay networks, when 1003 the underlay network is not trusted to provide at least data 1004 integrity, data encryption is needed to assure correct packet 1005 delivery. 1007 It is also desirable to restrict the types of information (e.g. 1008 topology information, such as discussed in Section 4.2.6) that can 1009 be exchanged between an NVO3 service and underlaying networks based 1010 upon their agreed security requirements. 1012 6. IANA Considerations 1014 IANA does not need to take any action for this draft. 1016 7. References 1018 7.1. Informative References 1020 [EVPN] Sajassi, A. et al, "BGP MPLS Based Ethernet VPN", draft- 1021 ietf-l2vpn-evpn (work in progress) 1023 [NVOPS] Narten, T. et al, "Problem Statement : Overlays for 1024 Network Virtualization", draft-ietf-nvo3-overlay-problem- 1025 statement (work in progress) 1027 [OPPSEC] Dukhovni, V. "Opportunistic Security: some protection most 1028 of the time", draft-dukhovni-opportunistic-security (work 1029 in progress) 1031 [RFC1191] Mogul, J. "Path MTU Discovery", RFC1191, November 1990 1033 [RFC1981] McCann, J. et al, "Path MTU Discovery for IPv6", RFC1981, 1034 August 1996 1036 [RFC2679] Almes, G. et al, "A One-way Delay Metric for IPPM", 1037 RFC2679, September 1999 1039 [RFC2680] Almes, G. et al, "A One-way Packet Loss Metric for IPPM", 1040 RFC2680, September 1999 1042 [RFC3148] Mathis, M. et al, "A Framework for Defining Empirical Bulk 1043 Transfer Capacity Metrics", RFC3148, July 2001 1045 [RFC3393] Demichelis, C. and Chimeto, P., "IP Packet Delay Variation 1046 Metric for IP Performance Metrics (IPPM)", RFC3393, 1047 November 2002 1049 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1050 Networks (VPNs)", RFC 4364, February 2006. 1052 [RFC4761] Kompella, K. et al, "Virtual Private LAN Service (VPLS) 1053 Using BGP for auto-discovery and Signaling", RFC4761, 1054 January 2007 1056 [RFC4762] Lasserre, M. et al, "Virtual Private LAN Service (VPLS) 1057 Using Label Distribution Protocol (LDP) Signaling", 1058 RFC4762, January 2007 1060 [RFC4821] Mathis, M. et al, "Packetization Layer Path MTU 1061 Discovery", RFC4821, March 2007 1063 [RFC6820] Narten, T. et al, "Address Resolution Problems in Large 1064 Data Center Networks", RFC6820, January 2013 1066 8. Acknowledgments 1068 In addition to the authors the following people have contributed to 1069 this document: 1071 Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas Narten, 1072 Larry Kreeger, David Black. 1074 This document was prepared using 2-Word-v2.0.template.dot. 1076 Authors' Addresses 1078 Marc Lasserre 1079 Alcatel-Lucent 1080 Email: marc.lasserre@alcatel-lucent.com 1082 Florin Balus 1083 Alcatel-Lucent 1084 777 E. Middlefield Road 1085 Mountain View, CA, USA 94043 1086 Email: florin.balus@alcatel-lucent.com 1088 Thomas Morin 1089 France Telecom Orange 1090 Email: thomas.morin@orange.com 1092 Nabil Bitar 1093 Verizon 1094 40 Sylvan Road 1095 Waltham, MA 02145 1096 Email: nabil.bitar@verizon.com 1098 Yakov Rekhter 1099 Juniper 1100 Email: yakov@juniper.net