idnits 2.17.1 draft-bitar-nvo3-vpn-applicability-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == The page length should not exceed 58 lines per page, but there was 52 longer pages, the longest (page 1) being 66 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 2 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 21, 2013) is 3839 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'NVo3-problem-statement' is mentioned on line 2082, but not defined == Missing Reference: 'NVo3-fmwk' is mentioned on line 1414, but not defined == Missing Reference: 'NVo3-dp-reqts' is mentioned on line 182, but not defined == Missing Reference: 'NVo3-cp-reqts' is mentioned on line 182, but not defined == Missing Reference: 'NVo3-frmwk' is mentioned on line 192, but not defined == Missing Reference: 'RFC7461' is mentioned on line 395, but not defined -- Looks like a reference, but probably isn't: '4762' on line 395 == Missing Reference: 'PE' is mentioned on line 395, but not defined == Missing Reference: 'NVo3-problem-statemnt' is mentioned on line 541, but not defined == Missing Reference: 'NV03-roblem-statement' is mentioned on line 755, but not defined == Missing Reference: 'E-VPN' is mentioned on line 2036, but not defined == Missing Reference: 'NV03-proble-statement' is mentioned on line 1077, but not defined == Missing Reference: 'RFC 4719' is mentioned on line 1255, but not defined == Missing Reference: 'RFC 4797' is mentioned on line 1255, but not defined == Missing Reference: 'NV03-fmwk' is mentioned on line 1799, but not defined == Missing Reference: 'RFC4684' is mentioned on line 1930, but not defined == Missing Reference: 'VPN-RSVP-TE' is mentioned on line 2086, but not defined == Unused Reference: 'RFC4719' is defined on line 2237, but no explicit reference was found in the text == Unused Reference: 'ARPproxy' is defined on line 2244, but no explicit reference was found in the text == Unused Reference: 'Nvo3-fmwk' is defined on line 2262, but no explicit reference was found in the text == Unused Reference: 'Nvo3-cp-reqts' is defined on line 2266, but no explicit reference was found in the text == Unused Reference: 'Nvo3-dp-reqts' is defined on line 2271, but no explicit reference was found in the text == Outdated reference: A later version (-06) exists of draft-ietf-l2vpn-pbb-vpls-interop-05 == Outdated reference: A later version (-11) exists of draft-ietf-l2vpn-evpn-04 == Outdated reference: A later version (-10) exists of draft-ietf-l2vpn-pbb-evpn-05 == Outdated reference: A later version (-07) exists of draft-raggarwa-data-center-mobility-05 == Outdated reference: A later version (-09) exists of draft-ietf-nvo3-framework-03 == Outdated reference: A later version (-03) exists of draft-ietf-nvo3-dataplane-requirements-01 Summary: 0 errors (**), 0 flaws (~~), 30 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 L2VPN Working Group 3 Internet Draft 4 Intended status: Informational 5 Expires: April 2014 6 Nabil Bitar 7 Verizon 9 Florin Balus 10 Marc Lasserre 11 Wim Henderickx 12 John Drake Alcatel-Lucent 13 Juniper Networks 14 Ali Sajassi 15 Luyuan Fang 16 Cisco 17 Lucy Yong 18 Huawei 19 Yuichi Ikejiri 20 Susan Hare NTT Communications 21 ADARA 22 Mircea Pisica 23 BT 25 October 21, 2013 27 Cloud Networking: VPN Applicability and NVo3 Gap Analysis 28 draft-bitar-nvo3-vpn-applicability-02.txt 30 Abstract 32 Multi-tenant data centers and clouds provide computing, 33 storage and network resources dedicated per tenant. The 34 current focus in the evolution of multi-tenant data-center and 35 cloud networks is to (1) support a large number of tenants 36 with a large number of communicating systems, (2) provide 37 isolation among tenant virtual networks, (3) provide for 38 efficient network utilization, and (4) support virtual machine 39 mobility and network elasticity that match compute and storage 40 elasticity. 42 The NVo3 work effort is initially targeted to identify the 43 requirements for large multi-tenant data centers, and develop 44 a framework architecture that addresses those requirements. In 45 addition, it is targeted to identify existing or evolving 46 solutions used in cloud networking, their applicability to 47 NVo3, and any gaps that they may have in addressing the NVo3 48 requirements. This document describes the applicability of 49 existing work in various IETF Working Groups (e.g., RFCs and 50 drafts developed or evolving in IETF L2VPN and L3VPN Working 51 Groups) to cloud networking and NVo3, as well as the gaps and 52 problems that need to be further addressed. 54 Status of this Memo 56 This Internet-Draft is submitted in full conformance with the 57 provisions of BCP 78 and BCP 79. 59 Internet-Drafts are working documents of the Internet 60 Engineering Task Force (IETF), its areas, and its working 61 groups. Note that other groups may also distribute working 62 documents as Internet-Drafts. 64 Internet-Drafts are draft documents valid for a maximum of six 65 months and may be updated, replaced, or obsoleted by other 66 documents at any time. It is inappropriate to use Internet- 67 Drafts as reference material or to cite them other than as 68 "work in progress." 70 The list of current Internet-Drafts can be accessed at 71 http://www.ietf.org/ietf/1id-abstracts.txt 73 The list of Internet-Draft Shadow Directories can be accessed 74 at http://www.ietf.org/shadow.html 76 This Internet-Draft will expire on April 21, 2014. 78 Copyright Notice 80 Copyright (c) 2013 IETF Trust and the persons identified as 81 the document authors. All rights reserved. 83 This document is subject to BCP 78 and the IETF Trust's Legal 84 Provisions Relating to IETF Documents 85 (http://trustee.ietf.org/license-info) in effect on the date 86 of publication of this document. Please review these documents 87 carefully, as they describe your rights and restrictions with 88 respect to this document. 90 Table of Contents 92 1. Introduction.............................................. 4 93 2. General terminology....................................... 6 94 2.1. Conventions used in this document.................... 7 95 3. Brief overview of Ethernet, L2VPN and L3VPN deployments... 7 96 4. Generic Cloud Networking Architecture..................... 9 97 5. Challenges in Existing Deployments....................... 13 98 5.1. VLAN Space Limitation............................... 14 99 5.2. MAC, IP, and ARP Issues............................. 14 100 5.3. Per VLAN flood containment.......................... 17 101 5.4. Convergence and multipath support................... 17 102 5.5. Optimal traffic forwarding.......................... 18 103 5.6. Efficient multicast................................. 20 104 5.7. L3 virtualization................................... 21 105 5.8. Connectivity to existing tenant VPN sites........... 21 106 5.9. DC Inter-connect requirements....................... 22 107 5.10. VM Mobility........................................ 22 108 6. L2VPN Applicability to Cloud Networking.................. 24 109 6.1. VLANs and L2VPN toolset............................. 24 110 6.2. E-VPN............................................... 27 111 6.3. PBB and L2VPN toolset............................... 30 112 6.3.1. Addressing VLAN space exhaustion and MAC 113 explosion............................................. 32 114 6.3.2. Fast convergence and L2 multi-pathing.......... 32 115 6.3.3. Per ISID flood containment..................... 34 116 6.3.4. Efficient multicast support.................... 34 117 6.3.5. Tunneling options for PBB ELAN: Ethernet, IP and 118 MPLS.................................................. 34 119 6.3.6. Use Case examples.............................. 35 120 6.3.7. NVo3 applicability............................. 38 121 6.3.8. Connectivity to existing VPN sites and Internet 40 122 6.3.9. DC Interconnect................................ 43 123 6.3.10. Interoperating with existing DC VLANs......... 44 124 6.4. TRILL and L2VPN toolset............................. 46 125 7. L3VPN applicability to Cloud Networking.................. 47 126 8. VM Mobility with E-VPN................................... 50 127 8.1. Layer 2 Extension Solution.......................... 50 128 8.2. VM Default Gateway Solutions........................ 53 129 8.2.1. VM Default Gateway Solution 1.................. 53 130 8.2.2. VM Default Gateway Solution 2.................. 54 131 9. Solutions and Considerations for other DC challenges..... 55 132 9.1. Addressing IP/ARP explosion......................... 55 133 9.2. Optimal traffic forwarding.......................... 55 134 9.3. VM Mobility......................................... 55 135 9.4. Dynamic provisioning of network services............ 56 136 9.5. Considerations for Layer2 and Layer3 VPNS on End- 137 systems.................................................. 57 138 10. Operator Considerations................................. 57 139 11. Security Considerations................................. 58 140 12. IANA Considerations..................................... 58 141 13. References.............................................. 58 142 13.1. Normative References............................... 58 143 13.2. Informative References............................. 59 144 14. Acknowledgments......................................... 61 146 1. Introduction 148 The initial Data Center (DC) networks were built to address the 149 needs of individual enterprises and/or individual applications. 150 Ethernet VLANs and regular IP routing were used to provide 151 connectivity between compute, storage resources and the related 152 customer sites. 154 The virtualization of compute resources in a Data Center (DC) 155 environment provides the foundation for providing compute and 156 storage resources to multiple tenants (customers), and/or for 157 providing application services to multiple tenants. For example, 158 a tenant may be provided a group of Virtual Machines (VMs) that 159 may reside on server blades distributed throughout a DC or across 160 DCs. In this latter case, the DCs may be owned and operated by a 161 cloud service provider connected to one or more network service 162 providers, two or more cloud service providers each connected to 163 one or more network service providers, or a hybrid of DCs 164 operated by the customer and the cloud service provider(s). In 165 addition, multiple tenants may be assigned resources on the same 166 compute and storage hardware. 168 In order to provide access for multiple tenants to the 169 virtualized compute and storage resources, the DC network and DC 170 interconnect have to evolve from the basic VLAN and IP routing 171 architecture to provide equivalent connectivity virtualization at 172 a large scale. 174 [NVo3-problem-statement] describes the problems faced in large 175 multi-tenant data centers, and motivates the need for overlays to 176 address these problems. The main problems highlighted are: (1) 177 support for a large number of tenants, (2) network infrastructure 178 scale, (3) isolation among tenant virtual networks with 179 overlapping address spaces across tenants, and (4) support for 180 virtual machine mobility, network elasticity, and accompanying 181 dynamic network provisioning. [NVo3-fmwk] describes a framework 182 architecture for NVo3, while [NVo3-dp-reqts] and [NVo3-cp-reqts] 183 describe NVo3 data plane and control plane requirements, 184 respectively. Prior to the NVo3 effort initiation, a number of 185 technologies had been used to address network virtualization. 186 Some had also been deployed in data centers and cloud networks, 187 and/or had been further evolved to address requirements of large 188 multi-tenant data centers and cloud networks. The natural 189 question is how these technologies address multi-tenant cloud 190 networking problems as described in [NVo3-problem-statement], 191 what challenges or gaps they need to still further address, and 192 how they compare to the NVo3 architecture framework [NVo3-frmwk]. 193 This document addresses that question. Further evolution of this 194 document may target a more detailed comparison of these 195 technologies to the evolving NVo3 data plane and control plane 196 requirements. 198 Virtual LAN bridging and Virtual Private Network (VPN) 199 technologies had been developed and deployed to support virtual 200 networks with overlapping address spaces over a common 201 infrastructure. Some of these technologies also use various 202 overlay technologies to enable the sharing and scale of an 203 undelay network infrastructure. Those technologies have been used 204 in data-center and cloud networks. However, these technologies 205 originally developed for relatively static environments in terms 206 of communicating endpoints, do not address all the requirements 207 arising in cloud-computing environments, and specifically multi- 208 tenant environments. 210 This document starts with a brief overview of Ethernet, Layer2 211 and Layer3 VPN deployments. It then describes generic data center 212 architecture. This architecture is used in subsequent sections as 213 basis for describing how different VPN technologies apply in DCs 214 and cloud networks, and what problems described in [Nvo3-problem- 215 statement] they address. In addition, it provides a comparison 216 among these technologies and the NVo3 architecture framework at 217 the functional level. 219 2. General terminology 221 Some general terminology is defined here; most of the 222 terminology used is from [802.1ah], [RFC4026] and [NVo3-fmwk]. 223 Terminology specific to this memo is introduced as needed in 224 later sections. 226 DC: Data Center 228 ELAN: MEF ELAN, multipoint-to-multipoint Ethernet service 230 EVPN: Ethernet VPN as defined in [EVPN] 232 PBB: Provider Backbone Bridging, new Ethernet encapsulation 233 designed to address VLAN exhaustion and MAC explosion issues; 234 specified in IEEE 802.1ah [802.1ah] 236 PBB-EVPN: defines how EVPN can be used to transport PBB frames 238 BMAC: Backbone MACs, the backbone source or destination MAC 239 address fields defined in the 802.1ah provider MAC 240 encapsulation header. 242 CMAC: Customer MACs, the customer source or destination MAC 243 address fields defined in the 802.1ah customer MAC 244 encapsulation header. 246 BEB: A backbone edge bridge positioned at the edge of a 247 provider backbone bridged network. It is usually the point in 248 the network where PBB encapsulation is added or removed from 249 the frame. 251 BCB: A backbone core bridge positioned in the core of a 252 provider backbone bridged network. It performs regular 253 Ethernet switching using the outer Ethernet header. 255 2.1. Conventions used in this document 257 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL 258 NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and 259 "OPTIONAL" in this document are to be interpreted as described 260 in RFC-2119 [RFC2119]. 262 In this document, these words will appear with that 263 interpretation only when in ALL CAPS. Lower case uses of 264 these words are not to be interpreted as carrying RFC-2119 265 significance. 267 3. Brief overview of Ethernet, L2VPN and L3VPN deployments 269 Initial Ethernet networks have been deployed in LAN 270 environments, where the total number of hosts (hence MAC 271 addresses) to manage was limited. Physical Ethernet topologies 272 in LANs were pretty simple. Hence, a simple loop resolution 273 protocol such as the Spanning Tree Protocol (STP) was 274 sufficient in the early days. Efficient utilisation of 275 physical links was not a major concern in LANs, while at the 276 same time leveraging existing and mature technologies. 278 As more hosts got connected to a LAN, or the need arose to 279 create multiple LANs on the same physical infrastructure, it 280 became necessary to partition the physical topology into 281 multiple Virtual LANs (VLANs) [802.1q]. A VLAN is identified 282 by a VLAN ID in the 802.1q VLAN tag inserted in the Ethernet 283 header. STP evolved to cope with multiple VLANs with Multiple- 284 STP (MSTP). Bridges/Switches evolved to learn behind which 285 VLAN specific MACs resided, a process known as qualified 286 learning, requiring MACs to be unique only in the VLAN 287 context. As Ethernet LANs moved into the provider space, the 288 12-bit VLAN space limitation (i.e. a total of 4094 VLANs, 289 VLANs 0 and 4095 reserved) led to VLAN stacking (Q-in-Q) and 290 later to Provider backbone Bridging (PBB). 292 With PBB, not only can over 16M virtual LAN instances (24-bit 293 Service I-SID) be supported, but also a clean separation 294 between customer and provider domains has been defined with 295 separate MAC address spaces (Customer-MACs (CMACs) versus 296 Provider Backbone-MACs (BMACs)). CMACs are only learned at the 297 edge of the PBB network on PBB Backbone Edge Bridges (BEBs) in 298 the context of an I-component while only B-MACs are learnt by 299 PBB Backbone Core Bridges (BCBs). This results in BEB switches 300 creating MAC-in-MAC tunnels to carry customer traffic, thereby 301 hiding C-MACs in the core. 303 In the meantime, interconnecting L2 domains across 304 geographical areas has become a necessity. VPN technologies 305 have been defined to carry both L2 and L3 traffic across 306 IP/MPLS core networks. The same technologies could also be 307 used within the same data center to provide for scale or for 308 interconnecting services across L3 domains, as needed. Virtual 309 Private LAN Service (VPLS) has been used to provide 310 transparent LAN services over IP/MPLS WANs while IP VPNs, 311 including BGP/MPLS IP VPNs and IPsec VPNs, have been used to 312 provide virtual IP routing instances over a common IP/MPLS 313 core network. 315 All these technologies have been combined to maximize their 316 respective benefits. At the edge of the network, such as in 317 access networks, VLAN and PBB are commonly used technologies. 318 Aggregation networks typically use VPLS or BGP/MPLS IP VPNs to 319 groom traffic on a common IP/MPLS core. 321 It should be noted that Ethernet has kept evolving because of 322 its attractive features, specifically its auto-discovery 323 capabilities and the ability of hosts to physically relocate 324 on the same LAN without requiring renumbering. In addition, 325 Ethernet switches have become commodity, creating a financial 326 incentive for interconnecting hosts in the same community with 327 Ethernet switches. The network layer (layer3), on the other 328 hand, has become pre-dominantly IP. Thus, communication across 329 LANs uses IP routing. 331 4. Generic Cloud Networking Architecture 333 A generic architecture for Cloud Networking is depicted in 334 Figure 1. 336 ,---------. 337 ,' `. 338 ( IP/MPLS ) 339 `. ,' 340 `-+------+' 341 +--+--+ +-+---+ 342 | GW |+-+| GW | 343 +-+---+ +-----+ 344 / \ 345 +----+---+ +---+-----+ 346 | Core | | Core | 347 | SW/Rtr | | SW/Rtr | 348 +-+----`.+ +-+---+---+ 349 / \ .' \ 350 +---+--+ +-`.+--+ +--+----+ 351 | ToR | | ToR | | ToR | 352 +-+--`.+ +-+-`.-+ +-+--+--+ 353 .' \ .' \ .' `. 354 __/_ _i./ i./_ _\__ 355 :VSw : :VSw : :VSw : :VSw : 356 '----' '----' '----' '----' 358 Figure 1 : A Generic Architecture for Cloud Networking 360 A cloud network is composed of intra-Data Center (DC) networks 361 and network services, and inter-DC network connectivity. DCs 362 may belong to a cloud service provider connected to one or 363 more network service providers, different cloud service 364 providers each connected to one or more network service 365 providers, or a hybrid of DCs operated by the enterprise 366 customers and the cloud service provider(s). It may also 367 provide access to the public and/or enterprise customers. 369 The following network components are present in a DC: 371 - VSw or virtual switch: software based Ethernet switch 372 running inside the server blades. VSw may be single or 373 dual-homed to the Top of Rack switches (ToRs). The 374 individual VMs appear to a VSw as IP hosts connected via 375 logical Ethernet interfaces. The VSw may evolve to 376 support IP routing functionality. 378 - ToR or Top of Rack: hardware-based Ethernet switch 379 aggregating all Ethernet links from the server blades in 380 a rack representing the entry point in the physical DC 381 network for the hosts. ToRs may also perform routing 382 functionality. ToRs are usually dual-homed to the Core 383 SW. Other deployment scenarios may use an EoR (End of 384 Row) switch to provide a similar function as a ToR. 386 - Core SW (switch): high capacity core node aggregating 387 multiple ToRs. This is usually a cost effective Ethernet 388 switch. Core switches can also support IP routing 389 capabilities. 391 - DC GW: gateway to the outside world providing DC 392 Interconnect and connectivity to Internet and VPN 393 customers. In the current DC network model, this may be a 394 Router with Virtual Routing capabilities and/or an IPVPN 395 [RFC4364]/L2VPN [RFC7461][RFC[4762] Provider Edge [PE]. 397 A DC network also contains other network services, such as 398 firewalls, load-balancers, IPsec gateways, and SSL 399 acceleration gateways. These network services are not 400 currently discussed in this document as the focus is on the 401 routing and switching services. The traditional DC deployment 402 employs VLANs to isolate different VM groups throughout the 403 Ethernet switching network within a DC. The VM Groups are 404 mapped to VLANs in the vSws. The ToRs and Core SWs may employ 405 VLAN trunking to eliminate provisioning touches in the DC 406 network. In some scenarios, IP routing is extended down to the 407 ToRs, and may be further extended to the hypervisor as 408 discussed earlier. However, this routing unless it provides 409 for virtual forwarding function, it would require it to be 410 limited to one IP domain addressed from the same address 411 realm. 413 Any new DC and cloud networking technology should to be able 414 to fit as seamlessly as possible with this existing DC model, 415 at least in a non-greenfield environment. In particular, it 416 should be possible to introduce enhancements to various tiers 417 in this model in a phased approach without disrupting the 418 other elements. 420 Depending upon the scale, DC distribution, operations model, 421 Capex and Opex aspects, DC switching elements can act as 422 strict L2 switches and/or provide IP routing capabilities, 423 including VPN routing and/or MPLS support if the operation 424 environment allows. In smaller DCs, it is likely that some 425 tier layers get collapsed, and that Internet connectivity, 426 inter-DC connectivity and VPN support will be handled by Core 427 Nodes that perform the DC GW role as well. 429 The DC network architecture described in this section can be 430 used to provide generic L2-L3 service connectivity to each 431 tenant as depicted in Figure 2: 433 ---+--. ---+--- 434 ....( VRF1 )...... ( VRF2 ) 435 | '-----' | '-----' 436 | Tenant1 |ELAN12 Tenant1| 437 |ELAN11 ....|........ |ELAN13 438 '':'''''''':' | | '':'''''''':' 439 ,'. ,'. ,+. ,+. ,'. ,'. 440 (VM )....(VM ) (VM )... (VM ) (VM )....(VM ) 441 `-' `-' `-' `-' `-' `-' 443 Figure 2 : Logical Service connectivity for one tenant 445 In this example one or more virtual routing contexts 446 distributed on multiple DC GWs and one or more ELANs (e.g., 447 one per Application) running on DC switches are assigned for 448 DC tenant 1. ELAN is a generic term for Ethernet multipoint 449 service, which in the current DC environment is implemented 450 using 12-bit VLAN tags. Other possible ELAN technologies are 451 discussed in section 6. 453 For a multi-tenant DC, this type of service connectivity or a 454 variation could be used for each tenant. In some cases only L2 455 connectivity is required, i.e., only an ELAN may be used to 456 interconnect VMs and customer sites. 458 5. Challenges in Existing Deployments 460 This section summarizes the challenges faced with the present 461 mode of operation described in the previous section and the 462 issues arising for next generation DC networks as described in 463 [NVo3-problem-statement]. 465 With the introduction of multi-tenant DCs, providing each 466 tenant dedicated virtual compute and storage resources and/or 467 application services, the DC network must also provide each 468 tenant access to these resources and services. In addition, 469 some tenants may require some aspect of their services 470 available to other businesses in a B-to-B model or to the 471 public. Every tenant requires service connectivity to its own 472 resources with secure separation from other tenant domains. 473 Connectivity needs to support various deployment models, 474 including interconnecting customer-hosted data center 475 resources to Cloud Service Provider (CSP) hosted resources 476 (Virtualized DC for the tenant). This connectivity may be at 477 layer2 or layer3. 479 Currently, large DCs are often built on a service architecture 480 where each tenant is assigned two or more VLANs. VLANs 481 configured in Ethernet edge and core switches are 482 interconnected by IP routing running in a few centralized 483 routers. There may be some cases though where IP routing might 484 be used in the DC core nodes or even in the TORs inside a DC. 486 5.1. VLAN Space Limitation 488 Existing DC deployments provide customer separation and flood 489 containment, including support for DC infrastructure 490 interconnectivity, using Ethernet VLANs [802.1q]. A 12-bit 491 VLAN tag provides support for a maximum of 4094 VLANs. 493 4094 VLANs are inadequate for a CSP looking to expand its 494 customer base. For example, there are a number of VPN 495 deployments (VPLS and IP VPN) that serve more than 20K 496 customers. If a VPN service provider with 20K VPN customers 497 wants to provide cloud services to these customers or teams up 498 with an independent CSP that does, and If 50% (10k) of these 499 customers are likely to become cloud customers each requiring 500 multiple VLANs in a multi-tenant DC, 4094 VLANs will not be 501 able to support the demand. In general, 4094 VLANs will 502 support less than 4K tenants in a multi-tenant DC unless 503 constraints are imposed on the VM placement so that the DC is 504 subdivided into multiple non-congruent domains, each with 4K 505 VLANs. 507 The cloud networking infrastructure needs to provide support 508 for a much bigger number of virtual Layer2 (L2) domains than 509 4K, as discussed in [NVo3-problem-statement] Section 2.7, 510 allowing for resource placement flexibility and efficient 511 resource utilization as discussed in [NVo3-problem-statement] 512 Section 2.2. 514 5.2. MAC, IP, and ARP Issues 516 Virtual Machines are the basic compute blocks provided to 517 cloud tenants. Every server blade typically supports 16-40 VMs 518 today with 100 or more VMs per server blade possibly becoming 519 common in the near future. Every VM may have multiple 520 interfaces for provider and enterprise management, VM mobility 521 and tenant access, each with its own MAC and IP addresses. For 522 a sizable DC, this may translate into millions of VM IP and 523 MAC addresses. From a cloud network viewpoint, this scale 524 number will be an order of magnitude higher. 526 Supporting this amount of IP and MAC addresses, including the 527 associated dynamic behavior (e.g., ARP), throughout the DC 528 Ethernet switches and routers is very challenging in an 529 Ethernet VLAN and regular routing environment. 531 A Core Ethernet switch supporting VLAN bridging domains 532 [802.1q] learns the MAC addresses for every single VM 533 interface that sends traffic through the switch albeit in the 534 context of VLANs to which these MACs belong. VLANs, as 535 discussed earlier, provide for MAC address separation across 536 tenants and therefore address the problem in [NVo3-problem- 537 statement] Section 2.5 for L2 bridged domains. Throwing 538 memory to increase the MAC Forwarding DataBase (FDB) size 539 affects the cost of these switches, and there could still be a 540 scale constraint. MAC address table scale is highlighted in 541 [NVo3-problem-statemnt] Section 2.3. In addition, as the 542 number of MACs that switches need to learn increases, 543 convergence time could increase, and flooding activity will 544 increase upon a topology change as the core switches flush and 545 re-learn the MAC addresses. Simple operational mistakes may 546 lead to duplicate MAC entries within the same VLAN domain and 547 security issues due to administrative MAC assignment used 548 today for VM interfaces. Similar concerns about memory 549 requirements and related cost apply to DC Edge switches 550 (ToRs/EoRs) and DC GWs. 552 From a router perspective, it is important to maximize the 553 utilization of available resources in both control and data 554 planes through flexible mapping of VMs and related VLANs to 555 routing interfaces. This is not easily done in the current 556 VLAN based deployment environment where the use of VLAN 557 trunking limits the allocation of VMs to only local routers. 559 The amount of ARP traffic grows linearly with the number of 560 hosts on a LAN. For 1 million VM hosts, it can be expected 561 that the amount of ARP traffic will be in the range of half 562 million ARPs per second at the peak, which corresponds to over 563 200 Mbps of ARP traffic [MYERS]. Similarly, on a server, the 564 amount of ARP traffic grows linearly with the number of 565 virtual L2 domains/ELANs instantiated on that server and the 566 number of VMs in that domain. Besides the link capacity 567 wasted, which may be small compared to the link capacities 568 deployed in DCs, the computational burden may be prohibitive. 569 In a large-DC environment, the large number of hosts and the 570 distribution of ARP traffic may lead to a number of 571 challenges: 573 - Processing overload and overload of ARP entries on the 574 Server/Hypervisor. This is caused by the increased number 575 of VMs per server blade and the size of related ELAN 576 domains. For example, a server blade with 100 VMs, each 577 in a separate L2 domain with 100 VMs each would need to 578 support 10K ARP entries and the associated ARP processing 579 while performing the other compute tasks. 580 - Processing overload and exhaustion of ARP entries on the 581 Routers/PEs and any other L3 Service Appliances (Firewall 582 (FW), Load-Balancer (LB) etc.). This issue is magnified 583 by the L3 virtualization at the service gateways. For 584 example, a gateway PE handling 10K ELANs each with 10 VMs 585 will result in 100K hosts sending/receiving traffic 586 to/from the PE, thus requiring the PE to learn 100K ARP 587 entries. It should be noted that if the PE supports 588 Integrated Routing and Bridging (IRB), it must support 589 the associated virtual IP RIBs/FIBs and MAC FDBs for 590 these hosts in addition to the ARP entries. 591 - Flood explosion throughout Ethernet switching network. 592 This is caused by the use of VLAN trunking and implicitly 593 by the lack of per VPN flood containment. 595 DC and DC-interconnect technologies, including control 596 plane, that minimize the negative impact of ARP, MAC and IP 597 entry explosion on individual network elements in a DC or 598 cloud network hierarchy are needed. 600 5.3. Per VLAN flood containment 602 From an operational perspective, DC operators try to minimize 603 the provisioning touches required for configuring a VLAN 604 domain by employing VLAN trunks on the L2 switches. This comes 605 at the cost of flooding broadcast, multicast and unknown 606 unicast frames outside of the boundaries of the actual VLAN 607 domain. Containment of a broadcast domain identified by a VLAN 608 ID to a POD, and connecting a broadcast domain to a local 609 router limits the L2 broadcast domain span but also limits the 610 flexibility of placing VMs across PODs in a DC or a cloud. 611 This is the problem identified in [NVo3-problem-statement] 612 Section 3.4. 614 The cloud-networking infrastructure needs to prevent 615 unnecessary traffic from being sent/leaked to undesired 616 locations. 618 5.4. Convergence and multipath support 620 Spanning Tree is used in the current DC environment for loop 621 avoidance in the Ethernet switching domain. 623 STP can take 30 to 50 seconds to repair a topology. Practical 624 experience shows that Rapid STP (RSTP) can also take multiple 625 seconds to converge, such as when the root bridge fails. 627 STP eliminates loops by disabling ports. The result is that 628 only one path is used to carry traffic. The capacity of 629 disabled links cannot be utilized, leading to inefficient use 630 of resources. 632 In a small DC deployment, multi-chassis LAG (MC-LAG) support 633 may be sufficient initially to provide for loop-free 634 redundancy as an STP alternative. However, in medium or large 635 DCs it is challenging to use MC-LAGs solely across the network 636 to provide for resiliency and loop-free paths without 637 introducing a layer2 routing protocol: i.e. for multi-homing 638 of server blades to ToRs, ToRs to Core SWs, Core SWs to DC 639 GWs. MC-LAG may work as a local mechanism but it has no 640 knowledge of the end-to-end paths so it does not provide any 641 degree of traffic steering across the network. 643 Efficient and mature link-state protocols, such as IS-IS, 644 provide rapid failover times, can compute optimal paths and 645 can fully utilize multiple parallel paths to forward traffic 646 between 2 nodes in the network. 648 Unlike OSPF, IS-IS runs directly at L2 (i.e. no reliance on 649 IP) and does not require any configuration. Therefore, IS-IS 650 based DC networks are to be favored over STP-based networks. 651 IEEE Shortest Path Bridging (SPB), based on IEEE 802.1aq and 652 IEEE 802.1Qbp, and IETF TRILL [RFC6325] are technologies that 653 enable Layer2 networks using IS-IS for Layer2 routing. 655 5.5. Optimal traffic forwarding 657 Optimal traffic forwarding requires (1) efficient utilization 658 of all available link capacity in a DC and DC-interconnect, 659 and (2) traffic forwarding on the shortest path between any 660 two communicating VMs within the DC or across DCs. 662 Optimizing traffic forwarding between any VM pair in the same 663 virtual domain is dependent on (1) the placement of these VMs 664 and their relative proximity from a network viewpoint, and (2) 665 the technology used for computing the routing/switching path 666 between these VMs. The latter is especially important in the 667 context of VM Mobility, moving a VM from one network location 668 to another, while maintaining its layer2 and Layer3 (IP) 669 addresses. 671 Ethernet-based forwarding between two VMs in traditional DCs 672 relies on the MAC-destination Address that is unique per VM 673 interface in the context of a virtual domain (e.g., VLAN). In 674 traditional IEEE technologies (e.g., 802.1q, 802.1ad, 802.1ah) 675 and IETF L2VPN (i.e., VPLS), Ethernet MAC reachability is 676 always learnt in the data plane. Other IEEE and IETF 677 technologies allow MAC reachability to be learnt in the 678 control plane as discussed further in Section 6. . In all 679 these cases, it is important that as a VM is moved from one 680 location to another: (1) VM MAC reachability convergence 681 happens fast to minimize traffic black-holing, and (2) 682 forwarding takes the shortest path. 684 IP-based forwarding relies on the destination IP address. ECMP 685 load balancing relies on flow-based criteria. An IP host 686 address is unique per VM interface. However, hosts on a LAN 687 share a subnet mask, and IP routing entries are based on that 688 subnet address. Thus, when VMs are on the same LAN and 689 traditional forwarding takes place, these VMs forward traffic 690 to each other by relying on ARP or IPv6 Neighbor discovery to 691 identify the MAC address of the destination and on the 692 underlying layer2 network to deliver the resulting MAC frame 693 to is destination. However, when VMs, as IP hosts across 694 layer2 virtual domains, need to communicate they rely on the 695 underlying IP routing infrastructure. 697 In addition, when a DC is an all-IP DC, VMs are assigned a 698 host address with /32 subnet in the IPv4 case, or /64 or /128 699 host address in the IPv6 case, and rely on the IP routing 700 infrastructure to route the IP packets among VMs. In this 701 latter case, there is really no need for layer2 awareness 702 potentially beyond the hypervisor switch at the server hosting 703 the VM. In either case, when a VM moves location from one 704 physical router to another while maintaining its IP identity 705 (address), the underlying IP network must be able to route the 706 traffic to the destination and must be able to do that on the 707 shortest path. 709 Thus, in the case of IP address aggregation as in a subnet, 710 optimality in traffic forwarding to a VM will require 711 reachability to the VM host address rather than only the 712 subnet. That is what is often referred to as punching a hole 713 in the aggregate at the expense of routing and forwarding 714 table size increase. 716 As in layer2, layer3 may capitalize on hierarchical tunneling 717 to optimize the routing/FIB resource utilization at different 718 places in the network. If a hybrid of subnet-based routing and 719 host-based routing (host-based routing here is used to refer 720 to hole-punching in the aggregate) is used, then during VM 721 mobility, routing transition can take place, and traffic may 722 be routed to a location based on subnet reachability or to a 723 location where the VM used to be attached. In either of these 724 cases, traffic must not be black-holed. It must be directed 725 potentially via tunneling to the location where the VM is. 726 This requires that the old routing gateway knows where the VM 727 is currently attached. How to obtain that information can be 728 based on different techniques with tradeoffs. However, this 729 traffic triangulation is not optimal and must only exist in 730 the transition until the network converges to a shortest path 731 to the destination. 733 5.6. Efficient multicast 735 STP bridges typically perform IGMP and/or PIM snooping in 736 order to optimize multicast data delivery. However, this 737 snooping is performed locally by each bridge following the STP 738 topology where all the traffic goes through the root bridge. 739 This may result in sub-optimal multicast traffic delivery. In 740 addition, each customer multicast group is associated with a 741 forwarding tree throughout the Ethernet switching network. 742 Solutions must provide for efficient Layer2 multicast. In an 743 all-IP network, explicit multicast trees in the DC network can 744 be built via multicast signaling protocols (e.g., PIM-SSM) 745 that follows the shortest path between the destinations and 746 source(s). In an IPVPN context, Multicast IPVPN based on 747 [MVPN] can be used to build multicast trees shared among 748 IPVPNs, specific to VPNs, and/or shared among multicast groups 749 across IPVPNs. 751 5.7. L3 virtualization 753 In order to provide tenant L3 separation while supporting 754 overlapping IP addressing and privacy across tenants, as 755 discussed in [NV03-roblem-statement] Section 2.5, a number of 756 schemes were implemented in the DC environment. Some of these 757 schemes, such as double NATing are operationally complex and 758 prone to operator errors. Virtual Routing contexts, Virtual 759 Device contexts, or dedicated hardware-routers are positioned 760 in the DC environment as an alternative to these mechanisms. 761 Every customer is assigned a dedicated routing context with 762 associated control plane protocols. For instance, every 763 customer gets an IP routing instance controlled by its own 764 routing. Assigning virtual or hardware routers to each 765 customer, while supporting thousands of customers in a DC, 766 is neither scalable nor cost-efficient. Section 6 further 767 discusses the applicability of BGP/MPLS IP VPNs to 768 L3vitualization. 770 5.8. Connectivity to existing tenant VPN sites 772 It is expected that cloud services will have to span larger 773 geographical areas in the near future and that existing VPN 774 customers will require access to VM and storage facilities 775 for virtualized data center applications. Hence, the DC 776 network virtualization must interoperate with deployed and 777 evolving VPN solutions (e.g., IP VPN, VPLS, VPWS, PBB-VPLS, 778 E-VPN and PBB-EVPN). 780 Section 6 discusses this type of connectivity. 782 5.9. DC Inter-connect requirements 784 Cloud computing requirements such as VM Mobility across DCs, 785 Management connectivity, and support for East-West traffic 786 between customer applications located in different DCs imply 787 that inter-DC connectivity must be supported. These DCs can be 788 part of a hybrid cloud operated by the cloud service 789 provider(s) and/or the end-customers. 791 Mature VPN technologies can be used to provide L2/L3 DC 792 interconnect among VLANs/virtual domains located in different 793 DCs. DC-interconnect using existing VPN technologies is 794 described in Section 6. 796 5.10. VM Mobility 798 The ability to move VMs within a resource pool, whether it is 799 a local move within the same DC to another server or to a 800 distant DC, offers multiple advantages for a number of 801 scenarios, for example: 803 - In the event of a possible natural disaster, moving VMs to a 804 safe DC location decreases downtime and allows for meeting 805 Service Level Agreement (SLA) requirements. 806 - 807 - Optimized resource location: VMs can be moved to locations 808 that offer significant cost reduction (e.g. power savings), 809 or locations close to the application users. They can also 810 be moved to simply load-balance across different locations. 812 When VMs change location, it is often important to maintain 813 the existing client sessions. The VM MAC and IP addresses must 814 be preserved, and the state of the VM sessions must be copied 815 to the new location. 817 Current VM mobility tools like VMware VMotion require L2 818 connectivity among the hypervisors on the servers 819 participating in a VMotion pool. This is in addition to 820 "tenant ELAN" connectivity that provides for communication 821 between the VM and the client(s). 823 A VMotion ELAN might need to cross multiple DC networks to 824 provide the required protection or load-balancing. In 825 addition, in the current VMotion procedure, the new VM 826 location must be part of the tenant ELAN domain. When the new 827 VM is activated, a Gratuitous ARP is sent so that the MAC FIB 828 entries in the tenant ELAN are updated to direct traffic 829 destined to that VM to the new VM location. In addition, if a 830 portion of the path requires IP forwarding, the VM 831 reachability information must be updated to direct the traffic 832 on the shortest path to the VM. 834 VM mobility requirements may be addressed through the use of 835 Inter-DC VLANs to address VMotion and "tenant ELANs". However, 836 expanding "tenant ELANs" across two or more DCs will 837 accelerate VLAN exhaustion and MAC explosion issues. In 838 addition, STP needs to run across DCs leading to increased 839 convergence times and the blocking of expensive WAN bandwidth. 840 VLAN trunking used throughout the network creates 841 indiscriminate flooding across DCs. 843 L2 VPN solutions over IP/MPLS are designed to interconnect 844 sites located across the WAN as described in Section 6. 846 6. L2VPN Applicability to Cloud Networking 848 The following sections will discuss different solution 849 alternatives, re-using IEEE and IETF technologies that can 850 provide a gradual migration path from the current Ethernet 851 switching VLAN-based model to more advanced Ethernet switching 852 and IP/MPLS based models. In addition, they discuss how these 853 solutions compare to the NVo3 framework [NVo3-fmwk] and the 854 problems in [Nvo3-problem-statement] that they would still 855 need to address. This evolution is targeted to address inter- 856 DC requirements, cost considerations, and the efficient use of 857 processing/memory resources on DC networking components. 859 6.1. VLANs and L2VPN toolset 861 One approach to address some of the DC challenges discussed in 862 the previous section is to gradually deploy additional 863 technologies within existing DC networks. For example, an 864 operator may start by breaking its DC VLAN domains into 865 different VLAN islands so that each island can support up to 866 4K VLANs. VLAN Domains can then be interconnected via VPLS 867 using the DC GW as a VPLS PE [RFC4761][RFC4762]. An ELAN 868 service can be identified with one VLAN ID in one island and 869 another VLAN ID in another island with the appropriate VLAN ID 870 processed at the GW. 872 As the number of tenants in individual VLAN islands surpasses 873 4K and no further sub-division of VLAN domains is feasible or 874 desired, the operator could push VPLS deployment deeper in the 875 DC network closer to tenant systems as defined in [NVo3-fmwk], 876 it is possible in the end to retain existing VLAN-based 877 solution only in VSw and to provide L2VPN support starting at 878 the ToRs. The ToR and DC core elements need to be MPLS enabled 879 with existing VPLS solutions. 881 VPLS represents a mature virtualization and overlay technology 882 for private LAN services. This is the way it has been deployed 883 in service provider networks. It also addresses many of the 884 problems described in Section 5 and in [NVo3-problem- 885 statement] but still lacks some capabilities to address 886 others. 888 Table 1 provides a comparison between the VPLS functional 889 elements and the NVo3 framework functional elements [NVo3- 890 fmwk]. 892 Table 1: Functional comparison between VPLS and NVo3 framework 894 Nvo3 Function Matching VPLS Function 896 ----------------------------------------------------------- 897 Virtual Access Point (VAP) Attachment Circuit (AC) 899 Network Virtual Edge (NVE) Provider Edge (PE) 901 Virtual Network Instance (VNI) Virtual Switching Instance 902 (VSI) 904 Virtual Network Context (VN A 20-bit MPLS label 905 Context) identifier 907 Overlay Module and tunneling -PWE3 over IP/GRE in an IP 908 network 910 -PWE3 and MPLS in an MPLS 911 network 913 Control Plane: TBD Control plane: 915 Service signaling 917 - PWE3 T-LDP or MP-BGP 918 Core Routing: 920 - IGP: OSPF/ISIS -(TE) 922 Core Signaling: 924 - RSVP or LDP for MPLS 925 LSPs 927 Depending on the implementation model, VPLS can address some 928 of the issues described in Section 5 and in [NVo3-problem- 929 statement], but not all: 931 -Dynamic Provisioning as described in [NVo3-problem- 932 statement] Section 2.1: This is not addressed today in 933 VPLS solutions, as it has not been in scope of that work. 934 VPLS provisioning today requires management of both VLAN 935 and L2VPN addressing, and mapping of service profiles. 936 Per VLAN, per port and per VPLS configurations are 937 required at the ToR, increasing the time it takes to 938 bring up service connectivity and complicating the 939 operational model. However, a mechanism may be developed 940 to perform such provisioning dynamically as compute 941 resources are configured. It should be noted that VPLS 942 currently supports auto-discovery of PEs with instances 943 of the same VPLS service, as a component of the dynamic 944 provision of a VPLS service. 946 -VM Mobility as also defined in [NVo3-problem-statement] 947 section 2.2: VPLS supports MAC discovery as in any LAN 948 switch based on MAC learning in the data plane. Thus, as 949 a VM moves, a VPLS may lean the location of a new MAC 950 from an ARP message initiated by the VM or by seeing 951 Ethernet frames from that VM. 953 -MAC table sizes in Switches as also described in [NVo3- 954 problem-statement] Section 2.3: As opposed to an 802.1q 955 based core Ethernet network, tenant VM addresses are only 956 learned at a VPLS PE with a corresponding service 957 instance. This is because VPLS is built as an overlay on 958 a core IP/MPLS network and the core interconnecting the 959 PEs will have no knowledge of the tenant MACs. 961 -VLAN limitation as also described in [NV03-proble- 962 statement] Section 2.7: VPLS enables service instance 963 scale in a DC as it connects VLAN domains as described 964 earlier and as the service identifier for a VPLS instance 965 at a PE is based on a 20-bit MPLS label. 967 This model does not solve the potential MAC explosion on VPLS 968 PEs, depending on how close to the tenant systems the PE 969 functionality is deployed. The closer to the systems, the 970 smaller the number of VPLS instances that need to be supported on 971 a VPLS PE, and the lower should be the MAC scale need. 973 6.2. E-VPN 975 Ethernet VPN (E-VPN) [E-VPN] is evolving work in IETF L2VPN WG. 976 Ethernet VPN provides private LAN service over an IP/MPLS core. 977 E-VPN was driven by some gaps in the existing VPLS solution, and 978 by large multi-tenant DC requirements. E-VPN, similar to VPLS, is 979 provided on a PE where an E-VPN instance (EVI) provides the 980 virtual LAN bridging service. E-VPN defines types of EVIs 981 depending on the bridging domains supported in an EVI. As opposed 982 to VPLS, E-VPN provides for active-active multi-homing of CEs to 983 different PEs while eliminating loops and traffic duplications. 984 In addition, it provides for effective load-balancing across the 985 IP/MPLS core to PEs with access to the same MAC address on 986 connected CEs. In addition, as opposed to IEEE 802.1q/ad/ah 987 standards and VPLS where MAC reachability is learned in the data 988 plane, E-VPNS distributes MAC reachability across the IP/MPLS 989 core using MP-BGP extensions. Along with MAC address 990 distribution, E-VPN also distributes the IP address(es) 991 associated with the MAC, equivalent in IPv4 to ARP entries. In 992 addition, as opposed to VPLS, and more in synergy with BGP/MPLS 993 VPNs [RFC4364], E-VPN uses (MP)-BGP extensions to discover and 994 signal the service MPLS label(s) among PEs across the IP/MPLS 995 core and does not require a Pseudowire (PW) mesh among PEs per E- 996 VPN. E-VPN also allows an option for flooding suppression of BUM 997 traffic. 999 E-VPN, can be implemented at the same network elements as VPLS 1000 discussed in the previous section. However, with reduced set of 1001 protocols needed, namely PW signaling via T-LDP, and in synergy 1002 with [endsystem], E-VPN could more likely be implemented at an 1003 end-system than VPLS. 1005 E-VPN represents an evolving virtualization and overlay 1006 technology for private LAN services, albeit capitalizing on the 1007 synergy with mature BGP/MPLS IPVPNs. It also addresses many of 1008 the problems described in Section 5 and in [NVo3-problem- 1009 statement] and some of the VPLS problems, but still lacks some 1010 capabilities to address others. 1012 Table 2 provides a comparison between the E-VPN functional 1013 elements and the NVo3 framework functional elements [NVo3-fmwk]. 1015 Table 2: Functional comparison between E-VPN and NVo3 1016 framework 1018 Nvo3 Function Matching E-VPN Function 1019 ----------------------------------------------------------- 1021 Virtual Access Point (VAP) Attachment Circuit (AC) 1022 based on VLAN ID 1024 Network Virtual Edge (NVE) PE 1026 Virtual Network Instance (VNI) EVPN Instance (EVI) 1028 Virtual Network Context (VN A 20-bit MPLS label 1029 Context) identifier 1031 Overlay Module and tunneling -MPLS over MPLS tunnels 1033 -MPLS over IP/GRE in an 1034 IP network 1036 Control Plane: TBD Control plane: 1038 - MP-BGP for E-VPN 1040 Core Routing: 1042 - IGP: OSPF/ISIS -(TE) 1044 Core Signaling: 1046 - RSVP or LDP for MPLS LSPs 1048 Depending on the implementation model, E-VPN can address some 1049 of the issues described in Section 5 and in [NVo3-problem- 1050 statement], but not all: 1052 -Dynamic Provisioning as described in [NVo3-problem-statement] 1053 Section 2.1: This is not addressed today in E-VPN solutions, 1054 as it has not been in scope of that work. E-VPN provisioning 1055 today requires management of VLAN and service profiles. Per 1056 VLAN, per port and per E-VPN configurations are required, 1057 increasing the time it takes to bring up service connectivity 1058 and complicating the operational model. However, a mechanism 1059 may be developed to perform such provisioning dynamically as 1060 compute resources are configured. It should be noted that E- 1061 VPN currently supports auto-discovery of PEs with instances of 1062 the same E-VPN service, as a component of the dynamic 1063 provisioning of an E-VPN service. 1065 -VM Mobility as also defined in [NVo3-problem-statement] 1066 section 2.2: E-VPN supports VM mobility as described in 1067 Section 8. 1069 -MAC-table sizes in Switches as also described in [NVo3- 1070 problem-statement] Section 2.3: As opposed to an 802.1q based 1071 core Ethernet network, tenant VM addresses are only learned at 1072 a E-VPN PE with a corresponding service instance. This is 1073 because E-VPN is built as an overlay on a core IP/MPLS network 1074 and the core interconnecting the PEs will have no knowledge of 1075 the tenant MACs. 1077 -VLAN limitation as also described in [NV03-proble-statement] 1078 Section 2.7: E-VPN enables service instance scale in a DC as 1079 it connects VLAN domains similarly to VPLS and as the service 1080 identifier for an E-VPN instance at a PE is based on a 20-bit 1081 MPLS label. 1083 This model does not solve the potential MAC explosion on E-VPN 1084 PEs, depending on how close to the tenant systems the PE 1085 functionality is deployed. The closer to the systems, the 1086 smaller the number of VPLS instances that need to be supported on 1087 a VPLS PE, and the lower should be the MAC scale need. E-VPN 1088 could be potentially implemented at an end-system hosting the VMs 1089 to which the E-VPN services are provided. 1091 6.3. PBB and L2VPN toolset 1093 As highlighted in Section 5, the expected large number of VM 1094 MAC addresses in the DC calls out for a VM MAC hiding solution 1095 so that the ToRs and the Core Switches only need to handle a 1096 limited number of MAC addresses. 1098 PBB IEEE 802.1ah encapsulation is a standard L2 technique 1099 developed by IEEE to achieve this goal. It was designed also 1100 to address other limitations of VLAN-based encapsulations 1101 while maintaining the native Ethernet operational model 1102 deployed in the DC network. 1104 A conceptual PBB encapsulation is described in Figure 3 (for 1105 detailed encapsulation see [802.1ah]): 1107 +-------------+ 1108 Backbone | BMAC DA,SA |12B 1109 Ethernet |-------------| 1110 Header |BVID optional| 4B 1111 |-------------| 1112 Service ID| PBB I-tag | 6B 1113 |-------------| 1114 Regular |VM MAC DA,SA | 1115 Payload |-------------| 1116 | | 1117 |VM IP Payload| 1118 | | 1119 +-------------+ 1121 Figure 3 PBB encapsulation 1123 The original Ethernet packet used in this example for Inter-VM 1124 communication is encapsulated in the following PBB header: 1126 - I-tag field - organized similarly with the 802.1q VLAN 1127 tag; it includes the Ethertype, PCP and DEI bits and a 24 1128 bit ISID tag which replaces the 12 bit VLAN tag, 1129 extending the number of virtual L2 domain support to 16 1130 Million. It should be noted that the PBB I-Tag includes 1131 also some reserved bits, and most importantly the C-MAC 1132 DA and SA. What is designated as 6 bytes in the figure is 1133 the I-tag information excluding the C-MAC DA and SA. 1135 - An optional Backbone VLAN field (BVLAN) may be used if 1136 grouping of tenant domains is desired. 1138 - An outer Backbone MAC header contains the source and 1139 destination MAC addresses for the related server blades, 1140 assuming the PBB encapsulation is done at the hypervisor 1141 virtual switch on the server blade. 1143 - The total resulting PBB overhead added to the VM- 1144 originated Ethernet frame is 18 or 22 Bytes (depending on 1145 whether the BVID is excluded or not). 1147 - Note that the original PBB encapsulation allows the use 1148 of CVLAN and SVLAN in between the VM MACs and IP Payload. 1149 These fields were removed from Figure 3 since in a VM 1150 environment these fields do not need to be used on the 1151 VSw, their function is relegated to the I-SID tag. 1153 6.3.1. Addressing VLAN space exhaustion and MAC explosion 1155 In a DC environment, PBB maintains traditional Ethernet 1156 forwarding plane and operational model. For example, a vSw 1157 implementation of PBB can make use of the 24 bit ISID tag 1158 instead of the 12 bit VLAN tag to identify the virtual 1159 bridging domains associated with different VM groups. The vSw 1160 uplink towards the ToR in Figure 1 can still be treated as an 1161 Ethernet backbone interface. A frame originated by a VM can be 1162 encapsulated with the ISID assigned to the VM vSw interface 1163 and with the outer DA and SA MACs associated with the 1164 respective destination and source server blades, and then sent 1165 to the ToR switch. Performing this encapsulation at the vSw 1166 distributes the VM MAC learning to server blades with 1167 instances in the corresponding layer2 domain, and therefore 1168 alleviates this load from ToRs that aggregate multiple server 1169 blades. Alternatively, the PBB encapsulation can be done at 1170 the ToR. 1172 With PBB encapsulation, ToRs and Core SWs do not have to 1173 handle VM MAC addresses so the size of their MAC FDB tables 1174 may decrease by two or more orders of magnitude, depending on 1175 the number of VMs configured in each server blade and the 1176 number of VM virtual interfaces and associated MACs. 1178 The original PBB specification [802.1ah] did not introduce any 1179 new control plane or new forwarding concepts for the PBB core. 1180 Spanning Tree and regular Ethernet switching based on MAC 1181 learning and flooding were maintained to provide a smooth 1182 technology introduction in existing Ethernet networks. 1184 6.3.2. Fast convergence and L2 multi-pathing 1186 Additional specification work for PBB control plane has been 1187 done since then in both IEEE and IETF L2VPN. 1189 As stated earlier, STP-based layer2 networks underutilize the 1190 available network capacity as links are put in an idle state 1191 to prevent loops. Similarly, existing VPLS technology for 1192 interconnecting Layer2 network-islands over an IP/MPLS core 1193 does not support active-active dual homing scenarios. 1195 IS-IS controlled layer2 networks allow traffic to flow on 1196 multiple parallel paths between any two servers, spreading 1197 traffic among available links on the path. IEEE 802.1aq 1198 Shortest Path Bridging (SPB) [802.1aq] and emerging IEEE 1199 802.1Qbp [802.1Qbp] are PBB control plane technologies that 1200 utilize different methods to compute parallel paths and 1201 forward traffic in order to maximize the utilization of 1202 available links in a DC. In addition, a BGP based solution 1203 [PBB-EVPN] is progressing in the IETF L2VPN WG. 1205 One or both mechanisms may be employed as required. IS-IS 1206 could be used inside the same administrative domain (e.g., a 1207 DC), while BGP may be employed to provide reachability among 1208 interconnected Autonomous Systems. Similar architectural 1209 models have been widely deployed in the Internet and for large 1210 VPN deployments. 1212 IS-IS and/or BGP are also used to advertise Backbone MAC 1213 addresses and to eliminate B-MAC learning and unknown unicast 1214 flooding in the forwarding plane, albeit with tradeoffs. The 1215 B-MAC FIB entries are populated as required from the resulting 1216 IS-IS or BGP RIBs. 1218 Legacy loop avoidance schemes using Spanning Tree and local 1219 Active/Active MC-LAG are no longer required as their function 1220 (layer2 routing) is replaced by the indicated routing 1221 protocols (IS-IS and BGP). 1223 6.3.3. Per ISID flood containment 1225 Service auto-discovery provided by 802.1aq SPB [802.1aq] and 1226 BGP [PBB-EVPN] is used to distribute ISID related information 1227 among DC nodes, eliminating any provisioning touches 1228 throughout the PBB infrastructure. This implicitly creates 1229 backbone distribution trees that provide per ISID automatic 1230 flood and multicast containment. 1232 6.3.4. Efficient multicast support 1234 IS-IS [802.1aq] and BGP [PBB-EVPN] could be used to build 1235 optimal multicast distribution trees. In addition, PBB and 1236 IP/MPLS tunnel hierarchy may be used to aggregate multiple 1237 customer multicast trees sharing the same nodes by associating 1238 them with the same backbone forwarding tree that may be 1239 represented by a common Group BMAC and optionally a P2MP LSP. 1240 More details will be discussed in a further version of the 1241 draft. 1243 6.3.5. Tunneling options for PBB ELAN: Ethernet, IP and MPLS 1245 A solution for DC ELAN domains based on PBB ISIDs, PBB 1246 encapsulation and IS-IS and/or BGP control plane was 1247 introduced. 1249 IETF L2 VPN specifications [PBB-VPLS] or [PBB-EVPN] enable the 1250 transport of PBB frames using PW over MPLS or simply MPLS, and 1251 implicitly allow the use of MPLS Traffic Engineering and 1252 resiliency toolset to provide for advanced traffic steering 1253 and faster convergence. 1255 Transport over IP/L2TPv3 [RFC 4719] or IP/GRE [RFC 4797] is 1256 also possible as an alternative to MPLS tunneling. Additional 1257 header optimization for PBB over IP/GRE encapsulated packets 1258 may be feasible. These specifications would allow for ISID 1259 based L2 overlay using a regular IP backbone. 1261 6.3.6. Use Case examples 1263 6.3.6.1. PBBN in DC, L2VPN in DC GW 1265 DC environments based on VLANs and native Ethernet operational 1266 model may want to consider using the native PBB option to 1267 provide L2 multi-tenancy, in effect the DC ELAN from Figure 2. 1268 An example of a network architecture that addresses this 1269 scenario is depicted in Figure 4: 1271 ,---------. 1272 ,' Inter-DC `. 1273 (L2VPN (PBB-VPLS) 1274 `.or PBB-EVPN),' 1275 `|-------|-' 1276 +--+--+ +-+---+ 1277 |PE GW|+-+|PE GW| 1278 .+-----+ +-----+. 1279 .' `-. 1280 .-' `\ 1281 ,' `. 1282 + Intra-DC PBBN \ 1283 | + 1284 : ; 1285 `\+------+ +------+ +--+----+-' 1286 | ToR |.. | ToR |..| ToR | 1287 +-+--+-+ +-+--+-+ +-+--+--+ 1288 .'PBB `. .'PBB `. .'PBB `. 1289 +--+-+ +-+-++ +-++-+ +-+--+ 1290 |VSw | :VSw : :VSw : :VSw : 1291 +----+ +----+ +----+ +----+ 1293 Figure 4 PBB in DC, PBB-VPLS or PBB-EVPN for DC Interconnect 1295 PBB inside the DC core interoperates seamlessly with VPLS used 1296 for L2 DC-Interconnect to extend ELAN domains across DCs. This 1297 expansion may be required to address VM Mobility requirements 1298 or to balance the load on DC PE gateways. Note than in PBB- 1299 VPLS case, just one or a handful of infrastructure B-VPLS 1300 instances are required, providing Backbone VLAN equivalent 1301 function. 1303 PBB encapsulation addresses the expansion of the ELAN service 1304 identification space with 16M ISIDs and solves MAC explosion 1305 through VM MAC hiding from the Ethernet core. 1307 PBB SPB [802.1aq] is used for core routing in the ToRs, Core 1308 SWs and PEs. If the DCs that need to be interconnected at L2 1309 are part of the same administrative domain, and scaling is not 1310 an issue, SPB/IS-IS may be extended across the VPLS 1311 infrastructure. If different AS domains are present, better 1312 load balancing is required between the DCs and the WAN, or IS- 1313 IS extension across DCs causes scaling issues, then BGP 1314 extensions described in [PBB-EVPN] must be employed. 1316 The forwarding plane, MAC FIB requirements and the Layer2 1317 operational model in the ToR and Core SW are maintained. The 1318 VSw sends PBB encapsulated frames to the ToR as described in 1319 the previous section. ToRs and Core SWs still perform standard 1320 Ethernet switching using the outer Ethernet header. 1322 From a control plane perspective, VSw uses a default gateway 1323 configuration to send traffic to the ToR, as in regular IP 1324 routing case. VSw BMAC learning on the ToR is done through 1325 either LLDP or VM Discovery Protocol (VDP) described in 1326 [802.1Qbg]. Identical mechanisms may be used for the ISID. 1327 Once this information is learned on the ToR it is 1328 automatically advertised through SPB. If PBB-EVPN is used in 1329 the DC GWs, MultiProtcol (MP)-BGP will be used to advertise 1330 the ISID and BMAC over the WAN as described in [PBB-EVPN]. 1332 6.3.6.2. PBBN in vSw, L2VPN in the ToR 1334 A variation of the use case example from the previous section 1335 is depicted in Figure 5: 1337 ,---------. 1338 ,' Inter-DC `. 1339 (L2VPN (PBB-VPLS) 1340 `.or PBB-EVPN),' 1341 `|-------|-' 1342 +--+--+ +-+---+ 1343 |PE GW|+-+|PE GW| 1344 .+-----+ +-----+. 1345 .' `-. 1346 .-' `\ 1347 ,' `. 1348 + Intra-DC L2VPN over \ 1349 | IP or MPLS tunneling + 1350 : ; 1351 `\+------+ +------+ +--+----+-' 1352 | ToR |.. | ToR |..| ToR | 1353 +-+--+-+ +-+--+-+ +-+--+--+ 1354 .'PBB `. .'PBB `. .'PBB `. 1355 +--+-+ +-+-++ +-++-+ +-+--+ 1356 |VSw | :VSw : :VSw : :VSw : 1357 +----+ +----+ +----+ +----+ 1359 Figure 5 PBB in VSw, L2VPN at the ToR 1361 The procedures from the previous section are used at the VSw: 1362 PBB encapsulation and Ethernet BVLANs can be used on the VSw 1363 uplink. L2VPN infrastructure is replacing the BVLAN at the ToR 1364 enabling the use of IP (GRE or L2TP) or MPLS tunneling. 1366 L2 networking still has the same control plane choices: IS-IS 1367 [802.1aq] and/or BGP [PBB-EVPN], independently from the 1368 tunneling choice. 1370 6.3.7. NVo3 applicability 1372 Table 3 provides a comparison between the PBB-VPLS VPN functional 1373 elements and the NVo3 framework functional elements [NVo3-fmwk]. 1375 Table 3: Functional comparison between PBB-VPLS and NVo3 1376 framework 1378 Nvo3 Function Matching PBB-VPLS 1379 Function 1381 ---------------------------------------------------------- 1382 Virtual Access Point (VAP) Attachment Circuit (AC) 1383 based on I-SID 1385 Network Virtual Edge (NVE) PE 1387 Virtual Network Instance (VNI) VSI 1389 Virtual Network Context (VN MPLS-label for PBB-VSI 1390 Context) identifier and I-SID if the PE is 1391 PBB edge 1393 Overlay Module and tunneling -MPLS over MPLS tunnels 1395 -MPLS over IP/GRE in an 1396 IP network 1398 Control Plane: TBD Control plane: 1400 - MP-BGP for auto- 1401 discovery 1402 - PWE3 T-LDP for PW 1403 signaling 1405 Core Routing: 1407 - IGP: OSPF/ISIS -(TE) 1409 Core Signaling: 1411 - RSVP or LDP for MPLS LSPs 1413 Table 4 provides a comparison between the PBB-EVPN functional 1414 elements and the NVo3 framework functional elements [NVo3-fmwk]. 1416 Table 4: Functional comparison between E-VPN and NVo3 1417 framework 1419 Nvo3 Function Matching PBB-EVPN 1420 Function 1422 ---------------------------------------------------------- 1423 Virtual Access Point (VAP) Attachment Circuit (AC) 1424 based on I-SID 1426 Network Virtual Edge (NVE) PE 1428 Virtual Network Instance (VNI) EVPN Instance (EVI) 1430 Virtual Network Context (VN MPLS label for PBB-EVI 1431 Context) identifier and I-SID if the PE is 1432 PBB edge 1434 Overlay Module and tunneling -MPLS over MPLS tunnels 1436 -MPLS over IP/GRE in an 1437 IP network 1439 Control Plane: TBD Control plane: 1441 - MP-BGP for E-VPN 1443 Core Routing: 1445 - IGP: OSPF/ISIS -(TE) 1447 Core Signaling: 1449 - RSVP or LDP for MPLS LSPs 1451 Depending on the implementation model, PBB-EVPN and PBB-VPLS 1452 can address some of the issues described in Section 5 and in 1453 [NVo3-problem-statement], but not all: 1455 -Dynamic Provisioning as described in [NVo3-problem- 1456 statement] Section 2.1: This is not addressed today in 1457 PBB and PBB-VPN solutions, as it has not been in scope of 1458 the work for either. However, a mechanism may be 1459 developed to perform such provisioning dynamically as 1460 compute resources are configured. It should be noted that 1461 PBB-VPLS and PBB-EVPN currently support auto-discovery of 1462 PEs with instances of the same VPLS or E-VPN service, as 1463 a component of the dynamic provisioning of a VPLS/E-VPN 1464 service. 1466 -VM Mobility as also defined in [NVo3-problem-statement] 1467 section 2.2: PBB-EVPN and PBB-VPLS support VM MAC 1468 mobility as the 802.1q and VPLS solution do based on MAC 1469 learning in the data plane. 1471 -MAC table sizes in Switches as also described in [NVo3- 1472 problem-statement] Section 2.3: As opposed to an 802.1q- 1473 based core Ethernet network, tenant VM addresses are only 1474 learned at a PBB edge. If the VsW implements PBB edge 1475 functionality and the ToR implements PBB-EVPN or PBB- 1476 VPLS, then the vsW will learn the MAC addresses of other 1477 VMs and devices in the same LAN, but the ToR will also 1478 learn the MAC addresses of Backbone bridges that will be 1479 on the order of number of servers not VMs, conserving MAC 1480 FDB entries on the ToR. This is because there are two 1481 layers of overlay, one at the VsW for PBB, and one at the 1482 ToR for VPLS or E-VPN, on a core IP/MPLS network. 1484 -VLAN limitation as also described in [NV03-proble- 1485 statement] Section 2.7: The number of service instances 1486 that can supported is 16 Millions. 1488 6.3.8. Connectivity to existing VPN sites and Internet 1489 The main reason for extending the ELAN space beyond the 4K 1490 VLANs is to be able to serve multiple DC tenants whereby the 1491 total number of service domains needed exceeds 4K. Figure 6 1492 represents the logical service view where PBB ELANs are used 1493 inside one or multiple DCs to connect to existing IP VPN 1494 sites. It should be noted that the PE GW should be able to 1495 perform integrated routing in a VPN context and bridging in 1496 VSI context: 1498 Tenant 1 sites connected over IP VPN 1500 ,--+-'. ;-`.--. 1501 ( PE ) VRFs on PEs . PE ) 1502 '-----' '-----' 1503 | | 1504 ,-------------------------------. 1505 ( IP VPN over IP/MPLS WAN ) 1506 `---.'-----------------------`.-' 1507 +--+--+ IP VPN VRF on PE GWs +-+---+ 1508 .....|PE GW|...... |PE GW| 1509 DC with PBB | +-----+ | +--+--+ 1510 Tenant 1 | |PBB ELAN12 | 1511 view PBB|ELAN11 ......|...... PBB|ELAN13 1512 '':'''''''':' | | '':'''''''':' 1513 ,'. ,'. ,+. ,+. ,'. ,'. 1514 (VM )....(VM ) (VM )... (VM ) (VM )....(VM ) 1515 `-' `-' `-' `-' `-' `-' 1516 Compute Resources inside DC 1518 Figure 6 Logical Service View with IP VPN 1520 DC ELANs are identified with 24-bit ISIDs instead of VLANs. At 1521 the PE GWs, an IP VPN VRF is configured for every DC tenant. 1522 Each "ISID ELAN" for Tenant 1 is seen as a logical Ethernet 1523 endpoint and is assigned an IP interface on the Tenant 1 VRF. 1524 Tenant 1 enterprise sites are connected to IP VPN PEs 1525 distributed across the WAN. IP VPN instances on PE GWs can be 1526 automatically discovered and connected to the WAN IP VPN using 1527 standard procedures [RFC4364]. 1529 In certain cases, the DC GW PEs are part of the IPVPN service 1530 provider network providing IPVPN services to the enterprise 1531 customers. In other cases, DC PEs are operated and managed by 1532 the DC/cloud provider and interconnect to multiple IPVPN 1533 service providers using inter-AS BGP/MPLS models A, B, or C 1534 [RFC4364]. The same discussion applies to the case of IPSec 1535 VPNs from a PBB ELAN termination perspective. 1537 If tenant sites are connected to the DC using WAN VPLS, the PE 1538 GWs need to implement the BEB function described in the PBB- 1539 VPLS PE model [PBB-VPLS] and the procedures from [PBB-Interop] 1540 to perform the required translation. Figure 7 describes the 1541 VPLS WAN scenario: 1543 Customer sites connected over VPLS 1545 ,--+-'. ;-`.--. 1546 ( PE ) VPLS on PEs . PE ) 1547 '-----' '-----' 1548 | | 1549 ,-------------------------------. 1550 ( VPLS over IP/MPLS WAN ) 1551 `---.'-----------------------`.-' 1552 +--+--+ +-+---+ 1553 |PE GW| <-- PBB-VPLS/BEB --> |PE GW| 1554 DC with PBB +--+--+ +--+--+ 1555 Tenant 1 | | 1556 view PBB|ELAN11 PBB|ELAN13 1557 '':'''''''':' '':'''''''':' 1558 ,'. ,'. ,'. ,'. 1559 (VM ) .. (VM ) (VM ) .. (VM ) 1560 `-' `-' `-' `-' 1561 Compute Resources inside DC 1563 Figure 7 Logical Service View with VPLS WAN 1565 One VSI is required at the PE GW for every DC ELAN domain. 1566 Same as in the IP VPN case, DC PE GWs may be fully integrated 1567 as part of the WAN provider network or using Inter-AS/Inter- 1568 provider models A,B or C. 1570 The VPN connectivity may be provided by one or multiple PE 1571 GWs, depending on capacity need and/or the operational model 1572 used by the DC/cloud operator. 1574 If a VM group is serving Internet connected customers, the 1575 related ISID ELAN will be terminated into a routing context 1576 (global public instance or another VRF) connected to the 1577 Internet. Same as in the IP VPN case, the 24bit ISID will be 1578 represented as a logical Ethernet endpoint on the Internet 1579 routing context and an IP interface will be allocated to it. 1580 Same PE GW may be used to provide both VPN and Internet 1581 connectivity with the routing contexts separated internally 1582 using the IP VPN models. 1584 6.3.9. DC Interconnect 1586 L2 DC interconnect may be required to expand the ELAN domains 1587 for Management, VM Mobility or when a VM Group needs to be 1588 distributed across DCs. 1590 PBB may be used to provide ELAN extension across multiple DCs 1591 as depicted in Figure 8: 1593 ,-------------------------------. 1594 ( IP/MPLS WAN ) 1595 `---.'------------------------`.' 1596 +--+--+ +-+---+ 1597 |PE GW| <----- PBB BCB ----> |PE GW| 1598 DC with PBB +--+--+ +--+--+ 1599 Tenant 1 | | 1600 view PBB|ELAN11 PBB|ELAN11 1601 '':'''''''':' '':'''''''':' 1602 ,'. ,'. ,'. ,'. 1603 (Hvz) .. (Hvz) (Hvz) .. (Hvz) 1604 `-' `-' `-' `-' 1605 Compute Resources inside DC 1607 Figure 8 PBB BCB providing VMotion ELAN 1609 ELAN11 is expanded across DC to provide interconnect for the 1610 pool of server blades assigned to the same VMotion domain. 1611 This time Hypervisors are connected directly to ELAN11. The PE 1612 GW operates in this case as a PBB Backbone Core Bridge (BCB) 1613 combined with PBB-EVPN capabilities [PBB-EVPN]. The I-SID 1614 ELANs do not require any additional provisioning touches and 1615 do not consume additional MPLS resources on the PE GWs. Per I- 1616 SID auto-discovery and flood containment is provided by IS- 1617 IS/SPB [802.1aq] and BGP [PBB-EVPN]. 1619 6.3.10. Interoperating with existing DC VLANs 1621 While green field deployments will definitely benefit from all 1622 the advantages described in the previous sections, in many 1623 other scenarios, existing DC VLAN environments will have to be 1624 gradually migrated to the new architecture. Figure 9 depicts 1625 an example of a possible migration scenario where both PBB and 1626 VLAN technologies are present: 1628 ,---------. 1629 ,' Inter-DC `. 1630 (L2VPN (PBB-VPLS) 1631 `.or PBB-EVPN),' 1632 `-/------\-' 1633 +---+-+ +-+---+ 1634 |PE GW|+-+|PE GW| 1635 .-+-----+ +-----+:-. 1636 .-' `-. 1637 ,' `-:. 1638 + PBBN/SPB DC \ 1639 | + 1640 : ; 1641 `-+------+ +------+ +--+----+-' 1642 | ToR |.. | ToR |..| ToR | 1643 +-+--+-+ +-+--+-+ +-+--+--+ 1644 .'PBB `. .' `. .'VLAN`. 1645 +--+-+ +-+-++ +-++-+ +-+--+ 1646 |VSw | :VSw : :VSw : :VSw : 1647 +----+ +----+ +----+ +----+ 1648 Figure 9 DC with PBB and VLANs 1650 This example assumes that the two VSWs on the right do not 1651 support PBB but the ToRs do. The VSw on the left side are 1652 running PBB while the ones on the right side are still using 1653 VLANs. The left ToR is performing only Ethernet switching 1654 whereas the one on the right is translating from VLANs to 1655 ISIDs and performing PBB encapsulation using the BEB function 1656 [802.1ah] and [PBB-VPLS]. The ToR in the middle is performing 1657 both functions: core Ethernet tunneling for the PBB VSw and 1658 BEB function for the VLAN VSw. 1660 The SPB control plane is still used between the ToRs, 1661 providing the benefits described in the previous section. The 1662 VLAN VSw must use regular multi-homing functions to the ToRs: 1663 for example STP or Multi-chassis-LAG. 1665 DC VLANs may be also present initially on some of the legacy 1666 ToRs or Core SWs. PBB interoperability will be performed as 1667 follows: 1669 -If VLANs are used in the ToRs, PBB BEB function may be 1670 performed by the Core SW(s) where the ToR uplink is 1671 connected. 1673 -If VLANs are used in the Core SW, PBB BEB function may 1674 be performed by the PE GWs where the Core SW uplink is 1675 connected. 1677 It is possible that some DCs may run PBB or PBB-VLAN 1678 combination while others may still be running VLANs. An 1679 example of this interoperability scenario is described in 1680 Figure 10: 1682 ,-------------------------------. 1684 ( IP/MPLS WAN ) 1685 `------/-----------------\-------' 1686 +--/--+ +--\--+ 1687 |PE GW|PBB-VPLS |PE GW|VPLS 1688 .'+-----+-' .'+------+. 1689 / \ / \ 1690 | | | | 1691 | PBB DC | | VLAN DC | 1692 \ / \ / 1693 +---+ +---+ +---+ +---+ 1694 |VSw|.|VSw| |VSw|.|VSw| 1695 +---+ +---+ +---+ +---+ 1696 Figure 10 Interoperability to a VLAN-based DC 1698 Interoperability with existing VLAN DC is required for DC 1699 interconnect. The PE-GW in the PBB DC or the PE GW in the VLAN 1700 DC must implement PBB-VPLS PE model described in [PBB-VPLS]. 1701 This interoperability scenario is addressed in detail in [PBB- 1702 Interop]. 1704 Connectivity to existing VPN customer sites (IP VPN, VPLS, 1705 IPSec) or Internet does not require any additional procedures 1706 beyond the ones described in the VPN connectivity section. The 1707 PE GW in the DC VLAN will aggregate DC ELANs through IP 1708 interfaces assigned to VLAN logical endpoints whereas the PE 1709 GW in the PBB DC will assign IP interfaces to ISID logical 1710 endpoints. 1712 If EVPN is used to interconnect the two DCs, PBB-EVPN 1713 functions described in [PBB-EVPN] must be implemented in one 1714 of the PE-GWs. 1716 6.4. TRILL and L2VPN toolset 1718 TRILL and SPB control planes provide similar functions. IS-IS 1719 is the base protocol used in both specifications to provide 1720 multi-pathing and fast convergence for core networking. [PBB- 1721 EVPN] describes how seamless Inter-DC connectivity can be 1722 provided over an MPLS/IP network for both TRILL [RFC6325] and 1723 SPB [802.1aq]/[802.1Qbp] networks. 1725 The main differences exist in the encapsulation and data plane 1726 forwarding. TRILL encapsulation [RFC6325] was designed 1727 initially for large enterprise and campus networks where 4k 1728 VLANs are sufficient. As a consequence the ELAN space in 1729 [RFC6325] is limited to 4K VLANs; however, this VLAN scale 1730 issue is being addressed in [Fine-Grained]. 1732 7. L3VPN applicability to Cloud Networking 1734 This section discusses the role of IP VPN technology in 1735 addressing the L3 Virtualization challenges described in 1736 section 5. 1738 IP VPN technology defined in L3VPN working group may be used 1739 to provide L3 virtualization in support of multi-tenancy in 1740 the DC network as depicted in Figure 11. 1742 ,-------------------------------. 1743 ( IP VPNs over IP/MPLS WAN ) 1744 `----.'------------------------`.' 1745 ,--+-'. ;-`.--. 1746 ..... VRF1 )...... . VRF2 ) 1747 | '-----' | '-----' 1748 | Tenant1 |ELAN12 Tenant1| 1749 |ELAN11 ....|........ |ELAN13 1750 '':'''''''':' | | '':'''''''':' 1751 ,'. ,'. ,+. ,+. ,'. ,'. 1752 (VM )....(VM ) (VM )... (VM ) (VM )....(VM ) 1753 `-' `-' `-' `-' `-' `-' 1754 Figure 11 Logical Service View with IP VPN 1756 Tenant 1 might buy Cloud Services in different DC locations 1757 and choose to associate the VMs in 3 different groups, each 1758 mapped to a different ELAN: ELAN11, ELAN12 and ELAN13. L3 1759 interconnect between the ELANs belonging to tenant1 is 1760 provided using a BGP/MPLS IPVPN and associated VRF1 and VRF2, 1761 possibly located in different DCs. Each tenant that requires 1762 L3 virtualization will be allocated a different IP VPN 1763 instance. Using full fledge IP VPN for L3 Virtualization 1764 inside a DC presents the following advantages compared with 1765 existing DC technologies like Virtual Routing: 1767 - Interoperates with existing WAN VPN technology 1769 - Deployment tested, provides a full networking toolset 1771 - Scalable core routing: only one MP-BGP routing instance 1772 is required compared with one per customer/tenant in the 1773 Virtual Routing case 1775 - Service Auto-discovery: automatic discovery and route 1776 distribution between related service instances 1778 - Well defined and deployed Inter-Provider/Inter-AS models 1780 - Supports a variety of VRF-to-VRF tunneling options 1781 accommodating different operational models: MPLS 1782 [RFC4364], IP or GRE [RFC4797] 1784 To provide Cloud services to related customer IP VPN instances 1785 located in the WAN the following connectivity models may be 1786 employed: 1788 - DC IP VPN instance may participate directly in the WAN IP 1789 VPN 1791 - Inter-AS Options A, B or C models may be employed with 1792 applicability to both Intra and Inter-Provider use cases 1793 [RFC4364] 1795 VRF implementation could be done in the endsystem [endsystem] 1796 to facilitate endsystem to endsystem direct communication. 1798 Table 5 summarizes the comparison between BGP/MPLS IPVPN 1799 functional elements and those of NVo3 [NV03-fmwk]. 1801 Table 5: Functional comparison between BGP/MPLS IPVPN and NV03 1802 functional elements 1804 Nvo3 Function Matching BGP/MPLS-IPVPN Fun 1805 ------------------------------------------------------------- 1807 Virtual Access Point (VAP) Attachment Circuit (AC) 1809 Network Virtual Edge (NVE) Provider Edge (PE) 1811 Virtual Network Instance (VNI) Virtual Routing and 1812 Forwarding (VRF) 1814 Virtual Network Context (VN A 20-bit MPLS label 1815 Context) identifier 1817 Overlay Module and tunneling -MPLS over MPLS tunnels 1819 -MPLS over IP/GRE in an 1820 IP network 1822 Control Plane: TBD Control plane: 1824 - MP-BGP for VPN 1825 signaling /routing 1827 Core Routing: 1829 - IGP: OSPF/ISIS -(TE) 1831 Core Signaling: 1833 - RSVP or LDP for MPLS LSPs 1835 Depending on the implementation model, BGP/MPLS IPVPN can 1836 address some of the issues described in Section 5 and in 1837 [NVo3-problem-statement], but not all: 1839 -Dynamic Provisioning as described in [NVo3-problem- 1840 statement] Section 2.1: This is not addressed today in 1841 the BGP/MPLS IPVPN solution, as it was not a requirement 1842 for that solution. However, a mechanism may be developed 1843 to perform such provisioning dynamically as compute 1844 resources are configured. Considerations must be given to 1845 the cases where VMs and the VRF, providing connectivity 1846 to these VMs, are co-located on the same end-system vs. 1847 being on different physical devices. It should be noted 1848 that BGP/MPLS IPVPN currently supports auto-discovery of 1849 PEs with instances of the same IPVPN, as a component of 1850 the dynamic provisioning of and IPVPN service. 1852 -VM Mobility as also defined in [NVo3-problem-statement] 1853 section 2.2: VM mobility is supported in [endsystem] when 1854 the NVE, being a VRF, is co-located with the VM(s) to 1855 which the VRF provides connectivity. However, further 1856 enhancements must provide support for VM mobility in 1857 other cases. 1859 -IP table sizes in edge routers as also described in 1860 [NVo3-problem-statement] Section 2.3: As opposed to an 1861 802.1q based core Ethernet network, tenant VM addresses 1862 are only learned at a PBB edge. If the VsW implements PBB 1863 edge functionality and the ToR implements PBB-EVPN or 1864 PBB-VPLS, then the vsW will learn the MAC addresses of 1865 other VMs and devices in the same LAN, but the ToR will 1866 also learn the MAC addresses of Backbone bridges that 1867 will be on the order of number of servers not VMs, 1868 conserving MAC FDB entries on the ToR. This is because 1869 there are two layers of overlay, one at the VsW for PBB, 1870 and one at the ToR for VPLS or E-VPN, on a core IP/MPLS 1871 network. 1873 -VLAN limitation as also described in [NV03-proble- 1874 statement] Section 2.7: The number of service instances 1875 that can supported is 16 Millions. 1877 8. VM Mobility with E-VPN 1879 8.1. Layer 2 Extension Solution 1881 This document illustrates a solution for the layer 2 extension 1882 based on E-VPNs [E-VPN]. That is, the L2 sites that contain VMs 1883 of a given L2-based Community User Group (CUG) or Virtual Network 1884 (VN) are interconnected together using E-VPN. Thus, a given E-VPN 1885 corresponds/associated with a single L2-based VN. An L2-based VN 1886 is associated with a single E-VPN Ethernet Tag Identifier. 1888 This section provides a brief overview of how E-VPN is used as 1889 the solution for the "layer 2 extension problem". Details of E- 1890 VPN operations can be found in [E-VPN]. 1892 A single L2 site could be as large as the whole network within a 1893 single data center, in which case the Data Center Border Routers 1894 (DCBRs) of that data center, in addition to acting as IP routers 1895 for the L2-based VNs present in the data center, also act as PEs. 1896 In this scenario, E-VPN is used to handle VM migration between 1897 servers in different data centers. 1899 A single L2 site could be as small as a single ToR with the 1900 server connected to it, in which case the ToR acts as a PE. In 1901 this scenario, E-VPN is used to handle VM migration between 1902 servers that are either in the same or in different data centers. 1903 Note that in this scenario this document assumes that DCBRs, in 1904 addition to acting as IP routers for the L2-based VNs present in 1905 their data center, also participate in the E-VPN procedures, 1906 acting as BGP Route Reflectors for the E-VPN routes originated by 1907 the ToRs acting as PEs. 1909 In the case where E-VPN is used to interconnect L2 sites in 1910 different data centers, the network that interconnects DCBRs of 1911 these data centers could provide either (a) only Ethernet or 1912 IP/MPLS connectivity service among these DCBRs, or (b) may offer 1913 the E-VPN service. In the former case DCBRs exchange E-VPN routes 1914 among themselves relying only on the Ethernet or IP/MPLS 1915 connectivity service provided by the network that interconnects 1916 these DCBRs. The network does not directly participate in the 1917 exchange of these E-VPN routes. In the latter case the routers at 1918 the edge of the network maybe either co-located with DCBRs, or 1919 may establish E-VPN peering with DCBRs. Either way, in this case 1920 the network facilitates exchange of E-VPN routes among DCBRs (as 1921 in this case DCBRs would not need to exchange E-VPN routes 1922 directly with each other). 1924 Please note that for the purpose of solving the layer 2 extension 1925 problem the propagation scope of E-VPN routes for a given L2- 1926 based VN is constrained by the scope of the PEs connected to the 1927 L2 sites that presently contain VMs of that VN. This scope is 1928 controlled by the Route Target of the E-VPN routes. Controlling 1929 propagation scope could be further facilitated by using Route 1930 Target Constrain [RFC4684]. 1932 Use of E-VPN ensures that traffic among members of the same L2- 1933 based VN is optimally forwarded, irrespective of whether members 1934 of that VN are within the same or in different data centers. This 1935 follows from the observation that E-VPN inherently enables 1936 (disaggregated) forwarding at the granularity of the MAC address 1937 of the VM. 1939 Optimal forwarding among VMs of a given L2-based VN that are 1940 within the same data center requires propagating VM MAC 1941 addresses, and comes at the cost of disaggregated forwarding 1942 within a given data center. However, such disaggregated 1943 forwarding is not necessary between data centers if a given L2- 1944 based VN spans multiple data centers. For example, when a given 1945 ToR acts as a PE, this ToR has to maintain MAC advertisement 1946 routes only to the VMs within its own data center (and 1947 furthermore, only to the VMs that belong to the L2-based VNs 1948 whose site(s) are connected to that ToR), and then point a 1949 "default" MAC route to one of the DCBRs of that data center. In 1950 this scenario a DCBR of a given data center, when it receives MAC 1951 advertisement routes from DCBR(s) in other data centers, does not 1952 re-advertise these routes to the PEs within its own data center, 1953 but just advertises a single "default" MAC advertisement route to 1954 these PEs. 1956 When a given VM moves to a new L2 site, if in the new site this 1957 VM is the only VM from its L2-based VN, then the PEs connected to 1958 the new site need to be provisioned with the E-VPN Instances 1959 (EVI) of the E-VPN associated with this L2-based VN. Likewise, if 1960 after the move the old site no longer has any VMs that are in the 1961 same L2 bas the VM that moved, the PEs connected to the old site 1962 need be de-provisioned with the EVI of the E-VPN. Procedures to 1963 accomplish this are outside the scope of this document. 1965 8.2. VM Default Gateway Solutions 1967 Once a VM moves to a new L2 site, solving the VM Default Gateway 1968 problem would require PEs connected to that L2 site to apply IP 1969 forwarding to the inter-L2VN/inter-subnet traffic originated from 1970 that VM. That implies that (a) PEs should be capable of 1971 performing both MAC-based and IP-based forwarding (although IP- 1972 based forwarding functionality could be limited to just 1973 forwarding either based on IP host routes, or based on the IP 1974 default route), and (b) PEs should be able to distinguish between 1975 intra-L2VN/intra-subnet and inter-L2VN/inter-subnet traffic 1976 originated by that VM (in order to apply MAC-based forwarding to 1977 the former and IP-based forwarding to the latter). 1979 As a VM moves to a new L2 site, the default gateway IP address of 1980 the VM may not change. Further, the ARP cache of the VM may not 1981 time out. Thus, the destination MAC address in the inter- 1982 L2VN/inter-subnet traffic originated by that VM would not change 1983 as the VM moves to the new site. Given that, how would PEs 1984 connected to the new L2 site be able to recognize inter- 1985 L2VN/inter-subnet traffic originated by that VM? The following 1986 describes two possible solutions. 1988 Both of the solutions assume that for inter-L2VN/inter-subnet 1989 traffic between a VM and its peers outside of VM's own data 1990 center, one or more DCBRs of that data center act as fully 1991 functional default gateways for that traffic. 1993 8.2.1. VM Default Gateway Solution 1 1995 The first solution relies on the use of an anycast default 1996 gateway IP address and an anycast default gateway MAC address. 1998 If DCBRs act as PEs for an E-VPN corresponding to a given L2- 1999 based VN, then these anycast addresses are configured on these 2000 DCBRs. Likewise, if ToRs act as PEs, then these anycast addresses 2001 are configured on these ToRs. All VMs of that L2-based VN are 2002 (auto)configured with the (anycast) IP address of the default 2003 gateway. This ensures that a particular DCBR (or ToR), acting as 2004 a PE, can always apply IP forwarding to the packets sent by a VM 2005 to the (anycast) default gateway MAC address. It also ensures 2006 that such DCBR (or ToR) can respond to the ARP Request generated 2007 by a VM for the default gateway (anycast) IP address. 2009 Note that with this approach when originating E-VPN MAC 2010 advertisement routes for the MAC address of the default gateways 2011 of a given L2-based VN, all these routes MUST indicate that this 2012 MAC address belongs to the same Ethernet Segment Identifier 2013 (ESI). 2015 8.2.2. VM Default Gateway Solution 2 2017 The second solution does not require configuration of the anycast 2018 default gateway IP and MAC address on the PEs. 2020 Each DCBR (or each ToR) that acts as a default gateway for a 2021 given L2-based VN advertises in the E-VPN control plane its 2022 default gateway IP and MAC address using the MAC advertisement 2023 route, and indicates that such route is associated with the 2024 default gateway. The MAC advertisement route MUST be advertised 2025 as per procedures in [E-VPN]. The MAC address in such an 2026 advertisement MUST be set to the default gateway MAC address of 2027 the DCBR (or ToR). The IP address in such an advertisement MUST 2028 be set to the default gateway IP address of the DCBR (or ToR). To 2029 indicate that such a route is associated with a default gateway, 2030 the route MUST carry the "Default Gateway" community. 2032 Each PE that receives this route and imports it as per procedures 2033 of[E-VPN] MUST create MAC forwarding state that enables it to 2034 apply IP forwarding to the packets destined to the MAC address 2035 carried in the route. The MES that receives this E-VPN route 2036 follows procedures in Section 12 of [E-VPN] when replying to ARP 2037 Requests that it receives if such Requests are for the IP address 2038 in the received E-VPN route. 2040 9. Solutions and Considerations for other DC challenges 2042 9.1. Addressing IP/ARP explosion 2044 This section will be updated in the next revision. 2046 9.2. Optimal traffic forwarding 2048 IP networks, built using links-state protocols such as OSPF or 2049 ISIS and BGP provide optimal traffic forwarding through the 2050 use of equal cost multiple path (ECMP) and ECMP traffic load- 2051 balancing, and the use of traffic engineering tools based on 2052 BGP and/or MPLS-TE as applicable. In the Layer2 case, SPB or 2053 TRILL based protocols provide for load-balancing across 2054 parallel paths or equal cost paths between two nodes. Traffic 2055 follows the shortest path. For multicast, data plane 2056 replication at layer2 or layer3 happens in the data plane 2057 albeit with different attributes after multicast trees are 2058 built via a control plane and/or snooping. In the presence of 2059 VM mobility, optimal forwarding relates to avoiding 2060 triangulation and providing for optimum forwarding between any 2061 two VMs. Solutions that provide for routing in presence of VM 2062 mobility are described in [VM-Mobility]. 2064 9.3. VM Mobility 2066 IP VPN technology may be used to support DC Interconnect for 2067 different functions like VM Mobility and Cloud Management. A 2068 description of VM Mobility between server blades located in 2069 different IP subnets using extensions to existing MP-BGP and 2070 IP VPN procedure is described in [VM-Mobility]. Support for VM 2071 mobility is also described in [endystems]. Other solutions can 2072 exist as well. What is needed is a solution that provides for 2073 fast convergence toward the steady state whereby communication 2074 among any two VMs can take place on the shortest path or most 2075 optimum path, transit triangulation time is minimized, traffic 2076 black-holing is avoided, and impact on routing scale for both 2077 IPv4 and IPv6 is controllable or minimized. 2079 9.4. Dynamic provisioning of network services 2081 The need for fast dynamic provisioning of virtual network 2082 services is described in [NVo3-problem-statement] to match the 2083 elasticity and mobility in compute and storage resource 2084 allocation. Such dynamic provisioning was not part of initial 2085 L2VPN or L3VPN work except for some work to provide for 2086 dynamic bandwidth access [VPN-RSVP-TE]. In current L2VPN and 2087 L3VPN targeted deployments, the customer equipment connected 2088 to a VPN PE is static in location. Thus, the logical 2089 forwarding instance on the connected PE (e.g., IPVPN VRF, VSI 2090 or EVI) and the attachment circuit to that instance as well 2091 any routing and forwarding policies within that instance are 2092 provisioned via network management systems upon a service 2093 order at a much larger time scale than needed in this case. In 2094 dynamic data centers, services (e.g., VRF, attachment circuit) 2095 need to be established and torn down at a much smaller time 2096 scale to match the dynamicity of compute resources connected 2097 via these services. Mechanisms to provide for such timely 2098 dynamic provisioning at a scale are needed. 2100 In addition, CEs in traditional L3VPN deployments are routers 2101 able to exchange signaling and routing protocol information 2102 with the PE, providing for the dynamic exchange of routing 2103 information and liveliness check between the CEs and the PEs. 2104 In NVo3, the CE equivalent is a TS that may be a virtualized 2105 or a physical CE with the same capabilities as the traditional 2106 CE in L3VPN deployments. However, in some other cases, VMs 2107 providing compute rather than network services may connect 2108 directly to the NVE providing Lyaer3 forwarding service 2109 (equivalent to a PE). In that case, control plane mechanisms 2110 that enable fast and dynamic connectivity of a VM to an NVE 2111 and reachability exchange among NVEs providing connectivity 2112 among VMs in the same NV must be provided. 2114 9.5. Considerations for Layer2 and Layer3 VPNS on End-systems 2116 With the advent of computing power on end-systems providing VM 2117 services, and to provide for more efficient communication 2118 among VMs minimizing middle boxes in the path, there is a 2119 strong requirement for enabling Layer2 and Layer3 VN 2120 forwarding on these servers. Layer2 VN forwarding today is 2121 supported via vSW implementations and is often limited to 2122 intra data centers. Evolving proprietary technologies such as 2123 vxlan and provide for L2 service transport over an IP network. 2124 If Layer2 and Layer3 VN forwarding solutions on end-systems 2125 are to leverage existing L2VPN and L3VPN solutions, 2126 considerations should be given to new PE models and 2127 specifically to decoupling of forwarding from control plane 2128 functions across different systems to best utilize compute 2129 resources of end-systems and provide for scale. [endystems] is 2130 one of the solutions being adopted for implementation of 2131 BGP/MPLS VPNs in a DC end-system environment. In that case, 2132 the end-system uses XMPP to exchange labeled IP VPN routes 2133 with a route server that supports MP-BGP labeled IPVPN route 2134 exchange with tradition VPN Route Reflectors and PEs. [sdn- 2135 control] proposes a more generic model for PE functionality 2136 decomposed across forwarding end-systems and control plane 2137 systems that control the forwarding function on these end- 2138 systems and interact with other systems such as other similar 2139 control systems, PEs and Route reflectors. These efforts 2140 targeting new PE models that best fit a scalable multi-tenant 2141 environment may also require extensions of existing protocols 2142 or definition of new ones. 2144 10. Operator Considerations 2146 To be filled in a later version of this document. 2148 11. Security Considerations 2150 No new security issues are introduced beyond those described 2151 already in the related L2VPN and L3VPN solutions drafts and 2152 RFCs in relation to the VPN technologies themselves when the 2153 deployment model and PE model remain the same. Allowing for 2154 dynamic provisioning of VPN services within a DC must ensure 2155 that tenant network privacy is preserved. In addition, when 2156 provisioning, dynamically or statically, VPN services for a 2157 tenant across domain boundaries, the tenant privacy must be 2158 preserved. Dynamic provisioning must include communication of 2159 a secure channel and ensure that the service is provided to an 2160 authorized tenant and connected to the right tenant service. 2161 In addition, changing the PE model by separating the 2162 forwarding plane and control plane must consider and address 2163 security implications. 2165 12. IANA Considerations 2167 IANA does not need to take any action for this draft. 2169 13. References 2171 13.1. Normative References 2173 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2174 Requirement Levels", BCP 14, RFC 2119, March 1997. 2176 [RFC4761] Kompella, K. and Rekhter, Y. (Editors), "Virtual 2177 Private LAN Service (VPLS) Using BGP for Auto- 2178 Discovery and Signaling", RFC 4761, January 2007. 2180 [RFC4762] Lasserre, M. and Kompella, V. (Editors), "Virtual 2181 Private LAN Service (VPLS) Using Label Distribution 2182 Protocol (LDP) Signaling", RFC 4762, January 2007. 2184 [PBB-VPLS] Balus, F. et al. "Extensions to VPLS PE model for 2185 Provider Backbone Bridging", draft-ietf-l2vpn-pbb- 2186 vpls-pe-model-07.txt (work in progress), June 2013. 2188 [PBB-Interop] Sajassi, A. et al. "VPLS Interoperability with 2189 Provider Backbone Bridging", draft-ietf-l2vpn-pbb- 2190 vpls-interop-05.txt (work in progress), July 2013. 2192 [802.1ah] IEEE 802.1ah "Virtual Bridged Local Area Networks, 2193 Amendment 6: Provider Backbone Bridges", Approved 2194 Standard June 12th, 2008. 2196 [802.1aq] IEEE Draft P802.1aq/D4.3 "Virtual Bridged Local Area 2197 Networks, Amendment: Shortest Path Bridging", Work 2198 in Progress, September 21, 2011. 2200 [RFC6325] Perlman, et al., "Routing Bridges (Rbridges): Base 2201 Protocol Specification", RFC 6325, July 2011. 2203 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual 2204 Private Networks (VPNs)", RFC 4364, February 2006. 2206 [RFC4797] Rosen, E. and Y. Rekhter, "Use of Provider Edge to 2207 Provider Edge (PE-PE) Generic Routing encapsulation 2208 (GRE) or IP in BGP/MPLS IP Virtual Private 2209 Networks", RFC 4797, January 2007. 2211 13.2. Informative References 2213 [RFC4026] Andersson, L. et Al., "Provider Provisioned Virtual 2214 Private Network (VPN) Terminology", RFC 4026, May 2215 2005. 2217 [802.1Qbp] IEEE Draft P802.1Qbp/D0.1 "Virtual Bridged Local 2218 Area Networks, Amendment: Equal Cost Multiple Paths 2219 (ECMP)", Work in Progress, October 13, 2011. 2221 [802.1Qbg] IEEE Draft P802.1Qbg/D1.8 "Virtual Bridged Local 2222 Area Networks, Amendment: Edge Virtual Bridging", 2223 Work in Progress, October 17, 2011. 2225 [EVPN] Raggarwa, R. et al. "BGP MPLS based Ethernet VPN", 2226 draft-ietf-l2vpn-evpn-04.txt (work in 2227 progress), July 2013. 2229 [PBB-EVPN] Sajassi, A. et al. "PBB-EVPN", draft-ietf-l2vpn- 2230 pbb-evpn-05.txt (work in progress), July 2013. 2232 [VM-Mobility] Raggarwa, R. et al. "Data Center Mobility based 2233 on BGP/MPLS, IP Routing and NHRP", draft-raggarwa- 2234 data-center-mobility-05.txt (work in progress), June 2235 2013. 2237 [RFC4719] Aggarwal, R. et al., "Transport of Ethernet over 2238 Layer 2 Tunneling Protocol Version 3 (L2TPv3)", RFC 2239 4719, November 2006. 2241 [MVPN] Rosen, E. and Raggarwa, R. "Multicast in MPLS/BGP IP 2242 VPN", RFC 6513, February 2012. 2244 [ARPproxy] Carl-Mitchell, S. and Quarterman, S., "Using ARP to 2245 implement transparent subnet gateways", RFC 1027, 2246 October 1987. 2248 [MYERS] Myers, A., Ng, E. and Zhang, H., "Rethinking the 2249 Service Model: Scaling Ethernet to a Million Nodes", 2250 http://www.cs.cmu.edu/~acm/papers/myers- 2251 hotnetsIII.pdf. 2253 [Fine-Grained] Eastlake, D. et Al., "RBridges: Fine-Grained 2254 Labeling", draft-eastlake-trill-rbridge-fine- 2255 labeling-02.txt (work in progress), October 2011. 2257 [Nvo3-problem-statement] Narten, T., et al., "Problem 2258 Statement: Overlays for Network Virtualization", 2259 draft-ietf-nvo3-overlay-problem-statement-04.txt 2260 (work in progress), July 2013. 2262 [Nvo3-fmwk] Lasserre, M., et al., "Framework for DC Network 2263 Virtualization", draft-ietf-nvo3-framework-03.txt 2264 (work in progress), July 2013. 2266 [Nvo3-cp-reqts] Kreeger, L., et al., "Network Virtualization 2267 Overlay Control Protocol Requirements", draft- 2268 kreeger-nvo3-overlay-cp-04.txt (work in progress), 2269 June 2013. 2271 [Nvo3-dp-reqts] Bitar, N., Lasserre, M., et al., "NVO3 Data 2272 Plane Requirements", draft-ietf-nvo3-dataplane- 2273 requirements-01.txt (work in progress), July 2274 2013. 2276 [endsystem] Marques, P., wt al., "BGP-signaled end-system 2277 IP/VPNs", draft-ietf-l3vpn-end-system-01.txt (work in 2278 progress), April 2013. 2280 14. Acknowledgments 2282 In addition to the authors the following people have 2283 contributed to this document: 2285 Javier Benitez, Colt 2287 Dimitrios Stiliadis, Alcatel-Lucent 2289 Samer Salam, Cisco 2291 Yakov Rekhter, Juniper 2293 Authors' Addresses 2295 Nabil Bitar 2296 Verizon 2297 40 Sylvan Road 2298 Waltham, MA 02145 2299 Email: nabil.bitar@verizon.com 2301 Florin Balus 2302 Alcatel-Lucent 2303 777 E. Middlefield Road 2304 Mountain View, CA, USA 94043 2305 Email: florin.balus@alcatel-lucent.com 2307 Marc Lasserre 2308 Alcatel-Lucent 2309 Email: marc.lasserre@alcatel-lucent.com 2310 Wim Henderickx 2311 Alcatel-Lucent 2312 Email: wim.henderickx@alcatel-lucent.com 2314 Ali Sajassi 2315 Cisco 2316 170 West Tasman Drive 2317 San Jose, CA 95134, USA 2318 Email: sajassi@cisco.com 2320 Luyuan Fang 2321 Cisco 2322 111 Wood Avenue South 2323 Iselin, NJ 08830 2324 Email: lufang@cisco.com 2326 Yuichi Ikejiri 2327 NTT Communications 2328 1-1-6, Uchisaiwai-cho, Chiyoda-ku 2329 Tokyo, 100-8019 Japan 2330 Email: y.ikejiri@ntt.com 2332 Mircea Pisica 2333 BT 2334 Telecomlaan 9 2335 Brussels 1831, Belgium 2336 Email: mircea.pisica@bt.com 2338 John E. Drake 2339 Juniper Networks 2340 Email: jnadeau@juniper.net 2342 Lucy Yong 2343 Huawei Technologies (USA) 2344 5340 Legacy Drive 2345 Plano, TX75025 2346 Email: lucy.yong@huawei.com 2348 Susan Hares 2349 ADARA 2350 Email: shares@ndzh.com