idnits 2.17.1 draft-ietf-bess-virtual-pe-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 11, 2014) is 3452 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Obsolete informational reference (is this intentional?): RFC 7159 (Obsoleted by RFC 8259) Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 INTERNET-DRAFT Luyuan Fang, Ed. 3 Intended Status: Informational Microsoft 4 Expires: May 12, 2015 Rex Fernando 5 Cisco 6 Maria Napierala 7 AT&T 8 Nabil Bitar 9 Verizon 10 Bruno Rijsman 11 Juniper 13 November 11, 2014 15 BGP/MPLS VPN Virtual PE 16 draft-ietf-bess-virtual-pe-00 18 Abstract 20 This document describes the architecture solutions for BGP/MPLS L3 21 and L2 Virtual Private Networks (VPNs) with virtual Provider Edge 22 (vPE) routers. It provides a functional description of the vPE 23 control, forwarding, and management. The proposed vPE solutions 24 support both the Software Defined Networks (SDN) approach which 25 allows physical decoupling of the control and the forwarding, and the 26 traditional distributed routing approach. A vPE can reside in any 27 network or compute devices, such as a server as co-resident with the 28 application virtual machines (VMs), or a Top-of-Rack (ToR) switch in 29 a Data Center (DC) network. 31 Status of this Memo 33 This Internet-Draft is submitted to IETF in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF), its areas, and its working groups. Note that 38 other groups may also distribute working documents as 39 Internet-Drafts. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 The list of current Internet-Drafts can be accessed at 47 http://www.ietf.org/1id-abstracts.html 49 The list of Internet-Draft Shadow Directories can be accessed at 50 http://www.ietf.org/shadow.html 52 Copyright and License Notice 54 Copyright (c) 2014 IETF Trust and the persons identified as the 55 document authors. All rights reserved. 57 This document is subject to BCP 78 and the IETF Trust's Legal 58 Provisions Relating to IETF Documents 59 (http://trustee.ietf.org/license-info) in effect on the date of 60 publication of this document. Please review these documents 61 carefully, as they describe your rights and restrictions with respect 62 to this document. Code Components extracted from this document must 63 include Simplified BSD License text as described in Section 4.e of 64 the Trust Legal Provisions and are provided without warranty as 65 described in the Simplified BSD License. 67 Table of Contents 69 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 70 1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 4 71 1.2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . 5 72 2. Virtual PE Architecture . . . . . . . . . . . . . . . . . . . . 6 73 2.1 Virtual PE definitions . . . . . . . . . . . . . . . . . . . 6 74 2.2 vPE Architecture and Design options . . . . . . . . . . . . 8 75 2.2.1 vPE-F host location . . . . . . . . . . . . . . . . . . 8 76 2.2.2 vPE control plane topology . . . . . . . . . . . . . . . 8 77 2.2.3 Data Center orchestration models . . . . . . . . . . . . 8 78 2.3 vPE Architecture reference models . . . . . . . . . . . . . 8 79 2.3.1 vPE-F in an end-device and vPE-C in the controller . . . 8 80 2.3.2 vPE-F and vPE-C on the same end-device . . . . . . . . . 10 81 2.3.3 vPE-F and vPE-C are on the ToR . . . . . . . . . . . . . 11 82 2.3.4 vPE-F on the ToR and vPE-C on the controller . . . . . . 12 83 2.3.5 The server view of a vPE . . . . . . . . . . . . . . . . 12 84 3. Control Plane . . . . . . . . . . . . . . . . . . . . . . . . . 13 85 3.1 vPE Control Plane (vPE-C) . . . . . . . . . . . . . . . . . 13 86 3.1.1 The SDN approach . . . . . . . . . . . . . . . . . . . . 13 87 3.1.2 Distributed control plane . . . . . . . . . . . . . . . 14 88 3.3 Use of router reflector . . . . . . . . . . . . . . . . . . 14 89 3.4 Use of Constrained Route Distribution [RFC4684] . . . . . . 14 90 4. Forwarding Plane . . . . . . . . . . . . . . . . . . . . . . . 14 91 4.1 Virtual Interface . . . . . . . . . . . . . . . . . . . . . 14 92 4.2 Virtual Provider Edge Forwarder (vPE-F) . . . . . . . . . . 15 93 4.3 Encapsulation . . . . . . . . . . . . . . . . . . . . . . . 15 94 4.4 Optimal forwarding . . . . . . . . . . . . . . . . . . . . . 15 95 4.5 Routing and Bridging Services . . . . . . . . . . . . . . . 16 96 5. Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . 17 97 5.1 IPv4 and IPv6 support . . . . . . . . . . . . . . . . . . . 17 98 5.2 Address space separation . . . . . . . . . . . . . . . . . . 17 99 6.0 Inter-connection considerations . . . . . . . . . . . . . . 17 100 7. Management, Control, and Orchestration . . . . . . . . . . . . 18 101 7.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . 18 102 7.2 Management/Orchestration system interfaces . . . . . . . . . 19 103 7.3 Service VM Management . . . . . . . . . . . . . . . . . . . 19 104 7.4 Orchestration and MPLS VPN inter-provisioning . . . . . . . 19 105 7.4.1 vPE Push model . . . . . . . . . . . . . . . . . . . . . 20 106 7.4.2 vPE Pull model . . . . . . . . . . . . . . . . . . . . . 21 107 8. Security Considerations . . . . . . . . . . . . . . . . . . . 21 108 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 109 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 110 10.1 Normative References . . . . . . . . . . . . . . . . . . . 22 111 10.2 Informative References . . . . . . . . . . . . . . . . . . 23 112 11. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 24 113 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 25 115 1 Introduction 117 Network virtualization enables multiple isolated individual networks 118 over a shared common network infrastructure. BGP/MPLS IP Virtual 119 Private Networks (IP VPNs) [RFC4364] have been widely deployed to 120 provide network based Layer 3 VPNs solutions. [RFC4364] provides 121 routing isolation among different customer VPNs and allow address 122 overlap among these VPNs through the implementation of per VPN 123 Virtual Routing and Forwarding instances (VRFs) at a Service Provider 124 Edge (PE) routers, while forwarding customer traffic over a common 125 IP/MPLS network. For L2 VPN, a similar technology is being defined in 126 [I-D.ietf-l2vpn-evpn] on the basis of BGP/MPLS, to provide switching 127 isolation and allow MAC address overlap. 129 With the advent of compute capabilities and the proliferation of 130 virtualization in Data Center servers, multi-tenant Data Centers are 131 becoming the norm. As applications and appliances are increasingly 132 being virtualized, support for virtual edge devices, such as virtual 133 L3/L2 VPN PE routers, becomes feasible and desirable for Service 134 Providers who want to extend their existing MPLS VPN deployments into 135 Data Centers to provide end-to-end Virtual Private Cloud (VPC) 136 services. Virtual PE work is also one of early effort for Network 137 Functions Virtualization (NFV). In general, scalability, agility, and 138 cost efficiency are primary motivations for vPE solutions. 140 The virtual Provider Edge (vPE) solution described in this document 141 allows for the extension of the PE functionality of L3/L2 VPN to an 142 end device, such as a server where the applications reside, or to a 143 first hop routing/switching device, such as a Top of the Rack (ToR) 144 switch in a DC. 146 The vPE solutions support both the Software Defined Networks (SDN) 147 approach, which allows physical decoupling of the control and the 148 forwarding, and the traditional distributed routing approach. 150 1.1 Terminology 152 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 153 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 154 document are to be interpreted as described in RFC 2119 [RFC2119]. 156 Term Definition 157 ----------- -------------------------------------------------- 158 ASBR Autonomous System Border Router 159 BGP Border Gateway Protocol 160 CE Customer Edge 161 Forwarder IP VPN forwarding function 162 GRE Generic Routing Encapsulation 163 Hypervisor Virtual Machine Manager 164 I2RS Interface to Routing Systems 165 LDP Label Distribution Protocol 166 MP-BGP Multi-Protocol Border Gateway Protocol 167 MPLS Multi-Protocol Label Switching 168 PCEF Policy Charging and Enforcement Function 169 QoS Quality of Service 170 RR Route Reflector 171 RT Route Target 172 RTC RT Constraint 173 SDN Software Defined Networks 174 ToR Top-of-Rack switch 175 VI Virtual Interface 176 vCE virtual Customer Edge Router 177 VM Virtual Machine 178 vPC virtual Private Cloud 179 vPE virtual Provider Edge Router 180 vPE-C virtual Provider Edge Control plane 181 vPE-F virtual Provider Edge Forwarder 182 VPN Virtual Private Network 183 vRR virtual Route Reflector 184 WAN Wide Area Network 186 End device: where Guest OS, Host OS/Hypervisor, applications, VMs, 187 and virtual router may reside. 189 1.2 Requirements 191 The following are key requirements for vPE solutions. 193 1) MUST support end device multi-tenancy, per tenant routing 194 isolation and traffic separation. 196 2) MUST support large scale MPLS VPNs in the Data Center, upto tens 197 of thousands of end devices and millions of VMs in the single Data 198 Center. 200 3) MUST support end-to-end MPLS VPN connectivity, e.g. MPLS VPN can 201 start from a DC end device, connect to a corresponding MPLS VPN in 202 the WAN, and terminate in another Data Center end device. 204 4) MUST allow physical decoupling of MPLS VPN PE control and 205 forwarding for network virtualization and abstraction. 207 5) MUST support the control plane with both SDN controller approach, 208 and the traditional distributed control plane approach with MP-BGP 209 protocol. 211 6) MUST support VM mobility. 213 7) MUST support orchestration/auto-provisioning deployment model. 215 8) SHOULD be capable to support service chaining as part of the 216 solution [I-D.rfernando-l3vpn-service-chaining], 217 [I-D.bitar-i2rs-service-chaining]. 219 The architecture and protocols defined in BGP/MPLS IP VPN [RFC4364] 220 and BGP/MPLS EVPN [I-D.ietf-l2vpn-evpn] provide the foundation for 221 vPE extension. Certain protocol extensions may be needed to support 222 the virtual PE solutions. 224 2. Virtual PE Architecture 226 2.1 Virtual PE definitions 228 As defined in [RFC4364] and [I-D.ietf-l2vpn-evpn], an MPLS VPN is 229 created by applying policies to form a subset of sites among all 230 sites connected to the backbone networks. It is a collection of 231 "sites". A site can be considered as a set of IP/ETH systems 232 maintaining IP/ETH inter-connectivity without direct connecting 233 through the backbone. The typical use of L3/L2 VPN has been to 234 inter-connect different sites of an Enterprise networks through a 235 Service Provider's BGP MPLS VPNs in the WAN. 237 A virtual PE (vPE) is a BGP/MPLS L3/L2 VPN PE software instance which 238 may reside in any network or computing devices. The control and 239 forwarding components of the vPE can be decoupled, they may reside in 240 the same physical device, or in different physical devices. 242 A virtualized Provider Edge Forwarder (vPE-F) is the forwarding 243 element of a vPE. vPE-F can reside in an end device, such as a server 244 in a Data Center where multiple application Virtual Machines (VMs) 245 are supported, or a Top-of-Rack switch (ToR) which is the first hop 246 switch from the Data Center edge. When a vPE-F is residing in a 247 server, its connection to a co-resident VM can be viewed as similar 248 to the PE- CE relationship in the regular BGP L3/L2 VPNs, but without 249 routing protocols or static routing between the virtual PE and end- 250 host because the connection is internal to the device. 252 The vPE Control plane (vPE-C) is the control element of a vPE. When 253 using the approach where control plane is decoupled from the physical 254 topology, the vPE-F may be in a server and co-resident with 255 application VMs, while one vPE-C can be in a separate device, such as 256 an SDN Controller where control plane elements and orchestration 257 functions are located. Alternatively, the vPE-C can reside in the 258 same physical device as the vPE-F. In this case, it is similar to the 259 traditional implementation of VPN PEs where, distributed MP-BGP is 260 used for L3/L2 VPN information exchange, though the vPE is not a 261 dedicated physical entity as it is in a physical PE implementation. 263 2.2 vPE Architecture and Design options 265 2.2.1 vPE-F host location 267 Option 1a. vPE-F is on an end device as co-resident with application 268 VMs. For example, the vPE-F is on a server in a Data Center. 270 Option 1b. vPE-F forwarder is on a ToR or other first hop devices in 271 a DC, not as co-resident with the application VMs. 273 Option 1c. vPE-F is on any network or compute devices in any types of 274 networks. 276 2.2.2 vPE control plane topology 278 Option 2a. vPE control plane is physically decoupled from the vPE-F. 279 The control plane may be located in a controller in a separate device 280 (a stand alone device or can be in the gateway as well) from the vPE 281 forwarding plane. 283 Option 2b. vPE control plane is supported through dynamic routing 284 protocols and located in the same physical device as the vPE-F. 286 2.2.3 Data Center orchestration models 288 Option 3a. Push model: It is a top down approach, push IP VPN 289 provisioning state from a network management system or other 290 centrally controlled provisioning system to the IP VPN network 291 elements. 293 Option 3b. Pull model: It is a bottom-up approach, pull state 294 information from network elements to network management/AAA based 295 upon data plane or control plane activity. 297 2.3 vPE Architecture reference models 299 2.3.1 vPE-F in an end-device and vPE-C in the controller 301 Figure 1 illustrates the reference model for a vPE solution with the 302 vPE-F in the end device co-resident with applications VMs, while the 303 vPE-C is physically decoupled and residing on a controller. 305 The Data Center is connected to the IP/MPLS core via the 306 Gatways/ASBRs. The MPLS VPN , e.g. VPN RED, has a single termination 307 point within the DC at one of the VPE-F, and is inter-connected in 308 the WAN to other member sites which belong to the same client, and 309 the remote ends of VPN RED can be a PE which has VPN RED attached to 310 it, or another vPE in a different Data Center. 312 Note that the DC fabrics/intermediate underlay devices in the DC do 313 not participate IP VPNs, their function is the same as provider 314 backbone routers in the IP/MPLS back bone and they do not maintain 315 the VPN states, nor they are VPN aware. 317 ,-----. 318 ( ') 319 .--(. '.---. 320 ( ' ' ) 321 ( IP/MPLS WAN ) 322 (. .) 323 ( ( .) 324 WAN ''--' '-''---' 325 ----------------|----------|------------------------ 326 Service/DC | | 327 Network +-------+ +-------+ 328 |Gateway|---|Gateway| * 329 | /ASBR | | /ASBR | * 330 +-------+ +-------+ * 331 | | +-------------+ 332 | ,---. | |Controller | 333 .---. ( '.---. |(vPE-C and | 334 ( ' ' ') |orchestrator)| 335 ( Data Center ) +-------------+ 336 (. Fabric ) * 337 ( ( ).--' * 338 / ''--' '-''--' \ * 339 / / \ \ * 340 +-------+ +-------+ +-------+ +-------+ 341 | vPE-F | | vPE-F | | vPE-F | | vPE-F | 342 +---+---+ +---+---+ +---+---+ +---+---+ 343 |VM |VM | |VM |VM | |VM |VM | |VM |VM | 344 +---+---+ +---+---+ +---+---+ +---+---+ 345 |VM |VM | |VM |VM | |VM |VM | |VM |VM | 346 +---+---+ +---+---+ +---+---+ +---+---+ 348 End Device End Device End Device End Device 350 Figure 1. Virtualized Data Center with vPE at 351 the end device and vPE-C and vPE-F physically decoupled 353 Note: 355 a) *** represents Controller logical connections to the all 356 Gateway/ASBRs and to all vPE-F. 358 b) ToR is assumed included in the Data Center cloud. 360 2.3.2 vPE-F and vPE-C on the same end-device 362 In this option, vPE-F and vPE-C functionality are both resident in 363 the end-device. The vPE functions the same as it is in a physical PE. 364 MP-BGP is used for the VPN control plane. Virtual or physical Route 365 Reflectors (RR) (not shown in the diagram) can be used to assist 366 scaling. 368 ,-----. 369 ( ') 370 .--(. '.---. 371 ( ' ' ) 372 ( IP/MPLS WAN ) 373 (. .) 374 ( ( .) 375 WAN ''--' '-''---' 376 ----------------|----------|---------------------- 377 Service/DC | | 378 Network +-------+ +-------+ 379 |Gateway|---|Gateway| 380 | /ASBR | | /ASBR | * 381 +-------+ +-------+ * 382 | | * MP-BGP 383 | ,---. | * 384 .---. ( '.---. * 385 ( ' ' ') * 386 ( Data Center ) * 387 (. Fabric ) * 388 ( ( ).--' * 389 / ''--' '-''--' \ * 390 / / \ \ * 391 +-------+ +-------+ +-------+ +-------+ 392 | vPE | | vPE | | vPE | | vPE | 393 +---+---+ +---+---+ +---+---+ +---+---+ 394 |VM |VM | |VM |VM | |VM |VM | |VM |VM | 395 +---+---+ +---+---+ +---+---+ +---+---+ 396 |VM |VM | |VM |VM | |VM |VM | |VM |VM | 397 +---+---+ +---+---+ +---+---+ +---+---+ 399 End Device End Device End Device End Device 401 Figure 2. Virtualized Data Center with vPE at 402 the end device, VPN control signal uses MP-BGP 404 Note: 406 a) *** represents the logical connections using MP-BGP among the 407 Gateway/ASBRs and to the vPEs on the end devices. 409 b) ToR is assumed included in the Data Center cloud. 411 2.3.3 vPE-F and vPE-C are on the ToR 413 In this option, vPE functionality is the same as a physical PE. MP- 414 BGP is used for the VPN control plane. Virtual or physical Route 415 Reflector (RR) (not shown in the diagram) can be used to assist 416 scaling. 418 ,-----. 419 ( ') 420 .--(. '.---. 421 ( ' ' ) 422 ( IP/MPLS WAN ) 423 (. .) 424 ( ( .) 425 WAN ''--' '-''---' 426 ----------------|----------|---------------------- 427 Service/DC | | 428 Network +-------+ +-------+ 429 |Gateway|---|Gateway| 430 | /ASBR | | /ASBR | * 431 +-------+ +-------+ * 432 | | * MP-BGP 433 | ,---. | * 434 .---. ( '.---. * 435 ( ' ' ') * 436 ( Data Center ) * 437 (. Fabric ) * 438 ( ( ).--' * 439 /''--' '-/'--' \ * 440 +---+---+ +---+---+ +---+---+ 441 |vPE| | |vPE| | |vPE| | 442 +---+ | +---+ | +---+ | 443 | ToR | | ToR | | ToR | 444 +-------+ +-------+ +-------+ 445 / \ / \ / \ 446 +-------+ +-------+ +-------+ +-------+ 447 | vPE | | vPE | | vPE | | vPE | 448 +---+---+ +---+---+ +---+---+ +---+---+ 449 |VM |VM | |VM |VM | |VM |VM | |VM |VM | 450 +---+---+ +---+---+ +---+---+ +---+---+ 451 |VM |VM | |VM |VM | |VM |VM | |VM |VM | 452 +---+---+ +---+---+ +---+---+ +---+---+ 453 End Device End Device End Device End Device 455 Figure 3. Virtualized Data Center with vPE at 456 the ToP, VPN control signal uses MP-BGP 458 Note: *** represents the logical connections using MP-BGP among the 459 Gateway/ASBRs and to the vPEs on the ToRs. 461 2.3.4 vPE-F on the ToR and vPE-C on the controller 463 In this option, the L3/L2 VPN termination is at the ToR, but the 464 control plane decoupled from the data plane and resided in a 465 controller, which can be on a stand alone device, or can be placed at 466 the Gateway/ASBR. 468 2.3.5 The server view of a vPE 470 An end device shown in Figure 4 is a virtualized server that hosts 471 multiple VMs. The virtual PE is co-resident in the server with 472 application VMs. The vPE supports multiple VRFs, VRF Red, VRF Grn, 473 VRF Yel, VRF Blu, etc. Each application VM is associated to a 474 particular VRF as a member of the particular VPN. For example, VM1 is 475 associated to VRF Red, VM2 and VM47 are associated to VRF Grn, etc. 476 Routing/switching isolation applies between VPNs for multi-tenancy 477 support. For example, VM1 and VM2 cannot communicate directly in a 478 simple intranet VPN topology as shown in the configuration. 480 The vPE connectivity relationship between vPE and the application VM 481 is similar to the PE-to-CE relationship in regular BGP VPNs. However, 482 as the vPE and end-host functions are co-resident in the same server, 483 the connection between them is an internal implementation of the 484 server. 486 +----------------------------------------------------+ 487 | +---------+ +---------+ +---------+ +---------+ | 488 | | VM1 | | VM2 | | VM47 | | VM48 | | 489 | |(VPN Red)| |(VPN Grn)|... |(VPN Grn)| |(VPN Blu)| | 490 | +----+----+ +---+-----+ +----+----+ +----+----+ | 491 | | | | | | 492 | +---+ | +-------------+ +---+ | 493 | | | | | | 494 to | +---+------+-+---------------------+---+ | 495 Gateway| | | | | | | | 496 PE | | +-+-+ ++-++ +---+ +-+-+ | | 497 | | |VRF| |VRF| ....... |VRF| |VRF| | | 498 <------+------+ |Red| |Grn| |Yel| |Blu| | | 499 | | +---+ +---+ +---+ +---+ | | 500 | | L3 VPN virtual PE | | 501 | +--------------------------------------+ | 502 | | 503 | End Device | 504 +----------------------------------------------------+ 505 Figure 4. Server View of vPE to VM relationship 507 An application VM may send packets to a vPE forwarder that need to be 508 bridged, either locally to another VM, or to a remote destination. In 509 this case, the vPE contains a virtual bridge instance to which the 510 application VMs (CEs) are attached. 512 +----------------------------------------------------+ 513 | +---------+ +---------+ +---------+ | 514 | | VM1 | | VM2 | | VM47 | | 515 | |(VPN Red)| |(VPN Grn)|...|(VPN Grn)| | 516 | +----+----+ +----+----+ +----+----+ | 517 | | | | | 518 | +---+ +----+ +----+ | 519 | | | | | 520 to | +---+------------+---+-----------------+ | 521 Gateway| | | | | | | 522 PE | | +-+--+--+ +-+---+-+ | | 523 | | |VBridge| |VBridge| ....... | | 524 <------+------+ |Red | |Grn | | | 525 | | +-------+ +-------+ | | 526 | | vPE | | 527 | +--------------------------------------+ | 528 | | 529 | End Device | 530 +----------------------------------------------------+ 532 Figure 4. Bridging Service at vPE 534 3. Control Plane 536 3.1 vPE Control Plane (vPE-C) 538 3.1.1 The SDN approach 540 This approach is appropriate when the vPE control and data planes are 541 physically decoupled. The control plane directing the data flow may 542 reside elsewhere, e.g. in a SDN controller. This approach requires a 543 standard interface to the routing system. The Interface to Routing 544 System (I2RS) is work in progress in IETF as described in 545 [I-D.ietf-i2rs-architecture], [I-D.ietf-i2rs-problem-statement]. 547 Although MP-BGP is often the de facto preferred choice between vPE 548 and gateway-PE/ASBR, the use of extensible signaling messaging 549 protocols MAY often be more practical in a Data Center environment. 550 One such proposal that uses this approach is detailed in 551 [I-D.ietf-l3vpn-end-system]. 553 3.1.2 Distributed control plane 555 In the distributed control plane approach, the vPE participates in 556 the overlay L3/L2 VPN control protocol: MP-BGP [RFC4364]. 558 When the vPE function is on a ToR, it participates the underlay 559 routing through IGP protocols (ISIS or OSPF) or BGP. 561 When the vPE function is on a server, it functions as a host attached 562 to a server. 564 3.3 Use of router reflector 566 Modern Data Centers can be very large in scale. For example, the 567 number of VPNs routes in a very large DC can surpass the scale of 568 those in a Service Provider backbone VPN networks. There may be tens 569 of thousands of end devices in a single DC. 571 Use of Router Reflector (RR) is necessary in large-scale IP VPN 572 networks to avoid a full iBGP mesh among all vPEs and PEs. The VPN 573 routes can be partitioned to a set of RRs, the partitioning 574 techniques are detailed in [RFC4364] and [I-D.ietf-l2vpn-evpn]. 576 When a RR software instance is residing in a physical device, e.g., a 577 server, which is partitioned to support multi-functions and 578 application VMs, the RR becomes a virtualized RR (vRR). Since RR 579 performs control functions only, a dedicated or virtualized server 580 with large scale of computing power and memory can be a good 581 candidate as host of vRRs. The vRR can also reside in a Gateway 582 PE/ASBR, or in an end device. 584 3.4 Use of Constrained Route Distribution [RFC4684] 586 The Constrained Route Distribution [RFC4684] is a powerful tool for 587 selective VPN route distribution. With RTC, only the BGP receivers 588 (e.g, PE/vPE/RR/vRR/ASBRs, etc.) with the particular IP VPNs attached 589 will receive the route update for the corresponding VPNs. It is 590 critical to use constrained route distribution to support large-scale 591 IP VPN developments. 593 4. Forwarding Plane 595 4.1 Virtual Interface 597 A Virtual Interface (VI) is an interface within an end device that is 598 used for connection of the vPE to the application VMs in the same end 599 device. Such application VMs are treated as CEs in the regular VPN's 600 view. 602 4.2 Virtual Provider Edge Forwarder (vPE-F) 604 The Virtual Provider Edge Forwarder (vPE-F) is the forwarding 605 component of a vPE where the tenant identifiers (for example, MPLS 606 VPN labels) are pushed/popped. 608 The vPE-F location options include: 610 1) Within the end device where the virtual interface and application 611 VMs are located. 613 2) In an external device such as a Top of the Rack switch (ToR) in a 614 DC into which the end device connects. 616 Multiple factors should be considered for the location of the vPE-F, 617 including device capabilities, overall solution economics, 618 QoS/firewall/NAT placement, optimal forwarding, latency and 619 performance, operational impact, etc. There are design tradeoffs, it 620 is worth the effort to study the traffic pattern and forwarding 621 looking trend in your own unique Data Center as part of the exercise. 623 4.3 Encapsulation 625 BGP/MPLS VPNs can be tunneled through the network as overlays using 626 MPLS-based or IP-based encapsulation. 628 In the case of MPLS-based encapsulation, most existing core 629 deployments use distributed protocols such as Label Distribution 630 Protocol (LDP), [RFC3032][RFC5036], or RSVP-TE [RFC3209]. 632 Due to its maturity, scalability, and header efficiency, MPLS Label 633 Stacking is gaining traction by service providers, and large-scale 634 cloud providers in particular, as the unified forwarding mechanism of 635 choice. 637 With the emergence of the SDN paradigm, label distribution may be 638 achieved through SDN controllers, or via a combination of centralized 639 control and distributed protocols. 641 In the case of IP-based encapsulation, MPLS VPN packets are 642 encapsulated in IP or Generic Routing Encapsulation (GRE), [RFC4023], 643 [RFC4797]. IP-based encapsulation has not been extensively deployed 644 for BGP/MPLS VPN in the core; however it is considered as one of the 645 tunneling options for carrying MPLS VPN overlays in the data center. 646 Note that when IP encapsulation is used, the associated security 647 properties must be analyzed carefully. 649 4.4 Optimal forwarding 650 Many large cloud service providers have reported the DC traffic is 651 now dominated by East-West across subnet traffic (between the end 652 device hosting different applications in different subnets) rather 653 than North-South traffic (going in/out of the Data Center and to/from 654 the WAN) or switched traffic within subnets. This is the primary 655 reason that newer DC design has moved away from traditional Layer-2 656 design to Layer-3, especially for the overlay networks. 658 When forwarding the traffic within the same VPN, the vPE SHOULD be 659 capable to provide direct communication among the VMs/application 660 senders/receivers without the need of going through Gateway devices. 661 If the senders and the receivers are on the same end device, the 662 traffic SHOULD NOT need to leave the device. If they are on different 663 end devices, optimal routing SHOULD be applied. 665 Extranet MPLS VPN techniques can be used for multiple VPNs access 666 without the need of Gateway facilitation. This is done through the 667 use of VPN policy control mechanisms. 669 In addition, ECMP is a built in IP mechanism for load sharing. 670 Optimal use of available bandwidth can be achieved by virtue of using 671 ECMP in the underlay, as long as the encapsulation includes certain 672 entropy in the header, VXLAN is such an example. 674 4.5 Routing and Bridging Services 676 A VPN forwarder (vPE-F) may support both IP forwarding as well as 677 Layer 2 bridging for traffic from attached end hosts. This traffic 678 may be between end hosts attached to the same VPN forwarder or to 679 different VPN forwarders. 681 In both cases, forwarding at a VPN forwarder takes place based on the 682 IP or MAC entries provisioned by the vPE controller. 684 When the vPE is providing Layer 3 service to the attached CEs, the 685 VPN forwarder has a VPN VRF instance with IP routes installed for 686 both locally attached end-hosts and ones reachable via other VPN 687 forwarders. The vPE may perform IP routing for all IP packets in this 688 mode. 690 When the vPE provides Layer 2 service to the attached end-hosts, the 691 VPN forwarder has an E-VPN instance with appropriate MAC entries. 693 The vPE may support an Integrated Routing and Bridging service, in 694 which case the relevant VPN forwarders will have both MAC and IP 695 table entries installed, and will appropriately route or switch 696 incoming packets. 698 The vPE controller performs the necessary provisioning functions to 699 support various services, as defined by an user. 701 5. Addressing 703 5.1 IPv4 and IPv6 support 705 IPv4 and IPv6 MUST be supported in the vPE solution. 707 This may present a challenge for older devices, but this normally is 708 not an issue for the newer generation of forwarding devices and 709 servers. Note that a server is replaced much more frequently than a 710 network router/switch, and newer equipment SHOULD be capable of IPv6 711 support. 713 5.2 Address space separation 715 The addresses used for the IP VPN overlay in a DC, SHOULD be taken 716 from separate address blocks outside the ones used for the underlay 717 infrastructure of the DC. This practice is to protect the DC 718 infrastructure from being attacked if the attacker gains access to 719 the tenant VPNs. 721 Similarity, the addresses used for the DC SHOULD be separated from 722 the WAN backbone addresses space. 724 6.0 Inter-connection considerations 726 The inter-connection considerations in this section are focused on 727 intra-DC inter-connections. 729 There are deployment scenarios where BGP/MPLS IP VPN may not be 730 supported in every segment of the networks to provide end-to-end IP 731 VPN connectivity. A vPE may be reachable only via an intermediate 732 inter-connecting network; interconnection may be needed in these 733 cases. 735 When multiple technologies are employed in the solution, a clear 736 demarcation should be preserved at the inter-connecting points. The 737 problems encountered in one domain SHOULD NOT impact other domains. 739 From an IP VPN point of view: An IP VPN vPE that implements [RFC4364] 740 is a component of the IP VPN network only. An IP VPN VRF on a 741 physical PE or vPE contains IP routes only, including routes learnt 742 over the locally attached network. 744 The IP VPN vPE should ideally be located as close to the "customer" 745 edge devices as possible. When this is not possible, simple existing 746 "IP VPN CE connectivity" mechanisms should be used, such as static, 747 or direct VM attachments such as described in the vCE 748 [I-D.fang-l3vpn-virtual-ce] option below. 750 Consider the following scenarios when BGP MPLS VPN technology is 751 considered as whole or partial deployment: 753 Scenario 1: All VPN sites (CEs/VMs) support IP connectivity. The most 754 suited BGP solution is to use IP VPNs [RFC4364] for all sites with PE 755 and/or vPE solutions. 757 Scenario 2: Legacy Layer 2 connectivity must be supported in certain 758 sites/CEs/VMs, and the rest of the sites/CEs/VMs need only Layer 3 759 connectivity. 761 One can consider using a combined vPE and vCE 762 [I-D.fang-l3vpn-virtual-ce] solution to solve the problem. Use IP VPN 763 for all sites with IP connectivity, and a physical or virtual CE 764 (vCE, may reside on the end device) to aggregate the Layer 2 sites 765 which for example, are in a single container in a Data Center. The 766 CE/vCE can be considered as inter-connecting points, where the Layer 767 2 network is terminated and the corresponding routes for connectivity 768 of the L2 network are inserted into IP VPN VRFs. The Layer 2 aspect 769 is transparent to the L3VPN in this case. 771 Reducing operation complicity and maintaining the robustness of the 772 solution are the primary reasons for the recommendations. 774 The interconnection of MPLS VPN in the data center and the MPLS core 775 through ASBR using existing inter-AS options is discussed in detail 776 in [I-D.fang-l3vpn-data-center-interconnect]. 778 7. Management, Control, and Orchestration 780 7.1 Assumptions 782 The discussion in this section is based on the following set of 783 assumptions: 785 - The WAN and the inter-connecting Data Center, MAY be under control 786 of separate administrative domains 788 - WAN Gateways/ASBRs/PEs are provisioned by existing WAN 789 provisioning systems 791 - If a single Gateway/ASBR/PE connecting to the WAN on one side, and 792 connecting to the Data Center network on the other side, then this 793 Gateway/ASBR/PE is the demarcation point between the two networks. 795 - vPEs and VMs are provisioned by Data Center Orchestration systems. 797 - Managing IP VPNs in the WAN is not within the scope of this 798 document except the inter-connection points. 800 7.2 Management/Orchestration system interfaces 802 The Management/Orchestration system CAN be used to communicate with 803 both the DC Gateway/ASBR, and the end devices. 805 The Management/Orchestration system MUST support standard, 806 programmatic interface for full-duplex, streaming state transfer in 807 and out of the routing system at the Gateway. 809 The programmatic interface is currently under definition in IETF 810 Interface to Routing Systems (I2RS)) initiative. 811 [I-D.ietf-i2rs-architecture], and [I-D.ietf-i2rs-problem-statement]. 813 Standard data modeling languages will be defined/identified in I2RS. 814 YANG - A Data Modeling Language for the Network Configuration 815 Protocol (NETCONF) [RFC6020] is a promising candidate currently under 816 investigation. 818 To support remote access between applications running on an end 819 device (e.g., a server) and routers in the network (e.g. the DC 820 Gateway), a standard mechanism is expected to be identified and 821 defined in I2RS to provide the transfer syntax, as defined by a 822 protocol, for communication between the application and the 823 network/routing systems. The protocol(s) SHOULD be lightweight and 824 familiar by the computing communities. Candidate examples include 825 ReSTful web services, JSON [RFC7159], NETCONF [RFC6241], XMPP 826 [RFC6120], and XML. [I-D.ietf-i2rs-architecture]. 828 7.3 Service VM Management 830 Service VM Management SHOULD be hypervisor agnostic, e.g. On demand 831 service VMs turning-up SHOULD be supported. 833 7.4 Orchestration and MPLS VPN inter-provisioning 835 The orchestration system 837 1) MUST support MPLS VPN service activation in virtualized DC. 839 2) MUST support automated cross-provisioning accounting correlation 840 between the WAN MPLS VPN and Data Center for the same tenant. 842 3) MUST support automated cross provisioning state correlation 843 between WAN MPLS VPN and Data Center for the same tenant 845 There are two primary approaches for IP VPN provisioning - push and 846 pull, both CAN be used for provisioning/orchestration. 848 7.4.1 vPE Push model 850 Push model: push IP VPN provisioning from management/orchestration 851 systems to the IP VPN network elements. 853 This approach supports service activation and it is commonly used in 854 existing MPLS VPN Enterprise deployments. When extending existing WAN 855 IP VPN solutions into the a Data Center, it MUST support off-line 856 accounting correlation between the WAN MPLS VPN and the cloud/DC MPLS 857 VPN for the tenant. The systems SHOULD be able to bind interface 858 accounting to particular tenant. It MAY requires offline state 859 correlation as well, for example, binding of interface state to 860 tenant. 862 Provisioning the vPE solution: 864 1) Provisioning process 866 a. The WAN provisioning system periodically provides to the DC 867 orchestration system the VPN tenant and RT context. 868 b. DC orchestration system configures vPE on a per request basis 870 2) Auto state correlation 872 3) Inter-connection options: 874 Inter-AS options defined in [RFC4364] may or may not be sufficient 875 for a given inter-connection scenario. BGP IP VPN inter-connection 876 with the Data Center is discussed in 877 [I-D.fang-l3vpn-data-center-interconnect]. 879 This model requires offline accounting correlation 881 1) Cloud/DC orchestration configures vPE 883 2) Orchestration initiates WAN IP VPN provisioning; passes connection 884 IDs (e.g., of VLAN/VXLAN) and tenant context to WAN IP VPN 885 provisioning systems. 887 3) WAN MPLS VPN provisioning system provisions PE VRF and policies as 888 in typical Enterprise IP VPN provisioning processes. 890 4) Cloud/DC Orchestration system or WAN IP VPN provisioning system 891 MUST have the knowledge of the connection topology between the DC 892 and WAN, including the particular interfaces on core router and 893 connecting interfaces on the DC PE and/or vPE. 895 In short, this approach requires off-line accounting correlation and 896 state correlation, and requires per WAN Service Provider integration. 898 Dynamic BGP sessions between PE/vPE and vCE MAY be used to automate 899 the PE provisioning in the PE-vCE model, that will remove the needs 900 for PE configuration. Caution: This is only under the assumption that 901 the DC provisioning system is trusted and can support dynamic 902 establishment of PE-vCE BGP neighbor relationships, for example, the 903 WAN network and the cloud/DC belong to the same Service Provider. 905 7.4.2 vPE Pull model 907 Pull model: pull from network elements to network management/AAA 908 based upon data plane or control plane activity. It supports service 909 activation. This approach is often used in broadband deployments. 910 Dynamic accounting correlation and dynamic state correlation are 911 supported. For example, session based accounting is implicitly 912 includes tenant context state correlation, as well as session-based 913 state that implicitly includes tenant context. Note that the pull 914 model is less common for vPE deployment solutions. 916 Provisioning process: 918 1) Cloud/DC orchestration configures vPE 920 2) Orchestration primes WAN MPLS VPN provisioning/AAA for new 921 service, passes connection IDs (e.g., VLAN/VXLAN) and tenant 922 context. 924 3) Cloud/DC ASBR detects new VLAN and sends Radius Access-Request (or 925 Diameter Base Protocol request message [RFC6733]). 927 4) Radius Access-Accept (or Diameter Answer) with VRF and other 928 policies 930 Auto accounting correlation and auto state correlation is supported. 932 8. Security Considerations 934 As vPE is an extended BGP/MPLS VPN solution, security threats and 935 defense techniques described in RFC 4111 [RFC4111] generally apply. 937 When the SDN approach is used, the protocols between the vPE agent 938 and the vPE-C in the controller MUST be mutually authenticated. Given 939 the potentially very large scale and the dynamic nature in the 940 cloud/DC environment, the choice of key management mechanisms need to 941 be further studied. 943 VMs in the servers can belong to different tenants with different 944 characteristics depending on the application. Classification of the 945 VMs must be done through the orchestration system and appropriate 946 security policies must be applied based on such classification before 947 turning on the services. 949 9. IANA Considerations 951 None. 953 10. References 955 10.1 Normative References 957 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 958 Requirement Levels", BCP 14, RFC 2119, March 1997. 960 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 961 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 962 Encoding", RFC 3032, January 2001. 964 [RFC3209] Awduche, D., et al., "RSVP-TE: Extension to RSVP for LSP 965 Tunnels", RFC 3209, December 2001. 967 [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., 968 "Encapsulating MPLS in IP or Generic Routing Encapsulation 969 (GRE)", RFC 4023, March 2005. 971 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 972 Networks (VPNs)", RFC 4364, February 2006. 974 [RFC4684] Marques, P., Bonica, R., Fang, L., Martini, L., Raszuk, 975 R., Patel, K., and J. Guichard, "Constrained Route 976 Distribution for Border Gateway Protocol/MultiProtocol 977 Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual 978 Private Networks (VPNs)", RFC 4684, November 2006. 980 [RFC5036] Andersson, L., Ed., Minei, I., Ed., and B. Thomas, Ed., 981 "LDP Specification", RFC 5036, October 2007. 983 [RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for 984 the Network Configuration Protocol (NETCONF)", RFC 6020, 985 October 2010. 987 [RFC6120] Saint-Andre, P., "Extensible Messaging and Presence 988 Protocol (XMPP): Core", RFC 6120, March 2011. 990 [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., 991 and A. Bierman, Ed., "Network Configuration Protocol 992 (NETCONF)", RFC 6241, June 2011. 994 [RFC6733] Fajardo, V., Ed., Arkko, J., Loughney, J., and G. Zorn, 995 Ed., "Diameter Base Protocol", RFC 6733, October 2012. 997 10.2 Informative References 999 [RFC4111] Fang, L., Ed., "Security Framework for Provider- 1000 Provisioned Virtual Private Networks (PPVPNs)", RFC 4111, 1001 July 2005. 1003 [RFC7159] Bray, T., "The JavaScript Object Notation (JSON) Data 1004 Interchange Format", RFC 7159, March 2014. 1006 [RFC4797] Rekhter, Y., Bonica, R., and E. Rosen, "Use of Provider 1007 Edge to Provider Edge (PE-PE) Generic Routing 1008 Encapsulation (GRE) or IP in BGP/MPLS IP Virtual Private 1009 Networks", RFC 4797, January 2007. 1011 [I-D.ietf-l3vpn-end-system] Marques, P., Fang, L., Pan, P., Shukla, 1012 A., Napierala, M., Bitar, N., "BGP-signaled end-system 1013 IP/VPNs", draft-ietf-l3vpn-end-system, work in progress. 1015 [I-D.rfernando-l3vpn-service-chaining] Fernando, R., Rao, D., Fang, 1016 L., Napierala, M., So, N., draft-rfernando-l3vpn-service- 1017 chaining, work in progress. 1019 [I-D.fang-l3vpn-virtual-ce] Fang, L., Evans, J., Ward, D., Fernando, 1020 R., Mullooly, J., So, N., Bitar., N., Napierala, M., "BGP 1021 IP VPN Virtual PE", draft-fang-l3vpn-virtual-ce, work in 1022 progress. 1024 [I-D.ietf-i2rs-architecture] Atlas, A., Halpern, J., Hares, S., Ward, 1026 D., and Nadeau, T., "An Architecture for the Interface to 1027 the Routing System", draft-ietf-i2rs-architecture, work in 1028 progress. 1030 [I-D.ietf-i2rs-problem-statement] Atlas, A., Nadeau, T., and Ward, 1031 D., "Interface to the Routing System Problem Statement", 1032 draft-ietf-i2rs-problem-statement, work in progress. 1034 [I-D.bitar-i2rs-service-chaining] Bitar, N., Geron, G., Fang, L., 1035 Krishnan, R., Leymann, N., Shah, H., Chakrabarti, S., 1036 Haddad, W., draft-bitar-i2rs-service-chaining, work in 1037 progress. 1039 [I-D.fang-l3vpn-data-center-interconnect] Fang, L., Fernando, R., 1040 Rao, D., Boutros, S., "BGP IP VPN Data Center 1041 Interconnect", draft-fang-l3vpn-data-center-interconnect, 1042 work in progress. 1044 [I-D.ietf-l2vpn-evpn] Sajassi, A., et al., "BGP MPLS Based Ethernet 1045 VPN", draft-ietf-l2vpn-evpn, work in progress. 1047 11. Contributors 1049 Wim Henderichx 1050 Alcatel-Lucent 1051 Email: wim.henderichx@alcatel-lucent.com 1053 Dhananjaya Rao 1054 Cisco 1055 170 W Tasman Dr 1056 San Jose, CA 1057 Email: dhrao@cisco.com 1059 Daniel Voyer 1060 Bell Canada 1061 Email: daniel.voyer@bell.ca 1063 David Ward 1064 Cisco 1065 170 W Tasman Dr 1066 San Jose, CA 95134 1067 Email: wardd@cisco.com 1069 Ning So 1070 Vinci Systems 1071 Plano, TX 75082, USA 1072 Email: ning.so@vinci-systems.com 1074 Jim Guichard 1075 Cisco 1076 Boxborough, MA 01719 1077 Email: jguichar@cisco.com 1078 Wen Wang 1079 CenturyLink 1080 2355 Dulles Corner Blvd. 1081 Herndon, VA 20171 1082 Email: Wen.Wang@CenturyLink.com 1084 Manuel Paul 1085 Deutsche Telekom 1086 Winterfeldtstr. 21-27 1087 10781 Berlin, Germany 1088 Email: manuel.paul@telekom.de 1090 Authors' Addresses 1092 Luyuan Fang 1093 Microsoft 1094 5600 148th Ave NE 1095 Redmond, WA 98052 1096 Email: lufang@microsoft.com 1098 Rex Fernando 1099 Cisco 1100 170 W Tasman Dr 1101 San Jose, CA 1102 Email: rex@cisco.com 1104 Maria Napierala 1105 AT&T 1106 200 Laurel Avenue 1107 Middletown, NJ 07748 1108 Email: mnapierala@att.com 1110 Nabil Bitar 1111 Verizon 1112 40 Sylvan Road 1113 Waltham, MA 02145 1114 Email: nabil.bitar@verizon.com 1116 Bruno Rijsman 1117 Juniper Networks 1118 10 Technology Park Drive 1119 Westford, MA 01886 1120 Email: brijsman@juniper.net