idnits 2.17.1 draft-dunbar-bess-bgp-sdwan-usage-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 27 instances of too long lines in the document, the longest one being 17 characters in excess of 72. == There are 1 instance of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 4, 2019) is 1634 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC 4364' is mentioned on line 142, but not defined == Missing Reference: 'RFC4456' is mentioned on line 273, but not defined == Missing Reference: 'RFC4301' is mentioned on line 817, but not defined == Missing Reference: 'RFC 3489' is mentioned on line 840, but not defined ** Obsolete undefined reference: RFC 3489 (Obsoleted by RFC 5389) == Unused Reference: 'RFC2119' is defined on line 946, but no explicit reference was found in the text == Unused Reference: 'RFC7432' is defined on line 955, but no explicit reference was found in the text == Unused Reference: 'RFC8192' is defined on line 963, but no explicit reference was found in the text == Unused Reference: 'RFC5521' is defined on line 966, but no explicit reference was found in the text == Unused Reference: 'BGP-SDWAN-Port' is defined on line 970, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-Gap' is defined on line 974, but no explicit reference was found in the text == Unused Reference: 'VPN-over-Internet' is defined on line 978, but no explicit reference was found in the text == Unused Reference: 'DMVPN' is defined on line 982, but no explicit reference was found in the text == Unused Reference: 'DSVPN' is defined on line 986, but no explicit reference was found in the text == Unused Reference: 'ITU-T-X1036' is defined on line 997, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-gap' is defined on line 1005, but no explicit reference was found in the text == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 == Outdated reference: A later version (-06) exists of draft-sajassi-bess-secure-evpn-01 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 -- Duplicate reference: draft-rosen-bess-secure-l3vpn, mentioned in 'SECURE-L3VPN', was also mentioned in 'VPN-over-Internet'. == Outdated reference: A later version (-07) exists of draft-dm-net2cloud-problem-statement-02 == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 -- Duplicate reference: draft-dm-net2cloud-gap-analysis, mentioned in 'Net2Cloud-gap', was also mentioned in 'Net2Cloud-Gap'. == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-10 Summary: 2 errors (**), 0 flaws (~~), 24 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft J. Guichard 3 Intended status: Informational Futurewei 4 Expires: May 4, 2020 Ali Sajassi 5 Cisco 6 J. Drake 7 Juniper 8 Ayan Barnerjee 9 D. Carrel 10 Cisco 12 November 4, 2019 14 BGP Usage for SDWAN Overlay Networks 15 draft-dunbar-bess-bgp-sdwan-usage-03 17 Abstract 19 The document describes three distinct SDWAN scenarios and discusses 20 the applicability of BGP for each of those scenarios. The goal of 21 the document is to make it easier for future SDWAN control plane 22 protocols discussion. 24 SDWAN edge nodes are commonly interconnected by multiple underlay 25 networks that are owned and managed by different network providers. 26 A BGP-based control plane is chosen for handling large number of 27 SDWAN edge nodes with little manual intervention. 29 Status of this Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. This document may not be modified, 36 and derivative works of it may not be created, except to publish it 37 as an RFC and to translate it into languages other than English. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF), its areas, and its working groups. Note that 41 other groups may also distribute working documents as Internet- 42 Drafts. 44 Internet-Drafts are draft documents valid for a maximum of six 45 months and may be updated, replaced, or obsoleted by other documents 46 at any time. It is inappropriate to use Internet-Drafts as 47 reference material or to cite them other than as "work in progress." 49 The list of current Internet-Drafts can be accessed at 50 http://www.ietf.org/ietf/1id-abstracts.txt 52 The list of Internet-Draft Shadow Directories can be accessed at 53 http://www.ietf.org/shadow.html 55 This Internet-Draft will expire on January 4, 2009. 57 Copyright Notice 59 Copyright (c) 2019 IETF Trust and the persons identified as the 60 document authors. All rights reserved. 62 This document is subject to BCP 78 and the IETF Trust's Legal 63 Provisions Relating to IETF Documents 64 (http://trustee.ietf.org/license-info) in effect on the date of 65 publication of this document. Please review these documents 66 carefully, as they describe your rights and restrictions with 67 respect to this document. Code Components extracted from this 68 document must include Simplified BSD License text as described in 69 Section 4.e of the Trust Legal Provisions and are provided without 70 warranty as described in the Simplified BSD License. 72 Table of Contents 74 1. Introduction...................................................3 75 2. Conventions used in this document..............................4 76 3. Use Case Scenario Description and Requirements.................5 77 3.1. Requirements..............................................5 78 3.1.1. Client Service Requirement...........................5 79 3.1.2. Application Flow based Segmentation..................6 80 3.1.3. SDWAN Node Provisioning..............................7 81 3.2. Scenarios #1: Homogeneous WAN.............................8 82 3.3. Scenario #2: SDWAN WAN ports to VPN's PEs and to Internet10 83 3.4. Scenario #3: SDWAN WAN ports to MPLS VPN and the Internet12 84 4. Provisioning Model............................................14 85 4.1. Client Service Provisioning Model........................14 86 4.2. WAN Ports Provisioning Model.............................14 87 4.2.1. Why BGP as Control Plane for SDWAN WAN Ports 88 Registration?..............................................15 89 5. SDWAN Traffic Forwarding Walk Through.........................16 90 5.1. SDWAN Network Startup Procedures.........................16 91 5.2. Packet Walk-Through for Scenario #1......................16 92 5.3. Packet Walk-Through for Scenario #2......................17 93 5.3.1. SDWAN node WAN Ports Properties Registration........19 94 5.3.2. Controller Facilitated IPsec SA & NAT management....19 95 5.3.3. BGP Based SDWAN client routes.......................21 96 5.4. Packet Walk-Through for Scenario #3......................22 97 6. Manageability Considerations..................................22 98 7. Security Considerations.......................................23 99 8. IANA Considerations...........................................23 100 9. References....................................................23 101 9.1. Normative References.....................................23 102 9.2. Informative References...................................23 103 10. Acknowledgments..............................................25 105 1. Introduction 107 An "SDWAN" network refers to an overlay network with many parallel 108 paths over different underlay networks, some of which are private 109 networks over which traffic can traverse without encryption, others 110 require encryption over untrusted public networks. 112 [Net2Cloud-Problem] describes the network related problems that 113 enterprises face today in transitioning their IT infrastructure to 114 support a digital economy, such as the need to connect enterprises' 115 branch offices to dynamic workloads in different Cloud DCs, or 116 aggregating multiple paths provided by different service providers 117 to achieve better performance. 119 Even though SDWAN has been positioned as a flexible way to reach 120 dynamic workloads in third party Cloud data centers over different 121 underlay networks, scaling becomes a major issue when there are 122 hundreds or thousands of nodes to be interconnected by an SDWAN 123 overlay network. 125 BGP is widely used by underlay networks. This document describes 126 using BGP to enhance the scaling properties of SDWAN overlay 127 networks. 129 2. Conventions used in this document 131 Cloud DC: Third party data centers that host applications and 132 workloads owned by different organizations or tenants. 134 Controller: Used interchangeably with SDWAN controller to manage 135 SDWAN overlay path creation/deletion and monitor the 136 path conditions between sites. 138 CPE: Customer Premise Equipment 140 CPE-Based VPN: Virtual Private Secure network formed among CPEs. 141 This is to differentiate from more commonly used PE- 142 based VPNs [RFC 4364]. 144 Homogeneous SDWAN: A type of SDWAN network in which all traffic 145 to/from the SDWAN edge nodes has to be encrypted 146 regardless of underlay networks. For lack of better 147 terminology, we call this Homogeneous SDWAN throughout 148 this document. 150 ISP: Internet Service Provider 152 NSP: Network Service Provider. NSP usually provides more 153 advanced network services, such as MPLS VPN, private 154 leased lines, or managed Secure WAN connections, many 155 times within a private trusted domain, whereas an ISP 156 usually provides plain internet services over public 157 untrusted domains. 159 PE: Provider Edge 161 SDWAN End-point: a port (logical or physical) of a SDWAN edge node. 163 SDWAN: Software Defined Wide Area Network. In this document, 164 "SDWAN" refers to the solutions of pooling WAN bandwidth 165 from multiple underlay networks to get better WAN 166 bandwidth management, visibility & control. When the 167 underlay networks are private, traffic can traverse 168 without additional encryption; when the underlay 169 networks are public, such as the Internet, some traffic 170 may need to be encrypted when traversing through 171 (depending on user provided policies). 173 SDWAN IPsec SA: IPsec Security Association between two SDWAN ports 174 or nodes. 176 SDWAN over Hybrid Networks: SDWAN over Hybrid Networks typically 177 have edge nodes utilizing bandwidth resources from 178 multiple service providers. In Hybrid SDWAN network, 179 packets over private networks can go natively without 180 encryption and are encrypted over the untrusted network, 181 such as the public Internet. 183 WAN Port: A Port or Interface facing an ISP or Network Service 184 Provider (NSP), with address (usually public routable 185 address) allocated by the ISP or the NSP. 187 C-PE: SDWAN Edge node, which can be CPE for customer managed 188 SDWAN, or PE that is for provider managed SDWAN 189 services). 191 ZTP: Zero Touch Provisioning 193 3. Use Case Scenario Description and Requirements 195 SDWAN networks can have different topologies and have different 196 traffic patterns. To make it easier for the focused discussion in 197 subsequent drafts on SDWAN control plane and data plane, this 198 section describes several SDWAN scenarios that may have different 199 need or impact to their corresponding control planes & data planes. 201 3.1. Requirements 203 3.1.1. Client Service Requirement 205 Client interface of SDWAN nodes can be IP or Ethernet based. 207 For Ethernet based client interfaces, SDWAN edge should support 208 VLAN-based service interfaces (EVI100), VLAN bundle service 209 interfaces (EVI200), or VLAN-Aware bundling service interfaces. EVPN 210 service requirements are applicable to the Client traffic, as 211 described in the Section 3.1 of RFC8388. 213 For IP based client interfaces, L3VPN service requirements are 214 applicable. 216 3.1.2. Application Flow based Segmentation 218 Application Flow based Segmentation refers to separate traffic based 219 on business and security needs, e.g. having different topology for 220 different traffic types or users/apps. 222 For example, retail business requires traffic from payment 223 applications in all branches only go to the Payment Gateway in its 224 HQ Data Centers, whereas other applications can be multi-point. 225 Segmentation is analogous to VLAN (in L2 network) and VRF (in L3 226 network) 228 Segmentation is a feature that can be provided or enabled for a 229 single SDWAN service (or domain). Each Segment can have its own 230 policy and topology. 232 In the figure below, the traffic from the Payment application is 233 along the Tree topology, whereas other traffic can be multipoint to 234 multi point topology as in VRF. 236 +--------+ 237 Payment traffic |Payment | 238 +------+----+-+gateway +------+----+-----+ 239 / / | +----+---+ | \ \ 240 / / | | | \ \ 241 +-+--+ +-+--+ +-+--+ | +-+--+ +-+--+ +-+--+ 242 |Site| |Site| |Site| | |Site| |Site| |Site| 243 | 1 | | 2 | | 3 | | |4 | | 5 | | 6 | 244 +--+-+ +--+-+ +--|-+ | +--|-+ +--|-+ +--|-+ 245 | | | | | | | 246 ==+=======+=======+====+======+=======+=======+=== 247 Multi-point connection for Other traffic 249 3.1.3. SDWAN Node Provisioning 251 Unlike traditional EVPN or L3VPN where PEs are deployed for long 252 term, SDWAN edge nodes (virtual or physical) deployment at a 253 specific location can be ephemeral. Therefore, Zero Touch 254 Provisioning (ZTP) is a common requirement for SDWAN. ZTP for SDWAN 255 can include many areas, but from network connectivity perspective, 256 ZTP should include the following: 258 - Upon power up, an SDWAN node can reach a central SDWAN 259 Controller (which can be burned or preconfigured in the device) 260 via a TLS or SSL secure channel. 262 - The Central SDWAN Controller can designate a Local Network 263 Controller in the proximity of the SDWAN node; the Local Network 264 Controller and the SDWAN nodes might be connected by third party 265 untrusted network. The Local controller does all the following 4 266 tasks: 268 1) ZTP 269 2) Auto-discovery of Network 270 3) (Auto)-Provisioning for IPsec SAs (initial provisioning 271 part) 272 4) Signaling of tenant's routes/info 273 BGP is well suited for (4), using Route Reflector (RR) [RFC4456] 274 to propagate network information among SDWAN edge nodes. The SDWAN 275 node can establish a secure connection (TLS, SSL, etc) to the 276 Local Network Controller (RR). 278 +---+ 279 Peer Group 1 |RR | Peer Group 2 280 +======+====+=+ +======+====+=====+ 281 / / | +---+ | \ \ 282 / / | | \ \ 283 +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ 284 |C-PE| |C-PE| |C-PE| |C-PE| |C-PE| |C-PE| 285 | 1 | | 2 | | 3 | |4 | | 5 | | 6 | 286 +----+ +----+ +----+ +----+ +----+ +----+ 287 Tenant 1 Tenant 2 288 Figure 1: Peer Groups managed by Local Controller 290 The SDWAN nodes (a.k.a. C-PEs throughout this document) belonging to 291 the same Tenant can be far apart and can be connected by third party 292 untrusted networks. Therefore, it is not appropriate for a SDWAN 293 edge node (C-PE) to advertise its SDWAN Port properties to its 294 neighbors. Each C-PE propagates its SDWAN Port attributes via the 295 secure channel (TLS, SSL, etc.) established with the Local 296 Controller. 298 C-PE-1 should include the following aspects in addition to managing 299 client routes: 300 - Register the SDWAN node's WAN port <-> local address mapping 301 to its Local Controller. The Local Controller propagates the 302 information to C-PE2 & C-PE3. 303 - Exchange IPsec property (capability such as the supported 304 encryption algorithms, etc.) and ports NAT property (e.g. 305 private addresses or dynamically assigned IP addresses) with 306 the Local Controller. 307 - C-PE2 and C-PE3 can establish IPsec SA with the C-PE1 after 308 receiving the information from the Local Controller. 309 - Then distribute the routes attached to the C-PE to its 310 authorized peers. 312 Tenant separation is achieved by the Local Controller creating 313 different Tenant based Peer Groups. 315 3.2. Scenarios #1: Homogeneous WAN 317 This is referring to a type of SDWAN network with edge nodes 318 encrypting all traffic over WAN to other edge nodes, regardless of 319 whether the underlay is private or public. For lack of better 320 terminology, we call this Homogeneous SDWAN throughout this 321 document. 323 Some typical scenarios for the use of a Homogeneous SDWAN network 324 are as follows: 326 - A small branch office connecting to its HQ offices via the 327 Internet. All sensitive traffic to/from this small branch office has 328 to be encrypted, which is usually achieved using IPsec SAs. 330 - A store in a shopping mall may need to securely connect to its 331 applications in one or more Cloud DCs via the Internet. A common way 332 of achieving this is to establish IPsec SAs to the Cloud DC gateway 333 to carry the sensitive data to/from the store. 335 As described in [SECURE-EVPN], the granularity of the IPsec SAs for 336 Homogeneous SDWAN can be per site, per subnet, per tenant, or per 337 address. Once the IPsec SA is established for a specific 338 subnet/tenant/site, all traffic to/from the subnets/tenants/site are 339 encrypted. 341 +---+ 342 +--------------|RR |------------+ 343 / Untrusted +-+-+ \ 344 / \ 345 / \ 346 +----+ +---------+ +------+ +----+ 347 | CN3|--| P1-----+ -------------+------ P1 |--| CN1| 348 +----+ | C-PE P2-----+ | | C-PE | +----+ 349 +----+ | A P3-----+ Wide +------ P2 B | +----+ 350 | CN2|--| | | area +------ P3 |--| CN3| 351 +----+ +---------+ | network | +------+ +----+ 352 | | 353 +----+ +---------+ | all packets | +------+ +----+ 354 | CN1|--| P1-----+ encrypted +------ P1 |--| CN1| 355 +----+ | C-PE P2-----+ by | | C-PE | +----+ 356 +----+ | C P3-----+ IPsec SAs +------ P2 D | +----+ 357 | CN2|--| P4-----+--------------+ | |--| CN2| 358 +----+ +---------+ +------+ +----+ 360 CN: Client Networks, which is same as Tenant Networks used by NVo3 362 Figure 1: Homogeneous SDWAN 364 One of the key properties of homogeneous SDWAN is that the SDWAN 365 Local Network Controller (RR)is connected to C-PEs via untrusted 366 public network, therefore, requiring secure connection between RR 367 and C-PEs (TLS, DTLS, etc.). 369 Homogeneous SDWAN has some similarity to commonly deployed IPsec 370 VPN, albeit the IPsec VPN is usually point-to-point among a small 371 number of endpoints and with heavy manual configuration for IPsec 372 between end-points, whereas an SDWAN network can have a large number 373 of end-points with an SDWAN controller to manage requiring zero 374 touch provisioning upon powering up. 376 Existing Private VPNs (e.g. MPLS based) can use homogeneous SDWAN to 377 extend over public network to remote sites to which the VPN operator 378 does not own or lease infrastructural connectivity, as described in 379 [SECURE-EVPN] and [SECURE-L3VPN] 381 3.3. Scenario #2: SDWAN WAN ports to VPN's PEs and to Internet 383 In this scenario, SDWAN edge nodes (a.k.a. C-PEs) have some WAN 384 ports connected to PEs of Private VPNs over which packets can be 385 forwarded natively without encryption, and some WAN ports connected 386 to the Internet over which sensitive traffic have to be encrypted 387 (usually by IPsec SA). 389 In this scenario, the SDWAN edge nodes' egress WAN ports are all 390 IP/Ethernet based, either egress to PEs of the VPNs or to the 391 Internet. Even if the VPN is a MPLS network, the VPN's PEs have 392 IP/Ethernet connections to the SDWAN edge (C-PEs). Throughout this 393 document, this scenario is also called CPE based SDWAN over Hybrid 394 Networks. 396 Even though IPsec SA can secure the packets traversing the Internet, 397 it does not offer the premium SLA commonly offered by Private VPNs, 398 especially over long distance. Clients need to have policies to 399 specify criteria for flows only traversing private VPNs or 400 traversing either as long as encrypted when over the Internet. For 401 example, client can have those polices for the flows: 403 1. A policy or criteria for sending the flows over a private network 404 without encryption (for better performance), 405 2. A policy or criteria for sending the flows over any networks as long 406 as the packets of the flows are encrypted when traversing untrusted 407 networks, or 408 3. A policy of not needing encryption at all. 410 If a flow traversing multiple segments, such as A<->B<->C<->D, has 411 either Policy 2 or 3 above, the flow can traverse different 412 underlays in different segments, such as over Private network 413 underlay between A<->B without encryption, or over the public 414 internet between B<->C in an IPsec SA. 416 As shown in the figure below, C-PE-1 has two different types of 417 interfaces (A1 to Internet and A2 & A3 to VPN). The C-PEs' loopback 418 addresses and addresses attached to C-PEs may or may not be visible 419 to the ISPs/NSPs. The addresses for the WAN ports can have addresses 420 allocated by the service providers or dynamically assigned (e.g. by 421 DHCP). One WAN port shown in the figure below (e.g. A1, A2, A3 etc.) 422 is a logical representation of potential multiple physical ports on 423 the C-PEs. 425 +---+ 426 +--------------|RR |----------+ 427 / Untrusted +-+-+ \ 428 / \ 429 / \ 430 +----+ +---------+ packets encrypted over +------+ +----+ 431 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 432 +----+ | C-PE A2-\ | C-PE | +----+ 433 +----+ | A A3--+--+ +---+---B2 B | +----+ 434 | CN2|--| | |PE+--------------+PE |---B3 |--| CN3| 435 +----+ +---------+ +--+ trusted +---+ +------+ +----+ 436 | WAN | 437 +----+ +---------+ +--+ packets +---+ +------+ +----+ 438 | CN1|--| C1--|PE| go natively |PE |-- D1 |--| CN1| 439 +----+ | C-PE C2--+--+ without encry+---+ | C-PE | +----+ 440 | C | +--------------+ | D | 441 | | | | 442 +----+ | | without encrypt over | | +----+ 443 | CN2|--| C3--+---- Untrusted --+------D2 |--| CN2| 444 +----+ +---------+ +------+ +----+ 446 CN: Client Network 447 Figure 2: Hybrid SDWAN 449 Some key characteristics of a Hybrid SDWAN overlay network are as 450 follows: 452 - one C-PE may be connected to different ISPs/NSPs, with some of its 453 WAN ports addresses being assigned by the ISPs/NSPs. 455 - The WAN ports connected to PEs of trusted private networks (e.g. MPLS 456 VPN) hand off IP/Ethernet packets, just like today's CPE that do not 457 handle MPLS packets and do not participate in the underlay VPN networks' 458 control plane. Traffic can flow natively without encryption when be 459 forwarded out through those WAN ports for better performance. 461 - The WAN ports connected to untrusted networks, e.g. the Internet, 462 requires sensitive traffic to be encrypted, i.e. encrypted by IPsec SA. 464 - An SDWAN local Network Controller (RR) is connected to C-PEs via 465 the untrusted public network, therefore, requiring secure 466 connection between RR and C-PEs via TLS, DTLS, etc. 468 - The SDWAN nodes' [loopback] addresses might not be routable nor 469 visible in the underlay ISP/NSP networks. Routes & services 470 attached to SDWAN edges at the SDWAN overlay layer are in 471 different address spaces than the underlay networks. 473 - There could be multiple SDWAN devices sharing a common property, 474 such as a geographic location. Some applications over SDWAN may 475 need to traverse specific geographic locations for various 476 reasons, such as to comply regulatory rules, to utilize specific 477 value added services, or others. 479 - The underlay path selection between sites can be a local decision. 480 Some policies allow one service from CPE1 -> CPE2 -> CPE3 using 481 one ISP/NSP underlay in the first segment (CPE1 -> CPE2), and 482 using a different ISP/NSP in the second segment (CPE2-> CPE3). 484 - Services may not be congruent, i.e. the packets from A-> B may 485 traverse one underlay network, and the packets from B -> A may 486 traverse a different underlay. 488 - Different services, routes, or VLANs attached to SDWAN nodes can 489 be aggregated over one underlay path; same service/routes/VLAN can 490 spread over multiple SDWAN underlays at different times depending 491 on the policies specified for the service. For example, one 492 tenant's packets to HQ need to be encrypted when sent over the 493 Internet or have to be sent over private networks, while the same 494 tenant's packets to Facebook can be sent over the Internet without 495 encryption. 497 3.4. Scenario #3: SDWAN WAN ports to MPLS VPN and the Internet 499 This scenario refers to existing VPN (e.g. MPLS based VPN, such as 500 EVPN or IPVPN) adding extra ports facing untrusted public networks 501 allowing PEs to offload some low priority traffic to those ports 502 facing public networks when the VPN MPLS paths are congested. 503 Throughout this document, this scenario is also called Internet 504 Offload for Private VPN, or PE based SDWAN. 506 In this scenario, it is important that the packets offloaded to 507 untrusted public network be encrypted. In this scenario, there is a 508 secure BGP connection between RR & PEs. 510 PE based SDWAN can be used by VPN service providers to temporarily 511 increase bandwidth between sites when they are not sure if the 512 demand will sustain for long period of time or as a temporary 513 solution before the permanent infrastructure is built or leased. 515 +---+ 516 +-------|PE2| 517 / +---+ 518 Internet / ^ 519 offload / || VPN 520 / VPN v 521 ++--+ ++-+ +---+ 522 |PE1| <====> |RR| <===> |PE3| 523 +-+-+ +--+ +-+-+ 524 | | 525 +--- Public Internet -- + 527 Figure 3: Additional Internet paths added to the VPN 529 Here are some key properties for PE based SDWAN: 531 - For MPLS based VPN, PEs continue having MPLS encapsulation 532 handoff to existing paths. 533 - The BGP RR is connected to PEs in the same way as VPN, i.e. 534 via the trusted network. 535 - For the added Internet ports, PEs have IP packets handoff, 536 i.e. sending and receiving IP data frames. Internally, PEs 537 can have the option to encapsulate the MPLS payload in IP, as 538 specified by RFC4023. 539 - The ports facing public internet might get IP addresses 540 assigned by ISPs, which may not be in the same address domain 541 as PEs'. 542 - Ports facing public internet are not as secure as the ports 543 facing private infrastructure. There could be spoofing, or 544 DDOS attacks to the ports facing public internet. Extra 545 consideration must be given when injecting the new routes 546 from public network into VRFs. 547 - Even though packets are encrypted over public internet, the 548 performance SLA is not guaranteed over public internet. 549 Therefore, clients may have policies only allowing some flows 550 to be offloaded to internet path. 552 4. Provisioning Model 554 4.1. Client Service Provisioning Model 556 The provisioning tasks described in Section 4 of RFC8388 are the 557 same for the SDWAN client traffic. When client traffic are multi- 558 homed to two (or more) C-PEs, the Non-Service-Specific parameters 559 need to be provisioned per the Section 4.1.1 of RFC8388. 561 Since most SDWAN nodes are ephemeral and have small number of IP 562 subnets or VLANs attached to the client ports of the SDWAN nodes, it 563 is recommended to have default and simplified Service-specific 564 parameters for each client port, remotely managed by the SDWAN 565 Network Controller (i.e. the RR) via the secure channel (TLS/DTLS) 566 between the controller and the C-PEs. 568 More details are to be added. 570 4.2. WAN Ports Provisioning Model 572 Since the deployment of PEs to MPLS VPN are for relatively long 573 term, the common provisioning procedure for PE's WAN ports is via 574 CLI. 576 A SDWAN node deployment can be ephemeral and its location can be in 577 remote locations, manual provisioning for its WAN ports is not 578 acceptable. In addition, a SDWAN WAN port's IP address can be 579 dynamically assigned or using private addresses. Therefore, it is 580 necessary to have a separate control protocol; something like NHRP 581 did for ATM, for a SDWAN node to register its WAN property to its 582 controller dynamically. 584 Unlike a PE to MPLS based VPN where its WAN ports are homogeneously 585 facing MPLS private network and all traffic are egressed in MPLS 586 data frames through its WAN ports, the WAN ports of a SDWAN node can 587 be connected to a PE of VPN, MPLS private network directly, the 588 public Internet, or the various combinations of all. 590 For Scenario #1 above, the WAN ports can face public internet or 591 VPN. 593 For Scenario #2 above, WAN ports are either configured as connecting 594 to PEs of VPN where traffic can be sent as IP/Ethernet without 595 encryption, or configured as connecting to public Internet. 597 For Scenario #3 above, the WAN ports are either configured as VPN 598 egress ports (hand off MPLS data frames), or as connecting to the 599 public internet that requires MPLS in IP in IPsec encapsulation. 601 4.2.1. Why BGP as Control Plane for SDWAN WAN Ports Registration? 603 For a small sized SDWAN network, traditional hub & spoke model using 604 NHRP or DSVPN/DMVPN with a hub node (or controller) managing SDWAN 605 node WAN ports mapping (e.g. local & public addresses and tunnel 606 identifiers mapping) can work reasonably well. However, for a large 607 SDWAN network, say more than 100 nodes with different types of 608 topologies, the traditional approach becomes very messy, complex and 609 error prone. 611 Here are some of the compelling reasons of using BGP instead of 612 extending NHRP/DSVPN/DMVPN. (Same as the reasons quoted by LSVR on 613 why using BGP): 615 - BGP already widely deployed as sole protocol (see RFC 7938) 617 - Robust and simple implementation 619 - Wide acceptance - minimal learning 621 - Reliable transport 623 - Guaranteed in-order delivery 625 - Incremental updates 627 - Incremental updates upon session restart 629 - No flooding and selective filtering 631 - RR already has the capability to apply policies to communications 632 among peers. 634 5. SDWAN Traffic Forwarding Walk Through 636 BGP based EVPN control plane are still applicable to routes attached 637 to the client ports of SDWAN nodes. Section 5 of RFC8388 describes 638 the BGP EVPN NLRI Usage for various routes of client traffic. The 639 procedures described in the Section 6 of RFC8388 are same for the 640 SDWAN client traffic. 642 The only additional consideration for SDWAN is to control how 643 traffic egress the SDWAN edge node to various WAN ports. 645 5.1. SDWAN Network Startup Procedures 647 A SDWAN network can add or delete SDWAN edge nodes on regular basis 648 depending on user requests. 650 - For Scenario #1: a SDWAN edge node in a shopping mall or Cloud DC can 651 be added or removed on demand. The Zero Touch Provisioning described 652 in 3.1.2 are required for the node startup. 653 - For Scenario #2: this can be Data Centers or enterprises upgrading 654 their CPEs to add extra bandwidth via public internet in addition to 655 VPN services that they already purchased. Before the node powers up 656 or upgraded, there should be links connected to the PEs of a provider 657 VPNs. 658 - For Scenario #3, the Internet facing WAN ports are added to (or 659 removed from) existing VPN PEs. 661 5.2. Packet Walk-Through for Scenario #1 663 Upon power up, a SDWAN node can learn client routes from the Client 664 facing ports, in the same way as EVPN described in RFC8388. 665 Controller facilitates the IPsec SA establishment and rekey 666 management as described in [SECURE-EVPN]. Controller manages how 667 client's routes are associated with individual IPSec SA. 669 [SECURE-L3VPN] describes how to extend the RFC4364 VPN to allow some 670 PEs being connected to other PEs via public networks. [SECURE-L3VPN] 671 introduces the concept of Red Interface & Black Interface on those 672 PEs, with RED interfaces face clients' routes within the VPN and the 673 Black Interfaces being WAN ports over which only IPsec-protected 674 packets to the Internet or other backbone network are sent so that 675 eliminating the need for MPLS transport in the backbone. 677 [SECURE-L3VPN] assumes PEs terminate MPLS packets, and use MPLS over 678 IPsec when sending over the Black Interfaces. 680 [SECURE-EVPN] describes a solution where BGP point-to-multipoint 681 signaling is leveraged as control plane for SDWAN Scenario #1. It 682 utilizes the BGP RR to facilitate the key and policy exchange among 683 PE devices to create private pair-wise IPsec Security Associations 684 without IKEv2 point-to-point signaling or any other direct peer-to- 685 peer session establishment messages. 687 When C-PEs do not support MPLS, the approaches described by RFC8365 688 can be used, with addition of IPsec encrypting the IP packets when 689 sending packets over the Black Interfaces. 691 5.3. Packet Walk-Through for Scenario #2 693 In this scenario, C-PEs have some WAN ports connected to the public 694 internet and some WAN ports connected to (i.e. directly connected 695 to) PEs of trusted VPN. The C-PEs in Scenario #2 are almost like 696 CPEs to MPLS VPN that have the IP/Ethernet data frames egress to the 697 PEs of the VPN, except the packets need encryption if egress to the 698 WAN ports facing public Internet. 700 Users specify the policy or criteria on which flows can only egress 701 WAN ports facing trusted VPN without encryption, which can egress 702 the WAN ports facing the public Internet with encryption, or which 703 can egress WAN ports facing the public Internet without encryption. 705 The Control Plane should not learn routes from the Public Network 706 facing WAN ports. Should strictly follow the policies specified by 707 the users. The internet facing WAN ports can face potential DDoS 708 attacks, additional anti-DDoS mechanism has to be implemented on WAN 709 ports facing those public networks. 711 The Scenario #2 SDWAN Control Plane has three distinct functional 712 components: 714 +---+ 715 +--------------|RR |----------+ 716 / Untrusted +-+-+ \ 717 / Network \ 718 / \ 719 +----+ +---------+ packets encrypted over +--------+ +----+ 720 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 721 +----+ | C-PE1 A2-----+ +-------B2 C-PE2 | +----+ 722 |10.1.1.1 | |10.1.2.1| 723 +----+ | | +--+ +---+ | | +----+ 724 | CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3| 725 +----+ +---------+ +--+ trusted +---+ +--------+ +----+ 726 | VPN | 727 +--------------+ 728 Figure 5: SDWAN Scenario #2 730 - SDWAN node's WAN ports property registration to the SDWAN 731 Network Controller (BGP RR). 732 o This is used to inform the SDWAN controller of all the 733 underlay networks to which the C-PE is connected. 734 o RR is responsible for propagating the C-PE WAN ports 735 properties to authorized peers. 737 - Controller Facilitated IPsec SA management and NAT information 738 distribution 739 o Used by the SDWAN controller to facilitate or manage the 740 IPsec configurations and peer authentications for all 741 IPsec SAs terminated at the SDWAN nodes. 742 o When WAN ports have private addresses, need exchange 743 between SDWAN edges and the RR about the type of NAT, and 744 mapping of the private addresses/ports <-> public 745 addresses/ports. 747 - Attached routes distribution via BGP RR, which can be EVPN, 748 IPVPN or others. 749 o This is for the overlay layer's route distribution, so 750 that a C-PE can establish the overlay routing table that 751 identifies the next hop for reaching a specific 752 route/service attached to remote nodes. [SECURE-EVPN] 753 describes EVPN and other options. 755 5.3.1. SDWAN node WAN Ports Properties Registration 757 In Figure 6, A1/A2/A3/B1/B2/B3 WAN ports can be from different 758 network providers. 760 +---+ 761 10.1.1.1 via +--------------|RR |----------+ 10.1.2.1 via 762 A1/A2/A3 / Untrusted +-+-+ \ B1/B2/B3 763 / \ 764 / \ 765 +----+ +---------+ packets encrypted over +--------+ +----+ 766 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 767 +----+ | C-PE1 A2-----+ +-------B2 C-PE2 | +----+ 768 |10.1.1.1 | |10.1.2.1| 769 +----+ | | +--+ +---+ | | +----+ 770 | CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3| 771 +----+ +---------+ +--+ trusted +---+ +--------+ +----+ 772 | VPN | 773 +--------------+ 774 Figure 6: SDWAN Scenario #2 WAN Ports Registration 776 Each SDWAN edge(C-PE) needs to register its WAN ports properties 777 along with its Loopback addresses to the SDWAN Network Controller 778 (RR). The policies that govern the communications among peers are 779 managed and controlled by the SDWAN Controller. Individual SDWAN 780 edge relies on its SDWAN Controller to determine which peers can 781 establish connections. The SDWAN controller is responsible for 782 propagating the mapping information to the authorized peers. If C- 783 PE-1 is not authorized to communicate with C-PE-n, C-PE-1's WAN 784 port<->Loopback address mapping will not be propagated to C-PE-n. 786 A C-PE's Loopback addresses & attached routes may not be visible to 787 some ISPs/NSPs to which the CPE's WAN port is connected. 789 5.3.2. Controller Facilitated IPsec SA & NAT management 791 One IPsec SA between two end points is straightforward. However, for 792 a network with many IPsec SAs among many end points, the 793 configuration and IPsec Key management for the entire network can be 794 complex. 796 For a 1,000-node network, each node is responsible for maintaining 797 and managing 999 keys to all their peers, which could potentially 798 result in 1,000,000 key exchanges to authenticate among all nodes. 799 In addition, when an edge node has multiple tenants attached, the 800 edge node may need to establish multiple tunnels for tenants. For 801 example, for a network with N nodes, a node A has 5 tenants app 802 attached to it, then the node A has to maintain 5*(N-1) number of 803 keys if each tenant needs to communicate with all other nodes. 805 In addition, all the IPsec keys have to be refreshed periodically, 806 which adds more complexity. Therefore, simplification facilitated by 807 an SDWAN controller is necessary for large-scale SDWAN deployment. 809 When the SDWAN IPsec SAs are fine-grained, such as per client 810 address, per client's VLAN, the number of IPsec SAs & Keys to be 811 managed can go much higher, leading to more IPsec management 812 complexity. It is better to aggregate multiple flows into one IPsec 813 SA. 815 SDWAN edge nodes can rely on the SDWAN controller to facilitate the 816 pair-wise IPsec key establishment and refreshment [RFC7296] and 817 maintain the Security Policy Database (SPD) [RFC4301]. 819 - In the Figure 5 SDWAN Scenario #2 above, if C-PE1 & C-PE2 each has 820 two ports facing two different ISPs networks, and their loopback 821 addresses are not visible to the ISPs, i.e. the C-PE1 & C-PE2 are 822 using a provider assigned IP addresses for A1/A2/B1/B2; you are 823 going to need minimum four IPsec SAs between C-PE1 & C-PE2. 824 - When C-PEs loopback addresses are visible to ISPs/NSPs, i.e. the 825 C-PEs' private source and destination IPs are part of a prefix 826 exported to the ISP(s) in each site, it is possible to have one 827 IPsec SA between C-PE1 & C-PE2. 829 The IP addresses of SDWAN WAN port can be dynamic (e.g. assigned by 830 DHCP) or private IP. Some SDWAN nodes are identified by "System-ID" 831 or Loopback addresses that are only locally significant. In some 832 SDWAN environments, "System-ID + PortID" are used to uniquely 833 identify a SDWAN WAN port. Sometimes, a SDWAN tunnel end-point can 834 be associated with "private IP" + "public IP" (if NAT is used.) 836 When CPE WAN ports are private addresses, an additional sub-TLV has 837 to be added to the [Tunnel-Encap] to describe the additional 838 information about the NAT property of SDWAN nodes' WAN ports. A 839 SDWAN node can inquire STUN (Session Traversal of UDP through 840 Network Address Translation [RFC 3489]) Server to get the NAT 841 property, the public IP address and the Public Port number to pass 842 to the authorized peers via the SDWAN Controller. 844 5.3.3. BGP Based SDWAN client routes 846 The client routes attached to SDWAN client ports have to be 847 distributed to all SDWAN edge nodes, just like BGP/MPLS IP VPN 848 [RFC4364], so that all SDWAN edges can establish the overlay routing 849 table that identifies the remote SDWAN edges to reach a specific 850 route/service. When C-PEs do not handle MPLS, RFC8365 can be used 851 for packets over WAN ports, albeit applying IPsec SA encryption when 852 sent over the WAN ports facing the public networks. 854 Using the terminologies described by [SECURE-L3VPN], the RED 855 interface are the clients' ports and the ports facing private 856 networks (e.g. connected to the PEs of MPLS VPN). Black Interfaces 857 are ports facing public networks. The behavior described in [SECURE- 858 L3VPN] applies to this scenario too, the C-PEs cannot mix the routes 859 learned from the Black Interfaces with the Routes from RED 860 Interfaces. 862 To minimize the burden on SDWAN edge nodes (especially low powered 863 virtual SDWAN edges), some SDWAN network can let SDWAN controller 864 take care of authenticating communications among SDWAN edge nodes 865 instead of pushing down policies to SDWAN edge nodes. SDWAN Edge 866 nodes might get clients routes from SDWAN controller instead of 867 learning from clients ports. 869 The Hybrid SDWAN control plane for distributing clients' routes is 870 more similar to overlay using EVPN [RFC8365], albeit the packets 871 sent over the internet facing ports have to be encrypted by IPsec 872 SA. 874 [Tunnel-Encap] can be used to associate client routes with specific 875 tunnels: 877 - C-PE1 can advertise the following properties to others C-PEs 878 via RR: 879 - Encapsulation capability of the Ports to VPN PE 880 - Encapsulation capability of the Ports to the Internet: 881 GRE-IPsec, or MPLS over GRE over IPsec 882 - with prior established IPsec SA 883 - NAT information if ports are private addresses 885 - The Remote Endpoint sub-TLV is NOT appropriate because 886 - The network to which a SDWAN port is connected might 887 have an identifier that is more than the AS number. The 888 SDWAN controller might use its own specific identifier 889 for the network. 890 - Suggest using an SDWAN overlay specific Transport- 891 Network-ID to represents the connected networks. 893 The underlay network selections to next hop C-PE can be a local 894 decision. Different services, routes, or VLANs can be aggregated to 895 one underlay network between two C-PEs; the same service/routes/VLAN 896 can spread over multiple SDWAN underlay networks at the next 897 segment. 899 5.4. Packet Walk-Through for Scenario #3 901 The behavior described in [SECURE-L3VPN] applies to this scenario, 902 except C-PEs not only have RED interfaces facing clients with within 903 the VPN but also have RED interface facing MPLS backbone, with 904 additional BLACK interfaces facing the untrusted public networks. 905 The C-PEs cannot mix the routes learned from the Black Interfaces 906 with the Routes from RED Interfaces. The routes learned from core- 907 facing RED interfaces are for underlay and cannot be mixed with the 908 routes learned over access-facing RED interfaces that are for 909 overlay. Furthermore, the routes learned over core-facing interfaces 910 (both RED and BLACK) can be shared in the same GLOBAL route table. 912 There may be some added risks of the packets from the ports facing 913 the Internet. Therefore, special consideration has to be given to 914 the routes from WAN ports facing the Internet. RFC4364 describes 915 using an RD to create different routes for reaching same system. A 916 similar approach can be considered to force packets received from 917 the Internet facing ports to go through special security functions 918 before being sent over to the VPN backbone WAN ports. 920 6. Manageability Considerations 922 SDWAN overlay networks utilize the SDWAN controller to facilitate 923 route distribution, central configurations, and others. To 924 minimize the burden on SDWAN edge nodes, SDWAN Edge nodes might 925 not need to learn the routes from clients. 927 7. Security Considerations 929 Having WAN ports facing the public Internet introduces the following 930 security risks: 932 1) Potential DDoS attack to the C-PEs with ports facing internet. 934 2) Potential risk of provider VPN network being injected with 935 illegal traffic coming from the public Internet WAN ports on the C- 936 PEs. 938 8. IANA Considerations 940 None 942 9. References 944 9.1. Normative References 946 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 947 Requirement Levels", BCP 14, RFC 2119, March 1997. 949 [RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private 950 networks (VPNs)", Feb 2006. 952 [RFC7296] C. Kaufman, et al, "Internet Key Exchange Protocol Version 953 2 (IKEv2)", Oct 2014. 955 [RFC7432] A. Sajassi, et al, "BGP MPLS-Based Ethernet VPN", Feb 956 2015. 958 [RFC8365] A. Sajassi, et al, "A network Virtualization Overlay 959 Solution Using Ethernet VPN (EVPN)", March 2018. 961 9.2. Informative References 963 [RFC8192] S. Hares, et al, "Interface to Network Security Functions 964 (I2NSF) Problem Statement and Use Cases", July 2017 966 [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation Subsequent 967 Address Family Identifier (SAFI) and the BGP Tunnel 968 Encapsulation Attribute", April 2009. 970 [BGP-SDWAN-Port] L. Dunbar, H. Wang, W. Hao, "BGP Extension for 971 SDWAN Overlay Networks", draft-dunbar-idr-bgp-sdwan- 972 overlay-ext-03, work-in-progress, Nov 2018. 974 [Net2Cloud-Gap] L. Dunbar, A. Malis, C. Jacquenet, "Gap Analysis of 975 Interconnecting Underlay with Cloud Overlay", draft-dm- 976 net2cloud-gap-analysis-02, work in progress, Oct. 2018. 978 [VPN-over-Internet] E. Rosen, "Provide Secure Layer L3VPNs over 979 Public Infrastructure", draft-rosen-bess-secure-l3vpn-00, 980 work-in-progress, July 2018 982 [DMVPN] Dynamic Multi-point VPN: 983 https://www.cisco.com/c/en/us/products/security/dynamic- 984 multipoint-vpn-dmvpn/index.html 986 [DSVPN] Dynamic Smart VPN: 987 http://forum.huawei.com/enterprise/en/thread-390771-1- 988 1.html 990 [SECURE-EVPN] A. Sajassi, et al, "Secure EVPN", draft-sajassi-bess- 991 secure-evpn-01, Work-in-progress, March 2019. 993 [SECURE-L3VPN] E. Rosen, R. Bonica, "Secure Layer L3VPN over Public 994 Infrastructure", draft-rosen-bess-secure-l3vpn-00, Work- 995 in-progress, June 2018. 997 [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation, 998 storage, distribution and enforcement of policies for 999 network security", Nov 2007. 1001 [Net2Cloud-Problem] L. Dunbar and A. Malis, "Seamless Interconnect 1002 Underlay to Cloud Overlay Problem Statement", draft-dm- 1003 net2cloud-problem-statement-02, June 2018 1005 [Net2Cloud-gap] L. Dunbar, A. Malis, and C. Jacquenet, "Gap Analysis 1006 of Interconnecting Underlay with Cloud Overlay", draft-dm- 1007 net2cloud-gap-analysis-02, work-in-progress, Aug 2018. 1009 [Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation 1010 Attribute", draft-ietf-idr-tunnel-encaps-10, Aug 2018. 1012 10. Acknowledgments 1014 Acknowledgements to Jim Guichard, John Scudder, Darren Dukes, Andy 1015 Malis and Donald Eastlake for their review and contributions. 1017 This document was prepared using 2-Word-v2.0.template.dot. 1019 Authors' Addresses 1021 Linda Dunbar 1022 Futurewei 1023 Email: ldunbar@futurewei.com 1025 James Guichard 1026 Futurewei 1027 Email: james.n.guichard@futurewei.com 1029 Ali Sajassi 1030 Cisco 1031 Email: sajassi@cisco.com 1033 John Drake 1034 Juniper 1035 Email: jdrake@juniper.net