idnits 2.17.1 draft-dunbar-bess-bgp-sdwan-usage-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 27 instances of too long lines in the document, the longest one being 5 characters in excess of 72. == There are 1 instance of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 8, 2019) is 1754 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC 4364' is mentioned on line 140, but not defined == Missing Reference: 'RFC4456' is mentioned on line 238, but not defined == Missing Reference: 'RFC4301' is mentioned on line 784, but not defined == Missing Reference: 'RFC 3489' is mentioned on line 807, but not defined ** Obsolete undefined reference: RFC 3489 (Obsoleted by RFC 5389) == Unused Reference: 'RFC2119' is defined on line 913, but no explicit reference was found in the text == Unused Reference: 'RFC7432' is defined on line 922, but no explicit reference was found in the text == Unused Reference: 'RFC8192' is defined on line 930, but no explicit reference was found in the text == Unused Reference: 'RFC5521' is defined on line 933, but no explicit reference was found in the text == Unused Reference: 'BGP-SDWAN-Port' is defined on line 937, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-Gap' is defined on line 941, but no explicit reference was found in the text == Unused Reference: 'VPN-over-Internet' is defined on line 945, but no explicit reference was found in the text == Unused Reference: 'DMVPN' is defined on line 949, but no explicit reference was found in the text == Unused Reference: 'DSVPN' is defined on line 953, but no explicit reference was found in the text == Unused Reference: 'ITU-T-X1036' is defined on line 964, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-gap' is defined on line 972, but no explicit reference was found in the text == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 == Outdated reference: A later version (-06) exists of draft-sajassi-bess-secure-evpn-01 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 -- Duplicate reference: draft-rosen-bess-secure-l3vpn, mentioned in 'SECURE-L3VPN', was also mentioned in 'VPN-over-Internet'. == Outdated reference: A later version (-07) exists of draft-dm-net2cloud-problem-statement-02 == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 -- Duplicate reference: draft-dm-net2cloud-gap-analysis, mentioned in 'Net2Cloud-gap', was also mentioned in 'Net2Cloud-Gap'. == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-10 Summary: 2 errors (**), 0 flaws (~~), 24 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft J. Guichard 3 Intended status: Informational Huawei 4 Expires: Dec 2019 Ali Sajassi 5 Cisco 6 J. Drake 7 Juniper 8 Ayan Barnerjee 9 D. Carrel 10 Cisco 12 July 8, 2019 14 BGP Usage for SDWAN Overlay Networks 15 draft-dunbar-bess-bgp-sdwan-usage-01 17 Abstract 19 The document describes three distinct SDWAN scenarios and discusses 20 the applicability of BGP for each of those scenarios. The goal of 21 the document is to make it easier for future SDWAN control plane 22 protocols discussion. 24 SDWAN edge nodes are commonly interconnected by multiple underlay 25 networks that are owned and managed by different network providers. 26 A BGP-based control plane is chosen for handling large number of 27 SDWAN edge nodes with little manual intervention. 29 Status of this Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. This document may not be modified, 36 and derivative works of it may not be created, except to publish it 37 as an RFC and to translate it into languages other than English. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF), its areas, and its working groups. Note that 41 other groups may also distribute working documents as Internet- 42 Drafts. 44 Internet-Drafts are draft documents valid for a maximum of six 45 months and may be updated, replaced, or obsoleted by other documents 46 at any time. It is inappropriate to use Internet-Drafts as 47 reference material or to cite them other than as "work in progress." 49 The list of current Internet-Drafts can be accessed at 50 http://www.ietf.org/ietf/1id-abstracts.txt 52 The list of Internet-Draft Shadow Directories can be accessed at 53 http://www.ietf.org/shadow.html 55 This Internet-Draft will expire on January 8, 2009. 57 Copyright Notice 59 Copyright (c) 2019 IETF Trust and the persons identified as the 60 document authors. All rights reserved. 62 This document is subject to BCP 78 and the IETF Trust's Legal 63 Provisions Relating to IETF Documents 64 (http://trustee.ietf.org/license-info) in effect on the date of 65 publication of this document. Please review these documents 66 carefully, as they describe your rights and restrictions with 67 respect to this document. Code Components extracted from this 68 document must include Simplified BSD License text as described in 69 Section 4.e of the Trust Legal Provisions and are provided without 70 warranty as described in the Simplified BSD License. 72 Table of Contents 74 1. Introduction...................................................3 75 2. Conventions used in this document..............................4 76 3. Use Case Scenario Description and Requirements.................5 77 3.1. Requirements..............................................6 78 3.1.1. Client Service Requirement...........................6 79 3.1.2. SDWAN Node Provisioning..............................6 80 3.2. Scenarios #1: Homogeneous WAN.............................8 81 3.3. Scenario #2: SDWAN WAN ports to VPN's PEs and to Internet.9 82 3.4. Scenario #3: SDWAN WAN ports to MPLS VPN and the Internet12 83 4. Provisioning Model............................................14 84 4.1. Client Service Provisioning Model........................14 85 4.2. WAN Ports Provisioning Model.............................14 86 4.2.1. Why BGP as Control Plane for SDWAN WAN Ports 87 Registration?..............................................15 88 5. SDWAN Traffic Forwarding Walk Through.........................16 89 5.1. SDWAN Network Startup Procedures.........................16 90 5.2. Packet Walk-Through for Scenario #1......................16 91 5.3. Packet Walk-Through for Scenario #2......................17 92 5.3.1. SDWAN node WAN Ports Properties Registration........19 93 5.3.2. Controller Facilitated IPsec SA & NAT management....19 94 5.3.3. BGP Based SDWAN client routes.......................21 95 5.4. Packet Walk-Through for Scenario #3......................22 96 6. Manageability Considerations..................................23 97 7. Security Considerations.......................................23 98 8. IANA Considerations...........................................23 99 9. References....................................................23 100 9.1. Normative References.....................................23 101 9.2. Informative References...................................24 102 10. Acknowledgments..............................................25 104 1. Introduction 106 An "SDWAN" network consists of many segments of parallel paths over 107 different underlay networks, some of which are private networks over 108 which traffic can traverse without encryption, others require 109 encryption over untrusted public networks. 111 [Net2Cloud-Problem] describes the network related problems that 112 enterprises face today in transitioning their IT infrastructure to 113 support a digital economy, such as the need to connect enterprises' 114 branch offices to dynamic workloads in different Cloud DCs, or 115 aggregating multiple paths provided by different service providers 116 to achieve better performance. 118 Even though SDWAN has been positioned as a flexible way to reach 119 dynamic workloads in third party data centers over multiple underlay 120 networks, scaling becomes a major issue when there are hundreds or 121 thousands of nodes to be interconnected by the SDWAN overlay paths. 123 BGP is widely used by underlay networks. This document describes 124 using BGP to enhance the scaling properties of SDWAN overlay 125 networks. 127 2. Conventions used in this document 129 Cloud DC: Third party data centers that host applications and 130 workloads owned by different organizations or tenants. 132 Controller: Used interchangeably with SDWAN controller to manage 133 SDWAN overlay path creation/deletion and monitor the 134 path conditions between sites. 136 CPE: Customer Premise Equipment 138 CPE-Based VPN: Virtual Private Secure network formed among CPEs. 139 This is to differentiate from more commonly used PE- 140 based VPNs [RFC 4364]. 142 Homogeneous SDWAN: A type of SDWAN network in which all traffic 143 to/from the SDWAN edge nodes has to be encrypted 144 regardless of underlay networks. For lack of better 145 terminology, we call this Homogeneous SDWAN throughout 146 this document. 148 ISP: Internet Service Provider 150 NSP: Network Service Provider. NSP usually provides more 151 advanced network services, such as MPLS VPN, private 152 leased lines, or managed Secure WAN connections, many 153 times within a private trusted domain, whereas an ISP 154 usually provides plain internet services over public 155 untrusted domains. 157 PE: Provider Edge 159 SDWAN End-point: a port (logical or physical) of a SDWAN edge node. 161 SDWAN: Software Defined Wide Area Network. In this document, 162 "SDWAN" refers to the solutions of pooling WAN bandwidth 163 from multiple underlay networks to get better WAN 164 bandwidth management, visibility & control. When the 165 underlay networks are private, traffic can traverse 166 without additional encryption; when the underlay 167 networks are public, such as the Internet, some traffic 168 may need to be encrypted when traversing through 169 (depending on user provided policies). 171 SDWAN IPsec SA: IPsec Security Association between two SDWAN ports 172 or nodes. 174 SDWAN over Hybrid Networks: SDWAN over Hybrid Networks typically 175 have edge nodes utilizing bandwidth resources from 176 multiple service providers. In Hybrid SDWAN network, 177 packets over private networks can go natively without 178 encryption and are encrypted over the untrusted network, 179 such as the public Internet. 181 WAN Port: A Port or Interface facing an ISP or Network Service 182 Provider (NSP), with address (usually public routable 183 address) allocated by the ISP or the NSP. 185 C-PE: SDWAN Edge node, which can be CPE for customer managed 186 SDWAN, or PE that is for provider managed SDWAN 187 services). 189 ZTP: Zero Touch Provisioning 191 3. Use Case Scenario Description and Requirements 193 SDWAN networks can have different topologies and have different 194 traffic patterns. To make it easier for the focused discussion in 195 subsequent drafts on SDWAN control plane and data plane, this 196 section describes several SDWAN scenarios that may have different 197 need or impact to their corresponding control planes & data planes. 199 3.1. Requirements 201 3.1.1. Client Service Requirement 203 Client interface of SDWAN nodes can be IP or Ethernet based. 205 For Ethernet based client interfaces, SDWAN edge should support 206 VLAN-based service interfaces (EVI100), VLAN bundle service 207 interfaces (EVI200), or VLAN-Aware bundling service interfaces. EVPN 208 service requirements are applicable to the Client traffic, as 209 described in the Section 3.1 of RFC8388. 211 For IP based client interfaces, L3VPN service requirements are 212 applicable. 214 3.1.2. SDWAN Node Provisioning 216 Unlike traditional EVPN or L3VPN where PEs are deployed for long 217 term, SDWAN edge nodes (virtual or physical) deployment at a 218 specific location can be ephemeral. Therefore, Zero Touch 219 Provisioning (ZTP) is a common requirement for SDWAN. ZTP for SDWAN 220 can include many areas, but from network connectivity perspective, 221 ZTP should include the following: 223 - Upon power up, an SDWAN node can reach a central SDWAN 224 Controller (which can be burned or preconfigured in the device) 225 via a TLS or SSL secure channel. 227 - The Central SDWAN Controller can designate a Local Network 228 Controller in the proximity of the SDWAN node; the Local Network 229 Controller and the SDWAN nodes might be connected by third party 230 untrusted network. The Local controller does all the following 4 231 tasks: 233 1) ZTP 234 2) Auto-discovery of Network 235 3) (Auto)-Provisioning for IPsec SAs (initial provisioning 236 part) 237 4) Signaling of tenant's routes/info 238 BGP is well suited for (4), using Route Reflector (RR) [RFC4456] 239 to propagate network information among SDWAN edge nodes. The SDWAN 240 node can establish a secure connection (TLS, SSL, etc) to the 241 Local Network Controller (RR). 243 +---+ 244 Peer Group 1 |RR | Peer Group 2 245 +======+====+=+ +======+====+=====+ 246 / / | +---+ | \ \ 247 / / | | \ \ 248 +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ 249 |C-PE| |C-PE|--|C-PE| |C-PE| |C-PE| |C-PE| 250 | 1 | | 2 | | 3 | |4 | | 5 | | 6 | 251 +----+ +----+ +----+ +----+ +----+ +----+ 252 Tenant 1 Tenant 2 253 Figure 1: Peer Groups managed by Local Controller 255 The SDWAN nodes (a.k.a. C-PEs throughout this document) belonging to 256 the same Tenant can be far apart and can be connected by third party 257 untrusted networks. Therefore, it is not appropriate for a SDWAN 258 edge node (C-PE) to advertise its SDWAN Port properties to its 259 neighbors. Each C-PE propagates its SDWAN Port attributes via the 260 secure channel (TLS, SSL, etc.) established with the Local 261 Controller. 263 C-PE-1 should include the following aspects in addition to managing 264 client routes: 265 - Register the SDWAN node's WAN port <-> local address mapping 266 to its Local Controller. The Local Controller propagates the 267 information to C-PE2 & C-PE3. 268 - Exchange IPsec property (capability such as the supported 269 encryption algorithms, etc.) and ports NAT property (e.g. 270 private addresses or dynamically assigned IP addresses) with 271 the Local Controller. 272 - C-PE2 and C-PE3 can establish IPsec SA with the C-PE1 after 273 receiving the information from the Local Controller. 274 - Then distribute the routes attached to the C-PE to its 275 authorized peers. 277 Tenant separation is achieved by the Local Controller creating 278 different Tenant based Peer Groups. 280 3.2. Scenarios #1: Homogeneous WAN 282 This is referring to a type of SDWAN network with edge nodes 283 encrypting all traffic over WAN to other edge nodes, regardless of 284 whether the underlay is private or public. For lack of better 285 terminology, we call this Homogeneous SDWAN throughout this 286 document. 288 Some typical scenarios for the use of a Homogeneous SDWAN network 289 are as follows: 291 - A small branch office connecting to its HQ offices via the 292 Internet. All sensitive traffic to/from this small branch office has 293 to be encrypted, which is usually achieved using IPsec SAs. 295 - A store in a shopping mall may need to securely connect to its 296 applications in one or more Cloud DCs via the Internet. A common way 297 of achieving this is to establish IPsec SAs to the Cloud DC gateway 298 to carry the sensitive data to/from the store. 300 As described in [SECURE-EVPN], the granularity of the IPsec SAs for 301 Homogeneous SDWAN can be per site, per subnet, per tenant, or per 302 address. Once the IPsec SA is established for a specific 303 subnet/tenant/site, all traffic to/from the subnets/tenants/site are 304 encrypted. 306 +---+ 307 +--------------|RR |------------+ 308 / Untrusted +-+-+ \ 309 / \ 310 / \ 311 +----+ +---------+ +------+ +----+ 312 | CN3|--| P1-----+ -------------+------ P1 |--| CN1| 313 +----+ | C-PE P2-----+ | | C-PE | +----+ 314 +----+ | A P3-----+ Wide +------ P2 B | +----+ 315 | CN2|--| | | area +------ P3 |--| CN3| 316 +----+ +---------+ | network | +------+ +----+ 317 | | 318 +----+ +---------+ | all packets | +------+ +----+ 319 | CN1|--| P1-----+ encrypted +------ P1 |--| CN1| 320 +----+ | C-PE P2-----+ by | | C-PE | +----+ 321 +----+ | C P3-----+ IPsec SAs +------ P2 D | +----+ 322 | CN2|--| P4-----+--------------+ | |--| CN2| 323 +----+ +---------+ +------+ +----+ 325 CN: Client Networks, which is same as Tenant Networks used by NVo3 327 Figure 1: Homogeneous SDWAN 329 One of the key properties of homogeneous SDWAN is that the SDWAN 330 Local Network Controller (RR)is connected to C-PEs via untrusted 331 public network, therefore, requiring secure connection between RR 332 and C-PEs (TLS, DTLS, etc.). 334 Homogeneous SDWAN has some similarity to commonly deployed IPsec 335 VPN, albeit the IPsec VPN is usually point-to-point among a small 336 number of endpoints and with heavy manual configuration for IPsec 337 between end-points, whereas an SDWAN network can have a large number 338 of end-points with an SDWAN controller to manage requiring zero 339 touch provisioning upon powering up. 341 Existing Private VPNs (e.g. MPLS based) can use homogeneous SDWAN to 342 extend over public network to remote sites to which the VPN operator 343 does not own or lease infrastructural connectivity, as described in 344 [SECURE-EVPN] and [SECURE-L3VPN] 346 3.3. Scenario #2: SDWAN WAN ports to VPN's PEs and to Internet 348 In this scenario, SDWAN edge nodes (a.k.a. C-PEs) have some WAN 349 ports connected to PEs of Private VPNs over which packets can be 350 forwarded natively without encryption, and some WAN ports connected 351 to the Internet over which sensitive traffic have to be encrypted 352 (usually by IPsec SA). 354 In this scenario, the SDWAN edge nodes' egress WAN ports are all 355 IP/Ethernet based, either egress to PEs of the VPNs or to the 356 Internet. Even if the VPN is a MPLS network, the VPN's PEs have 357 IP/Ethernet connections to the SDWAN edge (C-PEs). Throughout this 358 document, this scenario is also called CPE based SDWAN over Hybrid 359 Networks. 361 Even though IPsec SA can secure the packets traversing the Internet, 362 it does not offer the premium SLA commonly offered by Private VPNs, 363 especially over long distance. Clients need to have policies to 364 specify criteria for flows only traversing private VPNs or 365 traversing either as long as encrypted when over the Internet. For 366 example, client can have those polices for the flows: 368 1. A policy or criteria for sending the flows over a private network 369 without encryption (for better performance), 370 2. A policy or criteria for sending the flows over any networks as long 371 as the packets of the flows are encrypted when traversing untrusted 372 networks, or 373 3. A policy of not needing encryption at all. 375 If a flow traversing multiple segments, such as A<->B<->C<->D, has 376 either Policy 2 or 3 above, the flow can traverse different 377 underlays in different segments, such as over Private network 378 underlay between A<->B without encryption, or over the public 379 internet between B<->C in an IPsec SA. 381 As shown in the figure below, C-PE-1 has two different types of 382 interfaces (A1 to Internet and A2 & A3 to VPN). The C-PEs' loopback 383 addresses and addresses attached to C-PEs may or may not be visible 384 to the ISPs/NSPs. The addresses for the WAN ports can have addresses 385 allocated by the service providers or dynamically assigned (e.g. by 386 DHCP). One WAN port shown in the figure below (e.g. A1, A2, A3 etc.) 387 is a logical representation of potential multiple physical ports on 388 the C-PEs. 390 +---+ 391 +--------------|RR |----------+ 392 / Untrusted +-+-+ \ 393 / \ 394 / \ 395 +----+ +---------+ packets encrypted over +------+ +----+ 396 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 397 +----+ | C-PE A2-\ | C-PE | +----+ 398 +----+ | A A3--+--+ +---+---B2 B | +----+ 399 | CN2|--| | |PE+--------------+PE |---B3 |--| CN3| 400 +----+ +---------+ +--+ trusted +---+ +------+ +----+ 401 | WAN | 402 +----+ +---------+ +--+ packets +---+ +------+ +----+ 403 | CN1|--| C1--|PE| go natively |PE |-- D1 |--| CN1| 404 +----+ | C-PE C2--+--+ without encry+---+ | C-PE | +----+ 405 | C | +--------------+ | D | 406 | | | | 407 +----+ | | without encrypt over | | +----+ 408 | CN2|--| C3--+---- Untrusted --+------D2 |--| CN2| 409 +----+ +---------+ +------+ +----+ 411 CN: Client Network 412 Figure 2: Hybrid SDWAN 414 Some key characteristics of a Hybrid SDWAN overlay network are as 415 follows: 417 - one C-PE may be connected to different ISPs/NSPs, with some of its 418 WAN ports addresses being assigned by the ISPs/NSPs. 420 - The WAN ports connected to PEs of trusted private networks (e.g. MPLS 421 VPN) hand off IP/Ethernet packets, just like today's CPE that do not 422 handle MPLS packets and do not participate in the underlay VPN networks' 423 control plane. Traffic can flow natively without encryption when be 424 forwarded out through those WAN ports for better performance. 426 - The WAN ports connected to untrusted networks, e.g. the Internet, 427 requires sensitive traffic to be encrypted, i.e. encrypted by IPsec SA. 429 - An SDWAN local Network Controller (RR) is connected to C-PEs via 430 the untrusted public network, therefore, requiring secure 431 connection between RR and C-PEs via TLS, DTLS, etc. 433 - The SDWAN nodes' [loopback] addresses might not be routable nor 434 visible in the underlay ISP/NSP networks. Routes & services 435 attached to SDWAN edges at the SDWAN overlay layer are in 436 different address spaces than the underlay networks. 438 - There could be multiple SDWAN devices sharing a common property, 439 such as a geographic location. Some applications over SDWAN may 440 need to traverse specific geographic locations for various 441 reasons, such as to comply regulatory rules, to utilize specific 442 value added services, or others. 444 - The underlay path selection between sites can be a local section. 445 Some policies allow one service from CPE1 -> CPE2 -> CPE3 using 446 one ISP/NSP underlay in the first segment (CPE1 -> CPE2), and 447 using a different ISP/NSP in the second segment (CPE2-> CPE3). 449 - Services may not be congruent, i.e. the packets from A-> B may 450 traverse one underlay network, and the packets from B -> A may 451 traverse a different underlay. 453 - Different services, routes, or VLANs attached to SDWAN nodes can 454 be aggregated over one underlay path; same service/routes/VLAN can 455 spread over multiple SDWAN underlays at different times depending 456 on the policies specified for the service. For example, one 457 tenant's packets to HQ need to be encrypted when sent over the 458 Internet or have to be sent over private networks, while the same 459 tenant's packets to Facebook can be sent over the Internet without 460 encryption. 462 3.4. Scenario #3: SDWAN WAN ports to MPLS VPN and the Internet 464 This scenario refers to existing VPN (e.g. MPLS based VPN, such as 465 EVPN or IPVPN) adding extra ports facing untrusted public networks 466 allowing PEs to offload some low priority traffic to those ports 467 facing public networks when the VPN MPLS paths are congested. 468 Throughout this document, this scenario is also called Internet 469 Offload for Private VPN, or PE based SDWAN. 471 In this scenario, it is important that the packets offloaded to 472 untrusted public network be encrypted. In this scenario, there is a 473 secure BGP connection between RR & PEs. 475 PE based SDWAN can be used by VPN service providers to temporarily 476 increase bandwidth between sites when they are not sure if the 477 demand will sustain for long period of time or as a temporary 478 solution before the permanent infrastructure is built or leased. 480 +---+ 481 +-------|PE2| 482 / +---+ 483 Internet / ^ 484 offload / || VPN 485 / VPN v 486 ++--+ ++-+ +---+ 487 |PE1| <====> |RR| <===> |PE3| 488 +-+-+ +--+ +-+-+ 489 | | 490 +--- Public Internet -- + 492 Figure 3: Additional Internet paths added to the VPN 494 Here are some key properties for PE based SDWAN: 496 - For MPLS based VPN, PEs continue having MPLS encapsulation 497 handoff to existing paths. 498 - The BGP RR is connected to PEs in the same way as VPN, i.e. 499 via the trusted network. 500 - For the added Internet ports, PEs have IP packets handoff, 501 i.e. sending and receiving IP data frames. Internally, PEs 502 can have the option to encapsulate the MPLS payload in IP, as 503 specified by RFC4023. 504 - The ports facing public internet might get IP addresses 505 assigned by ISPs, which may not be in the same address domain 506 as PEs'. 508 - Ports facing public internet are not as secure as the ports 509 facing private infrastructure. There could be spoofing, or 510 DDOS attacks to the ports facing public internet. Extra 511 consideration must be given when injecting the new routes 512 from public network into VRFs. 513 - Even though packets are encrypted over public internet, the 514 performance SLA is not guaranteed over public internet. 515 Therefore, clients may have policies only allowing some flows 516 to be offloaded to internet path. 518 4. Provisioning Model 520 4.1. Client Service Provisioning Model 522 The provisioning tasks described in Section 4 of RFC8388 are the 523 same for the SDWAN client traffic. When client traffic are multi- 524 homed to two (or more) C-PEs, the Non-Service-Specific parameters 525 need to be provisioned per the Section 4.1.1 of RFC8388. 527 Since most SDWAN nodes are ephemeral and have small number of IP 528 subnets or VLANs attached to the client ports of the SDWAN nodes, it 529 is recommended to have default and simplified Service-specific 530 parameters for each client port, remotely managed by the SDWAN 531 Network Controller (i.e. the RR) via the secure channel (TLS/DTLS) 532 between the controller and the C-PEs. 534 More details are to be added. 536 4.2. WAN Ports Provisioning Model 538 Since the deployment of PEs to MPLS VPN are for relatively long 539 term, the common provisioning procedure for PE's WAN ports is via 540 CLI. 542 A SDWAN node deployment can be ephemeral and its location can be in 543 remote locations, manual provisioning for its WAN ports is not 544 acceptable. In addition, a SDWAN WAN port's IP address can be 545 dynamically assigned or using private addresses. Therefore, it is 546 necessary to have a separate control protocol; something like NHRP 547 did for ATM, for a SDWAN node to register its WAN property to its 548 controller dynamically. 550 Unlike a PE to MPLS based VPN where its WAN ports are homogeneously 551 facing MPLS private network and all traffic are egressed in MPLS 552 data frames through its WAN ports, the WAN ports of a SDWAN node can 553 be connected to a PE of VPN, MPLS private network directly, the 554 public Internet, or the various combinations of all. 556 For Scenario #1 above, the WAN ports can face public internet or 557 VPN. 559 For Scenario #2 above, WAN ports are either configured as connecting 560 to PEs of VPN where traffic can be sent as IP/Ethernet without 561 encryption, or configured as connecting to public Internet. 563 For Scenario #3 above, the WAN ports are either configured as VPN 564 egress ports (hand off MPLS data frames), or as connecting to the 565 public internet that requires MPLS in IP in IPsec encapsulation. 567 4.2.1. Why BGP as Control Plane for SDWAN WAN Ports Registration? 569 For a small sized SDWAN network, traditional hub & spoke model using 570 NHRP or DSVPN/DMVPN with a hub node (or controller) managing SDWAN 571 node WAN ports mapping (e.g. local & public addresses and tunnel 572 identifiers mapping) can work reasonably well. However, for a large 573 SDWAN network, say more than 100 nodes with different types of 574 topologies, the traditional approach becomes very messy, complex and 575 error prone. 577 Here are some of the compelling reasons of using BGP instead of 578 extending NHRP/DSVPN/DMVPN. (Same as the reasons quoted by LSVR on 579 why using BGP): 581 - BGP already widely deployed as sole protocol (see RFC 7938) 583 - Robust and simple implementation 585 - Wide acceptance - minimal learning 587 - Reliable transport 589 - Guaranteed in-order delivery 591 - Incremental updates 593 - 595 - Incremental updates upon session restart 596 - No flooding and selective filtering 598 - RR already has the capability to apply policies to communications 599 among peers. 601 5. SDWAN Traffic Forwarding Walk Through 603 BGP based EVPN control plane are still applicable to routes attached 604 to the client ports of SDWAN nodes. Section 5 of RFC8388 describes 605 the BGP EVPN NLRI Usage for various routes of client traffic. The 606 procedures described in the Section 6 of RFC8388 are same for the 607 SDWAN client traffic. 609 The only additional consideration for SDWAN is to control how 610 traffic egress the SDWAN edge node to various WAN ports. 612 5.1. SDWAN Network Startup Procedures 614 A SDWAN network can add or delete SDWAN edge nodes on regular basis 615 depending on user requests. 617 - For Scenario #1: a SDWAN edge node in a shopping mall or Cloud DC can 618 be added or removed on demand. The Zero Touch Provisioning described 619 in 3.1.2 are required for the node startup. 620 - For Scenario #2: this can be Data Centers or enterprises upgrading 621 their CPEs to add extra bandwidth via public internet in addition to 622 VPN services that they already purchased. Before the node powers up 623 or upgraded, there should be links connected to the PEs of a provider 624 VPNs. 625 - For Scenario #3, the Internet facing WAN ports are added to (or 626 removed from) existing VPN PEs. 628 5.2. Packet Walk-Through for Scenario #1 630 Upon power up, a SDWAN node can learn client routes from the Client 631 facing ports, in the same way as EVPN described in RFC8388. 632 Controller facilitates the IPsec SA establishment and rekey 633 management as described in [SECURE-EVPN]. Controller manages how 634 client's routes are associated with individual IPSec SA. 636 [SECURE-L3VPN] describes how to extend the RFC4364 VPN to allow some 637 PEs being connected to other PEs via public networks. [SECURE-L3VPN] 638 introduces the concept of Red Interface & Black Interface on those 639 PEs, with RED interfaces face clients' routes within the VPN and the 640 Black Interfaces being WAN ports over which only IPsec-protected 641 packets to the Internet or other backbone network are sent so that 642 eliminating the need for MPLS transport in the backbone. 644 [SECURE-L3VPN] assumes PEs terminate MPLS packets, and use MPLS over 645 IPsec when sending over the Black Interfaces. 647 [SECURE-EVPN] describes a solution where BGP point-to-multipoint 648 signaling is leveraged as control plane for SDWAN Scenario #1. It 649 utilizes the BGP RR to facilitate the key and policy exchange among 650 PE devices to create private pair-wise IPsec Security Associations 651 without IKEv2 point-to-point signaling or any other direct peer-to- 652 peer session establishment messages. 654 When C-PEs do not support MPLS, the approaches described by RFC8365 655 can be used, with addition of IPsec encrypting the IP packets when 656 sending packets over the Black Interfaces. 658 5.3. Packet Walk-Through for Scenario #2 660 In this scenario, C-PEs have some WAN ports connected to the public 661 internet and some WAN ports connected to (i.e. directly connected 662 to) PEs of trusted VPN. The C-PEs in Scenario #2 are almost like 663 CPEs to MPLS VPN that have the IP/Ethernet data frames egress to the 664 PEs of the VPN, except the packets need encryption if egress to the 665 WAN ports facing public Internet. 667 Users specify the policy or criteria on which flows can only egress 668 WAN ports facing trusted VPN without encryption, which can egress 669 the WAN ports facing the public Internet with encryption, or which 670 can egress WAN ports facing the public Internet without encryption. 672 The Control Plane should not learn routes from the Public Network 673 facing WAN ports. Should strictly follow the policies specified by 674 the users. The internet facing WAN ports can face potential DDoS 675 attacks, additional anti-DDoS mechanism has to be implemented on WAN 676 ports facing those public networks. 678 The Scenario #2 SDWAN Control Plane has three distinct functional 679 components: 681 +---+ 682 +--------------|RR |----------+ 683 / Untrusted +-+-+ \ 684 / Network \ 685 / \ 686 +----+ +---------+ packets encrypted over +--------+ +----+ 687 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 688 +----+ | C-PE1 A2-----+ +-------B2 C-PE2 | +----+ 689 |10.1.1.1 | |10.1.2.1| 690 +----+ | | +--+ +---+ | | +----+ 691 | CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3| 692 +----+ +---------+ +--+ trusted +---+ +--------+ +----+ 693 | VPN | 694 +--------------+ 695 Figure 5: SDWAN Scenario #2 697 - SDWAN node's WAN ports property registration to the SDWAN 698 Network Controller (BGP RR). 699 o This is used to inform the SDWAN controller of all the 700 underlay networks to which the C-PE is connected. 701 o RR is responsible for propagating the C-PE WAN ports 702 properties to authorized peers. 704 - Controller Facilitated IPsec SA management and NAT information 705 distribution 706 o Used by the SDWAN controller to facilitate or manage the 707 IPsec configurations and peer authentications for all 708 IPsec SAs terminated at the SDWAN nodes. 709 o When WAN ports have private addresses, need exchange 710 between SDWAN edges and the RR about the type of NAT, and 711 mapping of the private addresses/ports <-> public 712 addresses/ports. 714 - Attached routes distribution via BGP RR, which can be EVPN, 715 IPVPN or others. 716 o This is for the overlay layer's route distribution, so 717 that a C-PE can establish the overlay routing table that 718 identifies the next hop for reaching a specific 719 route/service attached to remote nodes. [SECURE-EVPN] 720 describes EVPN and other options. 722 5.3.1. SDWAN node WAN Ports Properties Registration 724 In Figure 6, A1/A2/A3/B1/B2/B3 WAN ports can be from different 725 network providers. 727 +---+ 728 10.1.1.1 via +--------------|RR |----------+ 10.1.2.1 via 729 A1/A2/A3 / Untrusted +-+-+ \ B1/B2/B3 730 / \ 731 / \ 732 +----+ +---------+ packets encrypted over +--------+ +----+ 733 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 734 +----+ | C-PE1 A2-----+ +-------B2 C-PE2 | +----+ 735 |10.1.1.1 | |10.1.2.1| 736 +----+ | | +--+ +---+ | | +----+ 737 | CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3| 738 +----+ +---------+ +--+ trusted +---+ +--------+ +----+ 739 | VPN | 740 +--------------+ 741 Figure 6: SDWAN Scenario #2 WAN Ports Registration 743 Each SDWAN edge(C-PE) needs to register its WAN ports properties 744 along with its Loopback addresses to the SDWAN Network Controller 745 (RR). The policies that govern the communications among peers are 746 managed and controlled by the SDWAN Controller. Individual SDWAN 747 edge relies on its SDWAN Controller to determine which peers can 748 establish connections. The SDWAN controller is responsible for 749 propagating the mapping information to the authorized peers. If C- 750 PE-1 is not authorized to communicate with C-PE-n, C-PE-1's WAN 751 port<->Loopback address mapping will not be propagated to C-PE-n. 753 A C-PE's Loopback addresses & attached routes may not be visible to 754 some ISPs/NSPs to which the CPE's WAN port is connected. 756 5.3.2. Controller Facilitated IPsec SA & NAT management 758 One IPsec SA between two end points is straightforward. However, for 759 a network with many IPsec SAs among many end points, the 760 configuration and IPsec Key management for the entire network can be 761 complex. 763 For a 1,000-node network, each node is responsible for maintaining 764 and managing 999 keys to all their peers, which could potentially 765 result in 1,000,000 key exchanges to authenticate among all nodes. 766 In addition, when an edge node has multiple tenants attached, the 767 edge node may need to establish multiple tunnels for tenants. For 768 example, for a network with N nodes, a node A has 5 tenants app 769 attached to it, then the node A has to maintain 5*(N-1) number of 770 keys if each tenant needs to communicate with all other nodes. 772 In addition, all the IPsec keys have to be refreshed periodically, 773 which adds more complexity. Therefore, simplification facilitated by 774 an SDWAN controller is necessary for large-scale SDWAN deployment. 776 When the SDWAN IPsec SAs are fine-grained, such as per client 777 address, per client's VLAN, the number of IPsec SAs & Keys to be 778 managed can go much higher, leading to more IPsec management 779 complexity. It is better to aggregate multiple flows into one IPsec 780 SA. 782 SDWAN edge nodes can rely on the SDWAN controller to facilitate the 783 pair-wise IPsec key establishment and refreshment [RFC7296] and 784 maintain the Security Policy Database (SPD) [RFC4301]. 786 - In the Figure 5 SDWAN Scenario #2 above, if C-PE1 & C-PE2 each has 787 two ports facing two different ISPs networks, and their loopback 788 addresses are not visible to the ISPs, i.e. the C-PE1 & C-PE2 are 789 using a provider assigned IP addresses for A1/A2/B1/B2; you are 790 going to need minimum four IPsec SAs between C-PE1 & C-PE2. 791 - When C-PEs loopback addresses are visible to ISPs/NSPs, i.e. the 792 C-PEs' private source and destination IPs are part of a prefix 793 exported to the ISP(s) in each site, it is possible to have one 794 IPsec SA between C-PE1 & C-PE2. 796 The IP addresses of SDWAN WAN port can be dynamic (e.g. assigned by 797 DHCP) or private IP. Some SDWAN nodes are identified by "System-ID" 798 or Loopback addresses that are only locally significant. In some 799 SDWAN environments, "System-ID + PortID" are used to uniquely 800 identify a SDWAN WAN port. Sometimes, a SDWAN tunnel end-point can 801 be associated with "private IP" + "public IP" (if NAT is used.) 803 When CPE WAN ports are private addresses, an additional sub-TLV has 804 to be added to the [Tunnel-Encap] to describe the additional 805 information about the NAT property of SDWAN nodes' WAN ports. A 806 SDWAN node can inquire STUN (Session Traversal of UDP through 807 Network Address Translation [RFC 3489]) Server to get the NAT 808 property, the public IP address and the Public Port number to pass 809 to the authorized peers via the SDWAN Controller. 811 5.3.3. BGP Based SDWAN client routes 813 The client routes attached to SDWAN client ports have to be 814 distributed to all SDWAN edge nodes, just like BGP/MPLS IP VPN 815 [RFC4364], so that all SDWAN edges can establish the overlay routing 816 table that identifies the remote SDWAN edges to reach a specific 817 route/service. When C-PEs do not handle MPLS, RFC8365 can be used 818 for packets over WAN ports, albeit applying IPsec SA encryption when 819 sent over the WAN ports facing the public networks. 821 Using the terminologies described by [SECURE-L3VPN], the RED 822 interface are the clients' ports and the ports facing private 823 networks (e.g. connected to the PEs of MPLS VPN). Black Interfaces 824 are ports facing public networks. The behavior described in [SECURE- 825 L3VPN] applies to this scenario too, the C-PEs cannot mix the routes 826 learned from the Black Interfaces with the Routes from RED 827 Interfaces. 829 To minimize the burden on SDWAN edge nodes (especially low powered 830 virtual SDWAN edges), some SDWAN network can let SDWAN controller 831 take care of authenticating communications among SDWAN edge nodes 832 instead of pushing down policies to SDWAN edge nodes. SDWAN Edge 833 nodes might get clients routes from SDWAN controller instead of 834 learning from clients ports. 836 The Hybrid SDWAN control plane for distributing clients' routes is 837 more similar to overlay using EVPN [RFC8365], albeit the packets 838 sent over the internet facing ports have to be encrypted by IPsec 839 SA. 841 [Tunnel-Encap] can be used to associate client routes with specific 842 tunnels: 844 - C-PE1 can advertise the following properties to others C-PEs 845 via RR: 846 - Encapsulation capability of the Ports to VPN PE 847 - Encapsulation capability of the Ports to the Internet: 848 GRE-IPsec, or MPLS over GRE over IPsec 849 - with prior established IPsec SA 850 - NAT information if ports are private addresses 852 - The Remote Endpoint sub-TLV is NOT appropriate because 853 - The network to which a SDWAN port is connected might 854 have an identifier that is more than the AS number. The 855 SDWAN controller might use its own specific identifier 856 for the network. 857 - Suggest using an SDWAN overlay specific Transport- 858 Network-ID to represents the connected networks. 860 The underlay network selections to next hop C-PE can be a local 861 decision. Different services, routes, or VLANs can be aggregated to 862 one underlay network between two C-PEs; the same service/routes/VLAN 863 can spread over multiple SDWAN underlay networks at the next 864 segment. 866 5.4. Packet Walk-Through for Scenario #3 868 The behavior described in [SECURE-L3VPN] applies to this scenario, 869 except C-PEs not only have RED interfaces facing clients with within 870 the VPN but also have RED interface facing MPLS backbone, with 871 additional BLACK interfaces facing the untrusted public networks. 872 The C-PEs cannot mix the routes learned from the Black Interfaces 873 with the Routes from RED Interfaces. The routes learned from core- 874 facing RED interfaces are for underlay and cannot be mixed with the 875 routes learned over access-facing RED interfaces that are for 876 overlay. Furthermore, the routes learned over core-facing interfaces 877 (both RED and BLACK) can be shared in the same GLOBAL route table. 879 There may be some added risks of the packets from the ports facing 880 the Internet. Therefore, special consideration has to be given to 881 the routes from WAN ports facing the Internet. RFC4364 describes 882 using an RD to create different routes for reaching same system. A 883 similar approach can be considered to force packets received from 884 the Internet facing ports to go through special security functions 885 before being sent over to the VPN backbone WAN ports. 887 6. Manageability Considerations 889 SDWAN overlay networks utilize the SDWAN controller to facilitate 890 route distribution, central configurations, and others. To 891 minimize the burden on SDWAN edge nodes, SDWAN Edge nodes might 892 not need to learn the routes from clients. 894 7. Security Considerations 896 Having WAN ports facing the public Internet introduces the following 897 security risks: 899 1) Potential DDoS attack to the C-PEs with ports facing internet. 901 2) Potential risk of provider VPN network being injected with 902 illegal traffic coming from the public Internet WAN ports on the C- 903 PEs. 905 8. IANA Considerations 907 None 909 9. References 911 9.1. Normative References 913 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 914 Requirement Levels", BCP 14, RFC 2119, March 1997. 916 [RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private 917 networks (VPNs)", Feb 2006. 919 [RFC7296] C. Kaufman, et al, "Internet Key Exchange Protocol Version 920 2 (IKEv2)", Oct 2014. 922 [RFC7432] A. Sajassi, et al, "BGP MPLS-Based Ethernet VPN", Feb 923 2015. 925 [RFC8365] A. Sajassi, et al, "A network Virtualization Overlay 926 Solution Using Ethernet VPN (EVPN)", March 2018. 928 9.2. Informative References 930 [RFC8192] S. Hares, et al, "Interface to Network Security Functions 931 (I2NSF) Problem Statement and Use Cases", July 2017 933 [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation Subsequent 934 Address Family Identifier (SAFI) and the BGP Tunnel 935 Encapsulation Attribute", April 2009. 937 [BGP-SDWAN-Port] L. Dunbar, H. Wang, W. Hao, "BGP Extension for 938 SDWAN Overlay Networks", draft-dunbar-idr-bgp-sdwan- 939 overlay-ext-03, work-in-progress, Nov 2018. 941 [Net2Cloud-Gap] L. Dunbar, A. Malis, C. Jacquenet, "Gap Analysis of 942 Interconnecting Underlay with Cloud Overlay", draft-dm- 943 net2cloud-gap-analysis-02, work in progress, Oct. 2018. 945 [VPN-over-Internet] E. Rosen, "Provide Secure Layer L3VPNs over 946 Public Infrastructure", draft-rosen-bess-secure-l3vpn-00, 947 work-in-progress, July 2018 949 [DMVPN] Dynamic Multi-point VPN: 950 https://www.cisco.com/c/en/us/products/security/dynamic- 951 multipoint-vpn-dmvpn/index.html 953 [DSVPN] Dynamic Smart VPN: 954 http://forum.huawei.com/enterprise/en/thread-390771-1- 955 1.html 957 [SECURE-EVPN] A. Sajassi, et al, "Secure EVPN", draft-sajassi-bess- 958 secure-evpn-01, Work-in-progress, March 2019. 960 [SECURE-L3VPN] E. Rosen, R. Bonica, "Secure Layer L3VPN over Public 961 Infrastructure", draft-rosen-bess-secure-l3vpn-00, Work- 962 in-progress, June 2018. 964 [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation, 965 storage, distribution and enforcement of policies for 966 network security", Nov 2007. 968 [Net2Cloud-Problem] L. Dunbar and A. Malis, "Seamless Interconnect 969 Underlay to Cloud Overlay Problem Statement", draft-dm- 970 net2cloud-problem-statement-02, June 2018 972 [Net2Cloud-gap] L. Dunbar, A. Malis, and C. Jacquenet, "Gap Analysis 973 of Interconnecting Underlay with Cloud Overlay", draft-dm- 974 net2cloud-gap-analysis-02, work-in-progress, Aug 2018. 976 [Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation 977 Attribute", draft-ietf-idr-tunnel-encaps-10, Aug 2018. 979 10. Acknowledgments 981 Acknowledgements to Jim Guichard, John Scudder, Darren Dukes, Andy 982 Malis and Donald Eastlake for their review and contributions. 984 This document was prepared using 2-Word-v2.0.template.dot. 986 Authors' Addresses 988 Linda Dunbar 989 Huawei 990 Email: ldunbar@futurewei.com 992 James Guichard 993 Huawei 994 Email: james.n.guichard@futurewei.com 996 Ali Sajassi 997 Cisco 998 Email: sajassi@cisco.com 1000 John Drake 1001 Juniper 1002 Email: jdrake@juniper.net