idnits 2.17.1 draft-ietf-bess-bgp-sdwan-usage-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 14 instances of too long lines in the document, the longest one being 6 characters in excess of 72. == There are 11 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (January 22, 2021) is 1189 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC 4364' is mentioned on line 157, but not defined == Missing Reference: 'RFC4456' is mentioned on line 337, but not defined == Missing Reference: 'Tunnel-encap' is mentioned on line 615, but not defined == Missing Reference: 'RFC4684' is mentioned on line 771, but not defined == Missing Reference: 'RFC4023' is mentioned on line 884, but not defined == Missing Reference: 'RFC3715' is mentioned on line 899, but not defined == Missing Reference: 'RFC3948' is mentioned on line 901, but not defined == Missing Reference: 'RFC3947' is mentioned on line 904, but not defined == Unused Reference: 'RFC2119' is defined on line 946, but no explicit reference was found in the text == Unused Reference: 'RFC4364' is defined on line 949, but no explicit reference was found in the text == Unused Reference: 'RFC7296' is defined on line 952, but no explicit reference was found in the text == Unused Reference: 'RFC7432' is defined on line 955, but no explicit reference was found in the text == Unused Reference: 'RFC8365' is defined on line 958, but no explicit reference was found in the text == Unused Reference: 'RFC8192' is defined on line 963, but no explicit reference was found in the text == Unused Reference: 'RFC5521' is defined on line 966, but no explicit reference was found in the text == Unused Reference: 'RFC8388' is defined on line 970, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-Gap' is defined on line 973, but no explicit reference was found in the text == Unused Reference: 'VPN-over-Internet' is defined on line 981, but no explicit reference was found in the text == Unused Reference: 'DMVPN' is defined on line 985, but no explicit reference was found in the text == Unused Reference: 'DSVPN' is defined on line 989, but no explicit reference was found in the text == Unused Reference: 'ITU-T-X1036' is defined on line 1000, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-gap' is defined on line 1008, but no explicit reference was found in the text == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 == Outdated reference: A later version (-04) exists of draft-dunbar-idr-sdwan-edge-discovery-01 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 == Outdated reference: A later version (-06) exists of draft-sajassi-bess-secure-evpn-01 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 -- Duplicate reference: draft-rosen-bess-secure-l3vpn, mentioned in 'SECURE-L3VPN', was also mentioned in 'VPN-over-Internet'. == Outdated reference: A later version (-07) exists of draft-dm-net2cloud-problem-statement-02 == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 -- Duplicate reference: draft-dm-net2cloud-gap-analysis, mentioned in 'Net2Cloud-gap', was also mentioned in 'Net2Cloud-Gap'. == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-10 Summary: 1 error (**), 0 flaws (~~), 32 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft J. Guichard 3 Intended status: Informational Futurewei 4 Expires: July 22, 2021 Ali Sajassi 5 Cisco 6 J. Drake 7 Juniper 8 B. Najem 9 Bell Canada 10 Ayan Barnerjee 11 D. Carrel 12 IPsec Research 14 January 22, 2021 16 BGP Usage for SDWAN Overlay Networks 17 draft-ietf-bess-bgp-sdwan-usage-02 19 Abstract 21 The document discusses the usage and applicability of BGP as control 22 plane for multiple SDWAN scenarios. The goal of the document is to 23 demonstrate how BGP-based control plane is used for large scale 24 SDWAN overlay networks with little manual intervention. 26 SDWAN edge nodes are commonly interconnected by multiple underlay 27 networks which can be owned and managed by different network 28 providers. 30 Status of this Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 This Internet-Draft is submitted in full conformance with the 36 provisions of BCP 78 and BCP 79. This document may not be modified, 37 and derivative works of it may not be created, except to publish it 38 as an RFC and to translate it into languages other than English. 40 Internet-Drafts are working documents of the Internet Engineering 41 Task Force (IETF), its areas, and its working groups. Note that 42 other groups may also distribute working documents as Internet- 43 Drafts. 45 Internet-Drafts are draft documents valid for a maximum of six 46 months and may be updated, replaced, or obsoleted by other documents 47 at any time. It is inappropriate to use Internet-Drafts as 48 reference material or to cite them other than as "work in progress." 50 The list of current Internet-Drafts can be accessed at 51 http://www.ietf.org/ietf/1id-abstracts.txt 53 The list of Internet-Draft Shadow Directories can be accessed at 54 http://www.ietf.org/shadow.html 56 This Internet-Draft will expire on July 22, 2021. 58 Copyright Notice 60 Copyright (c) 2021 IETF Trust and the persons identified as the 61 document authors. All rights reserved. 63 This document is subject to BCP 78 and the IETF Trust's Legal 64 Provisions Relating to IETF Documents 65 (http://trustee.ietf.org/license-info) in effect on the date of 66 publication of this document. Please review these documents 67 carefully, as they describe your rights and restrictions with 68 respect to this document. Code Components extracted from this 69 document must include Simplified BSD License text as described in 70 Section 4.e of the Trust Legal Provisions and are provided without 71 warranty as described in the Simplified BSD License. 73 Table of Contents 75 1. Introduction...................................................3 76 2. Conventions used in this document..............................4 77 3. Use Case Scenario Description and Requirements.................5 78 3.1. Requirements..............................................6 79 3.1.1. Supporting Multiple SDWAN Segmentations..............6 80 3.1.2. Client Service Requirement...........................6 81 3.1.3. Application Flow Based Segmentation..................7 82 3.1.4. Zero Touch Provisioning..............................8 83 3.1.5. Constrained Propagation of SDWAN Edge Properties.....9 85 3.2. Scenario #1: Homogeneous WAN.............................10 86 3.3. Scenario #2: Hybrid WAN Underlay.........................11 87 3.4. Scenario #3: Private VPN PE based SDWAN..................13 88 4. BGP Walk Through..............................................14 89 4.1. BGP Walk Through for Homogeneous SDWAN...................14 90 4.2. BGP Walk Through for Hybrid WAN Underlay.................16 91 4.3. BGP Walk Through for Application Flow Based Segmentation.17 92 4.4. Client Service Provisioning Model........................18 93 4.5. Benefit of Using Recursive Next Hop Resolution...........19 94 4.6. Why BGP as Control Plane for SDWAN?......................19 95 5. SDWAN Traffic Forwarding Walk Through.........................20 96 5.1. Packet Walk-Through for Scenario #1......................20 97 5.2. Packet Walk-Through for Scenario #2......................21 98 5.3. Packet Walk-Through for Scenario #3......................22 99 6. Manageability Considerations..................................22 100 7. Security Considerations.......................................23 101 8. IANA Considerations...........................................23 102 9. References....................................................23 103 9.1. Normative References.....................................23 104 9.2. Informative References...................................24 105 10. Acknowledgments..............................................25 107 1. Introduction 109 Here are some key characteristics of "SDWAN" networks: 111 - Augment of transport, which refers to utilizing overlay paths 112 over different underlay networks. Very often there are multiple 113 parallel overlay paths between any two SDWAN edges, some of them 114 are private networks over which traffic can traverse with or 115 without encryption, others require encryption, e.g. over 116 untrusted public networks. 117 - Enable direct Internet access from remote sites, instead hauling 118 all traffic to Corporate HQ for centralized policy control. 119 - Some traffic flows can be forwarded based on their application 120 identifiers instead of based on destination IP addresses, by the 121 edge nodes placing the traffic flows onto specific overlay paths 122 based on their application requirement. 123 - The traffic flows forwarding can also be based on specific 124 performance criteria (e.g. packets delay, packet loos, jitter) 125 to provide better application performance by choosing the right 126 underlay that meets or exceeds the specified criteria. 128 [Net2Cloud-Problem] describes the network related problems that 129 enterprises face to connect enterprises' branch offices to dynamic 130 workloads in different Cloud DCs, including using SDWAN to aggregate 131 multiple paths provided by different service providers to achieve 132 better performance and to accomplish application ID based 133 forwarding. 135 Even though SDWAN has been positioned as a flexible way to reach 136 dynamic workloads in third party Cloud data centers over different 137 underlay networks, scaling becomes a major issue when there are 138 hundreds or thousands of nodes to be interconnected by an SDWAN 139 overlay networks. 141 This document describes using BGP as control plane to scale large 142 SDWAN overlay networks. 144 2. Conventions used in this document 146 Cloud DC: Third party data centers that host applications and 147 workloads owned by different organizations or tenants. 149 Controller: Used interchangeably with SDWAN controller to manage 150 SDWAN overlay path creation/deletion and monitor the 151 path conditions between sites. 153 CPE: Customer Premise Equipment 155 CPE-Based VPN: Virtual Private Secure network formed among CPEs. 156 This is to differentiate from more commonly used PE- 157 based VPNs [RFC 4364]. 159 Homogeneous SDWAN: A type of SDWAN network in which all traffic 160 to/from the SDWAN edge nodes has to be encrypted 161 regardless of underlay networks. For lack of better 162 terminology, we call this Homogeneous SDWAN throughout 163 this document. 165 ISP: Internet Service Provider 166 NSP: Network Service Provider. NSP usually provides more 167 advanced network services, such as MPLS VPN, private 168 leased lines, or managed Secure WAN connections, many 169 times within a private trusted domain, whereas an ISP 170 usually provides plain internet services over public 171 untrusted domains. 173 PE: Provider Edge 175 SDWAN Edge Node: an edge node, which can be physical or virtual, 176 maps the attached clients' traffic to the wide area 177 network (WAN) overlay tunnels. 179 SDWAN: Software Defined Wide Area Network. A connectivity 180 service, offered by a Service Provider, that optimizes 181 transport of IP Packets over multiple underlay 182 connectivity services by recognizing applications at 183 Ingress and determining forwarding behavior by applying 184 policies to them. 186 SDWAN IPsec SA: IPsec Security Association between two SDWAN ports 187 or nodes. 189 SDWAN over Hybrid Networks: SDWAN over Hybrid Networks typically 190 have edge nodes utilizing bandwidth resources from 191 different types of underlay networks, some being private 192 networks and others being public Internet. 194 WAN Port: A Port or Interface facing an ISP or Network Service 195 Provider (NSP), with address allocated by the ISP or the 196 NSP. 198 C-PE: SDWAN Edge node, which can be CPE for customer managed 199 SDWAN, or PE for provider managed SDWAN services. 201 ZTP: Zero Touch Provisioning 203 3. Use Case Scenario Description and Requirements 205 SDWAN networks can have different topologies and have different 206 traffic patterns. To make it easier for the focused discussion in 207 subsequent drafts on SDWAN control plane and data plane, this 208 section describes several SDWAN scenarios that may have different 209 impact on their corresponding control planes & data planes. 211 3.1. Requirements 213 3.1.1. Supporting Multiple SDWAN Segmentations 215 The term "network segmentation", a.k.a. SDWAN instances, is 216 referring to the process of dividing the network into logical sub- 217 networks using isolation techniques on a forwarding device such as a 218 switch, router, or firewall. For a homogeneous network, such as MPLS 219 VPN or Layer 2 network, VRF or VLAN are used to achieve the network 220 segmentation. 222 As SDWAN is an overlay network arching over multiple types of 223 networks, MPLS L2VPN/L3VPN or pure L2 underlay can continue using 224 the VRF, VN-ID or VLAN to differentiate SDWAN network segmentations. 225 For public internet, the IPsec inner encapsulation header can carry 226 the SDWAN Instance Identifier to differentiate the packets belonging 227 to different SDWAN instances. 229 BGP already has the capability to differentiate control packets for 230 different network instances. When using BGP for SDWAN, the SDWAN 231 segmentations can be differentiated by the SDWAN Target ID in the 232 BGP Extended Community in the same way as VPN instances being 233 represented by the Route Target. Even though the SDWAN Target ID is 234 for the same purpose as the VPN ID in Route Target, it is better to 235 use a different name to differentiate from VPN if a CPE supports 236 traditional VPN with multiple VRFs and supports multiple SDWAN 237 Segmentations (instances). The actual SDWAN Target ID encoding is 238 specified by [SDWAN-EDGE-Discovery]. 240 3.1.2. Client Service Requirement 242 Client interface of SDWAN nodes can be IP or Ethernet based. 244 For Ethernet based client interfaces, SDWAN edge should support 245 VLAN-based service interfaces (EVI100), VLAN bundle service 246 interfaces (EVI200), or VLAN-Aware bundling service interfaces. EVPN 247 service requirements are applicable to the Client traffic, as 248 described in the Section 3.1 of RFC8388. 250 For IP based client interfaces, L3VPN service requirements are 251 applicable. 253 3.1.3. Application Flow Based Segmentation 255 Application Flow based Segmentation, also known as SDWAN Traffic 256 Segmentation, enables the separation of the traffic based on the 257 business and the security needs for different users' groups and/or 258 application requirements. Each user group and/or applications may 259 need different isolated topology and/or policies to fulfill the 260 business requirements. 262 The Application Flow based Segmentation concept is analogous to VLAN 263 (in L2 network) and VRF (in L3 network). 265 One can think about the Application Flow based Segmentation as a 266 feature that can be provided or enabled on a single SDWAN service 267 (or domain) to a single Subscriber. Each SDWAN Service can have one 268 or more overlay segments to support the business requirement; each 269 segment has its own policy, topology and application/user groups. 270 Applications/users' group can belong to more than one segment. 272 For example, a retail business requires the point-of-sales (PoS) 273 application in all stores to be isolated from other applications AND 274 routed only to the payment processing entity at a hub site (i.e. hub 275 and spoke); however, the same retail business allows other 276 applications to be routed to all sites (i.e. multipoint-to 277 multipoint) AND isolated from the PoS application. 279 In the figure below, the traffic from the PoS application follows a 280 Tree topology, whereas other traffic can be multipoint-to-multipoint 281 topology. 283 +--------+ 284 Payment traffic |Payment | 285 +------+----+-+gateway +------+----+-----+ 286 / / | +----+---+ | \ \ 287 / / | | | \ \ 288 +-+--+ +-+--+ +-+--+ | +-+--+ +-+--+ +-+--+ 289 |Site| |Site| |Site| | |Site| |Site| |Site| 290 | 1 | | 2 | | 3 | | |4 | | 5 | | 6 | 291 +--+-+ +--+-+ +--|-+ | +--|-+ +--|-+ +--|-+ 292 | | | | | | | 293 ==+=======+=======+====+======+=======+=======+=== 294 Multi-point connection for Other traffic 296 Another example is an enterprise who wants to isolate the traffic 297 for each department with each department has its own topology and 298 policy; the HR department may need to access certain applications 299 that are NOT accessible by the engineering department. In addition, 300 the contractors may have a limited access to the enterprise 301 resources. 303 3.1.4. Zero Touch Provisioning 305 Unlike traditional EVPN or L3VPN whose PEs are deployed for long 306 term, SDWAN edge nodes (virtual or physical) deployment at a 307 specific location can be ephemeral. Therefore, Zero Touch 308 Provisioning (ZTP), or Plug and Play, is a common requirement for 309 SDWAN. When an SDWAN edge is physically installed at a location or 310 instantiated on a VM in a Cloud DC, ZTP automates follow-up steps, 311 including updates to the OS, software version, and configuration 312 prior to connection. From network control perspective, ZTP includes 313 the following: 315 - Upon power up, an SDWAN node can establish transport layer 316 secure connection (such as TLS, SSL, etc.) to its controller whose 317 address can be burned or preconfigured on the device. 319 - The SDWAN Controller can designate a Local Network Controller 320 in the proximity of the SDWAN node; the Local Network Controller 321 manages and monitor the communication policies of the edge node. 323 3.1.5. Constrained Propagation of SDWAN Edge Properties 325 One SDWAN edge node may only be authorized to communicate with a 326 small number of other SDWAN edge nodes. Under this circumstance, the 327 property of the SDWAN edge node cannot be propagated to other nodes 328 that are not authorized to communicate. But a remote SDWAN edge node 329 upon powering up might not have the proper policies to know who the 330 authorized peers are. Therefore, it is very essential for SDWAN 331 deployment have a central point to distribute the properties of an 332 SDWAN edge node to its authorized peers. 334 BGP is well suited for this purpose. RFC 4684 has specified the 335 procedure to constrain the distribution of BGP UPDATE to only a 336 subset of nodes. Basically, each edge node informs the Route 337 Reflector (RR) [RFC4456] on its interested SDWAN instances. The RR 338 only propagates the BGP UPDATE for the relevant SDWAN instances to 339 the edge. 341 The connection between a SDWAN edge node and its RR can be over 342 insecure network. Therefore, upon power up, a SDWAN node needs to 343 establish a secure transport layer connection (TLS, SSL, etc.) to 344 its designated RR. The BGP UPDATE messages need to be sent over the 345 secure channel (TLS, SSL, etc.) to the RR. 347 +---+ 348 Peer Group 1 |RR | Peer Group 2 349 +======+====+=+ +======+====+=====+ 350 / / | +---+ | \ \ 351 / / | | \ \ 352 +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ 353 |C-PE| |C-PE| |C-PE| |C-PE| |C-PE| |C-PE| 354 | 1 | | 2 | | 3 | |4 | | 5 | | 6 | 355 +----+ +----+ +----+ +----+ +----+ +----+ 356 Tenant 1 Tenant 2 357 Figure 1: Peer Groups managed by RR 359 Tenant separation is achieved by the SDWAN instance identification 360 represented in control plane and data plane, respectively. 362 3.2. Scenario #1: Homogeneous WAN 364 This is referring to a type of SDWAN network with edge nodes 365 encrypting all traffic over WAN to other edge nodes, regardless of 366 whether the underlay is private or public. For lack of better 367 terminology, we call this Homogeneous SDWAN throughout this 368 document. 370 Some typical scenarios for the use of a Homogeneous SDWAN network 371 are as follows: 373 - A small branch office connecting to its HQ offices via the 374 Internet. All sensitive traffic to/from this small branch office has 375 to be encrypted, which is usually achieved using IPsec SAs. 377 - A store in a shopping mall may need to securely connect to its 378 applications in one or more Cloud DCs via the Internet. A common way 379 of achieving this is to establish IPsec SAs to the Cloud DC gateway 380 to carry the sensitive data to/from the store. 382 As described in [SECURE-EVPN], the granularity of the IPsec SAs for 383 Homogeneous SDWAN can be per site, per subnet, per tenant, or per 384 address. Once the IPsec SA is established for a specific 385 subnet/tenant/site, all traffic to/from the subnets/tenants/site are 386 encrypted. 388 +---+ 389 +--------------|RR |------------+ 390 / Untrusted +-+-+ \ 391 / \ 392 / \ 393 +----+ +---------+ +------+ +----+ 394 | CN3|--| P1-----+ -------------+------ P1 |--| CN1| 395 +----+ | C-PE1 P2-----+ | | C-PE2| +----+ 396 +----+ | P3-----+ Wide +------ P2 | +----+ 397 | CN2|--| | | area +------ P3 |--| CN3| 398 +----+ +---------+ | network | +------+ +----+ 399 | | 400 +----+ +---------+ | all packets | +------+ +----+ 401 | CN1|--| P1-----+ encrypted +------ P1 |--| CN1| 402 +----+ | C-PE3 P2-----+ by | | C-PE4| +----+ 403 +----+ | P3-----+ IPsec SAs +------ P2 | +----+ 404 | CN2|--| P4-----+--------------+ | |--| CN2| 405 +----+ +---------+ +------+ +----+ 407 CN: Client Networks, which is same as Tenant Networks used by NVo3 409 Figure 2: Homogeneous SDWAN 411 One of the key properties of homogeneous SDWAN is that the SDWAN 412 Local Network Controller (RR)is connected to C-PEs via untrusted 413 public network, therefore, requiring secure connection between RR 414 and C-PEs (TLS, DTLS, etc.). 416 Homogeneous SDWAN has some similarity to commonly deployed IPsec 417 VPN, albeit the IPsec VPN is usually point-to-point among a small 418 number of nodes and with heavy manual configuration for IPsec 419 between nodes, whereas an SDWAN network can have a large number of 420 edge nodes and require zero touch provisioning upon powering up. 422 Existing Private VPNs (e.g. MPLS based) can use homogeneous SDWAN to 423 extend over public network to remote sites to which the VPN operator 424 does not own or lease infrastructural connectivity, as described in 425 [SECURE-EVPN] and [SECURE-L3VPN] 427 3.3. Scenario #2: Hybrid WAN Underlay 429 In this scenario, SDWAN edge nodes (a.k.a. C-PEs) have some WAN 430 ports to private networks such as L3VPN and some WAN ports to the 431 public Internet. Since IPsec requires additional processing power 432 and encrypted traffic over Internet usually does not have the 433 premium SLA commonly offered by Private VPNs, especially over long 434 distance, this scenario is referring to a SDWAN network in which 435 traffic over IP VPN are forwarded natively without IPsec protection. 436 Only the traffic sent over the public Internet are protected by 437 IPsec SAs. 439 One C-PE might have Internet facing WAN ports managed by different 440 ISPs/NSPs and the WAN ports' addresses being assigned by the 441 corresponding ISPs/NSPs. Clients might have policies to specify: 443 1) some flows only traversing private VPNs, 444 2) some can traverse either if their packets are encrypted when over 445 the public Internet, and 446 3) Some flows, especially internet bound browsing ones can be handed off to 447 the internet without any encryption. 449 If a flow traversing multiple segments, such as A<->B<->C<->D, has 450 the Policy 2) above, the flow can traverse different underlays in 451 different segments, such as over Private network underlay between 452 A<->B without encryption, or over the public internet between B<->C 453 protected by an IPsec SA. 455 As shown in the figure below, C-PE-1 has two different types of 456 interfaces (A1 to Internet and A2 & A3 to VPN). The C-PEs' loopback 457 addresses and addresses attached to C-PEs may or may not be visible 458 to the ISPs/NSPs. The addresses for the WAN ports can have addresses 459 allocated by service providers or dynamically assigned (e.g. by 460 DHCP). 462 +---+ 463 +--------------|RR |----------+ 464 / Untrusted +-+-+ \ 465 / \ 466 / \ 467 +----+ +---------+ packets encrypted over +------+ +----+ 468 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 469 +----+ | C-PE1 A2-\ | C-PE2| +----+ 470 +----+ | A3--+--+ +---+---B2 | +----+ 471 | CN2|--| | |PE+--------------+PE |---B3 |--| CN3| 472 +----+ +---------+ +--+ trusted +---+ +------+ +----+ 473 | WAN | 474 +----+ +---------+ +--+ packets +---+ +------+ +----+ 475 | CN1|--| C1--|PE| go natively |PE |-- D1 |--| CN1| 476 +----+ | C-PE3 C2--+--+ without encry+---+ | C-PE4| +----+ 477 | | +--------------+ | | 478 | | | | 479 +----+ | | without encrypt over | | +----+ 480 | CN2|--| C3--+---- Untrusted --+------D2 |--| CN2| 481 +----+ +---------+ +------+ +----+ 483 CN: Client Network 484 Figure 3: Hybrid SDWAN 486 In addition, the connection between C-PEs and their Controller 487 (RR) might be via the untrusted public network. It is necessary to 488 encrypt the communication between RR and C-PEs, by TLS, DTLS, 489 etc.. 491 There could be multiple SDWAN edges (C-PEs) sharing a common 492 property, such as a geographic location. Some applications over 493 SDWAN may need to traverse specific geographic locations for 494 various reasons, such as to comply with regulatory rules, to 495 utilize specific value-added services, or others. 497 Services may not be congruent, i.e. the packets from A-> B may 498 traverse one underlay network, and the packets from B -> A may 499 traverse a different underlay. 501 3.4. Scenario #3: Private VPN PE based SDWAN 503 This scenario refers to existing VPN (e.g. MPLS based VPN, such as 504 EVPN or IPVPN) adding extra ports facing untrusted public networks 505 to allow PEs to offload some low priority traffic to public networks 506 when the VPN MPLS paths are congested. Throughout this document, 507 this scenario is also called Internet Offload for Private VPN, or PE 508 based SDWAN. 510 Here are some differences from the Scenario #2 (Section 3.3): 512 - The packets offloaded to untrusted public network are still 513 part of the VPN traffic, therefore, must be encrypted. 514 - For MPLS based VPN, PEs would have MPLS as payload 515 encapsulated within the IPsec tunnel egressing to the 516 Internet WAN ports, MPLS-in-IP-in-IPsec. 517 - The BGP RR is connected to PEs in the same way as VPN, i.e. 518 via the trusted network. 520 PE based SDWAN can be used by VPN service providers to temporarily 521 increase bandwidth between sites when not sure if the demand will 522 sustain for long period of time or as a temporary solution before 523 the permanent infrastructure is built or leased. 525 +---+ 526 +======>|PE2| 527 // +---+ 528 // ^ 529 // || VPN 530 // VPN v 531 ++--+ ++-+ +---+ 532 |PE1| <====> |RR| <===> |PE3| 533 +-+-+ +--+ +-+-+ 534 | | 535 +--- Public Internet -- + 536 Offload 538 Figure 4: Additional Internet paths added to the VPN 540 4. BGP Walk Through 542 4.1. BGP Walk Through for Homogeneous SDWAN 544 In the figure below, packets destined towards multiple routes 545 attached to the C-PE2 can be carried by one IPsec tunnel. Then one 546 BGP UPDATE can be announced by C-PE2 to its RR. 548 +---+ 549 +---------|RR |----------+ 550 / Untrusted+---+ \ 551 / \ 552 C-PE1/ \ 553 +---------+ +------+ 554 --+---+---------------------------------> |-10.1.x.x/16 555 | / | |C-PE2 |- VLAN = 15 556 | / | +-----> | 557 --|/1.1.1.1 | | | |-12.1.1.x/24 558 +---------+ | +------+ 559 | 2.2.2.2 560 | 561 C-PE3 | 562 +---------+ | 563 --|---+---------------------------+ 564 | / | 565 | / | 566 --|/3.3.3.3 | 567 +---------+ 568 Figure 5: Homogeneous SDWAN 570 The BGP UPDATE Message from C-PE2 to RR should have the client 571 routes encoded in the MP-NLRI Path Attribute and the IPsec Tunnel 572 associated information encoded in the Tunnel-Encap Path Attributes 573 as described in the [SECURE-EVPN]. 575 Alternatively, two separate BGP UPDATEs can be used to optimize the 576 BGP UPDATE packet size, as described by Section 4 and 8 of [Tunnel- 577 encap]. UPDATE U1 has its Nexthop to the node loopback address and 578 is reclusively resolved to the IPsec tunnel detailed attributes 579 advertised by the UPDATE U2 for the Node Loopback address: 581 Suppose that: 583 - a given packet P destined towards the client addresses attached 584 to C-PE2 (e.g. prefix 10.1.x.x/16) can be carried by any IPsec 585 tunnels terminated at C-PE2; 587 - The path along which P is to be forwarded is determined by BGP 588 UPDATE U1; 589 - UPDATE U1 does not have a Tunnel Encapsulation attribute; 590 - UPDATE U1 can include Encapsulation Extended Community and/or 591 Color Extended Community; 592 - The address of the next hop of UPDATE U1 is router C-PE2; 593 - The best route to router C-PE2 is a BGP route that was 594 advertised in UPDATE U2; 595 - UPDATE U2 has a Tunnel Encapsulation attribute to further 596 describe the IPsec detailed attributes. 598 UPDATE U1: 600 - MP-NLRI Path Attribute: 601 10.1.x.x/16 602 12.1.1.x/24 603 - Nexthop: 2.2.2.2 (C-PE2) 604 - Encapsulation Extended Community: Type = IPsec 606 UPDATE U2: 607 - MP-NLRI Path Attribute: 608 2.2.2.2 (C-PE2) 609 - Tunnel Encapsulation Path Attributes (as described in the 610 [SECURE-EVPN]) 612 If different client routes attached to C-PE2 needs to be reached by 613 separate IPsec tunnels, then The Color-Extended-Community [Tunnel- 614 encap] is used to associate routes with the tunnels. See Section 8 615 of [Tunnel-encap]. 617 If C-PE2 doesn't have the policy on authorized peers for the 618 specific client routes, RR needs to check the client routes policies 619 to propagate the BGP UPDATE messages to the remote authorized edge 620 nodes. 622 4.2. BGP Walk Through for Hybrid WAN Underlay 624 In this scenario, some client routes can be forwarded by any tunnels 625 terminated at the edge node and some client routes can be forwarded 626 by some specific tunnels (such as only MPLS VPN). 628 Color Extended Community (Section 4 & 8 of [Tunnel-Encap]) can be 629 used to represent specific tunnels for the client routes. 631 For example, in the Figure 5 above, suppose that Route 10.1.x.x/16 632 can be carried by either MPLS or IPsec, and Route 12.1.1.x/24 can 633 only be carried by MPLS, the following UDPATE messages can be used: 635 UPDATE #1a for Route Route 10.1.x.x/16: 637 - MP-NLRI Path Attribute: 638 10.1.x.x/16 639 Nexthop: 2.2.2.2 (C-PE2) 640 - Encapsulation Extended Community: Type = SDWAN-Hybrid 641 - Color Extended Community: RED 643 UPDATE #1b for Route Route 12.1.1.x/24: 644 - MP-NLRI Path Attribute: 645 12.1.1.x/24 646 Nexthop: 2.2.2.2 (C-PE2) 647 - Encapsulation Extended Community: Type= SDWAN-Hybrid 648 - Color Extended Community: YELLOW 650 UPDATE #2a: for IPsec tunnels terminated at the node: 651 - MP-NLRI Path Attribute: 652 2.2.2.2 (C-PE2) 653 - Tunnel Encapsulation Path Attributes: TYPE=SDWAN-Hybrid 654 - Color Extended Community: RED 656 UPDATE #2b: for MPLS-in-GRE terminated at the node: 657 - MP-NLRI Path Attribute: 658 2.2.2.2 (C-PE2) 659 - Tunnel Encapsulation Path Attributes: TYPE=SDWAN-Hybrid 660 - Color Extended Community: YELLOW 662 IANA Action: require a new value for SDWAN-Hybrid Tunnel Type. 664 4.3. BGP Walk Through for Application Flow Based Segmentation 666 If the applications are assigned with unique IP addresses, the 667 Application Flow based Segmentation described in Section 3.1.2 can 668 be achieved by advertising different BGP UPDATE messages to 669 different nodes. In the Figure below, the following BGP Updates can 670 be advertised to ensure that Payment Application only communicates 671 with the Payment Gateway: 673 BGP UPDATE #1a from C-PE2 to RR for the P2P topology that is only 674 propagated to Payment GW node: 676 UPDATE #1a (only to the Payment GW node): 678 - MP-NLRI Path Attribute: 679 - 30.1.1.x/24 680 - Nexthop: 2.2.2.2 681 - Encapsulation extended community: Tunneltype=IPSEC 682 - Color Extended Community: BLUE 684 BGP UPDATE #1b from C-PE2 to RR for the routes to be reached by C- 685 PE1 and C-PE2: 687 - MP-NLRI Path Attribute: 688 - 10.1.x.x 689 - 12.4.x.x 690 - Nexthop:2.2.2.2 691 - Encapsulation extended community: Tunnel-type=IPSEC 692 - Color Extended Community: RED 694 BGP UPDATE #2 describes the IPsec detailed attributes for IPsec 695 tunnels terminated at C-PE2 2.2.2.2. 697 UPDATE #2a: for all IPsec SAs terminated at the node: 698 - MP-NLRI Path Attribute: 699 2.2.2.2 (C-PE2) 700 - Tunnel Encapsulation Path Attributes: TYPE=IPsec (for all IPsec 701 SAs) 702 - Color Extended Community: RED 704 UPDATE #2b: for the IPsec SA to the Payment Gateway: 705 - MP-NLRI Path Attribute: 706 2.2.2.2 (C-PE2) 707 - Tunnel Encapsulation Path Attributes: TYPE=IPsec (for the IPsec 708 SA to Payment GW). 709 - Color Extended Community: Blue 711 +-------+ 712 |Payment| 713 +-------->| GW |<-------+ 714 /Hub-spoke +-------+ \ 715 /for Payment App \ 716 C-PE1 / \ C-PE2 717 +------/--+ +----\-+ 718 --|-----/ | | -|- 30.1.1.x/24 719 + ---------------------------------------> |-10.1.x.x/16 720 | | | |- 721 | | +------------> |- 12.1.1.x/24 722 --|---------------------------+ | | 723 +---------+ +------> |- VLAN=25; 724 / +------+ 22.1.1.x/24 725 +---------+ / 726 --| -----------------------------+ 727 | C-PE3 | / 728 | | / 729 --| --------------------------+ 730 +---------+ 731 Figure 6: Application Based SDWAN Segmentation 733 4.4. Client Service Provisioning Model 735 The provisioning tasks described in Section 4 of RFC8388 are the 736 same for the SDWAN client traffic. When client traffic is multi- 737 homed to two (or more) C-PEs, the Non-Service-Specific parameters 738 need to be provisioned per the Section 4.1.1 of RFC8388. 740 Since some SDWAN nodes are ephemeral and have small number of IP 741 subnets or VLANs attached to their client ports, it is recommended 742 to have default and simplified Service-specific parameters for each 743 client port, remotely managed by the SDWAN Network Controller via 744 the secure channel (TLS/DTLS) between the controller and the C-PEs. 746 4.5. Benefit of Using Recursive Next Hop Resolution 748 Using the Recursive Next Hop Resolution described in the Section 8 749 of [Tunnel-Encap], not only the clients' routes UPDATE messages 750 become very compact, but also they are not impacted by the underlay 751 network tunnels attributes changes. This method is especially useful 752 when the underlay tunnels are IPsec based which requires periodic 753 messages updates for the pairwise re-keying process. 755 4.6. Why BGP as Control Plane for SDWAN? 757 For a small sized SDWAN network, traditional hub & spoke model using 758 NHRP or DSVPN/DMVPN with a hub node (or controller) managing SDWAN 759 node WAN ports mapping (e.g. local & public addresses and tunnel 760 identifiers mapping) can work reasonably well. However, for a large 761 SDWAN network, say more than 100 nodes with different types of 762 topologies, the traditional approach becomes very messy, complex and 763 error prone. 765 Here are some of the compelling reasons of using BGP instead of 766 extending NHRP/DSVPN/DMVPN. (Same as the reasons quoted by LSVR on 767 why using BGP): 769 - BGP has the built-in capability to constrain the propagation of 770 SDWAN edge node properties to a small number of edge nodes 771 [RFC4684]. 773 - RR already has the capability to apply policies to communications 774 among peers. 776 - BGP is widely deployed as sole protocol (see RFC 7938) 778 - Robust and simple implementation 780 - Wide acceptance - minimal learning 782 - Reliable transport 784 - Guaranteed in-order delivery 786 - Incremental updates 788 - Incremental updates upon session restart 790 - No flooding and selective filtering 792 5. SDWAN Traffic Forwarding Walk Through 794 BGP based EVPN control plane are still applicable to routes attached 795 to the client ports of SDWAN nodes. Section 5 of RFC8388 describes 796 the BGP EVPN NLRI Usage for various routes of client traffic. The 797 procedures described in the Section 6 of RFC8388 are same for the 798 SDWAN client traffic. 800 The only additional consideration for SDWAN is to control how 801 traffic egress the SDWAN edge node to various WAN ports. 803 5.1. Packet Walk-Through for Scenario #1 805 A single IPsec security association (SA) protects data in one 806 direction. Under the Scenario #1, two SAs must be present to secure 807 traffic in both directions between any two end points, which can be 808 two C-PE nodes, two client ports, or two prefixes. 810 Upon power up, a SDWAN node can learn client routes from the Client 811 facing ports in the same way as EVPN described in RFC8388. 812 Controller, i.e. BGP-RR, facilitates the IPsec SA establishment and 813 rekey management as described in [SECURE-EVPN]. Controller manages 814 how client's routes are associated with individual IPSec SA. 816 Using Figure 2 of the Section 3.2 as an example. Let's assume that 817 the IPsec SAs terminate at the C-PE nodes. To enable full mesh 818 communication within client CN2 that are attached to C-PE1, C-PE3, 819 and C-PE4, six one directional IPsec SAs must be established: C-PE1 820 <-> C-PE3; C-PE1 <-> C-PE4; C-PE3 <-> C-PE4. The C-PE node address 821 (or loopback address) acts as the Next Hop address for the prefixes 822 attached to the C-PE node. 824 When a C-PE receives a packet from its client port, the C-PE uses 825 the IPsec SA whose destination address matches the Next Hop address 826 of the packet's destination to encrypt the packet and forward the 827 encrypted packet to the target C-PE via one of the C-PE's WAN ports. 829 When a C-PE receives an encrypted packet from its WAN port, it 830 decrypted the packet and forward the inner packet to the client port 831 based on the inner packet's destination address. 833 5.2. Packet Walk-Through for Scenario #2 835 In this scenario as shown in the Figure 3 of Section 3.3, there are 836 trusted VPN path and IPsec SAs over Public Internet between two C- 837 PEs. There could be user specified policy governing that some flows 838 can only be forwarded to the trusted VPN. 840 Upon receiving a packet from a client port, if the packet belongs to 841 a flow that can only be forwarded to the trusted VPN, the forwarding 842 processing is same as the VPN. If the packet belongs to a flow that 843 can be forwarded to either VPN or IPsec SA, the C-PE node can make 844 the local decision in choosing the least congested path to forward 845 the packet. Since it is not necessary to encrypt the packets 846 forwarded to the trusted VPN, it is more efficient for the IPsec SAs 847 to be terminated at the Internet facing WAN ports of the C-PE nodes. 848 For a C-PE with multiple WAN ports provided by different ISPs, the 849 C-PE can have multiple IPsec SAs terminated at one WAN port of a 850 remote C-PE. 852 If the IPsec SA is chosen, the packet is encrypted and encapsulated 853 in the inner payload of the IPsec SA and egress out the 854 corresponding WAN port of the IPsec SA. 856 Upon receiving a packet from a WAN port, especially from the 857 Internet facing WAN port, additional anti-DDoS mechanism has to be 858 enabled to prevent potential attacks from the Internet facing port. 859 Control Plane should not learn routes from the Internet facing WAN 860 ports. 862 +---+ 863 +--------------|RR |----------+ 864 / Untrusted +-+-+ \ 865 / Network \ 866 / \ 867 +----+ +---------+ packets encrypted over +--------+ +----+ 868 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 869 +----+ | C-PE-a A2-----+ +-------B2 C-PE-b| +----+ 870 |10.1.1.1 | |10.1.2.1| 871 +----+ | | +--+ +---+ | | +----+ 872 | CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3| 873 +----+ +---------+ +--+ trusted +---+ +--------+ +----+ 874 | VPN | 875 +--------------+ 876 Figure 8: SDWAN Scenario #2 878 5.3. Packet Walk-Through for Scenario #3 880 In this scenario all PEs have secure interfaces facing clients and 881 facing MPLS backbone with some PEs having additional connection by 882 unsecure public Internet. For the MPLS based PEs offloading some 883 MPLS packets to the internet path, the MPLS data frames need to be 884 encapsulated in IP [RFC4023] and secured by IPsec SA. The IP header 885 of the MPLS-in-IP packet becomes the outer IP header of the 886 resulting packet when IPsec transport mode is used to secure the 887 MPLS packet. This is followed by an IPsec header, followed by the 888 MPLS label stack. The IPsec header must set the payload type to MPLS 889 by using the IP protocol number specified in the section 3 of 890 RFC4023. If IPsec transport mode is applied on a MPLS-in-GRE packet, 891 the GRE header follows the IPsec header. 893 The end points of an IPsec SA between two PEs can be PEs' Internet 894 facing WAN ports addresses or the PEs' loopback addresses. The IPsec 895 SA end points should not be the client addresses unless the traffic 896 to/from those client addresses always go through the IPsec SA even 897 when the MPLS backbone has enough capacity to transport the traffic. 899 When the PEs' Internet facing ports are behind the NAT [RFC3715], an 900 outer UDP field can be added outside the encrypted payload 901 [RFC3948]. Three UDP ports must be open on the PEs: UDP port 4500 902 (used for NAT traversal), UDP port 500 (used for IKE) and IP 903 protocol 50 (ESP). IPsec IKE (Internet Key Exchange) between the two 904 PEs would be over the NAT [RFC3947] as well. 906 Upon receiving a packet from a client port, the forwarding and MPLS 907 processing is same as the MPLS VPN. If the MPLS backbone path to the 908 packets next hop is congested, the IPsec SA towards the target PEs 909 are used to encrypt the MPLS packet. 911 Upon receiving a packet from the Internet facing WAN port, the 912 packet is decrypted and the inner MPLS payload is extracted to be 913 sent to the MPLS VPN engine. 915 Same as the Scenario #2, additional anti-DDoS mechanism must be 916 enabled to prevent potential attacks from the Internet facing port. 917 Control Plane should not learn routes from the Internet facing WAN 918 ports. 920 6. Manageability Considerations 922 SDWAN overlay networks utilize the SDWAN controller to facilitate 923 route distribution, central configurations, and others. SDWAN Edge 924 nodes need to advertise the attached routes to their controller 925 (i.e. RR in BGP case). 927 7. Security Considerations 929 Having WAN ports facing the public Internet introduces the following 930 security risks: 932 1) Potential DDoS attack to the C-PEs with ports facing internet. 934 2) Potential risk of provider VPN network being injected with 935 illegal traffic coming from the public Internet WAN ports on the C- 936 PEs. 938 8. IANA Considerations 940 Need a new tunnel type: SDWAN-Hybrid 942 9. References 944 9.1. Normative References 946 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 947 Requirement Levels", BCP 14, RFC 2119, March 1997. 949 [RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private 950 networks (VPNs)", Feb 2006. 952 [RFC7296] C. Kaufman, et al, "Internet Key Exchange Protocol Version 953 2 (IKEv2)", Oct 2014. 955 [RFC7432] A. Sajassi, et al, "BGP MPLS-Based Ethernet VPN", Feb 956 2015. 958 [RFC8365] A. Sajassi, et al, "A network Virtualization Overlay 959 Solution Using Ethernet VPN (EVPN)", March 2018. 961 9.2. Informative References 963 [RFC8192] S. Hares, et al, "Interface to Network Security Functions 964 (I2NSF) Problem Statement and Use Cases", July 2017 966 [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation Subsequent 967 Address Family Identifier (SAFI) and the BGP Tunnel 968 Encapsulation Attribute", April 2009. 970 [RFC8388] J. Rabadan, et al, "Usage and Applicability of BGP MPLS- 971 Based Ethernet VPN", May 2018. 973 [Net2Cloud-Gap] L. Dunbar, A. Malis, C. Jacquenet, "Gap Analysis of 974 Interconnecting Underlay with Cloud Overlay", draft-dm- 975 net2cloud-gap-analysis-02, work in progress, Oct. 2018. 977 [SDWAN-EDGE-Discovery] L. Dunbar, S. Hares, R. Raszuk, K. Majumdar, 978 "BGP UPDATE for SDWAN Edge Discovery", draft-dunbar-idr- 979 sdwan-edge-discovery-01, work-in-progress, Nov 2020. 981 [VPN-over-Internet] E. Rosen, "Provide Secure Layer L3VPNs over 982 Public Infrastructure", draft-rosen-bess-secure-l3vpn-00, 983 work-in-progress, July 2018 985 [DMVPN] Dynamic Multi-point VPN: 986 https://www.cisco.com/c/en/us/products/security/dynamic- 987 multipoint-vpn-dmvpn/index.html 989 [DSVPN] Dynamic Smart VPN: 990 http://forum.huawei.com/enterprise/en/thread-390771-1- 991 1.html 993 [SECURE-EVPN] A. Sajassi, et al, "Secure EVPN", draft-sajassi-bess- 994 secure-evpn-01, Work-in-progress, March 2019. 996 [SECURE-L3VPN] E. Rosen, R. Bonica, "Secure Layer L3VPN over Public 997 Infrastructure", draft-rosen-bess-secure-l3vpn-00, Work- 998 in-progress, June 2018. 1000 [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation, 1001 storage, distribution and enforcement of policies for 1002 network security", Nov 2007. 1004 [Net2Cloud-Problem] L. Dunbar and A. Malis, "Seamless Interconnect 1005 Underlay to Cloud Overlay Problem Statement", draft-dm- 1006 net2cloud-problem-statement-02, June 2018 1008 [Net2Cloud-gap] L. Dunbar, A. Malis, and C. Jacquenet, "Gap Analysis 1009 of Interconnecting Underlay with Cloud Overlay", draft-dm- 1010 net2cloud-gap-analysis-02, work-in-progress, Aug 2018. 1012 [Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation 1013 Attribute", draft-ietf-idr-tunnel-encaps-10, Aug 2018. 1015 10. Acknowledgments 1017 Acknowledgements to Adrian Farrel, Joel Halpern, John Scudder, 1018 Darren Dukes, Andy Malis and Donald Eastlake for their review and 1019 contributions. 1021 This document was prepared using 2-Word-v2.0.template.dot. 1023 Authors' Addresses 1025 Linda Dunbar 1026 Futurewei 1027 Email: ldunbar@futurewei.com 1029 James Guichard 1030 Futurewei 1031 Email: james.n.guichard@futurewei.com 1033 Ali Sajassi 1034 Cisco 1035 Email: sajassi@cisco.com 1037 John Drake 1038 Juniper 1039 Email: jdrake@juniper.net 1041 Basil Najem 1042 Bell Canada 1043 Email: basil.najem@bell.ca 1045 David Carrel 1046 IPsec Research 1047 Email: carrel@ipsec.org 1049 Ayan Banerjee 1050 Cisco 1051 Email: ayabaner@cisco.com