idnits 2.17.1 draft-dunbar-bess-bgp-sdwan-usage-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 24 instances of too long lines in the document, the longest one being 5 characters in excess of 72. == There are 1 instance of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 29, 2020) is 1459 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC 4364' is mentioned on line 155, but not defined == Missing Reference: 'RFC8277' is mentioned on line 240, but not defined == Missing Reference: 'RFC4456' is mentioned on line 342, but not defined == Missing Reference: 'RFC4684' is mentioned on line 813, but not defined == Missing Reference: 'RFC4301' is mentioned on line 1000, but not defined == Missing Reference: 'RFC 3489' is mentioned on line 1025, but not defined ** Obsolete undefined reference: RFC 3489 (Obsoleted by RFC 5389) == Unused Reference: 'RFC2119' is defined on line 1076, but no explicit reference was found in the text == Unused Reference: 'RFC4364' is defined on line 1079, but no explicit reference was found in the text == Unused Reference: 'RFC7432' is defined on line 1085, but no explicit reference was found in the text == Unused Reference: 'RFC8365' is defined on line 1088, but no explicit reference was found in the text == Unused Reference: 'RFC8192' is defined on line 1093, but no explicit reference was found in the text == Unused Reference: 'RFC5521' is defined on line 1096, but no explicit reference was found in the text == Unused Reference: 'BGP-SDWAN-Port' is defined on line 1100, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-Gap' is defined on line 1104, but no explicit reference was found in the text == Unused Reference: 'VPN-over-Internet' is defined on line 1108, but no explicit reference was found in the text == Unused Reference: 'DMVPN' is defined on line 1112, but no explicit reference was found in the text == Unused Reference: 'DSVPN' is defined on line 1116, but no explicit reference was found in the text == Unused Reference: 'ITU-T-X1036' is defined on line 1127, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-gap' is defined on line 1135, but no explicit reference was found in the text == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 == Outdated reference: A later version (-06) exists of draft-sajassi-bess-secure-evpn-01 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 -- Duplicate reference: draft-rosen-bess-secure-l3vpn, mentioned in 'SECURE-L3VPN', was also mentioned in 'VPN-over-Internet'. == Outdated reference: A later version (-07) exists of draft-dm-net2cloud-problem-statement-02 == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 -- Duplicate reference: draft-dm-net2cloud-gap-analysis, mentioned in 'Net2Cloud-gap', was also mentioned in 'Net2Cloud-Gap'. == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-10 Summary: 2 errors (**), 0 flaws (~~), 28 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft J. Guichard 3 Intended status: Informational Futurewei 4 Expires: October 29, 2020 Ali Sajassi 5 Cisco 6 J. Drake 7 Juniper 8 B. Najem 9 Bell Canada 10 Ayan Barnerjee 11 D. Carrel 12 Cisco 14 April 29, 2020 16 BGP Usage for SDWAN Overlay Networks 17 draft-dunbar-bess-bgp-sdwan-usage-07 19 Abstract 21 The document describes three distinct SDWAN scenarios and discusses 22 the applicability of BGP for each of those scenarios. The goal of 23 the document is to demonstrate how BGP-based control plane is used 24 for large scale SDWAN overlay networks with little manual 25 intervention. 27 SDWAN edge nodes are commonly interconnected by multiple underlay 28 networks which can be owned and managed by different network 29 providers. 31 Status of this Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. This document may not be modified, 38 and derivative works of it may not be created, except to publish it 39 as an RFC and to translate it into languages other than English. 41 Internet-Drafts are working documents of the Internet Engineering 42 Task Force (IETF), its areas, and its working groups. Note that 43 other groups may also distribute working documents as Internet- 44 Drafts. 46 Internet-Drafts are draft documents valid for a maximum of six 47 months and may be updated, replaced, or obsoleted by other documents 48 at any time. It is inappropriate to use Internet-Drafts as 49 reference material or to cite them other than as "work in progress." 51 The list of current Internet-Drafts can be accessed at 52 http://www.ietf.org/ietf/1id-abstracts.txt 54 The list of Internet-Draft Shadow Directories can be accessed at 55 http://www.ietf.org/shadow.html 57 This Internet-Draft will expire on October 29, 2020. 59 Copyright Notice 61 Copyright (c) 2020 IETF Trust and the persons identified as the 62 document authors. All rights reserved. 64 This document is subject to BCP 78 and the IETF Trust's Legal 65 Provisions Relating to IETF Documents 66 (http://trustee.ietf.org/license-info) in effect on the date of 67 publication of this document. Please review these documents 68 carefully, as they describe your rights and restrictions with 69 respect to this document. Code Components extracted from this 70 document must include Simplified BSD License text as described in 71 Section 4.e of the Trust Legal Provisions and are provided without 72 warranty as described in the Simplified BSD License. 74 Table of Contents 76 1. Introduction...................................................3 77 2. Conventions used in this document..............................4 78 3. Use Case Scenario Description and Requirements.................5 79 3.1. Requirements..............................................6 80 3.1.1. Supporting Multiple SDWAN Segmentations..............6 81 3.1.2. Client Service Requirement...........................6 82 3.1.3. Application Flow Based Segmentation..................7 83 3.1.4. Zero Touch Provisioning..............................8 84 3.1.5. Constrained Propagation of SDWAN Edge Properties.....9 86 3.2. Scenarios #1: Homogeneous WAN............................10 87 3.3. Scenario #2: SDWAN WAN ports to VPN's PEs and to Internet11 88 3.4. Scenario #3: SDWAN WAN ports to MPLS VPN and the Internet14 89 4. BGP Walk Through..............................................15 90 4.1. BGP Walk Through for Homogeneous SDWAN...................15 91 4.2. BGP Walk Through for Application Flow Based Segmentation.18 92 4.3. Client Service Provisioning Model........................19 93 4.4. WAN Ports Provisioning Model.............................20 94 4.5. Why BGP as Control Plane for SDWAN?......................20 95 5. SDWAN Traffic Forwarding Walk Through.........................21 96 5.1. SDWAN Network Startup Procedures.........................21 97 5.2. Packet Walk-Through for Scenario #1......................22 98 5.3. Packet Walk-Through for Scenario #2......................22 99 5.3.1. SDWAN node WAN Ports Properties Registration........24 100 5.3.2. Controller Facilitated IPsec SA & NAT management....24 101 5.4. Packet Walk-Through for Scenario #3......................26 102 6. Manageability Considerations..................................26 103 7. Security Considerations.......................................26 104 8. IANA Considerations...........................................26 105 9. References....................................................27 106 9.1. Normative References.....................................27 107 9.2. Informative References...................................27 108 10. Acknowledgments..............................................28 110 1. Introduction 112 There are three key characteristics of "SDWAN" networks: 114 - Augment of transport, which refers to utilizing overlay paths 115 over different underlay networks. Very often there are multiple 116 parallel overlay paths between any two SDWAN edges, some of 117 which are private networks over which traffic can traverse with 118 or without encryption, others require encryption, e.g. over 119 untrusted public networks. 120 - Enable direct Internet access from remote sites, instead hauling 121 all traffic to Corporate HQ for centralized policy control. 122 - Some traffic are routed based on application IDs instead of 123 based on destination IP addresses. 125 [Net2Cloud-Problem] describes the network related problems that 126 enterprises face to connect enterprises' branch offices to dynamic 127 workloads in different Cloud DCs, including using SDWAN to aggregate 128 multiple paths provided by different service providers to achieve 129 better performance and to accomplish application ID based 130 forwarding. 132 Even though SDWAN has been positioned as a flexible way to reach 133 dynamic workloads in third party Cloud data centers over different 134 underlay networks, scaling becomes a major issue when there are 135 hundreds or thousands of nodes to be interconnected by an SDWAN 136 overlay networks. 138 BGP is widely used by underlay networks. This document describes 139 using BGP for edge nodes to exchange information across the SDWAN 140 overlay networks. 142 2. Conventions used in this document 144 Cloud DC: Third party data centers that host applications and 145 workloads owned by different organizations or tenants. 147 Controller: Used interchangeably with SDWAN controller to manage 148 SDWAN overlay path creation/deletion and monitor the 149 path conditions between sites. 151 CPE: Customer Premise Equipment 153 CPE-Based VPN: Virtual Private Secure network formed among CPEs. 154 This is to differentiate from more commonly used PE- 155 based VPNs [RFC 4364]. 157 Homogeneous SDWAN: A type of SDWAN network in which all traffic 158 to/from the SDWAN edge nodes has to be encrypted 159 regardless of underlay networks. For lack of better 160 terminology, we call this Homogeneous SDWAN throughout 161 this document. 163 ISP: Internet Service Provider 165 NSP: Network Service Provider. NSP usually provides more 166 advanced network services, such as MPLS VPN, private 167 leased lines, or managed Secure WAN connections, many 168 times within a private trusted domain, whereas an ISP 169 usually provides plain internet services over public 170 untrusted domains. 172 PE: Provider Edge 174 SDWAN End-point: a port (logical or physical) of a SDWAN edge node. 176 SDWAN: Software Defined Wide Area Network. In this document, 177 "SDWAN" refers to the solutions of pooling WAN bandwidth 178 from multiple underlay networks to get better WAN 179 bandwidth management, visibility & control. When the 180 underlay networks are private, traffic can traverse 181 without additional encryption; when the underlay 182 networks are public, such as the Internet, some traffic 183 may need to be encrypted when traversing through 184 (depending on user provided policies). 186 SDWAN IPsec SA: IPsec Security Association between two SDWAN ports 187 or nodes. 189 SDWAN over Hybrid Networks: SDWAN over Hybrid Networks typically 190 have edge nodes utilizing bandwidth resources from 191 multiple service providers. In Hybrid SDWAN network, 192 packets over private networks can go natively without 193 encryption and are encrypted over the untrusted network, 194 such as the public Internet. 196 WAN Port: A Port or Interface facing an ISP or Network Service 197 Provider (NSP), with address (usually public routable 198 address) allocated by the ISP or the NSP. 200 C-PE: SDWAN Edge node, which can be CPE for customer managed 201 SDWAN, or PE that is for provider managed SDWAN 202 services). 204 ZTP: Zero Touch Provisioning 206 3. Use Case Scenario Description and Requirements 208 SDWAN networks can have different topologies and have different 209 traffic patterns. To make it easier for the focused discussion in 210 subsequent drafts on SDWAN control plane and data plane, this 211 section describes several SDWAN scenarios that may have different 212 impact on their corresponding control planes & data planes. 214 3.1. Requirements 216 3.1.1. Supporting Multiple SDWAN Segmentations 218 The term "network segmentation", a.k.a. SDWAN instances, is 219 referring to the process of dividing the network into logical sub- 220 networks using isolation techniques on a forwarding device such as a 221 switch, router, or firewall. For a homogeneous network, such as MPLS 222 VPN or Layer 2 network, VRF or VLAN are used to achieve the network 223 segmentation. 225 As SDWAN is an overlay network arching over multiple types of 226 networks, VRF or VLAN can't be used directly to differentiate SDWAN 227 network segmentations. 229 However, BGP already has the capability to differentiate SDWAN 230 segmentations: 232 - Create a SDWAN Target ID in the BGP Extended Community to 233 represent different SDWAN Segmentations 234 - Same as Route Target, just use a different name to 235 differentiate from VPN if a CPE supports traditional VPN with 236 multiple VRFs and supports multiple SDWAN Segmentations 237 (instances). 238 - When the SDWAN Target ID is used, 239 - Use the similar approach as VPN Label carried by NLRI Path 240 Attribute [RFC8277] to identify routes belonging to different 241 SDWAN Segmentations. 242 - The MPLS VPN SAFI 128 & Route Distinguisher can be used for 243 routes belonging to different SDWAN instances. 245 3.1.2. Client Service Requirement 247 Client interface of SDWAN nodes can be IP or Ethernet based. 249 For Ethernet based client interfaces, SDWAN edge should support 250 VLAN-based service interfaces (EVI100), VLAN bundle service 251 interfaces (EVI200), or VLAN-Aware bundling service interfaces. EVPN 252 service requirements are applicable to the Client traffic, as 253 described in the Section 3.1 of RFC8388. 255 For IP based client interfaces, L3VPN service requirements are 256 applicable. 258 3.1.3. Application Flow Based Segmentation 260 Application Flow based Segmentation, also known as SDWAN Traffic 261 Segmentation, enables the separation of the traffic based on the 262 business and the security needs for different users' groups and/or 263 application requirements. Each user group and/or applications may 264 need different isolated topology and/or policies to fulfill the 265 business requirements. 267 The Application Flow based Segmentation concept is analogous to VLAN 268 (in L2 network) and VRF (in L3 network). 270 One can think about the Application Flow based Segmentation as a 271 feature that can be provided or enabled on a single SDWAN service 272 (or domain) to a single Subscriber. Each SDWAN Service can have one 273 or more overlay Segments to support the business requirement; each 274 Segment has its own policy, topology and application/user groups. 275 Applications/users' group can belong to more than one Segment. 277 For example, a retail business requires the point-of-sales (PoS) 278 application in all stores to be isolated from other applications AND 279 routed only to the payment processing entity at a hub site (i.e. hub 280 and spoke); however, the same retail business requires the other 281 applications to be routed to all sites (i.e. multipoint-to 282 multipoint) AND isolated from the PoS application. 284 In the figure below, the traffic from the PoS application follows a 285 Tree topology, whereas other traffic can be multipoint-to-multipoint 286 topology. 288 +--------+ 289 Payment traffic |Payment | 290 +------+----+-+gateway +------+----+-----+ 291 / / | +----+---+ | \ \ 292 / / | | | \ \ 293 +-+--+ +-+--+ +-+--+ | +-+--+ +-+--+ +-+--+ 294 |Site| |Site| |Site| | |Site| |Site| |Site| 295 | 1 | | 2 | | 3 | | |4 | | 5 | | 6 | 296 +--+-+ +--+-+ +--|-+ | +--|-+ +--|-+ +--|-+ 297 | | | | | | | 298 ==+=======+=======+====+======+=======+=======+=== 299 Multi-point connection for Other traffic 301 Another example is an enterprise who wants to isolate the traffic 302 for each department and have different topology and policy for 303 different department; the HR department may need to access certain 304 applications that are NOT accessible by the engineering department. 305 In addition, the contractors may have a limited access to the 306 enterprise resources. 308 3.1.4. Zero Touch Provisioning 310 Unlike traditional EVPN or L3VPN whose PEs are deployed for long 311 term, SDWAN edge nodes (virtual or physical) deployment at a 312 specific location can be ephemeral. Therefore, Zero Touch 313 Provisioning (ZTP), or Plug and Play, is a common requirement for 314 SDWAN. When an SDWAN edge is physically installed at a location or 315 instantiated on a VM in a Cloud DC, ZTP automates follow-up steps, 316 including updates to the OS, software version, and configuration 317 prior to connection. From network control perspective, ZTP includes 318 the following: 320 - Upon power up, an SDWAN node can establish transport layer 321 secure connection (such as TLS, SSL, etc.) to its controller whose 322 address can be burned or preconfigured on the device. 324 - The SDWAN Controller can designate a Local Network Controller 325 in the proximity of the SDWAN node; the Local Network Controller 326 manages and monitor the communication policies of the edge node. 328 3.1.5. Constrained Propagation of SDWAN Edge Properties 330 One SDWAN edge node may only be authorized to communicate with a 331 small number of other SDWAN edge nodes. Under this circumstance, the 332 property of the SDWAN edge node cannot be propagated to any other 333 nodes who are not authorized to communicate. But a remote SDWAN edge 334 node upon powering up might not have the proper policies to know who 335 the authorized peers are. Therefore, it is very essential for SDWAN 336 deployment have a central point to distribute the properties of each 337 SDWAN edge node to its authorized peers. 339 BGP is well suited for this purpose. RFC 4684 has specified the 340 procedure to constrain the distribution of BGP UPDATE to only a 341 subset of SDWAN edges. Basically, each edge node informs the Route 342 Reflector (RR) [RFC4456] on its interested SDWAN instances. The RR 343 only propagates the BGP UPDATE for the relevant SDWAN instances to 344 the edge. 346 Usually the connection between a SDWAN edge node and its RR is over 347 insecure network. Therefore, upon power up, a SDWAN node needs to 348 establish a secure transport layer connection (TLS, SSL, etc.) to 349 its designated RR. The BGP UPDATE messages need to be sent over the 350 secure channel (TLS, SSL, etc.) to the RR. 352 +---+ 353 Peer Group 1 |RR | Peer Group 2 354 +======+====+=+ +======+====+=====+ 355 / / | +---+ | \ \ 356 / / | | \ \ 357 +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ 358 |C-PE| |C-PE| |C-PE| |C-PE| |C-PE| |C-PE| 359 | 1 | | 2 | | 3 | |4 | | 5 | | 6 | 360 +----+ +----+ +----+ +----+ +----+ +----+ 361 Tenant 1 Tenant 2 362 Figure 1: Peer Groups managed by RR 364 Tenant separation is achieved by the RR creating different Tenant 365 based Peer Groups. 367 3.2. Scenarios #1: Homogeneous WAN 369 This is referring to a type of SDWAN network with edge nodes 370 encrypting all traffic over WAN to other edge nodes, regardless of 371 whether the underlay is private or public. For lack of better 372 terminology, we call this Homogeneous SDWAN throughout this 373 document. 375 Some typical scenarios for the use of a Homogeneous SDWAN network 376 are as follows: 378 - A small branch office connecting to its HQ offices via the 379 Internet. All sensitive traffic to/from this small branch office has 380 to be encrypted, which is usually achieved using IPsec SAs. 382 - A store in a shopping mall may need to securely connect to its 383 applications in one or more Cloud DCs via the Internet. A common way 384 of achieving this is to establish IPsec SAs to the Cloud DC gateway 385 to carry the sensitive data to/from the store. 387 As described in [SECURE-EVPN], the granularity of the IPsec SAs for 388 Homogeneous SDWAN can be per site, per subnet, per tenant, or per 389 address. Once the IPsec SA is established for a specific 390 subnet/tenant/site, all traffic to/from the subnets/tenants/site are 391 encrypted. 393 +---+ 394 +--------------|RR |------------+ 395 / Untrusted +-+-+ \ 396 / \ 397 / \ 398 +----+ +---------+ +------+ +----+ 399 | CN3|--| P1-----+ -------------+------ P1 |--| CN1| 400 +----+ | C-PE1 P2-----+ | | C-PE2| +----+ 401 +----+ | P3-----+ Wide +------ P2 | +----+ 402 | CN2|--| | | area +------ P3 |--| CN3| 403 +----+ +---------+ | network | +------+ +----+ 404 | | 405 +----+ +---------+ | all packets | +------+ +----+ 406 | CN1|--| P1-----+ encrypted +------ P1 |--| CN1| 407 +----+ | C-PE3 P2-----+ by | | C-PE4| +----+ 408 +----+ | P3-----+ IPsec SAs +------ P2 | +----+ 409 | CN2|--| P4-----+--------------+ | |--| CN2| 410 +----+ +---------+ +------+ +----+ 412 CN: Client Networks, which is same as Tenant Networks used by NVo3 414 Figure 2: Homogeneous SDWAN 416 One of the key properties of homogeneous SDWAN is that the SDWAN 417 Local Network Controller (RR)is connected to C-PEs via untrusted 418 public network, therefore, requiring secure connection between RR 419 and C-PEs (TLS, DTLS, etc.). 421 Homogeneous SDWAN has some similarity to commonly deployed IPsec 422 VPN, albeit the IPsec VPN is usually point-to-point among a small 423 number of endpoints and with heavy manual configuration for IPsec 424 between end-points, whereas an SDWAN network can have a large number 425 of end-points with an SDWAN controller to manage requiring zero 426 touch provisioning upon powering up. 428 Existing Private VPNs (e.g. MPLS based) can use homogeneous SDWAN to 429 extend over public network to remote sites to which the VPN operator 430 does not own or lease infrastructural connectivity, as described in 431 [SECURE-EVPN] and [SECURE-L3VPN] 433 3.3. Scenario #2: SDWAN WAN ports to VPN's PEs and to Internet 435 In this scenario, SDWAN edge nodes (a.k.a. C-PEs) have some WAN 436 ports connected to PEs of Private VPNs over which packets can be 437 forwarded natively without encryption, and some WAN ports connected 438 to the Internet over which sensitive traffic have to be encrypted 439 (usually by IPsec SA). 441 In this scenario, the SDWAN edge nodes' egress WAN ports are all 442 IP/Ethernet based, either egress to PEs of the VPNs or to the 443 Internet. Even if the VPN is a MPLS network, the VPN's PEs have 444 IP/Ethernet connections to the SDWAN edge (C-PEs). Throughout this 445 document, this scenario is also called CPE based SDWAN over Hybrid 446 Networks. 448 Even though IPsec SA can secure the packets traversing the Internet, 449 it does not offer the premium SLA commonly offered by Private VPNs, 450 especially over long distance. Clients need to have policies to 451 specify criteria for flows only traversing private VPNs or 452 traversing either as long as encrypted when over the Internet. For 453 example, client can have those polices for the flows: 455 1. A policy or criteria for sending the flows over a private 456 network without encryption (for better performance), 457 2. A policy or criteria for sending the flows over any networks 458 as long as the packets of the flows are encrypted when 459 traversing untrusted networks, or 460 3. A policy of not needing encryption at all. 462 If a flow traversing multiple segments, such as A<->B<->C<->D, has 463 either Policy 2 or 3 above, the flow can traverse different 464 underlays in different segments, such as over Private network 465 underlay between A<->B without encryption, or over the public 466 internet between B<->C in an IPsec SA. 468 As shown in the figure below, C-PE-1 has two different types of 469 interfaces (A1 to Internet and A2 & A3 to VPN). The C-PEs' loopback 470 addresses and addresses attached to C-PEs may or may not be visible 471 to the ISPs/NSPs. The addresses for the WAN ports can have addresses 472 allocated by service providers or dynamically assigned (e.g. by 473 DHCP). One WAN port shown in the figure below (e.g. A1, A2, A3 etc.) 474 is a logical representation of potential multiple physical ports on 475 the C-PEs. 477 +---+ 478 +--------------|RR |----------+ 479 / Untrusted +-+-+ \ 480 / \ 481 / \ 482 +----+ +---------+ packets encrypted over +------+ +----+ 483 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 484 +----+ | C-PE1 A2-\ | C-PE2| +----+ 485 +----+ | A3--+--+ +---+---B2 | +----+ 486 | CN2|--| | |PE+--------------+PE |---B3 |--| CN3| 487 +----+ +---------+ +--+ trusted +---+ +------+ +----+ 488 | WAN | 489 +----+ +---------+ +--+ packets +---+ +------+ +----+ 490 | CN1|--| C1--|PE| go natively |PE |-- D1 |--| CN1| 491 +----+ | C-PE3 C2--+--+ without encry+---+ | C-PE4| +----+ 492 | | +--------------+ | | 493 | | | | 494 +----+ | | without encrypt over | | +----+ 495 | CN2|--| C3--+---- Untrusted --+------D2 |--| CN2| 496 +----+ +---------+ +------+ +----+ 498 CN: Client Network 499 Figure 3: Hybrid SDWAN 501 Some key characteristics of a Hybrid SDWAN overlay network are as 502 follows: 504 - one C-PE may be connected to different ISPs/NSPs, with some of its 505 WAN ports addresses being assigned by different ISPs/NSPs. 507 - The WAN ports connected to PEs of trusted private networks (e.g. MPLS 508 VPN) hand off IP/Ethernet packets, just like today's CPE that do not 509 handle MPLS packets and do not participate in the underlay VPN networks' 510 control plane. Traffic can flow natively without encryption when be 511 forwarded out through those WAN ports for better performance. 513 - The WAN ports connected to untrusted networks, e.g. the Internet, 514 requires sensitive traffic to be encrypted, i.e. encrypted by IPsec SA. 516 - An SDWAN local Network Controller (RR) is connected to C-PEs via 517 the untrusted public network, therefore, requiring secure 518 connection between RR and C-PEs via TLS, DTLS, etc. 520 - The SDWAN nodes' [loopback] addresses might not be routable nor 521 visible in the underlay ISP/NSP networks. Routes & services 522 attached to SDWAN edges at the SDWAN overlay layer are in 523 different address spaces than the underlay networks. 525 - There could be multiple SDWAN devices sharing a common property, 526 such as a geographic location. Some applications over SDWAN may 527 need to traverse specific geographic locations for various 528 reasons, such as to comply with regulatory rules, to utilize 529 specific value added services, or others. 531 - The underlay path selection between sites can be a local decision. 532 Some policies allow one service from C-PE1 -> C-PE2 -> C-PE3 using 533 one ISP/NSP underlay in the first segment (C-PE1 -> C-PE2) and 534 using a different ISP/NSP in the second segment (C-PE2-> CPE3). 536 - Services may not be congruent, i.e. the packets from A-> B may 537 traverse one underlay network, and the packets from B -> A may 538 traverse a different underlay. 540 - Different services, routes, or VLANs attached to SDWAN nodes can 541 be aggregated over one underlay path; same service/routes/VLAN can 542 spread over multiple SDWAN underlays at different times depending 543 on the policies specified for the service. For example, one 544 tenant's packets to HQ need to be encrypted when sent over the 545 Internet or have to be sent over private networks, while the same 546 tenant's packets to Facebook can be sent over the Internet without 547 encryption. 549 3.4. Scenario #3: SDWAN WAN ports to MPLS VPN and the Internet 551 This scenario refers to existing VPN (e.g. MPLS based VPN, such as 552 EVPN or IPVPN) adding extra ports facing untrusted public networks 553 allowing PEs to offload some low priority traffic to ports facing 554 public networks when the VPN MPLS paths are congested. Throughout 555 this document, this scenario is also called Internet Offload for 556 Private VPN, or PE based SDWAN. 558 In this scenario, the packets offloaded to untrusted public network 559 must be encrypted. 561 PE based SDWAN can be used by VPN service providers to temporarily 562 increase bandwidth between sites when they are not sure if the 563 demand will sustain for long period of time or as a temporary 564 solution before the permanent infrastructure is built or leased. 566 +---+ 567 +-------|PE2| 568 / +---+ 569 Internet / ^ 570 offload / || VPN 571 / VPN v 572 ++--+ ++-+ +---+ 573 |PE1| <====> |RR| <===> |PE3| 574 +-+-+ +--+ +-+-+ 575 | | 576 +--- Public Internet -- + 578 Figure 4: Additional Internet paths added to the VPN 580 Here are some key properties for PE based SDWAN: 582 - For MPLS based VPN, PEs continue having MPLS encapsulation 583 handoff to existing paths. 585 - The BGP RR is connected to PEs in the same way as VPN, i.e. 586 via the trusted network. 587 - For the added Internet ports, PEs have IP packets handoff, 588 i.e. sending and receiving IP data frames. Internally, PEs 589 can have the option to encapsulate the MPLS payload in IP, as 590 specified by RFC4023. 591 - The ports facing public internet might get IP addresses 592 assigned by ISPs, which may not be in the same address domain 593 as PEs'. 594 - Ports facing public internet are not as secure as the ports 595 facing private infrastructure. There could be spoofing, or 596 DDOS attacks to the ports facing public internet. Extra 597 consideration must be given when injecting the new routes 598 learned from public network into VRFs. 599 - Even though packets are encrypted over public internet, the 600 performance SLA is not guaranteed over public internet. 601 Therefore, clients may have policies only allowing some flows 602 to be offloaded to internet path. 604 4. BGP Walk Through 606 4.1. BGP Walk Through for Homogeneous SDWAN 608 In the figure below, packets destined towards multiple routes 609 attached to the C-PE2 can be carried by one IPsec tunnel. Then one 610 BGP UPDATE can be announced by C-PE2 to its RR. 612 +---+ 613 +---------|RR |----------+ 614 / Untrusted+---+ \ 615 / \ 616 C-PE1/ \ 617 +---------+ +------+ 618 --+---+---------------------------------> |-10.1.x.x/16 619 | / | |C-PE2 |- VLAN = 15 620 | / | +-----> | 621 --|/ | | | |-12.1.1.x/24 622 +---------+ | +------+ 623 | 624 C-PE3 | 625 +---------+ | 626 --|---+---------------------------+ 627 | / | 628 | / | 629 --|/ | 630 +---------+ 631 Figure 5: (see *.pdf for more accurate figure) 633 The BGP UPDATE Message from C-PE2 to RR should have the client 634 routes encoded in the MP-NLRI Path Attribute and the IPsec Tunnel 635 associated information encoded in the Tunnel-Encap Path Attributes 636 as described in the [SECURE-EVPN]: 638 - MP-NLRI Path Attribute: to indicate multiple routes attached to 639 the C-PE2: 640 10.1.x.x/16 641 VLAN #15 642 12.1.1.x/24 643 - Tunnel-Encap Path Attribute: to describe the IPsec attributes 644 for routes encoded in the NLRI Path Attribute: 645 IPsec attributes for remote nodes to establish the IPsec 646 tunnel to C-PE2. 648 If different client routes attached to C-PE2 needs to be reached by 649 separate IPsec tunnels, then multiple BGP UPDATE messages need to be 650 sent to the remote nodes. If C-PE2 doesn't have the policy on 651 mapping of clients' routes to IPsec tunnels, RR needs to check the 652 client routes policies to send separate BGP UPDATE messages to the 653 remote edge nodes. 655 There could be policies governing the topologies of a client's 656 different routes attached to an edge node. For example, VLAN #25 and 657 route 22.1.1.x/24 could be the Payment Applications described in the 658 Section 3.1.2 that can only communicate with Payment Gateway 659 attached to C-PE3. If C-PEs don't have the policy to govern the 660 communication peers, RR can take over the responsibility of only 661 send BGP UPDATE for the Blue routes to C-PE1 and send the RED routes 662 to C-PE3. 664 +---+ 665 +-----------|RR |----------+ 666 / Untrusted +---+ \ 667 / \ 668 / \ 669 +---------+ +------+ 670 --+ --------------------------------------> |-10.1.x.x/16 671 | C-PE1 | |C-PE2 |- VLAN = 15 672 | | +-----------> |- 12.1.1.x/24 673 --|---------------------------+ | | 674 +---------+ +------> |- VLAN=25 675 / +------+ 22.1.1.x/24 676 +---------+ / 677 --| ----------------------------+ 678 | C-PE3 | / 679 | | / 680 --| -------------------------+ 681 +---------+ 683 Figure 6: (see *.pdf for more accurate figure) 685 UPDATE 1: 686 - MP-NLRI Path Attribute: 687 10.1.x.x/16 688 VLAN #15 689 12.1.1.x/24 691 - Tunnel-Encap Path Attribute: 692 IPsec SA attributes for IPsec tunnels to C-PE2 from any node 693 for reaching 10.1.x.x/16, VLAN #15, and 12.1.1.x/24. 695 UPDATE 2 (only sent to C-PE3) 696 - MP-NLRI Path Attribute: 697 VLAN #25 698 22.1.1.x/24 700 - Tunnel-Encap: 701 IPsec SA attributes for IPsec tunnels to C-PE2 from C-PE3 for 702 reaching VLAN #25 and subnet 22.1.1./24. 704 4.2. BGP Walk Through for Application Flow Based Segmentation 706 If the applications are assigned with unique IP addresses, the 707 Application Flow based Segmentation described in Section 3.1.2 can 708 be achieved by advertising different BGP UPDATE messages to 709 different nodes. In the Figure below, the following BGP Updates can 710 be advertised to ensure that Payment Application only communicates 711 with the Payment Gateway: 713 BGP UPDATE #1 from C-PE2 to RR for the RED P2P topology (only 714 propagated to Payment GW node: 716 - MP-NLRI Path Attribute: 717 - 30.1.1.x/24 718 - Tunnel Encap Path Attribute 719 - IPsec Attributes for PaymentGW ->C-PE2 721 BGP UPDATE #2 from C-PE2 to RR for the routes to be reached by 722 Purple: 724 - MP-NLRI Path Attribute: 725 - 10.1.x.x 726 - 12.4.x.x 727 - TunnelEncap Path Attribute: 728 - Any node to C-PE2 729 +-------+ 730 |Payment| 731 +-------->| GW |<-------+ 732 /Hub-spoke +-------+ \ 733 /for Payment Ppp \ 734 C-PE1 / \ C-PE2 735 +------/--+ +----\-+ 736 --|-----/ | | -|- 30.1.1.x/24 737 + ---------------------------------------> |-10.1.x.x/16 738 | | | |- 739 | | +------------> |- 12.1.1.x/24 740 --|---------------------------+ | | 741 +---------+ +------> |- VLAN=25; 742 / +------+ 22.1.1.x/24 743 +---------+ / 744 --| -----------------------------+ 745 | C-PE3 | / 746 | | / 747 --| --------------------------+ 748 +---------+ 749 Figure 7: (see *.pdf for more accurate figure) 751 4.3. Client Service Provisioning Model 753 The provisioning tasks described in Section 4 of RFC8388 are the 754 same for the SDWAN client traffic. When client traffic is multi- 755 homed to two (or more) C-PEs, the Non-Service-Specific parameters 756 need to be provisioned per the Section 4.1.1 of RFC8388. 758 Since most SDWAN nodes are ephemeral and have small number of IP 759 subnets or VLANs attached to their client ports, it is recommended 760 to have default and simplified Service-specific parameters for each 761 client port, remotely managed by the SDWAN Network Controller via 762 the secure channel (TLS/DTLS) between the controller and the C-PEs. 764 4.4. WAN Ports Provisioning Model 766 Since the deployment of PEs to MPLS VPN are for relatively long 767 term, the common provisioning procedure for PE's WAN ports is via 768 CLI. 770 A SDWAN node deployment can be ephemeral and its location can be in 771 remote locations, manual provisioning for its WAN ports is not 772 acceptable. In addition, a SDWAN WAN port's IP address can be 773 dynamically assigned or using private addresses. Therefore, it is 774 necessary to have a separate control protocol; something like NHRP 775 did for ATM, for a SDWAN node to register its WAN property to its 776 controller dynamically. 778 Unlike a PE to MPLS based VPN where its WAN ports are homogeneously 779 facing MPLS private network and all traffic are egressed in MPLS 780 data frames through its WAN ports, the WAN ports of a SDWAN node can 781 be connected to a PE of VPN with Ethernet/IP, MPLS private network 782 directly via MPLS headers, or the public Internet. 784 For Scenario #1 described in Section 3.2, the WAN ports can face 785 public internet or VPN. 787 For Scenario #2 described in Section 3.3, WAN ports are either 788 configured as connecting to PEs of VPN where traffic can be sent as 789 IP/Ethernet without encryption, or configured as connecting to 790 public Internet that requires encryption for packets egress out. 792 For Scenario #3 described in Section 3.4, the WAN ports are either 793 configured as VPN egress ports (hand off MPLS data frames), or as 794 connecting to the public internet that requires MPLS in IP in IPsec 795 encapsulation. 797 4.5. Why BGP as Control Plane for SDWAN? 799 For a small sized SDWAN network, traditional hub & spoke model using 800 NHRP or DSVPN/DMVPN with a hub node (or controller) managing SDWAN 801 node WAN ports mapping (e.g. local & public addresses and tunnel 802 identifiers mapping) can work reasonably well. However, for a large 803 SDWAN network, say more than 100 nodes with different types of 804 topologies, the traditional approach becomes very messy, complex and 805 error prone. 807 Here are some of the compelling reasons of using BGP instead of 808 extending NHRP/DSVPN/DMVPN. (Same as the reasons quoted by LSVR on 809 why using BGP): 811 - BGP has the built-in capability to constrain the propagation of 812 SDWAN edge node properties to a small number of edge nodes 813 [RFC4684]. 815 - RR already has the capability to apply policies to communications 816 among peers. 818 - BGP is widely deployed as sole protocol (see RFC 7938) 820 - Robust and simple implementation 822 - Wide acceptance - minimal learning 824 - Reliable transport 826 - Guaranteed in-order delivery 828 - Incremental updates 830 - Incremental updates upon session restart 832 - No flooding and selective filtering 834 5. SDWAN Traffic Forwarding Walk Through 836 BGP based EVPN control plane are still applicable to routes attached 837 to the client ports of SDWAN nodes. Section 5 of RFC8388 describes 838 the BGP EVPN NLRI Usage for various routes of client traffic. The 839 procedures described in the Section 6 of RFC8388 are same for the 840 SDWAN client traffic. 842 The only additional consideration for SDWAN is to control how 843 traffic egress the SDWAN edge node to various WAN ports. 845 5.1. SDWAN Network Startup Procedures 847 A SDWAN network can add or delete SDWAN edge nodes on regular basis 848 depending on user requests. 850 - For Scenario #1: a SDWAN edge node in a shopping mall or Cloud DC can 851 be added or removed on demand. The Zero Touch Provisioning described 852 in 3.1.2 are required for the node startup. 853 - For Scenario #2: this can be Data Centers or enterprises upgrading 854 their CPEs to add extra bandwidth via public internet in addition to 855 VPN services that they already purchased. Before the node powers up 856 or upgraded, there should be links connected to the PEs of a provider 857 VPNs. 858 - For Scenario #3, the Internet facing WAN ports are added to (or 859 removed from) existing VPN PEs. 861 5.2. Packet Walk-Through for Scenario #1 863 Upon power up, a SDWAN node can learn client routes from the Client 864 facing ports, in the same way as EVPN described in RFC8388. 865 Controller facilitates the IPsec SA establishment and rekey 866 management as described in [SECURE-EVPN]. Controller manages how 867 client's routes are associated with individual IPSec SA. 869 [SECURE-L3VPN] describes how to extend the RFC4364 VPN to allow some 870 PEs being connected to other PEs via public networks. [SECURE-L3VPN] 871 introduces the concept of Red Interface & Black Interface on those 872 PEs. RED interfaces face the VPN over which packets can be forwarded 873 natively without encryption. Black Interfaces face public network 874 over which only IPsec-protected packets are forwarded. 876 [SECURE-L3VPN] assumes PEs terminate MPLS packets, and use MPLS over 877 IPsec when sending over the Black Interfaces. 879 [SECURE-EVPN] describes a solution for SDWAN Scenario #1. It 880 utilizes the BGP RR to facilitate the key and policy exchange among 881 PE devices to create private pair-wise IPsec Security Associations 882 without IKEv2 point-to-point signaling or any other direct peer-to- 883 peer session establishment messages. 885 When C-PEs do not support MPLS, the approaches described by RFC8365 886 can be used, with addition of IPsec encrypting the IP packets when 887 sending packets over the Black Interfaces. 889 5.3. Packet Walk-Through for Scenario #2 891 In this scenario, C-PEs have some WAN ports connected to the public 892 internet and some WAN ports with direct connect to PEs of trusted 893 VPN. The C-PEs in Scenario #2 have the plain IP/Ethernet data frames 894 egress to the PEs of the VPN, encrypted data frames egress the WAN 895 ports facing the public Internet. 897 Users specify the policy or criteria on which flows can only egress 898 WAN ports facing the trusted VPN without encryption, which can 899 egress the WAN ports facing the public Internet with encryption, or 900 which can egress WAN ports facing the public Internet without 901 encryption. 903 The internet facing WAN ports can face potential DDoS attacks, 904 additional anti-DDoS mechanism has to be enabled on those WAN ports 905 and the Control Plane should not learn routes from the Public 906 Network facing WAN ports. 908 The Scenario #2 SDWAN Control Plane should include those three 909 distinct functional components: 911 +---+ 912 +--------------|RR |----------+ 913 / Untrusted +-+-+ \ 914 / Network \ 915 / \ 916 +----+ +---------+ packets encrypted over +--------+ +----+ 917 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 918 +----+ | C-PE-a A2-----+ +-------B2 C-PE-b| +----+ 919 |10.1.1.1 | |10.1.2.1| 920 +----+ | | +--+ +---+ | | +----+ 921 | CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3| 922 +----+ +---------+ +--+ trusted +---+ +--------+ +----+ 923 | VPN | 924 +--------------+ 925 Figure 8: SDWAN Scenario #2 927 - SDWAN node's WAN ports property registration to the SDWAN 928 Network Controller. 929 o See 5.3.1. for detail. 931 - Controller Facilitated IPsec SA management and NAT information 932 distribution 933 o See 5.3.2. for details. 935 - Attached routes distribution via BGP, which can be EVPN, IPVPN 936 or others. 937 o This is for the clients' route distribution, so that a C- 938 PE can establish the overlay routing table that identifies 939 the next hop for reaching a specific route/service 940 attached to remote nodes. [SECURE-EVPN] describes EVPN and 941 other options. 943 5.3.1. SDWAN node WAN Ports Properties Registration 945 In Figure 6, A1/A2/A3/B1/B2/B3 WAN ports can be from different 946 network providers. 948 +---+ 949 10.1.1.1 via +--------------|RR |----------+ 10.1.2.1 via 950 A1/A2/A3 / Untrusted +-+-+ \ B1/B2/B3 951 / \ 952 / \ 953 +----+ +---------+ packets encrypted over +--------+ +----+ 954 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 955 +----+ | C-PE1 A2-----+ +-------B2 C-PE2 | +----+ 956 |10.1.1.1 | |10.1.2.1| 957 +----+ | | +--+ +---+ | | +----+ 958 | CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3| 959 +----+ +---------+ +--+ trusted +---+ +--------+ +----+ 960 | VPN | 961 +--------------+ 962 Figure 9: SDWAN Scenario #2 WAN Ports Registration 964 Each SDWAN edge(C-PE) needs to register its WAN ports properties 965 along with its Loopback addresses to the SDWAN Network Controller. 966 The policies that govern the communications among peers are managed 967 and controlled by the SDWAN Controller. Individual SDWAN edge relies 968 on its SDWAN Controller to determine which peers can establish 969 connections. The SDWAN controller is responsible for propagating the 970 mapping information to the authorized peers. If C-PE-1 is not 971 authorized to communicate with C-PE-n, C-PE-1's WAN port<->Loopback 972 address mapping will not be propagated to C-PE-n. 974 A C-PE's Loopback addresses & attached routes may not be visible to 975 some ISPs/NSPs to which the CPE's WAN port is connected. 977 Section 4 describes how C-PEs use the BGP UPDATE messages to 978 propagate their local information to their corresponding RR. 980 5.3.2. Controller Facilitated IPsec SA & NAT management 982 Setting up and managing one IPsec SA between two points is 983 straightforward and simple. But managing multi-point IPsec SAs among 984 many points can be overwhelming. For a 1,000-node network, each node 985 may need to manage 999 IPSec SA keys to all their peers, which could 986 potentially result in 1,000,000 key exchanges to authenticate among 987 all nodes. In addition, when an edge node has multiple tenants 988 attached, the edge node may need to establish multiple tunnels to 989 isolate traffic from different tenants. When the SDWAN IPsec SAs are 990 fine-grained, such as per client address, per client's VLAN, the 991 number of IPsec SAs & Keys to be managed can go much higher, leading 992 to more IPsec management complexity. 994 All the IPsec keys have to be refreshed periodically. Therefore, 995 simplification facilitated by an SDWAN controller is necessary for 996 large-scale SDWAN deployment. 998 SDWAN edge nodes can rely on the SDWAN controller to facilitate the 999 pair-wise IPsec key establishment and refreshment [RFC7296] and 1000 maintain the Security Policy Database (SPD) [RFC4301]. 1002 - For the SDWAN Scenario #2 in described in Section 3.3. , if C-PE1 1003 & C-PE2 loopback addresses are not visible to the public network's 1004 ISP(s). C-PE1 & C-PE2 can use their provider assigned IP addresses 1005 for WAN ports A1/A2/B1/B2 to establish WAN Port based IPsec SA 1006 through the public network. Under this circumstance, there need to 1007 be minimum four IPsec SAs between C-PE1 & C-PE2 internet facing 1008 WAN ports. 1009 - When C-PEs loopback addresses are visible to ISPs/NSPs, i.e. the 1010 C-PEs' private source and destination IPs are part of a prefix 1011 exported to the ISP(s) in each site, it is possible to have one 1012 IPsec SA between C-PE1 & C-PE2. 1014 The IP addresses of SDWAN WAN port can be dynamic (e.g. assigned by 1015 DHCP) or private IP. Some SDWAN nodes are identified by "System-ID" 1016 or Loopback addresses that are only locally significant. In some 1017 SDWAN environments, "System-ID + PortID" are used to uniquely 1018 identify a SDWAN WAN port. Sometimes, a SDWAN tunnel end-point can 1019 be associated with "private IP" + "public IP" (if NAT is used.) 1021 When CPE WAN ports are private addresses, an additional sub-TLV has 1022 to be added to the [Tunnel-Encap] to describe the additional 1023 information about the NAT property of SDWAN nodes' WAN ports. A 1024 SDWAN node can inquire STUN (Session Traversal of UDP through 1025 Network Address Translation [RFC 3489]) Server to get the NAT 1026 property, the public IP address and the Public Port number to pass 1027 to the authorized peers via the SDWAN Controller. 1029 5.4. Packet Walk-Through for Scenario #3 1031 The behavior described in [SECURE-L3VPN] applies to this scenario, 1032 except C-PEs not only have RED interfaces facing clients but also 1033 have RED interface facing MPLS backbone, with additional BLACK 1034 interfaces facing the untrusted public networks for the WAN side. 1035 The C-PEs cannot mix the routes learned from the Black Interfaces 1036 with the Routes from RED Interfaces. The routes learned from core- 1037 facing RED interfaces are for underlay and cannot be mixed with the 1038 routes learned over access-facing RED interfaces that are for 1039 overlay. Furthermore, the routes learned over core-facing interfaces 1040 (both RED and BLACK) can be shared in the same GLOBAL route table. 1042 There may be some added risks of the packets from the ports facing 1043 the Internet. Therefore, special consideration has to be given to 1044 the routes from WAN ports facing the Internet. RFC4364 describes 1045 using an RD to create different routes for reaching same system. A 1046 similar approach can be considered to force packets received from 1047 the Internet facing ports to go through special security functions 1048 before being sent over to the VPN backbone WAN ports. 1050 6. Manageability Considerations 1052 SDWAN overlay networks utilize the SDWAN controller to facilitate 1053 route distribution, central configurations, and others. To 1054 minimize the burden on SDWAN edge nodes, SDWAN Edge nodes might 1055 not need to learn the routes from clients. 1057 7. Security Considerations 1059 Having WAN ports facing the public Internet introduces the following 1060 security risks: 1062 1) Potential DDoS attack to the C-PEs with ports facing internet. 1064 2) Potential risk of provider VPN network being injected with 1065 illegal traffic coming from the public Internet WAN ports on the C- 1066 PEs. 1068 8. IANA Considerations 1070 None 1072 9. References 1074 9.1. Normative References 1076 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1077 Requirement Levels", BCP 14, RFC 2119, March 1997. 1079 [RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private 1080 networks (VPNs)", Feb 2006. 1082 [RFC7296] C. Kaufman, et al, "Internet Key Exchange Protocol Version 1083 2 (IKEv2)", Oct 2014. 1085 [RFC7432] A. Sajassi, et al, "BGP MPLS-Based Ethernet VPN", Feb 1086 2015. 1088 [RFC8365] A. Sajassi, et al, "A network Virtualization Overlay 1089 Solution Using Ethernet VPN (EVPN)", March 2018. 1091 9.2. Informative References 1093 [RFC8192] S. Hares, et al, "Interface to Network Security Functions 1094 (I2NSF) Problem Statement and Use Cases", July 2017 1096 [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation Subsequent 1097 Address Family Identifier (SAFI) and the BGP Tunnel 1098 Encapsulation Attribute", April 2009. 1100 [BGP-SDWAN-Port] L. Dunbar, H. Wang, W. Hao, "BGP Extension for 1101 SDWAN Overlay Networks", draft-dunbar-idr-bgp-sdwan- 1102 overlay-ext-03, work-in-progress, Nov 2018. 1104 [Net2Cloud-Gap] L. Dunbar, A. Malis, C. Jacquenet, "Gap Analysis of 1105 Interconnecting Underlay with Cloud Overlay", draft-dm- 1106 net2cloud-gap-analysis-02, work in progress, Oct. 2018. 1108 [VPN-over-Internet] E. Rosen, "Provide Secure Layer L3VPNs over 1109 Public Infrastructure", draft-rosen-bess-secure-l3vpn-00, 1110 work-in-progress, July 2018 1112 [DMVPN] Dynamic Multi-point VPN: 1113 https://www.cisco.com/c/en/us/products/security/dynamic- 1114 multipoint-vpn-dmvpn/index.html 1116 [DSVPN] Dynamic Smart VPN: 1117 http://forum.huawei.com/enterprise/en/thread-390771-1- 1118 1.html 1120 [SECURE-EVPN] A. Sajassi, et al, "Secure EVPN", draft-sajassi-bess- 1121 secure-evpn-01, Work-in-progress, March 2019. 1123 [SECURE-L3VPN] E. Rosen, R. Bonica, "Secure Layer L3VPN over Public 1124 Infrastructure", draft-rosen-bess-secure-l3vpn-00, Work- 1125 in-progress, June 2018. 1127 [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation, 1128 storage, distribution and enforcement of policies for 1129 network security", Nov 2007. 1131 [Net2Cloud-Problem] L. Dunbar and A. Malis, "Seamless Interconnect 1132 Underlay to Cloud Overlay Problem Statement", draft-dm- 1133 net2cloud-problem-statement-02, June 2018 1135 [Net2Cloud-gap] L. Dunbar, A. Malis, and C. Jacquenet, "Gap Analysis 1136 of Interconnecting Underlay with Cloud Overlay", draft-dm- 1137 net2cloud-gap-analysis-02, work-in-progress, Aug 2018. 1139 [Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation 1140 Attribute", draft-ietf-idr-tunnel-encaps-10, Aug 2018. 1142 10. Acknowledgments 1144 Acknowledgements to Jim Guichard, John Scudder, Darren Dukes, Andy 1145 Malis and Donald Eastlake for their review and contributions. 1147 This document was prepared using 2-Word-v2.0.template.dot. 1149 Authors' Addresses 1151 Linda Dunbar 1152 Futurewei 1153 Email: ldunbar@futurewei.com 1155 James Guichard 1156 Futurewei 1157 Email: james.n.guichard@futurewei.com 1159 Ali Sajassi 1160 Cisco 1161 Email: sajassi@cisco.com 1163 John Drake 1164 Juniper 1165 Email: jdrake@juniper.net 1167 Basil Najem 1168 Bell Canada 1169 Email: basil.najem@bell.ca 1171 David Carrel 1172 Cisco 1173 Email: carrel@cisco.com 1175 Ayan Banerjee 1176 Cisco 1177 Email: ayabaner@cisco.com