idnits 2.17.1 draft-dunbar-bess-bgp-sdwan-usage-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 24 instances of too long lines in the document, the longest one being 5 characters in excess of 72. == There are 1 instance of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (January 31, 2020) is 1546 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC 4364' is mentioned on line 151, but not defined == Missing Reference: 'RFC4456' is mentioned on line 299, but not defined == Missing Reference: 'RFC4301' is mentioned on line 973, but not defined == Missing Reference: 'RFC 3489' is mentioned on line 998, but not defined ** Obsolete undefined reference: RFC 3489 (Obsoleted by RFC 5389) == Unused Reference: 'RFC2119' is defined on line 1048, but no explicit reference was found in the text == Unused Reference: 'RFC4364' is defined on line 1051, but no explicit reference was found in the text == Unused Reference: 'RFC7432' is defined on line 1057, but no explicit reference was found in the text == Unused Reference: 'RFC8365' is defined on line 1060, but no explicit reference was found in the text == Unused Reference: 'RFC8192' is defined on line 1065, but no explicit reference was found in the text == Unused Reference: 'RFC5521' is defined on line 1068, but no explicit reference was found in the text == Unused Reference: 'BGP-SDWAN-Port' is defined on line 1072, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-Gap' is defined on line 1076, but no explicit reference was found in the text == Unused Reference: 'VPN-over-Internet' is defined on line 1080, but no explicit reference was found in the text == Unused Reference: 'DMVPN' is defined on line 1084, but no explicit reference was found in the text == Unused Reference: 'DSVPN' is defined on line 1088, but no explicit reference was found in the text == Unused Reference: 'ITU-T-X1036' is defined on line 1099, but no explicit reference was found in the text == Unused Reference: 'Net2Cloud-gap' is defined on line 1107, but no explicit reference was found in the text == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 == Outdated reference: A later version (-06) exists of draft-sajassi-bess-secure-evpn-01 == Outdated reference: A later version (-01) exists of draft-rosen-bess-secure-l3vpn-00 -- Duplicate reference: draft-rosen-bess-secure-l3vpn, mentioned in 'SECURE-L3VPN', was also mentioned in 'VPN-over-Internet'. == Outdated reference: A later version (-07) exists of draft-dm-net2cloud-problem-statement-02 == Outdated reference: A later version (-04) exists of draft-dm-net2cloud-gap-analysis-02 -- Duplicate reference: draft-dm-net2cloud-gap-analysis, mentioned in 'Net2Cloud-gap', was also mentioned in 'Net2Cloud-Gap'. == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-10 Summary: 2 errors (**), 0 flaws (~~), 26 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft J. Guichard 3 Intended status: Informational Futurewei 4 Expires: July 31, 2020 Ali Sajassi 5 Cisco 6 J. Drake 7 Juniper 8 B. Najem 9 Bell Canada 10 Ayan Barnerjee 11 D. Carrel 12 Cisco 14 January 31, 2020 16 BGP Usage for SDWAN Overlay Networks 17 draft-dunbar-bess-bgp-sdwan-usage-04 19 Abstract 21 The document describes three distinct SDWAN scenarios and discusses 22 the applicability of BGP for each of those scenarios. The goal of 23 the document is to make it easier for future SDWAN control plane 24 protocols discussion. 26 SDWAN edge nodes are commonly interconnected by multiple underlay 27 networks which can be owned and managed by different network 28 providers. A BGP-based control plane is chosen for handling large 29 number of SDWAN edge nodes with little manual intervention. 31 Status of this Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. This document may not be modified, 38 and derivative works of it may not be created, except to publish it 39 as an RFC and to translate it into languages other than English. 41 Internet-Drafts are working documents of the Internet Engineering 42 Task Force (IETF), its areas, and its working groups. Note that 43 other groups may also distribute working documents as Internet- 44 Drafts. 46 Internet-Drafts are draft documents valid for a maximum of six 47 months and may be updated, replaced, or obsoleted by other documents 48 at any time. It is inappropriate to use Internet-Drafts as 49 reference material or to cite them other than as "work in progress." 51 The list of current Internet-Drafts can be accessed at 52 http://www.ietf.org/ietf/1id-abstracts.txt 54 The list of Internet-Draft Shadow Directories can be accessed at 55 http://www.ietf.org/shadow.html 57 This Internet-Draft will expire on July 31, 2020. 59 Copyright Notice 61 Copyright (c) 2020 IETF Trust and the persons identified as the 62 document authors. All rights reserved. 64 This document is subject to BCP 78 and the IETF Trust's Legal 65 Provisions Relating to IETF Documents 66 (http://trustee.ietf.org/license-info) in effect on the date of 67 publication of this document. Please review these documents 68 carefully, as they describe your rights and restrictions with 69 respect to this document. Code Components extracted from this 70 document must include Simplified BSD License text as described in 71 Section 4.e of the Trust Legal Provisions and are provided without 72 warranty as described in the Simplified BSD License. 74 Table of Contents 76 1. Introduction...................................................3 77 2. Conventions used in this document..............................4 78 3. Use Case Scenario Description and Requirements.................5 79 3.1. Requirements..............................................6 80 3.1.1. Client Service Requirement...........................6 81 3.1.2. Application Flow Based Segmentation..................6 82 3.1.3. SDWAN Node Provisioning..............................7 83 3.2. Scenarios #1: Homogeneous WAN.............................9 84 3.3. Scenario #2: SDWAN WAN ports to VPN's PEs and to Internet10 85 3.4. Scenario #3: SDWAN WAN ports to MPLS VPN and the Internet13 86 4. BGP Walk Through..............................................15 87 4.1. BGP Walk Through for Homogeneous SDWAN...................15 88 4.2. BGP Walk Through for Application Flow Based Segmentation.17 89 4.3. Client Service Provisioning Model........................18 90 4.4. WAN Ports Provisioning Model.............................19 91 4.5. Why BGP as Control Plane for SDWAN?......................19 92 5. SDWAN Traffic Forwarding Walk Through.........................20 93 5.1. SDWAN Network Startup Procedures.........................20 94 5.2. Packet Walk-Through for Scenario #1......................21 95 5.3. Packet Walk-Through for Scenario #2......................21 96 5.3.1. SDWAN node WAN Ports Properties Registration........22 97 5.3.2. Controller Facilitated IPsec SA & NAT management....23 98 5.4. Packet Walk-Through for Scenario #3......................24 99 6. Manageability Considerations..................................25 100 7. Security Considerations.......................................25 101 8. IANA Considerations...........................................25 102 9. References....................................................25 103 9.1. Normative References.....................................26 104 9.2. Informative References...................................26 105 10. Acknowledgments..............................................27 107 1. Introduction 109 There are three key characteristics of "SDWAN" networks: 111 - Augment of transport, which refers to utilizing overlay paths 112 over different underlay networks. Very often there are multiple 113 parallel overlay paths between any two SDWAN edges, some of 114 which are private networks over which traffic can traverse 115 without encryption, others require encryption, e.g. over 116 untrusted public networks. 117 - Enable direct Internet access from remote sites, instead hauling 118 all traffic to Corporate HQ for centralized policy control. 119 - Some traffic are routed based on application IDs instead of 120 based on destination IP addresses. 122 [Net2Cloud-Problem] describes the network related problems that 123 enterprises face to connect enterprises' branch offices to dynamic 124 workloads in different Cloud DCs, including using SDWAN to aggregate 125 multiple paths provided by different service providers to achieve 126 better performance. 128 Even though SDWAN has been positioned as a flexible way to reach 129 dynamic workloads in third party Cloud data centers over different 130 underlay networks, scaling becomes a major issue when there are 131 hundreds or thousands of nodes to be interconnected by an SDWAN 132 overlay networks. 134 BGP is widely used by underlay networks. This document describes 135 using BGP to enhance the scaling properties of SDWAN overlay 136 networks. 138 2. Conventions used in this document 140 Cloud DC: Third party data centers that host applications and 141 workloads owned by different organizations or tenants. 143 Controller: Used interchangeably with SDWAN controller to manage 144 SDWAN overlay path creation/deletion and monitor the 145 path conditions between sites. 147 CPE: Customer Premise Equipment 149 CPE-Based VPN: Virtual Private Secure network formed among CPEs. 150 This is to differentiate from more commonly used PE- 151 based VPNs [RFC 4364]. 153 Homogeneous SDWAN: A type of SDWAN network in which all traffic 154 to/from the SDWAN edge nodes has to be encrypted 155 regardless of underlay networks. For lack of better 156 terminology, we call this Homogeneous SDWAN throughout 157 this document. 159 ISP: Internet Service Provider 161 NSP: Network Service Provider. NSP usually provides more 162 advanced network services, such as MPLS VPN, private 163 leased lines, or managed Secure WAN connections, many 164 times within a private trusted domain, whereas an ISP 165 usually provides plain internet services over public 166 untrusted domains. 168 PE: Provider Edge 170 SDWAN End-point: a port (logical or physical) of a SDWAN edge node. 172 SDWAN: Software Defined Wide Area Network. In this document, 173 "SDWAN" refers to the solutions of pooling WAN bandwidth 174 from multiple underlay networks to get better WAN 175 bandwidth management, visibility & control. When the 176 underlay networks are private, traffic can traverse 177 without additional encryption; when the underlay 178 networks are public, such as the Internet, some traffic 179 may need to be encrypted when traversing through 180 (depending on user provided policies). 182 SDWAN IPsec SA: IPsec Security Association between two SDWAN ports 183 or nodes. 185 SDWAN over Hybrid Networks: SDWAN over Hybrid Networks typically 186 have edge nodes utilizing bandwidth resources from 187 multiple service providers. In Hybrid SDWAN network, 188 packets over private networks can go natively without 189 encryption and are encrypted over the untrusted network, 190 such as the public Internet. 192 WAN Port: A Port or Interface facing an ISP or Network Service 193 Provider (NSP), with address (usually public routable 194 address) allocated by the ISP or the NSP. 196 C-PE: SDWAN Edge node, which can be CPE for customer managed 197 SDWAN, or PE that is for provider managed SDWAN 198 services). 200 ZTP: Zero Touch Provisioning 202 3. Use Case Scenario Description and Requirements 204 SDWAN networks can have different topologies and have different 205 traffic patterns. To make it easier for the focused discussion in 206 subsequent drafts on SDWAN control plane and data plane, this 207 section describes several SDWAN scenarios that may have different 208 impact on their corresponding control planes & data planes. 210 3.1. Requirements 212 3.1.1. Client Service Requirement 214 Client interface of SDWAN nodes can be IP or Ethernet based. 216 For Ethernet based client interfaces, SDWAN edge should support 217 VLAN-based service interfaces (EVI100), VLAN bundle service 218 interfaces (EVI200), or VLAN-Aware bundling service interfaces. EVPN 219 service requirements are applicable to the Client traffic, as 220 described in the Section 3.1 of RFC8388. 222 For IP based client interfaces, L3VPN service requirements are 223 applicable. 225 3.1.2. Application Flow Based Segmentation 227 Application Flow based Segmentation, also known as SDWAN Traffic 228 Segmentation, enables the separation of the traffic based on the 229 business and the security needs for different users' groups and/or 230 application requirements. Each user group and/or applications may 231 need different isolated topology and/or policies to fulfill the 232 business requirements. 234 The Application Flow based Segmentation concept is analogous to VLAN 235 (in L2 network) and VRF (in L3 network). 237 One can think about the Application Flow based Segmentation as a 238 feature that can be provided or enabled on a single SDWAN service 239 (or domain) to a single Subscriber. Each SDWAN Service can have one 240 or more overlay Segments to support the business requirement; each 241 Segment has its own policy, topology and application/user groups. 242 Applications/users' group can belong to more than one Segment. 244 For example, a retail business requires the point-of-sales (PoS) 245 application in all stores to be isolated from other applications AND 246 routed only to the payment processing entity at a hub site (i.e. hub 247 and spoke); however, the same retail business requires the other 248 applications to be routed to all sites (i.e. multipoint-to 249 multipoint) AND isolated from the PoS application. 251 In the figure below, the traffic from the PoS application follows a 252 Tree topology, whereas other traffic can be multipoint-to-multipoint 253 topology. 255 +--------+ 256 Payment traffic |Payment | 257 +------+----+-+gateway +------+----+-----+ 258 / / | +----+---+ | \ \ 259 / / | | | \ \ 260 +-+--+ +-+--+ +-+--+ | +-+--+ +-+--+ +-+--+ 261 |Site| |Site| |Site| | |Site| |Site| |Site| 262 | 1 | | 2 | | 3 | | |4 | | 5 | | 6 | 263 +--+-+ +--+-+ +--|-+ | +--|-+ +--|-+ +--|-+ 264 | | | | | | | 265 ==+=======+=======+====+======+=======+=======+=== 266 Multi-point connection for Other traffic 268 Another example is an enterprise who wants to isolate the traffic 269 for each department and have different topology and policy for 270 different department; the HR department may need to access certain 271 applications that are NOT accessible by the engineering department. 272 In addition, the contractors may have a limited access to the 273 enterprise resources. 275 3.1.3. SDWAN Node Provisioning 277 Unlike traditional EVPN or L3VPN where PEs are deployed for long 278 term, SDWAN edge nodes (virtual or physical) deployment at a 279 specific location can be ephemeral. Therefore, Zero Touch 280 Provisioning (ZTP) is a common requirement for SDWAN. ZTP for SDWAN 281 can include many areas, but from network connectivity perspective, 282 ZTP should include the following: 284 - Upon power up, an SDWAN node can reach a central SDWAN 285 Controller (which can be burned or preconfigured in the device) 286 via a TLS or SSL secure channel. 288 - The Central SDWAN Controller can designate a Local Network 289 Controller in the proximity of the SDWAN node; the Local Network 290 Controller and the SDWAN nodes might be connected by third party 291 untrusted network. The Local controller does all the following 4 292 tasks: 294 1) ZTP 295 2) Auto-discovery of Network 296 3) (Auto)-Provisioning for IPsec SAs (initial provisioning 297 part) 298 4) Signaling of tenant's routes/info 299 BGP is well suited for (4), using Route Reflector (RR) [RFC4456] 300 to propagate network information among SDWAN edge nodes. The SDWAN 301 node can establish a secure connection (TLS, SSL, etc) to the 302 Local Network Controller (RR). 304 +---+ 305 Peer Group 1 |RR | Peer Group 2 306 +======+====+=+ +======+====+=====+ 307 / / | +---+ | \ \ 308 / / | | \ \ 309 +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ +-+--+ 310 |C-PE| |C-PE| |C-PE| |C-PE| |C-PE| |C-PE| 311 | 1 | | 2 | | 3 | |4 | | 5 | | 6 | 312 +----+ +----+ +----+ +----+ +----+ +----+ 313 Tenant 1 Tenant 2 314 Figure 1: Peer Groups managed by Local Controller 316 The SDWAN nodes (a.k.a. C-PEs throughout this document) belonging to 317 the same Tenant can be far apart and can be connected by third party 318 untrusted networks. Therefore, it is not appropriate for a SDWAN 319 edge node (C-PE) to advertise its SDWAN Port properties to its 320 neighbors. Each C-PE propagates its SDWAN Port attributes via the 321 secure channel (TLS, SSL, etc.) established with the Local 322 Controller. 324 C-PE-1 should include the following aspects in addition to managing 325 client routes: 326 - Register the SDWAN node's WAN port <-> local address mapping 327 to its Local Controller. The Local Controller propagates the 328 information to C-PE2 & C-PE3. 329 - Exchange IPsec property (capability such as the supported 330 encryption algorithms, etc.) and ports NAT property (e.g. 331 private addresses or dynamically assigned IP addresses) with 332 the Local Controller. 333 - C-PE2 and C-PE3 can establish IPsec SA with the C-PE1 after 334 receiving the information from the Local Controller. 335 - Then distribute the routes attached to the C-PE to its 336 authorized peers. 338 Tenant separation is achieved by the Local Controller creating 339 different Tenant based Peer Groups. 341 3.2. Scenarios #1: Homogeneous WAN 343 This is referring to a type of SDWAN network with edge nodes 344 encrypting all traffic over WAN to other edge nodes, regardless of 345 whether the underlay is private or public. For lack of better 346 terminology, we call this Homogeneous SDWAN throughout this 347 document. 349 Some typical scenarios for the use of a Homogeneous SDWAN network 350 are as follows: 352 - A small branch office connecting to its HQ offices via the 353 Internet. All sensitive traffic to/from this small branch office has 354 to be encrypted, which is usually achieved using IPsec SAs. 356 - A store in a shopping mall may need to securely connect to its 357 applications in one or more Cloud DCs via the Internet. A common way 358 of achieving this is to establish IPsec SAs to the Cloud DC gateway 359 to carry the sensitive data to/from the store. 361 As described in [SECURE-EVPN], the granularity of the IPsec SAs for 362 Homogeneous SDWAN can be per site, per subnet, per tenant, or per 363 address. Once the IPsec SA is established for a specific 364 subnet/tenant/site, all traffic to/from the subnets/tenants/site are 365 encrypted. 367 +---+ 368 +--------------|RR |------------+ 369 / Untrusted +-+-+ \ 370 / \ 371 / \ 372 +----+ +---------+ +------+ +----+ 373 | CN3|--| P1-----+ -------------+------ P1 |--| CN1| 374 +----+ | C-PE1 P2-----+ | | C-PE2| +----+ 375 +----+ | P3-----+ Wide +------ P2 | +----+ 376 | CN2|--| | | area +------ P3 |--| CN3| 377 +----+ +---------+ | network | +------+ +----+ 378 | | 379 +----+ +---------+ | all packets | +------+ +----+ 380 | CN1|--| P1-----+ encrypted +------ P1 |--| CN1| 381 +----+ | C-PE3 P2-----+ by | | C-PE4| +----+ 382 +----+ | P3-----+ IPsec SAs +------ P2 | +----+ 383 | CN2|--| P4-----+--------------+ | |--| CN2| 384 +----+ +---------+ +------+ +----+ 386 CN: Client Networks, which is same as Tenant Networks used by NVo3 388 Figure 2: Homogeneous SDWAN 390 One of the key properties of homogeneous SDWAN is that the SDWAN 391 Local Network Controller (RR)is connected to C-PEs via untrusted 392 public network, therefore, requiring secure connection between RR 393 and C-PEs (TLS, DTLS, etc.). 395 Homogeneous SDWAN has some similarity to commonly deployed IPsec 396 VPN, albeit the IPsec VPN is usually point-to-point among a small 397 number of endpoints and with heavy manual configuration for IPsec 398 between end-points, whereas an SDWAN network can have a large number 399 of end-points with an SDWAN controller to manage requiring zero 400 touch provisioning upon powering up. 402 Existing Private VPNs (e.g. MPLS based) can use homogeneous SDWAN to 403 extend over public network to remote sites to which the VPN operator 404 does not own or lease infrastructural connectivity, as described in 405 [SECURE-EVPN] and [SECURE-L3VPN] 407 3.3. Scenario #2: SDWAN WAN ports to VPN's PEs and to Internet 409 In this scenario, SDWAN edge nodes (a.k.a. C-PEs) have some WAN 410 ports connected to PEs of Private VPNs over which packets can be 411 forwarded natively without encryption, and some WAN ports connected 412 to the Internet over which sensitive traffic have to be encrypted 413 (usually by IPsec SA). 415 In this scenario, the SDWAN edge nodes' egress WAN ports are all 416 IP/Ethernet based, either egress to PEs of the VPNs or to the 417 Internet. Even if the VPN is a MPLS network, the VPN's PEs have 418 IP/Ethernet connections to the SDWAN edge (C-PEs). Throughout this 419 document, this scenario is also called CPE based SDWAN over Hybrid 420 Networks. 422 Even though IPsec SA can secure the packets traversing the Internet, 423 it does not offer the premium SLA commonly offered by Private VPNs, 424 especially over long distance. Clients need to have policies to 425 specify criteria for flows only traversing private VPNs or 426 traversing either as long as encrypted when over the Internet. For 427 example, client can have those polices for the flows: 429 1. A policy or criteria for sending the flows over a private 430 network without encryption (for better performance), 431 2. A policy or criteria for sending the flows over any networks 432 as long as the packets of the flows are encrypted when 433 traversing untrusted networks, or 434 3. A policy of not needing encryption at all. 436 If a flow traversing multiple segments, such as A<->B<->C<->D, has 437 either Policy 2 or 3 above, the flow can traverse different 438 underlays in different segments, such as over Private network 439 underlay between A<->B without encryption, or over the public 440 internet between B<->C in an IPsec SA. 442 As shown in the figure below, C-PE-1 has two different types of 443 interfaces (A1 to Internet and A2 & A3 to VPN). The C-PEs' loopback 444 addresses and addresses attached to C-PEs may or may not be visible 445 to the ISPs/NSPs. The addresses for the WAN ports can have addresses 446 allocated by service providers or dynamically assigned (e.g. by 447 DHCP). One WAN port shown in the figure below (e.g. A1, A2, A3 etc.) 448 is a logical representation of potential multiple physical ports on 449 the C-PEs. 451 +---+ 452 +--------------|RR |----------+ 453 / Untrusted +-+-+ \ 454 / \ 455 / \ 456 +----+ +---------+ packets encrypted over +------+ +----+ 457 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 458 +----+ | C-PE1 A2-\ | C-PE2| +----+ 459 +----+ | A3--+--+ +---+---B2 | +----+ 460 | CN2|--| | |PE+--------------+PE |---B3 |--| CN3| 461 +----+ +---------+ +--+ trusted +---+ +------+ +----+ 462 | WAN | 463 +----+ +---------+ +--+ packets +---+ +------+ +----+ 464 | CN1|--| C1--|PE| go natively |PE |-- D1 |--| CN1| 465 +----+ | C-PE3 C2--+--+ without encry+---+ | C-PE4| +----+ 466 | | +--------------+ | | 467 | | | | 468 +----+ | | without encrypt over | | +----+ 469 | CN2|--| C3--+---- Untrusted --+------D2 |--| CN2| 470 +----+ +---------+ +------+ +----+ 472 CN: Client Network 473 Figure 3: Hybrid SDWAN 475 Some key characteristics of a Hybrid SDWAN overlay network are as 476 follows: 478 - one C-PE may be connected to different ISPs/NSPs, with some of its 479 WAN ports addresses being assigned by different ISPs/NSPs. 481 - The WAN ports connected to PEs of trusted private networks (e.g. MPLS 482 VPN) hand off IP/Ethernet packets, just like today's CPE that do not 483 handle MPLS packets and do not participate in the underlay VPN networks' 484 control plane. Traffic can flow natively without encryption when be 485 forwarded out through those WAN ports for better performance. 487 - The WAN ports connected to untrusted networks, e.g. the Internet, 488 requires sensitive traffic to be encrypted, i.e. encrypted by IPsec SA. 490 - An SDWAN local Network Controller (RR) is connected to C-PEs via 491 the untrusted public network, therefore, requiring secure 492 connection between RR and C-PEs via TLS, DTLS, etc. 494 - The SDWAN nodes' [loopback] addresses might not be routable nor 495 visible in the underlay ISP/NSP networks. Routes & services 496 attached to SDWAN edges at the SDWAN overlay layer are in 497 different address spaces than the underlay networks. 499 - There could be multiple SDWAN devices sharing a common property, 500 such as a geographic location. Some applications over SDWAN may 501 need to traverse specific geographic locations for various 502 reasons, such as to comply with regulatory rules, to utilize 503 specific value added services, or others. 505 - The underlay path selection between sites can be a local decision. 506 Some policies allow one service from C-PE1 -> C-PE2 -> C-PE3 using 507 one ISP/NSP underlay in the first segment (C-PE1 -> C-PE2) and 508 using a different ISP/NSP in the second segment (C-PE2-> CPE3). 510 - Services may not be congruent, i.e. the packets from A-> B may 511 traverse one underlay network, and the packets from B -> A may 512 traverse a different underlay. 514 - Different services, routes, or VLANs attached to SDWAN nodes can 515 be aggregated over one underlay path; same service/routes/VLAN can 516 spread over multiple SDWAN underlays at different times depending 517 on the policies specified for the service. For example, one 518 tenant's packets to HQ need to be encrypted when sent over the 519 Internet or have to be sent over private networks, while the same 520 tenant's packets to Facebook can be sent over the Internet without 521 encryption. 523 3.4. Scenario #3: SDWAN WAN ports to MPLS VPN and the Internet 525 This scenario refers to existing VPN (e.g. MPLS based VPN, such as 526 EVPN or IPVPN) adding extra ports facing untrusted public networks 527 allowing PEs to offload some low priority traffic to ports facing 528 public networks when the VPN MPLS paths are congested. Throughout 529 this document, this scenario is also called Internet Offload for 530 Private VPN, or PE based SDWAN. 532 In this scenario, the packets offloaded to untrusted public network 533 must be encrypted. 535 PE based SDWAN can be used by VPN service providers to temporarily 536 increase bandwidth between sites when they are not sure if the 537 demand will sustain for long period of time or as a temporary 538 solution before the permanent infrastructure is built or leased. 540 +---+ 541 +-------|PE2| 542 / +---+ 543 Internet / ^ 544 offload / || VPN 545 / VPN v 546 ++--+ ++-+ +---+ 547 |PE1| <====> |RR| <===> |PE3| 548 +-+-+ +--+ +-+-+ 549 | | 550 +--- Public Internet -- + 552 Figure 4: Additional Internet paths added to the VPN 554 Here are some key properties for PE based SDWAN: 556 - For MPLS based VPN, PEs continue having MPLS encapsulation 557 handoff to existing paths. 558 - The BGP RR is connected to PEs in the same way as VPN, i.e. 559 via the trusted network. 560 - For the added Internet ports, PEs have IP packets handoff, 561 i.e. sending and receiving IP data frames. Internally, PEs 562 can have the option to encapsulate the MPLS payload in IP, as 563 specified by RFC4023. 564 - The ports facing public internet might get IP addresses 565 assigned by ISPs, which may not be in the same address domain 566 as PEs'. 567 - Ports facing public internet are not as secure as the ports 568 facing private infrastructure. There could be spoofing, or 569 DDOS attacks to the ports facing public internet. Extra 570 consideration must be given when injecting the new routes 571 learned from public network into VRFs. 572 - Even though packets are encrypted over public internet, the 573 performance SLA is not guaranteed over public internet. 575 Therefore, clients may have policies only allowing some flows 576 to be offloaded to internet path. 578 4. BGP Walk Through 580 4.1. BGP Walk Through for Homogeneous SDWAN 582 In the figure below, packets destined towards multiple routes 583 attached to the C-PE2 can be carried by one IPsec tunnel. Then one 584 BGP UPDATE can be announced by C-PE2 to its RR. 586 +---+ 587 +---------|RR |----------+ 588 / Untrusted+---+ \ 589 / \ 590 C-PE1/ \ 591 +---------+ +------+ 592 --+---+---------------------------------> |-10.1.x.x/16 593 | / | |C-PE2 |- VLAN = 15 594 | / | +-----> | 595 --|/ | | | |-12.1.1.x/24 596 +---------+ | +------+ 597 | 598 C-PE3 | 599 +---------+ | 600 --|---+---------------------------+ 601 | / | 602 | / | 603 --|/ | 604 +---------+ 605 Figure 5: (see *.pdf for more accurate figure) 607 The BGP UPDATE Message from C-PE2 to RR should have the client 608 routes encoded in the MP-NLRI Path Attribute and the IPsec Tunnel 609 associated information encoded in the Tunnel-Encap Path Attributes 610 as described in the [SECURE-EVPN]: 612 - MP-NLRI Path Attribute: to indicate multiple routes attached to 613 the C-PE2: 614 10.1.x.x/16 615 VLAN #15 616 12.1.1.x/24 618 - Tunnel-Encap Path Attribute: to describe the IPsec attributes 619 for routes encoded in the NLRI Path Attribute: 620 IPsec attributes for remote nodes to establish the IPsec 621 tunnel to C-PE2. 623 If different client routes attached to C-PE2 needs to be reached by 624 separate IPsec tunnels, then multiple BGP UPDATE messages need to be 625 sent to the remote nodes. If C-PE2 doesn't have the policy on 626 mapping of clients' routes to IPsec tunnels, RR needs to check the 627 client routes policies to send separate BGP UPDATE messages to the 628 remote edge nodes. 630 There could be policies governing the topologies of a client's 631 different routes attached to an edge node. For example, VLAN #25 and 632 route 22.1.1.x/24 could be the Payment Applications described in the 633 Section 3.1.2 that can only communicate with Payment Gateway 634 attached to C-PE3. If C-PEs don't have the policy to govern the 635 communication peers, RR can take over the responsibility of only 636 send BGP UPDATE for the Blue routes to C-PE1 and send the RED routes 637 to C-PE3. 639 +---+ 640 +-----------|RR |----------+ 641 / Untrusted +---+ \ 642 / \ 643 / \ 644 +---------+ +------+ 645 --+ --------------------------------------> |-10.1.x.x/16 646 | C-PE1 | |C-PE2 |- VLAN = 15 647 | | +-----------> |- 12.1.1.x/24 648 --|---------------------------+ | | 649 +---------+ +------> |- VLAN=25 650 / +------+ 22.1.1.x/24 651 +---------+ / 652 --| ----------------------------+ 653 | C-PE3 | / 654 | | / 655 --| -------------------------+ 656 +---------+ 658 Figure 6: (see *.pdf for more accurate figure) 660 UPDATE 1: 661 - MP-NLRI Path Attribute: 662 10.1.x.x/16 663 VLAN #15 664 12.1.1.x/24 666 - Tunnel-Encap Path Attribute: 667 IPsec SA attributes for IPsec tunnels to C-PE2 from any node 668 for reaching 10.1.x.x/16, VLAN #15, and 12.1.1.x/24. 670 UPDATE 2 (only sent to C-PE3) 671 - MP-NLRI Path Attribute: 672 VLAN #25 673 22.1.1.x/24 675 - Tunnel-Encap: 676 IPsec SA attributes for IPsec tunnels to C-PE2 from C-PE3 for 677 reaching VLAN #25 and subnet 22.1.1./24. 679 4.2. BGP Walk Through for Application Flow Based Segmentation 681 If the applications are assigned with unique IP addresses, the 682 Application Flow based Segmentation described in Section 3.1.2 can 683 be achieved by advertising different BGP UPDATE messages to 684 different nodes. In the Figure below, the following BGP Updates can 685 be advertised to ensure that Payment Application only communicates 686 with the Payment Gateway: 688 BGP UPDATE #1 from C-PE2 to RR for the RED P2P topology (only 689 propagated to Payment GW node: 691 - MP-NLRI Path Attribute: 692 - 30.1.1.x/24 693 - Tunnel Encap Path Attribute 694 - IPsec Attributes for PaymentGW ->C-PE2 696 BGP UPDATE #2 from C-PE2 to RR for the routes to be reached by 697 Purple: 699 - MP-NLRI Path Attribute: 701 - 10.1.x.x 702 - 12.4.x.x 703 - TunnelEncap Path Attribute: 704 - Any node to C-PE2 706 +-------+ 707 |Payment| 708 +-------->| GW |<-------+ 709 /Hub-spoke +-------+ \ 710 /for Payment Ppp \ 711 C-PE1 / \ C-PE2 712 +------/--+ +----\-+ 713 --|-----/ | | -|- 30.1.1.x/24 714 + ---------------------------------------> |-10.1.x.x/16 715 | | | |- 716 | | +------------> |- 12.1.1.x/24 717 --|---------------------------+ | | 718 +---------+ +------> |- VLAN=25; 719 / +------+ 22.1.1.x/24 720 +---------+ / 721 --| -----------------------------+ 722 | C-PE3 | / 723 | | / 724 --| --------------------------+ 725 +---------+ 726 Figure 7: (see *.pdf for more accurate figure) 728 4.3. Client Service Provisioning Model 730 The provisioning tasks described in Section 4 of RFC8388 are the 731 same for the SDWAN client traffic. When client traffic is multi- 732 homed to two (or more) C-PEs, the Non-Service-Specific parameters 733 need to be provisioned per the Section 4.1.1 of RFC8388. 735 Since most SDWAN nodes are ephemeral and have small number of IP 736 subnets or VLANs attached to their client ports, it is recommended 737 to have default and simplified Service-specific parameters for each 738 client port, remotely managed by the SDWAN Network Controller via 739 the secure channel (TLS/DTLS) between the controller and the C-PEs. 741 4.4. WAN Ports Provisioning Model 743 Since the deployment of PEs to MPLS VPN are for relatively long 744 term, the common provisioning procedure for PE's WAN ports is via 745 CLI. 747 A SDWAN node deployment can be ephemeral and its location can be in 748 remote locations, manual provisioning for its WAN ports is not 749 acceptable. In addition, a SDWAN WAN port's IP address can be 750 dynamically assigned or using private addresses. Therefore, it is 751 necessary to have a separate control protocol; something like NHRP 752 did for ATM, for a SDWAN node to register its WAN property to its 753 controller dynamically. 755 Unlike a PE to MPLS based VPN where its WAN ports are homogeneously 756 facing MPLS private network and all traffic are egressed in MPLS 757 data frames through its WAN ports, the WAN ports of a SDWAN node can 758 be connected to a PE of VPN with Ethernet/IP, MPLS private network 759 directly via MPLS headers, or the public Internet. 761 For Scenario #1 described in Section 3.2, the WAN ports can face 762 public internet or VPN. 764 For Scenario #2 described in Section 3.3, WAN ports are either 765 configured as connecting to PEs of VPN where traffic can be sent as 766 IP/Ethernet without encryption, or configured as connecting to 767 public Internet that requires encryption for packets egress out. 769 For Scenario #3 described in Section 3.4, the WAN ports are either 770 configured as VPN egress ports (hand off MPLS data frames), or as 771 connecting to the public internet that requires MPLS in IP in IPsec 772 encapsulation. 774 4.5. Why BGP as Control Plane for SDWAN? 776 For a small sized SDWAN network, traditional hub & spoke model using 777 NHRP or DSVPN/DMVPN with a hub node (or controller) managing SDWAN 778 node WAN ports mapping (e.g. local & public addresses and tunnel 779 identifiers mapping) can work reasonably well. However, for a large 780 SDWAN network, say more than 100 nodes with different types of 781 topologies, the traditional approach becomes very messy, complex and 782 error prone. 784 Here are some of the compelling reasons of using BGP instead of 785 extending NHRP/DSVPN/DMVPN. (Same as the reasons quoted by LSVR on 786 why using BGP): 788 - BGP already widely deployed as sole protocol (see RFC 7938) 790 - Robust and simple implementation 792 - Wide acceptance - minimal learning 794 - Reliable transport 796 - Guaranteed in-order delivery 798 - Incremental updates 800 - Incremental updates upon session restart 802 - No flooding and selective filtering 804 - RR already has the capability to apply policies to communications 805 among peers. 807 5. SDWAN Traffic Forwarding Walk Through 809 BGP based EVPN control plane are still applicable to routes attached 810 to the client ports of SDWAN nodes. Section 5 of RFC8388 describes 811 the BGP EVPN NLRI Usage for various routes of client traffic. The 812 procedures described in the Section 6 of RFC8388 are same for the 813 SDWAN client traffic. 815 The only additional consideration for SDWAN is to control how 816 traffic egress the SDWAN edge node to various WAN ports. 818 5.1. SDWAN Network Startup Procedures 820 A SDWAN network can add or delete SDWAN edge nodes on regular basis 821 depending on user requests. 823 - For Scenario #1: a SDWAN edge node in a shopping mall or Cloud DC can 824 be added or removed on demand. The Zero Touch Provisioning described 825 in 3.1.2 are required for the node startup. 826 - For Scenario #2: this can be Data Centers or enterprises upgrading 827 their CPEs to add extra bandwidth via public internet in addition to 828 VPN services that they already purchased. Before the node powers up 829 or upgraded, there should be links connected to the PEs of a provider 830 VPNs. 831 - For Scenario #3, the Internet facing WAN ports are added to (or 832 removed from) existing VPN PEs. 834 5.2. Packet Walk-Through for Scenario #1 836 Upon power up, a SDWAN node can learn client routes from the Client 837 facing ports, in the same way as EVPN described in RFC8388. 838 Controller facilitates the IPsec SA establishment and rekey 839 management as described in [SECURE-EVPN]. Controller manages how 840 client's routes are associated with individual IPSec SA. 842 [SECURE-L3VPN] describes how to extend the RFC4364 VPN to allow some 843 PEs being connected to other PEs via public networks. [SECURE-L3VPN] 844 introduces the concept of Red Interface & Black Interface on those 845 PEs. RED interfaces face the VPN over which packets can be forwarded 846 natively without encryption. Black Interfaces face public network 847 over which only IPsec-protected packets are forwarded. 849 [SECURE-L3VPN] assumes PEs terminate MPLS packets, and use MPLS over 850 IPsec when sending over the Black Interfaces. 852 [SECURE-EVPN] describes a solution for SDWAN Scenario #1. It 853 utilizes the BGP RR to facilitate the key and policy exchange among 854 PE devices to create private pair-wise IPsec Security Associations 855 without IKEv2 point-to-point signaling or any other direct peer-to- 856 peer session establishment messages. 858 When C-PEs do not support MPLS, the approaches described by RFC8365 859 can be used, with addition of IPsec encrypting the IP packets when 860 sending packets over the Black Interfaces. 862 5.3. Packet Walk-Through for Scenario #2 864 In this scenario, C-PEs have some WAN ports connected to the public 865 internet and some WAN ports with direct connect to PEs of trusted 866 VPN. The C-PEs in Scenario #2 have the plain IP/Ethernet data frames 867 egress to the PEs of the VPN, encrypted data frames egress the WAN 868 ports facing the public Internet. 870 Users specify the policy or criteria on which flows can only egress 871 WAN ports facing the trusted VPN without encryption, which can 872 egress the WAN ports facing the public Internet with encryption, or 873 which can egress WAN ports facing the public Internet without 874 encryption. 876 The internet facing WAN ports can face potential DDoS attacks, 877 additional anti-DDoS mechanism has to be enabled on those WAN ports 878 and the Control Plane should not learn routes from the Public 879 Network facing WAN ports. 881 The Scenario #2 SDWAN Control Plane should include those three 882 distinct functional components: 884 +---+ 885 +--------------|RR |----------+ 886 / Untrusted +-+-+ \ 887 / Network \ 888 / \ 889 +----+ +---------+ packets encrypted over +--------+ +----+ 890 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 891 +----+ | C-PE-a A2-----+ +-------B2 C-PE-b| +----+ 892 |10.1.1.1 | |10.1.2.1| 893 +----+ | | +--+ +---+ | | +----+ 894 | CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3| 895 +----+ +---------+ +--+ trusted +---+ +--------+ +----+ 896 | VPN | 897 +--------------+ 898 Figure 8: SDWAN Scenario #2 900 - SDWAN node's WAN ports property registration to the SDWAN 901 Network Controller. 902 o See 5.3.1. for detail. 904 - Controller Facilitated IPsec SA management and NAT information 905 distribution 906 o See 5.3.2. for details. 908 - Attached routes distribution via BGP, which can be EVPN, IPVPN 909 or others. 910 o This is for the clients' route distribution, so that a C- 911 PE can establish the overlay routing table that identifies 912 the next hop for reaching a specific route/service 913 attached to remote nodes. [SECURE-EVPN] describes EVPN and 914 other options. 916 5.3.1. SDWAN node WAN Ports Properties Registration 918 In Figure 6, A1/A2/A3/B1/B2/B3 WAN ports can be from different 919 network providers. 921 +---+ 922 10.1.1.1 via +--------------|RR |----------+ 10.1.2.1 via 923 A1/A2/A3 / Untrusted +-+-+ \ B1/B2/B3 924 / \ 925 / \ 926 +----+ +---------+ packets encrypted over +--------+ +----+ 927 | CN3|--| A1-----+ Untrusted +------ B1 |--| CN1| 928 +----+ | C-PE1 A2-----+ +-------B2 C-PE2 | +----+ 929 |10.1.1.1 | |10.1.2.1| 930 +----+ | | +--+ +---+ | | +----+ 931 | CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3| 932 +----+ +---------+ +--+ trusted +---+ +--------+ +----+ 933 | VPN | 934 +--------------+ 935 Figure 9: SDWAN Scenario #2 WAN Ports Registration 937 Each SDWAN edge(C-PE) needs to register its WAN ports properties 938 along with its Loopback addresses to the SDWAN Network Controller. 939 The policies that govern the communications among peers are managed 940 and controlled by the SDWAN Controller. Individual SDWAN edge relies 941 on its SDWAN Controller to determine which peers can establish 942 connections. The SDWAN controller is responsible for propagating the 943 mapping information to the authorized peers. If C-PE-1 is not 944 authorized to communicate with C-PE-n, C-PE-1's WAN port<->Loopback 945 address mapping will not be propagated to C-PE-n. 947 A C-PE's Loopback addresses & attached routes may not be visible to 948 some ISPs/NSPs to which the CPE's WAN port is connected. 950 Section 4 describes how C-PEs use the BGP UPDATE messages to 951 propagate their local information to their corresponding RR. 953 5.3.2. Controller Facilitated IPsec SA & NAT management 955 Setting up and managing one IPsec SA between two points is 956 straightforward and simple. But managing multi-point IPsec SAs among 957 many points can be overwhelming. For a 1,000-node network, each node 958 may need to manage 999 IPSec SA keys to all their peers, which could 959 potentially result in 1,000,000 key exchanges to authenticate among 960 all nodes. In addition, when an edge node has multiple tenants 961 attached, the edge node may need to establish multiple tunnels to 962 isolate traffic from different tenants. When the SDWAN IPsec SAs are 963 fine-grained, such as per client address, per client's VLAN, the 964 number of IPsec SAs & Keys to be managed can go much higher, leading 965 to more IPsec management complexity. 967 All the IPsec keys have to be refreshed periodically. Therefore, 968 simplification facilitated by an SDWAN controller is necessary for 969 large-scale SDWAN deployment. 971 SDWAN edge nodes can rely on the SDWAN controller to facilitate the 972 pair-wise IPsec key establishment and refreshment [RFC7296] and 973 maintain the Security Policy Database (SPD) [RFC4301]. 975 - For the SDWAN Scenario #2 in described in Section 3.3. , if C-PE1 976 & C-PE2 loopback addresses are not visible to the public network's 977 ISP(s). C-PE1 & C-PE2 can use their provider assigned IP addresses 978 for WAN ports A1/A2/B1/B2 to establish WAN Port based IPsec SA 979 through the public network. Under this circumstance, there need to 980 be minimum four IPsec SAs between C-PE1 & C-PE2 internet facing 981 WAN ports. 982 - When C-PEs loopback addresses are visible to ISPs/NSPs, i.e. the 983 C-PEs' private source and destination IPs are part of a prefix 984 exported to the ISP(s) in each site, it is possible to have one 985 IPsec SA between C-PE1 & C-PE2. 987 The IP addresses of SDWAN WAN port can be dynamic (e.g. assigned by 988 DHCP) or private IP. Some SDWAN nodes are identified by "System-ID" 989 or Loopback addresses that are only locally significant. In some 990 SDWAN environments, "System-ID + PortID" are used to uniquely 991 identify a SDWAN WAN port. Sometimes, a SDWAN tunnel end-point can 992 be associated with "private IP" + "public IP" (if NAT is used.) 994 When CPE WAN ports are private addresses, an additional sub-TLV has 995 to be added to the [Tunnel-Encap] to describe the additional 996 information about the NAT property of SDWAN nodes' WAN ports. A 997 SDWAN node can inquire STUN (Session Traversal of UDP through 998 Network Address Translation [RFC 3489]) Server to get the NAT 999 property, the public IP address and the Public Port number to pass 1000 to the authorized peers via the SDWAN Controller. 1002 5.4. Packet Walk-Through for Scenario #3 1004 The behavior described in [SECURE-L3VPN] applies to this scenario, 1005 except C-PEs not only have RED interfaces facing clients but also 1006 have RED interface facing MPLS backbone, with additional BLACK 1007 interfaces facing the untrusted public networks for the WAN side. 1008 The C-PEs cannot mix the routes learned from the Black Interfaces 1009 with the Routes from RED Interfaces. The routes learned from core- 1010 facing RED interfaces are for underlay and cannot be mixed with the 1011 routes learned over access-facing RED interfaces that are for 1012 overlay. Furthermore, the routes learned over core-facing interfaces 1013 (both RED and BLACK) can be shared in the same GLOBAL route table. 1015 There may be some added risks of the packets from the ports facing 1016 the Internet. Therefore, special consideration has to be given to 1017 the routes from WAN ports facing the Internet. RFC4364 describes 1018 using an RD to create different routes for reaching same system. A 1019 similar approach can be considered to force packets received from 1020 the Internet facing ports to go through special security functions 1021 before being sent over to the VPN backbone WAN ports. 1023 6. Manageability Considerations 1025 SDWAN overlay networks utilize the SDWAN controller to facilitate 1026 route distribution, central configurations, and others. To 1027 minimize the burden on SDWAN edge nodes, SDWAN Edge nodes might 1028 not need to learn the routes from clients. 1030 7. Security Considerations 1032 Having WAN ports facing the public Internet introduces the following 1033 security risks: 1035 1) Potential DDoS attack to the C-PEs with ports facing internet. 1037 2) Potential risk of provider VPN network being injected with 1038 illegal traffic coming from the public Internet WAN ports on the C- 1039 PEs. 1041 8. IANA Considerations 1043 None 1045 9. References 1046 9.1. Normative References 1048 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1049 Requirement Levels", BCP 14, RFC 2119, March 1997. 1051 [RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private 1052 networks (VPNs)", Feb 2006. 1054 [RFC7296] C. Kaufman, et al, "Internet Key Exchange Protocol Version 1055 2 (IKEv2)", Oct 2014. 1057 [RFC7432] A. Sajassi, et al, "BGP MPLS-Based Ethernet VPN", Feb 1058 2015. 1060 [RFC8365] A. Sajassi, et al, "A network Virtualization Overlay 1061 Solution Using Ethernet VPN (EVPN)", March 2018. 1063 9.2. Informative References 1065 [RFC8192] S. Hares, et al, "Interface to Network Security Functions 1066 (I2NSF) Problem Statement and Use Cases", July 2017 1068 [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation Subsequent 1069 Address Family Identifier (SAFI) and the BGP Tunnel 1070 Encapsulation Attribute", April 2009. 1072 [BGP-SDWAN-Port] L. Dunbar, H. Wang, W. Hao, "BGP Extension for 1073 SDWAN Overlay Networks", draft-dunbar-idr-bgp-sdwan- 1074 overlay-ext-03, work-in-progress, Nov 2018. 1076 [Net2Cloud-Gap] L. Dunbar, A. Malis, C. Jacquenet, "Gap Analysis of 1077 Interconnecting Underlay with Cloud Overlay", draft-dm- 1078 net2cloud-gap-analysis-02, work in progress, Oct. 2018. 1080 [VPN-over-Internet] E. Rosen, "Provide Secure Layer L3VPNs over 1081 Public Infrastructure", draft-rosen-bess-secure-l3vpn-00, 1082 work-in-progress, July 2018 1084 [DMVPN] Dynamic Multi-point VPN: 1085 https://www.cisco.com/c/en/us/products/security/dynamic- 1086 multipoint-vpn-dmvpn/index.html 1088 [DSVPN] Dynamic Smart VPN: 1089 http://forum.huawei.com/enterprise/en/thread-390771-1- 1090 1.html 1092 [SECURE-EVPN] A. Sajassi, et al, "Secure EVPN", draft-sajassi-bess- 1093 secure-evpn-01, Work-in-progress, March 2019. 1095 [SECURE-L3VPN] E. Rosen, R. Bonica, "Secure Layer L3VPN over Public 1096 Infrastructure", draft-rosen-bess-secure-l3vpn-00, Work- 1097 in-progress, June 2018. 1099 [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation, 1100 storage, distribution and enforcement of policies for 1101 network security", Nov 2007. 1103 [Net2Cloud-Problem] L. Dunbar and A. Malis, "Seamless Interconnect 1104 Underlay to Cloud Overlay Problem Statement", draft-dm- 1105 net2cloud-problem-statement-02, June 2018 1107 [Net2Cloud-gap] L. Dunbar, A. Malis, and C. Jacquenet, "Gap Analysis 1108 of Interconnecting Underlay with Cloud Overlay", draft-dm- 1109 net2cloud-gap-analysis-02, work-in-progress, Aug 2018. 1111 [Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation 1112 Attribute", draft-ietf-idr-tunnel-encaps-10, Aug 2018. 1114 10. Acknowledgments 1116 Acknowledgements to Jim Guichard, John Scudder, Darren Dukes, Andy 1117 Malis and Donald Eastlake for their review and contributions. 1119 This document was prepared using 2-Word-v2.0.template.dot. 1121 Authors' Addresses 1123 Linda Dunbar 1124 Futurewei 1125 Email: ldunbar@futurewei.com 1127 James Guichard 1128 Futurewei 1129 Email: james.n.guichard@futurewei.com 1131 Ali Sajassi 1132 Cisco 1133 Email: sajassi@cisco.com 1135 John Drake 1136 Juniper 1137 Email: jdrake@juniper.net 1139 Basil Najem 1140 Bell Canada 1141 Email: basil.najem@bell.ca 1143 David Carrel 1144 Cisco 1145 Email: carrel@cisco.com 1147 Ayan Banerjee 1148 Cisco 1149 Email: ayabaner@cisco.com