idnits 2.17.1 draft-hares-idr-bgp-ipsec-analysis-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 14, 2020) is 1375 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-04) exists of draft-dunbar-idr-sdwan-edge-discovery-00 == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-16 == Outdated reference: A later version (-06) exists of draft-sajassi-bess-secure-evpn-03 == Outdated reference: A later version (-09) exists of draft-ietf-rtgwg-net2cloud-gap-analysis-06 == Outdated reference: A later version (-39) exists of draft-ietf-rtgwg-net2cloud-problem-statement-10 -- Obsolete informational reference (is this intentional?): RFC 5512 (Obsoleted by RFC 9012) -- Obsolete informational reference (is this intentional?): RFC 5566 (Obsoleted by RFC 9012) Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 IDR Working Group S. Hares 3 Internet-Draft Hickory Hill Consulting 4 Intended status: Informational July 14, 2020 5 Expires: January 15, 2021 7 BGP IPSEC VPNs - A Solution Analysis 8 draft-hares-idr-bgp-ipsec-analysis-00 10 Abstract 12 This draft describes problems with IGP convergence time in some IPRAN 13 networks that use a physical topology of grid backbones that connect 14 rings of routers. Part of these IPRAN network topologies exist in 15 data centers with sufficient power and interconnections, but some 16 network equipment sits in remote sites impacted by power loss. In 17 some geographic areas these remote sites may be subject to rolling 18 blackouts. These rolling power blackouts could cause multiple 19 simultaneous node and link failures. In these remote networks with 20 blackouts, it is often critical that the IPRAN phone network re- 21 converge quickly. 23 The IGP running in these networks may run in a single level of the 24 IGP. This document seeks to briefly describe these problems to 25 determine if the emerging IGP technologies (flexible algorithms, 26 dynamic flooding, layers of hierarchy in IGPs) can be applied to help 27 reduce convergence times. It also seeks to determine if the 28 improvements of these algorithms or the IP-Fast re-route algorithms 29 are thwarted by the failure of multiple link and nodes. 31 Status of This Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF). Note that other groups may also distribute 38 working documents as Internet-Drafts. The list of current Internet- 39 Drafts is at https://datatracker.ietf.org/drafts/current/. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 This Internet-Draft will expire on January 15, 2021. 48 Copyright Notice 50 Copyright (c) 2020 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents 55 (https://trustee.ietf.org/license-info) in effect on the date of 56 publication of this document. Please review these documents 57 carefully, as they describe your rights and restrictions with respect 58 to this document. Code Components extracted from this document must 59 include Simplified BSD License text as described in Section 4.e of 60 the Trust Legal Provisions and are provided without warranty as 61 described in the Simplified BSD License. 63 Table of Contents 65 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 66 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 67 1.2. IPSEC deployments . . . . . . . . . . . . . . . . . . . . 3 68 1.3. History of BGP passing Tunnel Endpoints . . . . . . . . . 4 69 1.4. Overview of proposals . . . . . . . . . . . . . . . . . . 5 70 1.5. Method of analysis . . . . . . . . . . . . . . . . . . . 7 71 2. Security and Privacy . . . . . . . . . . . . . . . . . . . . 8 72 2.1. Option 1 - Configuration Plus BGP Routes with Tunnel SA 73 IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 74 2.2. Option 2- BGP passes client routes with SA-ID plus NLRI 75 passes underlay SAs . . . . . . . . . . . . . . . . . . . 13 76 2.3. Option 3: Secure EVPN (client routes + SA information) . 14 77 2.4. comparison of security issues . . . . . . . . . . . . . . 16 78 3. Manageability and Scaling . . . . . . . . . . . . . . . . . . 16 79 3.1. Configuration sizes - used for theoretical comparison . . 17 80 3.2. BGP Route sizes for theoretical comparison . . . . . . . 18 81 3.2.1. Size of Tunnel encapsulation attribute with 1 SA per 82 tunnel endpoint . . . . . . . . . . . . . . . . . . . 18 83 3.2.2. Size of Tunnel encapsulation attribute with 10 SAs 84 per tunnel . . . . . . . . . . . . . . . . . . . . . 19 85 3.3. Network Scenario 1 . . . . . . . . . . . . . . . . . . . 19 86 3.4. Network scenario 2 . . . . . . . . . . . . . . . . . . . 19 87 3.5. Scaling Memory sizes . . . . . . . . . . . . . . . . . . 19 88 4. Key differences between the options . . . . . . . . . . . . . 19 89 5. Processing of BGP routes . . . . . . . . . . . . . . . . . . 19 90 6. Future Issues - SBGP and Secure IPSEC VPNs . . . . . . . . . 19 91 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 92 8. IANA considerations . . . . . . . . . . . . . . . . . . . . . 20 93 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 94 9.1. Normative References . . . . . . . . . . . . . . . . . . 20 95 9.2. Informative References . . . . . . . . . . . . . . . . . 20 97 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 21 99 1. Introduction 101 This document analyzes three solutions for using a BGP based approach 102 on SDWAN edge nodes to establish secure IPSec tunnels for the overlay 103 routes distribution. The solutions are: 105 o [I-D.hujun-idr-bgp-ipsec] 107 o [I-D.dunbar-idr-sdwan-edge-discovery] 109 o [I-D.sajassi-bess-secure-evpn] 111 These three drafts propose an IPsec related tunnel type for an 112 augmentation of [I-D.ietf-idr-tunnel-encaps] to support IPsec 113 tunnels. At IETF 105, IDR and BESS WG held a session to discuss the 114 security issues in these emerging drafts with security directorate 115 people. The security people provided excellent feedback on on how to 116 approach security, privacy, and scaling. The IDR/BESS working 117 members provided provided feedback on the scaling and concepts. As a 118 result of this session, it became evident that each proposal has 119 started with a unique user scenario. 121 Therefore, this draft simply reviews the technical qualities in terms 122 of: 1) the security and privacy of each technology, and 2) how the 123 technology is managed (manageability) and how the technology scales. 125 The purpose of this draft to grow our joint understanding of each 126 proposed IPSec tunnel type so that IDR and BESS can make informed 127 decisions. It is non-goal to determine which pf these 3 solutions 128 fits a particular use case using VPN using BGP to pass IPsec tunnel 129 end points. 131 1.1. Requirements Language 133 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 134 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 135 document are to be interpreted as described in RFC 2119 [RFC2119]. 137 1.2. IPSEC deployments 139 Secure VPNs based on IPsec tunnels began appearing around 2000. 140 IPsec tunnels were used for secure link transport. The IPSEC VPNs 141 utilized IPsec tunnels over physical links or underlay networks to 142 virtual links for these VPNs. These IPsec tunnels can be created by 143 configuring routers with tunnel endpoints and setting up security 144 associations for these tunnels. Automated processes can use IETF 145 network management protocols (NETCONF and RESTCONF) to configure Yang 146 modules in the routers to set up these tunnels. 148 Enterprise VPNs were created from these secure tunnels. EVPNs and 149 SDWANs have also deployed VPNs using IPSEC. 151 The BGP client routes which use the tunnel as a pathway also 152 distribute pathway information (endpoint, inner encapsulation, outer 153 encapsulation) via BGP with tunnel attribute 154 [I-D.ietf-idr-tunnel-encaps] For IPsec tunnels, there are three 155 approaches to what security association information is included with 156 the tunnel attribute. (See [I-D.hujun-idr-bgp-ipsec], 157 [I-D.dunbar-idr-sdwan-edge-discovery] and 158 [I-D.sajassi-bess-secure-evpn]. 160 One of the newly defined user scenarios for the secure VPN is the 161 SDWAN. [ [I-D.ietf-rtgwg-net2cloud-problem-statement] describes the 162 problems faced by SDWAN. [I-D.ietf-rtgwg-net2cloud-gap-analysis] 163 describes the gaps in IETF technology for this use case. 164 [I-D.dunbar-bess-bgp-sdwan-usage] describes how BGP is used as 165 control plane to setup the SDWAN networks for various SDWAN use 166 cases. SDWAN overlay networks can run over physical forwarding by a 167 wide variety of underlay networks. SDWAN is one of the more recent 168 developments in IPsec based VPNS created by an SDN controller. 170 The author welcomes additional information on other use cases. 172 1.3. History of BGP passing Tunnel Endpoints 174 [RFC5512] defined SAFI to pass tunnel endpoint encapsulation 175 information. However, many operators and vendors preferred to pass 176 this information in a BGP attribute. [I-D.ietf-idr-tunnel-encaps] 177 defines a BGP attribute for tunnels to replace [RFC5512] 178 functionality, but does not address how to use RFC5566 without the 179 encapsulation SAFI. EVPN [RFC8365] also defined tunnel types for 180 encapsulation. The tunnel types registered with IANA (www.IANA.org) 181 list the following tunnel types from [RFC5512], [RFC5566], and 182 [RFC8365]: 184 o L2TPv3 over IP [RFC5512] [value 1], 186 o GRE [RFC5512] [value 2] 188 o Transmit tunnel endpoint [RFC5566][value 3] 190 o IPsec in tunnel mode [RFC5566] [value 4] 192 o IP in IP tunnel with IPsec Transport mode [RFC5566][value 5] 193 o MPLS-in-IP tunnel with IPsec Transport mode [RFC5566][value 6] 195 o IP in IP [RFC5566] [value 7] 197 o VXLAN encapsulation [RFC8365][value 8] 199 o NVGRE encapsulation [RFC8365][value 9] 201 o MPLS Encapsulation [RFC8365][value 10] 203 o MPLS in GPE encapsulation [RFC8365] [value 11] 205 o VXLAN GPE encapsulation [RFC8365] [value 12] 207 [I-D.ietf-idr-tunnel-encaps] has been created to address deficiencies 208 in RFC5512 [RFC5512]. These deficiencies include: operational costs 209 of using SAFI for tunnel identifiers, inability to specify egress 210 endpoint of tunnel, unclear prefix-tunnel association, and inability 211 to specify inner/outer encapsulation. [I-D.ietf-idr-tunnel-encaps] 212 defines new Sub-TLVs to support inner and outer encapsulation for 213 these encapsulation types, and will become the main reference for 214 these tunnel types. 216 RFC5566 [RFC5566] defined the IP Tunnel Authenticator Sub-TLV for use 217 in the SAFI, but these recent proposals have suggested different 218 alternatives for replacing the Tunnel Authenticator function. 220 1.4. Overview of proposals 222 This section provides a technical overview of the 3 proposals 223 [I-D.hujun-idr-bgp-ipsec], [I-D.dunbar-idr-sdwan-edge-discovery], and 224 [I-D.sajassi-bess-secure-evpn]. 226 [I-D.hujun-idr-bgp-ipsec] proposes 3 new Sub-TLVs: local/remote 227 Prefix Sub-TLV, Public Routing Instance Sub-TLV, and IPSec 228 Configuration Sub-TLV (IPsec-Config). The local/remote prefix Sub- 229 TLV will not be discussed here as it does not clearly align to 230 [I-D.ietf-idr-tunnel-encaps]. The optional Public Routing Instance 231 (PRI) is used instead of a route target community so that local 232 policy can filter routes for a specific community. This feature 233 provides the same feature as a Route target for a pre-configured set 234 of PRIs. 236 The IPsec Configuration Sub-TLV contains 4 octet opaque value to link 237 the tunnel to the Tunnel Authentication entry found in a security 238 association table on the local node. This table will need to include 239 which tunnel endpoints this security association is valid for. This 240 analysis assumes the IETF protocols NETCONF RESTCONF configure a YANG 241 module that has these security associations. 243 [I-D.dunbar-idr-sdwan-edge-discovery] proposes UPDATEs from client 244 routers to include the IPsec SA identifiers (ID) to reference the 245 IPsec SA attributes being advertised by separate Underlay Property 246 BGP UPDATE messages. The security association table is built 247 dynamically from the information passed in these Underlay Property 248 BGP Updates plus some local configuration. If a client route can be 249 encrypted by multiple IPsec SAs, then multiple IPsec SA IDs are 250 included in the Tunnel-Encaps Path attribute for the client route. 251 This draft proposes two new Sub-TLVs: IPsec-SA-ID and IPsec-SA-Group. 252 The IPsec-SA-ID is similar to [I-D.hujun-idr-bgp-ipsec] IPSec Config 253 Sub-TLV passing a 2 octet pointer to into a security association 254 table. IPsec-SA-Group Sub-TLV optimizes passing the same information 255 when multiple IPsec SAs with the same inner encapsulation header. 257 [I-D.dunbar-idr-sdwan-edge-discovery] proposes underlay tunnel 258 topology information for SDWAN in BGP UPDATEs. The information is 259 passed in a combination of NLRI with an SAFI=74 (SDWAN SAFI) and a 260 Tunnel Encapsulation attribute with tunnel type being SDWAN-Underlay. 262 Security association information for the tunnels in this underlay 263 will be passed in the Tunnel Attribute using in the SDWAN Underlay 264 tunnel type. This new tunnel type which will support the current 265 tunnel Sub-TLV plus the newly proposed IPSec SA Sub-TLV(s). There 266 are two types of IPsec SA Sub-TLVs proposed by 267 [I-D.dunbar-idr-sdwan-edge-discovery], one is for general purpose 268 deployment which requires a full-set of Security Association, 269 including Nonce, Public Key, Proposal and Transform Sub-TLVs in the 270 SDWAN SAFI NLRI (SA-TYPE =2). Another type is for simple deployment 271 which only needs one simple IPsec SA Sub-TLV included (SA-TYPE=1). 272 In addition, it can also include other optional Sub-TLVs like NAT, 273 WAN Port, Geo-location with the SDWAN SAFI route. 275 [I-D.sajassi-bess-secure-evpn] proposes defines 2 new tunnel types 276 (ESP-Transport and ESP-in-UDP-Transport) and 3 new Sub-TLVS (DIM Sub- 277 TLV, Key-Exchange Sub-TLV, and proposal Sub-TLV) for these new tunnel 278 types. The new Sub-TLVs pass information regarding security 279 associations. The DIM Sub-TLV is required to be supported for the 280 two new tunnel types. As noted above, the SDWAN-WAN-Underlay tunnel 281 type from [I-D.dunbar-idr-sdwan-edge-discovery] supports equivalent 282 features to IPsec-SA, Public-key, and SA-Transforms. 284 [I-D.dunbar-idr-sdwan-edge-discovery] and 285 [I-D.sajassi-bess-secure-evpn] differ in the information included in 286 the client routes. [I-D.sajassi-bess-secure-evpn] attaches all the 287 IPsec SA information to the actual client routes, whereas the 289 [I-D.dunbar-idr-sdwan-edge-discovery] only includes the IPsec SA IDs 290 for the client routes. The IPsec SA IDs used by 291 [I-D.dunbar-idr-sdwan-edge-discovery] reference (point) to the SA- 292 Information which is advertised separately. All the SA-Information 293 are attached to routes which describe the SDWAN underlay, such as WAN 294 Ports or Node address. 296 [I-D.sajassi-bess-secure-evpn] supports tunnel types of ESP-Transport 297 and ESP-in-UDP transport, but not traditional IPsec tunnel types 298 (IPsec in tunnel mode, IP in IP tunnel with IPsec transport, MPLS-in- 299 IP tunnel with transport mode). The use of the new tunnel type could 300 be used in a similar fashion to [I-D.dunbar-idr-sdwan-edge-discovery] 301 to pass SA-information regarding the underlay. 302 [I-D.sajassi-bess-secure-evpn] seems to point to passing client 303 routes upon a rekeying request. This method will increase the amount 304 of BGP traffic passed in the crash or initial start-up in the tunnel 305 encapsulation attribute. 307 Since [I-D.sajassi-bess-secure-evpn] draft has not recently been 308 updated, it is not clear if the recent changes to 309 [I-D.ietf-idr-tunnel-encaps] are reflected in this draft. 310 [I-D.sajassi-bess-secure-evpn] depends on 311 [I-D.carrel-ipsecme-controller-ike] which received many security 312 comments at IETF 105. Therefore, the author has analyzed 313 [I-D.sajassi-bess-secure-evpn] solutions based on the following 314 assumptions: 316 o ESP-Transport and ESP-in-UDP would have been aligned with the 317 latest version of the [I-D.ietf-idr-tunnel-encaps], 319 o Only the DIM Sub-TLV is required to be sent during initialization, 320 PE rekey requests, routing periodic updates, and node restarts 321 (crash/load) for shared security controller policies. 323 o The multiple policy environments may increase the size of Tunnel 324 Encapsulation attribute as transforms and transform attributes are 325 sent. 327 1.5. Method of analysis 329 The things matter to the network operator of IP-SEC VPN in SDWAN: 330 security, manageability, scaling, and privacy. Each deployment of an 331 IPSec VPN may combine different underlay networks with different 332 challenges to security, manageability, scaling and privacy. This 333 analysis compares the basic technologies of these proposals in terms 334 of two groups of features: 1) security and privacy, and 2) 335 manageability and scaling. This analysis drafts looks at each 336 solution based on the strengths are weaknesses of each type. 338 Analyzing scaling can either be done at the 50,000 foot level or in 339 excruciating detail. This analysis will be at the 50,000 foot level 340 using two example scenarios (small and very large) 342 Scenario 1: The 3 Route Reflectors (RR) each have 5 client routers 343 per router reflector. The client routers have a potential of 5 344 tunnels with 1 security association per tunnel. Each client router 345 has 200 routes. The total number of configured tunnels is 20 tunnels 346 per RR cluster and the total number of client routes is 3000. 347 Diagram 1 in section 1.3 showed this simple topology for these route 348 reflectors. 350 Scenario 2: The 3 Route Reflectors (RR) each have 10,000 client 351 routers. Each client router supports 100 tunnels, 10K routes, and 10 352 security associations per tunnel. Each Route Reflectors will receive 353 from its client routers a total of 100 million client routes with 1 354 million tunnels client tunnels (100*10K client routers), and 10 355 million security associations. The totals for all 3 RR may be up to 356 3 times this level (300 Million client routes, 3 million tunnels, and 357 10 million security associations), but it is likely the RR will 358 contain some redundancy. Our scenario focuses on the challenges 359 within a single RR clustr. 361 The BGP scaling in these two scenarios contrast small IPsec VPNs and 362 very large IPsec VPNs. BGP routing products handle route 363 distribution of over 100 Million routes so this scaling is well 364 within the range of the BGP products. 366 2. Security and Privacy 368 During an initial security review of this information, Ben Kaduk made 369 the following comments: 371 "First off, when we start to get IPsec configuration via BGP, it's 372 helpful to think of what other information we get in the same way, 373 and to analyze the effects of misconfiguration or malicious 374 configuration both on IPsec and the broader system. For example, 375 if we are getting NLRI from BGP, then a misconfiguration that 376 gives us parameters that are incompatible with a peer's is not 377 causing particularly new harm, since we could just as easily be 378 told that peer is unreachable and we wouldn't try to talk to them 379 anyway. On the other hand, we could be given configuration to use 380 computationally expensive algorithms which would increase the DOS 381 risk in a way that may not (or may!) be already possible. " (email 382 to IDR and BESS WGs after IETF 105)." 384 The security analysis in this draft assumes that the route 385 distribution for BGP routes is done via a mesh of Route Reflectors 386 which have route reflector clients associated. The IBGP mesh of 387 route reflectors within a domain is assumed to run over secure 388 transport links (such as TLS). If privacy is an issue for BGP route 389 distribution, the TLS encrypts/decrypts the data in the IBGP mesh. 390 If a single AS IBGP mesh of Route Reflectors connects to another AS, 391 the EBGP connection is also over a secure transport (such as TLS). 393 [Full mesh topology within a IBGP cloud 394 LAN used to simply drawing) 395 TLS 396 =============================== 397 | | | 398 RR1---------- RR2 ------------ RR3 399 TLS / | \ / | \ / | \ 400 C1 C2 Cn C1 C2 Cn C1 C2 Cn 402 Route Reflector Topology 404 This security topology has the same transport link secure topology as 405 the NETCONF/RESTCONF security of set of NETCONF/RESTCONF servers. 406 The example NETCONF topology is below. 408 Back-end configuration database 409 TLS 410 =============================== 411 | | | 412 Netconf Netconf Netconf 413 Client1 -------Client2 ------- Client 3 414 TLS / | \ / | \ / | \ 415 NS1 NS2 NSn NS1 NS2 NSn NS1 NS2 NSn 417 NETCONF Topology 419 The security aspects of using network management protocols NETCONF or 420 RESTCONF servers to control IPsec SA distribution has been considered 421 as part of SDN-based IPSEC flow consideration (see [RFC8192]). The 422 user data traffic runs over secure IP tunnels whether the 423 configuration is via NETCONF or BGP RR mesh. Figure 3 below shows a 424 Route Reflector topology with BGP sessions in a RR mesh and client 425 traffic over IPSEC tunnels. 427 Figure 3 - pending [Editor's note: Large scale topology needs svg 428 drawings] 430 Figure 4 shows an equivalent topology can be used with NETCONF 431 client-servers. Notice that the NETCONF topology requires a common 432 database behind the network clients to provide the correct 433 configurations. If the NETCONF servers work across administrative 434 domains, a shared database must be developed to provide the 435 appropriate information given the correct policy filters and access 436 (NETCONF NACM). 438 Diagram 4 - pending [Editor's note: Large scale BGP topologies need 439 .svg ] 441 There are two parts of the security for control plane traffic: link 442 security and data security. Link security entails making sure the 443 data is secure as the data is transmitted across the link. 445 Link security in the NETCONF configuration cloud shown in diagram 2 446 entails making sure the configuration data passes across each of the 447 links. The links from the configuration database (DB in diagram) to 448 each client server must be secured via TLS. The links from the 449 netconf client to the netconf server on the node must be secured via 450 TLS. Data security in the NETCONF configuration cloud entails making 451 sure the data from the configuration data base (DB) travels through 452 the netconf clients (e.g. netconf client1) to the node's netconf 453 server (e.g. NS1) without change. Data privacy for configuration 454 pathways traffic entails making sure no other party snoops on the 455 data while it travels from configuration database (DB) to the netconf 456 client to the server. 458 NETCONF client/servers are designed to operate in a single 459 administrative domain. NETCONF client/servers require additional 460 policy filters and checks to run between multiple administrative 461 domains. The Database to NETCONF client link is not standardized by 462 IETF. 464 Correspondingly, the link security in a BGP RR mesh requires that the 465 data is secure across any link in the BGP RR mesh (RR to/from any RR 466 client or within the RR mesh). Data security for control plane 467 traffic entails ensuring that the data placed into the BGP mesh (from 468 RR clients or RR) arrives at the appropriate destination without 469 change. BGP does not provide data security on control plane traffic 470 as the data may be modified via policy at each node. SBGP does 471 provide data security. 473 For most networks, physical security of each node and link security 474 is considered sufficient. 476 The data written into a node using configuration data writes (NETCONF 477 edit-config or RESTCONF PUT/POST) uses the NETCONF client to write to 478 the network server on the client router. The data which is sent from 479 the route-reflector to the RR client routers is sent via BGP, saved 480 in the BGP RIBs, and installed in the router. 482 The network management protocols (NETCONF/RESTCONF) and BGP both have 483 access policy that controls the data is written into the router. The 484 error handling for incorrect data is different between these two 485 network management protocols (NETCONF/RESTCONF) and BGP. If netconf 486 tries save data with the wrong format, it will provide an error 487 information in the response (rpc-error). The BGP error handling of a 488 malformed Tunnel Attribute in the TLV simply ignores the tunnel 489 attribute while accepting the route. 491 The common resolution is that NETCONF, RESTCONF, and BGP write error 492 information to a local log. Error reports can be tracked in a Yang 493 module which can be automatically streamed to central controller via 494 an alternate channel NETCONF/RESTCONF logging channel. 496 [Editorial note: Should I give the details of the NETCONF/RESTCONF 497 logging channel?] 499 The SDWAN environment or any VPN that uses BGP to transfer tunnel 500 configuration and security association information SHOULD consider 501 augmenting the base BGP Yang model with BGP tunnel encapsulation Yang 502 model for all tunnel types including IPSEC. The logging features or 503 the reporting of the BGP errors can be combined with any error 504 reporting on NETCONF/RESTCONF configuration or any operational state 505 from the tunnel interface. The NETCONF/RESTCONF logging feature 506 providing throttling so any type of error reporting can be configured 507 to be manageable within a large network. 509 This implies the SDWAN environment should design a BGP Yang model 510 augmenting the BGP base model for the BGP tunnel encapsulation 511 functionality for all tunnel types including IPSEC AND provides 512 logging features the reporting can be the same as NETCONF/RESTCONF. 514 Given Ben's rule of thumb, the transmission of the routes, the tunnel 515 end points, and the link to the security association information via 516 the BGP protocol does not cause extra security risks. 518 The next 3 sections summarize the security and privacy of each 519 technology in terms of: 521 o what is distributed via netconf, 523 o what is distributed via BGP, 525 o link security provided, 527 o data security provided, 529 o suggested Yang models that will augment error handling, 530 o privacy issues. 532 2.1. Option 1 - Configuration Plus BGP Routes with Tunnel SA IDs 534 Document: draft-hujun-idr-bgp-ipsec-02.txt [I-D.hujun-idr-bgp-ipsec] 536 What is distributed via NETCONF: Tunnel Configuration, Security 537 associations, and the mapping of the security association to a tunnel 538 end point (identified y IPsec tunnel identifier), and SA (security 539 association) for the each IPSec tunnel. 541 What is distributed via BGP: Client Routes with IPsec Sub-TLV per 542 tunnel attribute with IP-SEC tunnel. Optionally, the Public Instance 543 Sub-TLV may augment the BGP tunnel attributes Sub-TLV for tunnel 544 endpoint. 546 Sub-TLVs added: 548 IPsec SUB-TLV in IP-Sec Tunnel Attribute (proposed): 4 octet 549 opaque tag. 551 Public Instance Sub-TLV: identifies the remote instance the Sub- 552 TLV for Tunnel End-Point Identifier takes its address from. 554 NETCONF Link Security: Distribution is secured by Client-server TLS 556 Configuration Data security: Configuration clients SHOULD have host 557 and data security. This is beyond NETCONF/RESTCONF security. Client 558 synchronization of data with other clients must have security links 559 and security mechanisms. 561 Suggested Yang Models for Configuration and Reporting 563 o Tunnels configuration and operational state 565 o SA configuration and operational state, 567 o BGP Tunnel Attribute Yang model augmenting base BGP model with 568 tunnel attributes data and error log. (Tunnel attribute 569 information includes the tunnel attributes plus the mapping of 570 routes to tuples of [tunnel endpoint, security association, and 571 encryption].) 573 Privacy: Link privacy assumes the ability to encrypt the link data to 574 provide privacy. Node Privacy requires software secure containers 575 within the NETCONF/RESTCONF clients/servers and BGP modules for each 576 of these models. 578 2.2. Option 2- BGP passes client routes with SA-ID plus NLRI passes 579 underlay SAs 581 Document: [I-D.dunbar-idr-sdwan-edge-discovery] 583 What is distributed via NETCONF/RESTCONF or locally configured: 584 Policy and/or templates so that automation may use NLRI with SDWAN 585 SAFI to configure tunnels. This policy may be expressed in as little 586 as 1 line of local configuration. 588 What is distributed via BGP: Client Routes with tunnel attribute with 589 IPsec Tunnel type, IPSec SA ID(s) which reference Security 590 Association attributes being advertised by SDWAN-Underlay UPDATE. 591 The Sub-TLVs added include: 593 IPSec-SA-ID SUB-TLV (proposed): 2 octet pointer into SA table 595 IPsec-SA-Group SUB-TLV: list of pointers into SA table grouped by 596 inner encapsulation type 598 The Underlay property is NLRI attached to port addresses or node 599 address with SDWAN SAFI: Includes Site Type, IPSEC-SA-Type, Port- 600 Local-ID, SDWAN-Site-ID, SDWAN Node ID. Depending on the SITE-Type 601 and IPSec-SA-type, this SAFI carries either template references for 602 pre-configured security association (SAs) or full SA information. 604 Note: Since the Security association information is carried in a 605 different AFI/SAFI pair, this information may be transmitted in a 606 different BGP update than the client routes with the Tunnel 607 attribute. 609 Link Security: Distribution is secured between RR to RR clients and 610 between RR in the RR mesh is secured with Transport layer security. 611 If the RR mesh with underlay information is compromised, it does not 612 mean the route with tunnel attributes will be compromised. 614 Data Security: Data distribution security of tunnel endpoints, SA 615 (security association), routes and mapping (tunnel endpoint, SA, 616 routes) SHOULD have RR and RR Client security on modules processing 617 the data. Full data security (with certificates that the data 618 originated is what arrives at the final destination) for the BGP 619 routes and attributes is beyond the normal mechanism of BGP. These 620 features may be available in SBGP. SBGP signature processing is 621 computationally expensive and requires additional memory space. 622 Synchronization of the routing information on RR (routes, tunnel 623 endpoints, SA-links) and underlay security association information 624 (from AFI/SAFI SDWAN) may be impacted policy that distributes the 625 data. 627 Technical Note: Many ISPs have chosen to only validate the route 628 origin attribute of the BGP route to insure reduction of "human 629 errors" and some classes of attacks. 631 Suggested Yang Models for Reporting Errors 633 o Tunnels configuration and operational state of tunnels, 635 o SA configuration and operational state of SA information, 637 o BGP Tunnel Attribute Yang model augmenting base BGP model with 638 tunnel attributes data and error log. (Tunnel attribute 639 information includes the tunnel attributes plus the mapping of 640 routes to tuples of [tunnel endpoint, security association, and 641 encryption]. 643 o Augmentation to base BGP Model to display information passed in 644 NLRI with SDWAN SAFI 646 Privacy: (Same as option 1) 648 o Link privacy assumes the ability to encrypt the link data to 649 provide privacy. 651 o Node Privacy requires secure containers within the netconf/ 652 restconf clients/servers and BGP modules for the BGP control plane 653 data. 655 2.3. Option 3: Secure EVPN (client routes + SA information) 657 Document: draft-ietf-sajassi-bess-secure-evpn 658 [I-D.sajassi-bess-secure-evpn] 660 What is distributed: Client routes with tunnel attribute with ESP- 661 Transport and ESP-in-UDP-Transport tunnel types. 663 SUB-TLVs in ESP-Transport and ESP-in-UDP-Transport Attribute 664 (proposed): BASE DIM, Key-Exchange, ESP SA, and Transform Sub- 665 Structure. 667 This solution would require the Tunnel TLV for the IPsec to 668 contain: Tunnel Endpoint TLV and the DIM TLV. 670 The DIM SUB-TLV has the following fields: 672 * ID-length 674 * Nonce length, 675 * I-flag 677 * Flags 679 * Re-key counter 681 * Originator ID + (Tenant ID) + (Subnet ID) + (Tenant Address 683 * Nonce data 685 Technical Note: The data rate for retransmitting the client routes 686 with the DIM Sub-TLV must be done at the rekeying rate. This 687 automatic re-key counter is distributed with the data. 689 Link Security: Distribution is secured between RR to RR clients and 690 the RR mesh is secured with transport link security. The regular 691 data distribution of the SA nonce and the rekeying counter provides a 692 potential attack vector for man-in-the middle attacks if the link 693 security is compromised. 695 Data Security: Data distribution security of tunnel endpoints, SA 696 (security association), routes and mapping (tunnel endpoint, SA, 697 routes) SHOULD have RR and RR Client security on modules processing 698 the data. In addition the processes handling SA information 699 [I-D.carrel-ipsecme-controller-ike] should exist in a protected 700 process. 702 Full data security for the BGP routes and attributes is beyond the 703 normal mechanism of BGP, but may be available in SBGP. SBGP 704 signature processing is computationally expensive and requires 705 additional memory space. Synchronization of the routing information 706 on RR (routes, tunnel endpoints, SA-links) and underlay security 707 association information (from AFI/SAFI SDWAN) may be impacted policy 708 that distributes the data. 710 Yang Models for Reporting Errors 712 o Tunnels configuration and operational state, 714 o configuration and operational state, 716 o BGP Tunnel Attribute Yang model augmenting base BGP model with 717 tunnel attributes data and error log. (Tunnel attribute 718 information includes the tunnel attributes plus the mapping of 719 routes to tuples of [tunnel endpoint, security association, and 720 encryption].) 722 o Yang models for the operational state in 723 [I-D.carrel-ipsecme-controller-ike]. 725 Privacy: Same as option 1 and 2. 727 2.4. comparison of security issues 729 The security of each of these solutions utilizes similar distribution 730 and error reporting. Man in the Middle attacks based on snooping, 731 would need to break the TLS security and encryption for privacy. The 732 [I-D.sajassi-bess-secure-evpn] provides more data directly linked to 733 the routes which could allow an attack vector. 735 [I-D.hujun-idr-bgp-ipsec] and [I-D.dunbar-idr-sdwan-edge-discovery] 736 provide the route, tunnel information, and link to the SA 737 information. This indirect access to SA information could lessen the 738 attack vector for the tunnel. 740 [I-D.dunbar-idr-sdwan-edge-discovery] and 741 [I-D.sajassi-bess-secure-evpn] have options send the SA information 742 on unique tunnel types. [I-D.dunbar-idr-sdwan-edge-discovery] 743 placement of the SA data in a NLRI can allow a separate encryption 744 between the SA data and the route/tunnel information. 746 While all 3 solutions can be used with automated tools (SDN based on 747 simply configuration based), the each solution has benefits and 748 deficits. 750 3. Manageability and Scaling 752 Manageability involves how much manual effort is involved to set up 753 IPSec tunnels using each of the three options. The manageability 754 must handle the following: initial set-up of nodes, reporting of 755 status or errors, and rekeying efforts. BGP data distribution and 756 processing of routes to set-up forwarding is stressed during: initial 757 start-up, crash of a RR, and start-up of RR. 759 The scaling of the system should handle the data distribution and the 760 manageability should handle both the network scenario 1 and network 761 scenario 2. Scenario 1 and scenario 2 both consider one security 762 association per tunnel and 10 security associations per 10. This 763 comparison is given to help understand the impact of rekeying the 764 security associations. [I-D.hujun-idr-bgp-ipsec] would need to send 765 rekeying via NETCONF/RESTCONF, but the rekeying that causes a tunnel 766 to switch security associations can be sent via BGP. 767 [I-D.hujun-idr-bgp-ipsec] use of the NETCONF/RETCONF to send a 768 configuration becomes a bottleneck if the network sizes reach 769 scenario 2. [I-D.dunbar-idr-sdwan-edge-discovery] uses two parallel 770 BGP NLRI processes where one passes routes and security association 771 identifiers, and the second process sends rekeying based on topology 772 information. Rekeying information is transmitted prior to BGP passes 773 the rekeying of the tunnel. [I-D.sajassi-bess-secure-evpn] passes 774 the rekeying information with the client routes. During initial 775 start-up or RR crash, this rekeying data substantially increases the 776 memory footprint. A continual rekeying process in 777 [I-D.sajassi-bess-secure-evpn] could also cause periodic BGP updates 778 to continue to use bandwidth in the network. 780 One alternative to the periodic rekeying is to allow the association 781 of more than 1 security association (SA) per tunnel, and allow a 782 local mechanism to switch security associations are a particular 783 time. This analysis looks at the scaling issues of 10 SAs per tunnel 784 allows this analysis to look at the scaling in terms of memory 785 required for this mechanism of rekeying. 787 The estimate of 10 security associations is admittedly imperfect, but 788 it may help to start the discussion on the memory usage during 789 rekeying. 791 NETCONF/RESTCONF data distribution scales when the client to netconf/ 792 restconf server ratio is low. 1 client per server is best, but 5 793 servers per client is a low level. 1 client configuring 10K servers 794 on network nodes is beyond most NETCONF/RESTCONF servers. Pushing 795 multiple types of data may also cause stress on the client ability to 796 pull data from the back-end configuration database. 798 The difference between NETCONF/RESTCONF and BGP mechanisms matter in 799 for SDWAN deployments. NETCONF/RESTCONF is optimized for a single 800 administrative domain and BGP is optimized for inter-domain policy. 801 In SDWAN the nodes are distributed across multiple administrative 802 domains. BGP implementations have many levels of policy. Using BGP 803 each node can be under different RR. Each node can have default SA 804 attributes such as supported encryption algorithms, the nonce, and 805 the public key. The SA ID is only locally significant to the node 806 (or to the port), which is less prone to misconfiguration. BGP also 807 has policy at the route level. Using BGP built-in RT constraint 808 distribution, BGP implementations distribute the SA information to 809 the nodes specified as authorized peers. 811 3.1. Configuration sizes - used for theoretical comparison 813 To provide a simple estimate, it is assumed that 100 items needed to 814 be configured in the Yang modules prior to starting the IPsec VPN. 816 o BGP peer items per node: 20 817 o Tunnel configuration items node: 20 819 o SA Configuration items per node: 40 821 o Monitoring configuration per node: 20 823 3.2. BGP Route sizes for theoretical comparison 825 The following estimates are for route and the tunnel Attribute are 826 used for this comparison: 828 o average of 4 bytes for IPv4 prefix 830 o average of 8 bytes for IPv6 prefix. 832 3.2.1. Size of Tunnel encapsulation attribute with 1 SA per tunnel 833 endpoint 835 The space required in the BGP packet for the tunnel attribute per 1 836 tunnel with 1 Security association (SA) for each of the options is as 837 follows: 839 o Tunnel TLV header [4 bytes] 841 o Sub-TLV for tunnel endpoint for IPv4 [12 bytes] 843 o IP-Sec Sub-TLVs (required) with 1 SA per tunnel endpoint: 845 * Option 1 [I-D.hujun-idr-bgp-ipsec]: 6 octets 847 * Option 2 [I-D.dunbar-idr-sdwan-edge-discovery] 4 octets 849 * Option 3 [I-D.sajassi-bess-secure-evpn] : 35 octets 851 + Sub-TLV header: 3 octets 853 + Dim: 32 octets (header (4), rekey (8), address (8), Nonce 854 (12)) 856 The total space in the tunnel attribute for each type per tunnel 857 endpoint with one security association is the following: 859 Option 1 [I-D.hujun-idr-bgp-ipsec]: 22 octets 861 Option 2 [I-D.dunbar-idr-sdwan-edge-discovery]: 20 octets 863 Option 3 [I-D.sajassi-bess-secure-evpn] : 52 octets 865 Encapsulation mechanisms such as GRE and VXLAN may add 6-16 octets 866 per tunnel to the Tunnel attribute per tunnel. This addition is due 867 to adding encodings for inner mechanisms (4-12), and outer encodings 868 (2-4) 870 The total space with encapsulations would then be: 872 Option 1 [I-D.hujun-idr-bgp-ipsec]: 28-38 octets 874 Option 2: [I-D.dunbar-idr-sdwan-edge-discovery]:: 26-36 octets 876 Option 3 [I-D.sajassi-bess-secure-evpn] : 56-68 octets 878 3.2.2. Size of Tunnel encapsulation attribute with 10 SAs per tunnel 880 This will be completed in version -01 882 3.3. Network Scenario 1 884 This will be completed in version -01 886 3.4. Network scenario 2 888 This will be completed in version -01.txt 890 3.5. Scaling Memory sizes 892 This section includes scaling for network scenario 1 and 2. 894 4. Key differences between the options 896 (to be completed in version -01) 898 5. Processing of BGP routes 900 (to be completed in version -01) 902 6. Future Issues - SBGP and Secure IPSEC VPNs 904 (to be completed in version -01) 906 7. Security Considerations 908 This draft is analysis that includes security and privacy. The draft 909 does not cause any further security issues, but hopes to enhance the 910 security considerations in other drafts. 912 8. IANA considerations 914 This draft does not make any requests to IANA for allocations. It is 915 an analysis for review of future allocations in the BGP registry. 917 9. References 919 9.1. Normative References 921 [I-D.carrel-ipsecme-controller-ike] 922 Carrel, D. and B. Weis, "IPsec Key Exchange using a 923 Controller", draft-carrel-ipsecme-controller-ike-01 (work 924 in progress), March 2019. 926 [I-D.dunbar-idr-sdwan-edge-discovery] 927 Dunbar, L., Hares, S., Raszuk, R., and K. Majumdar, "BGP 928 UPDATE for SDWAN Edge Discovery", draft-dunbar-idr-sdwan- 929 edge-discovery-00 (work in progress), July 2020. 931 [I-D.hujun-idr-bgp-ipsec] 932 Hu, J., "BGP Provisioned IPsec Tunnel Configuration", 933 draft-hujun-idr-bgp-ipsec-02 (work in progress), March 934 2020. 936 [I-D.ietf-idr-tunnel-encaps] 937 Patel, K., Velde, G., Ramachandra, S., and J. Scudder, 938 "The BGP Tunnel Encapsulation Attribute", draft-ietf-idr- 939 tunnel-encaps-16 (work in progress), July 2020. 941 [I-D.sajassi-bess-secure-evpn] 942 Sajassi, A., Banerjee, A., Thoria, S., Carrel, D., Weis, 943 B., and J. Drake, "Secure EVPN", draft-sajassi-bess- 944 secure-evpn-03 (work in progress), July 2020. 946 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 947 Requirement Levels", BCP 14, RFC 2119, 948 DOI 10.17487/RFC2119, March 1997, 949 . 951 9.2. Informative References 953 [I-D.dunbar-bess-bgp-sdwan-usage] 954 Dunbar, L., Guichard, J., Sajassi, A., Drake, J., Najem, 955 B., and D. Carrel, "BGP Usage for SDWAN Overlay Networks", 956 draft-dunbar-bess-bgp-sdwan-usage-08 (work in progress), 957 July 2020. 959 [I-D.ietf-rtgwg-net2cloud-gap-analysis] 960 Dunbar, L., Malis, A., and C. Jacquenet, "Networks 961 Connecting to Hybrid Cloud DCs: Gap Analysis", draft-ietf- 962 rtgwg-net2cloud-gap-analysis-06 (work in progress), May 963 2020. 965 [I-D.ietf-rtgwg-net2cloud-problem-statement] 966 Dunbar, L., Jacquenet, C., and M. Toy, "Dynamic Networks 967 to Hybrid Cloud DCs Problem Statement", draft-ietf-rtgwg- 968 net2cloud-problem-statement-10 (work in progress), May 969 2020. 971 [RFC5512] Mohapatra, P. and E. Rosen, "The BGP Encapsulation 972 Subsequent Address Family Identifier (SAFI) and the BGP 973 Tunnel Encapsulation Attribute", RFC 5512, 974 DOI 10.17487/RFC5512, April 2009, 975 . 977 [RFC5566] Berger, L., White, R., and E. Rosen, "BGP IPsec Tunnel 978 Encapsulation Attribute", RFC 5566, DOI 10.17487/RFC5566, 979 June 2009, . 981 [RFC8192] Hares, S., Lopez, D., Zarny, M., Jacquenet, C., Kumar, R., 982 and J. Jeong, "Interface to Network Security Functions 983 (I2NSF): Problem Statement and Use Cases", RFC 8192, 984 DOI 10.17487/RFC8192, July 2017, 985 . 987 [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., 988 Uttaro, J., and W. Henderickx, "A Network Virtualization 989 Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, 990 DOI 10.17487/RFC8365, March 2018, 991 . 993 Author's Address 995 Susan Hares 996 Hickory Hill Consulting 997 US 999 Email: shares@ndzh.com