BESS Workgroup A. Sajassi, Ed. INTERNET-DRAFT A. Banerjee Intended Status: Standards Track S. Thoria D. Carrel Cisco B. Weis Individual J. Drake Juniper Expires: January 13, 2021 July 13, 2020 Secure EVPN draft-sajassi-bess-secure-evpn-03 Abstract The applications of EVPN-based solutions ([RFC7432] and [RFC8365]) have become pervasive in Data Center, Service Provider, and Enterprise segments. It is being used for fabric overlays and inter- site connectivity in the Data Center market segment, for Layer-2, Layer-3, and IRB VPN services in the Service Provider market segment, and for fabric overlay and WAN connectivity in Enterprise networks. For Data Center and Enterprise applications, there is a need to provide inter-site and WAN connectivity over public Internet in a secured manner with same level of privacy, integrity, and authentication for tenant's traffic as IPsec tunneling using IKEv2. This document presents a solution where BGP point-to-multipoint signaling is leveraged for key and policy exchange among PE devices to create private pair-wise IPsec Security Associations without IKEv2 point-to-point signaling or any other direct peer-to-peer session establishment messages. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months Sajassi et al. Expires January 13, 2021 [Page 1] INTERNET DRAFT Secure EVPN July 13, 2020 and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Copyright and License Notice Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1 Tenant's Layer-2 and Layer-3 data & control traffic . . . . 7 2.2 Tenant's Unicast & Multicast Data Protection . . . . . . . . 7 2.3 P2MP Signaling for SA setup and Maintenance . . . . . . . . 7 2.4 Granularity of Security Association Tunnels . . . . . . . . 7 2.5 Support for Policy and DH-Group List . . . . . . . . . . . . 8 3 BGP Component . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.1 Zero Touch Bring-up (ZTB) . . . . . . . . . . . . . . . . . 8 3.2 Configuration Management . . . . . . . . . . . . . . . . . . 8 3.3 Orchestration . . . . . . . . . . . . . . . . . . . . . . . 9 3.4 Signaling . . . . . . . . . . . . . . . . . . . . . . . . . 9 4 Solution Description . . . . . . . . . . . . . . . . . . . . . 9 4.1 Inheritance of Security Policies . . . . . . . . . . . . . . 10 4.2 Distribution of Public Keys and Policies . . . . . . . . . 11 4.2.1 Minimal DIM . . . . . . . . . . . . . . . . . . . . . . 11 4.2.2 Multiple Policies . . . . . . . . . . . . . . . . . . . 12 4.2.2.1 Multiple DH-groups . . . . . . . . . . . . . . . . 12 Sajassi et al. Expires January 13, 2021 [Page 2] INTERNET DRAFT Secure EVPN July 13, 2020 4.2.2.2 Multiple or Single ESP SA policies . . . . . . . . 12 4.3 Initial IPsec SAs Generation . . . . . . . . . . . . . . . 13 4.4 Re-Keying . . . . . . . . . . . . . . . . . . . . . . . . . 13 4.5 IPsec Databases . . . . . . . . . . . . . . . . . . . . . . 13 5 Encapsulation . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.1 Standard ESP Encapsulation . . . . . . . . . . . . . . . . . 14 5.2 ESP Encapsulation within UDP packet . . . . . . . . . . . . 15 6 BGP Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . 16 6.1 The Base (Minimal Set) DIM Sub-TLV . . . . . . . . . . . . . 16 6.2 Key Exchange Sub-TLV . . . . . . . . . . . . . . . . . . . . 17 6.3 ESP SA Proposals Sub-TLV . . . . . . . . . . . . . . . . . . 18 6.3.1 Transform Substructure . . . . . . . . . . . . . . . . . 19 7 Applicability to other VPN types . . . . . . . . . . . . . . . 19 8 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 20 9 Security Considerations . . . . . . . . . . . . . . . . . . . . 20 10 IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 10 References . . . . . . . . . . . . . . . . . . . . . . . . . . 20 11.1 Normative References . . . . . . . . . . . . . . . . . . . 20 11.2 Informative References . . . . . . . . . . . . . . . . . . 21 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 22 Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. AC: Attachment Circuit. ARP: Address Resolution Protocol. BD: Broadcast Domain. As per [RFC7432], an EVI consists of a single or multiple BDs. In case of VLAN-bundle and VLAN-based service models (see [RFC7432]), a BD is equivalent to an EVI. In case of VLAN-aware bundle service model, an EVI contains multiple BDs. Also, in this document, BD and subnet are equivalent terms. BD Route Target: refers to the Broadcast Domain assigned Route Target [RFC4364]. In case of VLAN-aware bundle service model, all the BD instances in the MAC-VRF share the same Route Target. BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as per [RFC7432]. DGW: Data Center Gateway. Sajassi et al. Expires January 13, 2021 [Page 3] INTERNET DRAFT Secure EVPN July 13, 2020 Ethernet A-D route: Ethernet Auto-Discovery (A-D) route, as per [RFC7432]. Ethernet NVO tunnel: refers to Network Virtualization Overlay tunnels with Ethernet payload. Examples of this type of tunnels are VXLAN or GENEVE. EVI: EVPN Instance spanning the NVE/PE devices that are participating on that EVPN, as per [RFC7432]. EVPN: Ethernet Virtual Private Networks, as per [RFC7432]. GRE: Generic Routing Encapsulation. GW IP: Gateway IP Address. IPL: IP Prefix Length. IP NVO tunnel: it refers to Network Virtualization Overlay tunnels with IP payload (no MAC header in the payload). IP-VRF: A VPN Routing and Forwarding table for IP routes on an NVE/PE. The IP routes could be populated by EVPN and IP-VPN address families. An IP-VRF is also an instantiation of a layer 3 VPN in an NVE/PE. IRB: Integrated Routing and Bridging interface. It connects an IP-VRF to a BD (or subnet). MAC-VRF: A Virtual Routing and Forwarding table for Media Access Control (MAC) addresses on an NVE/PE, as per [RFC7432]. A MAC-VRF is also an instantiation of an EVI in an NVE/PE. ML: MAC address length. ND: Neighbor Discovery Protocol. NVE: Network Virtualization Edge. GENEVE: Generic Network Virtualization Encapsulation, [GENEVE]. NVO: Network Virtualization Overlays. RT-2: EVPN route type 2, i.e., MAC/IP advertisement route, as defined in [RFC7432]. RT-5: EVPN route type 5, i.e., IP Prefix route. As defined in Section 3 of [EVPN-PREFIX]. Sajassi et al. Expires January 13, 2021 [Page 4] INTERNET DRAFT Secure EVPN July 13, 2020 SBD: Supplementary Broadcast Domain. A BD that does not have any ACs, only IRB interfaces, and it is used to provide connectivity among all the IP-VRFs of the tenant. The SBD is only required in IP-VRF- to-IP- VRF use-cases (see Section 4.4.). SN: Subnet. TS: Tenant System. VA: Virtual Appliance. VNI: Virtual Network Identifier. As in [RFC8365], the term is used as a representation of a 24-bit NVO instance identifier, with the understanding that VNI will refer to a VXLAN Network Identifier in VXLAN, or Virtual Network Identifier in GENEVE, etc. unless it is stated otherwise. VTEP: VXLAN Termination End Point, as in [RFC7348]. VXLAN: Virtual Extensible LAN, as in [RFC7348]. This document also assumes familiarity with the terminology of [RFC7432], [RFC8365] and [RFC7365]. Sajassi et al. Expires January 13, 2021 [Page 5] INTERNET DRAFT Secure EVPN July 13, 2020 1 Introduction The applications of EVPN-based solutions have become pervasive in Data Center, Service Provider, and Enterprise segments. It is being used for fabric overlays and inter-site connectivity in the Data Center market segment, for Layer-2, Layer-3, and IRB VPN services in the Service Provider market segment, and for fabric overlay and WAN connectivity in the Enterprise networks. For Data Center and Enterprise applications, there is a need to provide inter-site and WAN connectivity over public Internet in a secured manner with the same level of privacy, integrity, and authentication for tenant's traffic as used in IPsec tunneling using IKEv2. This document presents a solution where BGP point-to-multipoint signaling is leveraged for key and policy exchange among PE devices to create private pair-wise IPsec Security Associations without IKEv2 point-to- point signaling or any other direct peer-to-peer session establishment messages. EVPN uses BGP as control-plane protocol for distribution of information needed for discovery of PEs participating in a VPN, discovery of PEs participating in a redundancy group, customer MAC addresses and IP prefixes/addresses, aliasing information, tunnel encapsulation types, multicast tunnel types, multicast group memberships, and other info. The advantages of using BGP control plane in EVPN are well understood including the following: 1) A full mesh of BGP sessions among PE devices can be avoided by using Route Reflector (RR) where a PE only needs to setup a single BGP session between itself and the RR as opposed to setting up N BGP sessions to N other remote PEs; therefore, reducing number of BGP sessions from O(N^2) to O(N) in the network. Furthermore, RR hierarchy can be leveraged to scale the number of BGP routes on the RR. 2) MP-BGP route filtering and constrained route distribution can be leveraged to ensure that the control-plane traffic for a given VPN is only distributed to the PEs participating in that VPN. For setting up point-to-point security association (i.e., IPsec tunnel) between a pair of EVPN PEs, it is important to leverage BGP point-to-multipoint singling architecture using the RR along with its route filtering and constrain mechanisms to achieve the performance and the scale needed for large number of security associations (IPsec tunnels) along with their frequent re-keying requirements. Using BGP signaling along with the RR (instead of peer-to-peer protocol such as IKEv2) reduces number of message exchanges needed for SAs establishment and maintenance from O(N^2) to O(N) in the network. Sajassi et al. Expires January 13, 2021 [Page 6] INTERNET DRAFT Secure EVPN July 13, 2020 2 Requirements The requirements for secured EVPN are captured in the following subsections. 2.1 Tenant's Layer-2 and Layer-3 data & control traffic Tenant's layer-2 and layer-3 data and control traffic must be protected by IPsec cryptographic methods. This implies not only tenant's data traffic must be protected by IPsec but also tenant's control and routing information that are advertised in BGP must also be protected by IPsec. This in turn implies that BGP session must be protected by IPsec. 2.2 Tenant's Unicast & Multicast Data Protection Tenant's layer-2 and layer-3 unicast traffic must be protected by IPsec. In addition to that, tenant's layer-2 broadcast, unknown unicast, and multicast traffic as well as tenant's layer-3 multicast traffic must be protected by IPsec when ingress replication or assisted replication are used. The use of BGP P2MP signaling for setting up P2MP SAs in P2MP multicast tunnels is for future study. 2.3 P2MP Signaling for SA setup and Maintenance BGP P2MP signaling must be used for IPsec SAs setup and maintenance. The BGP signaling must follow P2MP signaling framework per [CONTROLLER-IKE] for IPsec SAs setup and maintenance in order to reduce the number of message exchanges from O(N^2) to O(N) among the participant PE devices. 2.4 Granularity of Security Association Tunnels The solution must support the setup and maintenance of IPsec SAs at the following level of granularities: 1) Per PE: A single IPsec tunnel between a pair of PEs to be used for all tenants' traffic supported by the pair of PEs. 2) Per tenant: A single IPsec tunnel per tenant per pair of PEs. For example, if there are 1000 tenants supported on a pair of PEs, then 1000 IPsec tunnels are required between that pair of PEs. 3) Per subnet: A single IPsec tunnel per subnet (e.g., per VLAN/EVI) of a tenant on a pair of PEs. 4) Per IP address: A single IPsec tunnel per pair of IP addresses of a tenant on a pair of PEs. Sajassi et al. Expires January 13, 2021 [Page 7] INTERNET DRAFT Secure EVPN July 13, 2020 5) Per MAC address: A single IPsec tunnel per pair of MAC addresses of a tenant on a pair of PEs. 6) Per Attachment Circuit: A single IPsec tunnel per pair of Attachment Circuits between a pair of PEs. 2.5 Support for Policy and DH-Group List The solution must support a single policy and DH group for all SAs as well as supporting multiple policies and DH groups among the SAs. 3 BGP Component The architecture that encompasses device-to-controller trust model, has several components among which is the signaling component. Secure EVPN Signaling, as defined in this document, is the BGP signaling component of the overall Architecture. We will briefly describe this Architecture here to further facilitate understanding how Secure EVPN fits into the overall architecture. The Architecture describes the components needed to create BGP based SD-WANs and how these components work together. Our intention is to list these components here along with their brief description and to describe this Architecture in details in a separate document where to specify the details for other parts of this architecture besides the BGP signaling component which is described in this document. The Architecture consists of four components. These components are Zero Touch Bring-up, Configuration Management, Orchestration, and Signaling. In addition to these components, secure communications must be provided between the edge nodes and all servers/devices providing the architecture components. 3.1 Zero Touch Bring-up (ZTB) The first component is a zero touch capability that allows an edge device to find and join its SD-WAN with little to no assistance other than power and network connectivity. The goal is to use existing work in this area. The requirements are that an edge device can locate its ZTB server/component of its SD-WAN controller in a secure manner and to proceed to receive its configuration. 3.2 Configuration Management After an edge device joins its SD-WAN, it needs to be configured. Sajassi et al. Expires January 13, 2021 [Page 8] INTERNET DRAFT Secure EVPN July 13, 2020 Configuration covers all device configuration, not just the configuration related to Secure EVPN. The previous Zero Touch Bring- up component will have directed the edge device, either directly or indirectly, to its configuration server/component. One example of a configuration server is the I2NSF Controller. After a device has been configured, it can engage in the next two components. Configuration may include updates over time and is not a one time only component. 3.3 Orchestration This component is optional. It allows for more dynamic updates of configuration and statistics information. Orchestration can be more dynamic than configuration. 3.4 Signaling Signaling is the component described in this document. The functionality of a Route Reflector is well understood. Here we describe the signaling component of BGP SD-WAN Architecture and the BGP extension/signaling for IPsec key management and policy. 4 Solution Description This solution uses BGP P2MP signaling where an originating PE only send a message to the Route Reflector (RR) and then the RR reflects that message to the interested recipient PEs. The framework for such signaling is described in [CONTROLLER-IKE] and it is referred to as device-to-controller trust model. This trust model is significantly different than the traditional peer-to-peer trust model where a P2P signaling protocol such as IKEv2 [RFC7296] is used in which the PE devices directly authenticate each other and agree upon security policy and keying material to protect communications between themselves. The device-to-controller trust model leverages P2MP signaling via the controller (e.g., the RR) to achieve much better scale and performance for establishment and maintenance of large number of pair-wise Security Associations (SAs) among the PEs. This device-to-controller trust model first secures the control channel between each device and the controller using peer-to-peer protocol such as IKEv2 [RFC7296] to establish P2P SAs between each PE and the RR. It then uses this secured control channel for P2MP signaling in establishment of P2P SAs between each pair of PE devices. Sajassi et al. Expires January 13, 2021 [Page 9] INTERNET DRAFT Secure EVPN July 13, 2020 Each PE advertises to other PEs via the RR the information needed in establishment of pair-wise SAs between itself an every other remote PEs. These pieces of information are sent as Sub-TLVs of IPSec tunnel type in BGP Tunnel Encapsulation attribute. These Sub-TLVs are detailed in section 5 and are based on the DIM message components from [CONTROLLER-IKE] and the IKEv2 specification [RFC7296]. The IPsec tunnel TLVs along with its Sub-TLVs are sent along with the BGP route (NLRI) for a given level of granularity. If only a single SA is required per pair of PE devices to multiplex user traffic for all tenants, then IPsec tunnel TLV is advertised along with IPv4 or IPv6 NLRI representing loopback address of the originating PE. It should be noted that this is not a VPN route but rather an IPv4 or IPv6 route. If a SA is required per tenant between a pair of PE devices, then IPsec tunnel TLV can be advertised along with EVPN IMET route representing the tenant or can be advertised along with a new EVPN route representing the tenant. If a SA is required per tenant's subnet (e.g., per VLAN) between a pair of PE devices, then IPsec tunnel TLV is advertised along with EVPN IMET route. If a SA is required between a pair of tenant's devices represented by a pair of IP addresses, then IPsec tunnel TLV is advertised along with EVPN IP Prefix Advertisement Route or EVPN MAC/IP Advertisement route. If a SA is required between a pair of tenant's devices represented by a pair of MAC addresses, then IPsec tunnel TLV is advertised along with EVPN MAC/IP Advertisement route. If a SA is required between a pair of Attachment Circuits (ACs) on two PE devices (where an AC can be represented by ), then IPsec tunnel TLV is advertised along with EVPN Ethernet AD route. 4.1 Inheritance of Security Policies Operationally, it is easy to configure a security association between a pair of PEs using BGP signaling. This is the default security association that is used for traffic that flows between peers. However, in the event more finer granularity of security association is desired on the traffic flows, it is possible to set up SAs between a pair of tenants, a pair of subnets within a tenant, a pair of IPs between a subnet, and a pair of MACs between a subnet using the appropriate EVPN routes as described above. In the event, there are no security TLVs associated with an EVPN route, there is a strict Sajassi et al. Expires January 13, 2021 [Page 10] INTERNET DRAFT Secure EVPN July 13, 2020 order in the manner security associations are inherited for such a route. This results in an EVPN route inheriting the security associations of the parent in a hierarchical fashion. For example, traffic between an IP pair is protected using security TLVs announced along with the EVPN IP Prefix Advertisement Route or EVPN MAC/IP Advertisement route as a first choice. If such TLVs are missing with the associated route, then one checks to see if the subnets the IPs are associated with has security TLVs with the EVPN IMET route. If they are present, those associations are used in securing the traffic. In the absence of them, the peer security associations are used. The order in which security associations are inherited are from the granular to the coarser, namely, IP/MAC associated TLVs with the EVPN route being the first preference, and the subnet, the tenant, and the peer associations preferred in that fashion. It should be noted that when a security association is made it is possible for it to be re-used by a large number of traffic flows. For example, a tenant security association may be associated with a number of child subnet routes. Clearly it is mandatory to keep a tenant security association alive, if there are one or more subnet routes that want to use that association. Logically, the security associations between a pair of entities creates a single secure tunnel. It is thus possible to classify the incoming traffic in the most granular sense {IP/MAC, subnet, tenant, peer} to a particular secure tunnel that falls within its route hierarchy. The policy that is applied to such traffic is independent from its use of an existing or a new secure tunnel. It is clear that since any number of classified traffic flows can use a security association, such a security association will not be torn down, if at least there is one policy using such a secure tunnel. 4.2 Distribution of Public Keys and Policies One of the requirements for this solution is to support a single DH group and a single policy for all SAs as well as to support multiple DH groups and policies among the SAs. The following subsections describe what pieces of information (what Sub-TLVs) are needed to be exchanged to support a single DH group and a single policy versus multiple DH groups and multiple policies. 4.2.1 Minimal DIM For SA establishment, at the minimum, a PE needs to advertise to other PEs, its DIM values as specified in [CONTROLLER-IKE]. These include: ID Tunnel ID N Nonce Sajassi et al. Expires January 13, 2021 [Page 11] INTERNET DRAFT Secure EVPN July 13, 2020 RC Rekey Counter I Indication of initial policy distribution KE DH public value. When this minimal set of DIM values is sent, then it is assumed that all peer PEs share the same policy for which DH group to use, as well as which IPSec SA policy to employ. Section 5.1 defines the Minimal DIM sub-TLV as part of IPsec tunnel TLV in BGP Tunnel Encapsulation Attribute. 4.2.2 Multiple Policies There can be scenarios for which there is a need to have multiple policy options. This can happen when there is a need for policy change and smooth migration among all PE devices to the new policy is required. It can also happen if different PE devices have different capabilities within the network. In these scenarios, PE devices need to be able to choose the correct policy to use for each other. This multi-policy scheme is described in section 6 of [CONTROLLER-IKE]. In order to support this multi-policy feature, a PE device MUST distribute a policy list. This list consists of multiple distinct policies in order of preference, where the first policy is the most preferred one. The receiving PE selects the policy by taking the received list (starting with the first policy) and comparing that against its own list and choosing the first one found in common. If there is no match, this indicates a configuration error and the PEs MUST NOT establish new SAs until a message is received that does produce a match. 4.2.2.1 Multiple DH-groups It can be the case that not all peers use the same DH group. When multiple DH groups are supported, the peer may include multiple KE Sub-TLVs. The order of the KE Sub-TLVs determines the preference. The preference and selection methods are specified in Section 6 of [CONTROLLER-IKE]. 4.2.2.2 Multiple or Single ESP SA policies In order to specify an ESP SA Policy, a DIM may include one or more SA Sub-TLVs. When all peers are configured by a controller with the same ESP SA policy, they MAY leave the SA out of the DIM. This minimizes messaging when group configuration is static and known. However, it may also be desirable to include the SA. If a single SA is included, the peer is indicating what ESP SA policy it uses, but is not willing to negotiate. If multiple SA Sub-TLVs are included, the peer is indicating that it is willing to negotiate. The order of Sajassi et al. Expires January 13, 2021 [Page 12] INTERNET DRAFT Secure EVPN July 13, 2020 the SA Sub-TLVs determines the preference. The preference and selection methods are specified in Section 6 of [CONTROLLER-IKE]. 4.3 Initial IPsec SAs Generation The procedure for generation of initial IPsec SAs is described in section 3 of [CONTROLLER-IKE]. This section gives a summary of it in context of BGP signaling. When a PE device first comes up and wants to setup an IPsec SA between itself and each of the interested remote PEs, it generates a DH pair along for each [what word here? "tennant"?] using an algorithm defined in the IKEv2 Diffie-Hellman Group Transform IDs [IKEv2-IANA]. The originating PE distributes the DH public value along with the other values in the DIM (using IPsec Tunnel TLV in Tunnel Encapsulation Attribute) to other remote PEs via the RR. Each receiving PE uses this DH public number and the corresponding nonce in creation of IPsec SA pair to the originating PE - i.e., an outbound SA and an inbound SA. The detail procedures are described in section 5.2 of [CONTROLLER-IKE]. 4.4 Re-Keying A PE can initiate re-keying at any time due to local time or volume based policy or due to the result of cipher counter nearing its final value. The rekey process is performed individually for each remote PE. If rekeying is performed with multiple PEs simultaneously, then the decision process and rules described in this rekey are performed independently for each PE. Section 4 of [CONTROLLER-IKE] describes this rekeying process in details and gives examples for a single IPsec device (e.g., a single PE) rekey versus multiple PE devices rekey simultaneously. 4.5 IPsec Databases The Peer Authorization Database (PAD), the Security Policy Database (SPD), and the Security Association Database (SAD) all need to be setup as defined in the IPsec Security Architecture [RFC4301]. Section 5 of [CONTROLLER-IKE] gives a summary description of how these databases are setup for the controller-based model where key is exchanged via P2MP signaling via the controller (i.e., the RR) and the policy can be either signaled via the RR (in case of multiple policies) or configured by the management station (in case of single policy). 5 Encapsulation Sajassi et al. Expires January 13, 2021 [Page 13] INTERNET DRAFT Secure EVPN July 13, 2020 Vast majority of Encapsulation for Network Virtualization Overlay (NVO) networks in deployment are based on UDP/IP with UDP destination port ID indicating the type of NVO encapsulation (e.g., VxLAN, GPE, GENEVE, GUE) and UDP source port ID representing flow entropy for load-balancing of the traffic within the fabric based on n-tuple that includes UDP header. When encrypting NVO encapsulated packets using IP Encapsulating Security Payload (ESP), the following two options can be used: a) adding a UDP header before ESP header (e.g., UDP header in clear) and b) no UDP header before ESP header (e.g., standard ESP encapsulation). The following subsection describe these encapsulation in further details. 5.1 Standard ESP Encapsulation When standard IP Encapsulating Security Payload (ESP) is used (without outer UDP header) for encryption of NVO packets, it is used in transport mode as depicted below. When such encapsulation is used, for BGP signaling, the Tunnel Type of Tunnel Encapsulation TLV is set to ESP-Transport and the Tunnel Type of Encapsulation Extended Community is set to NVO encapsulation type (e.g., VxLAN, GENEVE, GPE, etc.). This implies that the customer packets are first encapsulated using NVO encapsulation type and then it is further encapsulated & encrypted using ESP-Transport mode. Sajassi et al. Expires January 13, 2021 [Page 14] INTERNET DRAFT Secure EVPN July 13, 2020 +-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+ | MAC Header | | MAC Header | +-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+ | Eth Type = IPv4/IPv6 | | Eth Type = IPv4/IPv6 | +-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+ | IP Header | | IP Header | | Protocol = UDP | | Protocol = ESP | +-----------------------+ +-----------------------+ | UDP Header | | ESP Header | | Dest Port = VxLAN | +-----------------------+ +-----------------------+ | UDP Header | | VxLAN Header | | Dest Port = VxLAN | +-----------------------+ +-----------------------+ | Inner MAC Header | | VxLAN Header | +-----------------------+ +-----------------------+ | Inner Eth Payload | | Inner MAC Header | +-----------------------+ +-----------------------+ | CRC | | Inner Eth Payload | +-----------------------+ +-----------------------+ | ESP Trailer (NP=UDP) | +-----------------------+ | CRC | +-----------------------+ Figure 3: VxLAN Encapsulation within ESP 5.2 ESP Encapsulation within UDP packet In scenarios where NAT traversal is required ([RFC3948]) or where load balancing using UDP header is required, then ESP encapsulation within UDP packet as depicted in the following figure is used. The ESP for NVO applications is in transport mode. The outer UDP header (before the ESP header) has its source port set to flow entropy and its destination port set to 4500 (indicating ESP header follows). A non-zero SPI value in ESP header implies that this is a data packet (i.e., it is not an IKE packet). The Next Protocol field in the ESP trailer indicates what follows the ESP header, is a UDP header. This inner UDP header has a destination port ID that identifies NVO encapsulation type (e.g., VxLAN). Optimization of this packet format where only a single UDP header is used (only the outer UDP header) is for future study. When such encapsulation is used, for BGP signaling, the Tunnel Type of Tunnel Encapsulation TLV is set to ESP-in-UDP-Transport and the Tunnel Type of Encapsulation Extended Community is set to NVO Sajassi et al. Expires January 13, 2021 [Page 15] INTERNET DRAFT Secure EVPN July 13, 2020 encapsulation type (e.g., VxLAN, GENEVE, GPE, etc.). This implies that the customer packets are first encapsulated using NVO encapsulation type and then it is further encapsulated & encrypted using ESP-in-UDP with Transport mode. [RFC3948] +-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+ | MAC Header | | MAC Header | +-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+ | Eth Type = IPv4/IPv6 | | Eth Type = IPv4/IPv6 | +-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+ | IP Header | | IP Header | | Protocol = UDP | | Protocol = UDP | +-----------------------+ +-----------------------+ | UDP Header | | UDP Header | | Dest Port = VxLAN | | Dest Port = 4500(ESP) | +-----------------------+ +-----------------------+ | VxLAN Header | | ESP Header | +-----------------------+ +-----------------------+ | Inner MAC Header | | UDP Header | +-----------------------+ | Dest Port = VxLAN | | Inner Eth Payload | +-----------------------+ +-----------------------+ | VxLAN Header | | CRC | +-----------------------+ +-----------------------+ | Inner MAC Header | +-----------------------+ | Inner Eth Payload | +-----------------------+ | ESP Trailer (NP=UDP) | +-----------------------+ | CRC | +-----------------------+ Figure 4: VxLAN Encapsulation within ESP Within UDP 6 BGP Encoding This document defines two new Tunnel Types along with its associated sub-TLVs for The Tunnel Encapsulation Attribute [TUNNEL-ENCAP]. These tunnel types correspond to ESP-Transport and ESP-in-UDP-Transport as described in section 4. The following sub-TLVs apply to both tunnel types unless stated otherwise. 6.1 The Base (Minimal Set) DIM Sub-TLV Sajassi et al. Expires January 13, 2021 [Page 16] INTERNET DRAFT Secure EVPN July 13, 2020 The Base DIM is described in 3.2.1. One and only one Base DIM may be sent in the IPSec Tunnel TLV. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ID Length | Nonce Length |I| Flags | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Rekey | | Counter | +---------------------------------------------------------------+ | | ~ Originator ID + (Tenant ID) + (Subnet ID) + (Tenant Address) ~ | | +---------------------------------------------------------------+ | | ~ Nonce Data ~ | | +---------------------------------------------------------------+ Figure 5: The Base DIM Sub-TLV ID Length (16 bits) is the length of the Originator ID + (Tenant ID) + (Subnet ID) + (Tenant Address) in bytes. Nonce Length (8 bits) is the length of the Nonce Data in bytes I (1 bit) is the initial contact flag from [CONTROLLER-IKE] Flags (7 bits) are reserved and MUST be set to zero on transmit and ignored on receipt. The Rekey Counter is a 64 bit rekey counter as specified in [CONTROLLER-IKE] The Originator ID + (Tenant ID) + (Subnet ID) + (Tenant Address) is the tunnel identifier and uniquely identifies the tunnel. Depending on the granularity of the tunnel, the fields in () may not be used - i.e., for a tunnel at the PE level of granularity, only Originator ID is required. The Nonce Data is the nonce described in [CONTROLLER-IKE]. Its length is a multiple of 32 bits. Nonce lengths should be chosen to meet minimum requirements described in IKEv2 [RFC7296]. 6.2 Key Exchange Sub-TLV Sajassi et al. Expires January 13, 2021 [Page 17] INTERNET DRAFT Secure EVPN July 13, 2020 The KE Sub-TLV is described in 3.2.1 and 3.2.2.1. A KE is always required. One or more KE Sub-TLVs may be included in the IPSec Tunnel TLV. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Diffie-Hellman Group Num | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Key Exchange Data ~ | | +---------------------------------------------------------------+ Figure 6: Key Exchange Sub-TLV Diffie-Hellman Group Num 916 bits) identifies the Diffie-Hellman group in the Key Exchange Data was computed. Diffie-Hellman group numbers are discussed in IKEv2 [RFC7296] Appendix B and [RFC5114]. The Key Exchange payload is constructed by copying one's Diffie- Hellman public value into the "Key Exchange Data" portion of the payload. The length of the Diffie-Hellman public value is described for MOPD groups in [RFC7296] and for ECP groups in [RFC4753]. 6.3 ESP SA Proposals Sub-TLV The SA Sub-TLV is described in 3.2.2.2. Zero or more SA Sub-TLVs may be included in the IPSec Tunnel TLV. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ||Num Transforms| Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Transforms ~ | | +---------------------------------------------------------------+ Figure 8: ESP SA Proposals Sub-TLV Num Transforms is the number of transforms included. Reserved is not used and MUST be set to zero on transmit and MUST be ignored on receipt. Sajassi et al. Expires January 13, 2021 [Page 18] INTERNET DRAFT Secure EVPN July 13, 2020 6.3.1 Transform Substructure 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Transform Attr Length |Transform Type | Reserved. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Transform ID | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Transform Attributes ~ | | +---------------------------------------------------------------+ Figure 9: Transform Substructure Sub-TLV The Transform Attr Length is the length of the Transform Attributes field. The Transform Type is from Section 3.3.2 of [RFC7296] and [IKEV2IANA]. Only the values ENCR, INTEG, and ESN are allowed. The Transform ID specifies the transform identification value from [IKEV2IANA]. Reserved is unused and MUST be zero on transmit and MUST be ignored on receipt. The Transform Attributes are taken directly from 3.3.5 of [RFC7296]. 7 Applicability to other VPN types Although P2MP BGP signaling for establishment and maintenance of SAs among PE devices is described in this document in context of EVPN, there is no reason why it cannot be extended to other VPN technologies such as IP-VPN [RFC4364], VPLS [RFC4761] & [RFC4762], and MVPN [RFC6513] & [RFC6514] with ingress replication. The reason EVPN has been chosen is because of its pervasiveness in DC, SP, and Enterprise applications and because of its ability to support SA establishment at different granularity levels such as: per PE, Per tenant, per subnet, per Ethernet Segment, per IP address, and per MAC. For other VPN technology types, a much smaller granularity levels can be supported. For example for VPLS, only the granularity of per PE and per subnet can be supported. For per-PE granularity level, the mechanism is the same among all the VPN technologies as IPsec tunnel type (and its associated TLV and sub-TLVs) are sent along with the PE's loopback IPv4 (or IPv6) address. For VPLS, if Sajassi et al. Expires January 13, 2021 [Page 19] INTERNET DRAFT Secure EVPN July 13, 2020 per-subnet (per bridge domain) granularity level needs to be supported, then the IPsec tunnel type and TLV are sent along with VPLS AD route. The following table lists what level of granularity can be supported by a given VPN technology and with what BGP route. +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Functionality | EVPN | IP-VPN | MVPN | VPLS | +---------------+-------------+-------------+-----------+---------+ | per PE |IPv4/v6 route|IPv4/v6 route|IPv4/v6 rte|IPv4/v6 | +---------------+-------------+-------------+-----------+---------+ | per tenant |IMET (or new)|lpbk (or new)| I-PMSI | N/A | +---------------+-------------+-------------+-----------+---------+ | per subnet | IMET | N/A | N/A | VPLS AD | +---------------+-------------+-------------+-----------+---------+ | per IP |EVPN RT2/RT5 | VPN IP rt | *,G or S,G| N/A | +---------------+-------------+-------------+-----------+---------+ | per MAC | EVPN RT2 | N/A | N/A | N/A | +---------------+-------------+-------------+-----------+---------+ 8 Acknowledgements 9 Security Considerations 10 IANA Considerations A new transitive extended community Type of 0x06 and Sub-Type of TBD for EVPN Attachment Circuit Extended Community needs to be allocated by IANA. 10 References 11.1 Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC2119 Sajassi et al. Expires January 13, 2021 [Page 20] INTERNET DRAFT Secure EVPN July 13, 2020 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017. [RFC7432] Sajassi et al., "BGP MPLS Based Ethernet VPN", RFC 7432, February, 2015. [RFC8365] Sajassi et al., "A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, March, 2018. [TUNNEL-ENCAP] Rosen et al., "The BGP Tunnel Encapsulation Attribute", draft-ietf-idr-tunnel-encaps-03, November 2016. [CONTROLLER-IKE] Carrel et al., "IPsec Key Exchange using a Controller", draft-carrel-ipsecme-controller-ike-00, July, 2018. [IKEV2IANA] IANA, "Internet Key Exchange Version 2 (IKEv2) Parameters", . [RFC3948] Huttunen et al., "UDP Encapsulation of IPsec ESP Packets", RFC 3948, January 2005. [IKEV2-IANA] IANA, "Internet Key Exchange Version 2 (IKEv2) Parameters", February 2016, www.iana.org/assignments/ikev2-parameters/ikev2- parameters.xhtml. [RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, December 2005. 11.2 Informative References [RFC4364] Rosen, E., et. al., "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, February 2006. [RFC4761] Kompella, K., et. al., "Virtual Private LAN Service (VPLS) Using BGP for Auto-Discovery and Signaling", RFC 4761, January 2007. [RFC4762] Kompella, K., et. al., "Virtual Private LAN Service (VPLS) Using Label Distribution Protocol (LDP) Signaling", RFC 4762, January 2007. [RFC6513] Rosen, E., et. al., "Multicast in MPLS/BGP IP VPNs", RFC Sajassi et al. Expires January 13, 2021 [Page 21] INTERNET DRAFT Secure EVPN July 13, 2020 6513, February 2012. [RFC6514] Rosen, E., et. al., "BGP Encodings and Procedures for Multicast in MPLS/BGP IP VPNs", RFC 6514, February 2012. [RFC7606] Chen, E., Scudder, J., Mohapatra, P., and K. Patel, "Revised Error Handling for BGP UPDATE Messages", RFC 7606, August 2015, . [802.1Q] "IEEE Standard for Local and metropolitan area networks - Media Access Control (MAC) Bridges and Virtual Bridged Local Area Networks", IEEE Std 802.1Q(tm), 2014 Edition, November 2014. [RFC7348] Mahalingam, M., et al., "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014. [GENEVE] Gross, J., et al., "Geneve: Generic Network Virtualization Encapsulation", Work in Progress, draft-ietf-nvo3-geneve-06, March 2018. Authors' Addresses Ali Sajassi Cisco Email: sajassi@cisco.com Ayan Banerjee Cisco Email: ayabaner@cisco.com Samir Thoria Cisco Email: sthoria@cisco.com David Carrel Cisco Email: carrel@cisco.com Brian Weis Individual Sajassi et al. Expires January 13, 2021 [Page 22] INTERNET DRAFT Secure EVPN July 13, 2020 Email: bew.stds@gmail.com John Drake Juniper Email: jdrake@juniper.net Sajassi et al. Expires January 13, 2021 [Page 23]