idnits 2.17.1 draft-ietf-bess-evpn-ipvpn-interworking-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of lines with control characters in the document. ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 399: '... MAY be filled with an Autonomous ...' RFC 2119 keyword, line 461: '...) The gateway PE MUST NOT add the D-PA...' RFC 2119 keyword, line 467: '... looped ISF route and MUST be dropped....' RFC 2119 keyword, line 487: '...PE that imports an ISF route MUST flag...' RFC 2119 keyword, line 521: '... following rules MUST be observed by t...' (20 more instances...) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 6, 2019) is 1876 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC4364' is mentioned on line 169, but not defined == Missing Reference: 'RFC2119' is mentioned on line 929, but not defined == Missing Reference: 'RFC8174' is mentioned on line 929, but not defined == Unused Reference: 'ENCAP-ATT' is defined on line 967, but no explicit reference was found in the text == Outdated reference: A later version (-15) exists of draft-ietf-bess-evpn-inter-subnet-forwarding-05 == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-10 Summary: 2 errors (**), 0 flaws (~~), 7 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BESS Workgroup J. Rabadan, Ed. 3 Internet Draft Nokia 4 Intended status: Standards Track A. Sajassi, Ed. 5 Cisco 7 E. Rosen 8 J. Drake 9 W. Lin 10 Juniper 12 J. Uttaro 13 AT&T 15 A. Simpson 16 Nokia 18 Expires: September 7, 2019 March 6, 2019 20 EVPN Interworking with IPVPN 21 draft-ietf-bess-evpn-ipvpn-interworking-00 23 Abstract 25 EVPN is used as a unified control plane for tenant network intra and 26 inter-subnet forwarding. When a tenant network spans not only EVPN 27 domains but also domains where IPVPN provides inter-subnet 28 forwarding, there is a need to specify the interworking aspects 29 between both EVPN and IPVPN domains, so that the end to end tenant 30 connectivity can be accomplished. This document specifies how EVPN 31 should interwork with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families 32 for inter-subnet forwarding. 34 Status of this Memo 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF), its areas, and its working groups. Note that 41 other groups may also distribute working documents as Internet- 42 Drafts. 44 Internet-Drafts are draft documents valid for a maximum of six months 45 and may be updated, replaced, or obsoleted by other documents at any 46 time. It is inappropriate to use Internet-Drafts as reference 47 material or to cite them other than as "work in progress." 49 The list of current Internet-Drafts can be accessed at 50 http://www.ietf.org/ietf/1id-abstracts.txt 52 The list of Internet-Draft Shadow Directories can be accessed at 53 http://www.ietf.org/shadow.html 55 This Internet-Draft will expire on September 7, 2019. 57 Copyright Notice 59 Copyright (c) 2019 IETF Trust and the persons identified as the 60 document authors. All rights reserved. 62 This document is subject to BCP 78 and the IETF Trust's Legal 63 Provisions Relating to IETF Documents 64 (http://trustee.ietf.org/license-info) in effect on the date of 65 publication of this document. Please review these documents 66 carefully, as they describe your rights and restrictions with respect 67 to this document. Code Components extracted from this document must 68 include Simplified BSD License text as described in Section 4.e of 69 the Trust Legal Provisions and are provided without warranty as 70 described in the Simplified BSD License. 72 Table of Contents 74 1. Introduction and Problem Statement . . . . . . . . . . . . . . 3 75 2. Terminology and Interworking PE Components . . . . . . . . . . 3 76 3. Domain Path Attribute (D-PATH) . . . . . . . . . . . . . . . . 9 77 3.1. D-PATH and Loop Prevention . . . . . . . . . . . . . . . . 11 78 4. BGP Path Attribute Propagation across ISF SAFIs . . . . . . . . 12 79 4.1. No-Propagation-Mode . . . . . . . . . . . . . . . . . . . . 12 80 4.2. Uniform-Propagation-Mode . . . . . . . . . . . . . . . . . 12 81 4.3. Aggregation of Routes and Path Attribute Propagation . . . 13 82 5. Route Selection Process between EVPN and other ISF SAFIs . . . 14 83 6. Composite PE Procedures . . . . . . . . . . . . . . . . . . . . 15 84 7. Gateway PE Procedures . . . . . . . . . . . . . . . . . . . . . 17 85 8. Interworking Use-Cases . . . . . . . . . . . . . . . . . . . . 19 86 9. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 21 87 10. Conventions used in this document . . . . . . . . . . . . . . 21 88 11. Security Considerations . . . . . . . . . . . . . . . . . . . 21 89 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 90 13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 91 13.1. Normative References . . . . . . . . . . . . . . . . . . . 21 92 13.2. Informative References . . . . . . . . . . . . . . . . . . 22 93 14. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 94 15. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 22 95 16. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 22 97 1. Introduction and Problem Statement 99 EVPN is used as a unified control plane for tenant network intra and 100 inter-subnet forwarding. When a tenant network spans not only EVPN 101 domains but also domains where IPVPN provides inter-subnet 102 forwarding, there is a need to specify the interworking aspects 103 between both EVPN and IPVPN domains, so that the end to end tenant 104 connectivity can be accomplished. This document specifies how EVPN 105 should interwork with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families 106 for inter-subnet forwarding. 108 EVPN supports the advertisement of IPv4 or IPv6 prefixes in two 109 different route types: 111 o Route Type 2 - MAC/IP route (only for /32 and /128 host routes), as 112 described by [INTER-SUBNET]. 114 o Route Type 5 - IP Prefix route, as described by [IP-PREFIX]. 116 When interworking with other BGP address families (AFIs/SAFIs) for 117 inter-subnet forwarding, the IP prefixes in those two EVPN route 118 types must be propagated to other domains using different SAFIs. Some 119 aspects of that propagation must be clarified. Examples of these 120 aspects or procedures across BGP families are: route selection, loop 121 prevention or BGP Path attribute propagation. The Interworking PE 122 concepts are defined in section 2, and the rest of the document 123 describes the interaction between Interworking PEs and other PEs for 124 end-to-end inter-subnet forwarding. 126 2. Terminology and Interworking PE Components 128 This section summarizes the terminology related to the "Interworking 129 PE" concept that will be used throughout the rest of the document. 131 +-------------------------------------------------------------+ 132 | | 133 | +------------------+ Interworking PE | 134 | Attachment | +------------------+ | 135 | Circuit(AC1) | | +----------+ | MPLS/NVO tnl 136 ----------------------*Bridge | | +------ 137 | | | |Table(BT1)| | +-----------+ / \ \ 138 MPLS/NVO tnl +-------->| *---------* |<--> | Eth | 139 -------+ | | | |Eth-Tag x + |IRB1| | \ / / 140 / Eth / \<-+ | | +----------+ | | | +------ 141 | | | | | ... | | IP-VRF1 | | 142 \ \ /<-+ | | +----------+ | | RD2/RT2 |MPLS/NVO tnl 143 -------+ | | | |Bridge | | | | +------ 144 | +-------->|Table(BT2)| |IRB2| | / \ \ 145 | | | | *---------* |<--> | IP | 146 ----------------------*Eth-Tag y | | +-----*-----+ \ / / 147 | AC2 | | +----------+ | AC3| +------ 148 | | | MAC-VRF1 | | | 149 | +-+ RD1/RT1 | | | 150 | +------------------+ | SAFIs | 151 | | 1 +---+ | 152 -------------------------------------------------+ 128 |BGP| | 153 | EVPN +---+ | 154 | | 155 +-------------------------------------------------------------+ 157 Figure 1 EVPN-IPVPN Interworking PE 159 o ISF SAFI: Inter-Subnet Forwarding (ISF) SAFI is a MP-BGP Sub- 160 Address Family that advertises reachability for IP prefixes and can 161 be used for inter-subnet forwarding within a given tenant network. 162 The ISF SAFIs are 1 (including IPv4 and IPv6 AFIs), 128 (including 163 IPv4 and IPv6 AFIs) and 70 (EVPN, including only AFI 25). 165 o ISF route: a route for a given prefix whose ISF SAFI may change as 166 it transits different domains. 168 o IP-VRF: an IP Virtual Routing and Forwarding table, as defined in 169 [RFC4364]. It is also the instantiation of an IPVPN in a PE. Route 170 Distinguisher and Route Target(s) are required properties of an IP- 171 VRF. 173 o MAC-VRF: a MAC Virtual Routing and Forwarding table, as defined in 174 [RFC7432]. It is also the instantiation of an EVI (EVPN Instance) 175 in a PE. Route Distinguisher and Route Target(s) are required 176 properties and they are normally different than the ones defined in 177 the associated IP-VRF. 179 o BT: a Bridge Table, as defined in [RFC7432]. A BT is the 180 instantiation of a Broadcast Domain in a PE. When there is a single 181 Broadcast Domain in a given EVI, the MAC-VRF in each PE will 182 contain a single BT. When there are multiple BTs within the same 183 MAC-VRF, each BT is associated to a different Ethernet Tag. The 184 EVPN routes specific to a BT, will indicate which Ethernet Tag the 185 route corresponds to. 187 Example: In Figure 1, MAC-VRF1 has two BTs: BT1 and BT2. Ethernet 188 Tag x is defined in BT1 and Ethernet Tag y in BT2. 190 o AC: Attachment Circuit or logical interface associated to a given 191 BT or IP-VRF. To determine the AC on which a packet arrived, the PE 192 will examine the combination of a physical port and VLAN tags 193 (where the VLAN tags can be individual c-tags, s-tags or ranges of 194 both). 196 Example: In Figure 1, AC1 is associated to BT1, AC2 to BT2 and AC3 197 to IP-VRF1. 199 o IRB: Integrated Routing and Bridging interface. It refers to the 200 logical interface that connects a BT to an IP-VRF and allows to 201 forward packets with destination in a different subnet. 203 o MPLS/NVO tnl: It refers to a tunnel that can be MPLS or NVO-based 204 (Network Virtualization Overlays) and it is used by MAC-VRFs and 205 IP-VRFs. Irrespective of the type, the tunnel may carry an Ethernet 206 or an IP payload. MAC-VRFs can only use tunnels with Ethernet 207 payloads (setup by EVPN), whereas IP-VRFs can use tunnels with 208 Ethernet (setup by EVPN) or IP payloads (setup by EVPN or IPVPN). 209 IPVPN-only PEs have IP-VRFs but they cannot send or receive traffic 210 on tunnels with Ethernet payloads. 212 Example: Figure 1 shows an MPLS/NVO tunnel that is used to 213 transport Ethernet frames to/from MAC-VRF1. The PE determines the 214 MAC-VRF and BT the packets belong to based on the EVPN label (MPLS 215 or VNI). Figure 1 also shows two MPLS/NVO tunnels being used by IP- 216 VRF1, one carrying Ethernet frames and the other one carrying IP 217 packets. 219 o RT-2: Route Type 2 or MAC/IP route, as per [RFC7432]. 221 o RT-5: Route Type 5 or IP Prefix route, as per [IP-PREFIX]. 223 o Domain: Two PEs are in the same domain if they are attached to the 224 same tenant and the packets between them do not require a data path 225 IP lookup (in the tenant space) in any intermediate router. A 226 gateway PE is always configured with multiple Domain-IDs. 228 Example 1: Figure 4 depicts an example where TS1 and TS2 belong to 229 the same tenant, and they are located in different Data Centers 230 that are connected by gateway PEs (see the gateway PE definition 231 later). These gateway PEs use IPVPN in the WAN. When TS1 sends 232 traffic to TS2, the intermediate routers between PE1 and PE2 233 require a tenant IP lookup in their IP-VRFs so that the packets can 234 be forwarded. In this example there are three different domains. 235 The gateway PEs connect the EVPN domains to the IPVPN domain. 237 GW1------------GW3 238 +------+ +------+ 239 +-------------|IP-VRF| |IP-VRF|-------------+ 240 PE1 +------+ +------+ PE2 241 +------+ DC1 | WAN | DC2 +------+ 242 TS1-|IP-VRF| EVPN | IPVPN | EVPN |IP-VRF|-TS2 243 +------+ GW2 GW4 +---+--+ 244 | +------+ +------+ | 245 +-------------|IP-VRF| |IP-VRF|-------------+ 246 +------+ +------+ 247 +--------------+ 248 DOMAIN 1 DOMAIN 2 DOMAIN 3 249 <---------------> <------------> <----------------> 251 Figure 4 Multiple domain DCI example 253 Example 2: Figure 5 illustrates a similar example, but PE1 and PE2 254 are now connected by a BGP-LU (BGP Labeled Unicast) tunnel, and 255 they have a BGP peer relationship for EVPN. Contrary to Example 1, 256 there is no need for tenant IP lookups on the intermediate routers 257 in order to forward packets between PE1 and PE2. Therefore, there 258 is only one domain in the network and PE1/PE2 belong to it. 260 EVPN 261 <-------------------------------------------------> 262 BGP-LU 263 <-------------------------------------------------> 265 ASBR------------ASBR 266 +------+ +------+ 267 +-------------| | | |-------------+ 268 PE1 +------+ +--+---+ PE2 269 +------+ DC1 | WAN | DC2 +------+ 270 TS1-|IP-VRF| EVPN | | EVPN |IP-VRF|-TS2 271 +------+ ASBR ASBR +---+--+ 272 | +------+ +------+ | 273 +-------------| | | |-------------+ 274 +------+ +------+ 275 +--------------+ 277 <--------------------DOMAIN-1---------------------> 279 Figure 5 Single domain DCI example 281 o Regular Domain: a domain in which a single control plane, IPVPN or 282 EVPN, is used and which is composed of regular PEs, see below. In 283 Figures 4 and 5, above, all domains are regular domains. 285 o Composite Domain: a domain in which multiple control planes, IPVPN 286 and EVPN, are used and which is composed of regular PEs, see below, 287 and composite PEs, see below. 289 o Regular PE: a PE that is attached to a domain, either regular or 290 composite, and which uses one of the control plane protocols (IPVPN 291 or EVPN) operating in the domain. 293 o Interworking PE: a PE that may advertise a given prefix with an 294 EVPN ISF route (RT-2 or RT-5) and/or an IPVPN ISF route. An 295 interworking PE has one IP-VRF per tenant, and one or multiple MAC- 296 VRFs per tenant. Each MAC-VRF may contain one or more BTs, where 297 each BT may be attached to that IP-VRF via IRB. There are two types 298 of Interworking PEs: composite PEs and gateway PEs. Both PE 299 functions can be independently implemented per tenant and they may 300 both be implemented for the same tenant. 302 Example: Figure 1 shows an interworking PE of type gateway, where 303 ISF SAFIs 1, 128 and 70 are enabled. IP-VRF1 and MAC-VRF1 are 304 instantiated on the PE, and together provide inter-subnet 305 forwarding for the tenant. 307 o Composite PE: an interworking PE that is attached to a composite 308 domain and which advertises a given prefix to an IPVPN peer with an 309 IPVPN ISF route, to an EVPN peer with an EVPN ISF route, and to a 310 route reflector with both an IPVPN and EVPN ISF route. A composite 311 PE performs the procedures of Sections 5 and 6. 313 Example: Figure 2 shows an example where PE1 is a composite PE 314 since PE1 has EVPN and another ISF SAFI enabled to the same route- 315 reflector, and PE1 advertises a given IP prefix IPn/x twice, one 316 using EVPN and another one using ISF SAFI 128. PE2 and PE3 are not 317 composite PEs. 319 +---+ 320 |PE2| 321 +---+ 322 ^ 323 |EVPN 324 IW EVPN v 325 +---+ IPVPN ++-+ +---+ 326 |PE1| <----> |RR| <---> |PE3| 327 +---+ +--+ IPVPN +---+ 328 Composite 330 Figure 2 Interworking composite PE example 332 o Gateway PE: an interworking PE that is attached to two domains, 333 each either regular or composite, and which, based on 334 configuration, does one of the following: 336 - Propagates the same control plane protocol, either IPVPN or EVPN, 337 between the two domains. 339 - Propagates an ISF route with different ISF SAFIs between the two 340 domains. E.g., propagate an EVPN ISF route in one domain as an 341 IPVPN ISF route in the other domain and vice versa. A gateway PE 342 performs the procedures of Sections 3, 4, 5 and 7. 344 A gateway PE is always configured with multiple Domain-IDs. The 345 Domain-ID is encoded in the Domain Path Attribute (D-PATH), and 346 advertised along with EVPN and other ISF SAFI routes. Section 3 347 describes the D-PATH attribute. 349 Example: Figure 3 illustrates an example where PE1 is a gateway 350 PE since the EVPN and IPVPN SAFIs are enabled on different BGP 351 peers, and a given local IP prefix IPn/x is sent to both BGP 352 peers for the same tenant. PE2 and PE1 are in one domain and PE3 353 and PE1 are in another domain. 355 IW 356 +---+ EVPN +---+ IPVPN +---+ 357 |PE2| <----> |PE1| <----> |PE3| 358 +---+ +---+ +---+ 359 Gateway 361 Figure 3 Interworking gateway PE example 363 o Composite/Gateway PE: an interworking PE that is both a composite 364 PE and a gateway PE that is attached to two domains, one regular 365 and one composite, and which does the following: 367 - Propagates an ISF route, either IPVPN or EVPN, from the regular 368 domain into the composite domain. Within the composite domain it 369 acts as a composite PE. 371 - Propagates an ISF route, either IPVPN or EVPN, from the composite 372 domain into the regular domain. Within the regular domain it is 373 propagated as an ISF route using the ISF SAFI for that domain. 375 This is particularly useful when a tenant network is attached to 376 both IPVPN and EVPN domains, any-to-any connectivity is required, 377 and end-to-end control plane consistency, when possible, is 378 desired. 380 It would be instantiated by attaching the disparate, regular 381 IPVPN and EVPN domains via these PEs to a central composite 382 domain. 384 3. Domain Path Attribute (D-PATH) 386 The BGP Domain Path (D-PATH) attribute is an optional and transitive 387 BGP path attribute. 389 Similar to AS_PATH, D-PATH is composed of a length field followed by 390 a sequence of Domain segments, where each domain segment is 391 represented by . 393 o The length field is a 1-octet field, containing the number of 394 domain segments. 396 o DOMAIN-ID is a 6-octet field that represents a domain. It is 397 composed of a 4-octet Global Administrator sub-field and a 2-octet 398 Local Administrator sub-field. The Global Administrator sub-field 399 MAY be filled with an Autonomous System Number (ASN), an IPv4 400 address, or any value that guarantees the uniqueness of the DOMAIN- 401 ID when the tenant network is connected to multiple Operators. 403 o ISF_SAFI_TYPE is a 1-octet field that indicates the Inter-Subnet 404 Forwarding SAFI type in which a route was advertised in the DOMAIN. 405 The following types are valid in this document: 407 Value Type 409 1 SAFI 1 410 70 EVPN 411 128 SAFI 128 413 About the BGP D-PATH attribute: 415 a) Identifies the sequence of domains, each identified by a through which a given ISF route has passed. 418 - This attribute list may contain zero, one or more entries. 420 - The first entry in the list (leftmost) is the from which a gateway PE is propagating an ISF 422 route. The last entry in the list (rightmost) is the from which a gateway PE received an ISF route 424 without a D-PATH attribute. Intermediate entries in the list are 425 domains that the ISF route has transited. 427 - As an example, an ISF route received with a D-PATH attribute of 428 {<6500:2:IPVPN>,<6500:1:EVPN>} indicates that the ISF route was 429 originated in EVPN domain 6500:1, and propagated into IPVPN 430 domain 6500:2. 432 b) It is added/modified by a gateway PE when propagating an update to 433 a different domain: 435 - A gateway PE's IP-VRF, that connects two domains, belongs to two 436 DOMAIN-IDs, e.g. 6500:1 for EVPN and 6500:2 for IPVPN. 438 - Whenever a prefix arrives at a gateway PE in a particular ISF 439 SAFI route, if the gateway PE needs to export that prefix to a 440 BGP peer, the gateway PE will prepend a segment to the list of segments in the 442 received D-PATH. 444 - For instance, in an IP-VRF configured with DOMAIN-IDs 6500:1 for 445 EVPN and 6500:2 for IPVPN, if an EVPN route for prefix P is 446 received and P installed in the IP-VRF, the IPVPN route for P 447 that is exported to an IPVPN peer will prepend the segment 448 <6500:1:EVPN> to the previously received D-PATH attribute. 449 Likewise, IP-VRF prefixes that are received from IP-VPN, will be 450 exported to EVPN peers with the additional segment 451 <6500:2:IPVPN>. 453 - In the above example, if the EVPN route is received without D- 454 PATH, the gateway PE will add the D-PATH attribute with segment 455 <6500:1:EVPN> when re-advertising to domain 6500:2. 457 - Within the originating domain, the update does not contain a D- 458 PATH attribute because the update has not passed through a 459 gateway PE yet. 461 c) The gateway PE MUST NOT add the D-PATH attribute to ISF routes 462 generated for IP-VRF prefixes that are not learned via any ISF 463 SAFI, for instance, local prefixes. 465 d) An ISF route received by a gateway PE with a D-PATH attribute that 466 contains one or more of its locally configured domains for the IP- 467 VRF is considered to be a looped ISF route and MUST be dropped. 469 e) The number of domain segments in the D-PATH attribute indicates 470 the number of gateway PEs that the ISF route update has transited. 472 3.1. D-PATH and Loop Prevention 474 The D-PATH attribute is used to prevent loops in interworking PE 475 networks. For instance, in the example of Figure 4, gateway GW1 476 receives TS1 prefix in two different ISF routes: 478 o In an EVPN RT-5 with next-hop PE1 and no D-PATH attribute. 480 o In a SAFI 128 route with next-hop GW2 and D-PATH = (6500:1:EVPN), 481 assuming that DOMAIN-ID for domain 1 is 6500:1. 483 Gateway GW1 flags the SAFI 128 route as a loop, and does not re- 484 advertise it to the EVPN neighbors since the route includes the GW1's 485 local domain. 487 In general, any interworking PE that imports an ISF route MUST flag 488 the route as "looped" if its D-PATH contains a segment, where DOMAIN-ID matches a local DOMAIN-ID 490 in the tenant IP-VRF. 492 4. BGP Path Attribute Propagation across ISF SAFIs 494 Based on configurations a gateway PE is required to propagate an ISF 495 route with different ISF SAFIs between two domains. This requires a 496 definition of what a gateway PE is to do with Path attributes 497 attached to the ISF route that it is propagating. 499 4.1. No-Propagation-Mode 501 This is the default mode of operation. In this mode, the gateway PE 502 will simply re-initialize the Path Attributes when propagating an ISF 503 route, as though it would for direct or local IP prefixes. This model 504 may be enough in those use-cases where the EVPN domain is considered 505 an "abstracted" CE and remote IPVPN/IP PEs don't need to consider the 506 original EVPN Attributes for path calculations. 508 Since this mode of operation does not propagate the D-PATH attribute 509 either, redundant gateway PEs are exposed to routing loops. Those 510 loops may be resolved by policies and the use of other attributes, 511 such as the Route Origin extended community [RFC4360], however not 512 all the loop situations may be solved. 514 4.2. Uniform-Propagation-Mode 516 In this mode, the gateway PE simply keeps accumulating or mapping 517 certain key commonly used Path Attributes when propagating an ISF 518 route. This mode is typically used in networks where EVPN and IPVPN 519 SAFIs are used seamlessly to distribute IP prefixes. 521 The following rules MUST be observed by the gateway PE when 522 propagating Path Attributes: 524 o The gateway PE imports an ISF route in the IP-VRF and stores the 525 original Path Attributes. The following set of Path Attributes 526 SHOULD be propagated by the gateway PE to other ISF SAFIs (other 527 Path Attributes SHOULD NOT be propagated): 529 - AS_PATH 530 - D-PATH 531 - IBGP-only Path Attributes: LOCAL_PREF, ORIGINATOR_ID, CLUSTER_ID 532 - MED 533 - AIGP 534 - Communities, (non-EVPN) Extended Communities and Large 535 Communities 537 o When propagating an ISF route to a different ISF SAFI and IBGP 538 peer, the gateway PE SHOULD copy the AS_PATH of the originating 539 family and add it to the destination family without any 540 modification. When re-advertising to a different ISF SAFI and EBGP 541 peer, the gateway PE SHOULD copy the AS_PATH of the originating 542 family and prepend the IP-VRF's AS before sending the route. 544 o When propagating an ISF route to IBGP peers, the gateway PE SHOULD 545 copy the IBGP-only Path Attributes from the originating SAFI to the 546 re-advertised route. 548 o Communities, non-EVPN Extended Communities and Large Communities 549 SHOULD be copied by the gateway PE from the originating SAFI route. 551 4.3. Aggregation of Routes and Path Attribute Propagation 553 Instead of propagating a high number of (host) ISF routes between ISF 554 SAFIs, a gateway PE that receives multiple ISF routes of one ISF SAFI 555 MAY choose to propagate a single ISF aggregate route with a different 556 ISF SAFI. In this document, aggregation is used to combine the 557 characteristics of multiple ISF routes of the same ISF SAFI in such 558 way that a single aggregate ISF route of a different ISF SAFI can be 559 propagated. Aggregation of multiple ISF routes of one ISF SAFI into 560 an aggregate ISF route of a different ISF SAFI is only done by a 561 gateway PE. 563 Aggregation on gateway PEs may use either the No-Propagation-Mode or 564 the Uniform-Propagation-Mode explained in Sections 4.1. and 4.2, 565 respectively. 567 When using Uniform-Propagation-Mode, Path Attributes of the same type 568 code MAY be aggregated according to the following rules: 570 o AS_PATH is aggregated based on the rules in [RFC4271]. The gateway 571 PEs SHOULD NOT receive AS_PATH attributes with path segments of 572 type AS_SET [RFC6472]. Routes received with AS_PATH attributes 573 including AS_SET path segments MUST NOT be aggregated. 575 o ISF routes that have different attributes of the following type 576 codes MUST NOT be aggregated: D-PATH, LOCAL_PREF, ORIGINATOR_ID, 577 CLUSTER_ID, MED or AIGP. 579 o The Community, Extended Community and Large Community attributes of 580 the aggregate ISF route MUST contain all the Communities/Extended 581 Communities/Large Communities from all of the aggregated ISF 582 routes. 584 Assuming the aggregation can be performed (the above rules are 585 applied), the operator should consider aggregation to deal with 586 scaled tenant networks where a significant number of host routes 587 exists. For a example, large Data Centers. 589 5. Route Selection Process between EVPN and other ISF SAFIs 591 A PE may receive an IP prefix in ISF routes with different ISF SAFIs, 592 from the same or different BGP peer. It may also receive the same IP 593 prefix (host route) in an EVPN RT-2 and RT-5. A route selection 594 algorithm across all ISF SAFIs is needed so that: 596 o Different gateway and composite PEs have a consistent and 597 deterministic view on how to reach a given prefix. 599 o Prefixes advertised in EVPN and other ISF SAFIs can be compared 600 based on path attributes commonly used by operators across 601 networks. 603 o Equal Cost Multi-Path (ECMP) is allowed across EVPN and other ISF 604 SAFI routes. 606 For a given prefix advertised in one or more non-EVPN ISF routes, the 607 BGP best path selection procedure will produce a set of "non-EVPN 608 best paths". For a given prefix advertised in one or more EVPN ISF 609 routes, the BGP best path selection procedure will produce a set of 610 "EVPN best paths". To support IP/EVPN interworking, it is then 611 necessary to run a tie-breaking selection algorithm on the union of 612 these two sets. This tie-breaking algorithm begins by considering all 613 EVPN and other ISF SAFI routes, equally preferable routes to the same 614 destination, and then selects routes to be removed from 615 consideration. The process terminates as soon as only one route 616 remains in consideration. 618 The route selection algorithm must remove from consideration the 619 routes following the rules and the order defined in [RFC4271], with 620 the following exceptions and in the following order: 622 1- Immediately after removing from consideration all routes that are 623 not tied for having the highest Local Preference, any routes that 624 do not have the shortest D-PATH are also removed from 625 consideration. Routes with no D-PATH are considered to have a 626 zero-length D-PATH. 628 2- Then regular [RFC4271] selection criteria is followed. 630 3- At the end of the selection algorithm, if at least one route still 631 under consideration is an RT-2 route, remove from consideration 632 any RT-5 routes. 634 4- Steps 1-3 could possibly leave Equal Cost Multi-Path (ECMP) 635 between IP and EVPN paths. By default, the EVPN path is considered 636 (and the IP path removed from consideration). However, if ECMP 637 across ISF SAFIs is enabled by policy, and an "IP path" and an 638 "EVPN path" remain at the end of step 3, both path types will be 639 used. 641 Example 1 - PE1 receives the following routes for IP1/32, that are 642 candidate to be imported in IP-VRF-1: 644 {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=(100,200)} 645 {SAFI=EVPN, RT-5, Local-Pref=100, AS-Path=(100,200)} 646 {SAFI=128, Local-Pref=100, AS-Path=(100,200)} 648 Selected route: {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=100,200] 649 (due to step 3, and no ECMP) 651 Example 2 - PE1 receives the following routes for IP2/24, that are 652 candidate to be imported in IP-VRF-1: 654 {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS-Path=(100,200), 655 MED=10} 656 {SAFI=128, D-PATH=(6500:1:EVPN,6500:2:IPVPN), AS-Path=(200), 657 MED=200} 659 Selected route: {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS- 660 Path=(100,200), MED=10} (due to step 1) 662 6. Composite PE Procedures 664 As described in Section 2, composite PEs are typically used in tenant 665 networks where EVPN and IPVPN are both used to provide inter-subnet 666 forwarding within the same composite domain. 668 Figure 6 depicts an example of a composite domain, where PE1/PE2/PE4 669 are composite PEs (they support EVPN and IPVPN ISF SAFIs on their 670 peering to the Route Reflector), and PE3 is a regular IPVPN PE. 672 +-----------------------------------+ 673 | | 674 | MPLS IPVPN PE3 675 | Network +----------+ IP3/24 676 | IPVPN |+------+ | +---+ 677 | +----->||IP-VRF|------|CE3| 678 Composite PE1 | |+------+ | +---+ 679 +---------------+ | +----------+ 680 | +------+ | EVPN v | 681 | |IP-VRF| | IPVPN +--+ | 682 | +----| | | <------> |RR| | 683 +---+ | | +------+ | +--+ Composite PE4 684 |CE2|----|MAC-VRF| | ^ ^ +---------+ IP4/24 685 +---+ | +-------+ | EVPN | | EVPN |+------+ | +---+ 686 +---|-----------+ IPVPN | | IPVPN ||IP-VRF|-----|CE4| 687 | | +----+ +-------->|+------+ | +---+ 688 IP1/24 | | v +---------+ 689 +---+ | | +---------------+ | 690 |CE1|--+ +----| +------+ +--------------+ 691 +---+ | |IP-VRF| | 692 | | +----| | | 693 | | | +------+ | 694 +--------------|MAC-VRF| | 695 | +-------+ | 696 +---------------+ 697 Composite PE2 699 Figure 6 Composite PE example 701 In a composite domain with composite and regular PEs: 703 o The composite PEs advertise the same IP prefixes in each ISF SAFI 704 to the RR. For example, in Figure 6, the prefix IP1/24 is 705 advertised by PE1 and PE2 to the RR in two separate NLRIs, one for 706 AFI/SAFI 1/128 and another one for EVPN. 708 o The RR does not forward EVPN routes to PE3 (since the RR does not 709 have the EVPN SAFI enabled on its BGP session to PE3), whereas the 710 IPVPN routes are forwarded to all the PEs. 712 o PE3 receives only the IPVPN route for IP1/24 and resolves the BGP 713 next-hop to an MPLS tunnel (with IP payload) to PE1 and/or PE2. 715 o Composite PE4 receives IP1/24 encoded in EVPN and another ISF SAFI 716 route (EVPN RT-5 and IPVPN). The route selection follows the 717 procedures in Section 5. Assuming an EVPN route is selected, PE4 718 resolves the BGP next-hop to an MPLS tunnel (with Ethernet or IP 719 payload) to PE1 and/or PE2. As described in Section 2, two EVPN PEs 720 may use tunnels with Ethernet or IP payloads to connect their IP- 721 VRFs, depending on the [IP-PREFIX] model implemented. If some 722 attributes are modified so that the route selection process 723 (Section 5) results in PE4 selecting the IPVPN path instead of the 724 EVPN path, the operator should be aware that the EVPN advanced 725 forwarding features, e.g. recursive resolution to overlay indexes, 726 will be lost for PE4. 728 o The other composite PEs (PE1 and PE2) receive also the same IP 729 prefix via EVPN and IPVPN SAFIs and they also follow the route 730 selection in Section 5. 732 o When a given route has been selected as the route for a particular 733 packet, the transmission of the packet is done according to the 734 rules for that route's AFI/SAFI. 736 o It is important to note that in composite domains, such as the one 737 in Figure 6, the EVPN advanced forwarding features will only be 738 available to composite and EVPN PEs (assuming they select an RT-5 739 to forward packets for a given IP prefix), and not to IPVPN PEs. 740 For example, assuming PE1 sends IP1/24 in an EVPN and an IPVPN 741 route and the EVPN route is the best one in the selection, the 742 recursive resolution of the EVPN RT-5s can only be used in PE2 and 743 PE4 (composite PEs), and not in PE3 (IPVPN PE). As a consequence of 744 this, the indirection provided by the RT5's recursive resolution 745 and its benefits in a scaled network, will not be available in all 746 the PEs in the network. 748 7. Gateway PE Procedures 750 Section 2 defines a gateway PE as an Interworking PE that advertises 751 IP prefixes to different BGP peers, using EVPN to one BGP peer and 752 another ISF SAFI to another BGP peer. Examples of gateway PEs are 753 Data Center gateways connecting domains that make use of EVPN and 754 other ISF SAFIs for a given tenant. Figure 7 illustrates this use- 755 case, in which PE1 and PE2 (and PE3/PE4) are gateway PEs 756 interconnecting domains for the same tenant. 758 <----EVPN----> <----------IPVPN---------> <----EVPN----> 759 6500:1:EVPN 6500:2:IPVPN 6500:3:EVPN 760 761 +-----------------------+ 762 Gateway PE1 Gateway PE3 763 +----------+ +----------+ 764 +-----------|+------+ | MPLS tnls |+------+ |-------------+ 765 | ||IP-VRF| | ||IP-VRF| | | 766 PE5 |+------+ | |+------+ | PE6 767 +------+ +----------+ +----------+ +------+ 768 |IP-VRF| NVO tnls | | | | NVO tnls |IP-VRF| 769 | | | | | | | | 770 +------+ +----------+ +----------+ +------+ 771 IP1/24--> |+------+ | |+------+ | | 772 | ||IP-VRF| | ||IP-VRF| | | 773 +-----------|+------+ | |+------+ |-------------+ 774 +----------+ +----------+ 775 Gateway PE2 +------+ Gateway PE4 776 +-------|IP-VRF|---------+ 777 | | 778 +------+ 779 PE7 781 Figure 7 Gateway PE example 783 The gateway PE procedures are described as follows: 785 o A gateway PE that imports an ISF SAFI-x route to prefix P in an IP- 786 VRF, MUST export P in ISF SAFI-y if: 788 1. P is installed in the IP-VRF (hence the SAFI-x route is the best 789 one for P) and 791 2. PE has a BGP peer for SAFI-y (enabled for the same IP-VRF) and 793 3. Either x or y is EVPN. 795 In the example of Figure 7, gateway PE1 and PE2 receive an EVPN 796 RT-5 with IP1/24, install the prefix in the IP-VRF and re- 797 advertise it using SAFI 128. 799 o ISF SAFI routes advertised by a gateway PE MUST include a D-PATH 800 attribute, so that loops can be detected in remote gateway PEs. 801 When a gateway PE propagates an IP prefix between EVPN and another 802 ISF SAFI, it MUST prepend a to the 803 received D-PATH attribute. The DOMAIN-ID and ISF_SAFI_TYPE fields 804 refer to the domain over which the gateway PE received the IP 805 prefix and the ISF SAFI of the route, respectively. If the received 806 IP prefix route did not include any D-PATH attribute, the gateway 807 IP MUST add the D-PATH when readvertising. The D-PATH in this case 808 will have only one segment on the list, the of the received route. 811 In the example of Figure 7, gateway PE1/PE2 receive the EVPN RT-5 812 with no D-PATH attribute since the route is originated at PE5. 813 Therefore PE1 and PE2 will add the D-PATH attribute including 814 = <6500:1:EVPN>. Gateways PE3/PE4 will 815 propagate the route again, now prepending their = <6500:2:IPVPN>. PE6 receives the EVPN RT-5 817 routes with D-PATH = {<6500:2:IPVPN>,<6500:1:EVPN>} and can use 818 that information to make BGP path decisions. 820 o The gateway PE MAY use the Route Distinguisher of the IP-VRF to 821 readvertise IP prefixes in EVPN or the other ISF SAFI. 823 o The label allocation used by each gateway PE is a local 824 implementation matter. The IP-VRF advertising IP prefixes for EVPN 825 and another ISF SAFI may use a label per-VRF, per-prefix, etc. 827 o The gateway PE MUST be able to use the same or different set of 828 Route Targets per ISF SAFI on the same IP-VRF. In particular, if 829 different domains use different set of Route Targets for the same 830 tenant, the gateway PE MUST be able to import and export routes 831 with the different sets. 833 o Even though Figure 7 only shows two domains per gateway PE, the 834 gateway PEs may be connected to more than two domains. 836 o There is no limitation of gateway PEs that a given IP prefix can 837 pass through until it reaches a given PE. 839 o It is worth noting that an IP prefix that was originated in an EVPN 840 domain but traversed a different ISF SAFI domain, will lose EVPN- 841 specific attributes that are used in advanced EVPN procedures. For 842 example, even if PE1 advertises IP1/24 along with a given non-zero 843 ESI (for recursive resolution to that ESI), when PE6 receives the 844 IP prefix in an EVPN route, the ESI value will be zero. This is 845 because the route traverses an ISF SAFI domain that is different 846 than EVPN. 848 8. Interworking Use-Cases 850 While Interworking PE networks may well be similar to the examples 851 described in Sections 6 and 7, in some cases a combination of both 852 functions may be required. Figure 8 illustrates an example where the 853 gateway PEs are also composite PEs, since not only they need to re- 854 advertise IP prefixes from EVPN routes to another ISF SAFI routes, 855 but they also need to interwork with IPVPN-only PEs in a domain with 856 a mix of composite and IPVPN-only PEs. 858 +-----------------------------------+ 859 | | 860 | MPLS IPVPN PE3 861 | Network +---------+ 862 | IPVPN |+------+ | 863 | +----->||IP-VRF|---TS3 864 (GW+composite) PE1 | |+------+ | 865 +---------------+ | +---------+ 866 | +------+ | EVPN v | 867 | |IP+VRF| | IPVPN +-++ | 868 | +----| | | <------> |RR| | 869 +--------| | +------+ | +--+ Composite PE4 870 | | |MAC+VRF| | ^ ^ +---------+ 871 | | +-------+ | EVPN | | EVPN |+------+ | 872 +----+ +---------------+ IPVPN | | IPVPN ||IP-VRF|---TS4 873 TS1-|NVE1| | +----+ +-------->|+------+ | 874 +----+ | v +---------+ 875 | EVPN DC | +---------------+ | 876 | NVO tnls +----| +------+ |-------------+ 877 | | |IP+VRF| | 878 | | +----| | | 879 | | | +------+ | 880 | +----+ | |MAC+VRF| | 881 +-----|NVE2|---------| +-------+ | 882 +----+ +---------------+ 883 | (GW+composite) PE2 884 TS2 886 Figure 8 Gateway and composite combined functions - example 888 In the example above, PE1 and PE2 MUST follow the procedures 889 described in Sections 6 and 7. Compared to section 7, PE1 and PE2 now 890 need to also propagate prefixes from EVPN to EVPN, in addition to 891 propagating prefixes from EVPN to IPVPN. 893 It is worth noting that PE1 and PE2 will receive TS4's IP prefix via 894 IPVPN and RT-5 routes. When readvertising to NVE1 and NVE2, PE1 and 895 PE2 will consider the D-PATH rules and attributes of the selected 896 route for TS4 (Section 5 describes the Route Selection Process). 898 9. Conclusion 900 This document describes the procedures required in PEs that use EVPN 901 and another Inter-Subnet Forwarding SAFI to import and export IP 902 prefixes for a given tenant. In particular, this document defines: 904 o A route selection algorithm so that a PE can determine what path to 905 choose between EVPN paths and other ISF SAFI paths. 907 o A new BGP Path attribute called D-PATH that provides loop 908 protection and visibility on the domains a particular route has 909 traversed. 911 o The way Path attributes should be propagated between EVPN and 912 another ISF SAFI. 914 o The procedures that must be followed on Interworking PEs that 915 behave as composite PEs, gateway PEs or a combination of both. 917 The above procedures provide an operator with the required tools to 918 build large tenant networks that may span multiple domains, use 919 different ISF SAFIs to handle IP prefixes, in a deterministic way and 920 with routing loop protection. 922 10. Conventions used in this document 924 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 925 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 926 "OPTIONAL" in this document are to be interpreted as described in BCP 927 14 [RFC2119] [RFC8174] when, and only when, they appear in all 928 capitals, as shown here. 930 11. Security Considerations 932 This section will be added in future versions. 934 12. IANA Considerations 936 This document defines a new BGP path attribute known as the BGP 937 Domain Path (D-PATH) attribute and requests IANA to assign a new 938 attribute code type from the "BGP Path Attributes" subregistry under 939 the "Border Gateway Protocol (BGP) Parameters" registry. 941 13. References 943 13.1. Normative References 945 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 946 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based Ethernet 947 VPN", RFC 7432, DOI 10.17487/RFC7432, February 2015, . 950 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 951 Border Gateway Protocol 4 (BGP-4)", RFC 4271, DOI 10.17487/RFC4271, 952 January 2006, . 954 13.2. Informative References 956 [RFC4360] Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended 957 Communities Attribute", RFC 4360, DOI 10.17487/RFC4360, February 958 2006, . 960 [IP-PREFIX] Rabadan et al., "IP Prefix Advertisement in EVPN", 961 draft-ietf-bess-evpn-prefix-advertisement-11, May, 2018. 963 [INTER-SUBNET] Sajassi et al., "IP Inter-Subnet Forwarding in EVPN", 964 draft-ietf-bess-evpn-inter-subnet-forwarding-05.txt, work in 965 progress, July, 2018 967 [ENCAP-ATT] Rosen et al., "The BGP Tunnel Encapsulation Attribute", 968 draft-ietf-idr-tunnel-encaps-10.txt, work in progress, August, 2018. 970 [RFC6472] Kumari, W. and K. Sriram, "Recommendation for Not Using 971 AS_SET and AS_CONFED_SET in BGP", BCP 172, RFC 6472, DOI 972 10.17487/RFC6472, December 2011, . 975 14. Acknowledgments 977 15. Contributors 979 16. Authors' Addresses 981 Jorge Rabadan (editor) 982 Nokia 983 777 E. Middlefield Road 984 Mountain View, CA 94043 USA 985 Email: jorge.rabadan@nokia.com 986 Ali Sajassi (editor) 987 Cisco 988 170 West Tasman Drive 989 San Jose, CA 95134, US 990 EMail: sajassi@cisco.com 992 Eric C. Rosen 993 Juniper Networks, Inc. 994 EMail: erosen@juniper.net 996 John Drake 997 Juniper Networks, Inc. 998 EMail: jdrake@juniper.net 1000 Wen Lin 1001 Juniper Networks, Inc. 1002 EMail: wlin@juniper.net 1004 Jim Uttaro 1005 AT&T 1006 Email: ju1738@att.com 1008 Adam Simpson 1009 Nokia 1010 Email: adam.1.simpson@nokia.com