idnits 2.17.1 draft-ietf-bess-evpn-ipvpn-interworking-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 3 instances of lines with control characters in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 29, 2019) is 1603 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-15) exists of draft-ietf-bess-evpn-inter-subnet-forwarding-08 Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BESS Workgroup J. Rabadan, Ed. 3 Internet Draft Nokia 4 Intended status: Standards Track A. Sajassi, Ed. 5 Cisco 7 E. Rosen 8 Individual 10 J. Drake 11 W. Lin 12 Juniper 14 J. Uttaro 15 AT&T 17 A. Simpson 18 Nokia 20 Expires: June 1, 2020 November 29, 2019 22 EVPN Interworking with IPVPN 23 draft-ietf-bess-evpn-ipvpn-interworking-02 25 Abstract 27 EVPN is used as a unified control plane for tenant network intra and 28 inter-subnet forwarding. When a tenant network spans not only EVPN 29 domains but also domains where IPVPN provides inter-subnet 30 forwarding, there is a need to specify the interworking aspects 31 between both EVPN and IPVPN domains, so that the end to end tenant 32 connectivity can be accomplished. This document specifies how EVPN 33 should interwork with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families 34 for inter-subnet forwarding. 36 Status of this Memo 38 This Internet-Draft is submitted in full conformance with the 39 provisions of BCP 78 and BCP 79. 41 Internet-Drafts are working documents of the Internet Engineering 42 Task Force (IETF), its areas, and its working groups. Note that 43 other groups may also distribute working documents as Internet- 44 Drafts. 46 Internet-Drafts are draft documents valid for a maximum of six months 47 and may be updated, replaced, or obsoleted by other documents at any 48 time. It is inappropriate to use Internet-Drafts as reference 49 material or to cite them other than as "work in progress." 51 The list of current Internet-Drafts can be accessed at 52 http://www.ietf.org/ietf/1id-abstracts.txt 54 The list of Internet-Draft Shadow Directories can be accessed at 55 http://www.ietf.org/shadow.html 57 This Internet-Draft will expire on June 1, 2020. 59 Copyright Notice 61 Copyright (c) 2019 IETF Trust and the persons identified as the 62 document authors. All rights reserved. 64 This document is subject to BCP 78 and the IETF Trust's Legal 65 Provisions Relating to IETF Documents 66 (http://trustee.ietf.org/license-info) in effect on the date of 67 publication of this document. Please review these documents 68 carefully, as they describe your rights and restrictions with respect 69 to this document. Code Components extracted from this document must 70 include Simplified BSD License text as described in Section 4.e of 71 the Trust Legal Provisions and are provided without warranty as 72 described in the Simplified BSD License. 74 Table of Contents 76 1. Introduction and Problem Statement . . . . . . . . . . . . . . 3 77 2. Terminology and Interworking PE Components . . . . . . . . . . 3 78 3. Domain Path Attribute (D-PATH) . . . . . . . . . . . . . . . . 9 79 3.1. D-PATH and Loop Prevention . . . . . . . . . . . . . . . . 11 80 4. BGP Path Attribute Propagation across ISF SAFIs . . . . . . . . 12 81 4.1. No-Propagation-Mode . . . . . . . . . . . . . . . . . . . . 12 82 4.2. Uniform-Propagation-Mode . . . . . . . . . . . . . . . . . 12 83 4.3. Aggregation of Routes and Path Attribute Propagation . . . 13 84 5. Route Selection Process between EVPN and other ISF SAFIs . . . 14 85 6. Composite PE Procedures . . . . . . . . . . . . . . . . . . . . 15 86 7. Gateway PE Procedures . . . . . . . . . . . . . . . . . . . . . 17 87 8. Interworking Use-Cases . . . . . . . . . . . . . . . . . . . . 19 88 9. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 21 89 10. Conventions used in this document . . . . . . . . . . . . . . 21 90 11. Security Considerations . . . . . . . . . . . . . . . . . . . 21 91 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 92 13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22 93 13.1. Normative References . . . . . . . . . . . . . . . . . . . 22 94 13.2. Informative References . . . . . . . . . . . . . . . . . . 22 95 14. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 23 96 15. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 23 97 16. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 23 99 1. Introduction and Problem Statement 101 EVPN is used as a unified control plane for tenant network intra and 102 inter-subnet forwarding. When a tenant network spans not only EVPN 103 domains but also domains where IPVPN provides inter-subnet 104 forwarding, there is a need to specify the interworking aspects 105 between both EVPN and IPVPN domains, so that the end to end tenant 106 connectivity can be accomplished. This document specifies how EVPN 107 should interwork with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families 108 for inter-subnet forwarding. 110 EVPN supports the advertisement of IPv4 or IPv6 prefixes in two 111 different route types: 113 o Route Type 2 - MAC/IP route (only for /32 and /128 host routes), as 114 described by [INTER-SUBNET]. 116 o Route Type 5 - IP Prefix route, as described by [IP-PREFIX]. 118 When interworking with other BGP address families (AFIs/SAFIs) for 119 inter-subnet forwarding, the IP prefixes in those two EVPN route 120 types must be propagated to other domains using different SAFIs. Some 121 aspects of that propagation must be clarified. Examples of these 122 aspects or procedures across BGP families are: route selection, loop 123 prevention or BGP Path attribute propagation. The Interworking PE 124 concepts are defined in section 2, and the rest of the document 125 describes the interaction between Interworking PEs and other PEs for 126 end-to-end inter-subnet forwarding. 128 2. Terminology and Interworking PE Components 130 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 131 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 132 "OPTIONAL" in this document are to be interpreted as described in BCP 133 14 [RFC2119] [RFC8174] when, and only when, they appear in all 134 capitals, as shown here. 136 In addition, this section summarizes the terminology related to the 137 "Interworking PE" concept that will be used throughout the rest of 138 the document. 140 +-------------------------------------------------------------+ 141 | | 142 | +------------------+ Interworking PE | 143 | Attachment | +------------------+ | 144 | Circuit(AC1) | | +----------+ | MPLS/NVO tnl 145 ----------------------*Bridge | | +------ 146 | | | |Table(BT1)| | +-----------+ / \ \ 147 MPLS/NVO tnl +-------->| *---------* |<--> | Eth | 148 -------+ | | | |Eth-Tag x + |IRB1| | \ / / 149 / Eth / \<-+ | | +----------+ | | | +------ 150 | | | | | ... | | IP-VRF1 | | 151 \ \ /<-+ | | +----------+ | | RD2/RT2 |MPLS/NVO tnl 152 -------+ | | | |Bridge | | | | +------ 153 | +-------->|Table(BT2)| |IRB2| | / \ \ 154 | | | | *---------* |<--> | IP | 155 ----------------------*Eth-Tag y | | +-----*-----+ \ / / 156 | AC2 | | +----------+ | AC3| +------ 157 | | | MAC-VRF1 | | | 158 | +-+ RD1/RT1 | | | 159 | +------------------+ | SAFIs | 160 | | 1 +---+ | 161 -------------------------------------------------+ 128 |BGP| | 162 | EVPN +---+ | 163 | | 164 +-------------------------------------------------------------+ 166 Figure 1 EVPN-IPVPN Interworking PE 168 o ISF SAFI: Inter-Subnet Forwarding (ISF) SAFI is a MP-BGP Sub- 169 Address Family that advertises reachability for IP prefixes and can 170 be used for inter-subnet forwarding within a given tenant network. 171 The ISF SAFIs are 1 (including IPv4 and IPv6 AFIs), 128 (including 172 IPv4 and IPv6 AFIs) and 70 (EVPN, including only AFI 25). 174 o ISF route: a route for a given prefix whose ISF SAFI may change as 175 it transits different domains. 177 o IP-VRF: an IP Virtual Routing and Forwarding table, as defined in 178 [RFC4364]. It is also the instantiation of an IPVPN in a PE. Route 179 Distinguisher and Route Target(s) are required properties of an IP- 180 VRF. 182 o MAC-VRF: a MAC Virtual Routing and Forwarding table, as defined in 184 [RFC7432]. It is also the instantiation of an EVI (EVPN Instance) 185 in a PE. Route Distinguisher and Route Target(s) are required 186 properties and they are normally different than the ones defined in 187 the associated IP-VRF. 189 o BT: a Bridge Table, as defined in [RFC7432]. A BT is the 190 instantiation of a Broadcast Domain in a PE. When there is a single 191 Broadcast Domain in a given EVI, the MAC-VRF in each PE will 192 contain a single BT. When there are multiple BTs within the same 193 MAC-VRF, each BT is associated to a different Ethernet Tag. The 194 EVPN routes specific to a BT, will indicate which Ethernet Tag the 195 route corresponds to. 197 Example: In Figure 1, MAC-VRF1 has two BTs: BT1 and BT2. Ethernet 198 Tag x is defined in BT1 and Ethernet Tag y in BT2. 200 o AC: Attachment Circuit or logical interface associated to a given 201 BT or IP-VRF. To determine the AC on which a packet arrived, the PE 202 will examine the combination of a physical port and VLAN tags 203 (where the VLAN tags can be individual c-tags, s-tags or ranges of 204 both). 206 Example: In Figure 1, AC1 is associated to BT1, AC2 to BT2 and AC3 207 to IP-VRF1. 209 o IRB: Integrated Routing and Bridging interface. It refers to the 210 logical interface that connects a BT to an IP-VRF and allows to 211 forward packets with destination in a different subnet. 213 o MPLS/NVO tnl: It refers to a tunnel that can be MPLS or NVO-based 214 (Network Virtualization Overlays) and it is used by MAC-VRFs and 215 IP-VRFs. Irrespective of the type, the tunnel may carry an Ethernet 216 or an IP payload. MAC-VRFs can only use tunnels with Ethernet 217 payloads (setup by EVPN), whereas IP-VRFs can use tunnels with 218 Ethernet (setup by EVPN) or IP payloads (setup by EVPN or IPVPN). 219 IPVPN-only PEs have IP-VRFs but they cannot send or receive traffic 220 on tunnels with Ethernet payloads. 222 Example: Figure 1 shows an MPLS/NVO tunnel that is used to 223 transport Ethernet frames to/from MAC-VRF1. The PE determines the 224 MAC-VRF and BT the packets belong to based on the EVPN label (MPLS 225 or VNI). Figure 1 also shows two MPLS/NVO tunnels being used by IP- 226 VRF1, one carrying Ethernet frames and the other one carrying IP 227 packets. 229 o RT-2: Route Type 2 or MAC/IP route, as per [RFC7432]. 231 o RT-5: Route Type 5 or IP Prefix route, as per [IP-PREFIX]. 233 o Domain: Two PEs are in the same domain if they are attached to the 234 same tenant and the packets between them do not require a data path 235 IP lookup (in the tenant space) in any intermediate router. A 236 gateway PE is always configured with multiple Domain-IDs. 238 Example 1: Figure 2 depicts an example where TS1 and TS2 belong to 239 the same tenant, and they are located in different Data Centers 240 that are connected by gateway PEs (see the gateway PE definition 241 later). These gateway PEs use IPVPN in the WAN. When TS1 sends 242 traffic to TS2, the intermediate routers between PE1 and PE2 243 require a tenant IP lookup in their IP-VRFs so that the packets can 244 be forwarded. In this example there are three different domains. 245 The gateway PEs connect the EVPN domains to the IPVPN domain. 247 GW1------------GW3 248 +------+ +------+ 249 +-------------|IP-VRF| |IP-VRF|-------------+ 250 PE1 +------+ +------+ PE2 251 +------+ DC1 | WAN | DC2 +------+ 252 TS1-|IP-VRF| EVPN | IPVPN | EVPN |IP-VRF|-TS2 253 +------+ GW2 GW4 +---+--+ 254 | +------+ +------+ | 255 +-------------|IP-VRF| |IP-VRF|-------------+ 256 +------+ +------+ 257 +--------------+ 258 DOMAIN 1 DOMAIN 2 DOMAIN 3 259 <---------------> <------------> <----------------> 261 Figure 2 Multiple domain DCI example 263 Example 2: Figure 3 illustrates a similar example, but PE1 and PE2 264 are now connected by a BGP-LU (BGP Labeled Unicast) tunnel, and 265 they have a BGP peer relationship for EVPN. Contrary to Example 1, 266 there is no need for tenant IP lookups on the intermediate routers 267 in order to forward packets between PE1 and PE2. Therefore, there 268 is only one domain in the network and PE1/PE2 belong to it. 270 EVPN 271 <-------------------------------------------------> 272 BGP-LU 273 <-------------------------------------------------> 275 ASBR------------ASBR 276 +------+ +------+ 277 +-------------| | | |-------------+ 278 PE1 +------+ +--+---+ PE2 279 +------+ DC1 | WAN | DC2 +------+ 280 TS1-|IP-VRF| EVPN | | EVPN |IP-VRF|-TS2 281 +------+ ASBR ASBR +---+--+ 282 | +------+ +------+ | 283 +-------------| | | |-------------+ 284 +------+ +------+ 285 +--------------+ 287 <--------------------DOMAIN-1---------------------> 289 Figure 3 Single domain DCI example 291 o Regular Domain: a domain in which a single control plane, IPVPN or 292 EVPN, is used and which is composed of regular PEs, see below. In 293 Figures 2 and 3, above, all domains are regular domains. 295 o Composite Domain: a domain in which multiple control planes, IPVPN 296 and EVPN, are used and which is composed of regular PEs, see below, 297 and composite PEs, see below. 299 o Regular PE: a PE that is attached to a domain, either regular or 300 composite, and which uses one of the control plane protocols (IPVPN 301 or EVPN) operating in the domain. 303 o Interworking PE: a PE that may advertise a given prefix with an 304 EVPN ISF route (RT-2 or RT-5) and/or an IPVPN ISF route. An 305 interworking PE has one IP-VRF per tenant, and one or multiple MAC- 306 VRFs per tenant. Each MAC-VRF may contain one or more BTs, where 307 each BT may be attached to that IP-VRF via IRB. There are two types 308 of Interworking PEs: composite PEs and gateway PEs. Both PE 309 functions can be independently implemented per tenant and they may 310 both be implemented for the same tenant. 312 Example: Figure 1 shows an interworking PE of type gateway, where 313 ISF SAFIs 1, 128 and 70 are enabled. IP-VRF1 and MAC-VRF1 are 314 instantiated on the PE, and together provide inter-subnet 315 forwarding for the tenant. 317 o Composite PE: an interworking PE that is attached to a composite 318 domain and which advertises a given prefix to an IPVPN peer with an 319 IPVPN ISF route, to an EVPN peer with an EVPN ISF route, and to a 320 route reflector with both an IPVPN and EVPN ISF route. A composite 321 PE performs the procedures of Sections 5 and 6. 323 Example: Figure 4 shows an example where PE1 is a composite PE 324 since PE1 has EVPN and another ISF SAFI enabled to the same route- 325 reflector, and PE1 advertises a given IP prefix IPn/x twice, one 326 using EVPN and another one using ISF SAFI 128. PE2 and PE3 are not 327 composite PEs. 329 +---+ 330 |PE2| 331 +---+ 332 ^ 333 |EVPN 334 IW EVPN v 335 +---+ IPVPN ++-+ +---+ 336 |PE1| <----> |RR| <---> |PE3| 337 +---+ +--+ IPVPN +---+ 338 Composite 340 Figure 4 Interworking composite PE example 342 o Gateway PE: an interworking PE that is attached to two domains, 343 each either regular or composite, and which, based on 344 configuration, does one of the following: 346 - Propagates the same control plane protocol, either IPVPN or EVPN, 347 between the two domains. 349 - Propagates an ISF route with different ISF SAFIs between the two 350 domains. E.g., propagate an EVPN ISF route in one domain as an 351 IPVPN ISF route in the other domain and vice versa. A gateway PE 352 performs the procedures of Sections 3, 4, 5 and 7. 354 A gateway PE is always configured with multiple Domain-IDs. The 355 Domain-ID is encoded in the Domain Path Attribute (D-PATH), and 356 advertised along with EVPN and other ISF SAFI routes. Section 3 357 describes the D-PATH attribute. 359 Example: Figure 5 illustrates an example where PE1 is a gateway 360 PE since the EVPN and IPVPN SAFIs are enabled on different BGP 361 peers, and a given local IP prefix IPn/x is sent to both BGP 362 peers for the same tenant. PE2 and PE1 are in one domain and PE3 363 and PE1 are in another domain. 365 IW 366 +---+ EVPN +---+ IPVPN +---+ 367 |PE2| <----> |PE1| <----> |PE3| 368 +---+ +---+ +---+ 369 Gateway 371 Figure 5 Interworking gateway PE example 373 o Composite/Gateway PE: an interworking PE that is both a composite 374 PE and a gateway PE that is attached to two domains, one regular 375 and one composite, and which does the following: 377 - Propagates an ISF route, either IPVPN or EVPN, from the regular 378 domain into the composite domain. Within the composite domain it 379 acts as a composite PE. 381 - Propagates an ISF route, either IPVPN or EVPN, from the composite 382 domain into the regular domain. Within the regular domain it is 383 propagated as an ISF route using the ISF SAFI for that domain. 385 This is particularly useful when a tenant network is attached to 386 both IPVPN and EVPN domains, any-to-any connectivity is required, 387 and end-to-end control plane consistency, when possible, is 388 desired. 390 It would be instantiated by attaching the disparate, regular 391 IPVPN and EVPN domains via these PEs to a central composite 392 domain. 394 3. Domain Path Attribute (D-PATH) 396 The BGP Domain Path (D-PATH) attribute is an optional and transitive 397 BGP path attribute. 399 Similar to AS_PATH, D-PATH is composed of a sequence of Domain 400 segments. Each Domain segment is comprised of , where the domain segment value is a sequence 402 of one or more Domains. Each domain is represented by . 405 o The domain segment length field is a 1-octet field, containing the 406 number of domains in the segment. 408 o DOMAIN-ID is a 6-octet field that represents a domain. It is 409 composed of a 4-octet Global Administrator sub-field and a 2-octet 410 Local Administrator sub-field. The Global Administrator sub-field 411 MAY be filled with an Autonomous System Number (ASN), an IPv4 412 address, or any value that guarantees the uniqueness of the DOMAIN- 413 ID when the tenant network is connected to multiple Operators. 415 o ISF_SAFI_TYPE is a 1-octet field that indicates the Inter-Subnet 416 Forwarding SAFI type in which a route was advertised in the DOMAIN. 417 The following types are valid in this document: 419 Value Type 421 1 SAFI 1 422 70 EVPN 423 128 SAFI 128 425 About the BGP D-PATH attribute: 427 a) Identifies the sequence of domains, each identified by a through which a given ISF route has passed. 430 - This attribute list may contain zero, one or more segments. 432 - The first entry in the list (leftmost) is the from which a gateway PE is propagating an ISF 434 route. The last entry in the list (rightmost) is the from which a gateway PE received an ISF route 436 without a D-PATH attribute. Intermediate entries in the list are 437 domains that the ISF route has transited. 439 - As an example, an ISF route received with a D-PATH attribute 440 containing a domain segment of {length=2, 441 <6500:2:IPVPN>,<6500:1:EVPN>} indicates that the ISF route was 442 originated in EVPN domain 6500:1, and propagated into IPVPN 443 domain 6500:2. 445 b) It is added/modified by a gateway PE when propagating an update to 446 a different domain: 448 - A gateway PE's IP-VRF, that connects two domains, belongs to two 449 DOMAIN-IDs, e.g. 6500:1 for EVPN and 6500:2 for IPVPN. 451 - Whenever a prefix arrives at a gateway PE in a particular ISF 452 SAFI route, if the gateway PE needs to export that prefix to a 453 BGP peer, the gateway PE will prepend a to the list of domains in the received D-PATH. 456 - For instance, in an IP-VRF configured with DOMAIN-IDs 6500:1 for 457 EVPN and 6500:2 for IPVPN, if an EVPN route for prefix P is 458 received and P installed in the IP-VRF, the IPVPN route for P 459 that is exported to an IPVPN peer will prepend the domain 460 <6500:1:EVPN> to the previously received D-PATH attribute. 461 Likewise, IP-VRF prefixes that are received from IP-VPN, will be 462 exported to EVPN peers with the domain <6500:2:IPVPN> added to 463 the segment. 465 - In the above example, if the EVPN route is received without D- 466 PATH, the gateway PE will add the D-PATH attribute with one 467 segment {length=1, <6500:1:EVPN>} when re-advertising to domain 468 6500:2. 470 - Within the originating domain, the update does not contain a D- 471 PATH attribute because the update has not passed through a 472 gateway PE yet. 474 c) The gateway PE MUST NOT add the D-PATH attribute to ISF routes 475 generated for IP-VRF prefixes that are not learned via any ISF 476 SAFI, for instance, local prefixes. 478 d) An ISF route received by a gateway PE with a D-PATH attribute that 479 contains one or more of its locally configured domains for the IP- 480 VRF is considered to be a looped ISF route and MUST be dropped. 482 e) The number of domains in the D-PATH attribute indicates the number 483 of gateway PEs that the ISF route update has transited. 485 3.1. D-PATH and Loop Prevention 487 The D-PATH attribute is used to prevent loops in interworking PE 488 networks. For instance, in the example of Figure 4, gateway GW1 489 receives TS1 prefix in two different ISF routes: 491 o In an EVPN RT-5 with next-hop PE1 and no D-PATH attribute. 493 o In a SAFI 128 route with next-hop GW2 and D-PATH = {length=1, 494 <6500:1:EVPN>}, assuming that DOMAIN-ID for domain 1 is 6500:1. 496 Gateway GW1 flags the SAFI 128 route as a loop, and does not re- 497 advertise it to the EVPN neighbors since the route includes the GW1's 498 local domain. 500 In general, any interworking PE that imports an ISF route MUST flag 501 the route as "looped" if its D-PATH contains a segment, where DOMAIN-ID matches a local DOMAIN-ID 503 in the tenant IP-VRF. 505 4. BGP Path Attribute Propagation across ISF SAFIs 507 Based on configurations a gateway PE is required to propagate an ISF 508 route with different ISF SAFIs between two domains. This requires a 509 definition of what a gateway PE has to do with Path attributes 510 attached to the ISF route that it is propagating. 512 4.1. No-Propagation-Mode 514 This is the default mode of operation. In this mode, the gateway PE 515 will simply re-initialize the Path Attributes when propagating an ISF 516 route, as though it would for direct or local IP prefixes. This model 517 may be enough in those use-cases where the EVPN domain is considered 518 an "abstracted" CE and remote IPVPN/IP PEs don't need to consider the 519 original EVPN Attributes for path calculations. 521 Since this mode of operation does not propagate the D-PATH attribute 522 either, redundant gateway PEs are exposed to routing loops. Those 523 loops may be resolved by policies and the use of other attributes, 524 such as the Route Origin extended community [RFC4360], however not 525 all the loop situations may be solved. 527 4.2. Uniform-Propagation-Mode 529 In this mode, the gateway PE simply keeps accumulating or mapping 530 certain key commonly used Path Attributes when propagating an ISF 531 route. This mode is typically used in networks where EVPN and IPVPN 532 SAFIs are used seamlessly to distribute IP prefixes. 534 The following rules MUST be observed by the gateway PE when 535 propagating Path Attributes: 537 o The gateway PE imports an ISF route in the IP-VRF and stores the 538 original Path Attributes. The following set of Path Attributes 539 SHOULD be propagated by the gateway PE to other ISF SAFIs (other 540 Path Attributes SHOULD NOT be propagated): 542 - AS_PATH 543 - D-PATH 544 - IBGP-only Path Attributes: LOCAL_PREF, ORIGINATOR_ID, CLUSTER_ID 545 - MED 546 - AIGP 547 - Communities, (non-EVPN) Extended Communities and Large 548 Communities 550 o When propagating an ISF route to a different ISF SAFI and IBGP 551 peer, the gateway PE SHOULD copy the AS_PATH of the originating 552 family and add it to the destination family without any 553 modification. When re-advertising to a different ISF SAFI and EBGP 554 peer, the gateway PE SHOULD copy the AS_PATH of the originating 555 family and prepend the IP-VRF's AS before sending the route. 557 o When propagating an ISF route to IBGP peers, the gateway PE SHOULD 558 copy the IBGP-only Path Attributes from the originating SAFI to the 559 re-advertised route. 561 o Communities, non-EVPN Extended Communities and Large Communities 562 SHOULD be copied by the gateway PE from the originating SAFI route. 564 4.3. Aggregation of Routes and Path Attribute Propagation 566 Instead of propagating a high number of (host) ISF routes between ISF 567 SAFIs, a gateway PE that receives multiple ISF routes of one ISF SAFI 568 MAY choose to propagate a single ISF aggregate route with a different 569 ISF SAFI. In this document, aggregation is used to combine the 570 characteristics of multiple ISF routes of the same ISF SAFI in such 571 way that a single aggregate ISF route of a different ISF SAFI can be 572 propagated. Aggregation of multiple ISF routes of one ISF SAFI into 573 an aggregate ISF route of a different ISF SAFI is only done by a 574 gateway PE. 576 Aggregation on gateway PEs may use either the No-Propagation-Mode or 577 the Uniform-Propagation-Mode explained in Sections 4.1. and 4.2, 578 respectively. 580 When using Uniform-Propagation-Mode, Path Attributes of the same type 581 code MAY be aggregated according to the following rules: 583 o AS_PATH is aggregated based on the rules in [RFC4271]. The gateway 584 PEs SHOULD NOT receive AS_PATH attributes with path segments of 585 type AS_SET [RFC6472]. Routes received with AS_PATH attributes 586 including AS_SET path segments MUST NOT be aggregated. 588 o ISF routes that have different attributes of the following type 589 codes MUST NOT be aggregated: D-PATH, LOCAL_PREF, ORIGINATOR_ID, 590 CLUSTER_ID, MED or AIGP. 592 o The Community, Extended Community and Large Community attributes of 593 the aggregate ISF route MUST contain all the Communities/Extended 594 Communities/Large Communities from all of the aggregated ISF 595 routes. 597 Assuming the aggregation can be performed (the above rules are 598 applied), the operator should consider aggregation to deal with 599 scaled tenant networks where a significant number of host routes 600 exists. For a example, large Data Centers. 602 5. Route Selection Process between EVPN and other ISF SAFIs 604 A PE may receive an IP prefix in ISF routes with different ISF SAFIs, 605 from the same or different BGP peer. It may also receive the same IP 606 prefix (host route) in an EVPN RT-2 and RT-5. A route selection 607 algorithm across all ISF SAFIs is needed so that: 609 o Different gateway and composite PEs have a consistent and 610 deterministic view on how to reach a given prefix. 612 o Prefixes advertised in EVPN and other ISF SAFIs can be compared 613 based on path attributes commonly used by operators across 614 networks. 616 o Equal Cost Multi-Path (ECMP) is allowed across EVPN and other ISF 617 SAFI routes. 619 For a given prefix advertised in one or more non-EVPN ISF routes, the 620 BGP best path selection procedure will produce a set of "non-EVPN 621 best paths". For a given prefix advertised in one or more EVPN ISF 622 routes, the BGP best path selection procedure will produce a set of 623 "EVPN best paths". To support IP/EVPN interworking, it is then 624 necessary to run a tie-breaking selection algorithm on the union of 625 these two sets. This tie-breaking algorithm begins by considering all 626 EVPN and other ISF SAFI routes, equally preferable routes to the same 627 destination, and then selects routes to be removed from 628 consideration. The process terminates as soon as only one route 629 remains in consideration. 631 The route selection algorithm must remove from consideration the 632 routes following the rules and the order defined in [RFC4271], with 633 the following exceptions and in the following order: 635 1- Immediately after removing from consideration all routes that are 636 not tied for having the highest Local Preference, any routes that 637 do not have the shortest D-PATH are also removed from 638 consideration. Routes with no D-PATH are considered to have a 639 zero-length D-PATH. 641 2- Then regular [RFC4271] selection criteria is followed. 643 3- At the end of the selection algorithm, if at least one route still 644 under consideration is an RT-2 route, remove from consideration 645 any RT-5 routes. 647 4- Steps 1-3 could possibly leave Equal Cost Multi-Path (ECMP) 648 between IP and EVPN paths. By default, the EVPN path is considered 649 (and the IP path removed from consideration). However, if ECMP 650 across ISF SAFIs is enabled by policy, and an "IP path" and an 651 "EVPN path" remain at the end of step 3, both path types will be 652 used. 654 Example 1 - PE1 receives the following routes for IP1/32, that are 655 candidate to be imported in IP-VRF-1: 657 {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=(100,200)} 658 {SAFI=EVPN, RT-5, Local-Pref=100, AS-Path=(100,200)} 659 {SAFI=128, Local-Pref=100, AS-Path=(100,200)} 661 Selected route: {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=100,200] 662 (due to step 3, and no ECMP) 664 Example 2 - PE1 receives the following routes for IP2/24, that are 665 candidate to be imported in IP-VRF-1: 667 {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS-Path=(100,200), 668 MED=10} 669 {SAFI=128, D-PATH=(6500:1:EVPN,6500:2:IPVPN), AS-Path=(200), 670 MED=200} 672 Selected route: {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS- 673 Path=(100,200), MED=10} (due to step 1) 675 6. Composite PE Procedures 677 As described in Section 2, composite PEs are typically used in tenant 678 networks where EVPN and IPVPN are both used to provide inter-subnet 679 forwarding within the same composite domain. 681 Figure 6 depicts an example of a composite domain, where PE1/PE2/PE4 682 are composite PEs (they support EVPN and IPVPN ISF SAFIs on their 683 peering to the Route Reflector), and PE3 is a regular IPVPN PE. 685 +-----------------------------------+ 686 | | 687 | MPLS IPVPN PE3 688 | Network +----------+ IP3/24 689 | IPVPN |+------+ | +---+ 690 | +----->||IP-VRF|------|CE3| 691 Composite PE1 | |+------+ | +---+ 692 +---------------+ | +----------+ 693 | +------+ | EVPN v | 694 | |IP-VRF| | IPVPN +--+ | 695 | +----| | | <------> |RR| | 696 +---+ | | +------+ | +--+ Composite PE4 697 |CE2|----|MAC-VRF| | ^ ^ +---------+ IP4/24 698 +---+ | +-------+ | EVPN | | EVPN |+------+ | +---+ 699 +---|-----------+ IPVPN | | IPVPN ||IP-VRF|-----|CE4| 700 | | +----+ +-------->|+------+ | +---+ 701 IP1/24 | | v +---------+ 702 +---+ | | +---------------+ | 703 |CE1|--+ +----| +------+ +--------------+ 704 +---+ | |IP-VRF| | 705 | | +----| | | 706 | | | +------+ | 707 +--------------|MAC-VRF| | 708 | +-------+ | 709 +---------------+ 710 Composite PE2 712 Figure 6 Composite PE example 714 In a composite domain with composite and regular PEs: 716 o The composite PEs advertise the same IP prefixes in each ISF SAFI 717 to the RR. For example, in Figure 6, the prefix IP1/24 is 718 advertised by PE1 and PE2 to the RR in two separate NLRIs, one for 719 AFI/SAFI 1/128 and another one for EVPN. 721 o The RR does not forward EVPN routes to PE3 (since the RR does not 722 have the EVPN SAFI enabled on its BGP session to PE3), whereas the 723 IPVPN routes are forwarded to all the PEs. 725 o PE3 receives only the IPVPN route for IP1/24 and resolves the BGP 726 next-hop to an MPLS tunnel (with IP payload) to PE1 and/or PE2. 728 o Composite PE4 receives IP1/24 encoded in EVPN and another ISF SAFI 729 route (EVPN RT-5 and IPVPN). The route selection follows the 730 procedures in Section 5. Assuming an EVPN route is selected, PE4 731 resolves the BGP next-hop to an MPLS tunnel (with Ethernet or IP 732 payload) to PE1 and/or PE2. As described in Section 2, two EVPN PEs 733 may use tunnels with Ethernet or IP payloads to connect their IP- 734 VRFs, depending on the [IP-PREFIX] model implemented. If some 735 attributes are modified so that the route selection process 736 (Section 5) results in PE4 selecting the IPVPN path instead of the 737 EVPN path, the operator should be aware that the EVPN advanced 738 forwarding features, e.g. recursive resolution to overlay indexes, 739 will be lost for PE4. 741 o The other composite PEs (PE1 and PE2) receive also the same IP 742 prefix via EVPN and IPVPN SAFIs and they also follow the route 743 selection in Section 5. 745 o When a given route has been selected as the route for a particular 746 packet, the transmission of the packet is done according to the 747 rules for that route's AFI/SAFI. 749 o It is important to note that in composite domains, such as the one 750 in Figure 6, the EVPN advanced forwarding features will only be 751 available to composite and EVPN PEs (assuming they select an RT-5 752 to forward packets for a given IP prefix), and not to IPVPN PEs. 753 For example, assuming PE1 sends IP1/24 in an EVPN and an IPVPN 754 route and the EVPN route is the best one in the selection, the 755 recursive resolution of the EVPN RT-5s can only be used in PE2 and 756 PE4 (composite PEs), and not in PE3 (IPVPN PE). As a consequence of 757 this, the indirection provided by the RT5's recursive resolution 758 and its benefits in a scaled network, will not be available in all 759 the PEs in the network. 761 7. Gateway PE Procedures 763 Section 2 defines a gateway PE as an Interworking PE that advertises 764 IP prefixes to different BGP peers, using EVPN to one BGP peer and 765 another ISF SAFI to another BGP peer. Examples of gateway PEs are 766 Data Center gateways connecting domains that make use of EVPN and 767 other ISF SAFIs for a given tenant. Figure 7 illustrates this use- 768 case, in which PE1 and PE2 (and PE3/PE4) are gateway PEs 769 interconnecting domains for the same tenant. 771 <----EVPN----> <----------IPVPN---------> <----EVPN----> 772 6500:1:EVPN 6500:2:IPVPN 6500:3:EVPN 773 774 +-----------------------+ 775 Gateway PE1 Gateway PE3 776 +----------+ +----------+ 777 +-----------|+------+ | MPLS tnls |+------+ |-------------+ 778 | ||IP-VRF| | ||IP-VRF| | | 779 PE5 |+------+ | |+------+ | PE6 780 +------+ +----------+ +----------+ +------+ 781 |IP-VRF| NVO tnls | | | | NVO tnls |IP-VRF| 782 | | | | | | | | 783 +------+ +----------+ +----------+ +------+ 784 IP1/24--> |+------+ | |+------+ | | 785 | ||IP-VRF| | ||IP-VRF| | | 786 +-----------|+------+ | |+------+ |-------------+ 787 +----------+ +----------+ 788 Gateway PE2 +------+ Gateway PE4 789 +-------|IP-VRF|---------+ 790 | | 791 +------+ 792 PE7 794 Figure 7 Gateway PE example 796 The gateway PE procedures are described as follows: 798 o A gateway PE that imports an ISF SAFI-x route to prefix P in an IP- 799 VRF, MUST export P in ISF SAFI-y if: 801 1. P is installed in the IP-VRF (hence the SAFI-x route is the best 802 one for P) and 804 2. PE has a BGP peer for SAFI-y (enabled for the same IP-VRF) and 806 3. Either x or y is EVPN. 808 In the example of Figure 7, gateway PE1 and PE2 receive an EVPN 809 RT-5 with IP1/24, install the prefix in the IP-VRF and re- 810 advertise it using SAFI 128. 812 o ISF SAFI routes advertised by a gateway PE MUST include a D-PATH 813 attribute, so that loops can be detected in remote gateway PEs. 814 When a gateway PE propagates an IP prefix between EVPN and another 815 ISF SAFI, it MUST prepend a to the 816 received D-PATH attribute. The DOMAIN-ID and ISF_SAFI_TYPE fields 817 refer to the domain over which the gateway PE received the IP 818 prefix and the ISF SAFI of the route, respectively. If the received 819 IP prefix route did not include any D-PATH attribute, the gateway 820 IP MUST add the D-PATH when readvertising. The D-PATH in this case 821 will have only one segment on the list, the of the received route. 824 In the example of Figure 7, gateway PE1/PE2 receive the EVPN RT-5 825 with no D-PATH attribute since the route is originated at PE5. 826 Therefore PE1 and PE2 will add the D-PATH attribute including 827 = <6500:1:EVPN>. Gateways PE3/PE4 will 828 propagate the route again, now prepending their = <6500:2:IPVPN>. PE6 receives the EVPN RT-5 830 routes with D-PATH = {<6500:2:IPVPN>,<6500:1:EVPN>} and can use 831 that information to make BGP path decisions. 833 o The gateway PE MAY use the Route Distinguisher of the IP-VRF to 834 readvertise IP prefixes in EVPN or the other ISF SAFI. 836 o The label allocation used by each gateway PE is a local 837 implementation matter. The IP-VRF advertising IP prefixes for EVPN 838 and another ISF SAFI may use a label per-VRF, per-prefix, etc. 840 o The gateway PE MUST be able to use the same or different set of 841 Route Targets per ISF SAFI on the same IP-VRF. In particular, if 842 different domains use different set of Route Targets for the same 843 tenant, the gateway PE MUST be able to import and export routes 844 with the different sets. 846 o Even though Figure 7 only shows two domains per gateway PE, the 847 gateway PEs may be connected to more than two domains. 849 o There is no limitation of gateway PEs that a given IP prefix can 850 pass through until it reaches a given PE. 852 o It is worth noting that an IP prefix that was originated in an EVPN 853 domain but traversed a different ISF SAFI domain, will lose EVPN- 854 specific attributes that are used in advanced EVPN procedures. For 855 example, even if PE1 advertises IP1/24 along with a given non-zero 856 ESI (for recursive resolution to that ESI), when PE6 receives the 857 IP prefix in an EVPN route, the ESI value will be zero. This is 858 because the route traverses an ISF SAFI domain that is different 859 than EVPN. 861 8. Interworking Use-Cases 863 While Interworking PE networks may well be similar to the examples 864 described in Sections 6 and 7, in some cases a combination of both 865 functions may be required. Figure 8 illustrates an example where the 866 gateway PEs are also composite PEs, since not only they need to re- 867 advertise IP prefixes from EVPN routes to another ISF SAFI routes, 868 but they also need to interwork with IPVPN-only PEs in a domain with 869 a mix of composite and IPVPN-only PEs. 871 +-----------------------------------+ 872 | | 873 | MPLS IPVPN PE3 874 | Network +---------+ 875 | IPVPN |+------+ | 876 | +----->||IP-VRF|---TS3 877 (GW+composite) PE1 | |+------+ | 878 +---------------+ | +---------+ 879 | +------+ | EVPN v | 880 | |IP+VRF| | IPVPN +-++ | 881 | +----| | | <------> |RR| | 882 +--------| | +------+ | +--+ Composite PE4 883 | | |MAC+VRF| | ^ ^ +---------+ 884 | | +-------+ | EVPN | | EVPN |+------+ | 885 +----+ +---------------+ IPVPN | | IPVPN ||IP-VRF|---TS4 886 TS1-|NVE1| | +----+ +-------->|+------+ | 887 +----+ | v +---------+ 888 | EVPN DC | +---------------+ | 889 | NVO tnls +----| +------+ |-------------+ 890 | | |IP+VRF| | 891 | | +----| | | 892 | | | +------+ | 893 | +----+ | |MAC+VRF| | 894 +-----|NVE2|---------| +-------+ | 895 +----+ +---------------+ 896 | (GW+composite) PE2 897 TS2 899 Figure 8 Gateway and composite combined functions - example 901 In the example above, PE1 and PE2 MUST follow the procedures 902 described in Sections 6 and 7. Compared to section 7, PE1 and PE2 now 903 need to also propagate prefixes from EVPN to EVPN, in addition to 904 propagating prefixes from EVPN to IPVPN. 906 It is worth noting that PE1 and PE2 will receive TS4's IP prefix via 907 IPVPN and RT-5 routes. When readvertising to NVE1 and NVE2, PE1 and 908 PE2 will consider the D-PATH rules and attributes of the selected 909 route for TS4 (Section 5 describes the Route Selection Process). 911 9. Conclusion 913 This document describes the procedures required in PEs that use EVPN 914 and another Inter-Subnet Forwarding SAFI to import and export IP 915 prefixes for a given tenant. In particular, this document defines: 917 o A route selection algorithm so that a PE can determine what path to 918 choose between EVPN paths and other ISF SAFI paths. 920 o A new BGP Path attribute called D-PATH that provides loop 921 protection and visibility on the domains a particular route has 922 traversed. 924 o The way Path attributes should be propagated between EVPN and 925 another ISF SAFI. 927 o The procedures that must be followed on Interworking PEs that 928 behave as composite PEs, gateway PEs or a combination of both. 930 The above procedures provide an operator with the required tools to 931 build large tenant networks that may span multiple domains, use 932 different ISF SAFIs to handle IP prefixes, in a deterministic way and 933 with routing loop protection. 935 10. Conventions used in this document 937 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 938 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 939 "OPTIONAL" in this document are to be interpreted as described in BCP 940 14 [RFC2119] [RFC8174] when, and only when, they appear in all 941 capitals, as shown here. 943 11. Security Considerations 945 This section will be added in future versions. 947 12. IANA Considerations 949 This document defines a new BGP path attribute known as the BGP 950 Domain Path (D-PATH) attribute. 952 IANA has assigned a new attribute code type from the "BGP Path 953 Attributes" subregistry under the "Border Gateway Protocol (BGP) 954 Parameters" registry: 956 Path Attribute Value Code Reference 957 -------------------- ------------------------ --------------- 958 36 BGP Domain Path (D-PATH) [This document] 960 13. References 962 13.1. Normative References 964 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 965 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based Ethernet 966 VPN", RFC 7432, DOI 10.17487/RFC7432, February 2015, . 969 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 970 Border Gateway Protocol 4 (BGP-4)", RFC 4271, DOI 10.17487/RFC4271, 971 January 2006, . 973 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 974 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 2006, 975 . 977 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 978 Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 979 1997, . 981 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 982 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, 983 . 985 13.2. Informative References 987 [RFC4360] Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended 988 Communities Attribute", RFC 4360, DOI 10.17487/RFC4360, February 989 2006, . 991 [IP-PREFIX] Rabadan et al., "IP Prefix Advertisement in EVPN", 992 draft-ietf-bess-evpn-prefix-advertisement-11, May, 2018. 994 [INTER-SUBNET] Sajassi et al., "IP Inter-Subnet Forwarding in EVPN", 995 draft-ietf-bess-evpn-inter-subnet-forwarding-08.txt, work in 996 progress, March, 2019. 998 [RFC6472] Kumari, W. and K. Sriram, "Recommendation for Not Using 999 AS_SET and AS_CONFED_SET in BGP", BCP 172, RFC 6472, DOI 1000 10.17487/RFC6472, December 2011, . 1003 14. Acknowledgments 1005 The authors want to thank Russell Kelly for his review of the D-PATH 1006 section and suggestions. 1008 15. Contributors 1010 16. Authors' Addresses 1012 Jorge Rabadan (editor) 1013 Nokia 1014 777 E. Middlefield Road 1015 Mountain View, CA 94043 USA 1016 Email: jorge.rabadan@nokia.com 1018 Ali Sajassi (editor) 1019 Cisco 1020 170 West Tasman Drive 1021 San Jose, CA 95134, US 1022 EMail: sajassi@cisco.com 1024 Eric C. Rosen 1025 EMail: erosen52@gmail.com 1027 John Drake 1028 Juniper Networks, Inc. 1029 EMail: jdrake@juniper.net 1031 Wen Lin 1032 Juniper Networks, Inc. 1033 EMail: wlin@juniper.net 1035 Jim Uttaro 1036 AT&T 1037 Email: ju1738@att.com 1039 Adam Simpson 1040 Nokia 1041 Email: adam.1.simpson@nokia.com