idnits 2.17.1 draft-ietf-bess-evpn-virtual-eth-segment-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 9, 2020) is 1502 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'ETH-OAM' is mentioned on line 611, but not defined == Missing Reference: 'MPLS-OAM' is mentioned on line 617, but not defined == Missing Reference: 'PW-OAM' is mentioned on line 617, but not defined == Missing Reference: 'EVPN-IRB' is mentioned on line 715, but not defined == Outdated reference: A later version (-09) exists of draft-ietf-bess-pbb-evpn-isid-cmacflush-00 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BESS WorkGroup A. Sajassi 3 Internet-Draft P. Brissette 4 Intended status: Standards Track Cisco Systems 5 Expires: September 10, 2020 R. Schell 6 Verizon 7 J. Drake 8 Juniper 9 J. Rabadan 10 Nokia 11 March 9, 2020 13 EVPN Virtual Ethernet Segment 14 draft-ietf-bess-evpn-virtual-eth-segment-06 16 Abstract 18 EVPN and PBB-EVPN introduce a family of solutions for multipoint 19 Ethernet services over MPLS/IP network with many advanced features 20 among which their multi-homing capabilities. These solutions 21 introduce Single-Active and All-Active for an Ethernet Segment (ES), 22 itself defined as a set of physical links between the multi-homed 23 device/network and a set of PE devices that they are connected to. 24 This document extends the Ethernet Segment concept so that an ES can 25 be associated to a set of EVCs (e.g., VLANs) or other objects such as 26 MPLS Label Switch Paths (LSPs) or Pseudowires (PWs), referred to as 27 Virtual Ethernet Segments (vES). This draft describes the 28 requirements and the extensions needed to support vES in EVPN and 29 PBB-EVPN. 31 Requirements Language 33 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 34 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 35 document are to be interpreted as described in RFC 2119 [RFC2119] and 36 RFC 8174 [RFC8174]. 38 Status of This Memo 40 This Internet-Draft is submitted in full conformance with the 41 provisions of BCP 78 and BCP 79. 43 Internet-Drafts are working documents of the Internet Engineering 44 Task Force (IETF). Note that other groups may also distribute 45 working documents as Internet-Drafts. The list of current Internet- 46 Drafts is at https://datatracker.ietf.org/drafts/current/. 48 Internet-Drafts are draft documents valid for a maximum of six months 49 and may be updated, replaced, or obsoleted by other documents at any 50 time. It is inappropriate to use Internet-Drafts as reference 51 material or to cite them other than as "work in progress." 53 This Internet-Draft will expire on September 10, 2020. 55 Copyright Notice 57 Copyright (c) 2020 IETF Trust and the persons identified as the 58 document authors. All rights reserved. 60 This document is subject to BCP 78 and the IETF Trust's Legal 61 Provisions Relating to IETF Documents 62 (https://trustee.ietf.org/license-info) in effect on the date of 63 publication of this document. Please review these documents 64 carefully, as they describe your rights and restrictions with respect 65 to this document. Code Components extracted from this document must 66 include Simplified BSD License text as described in Section 4.e of 67 the Trust Legal Provisions and are provided without warranty as 68 described in the Simplified BSD License. 70 Table of Contents 72 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 73 1.1. Virtual Ethernet Segments in Access Ethernet Networks . . 3 74 1.2. Virtual Ethernet Segments in Access MPLS Networks . . . . 5 75 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 76 3. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 7 77 3.1. Single-Homed and Multi-Homed vES . . . . . . . . . . . . 7 78 3.2. Scalability . . . . . . . . . . . . . . . . . . . . . . . 8 79 3.3. Local Switching . . . . . . . . . . . . . . . . . . . . . 8 80 3.4. EVC Service Types . . . . . . . . . . . . . . . . . . . . 9 81 3.5. Designated Forwarder (DF) Election . . . . . . . . . . . 9 82 3.6. OAM . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 83 3.7. Failure and Recovery . . . . . . . . . . . . . . . . . . 10 84 3.8. Fast Convergence . . . . . . . . . . . . . . . . . . . . 10 85 4. Solution Overview . . . . . . . . . . . . . . . . . . . . . . 11 86 4.1. EVPN DF Election for vES . . . . . . . . . . . . . . . . 12 87 5. Failure Handling and Recovery . . . . . . . . . . . . . . . . 13 88 5.1. EVC Failure Handling for Single-Active vES in EVPN . . . 15 89 5.2. EVC Failure Handling for Single-Active vES in PBB-EVPN . 15 90 5.3. Port Failure Handling for Single-Active vESes in EVPN . . 16 91 5.4. Port Failure Handling for Single-Active vESes in PBB-EVPN 17 92 5.5. Fast Convergence in (PBB-)EVPN . . . . . . . . . . . . . 18 93 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 94 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 95 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 96 9. Intellectual Property Considerations . . . . . . . . . . . . 20 97 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 98 10.1. Normative References . . . . . . . . . . . . . . . . . . 20 99 10.2. Informative References . . . . . . . . . . . . . . . . . 21 100 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21 102 1. Introduction 104 [RFC7432] and [RFC7623] introduce a family of solutions for 105 multipoint Ethernet services over MPLS/IP network with many advanced 106 features among which their multi-homing capabilities. These 107 solutions introduce Single-Active and All-Active for an Ethernet 108 Segment (ES), itself defined as a set of links between the multi- 109 homed device/network and a set of PE devices that they are connected 110 to. 112 This document extends the Ethernet Segment concept so that an ES can 113 be associated to a set of EVCs (e.g., VLANs) or other objects such as 114 MPLS Label Switch Paths (LSPs) or Pseudowires (PWs), referred to as 115 Virtual Ethernet Segments (vES). This draft describes the 116 requirements and the extensions needed to support vES in EVPN and 117 PBB-EVPN. 119 1.1. Virtual Ethernet Segments in Access Ethernet Networks 121 Some Service Providers (SPs) want to extend the concept of the 122 physical links in an ES to Ethernet Virtual Circuits (EVCs) where 123 many of such EVCs (e.g., VLANs) can be aggregated on a single 124 physical External Network-to-Network Interface (ENNI). An ES that 125 consists of a set of EVCs instead of physical links is referred to as 126 a virtual ES (vES). Figure 1 depicts two PE devices (PE1 and PE2) 127 each with an ENNI where a number of vESes are aggregated on - each of 128 which through its associated EVC. 130 Carrier 131 Ethernet 132 +-----+ Network 133 | CE11|EVC1 +---------+ 134 +-----+ \ | | +---+ 135 Cust. A \-0=========0--ENNI1| | 136 +-----+ | | ENNI1| | +-------+ +---+ 137 | CE12|EVC2--0=========0--ENNI1|PE1|---| | | | 138 +-----+ | | ENNI1| | | |---|PE3|- 139 | ==0--ENNI1| | |IP/MPLS| | | \ +---+ 140 +-----+ | / | +---+ |Network| +---+ \-| | 141 | CE22|EVC3--0==== / | | | |CE4| 142 +-----+ | X | | | +---+ | | 143 Cust. B | / \ | +---+ | | | | /-| | 144 +-----+ -0=== ===0--ENNI2| | | |---|PE4|-/ +---+ 145 | CE3 |EVC4/ | | ENNI2|PE2|---| | | | 146 | |EVC5--0=========0--ENNI2| | +-------+ +---+ 147 +-----+ | | +---+ 148 Cust. C +---------+ /\ 149 /\ || 150 || ENNI 151 EVCs Interface 152 <--------802.1Q----------> <---- EVPN Network -----> <-802.1Q-> 154 Figure 1: DHD/DHN (both SA/AA) and SH on same ENNI 156 ENNIs are commonly used to reach off-network / out-of-franchise 157 customer sites via independent Ethernet access networks or third- 158 party Ethernet Access Providers (EAP) (see Figure 1). ENNIs can 159 aggregate traffic from hundreds to thousands of vESes, where each vES 160 is represented by its associated EVC on that ENNI. As a result, 161 ENNIs and their associated EVCs are a key element of SP off-networks 162 that are carefully designed and closely monitored. 164 In order to meet customers' Service Level Agreements (SLA), SPs build 165 redundancy via multiple EVPN PEs and across multiple ENNIs (as shown 166 in Figure 1) where a given vES can be multi-homed to two or more EVPN 167 PE devices (on two or more ENNIs) via their associated EVCs. Just 168 like physical ES's in [RFC7432] and [RFC7623] solutions, these vESes 169 can be single-homed or multi-homed ES's and when multi-homed, then 170 can operate in either Single-Active or All-Active redundancy modes. 171 In a typical SP off-network scenario, an ENNI can be associated with 172 several thousands of single-homed vESes, several hundreds of Single- 173 Active vESes and it may also be associated with tens or hundreds of 174 All-Active vESes. 176 1.2. Virtual Ethernet Segments in Access MPLS Networks 178 Other Service Providers (SPs) want to extend the concept of the 179 physical links in an ES to individual Pseudowires (PWs) or to MPLS 180 Label Switched Paths (LSPs) in Access MPLS networks - i.e., a vES 181 consisting of a set of PWs or a set of LSPs. Figure 2 illustrates 182 this concept. 184 MPLS Aggregation 185 Network 186 +-----+ +-----------------+ 187 | CE11|EVC1 | | 188 +-----+ \+AG1--+ PW1 +-----+ 189 Cust. A -0----|===========| | 190 +-----+ | ---+===========| | +-------+ +---+ 191 | CE12|EVC2-0/ | PW2 /\ | PE1 +---+ | | | 192 +-----+ ++---+ ==||=| | | +---+PE3+- 193 | //=||=| | |IP/MPLS| | | \ +---+ 194 | // \/ ++----+ |Network| +---+ \-+ | 195 +-----+EVC3 | PW3// LSP1 | | | |CE4| 196 | CE13| +AG2--+===/PW4 | | | +---+ | | 197 +-----+ 0 |=== /\ ++----+ | | | | /-+ | 198 0 |==PW5===||=| | | +---+PE4+-/ +---+ 199 +-----+ /++---+==PW6===||=| PE2 +---+ | | | 200 | CE14|EVC4 | \/ | | +-------+ +---+ 201 +-----+ | LSP2+-----+ 202 Cust. C +-----------------+ 203 /\ 204 || 205 EVCs 206 <--802.1Q---><------MPLS------> <---- EVPN Network ---> <-802.1Q-> 208 Figure 2: DHN and SH on Access MPLS networks 210 In some cases, Service Providers use Access MPLS Networks that belong 211 to separate administrative entities or third parties as a way to get 212 access to the their own IP/MPLS network infrastructure. This is the 213 case illustrated in Figure 2. 215 In such scenarios, a virtual ES (vES) is defined as a set of 216 individual PWs if they cannot be aggregated into a common LSP. If 217 the aggregation of PWs is possible, the vES can be associated to an 218 LSP in a given PE. In the example of Figure 2, EVC3 is connected to 219 a VPWS instance in AG2 that is connected to PE1 and PE2 via PW3 and 220 PW5 respectively. EVC4 is connected to a separate VPWS instance on 221 AG2 that gets connected to an EVI on PE1 and PE2 via PW4 and PW6, 222 respectively. Since the PWs for the two VPWS instances can be 223 aggregated into the same LSPs going to the MPLS network, a common 224 virtual ES can be defined for LSP1 and LSP2. This vES will be shared 225 by two separate EVIs in the EVPN network. 227 In some cases, this aggregation of PWs into common LSPs may not be 228 possible. For instance, if PW3 were terminated into a third PE, e.g. 229 PE3, instead of PE1, the vES would need to be defined on a per 230 individual PW on each PE, i.e. PW3 and PW5 would belong to ES-1, 231 whereas PW4 and PW6 would be associated to ES-2. 233 For MPLS/IP access networks where a vES represents a set of PWs or 234 LSPs, this document extends Single-Active multi-homing procedures of 235 [RFC7432] and [RFC7623] to vES. The vES extension to All-Active 236 multi- homing is outside of the scope of this document for MPLS/IP 237 access networks. 239 This draft describes requirements and the extensions needed to 240 support a vES in [RFC7432] and [RFC7623]. Section 3 lists the set of 241 requirements for a vES. Section 4 describes extensions for a vES 242 that are applicable to EVPN solutions including [RFC7432] and 243 [RFC7209]. Furthermore, these extensions meet the requirements 244 described in Section 3. Section 4 gives solution overview and 245 Section 5 describes failure handling, recovery, scalability, and fast 246 convergence of [RFC7432] and [RFC7623] for vESes. 248 2. Terminology 250 AC: Attachment Circuit 252 BEB: Backbone Edge Bridge 254 B-MAC: Backbone MAC Address 256 CE: Customer Edge 258 CFM: Connectivity Fault Management (802.1ag) 260 C-MAC: Customer/Client MAC Address 262 DHD: Dual-homed Device 264 DHN: Dual-homed Network 266 ENNI: External Network-Network Interface 267 ES: Ethernet Segment 269 ESI: Ethernet Segment Identifier 271 EVC: Ethernet Virtual Circuit 273 EVPN: Ethernet VPN 275 I-SID: Service Instance Identifier (24 bits and global within a PBB 276 network see [RFC7080]) 278 LACP: Link Aggregation Control Protocol 280 PBB: Provider Backbone Bridge 282 PBB-EVPN: Provider Backbone Bridge EVPN 284 PE: Provider Edge 286 SH: Single-Homed 288 Single-Active Redundancy Mode (SA): When only a single PE, among a 289 group of PEs attached to an Ethernet Segment, is allowed to forward 290 traffic to/from that Ethernet Segment, then the Ethernet Segment is 291 defined to be operating in Single-Active redundancy mode. 293 All-Active Redundancy Mode (AA): When all PEs attached to an Ethernet 294 segment are allowed to forward traffic to/from that Ethernet Segment, 295 then the Ethernet Segment is defined to be operating in All-Active 296 redundancy mode. 298 3. Requirements 300 This section describes the requirements specific to virtual Ethernet 301 Segment (vES) for (PBB-)EVPN solutions. These requirements are in 302 addition to the ones described in [RFC8214], [RFC7432], and 303 [RFC7623]. 305 3.1. Single-Homed and Multi-Homed vES 307 A PE needs to support the following types of vESes: 309 (R1a) A PE MUST handle single-homed vESes on a single physical port 310 (e.g., single ENNI) 312 (R1b) A PE MUST handle a mix of Single-Homed vESes and Single-Active 313 multi-homed vESes simultaneously on a single physical port (e.g., 314 single ENNI). Single-Active multi-homed vESes will be simply 315 referred to as Single-Active vESes through the rest of this document. 317 (R1c) A PE MAY handle All-Active multi-homed vESes on a single 318 physical port. All-Active multi-homed vESes will be simply referred 319 to as All-Active vESes through the rest of this document. 321 (R1d) A PE MAY handle a mixed of All-Active vESes along with other 322 types of vESes on a single physical port. 324 (R1e) A Multi-Homed vES (Single-Active or All-Active) can be spread 325 across two or more ENNIs, on any two or more PEs. 327 3.2. Scalability 329 A single physical port (e.g., ENNI) can be associated with many 330 vESes. The following requirements give a quantitative measure for 331 each vES type. 333 (R2a) A PE SHOULD handle very large number of Single-Homed vESes on a 334 single physical port (e.g., thousands of vESes on a single ENNI). 336 (R2b) A PE SHOULD handle large number of Single-Active vESes on a 337 single physical port (e.g., hundreds of vESes on a single ENNI). 339 (R2c) A PE MAY handle large number of All-Active vESes on a single 340 physical port (e.g., hundreds of vESes on a single ENNI). 342 (R2d) A PE SHOULD handle the above scale for a mix of Single-homed 343 vESes and Single-Active vESes simultaneously on a single physical 344 port (e.g., single ENNI). 346 (R2e) A PE MAY handle the above sale for a mixed of All-Active vESes 347 along with other types of vESes on a single physical port. 349 3.3. Local Switching 351 Many vESes of different types can be aggregated on a single physical 352 port on a PE device and some of these vES can belong to the same 353 service instance (or customer). This translates into the need for 354 supporting local switching among the vESes of the same service 355 instance on the same physical port (e.g., ENNI) of the PE. 357 (R3a) A PE MUST support local switching among different vESes 358 belonging to the same service instance (or customer) on a single 359 physical port. For example, in Figure 1, PE1 MUST support local 360 switching between CE11 and CE12 (both belonging to customer A) that 361 are mapped to two Single-homed vESes on ENNI1. In case of Single- 362 Active vESes, the local switching is performed among active EVCs 363 belonging to the same service instance on the same ENNI. 365 3.4. EVC Service Types 367 A physical port (e.g., ENNI) of a PE can aggregate many EVCs each of 368 which is associated with a vES. Furthermore, an EVC may carry one or 369 more VLANs. Typically, an EVC carries a single VLAN and thus it is 370 associated with a single broadcast domain. However, there is no 371 restriction on an EVC to carry more than one VLAN. 373 (R4a) An EVC can be associated with a single broadcast domain - e.g., 374 VLAN-based service or VLAN bundle service. 376 (R4b) An EVC MAY be associated with several broadcast domains - e.g., 377 VLAN-aware bundle service. 379 In the same way, a PE can aggregate many LSPs and PWs. In the case 380 of individual PWs per vES, typically a PW is associated with a single 381 broadcast domain, but there is no restriction on the PW to carry more 382 than one VLAN if the PW is of type Raw mode. 384 (R4c) A PW can be associated with a single broadcast domain - e.g., 385 VLAN-based service or VLAN bundle service. 387 (R4d) An PW MAY be associated with several broadcast domains - e.g., 388 VLAN-aware bundle service. 390 3.5. Designated Forwarder (DF) Election 392 Section 8.5 of [RFC7432] describes the default procedure for DF 393 election in EVPN which is also used in [RFC7623] and [RFC8214]. This 394 default DF election procedure is performed at the granularity of 395 (ESI, Ethernet Tag). In case of a vES, the same EVPN default 396 procedure for DF election also applies; however, at the granularity 397 of (vESI, Ethernet Tag); where vESI is the virtual Ethernet Segment 398 Identifier and the Ethernet Tag field is represented by and I-SID in 399 PBB-EVPN and by a VLAN ID (VID) in EVPN. As in [RFC7432], this 400 defult procedure for DF election at the granularity of (vESI, 401 Ethernet Tag) is also referred to as "service carving". With service 402 carving, it is desireable to evenly partition the DFs for different 403 vES's among different PEs, thus evenly distributing the traffic among 404 different PEs. The following list the requirements apply to DF 405 election of vES's for (PBB-)EVPN. 407 (R5a) A vES with m EVCs can be distributed among n ENNIs belonging to 408 p PEs in any arbitrary order; where n >= p >= m. For example, if 409 there is an vES with 2 EVCs and there are 5 ENNIs on 5 PEs (PE1 410 through PE5), then vES can be dual-homed to PE2 and PE4 and the DF 411 election must be performed between PE2 and PE4. 413 (R5b) Each vES MUST be identified by its own virtual ESI (vESI). 415 3.6. OAM 417 In order to detect the failure of an individual EVC and perform DF 418 election for its associated vES as the result of this failure, each 419 EVC should be monitored independently. 421 (R6a) Each EVC SHOULD be monitored for its health independently. 423 (R6b) A single EVC failure (among many aggregated on a single 424 physical port/ENNI) MUST trigger DF election for its associated vES. 426 3.7. Failure and Recovery 428 (R7a) Failure and failure recovery of an EVC for a Single-homed vES 429 SHALL NOT impact any other EVCs within its service instance or any 430 other service instances. In other words, for PBB-EVPN, it SHALL NOT 431 trigger any MAC flushing both within its own I-SID as well as other 432 I-SIDs. 434 (R7b) In case of All-Active vES, failure and failure recovery of an 435 EVC for that vES SHALL NOT impact any other EVCs within its service 436 instance or any other service instances. In other words, for PBB- 437 EVPN, it SHALL NOT trigger any MAC flushing both within its own I-SID 438 as well as other I-SIDs. 440 (R7c) Failure and failure recovery of an EVC for a Single-Active vES 441 SHALL impact only its own service instance. In other words, for PBB- 442 EVPN, MAC flushing SHALL be limited to the associated I-SID only and 443 SHALL NOT impact any other I-SIDs. 445 (R7d) Failure and failure recovery of an EVC for a Single-Active vES 446 MAY only impact C-MACs associated with MHD/MHNs for that service 447 instance. In other words, MAC flushing SHOULD be limited to single 448 service instance (I-SID in the case of PBB-EVPN) and only CMACs for 449 Single-Active MHD/MHNs. 451 3.8. Fast Convergence 453 Since large number of EVCs (and their associated vESes) are 454 aggregated via a single physical port (e.g., ENNI), then the failure 455 of that physical port impacts large number of vESes and triggers 456 large number of ES route withdrawals. Formulating, sending, 457 receiving, and processing such large number of BGP messages can 458 introduce delay in DF election and convergence time. As such, it is 459 highly desirable to have a mass-withdraw mechanism similar to the one 460 in the [RFC7432] for withdrawing large number of Ethernet A-D routes. 462 (R8a) There SHOULD be a mechanism equivalent to EVPN mass-withdraw 463 such that upon an ENNI failure, only a single BGP message is needed 464 to indicate to the remote PEs to trigger DF election for all impacted 465 vES associated with that ENNI. 467 4. Solution Overview 469 The solutions described in [RFC7432] and [RFC7623] are leveraged as- 470 is with the modification that the ESI assignment is performed for an 471 EVC or a group of EVCs or LSPs/PWs instead of a link or a group of 472 physical links. In other words, the ESI is associated with a virtual 473 ES (vES), hereby referred to as vESI. 475 For the EVPN solution, everything basically remains the same except 476 for the handling of physical port failure where many vESes can be 477 impacted. Sections 5.1 and 5.3 below describe the handling of 478 physical port/link failure for EVPN. In a typical multi-homed 479 operation, MAC addresses are learned behind a vES and are advertised 480 with the ESI corresponding to the vES (i.e., vESI). EVPN aliasing 481 and mass- withdraw operations are performed with respect to vES. In 482 other words, the Ethernet A-D routes for these operations are 483 advertised with vESI instead of ESI. 485 For PBB-EVPN solution, the main change is with respect to the BMAC 486 address assignment which is performed similar to what is described in 487 section 7.2.1.1 of [RFC7623] with the following refinements: 489 o One shared BMAC address SHOULD be used per PE for the single-homed 490 vESes. In other words, a single BMAC is shared for all single- 491 homed vESes on that PE. 493 o One shared BMAC address SHOULD be used per PE per physical port 494 (e.g., ENNI) for the Single-Active vESes. In other words, a 495 single BMAC is shared for all Single-Active vESes that share the 496 same ENNI. 498 o One shared BMAC address MAY be used for all Single-Active vESes on 499 that PE. 501 o One BMAC address SHOULD be used per set of EVCs representing an 502 All-Active vES. In other words, a single BMAC address is used per 503 vES for All-Active scenarios. 505 o A single BMAC address MAY also be used per vES per PE for Single- 506 Active scenarios. 508 BEB +--------------+ BEB 509 || | | || 510 \/ | | \/ 511 +----+ EVC1 +----+ | | +----+ +----+ 512 | CE1|------| | | | | |---| CE2| 513 +----+\ | PE1| | IP/MPLS | | PE3| +----+ 514 \ +----+ | Network | +----+ 515 \ | | 516 EVC2\ +----+ | | 517 \ | | | | 518 \| PE2| | | 519 +----+ | | 520 /\ +--------------+ 521 || 522 BEB 523 <--802.1Q--><---------- PBB-EVPN --------><--802.1Q-> 525 Figure 3: PBB-EVPN Network 527 4.1. EVPN DF Election for vES 529 The procedure for service carving for virtual Ethernet Segments is 530 the same as the one outlined in section 8.5 of [RFC7432] except for 531 the fact that ES is replaced with vES. For the sake of clarity and 532 completeness, this procedure is repeated below: 534 1. When a PE discovers the vESI or is configured with the vESI 535 associated with its attached vES, it advertises an Ethernet 536 Segment route with the associated ES-Import extended community 537 attribute. 539 2. The PE then starts a timer (default value = 3 seconds) to allow 540 the reception of Ethernet Segment routes from other PE nodes 541 connected to the same vES. This timer value MUST be same across 542 all PEs connected to the same vES. 544 3. When the timer expires, each PE builds an ordered list of the IP 545 addresses of all the PE nodes connected to the vES (including 546 itself), in increasing numeric value. Each IP address in this 547 list is extracted from the "Originator Router's IP address" field 548 of the advertised Ethernet Segment route. Every PE is then given 549 an ordinal indicating its position in the ordered list, starting 550 with 0 as the ordinal for the PE with the numerically lowest IP 551 address. The ordinals are used to determine which PE node will 552 be the DF for a given EVPN instance on the vES using the 553 following rule: Assuming a redundancy group of N PE nodes, the PE 554 with ordinal i is the DF for an EVPN instance with an associated 555 Ethernet Tag value of V when (V mod N) = i. It should be noted 556 that using "Originator Router's IP address" field in the Ethernet 557 Segment route to get the PE IP address needed for the ordered 558 list, allows for a CE to be multi-homed across different ASes if 559 such need ever arises. 561 4. The PE that is elected as a DF for a given EVPN instance will 562 unblock traffic for that EVPN instance. Note that the DF PE 563 unblocks all traffic in both ingress and egress directions for 564 Single-Active vES and unblocks multi-destination in egress 565 direction for All-Active Multi-homed vES. All non-DF PEs block 566 all traffic in both ingress and egress directions for Single- 567 Active vES and block multi-destination traffic in the egress 568 direction for All-Active vES. 570 In the case of an EVC failure, the affected PE withdraws its Virtual 571 Ethernet Segment route if there are no more EVCs associated to the 572 vES in the PE. This will re-trigger the DF Election procedure on all 573 the PEs in the Redundancy Group. For PE node failure, or upon PE 574 commissioning or decommissioning, the PEs re-trigger the DF Election 575 Procedure across all affected vESes. In case of a Single-Active, 576 when a service moves from one PE in the Redundancy Group to another 577 PE as a result of DF re-election, the PE, which ends up being the 578 elected DF for the service, SHOULD trigger a MAC address flush 579 notification towards the associated vES. This can be done, for e.g. 580 using IEEE 802.1ak MVRP 'new' declaration. 582 For LSP-baesd and PW-based vES, the non-DF PE SHOULD signal PW-status 583 'standby' to the Aggregation PE (e.g., AG PE in Figure 2), and a new 584 DF PE MAY send an LDP MAC withdraw message as a MAC address flush 585 notification. It should be noted that the PW-status is signaled for 586 the scenarios where there is a one-to-one mapping between EVI/BD and 587 the PW. 589 5. Failure Handling and Recovery 591 There are a number of failure scenarios to consider such as: 593 A: CE uplink port failure 595 B: Ethernet Access Network failure 596 C: PE access-facing port or link failure 598 D: PE node failure 600 E: PE isolation from IP/MPLS network 602 [RFC7432], [RFC7623], and [RFC8214] solutions provide protection 603 against such failures as described in the corresponding references. 604 In the presence of virtual Ethernet Segments (vESes) in these 605 solutions, besides the above failure scenarios, EVC failure is an 606 additional scenario to consider. Handling vES failure scenarios 607 implies that individual EVCs or PWs need to be monitored and upon 608 detection of failure or restoration of services, appropriate DF 609 election and failure recovery mechanisms are executed. 611 [ETH-OAM] is used for monitoring EVCs and upon failure detection of a 612 given EVC, DF election procedure per section [4.1] is executed. For 613 PBB-EVPN, some extensions are needed to handle the failure and 614 recovery procedures of [RFC7623] in order to meet the above 615 requirements. These extensions are described in the next section. 617 [MPLS-OAM] and [PW-OAM] are used for monitoring the status of LSPs 618 and/or PWs associated to vES. 620 B D 621 || || 622 \/ \/ 623 +-----+ 624 +-----+ | | +---+ 625 | CE1 |EVC2--0=====0--ENNI1| | +-------+ 626 +-----+ | =0--ENNI1|PE1|---| | +---+ +---+ 627 Cust. A | / | | | |IP/MPLS|--|PE3|--|CE4| 628 +-----+ | / | +---+ |Network| | | +---+ 629 | |EVC2--0== | | | +---+ 630 | CE2 | | | +---+ | | 631 | |EVC3--0=====0--ENNI2|PE2|---| | 632 +-----+ | | | | +-------+ 633 +-----+ +---+ 634 /\ /\ /\ 635 || || || 636 A C E 638 Figure 4: Failure Scenarios A,B,C,D and E 640 5.1. EVC Failure Handling for Single-Active vES in EVPN 642 In RFC7432, when a DF PE connected to a Single-Active multi-homed 643 Ethernet Segment loses connectivity to the segment, due to link or 644 port failure, it signals to the remote PEs to withdraw all MAC 645 addresses associated with that Ethernet Segment. This is done by 646 advertising a mass-withdraw message using Ethernet A-D per-ES route. 647 It should be noted that for dual-homing use cases where there is only 648 a single backup path, MAC withdraw can be avoided by the remote PEs 649 as they can simply update their nexthop associated with the affected 650 MAC entries to the backup path per procedure described in section 8.2 651 of [RFC7432]. 653 In case of an EVC failure which impacts a single vES, the exact same 654 EVPN procedure is used. In this case, the message using Ethernet A-D 655 per-vES route carries the vESI representing the vES which in turn is 656 associated with the failed EVC. The remote PEs upon receiving this 657 message perform the same procedures outlined in section 8.2 of 658 [RFC7432]. 660 5.2. EVC Failure Handling for Single-Active vES in PBB-EVPN 662 In [RFC7432], when a PE connected to a Single-Active Ethernet Segment 663 loses connectivity to the segment, due to link or port failure, it 664 signals the remote PE to flush all CMAC addresses associated with 665 that Ethernet Segment. This is done by advertising a BMAC route 666 along with MAC Mobility Extended community. 668 In case of an EVC failure that impacts a single vES, if the above 669 PBB-EVPN procedure is used, it results in excessive CMAC flushing 670 because a single physical port can support large number of EVCs (and 671 their associated vESes) and thus advertising a BMAC corresponding to 672 the physical port with MAC mobility Extended community will result in 673 flushing CMAC addresses not just for the impacted EVC but for all 674 other EVCs on that port. 676 In order to reduce the scope of CMAC flushing to only the impacted 677 service instances (the service instance(s) impacted by the EVC 678 failure), the PBB-EVPN CMAC flushing needs to be adapted on a per 679 service instance basis (i.e., per I-SID). 680 [I-D.ietf-bess-pbb-evpn-isid-cmacflush] introduces BMAC/I-SID route 681 where existing PBB-EVPN BMAC route is modified to carry an I-SID in 682 the "Ethernet Tag ID" field instead of NULL value. This field 683 indicates to the receiving PE, to flush all CMAC addresses associated 684 with that I-SID for that BMAC. This CMAC flushing mechanism per 685 I-SID SHOULD be used in case of EVC failure impacting a vES. Since 686 typically an EVC maps to a single broadcast domain and thus a single 687 service instance, the affected PE only needs to advertise a single 688 BMAC/I-SID route. However, if the failed EVC carries multiple VLANs 689 each with its own broadcast domain, then the affected PE needs to 690 advertise multiple BMAC/I-SID routes - one for each VLAN (broadcast 691 domain) - i.e., one for each I-SID. Each BMAC/I-SID route basically 692 instructs the remote PEs to perform flushing for CMACs corresponding 693 to the advertised BMAC only for the advertised I-SID. 695 The CMAC flushing based on BMAC/I-SID route works fine when there are 696 only a few VLANs (e.g., I-SIDs) per EVC. However if the number of 697 I-SIDs associated with a failed EVC is large, then it is recommended 698 to assign a BMAC per vES and upon EVC failure, the affected PE simply 699 advertise BMAC withdraw message to other PEs. 701 5.3. Port Failure Handling for Single-Active vESes in EVPN 703 When a large number of EVCs are aggregated via a single physical port 704 on a PE; where each EVC corresponds to a vES, then the port failure 705 impacts all the associated EVCs and their corresponding vESes. If 706 the number of EVCs corresponding to the Single-Active vESes for that 707 physical port is in thousands, then thousands of service instances 708 are impacted. Therefore, the BGP flush message need to be inclusive 709 of all these impacted service instances. In order to achieve this, 710 the following extensions are added to the baseline EVPN mechanism: 712 1. When a PE advertises an Ethernet A-D per-ES route for a given 713 vES, it colors it with the MAC address of the physical port which 714 is associated with that vES using EVPN Router's MAC Extended 715 Community per [EVPN-IRB]. The receiving PEs take note of this 716 color and create a list of vESes for this color. 718 2. Upon a port failure (e.g., ENNI failure), the PE advertise a 719 special mass-withdraw message with the MAC address of the failed 720 port (i.e., the color of the port) encoded in the ESI field. For 721 this encoding, type 3 ESI (RFC7432 section 5) is used with the 722 MAC field set to the MAC address of the port and the 3-octet 723 local discriminator field set to 0xFFFFFF. This mass-withdraw 724 route is advertised with a list of Route Targets corresponding to 725 the impacted service instances. If the number of Route Targets 726 is more than can fit into a single attribute, then a set of 727 Ethernet A-D per ES routes are advertised. 729 3. Upon a port failure (e.g., ENNI failure), the PE advertise a 730 special mass-withdraw message with the MAC address of the failed 731 port. 733 4. The remote PEs upon receiving this message, based on ESI Type 3 734 and 0xFFFFFF Local Discrimnator values, detect the special vES 735 mass-withdraw message. The remote PEs then access the list of 736 the vES's for the specified color created in (1) and initialte 737 locally mass-withdraw procedures for each of the vES's in the 738 list. 740 In scenarios where a logical ENNI is used the above procedure equally 741 applies. The logical ENNI is represented by a Type 3 ESI and the MAC 742 address used in the ENNI's ESI is used as a color for vESes as 743 described above. 745 5.4. Port Failure Handling for Single-Active vESes in PBB-EVPN 747 When a large number of EVCs are aggregated via a single physical port 748 on a PE, where each EVC corresponds to a vES, then the port failure 749 impacts all the associated EVCs and their corresponding vESes. If 750 the number of EVCs corresponding to the Single-Active vESes for that 751 physical port is in thousands, then thousands of service instances 752 (I-SIDs) are impacted. In such failure scenarios, the following two 753 MAC flushing mechanisms per [RFC7623] can be performed. 755 1. If the MAC address of the physical port is used for PBB 756 encapsulation as BMAC SA, then upon the port failure, the PE MUST 757 use the EVPN MAC route withdrawal message to signal the flush. 759 2. If the PE shared MAC address is used for PBB encapsulation as 760 BMAC SA, then upon the port failure, the PE MUST re-advertise 761 this MAC route with the MAC Mobility Extended Community to signal 762 the flush. 764 The first method is recommended because it reduces the scope of 765 flushing the most. 767 If there are large number of service instances (i.e., I-SIDs) 768 associated with each EVC, and if there is a BMAC assigned per vES as 769 recommended in the above section, then in order to handle port 770 failure efficiently, each vES MAY be color with another MAC 771 representing the physical port similar to the coloring mechanism for 772 EVPN. In other words, each BMAC representing a vES is advertised 773 with the EVPN Router's MAC Extended Community carrying the MAC 774 address of the physical port.The difference between coloring 775 mechanism for EVPN and PBB-EVPN is that for EVPN, the extended 776 community is advertised with the Ethernet A-D per ES route; whereas, 777 for PBB-EVPN, the extended community is advertised with the BMAC 778 route. As noted above, the advertisement of the extended community 779 along with BMAC route for coloring purpoes is optional and only 780 recommended when there are many vESes per physical port and each vES 781 is associated with very large number of service instances (i.e., 782 large numbe of I-SIDs). 784 When coloring mechanism is used, the receiving PEs take note of the 785 color being advertised along with the BMAC route and for each such 786 color, they create a list of vESes associated with this color (i.e., 787 associated with this MAC address). Now, when a port failure occurs, 788 the impacted PE needs to notify the other PEs of this color so that 789 these PEs can identify all the impacted vESes associated with this 790 color (from the above list) and flush CMACs associated with the 791 failed physical port. This is accomplished by withdrawing the MAC 792 route associated with the failed port. 794 5.5. Fast Convergence in (PBB-)EVPN 796 As described above, when a large number of EVCs are aggregated via a 797 physical port on a PE, and where each EVC corresponds to a vES, then 798 the port failure impacts all the associated EVCs and their 799 corresponding vESes. Two actions must be taken as the result of such 800 port failure: 802 o For EVPN initiate mass-withdraw procedure for all vESes associated 803 with the failed port and for PBB-EVPN flush all CMACs associated 804 with the failed port across all vESes and the impacted I-SIDs 806 o DF election for all impacted vESes associated with the failed port 808 Section 5.3 already describes how perform mass-withdraw for all 809 affected vESes using a single BGP advertisment. Section 5.4 810 describes how to only flush CMAC address associated with the failed 811 physical port (e.g., optimum CMAC flushing). This section describes 812 how to perform DF election in the most optimum way - e.g., to trigger 813 DF election for all impacted vESes (which can be very large) among 814 the participating PEs via a single BGP message as opposed to sending 815 large number of BGP messages - one per vES. This section assumes 816 that the MAC flushing mechanism described in section 5.4, bullet (1) 817 is used. 819 +-----+ 820 +----+ | | +---+ 821 | CE1|AC1--0=====0--ENNI1| | +-------+ 822 | |AC2--0 | |PE1|--| | 823 +----+ |\ ==0--ENNI2| | | | 824 | \/ | +---+ | | 825 | /\ | |IP/MPLS| 826 +----+ |/ \ | +---+ |Network| +---+ +---+ 827 | CE2|AC4--0 =0--ENNI3| | | |---|PE4|--|CE4| 828 | |AC4--0=====0--ENNI3|PE2|--| | +---+ +---+ 829 +----+ | ====0--ENNI3| | | | 830 |/ | +---+ | | 831 0 | | | 832 +----+ /| | +---+ | | 833 | CE3|AC5- | | |PE3|--| | 834 | |AC6--0=====0--ENNI4| | +-------+ 835 +----+ | | +---+ 836 +-----+ 838 Figure 5: Fast Convergence Upon ENNI Failure 840 The following describes the procedure for coloring vESes and fast 841 convergence for DF election using this color: 843 1. When a vES is configured, the PE colors the vES with the MAC 844 address of the corresponding physical port and advertises the 845 Ethernet Segment route for this vES with this color. 847 2. All other PEs (in the redundancy group) take note of this color 848 and add the vES to the list for this color. 850 3. Upon the occurrence of a port failure (e.g., an ENNI failure), 851 the PE withdraw the previously advertised MAC address associated 852 with the failed port. The PE should prioritize sending this MAC 853 address withdraw message over vES route withdrawal messages of 854 impacted vESes. 856 4. On reception of this MAC withdraw message, other PEs in the 857 redundancy group use this info to initiate DF election procedures 858 across all their affected vESes. 860 5. The PE with the physical port failure (ENNI failure), also sends 861 vES route withdrawal for every impacted vESes. The other PEs 862 upon receiving these messages, clear up their BGP tables. It 863 should be noted the vES route withdrawal messages are not used 864 for executing DF election procedures by the receiving PEs. 866 6. Acknowledgements 868 The authors would like to thanks Mei Zhang, Jose Liste, and Luc Andre 869 Burdet for their reviews and feedbacks of this document. 871 7. Security Considerations 873 All the security considerations in [RFC7432] and [RFC7623] apply 874 directly to this document because this document leverages the control 875 and data plane procedures described in those documents. 877 This document does not introduce any new security considerations 878 beyond that of [RFC7432] and [RFC7623] because advertisements and 879 processing of Ethernet Segment route for vES in this document follows 880 that of physical ES in those RFCs. 882 8. IANA Considerations 884 IANA has allocated sub-type value 7 in the "EVPN Extended Community 885 Sub-Types" registry defined in "https://www.iana.org/assignments/bgp- 886 extended-communities/bgp-extended-communities.xhtml#evpn" as follows: 888 SUB-TYPE NAME Reference 889 ---- -------------- ------------- 890 0x07 I-SID Ext Comm [draft-ietf-bess-evpn-virtual-eth-segment] 892 It is requested from IANA to update the reference to this document. 894 9. Intellectual Property Considerations 896 This document is being submitted for use in IETF standards 897 discussions. 899 10. References 901 10.1. Normative References 903 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 904 Requirement Levels", BCP 14, RFC 2119, 905 DOI 10.17487/RFC2119, March 1997, 906 . 908 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 909 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 910 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 911 2015, . 913 [RFC7623] Sajassi, A., Ed., Salam, S., Bitar, N., Isaac, A., and W. 914 Henderickx, "Provider Backbone Bridging Combined with 915 Ethernet VPN (PBB-EVPN)", RFC 7623, DOI 10.17487/RFC7623, 916 September 2015, . 918 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 919 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 920 May 2017, . 922 [RFC8214] Boutros, S., Sajassi, A., Salam, S., Drake, J., and J. 923 Rabadan, "Virtual Private Wire Service Support in Ethernet 924 VPN", RFC 8214, DOI 10.17487/RFC8214, August 2017, 925 . 927 10.2. Informative References 929 [I-D.ietf-bess-pbb-evpn-isid-cmacflush] 930 Rabadan, J., Sathappan, S., Nagaraj, K., Miyake, M., and 931 T. Matsuda, "PBB-EVPN ISID-based CMAC-Flush", draft-ietf- 932 bess-pbb-evpn-isid-cmacflush-00 (work in progress), 933 October 2019. 935 [RFC7080] Sajassi, A., Salam, S., Bitar, N., and F. Balus, "Virtual 936 Private LAN Service (VPLS) Interoperability with Provider 937 Backbone Bridges", RFC 7080, DOI 10.17487/RFC7080, 938 December 2013, . 940 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 941 Henderickx, W., and A. Isaac, "Requirements for Ethernet 942 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 943 . 945 Authors' Addresses 947 Ali Sajassi 948 Cisco Systems 949 MILPITAS, CALIFORNIA 95035 950 UNITED STATES 952 Email: sajassi@cisco.com 954 Patrice Brissette 955 Cisco Systems 957 Email: pbrisset@cisco.com 958 Rick Schell 959 Verizon 961 Email: richard.schell@verizon.com 963 John E Drake 964 Juniper 966 Email: jdrake@juniper.net 968 Jorge Rabadan 969 Nokia 971 Email: jorge.rabadan@nokia.com