idnits 2.17.1 draft-ietf-bess-evpn-inter-subnet-forwarding-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (September 3, 2020) is 1329 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'GENEVE' is mentioned on line 153, but not defined == Outdated reference: A later version (-17) exists of draft-ietf-bess-evpn-irb-extended-mobility-03 == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-17 Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BESS WorkGroup A. Sajassi 3 Internet-Draft S. Salam 4 Intended status: Standards Track S. Thoria 5 Expires: March 7, 2021 Cisco Systems 6 J. Drake 7 Juniper 8 J. Rabadan 9 Nokia 10 September 3, 2020 12 Integrated Routing and Bridging in EVPN 13 draft-ietf-bess-evpn-inter-subnet-forwarding-10 15 Abstract 17 Ethernet VPN (EVPN) provides an extensible and flexible multi-homing 18 VPN solution over an MPLS/IP network for intra-subnet connectivity 19 among Tenant Systems and End Devices that can be physical or virtual. 20 However, there are scenarios for which there is a need for a dynamic 21 and efficient inter-subnet connectivity among these Tenant Systems 22 and End Devices while maintaining the multi-homing capabilities of 23 EVPN. This document describes an Integrated Routing and Bridging 24 (IRB) solution based on EVPN to address such requirements. 26 Requirements Language 28 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 29 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 30 "OPTIONAL" in this document are to be interpreted as described in RFC 31 2119 [RFC2119] and RFC 8174 [RFC8174] when, and only when, they 32 appear in all capitals, as shown here. 34 Status of This Memo 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF). Note that other groups may also distribute 41 working documents as Internet-Drafts. The list of current Internet- 42 Drafts is at https://datatracker.ietf.org/drafts/current/. 44 Internet-Drafts are draft documents valid for a maximum of six months 45 and may be updated, replaced, or obsoleted by other documents at any 46 time. It is inappropriate to use Internet-Drafts as reference 47 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on March 7, 2021. 50 Copyright Notice 52 Copyright (c) 2020 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (https://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 68 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 69 3. EVPN PE Model for IRB Operation . . . . . . . . . . . . . . . 6 70 4. Symmetric and Asymmetric IRB . . . . . . . . . . . . . . . . 7 71 4.1. IRB Interface and its MAC and IP addresses . . . . . . . 10 72 5. Symmetric IRB Procedures . . . . . . . . . . . . . . . . . . 12 73 5.1. Control Plane - Advertising PE . . . . . . . . . . . . . 12 74 5.2. Control Plane - Receiving PE . . . . . . . . . . . . . . 13 75 5.3. Subnet route advertisement . . . . . . . . . . . . . . . 14 76 5.4. Data Plane - Ingress PE . . . . . . . . . . . . . . . . . 14 77 5.5. Data Plane - Egress PE . . . . . . . . . . . . . . . . . 15 78 6. Asymmetric IRB Procedures . . . . . . . . . . . . . . . . . . 16 79 6.1. Control Plane - Advertising PE . . . . . . . . . . . . . 16 80 6.2. Control Plane - Receiving PE . . . . . . . . . . . . . . 16 81 6.3. Data Plane - Ingress PE . . . . . . . . . . . . . . . . . 18 82 6.4. Data Plane - Egress PE . . . . . . . . . . . . . . . . . 18 83 7. Mobility Procedure . . . . . . . . . . . . . . . . . . . . . 19 84 7.1. Initiating a gratutious ARP upon a Move . . . . . . . . . 20 85 7.2. Sending Data Traffic without an ARP Request . . . . . . . 20 86 7.3. Silent Host . . . . . . . . . . . . . . . . . . . . . . . 22 87 8. BGP Encoding . . . . . . . . . . . . . . . . . . . . . . . . 23 88 8.1. Router's MAC Extended Community . . . . . . . . . . . . . 23 89 9. Operational Models for Symmetric Inter-Subnet Forwarding . . 24 90 9.1. IRB forwarding on NVEs for Tenant Systems . . . . . . . . 24 91 9.1.1. Control Plane Operation . . . . . . . . . . . . . . . 25 92 9.1.2. Data Plane Operation . . . . . . . . . . . . . . . . 27 93 9.2. IRB forwarding on NVEs for Subnets behind Tenant Systems 28 94 9.2.1. Control Plane Operation . . . . . . . . . . . . . . . 29 95 9.2.2. Data Plane Operation . . . . . . . . . . . . . . . . 30 97 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 31 98 11. Security Considerations . . . . . . . . . . . . . . . . . . . 31 99 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 100 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 32 101 13.1. Normative References . . . . . . . . . . . . . . . . . . 32 102 13.2. Informative References . . . . . . . . . . . . . . . . . 33 103 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 34 105 1. Terminology 107 AC: Attachment Circuit 109 ARP: Address Resolution Protocol 111 BD: Broadcast Domain. As per [RFC7432], an EVI consists of a single 112 or multiple BDs. In case of VLAN-bundle and VLAN-based service 113 models (see [RFC7432]), a BD is equivalent to an EVI. In case of 114 VLAN-aware bundle service model, an EVI contains multiple BDs. Also, 115 in this document, BD and subnet are equivalent terms and wherever 116 "subnet" is used, it means "IP subnet" 118 BD Route Target: refers to the Broadcast Domain assigned Route Target 119 [RFC4364]. In case of VLAN-aware bundle service model, all the BD 120 instances in the MAC-VRF share the same Route Target 122 BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as per 123 [RFC7432]. 125 Ethernet NVO tunnel: refers to Network Virtualization Overlay tunnels 126 with Ethernet payload. Examples of this type of tunnels are VXLAN or 127 GENEVE. 129 EVI: EVPN Instance spanning the NVE/PE devices that are participating 130 on that EVPN, as per [RFC7432]. 132 EVPN: Ethernet Virtual Private Networks, as per [RFC7432]. 134 IP NVO tunnel: it refers to Network Virtualization Overlay tunnels 135 with IP payload (no MAC header in the payload). 137 IP-VRF: A VPN Routing and Forwarding table for IP routes on an NVE/ 138 PE. The IP routes could be populated by EVPN and IP-VPN address 139 families. An IP-VRF is also an instantiation of a layer 3 VPN in an 140 NVE/PE. 142 IRB: Integrated Routing and Bridging interface. It connects an IP- 143 VRF to a BD (or subnet). 145 MAC-VRF: A Virtual Routing and Forwarding table for Media Access 146 Control (MAC) addresses on an NVE/PE, as per [RFC7432]. A MAC-VRF is 147 also an instantiation of an EVI in an NVE/PE. 149 ND: Neighbor Discovery Protocol 151 NVE: Network Virtualization Edge 153 GENEVE: Generic Network Virtualization Encapsulation, [GENEVE] 155 NVGRE: Network Virtualization Generic Routing Encapsulation 157 NVO: Network Virtualization Overlays 159 RT-2: EVPN route type 2, i.e., MAC/IP Advertisement route, as defined 160 in [RFC7432] 162 RT-5: EVPN route type 5, i.e., IP Prefix route. As defined in 163 Section 3 of [I-D.ietf-bess-evpn-prefix-advertisement] 165 TS: Tenant System 167 VA: Virtual Appliance 169 VNI: Virtual Network Identifier. As in [RFC8365], the term is used 170 as a representation of a 24-bit NVO instance identifier, with the 171 understanding that VNI will refer to a VXLAN Network Identifier in 172 VXLAN, or Virtual Network Identifier in GENEVE, etc. unless it is 173 stated otherwise. 175 VTEP: VXLAN Termination End Point, as in [RFC7348]. 177 VXLAN: Virtual Extensible LAN, as in [RFC7348]. 179 This document also assumes familiarity with the terminology of 180 [RFC7432], [RFC8365] and [RFC7365]. 182 2. Introduction 184 EVPN [RFC7432] provides an extensible and flexible multi-homing VPN 185 solution over an MPLS/IP network for intra-subnet connectivity among 186 Tenant Systems (TSes) and End Devices that can be physical or 187 virtual; where an IP subnet is represented by an EVI for a VLAN-based 188 service or by an (EVI, VLAN) for a VLAN-aware bundle service. 189 However, there are scenarios for which there is a need for a dynamic 190 and efficient inter-subnet connectivity among these Tenant Systems 191 and End Devices while maintaining the multi-homing capabilities of 192 EVPN. This document describes an Integrated Routing and Bridging 193 (IRB) solution based on EVPN to address such requirements. 195 The inter-subnet communication is traditionally achieved at 196 centralized L3 Gateway (L3GW) devices where all the inter-subnet 197 forwarding is performed and all the inter-subnet communication 198 policies are enforced. When two TSes belonging to two different 199 subnets connected to the same PE wanted to communicate with each 200 other, their traffic needed to be back hauled from the PE all the way 201 to the centralized gateway where inter-subnet switching is performed 202 and then back to the PE. For today's large multi-tenant data center, 203 this scheme is very inefficient and sometimes impractical. 205 In order to overcome the drawback of centralized layer-3 GW approach, 206 IRB functionality is needed on the PEs (also referred to as EVPN 207 NVEs) attached to TSes in order to avoid inefficient forwarding of 208 tenant traffic (i.e., avoid back-hauling and hair-pinning). When a 209 PE with IRB capability receives tenant traffic over an Attachment 210 Circuit (AC), it can not only locally bridge the tenant intra-subnet 211 traffic but also can locally route the tenant inter-subnet traffic on 212 a packet by packet basis thus meeting the requirements for both intra 213 and inter-subnet forwarding and avoiding non-optimal traffic 214 forwarding associated with centralized layer-3 GW approach. 216 Some TSes run non-IP protocols in conjunction with their IP traffic. 217 Therefore, it is important to handle both kinds of traffic optimally 218 - e.g., to bridge non-IP and intra-subnet traffic and to route inter- 219 subnet IP traffic. Therefore, the solution needs to meet the 220 following requirements: 222 R1: The solution MUST allow for both inter-subnet and intra-subnet 223 traffic belonging to the same tenant to be locally routed and bridged 224 respectively. The solution MUST provide IP routing for inter-subnet 225 traffic and Ethernet Bridging for intra-subnet traffic. It should be 226 noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE 227 receives IPv4 traffic on the corresponding VLAN, then the IPv4 228 traffic is treated as L2 traffic and it is bridged. Also vise versa, 229 if an IP-VRF in a NVE is configured for IPv4 and that NVE receives 230 IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is 231 treated as L2 traffic and it is bridged. 233 R2: The solution MUST support bridging for non-IP traffic. 235 R3: The solution MUST allow inter-subnet switching to be disabled on 236 a per VLAN basis on PEs where the traffic needs to be back hauled to 237 another node (i.e., for performing FW or DPI functionality). 239 3. EVPN PE Model for IRB Operation 241 Since this document discusses IRB operation in relationship to EVPN 242 MAC-VRF, IP-VRF, EVI, Bridge Domain (BD), Bridge Table (BT), and IRB 243 interfaces, it is important to understand the relationship among 244 these components. Therefore, the following PE model is illustrated 245 below to a) describe these components and b) illustrate the 246 relationship among them. 248 +-------------------------------------------------------------+ 249 | | 250 | +------------------+ IRB PE | 251 | Attachment | +------------------+ | 252 | Circuit(AC1) | | +----------+ | MPLS/NVO tnl 253 ----------------------*Bridge | | +----- 254 | | | |Table(BT1)| | +-----------+ / \ \ 255 | | | | *---------* |<--> |Eth| 256 | | | | VLAN x | |IRB1| | \ / / 257 | | | +----------+ | | | +----- 258 | | | ... | | IP-VRF1 | | 259 | | | +----------+ | | RD2/RT2 |MPLS/NVO tnl 260 | | | |Bridge | | | | +----- 261 | | | |Table(BT2)| |IRB2| | / \ \ 262 | | | | *---------* |<--> |IP | 263 ----------------------* VLAN y | | +-----------+ \ / / 264 | AC2 | | +----------+ | +----- 265 | | | MAC-VRF1 | | 266 | +-+ RD1/RT1 | | 267 | +------------------+ | 268 | | 269 | | 270 +-------------------------------------------------------------+ 272 Figure 1: EVPN IRB PE Model 274 A tenant needing IRB services on a PE, requires an IP Virtual Routing 275 and Forwarding table (IP-VRF) along with one or more MAC Virtual 276 Routing and Forwarding tables (MAC-VRFs). An IP-VRF, as defined in 277 [RFC4364], is the instantiation of an IPVPN instance in a PE. A MAC- 278 VRF, as defined in [RFC7432], is the instantiation of an EVI (EVPN 279 Instance) in a PE. A MAC-VRF consists of one or more Bridge Tables 280 (BTs) where each BT corresponds to a VLAN (broadcast domain - BD). 281 If service interfaces for an EVPN PE are configured in VLAN- Based 282 mode (i.e., section 6.1 of RFC7432), then there is only a single BT 283 per MAC-VRF (per EVI) - i.e., there is only one tenant VLAN per EVI. 284 However, if service interfaces for an EVPN PE are configured in VLAN- 285 Aware Bundle mode (i.e., section 6.3 of RFC7432), then there are 286 several BTs per MAC-VRF (per EVI) - i.e., there are several tenant 287 VLANs per EVI. 289 Each BT is connected to a IP-VRF via a L3 interface called IRB 290 interface. Since a single tenant subnet is typically (and in this 291 document) represented by a VLAN (and thus supported by a single BT), 292 for a given tenant there are as many BTs as there are subnets and 293 thus there are also as many IRB interfaces between the tenant IP-VRF 294 and the associated BTs as shown in the PE model above. 296 IP-VRF is identified by its corresponding route target and route 297 distinguisher and MAC-VRF is also identified by its corresponding 298 route target and route distinguisher. If operating in EVPN VLAN- 299 Based mode, then a receiving PE that receives an EVPN route with MAC- 300 VRF route target can identify the corresponding BT; however, if 301 operating in EVPN VLAN-Aware Bundle mode, then the receiving PE needs 302 both the MAC-VRF route target and VLAN ID in order to identify the 303 corresponding BT. 305 4. Symmetric and Asymmetric IRB 307 This document defines and describes two types of IRB solutions - 308 namely symmetric and asymmetric IRB. The description of symmetric 309 and asymmetric IRB procedures relating to data path operations and 310 tables in this document is a logical view of data path lookups and 311 related tables. Actual implementations, while following this logical 312 view, may not strictly adhere to it for performance tradeoffs. 313 Specifically, 315 o references to ARP table in the context of asymmetric IRB is a 316 logical view of a forwarding table that maintains an IP to MAC 317 binding entry on a layer 3 interface for both IPv4 and IPv6. 318 These entries are not subject to ARP or ND protocol. For IP to 319 MAC bindings learnt via EVPN, an implementation may choose to 320 import these bindings directly to the respective forwarding table 321 (such as an adjacency/next-hop table) as opposed to importing them 322 to ARP or ND protocol tables. 324 o references to host IP lookup followed by a host MAC lookup in the 325 context of asymmetric IRB MAY be collapsed into a single IP lookup 326 in a hardware implementation. 328 In symmetric IRB as its name implies, the lookup operation is 329 symmetric at both ingress and egress PEs - i.e., both ingress and 330 egress PEs perform lookups on both MAC and IP addresses. The ingress 331 PE performs a MAC lookup followed by an IP lookup and the egress PE 332 performs a IP lookup followed by a MAC lookup as depicted in the 333 following figure. 335 Ingress PE Egress PE 336 +-------------------+ +------------------+ 337 | | | | 338 | +-> IP-VRF ----|---->---|-----> IP-VRF -+ | 339 | | | | | | 340 | BT1 BT2 | | BT3 BT2 | 341 | | | | | | 342 | ^ | | v | 343 | | | | | | 344 +-------------------+ +------------------+ 345 ^ | 346 | | 347 TS1->-+ +->-TS2 348 Figure 2: Symmetric IRB 350 In symmetric IRB as shown in figure-2, the inter-subnet forwarding 351 between two PEs is done between their associated IP-VRFs. Therefore, 352 the tunnel connecting these IP-VRFs can be either IP-only tunnel 353 (e.g., in case of MPLS or GENEVE encapsulation) or Ethernet NVO 354 tunnel (e.g., in case of VxLAN encapsulation). If it is an Ethernet 355 NVO tunnel, the TS1's IP packet is encapsulated in an Ethernet header 356 consisting of ingress and egress PEs MAC addresses - i.e., there is 357 no need for ingress PE to use the destination TS2's MAC address. 358 Therefore, in symmetric IRB, there is no need for the ingress PE to 359 maintain ARP entries for destination TS2's IP and MAC addresses 360 association in its ARP table. Each PE participating in symmetric IRB 361 only maintains ARP entries for locally connected hosts and maintains 362 MAC-VRFs/BTs for only locally configured subnets. 364 In asymmetric IRB, the lookup operation is asymmetric and the ingress 365 PE performs three lookups; whereas the egress PE performs a single 366 lookup - i.e., the ingress PE performs a MAC lookup, followed by an 367 IP lookup, followed by a MAC lookup again; whereas, the egress PE 368 performs just a single MAC lookup as depicted in figure 3 below. 370 Ingress PE Egress PE 371 +-------------------+ +------------------+ 372 | | | | 373 | +-> IP-VRF -> | | IP-VRF | 374 | | | | | | 375 | BT1 BT2 | | BT3 BT2 | 376 | | | | | | | | 377 | | +--|--->----|--------------+ | | 378 | | | | v | 379 +-------------------+ +----------------|-+ 380 ^ | 381 | | 382 TS1->-+ +->-TS2 383 Figure 3: Asymmetric IRB 385 In asymmetric IRB as shown in figure-3, the inter-subnet forwarding 386 between two PEs is done between their associated MAC-VRFs/BTs. 387 Therefore, the MPLS or NVO tunnel used for inter-subnet forwarding 388 MUST be of type Ethernet. Since only MAC lookup is performed at the 389 egress PE (e.g., no IP lookup), the TS1's IP packets need to be 390 encapsulated with the destination TS2's MAC address. In order for 391 ingress PE to perform such encapsulation, it needs to maintain TS2's 392 IP and MAC address association in its ARP table. Furthermore, it 393 needs to maintain destination TS2's MAC address in the corresponding 394 BT even though it may not have any TSes of the corresponding subnet 395 locally attached. In other words, each PE participating in 396 asymmetric IRB MUST maintain ARP entries for remote hosts (hosts 397 connected to other PEs) as well as maintain MAC-VRFs/BTs and IRB 398 interfaces for ALL subnets in an IP VRF including subnets that may 399 not be locally attached. Therefore, careful consideration of PE 400 scale aspects for its ARP table size, its IRB interfaces, number and 401 size of its bridge tables should be given for application of 402 asymmetric IRB. 404 The following subsection defines the control and data planes 405 procedures for symmetric and asymmetric IRB on ingress and egress 406 PEs. The following figure is used in description of these procedures 407 where it shows a single IP-VRF and a number of BTs on each PE for a 408 given tenant. The IP-VRF of the tenant (i.e., IP-VRF1) is connected 409 to each BT via its associated IRB interface. Each BT on a PE is 410 associated with a unique VLAN (e.g., with a BD) where in turn it is 411 associated with a single MAC-VRF in case of VLAN-Based mode or a 412 number of BTs can be associated with a single MAC-VRF in case of 413 VLAN-Aware Bundle mode. Whether the service interface on a PE is 414 VLAN-Based or VLAN-Aware Bundle mode does not impact the IRB 415 operation and procedures. It mainly impacts the setting of Ethernet 416 tag field in EVPN BGP routes as described in section 6 of [RFC7432]. 418 PE 1 +---------+ 419 +-------------+ | | 420 TS1-----| MACx| | | PE2 421 (IP1/M1) |(BT1) | | | +-------------+ 422 TS5-----| \ | | MPLS/ | |MACy (BT3) |-----TS3 423 (IP5/M5) |IPx/Mx \ | | VxLAN/ | | / | (IP3/M3) 424 | (IP-VRF1)|----| NVGRE |---|(IP-VRF1) | 425 | / | | | | \ | 426 TS2-----|(BT2) / | | | | (BT1) |-----TS4 427 (IP2/M2) | | | | | | (IP4/M4) 428 +-------------+ | | +-------------+ 429 | | 430 +---------+ 432 Figure 4: IRB forwarding 434 4.1. IRB Interface and its MAC and IP addresses 436 To support inter-subnet forwarding on a PE, the PE acts as an IP 437 Default Gateway from the perspective of the attached Tenant Systems 438 where default gateway MAC and IP addresses are configured on each IRB 439 interface associated with its subnet and falls into one of the 440 following two options: 442 1. All the PEs for a given tenant subnet use the same anycast 443 default gateway IP and MAC addresses. On each PE, this default 444 gateway IP and MAC addresses correspond to the IRB interface 445 connecting the BT associated with the tenant's VLAN to the 446 corresponding tenant's IP-VRF. 448 2. Each PE for a given tenant subnet uses the same anycast default 449 gateway IP address but its own MAC address. These MAC addresses 450 are aliased to the same anycast default gateway IP address 451 through the use of the Default Gateway extended community as 452 specified in [RFC7432], which is carried in the EVPN MAC/IP 453 Advertisement routes. On each PE, this default gateway IP 454 address along with its associated MAC addresses correspond to the 455 IRB interface connecting the BT associated with the tenant's VLAN 456 to the corresponding tenant's IP-VRF. 458 It is worth noting that if the applications that are running on the 459 TSes are employing or relying on any form of MAC security, then the 460 first option (i.e. using anycast MAC address) should be used to 461 ensure that the applications receive traffic from the same IRB 462 interface MAC address that they are sending to. If the second option 463 is used, then the IRB interface MAC address MUST be the one used in 464 the initial ARP reply or ND Neighbor Advertisement (NA)for that TS. 466 Although both of these options are applicable to both symmetric and 467 asymmetric IRB, the option-1 is recommended because of the ease of 468 anycast MAC address provisioning on not only the IRB interface 469 associated with a given subnet across all the PEs corresponding to 470 that VLAN but also on all IRB interfaces associated with all the 471 tenant's subnets across all the PEs corresponding to all the VLANs 472 for that tenant. Furthermore, it simplifies the operation as there 473 is no need for Default Gateway extended community advertisement and 474 its associated MAC aliasing procedure. Yet another advantage is that 475 following host mobility, the host does not need to refresh the 476 default GW ARP/ND entry. 478 If option-1 is used, an implementation MAY choose to auto-derive the 479 anycast MAC address. If auto-derivation is used, the anycast MAC 480 MUST be auto-derived out of the following ranges (which are defined 481 in [RFC5798]): 483 o Anycast IPv4 IRB case: 00-00-5E-00-01-{VRID} (in hex, in Internet 484 standard bit-order) 486 o Anycast IPv6 IRB case: 00-00-5E-00-02-{VRID} (in hex, in Internet 487 standard bit-order) 489 Where the last octet is generated based on a configurable Virtual 490 Router ID (VRID, range 1-255)). If not explicitly configured, the 491 default value for the VRID octet is '1'. Auto-derivation of the 492 anycast MAC can only be used if there is certainty that the auto- 493 derived MAC does not collide with any customer MAC address. 495 In addition to IP anycast addresses, IRB interfaces can be configured 496 with non-anycast IP addresses for the purpose of OAM (such as 497 traceroute/ping to these interfaces) for both symmetric and 498 asymmetric IRB. These IP addresses need to be distributed as VPN 499 routes when PEs operate in symmetric IRB mode. However, they don't 500 need to be distributed if the PEs are operating in asymmetric IRB 501 mode as the non-anycast IP addresses are configured along with their 502 individual MACs and they get distributed via EVPN route type-2 503 advertisement. 505 For option-1, irrespective of using only the anycast MAC address or 506 both anycast and non-anycast MAC addresses (where the latter one is 507 used for the purpose of OAM) on the same IRB, when a TS sends an ARP 508 request or ND Neighbor Solicitation (NS) to the PE that is attached 509 to, the request is sent for the anycast IP address of the IRB 510 interface associated with the TS's subnet and then the reply will use 511 anycast MAC address (in both Source MAC in the Ethernet header and 512 Sender hardware address in the payload). For example, in figure 4, 513 TS1 is configured with the anycast IPx address as its default gateway 514 IP address and thus when it sends an ARP request for IPx (anycast IP 515 address of the IRB interface for BT1), the PE1 sends an ARP reply 516 with the MACx which is the anycast MAC address of that IRB interface. 517 Traffic routed from IP-VRF1 to TS1 uses the anycast MAC address as 518 source MAC address. 520 5. Symmetric IRB Procedures 522 5.1. Control Plane - Advertising PE 524 When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address of 525 a TS (e.g., via an ARP request or Neighbor Solicitation), it adds the 526 MAC address to the corresponding MAC-VRF/BT of that tenant's subnet 527 and adds the IP address to the IP-VRF for that tenant. Furthermore, 528 it adds this TS's MAC and IP address association to its ARP table or 529 NDP cahce. It then builds an EVPN MAC/IP Advertisement route (type 530 2) as follows and advertises it to other PEs participating in that 531 tenant's VPN. 533 o The Length field of the BGP EVPN NLRI for an EVPN MAC/IP 534 Advertisement route MUST be either 40 (if IPv4 address is carried) 535 or 52 (if IPv6 address is carried). 537 o Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet 538 Tag ID, MAC Address Length, MAC Address, IP Address Length, IP 539 Address, and MPLS Label1 fields MUST be set per [RFC7432] and 540 [RFC8365]. 542 o The MPLS Label2 field is set to either an MPLS label or a VNI 543 corresponding to the tenant's IP-VRF. In case of an MPLS label, 544 this field is encoded as 3 octets, where the high-order 20 bits 545 contain the label value. 547 Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, 548 MAC Address, IP Address Length, and IP Address fields are part of the 549 route key used by BGP to compare routes. The rest of the fields are 550 not part of the route key. 552 This route is advertised along with the following two extended 553 communities: 555 1. Tunnel Type Extended Community 557 2. Router's MAC Extended Community 559 For symmetric IRB mode, Router's MAC EC is needed to carry the PE's 560 overlay MAC address (e.g., inner MAC address in NVO encapsulation) 561 which is used for IP-VRF to IP-VRF communications with Ethernet NVO 562 tunnel. If MPLS or IP-only NVO tunnel is used, then there is no need 563 to send Router's MAC Extended Community along with this route. 565 This route MUST be advertised with two route targets, one 566 corresponding to the MAC-VRF of the tenant's subnet and another 567 corresponding to the tenant's IP-VRF. 569 5.2. Control Plane - Receiving PE 571 When a PE (e.g., PE2 in figure 4 above) receives this EVPN MAC/IP 572 Advertisement route, it performs the following: 574 o Using MAC-VRF Route Target (and Ethernet Tag if different from 575 zero), it identifies the corresponding MAC-VRF (and BT). If the 576 MAC- VRF (and BT) exists (e.g., it is locally configured) then it 577 imports the MAC address into it. Otherwise, it does not import 578 the MAC address. 580 o Using IP-VRF route target, it identifies the corresponding IP-VRF 581 and imports the IP address into it. 583 The inclusion of MPLS label2 field in this route signals to the 584 receiving PE that this route is for symmetric IRB mode and MPLS 585 label2 needs to be installed in forwarding path to identify the 586 corresponding IP-VRF. 588 If the receiving PE receives this route with both the MAC-VRF and IP- 589 VRF route targets but the MAC/IP Advertisement route does not include 590 MPLS label2 field and if the receiving PE supports asymmetric IRB 591 mode, then the receiving PE installs the MAC address in the 592 corresponding MAC-VRF and (IP, MAC) association in the ARP table for 593 that tenant (identified by the corresponding IP-VRF route target). 595 If the receiving PE receives this route with both the MAC-VRF and IP- 596 VRF route targets and if the receiving PE does not support either 597 asymmetric or symmetric IRB modes, then if it has the corresponding 598 MAC-VRF, it only imports the MAC address. Otherwise, if it doesn't 599 have the corresponding MAC-VRF, it must not import this route. 601 If the receiving PE receives this route with both the MAC-VRF and IP- 602 VRF route targets and the MAC/IP Advertisement route includes MPLS 603 label2 field but the receiving PE only supports asymmetric IRB mode, 604 then the receiving PE MUST ignore MPLS label2 field and install the 605 MAC address in the corresponding MAC-VRF and (IP, MAC) association in 606 the ARP table for that tenant (identified by the corresponding IP-VRF 607 route target). 609 5.3. Subnet route advertisement 611 In case of symmetric IRB, a layer-3 subnet and IRB interface 612 corresponding to a MAC-VRF/BT is required to be provisioned at a PE 613 only if that PE has locally attached hosts in that subnet. In order 614 to enable inter-subnet routing across PEs in a deployment where not 615 all subnets are provisioned at all PEs participating in an EVPN IRB 616 instance, PEs MUST advertise local subnet routes as RT-5. These 617 subnet routes are required for bootstrapping host (MAC,IP) learning 618 using gleaning procedures initiated by an inter-subnet data packet. 620 Consider a subnet A that is locally attached to PE1 and subnet B that 621 is locally attached to PE2 and to PE3. Host A in subnet A, that is 622 attached to PE1 initiates a data packet destined to host B in subnet 623 B that is attached to PE3. If host B's (MAC, IP) has not yet been 624 learnt either via a gratuitous ARP OR via a prior gleaning procedure, 625 a new gleaning procedure MUST be triggered for host B's (MAC, IP) to 626 be learnt and advertised across the EVPN network. Since host B's 627 subnet is not local to PE1, an IP lookup for host B at PE1 will not 628 trigger this gleaning procedure for host B's (MAC, IP). Therefore, 629 PE1 MUST learn subnet B's prefix route via RT-5 advertised from PE2 630 and PE3, so it can route the packet to one of the PEs that have 631 subnet B locally attached. Once the packet is received at PE2 OR 632 PE3, and the route lookup yields a glean result, an ARP request is 633 triggered and flooded across the layer-2 overlay. This ARP request 634 would be received and replied to by host B, resulting in host B (MAC, 635 IP) learning at PE3, and its advertisement across the EVPN network. 636 Packets from host A to host B can now be routed directly from PE1 to 637 PE3. Advertisement of local subnet RT-5 for an IP VRF MAY typically 638 be achieved via provisioning connected route redistribution to BGP. 640 5.4. Data Plane - Ingress PE 642 When an Ethernet frame is received by an ingress PE (e.g., PE1 in 643 figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify 644 the associated MAC-VRF/BT and it performs a lookup on the destination 645 MAC address. If the MAC address corresponds to its IRB Interface MAC 646 address, the ingress PE deduces that the packet must be inter-subnet 647 routed. Hence, the ingress PE performs an IP lookup in the 648 associated IP-VRF table. The lookup identifies BGP next hop of 649 egress PE along with the tunnel/encapsulation type and the associated 650 MPLS/VNI values. 652 If the tunnel type is that of MPLS or IP-only NVO tunnel, then TS's 653 IP packet is sent over the tunnel without any Ethernet header. 654 However, if the tunnel type is that of Ethernet NVO tunnel, then an 655 Ethernet header needs to be added to the TS's IP packet. The source 656 MAC address of this inner Ethernet header is set to the ingress PE's 657 router MAC address and the destination MAC address of this inner 658 Ethernet header is set to the egress PE's router MAC address learnt 659 via Router's MAC extended community attached to the route. MPLS VPN 660 label is set to the received label2 in the route. In case of 661 Ethernet NVO tunnel type, VNI may be set one of two ways: 663 o downstream mode: VNI is set to the received label2 in the route 664 which is downstream assigned. 666 o global mode: VNI is set to the received label2 in the route which 667 is domain-wide assigned. This VNI value from received label2 MUST 668 be the same as the locally configured VNI for the IP VRF as all 669 PEs in the NVO MUST be configured with the same IP VRF VNI for 670 this mode of operation. 672 PEs may be configured to operate in one of these two modes depending 673 on the administrative domain boundaries across PEs participating in 674 the NVO, and PE's capability to support downstream VNI mode. 676 In case of NVO tunnel encapsulation, the outer source and destination 677 IP addresses are set to the ingress and egress PE BGP next-hop IP 678 addresses respectively. 680 5.5. Data Plane - Egress PE 682 When the tenant's MPLS or NVO encapsulated packet is received over an 683 MPLS or NVO tunnel by the egress PE, the egress PE removes NVO tunnel 684 encapsulation and uses the VPN MPLS label (for MPLS encapsulation) or 685 VNI (for NVO encapsulation) to identify the IP-VRF in which IP lookup 686 needs to be performed. If the VPN MPLS label or VNI identifies a 687 MAC- VRF instead of an IP-VRF, then the procedures in section 6.4 for 688 asymmetric IRB are executed. 690 The lookup in the IP-VRF identifies a local adjacency to the IRB 691 interface associated with the egress subnet's MAC-VRF/BT. 693 The egress PE gets the destination TS's MAC address for that TS's IP 694 address from its ARP table or NDP cache, it encapsulates the packet 695 with that destination MAC address and a source MAC address 696 corresponding to that IRB interface and sends the packet to its 697 destination subnet MAC-VRF/BT. 699 The destination MAC address lookup in the MAC-VRF/BT results in local 700 adjacency (e.g., local interface) over which the Ethernet frame is 701 sent on. 703 6. Asymmetric IRB Procedures 705 6.1. Control Plane - Advertising PE 707 When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address of 708 an attached TS (e.g., via an ARP request or ND Neighbor 709 Solicitation), it populates its MAC-VRF/BT, IP-VRF, and ARP table or 710 NDP cache just as in the case for symmetric IRB. It then builds an 711 EVPN MAC/IP Advertisement route (type 2) as follows and advertises it 712 to other PEs participating in that tenant's VPN. 714 o The Length field of the BGP EVPN NLRI for an EVPN MAC/IP 715 Advertisement route MUST be either 37 (if IPv4 address is carried) 716 or 49 (if IPv6 address is carried). 718 o Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet 719 Tag ID, MAC Address Length, MAC Address, IP Address Length, IP 720 Address, and MPLS Label1 fields MUST be set per [RFC7432] and 721 [RFC8365]. 723 o The MPLS Label2 field MUST NOT be included in this route. 725 Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, 726 MAC Address, IP Address Length, and IP Address fields are part of the 727 route key used by BGP to compare routes. The rest of the fields are 728 not part of the route key. 730 This route is advertised along with the following extended community: 732 o Tunnel Type Extended Community 734 For asymmetric IRB mode, Router's MAC EC is not needed because 735 forwarding is performed using destination TS's MAC address which is 736 carried in this EVPN route type-2 advertisement. 738 This route MUST always be advertised with the MAC-VRF route target. 739 It MAY also be advertised with a second route target corresponding to 740 the IP-VRF. 742 6.2. Control Plane - Receiving PE 744 When a PE (e.g., PE2 in figure 4 above) receives this EVPN MAC/IP 745 Advertisement route, it performs the following: 747 o Using MAC-VRF route target, it identifies the corresponding MAC- 748 VRF and imports the MAC address into it. For asymmetric IRB mode, 749 it is assumed that all PEs participating in a tenant's VPN are 750 configured with all subnets (i.e., all VLANs) and corresponding 751 MAC-VRFs/BTs even if there are no locally attached TSes for some 752 of these subnets. The reason for this is because ingress PE needs 753 to do forwarding based on destination TS's MAC address and perform 754 NVO tunnel encapsulation as a property of a lookup in MAC-VRF/BT. 756 o If only MAC-VRF route target is used, then the receiving PE uses 757 the MAC-VRF route target to identify the corresponding IP-VRF -- 758 i.e., many MAC-VRF route targets map to the same IP-VRF for a 759 given tenant. In this case, MAC-VRF may be used by the receiving 760 PE to identify the corresponding IP VRF via the IRB interface 761 associated with the subnet MAC-VRF/BT. This would equivalent to 762 how ARP table or NDP cache entries are typically mapped to IRB 763 interface of an IP VRF for installing attached host routes in an 764 IP VRF. Since in asymmetric IRB mode, each PE is configured with 765 all VLANs of a tenant, indirect import to IP VRF via the 766 corresponding MAC-VRF route target is a viable alternative. 768 o Using MAC-VRF route target, the receiving PE identifies the 769 corresponding ARP table or NDP cache for the tenant and it adds an 770 entry to the ARP table or NDP cache for the TS's MAC and IP 771 address association. It should be noted that the tenant's ARP 772 table or NDP cache at the receiving PE is identified by all the 773 MAC- VRF route targets for that tenant. 775 o If IP-VRF route target is included, it may be used to import the 776 route to IP-VRF. If IP-VRF route-target is not included, MAC-VRF 777 is used to derive corresponding IP-VRF for import, as explained in 778 prior section. In both cases, IP-VRF route is installed with the 779 TS MAC binding included in the received route. 781 If the receiving PE receives the MAC/IP Advertisement route with MPLS 782 label2 field but the receiving PE only supports asymmetric IRB mode, 783 then the receiving PE MUST ignore MPLS label2 field and install the 784 MAC address in the corresponding MAC-VRF and (IP, MAC) association in 785 the ARP table or NDP cache for that tenant (with IRB interface 786 identified by the MAC-VRF). 788 If the receiving PE receives the MAC/IP Advertisement route with MPLS 789 label2 field and it can support symmetric IRB mode, then it should 790 use the MAC-VRF route target to identify its corresponding MAC-VRF 791 table and import the MAC address. It should use the IP-VRF route 792 target to identify the corresponding IP-VRF table and import the IP 793 address, as specified in symmetric IRB handling. It MUST NOT import 794 (IP, MAC) association into its ARP table or NDP cache. 796 6.3. Data Plane - Ingress PE 798 When an Ethernet frame is received by an ingress PE (e.g., PE1 in 799 figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify 800 the associated MAC-VRF/BT and it performs a lookup on the destination 801 MAC address. If the MAC address corresponds to its IRB Interface MAC 802 address, the ingress PE deduces that the packet must be inter-subnet 803 routed. Hence, the ingress PE performs an IP lookup in the 804 associated IP-VRF table. The lookup identifies a local adjacency to 805 the IRB interface associated with the egress subnet's MAC-VRF/BT. 807 The ingress PE gets the destination TS's MAC address for that TS's IP 808 address from its ARP table or NDP cache, it encapsulates the packet 809 with that destination MAC address and a source MAC address 810 corresponding to that IRB interface and sends the packet to its 811 destination subnet MAC-VRF/BT. 813 The destination MAC address lookup in the MAC-VRF/BT results in BGP 814 next hop address of egress PE along with label-1 (L2 VPN MPLS label 815 or VNI). The ingress PE encapsulates the packet using Ethernet NVO 816 tunnel of the choice (e.g., VxLAN or GENEVE) and sends the packet to 817 the egress PE. Since the packet forwarding is between ingress PE's 818 MAC-VRF/BT and egress PE's MAC-VRF/BT, the packet encapsulation 819 procedures follows that of [RFC7432] for MPLS and [RFC8365] for VxLAN 820 encapsulations. 822 6.4. Data Plane - Egress PE 824 When a tenant's Ethernet frame is received over an NVO tunnel by the 825 egress PE, the egress PE removes NVO tunnel encapsulation and uses 826 the VPN MPLS label (for MPLS encapsulation) or VNI (for NVO 827 encapsulation) to identify the MAC-VRF/BT in which MAC lookup needs 828 to be performed. 830 The MAC lookup results in local adjacency (e.g., local interface) 831 over which the packet needs to get sent. 833 Note that the forwarding behavior on the egress PE is the same as 834 EVPN intra-subnet forwarding described in [RFC7432] for MPLS and 835 [RFC8365] for NVO networks. In other words, all the packet 836 processing associated with the inter-subnet forwarding semantics is 837 confined to the ingress PE for asymmetric IRB mode. 839 It should also be noted that [RFC7432] provides different level of 840 granularity for the EVPN label. Besides identifying bridge domain 841 table, it can be used to identify the egress interface or a 842 destination MAC address on that interface. If EVPN label is used for 843 egress interface or individual MAC address identification, then no 844 MAC lookup is needed in the egress PE for MPLS encapsulation and the 845 packet can be directly forwarded to the egress interface just based 846 on EVPN label lookup. 848 7. Mobility Procedure 850 When a TS moves from one NVE (aka source NVE) to another NVE (aka 851 target NVE), it is important that the MAC mobility procedures are 852 properly executed and the corresponding MAC-VRF and IP-VRF tables on 853 all participating NVEs are updated. [RFC7432] describes the MAC 854 mobility procedures for L2-only services for both single-homed TS and 855 multi-homed TS. This section describes the incremental procedures 856 and BGP Extended Communities needed to handle the MAC mobility for 857 IRB. In order to place the emphasis on the differences between 858 L2-only and IRB use cases, the incremental procedure is described for 859 single-homed TS with the expectation that the additional steps needed 860 for multi-homed TS, can be extended per section 15 of [RFC7432]. 861 This section describes mobility procedures for both symmetric and 862 asymmetric IRB. Although the language used in this section is for 863 IPv4 ARP, it equally applies to IPv6 ND. 865 When a TS moves from a source NVE to a target NVE, it can behave in 866 one of the following three ways: 868 1. TS initiates an ARP request upon a move to the target NVE 870 2. TS sends data packet without first initiating an ARP request to 871 the target NVE 873 3. TS is a silent host and neither initiates an ARP request nor 874 sends any packets 876 Depending on the expexted TS's behavior, an NVE needs to handle at 877 least the first bullet and should be able to handle the 2nd and the 878 3rd bullet. The following subsections describe the procedures for 879 each of them where it is assumed that the MAC and IP addresses of a 880 TS have one-to-one relationship (i.e., there is one IP address per 881 MAC address and vice versa). If there is many- to-one relationship 882 such that there are many host IP addresses correspond to a single 883 host MAC address or there are many host MAC addresses correspond to a 884 single IP address, then to detect host mobility, the procedures in 885 [I-D.ietf-bess-evpn-irb-extended-mobility] must be exercised followed 886 by the procedures described below. 888 7.1. Initiating a gratutious ARP upon a Move 890 In this scenario when a TS moves from a source NVE to a target NVE, 891 the TS initiates a gratuitous ARP upon the move to the target NVE. 893 The target NVE upon receiving this ARP message, updates its MAC-VRF, 894 IP-VRF, and ARP table with the host MAC, IP, and local adjacency 895 information (e.g., local interface). 897 Since this NVE has previously learned the same MAC and IP addresses 898 from the source NVE, it recognizes that there has been a MAC move and 899 it initiates MAC mobility procedures per [RFC7432] by advertising an 900 EVPN MAC/IP Advertisement route with both the MAC and IP addresses 901 filled in (per sections 5.1 and 6.1) along with MAC Mobility Extended 902 Community with the sequence number incremented by one. The target 903 NVE also exercises the MAC duplication detection procedure in section 904 15.1 of [RFC7432]. 906 The source NVE upon receiving this MAC/IP Advertisement route, 907 realizes that the MAC has moved to the target NVE. It updates its 908 MAC-VRF and IP-VRF table accordingly with the adjacency information 909 of the target NVE. In case of the asymmetric IRB, the source NVE 910 also updates its ARP table with the received adjacency information 911 and in case of the symmetric IRB, the source NVE removes the entry 912 associated with the received (MAC, IP) from its local ARP table. It 913 then withdraws its EVPN MAC/IP Advertisement route. Furthermore, it 914 sends an ARP probe locally to ensure that the MAC is gone. If an ARP 915 response is received, the source NVE updates its ARP entry for that 916 (IP, MAC) and re-advertises an EVPN MAC/IP Advertisement route for 917 that (IP, MAC) along with MAC Mobility Extended Community with the 918 sequence number incremented by one. The source NVE also exercises 919 the MAC duplication detection procedure in section 15.1 of [RFC7432]. 921 All other remote NVE devices upon receiving the MAC/IP Advertisement 922 route with MAC Mobility extended community compare the sequence 923 number in this advertisement with the one previously received. If 924 the new sequence number is greater than the old one, then they update 925 the MAC/IP addresses of the TS in their corresponding MAC-VRF and IP- 926 VRF tables to point to the target NVE. Furthermore, upon receiving 927 the MAC/IP withdraw for the TS from the source NVE, these remote PEs 928 perform the cleanups for their BGP tables. 930 7.2. Sending Data Traffic without an ARP Request 932 In this scenario when a TS moves from a source NVE to a target NVE, 933 the TS starts sending data traffic without first initiating an ARP 934 request. 936 The target NVE upon receiving the first data packet, learns the MAC 937 address of the TS in data plane and updates its MAC-VRF table with 938 the MAC address and the local adjacency information (e.g., local 939 interface) accordingly. The target NVE realizes that there has been 940 a MAC move because the same MAC address has been learned remotely 941 from the source NVE. 943 If EVPN-IRB NVEs are configured to advertise MAC-only routes in 944 addition to MAC-and-IP EVPN routes, then the following steps are 945 taken: 947 o The target NVE upon learning this MAC address in data-plane, 948 updates this MAC address entry in the corresponding MAC-VRF with 949 the local adjacency information (e.g., local interface). It also 950 recognizes that this MAC has moved and initiates MAC mobility 951 procedures per [RFC7432] by advertising an EVPN MAC/IP 952 Advertisement route with only the MAC address filled in along with 953 MAC Mobility Extended Community with the sequence number 954 incremented by one. 956 o The source NVE upon receiving this MAC/IP Advertisement route, 957 realizes that the MAC has moved to the new NVE. It updates its 958 MAC-VRF table with the adjacency information for that MAC address 959 to point to the target NVE and withdraws its EVPN MAC/IP 960 Advertisement route that has only the MAC address (if it has 961 advertised such route previously). Furthermore, it searches for 962 the corresponding MAC-IP entry and sends an ARP probe for this 963 (MAC,IP) pair. The ARP request message is sent both locally to 964 all attached TSes in that subnet as well as it is sent to other 965 NVEs participating in that subnet including the target NVE. Note 966 that the PE needs to maintain a correlation between MAC and MAC-IP 967 route entries in the MAC-VRF to accomplish this. 969 o The target NVE passes the ARP request to its locally attached TSes 970 and when it receives the ARP response, it updates its IP-VRF and 971 ARP table with the host (MAC, IP) information. It also sends an 972 EVPN MAC/IP Advertisement route with both the MAC and IP addresses 973 filled in along with MAC Mobility Extended Community with the 974 sequence number set to the same value as the one for MAC-only 975 advertisement route it sent previously. 977 o When the source NVE receives the EVPN MAC/IP Advertisement route, 978 it updates its IP-VRF table with the new adjacency information 979 (pointing to the target NVE). In case of the asymmetric IRB, the 980 source NVE also updates its ARP table with the received adjacency 981 information and in case of the symmetric IRB, the source NVE 982 removes the entry associated with the received (MAC, IP) from its 983 local ARP table. Furthermore, it withdraws its previously 984 advertised EVPN MAC/IP route with both the MAC and IP address 985 fields filled in. 987 o All other remote NVE devices upon receiving the MAC/IP 988 advertisement route with MAC Mobility extended community compare 989 the sequence number in this advertisement with the one previously 990 received. If the new sequence number is greater than the old one, 991 then they update the MAC/IP addresses of the TS in their 992 corresponding MAC-VRF, IP-VRF, and ARP tables (in case of 993 asymmetric IRB) to point to the new NVE. Furthermore, upon 994 receiving the MAC/IP withdraw for the TS from the old NVE, these 995 remote PEs perform the cleanups for their BGP tables. 997 If EVPN-IRB NVEs are configured not to advertise MAC-only routes, 998 then upon receiving the first data packet, it learns the MAC address 999 of the TS and updates the MAC entry in the corresponding MAC-VRF 1000 table with the local adjacency information (e.g., local interface). 1001 It also realizes that there has been a MAC move because the same MAC 1002 address has been learned remotely from the source NVE. It uses the 1003 local MAC route to find the corresponding local MAC-IP route, and 1004 sends a unicast ARP request to the host and when receiving an ARP 1005 response, it follows the procedure outlined in section 7.1. In the 1006 prior case, where MAC-only routes are also advertised, this procedure 1007 of triggering a unicast ARP probe at the target PE MAY also be used 1008 in addition to the source PE broadcast ARP probing procedure 1009 described earlier for better convergence. 1011 7.3. Silent Host 1013 In this scenario when a TS moves from a source NVE to a target NVE, 1014 the TS is silent and it neither initiates an ARP request nor it sends 1015 any data traffic. Therefore, neither the target nor the source NVEs 1016 are aware of the MAC move. 1018 On the source NVE, an age-out timer (for the silent host that has 1019 moved) is used to trigger an ARP probe. This age-out timer can be 1020 either ARP timer or MAC age-out timer and this is an implementation 1021 choice. The ARP request gets sent both locally to all the attached 1022 TSes on that subnet as well as it gets sent to all the remote NVEs 1023 (including the target NVE) participating in that subnet. The source 1024 NVE also withdraw the EVPN MAC/IP Advertisement route with only the 1025 MAC address (if it has previously advertised such a route). 1027 The target NVE passes the ARP request to its locally attached TSes 1028 and when it receives the ARP response, it updates its MAC-VRF, IP- 1029 VRF, and ARP table with the host (MAC, IP) and local adjacency 1030 information (e.g., local interface). It also sends an EVPN MAC/IP 1031 advertisement route with both the MAC and IP address fields filled in 1032 along with MAC Mobility Extended Community with the sequence number 1033 incremented by one. 1035 When the source NVE receives the EVPN MAC/IP Advertisement route, it 1036 updates its IP-VRF table with the new adjacency information (pointing 1037 to the target NVE). In case of the asymmetric IRB, the source NVE 1038 also updates its ARP table with the received adjacency information 1039 and in case of the symmetric IRB, the source NVE removes the entry 1040 associated with the received (MAC, IP) from its local ARP table. 1041 Furthermore, it withdraws its previously advertised EVPN MAC/IP route 1042 with both the MAC and IP address fields filled in. 1044 All other remote NVE devices upon receiving the MAC/IP Advertisement 1045 route route with MAC Mobility extended community compare the sequence 1046 number in this advertisement with the one previously received. If 1047 the new sequence number is greater than the old one, then they update 1048 the MAC/IP addresses of the TS in their corresponding MAC-VRF, IP- 1049 VRF, and ARP (in case of asymmetric IRB) tables to point to the new 1050 NVE. Furthermore, upon receiving the MAC/IP withdraw for the TS from 1051 the old NVE, these remote PEs perform the cleanups for their BGP 1052 tables. 1054 8. BGP Encoding 1056 This document defines one new BGP Extended Community for EVPN. 1058 8.1. Router's MAC Extended Community 1060 A new EVPN BGP Extended Community called Router's MAC is introduced 1061 here. This new extended community is a transitive extended community 1062 with the Type field of 0x06 (EVPN) and the Sub-Type of 0x03. It may 1063 be advertised along with Encapsulation Extended Community defined in 1064 section 4.1 of [I-D.ietf-idr-tunnel-encaps]. 1066 The Router's MAC Extended Community is encoded as an 8-octet value as 1067 follows: 1069 0 1 2 3 1070 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1071 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1072 | Type=0x06 | Sub-Type=0x03 | Router's MAC | 1073 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1074 | Router's MAC Cont'd | 1075 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1077 Figure 5: Router's MAC Extended Community 1079 This extended community is used to carry the PE's MAC address for 1080 symmetric IRB scenarios and it is sent with RT-2. 1082 9. Operational Models for Symmetric Inter-Subnet Forwarding 1084 The following sections describe two main symmetric IRB forwarding 1085 scenarios (within a DC -- i.e., intra-DC) along with the 1086 corresponding procedures. In the following scenarios, without loss 1087 of generality, it is assumed that a given tenant is represented by a 1088 single IP-VPN instance. Therefore, on a given PE, a tenant is 1089 represented by a single IP-VRF table and one or more MAC-VRF tables. 1091 9.1. IRB forwarding on NVEs for Tenant Systems 1093 This section covers the symmetric IRB procedures for the scenario 1094 where each Tenant System (TS) is attached to one or more NVEs and its 1095 host IP and MAC addresses are learned by the attached NVEs and are 1096 distributed to all other NVEs that are interested in participating in 1097 both intra-subnet and inter-subnet communications with that TS. 1099 In this scenario, without loss of generality, it is assumed that NVEs 1100 operate in VLAN-based service interface mode with one Bridge 1101 Table (BT) per MAC-VRF. Thus, for a given tenant, an NVE has one 1102 MAC-VRF for each tenant subnet (e.g., each VLAN) that is configured 1103 for extension via VxLAN or GENEVE encapsulation. In case of VLAN- 1104 aware bundling, then each MAC-VRF consists of multiple Bridge Tables 1105 (e.g., one BT per VLAN). The MAC-VRFs on an NVE for a given tenant 1106 are associated with an IP-VRF corresponding to that tenant (or IP-VPN 1107 instance) via their IRB interfaces. 1109 Since VxLAN and GENEVE encapsulations require inner Ethernet header 1110 (inner MAC SA/DA), and since for inter-subnet traffic, TS MAC address 1111 cannot be used, the ingress NVE's MAC address is used as inner MAC 1112 SA. The NVE's MAC address is the device MAC address and it is common 1113 across all MAC-VRFs and IP-VRFs. This MAC address is advertised 1114 using the new EVPN Router's MAC Extended Community (section 8.1). 1116 Figure 6 below illustrates this scenario where a given tenant (e.g., 1117 an IP-VPN instance) has three subnets represented by MAC-VRF1, MAC- 1118 VRF2, and MAC-VRF3 across two NVEs. There are five TSes that are 1119 associated with these three MAC-VRFs -- i.e., TS1, TS4, and TS5 are 1120 on the same subnet (e.g., same MAC-VRF/VLAN). TS1 and TS5 are 1121 associated with MAC-VRF1 on NVE1, while TS4 is associated with MAC- 1122 VRF1 on NVE2. TS2 is associated with MAC-VRF2 on NVE1, and TS3 is 1123 associated with MAC-VRF3 on NVE2. MAC-VRF1 and MAC-VRF2 on NVE1 are 1124 in turn associated with IP-VRF1 on NVE1 and MAC-VRF1 and MAC-VRF3 on 1125 NVE2 are associated with IP-VRF1 on NVE2. When TS1, TS5, and TS4 1126 exchange traffic with each other, only L2 forwarding (bridging) part 1127 of the IRB solution is exercised because all these TSes belong to the 1128 same subnet. However, when TS1 wants to exchange traffic with TS2 or 1129 TS3 which belong to different subnets, both bridging and routing 1130 parts of the IRB solution are exercised. The following subsections 1131 describe the control and data planes operations for this IRB scenario 1132 in details. 1134 NVE1 +---------+ 1135 +-------------+ | | 1136 TS1-----| MACx| | | NVE2 1137 (IP1/M1) |(MAC- | | | +-------------+ 1138 TS5-----| VRF1)\ | | MPLS/ | |MACy (MAC- |-----TS3 1139 (IP5/M5) | \ | | VxLAN/ | | / VRF3) | (IP3/M3) 1140 | (IP-VRF1)|----| NVGRE |---|(IP-VRF1) | 1141 | / | | | | \ | 1142 TS2-----|(MAC- / | | | | (MAC- |-----TS4 1143 (IP2/M2) | VRF2) | | | | VRF1) | (IP4/M4) 1144 +-------------+ | | +-------------+ 1145 | | 1146 +---------+ 1148 Figure 6: IRB forwarding on NVEs for Tenant Systems 1150 9.1.1. Control Plane Operation 1152 Each NVE advertises a MAC/IP Advertisement route (i.e., Route Type 2) 1153 for each of its TSes with the following field set: 1155 o RD and ESI per [RFC7432] 1157 o Ethernet Tag = 0; assuming VLAN-based service 1159 o MAC Address Length = 48 1161 o MAC Address = Mi ; where i = 1,2,3,4, or 5 in the above example 1163 o IP Address Length = 32 or 128 1165 o IP Address = IPi ; where i = 1,2,3,4, or 5 in the above example 1167 o Label-1 = MPLS Label or VNI corresponding to MAC-VRF 1169 o Label-2 = MPLS Label or VNI corresponding to IP-VRF 1171 Each NVE advertises an RT-2 route with two Route Targets (one 1172 corresponding to its MAC-VRF and the other corresponding to its IP- 1173 VRF. Furthermore, the RT-2 is advertised with two BGP Extended 1174 Communities. The first BGP Extended Community identifies the tunnel 1175 type per section 4.5 of [I-D.ietf-idr-tunnel-encaps] and the second 1176 BGP Extended Community includes the MAC address of the NVE (e.g., 1177 MACx for NVE1 or MACy for NVE2) as defined in section 8.1. This 1178 second Extended Community (for the MAC address of NVE) is only 1179 required when Ethernet NVO tunnel type is used. If IP NVO tunnel 1180 type is used, then there is no need to send this second Extended 1181 Community. It should be noted that IP NVO tunnel type is only 1182 applicable to symmetric IRB procedures. 1184 Upon receiving this advertisement, the receiving NVE performs the 1185 following: 1187 o It uses Route Targets corresponding to its MAC-VRF and IP-VRF for 1188 identifying these tables and subsequently importing the MAC and IP 1189 addresses into them respectively. 1191 o It imports the MAC address from MAC/IP Advertisement route into 1192 the MAC-VRF with BGP Next Hop address as underlay tunnel 1193 destination address (e.g., VTEP DA for VxLAN encapsulation) and 1194 Label-1 as VNI for VxLAN encapsulation or EVPN label for MPLS 1195 encapsulation. 1197 o If the route carries the new Router's MAC Extended Community, and 1198 if the receiving NVE uses Ethernet NVO tunnel, then the receiving 1199 NVE imports the IP address into IP-VRF with NVE's MAC address 1200 (from the new Router's MAC Extended Community) as inner MAC DA and 1201 BGP Next Hop address as underlay tunnel destination address, VTEP 1202 DA for VxLAN encapsulation and Label-2 as IP-VPN VNI for VxLAN 1203 encapsulation. 1205 o If the receiving NVE uses MPLS encapsulation, then the receiving 1206 NVE imports the IP address into IP-VRF with BGP Next Hop address 1207 as underlay tunnel destination address, and Label-2 as IP-VPN 1208 label for MPLS encapsulation. 1210 If the receiving NVE receives a RT-2 with only Label-1 and only a 1211 single Route Target corresponding to IP-VRF, or if it receives a RT-2 1212 with only a single Route Target corresponding to MAC-VRF but with 1213 both Label-1 and Label-2, or if it receives a RT-2 with MAC Address 1214 Length of zero, then it MUST treat the route as withdraw [RFC7606] 1215 and SHOULD log an error message. 1217 9.1.2. Data Plane Operation 1219 The following description of the data-plane operation describes just 1220 the logical functions and the actual implementation may differ. Lets 1221 consider data-plane operation when TS1 in subnet-1 (MAC-VRF1) on NVE1 1222 wants to send traffic to TS3 in subnet-3 (MAC-VRF3) on NVE2. 1224 o NVE1 receives a packet with MAC DA corresponding to the MAC-VRF1 1225 IRB interface on NVE1 (the interface between MAC-VRF1 and IP- 1226 VRF1), and VLAN-tag corresponding to MAC-VRF1. 1228 o Upon receiving the packet, the NVE1 uses VLAN-tag to identify the 1229 MAC-VRF1. It then looks up the MAC DA and forwards the frame to 1230 its IRB interface. 1232 o The Ethernet header of the packet is stripped and the packet is 1233 fed to the IP-VRF where IP lookup is performed on the destination 1234 IP address. This lookup yields the outgoing NVO tunnel and the 1235 required encapsulation. If the encapsulation is for Ethernet NVO 1236 tunnel, then it includes the egress NVE's MAC address as inner MAC 1237 DA, the egress NVE's IP address (e.g., BGP Next Hop address) as 1238 the VTEP DA, and the VPN-ID as the VNI. The inner MAC SA and VTEP 1239 SA are set to NVE's MAC and IP addresses respectively. If it is a 1240 MPLS encapsulation, then corresponding EVPN and LSP labels are 1241 added to the packet. The packet is then forwarded to the egress 1242 NVE. 1244 o On the egress NVE, if the packet arrives on Ethernet NVO tunnel 1245 (e.g., it is VxLAN encapsulated), then the NVO tunnel header is 1246 removed. Since the inner MAC DA is the egress NVE's MAC address, 1247 the egress NVE knows that it needs to perform an IP lookup. It 1248 uses the VNI to identify the IP-VRF table. If the packet is MPLS 1249 encapsulated, then the EVPN label lookup identifies the IP-VRF 1250 table. Next, an IP lookup is performed for the destination TS 1251 (TS3) which results in access-facing IRB interface over which the 1252 packet is sent. Before sending the packet over this interface, 1253 the ARP table is consulted to get the destination TS's MAC 1254 address. 1256 o The IP packet is encapsulated with an Ethernet header with MAC SA 1257 set to that of IRB interface MAC address (i.e, IRB interface 1258 between MAC-VRF3 and IP-VRF1 on NVE2) and MAC DA set to that of 1259 destination TS (TS3) MAC address. The packet is sent to the 1260 corresponding MAC-VRF (i.e., MAC-VRF3) and after a lookup of MAC 1261 DA, is forwarded to the destination TS (TS3) over the 1262 corresponding interface. 1264 In this symmetric IRB scenario, inter-subnet traffic between NVEs 1265 will always use the IP-VRF VNI/MPLS label. For instance, traffic 1266 from TS2 to TS4 will be encapsulated by NVE1 using NVE2's IP-VRF VNI/ 1267 MPLS label, as long as TS4's host IP is present in NVE1's IP-VRF. 1269 9.2. IRB forwarding on NVEs for Subnets behind Tenant Systems 1271 This section covers the symmetric IRB procedures for the scenario 1272 where some Tenant Systems (TSes) support one or more subnets and 1273 these TSes are associated with one or more NVEs. Therefore, besides 1274 the advertisement of MAC/IP addresses for each TS which can be multi- 1275 homed with All-Active redundancy mode, the associated NVE needs to 1276 also advertise the subnets statically configured on each TS. 1278 The main difference between this solution and the previous one is the 1279 additional advertisement corresponding to each subnet. These subnet 1280 advertisements are accomplished using EVPN IP Prefix route defined in 1281 [I-D.ietf-bess-evpn-prefix-advertisement]. These subnet prefixes are 1282 advertised with the IP address of their associated TS (which is in 1283 overlay address space) as their next hop. The receiving NVEs perform 1284 recursive route resolution to resolve the subnet prefix with its 1285 advertising NVE so that they know which NVE to forward the packets to 1286 when they are destined for that subnet prefix. 1288 The advantage of this recursive route resolution is that when a TS 1289 moves from one NVE to another, there is no need to re-advertise any 1290 of the subnet prefixes for that TS. All it is needed is to advertise 1291 the IP/MAC addresses associated with the TS itself and exercise MAC 1292 mobility procedures for that TS. The recursive route resolution 1293 automatically takes care of the updates for the subnet prefixes of 1294 that TS. 1296 Figure below illustrates this scenario where a given tenant (e.g., an 1297 IP-VPN service) has three subnets represented by MAC-VRF1, MAC-VRF2, 1298 and MAC-VRF3 across two NVEs. There are four TSes associated with 1299 these three MAC-VRFs -- i.e., TS1 is connected to MAC-VRF1 on NVE1, 1300 TS2 is connected to MAC-VRF2 on NVE1, TS3 is connected to MAC- VRF3 1301 on NVE2, and TS4 is connected to MAC-VRF1 on NVE2. TS1 has two 1302 subnet prefixes (SN1 and SN2) and TS3 has a single subnet prefix, 1303 SN3. The MAC-VRFs on each NVE are associated with their 1304 corresponding IP-VRF using their IRB interfaces. When TS4 and TS1 1305 exchange intra- subnet traffic, only L2 forwarding (bridging) part of 1306 the IRB solution is used (i.e., the traffic only goes through their 1307 MAC- VRFs); however, when TS3 wants to forward traffic to SN1 or SN2 1308 sitting behind TS1 (inter-subnet traffic), then both bridging and 1309 routing parts of the IRB solution are exercised (i.e., the traffic 1310 goes through the corresponding MAC-VRFs and IP-VRFs). The following 1311 subsections describe the control and data planes operations for this 1312 IRB scenario in details. 1314 NVE1 +----------+ 1315 SN1--+ +-------------+ | | 1316 |--TS1-----|(MAC- \ | | | 1317 SN2--+ IP1/M1 | VRF1) \ | | | 1318 | (IP-VRF)|---| | 1319 | / | | | 1320 TS2-----|(MAC- / | | MPLS/ | 1321 IP2/M2 | VRF2) | | VxLAN/ | 1322 +-------------+ | NVGRE | 1323 +-------------+ | | 1324 SN3--+--TS3-----|(MAC-\ | | | 1325 IP3/M3 | VRF3)\ | | | 1326 | (IP-VRF)|---| | 1327 | / | | | 1328 TS4-----|(MAC- / | | | 1329 IP4/M4 | VRF1) | | | 1330 +-------------+ +----------+ 1331 NVE2 1333 Figure 7: IRB forwarding on NVEs for subnets behind TSes 1335 9.2.1. Control Plane Operation 1337 Each NVE advertises a Route Type-5 (RT-5, IP Prefix Route defined in 1338 [I-D.ietf-bess-evpn-prefix-advertisement]) for each of its subnet 1339 prefixes with the IP address of its TS as the next hop (gateway 1340 address field) as follow: 1342 o RD associated with the IP-VRF 1344 o ESI = 0 1346 o Ethernet Tag = 0; 1348 o IP Prefix Length = 0 to 32 or 0 to 128 1350 o IP Prefix = SNi 1352 o Gateway Address = IPi; IP address of TS 1354 o MPLS Label = 0 1355 This RT-5 is advertised with one or more Route Targets associated 1356 with the IP-VRF from which the route is originated. 1358 Each NVE also advertises an RT-2 (MAC/IP Advertisement Route) along 1359 with their associated Route Targets and Extended Communities for each 1360 of its TSes exactly as described in section 9.1.1. 1362 Upon receiving the RT-5 advertisement, the receiving NVE performs the 1363 following: 1365 o It uses the Route Target to identify the corresponding IP-VRF 1367 o It imports the IP prefix into its corresponding IP-VRF that is 1368 configured with an import RT that is one of the RTs being carried 1369 by the RT-5 route along with the IP address of the associated TS 1370 as its next hop. 1372 When receiving the RT-2 advertisement, the receiving NVE imports MAC/ 1373 IP addresses of the TS into the corresponding MAC-VRF and IP-VRF per 1374 section 9.1.1. When both routes exist, recursive route resolution is 1375 performed to resolve the IP prefix (received in RT-5) to its 1376 corresponding NVE's IP address (e.g., its BGP next hop). BGP next 1377 hop will be used as underlay tunnel destination address (e.g., VTEP 1378 DA for VxLAN encapsulation) and Router's MAC will be used as inner 1379 MAC for VxLAN encapsulation. 1381 9.2.2. Data Plane Operation 1383 The following description of the data-plane operation describes just 1384 the logical functions and the actual implementation may differ. Lets 1385 consider data-plane operation when a host on SN1 sitting behind TS1 1386 wants to send traffic to a host sitting behind SN3 behind TS3. 1388 o TS1 send a packet with MAC DA corresponding to the MAC-VRF1 IRB 1389 interface of NVE1, and VLAN-tag corresponding to MAC-VRF1. 1391 o Upon receiving the packet, the ingress NVE1 uses VLAN-tag to 1392 identify the MAC-VRF1. It then looks up the MAC DA and forwards 1393 the frame to its IRB interface just like section 9.1.1. 1395 o The Ethernet header of the packet is stripped and the packet is 1396 fed to the IP-VRF; where, IP lookup is performed on the 1397 destination address. This lookup yields the fields needed for 1398 VxLAN encapsulation with NVE2's MAC address as the inner MAC DA, 1399 NVE'2 IP address as the VTEP DA, and the VNI. MAC SA is set to 1400 NVE1's MAC address and VTEP SA is set to NVE1's IP address. 1402 o The packet is then encapsulated with the proper header based on 1403 the above info and is forwarded to the egress NVE (NVE2). 1405 o On the egress NVE (NVE2), assuming the packet is VxLAN 1406 encapsulated, the VxLAN and the inner Ethernet headers are removed 1407 and the resultant IP packet is fed to the IP-VRF associated with 1408 that the VNI. 1410 o Next, a lookup is performed based on IP DA (which is in SN3) in 1411 the associated IP-VRF of NVE2. The IP lookup yields the access- 1412 facing IRB interface over which the packet needs to be sent. 1413 Before sending the packet over this interface, the ARP table is 1414 consulted to get the destination TS (TS3) MAC address. 1416 o The IP packet is encapsulated with an Ethernet header with the MAC 1417 SA set to that of the access-facing IRB interface of the egress 1418 NVE (NVE2) and the MAC DA is set to that of destination TS (TS3) 1419 MAC address. The packet is sent to the corresponding MAC-VRF3 and 1420 after a lookup of MAC DA, is forwarded to the destination TS (TS3) 1421 over the corresponding interface. 1423 10. Acknowledgements 1425 The authors would like to thank Sami Boutros, Jeffrey Zhang, 1426 Krzysztof Szarkowicz, Lukas Krattiger and Neeraj Malhotra for their 1427 valuable comments. The authors would also like to thank Linda 1428 Dunbar, Florin Balus, Yakov Rekhter, Wim Henderickx, Lucy Yong, and 1429 Dennis Cai for their feedbacks and contributions. 1431 11. Security Considerations 1433 The security considerations for layer-2 forwarding in this document 1434 follow that of [RFC7432] for MPLS encapsulation and it follows that 1435 of [RFC8365] for VxLAN or GENEVE encapsulations. This section 1436 describes additional considerations. 1438 This document describes a set of procedures for Inter-Subnet 1439 Forwarding of tenant traffic across PEs (or NVEs). These procedures 1440 include both layer-2 forwarding and layer-3 routing on a packet by 1441 packet basis. The security consideration for layer-3 routing is this 1442 document follows that of [RFC4365] with the exception for application 1443 of routing protocols between CEs and PEs. Contrary to [RFC4364], 1444 this document does not describe route distribution techniques between 1445 CEs and PEs, but rather considers the CEs as TSes or VAs that do not 1446 run dynamic routing protocols. This can be considered a security 1447 advantage, since dynamic routing protocols can be blocked on the NVE/ 1448 PE ACs, not allowing the tenant to interact with the infrastructure's 1449 dynamic routing protocols. 1451 The VPN scheme described in this document does not provide the 1452 quartet of security properties mentioned in [RFC4365] 1453 (confidentiality protection, source authentication, integrity 1454 protection, replay protection). If these are desired, they must be 1455 provided by mechanisms that are outside the scope of the VPN 1456 mechanisms. 1458 In this document, the RT-5 is used for certain scenarios. This route 1459 uses an Overlay Index that requires a recursive resolution to a 1460 different EVPN route (an RT-2). Because of this, it is worth noting 1461 that any action that ends up filtering or modifying the RT-2 route 1462 used to convey the Overlay Indexes, will modify the resolution of the 1463 RT-5 and therefore the forwarding of packets to the remote subnet. 1465 12. IANA Considerations 1467 IANA has allocated a new transitive extended community Type of 0x06 1468 and Sub-Type of 0x03 for EVPN Router's MAC Extended Community. 1470 13. References 1472 13.1. Normative References 1474 [I-D.ietf-bess-evpn-prefix-advertisement] 1475 Rabadan, J., Henderickx, W., Drake, J., Lin, W., and A. 1476 Sajassi, "IP Prefix Advertisement in EVPN", draft-ietf- 1477 bess-evpn-prefix-advertisement-11 (work in progress), May 1478 2018. 1480 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1481 Requirement Levels", BCP 14, RFC 2119, 1482 DOI 10.17487/RFC2119, March 1997, 1483 . 1485 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1486 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1487 2006, . 1489 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 1490 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 1491 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 1492 2015, . 1494 [RFC7606] Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K. 1495 Patel, "Revised Error Handling for BGP UPDATE Messages", 1496 RFC 7606, DOI 10.17487/RFC7606, August 2015, 1497 . 1499 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1500 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1501 May 2017, . 1503 [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., 1504 Uttaro, J., and W. Henderickx, "A Network Virtualization 1505 Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, 1506 DOI 10.17487/RFC8365, March 2018, 1507 . 1509 13.2. Informative References 1511 [I-D.ietf-bess-evpn-irb-extended-mobility] 1512 Malhotra, N., Sajassi, A., Pattekar, A., Lingala, A., 1513 Rabadan, J., and J. Drake, "Extended Mobility Procedures 1514 for EVPN-IRB", draft-ietf-bess-evpn-irb-extended- 1515 mobility-03 (work in progress), May 2020. 1517 [I-D.ietf-idr-tunnel-encaps] 1518 Patel, K., Velde, G., Sangli, S., and J. Scudder, "The BGP 1519 Tunnel Encapsulation Attribute", draft-ietf-idr-tunnel- 1520 encaps-17 (work in progress), July 2020. 1522 [RFC4365] Rosen, E., "Applicability Statement for BGP/MPLS IP 1523 Virtual Private Networks (VPNs)", RFC 4365, 1524 DOI 10.17487/RFC4365, February 2006, 1525 . 1527 [RFC5798] Nadas, S., Ed., "Virtual Router Redundancy Protocol (VRRP) 1528 Version 3 for IPv4 and IPv6", RFC 5798, 1529 DOI 10.17487/RFC5798, March 2010, 1530 . 1532 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 1533 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 1534 eXtensible Local Area Network (VXLAN): A Framework for 1535 Overlaying Virtualized Layer 2 Networks over Layer 3 1536 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, 1537 . 1539 [RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. 1540 Rekhter, "Framework for Data Center (DC) Network 1541 Virtualization", RFC 7365, DOI 10.17487/RFC7365, October 1542 2014, . 1544 Authors' Addresses 1546 Ali Sajassi 1547 Cisco Systems 1548 MILPITAS, CALIFORNIA 95035 1549 UNITED STATES 1551 Email: sajassi@cisco.com 1553 Samer Salam 1554 Cisco Systems 1556 Email: ssalam@cisco.com 1558 Samir Thoria 1559 Cisco Systems 1561 Email: sthoria@cisco.com 1563 John E Drake 1564 Juniper 1566 Email: jdrake@juniper.net 1568 Jorge Rabadan 1569 Nokia 1571 Email: jorge.rabadan@nokia.com