idnits 2.17.1 draft-ietf-bess-evpn-inter-subnet-forwarding-13.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 10, 2021) is 1169 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Downref: Normative reference to an Informational RFC: RFC 7348 ** Downref: Normative reference to an Informational RFC: RFC 7637 == Outdated reference: A later version (-17) exists of draft-ietf-bess-evpn-irb-extended-mobility-03 == Outdated reference: A later version (-13) exists of draft-ietf-nvo3-vxlan-gpe-10 Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BESS WorkGroup A. Sajassi 3 Internet-Draft S. Salam 4 Intended status: Standards Track S. Thoria 5 Expires: August 14, 2021 Cisco Systems 6 J. Drake 7 Juniper 8 J. Rabadan 9 Nokia 10 February 10, 2021 12 Integrated Routing and Bridging in EVPN 13 draft-ietf-bess-evpn-inter-subnet-forwarding-13 15 Abstract 17 Ethernet VPN (EVPN) provides an extensible and flexible multi-homing 18 VPN solution over an MPLS/IP network for intra-subnet connectivity 19 among Tenant Systems and End Devices that can be physical or virtual. 20 However, there are scenarios for which there is a need for a dynamic 21 and efficient inter-subnet connectivity among these Tenant Systems 22 and End Devices while maintaining the multi-homing capabilities of 23 EVPN. This document describes an Integrated Routing and Bridging 24 (IRB) solution based on EVPN to address such requirements. 26 Requirements Language 28 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 29 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 30 "OPTIONAL" in this document are to be interpreted as described in RFC 31 2119 [RFC2119] and RFC 8174 [RFC8174] when, and only when, they 32 appear in all capitals, as shown here. 34 Status of This Memo 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF). Note that other groups may also distribute 41 working documents as Internet-Drafts. The list of current Internet- 42 Drafts is at https://datatracker.ietf.org/drafts/current/. 44 Internet-Drafts are draft documents valid for a maximum of six months 45 and may be updated, replaced, or obsoleted by other documents at any 46 time. It is inappropriate to use Internet-Drafts as reference 47 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on August 14, 2021. 50 Copyright Notice 52 Copyright (c) 2021 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (https://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 68 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 69 3. EVPN PE Model for IRB Operation . . . . . . . . . . . . . . . 6 70 4. Symmetric and Asymmetric IRB . . . . . . . . . . . . . . . . 7 71 4.1. IRB Interface and its MAC and IP addresses . . . . . . . 10 72 5. Symmetric IRB Procedures . . . . . . . . . . . . . . . . . . 12 73 5.1. Control Plane - Advertising PE . . . . . . . . . . . . . 12 74 5.2. Control Plane - Receiving PE . . . . . . . . . . . . . . 13 75 5.3. Subnet route advertisement . . . . . . . . . . . . . . . 14 76 5.4. Data Plane - Ingress PE . . . . . . . . . . . . . . . . . 15 77 5.5. Data Plane - Egress PE . . . . . . . . . . . . . . . . . 15 78 6. Asymmetric IRB Procedures . . . . . . . . . . . . . . . . . . 16 79 6.1. Control Plane - Advertising PE . . . . . . . . . . . . . 16 80 6.2. Control Plane - Receiving PE . . . . . . . . . . . . . . 17 81 6.3. Data Plane - Ingress PE . . . . . . . . . . . . . . . . . 18 82 6.4. Data Plane - Egress PE . . . . . . . . . . . . . . . . . 18 83 7. Mobility Procedure . . . . . . . . . . . . . . . . . . . . . 19 84 7.1. Initiating a gratutious ARP upon a Move . . . . . . . . . 20 85 7.2. Sending Data Traffic without an ARP Request . . . . . . . 21 86 7.3. Silent Host . . . . . . . . . . . . . . . . . . . . . . . 23 87 8. BGP Encoding . . . . . . . . . . . . . . . . . . . . . . . . 23 88 8.1. Router's MAC Extended Community . . . . . . . . . . . . . 24 89 9. Operational Models for Symmetric Inter-Subnet Forwarding . . 24 90 9.1. IRB forwarding on NVEs for Tenant Systems . . . . . . . . 24 91 9.1.1. Control Plane Operation . . . . . . . . . . . . . . . 26 92 9.1.2. Data Plane Operation . . . . . . . . . . . . . . . . 27 93 9.2. IRB forwarding on NVEs for Subnets behind Tenant Systems 29 94 9.2.1. Control Plane Operation . . . . . . . . . . . . . . . 30 95 9.2.2. Data Plane Operation . . . . . . . . . . . . . . . . 31 97 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 32 98 11. Security Considerations . . . . . . . . . . . . . . . . . . . 32 99 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 100 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 33 101 13.1. Normative References . . . . . . . . . . . . . . . . . . 33 102 13.2. Informative References . . . . . . . . . . . . . . . . . 34 103 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35 105 1. Terminology 107 AC: Attachment Circuit 109 ARP: Address Resolution Protocol 111 BD: Broadcast Domain. As per [RFC7432], an EVI consists of a single 112 or multiple BDs. In the case of VLAN-bundle and VLAN-based service 113 models (see [RFC7432]), a BD is equivalent to an EVI. In the case of 114 VLAN-aware bundle service model, an EVI contains multiple BDs. Also, 115 in this document, BD and subnet are equivalent terms and wherever 116 "subnet" is used, it means "IP subnet" 118 BD Route Target: refers to the Broadcast Domain assigned Route Target 119 [RFC4364]. In the case of VLAN-aware bundle service model, all the 120 BD instances in the MAC-VRF share the same Route Target 122 BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as per 123 [RFC7432]. 125 Ethernet NVO tunnel: refers to Network Virtualization Overlay tunnels 126 with Ethernet payload as specified for VxLAN in [RFC7348] and for 127 NVGRE in [RFC7637]. 129 EVI: EVPN Instance spanning the NVE/PE devices that are participating 130 on that EVPN, as per [RFC7432]. 132 EVPN: Ethernet Virtual Private Networks, as per [RFC7432]. 134 IP NVO tunnel: it refers to Network Virtualization Overlay tunnels 135 with IP payload (no MAC header in the payload) as specified for GPE 136 in [I-D.ietf-nvo3-vxlan-gpe]. 138 IP-VRF: A Virtual Routing and Forwarding table for IP routes on an 139 NVE/PE. The IP routes could be populated by EVPN and IP-VPN address 140 families. An IP-VRF is also an instantiation of a layer 3 VPN in an 141 NVE/PE. 143 IRB: Integrated Routing and Bridging interface. It connects an IP- 144 VRF to a BD (or subnet). 146 MAC-VRF: A Virtual Routing and Forwarding table for Media Access 147 Control (MAC) addresses on an NVE/PE, as per [RFC7432]. A MAC-VRF is 148 also an instantiation of an EVI in an NVE/PE. 150 ND: Neighbor Discovery Protocol 152 NVE: Network Virtualization Edge 154 NVGRE: Network Virtualization Generic Routing Encapsulation, 155 [RFC7637] 157 NVO: Network Virtualization Overlays 159 RT-2: EVPN route type 2, i.e., MAC/IP Advertisement route, as defined 160 in [RFC7432] 162 RT-5: EVPN route type 5, i.e., IP Prefix route. As defined in 163 Section 3 of [I-D.ietf-bess-evpn-prefix-advertisement] 165 TS: Tenant System 167 VA: Virtual Appliance 169 VNI: Virtual Network Identifier. As in [RFC8365], the term is used 170 as a representation of a 24-bit NVO instance identifier, with the 171 understanding that VNI will refer to a VXLAN Network Identifier in 172 VXLAN, or Virtual Subnet Identifier in NVGRE, etc. unless it is 173 stated otherwise. 175 VTEP: VXLAN Termination End Point, as in [RFC7348]. 177 VXLAN: Virtual Extensible LAN, as in [RFC7348]. 179 This document also assumes familiarity with the terminology of 180 [RFC7432], [RFC8365] and [RFC7365]. 182 2. Introduction 184 EVPN [RFC7432] provides an extensible and flexible multi-homing VPN 185 solution over an MPLS/IP network for intra-subnet connectivity among 186 Tenant Systems (TSes) and End Devices that can be physical or 187 virtual; where an IP subnet is represented by an EVPN Instance (EVI) 188 for a VLAN-based service or by an (EVI, VLAN) for a VLAN-aware bundle 189 service. However, there are scenarios for which there is a need for 190 a dynamic and efficient inter-subnet connectivity among these Tenant 191 Systems and End Devices while maintaining the multi-homing 192 capabilities of EVPN. This document describes an Integrated Routing 193 and Bridging (IRB) solution based on EVPN to address such 194 requirements. 196 The inter-subnet communication is traditionally achieved at 197 centralized L3 Gateway (L3GW) devices where all the inter-subnet 198 forwarding is performed and all the inter-subnet communication 199 policies are enforced. When two TSes belonging to two different 200 subnets connected to the same PE wanted to communicate with each 201 other, their traffic needed to be backhauled from the PE all the way 202 to the centralized gateway where inter-subnet switching is performed 203 and then back to the PE. For today's large multi-tenant data center, 204 this scheme is very inefficient and sometimes impractical. 206 In order to overcome the drawback of the centralized layer-3 GW 207 approach, IRB functionality is needed on the PEs (also referred to as 208 EVPN NVEs) attached to TSes in order to avoid inefficient forwarding 209 of tenant traffic (i.e., avoid back-hauling and hair-pinning). When 210 a PE with IRB capability receives tenant traffic over an Attachment 211 Circuit (AC), it can not only locally bridge the tenant intra-subnet 212 traffic but also can locally route the tenant inter-subnet traffic on 213 a packet by packet basis thus meeting the requirements for both intra 214 and inter-subnet forwarding and avoiding non-optimal traffic 215 forwarding associated with centralized layer-3 GW approach. 217 Some TSes run non-IP protocols in conjunction with their IP traffic. 218 Therefore, it is important to handle both kinds of traffic optimally 219 - e.g., to bridge non-IP and intra-subnet traffic and to route inter- 220 subnet IP traffic. Therefore, the solution needs to meet the 221 following requirements: 223 R1: The solution must allow for both inter-subnet and intra-subnet 224 traffic belonging to the same tenant to be locally routed and bridged 225 respectively. The solution must provide IP routing for inter-subnet 226 traffic and Ethernet Bridging for intra-subnet traffic. It should be 227 noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE 228 receives IPv4 traffic on the corresponding VLAN, then the IPv4 229 traffic is treated as L2 traffic and it is bridged. Also vise versa, 230 if an IP-VRF in a NVE is configured for IPv4 and that NVE receives 231 IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is 232 treated as L2 traffic and it is bridged. 234 R2: The solution must support bridging for non-IP traffic. 236 R3: The solution must allow inter-subnet switching to be disabled on 237 a per VLAN basis on PEs where the traffic needs to be backhauled to 238 another node (i.e., for performing FW or DPI functionality). 240 3. EVPN PE Model for IRB Operation 242 Since this document discusses IRB operation in relationship to EVPN 243 MAC-VRF, IP-VRF, EVI, Bridge Domain (BD), Bridge Table (BT), and IRB 244 interfaces, it is important to understand the relationship between 245 these components. Therefore, the following PE model is illustrated 246 below to a) describe these components and b) illustrate the 247 relationship among them. 249 +-------------------------------------------------------------+ 250 | | 251 | +------------------+ IRB PE | 252 | Attachment | +------------------+ | 253 | Circuit(AC1) | | +----------+ | MPLS/NVO tnl 254 ----------------------*Bridge | | +----- 255 | | | |Table(BT1)| | +-----------+ / \ \ 256 | | | | *---------* |<--> |Eth| 257 | | | | VLAN x | |IRB1| | \ / / 258 | | | +----------+ | | | +----- 259 | | | ... | | IP-VRF1 | | 260 | | | +----------+ | | RD2/RT2 |MPLS/NVO tnl 261 | | | |Bridge | | | | +----- 262 | | | |Table(BT2)| |IRB2| | / \ \ 263 | | | | *---------* |<--> |IP | 264 ----------------------* VLAN y | | +-----------+ \ / / 265 | AC2 | | +----------+ | +----- 266 | | | MAC-VRF1 | | 267 | +-+ RD1/RT1 | | 268 | +------------------+ | 269 | | 270 | | 271 +-------------------------------------------------------------+ 273 Figure 1: EVPN IRB PE Model 275 A tenant needing IRB services on a PE, requires an IP Virtual Routing 276 and Forwarding table (IP-VRF) along with one or more MAC Virtual 277 Routing and Forwarding tables (MAC-VRFs). An IP-VRF, as defined in 278 [RFC4364], is the instantiation of an IPVPN instance in a PE. A MAC- 279 VRF, as defined in [RFC7432], is the instantiation of an EVI (EVPN 280 Instance) in a PE. A MAC-VRF consists of one or more Bridge Tables 281 (BTs) where each BT corresponds to a VLAN (broadcast domain - BD). 282 If service interfaces for an EVPN PE are configured in VLAN- Based 283 mode (i.e., section 6.1 of RFC7432), then there is only a single BT 284 per MAC-VRF (per EVI) - i.e., there is only one tenant VLAN per EVI. 285 However, if service interfaces for an EVPN PE are configured in VLAN- 286 Aware Bundle mode (i.e., section 6.3 of RFC7432), then there are 287 several BTs per MAC-VRF (per EVI) - i.e., there are several tenant 288 VLANs per EVI. 290 Each BT is connected to an IP-VRF via an L3 interface called IRB 291 interface. Since a single tenant subnet is typically (and in this 292 document) represented by a VLAN (and thus supported by a single BT), 293 for a given tenant there are as many BTs as there are subnets and 294 thus there are also as many IRB interfaces between the tenant IP-VRF 295 and the associated BTs as shown in the PE model above. 297 IP-VRF is identified by its corresponding route target and route 298 distinguisher and MAC-VRF is also identified by its corresponding 299 route target and route distinguisher. If operating in EVPN VLAN- 300 Based mode, then a receiving PE that receives an EVPN route with MAC- 301 VRF route target can identify the corresponding BT; however, if 302 operating in EVPN VLAN-Aware Bundle mode, then the receiving PE needs 303 both the MAC-VRF route target and VLAN ID in order to identify the 304 corresponding BT. 306 4. Symmetric and Asymmetric IRB 308 This document defines and describes two types of IRB solutions - 309 namely symmetric and asymmetric IRB. The description of symmetric 310 and asymmetric IRB procedures relating to data path operations and 311 tables in this document is a logical view of data path lookups and 312 related tables. Actual implementations, while following this logical 313 view, may not strictly adhere to it for performance tradeoffs. 314 Specifically, 316 o references to ARP table in the context of asymmetric IRB is a 317 logical view of a forwarding table that maintains an IP to MAC 318 binding entry on a layer 3 interface for both IPv4 and IPv6. 319 These entries are not subject to ARP or ND protocol. For IP to 320 MAC bindings learnt via EVPN, an implementation may choose to 321 import these bindings directly to the respective forwarding table 322 (such as an adjacency/next-hop table) as opposed to importing them 323 to ARP or ND protocol tables. 325 o references to host IP lookup followed by a host MAC lookup in the 326 context of asymmetric IRB MAY be collapsed into a single IP lookup 327 in a hardware implementation. 329 In symmetric IRB as its name implies, the lookup operation is 330 symmetric at both ingress and egress PEs - i.e., both ingress and 331 egress PEs perform lookups on both MAC and IP addresses. The ingress 332 PE performs a MAC lookup followed by an IP lookup and the egress PE 333 performs an IP lookup followed by a MAC lookup as depicted in the 334 following figure. 336 Ingress PE Egress PE 337 +-------------------+ +------------------+ 338 | | | | 339 | +-> IP-VRF ----|---->---|-----> IP-VRF -+ | 340 | | | | | | 341 | BT1 BT2 | | BT3 BT2 | 342 | | | | | | 343 | ^ | | v | 344 | | | | | | 345 +-------------------+ +------------------+ 346 ^ | 347 | | 348 TS1->-+ +->-TS2 349 Figure 2: Symmetric IRB 351 In symmetric IRB as shown in figure-2, the inter-subnet forwarding 352 between two PEs is done between their associated IP-VRFs. Therefore, 353 the tunnel connecting these IP-VRFs can be either IP-only tunnel 354 (e.g., in case of MPLS or GPE encapsulation) or Ethernet NVO tunnel 355 (e.g., in case of VxLAN encapsulation). If it is an Ethernet NVO 356 tunnel, the TS1's IP packet is encapsulated in an Ethernet header 357 consisting of ingress and egress PEs MAC addresses - i.e., there is 358 no need for ingress PE to use the destination TS2's MAC address. 359 Therefore, in symmetric IRB, there is no need for the ingress PE to 360 maintain ARP entries for destination TS2's IP and MAC addresses 361 association in its ARP table. Each PE participating in symmetric IRB 362 only maintains ARP entries for locally connected hosts and maintains 363 MAC-VRFs/BTs for only locally configured subnets. 365 In asymmetric IRB, the lookup operation is asymmetric and the ingress 366 PE performs three lookups; whereas the egress PE performs a single 367 lookup - i.e., the ingress PE performs a MAC lookup, followed by an 368 IP lookup, followed by a MAC lookup again; whereas, the egress PE 369 performs just a single MAC lookup as depicted in figure 3 below. 371 Ingress PE Egress PE 372 +-------------------+ +------------------+ 373 | | | | 374 | +-> IP-VRF -> | | IP-VRF | 375 | | | | | | 376 | BT1 BT2 | | BT3 BT2 | 377 | | | | | | | | 378 | | +--|--->----|--------------+ | | 379 | | | | v | 380 +-------------------+ +----------------|-+ 381 ^ | 382 | | 383 TS1->-+ +->-TS2 384 Figure 3: Asymmetric IRB 386 In asymmetric IRB as shown in figure-3, the inter-subnet forwarding 387 between two PEs is done between their associated MAC-VRFs/BTs. 388 Therefore, the MPLS or NVO tunnel used for inter-subnet forwarding 389 MUST be of type Ethernet. Since only MAC lookup is performed at the 390 egress PE (e.g., no IP lookup), the TS1's IP packets need to be 391 encapsulated with the destination TS2's MAC address. In order for 392 ingress PE to perform such encapsulation, it needs to maintain TS2's 393 IP and MAC address association in its ARP table. Furthermore, it 394 needs to maintain destination TS2's MAC address in the corresponding 395 BT even though it may not have any TSes of the corresponding subnet 396 locally attached. In other words, each PE participating in 397 asymmetric IRB MUST maintain ARP entries for remote hosts (hosts 398 connected to other PEs) as well as maintain MAC-VRFs/BTs and IRB 399 interfaces for ALL subnets in an IP VRF including subnets that may 400 not be locally attached. Therefore, careful consideration of PE 401 scale aspects for its ARP table size, its IRB interfaces, number and 402 size of its bridge tables should be given for the application of 403 asymmetric IRB. 405 It should be noted that whenever a PE performs a host IP lookup for a 406 packet that is routed, IPv4 TTL or IPv6 hop limit for that packet is 407 decremented by one and if it reaches zero, the packet is discarded. 408 In the case of symmetric IRB, the TTL/hop limit is decremented by 409 both ingress and egress PEs (once by each); whereas, in the case of 410 asymmetric IRB, the TTL/hop limit is decremented only once by the 411 ingress PE. 413 The following subsection defines the control and data planes 414 procedures for symmetric and asymmetric IRB on ingress and egress 415 PEs. The following figure is used to describe these procedures where 416 it shows a single IP-VRF and a number of BTs on each PE for a given 417 tenant. The IP-VRF of the tenant (i.e., IP-VRF1) is connected to 418 each BT via its associated IRB interface. Each BT on a PE is 419 associated with a unique VLAN (e.g., with a BD) where in turn it is 420 associated with a single MAC-VRF in the case of VLAN-Based mode or a 421 number of BTs can be associated with a single MAC-VRF in the case of 422 VLAN-Aware Bundle mode. Whether the service interface on a PE is 423 VLAN-Based or VLAN-Aware Bundle mode does not impact the IRB 424 operation and procedures. It mainly impacts the setting of Ethernet 425 tag field in EVPN BGP routes as described in section 6 of [RFC7432]. 427 PE 1 +---------+ 428 +-------------+ | | 429 TS1-----| MACx| | | PE2 430 (IP1/M1) |(BT1) | | | +-------------+ 431 TS5-----| \ | | MPLS/ | |MACy (BT3) |-----TS3 432 (IP5/M5) |IPx/Mx \ | | VxLAN/ | | / | (IP3/M3) 433 | (IP-VRF1)|----| NVGRE |---|(IP-VRF1) | 434 | / | | | | \ | 435 TS2-----|(BT2) / | | | | (BT1) |-----TS4 436 (IP2/M2) | | | | | | (IP4/M4) 437 +-------------+ | | +-------------+ 438 | | 439 +---------+ 441 Figure 4: IRB forwarding 443 4.1. IRB Interface and its MAC and IP addresses 445 To support inter-subnet forwarding on a PE, the PE acts as an IP 446 Default Gateway from the perspective of the attached Tenant Systems 447 where default gateway MAC and IP addresses are configured on each IRB 448 interface associated with its subnet and falls into one of the 449 following two options: 451 1. All the PEs for a given tenant subnet use the same anycast 452 default gateway IP and MAC addresses. On each PE, this default 453 gateway IP and MAC addresses correspond to the IRB interface 454 connecting the BT associated with the tenant's VLAN to the 455 corresponding tenant's IP-VRF. 457 2. Each PE for a given tenant subnet uses the same anycast default 458 gateway IP address but its own MAC address. These MAC addresses 459 are aliased to the same anycast default gateway IP address 460 through the use of the Default Gateway extended community as 461 specified in [RFC7432], which is carried in the EVPN MAC/IP 462 Advertisement routes. On each PE, this default gateway IP 463 address along with its associated MAC addresses correspond to the 464 IRB interface connecting the BT associated with the tenant's VLAN 465 to the corresponding tenant's IP-VRF. 467 It is worth noting that if the applications that are running on the 468 TSes are employing or relying on any form of MAC security, then the 469 first option (i.e. using anycast MAC address) should be used to 470 ensure that the applications receive traffic from the same IRB 471 interface MAC address that they are sending to. If the second option 472 is used, then the IRB interface MAC address MUST be the one used in 473 the initial ARP reply or ND Neighbor Advertisement (NA)for that TS. 475 Although both of these options are applicable to both symmetric and 476 asymmetric IRB, the option-1 is recommended because of the ease of 477 anycast MAC address provisioning on not only the IRB interface 478 associated with a given subnet across all the PEs corresponding to 479 that VLAN but also on all IRB interfaces associated with all the 480 tenant's subnets across all the PEs corresponding to all the VLANs 481 for that tenant. Furthermore, it simplifies the operation as there 482 is no need for Default Gateway extended community advertisement and 483 its associated MAC aliasing procedure. Yet another advantage is that 484 following host mobility, the host does not need to refresh the 485 default GW ARP/ND entry. 487 If option-1 is used, an implementation MAY choose to auto-derive the 488 anycast MAC address. If auto-derivation is used, the anycast MAC 489 MUST be auto-derived out of the following ranges (which are defined 490 in [RFC5798]): 492 o Anycast IPv4 IRB case: 00-00-5E-00-01-{VRID} (in hex, in Internet 493 standard bit-order) 495 o Anycast IPv6 IRB case: 00-00-5E-00-02-{VRID} (in hex, in Internet 496 standard bit-order) 498 Where the last octet is generated based on a configurable Virtual 499 Router ID (VRID, range 1-255)). If not explicitly configured, the 500 default value for the VRID octet is '1'. Auto-derivation of the 501 anycast MAC can only be used if there is certainty that the auto- 502 derived MAC does not collide with any customer MAC address. 504 In addition to IP anycast addresses, IRB interfaces can be configured 505 with non-anycast IP addresses for the purpose of OAM (such as 506 traceroute/ping to these interfaces) for both symmetric and 507 asymmetric IRB. These IP addresses need to be distributed as VPN 508 routes when PEs operate in symmetric IRB mode. However, they don't 509 need to be distributed if the PEs are operating in asymmetric IRB 510 mode as the non-anycast IP addresses are configured along with their 511 individual MACs and they get distributed via EVPN route type-2 512 advertisement. 514 For option-1, irrespective of using only the anycast MAC address or 515 both anycast and non-anycast MAC addresses (where the latter one is 516 used for the purpose of OAM) on the same IRB, when a TS sends an ARP 517 request or ND Neighbor Solicitation (NS) to the PE that is attached 518 to, the request is sent for the anycast IP address of the IRB 519 interface associated with the TS's subnet and then the reply will use 520 anycast MAC address (in both Source MAC in the Ethernet header and 521 Sender hardware address in the payload). For example, in figure 4, 522 TS1 is configured with the anycast IPx address as its default gateway 523 IP address and thus when it sends an ARP request for IPx (anycast IP 524 address of the IRB interface for BT1), the PE1 sends an ARP reply 525 with the MACx which is the anycast MAC address of that IRB interface. 526 Traffic routed from IP-VRF1 to TS1 uses the anycast MAC address as 527 source MAC address. 529 5. Symmetric IRB Procedures 531 5.1. Control Plane - Advertising PE 533 When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address of 534 a TS (e.g., via an ARP request or Neighbor Solicitation), it adds the 535 MAC address to the corresponding MAC-VRF/BT of that tenant's subnet 536 and adds the IP address to the IP-VRF for that tenant. Furthermore, 537 it adds this TS's MAC and IP address association to its ARP table or 538 NDP cache. It then builds an EVPN MAC/IP Advertisement route (type 539 2) as follows and advertises it to other PEs participating in that 540 tenant's VPN. 542 o The Length field of the BGP EVPN NLRI for an EVPN MAC/IP 543 Advertisement route MUST be either 40 (if IPv4 address is carried) 544 or 52 (if IPv6 address is carried). 546 o Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet 547 Tag ID, MAC Address Length, MAC Address, IP Address Length, IP 548 Address, and MPLS Label1 fields MUST be set per [RFC7432] and 549 [RFC8365]. 551 o The MPLS Label2 field is set to either an MPLS label or a VNI 552 corresponding to the tenant's IP-VRF. In the case of an MPLS 553 label, this field is encoded as 3 octets, where the high-order 20 554 bits contain the label value. 556 Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, 557 MAC Address, IP Address Length, and IP Address fields are part of the 558 route key used by BGP to compare routes. The rest of the fields are 559 not part of the route key. 561 This route is advertised along with the following two extended 562 communities: 564 1. Encapsulation Extended Community 566 2. Router's MAC Extended Community 568 For symmetric IRB mode, Router's MAC EC is needed to carry the PE's 569 overlay MAC address (e.g., inner MAC address in NVO encapsulation) 570 which is used for IP-VRF to IP-VRF communications with Ethernet NVO 571 tunnel. If MPLS or IP-only NVO tunnel is used, then there is no need 572 to send Router's MAC Extended Community along with this route. 574 This route MUST be advertised with two route targets, one 575 corresponding to the MAC-VRF of the tenant's subnet and another 576 corresponding to the tenant's IP-VRF. 578 5.2. Control Plane - Receiving PE 580 When a PE (e.g., PE2 in figure 4 above) receives this EVPN MAC/IP 581 Advertisement route, it performs the following: 583 o Using MAC-VRF Route Target (and Ethernet Tag if different from 584 zero), it identifies the corresponding MAC-VRF (and BT). If the 585 MAC- VRF (and BT) exists (e.g., it is locally configured) then it 586 imports the MAC address into it. Otherwise, it does not import 587 the MAC address. 589 o Using IP-VRF route target, it identifies the corresponding IP-VRF 590 and imports the IP address into it. 592 The inclusion of MPLS label2 field in this route signals to the 593 receiving PE that this route is for symmetric IRB mode and MPLS 594 label2 needs to be installed in forwarding path to identify the 595 corresponding IP-VRF. 597 If the receiving PE receives this route with both the MAC-VRF and IP- 598 VRF route targets but the MAC/IP Advertisement route does not include 599 MPLS label2 field and if the receiving PE supports asymmetric IRB 600 mode, then the receiving PE installs the MAC address in the 601 corresponding MAC-VRF and (IP, MAC) association in the ARP table for 602 that tenant (identified by the corresponding IP-VRF route target). 604 If the receiving PE receives this route with both the MAC-VRF and IP- 605 VRF route targets and if the receiving PE does not support either 606 asymmetric or symmetric IRB modes, then if it has the corresponding 607 MAC-VRF, it only imports the MAC address. Otherwise, if it doesn't 608 have the corresponding MAC-VRF, it must not import this route. 610 If the receiving PE receives this route with both the MAC-VRF and IP- 611 VRF route targets and the MAC/IP Advertisement route includes MPLS 612 label2 field but the receiving PE only supports asymmetric IRB mode, 613 then the receiving PE MUST ignore MPLS label2 field and install the 614 MAC address in the corresponding MAC-VRF and (IP, MAC) association in 615 the ARP table for that tenant (identified by the corresponding IP-VRF 616 route target). 618 5.3. Subnet route advertisement 620 In the case of symmetric IRB, a layer-3 subnet and IRB interface 621 corresponding to a MAC-VRF/BT is required to be provisioned at a PE 622 only if that PE has locally attached hosts in that subnet. In order 623 to enable inter-subnet routing across PEs in a deployment where not 624 all subnets are provisioned at all PEs participating in an EVPN IRB 625 instance, PEs MUST advertise local subnet routes as EVPN RT-5. These 626 subnet routes are required for bootstrapping host (MAC,IP) learning 627 using gleaning procedures initiated by an inter-subnet data packet. 629 Consider a subnet A that is locally attached to PE1 and subnet B that 630 is locally attached to PE2 and to PE3. Host A in subnet A, that is 631 attached to PE1 initiates a data packet destined to host B in subnet 632 B that is attached to PE3. If host B's (MAC, IP) has not yet been 633 learnt either via a gratuitous ARP OR via a prior gleaning procedure, 634 a new gleaning procedure MUST be triggered for host B's (MAC, IP) to 635 be learnt and advertised across the EVPN network. Since host B's 636 subnet is not local to PE1, an IP lookup for host B at PE1 will not 637 trigger this gleaning procedure for host B's (MAC, IP). Therefore, 638 PE1 MUST learn subnet B's prefix route via EVPN RT-5 advertised from 639 PE2 and PE3, so it can route the packet to one of the PEs that have 640 subnet B locally attached. Once the packet is received at PE2 OR 641 PE3, and the route lookup yields a glean result, an ARP request is 642 triggered and flooded across the layer-2 overlay. This ARP request 643 would be received and replied to by host B, resulting in host B (MAC, 644 IP) learning at PE3, and its advertisement across the EVPN network. 645 Packets from host A to host B can now be routed directly from PE1 to 646 PE3. Advertisement of local subnet EVPN RT-5 for an IP VRF MAY 647 typically be achieved via provisioning connected route redistribution 648 to BGP. 650 5.4. Data Plane - Ingress PE 652 When an Ethernet frame is received by an ingress PE (e.g., PE1 in 653 figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify 654 the associated MAC-VRF/BT and it performs a lookup on the destination 655 MAC address. If the MAC address corresponds to its IRB Interface MAC 656 address, the ingress PE deduces that the packet must be inter-subnet 657 routed. Hence, the ingress PE performs an IP lookup in the 658 associated IP-VRF table. The lookup identifies BGP next hop of 659 egress PE along with the tunnel/encapsulation type and the associated 660 MPLS/VNI values. The ingress PE also decrements the TTL/hop limit 661 for that packet by one and if it reaches zero, the ingress PE 662 discards the packet. 664 If the tunnel type is that of MPLS or IP-only NVO tunnel, then TS's 665 IP packet is sent over the tunnel without any Ethernet header. 666 However, if the tunnel type is that of Ethernet NVO tunnel, then an 667 Ethernet header needs to be added to the TS's IP packet. The source 668 MAC address of this inner Ethernet header is set to the ingress PE's 669 router MAC address and the destination MAC address of this inner 670 Ethernet header is set to the egress PE's router MAC address learnt 671 via Router's MAC extended community attached to the route. MPLS VPN 672 label is set to the received label2 in the route. In the case of 673 Ethernet NVO tunnel type, VNI may be set one of two ways: 675 o downstream mode: VNI is set to the received label2 in the route 676 which is downstream assigned. 678 o global mode: VNI is set to the received label2 in the route which 679 is domain-wide assigned. This VNI value from received label2 MUST 680 be the same as the locally configured VNI for the IP VRF as all 681 PEs in the NVO MUST be configured with the same IP VRF VNI for 682 this mode of operation. 684 PEs may be configured to operate in one of these two modes depending 685 on the administrative domain boundaries across PEs participating in 686 the NVO, and PE's capability to support downstream VNI mode. 688 In the case of NVO tunnel encapsulation, the outer source and 689 destination IP addresses are set to the ingress and egress PE BGP 690 next-hop IP addresses respectively. 692 5.5. Data Plane - Egress PE 694 When the tenant's MPLS or NVO encapsulated packet is received over an 695 MPLS or NVO tunnel by the egress PE, the egress PE removes NVO tunnel 696 encapsulation and uses the VPN MPLS label (for MPLS encapsulation) or 697 VNI (for NVO encapsulation) to identify the IP-VRF in which IP lookup 698 needs to be performed. If the VPN MPLS label or VNI identifies a 699 MAC- VRF instead of an IP-VRF, then the procedures in section 6.4 for 700 asymmetric IRB are executed. 702 The lookup in the IP-VRF identifies a local adjacency to the IRB 703 interface associated with the egress subnet's MAC-VRF/BT. The egress 704 PE also decrements the TTL/hop limit for that packet by one and if it 705 reaches zero, the egress PE discards the packet. 707 The egress PE gets the destination TS's MAC address for that TS's IP 708 address from its ARP table or NDP cache, it encapsulates the packet 709 with that destination MAC address and a source MAC address 710 corresponding to that IRB interface and sends the packet to its 711 destination subnet MAC-VRF/BT. 713 The destination MAC address lookup in the MAC-VRF/BT results in local 714 adjacency (e.g., local interface) over which the Ethernet frame is 715 sent on. 717 6. Asymmetric IRB Procedures 719 6.1. Control Plane - Advertising PE 721 When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address of 722 an attached TS (e.g., via an ARP request or ND Neighbor 723 Solicitation), it populates its MAC-VRF/BT, IP-VRF, and ARP table or 724 NDP cache just as in the case for symmetric IRB. It then builds an 725 EVPN MAC/IP Advertisement route (type 2) as follows and advertises it 726 to other PEs participating in that tenant's VPN. 728 o The Length field of the BGP EVPN NLRI for an EVPN MAC/IP 729 Advertisement route MUST be either 37 (if IPv4 address is carried) 730 or 49 (if IPv6 address is carried). 732 o Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet 733 Tag ID, MAC Address Length, MAC Address, IP Address Length, IP 734 Address, and MPLS Label1 fields MUST be set per [RFC7432] and 735 [RFC8365]. 737 o The MPLS Label2 field MUST NOT be included in this route. 739 Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, 740 MAC Address, IP Address Length, and IP Address fields are part of the 741 route key used by BGP to compare routes. The rest of the fields are 742 not part of the route key. 744 This route is advertised along with the following extended community: 746 o Tunnel Type Extended Community 748 For asymmetric IRB mode, Router's MAC EC is not needed because 749 forwarding is performed using destination TS's MAC address which is 750 carried in this EVPN route type-2 advertisement. 752 This route MUST always be advertised with the MAC-VRF route target. 753 It MAY also be advertised with a second route target corresponding to 754 the IP-VRF. 756 6.2. Control Plane - Receiving PE 758 When a PE (e.g., PE2 in figure 4 above) receives this EVPN MAC/IP 759 Advertisement route, it performs the following: 761 o Using MAC-VRF route target, it identifies the corresponding MAC- 762 VRF and imports the MAC address into it. For asymmetric IRB mode, 763 it is assumed that all PEs participating in a tenant's VPN are 764 configured with all subnets (i.e., all VLANs) and corresponding 765 MAC-VRFs/BTs even if there are no locally attached TSes for some 766 of these subnets. The reason for this is because ingress PE needs 767 to do forwarding based on destination TS's MAC address and perform 768 NVO tunnel encapsulation as a property of a lookup in MAC-VRF/BT. 770 o If only MAC-VRF route target is used, then the receiving PE uses 771 the MAC-VRF route target to identify the corresponding IP-VRF -- 772 i.e., many MAC-VRF route targets map to the same IP-VRF for a 773 given tenant. In this case, MAC-VRF may be used by the receiving 774 PE to identify the corresponding IP VRF via the IRB interface 775 associated with the subnet MAC-VRF/BT. This would equivalent to 776 how ARP table or NDP cache entries are typically mapped to IRB 777 interface of an IP VRF for installing attached host routes in an 778 IP VRF. Since in asymmetric IRB mode, each PE is configured with 779 all VLANs of a tenant, indirect import to IP VRF via the 780 corresponding MAC-VRF route target is a viable alternative. 782 o Using MAC-VRF route target, the receiving PE identifies the 783 corresponding ARP table or NDP cache for the tenant and it adds an 784 entry to the ARP table or NDP cache for the TS's MAC and IP 785 address association. It should be noted that the tenant's ARP 786 table or NDP cache at the receiving PE is identified by all the 787 MAC- VRF route targets for that tenant. 789 o If IP-VRF route target is included, it may be used to import the 790 route to IP-VRF. If IP-VRF route-target is not included, MAC-VRF 791 is used to derive corresponding IP-VRF for import, as explained in 792 the prior section. In both cases, IP-VRF route is installed with 793 the TS MAC binding included in the received route. 795 If the receiving PE receives the MAC/IP Advertisement route with MPLS 796 label2 field but the receiving PE only supports asymmetric IRB mode, 797 then the receiving PE MUST ignore MPLS label2 field and install the 798 MAC address in the corresponding MAC-VRF and (IP, MAC) association in 799 the ARP table or NDP cache for that tenant (with IRB interface 800 identified by the MAC-VRF). 802 If the receiving PE receives the MAC/IP Advertisement route with MPLS 803 label2 field and it uses symmetric IRB mode, then it should use the 804 MAC-VRF route target to identify its corresponding MAC-VRF table and 805 import the MAC address. It should use the IP-VRF route target to 806 identify the corresponding IP-VRF table and import the IP address, as 807 specified in symmetric IRB handling. It MUST NOT import (IP, MAC) 808 association into its ARP table or NDP cache. 810 6.3. Data Plane - Ingress PE 812 When an Ethernet frame is received by an ingress PE (e.g., PE1 in 813 figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify 814 the associated MAC-VRF/BT and it performs a lookup on the destination 815 MAC address. If the MAC address corresponds to its IRB Interface MAC 816 address, the ingress PE deduces that the packet must be inter-subnet 817 routed. Hence, the ingress PE performs an IP lookup in the 818 associated IP-VRF table. The lookup identifies a local adjacency to 819 the IRB interface associated with the egress subnet's MAC-VRF/BT. 820 The ingress PE also decrements the TTL/hop limit for that packet by 821 one and if it reaches zero, the ingress PE discards the packet. 823 The ingress PE gets the destination TS's MAC address for that TS's IP 824 address from its ARP table or NDP cache, it encapsulates the packet 825 with that destination MAC address and a source MAC address 826 corresponding to that IRB interface and sends the packet to its 827 destination subnet MAC-VRF/BT. 829 The destination MAC address lookup in the MAC-VRF/BT results in BGP 830 next hop address of egress PE along with label1 (L2 VPN MPLS label or 831 VNI). The ingress PE encapsulates the packet using Ethernet NVO 832 tunnel of the choice (e.g., VxLAN or NVGRE) and sends the packet to 833 the egress PE. Because the packet forwarding is between ingress PE's 834 MAC-VRF/BT and egress PE's MAC-VRF/BT, the packet encapsulation 835 procedures follow that of [RFC7432] for MPLS and [RFC8365] for VxLAN 836 encapsulations. 838 6.4. Data Plane - Egress PE 840 When a tenant's Ethernet frame is received over an NVO tunnel by the 841 egress PE, the egress PE removes NVO tunnel encapsulation and uses 842 the VPN MPLS label (for MPLS encapsulation) or VNI (for NVO 843 encapsulation) to identify the MAC-VRF/BT in which MAC lookup needs 844 to be performed. 846 The MAC lookup results in local adjacency (e.g., local interface) 847 over which the packet needs to get sent. 849 Note that the forwarding behavior on the egress PE is the same as 850 EVPN intra-subnet forwarding described in [RFC7432] for MPLS and 851 [RFC8365] for NVO networks. In other words, all the packet 852 processing associated with the inter-subnet forwarding semantics is 853 confined to the ingress PE for asymmetric IRB mode. 855 It should also be noted that [RFC7432] provides a different level of 856 granularity for the EVPN label. Besides identifying the bridge 857 domain table, it can be used to identify the egress interface or a 858 destination MAC address on that interface. If EVPN label is used for 859 egress interface or individual MAC address identification, then no 860 MAC lookup is needed in the egress PE for MPLS encapsulation and the 861 packet can be directly forwarded to the egress interface just based 862 on EVPN label lookup. 864 7. Mobility Procedure 866 When a TS moves from one NVE (aka source NVE) to another NVE (aka 867 target NVE), it is important that the MAC mobility procedures are 868 properly executed and the corresponding MAC-VRF and IP-VRF tables on 869 all participating NVEs are updated. [RFC7432] describes the MAC 870 mobility procedures for L2-only services for both single-homed TS and 871 multi-homed TS. This section describes the incremental procedures 872 and BGP Extended Communities needed to handle the MAC mobility for 873 IRB. In order to place the emphasis on the differences between 874 L2-only and IRB use cases, the incremental procedure is described for 875 single-homed TS with the expectation that the additional steps needed 876 for multi-homed TS, can be extended per section 15 of [RFC7432]. 877 This section describes mobility procedures for both symmetric and 878 asymmetric IRB. Although the language used in this section is for 879 IPv4 ARP, it equally applies to IPv6 ND. 881 When a TS moves from a source NVE to a target NVE, it can behave in 882 one of the following three ways: 884 1. TS initiates an ARP request upon a move to the target NVE 886 2. TS sends data packet without first initiating an ARP request to 887 the target NVE 889 3. TS is a silent host and neither initiates an ARP request nor 890 sends any packets 892 Depending on the expexted TS's behavior, an NVE needs to handle at 893 least the first bullet and should be able to handle the 2nd and the 894 3rd bullet. The following subsections describe the procedures for 895 each of them where it is assumed that the MAC and IP addresses of a 896 TS have one-to-one relationship (i.e., there is one IP address per 897 MAC address and vice versa). The procedures for host mobility 898 detection in the presence of many-to-one relationship is outside the 899 scope of this document and it is covered in 900 [I-D.ietf-bess-evpn-irb-extended-mobility]. The many-to-one 901 relationship means many host IP addresses corresponding to a single 902 host MAC address or many host MAC addresses corresponding to a single 903 IP address. It should be noted that in case of IPv6, a Link Local IP 904 address does not count in many-to-one relationship because that 905 address is confined to single Ethernet Segment and it is not used for 906 host moblity (i.e., by definition host mobility is between two 907 different Ethernet Segments). Therefore, when an IPv6 host is 908 configured with both a Global Unicast address (or a Unique Local 909 address) and a Link Local address, for the purpose of host mobility, 910 it is considered with a single IP address. 912 7.1. Initiating a gratutious ARP upon a Move 914 In this scenario when a TS moves from a source NVE to a target NVE, 915 the TS initiates a gratuitous ARP upon the move to the target NVE. 917 The target NVE upon receiving this ARP message, updates its MAC-VRF, 918 IP-VRF, and ARP table with the host MAC, IP, and local adjacency 919 information (e.g., local interface). 921 Since this NVE has previously learned the same MAC and IP addresses 922 from the source NVE, it recognizes that there has been a MAC move and 923 it initiates MAC mobility procedures per [RFC7432] by advertising an 924 EVPN MAC/IP Advertisement route with both the MAC and IP addresses 925 filled in (per sections 5.1 and 6.1) along with MAC Mobility Extended 926 Community with the sequence number incremented by one. The target 927 NVE also exercises the MAC duplication detection procedure in section 928 15.1 of [RFC7432]. 930 The source NVE upon receiving this MAC/IP Advertisement route, 931 realizes that the MAC has moved to the target NVE. It updates its 932 MAC-VRF and IP-VRF table accordingly with the adjacency information 933 of the target NVE. In the case of the asymmetric IRB, the source NVE 934 also updates its ARP table with the received adjacency information 935 and in the case of the symmetric IRB, the source NVE removes the 936 entry associated with the received (MAC, IP) from its local ARP 937 table. It then withdraws its EVPN MAC/IP Advertisement route. 938 Furthermore, it sends an ARP probe locally to ensure that the MAC is 939 gone. If an ARP response is received, the source NVE updates its ARP 940 entry for that (IP, MAC) and re-advertises an EVPN MAC/IP 941 Advertisement route for that (IP, MAC) along with MAC Mobility 942 Extended Community with the sequence number incremented by one. The 943 source NVE also exercises the MAC duplication detection procedure in 944 section 15.1 of [RFC7432]. 946 All other remote NVE devices upon receiving the MAC/IP Advertisement 947 route with MAC Mobility extended community compare the sequence 948 number in this advertisement with the one previously received. If 949 the new sequence number is greater than the old one, then they update 950 the MAC/IP addresses of the TS in their corresponding MAC-VRF and IP- 951 VRF tables to point to the target NVE. Furthermore, upon receiving 952 the MAC/IP withdraw for the TS from the source NVE, these remote PEs 953 perform the cleanups for their BGP tables. 955 7.2. Sending Data Traffic without an ARP Request 957 In this scenario when a TS moves from a source NVE to a target NVE, 958 the TS starts sending data traffic without first initiating an ARP 959 request. 961 The target NVE upon receiving the first data packet, learns the MAC 962 address of the TS in the data plane and updates its MAC-VRF table 963 with the MAC address and the local adjacency information (e.g., local 964 interface) accordingly. The target NVE realizes that there has been 965 a MAC move because the same MAC address has been learned remotely 966 from the source NVE. 968 If EVPN-IRB NVEs are configured to advertise MAC-only routes in 969 addition to MAC-and-IP EVPN routes, then the following steps are 970 taken: 972 o The target NVE upon learning this MAC address in the data plane, 973 updates this MAC address entry in the corresponding MAC-VRF with 974 the local adjacency information (e.g., local interface). It also 975 recognizes that this MAC has moved and initiates MAC mobility 976 procedures per [RFC7432] by advertising an EVPN MAC/IP 977 Advertisement route with only the MAC address filled in along with 978 MAC Mobility Extended Community with the sequence number 979 incremented by one. 981 o The source NVE upon receiving this MAC/IP Advertisement route, 982 realizes that the MAC has moved to the new NVE. It updates its 983 MAC-VRF table with the adjacency information for that MAC address 984 to point to the target NVE and withdraws its EVPN MAC/IP 985 Advertisement route that has only the MAC address (if it has 986 advertised such route previously). Furthermore, it searches for 987 the corresponding MAC-IP entry and sends an ARP probe for this 988 (MAC,IP) pair. The ARP request message is sent both locally to 989 all attached TSes in that subnet as well as it is sent to other 990 NVEs participating in that subnet including the target NVE. Note 991 that the PE needs to maintain a correlation between MAC and MAC-IP 992 route entries in the MAC-VRF to accomplish this. 994 o The target NVE passes the ARP request to its locally attached TSes 995 and when it receives the ARP response, it updates its IP-VRF and 996 ARP table with the host (MAC, IP) information. It also sends an 997 EVPN MAC/IP Advertisement route with both the MAC and IP addresses 998 filled in along with MAC Mobility Extended Community with the 999 sequence number set to the same value as the one for MAC-only 1000 advertisement route it sent previously. 1002 o When the source NVE receives the EVPN MAC/IP Advertisement route, 1003 it updates its IP-VRF table with the new adjacency information 1004 (pointing to the target NVE). In the case of the asymmetric IRB, 1005 the source NVE also updates its ARP table with the received 1006 adjacency information and in the case of the symmetric IRB, the 1007 source NVE removes the entry associated with the received (MAC, 1008 IP) from its local ARP table. Furthermore, it withdraws its 1009 previously advertised EVPN MAC/IP route with both the MAC and IP 1010 address fields filled in. 1012 o All other remote NVE devices upon receiving the MAC/IP 1013 advertisement route with MAC Mobility extended community compare 1014 the sequence number in this advertisement with the one previously 1015 received. If the new sequence number is greater than the old one, 1016 then they update the MAC/IP addresses of the TS in their 1017 corresponding MAC-VRF, IP-VRF, and ARP tables (in the case of 1018 asymmetric IRB) to point to the new NVE. Furthermore, upon 1019 receiving the MAC/IP withdraw for the TS from the old NVE, these 1020 remote PEs perform the cleanups for their BGP tables. 1022 If EVPN-IRB NVEs are configured not to advertise MAC-only routes, 1023 then upon receiving the first data packet, it learns the MAC address 1024 of the TS and updates the MAC entry in the corresponding MAC-VRF 1025 table with the local adjacency information (e.g., local interface). 1026 It also realizes that there has been a MAC move because the same MAC 1027 address has been learned remotely from the source NVE. It uses the 1028 local MAC route to find the corresponding local MAC-IP route, and 1029 sends a unicast ARP request to the host and when receiving an ARP 1030 response, it follows the procedure outlined in section 7.1. In the 1031 prior case, where MAC-only routes are also advertised, this procedure 1032 of triggering a unicast ARP probe at the target PE MAY also be used 1033 in addition to the source PE broadcast ARP probing procedure 1034 described earlier for better convergence. 1036 7.3. Silent Host 1038 In this scenario when a TS moves from a source NVE to a target NVE, 1039 the TS is silent and it neither initiates an ARP request nor it sends 1040 any data traffic. Therefore, neither the target nor the source NVEs 1041 are aware of the MAC move. 1043 On the source NVE, an age-out timer (for the silent host that has 1044 moved) is used to trigger an ARP probe. This age-out timer can be 1045 either ARP timer or MAC age-out timer and this is an implementation 1046 choice. The ARP request gets sent both locally to all the attached 1047 TSes on that subnet as well as it gets sent to all the remote NVEs 1048 (including the target NVE) participating in that subnet. The source 1049 NVE also withdraw the EVPN MAC/IP Advertisement route with only the 1050 MAC address (if it has previously advertised such a route). 1052 The target NVE passes the ARP request to its locally attached TSes 1053 and when it receives the ARP response, it updates its MAC-VRF, IP- 1054 VRF, and ARP table with the host (MAC, IP) and local adjacency 1055 information (e.g., local interface). It also sends an EVPN MAC/IP 1056 advertisement route with both the MAC and IP address fields filled in 1057 along with MAC Mobility Extended Community with the sequence number 1058 incremented by one. 1060 When the source NVE receives the EVPN MAC/IP Advertisement route, it 1061 updates its IP-VRF table with the new adjacency information (pointing 1062 to the target NVE). In the case of the asymmetric IRB, the source 1063 NVE also updates its ARP table with the received adjacency 1064 information and in the case of the symmetric IRB, the source NVE 1065 removes the entry associated with the received (MAC, IP) from its 1066 local ARP table. Furthermore, it withdraws its previously advertised 1067 EVPN MAC/IP route with both the MAC and IP address fields filled in. 1069 All other remote NVE devices upon receiving the MAC/IP Advertisement 1070 route with MAC Mobility extended community compare the sequence 1071 number in this advertisement with the one previously received. If 1072 the new sequence number is greater than the old one, then they update 1073 the MAC/IP addresses of the TS in their corresponding MAC-VRF, IP- 1074 VRF, and ARP (in the case of asymmetric IRB) tables to point to the 1075 new NVE. Furthermore, upon receiving the MAC/IP withdraw for the TS 1076 from the old NVE, these remote PEs perform the cleanups for their BGP 1077 tables. 1079 8. BGP Encoding 1081 This document defines one new BGP Extended Community for EVPN. 1083 8.1. Router's MAC Extended Community 1085 A new EVPN BGP Extended Community called Router's MAC is introduced 1086 here. This new extended community is a transitive extended community 1087 with the Type field of 0x06 (EVPN) and the Sub-Type of 0x03. It may 1088 be advertised along with Encapsulation Extended Community defined in 1089 section 4.1 of [I-D.ietf-idr-tunnel-encaps]. 1091 The Router's MAC Extended Community is encoded as an 8-octet value as 1092 follows: 1094 0 1 2 3 1095 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1096 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1097 | Type=0x06 | Sub-Type=0x03 | Router's MAC | 1098 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1099 | Router's MAC Cont'd | 1100 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1102 Figure 5: Router's MAC Extended Community 1104 This extended community is used to carry the PE's MAC address for 1105 symmetric IRB scenarios and it is sent with EVPN RT-2. The 1106 advertising PE SHALL only attach a single Router's MAC Extended 1107 Community to a route. In case the receiving PE receives more than 1108 one Router's MAC Extended Community with a route, it SHALL process 1109 the first one in the list and not store and propagate the others. 1111 9. Operational Models for Symmetric Inter-Subnet Forwarding 1113 The following sections describe two main symmetric IRB forwarding 1114 scenarios (within a DC -- i.e., intra-DC) along with the 1115 corresponding procedures. In the following scenarios, without loss 1116 of generality, it is assumed that a given tenant is represented by a 1117 single IP-VPN instance. Therefore, on a given PE, a tenant is 1118 represented by a single IP-VRF table and one or more MAC-VRF tables. 1120 9.1. IRB forwarding on NVEs for Tenant Systems 1122 This section covers the symmetric IRB procedures for the scenario 1123 where each Tenant System (TS) is attached to one or more NVEs and its 1124 host IP and MAC addresses are learned by the attached NVEs and are 1125 distributed to all other NVEs that are interested in participating in 1126 both intra-subnet and inter-subnet communications with that TS. 1128 In this scenario, without loss of generality, it is assumed that NVEs 1129 operate in VLAN-based service interface mode with one Bridge 1130 Table (BT) per MAC-VRF. Thus, for a given tenant, an NVE has one 1131 MAC-VRF for each tenant subnet (e.g., each VLAN) that is configured 1132 for extension via VxLAN or NVGRE encapsulation. In the case of VLAN- 1133 aware bundling, then each MAC-VRF consists of multiple Bridge Tables 1134 (e.g., one BT per VLAN). The MAC-VRFs on an NVE for a given tenant 1135 are associated with an IP-VRF corresponding to that tenant (or IP-VPN 1136 instance) via their IRB interfaces. 1138 Since VxLAN and NVGRE encapsulations require inner Ethernet header 1139 (inner MAC SA/DA), and since for inter-subnet traffic, TS MAC address 1140 cannot be used, the ingress NVE's MAC address is used as inner MAC 1141 SA. The NVE's MAC address is the device MAC address and it is common 1142 across all MAC-VRFs and IP-VRFs. This MAC address is advertised 1143 using the new EVPN Router's MAC Extended Community (section 8.1). 1145 Figure 6 below illustrates this scenario where a given tenant (e.g., 1146 an IP-VPN instance) has three subnets represented by MAC-VRF1, MAC- 1147 VRF2, and MAC-VRF3 across two NVEs. There are five TSes that are 1148 associated with these three MAC-VRFs -- i.e., TS1, TS4, and TS5 are 1149 on the same subnet (e.g., same MAC-VRF/VLAN). TS1 and TS5 are 1150 associated with MAC-VRF1 on NVE1, while TS4 is associated with MAC- 1151 VRF1 on NVE2. TS2 is associated with MAC-VRF2 on NVE1, and TS3 is 1152 associated with MAC-VRF3 on NVE2. MAC-VRF1 and MAC-VRF2 on NVE1 are 1153 in turn associated with IP-VRF1 on NVE1 and MAC-VRF1 and MAC-VRF3 on 1154 NVE2 are associated with IP-VRF1 on NVE2. When TS1, TS5, and TS4 1155 exchange traffic with each other, only the L2 forwarding (bridging) 1156 part of the IRB solution is exercised because all these TSes belong 1157 to the same subnet. However, when TS1 wants to exchange traffic with 1158 TS2 or TS3 which belong to different subnets, both bridging and 1159 routing parts of the IRB solution are exercised. The following 1160 subsections describe the control and data planes operations for this 1161 IRB scenario in details. 1163 NVE1 +---------+ 1164 +-------------+ | | 1165 TS1-----| MACx| | | NVE2 1166 (IP1/M1) |(MAC- | | | +-------------+ 1167 TS5-----| VRF1)\ | | MPLS/ | |MACy (MAC- |-----TS3 1168 (IP5/M5) | \ | | VxLAN/ | | / VRF3) | (IP3/M3) 1169 | (IP-VRF1)|----| NVGRE |---|(IP-VRF1) | 1170 | / | | | | \ | 1171 TS2-----|(MAC- / | | | | (MAC- |-----TS4 1172 (IP2/M2) | VRF2) | | | | VRF1) | (IP4/M4) 1173 +-------------+ | | +-------------+ 1174 | | 1175 +---------+ 1177 Figure 6: IRB forwarding on NVEs for Tenant Systems 1179 9.1.1. Control Plane Operation 1181 Each NVE advertises a MAC/IP Advertisement route (i.e., Route Type 2) 1182 for each of its TSes with the following field set: 1184 o RD and ESI per [RFC7432] 1186 o Ethernet Tag = 0; assuming VLAN-based service 1188 o MAC Address Length = 48 1190 o MAC Address = Mi ; where i = 1,2,3,4, or 5 in the above example 1192 o IP Address Length = 32 or 128 1194 o IP Address = IPi ; where i = 1,2,3,4, or 5 in the above example 1196 o Label1 = MPLS Label or VNI corresponding to MAC-VRF 1198 o Label2 = MPLS Label or VNI corresponding to IP-VRF 1200 Each NVE advertises an EVPN RT-2 route with two Route Targets (one 1201 corresponding to its MAC-VRF and the other corresponding to its IP- 1202 VRF. Furthermore, the EVPN RT-2 is advertised with two BGP Extended 1203 Communities. The first BGP Extended Community identifies the tunnel 1204 type and it is called Encapsulation Extended Community as defined in 1205 [I-D.ietf-idr-tunnel-encaps] and the second BGP Extended Community 1206 includes the MAC address of the NVE (e.g., MACx for NVE1 or MACy for 1207 NVE2) as defined in section 8.1. The Router's MAC Extended community 1208 MUST be added when Ethernet NVO tunnel is used. If IP NVO tunnel 1209 type is used, then there is no need to send this second Extended 1210 Community. It should be noted that IP NVO tunnel type is only 1211 applicable to symmetric IRB procedures. 1213 Upon receiving this advertisement, the receiving NVE performs the 1214 following: 1216 o It uses Route Targets corresponding to its MAC-VRF and IP-VRF for 1217 identifying these tables and subsequently importing the MAC and IP 1218 addresses into them respectively. 1220 o It imports the MAC address from MAC/IP Advertisement route into 1221 the MAC-VRF with BGP Next Hop address as the underlay tunnel 1222 destination address (e.g., VTEP DA for VxLAN encapsulation) and 1223 Label1 as VNI for VxLAN encapsulation or EVPN label for MPLS 1224 encapsulation. 1226 o If the route carries the new Router's MAC Extended Community, and 1227 if the receiving NVE uses Ethernet NVO tunnel, then the receiving 1228 NVE imports the IP address into IP-VRF with NVE's MAC address 1229 (from the new Router's MAC Extended Community) as inner MAC DA and 1230 BGP Next Hop address as the underlay tunnel destination address, 1231 VTEP DA for VxLAN encapsulation and Label2 as IP-VPN VNI for VxLAN 1232 encapsulation. 1234 o If the receiving NVE uses MPLS encapsulation, then the receiving 1235 NVE imports the IP address into IP-VRF with BGP Next Hop address 1236 as the underlay tunnel destination address, and Label2 as IP-VPN 1237 label for MPLS encapsulation. 1239 If the receiving NVE receives an EVPN RT-2 with only Label1 and only 1240 a single Route Target corresponding to IP-VRF, or if it receives an 1241 EVPN RT-2 with only a single Route Target corresponding to MAC-VRF 1242 but with both Label1 and Label2, or if it receives an EVPN RT-2 with 1243 MAC Address Length of zero, then it MUST use the treat-as-withdraw 1244 approach [RFC7606] and SHOULD log an error message. 1246 9.1.2. Data Plane Operation 1248 The following description of the data-plane operation describes just 1249 the logical functions and the actual implementation may differ. Lets 1250 consider data-plane operation when TS1 in subnet-1 (MAC-VRF1) on NVE1 1251 wants to send traffic to TS3 in subnet-3 (MAC-VRF3) on NVE2. 1253 o NVE1 receives a packet with MAC DA corresponding to the MAC-VRF1 1254 IRB interface on NVE1 (the interface between MAC-VRF1 and IP- 1255 VRF1), and VLAN-tag corresponding to MAC-VRF1. 1257 o Upon receiving the packet, the NVE1 uses VLAN-tag to identify the 1258 MAC-VRF1. It then looks up the MAC DA and forwards the frame to 1259 its IRB interface. 1261 o The Ethernet header of the packet is stripped and the packet is 1262 fed to the IP-VRF where an IP lookup is performed on the 1263 destination IP address. NVE1 also decrements the TTL/hop limit 1264 for that packet by one and if it reaches zero, NVE1 discards the 1265 packet. This lookup yields the outgoing NVO tunnel and the 1266 required encapsulation. If the encapsulation is for Ethernet NVO 1267 tunnel, then it includes the egress NVE's MAC address as inner MAC 1268 DA, the egress NVE's IP address (e.g., BGP Next Hop address) as 1269 the VTEP DA, and the VPN-ID as the VNI. The inner MAC SA and VTEP 1270 SA are set to NVE's MAC and IP addresses respectively. If it is a 1271 MPLS encapsulation, then corresponding EVPN and LSP labels are 1272 added to the packet. The packet is then forwarded to the egress 1273 NVE. 1275 o On the egress NVE, if the packet arrives on Ethernet NVO tunnel 1276 (e.g., it is VxLAN encapsulated), then the NVO tunnel header is 1277 removed. Since the inner MAC DA is the egress NVE's MAC address, 1278 the egress NVE knows that it needs to perform an IP lookup. It 1279 uses the VNI to identify the IP-VRF table. If the packet is MPLS 1280 encapsulated, then the EVPN label lookup identifies the IP-VRF 1281 table. Next, an IP lookup is performed for the destination TS 1282 (TS3) which results in an access-facing IRB interface over which 1283 the packet is sent. Before sending the packet over this 1284 interface, the ARP table is consulted to get the destination TS's 1285 MAC address. NVE2 also decrements the TTL/hop limit for that 1286 packet by one and if it reaches zero, NVE2 discards the packet. 1288 o The IP packet is encapsulated with an Ethernet header with MAC SA 1289 set to that of IRB interface MAC address (i.e, IRB interface 1290 between MAC-VRF3 and IP-VRF1 on NVE2) and MAC DA set to that of 1291 destination TS (TS3) MAC address. The packet is sent to the 1292 corresponding MAC-VRF (i.e., MAC-VRF3) and after a lookup of MAC 1293 DA, is forwarded to the destination TS (TS3) over the 1294 corresponding interface. 1296 In this symmetric IRB scenario, inter-subnet traffic between NVEs 1297 will always use the IP-VRF VNI/MPLS label. For instance, traffic 1298 from TS2 to TS4 will be encapsulated by NVE1 using NVE2's IP-VRF VNI/ 1299 MPLS label, as long as TS4's host IP is present in NVE1's IP-VRF. 1301 9.2. IRB forwarding on NVEs for Subnets behind Tenant Systems 1303 This section covers the symmetric IRB procedures for the scenario 1304 where some Tenant Systems (TSes) support one or more subnets and 1305 these TSes are associated with one or more NVEs. Therefore, besides 1306 the advertisement of MAC/IP addresses for each TS which can be multi- 1307 homed with All-Active redundancy mode, the associated NVE needs to 1308 also advertise the subnets statically configured on each TS. 1310 The main difference between this solution and the previous one is the 1311 additional advertisement corresponding to each subnet. These subnet 1312 advertisements are accomplished using the EVPN IP Prefix route 1313 defined in [I-D.ietf-bess-evpn-prefix-advertisement]. These subnet 1314 prefixes are advertised with the IP address of their associated TS 1315 (which is in overlay address space) as their next hop. The receiving 1316 NVEs perform recursive route resolution to resolve the subnet prefix 1317 with its advertising NVE so that they know which NVE to forward the 1318 packets to when they are destined for that subnet prefix. 1320 The advantage of this recursive route resolution is that when a TS 1321 moves from one NVE to another, there is no need to re-advertise any 1322 of the subnet prefixes for that TS. All it is needed is to advertise 1323 the IP/MAC addresses associated with the TS itself and exercise MAC 1324 mobility procedures for that TS. The recursive route resolution 1325 automatically takes care of the updates for the subnet prefixes of 1326 that TS. 1328 Figure 7 illustrates this scenario where a given tenant (e.g., an IP- 1329 VPN service) has three subnets represented by MAC-VRF1, MAC-VRF2, and 1330 MAC-VRF3 across two NVEs. There are four TSes associated with these 1331 three MAC-VRFs -- i.e., TS1 is connected to MAC-VRF1 on NVE1, TS2 is 1332 connected to MAC-VRF2 on NVE1, TS3 is connected to MAC- VRF3 on NVE2, 1333 and TS4 is connected to MAC-VRF1 on NVE2. TS1 has two subnet 1334 prefixes (SN1 and SN2) and TS3 has a single subnet prefix, SN3. The 1335 MAC-VRFs on each NVE are associated with their corresponding IP-VRF 1336 using their IRB interfaces. When TS4 and TS1 exchange intra- subnet 1337 traffic, only L2 forwarding (bridging) part of the IRB solution is 1338 used (i.e., the traffic only goes through their MAC- VRFs); however, 1339 when TS3 wants to forward traffic to SN1 or SN2 sitting behind TS1 1340 (inter-subnet traffic), then both bridging and routing parts of the 1341 IRB solution are exercised (i.e., the traffic goes through the 1342 corresponding MAC-VRFs and IP-VRFs). The following subsections 1343 describe the control and data planes operations for this IRB scenario 1344 in details. 1346 NVE1 +----------+ 1347 SN1--+ +-------------+ | | 1348 |--TS1-----|(MAC- \ | | | 1349 SN2--+ IP1/M1 | VRF1) \ | | | 1350 | (IP-VRF)|---| | 1351 | / | | | 1352 TS2-----|(MAC- / | | MPLS/ | 1353 IP2/M2 | VRF2) | | VxLAN/ | 1354 +-------------+ | NVGRE | 1355 +-------------+ | | 1356 SN3--+--TS3-----|(MAC-\ | | | 1357 IP3/M3 | VRF3)\ | | | 1358 | (IP-VRF)|---| | 1359 | / | | | 1360 TS4-----|(MAC- / | | | 1361 IP4/M4 | VRF1) | | | 1362 +-------------+ +----------+ 1363 NVE2 1365 Figure 7: IRB forwarding on NVEs for subnets behind TSes 1367 9.2.1. Control Plane Operation 1369 Each NVE advertises a Route Type-5 (EVPN RT-5, IP Prefix Route 1370 defined in [I-D.ietf-bess-evpn-prefix-advertisement]) for each of its 1371 subnet prefixes with the IP address of its TS as the next hop 1372 (gateway address field) as follows: 1374 o RD associated with the IP-VRF 1376 o ESI = 0 1378 o Ethernet Tag = 0; 1380 o IP Prefix Length = 0 to 32 or 0 to 128 1382 o IP Prefix = SNi 1384 o Gateway Address = IPi; IP address of TS 1386 o MPLS Label = 0 1388 This EVPN RT-5 is advertised with one or more Route Targets 1389 associated with the IP-VRF from which the route is originated. 1391 Each NVE also advertises an EVPN RT-2 (MAC/IP Advertisement Route) 1392 along with their associated Route Targets and Extended Communities 1393 for each of its TSes exactly as described in section 9.1.1. 1395 Upon receiving the EVPN RT-5 advertisement, the receiving NVE 1396 performs the following: 1398 o It uses the Route Target to identify the corresponding IP-VRF 1400 o It imports the IP prefix into its corresponding IP-VRF that is 1401 configured with an import RT that is one of the RTs being carried 1402 by the EVPN RT-5 route along with the IP address of the associated 1403 TS as its next hop. 1405 When receiving the EVPN RT-2 advertisement, the receiving NVE imports 1406 MAC/IP addresses of the TS into the corresponding MAC-VRF and IP-VRF 1407 per section 9.1.1. When both routes exist, recursive route 1408 resolution is performed to resolve the IP prefix (received in EVPN 1409 RT-5) to its corresponding NVE's IP address (e.g., its BGP next hop). 1410 BGP next hop will be used as the underlay tunnel destination address 1411 (e.g., VTEP DA for VxLAN encapsulation) and Router's MAC will be used 1412 as inner MAC for VxLAN encapsulation. 1414 9.2.2. Data Plane Operation 1416 The following description of the data-plane operation describes just 1417 the logical functions and the actual implementation may differ. Lets 1418 consider data-plane operation when a host on SN1 sitting behind TS1 1419 wants to send traffic to a host sitting behind SN3 behind TS3. 1421 o TS1 send a packet with MAC DA corresponding to the MAC-VRF1 IRB 1422 interface of NVE1, and VLAN-tag corresponding to MAC-VRF1. 1424 o Upon receiving the packet, the ingress NVE1 uses VLAN-tag to 1425 identify the MAC-VRF1. It then looks up the MAC DA and forwards 1426 the frame to its IRB interface just like section 9.1.1. 1428 o The Ethernet header of the packet is stripped and the packet is 1429 fed to the IP-VRF; where, IP lookup is performed on the 1430 destination address. This lookup yields the fields needed for 1431 VxLAN encapsulation with NVE2's MAC address as the inner MAC DA, 1432 NVE'2 IP address as the VTEP DA, and the VNI. MAC SA is set to 1433 NVE1's MAC address and VTEP SA is set to NVE1's IP address. NVE1 1434 also decrements the TTL/hop limit for that packet by one and if it 1435 reaches zero, NVE1 discards the packet. 1437 o The packet is then encapsulated with the proper header based on 1438 the above info and is forwarded to the egress NVE (NVE2). 1440 o On the egress NVE (NVE2), assuming the packet is VxLAN 1441 encapsulated, the VxLAN and the inner Ethernet headers are removed 1442 and the resultant IP packet is fed to the IP-VRF associated with 1443 that the VNI. 1445 o Next, a lookup is performed based on IP DA (which is in SN3) in 1446 the associated IP-VRF of NVE2. The IP lookup yields the access- 1447 facing IRB interface over which the packet needs to be sent. 1448 Before sending the packet over this interface, the ARP table is 1449 consulted to get the destination TS (TS3) MAC address. NVE2 also 1450 decrements the TTL/hop limit for that packet by one and if it 1451 reaches zero, NVE2 discards the packet. 1453 o The IP packet is encapsulated with an Ethernet header with the MAC 1454 SA set to that of the access-facing IRB interface of the egress 1455 NVE (NVE2) and the MAC DA is set to that of destination TS (TS3) 1456 MAC address. The packet is sent to the corresponding MAC-VRF3 and 1457 after a lookup of MAC DA, is forwarded to the destination TS (TS3) 1458 over the corresponding interface. 1460 10. Acknowledgements 1462 The authors would like to thank Sami Boutros, Jeffrey Zhang, 1463 Krzysztof Szarkowicz, Lukas Krattiger and Neeraj Malhotra for their 1464 valuable comments. The authors would also like to thank Linda 1465 Dunbar, Florin Balus, Yakov Rekhter, Wim Henderickx, Lucy Yong, and 1466 Dennis Cai for their feedback and contributions. 1468 11. Security Considerations 1470 The security considerations for layer-2 forwarding in this document 1471 follow that of [RFC7432] for MPLS encapsulation and it follows that 1472 of [RFC8365] for VxLAN or NVGRE encapsulations. This section 1473 describes additional considerations. 1475 This document describes a set of procedures for Inter-Subnet 1476 Forwarding of tenant traffic across PEs (or NVEs). These procedures 1477 include both layer-2 forwarding and layer-3 routing on a packet by 1478 packet basis. The security consideration for layer-3 routing in this 1479 document follows that of [RFC4365] with the exception for the 1480 application of routing protocols between CEs and PEs. Contrary to 1481 [RFC4364], this document does not describe route distribution 1482 techniques between CEs and PEs, but rather considers the CEs as TSes 1483 or VAs that do not run dynamic routing protocols. This can be 1484 considered a security advantage, since dynamic routing protocols can 1485 be blocked on the NVE/PE ACs, not allowing the tenant to interact 1486 with the infrastructure's dynamic routing protocols. 1488 The VPN scheme described in this document does not provide the 1489 quartet of security properties mentioned in [RFC4365] 1490 (confidentiality protection, source authentication, integrity 1491 protection, replay protection). If these are desired, they must be 1492 provided by mechanisms that are outside the scope of the VPN 1493 mechanisms. 1495 In this document, the EVPN RT-5 is used for certain scenarios. This 1496 route uses an Overlay Index that requires a recursive resolution to a 1497 different EVPN route (an EVPN RT-2). Because of this, it is worth 1498 noting that any action that ends up filtering or modifying the EVPN 1499 RT-2 route used to convey the Overlay Indexes, will modify the 1500 resolution of the EVPN RT-5 and therefore the forwarding of packets 1501 to the remote subnet. 1503 12. IANA Considerations 1505 IANA has allocated a new transitive extended community Type of 0x06 1506 and Sub-Type of 0x03 for EVPN Router's MAC Extended Community. 1508 13. References 1510 13.1. Normative References 1512 [I-D.ietf-bess-evpn-prefix-advertisement] 1513 Rabadan, J., Henderickx, W., Drake, J., Lin, W., and A. 1514 Sajassi, "IP Prefix Advertisement in EVPN", draft-ietf- 1515 bess-evpn-prefix-advertisement-11 (work in progress), May 1516 2018. 1518 [I-D.ietf-idr-tunnel-encaps] 1519 Patel, K., Velde, G., Sangli, S., and J. Scudder, "The BGP 1520 Tunnel Encapsulation Attribute", draft-ietf-idr-tunnel- 1521 encaps-22 (work in progress), January 2021. 1523 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1524 Requirement Levels", BCP 14, RFC 2119, 1525 DOI 10.17487/RFC2119, March 1997, 1526 . 1528 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1529 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1530 2006, . 1532 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 1533 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 1534 eXtensible Local Area Network (VXLAN): A Framework for 1535 Overlaying Virtualized Layer 2 Networks over Layer 3 1536 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, 1537 . 1539 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 1540 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 1541 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 1542 2015, . 1544 [RFC7606] Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K. 1545 Patel, "Revised Error Handling for BGP UPDATE Messages", 1546 RFC 7606, DOI 10.17487/RFC7606, August 2015, 1547 . 1549 [RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network 1550 Virtualization Using Generic Routing Encapsulation", 1551 RFC 7637, DOI 10.17487/RFC7637, September 2015, 1552 . 1554 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1555 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1556 May 2017, . 1558 [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., 1559 Uttaro, J., and W. Henderickx, "A Network Virtualization 1560 Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, 1561 DOI 10.17487/RFC8365, March 2018, 1562 . 1564 13.2. Informative References 1566 [I-D.ietf-bess-evpn-irb-extended-mobility] 1567 Malhotra, N., Sajassi, A., Pattekar, A., Lingala, A., 1568 Rabadan, J., and J. Drake, "Extended Mobility Procedures 1569 for EVPN-IRB", draft-ietf-bess-evpn-irb-extended- 1570 mobility-03 (work in progress), May 2020. 1572 [I-D.ietf-nvo3-vxlan-gpe] 1573 Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol 1574 Extension for VXLAN (VXLAN-GPE)", draft-ietf-nvo3-vxlan- 1575 gpe-10 (work in progress), July 2020. 1577 [RFC4365] Rosen, E., "Applicability Statement for BGP/MPLS IP 1578 Virtual Private Networks (VPNs)", RFC 4365, 1579 DOI 10.17487/RFC4365, February 2006, 1580 . 1582 [RFC5798] Nadas, S., Ed., "Virtual Router Redundancy Protocol (VRRP) 1583 Version 3 for IPv4 and IPv6", RFC 5798, 1584 DOI 10.17487/RFC5798, March 2010, 1585 . 1587 [RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. 1588 Rekhter, "Framework for Data Center (DC) Network 1589 Virtualization", RFC 7365, DOI 10.17487/RFC7365, October 1590 2014, . 1592 Authors' Addresses 1594 Ali Sajassi 1595 Cisco Systems 1596 MILPITAS, CALIFORNIA 95035 1597 UNITED STATES 1599 Email: sajassi@cisco.com 1601 Samer Salam 1602 Cisco Systems 1604 Email: ssalam@cisco.com 1606 Samir Thoria 1607 Cisco Systems 1609 Email: sthoria@cisco.com 1611 John E Drake 1612 Juniper 1614 Email: jdrake@juniper.net 1616 Jorge Rabadan 1617 Nokia 1619 Email: jorge.rabadan@nokia.com