idnits 2.17.1 draft-ietf-bess-evpn-irb-extended-mobility-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (2 October 2021) is 936 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC 826' is mentioned on line 145, but not defined == Missing Reference: 'RFC 4861' is mentioned on line 148, but not defined == Missing Reference: 'RFC 7432' is mentioned on line 1034, but not defined == Missing Reference: 'EVPN-INTER-SUBNET' is mentioned on line 230, but not defined == Missing Reference: 'VM-IP1' is mentioned on line 354, but not defined == Missing Reference: 'VM-IP2' is mentioned on line 354, but not defined == Missing Reference: 'VM-IP3' is mentioned on line 354, but not defined == Missing Reference: 'VM-IP4' is mentioned on line 354, but not defined == Missing Reference: 'VM-IP5' is mentioned on line 354, but not defined == Missing Reference: 'VM-IP6' is mentioned on line 354, but not defined == Missing Reference: 'IP1' is mentioned on line 429, but not defined == Missing Reference: 'MAC1' is mentioned on line 359, but not defined == Missing Reference: 'PE1' is mentioned on line 409, but not defined == Missing Reference: 'PE2' is mentioned on line 409, but not defined == Missing Reference: 'PE3' is mentioned on line 483, but not defined == Missing Reference: 'PE4' is mentioned on line 483, but not defined == Missing Reference: 'MAC2' is mentioned on line 363, but not defined == Missing Reference: 'VM-IP1-M1' is mentioned on line 404, but not defined == Missing Reference: 'VM-IP2-M2' is mentioned on line 404, but not defined == Missing Reference: 'VM-IP3-M3' is mentioned on line 404, but not defined == Missing Reference: 'VM-IP4-M4' is mentioned on line 404, but not defined == Missing Reference: 'VM-IP5-M5' is mentioned on line 404, but not defined == Missing Reference: 'VM-IP6-M6' is mentioned on line 404, but not defined == Missing Reference: 'IP7' is mentioned on line 427, but not defined == Missing Reference: 'M1' is mentioned on line 797, but not defined == Missing Reference: 'I1' is mentioned on line 797, but not defined == Missing Reference: 'M2' is mentioned on line 797, but not defined == Missing Reference: 'I3' is mentioned on line 797, but not defined == Missing Reference: 'RFC 7814' is mentioned on line 865, but not defined == Unused Reference: 'RFC7814' is defined on line 1114, but no explicit reference was found in the text == Outdated reference: A later version (-15) exists of draft-ietf-bess-evpn-inter-subnet-forwarding-13 == Outdated reference: A later version (-16) exists of draft-ietf-bess-evpn-proxy-arp-nd-11 ** Downref: Normative reference to an Informational RFC: RFC 7814 Summary: 1 error (**), 0 flaws (~~), 33 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BESS WorkGroup N. Malhotra, Ed. 3 Internet-Draft A. Sajassi 4 Intended status: Standards Track A. Pattekar 5 Expires: 5 April 2022 Cisco Systems 6 J. Rabadan 7 Nokia 8 A. Lingala 9 ATT 10 J. Drake 11 Juniper Networks 12 2 October 2021 14 Extended Mobility Procedures for EVPN-IRB 15 draft-ietf-bess-evpn-irb-extended-mobility-07 17 Abstract 19 Procedure to handle host mobility in a layer 2 Network with EVPN 20 control plane is defined as part of RFC 7432. EVPN has since evolved 21 to find wider applicability across various IRB use cases that include 22 distributing both MAC and IP reachability via a common EVPN control 23 plane. MAC Mobility procedures defined in RFC 7432 are extensible to 24 IRB use cases if a fixed 1:1 mapping between VM IP and MAC is assumed 25 across VM moves. Generic mobility support for IP and MAC that allows 26 these bindings to change across moves is required to support a 27 broader set of EVPN IRB use cases, and requires further 28 consideration. EVPN all-active multi-homing further introduces 29 scenarios that require additional consideration from mobility 30 perspective. This document enumerates a set of design considerations 31 applicable to mobility across these EVPN IRB use cases and defines 32 generic sequence number assignment procedures to address these IRB 33 use cases. 35 Status of This Memo 37 This Internet-Draft is submitted in full conformance with the 38 provisions of BCP 78 and BCP 79. 40 Internet-Drafts are working documents of the Internet Engineering 41 Task Force (IETF). Note that other groups may also distribute 42 working documents as Internet-Drafts. The list of current Internet- 43 Drafts is at https://datatracker.ietf.org/drafts/current/. 45 Internet-Drafts are draft documents valid for a maximum of six months 46 and may be updated, replaced, or obsoleted by other documents at any 47 time. It is inappropriate to use Internet-Drafts as reference 48 material or to cite them other than as "work in progress." 49 This Internet-Draft will expire on 5 April 2022. 51 Copyright Notice 53 Copyright (c) 2021 IETF Trust and the persons identified as the 54 document authors. All rights reserved. 56 This document is subject to BCP 78 and the IETF Trust's Legal 57 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 58 license-info) in effect on the date of publication of this document. 59 Please review these documents carefully, as they describe your rights 60 and restrictions with respect to this document. Code Components 61 extracted from this document must include Simplified BSD License text 62 as described in Section 4.e of the Trust Legal Provisions and are 63 provided without warranty as described in the Simplified BSD License. 65 Table of Contents 67 1. Requirements Language and Terminology . . . . . . . . . . . . 3 68 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 69 2.1. Document Structure . . . . . . . . . . . . . . . . . . . 6 70 3. Optional MAC only RT-2 . . . . . . . . . . . . . . . . . . . 6 71 4. Mobility Use Cases . . . . . . . . . . . . . . . . . . . . . 6 72 4.1. Host MAC+IP Move . . . . . . . . . . . . . . . . . . . . 7 73 4.2. Host IP Move to new MAC . . . . . . . . . . . . . . . . . 7 74 4.2.1. VM Reload . . . . . . . . . . . . . . . . . . . . . . 7 75 4.2.2. MAC Sharing . . . . . . . . . . . . . . . . . . . . . 7 76 4.2.3. Problem . . . . . . . . . . . . . . . . . . . . . . . 8 77 4.3. Host MAC move to new IP . . . . . . . . . . . . . . . . . 9 78 4.3.1. Problem . . . . . . . . . . . . . . . . . . . . . . . 9 79 5. EVPN All Active multi-homed ES . . . . . . . . . . . . . . . 10 80 6. Design Considerations . . . . . . . . . . . . . . . . . . . . 12 81 7. Solution Components . . . . . . . . . . . . . . . . . . . . . 12 82 7.1. Sequence Number Inheritance . . . . . . . . . . . . . . . 12 83 7.2. MAC Sharing . . . . . . . . . . . . . . . . . . . . . . . 13 84 7.3. Multi-homing Mobility Synchronization . . . . . . . . . . 14 85 8. Requirements for Sequence Number Assignment . . . . . . . . . 15 86 8.1. LOCAL MAC-IP learning . . . . . . . . . . . . . . . . . . 15 87 8.2. LOCAL MAC learning . . . . . . . . . . . . . . . . . . . 15 88 8.3. Remote MAC OR MAC-IP Update . . . . . . . . . . . . . . . 16 89 8.4. REMOTE (SYNC) MAC update . . . . . . . . . . . . . . . . 16 90 8.5. REMOTE (SYNC) MAC-IP update . . . . . . . . . . . . . . . 16 91 8.6. Inter-op . . . . . . . . . . . . . . . . . . . . . . . . 17 92 8.7. MAC Sharing Race Condition . . . . . . . . . . . . . . . 17 93 8.8. Mobility Convergence . . . . . . . . . . . . . . . . . . 18 94 8.8.1. Generalized Probing Logic . . . . . . . . . . . . . . 18 95 9. Routed Overlay . . . . . . . . . . . . . . . . . . . . . . . 19 96 10. Duplicate Host Detection . . . . . . . . . . . . . . . . . . 20 97 10.1. Scenario A . . . . . . . . . . . . . . . . . . . . . . . 20 98 10.2. Scenario B . . . . . . . . . . . . . . . . . . . . . . . 20 99 10.2.1. Duplicate IP Detection Procedure for Scenario B . . 21 100 10.3. Scenario C . . . . . . . . . . . . . . . . . . . . . . . 21 101 10.4. Duplicate Host Recovery . . . . . . . . . . . . . . . . 22 102 10.4.1. Route Un-freezing Configuration . . . . . . . . . . 22 103 10.4.2. Route Clearing Configuration . . . . . . . . . . . . 23 104 11. Security Considerations . . . . . . . . . . . . . . . . . . . 23 105 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 106 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 23 107 14. Normative References . . . . . . . . . . . . . . . . . . . . 23 108 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 110 1. Requirements Language and Terminology 112 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 113 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 114 document are to be interpreted as described in [RFC2119]. 116 * EVPN-IRB: A BGP-EVPN distributed control plane based integrated 117 routing and bridging fabric overlay discussed in [EVPN-IRB] 119 * Underlay: IP or MPLS fabric core network that provides IP or MPLS 120 routed reachability between EVPN PEs. 122 * Overlay: VPN or service layer network consisting of EVPN PEs OR 123 VPN provider-edge (PE) switch-router devices that runs on top of 124 an underlay routed core. 126 * EVPN PE: A PE switch-router in a data-center fabric that runs 127 overlay BGP-EVPN control plane and connects to overlay CE host 128 devices. An EVPN PE may also be the first-hop layer-3 gateway for 129 CE/host devices. This document refers to EVPN PE as a logical 130 function in a data-center fabric. This EVPN PE function may be 131 physically hosted on a top-of-rack switching device (ToR) OR at 132 layer(s) above the ToR in the Clos fabric. An EVPN PE is 133 typically also an IP or MPLS tunnel end-point for overlay VPN flow 135 * Symmetric EVPN-IRB: An overlay fabric first-hop routing 136 architecture as defined in [EVPN-IRB], wherein, overlay host-to- 137 host routed inter-subnet flows are routed at both ingress and 138 egress EVPN PEs. 140 * Asymmetric EVPN-IRB: An overlay fabric first-hop routing 141 architecture as defined in [EVPN-IRB], wherein, overlay host-to- 142 host routed inter-subnet flows are routed and bridged at ingress 143 PE and bridged at egress PEs. 145 * ARP: Address Resolution Protocol [RFC 826]. ARP references in 146 this document are equally applicable to ND as well. 148 * ND: IPv6 Neighbor Discovery Protocol [RFC 4861]. 150 * Ethernet-Segment: physical Ethernet or LAG port that connects an 151 access device to an EVPN PE, as defined in [RFC 7432]. 153 * ESI: Ethernet Segment Identifier as defined in [RFC 7432]. 155 * LAG: Layer-2 link-aggregation, also known as layer-2 bundle port- 156 channel, or bond interface. 158 * EVPN all-active multi-homing: PE-CE all-active multi-homing 159 achieved via a multi-homed layer-2 LAG interface on a CE with 160 member links to multiple PEs and related EVPN procedures on the 161 PEs. 163 * RT-2: EVPN route type 2 carrying both MAC and IP reachability. 165 * RT-5: EVPN route type 5 carrying IP prefix reachability. 167 * MAC-IP: IP association for a MAC, referred to in this document may 168 be IPv4, IPv6 or both. 170 * SYNC MAC route: In the context of EVPN multi-homing, this refers 171 to a local MAC route SYNCed from another PE sharing the same ESI. 173 * SYNC MAC-IP route: In the context of EVPN multi-homing, this 174 refers to a local MAC-IP route SYNCed from another PE sharing the 175 same ESI. 177 * SYNC MAC sequence number: In the context of EVPN multi-homing, 178 this refers to sequencee number received with a SYNC MAC route. 180 * SYNC MAC-IP sequence number: In the context of EVPN multi-homing, 181 this refers to sequencee number received with a SYNC MAC-IP route. 183 2. Introduction 185 EVPN-IRB enables capability to advertise both MAC and IP routes via a 186 single MAC+IP RT-2 advertisement. MAC is imported into local bridge 187 MAC table and enables L2 bridged traffic across the network overlay. 188 IP is imported into the local ARP table in an asymmetric IRB design 189 OR imported into the IP routing table in a symmetric IRB design, and 190 enables routed traffic across the layer 2 network overlay. Please 191 refer to [EVPN-IRB] for more background on EVPN IRB forwarding modes. 193 To support EVPN mobility procedure, a single sequence number mobility 194 attribute is advertised with the combined MAC+IP route. A single 195 sequence number advertised with the combined MAC+IP route to resolve 196 both MAC and IP reachability implicitly assumes a 1:1 fixed mapping 197 between IP and MAC. While a fixed 1:1 mapping between IP and MAC is 198 a common use case that could be addressed via existing MAC mobility 199 procedure, additional IRB scenarios need to be considered, that don't 200 necessarily adhere to this assumption. Following IRB mobility 201 scenarios are considered: 203 * VM move results in VM IP and MAC moving together 205 * VM move results in VM IP moving to a new MAC association 207 * VM move results in VM MAC moving to a new IP association 209 While existing MAC mobility procedure can be leveraged for MAC+IP 210 move in the first scenario, subsequent scenarios result in a new MAC- 211 IP association. As a result, a single sequence number assigned 212 independently per-[MAC, IP] is not sufficient to determine most 213 recent reachability for both MAC and IP, unless the sequence number 214 assignment algorithm is designed to allow for changing MAC-IP 215 bindings across moves. 217 Purpose of this draft is to define additional sequence number 218 assignment and handling procedures to adequately address generic 219 mobility support across EVPN-IRB overlay use cases that allow MAC-IP 220 bindings to change across VM moves and can support mobility for both 221 MAC and IP components carried in an EVPN RT-2 for these use cases. 223 In addition, for hosts on an ESI multi-homed to multiple GW devices, 224 additional procedure is proposed to ensure synchronized sequence 225 number assignments across the multi-homing devices. 227 Content presented in this draft is independent of data plane 228 encapsulation used in the overlay being MPLS or NVO Tunnels. It is 229 also largely independent of the EVPN IRB solution being based on 230 symmetric OR asymmetric IRB design as defined in [EVPN-INTER-SUBNET]. 232 In addition to symmetric and asymmetric IRB, mobility solution for a 233 routed overlay, where traffic to an end host in the overlay is always 234 IP routed using EVPN RT-5 is also presented in this document. 236 To summarize, this draft covers mobility mobility for the following 237 independent of the overlay encapsulation being MPLS or an NVO Tunnel: 239 * Symmetric EVPN IRB overlay 240 * Asymmetric EVPN IRB overlay 242 * Routed EVPN overlay 244 2.1. Document Structure 246 Following sections of the document should be condidered informnative: 248 * section 4 and 5 provide the necessary background and problem 249 statement being addressed in this document. 251 * section 6 lists the resulting design considerations for the 252 document. 254 Following sections of the document should be condidered normative: 256 * section 8 describes the mobility and sequence number assigment 257 procedures in an EVPN-IRB overlay required to address the 258 scenarios described in section 4. 260 * section 9 describes the mobility procedures for a routed overlay 261 nextwork as opposed to an IRB overlay. 263 * section 10 describes corresponding duplicate detection procedures 264 for EVPN-IRB and routed overlays. 266 3. Optional MAC only RT-2 268 In an EVPN IRB scenario, where a single MAC+IP RT-2 advertisement 269 carries both IP and MAC routes, a MAC only RT-2 advertisement is 270 redundant for host MACs that are advertised via MAC+IP RT-2. As a 271 result, a MAC only RT-2 is an optional route that may not be 272 advertised from or received at an EVPN PE. This is an important 273 consideration for mobility scenarios discussed in subsequent 274 sections. 276 MAC only RT-2 may still be advertised for non-IP host MACs that are 277 not advertised via MAC+IP RT-2. 279 4. Mobility Use Cases 281 This section describes the IRB mobility use cases considered in this 282 document. Procedures to address them are covered later in section 6 283 and section 7. 285 * Host move results in Host IP and MAC moving together 287 * Host move results in Host IP moving to a new MAC association 288 * Host move results in Host MAC moving to a new IP association 290 4.1. Host MAC+IP Move 292 This is the baseline case, wherein a host move results in both host 293 MAC and IP moving together with no change in MAC-IP binding across a 294 move. Existing MAC mobility defined in RFC 7432 may be leveraged to 295 apply to corresponding MAC+IP route to support this mobility 296 scenario. 298 4.2. Host IP Move to new MAC 300 This is the case, where a host move results in VM IP moving to a new 301 MAC binding. 303 4.2.1. VM Reload 305 A host reload or an orchestrated host move that results in host being 306 re-spawned at a new location may result in host getting a new MAC 307 assignment, while maintaining existing IP address. This results in a 308 host IP move to a new MAC binding: 310 IP-a, MAC-a ---> IP-a, MAC-b 312 4.2.2. MAC Sharing 314 This takes into account scenarios, where multiple hosts, each with a 315 unique IP, may share a common MAC binding, and a host move results in 316 a new MAC binding for the host IP. 318 As an example, hosts running on a single physical server, each with a 319 unique IP, may share the same physical server MAC. In yet another 320 scenario, an L2 access network may be behind a firewall, such that 321 all hosts IPs on the access network are learnt with a common firewall 322 MAC. In all such "shared MAC" use cases, multiple local MAC-IP ARP 323 entries may be learnt with the same MAC. A host IP move, in such 324 scenarios (for e.g., to a new physical server), could result in new 325 MAC association for the host IP. 327 4.2.3. Problem 329 In both of the above scenarios, a combined MAC+IP EVPN RT-2 330 advertised with a single sequence number attribute implicitly assumes 331 a fixed IP to MAC mapping. A host IP move to a new MAC breaks this 332 assumption and results in a new MAC+IP route. If this new MAC+IP 333 route is independently assigned a new sequence number, the sequence 334 number can no longer be used to determine most recent host IP 335 reachability in a symmetric EVPN-IRB design OR the most recent IP to 336 MAC binding in an asymmetric EVPN-IRB design. 338 +------------------------+ 339 | Underlay Network Fabric| 340 +------------------------+ 342 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ 343 | PE1 | | PE2 | | PE3 | | PE4 | | PE5 | | PE6 | 344 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ 345 \ / \ / \ / 346 \ ESI-1 / \ ESI-2 / \ ESI-3 / 347 \ / \ / \ / 348 +\---/+ +\---/+ +\---/+ 349 | \ / | | \ / | | \ / | 350 +--+--+ +--+--+ +--+--+ 351 | | | 352 Server-MAC1 Server-MAC2 Server-MAC3 353 | | | 354 [VM-IP1, VM-IP2] [VM-IP3, VM-IP4] [VM-IP5, VM-IP6] 356 Figure 1 358 As an example, consider a topology shown in Figure 1, with host VMs 359 sharing the physical server MAC. In steady state, [IP1, MAC1] route 360 is learnt at [PE1, PE2] and advertised to remote PEs with a sequence 361 number N. Now, VM-IP1 is moved to Server-MAC2. ARP or ND based 362 local learning at [PE3, PE4] would now result in a new [IP1, MAC2] 363 route being learnt. If route [IP1, MAC2] is learnt as a new MAC+IP 364 route and assigned a new sequence number of say 0, mobility procedure 365 for VM-IP1 will not trigger across the overlay network. 367 A sequence number assignment procedure needs to be defined to 368 unambiguously determine the most recent IP reachability, IP to MAC 369 binding, and MAC reachability for such a MAC sharing scenario. 371 4.3. Host MAC move to new IP 373 This is a scenario where host move or re-provisioning behind a new 374 gateway location may result in host getting a new IP address 375 assigned, while keeping the same MAC. 377 4.3.1. Problem 379 Complication with this scenario is that MAC reachability could be 380 carried via a combined MAC+IP route while a MAC only route may not be 381 advertised at all. A single sequence number association with the 382 MAC+IP route again implicitly assumes a fixed mapping between MAC and 383 IP. A MAC move resulting in a new IP association for the host MAC 384 breaks this assumption and results in a new MAC+IP route. If this 385 new MAC+IP route independently assumes a new sequence number, this 386 mobility attribute can no longer be used to determine most recent 387 host MAC reachability. 389 +------------------------+ 390 | Underlay Network Fabric| 391 +------------------------+ 392 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ 393 | PE1 | | PE2 | | PE3 | | PE4 | | PE5 | | PE6 | 394 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ 395 \ / \ / \ / 396 \ ESI-1 / \ ESI-2 / \ ESI-3 / 397 \ / \ / \ / 398 +\---/+ +\---/+ +\---/+ 399 | \ / | | \ / | | \ / | 400 +--+--+ +--+--+ +--+--+ 401 | | | 402 Server1 Server2 Server3 403 | | | 404 [VM-IP1-M1, VM-IP2-M2] [VM-IP3-M3, VM-IP4-M4] [VM-IP5-M5, VM-IP6-M6] 406 Figure 2 408 As an example, consider a host VM IP1-M1 that is learnt locally at 409 [PE1, PE2] and advertised to remote hosts with a sequence number N. 410 Consider a scenario where this VM with MAC M1 is re-provisioned at 411 server 2, however, as part of this re-provisioning, assigned a 412 different IP address say IP7. [IP7, M1] is learnt as a new route at 413 [PE3, PE4] and advertised to remote PEs with a sequence number of 0. 414 As a result, L3 reachability to IP7 would be established across the 415 overlay, however, MAC mobility procedure for MAC1 will not trigger as 416 a result of this MAC-IP route advertisement. If an optional MAC only 417 route is also advertised, sequence number associated with the MAC 418 only route would trigger MAC mobility as per [RFC7432]. However, in 419 the absence of an additional MAC only route advertisement, a single 420 sequence number advertised with a combined MAC+IP route may not be 421 sufficient to update MAC reachability across the overlay. 423 A MAC-IP sequence number assignment procedure needs to be defined to 424 unambiguously determine the most recent MAC reachability in such a 425 scenario without a MAC only route being advertised. 427 Further, PE1/PE2, on learning new reachability for [IP7, M1] via PE3/ 428 PE4 MUST probe and delete any local IPs associated with MAC M1, such 429 as [IP1, M1] in the above example. 431 Arguably, MAC mobility sequence number defined in [RFC7432], could be 432 interpreted to apply only to the MAC part of MAC-IP route, and would 433 hence cover this scenario. It could hence be interpreted as a 434 clarification to [RFC7432] and one of the considerations for a common 435 sequence number assignment procedure across all MAC-IP mobility 436 scenarios detailed in this document. 438 5. EVPN All Active multi-homed ES 440 +------------------------+ 441 | Underlay Network Fabric| 442 +------------------------+ 444 +-----+ +-----+ 445 | PE1 | | PE2 | 446 +-----+ +-----+ 447 \\ // 448 \\ ESI-1 // 449 \\ /X 450 +\\---//+ 451 | \\ // | 452 +---+---+ 453 | 454 CE1 456 Figure 3 458 Consider an EVPN-IRB overlay network shown in Figure 2, with hosts 459 multi-homed to two or more PE devices via an all-active multi-homed 460 ES. MAC and ARP entries learnt on a local ES may also be 461 synchronized across the multi-homing PE devices sharing this ES. 462 This MAC and ARP SYNC enables local switching of intra and inter 463 subnet ECMP traffic flows from remote hosts. In other words, local 464 MAC and ARP entries on a given ES may be learnt via local learning 465 and / or via sync from another PE device sharing the same ES. 467 For a host that is multi-homed to multiple PE devices via an all- 468 active ES interface, local learning of host MAC and MAC-IP at each PE 469 device is an independent asynchronous event, that is dependent on 470 traffic flow and or ARP / ND response from the host hashing to a 471 directly connected PE on the MC-LAG interface. As a result, sequence 472 number mobility attribute value assigned to a locally learnt MAC or 473 MAC-IP route at each device may not always be the same, depending on 474 transient states on the device at the time of local learning. 476 As an example, consider a host VM that is deleted from ESI-2 and 477 moved to ESI-1. It is possible for host to be learnt on say, PE1 478 following deletion of the remote route from [PE3, PE4], while being 479 learnt on PE2 prior to deletion of remote route from [PE3, PE4]. If 480 so, PE1 would process local host route learning as a new route and 481 assign a sequence number of 0, while PE2 would process local host 482 route learning as a remote to local move and assign a sequence number 483 of N+1, N being the existing sequence number assigned at [PE3, PE4]. 485 Inconsistent sequence numbers advertised from multi-homing devices 486 introduces: 488 * Ambiguity with respect to how the remote PEs should handle paths 489 with same ESI and different sequence numbers. A remote PE may not 490 program ECMP paths if it receives routes with different sequence 491 numbers from a set of multi-homing PEs sharing the same ESI. 493 * Breaks consistent route versioning across the network overlay that 494 is needed for EVPN mobility procedures to work. 496 As an example, in this inconsistent state, PE2 would drop a remote 497 route received for the same host with sequence number N (as its local 498 sequence number is N+1), while PE1 would install it as the best route 499 (as its local sequence number is 0). 501 There is need for a mechanism to ensure consistency of sequence 502 numbers advertised from a set of multi-homing devices for EVPN 503 mobility to work reliably. 505 In order to support mobility for multi-homed hosts using the sequence 506 number mobility attribute, local MAC and MAC-IP routes learnt on a 507 multi-homed ES MUST be advertised with the same sequence number by 508 all PE devices that the ES is multi-homed to. There is need for a 509 mechanism to ensure consistency of sequence numbers assigned across 510 these PEs. 512 6. Design Considerations 514 To summarize, sequence number assignment scheme and implementation 515 must take following considerations into account: 517 * MAC+IP may be learnt on an ES multi-homed to multiple PE devices, 518 hence requires sequence numbers to be synchronized across multi- 519 homing PE devices. 521 * MAC only RT-2 is optional in an IRB scenario and may not 522 necessarily be advertised in addition to MAC+IP RT-2. 524 * Single MAC may be associated with multiple IPs, i.e., multiple 525 host IPs may share a common MAC. 527 * Host IP move could result in host moving to a new MAC, resulting 528 in a new IP to MAC association and a new MAC+IP route. 530 * Host MAC move to a new location could result in host MAC being 531 associated with a different IP address, resulting in a new MAC to 532 IP association and a new MAC+IP route. 534 * LOCAL MAC-IP learn via ARP would always accompanied by a LOCAL MAC 535 learn event resulting from the ARP packet. MAC and MAC-IP 536 learning, however, could happen in any order. 538 * Use cases discussed earlier that do not maintain a constant 1:1 539 MAC-IP mapping across moves could potentially be addressed by 540 using separate sequence numbers associated with MAC and IP 541 components of MAC+IP route. Maintaining two separate sequence 542 numbers however adds significant overhead with respect to 543 complexity, debugability, and backward compatibility. Hence, this 544 document addresses these requirements via a single sequence number 545 attribute. 547 7. Solution Components 549 This section goes over main components of the EVPN IRB mobility 550 solution proposed in this draft. Later sections will go over exact 551 sequence number assignment procedures resulting from concepts 552 described in this section. 554 7.1. Sequence Number Inheritance 556 Main idea presented here is to view a LOCAL MAC-IP route as a child 557 of the corresponding LOCAL MAC only route that inherits the sequence 558 number attribute from the parent LOCAL MAC only route: 560 Mx-IPx -----> Mx (seq# = N) 562 As a result, both parent MAC and child MAC-IP routes share one common 563 sequence number associated with the parent MAC route. Doing so 564 ensures that a single sequence number attribute carried in a combined 565 MAC+IP route represents sequence number for both a MAC only route as 566 well as a MAC+IP route, and hence makes the MAC only route truly 567 optional. As a result, optional MAC only route with its own sequence 568 number is not required to establish most recent reachability for a 569 MAC in the overlay network. Specifically, this enables a MAC to 570 assume a different IP address on a move, and still be able to 571 establish most recent reachability to the MAC across the overlay 572 network via mobility attribute associated with the MAC+IP route 573 advertisement. As an example, when Mx moves to a new location, it 574 would result in LOCAL Mx being assigned a higher sequence number at 575 its new location as per RFC 7432. If this move results in Mx 576 assuming a different IP address, IPz, LOCAL Mx+IPz route would 577 inherit the new sequence number from Mx. 579 LOCAL MAC and LOCAL MAC-IP routes would typically be sourced from 580 data plane learning and ARP learning respectively, and could get 581 learnt in control plane in any order. Implementation could either 582 replicate inherited sequence number in each MAC-IP entry OR maintain 583 a single attribute in the parent MAC by creating a forward reference 584 LOCAL MAC object for cases where a LOCAL MAC-IP is learnt before the 585 LOCAL MAC. 587 Arguably, this inheritance may be assumed from RFC 7432, in which 588 case, the above may be interpreted as a clarification with respect to 589 interpretation of a MAC sequence number in a MAC-IP route. 591 7.2. MAC Sharing 593 Further, for the shared MAC scenario, this would result in multiple 594 LOCAL MAC-IP siblings inheriting sequence number attribute from a 595 common parent MAC route: 597 Mx-IP1 ----- 598 | | 599 Mx-IP2 ----- 600 . | 601 . +---> Mx (seq# = N) 602 . | 603 Mx-IPw ----- 604 | | 605 Mx-IPx ----- 607 Figure 4 609 In such a case, a host-IP move to a different physical server would 610 result in IP moving to a new MAC binding. A new MAC-IP route 611 resulting from this move must now be advertised with a sequence 612 number that is higher than the previous MAC-IP route for this IP, 613 advertised from the prior location. As an example, consider a route 614 Mx-IPx that is currently advertised with sequence number N from PE1. 615 IPx moving to a new physical server behind PE2 results in IPx being 616 associated with MAC Mz. A new local Mz-IPx route resulting from this 617 move at PE2 must now be advertised with a sequence number higher than 618 N. This is so that PE devices, including PE1, PE2, and other remote 619 PE devices that are part of the overlay can clearly determine and 620 program the most recent MAC binding and reachability for the IP. 621 PE1, on receiving this new Mz-IPx route with sequence number say, 622 N+1, for symmetric IRB case, would update IPx reachability via PE2 in 623 forwarding, for asymmetric IRB case, would update IPx's ARP binding 624 to Mz. In addition, PE1 would clear and withdraw the stale Mx-IPx 625 route with the lower sequence number. 627 This also implies that sequence number associated with local MAC Mz 628 and all local MAC-IP children of Mz at PE2 must now be incremented to 629 N+1, and re-advertised across the overlay. While this re- 630 advertisement of all local MAC-IP children routes affected by the 631 parent MAC route is an overhead, it avoids the need for two separate 632 sequence number attributes to be maintained and advertised for IP and 633 MAC components of MAC+IP RT-2. Implementation would need to be able 634 to lookup MAC-IP routes for a given IP and update sequence number for 635 it's parent MAC and its MAC-IP children. 637 7.3. Multi-homing Mobility Synchronization 639 In order to support mobility for multi-homed hosts, local MAC and 640 MAC-IP routes learnt on a shared ES MUST be advertised with the same 641 sequence number by all PE devices that the ES is multi-homed to. 642 This also applies to local MAC only routes. LOCAL MAC and MAC-IP may 643 be learnt natively via data plane and ARP/ND respectively as well as 644 via SYNC from another multi-homing PE to achieve local switching. 645 Local and SYNC route learning can happen in any order. Local MAC-IP 646 routes advertised by all multi-homing PE devices sharing the ES must 647 carry the same sequence number, independent of the order in which 648 they are learnt. This implies: 650 * On local or SYNC MAC-IP route learning, sequence number for the 651 local MAC-IP route MUST be compared and updated to the higher 652 value. 654 * On local or SYNC MAC route learning, sequence number for the local 655 MAC route MUST be compared and updated to the higher value. 657 If an update to local MAC-IP sequence number is required as a result 658 of above comparison with SYNC MAC-IP route, it would essentially 659 amount to a sequence number update on the parent local MAC, resulting 660 in inherited sequence number update on the MAC-IP route. 662 8. Requirements for Sequence Number Assignment 664 Following sections summarize sequence number assignment procedure 665 needed on local and SYNC MAC and MAC-IP route learning events in 666 order to accomplish the above. 668 8.1. LOCAL MAC-IP learning 670 A local Mx-IPx learning via ARP or ND should result in computation OR 671 re-computation of parent MAC Mx's sequence number, following which 672 the MAC-IP route Mx-IPx would simply inherit parent MAC's sequence 673 number. Parent MAC Mx Sequence number should be computed as follows: 675 * MUST be higher than any existing remote MAC route for Mx, as per 676 RFC 7432. 678 * MUST be at least equal to corresponding SYNC MAC sequence number 679 if one is present. 681 * If the IP is also associated with a different remote MAC "Mz", 682 MUST be higher than "Mz" sequence number. 684 Once new sequence number for MAC route Mx is computed as per above, 685 all LOCAL MAC-IPs associated with MAC Mx MUST inherit the updated 686 sequence number. 688 8.2. LOCAL MAC learning 690 Local MAC Mx Sequence number should be computed as follows: 692 * MUST be higher than any existing remote MAC route for Mx, as per 693 RFC 7432. 695 * MUST be at least equal to corresponding SYNC MAC sequence number 696 if one is present. 698 * Once new sequence number for MAC route Mx is computed as per 699 above, all LOCAL MAC-IPs associated with MAC Mx MUST inherit the 700 updated sequence number. 702 Note that the local MAC sequence number might already be present if 703 there was a local MAC-IP learnt prior to the local MAC, in which case 704 the above may not result in any change in local MAC's sequence 705 number. 707 8.3. Remote MAC OR MAC-IP Update 709 On receiving a remote MAC OR MAC-IP route update associated with a 710 MAC Mx with a sequence number that is 712 * either higher than the sequence number assigned to a LOCAL route 713 for MAC Mx, 715 * or equal to the sequence number assigned to a LOCAL route for MAC 716 Mx, but the remote route is selected as best because of lower VTEP 717 IP as per [RFC 7432], 719 following handling is required on the receiving PE: 721 * PE MUST trigger probe and deletion procedure for all LOCAL IPs 722 associated with MAC Mx. 724 * PE MUST trigger deletion procedure for LOCAL MAC route for Mx. 726 8.4. REMOTE (SYNC) MAC update 728 Corresponding local MAC Mx (if present) sequence number should be re- 729 computed as follows: 731 * If the current sequence number is less than the received SYNC MAC 732 sequence number, it MUST be increased to be equal to received SYNC 733 MAC sequence number. 735 * If a LOCAL MAC sequence number is updated as a result of the 736 above, all LOCAL MAC-IPs associated with MAC Mx MUST inherit the 737 updated sequence number. 739 8.5. REMOTE (SYNC) MAC-IP update 741 If this is a SYNCed MAC-IP on a local ES, it would also result in a 742 derived SYNC MAC Mx route entry, as MAC only RT-2 advertisement is 743 optional. Corresponding local MAC Mx (if present) sequence number 744 should be re-computed as follows: 746 * If the current sequence number is less than the received SYNC MAC 747 sequence number, it MUST be increased to be equal to received SYNC 748 MAC sequence number. 750 * If a LOCAL MAC sequence number is updated as a result of the 751 above, all LOCAL MAC-IPs associated with MAC Mx MUST inherit the 752 updated sequence number. 754 8.6. Inter-op 756 In general, if all PE nodes in the overlay network follow the above 757 sequence number assignment procedure, and the PE is advertising both 758 MAC+IP and MAC routes, sequence number advertised with the MAC and 759 MAC+IP routes with the same MAC would always be the same. However, 760 an inter-op scenario with a different implementation could arise, 761 where a PE implementation non-compliant with this document or with 762 RFC 7432 assigns and advertises independent sequence numbers to MAC 763 and MAC+IP routes. To handle this case, if different sequence 764 numbers are received for remote MAC+IP and corresponding remote MAC 765 routes from a remote PE, sequence number associated with the remote 766 MAC route should be computed as: 768 * Highest of the all received sequence numbers with remote MAC+IP 769 and MAC routes with the same MAC. 771 * MAC sequence number would be re-computed on a MAC or MAC+IP route 772 withdraw as per above. 774 A MAC and / or IP move to the local PE would now result in the MAC 775 (and hence all MAC-IP) sequence numbers incremented from the above 776 computed remote MAC sequence number. 778 If MAC only routes are not advertised at all, and different sequence 779 numbers are received with multiple MAC+IP routes for a given MAC, 780 sequence number associated with the derived remote MAC route should 781 still be computed as the highest of the all received MAC+IP sequence 782 numbers with the same MAC. 784 8.7. MAC Sharing Race Condition 786 In a MAC sharing use case described in section 6.2, a race condition 787 is possible with simultaneous host moves between a pair of PEs. As 788 an example, consider PE1 with local host IPs I1 and I2 sharing MAC 789 M1, and PE2 with local host IPs I3 and I4 sharing MAC M2. A 790 simultaneous move of I1 from PE1 to PE2 and of I3 from PE2 to PE1, 791 such that I3 is learnt on PE1 before I1's local entry has been probed 792 out on PE1 and/or I1 is learnt on PE2 before I3's local entry has 793 been probed out on PE2 may trigger a race condition. This race 794 condition together with MAC sequence number assignment rules defined 795 in section 7.1 can cause new mac-ip routes [I1, M2] and [I3, M1] to 796 bounce a couple of times with an incremented sequence number until 797 stale entries [I1, M1] and [I3, M2] have been probed out from PE1 and 798 PE2 respectively. An implementation MUST ensure proper probing 799 procedures to remove stale ARP, ND, and local MAC entries, following 800 a move, on learning remote routes as defined in section 7.3 (and as 801 per [EVPN-IRB]) to minimize exposure to this race condition. 803 8.8. Mobility Convergence 805 This sections is to be treated as optional and details ARP and ND 806 probing procedures that MAY be implemented to achieve faster host re- 807 learning and convergence on mobility events. 809 * Following a host move from PE1 to PE2, the host's MAC is 810 discovered at PE2 as a local MAC via a data frames received from 811 the host. If PE2 has a prior REMOTE MAC-IP host route for this 812 MAC from PE1, an ARP/ND probe MAY be triggered at PE2 to learn the 813 MAC-IP as a local adjacency and trigger EVPN RT-2 advertisement 814 for this MAC-IP across the overlay with new reachability via PE2. 815 This results in a reliable "event based" host IP learning 816 triggered by a "MAC learning event" across the overlay, and hence 817 faster convergence of overlay routed flows to the host. 819 * Following a host move from PE1 to PE2, once PE1 receives a MAC or 820 MAC-IP route from PE2 with a higher sequence number, an ARP/ND 821 probe MAY be triggered at PE1 to clear the stale local MAC-IP 822 neighbor adjacency OR re-learn the local MAC-IP in case the host 823 has moved back or is duplicate. 825 * Following a local MAC age-out, if there is a local IP adjacency 826 with this MAC, an ARP/ND probe MAY be triggered for this IP to 827 either re-learn the local MAC and maintain local l3 and l2 828 reachability to this host OR to clear the ARP/ND entry in case the 829 host is indeed no longer local. Note that this accomplishes 830 clearing of stale ARP entries, triggered by a MAC age-out event 831 even when the ARP refresh timer was longer than the MAC age-out 832 timer. Clearing of stale IP neighbor entries in-turn facilitates 833 traffic convergence in the event that the host was silent and not 834 discovered at its new location. Once stale neighbor entry for the 835 host is cleared, routed traffic flow destined for the host can re- 836 trigger ARP/ND discovery for this host at the new location. 838 8.8.1. Generalized Probing Logic 840 Above probing logic may be generalized as probing for an IP neighbor 841 anytime a resolving parent MAC route is "inconsistent" with the MAC- 842 IP neighbor route, where being inconsistent is defined as being not 843 present OR conflicting in terms of the route source being local OR 844 remote. MAC-IP to MAC parent relationship described earlier in this 845 document in section 6.1 MAY be used to achieve this logic. 847 9. Routed Overlay 849 An additional use case is possible, such that traffic to an end host 850 in the overlay is always IP routed. In a purely routed overlay such 851 as this: 853 * A host MAC is never advertised in EVPN overlay control plane. 855 * Host /32 or /128 IP reachability is distributed across the overlay 856 via EVPN route type 5 (RT-5) along with a zero or non- zero ESI. 858 * An overlay IP subnet may still be stretched across the underlay 859 fabric, however, intra-subnet traffic across the stretched overlay 860 is never bridged. 862 * Both inter-subnet and intra-subnet traffic, in the overlay is IP 863 routed at the EVPN PE. 865 Please refer to [RFC 7814] for more details. 867 Host mobility within the stretched subnet would still need to be 868 supported for this use. In the absence of any host MAC routes, 869 sequence number mobility EXT-COMM specified in [RFC7432], section 7.7 870 may be associated with a /32 OR /128 host IP prefix advertised via 871 EVPN route type 5. MAC mobility procedures defined in RFC 7432 can 872 now be applied as is to host IP prefixes: 874 * On LOCAL learning of a host IP, on a new ESI, host IP MUST be 875 advertised with a sequence number attribute that is higher than 876 what is currently advertised with the old ESI. 878 * On receiving a host IP route advertisement with a higher sequence 879 number, a PE MUST trigger ARP/ND probe and deletion procedure on 880 any LOCAL route for that IP with a lower sequence number. A PE 881 would essentially move the forwarding entry to point to the remote 882 route with a higher sequence number and send an ARP/ND PROBE for 883 the local IP route. If the IP has indeed moved, PROBE would 884 timeout and the local IP host route would be deleted. 886 Note that there is still only one sequence number associated with a 887 host route at any time. For earlier use cases where a host MAC is 888 advertised along with the host IP, a sequence number is only 889 associated with a MAC. Only if the MAC is not advertised at all, as 890 in this use case, is a sequence number associated with a host IP. 892 Note that this mobility procedure would not apply to "anycast IPv6" 893 hosts advertised via NA messages with 0-bit=0. Please refer to 894 [EVPN-PROXY-ARP]. 896 10. Duplicate Host Detection 898 Duplicate host detection scenarios across EVPN IRB can be classified 899 as follows: 901 * Scenario A: where two hosts have the same MAC (host IPs may or may 902 not be duplicate). 904 * Scenario B: where two hosts have the same IP but different MACs. 906 * Scenario C: where two hosts have the same IP and host MAC is not 907 advertised at all. 909 Duplicate detection procedures for scenario B and C would not apply 910 to "anycast IPv6" hosts advertised via NA messages with 0-bit=0. 911 Please refer to [EVPN-PROXY-ARP]. 913 10.1. Scenario A 915 For all use cases where duplicate hosts have the same MAC, MAC is 916 detected as duplicate via duplicate MAC detection procedure described 917 in RFC 7432. Corresponding MAC-IP routes with the same MAC do not 918 require duplicate detection and MUST simply inherit the DUPLICATE 919 property from the corresponding MAC route. In other words, if a MAC 920 route is in DUPLICATE state, all corresponding MAC-IP routes MUST 921 also be treated as DUPLICATE. Duplicate detection procedure need 922 only be applied to MAC routes. 924 10.2. Scenario B 926 Due to misconfiguration, a situation may arise where hosts with 927 different MACs are configured with the same IP. This scenario would 928 not be detected by existing duplicate MAC detection procedure and 929 would result in incorrect forwarding of routed traffic destined to 930 this IP. 932 Such a situation, on LOCAL MAC-IP learning, would be detected as a 933 move scenario via the following local MAC sequence number computation 934 procedure described earlier in section 6.1: 936 * If the IP is also associated with a different remote MAC "Mz", 937 MUST be higher than "Mz" sequence number. 939 Such a move that results in sequence number increment on local MAC 940 because of a remote MAC-IP route associated with a different MAC MUST 941 be counted as an "IP move" against the "IP" independent of MAC. 942 Duplicate detection procedure described in RFC 7432 can now be 943 applied to an "IP" entity independent of MAC. Once an IP is detected 944 as DUPLICATE, corresponding MAC-IP route should be treated as 945 DUPLICATE. Associated MAC routes and any other MAC-IP routes 946 associated with this MAC should not be affected. 948 10.2.1. Duplicate IP Detection Procedure for Scenario B 950 Duplicate IP detection procedure for such a scenario is specified in 951 [EVPN-PROXY-ARP]. What counts as an "IP move" in this scenario is 952 further clarified as follows: 954 * On learning a LOCAL MAC-IP route Mx-IPx, check if there is an 955 existing REMOTE OR LOCAL route for IPx with a different MAC 956 association, say, Mz-IPx. If so, count this as an "IP move" count 957 for IPx, independent of the MAC. 959 * On learning a REMOTE MAC-IP route Mz-IPx, check if there is an 960 existing LOCAL route for IPx with a different MAC association, 961 say, Mx-IPx. If so, count this as an "IP move" count for IPx, 962 independent of the MAC. 964 A MAC-IP route SHOULD be treated as DUPLICATE if either of the 965 following two conditions are met: 967 * Corresponding MAC route is marked as DUPLICATE via existing 968 duplicate detection procedure. 970 * Corresponding IP is marked as DUPLICATE via extended procedure 971 described above. 973 10.3. Scenario C 975 For a purely routed overlay scenario described in section 8, where 976 only a host IP is advertised via EVPN RT-5, together with a sequence 977 number mobility attribute, duplicate MAC detection procedures 978 specified in RFC 7432 can be intuitively applied to IP only host 979 routes for the purpose of duplicate IP detection. 981 * On learning a LOCAL host IP route IPx, check if there is an 982 existing REMOTE OR LOCAL route for IPx with a different ESI 983 association. If so, count this as an "IP move" count for IPx. 985 * On learning a REMOTE host IP route IPx, check if there is an 986 existing LOCAL route for IPx with a different ESI association. If 987 so, count this as an "IP move" count for IPx. 989 * With configurable parameters "N" and "M", If "N" IP moves are 990 detected within "M" seconds for IPx, treat IPx as DUPLICATE. 992 10.4. Duplicate Host Recovery 994 Once a MAC or IP is marked as DUPLICATE and FROZEN, corrective action 995 must be taken to un-provision one of the duplicate MAC or IP. Un- 996 provisioning a duplicate MAC or IP in this context refers to a 997 corrective action taken on the host side. Once one of the duplicate 998 MAC or IP is un-provisioned, normal operation would not resume until 999 the duplicate MAC or IP ages out, following this correction, unless 1000 additional action is taken to speed up recovery. 1002 This section lists possible additional corrective actions that could 1003 be taken to achieve faster recovery to normal operation. 1005 10.4.1. Route Un-freezing Configuration 1007 Unfreezing the DUPLICATE OR FROZEN MAC or IP via a CLI can be 1008 leveraged to recover from DUPLICATE and FROZEN state following 1009 corrective un-provisioning of the duplicate MAC or IP. 1011 Unfreezing the frozen MAC or IP via a CLI at a PE should result in 1012 that MAC OR IP being advertised with a sequence number that is higher 1013 than the sequence number advertised from the other location of that 1014 MAC or IP. 1016 Two possible corrective un-provisioning scenarios exist: 1018 * Scenario A: A duplicate MAC or IP may have been un-provisioned at 1019 the location where it was NOT marked as DUPLICATE and FROZEN. 1021 * Scenario B: A duplicate MAC or IP may have been un-provisioned at 1022 the location where it was marked as DUPLICATE and FROZEN. 1024 Unfreezing the DUPLICATE and FROZEN MAC or IP, following the above 1025 corrective un-provisioning scenarios would result in recovery to 1026 steady state as follows: 1028 * Scenario A: If the duplicate MAC or IP was un-provisioned at the 1029 location where it was NOT marked as DUPLICATE, unfreezing the 1030 route at the FROZEN location will result in the route being 1031 advertised with a higher sequence number. This would in-turn 1032 result in automatic clearing of local route at the PE location, 1033 where the host was un-provisioned via ARP/ND PROBE and DELETE 1034 procedure specified earlier in section 8 and in [RFC 7432]. 1036 * Scenario B: If the duplicate host is un-provisioned at the 1037 location where it was marked as DUPLICATE, unfreezing the route 1038 will trigger an advertisement with a higher sequence number to the 1039 other location. This would in-turn trigger re-learning of local 1040 route at the remote location, resulting in another advertisement 1041 with a higher sequence number from the remote location. Route at 1042 the local location would now be cleared on receiving this remote 1043 route advertisement, following the ARP/ND PROBE. 1045 Note that the probes referred to in the above scenarios are event 1046 driven probes resulting from receiving a route with a higher sequence 1047 number. Periodic probes resulting from refresh timers may also occur 1048 in addition as completely independent probes. 1050 10.4.2. Route Clearing Configuration 1052 In addition to the above, route clearing CLIs may also be leveraged 1053 to clear the local MAC or IP route, to be executed AFTER the 1054 duplicate host is un-provisioned: 1056 * clear mac CLI: A clear MAC CLI can be leveraged to clear a 1057 DUPLICATE MAC route, to recover from a duplicate MAC scenario. 1059 * clear ARP/ND: A clear ARP/ND CLI may be leveraged to clear a 1060 DUPLICATE IP route to recover from a duplicate IP scenario. 1062 Note that the route unfreeze CLI may still need to be run if the 1063 route was un-provisioned and cleared from the NON-DUPLICATE / NON- 1064 FROZEN location. Given that unfreezing of the route via the un- 1065 freeze CLI would any ways result in auto-clearing of the route from 1066 the "un- provisioned" location, as explained in the prior section, 1067 need for a route clearing CLI for recovery from DUPLICATE / FROZEN 1068 state is truly optional. 1070 11. Security Considerations 1072 This document raises no new security issues for EVPN. 1074 12. IANA Considerations 1076 None. 1078 13. Acknowledgements 1080 Authors would like to thank Vibov Bhan and Patrice Brisset for 1081 feedback the process of design and implementation of procedures 1082 defined in this document. Authors would like to thank Wen Lin for a 1083 detailed review and valuable comments related to MAC sharing race 1084 conditions. Authors would also like to thank Saumya Dikshit for a 1085 detailed review and valuable comments across the document. 1087 14. Normative References 1089 [EVPN-IRB] Sajassi, A., Salam, S., Thoria, S., Drake, J., and J. 1090 Rabadan, "Integrated Routing and Bridging in EVPN", Work 1091 in Progress, Internet-Draft, draft-ietf-bess-evpn-inter- 1092 subnet-forwarding-13, 10 February 2021, 1093 . 1096 [EVPN-PROXY-ARP] 1097 Rabadan, J., Sathappan, S., Nagaraj, K., Hankins, G., and 1098 T. King, "Operational Aspects of Proxy-ARP/ND in EVPN 1099 Networks", Work in Progress, Internet-Draft, draft-ietf- 1100 bess-evpn-proxy-arp-nd-11, 7 January 2021, 1101 . 1104 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1105 Requirement Levels", BCP 14, RFC 2119, 1106 DOI 10.17487/RFC2119, March 1997, 1107 . 1109 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 1110 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 1111 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 1112 2015, . 1114 [RFC7814] Xu, X., Jacquenet, C., Raszuk, R., Boyes, T., and B. Fee, 1115 "Virtual Subnet: A BGP/MPLS IP VPN-Based Subnet Extension 1116 Solution", RFC 7814, DOI 10.17487/RFC7814, March 2016, 1117 . 1119 Authors' Addresses 1121 Neeraj Malhotra (editor) 1122 Cisco Systems 1123 170 W. Tasman Drive 1124 San Jose, CA 95134 1125 United States of America 1127 Email: nmalhotr@cisco.com 1129 Ali Sajassi 1130 Cisco Systems 1131 170 W. Tasman Drive 1132 San Jose, CA 95134 1133 United States of America 1135 Email: sajassi@cisco.com 1136 Aparna Pattekar 1137 Cisco Systems 1138 170 W. Tasman Drive 1139 San Jose, CA 95134 1140 United States of America 1142 Email: apjoshi@cisco.com 1144 Jorge Rabadan 1145 Nokia 1146 777 E. Middlefield Road 1147 Mountain View, CA 94043 1148 United States of America 1150 Email: jorge.rabadan@nokia.com 1152 Avinash Lingala 1153 ATT 1154 200 S. Laurel Avenue 1155 Middletown, CA 07748 1156 United States of America 1158 Email: ar977m@att.com 1160 John Drake 1161 Juniper Networks 1163 Email: jdrake@juniper.net