EVPN Egress Protection

A principal feature of EVPN is the ability to support multi-homing from a customer equipment (CE) to multiple PE with Ethernet segment (ES) links. This draft specifies a egress protection mechanism in the multi-homed cases and enhance EVPN convergency on the egress PE node failures and egress ES link failures, especially for the convergency of BUM packets () and the unified signallings () for the MPLS SPEs and TPEs.

This document uses the following acronyms and terms:

All-Active Redundancy Mode - When a device is multihomed to a group of two or more PEs and when all PEs in such redundancy group can forward traffic to/from the multihomed device or network for a given VLAN.
Backup egress router - Given an egress-protected tunnel and its egress router, this is another router that has connectivity with all or a subset of the destinations of the egress-protected services carried by the egress-protected tunnel.
SPE - Stitching PE, the PEs to do label swapping operation for the EVPN labels. It is similar to the SPE of MS-PWs.
TPE - Target PE, the PEs to do EVPN forwarding for the overlay network.
BUM - Broadcast, Unknown unicast, and Multicast.
CE - Customer Edge equipment.
DCI - Data Center Interconnect.
EELP bypass tunnel - Egress ESI Link Protection bypass tunnel - A tunnel used to reroute service packets upon an egress ESI link failure.
Egress failure - An egress node failure or an egress link failure.
Egress link failure - A failure of the egress link (e.g., PE-CE link, attachment circuit) of a service.
Egress loopback - the loopback interface on the Egress router, whose IP address is the destination of the Egress-protected tunnel.
Egress node failure - A failure of an egress router.
Egress router - A router at the egress endpoint of a tunnel. It hosts service instances for all the services carried by the tunnel and has connectivity with the destinations of the services.
ESI - Ethernet Segment Identifier - A unique non-reserved identifier that identifies an Ethernet segment.
NVE - Network Virtualization Edge.
OPE - Originating PE - the original Router of an EVPN route.
PE - Provider Edge equipment. Note that VTEP/NVE are also called as PE in this draft.
PLR - A router at the point of local repair. In egress node protection, it is the penultimate hop router on an egress-protected tunnel. In egress link protection, it is the egress router of the egress- protected tunnel.
Protector - A role acted by a router as an alternate of a protected egress router, to handle service packets in the event of an egress failure. A protector is physically independent of the egress router.
Protector loopback - the loopback interface on the Protector, whose IP address is the destination of the Egress-protected tunnel.
Single-Active Redundancy Mode - When a device or a network is multihomed to a group of two or more PEs and when only a single PE in such a redundancy group can forward traffic to/from the multihomed device or network for a given VLAN.
VTEP - VXLAN Tunnel End Point.
VXLAN - Virtual eXtensible Local Area Network [RFC7348].
GRT - Global Routing Table.
EVPN SID - SRv6 SID for EVPN Instances, e.g. End.DT2M SID, End.DT2U SID, End.DX2 SID, End.DX2V SID.
DF - Designated Forwarder.
NDF - non-DF, non Designated-Forwarder.
NDF-Bias - An exception for filtering bypassed BUM packets. It says that when an outgoing AC is a NDF on its ES, the bypass-BUM filter rules will not be applied for that AC.

In the scenario illustrated in , where an CE1 is dual-homed to PE1 and PE2 to access the VXLAN/SRv6 network, which enhances network access reliability. When one PE fails, services can be rapidly switched to the other PE, minimizing the impact on services. As shown in , the EVPN instance EVI1 has three PEs, PE1, PE2 and PE3. The PE address of PE1 is IP1 and the PE address of PE2 is IP2, the PE address of PE3 is IP3, they are three different IP addresses. The BGP update-source of PE1 is IP_N1, of PE2 is IP_N2, and of PE3 is IP_N3. LOC1 is the prefix from which IP1 is allocated, LOC2 is the prefix from which IP2 is allocated, LOC3 is the prefix from which IP3 is allocated. Note that IP_N1 must not be allocated in prefix LOC1, IP_N2 must not be allocated in prefix LOC2. But IP_N3 may be the same as IP3.

Both PE1 and PE2 will advertise an IGP route for the prefix LOC1. The prefix LOC1 on PE1 is called LOC1_E, The prefix LOC1 on PE2 is called LOC1_P. Then we make the metric of LOC1_P lower than LOC1_E. Then we do the same for LOC2. As a result, the PLR node will install an egress-FRR entry for LOC1 and LOC2. The primary egress for LOC1 is PE1, the backup egress for LOC1 is PE2. So when PE3 send a EVPN data packet to an IP address of LOC1, the PLR node will use PE1 as the primary path, and use PE2 as the backup path. The different settings between VXLAN EVPN scenario and SRv6 EVPN scenario are described in the following list:

In VXLAN EVPN scenario: LOC1_E is a loopback interface on PE1, LOC1_P is a loopback interface on PE2. Both of the two loopback interfaces are configured with the sam IP address IP1. LOC1 is the /32 or /128 prefix for IP1. So we also use LOC1 to stand for IP1 in VXLAN context in the remainder of this draft. The loopback interface for LOC1_E is IP1's egress loopback interface. The loopback interface for LOC1_P is IP1's protector loopback interface. Then we do the same for IP2.
In SRv6 EVPN scenario: IP1 is the node SID of PE1, LOC1_E is the SRv6 locator for IP1. IP2 is the node SID of PE2, LOC2_E is the SRv6 locator for IP2. LOC1_P is the mirror of LOC1_E in PE2's GRT. LOC2_P is the mirror of LOC2_E in PE1's GRT.

When PE2 receives an EVPN route R0 whose nexthop matches the prefix LOC1, PE2 may discard the route R0 because its nexthop is considered to be PE2's own address. Even though PE2 don't disccard R0, PE2 cannot use its nexthop to send an EVPN data packet to PE1. Because that a destination IP within prefix LOC1 (in forms of LOC1_P) will be considered to be sent to PE2 itself. So we should use IP_N1 and IP_N2 to establish the bypass path between PE1 and PE2 instead of LOC1 and LOC2.

When PE2 receives the EVPN routes from PE3, only the VXLAN tunnel <LOC2_E, IP_N3> will be installed according to [RFC8365]. The VXLAN tunnel <LOC1_P, IP_N3> will not be installed. So when PE1 fails, although the packets to PE1 are fast-rerouted to PE2 by PLR, PE2 may discard these packets because of the absent of the corresponding VXLAN tunnel entity for their SIP and DIP.

The PLR will not be expected to support any Segment Routing extensions at all, it is just asummed to be an ordinary IPv6 router. When PE2 receives an EVPN data packet whose bottommost SID is an EVPN SID from LOC1. Although the EVPN SID can match the prefix LOC1_P on PE2, the EVPN data packet will be dropped because of the absence of SRv6 function indications.

ESI-label is used for MPLS EVPN, SRv6 EVPN and Geneve EVPN, it is typically downstream-assigned in ingress-replication scenarios. When a packet is received from a mirrored End.DT2M SID, the ESI-label in the SID's ARG.FE2 part have to be lookup in a label space that is different from the native ESI-label's label space. Otherwise the packet should be dropped. In multi-homing scenarios, the mirrored End.DT2M SIDs for differnt OPE must do ESI-label lookup in defferent label space, at the same time, the mirrored End.DT2M SIDs for the same OPE should do ESI-label lookup in the same label space. But ESI labels are used only when two PEs from the same egress protection group send BUM packets to each other. In such case, they will use the ESI/EVI label of each egress PE to transport these BUM packets accordingly. So the context-specific ESI/EVI-label lookup is typically not necessary on that egress PE, even if it is in the MPLS EVPN scenarios. But when the egress PE node fails, and the PLR executes egress node protection procedures for that egress PE, that ingress PE may perform context-specific EVI-label lookup consequently at that moment. This will be discussed in again more detailedly.

The above figure is a combination of and 's Figure 6. The TPE1/SPE1/SPE2/TPE2 above is the TPE1/SPE1/SPE2/TPE2 of 's Figure 6, But TPE2 is also the PE1 of , and TPE3 is the PE2, SPE1 is the PE3. When TPE2 advertises an EVPN route (say R9), the same R9 will be advertised to both the two SPEs and TPE3. When TPE3 receives R9, they will do EVPN egress protection. When SPE1 or SPE2 receives the same R9, SPE1/SPE2 will advertise R9 to TPE1 with the same nexthop (the anycast tunnel address of SPE1 and SPE2) following 's section 5.4. Then the requirement here is clear that we want TPE2 use the same route attributes to advertise R9 to both the SPEs and the TPEs. In addition, Note that when the BUM tunnel (T1) from PE1 (TPE2) to PE2 (TPE3) travels through the PLR1, and the PLR1 reroutes these packets (destined to PE2) back to PE1 when PE2 fails, at that moment, PE1 should drop these packets because their EVI label are mirrored EVI labels (in context-specific label space) but their ESI labels are not absent.

This sections describe the extensions specified to meeting the requirements given in and enhance VXLAN EVPN convergency. This document reuse the OPE TLV defined in section 3. The OPE TLV carries the BGP update-source on corresponding PE. The PEs with egress protection procedures described in this document will add the OPE TLV in the EVPN routes that they are about to advertise. Note that the ESI label or leaf Label is not used in VXLAN packet, so the usage for OPE TLV here won't conflict with the usage in .

We will discuss the common procedures for VXLAN EVPN and SRv6 EVPN first. Then we will discuss the procedures specific to VXLAN EVPN or SRv6 EVPN in the following sections. Using the topology in , the common procedures are described in the following list:

PE3 sends a MAC/IP route R1 and an IMET route R2 to PE1 and PE2. The nexthop of these routes is IP_N3 (we assume that IP_N3=IP3). PE3 won't add the OPE TLV to these routes because it works as a normal EVPN PE.
PE1 and PE2 receive R1 and R2 from PE3.
PE1 sends a MAC/IP route R4, an EAD/EVI route R5 and an IMET route R6 to PE2 and PE3. The nexthop of these routes is IP1. PE2 sends a MAC/IP route R7, an EAD/EVI route R8 and an IMET route R9 to PE1 and PE3. The nexthop of these route is IP2. PE1 and PE2 will both add the OPE TLV to these routes because they are configured with protector LOCs. The OPE TLV carries their BGP update-source IP address (IP_N1 or IP_N2).
When PE2 receives R4, R5 and R6 from PE1, it installs the bypass tunnel <IP_N2, IP_N1>. Because that their nexthops is an IP address within the prefix LOC1_P on PE2. The bypass tunnel <IP_N1, IP_N2> is called Egress ESI Link Protection (EELP) bypass tunnel. and PE2 will apply the egress link protection procedures to the received EAD/EVI route R5 following the second approach of section 6. Please see [Section ] for details. Note that the MAC/IP entries from PE1 is installed in GRT by PE2 as mirrored entries. The procedures when PE1 receives R7, R8 and R9 are simlar to the above.
When PE3 receives the EVPN routes from PE1/PE2, it will ignore the OPE TLV because the route's tunnel encapsulation is VXLAN or SRv6 and the nexthop is not a local address on PE3.

First of all, we assume that the VNI for the same EVI on PE1 and PE2 must be the same. The VXLAN-specific procedures are defined in the following list:

In , when PE1 receives the MAC/IP route from PE3, it constructs two VXLAN tunnels: <LOC1_E, IP_N3> and <LOC2_P, IP_N3>. Because it is configured with egress loopback and protector loopback.
In , the bypass tunnel is a VXLAN tunnel.

In VXLAN EVPN, the VXLAN tunnels are constructed for packet-validating purpose. In SRv6 EVPN, there aren't such packet-validating tunnels. So when a SRv6 PE receives EVPN routes from other PEs, no packet-validating tunnels will be installed. But the bypass tunnels aren't constructed for packet-validating purpose, they are used to transport flows among the PEs of the same egress protection group. So the bypass tunnel between PE1 and PE2 must also be installed in SRv6 EVPN scenarios. The SRv6-specific procedures are defined in the following list:

In , The bypass tunnel is a SRv6 tunnel.
In , When PE2 receives R4, R5 from PE1, it mirrors the remote EVPN-SID of these routes in the GRT. This is for the requirements from [Section ]. Note that the End.DT2M SID of IMET route R5 SHOULD not be mirrored by default on PE2. The reason will be described in .
In , PE2 may apply the egress link protection control-plane procedures to the received EAD/EVI route R5 following the first approach of section 6. The corresponding data-plane details will be described in .

First of all, We reserve a portion of the label space for assignment by a central authority. We refer to this reserved portion as the "Domain-wide Common Block" (DCB) of labels. This is analogous to the DCB that is described in 's section 5.4. The DCB is taken from the same label space that is used for downstream-assigned labels, but each PE would know not to allocate local labels from that space. A PE would know by provisioning which label from the DCB corresponds to itself, and each of other labels from the DCB corresponds to each PE of the domain. Note that the PEs don't have to know exactly which label corresponds to a specified PE, They just need know which label is for itself, and other labels is not for itself. The MPLS-specific procedures are defined in the following list:

In , when TPE2(PE1) advertise R4/R5/R6, It following 's section 5.4. This means that a Downstream-CLS ID EC will be advertised along with R4/R5/R6. And this EC carries the label (in DCB) that identifying TPE2(PE1) itself.
In , when TPE3(PE2) receives R4/R5/R6, It install the mirrored ILM entry in a context-specific labels space (say CLS23). The CLS23 is identified by the Downstream-CLS ID EC (say CIL2) of R4/R5/R6. The mirrored ILM entry is called as a CLS-specific ILM entry (CLS-ILM).
In , when SPE1(PE3) receives R4/R5/R6, It should impose the context-identifying label (CIL) carried in R4/R5/R6's Downstream-CLS ID EC onto the label stack following 's section 5.4. That CIL is the outer label of the EVPN label of R4/R5/R6. In addition, SPE1(PE3) will aplly the procedures of 's section 5.4 too. Although these procedures is not of EVPN egress protection schema, they share the same signalling with EVPN protection. This simplifies the signalling procedures, because there no longer will be a requirement to advertise different route attributes to different PEs.

This section describes how Layer 2 unicast and BUM (Broadcast, Unknown unicast, and Multicast) packet forwarding are protected. A description of how Layer 3 packet forwarding are protected will be provided in a furture version of this document.

The following two subsections discuss EENP procedures for BUM forwarding and Unicast Forwarding.

PE3 will do ingress replication to PE1 and PE2 for BUM packets, one copy for each PE. So BUM packets need not to be protected by the egress node protection mechanism. But there will be another issue along with the BUM packets. It is:

PE1 and PE2 will receive a copy of BUM packet from PE3 separately, and the DF node for the <ESI2, EVI> will forward it to the CE node. When the non-DF node of them fails, the BUM packets destined to it will be re-routed to the other one, which will be the DF-node for that <ESI2, EVI>, so these BUM packets will be forwarded to CE2. But PE3 has sent another copy of BUM packets directly to the DF-node, and this copy will be forwarded to CE2 either. So CE2 will receive duplicated BUM packets from the DF-node after the FRR switch of PLR untill the global repair finishes.

In order to avoid the excessive BUM packets, some specific rules are defined in the following list:

For VXLAN EVPN: The BUM packet received via LOC1_P or LOC2_P will be dropped.
For SRv6 EVPN: Because the End.DT2M SIDs are not mirrored according to [Section ], those duplicating BUM packets will be dropped due to the absence of SRv6 function indication.

The bypass-BUM-filter also insures that when PE1 send BUM packet to PE2 but PE2 fails, The FRR-switch on PLR node will not bring out duplicated BUM packets to PE1's local CEs. The tunnel address from PE1 to PE2 for BUM packets is called T1, The tunnel address from PE3 to PE2 for BUM packets is called T2, The bypass-BUM-filter also insures that T1 can be the same as T2. Note that the BUM tunnel T1 may travel through PLR too.

When PE1 node fails, the DF-election between PE1 and PE2 will be restarted, and the PLR will do FRR-switch. We assume that the PLR FRR-switch may be faster than the DF re-election. So if PE1 is the DF node for <ESI2, EVI1> before its failure, the rules that described in [Section ] will cause packet drop before the DF re-election finishes. Because that PE2 will be the non-DF (NDF) node for <ESI2, EVI1> at that time. In order to accelerate the convergence of bypass-BUM packets, some specific rules are defined in the following list:

For VXLAN EVPN: The BUM packet received via LOC1_P or LOC2_P will not be dropped when it is about to forwarded to an AC whose DF-role is NDF.
For SRv6 EVPN: The End.DT2M SIDs need to be mirrored, those BUM packets received via a mirrored End.DT2M SID will not be dropped only when it is about to forwarded to an AC whose DF-role is NDF. Note that a single-homing AC is always considered as DF-role, so it will filter all bypass BUM packets. When BUM packets are received via a mirrored End.DT2M SID, and their Arg.FE2 parts are not empty, such BUM packets will be dropped. Because that such BUM packets just originated inside the same protection group of current PE node.

These rules are called as "NDF-Bias" rules in this draft.

When PE1 fails, the data packets (destined to CE2) from PE3 to PE1 are fast- rerouted to PE2 by the PLR node in the underlay network, the PE2 won't discard these packets because of the existence of VXLAN tunnel<IP3, IP1_P> or mirrored EVPN SID or mirrored CLS-ILM on PE2 itself. The PE2 will forward them to CE. Note that in MPLS scenario (), SPE1(PE3) will impose CIL2 onto the label stack, so the PLR1 wouldn't impose CIL2 (in fact that CIL2 need not be advertised in the underlay network) again. PLR1 just do ordinary anycast FRR or TI-LFA.

The EELP <ESI, EVI> forwarding entry on PE1 will take the ESI link as primary forwarding path, and take the EAD/EVI route from PE2 as backup forwarding path. This procedure follows the second approach of section 6. When the ESI2 link fails, the backup path will be activated on the result of a FRR switch by the overlay network. Note that even when the ESI is All-Active redundancy mode the EELP will follow the FRR behavior. The EELP behavior is the same for All- Active redundancy mode and Single-Active redundancy mode. When ESI is All-Active redundancy mode PE3 will performing overlay ECMP via EAD/EVI routes to PE1/PE2, When the ESI link on PE1 fails, PE1 will forwarding the packets via EELP bypass tunnel before PE3 delete the EAD/EVI routes. But the bypass forwarding is temporary, after PE3 delete the EAD/EVI routes upon the withdraw of the EAD/EVI route from PE1, there won't be any bypass forwarding again. Given that the destination IP address of the EELP bypass tunnel from PE1 to PE2 is called T3, note that T3 may be not the same as T1 or T2 of section . When T3 is a different IP address, different forwarding behaviors can be applied. For example, T3 should not be protected by PLR's egress node protection procedures. When T3 is different from T1/T2, can be used. Otherwise, only the second approach of section 6 can be used if the travel from PE1 to PE2 passes the PLR node. Note that when ESI is Single-Active redundancy mode, there is no importance for PE3 to use the EAD/EVI routes from PE1/PE2. But the EAD/EVI route is still useful between PE1 and PE2 for EELP procedures in Single- Active redundancy mode.

When a PE of an egress protection group receives packets from an EELP bypass tunnel, that PE MUST not send it to another PE in the same egress protection group over any of the bypass tunnels. But when a PE receives a VXLAN/SRv6 encapsulated data packet from an ordinary underlay destination IP address, that PE can bypass that packet following a bypass tunnel.

When the ESI2 link on PE1 fails, the bypassed EVPN packet's underlay destination IP adrress can't be the EVPN SID directly. The EVPN SID have to be hiden in the SRH header in order to avoid the problems described in [Section ], even if the bypass tunnel is a SR-BE tunnel.
Note that there are no egress link protection for BUM packets, all the bypass-BUM packets are the result of egress node protection on the PLR. But when CE1 requests CE3's MAC address, PE1 can't forward the ARP request to PE2 using SRv6 BE encapsualtion. Although these ARP packets are not bypassed packets, the EVPN SID have to be hiden in the SRH header too, in order to avoid the same problems as above.

When is applied, PE1 will use the EVPN SID of itself to encapsulate the bypassed EVPN packet, not use the EVPN SID from the received EAD/EVI route. When that bypassed EVPN packet is received by PE2, the packet will match a mirrored EVPN SID on PE2. So PE2 will know that the packet is a bypassed data packet. The bypassed data packets will be forwarded to local CEs only.
When is applied, the bypass tunnel's destination must be an IP-address for whom the PLR will not do egress node protection. Otherwise micro-loops will arise when the FRR-switch of that egress node protection is triggered on the PLR node.

The first approach of section 6 should be applied, and the bypass tunnel's destination must be an IP-address for whom the PLR1 will not do egress node protection. Otherwise micro-loops will arise when the FRR-switch for that egress node protection is triggered on the PLR node. It means that the sender TPE will impose the EVPN label and CIL of itself onto the label stack, on the failure of local ACs.

IANA Considerations for OPE TLV following .

This section will be added in future versions.

The authors would like to thank the following for their comments and review of this document: Chunning Dai, Bing Song, Zheng Zhou.