idnits 2.17.1 draft-ietf-bess-evpn-inter-subnet-forwarding-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 4 instances of too long lines in the document, the longest one being 3 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: For asymmetric IRB mode, the MPLS label2 field SHOULD not be included in the route; however, if the receiving PE receives this route with the MPLS label2 field, then it SHOULD ignore it. -- The document date (July 2, 2018) is 2119 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC8174' is mentioned on line 112, but not defined == Missing Reference: 'RFC7432' is mentioned on line 714, but not defined == Missing Reference: 'RFC4364' is mentioned on line 293, but not defined == Missing Reference: 'GENEVE' is mentioned on line 171, but not defined == Missing Reference: 'RFC8365' is mentioned on line 710, but not defined == Missing Reference: 'RFC7348' is mentioned on line 200, but not defined == Missing Reference: 'RFC7365' is mentioned on line 203, but not defined == Missing Reference: 'LS' is mentioned on line 1238, but not defined == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-03 == Outdated reference: A later version (-11) exists of draft-ietf-bess-evpn-prefix-advertisement-03 == Outdated reference: A later version (-02) exists of draft-sajassi-l2vpn-evpn-ipvpn-interop-01 == Outdated reference: A later version (-07) exists of draft-raggarwa-data-center-mobility-05 Summary: 1 error (**), 0 flaws (~~), 14 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 L2VPN Workgroup A. Sajassi, Ed. 3 INTERNET-DRAFT S. Salam 4 Intended Status: Standards Track S. Thoria 5 Cisco 6 J. Drake 7 Juniper 8 J. Rabadan 9 Nokia 11 Expires: January 2, 2019 July 2, 2018 13 Integrated Routing and Bridging in EVPN 14 draft-ietf-bess-evpn-inter-subnet-forwarding-04 16 Abstract 18 EVPN provides an extensible and flexible multi-homing VPN solution 19 over an MPLS/IP network for intra-subnet connectivity among Tenant 20 Systems and End Devices that can be physical or virtual. However, 21 there are scenarios for which there is a need for a dynamic and 22 efficient inter-subnet connectivity among these Tenant Systems and 23 End Devices while maintaining the multi-homing capabilities of EVPN. 24 This document describes an Integrated Routing and Bridging (IRB) 25 solution based on EVPN to address such requirements. 27 Status of this Memo 29 This Internet-Draft is submitted to IETF in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF), its areas, and its working groups. Note that 34 other groups may also distribute working documents as 35 Internet-Drafts. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 The list of current Internet-Drafts can be accessed at 43 http://www.ietf.org/1id-abstracts.html 45 The list of Internet-Draft Shadow Directories can be accessed at 46 http://www.ietf.org/shadow.html 48 Copyright and License Notice 50 Copyright (c) 2014 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents 55 (http://trustee.ietf.org/license-info) in effect on the date of 56 publication of this document. Please review these documents 57 carefully, as they describe your rights and restrictions with respect 58 to this document. Code Components extracted from this document must 59 include Simplified BSD License text as described in Section 4.e of 60 the Trust Legal Provisions and are provided without warranty as 61 described in the Simplified BSD License. 63 Table of Contents 65 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 6 66 2 EVPN PE Model for IRB Operation . . . . . . . . . . . . . . . . 7 67 3 Symmetric and Asymmetric IRB . . . . . . . . . . . . . . . . . 8 68 3.1 IRB Interface and its MAC & IP addresses . . . . . . . . . . 11 69 3.2 Symmetric IRB Procedures . . . . . . . . . . . . . . . . . . 12 70 3.2.1 Control Plane - Ingress PE . . . . . . . . . . . . . . . 12 71 3.2.2 Control Plane - Egress PE . . . . . . . . . . . . . . . 13 72 3.2.3 Data Plane - Ingress PE . . . . . . . . . . . . . . . . 14 73 3.2.4 Data Plane - Egress PE . . . . . . . . . . . . . . . . . 14 74 3.3 Asymmetric IRB Procedures . . . . . . . . . . . . . . . . . 15 75 3.3.1 Control Plane - Ingress PE . . . . . . . . . . . . . . . 15 76 3.3.2 Control Plane - Egress PE . . . . . . . . . . . . . . . 15 77 3.3.3 Data Plane - Ingress PE . . . . . . . . . . . . . . . . 16 78 3.3.4 Data Plane - Egress PE . . . . . . . . . . . . . . . . . 17 79 4 BGP Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . 17 80 4.1 Router's MAC Extended Community . . . . . . . . . . . . . . 17 81 5 Operational Models for Symmetric Inter-Subnet Forwarding . . . . 18 82 5.1 IRB forwarding on NVEs for Tenant Systems . . . . . . . . . 18 83 5.1.1 Control Plane Operation . . . . . . . . . . . . . . . . 19 84 5.1.2 Data Plane Operation - Inter Subnet . . . . . . . . . . 21 85 5.1.3 TS Move Operation . . . . . . . . . . . . . . . . . . . 22 86 5.2 IRB forwarding on NVEs for Subnets behind Tenant Systems . . 23 87 5.2.1 Control Plane Operation . . . . . . . . . . . . . . . . 24 88 5.2.2 Data Plane Operation . . . . . . . . . . . . . . . . . . 25 89 6 Inter-Subnet DCI Scenarios . . . . . . . . . . . . . . . . . . 26 90 6.1 Switching among IP subnets in different DCs without GW . . . 27 91 6.2 Switching among IP subnets in different DCs with GW . . . . 29 92 7 TS Mobility . . . . . . . . . . . . . . . . . . . . . . . . . . 31 93 7.1 TS Mobility & Optimum Forwarding for TS Outbound Traffic . . 31 94 7.2 TS Mobility & Optimum Forwarding for TS Inbound Traffic . . 31 95 7.2.1 Mobility without Route Aggregation . . . . . . . . . . . 31 96 8 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 32 97 9 Security Considerations . . . . . . . . . . . . . . . . . . . . 32 98 10 IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 99 11 References . . . . . . . . . . . . . . . . . . . . . . . . . . 32 100 11.1 Normative References . . . . . . . . . . . . . . . . . . . 32 101 11.2 Informative References . . . . . . . . . . . . . . . . . . 32 102 12 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 33 103 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 33 105 Terminology 107 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 108 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 109 "OPTIONAL" in this document are to be interpreted as described in BCP 110 14 [RFC2119] [RFC8174] when, and only when, they appear in all 111 capitals, as shown here. 113 AC: Attachment Circuit. 115 ARP: Address Resolution Protocol. 117 BD: Broadcast Domain. As per [RFC7432], an EVI consists of a single 118 or multiple BDs. In case of VLAN-bundle and VLAN-based service models 119 (see [RFC7432]), a BD is equivalent to an EVI. In case of VLAN-aware 120 bundle service model, an EVI contains multiple BDs. Also, in this 121 document, BD and subnet are equivalent terms. 123 BD Route Target: refers to the Broadcast Domain assigned Route Target 124 [RFC4364]. In case of VLAN-aware bundle service model, all the BD 125 instances in the MAC-VRF share the same Route Target. 127 BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as per 128 [RFC7432]. 130 DGW: Data Center Gateway. 132 Ethernet A-D route: Ethernet Auto-Discovery (A-D) route, as per 133 [RFC7432]. 135 Ethernet NVO tunnel: refers to Network Virtualization Overlay tunnels 136 with Ethernet payload. Examples of this type of tunnels are VXLAN or 137 GENEVE. 139 EVI: EVPN Instance spanning the NVE/PE devices that are participating 140 on that EVPN, as per [RFC7432]. 142 EVPN: Ethernet Virtual Private Networks, as per [RFC7432]. 144 GRE: Generic Routing Encapsulation. 146 GW IP: Gateway IP Address. 148 IPL: IP Prefix Length. 150 IP NVO tunnel: it refers to Network Virtualization Overlay tunnels 151 with IP payload (no MAC header in the payload). 153 IP-VRF: A VPN Routing and Forwarding table for IP routes on an 154 NVE/PE. The IP routes could be populated by EVPN and IP-VPN address 155 families. An IP-VRF is also an instantiation of a layer 3 VPN in an 156 NVE/PE. 158 IRB: Integrated Routing and Bridging interface. It connects an IP-VRF 159 to a BD (or subnet). 161 MAC-VRF: A Virtual Routing and Forwarding table for Media Access 162 Control (MAC) addresses on an NVE/PE, as per [RFC7432]. A MAC-VRF is 163 also an instantiation of an EVI in an NVE/PE. 165 ML: MAC address length. 167 ND: Neighbor Discovery Protocol. 169 NVE: Network Virtualization Edge. 171 GENEVE: Generic Network Virtualization Encapsulation, [GENEVE]. 173 NVO: Network Virtualization Overlays. 175 RT-2: EVPN route type 2, i.e., MAC/IP advertisement route, as defined 176 in [RFC7432]. 178 RT-5: EVPN route type 5, i.e., IP Prefix route. As defined in Section 179 3 of [EVPN-PREFIX]. 181 SBD: Supplementary Broadcast Domain. A BD that does not have any ACs, 182 only IRB interfaces, and it is used to provide connectivity among all 183 the IP-VRFs of the tenant. The SBD is only required in IP-VRF- to-IP- 184 VRF use-cases (see Section 4.4.). 186 SN: Subnet. 188 TS: Tenant System. 190 VA: Virtual Appliance. 192 VNI: Virtual Network Identifier. As in [RFC8365], the term is used as 193 a representation of a 24-bit NVO instance identifier, with the 194 understanding that VNI will refer to a VXLAN Network Identifier in 195 VXLAN, or Virtual Network Identifier in GENEVE, etc. unless it is 196 stated otherwise. 198 VTEP: VXLAN Termination End Point, as in [RFC7348]. 200 VXLAN: Virtual Extensible LAN, as in [RFC7348]. 202 This document also assumes familiarity with the terminology of 203 [RFC7432], [RFC8365] and [RFC7365]. 205 1 Introduction 207 EVPN provides an extensible and flexible multi-homing VPN solution 208 over an MPLS/IP network for intra-subnet connectivity among Tenant 209 Systems (TS's) and End Devices that can be physical or virtual; where 210 an IP subnet is represented by an EVI for a VLAN-based service or by 211 an for a VLAN-aware bundle service. However, there are 212 scenarios for which there is a need for a dynamic and efficient 213 inter-subnet connectivity among these Tenant Systems and End Devices 214 while maintaining the multi-homing capabilities of EVPN. This 215 document describes an Integrated Routing and Bridging (IRB) solution 216 based on EVPN to address such requirements. 218 The inter-subnet communication is traditionally achieved at 219 centralized L3 Gateway (L3GW) nodes where all the inter-subnet 220 communication policies are enforced. When two Tenant Systems (TS's) 221 belonging to two different subnets connected to the same PE node, 222 wanted to communicate with each other, their traffic needed to be 223 back hauled from the PE node all the way to the centralized gateway 224 nodes where inter-subnet switching is performed and then back to the 225 PE node. For today's large multi-tenant data center, this scheme is 226 very inefficient and sometimes impractical. 228 In order to overcome the drawback of centralized L3GW approach, IRB 229 functionality is needed on the PE nodes (also referred to as EVPN 230 NVEs) attached to TS's in order to avoid inefficient forwarding of 231 tenant traffic (i.e., avoid back-hauling and hair-pinning). A PE with 232 IRB capability, can not only locally bridged the tenant intra-subnet 233 traffic but also can locally route the tenant inter-subnet traffic on 234 a packet by packet basis thus meeting the requirements for both intra 235 and inter-subnet forwarding and avoiding non-optimum traffic 236 forwarding associate with centralized L3GW approach. 238 Some TS's run non-IP protocols in conjunction with their IP traffic. 239 Therefore, it is important to handle both kinds of traffic optimally 240 - e.g., to bridge non-IP and intra-subnet traffic and to route inter- 241 subnet IP traffic. Therefore, the solution needs to meet the 242 following requirements: 244 R1: The solution MUST allow for both inter-subnet and intra-subnet 245 traffic belonging to the same tenant to be locally routed and bridged 246 respectively. The solution MUST provide IP routing for inter-subnet 247 traffic and Ethernet Bridging for intra-subnet traffic. 249 R2: The solution MUST support bridging of non-IP traffic. 251 R3: The solution MUST allow inter-subnet switching to be disabled on 252 a per VLAN basis on PEs where the traffic needs to be back hauled to 253 another node (i.e., for performing FW or DPI functionality). 255 2 EVPN PE Model for IRB Operation 257 Since this document discusses IRB operation in relationship to EVPN 258 MAC-VRF, IP-VRF, EVI, Bridge Domain (BD), Bridge Table (BT), and IRB 259 interfaces, it is important to understand the relationship among 260 these components. Therefore, the following PE model is demonstrated 261 below to a) describe these components and b) illustrate the 262 relationship among them. 264 +-------------------------------------------------------------+ 265 | | 266 | +------------------+ IRB PE | 267 | Attachment | +------------------+ | 268 | Circuit(AC1) | | +----------+ | MPLS/NVO tnl 269 ----------------------*Bridge | | +----- 270 | | | |Table(BT1)| | +-----------+ / \ \ 271 | | | | *---------* |<--> |Eth| 272 | | | |Eth-Tag x | |IRB1| | \ / / 273 | | | +----------+ | | | +----- 274 | | | ... | | IP-VRF1 | | 275 | | | +----------+ | | RD2/RT2 |MPLS/NVO tnl 276 | | | |Bridge | | | | +----- 277 | | | |Table(BT2)| |IRB2| | / \ \ 278 | | | | *---------* |<--> |IP | 279 ----------------------*Eth-Tag y | | +-----------+ \ / / 280 | AC2 | | +----------+ | +----- 281 | | | MAC-VRF1 | | 282 | +-+ RD1/RT1 | | 283 | +------------------+ | 284 | | 285 | | 286 +-------------------------------------------------------------+ 288 Figure 1: EVPN IRB PE Model 290 A tenant needing IRB services on a PE, requires an IP Virtual Routing 291 and Forwarding table (IP-V RF) along with one or more MAC Virtual 292 Routing and Forwarding tables (MAC-VRFs). An IP-VRF, as defined in 293 [RFC4364], is the instantiation of an IPVPN in a PE. A MAC-VRF, as 294 defined in [RFC7432], is the instantiation of an EVI (EVPN Instancce) 295 in a PE. A MAC-VRF can consists of one or more Bridge Tables (BTs) 296 where each BT corresponds to a VLAN (broadcast domain - BD). If the 297 service interface for the EVPN PE is configured in VLAN-Based mode 298 (i.e., section 6.1 of [RFC7432]), then there is only a single BT per 299 MAC-VRF (per EVI) - i.e., there is only one tenant VLAN per EVI. 300 However, if the service interface for the EVPN PE is configured in 301 VLAN-Aware Bundle mode (i.e., section 6.3 of [RFC7432]), then there 302 are several BTs per MAC-VRF (per EVI) - i.e., there are several 303 tenant VLANs per EVI. 305 Each BT is connected to a IP-VRF via a L3 interface called IRB 306 interface. Since a single tenant subnet is typically (and in this 307 document) represented by a VLAN (and thus supported by a single BT), 308 for a given tenant there are as many BTs as there are subnets and 309 thus there are also as many IRB interfaces between the tenant IP-VRF 310 and the associated BTs as shown in the PE model above. 312 IP-VRF is identified by its corresponding route target and route 313 distinguisher and MAC-VRF is also identified by its corresponding 314 route target and route distinguisher. If operating in EVPN VLAN-Based 315 mode, then a receiving PE that receives an EVPN route with MAC-VRF 316 route target can identify the corresponding BT; however, if operating 317 in EVPN VLAN-Aware Bundle mode, then the receiving PE needs both the 318 MAC-VRF route target and VLAN ID in order to identify the 319 corresponding BT. 321 3 Symmetric and Asymmetric IRB 323 This document defines and describes two types of IRB solutions - 324 namely symmetric and asymmetric IRB. In symmetric IRB as its name 325 implies, the lookup operation is symmetric at both ingress and egress 326 PEs - i.e., both ingress and egress PEs perform lookups on both TS's 327 MAC and IP addresses - i.e., ingress PE performs lookup on 328 destination TS's MAC address followed by its IP address and egress PE 329 performs lookup on destination TS's IP address followed by its MAC 330 address as depicted in figure 2. 332 Ingress PE Egress PE 333 +-------------------+ +------------------+ 334 | | | | 335 | +-> IP-VFF ----|---->---|-----> IP-VRF -+ | 336 | | | | | | 337 | BT1 BT2 | | BT3 BT2 | 338 | | | | | | 339 | ^ | | v | 340 | | | | | | 341 +-------------------+ +------------------+ 342 ^ | 343 | | 344 TS1->-+ +->-TS2 345 Figure 2: Symmetric IRB 347 In symmetric IRB as shown in figure-2, the inter-subnet forwarding 348 between two PEs is done between their associated IP-VRFs. Therefore, 349 the tunnel connecting these IP-VRFs can be either IP-only tunnel (in 350 case of MPLS or GENEVE encapsulation) or Ethernet NVO tunnel (in case 351 of VxLAN encapsulation). If it is Ethernet NOV tunnel, the TS's IP 352 packet is encapsulated in an Ethernet header consisting of ingress 353 and egress PEs MAC addresses - i.e., there is no need for ingress PE 354 to use the destination TS's MAC address. Therefore, in symmetric IRB, 355 there is no need for the ingress PE to hold destination TS's IP and 356 MAC association in its ARP table. Each PE participating in symmetric 357 IRB only maintains ARP entries for locally connected hosts and 358 maintain MAC-VRFs/BTs for only locally configured subnets. 360 In asymmetric IRB, the lookup operation is asymmetric and the ingress 361 PE performs three lookups; whereas the egress PE performs a single 362 lookup - i.e., the ingress PE performs lookups on destination TS's 363 MAC address, followed by its IP address, followed by its MAC address 364 again; whereas, the egress PE performs just a single lookup on 365 destination TS's MAC address as depicted in figure 3 below. 367 Ingress PE Egress PE 368 +-------------------+ +------------------+ 369 | | | | 370 | +-> IP-VFF -> | | IP-VRF | 371 | | | | | | 372 | BT1 BT2 | | BT3 BT2 | 373 | | | | | | | | 374 | | +--|--->----|--------------+ | | 375 | | | | v | 376 +-------------------+ +----------------|-+ 377 ^ | 378 | | 379 TS1->-+ +->-TS2 380 Figure 3: Asymmetric IRB 382 In asymmetric IRB as shown in figure-2, the inter-subnet forwarding 383 between two PEs is done between their associated MAC-VRFs/BTs. 384 Therefore, the MPLS or NVO tunnel used for inter-subnet forwarding 385 MUST be of type Ethernet. Since at the egress PE only MAC lookup is 386 performed (e.g., no IP lookup), the TS's IP packet needs to be 387 encapsulated with the destination TS's MAC address. In order for 388 ingress PE to perform such encapsulation, it needs to maintain TS's 389 IP and MAC address association in its ARP table. Furthermore, it 390 needs to maintain destination TS's MAC address in the corresponding 391 BT even though it does not have the corresponding subnet locally 392 configured. In other words, each PE participating in asymmetric IRB 393 MUST maintain ARP entries for remote hosts (hosts connected to other 394 PEs) as well as maintaining MAC-VRFs/BTs for subnets that are not 395 locally present on that PE. 397 The following subsection defines the control and data planes 398 procedures for symmetric and asymmetric IRB on ingress and egress 399 PEs. The following figure is used in description of these procedures 400 where it shows a single IP-VRF and a number of BTs on each PE for a 401 given tenant. The IP-VRF of the tenant (i.e., IP-VRF1) is connected 402 to each BT via its associated IRB interface. Each BT on a PE is 403 associated with a unique VLAN (e.g., with a BD) where in turn is 404 associated with a single MAC-VRF in case of VLAN-Based mode or a 405 number of BTs can be associated with a single MAC-VRF in case of 406 VLAN-Aware Bundle mode. Whether the service interface on a PE is 407 VLAN-Based or VLAN-Aware Bundle mode does not impact the IRB 408 operation and procedures. It only impacts the setting of Ethernet tag 409 field in EVPN routes as described in [RFC7432]. 411 PE 1 +---------+ 412 +-------------+ | | 413 TS1-----| MACx| | | PE2 414 (IP1/M1) |(BT1) | | | +-------------+ 415 TS5-----| \ | | MPLS/ | |MACy (BT3) |-----TS3 416 (IP5/M5) |Mx/IPx \ | | VxLAN/ | | / | (IP3/M3) 417 | (IP-VRF1)|----| NVGRE |---|(IP-VRF1) | 418 | / | | | | \ | 419 TS2-----|(BT2) / | | | | (BT1) |-----TS4 420 (IP2/M2) | | | | | | (IP4/M4) 421 +-------------+ | | +-------------+ 422 | | 423 +---------+ 425 Figure 4: IRB forwarding 427 3.1 IRB Interface and its MAC & IP addresses 429 To support inter-subnet forwarding on a PE, the PE acts as an IP 430 Default Gateway from the perspective of the attached Tenant Systems 431 where default gateway MAC and IP addresses are configured on each IRB 432 interface associated with its subnet and falls into one of the 433 following two options: 435 1. All the PEs for a given tenant subnet use the same anycast default 436 gateway IP and MAC addresses . On each PE, this default gateway IP 437 and MAC addresses correspond to the IRB interface connecting the BT 438 associated with the tenant's to the corresponding 439 tenant's IP-VRF. 441 2. Each PE for a given tenant subnet uses the same anycast default 442 gateway IP address but its own MAC address. These MAC addresses are 443 aliased to the same anycast default gateway IP address through the 444 use of the Default Gateway extended community as specified in [EVPN], 445 which is carried in the EVPN MAC/IP Advertisement routes. On each PE, 446 this default gateway IP address along with its associated MAC 447 addresses correspond to the IRB interface connecting the BT 448 associated with the tenant's to the corresponding 449 tenant's IP-VRF. 451 It is worth noting that if the applications that are running on the 452 TS's are employing or relying on any form of MAC security, then 453 either the first model (i.e. using anycast MAC address) should be 454 used to ensure that the applications receive traffic from the same 455 IRB interface MAC address that they are sending to, or if the second 456 model is used, then the IRB interface MAC address MUST be the one 457 used in the initial ARP reply for that TS. 459 Although both of these options are equally applicable to both 460 symmetric and asymmetric IRB, the option-1 is recommended because of 461 the ease of anycast MAC address provisioning on not only the IRB 462 interface associated with a given subnet across all the PEs 463 corresponding to that EVI but also on all IRB interfaces associated 464 with all the tenant's subnets across all the PEs corresponding to all 465 the EVIs for that tenant. Furthermore, it simplifies the operation as 466 there is no need for Default Gateway extended community advertisement 467 and its associated MAC aliasing procedure. 469 When a TS sends an ARP request to the PE that is attached to, the ARP 470 request is sent for the IP address of the IRB interface associated 471 with the TS's subnet. For example, in figure 4, TS1 is configured 472 with the anycast IPx address as its default gateway IP address and 473 thus when it sends an ARP request for IPx (IP address of the IRB 474 interface for BT1), the PE1 sends an ARP reply with the MACx which is 475 the MAC address of that IRB interface. 477 In addition to anycast addresses, IRB interfaces can be configured 478 with non-anycast IP addresses for the purpose of OAM (such as 479 traceroute/ping to these interfaces) for both symmetric and 480 asymmetric IRB. These IP addresses need to be distributed as VPN 481 routes when PEs operating in symmetric IRB mode. However, they don't 482 need to be distributed if the PEs are operating in asymmetric IRB 483 mode and the IRB interfaces are configured with individual MACs. 485 3.2 Symmetric IRB Procedures 487 3.2.1 Control Plane - Ingress PE 489 When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address of 490 a TS (via an ARP request), it adds the MAC address to the 491 corresponding MAC-VRF/BT of that tenant's subnet and adds the IP 492 address to the IP-VRF for that tenant. Furthermore, it adds this TS's 493 MAC and IP address association to its ARP table. It then builds an 494 EVPN MAC/IP Advertisement route (type 2) as follow and advertises it 495 to other PEs participating in that tenant's VPN. 497 - The Length field of the BGP EVPN NLRI for an EVPN MAC/IP 498 Advertisement route MUST be either 40 (if IPv4 address is carried) or 499 52 (if IPv6 address is carried). 501 - Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet Tag 502 ID, MAC Address Length, MAC Address, IP Address Length, IP Address, 503 and MPLS Label1 fields MUST be used as defined in [RFC7432] and 504 [RFC8365]. 506 - The MPLS Label2 field is set to either an MPLS label or a VNI 507 corresponding to the tenant's IP-VRF. In case of an MPLS label, this 508 field is encoded as 3 octets, where the high-order 20 bits contain 509 the label value. 511 Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, 512 MAC Address, IP Address Length, and IP Address fields are part of 513 the route key used by BGP to compare routes. The rest of the fields 514 are not part of the route key. 516 This route is advertised along with the following two extended 517 communities: 518 1) Tunnel Type Extended Community 519 2) Router's MAC Extended Community 521 For symmetric IRB mode, Router's MAC EC is needed to carry the PE's 522 overlay MAC address which is used for IP-VRF to IP-VRF communications 523 with Ethernet NVO tunnel. If MPLS or IP-only NVO tunnel is used, then 524 there is no need to send Router's MAC Extended Community along with 525 this route. 527 This route MUST be advertised with two route targets - one 528 corresponding to the MAC-VRF of the tenant's subnet and another 529 corresponding to the tenant's IP-VRF. 531 3.2.2 Control Plane - Egress PE 533 When a PE (e.g., PE2 in figure 4 above) receives this EVPN MAC/IP 534 Advertisement route advertisement, it performs the following: 536 - Using MAC-VRF route target, it identifies the corresponding MAC- 537 VRF. If the MAC-VRF exists (e.g., it is locally configured) then it 538 imports the MAC address into it. Otherwise, it does not import the 539 MAC address. 541 - Using IP-VRF route target, it identifies the corresponding IP-VRF 542 and imports the IP address into it. 544 The inclusion of MPLS label2 field in this route signals to the 545 receiving PE that this route is for symmetric IRB mode and MPLS 546 label2 needs to be installed in forwarding path to identify the 547 corresponding IP-VRF. 549 If the receiving PE receives this route with both the MAC-VRF and IP- 550 VRF route targets but the MAC/IP Advertisement route does not include 551 MPLS label2 field and if the receiving PE does not support asymmetric 552 IRB mode, then if it has the corresponding MAC-VRF, it only imports 553 the MAC address; otherwise, if it doesn't have the corresponding MAC- 554 VRF, it MUST treat the route as withdraw [RFC7606]. 556 3.2.3 Data Plane - Ingress PE 558 When an Ethernet frame is received by an ingress PE (e.g., PE1 in 559 figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify 560 the associated MAC-VRF/BT and it performs a lookup on the destination 561 MAC address. If the MAC address corresponds to its IRB Interface MAC 562 address, the ingress PE deduces that the packet must be inter-subnet 563 routed. Hence, the ingress PE performs an IP lookup in the associated 564 IP-VRF table. The lookup identifies BGP next hop of egress PE along 565 with the tunnel/encapsulation type and the associated MPLS/VNI 566 values. 568 If the tunnel type is that of MPLS or IP-only NVO tunnel, then TS's 569 IP packet is sent over the tunnel without any Ethernet header. 570 However, if the tunnel type is that of Eternet NVO tunnel, then an 571 Ethernet header needs to be added to the TS's IP packet. The source 572 MAC address of this Ethernet header is set to the ingress PE's router 573 MAC address and the destination MAC address of this Ethernet header 574 is set to the egress PE's router MAC address. The MPLS VPN label or 575 VNI fields are set accordingly and the packet is forwarded to the 576 egress PE. 578 If case of NVO tunnel encapsulation, the outer source IP address is 579 set to the ingress PE's BGP next-hop address and outer destination IP 580 address is set to the egress PE's BGP next-hop address. 582 3.2.4 Data Plane - Egress PE 584 When the tenant's MPLS or NVO encapsulated packet is received over an 585 MPLS or NVO tunnel by the egress PE, the egress PE removes NVO tunnel 586 encapsulation and uses the VPN MPLS label (for MPLS encapsulation) or 587 VNI (for VxLAN encapsulation) to identify the IP-VRF in which IP 588 lookup needs to be performed. 590 The lookup identifies a local adjacency to the IRB interface 591 associated with the egress subnet's MAC-VRF/BT. 593 The egress PE gets the destination TS's MAC address for that TS's IP 594 address from its ARP table, it encapsulates the packet with that 595 destination MAC address and a source MAC address corresponding to 596 that IRB interface and sends the packet to its destination subnet 597 MAC-VRF/BT. 599 The destination MAC address lookup in the MAC-VRF/BT results in local 600 adjacency (e.g., local interface) over which the Ethernet frame is 601 sent on. 603 3.3 Asymmetric IRB Procedures 605 3.3.1 Control Plane - Ingress PE 607 When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address of 608 a TS (via an ARP request), it populates its MAC-VRF/BT, IP-VRF, and 609 ARP table just as in the case for symmetric IRB. It then builds an 610 EVPN MAC/IP Advertisement route (type 2) as follow and advertises it 611 to other PEs participating in that tenant's VPN. 613 - The Length field of the BGP EVPN NLRI for an EVPN MAC/IP 614 Advertisement route MUST be either 37 (if IPv4 address is carried) or 615 49 (if IPv6 address is carried). 617 - Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet Tag 618 ID, MAC Address Length, MAC Address, IP Address Length, IP Address, 619 and MPLS Label1 fields MUST be used as defined in [RFC7432] and 620 [RFC8365]. 622 - The MPLS Label2 field MUST NOT be included in this route. 624 Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, 625 MAC Address, IP Address Length, and IP Address fields are part of 626 the route key used by BGP to compare routes. The rest of the fields 627 are not part of the route key. 629 This route is advertised along with the following extended 630 communitiy: 631 1) Tunnel Type Extended Community 633 For asymmetric IRB mode, Router's MAC EC is not needed because 634 forwarding is performed using destination TS's MAC address which is 635 carried in this route advertisement. 637 This route MUST always be advertised with MAC-VRF route target. It 638 MAY also be advertised with a second route target corresponding to 639 the IP-VRF. If only MAC-VRF route target is used, then the receiving 640 PE uses the MAC-VRF route target to identify the corresponding IP-VRF 641 - i.e., many MAC-VRF route targets map to the same IP-VRF for a given 642 tenant. 644 3.3.2 Control Plane - Egress PE 645 When a PE (e.g., PE2 in figure 4 above) receives this EVPN MAC/IP 646 Advertisement route advertisement, it performs the following: 648 - Using MAC-VRF route target, it identifies the corresponding MAC-VRF 649 and imports the MAC address into it. For asymmetric IRB mode, it is 650 assumed that all PEs participating in a tenant's VPN are configured 651 with all subnets and corresponding MAC-VRFs/BTs even if there are no 652 locally attached TS's for some of these subnets. The reason for this 653 is because ingress PE needs to do forwarding based on destination 654 TS's MAC address and does proper NVO tunnel encapsulation which are 655 property of a lookup in MAC-VRF/BT. An implementation may choose to 656 consolidate the lookup at the ingress PE's IP-VRF with the lookup at 657 the ingress PE's destination subnet MAC-VRF. Consideration for such 658 consolidation of lookups is outside the scope of this document. 660 - Using MAC-VRF route target, it identifies the corresponding ARP 661 table for the tenant and it adds an entry to the ARP table for the 662 TS's MAC and IP address association. It should be noted that the 663 tenant's ARP table at the receiving PE is identified by all the MAC- 664 VRF route targets for that tenant. If IP-VRF route target is included 665 with this route advertisement, then it MAY be used for the 666 identification of tenant's ARP table. 668 For asymmetric IRB mode, the MPLS label2 field SHOULD not be included 669 in the route; however, if the receiving PE receives this route with 670 the MPLS label2 field, then it SHOULD ignore it. 672 3.3.3 Data Plane - Ingress PE 674 When an Ethernet frame is received by an ingress PE (e.g., PE1 in 675 figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify 676 the associated MAC-VRF/BT and it performs a lookup on the destination 677 MAC address. If the MAC address corresponds to its IRB Interface MAC 678 address, the ingress PE deduces that the packet must be inter-subnet 679 routed. Hence, the ingress PE performs an IP lookup in the associated 680 IP-VRF table. The lookup identifies a local adjacency to the IRB 681 interface associated with the egress subnet's MAC-VRF/BT. 683 The ingress PE gets the destination TS's MAC address for that TS's IP 684 address from its ARP table, it encapsulates the packet with that 685 destination MAC address and a source MAC address corresponding to 686 that IRB interface and sends the packet to its destination subnet 687 MAC-VRF/BT. 689 The destination MAC address lookup in the MAC-VRF/BT results in BGP 690 next hop address of egress PE. The ingress PE encapsulates the packet 691 using Ethernet NVO tunnel of the choice (e.g., VxLAN or GENEVE) and 692 sends the packet to the egress PE. Since the packet forwarding is 693 between ingress PE's MAC-VRF/BT and egress PE's MAC-VRF/BT, the 694 packet encapsulation procedures follows that of [RFC7432] for MPLS 695 and [RFC8365] for VxLAN encapsulations. 697 3.3.4 Data Plane - Egress PE 699 When a tenant's Ethernet frame is received over an NVO tunnel by the 700 egress PE, the egress PE removes NVO tunnel encapsulation and uses 701 the VPN MPLS label (for MPLS encapsulation) or VNI (for VxLAN 702 encapsulation) to identify the MAC-VRF/BT in which MAC lookup needs 703 to be performed. 705 The MAC lookup results in local adjacency (e.g., local interface) 706 over which the packet needs to get sent. 708 Note that the forwarding behavior on the egress PE is the same as 709 EVPN intra-subnet forwarding described in [RFC7432] for MPLS and 710 [RFC8365] for VxLAN networks. In other words, all the packet 711 processing associated with the inter-subnet forwarding semantics is 712 confined to the ingress PE. 714 It should also be noted that [RFC7432] provides different level of 715 granularity for the EVPN label. Besides identifying bridge domain 716 table, it can be used to identify the egress interface or a 717 destination MAC address on that interface. If EVPN label is used for 718 egress interface or individual MAC address identification, then no 719 MAC lookup is needed in the egress PE for MPLS encapsulation and the 720 packet can be directly forwarded to the egress interface just based 721 on EVPN label lookup. 723 4 BGP Encoding 725 This document defines one new BGP Extended Community for EVPN. 727 4.1 Router's MAC Extended Community 729 A new EVPN BGP Extended Community called Router's MAC is introduced 730 here. This new extended community is a transitive extended community 731 with the Type field of 0x06 (EVPN) and the Sub-Type of 0x03. It may 732 be advertised along with BGP Encapsulation Extended Community define 733 in section 4.5 of [TUNNEL-ENCAP]. 735 The Router's MAC Extended Community is encoded as an 8-octet value as 736 follows: 738 0 1 2 3 739 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 740 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 741 | Type=0x06 | Sub-Type=0x03 | Router's MAC | 742 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 743 | Router's MAC Cont'd | 744 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 746 Figure 5: Router's MAC Extended Community 748 This extended community is used to carry the PE's MAC address for 749 symmetric IRB scenarios and it is sent with RT-2. 751 5 Operational Models for Symmetric Inter-Subnet Forwarding 753 The following sections describe two main symmetric IRB forwarding 754 scenarios (within a DC - i.e., intra-DC) along with their 755 corresponding procedures. In the following scenarios, without loss of 756 generality, it is assumed that a given tenant is represented by a 757 single IP-VPN instance. Therefore, on a given PE, a tenant is 758 represented by a single IP-VRF table and one or more MAC-VRF tables. 760 5.1 IRB forwarding on NVEs for Tenant Systems 762 This section covers the symmetric IRB procedures for the scenario 763 where each Tenant System (TS) is attached to one or more NVEs and its 764 host IP and MAC addresses are learned by the attached NVEs and are 765 distributed to all other NVEs that are interested in participating in 766 both intra-subnet and inter-subnet communications with that TS. 768 In this scenario, for a given tenant, an NVE has typically one MAC- 769 VRF for each tenant's subnet (VLAN) that is configured for, assuming 770 VLAN-based service which is typically the case for VxLAN and NVGRE 771 encapsulation and each MAC-VRF consists of a single bridge domain. In 772 case of MPLS encapsulation with VLAN-aware bundling, then each MAC- 773 VRF consists of multiple bridge domains (one bridge domain per VLAN). 774 The MAC-VRFs on an NVE for a given tenant are associated with an IP- 775 VRF corresponding to that tenant (or IP-VPN instance) via their IRB 776 interfaces. 778 Each NVE MUST support QoS, Security, and OAM policies per IP-VRF 779 to/from the core network. This is not to be confused with the QoS, 780 Security, and OAM policies per Attachment Circuits (AC) to/from the 781 Tenant Systems. How this requirement is met is an implementation 782 choice and it is outside the scope of this document. 784 Since VxLAN and NVGRE encapsulations require inner Ethernet header 785 (inner MAC SA/DA), and since for inter-subnet traffic, TS MAC address 786 cannot be used, the ingress NVE's MAC address is used as inner MAC 787 SA. The NVE's MAC address is the device MAC address and it is common 788 across all MAC-VRFs and IP-VRFs. This MAC address is advertised using 789 the new EVPN Router's MAC Extended Community (section 6.1). 791 Figure 6 below illustrates this scenario where a given tenant (e.g., 792 an IP-VPN instance) has three subnets represented by MAC-VRF1, MAC- 793 VRF2, and MAC-VRF3 across two NVEs. There are five TS's that are 794 associated with these three MAC-VRFs - i.e., TS1, TS4, and TS5 are 795 sitting on the same subnet (e.g., same MAC-VRF/VLAN);where, TS1 and 796 TS5 are associated with MAC-VRF1 on NVE1, TS4 is associated with MAC- 797 VRF1 on NVE2. TS2 is associated with MAC-VRF2 on NVE1, and TS3 is 798 associated with MAC-VRF3 on NVE2. MAC-VRF1 and MAC-VRF2 on NVE1 are 799 in turn associated with IP-VRF1 on NVE1 and MAC-VRF1 and MAC-VRF3 on 800 NVE2 are associated with IP-VRF1 on NVE2. When TS1, TS5, and TS4 801 exchange traffic with each other, only L2 forwarding (bridging) part 802 of the IRB solution is exercised because all these TS's sit on the 803 same subnet. However, when TS1 wants to exchange traffic with TS2 or 804 TS3 which belong to different subnets, then both bridging and routing 805 parts of the IRB solution are exercised. The following subsections 806 describe the control and data planes operations for this IRB scenario 807 in details. 809 NVE1 +---------+ 810 +-------------+ | | 811 TS1-----| MACx| | | NVE2 812 (IP1/M1) |(MAC- | | | +-------------+ 813 TS5-----| VRF1)\ | | MPLS/ | |MACy (MAC- |-----TS3 814 (IP5/M5) | \ | | VxLAN/ | | / VRF3) | (IP3/M3) 815 | (IP-VRF1)|----| NVGRE |---|(IP-VRF1) | 816 | / | | | | \ | 817 TS2-----|(MAC- / | | | | (MAC- |-----TS4 818 (IP2/M2) | VRF2) | | | | VRF1) | (IP4/M4) 819 +-------------+ | | +-------------+ 820 | | 821 +---------+ 823 Figure 6: IRB forwarding on NVEs for Tenant Systems 825 5.1.1 Control Plane Operation 827 Each NVE advertises a MAC/IP Advertisement route (i.e., Route Type 2) 828 for each of its TS's with the following field set: 830 - RD and ESI per [EVPN] 831 - Ethernet Tag = 0; assuming VLAN-based service 832 - MAC Address Length = 48 833 - MAC Address = Mi ; where i = 1,2,3,4, or 5 in the above example 834 - IP Address Length = 32 or 128 835 - IP Address = IPi ; where i = 1,2,3,4, or 5 in the above example 836 - Label-1 = MPLS Label or VNID corresponding to MAC-VRF 837 - Label-2 = MPLS Label or VNID corresponding to IP-VRF 839 Each NVE advertises an RT-2 route with two Route Targets (one 840 corresponding to its MAC-VRF and the other corresponding to its IP- 841 VRF. Furthermore, the RT-2 is advertised with two BGP Extended 842 Communities. The first BGP Extended Community identifies the tunnel 843 type per section 4.5 of [TUNNEL-ENCAP] and the second BGP Extended 844 Community includes the MAC address of the NVE (e.g., MACx for NVE1 or 845 MACy for NVE2) as defined in section 6.1. This second Extended 846 Community (for the MAC address of NVE) is only required when Ethernet 847 NVO tunnel type is used. If IP NVO tunnel type is used, then there is 848 no need to send this second Extended Community. It should be noted 849 that IP NVO tunnel type is only applicable to symmetric IRB 850 procedures. 852 Upon receiving this advertisement, the receiving NVE performs the 853 following: 855 - It uses Route Targets corresponding to its MAC-VRF and IP-VRF for 856 identifying these tables and subsequently importing this route into 857 them. 859 - It imports the MAC address from MAC/IP Advertisement route into the 860 MAC-VRF with BGP Next Hop address as underlay tunnel destination 861 address (e.g., VTEP DA for VxLAN encapsulation) and Label-1 as VNID 862 for VxLAN encapsulation or EVPN label for MPLS encapsulation. 864 - If the route carries the new Router's MAC Extended Community, and 865 if the receiving NVE is using Ethernet NVO tunnel, then the receiving 866 NVE imports the IP address into IP-VRF with NVE's MAC address (from 867 the new Router's MAC Extended Community) as inner MAC DA and BGP Next 868 Hop address as underlay tunnel destination address, VTEP DA for VxLAN 869 encapsulation and Label-2 as IP-VPN VNID for VxLAN encapsulation. 871 - If the receiving NVE is going to use MPLS encapsulation, then the 872 receiving NVE imports the IP address into IP-VRF with BGP Next Hop 873 address as underlay tunnel destination address, and Label-2 as IP-VPN 874 label for MPLS encapsulation. 876 If the receiving NVE receives a RT-2 with only a single Route Target 877 corresponding to IP-VRF and Label-1, or if it receives a RT-2 with 878 only a single Route Target corresponding to MAC-VRF but with both 879 Label-1 and Label-2, or if it receives a RT-2 with MAC Address Length 880 of zero, then it must not import it to either IP-VRF or MAC-VRF and 881 it must log an error. 883 5.1.2 Data Plane Operation - Inter Subnet 885 The following description of the data-plane operation describes just 886 the logical functions and the actual implementation may differ. Lets 887 consider data-plane operation when TS1 in subnet-1 (MAC-VRF1) on NVE1 888 wants to send traffic to TS3 in subnet-3 (MAC-VRF3) on NVE2. 890 - NVE1 receives a packet with MAC DA corresponding to the MAC-VRF1 891 IRB interface on NVE1 (the interface between MAC-VRF1 and IP-VRF1), 892 and VLAN-tag corresponding to MAC-VRF1. 894 - Upon receiving the packet, the NVE1 uses VLAN-tag to identify the 895 MAC-VRF1. It then looks up the MAC DA and forwards the frame to its 896 IRB interface. 898 - The Ethernet header of the packet is stripped and the packet is 899 fed to the IP-VRF where IP lookup is performed on the destination 900 address. This lookup yields an outgoing interface and the required 901 encapsulation. If the encapsulation is for Ethernet NVO tunnel, then 902 it includes a MAC address to be used as inner MAC DA, an IP address 903 to be used as VTEP DA, and a VPN-ID to be used as VNID. The inner MAC 904 SA and VTEP SA is set to NVE's MAC and IP addresses respectively. If 905 it is a MPLS encapsulation, then corresponding EVPN and LSP labels 906 are added to the packet. The packet is then forwarded to the egress 907 NVE. 909 - On the egress NVE, if the packet arrives on Ethernet NVO tunnel 910 (e.g., it is VxLAN encapsulated), then the VxLAN header is removed. 911 Since the inner MAC DA is the egress NVE's MAC address, the egress 912 NVE knows that it needs to perform an IP lookup. It uses VNID to 913 identify the IP-VRF table. If the packet is MPLS encapsulated, then 914 the EVPN label lookup identifies the IP-VRF table. Next, an IP lookup 915 is performed for the destination TS (TS3) which results in access- 916 facing IRB interface over which the packet is sent. Before sending 917 the packet over this interface, the ARP table is consulted to get the 918 destination TS's MAC address. 920 - The IP packet is encapsulated with an Ethernet header with MAC SA 921 set to that of IRB interface MAC address and MAC DA set to that of 922 destination TS (TS3) MAC address. The packet is sent to the 923 corresponding MAC-VRF3 and after a lookup of MAC DA, is forwarded to 924 the destination TS (TS3) over the corresponding interface. 926 In this symmetric IRB scenario, inter-subnet traffic between NVEs 927 will always use the IP-VRF VNID/MPLS label. For instance, traffic 928 from TS2 to TS4 will be encapsulated by NVE1 using NVE2's IP-VRF 929 VNID/MPLS label, as long as TS4's host IP is present in NVE1's IP- 930 VRF. 932 5.1.3 TS Move Operation 934 When a TS move from one NVE to other, it is important that the MAC 935 mobility procedures are properly executed and the corresponding MAC- 936 VRF and IP-VRF tables on all participating NVEs are updated. [EVPN] 937 describes the MAC mobility procedures for L2-only services for both 938 single-homed TS and multi-homed TS. This section describes the 939 incremental procedures and BGP Extended Communities needed to handle 940 the MAC mobility for a mixed of L2 and L3 connectivity (aka IRB). In 941 order to place the emphasis on the differences between L2-only versus 942 L2-and-L3 use cases, the incremental procedure is described for 943 single-homed TS with the expectation that the reader can easily 944 extrapolate multi-homed TS based on the procedures described in 945 section 15 of [EVPN]. 947 Lets consider TS1 in figure-6 above where it moves from NVE1 to NVE2. 948 In such move, NVE2 discovers IP1/MAC1 of TS1 and realizes that it is 949 a MAC move and it advertises a MAC/IP route per section 5.1.1 above 950 with MAC Mobility Extended Community. 952 Since NVE2 learns TS1's MAC/IP addresses locally, it updates its MAC- 953 VRF1 and IP-VRF1 for TS1 with its local interface. 955 If the local learning at NVE1 is performed using control or 956 management planes, then these interactions serve as the trigger for 957 NVE1 to withdraw the MAC and IP addresses associated with TS1. 958 However, if the local learning at NVE1 is performed using data-plane 959 learning, then the reception of the MAC/IP Advertisement route (for 960 TS1) from NVE2 with MAC Mobility extended community serve as the 961 trigger for NVE1 to withdraw the MAC and IP addresses associated with 962 TS1. 964 All other remote NVE devices upon receiving the MAC/IP advertisement 965 route for TS1 from NVE2 with MAC Mobility extended community compare 966 the sequence number in this advertisement with the one previously 967 received. If the new sequence number is greater than the old one, 968 then they update the MAC/IP addresses of TS1 in their corresponding 969 MAC-VRFs and IP-VRFs to point to NVE2. Furthermore, upon receiving 970 the MAC/IP withdraw for TS1 from NVE1, these remote PEs perform the 971 cleanups for their BGP tables. 973 5.2 IRB forwarding on NVEs for Subnets behind Tenant Systems 975 This section covers the symmetric IRB procedures for the scenario 976 where some Tenant Systems (TS's) support one or more subnets and 977 these TS's are associated with one or more NVEs. Therefore, besides 978 the advertisement of MAC/IP addresses for each TS which can be multi- 979 homed with All-Active redundancy mode, the associated NVE needs to 980 also advertise the subnets statically configured on each TS. 982 The main difference between this solution and the previous one is the 983 additional advertisement corresponding to each subnet. These subnet 984 advertisements are accomplished using EVPN IP Prefix route defined in 985 [EVPN-PREFIX]. These subnet prefixes are advertised with the IP 986 address of their associated TS (which is in overlay address space) as 987 their next hop. The receiving NVEs perform recursive route resolution 988 to resolve the subnet prefix with its associated ingress NVE so that 989 they know which NVE to forward the packets to when they are destined 990 for that subnet prefix. 992 The advantage of this recursive route resolution is that when a TS 993 moves from one NVE to another, there is no need to re-advertise any 994 of the subnet prefixes for that TS. All it is needed is to advertise 995 the IP/MAC addresses associated with the TS itself and exercise MAC 996 mobility procedures for that TS. The recursive route resolution 997 automatically takes care of the updates for the subnet prefixes of 998 that TS. 1000 Figure below illustrates this scenario where a given tenant (e.g., an 1001 IP-VPN service) has three subnets represented by MAC-VRF1, MAC-VRF2, 1002 and MAC-VRF3 across two NVEs. There are four TS's associated with 1003 these three MAC-VRFs - i.e., TS1, TS5 are connected to MAC-VRF1 on 1004 NVE1, TS2 is connected to MAC-VRF2 on NVE1, TS3 is connected to MAC- 1005 VRF3 on NVE2, and TS4 is connected to MAC-VRF1 on NVE2. TS1 has two 1006 subnet prefixes (SN1 and SN2) and TS3 has a single subnet prefix, 1007 SN3. The MAC-VRFs on each NVE are associated with their corresponding 1008 IP-VRF using their IRB interfaces. When TS4 and TS1 exchange intra- 1009 subnet traffic, only L2 forwarding (bridging) part of the IRB 1010 solution is used (i.e., the traffic only goes through their MAC- 1011 VRFs); however, when TS3 wants to forward traffic to SN1 or SN2 1012 sitting behind TS1 (inter-subnet traffic), then both bridging and 1013 routing parts of the IRB solution are exercised (i.e., the traffic 1014 goes through the corresponding MAC-VRFs and IP-VRFs). The following 1015 subsections describe the control and data planes operations for this 1016 IRB scenario in details. 1018 NVE1 +----------+ 1019 SN1--+ +-------------+ | | 1020 |--TS1-----|(MAC- \ | | | 1021 SN2--+ IP1/M1 | VRF1) \ | | | 1022 | (IP-VRF)|---| | 1023 | / | | | 1024 TS2-----|(MAC- / | | MPLS/ | 1025 IP2/M2 | VRF2) | | VxLAN/ | 1026 +-------------+ | NVGRE | 1027 +-------------+ | | 1028 SN3--+--TS3-----|(MAC-\ | | | 1029 IP3/M3 | VRF3)\ | | | 1030 | (IP-VRF)|---| | 1031 | / | | | 1032 TS4-----|(MAC- / | | | 1033 IP4/M4 | VRF1) | | | 1034 +-------------+ +----------+ 1035 NVE2 1037 Figure 7: IRB forwarding on NVEs for Tenant Systems with configured subnets 1039 5.2.1 Control Plane Operation 1041 Each NVE advertises a Route Type-5 (RT-5, IP Prefix Route defined in 1042 [EVPN-PREFIX]) for each of its subnet prefixes with the IP address of 1043 its TS as the next hop (gateway address field) as follow: 1045 - RD associated with the IP-VRF 1046 - ESI = 0 1047 - Ethernet Tag = 0; 1048 - IP Prefix Length = 32 or 128 1049 - IP Prefix = SNi 1050 - Gateway Address = IPi; IP address of TS 1051 - Label = 0 1053 This RT-5 is advertised with one or more Route Targets that have been 1054 configured as "export route targets" of the IP-VRF from which the 1055 route is originated. 1057 Each NVE also advertises an RT-2 (MAC/IP Advertisement Route) along 1058 with their associated Route Targets and Extended Communities for each 1059 of its TS's exactly as described in section 5.1.1. 1061 Upon receiving the RT-5 advertisement, the receiving NVE performs the 1062 following: 1064 - It uses the Route Target to identify the corresponding IP-VRF 1065 - It imports the IP prefix into its corresponding IP-VRF that is 1066 configured with an import RT that is one of the RTs being carried by 1067 the RT-5 route along with the IP address of the associated TS as its 1068 next hop. 1070 When receiving the RT-2 advertisement, the receiving NVE imports 1071 MAC/IP addresses of the TS into the corresponding MAC-VRF and IP-VRF 1072 per section 5.1.1. When both routes exist, recursive route resolution 1073 is performed to resolve the IP prefix (received in RT-5) to its 1074 corresponding NVE's IP address (e.g., its BGP next hop). BGP next hop 1075 will be used as underlay tunnel destination address (e.g., VTEP DA 1076 for VxLAN encapsulation) and Router's MAC will be used as inner MAC 1077 for VxLAN encapsulation. 1079 5.2.2 Data Plane Operation 1081 The following description of the data-plane operation describes just 1082 the logical functions and the actual implementation may differ. Lets 1083 consider data-plane operation when a host on SN1 sitting behind TS1 1084 wants to send traffic to a host sitting behind SN3 behind TS3. 1086 - TS1 send a packet with MAC DA corresponding to the MAC-VRF1 IRB 1087 interface of NVE1, and VLAN-tag corresponding to MAC-VRF1. 1089 - Upon receiving the packet, the ingress NVE1 uses VLAN-tag to 1090 identify the MAC-VRF1. It then looks up the MAC DA and forwards the 1091 frame to its IRB interface just like section 5.1.1. 1093 - The Ethernet header of the packet is stripped and the packet is fed 1094 to the IP-VRF; where, IP lookup is performed on the destination 1095 address. This lookup yields the fields needed for VxLAN encapsulation 1096 with NVE2's MAC address as the inner MAC DA, NVE'2 IP address as the 1097 VTEP DA, and the VNID. MAC SA is set to NVE1's MAC address and VTEP 1098 SA is set to NVE1's IP address. 1100 - The packet is then encapsulated with the proper header based on 1101 the above info and is forwarded to the egress NVE (NVE2). 1103 - On the egress NVE (NVE2), assuming the packet is VxLAN 1104 encapsulated, the VxLAN and the inner Ethernet headers are removed 1105 and the resultant IP packet is fed to the IP-VRF associated with that 1106 the VNID. 1108 - Next, a lookup is performed based on IP DA (which is in SN3) in the 1109 associated IP-VRF of NVE2. The IP lookup yields the access-facing IRB 1110 interface over which the packet needs to be sent. Before sending the 1111 packet over this interface, the ARP table is consulted to get the 1112 destination TS (TS3) MAC address. 1114 - The IP packet is encapsulated with an Ethernet header with the MAC 1115 SA set to that of the access-facing IRB interface of the egress NVE 1116 (NVE2) and the MAC DA is set to that of destination TS (TS3) MAC 1117 address. The packet is sent to the corresponding MAC-VRF3 and after a 1118 lookup of MAC DA, is forwarded to the destination TS (TS3) over the 1119 corresponding interface. 1121 6 Inter-Subnet DCI Scenarios 1123 The inter-subnet DCI scenarios can be categorized into the following 1124 four categories. The last two scenarios, along with its corresponding 1125 solution, are described in [EVPN-IPVPN-INTEROP]. The first two 1126 scenarios are covered in this document. 1128 1. Switching among IP subnets in different DCs using EVPN without GW 1130 2. Switching among IP subnets in different DCs using EVPN with GW 1132 3. Switching among IP subnets spread across IP-VPN and EVPN networks 1133 with GW 1135 4. Switching among IP subnets spread across IP-VPN and EVPN networks 1136 without GW 1138 In the above scenario, the term "GW" refers to the case where a node 1139 situated at the WAN edge of the data center network behaves as a 1140 default gateway (GW) for all the destinations that are outside the 1141 data center. The absence of GW refers to the scenario where NVEs 1142 within a data center maintain individual (host) routes that are 1143 outside of the data center. 1145 In the case (3), the WAN edge node also performs route aggregation 1146 for all the destinations within its own data center, and acts as an 1147 interworking unit between EVPN and IP VPN (it implements both EVPN 1148 and IP-VPN functionality). 1150 +---+ Enterprise Site 1 1151 |PE1|----- H1 1152 +---+ 1153 / 1154 ,---------. Enterprise Site 2 1155 ,' `. +---+ 1156 ,---------. /( MPLS/IP )---|PE2|----- H2 1157 ' DCN 3 `./ `. Core ,' +---+ 1158 `-+------+' `-+------+' 1159 __/__ / / \ \ 1160 :NVE4 : +---+ \ \ 1161 '-----' ,----|GW |. \ \ 1162 | ,' +---+ `. ,---------. 1163 TS6 ( DCN 1 ) ,' `. 1164 `. ,' ( DCN 2 ) 1165 `-+------+' `. ,' 1166 __/__ `-+------+' 1167 :NVE1 : __/__ __\__ 1168 '-----' :NVE2 : :NVE3 : 1169 | | '-----' '-----' 1170 TS1 TS2 | | | 1171 TS3 TS4 TS5 1173 Figure 8: Interoperability Use-Cases 1175 In what follows, we will describe scenarios 1 and 2 in more details. 1177 6.1 Switching among IP subnets in different DCs without GW 1179 This case is similar to that of section 2.1 above albeit for the fact 1180 that the TS's belong to different data centers that are 1181 interconnected over a WAN (e.g. MPLS/IP PSN). The data centers in 1182 question here are seamlessly interconnected to the WAN, i.e., the WAN 1183 edge devices do not maintain any TS-specific addresses in the 1184 forwarding path - e.g., there is no WAN edge GW(s) between these DCs. 1186 As an example, consider TS3 and TS6 of Figure 2 above. Assume that 1187 connectivity is required between these two TS's where TS3 belongs to 1188 the SN3 whereas TS6 belongs to the SN6. NVE2 has an EVI3 associated 1189 with SN3 and NVE4 has an EVI6 associated with the SN6. Both SN3 and 1190 SN6 are part of the same IP-VRF. 1192 When an EVPN MAC advertisement route is received by a NVE, the IP 1193 address associated with the route is used to populate the IP-VRF 1194 table, whereas the MAC address associated with the route is used to 1195 populate both the MAC-VRF table, as well as the adjacency associated 1196 with the IP route in the IP-VRF table (i.e., ARP table). 1198 When an Ethernet frame is received by an ingress NVE, it performs a 1199 lookup on the destination MAC address in the associated EVI. If the 1200 MAC address corresponds to its IRB Interface MAC address, the ingress 1201 NVE deduces that the packet MUST be inter-subnet routed. Hence, the 1202 ingress NVE performs an IP lookup in the associated IP-VRF table. The 1203 lookup identifies an adjacency that contains a MAC rewrite and in 1204 turn the next-hop (i.e. egress) Gateway to which the packet must be 1205 forwarded along with the associated MPLS label stack. The MAC rewrite 1206 holds the MAC address associated with the destination host (as 1207 populated by the EVPN MAC route), instead of the MAC address of the 1208 next-hop Gateway. The ingress NVE then rewrites the destination MAC 1209 address in the packet with the address specified in the adjacency. It 1210 also rewrites the source MAC address with its IRB Interface MAC 1211 address for the destination subnet. The ingress NVE, then, forwards 1212 the frame to the next-hop (i.e. egress) Gateway after encapsulating 1213 it with the MPLS label stack. 1215 Note that this label stack includes the LSP label as well as an EVPN 1216 label. The EVPN label could be either advertised by the ingress 1217 Gateway, if inter-AS option B is used, or advertised by the egress 1218 NVE, if inter-AS option C is used. When the MPLS encapsulated packet 1219 is received by the ingress Gateway, the processing again differs 1220 depending on whether inter-AS option B or option C is employed: in 1221 the former case, the ingress Gateway swaps the EVPN label in the 1222 packets with the EVPN label value received from the egress Gateway. 1223 In the latter case, the ingress Gateway does not modify the EVPN 1224 label and performs normal label switching on the LSP label. 1225 Similarly on the egress Gateway, for option B, the egress Gateway 1226 swaps the EVPN label with the value advertised by the egress NVE. 1227 Whereas, for option C, the egress Gateway does not modify the EVPN 1228 label, and performs normal label switching on the LSP label. When the 1229 MPLS encapsulated packet is received by the egress NVE, it uses the 1230 EVPN label to identify the bridge-domain table. It then performs a 1231 MAC lookup in that table, which yields the outbound interface to 1232 which the Ethernet frame must be forwarded. Figure 3 below depicts 1233 the packet flow. 1235 NVE1 ASBR1 ASBR2 NVE2 1236 +------------+ +------------+ +------------+ +------------+ 1237 | | | | | | | | 1238 |(MAC - (IP | | [LS] | | [LS] | |(IP - (MAC | 1239 | VRF) VRF)| | | | | | VRF) VRF)| 1240 | | | | | | | | | | | | | | | | 1241 +------------+ +------------+ +------------+ +------------+ 1242 ^ v ^ V ^ V ^ V 1243 | | | | | | | | 1244 TS1->-+ +-->--------+ +------------+ +---------------+ +->-TS2 1246 Figure 9: Inter-Subnet Forwarding Among EVPN NVEs in Different DCs 1247 without GW 1249 6.2 Switching among IP subnets in different DCs with GW 1251 In this scenario, connectivity is required between TS's in different 1252 data centers, and those hosts belong to different IP subnets. What 1253 makes this case different from that of Section 2.2 is that at least 1254 one of the data centers has a gateway as the WAN edge switch. Because 1255 of that, the NVE's IP-VRF within that data center need not maintain 1256 (host) routes to individual TS's outside of that data center. 1258 As an example, consider a tenant with TS1 and TS5 of Figure 2 above. 1259 Assume that connectivity is required between these two TS's where TS1 1260 belongs to the SN1 whereas TS5 belongs to the SN5. NVE3 has an EVI5 1261 associated with the SN5 and this EVI is represented by the MAC-VRF 1262 which is connected to the IP-VRF via an IRB interface. NVE1 has an 1263 EVI1 associated with the SN1 and this EVI is represented by the MAC- 1264 VRF which is connected to the IP-VRF representing the same tenant. 1265 Due to the gateway at the edge of DCN 1, NVE1's IP-VRF does not need 1266 to have the address of TS5 but instead it has a default route in its 1267 IP-VRF with the next-hop being the GW. 1269 In this scenario, the NVEs within a given data center do not have 1270 entries for the MAC/IP addresses of hosts in remote data centers. 1271 Rather, the NVEs have a default IP route pointing to the WAN gateway 1272 for each VRF. This is accomplished by the WAN gateway advertising for 1273 a given EVPN that spans multiple DC a default VPN-IP route that is 1274 imported by the NVEs of that VPN that are in the gateway's own DC. 1276 When an Ethernet frame is received by an ingress NVE, it performs a 1277 lookup on the destination MAC address in the associated MAC-VRF 1278 table. If the MAC address corresponds to the IRB Interface MAC 1279 address, the ingress NVE deduces that the packet MUST be inter-subnet 1280 routed. Hence, the ingress NVE performs an IP lookup in the 1281 associated IP-VRF table. The lookup, in this case, matches the 1282 default host route which points to the local WAN gateway (GW1). The 1283 ingress NVE (NVE1) then rewrites the destination MAC address in the 1284 packet with the MAC address of core-facing IRB interface of GW1 (not 1285 shown in the figure) or it can rewrite it with the router's MAC 1286 address of GW1. It also rewrites the source MAC address with its own 1287 core-facing IRB Interface's MAC address for the destination subnet 1288 (i.e., the subnet between NVE1 and GW1) or it can rewrite it with its 1289 own router's MAC address of NVE1. The ingress NVE, then, forwards the 1290 frame to GW1 after encapsulating it with the MPLS label stack. Note 1291 that this label stack includes the LSP label as well as the label for 1292 default host route that was advertised by the local WAN gateway. When 1293 the MPLS encapsulated packet is received by GW1, it uses the default 1294 host route MPLS label to identify the core-facing MAC-VRF. It does a 1295 MAC-DA lookup and forwards the packet to the IP-VRF after stripping 1296 the Ethernet header. It then performs an IP lookup in that table. The 1297 lookup identifies an adjacency that contains a MAC rewrite and in 1298 turn the remote WAN gateway (GW2) to which the packet must be 1299 forwarded along with the associated MPLS label stack. The MAC rewrite 1300 holds the MAC address associated with the ultimate destination host 1301 (as populated by the EVPN MAC route). GW1 then rewrites the 1302 destination MAC address in the packet with the address specified in 1303 the adjacency. It also rewrites the source MAC address with the MAC 1304 address of its core-facing IRB interface (not shown in the figure) or 1305 its router's MAC address. GW1, then, forwards the frame to the GW2 1306 after encapsulating it with the MPLS label stack. Note that this 1307 label stack includes the LSP label as well as a EVPN label that was 1308 advertised by GW2. When the MPLS encapsulated packet is received by 1309 GW2, it uses the EVPN label to identify the destination MAC-VRF. It 1310 then performs a MAC-DA lookup and grabs the EVPN label advertised by 1311 NVE2 along with adjacencies info. It then encapsulates the packet 1312 with the corresponding label stack and forwards the packet to NVE2. 1313 It should be noted that no MAC header re-write is performed on GW2. 1314 This implies that both GW1 and GW2 need to keep the remote host MAC 1315 addresses along with the corresponding EVPN labels in their tables. 1316 The egress NVE (NVE2) then upon receiving the packet, performs a MAC 1317 lookup in the MAC-VRF (identified by the received EVPN label) to 1318 determine the outbound port to send the traffic on. 1320 Figure 4 below depicts the forwarding model. 1322 NVE1 GW1 GW2 NVE2 1323 +------------+ +------------+ +------------+ +------------+ 1324 | | | | | | | | 1325 |(MAC - (IP | |(IP - (MAC | | (MAC | |(IP - (MAC | 1326 | VRF) VRF)| | VRF) VRF)| | VRF) | | VRF) VRF)| 1327 | | | | | | | | | | | | | | | | 1328 +------------+ +------------+ +------------+ +------------+ 1329 ^ v ^ V ^ V ^ V 1330 | | | | | | | | 1331 TS1->-+ +-->-----+ +---------------+ +---------------+ +->-TS2 1333 Figure 10: Inter-Subnet Forwarding Among EVPN NVEs in Different DCs 1334 with GW 1336 7 TS Mobility 1338 7.1 TS Mobility & Optimum Forwarding for TS Outbound Traffic 1340 Optimum forwarding for the TS outbound traffic, upon TS mobility, can 1341 be achieved using either the anycast default Gateway MAC and IP 1342 addresses, or using the address aliasing as discussed in [DC- 1343 MOBILITY]. 1345 7.2 TS Mobility & Optimum Forwarding for TS Inbound Traffic 1347 For optimum forwarding of the TS inbound traffic, upon TS mobility, 1348 all the NVEs and/or IP-VPN PEs need to know the up to date location 1349 of the TS. Two scenarios must be considered, as discussed next. 1351 In what follows, we use the following terminology: 1353 - source NVE refers to the NVE behind which the TS used to reside 1354 prior to the TS mobility event. 1356 - target NVE refers to the new NVE behind which the TS has moved 1357 after the mobility event. 1359 7.2.1 Mobility without Route Aggregation 1361 In this scenario, when a target NVE detects that a MAC mobility event 1362 has occurred, it initiates the MAC mobility handshake in BGP as 1363 specified in section 5.1.3. The WAN Gateways, acting as ASBRs in this 1364 case, re-advertise the MAC route of the target NVE with the MAC 1365 Mobility extended community attribute unmodified. Because the WAN 1366 Gateway for a given data center re-advertises BGP routes received 1367 from the WAN into the data center, the source NVE will receive the 1368 MAC Advertisement route of the target NVE (with the next hop 1369 attribute adjusted depending on which inter-AS option is employed). 1370 The source NVE will then withdraw its original MAC Advertisement 1371 route as a result of evaluating the Sequence Number field of the MAC 1372 Mobility extended community in the received MAC Advertisement route. 1373 This is per the procedures already defined in [EVPN]. 1375 8 Acknowledgements 1377 The authors would like to thank Sami Boutros and Jeffrey Zhang for 1378 their valuable comments. 1380 9 Security Considerations 1382 The security considerations discussed in [EVPN] apply to this 1383 document. 1385 10 IANA Considerations 1387 IANA has allocated a new transitive extended community Type of 0x06 1388 and Sub-Type of 0x03 for EVPN Router's MAC Extended Community. 1390 11 References 1392 11.1 Normative References 1394 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1395 Requirement Levels", BCP 14, RFC 2119, March 1997. 1397 [EVPN] Sajassi et al., "BGP MPLS Based Ethernet VPN", RFC 7432, 1398 February, 2015. 1400 [TUNNEL-ENCAP] Rosen et al., "The BGP Tunnel Encapsulation 1401 Attribute", draft-ietf-idr-tunnel-encaps-03, November 1402 2016. 1404 [EVPN-PREFIX] Rabadan et al., "IP Prefix Advertisement in EVPN", 1405 draft-ietf-bess-evpn-prefix-advertisement-03, September, 1406 2016. 1408 11.2 Informative References 1410 [RFC7606] Chen, E., Scudder, J., Mohapatra, P., and K. Patel, 1411 "Revised Error Handling for BGP UPDATE Messages", RFC 7606, August 1412 2015, . 1414 [802.1Q] "IEEE Standard for Local and metropolitan area networks - 1415 Media Access Control (MAC) Bridges and Virtual Bridged Local Area 1416 Networks", IEEE Std 802.1Q(tm), 2014 Edition, November 2014. 1418 [EVPN-IPVPN-INTEROP] Sajassi et al., "EVPN Seamless Interoperability 1419 with IP-VPN", draft-sajassi-l2vpn-evpn-ipvpn-interop-01, work in 1420 progress, October, 2012. 1422 [DC-MOBILITY] Aggarwal et al., "Data Center Mobility based on 1423 BGP/MPLS, IP Routing and NHRP", draft-raggarwa-data-center-mobility- 1424 05.txt, work in progress, June, 2013. 1426 12 Contributors 1428 In addition to the authors listed on the front page, the following 1429 co-authors have also contributed to this document: 1431 Florin Balus 1432 Cisco 1434 Yakov Rekhter 1435 Juniper 1437 Wim Henderickx 1438 Nokia 1440 Linda Dunbar 1441 Huawei 1443 Dennis Cai 1444 Alibaba 1446 Authors' Addresses 1448 Ali Sajassi (Editor) 1449 Cisco 1450 Email: sajassi@cisco.com 1452 Samer Salam 1453 Cisco 1454 Email: sslam@cisco.com 1456 Samir Thoria 1457 Cisco 1458 Email: sthoria@cisco.com 1459 John E. Drake 1460 Juniper Networks 1461 Email: jdrake@juniper.net 1463 Lucy Yong 1464 Huawei Technologies 1465 Email: lucy.yong@huawei.com 1467 Jorge Rabadan 1468 Nokia 1469 Email: jorge.rabadan@nokia.com