idnits 2.17.1 draft-ietf-trill-irb-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- -- The document has examples using IPv4 documentation addresses according to RFC6890, but does not use any IPv6 documentation addresses. Maybe there should be IPv6 examples, too? Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (January 28, 2016) is 3010 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. 'IS-IS' Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 TRILL Working Group W. Hao 2 INTERNET-DRAFT Y. Li 3 Intended Status: Standard Track Huawei 4 A. Qu 5 MediaTec 6 M. Durrani 7 Cisco 8 P. Sivamurugan 9 IP Infusion 10 L. Xia 11 Huawei 12 Expires: July 28, 2016 January 28, 2016 14 TRILL Distributed Layer 3 Gateway 15 draft-ietf-trill-irb-10.txt 17 Abstract 19 The base TRILL protocol provides optimal pair-wise data frame 20 forwarding for layer 2 intra-subnet traffic but not for layer 3 21 inter-subnet traffic. A centralized gateway solution is typically 22 used for layer 3 inter-subnet traffic forwarding but has the 23 following issues: 25 1. Sub-optimum forwarding paths for inter-subnet traffic. 27 2. A centralized gateway may need to support a very large 28 number of gateway interfaces in a data center, one per tenant per 29 data label used by that tenant, to provide interconnect 30 functionality for all the layer 2 virtual networks in a TRILL campus. 32 3. A traffic bottleneck at the gateway. 34 This document specifies an optional TRILL distributed gateway 35 solution that resolves these centralized gateway issues. 37 Status of this Memo 39 This Internet-Draft is submitted in full conformance with the 40 provisions of BCP 78 and BCP 79. 42 Internet-Drafts are working documents of the Internet Engineering 43 Task Force (IETF), its areas, and its working groups. Note that 44 other groups may also distribute working documents as Internet- 45 Drafts. 47 Internet-Drafts are draft documents valid for a maximum of six 48 months and may be updated, replaced, or obsoleted by other documents 49 at any time. It is inappropriate to use Internet-Drafts as 50 reference material or to cite them other than as "work in progress." 52 The list of current Internet-Drafts can be accessed at 53 http://www.ietf.org/1id-abstracts.html 55 The list of Internet-Draft Shadow Directories can be accessed at 56 http://www.ietf.org/shadow.html. 58 Copyright Notice 60 Copyright (c) 2015 IETF Trust and the persons identified as the 61 document authors. All rights reserved. 63 This document is subject to BCP 78 and the IETF Trust's Legal 64 Provisions Relating to IETF Documents 65 (http://trustee.ietf.org/license-info) in effect on the date of 66 publication of this document. Please review these documents 67 carefully, as they describe your rights and restrictions with 68 respect to this document. Code Components extracted from this 69 document must include Simplified BSD License text as described in 70 Section 4.e of the Trust Legal Provisions and are provided without 71 warranty as described in the Simplified BSD License. 73 Table of Contents 75 1. Introduction ................................................ 3 76 1.1. Document Organization................................... 3 77 2. Conventions used in this document............................ 4 78 3. Simplified Example and Problem Statement..................... 5 79 3.1. Simplified Example...................................... 5 80 3.2. Problem Statement Summary............................... 8 81 4. Layer 3 Traffic Forwarding Model............................. 9 82 5. Distributed Gateway Solution Details......................... 9 83 5.1. Local Routing Information.............................. 10 84 5.2. Local Routing Information Synchronization.............. 11 85 5.3. Active-active Access................................... 13 86 5.4. Data Traffic Forwarding Process........................ 14 87 6. Distributed Layer 3 Gateway Process Example................. 15 88 6.1. Control plane process.................................. 16 89 6.2. Data Plane Process..................................... 17 90 7. TRILL Protocol Extensions................................... 18 91 7.1. The Tenant Label and Gateway MAC APPsub-TLV............ 18 92 7.2. "SE" Flag in NickFlags APPsub-TLV...................... 19 93 7.3. The IPv4 Prefix APPsub-TLV............................. 19 94 7.4. The IPv6 Prefix APPsub-TLV............................. 20 95 8. Security Considerations..................................... 21 96 9. IANA Considerations ........................................ 22 97 10. Normative References....................................... 22 98 11. Informative References..................................... 23 99 Acknowledgments ............................................... 23 100 Authors' Addresses ............................................ 23 102 1. Introduction 104 The TRILL (Transparent Interconnection of Lots of Links) protocol 105 [RFC6325] provides a solution for least cost transparent routing in 106 multi-hop networks with arbitrary topologies and link technologies, 107 using [IS-IS] [RFC7176] link-state routing and a hop count. TRILL 108 switches are sometimes called RBridges (Routing Bridges). 110 The base TRILL protocol provides optimal unicast forwarding for 111 Layer 2 intra-subnet traffic but not for Layer 3 inter-subnet 112 traffic, where subnet means different IP address prefix and 113 typically a different Data Label (VLAN or FGL). In this document, a 114 TRILL-based distributed Layer 3 gateway solution is specified that 115 provides optimal unicast forwarding for Layer 3 inter-subnet traffic. 116 With distributed gateway support, an edge RBridge provides both 117 routing based on Layer 2 identity (address and virtual network (VN, 118 i.e. Data Label)) among end stations (ESs) that belong to same 119 subnet and also provides routing based on Layer 3 identity among ESs 120 that belong to different subnets of the same routing domain. An edge 121 RBridge supporting this feature needs to provide routing instances 122 and Layer 3 gateway interfaces for locally connected ESs. Such 123 routing instances provide IP address isolation between tenants. In 124 the TRILL distributed Layer 3 gateway solution, inter-subnet traffic 125 can be fully spread over edge RBridges, so there is no single 126 bottleneck. 128 1.1. Document Organization 130 This document is organized as follows: Section 3 gives a simplified 131 example and also a more detailed problem statement. Section 4 gives 132 the Layer 3 traffic forwarding model. Section 5 provides a 133 distributed gateway solution overview. Section 6 gives a detailed 134 distributed gateway solution example. And Section 7 describes the 135 TRILL protocol extensions needed to support this distributed gateway 136 solution. 138 2. Conventions used in this document 140 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 141 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 142 document are to be interpreted as described in RFC 2119 [RFC2119]. 144 The terms and acronyms in [RFC6325] are used with the following 145 additions: 147 ARP: Address Resolution Protocol [RFC826]. 149 Campus: The name for a network using the TRILL protocol in the 150 same sense that a ''bridged LAN'' is the name for a network using 151 bridging. In TRILL, the word ''campus'' has no academic implication. 153 Data Label: VLAN or FGL [RFC7172]. 155 DC: Data Center. 157 Edge RBridge: An RBridges that connects to one or more End 158 Stations without any intervening RBridges. 160 FGL: Fine Grained Label [RFC7172]. 162 ES: End Station. A VM (Virtual Machine) or physical server, 163 whose address is either the destination or source of a data frame. 165 Gateway interface: A Layer 3 virtual interface that terminates 166 layer 2 forwarding and forwards IP traffic to the destination using 167 IP forwarding rules. Incoming traffic from a physical port on a 168 gateway will be distributed to its virtual gateway interface based 169 on Data Label (VLAN or FGL). 171 Inner.MacDA: The inner MAC destination address in a TRILL Data 172 packet [RFC6325]. 174 Inner.MacSA: The inner MAC source address in a TRILL Data 175 packet [RFC6325]. 177 L2: Layer 2. 179 L3: IP Layer 3. 181 ND: IPv6's Neighbor Discovery [RFC4861]. 183 ToR: Top of Rack. 185 VN: Virtual Network. In a TRILL campus, a unique 12-bit VLAN 186 ID or a 24-bit Fine Grained Label [RFC7172] identifies each virtual 187 network. 189 VRF: Virtual Routing and Forwarding. In IP-based computer 190 networks, Virtual Routing and Forwarding (VRF) technology supports 191 multiple instances of routing tables existing within the same router 192 at the same time. 194 3. Simplified Example and Problem Statement 196 There is normally a Data Label (VLAN or FGL) associated with each IP 197 subnet. For traffic within a subnet, that is IP traffic to another 198 end station in the same Data Label attached to the TRILL campus, the 199 end station just ARPs for the MAC address of the destination end 200 station's IP. It then uses this MAC address for traffic to that 201 destination. TRILL routes the ingressed TRILL data packets to the 202 destination's edge RBridge based on the egress nickname for that 203 destination MAC address and Data Label. This is the regular TRILL 204 base protocol [RFC6325] process. 206 If two end stations of the same tenant are on different subnets and 207 need to communicate with each other, their packets are typically 208 forwarded to an IP Layer 3 gateway that performs L3 routing and, if 209 necessary, changes the Data Label. Either a centralized layer 3 210 gateway solution or the distributed layer 3 gateway solution 211 specified in this document can be used for the inter-subnet traffic 212 forwarding. 214 Section 3.1 gives a simplified example in a TRILL campus with and 215 without a distributed layer 3 gateway using VLAN Data Labels. 216 Section 3.2 gives the detailed description of the problem without a 217 distributed layer 3 gateway. The remainder of this document, 218 particularly Section 5, describes the distributed gateway solution 219 in detail. 221 3.1. Simplified Example 222 ------- -------- 223 | COR1| | COR2 | 224 ------- -------- 225 | | 226 ------- ------- 227 |AGG1 | |AGG2 | 228 ------- ------- 229 | | 230 ----------------------------------------------------- 231 | -------------|------------------|----------------| 232 | | | | | | | | 233 ------- ------- ------- ------- 234 | RB1 | | RB2 | | RB3 | | RB4 | 235 |TOR1 | |TOR2 | |TOR3 | |TOR4 | 236 ------- ------- ------- ------- 237 | | | | | | | | 238 ----- ----- ----- ----- ----- ----- ----- ----- 239 |ES1| |ES2| |ES3| |ES4| |ES5| |ES6| |ES7| |ES8| 240 ----- ----- ----- ----- ----- ----- ----- ----- 242 Figure 1. A Typical TRILL DC Network 244 Figure 1 depicts a TRILL Data Center Network where Top of Rack (ToR) 245 switches are edge RBridges. ES1 to ES8 belong to one tenant network 246 and the tenant has four subnets with each subnet corresponding to 247 one VLAN (which indicates one individual layer 2 virtual 248 network). Each ES's IP address, VLAN and subnet are listed below: 250 +----+----------------+-----------------+----------+ 251 | ES | IP Address | Subnet | VLAN | 252 +----+----------------+-----------------+----------+ 253 | ES1| 192.0.2.2 | 192.0.2.0/24 | 10 | 254 +----+----------------+-----------------+----------+ 255 | ES2| 198.51.100.2 | 198.51.100.0/24 | 11 | 256 +----+----------------+-----------------+----------+ 257 | ES3| 192.0.2.3 | 192.0.2.0/24 | 10 | 258 +----+----------------+-----------------+----------+ 259 | ES4| 198.51.100.3 | 198.51.100.0/24 | 11 | 260 +----+----------------+-----------------+----------+ 261 | ES5| 203.0.113.2 | 203.0.113.0/25 | 12 | 262 +----+----------------+-----------------+----------+ 263 | ES6| 203.0.113.130 | 203.0.113.128/25| 13 | 264 +----+----------------+-----------------+----------+ 265 | ES7| 203.0.113.3 | 203.0.113.0/25 | 12 | 266 +----+----------------+-----------------+----------+ 267 | ES8| 203.0.113.131 | 203.0.113.128/25| 13 | 268 +----+----------------+-----------------+----------+ 269 Assume a centralized gateway solution is used with both COR1 and 270 COR2 acting as centralized gateways for redundancy in figure 1. COR1 271 and COR2 each have four gateway interfaces for the four subnets in 272 the tenant. In centralized layer 3 gateway solution, all traffic 273 within the tenant between different VLANs must go through the 274 centralized layer 3 gateway device of COR1 or COR2, even if the 275 traffic is between two end stations connected to the same edge 276 RBridge, because only the layer 3 gateway can change the VLAN 277 labeling of the traffic. 279 This is generally sub-optimal because the two end stations may be 280 connected to the same ToR where L3 switching could have been 281 performed locally. For example, in above Figure 1, the unicast IP 282 traffic between ES1 and ES2 has to go through a centralized gateway 283 of COR1 or COR2. It can't be locally routed between them on TOR1. 284 However, if an edge RBridge has the distributed gateway capabilities 285 specified in this document, then it can still perform optimum L2 286 forwarding for intra-subnet traffic and, in addition, optimum L3 287 forwarding for inter-subnet traffic, thus delivering optimum 288 forwarding for unicast packets in all important cases. 290 With a distributed layer 3 gateway, each edge RBridge acts as a 291 default layer 3 gateway for local connecting ESs and has IP router 292 capabilities to direct IP communications to other edge RBridges. 293 Each edge RBridge only needs gateway interfaces for local connecting 294 ESs, i.e., RB1 and RB2 need gateway interfaces only for VLAN 10 and 295 VLAN 11 while RB3 and RB4 need gateway interfaces only for VLAN 12 296 and VLAN 13. No device needs to maintain gateway interfaces for all 297 VLANs in entire network. This will enhance the scalability in terms 298 of number of tenants and subnets per tenant. 300 When each end station ARPs for their layer 3 gateway, that is, their 301 IP router, the edge RBridge to which it is connected will respond 302 with that RBridge's 'gateway MAC'. When the end station later sends 303 IP traffic to the layer 3 gateway, which it does if the destination 304 IP is outside of its subnet, the edge RBridge intercepts the IP 305 packet because the destination MAC is its gateway MAC. That RBridge 306 routes the IP packet using the routing instance associated with that 307 tenant, handling it in one of three ways: 309 (1) ES1 communicates with ES2. The destination IP is 310 connected to the same edge RBridge, the RBridge of TOR1 can simply 311 transmit the IP packet out the right edge port in the destination 312 VLAN. 314 (2) If the destination IP is located in an outside network, 315 the edge RBridge encapsulates it as a TRILL Data packet and sends it 316 to the actual TRILL campus edge RBridge connecting to an external IP 317 router. 319 (3) ES1 communicates with ES4. The destination end station 320 is connected to a different edge RBridge, the ingress RBridge TOR1 321 uses TRILL encapsulation to route the IP packet to the correct 322 egress RBridge TOR2, using the egress RBridge's gateway MAC and an 323 Inner.VLAN identifying the tenant. Finally, the egress RBridge 324 terminates the TRILL encapsulation and routes the IP packet to the 325 destination end station based on the routing instance for that 326 tenant. 328 3.2. Problem Statement Summary 330 With Fine Grained Labeling [RFC7172], in theory, up to 16 million 331 Layer 2 VN can be supported in a TRILL campus. To support inter- 332 subnet traffic, a very large number of Layer 3 gateway interfaces 333 could be needed on a centralized gateway, if each VN corresponds to 334 a subnet and there are many tenant with many subnets per tenant. It 335 is a big burden for the centralized gateway to support so many 336 interfaces. In addition all inter-subnet traffic will go through a 337 centralized gateway that may become the traffic bottleneck. 339 The centralized gateway has the following issues: 341 1. Sub-optimum forwarding paths for inter-subnet traffic 342 due to the requirements to perform IP routing and possibly change 343 Data Labels at a centralized gateway. 345 2. The centralized gateway may need to support a very large 346 number of gateway interfaces, in a data center one per tenant per 347 data label used by that tenant, to provide interconnect 348 functionality for all the layer 2 virtual networks in the TRILL 349 campus. 351 3. There may be a traffic bottleneck at the centralized 352 gateway. 354 A distributed gateway on edge RBridges addresses these issues. 355 Through the distributed layer 3 gateway solution, the inter-subnet 356 traffic is fully dispersed and is transmitted along optimal pair- 357 wise forwarding path, improving network efficiency. 359 4. Layer 3 Traffic Forwarding Model 361 +---------------------------------------------+ 362 | | 363 | +-----------+ +-----------+ | 364 | | Tenant n |---------| VRF n | | 365 | +------------+ | +------------+ | | 366 | | +-----+ | | | | | | 367 | | | VN1 | | | | | | | 368 | | +-----+ | | | VRF 1 | | | 369 | | .. +-------+ | | | 370 | | +-----+ | | | | | | 371 | | | VNm | | | | | | | 372 | | +-----+ | | | | | | 373 | | Tenant 1 |-+ | | | | 374 | +------------+ | | | | 375 | +------------+ +------------+ | 376 | | 377 | Edge RBridge | 378 +---------------------------------------------+ 380 Figure 2. Edge RBridge Model as Distributed Gateway 382 In a data center network, each tenant has one or more Layer 2 383 virtual networks and, in normal cases, each tenant corresponds to 384 one routing domain. Normally each Layer 2 virtual network uses a 385 different Data Label and corresponds to one or more IP subnets. 387 Each Layer 2 virtual network in a TRILL campus is identified by a 388 unique 12-bit VLAN ID or 24-bit Fine Grained Label [RFC7172]. 389 Different routing domains may have overlapping address space but 390 need distinct and separate routes. The end stations that belong to 391 the same subnet communicate through L2 forwarding, end stations of 392 the same tenant that belong to different subnets communicate through 393 L3 routing. 395 Figure 2 depicts the model where there are n VRFs corresponding to n 396 tenants with each tenant having up to m segments/subnets (virtual 397 network). 399 5. Distributed Gateway Solution Details 401 With the TRILL distributed gateway solution, an edge RBridge 402 continues to perform Layer 2 routing for the ESs that are on the 403 same subnet but performs IP routing for the ESs that are on the 404 different subnets of the same tenant. 406 As the IP address space in different routing domains can overlap, 407 VRF instances need to be created on each edge RBridge to isolate the 408 IP forwarding process for different routing domains present on the 409 edge RBridge. A globally unique tenant ID identifies each routing 410 domain. The network operator MUST ensure the consistency of the 411 tenant ID on each edge RBridge for each routing domain. If a routing 412 domain spreads over multiple edge RBridges, routing information for 413 the routing domain is synchronized among these edge RBridges to 414 ensure reachability to all ESs in that routing domain. The routing 415 information is, in effect, labeled with the Tenant ID to 416 differentiate the routing domains. 418 From the data plane perspective, all edge RBridges are connected to 419 each other via one or more TRILL hops, however they are always just 420 a single IP hop away. When an ingress RBridge receives inter-subnet 421 IP traffic from a local ES whose destination MAC is the edge 422 RBridge's gateway MAC, that RBridge will perform Ethernet header 423 termination and look up in its IP routing table how to route the 424 traffic to the IP next hop. If the destination ES is connected to a 425 remote edge RBridge, the remote RBridge will be the IP next hop for 426 traffic forwarding. For such inter-subnet traffic, the ingress 427 RBridge will rewrite the original Ethernet header with the ingress 428 RBridge's gateway MAC address as the Inner.MacSA and the egress 429 RBridge's gateway MAC address as the Inner.MacDA and then perform 430 TRILL encapsulation to the remote RBridge's nickname. TRILL then 431 routes it to the remote edge RBridge through the TRILL campus. 433 When that remote edge RBridge receives the traffic, it will 434 decapsulate the TRILL data packet and see that the inner destination 435 MAC is its gateway MAC. It then terminates the inner Ethernet 436 encapsulation and looks up the destination IP in the RBridge's IP 437 forwarding table for the tenant indicated by the inner Data Label to 438 route it to the destination ES. 440 Through this method, TRILL with distributed gateways provides 441 optimum pair-wise data routing for inter-subnet traffic. 443 5.1. Local Routing Information 445 An ES can be locally connected to an edge RBridges through a layer 2 446 network (such as a point-to-point Ethernet link or a bridged LAN) or 447 externally connected through a layer 3 IP network. 449 If the ES is connected to an edge RBridge through a Layer 2 network, 450 then the edge RBridge acts as a Layer 3 Gateway for the ES. A 451 gateway interface is established on the edge RBridge for the 452 connecting ES. Because the ESs in a subnet may be spread over 453 multiple edge RBridges, in each of these edge RBridges which 454 establishes its gateway interface for the subnet the edge RBridges 455 SHOULD share the same gateway MAC and gateway IP address 456 configuration. Sharing the configuration and insuring configuration 457 consistency can be done by local configuration and netconf/Yang 458 models. 460 With distributed gateway, the edge RBridge to which an end station 461 is connected appears to be the local IP router on its link. As in 462 any IP network, before the end station starts to send inter-subnet 463 traffic, it acquires its gateway's MAC through the ARP/ND process. 464 Local connecting edge RBridges that support this distributed gateway 465 feature always respond with the gateway MAC address when receiving 466 ARP/ND requests for the gateway IP. Through the ARP/ND process, the 467 edge RBridge can learn the IP and MAC correspondence of a local ES 468 connected to the edge RBridge by Layer 2 and then generate local IP 469 routing entries for that ES in the corresponding routing domain. 471 An IP router connected to an edge RBridge looks to TRILL like an ES. 472 If a router/ES is located in an external IP network, normally it 473 provides access to one or more IP prefixes. The router/ES should run 474 an IP routing protocol with the connecting TRILL edge Rbridge. The 475 edge RBridge will learn the IP prefixes behind the router/ES through 476 that IP routing protocol, then the RBridge will generate local IP 477 routing entries in the corresponding routing domain. 479 5.2. Local Routing Information Synchronization 481 When a routing instance is created on an edge RBridge, the tenant ID, 482 tenant Label (VLAN or FGL), tenant gateway MAC, and their 483 correspondence should be set and globally advertised (see Section 484 7.1). 486 When an ingress RBridge performs inter-subnet traffic TRILL 487 encapsulation, the ingress RBridge uses the Label advertised by the 488 egress RBridge as the inner VLAN or FGL and uses the tenant gateway 489 MAC advertised by the egress RBridge as the Inner.MacDA. The egress 490 RBridge relies on this tenant Data Label to find the local VRF 491 instance for the IP forwarding process when receiving inter-subnet 492 traffic from the TRILL campus. (The role of tenant Label is akin to 493 an MPLS VPN Label in an MPLS IP/MPLS VPN network.) Tenant Data 494 Labels are independently allocated on each edge RBridge for each 495 routing domain. An edge RBridge can use an access Data Label from a 496 routing domain to act as the inter-subnet Label, or the edge RBridge 497 can use a Label different from any access Labels to be a tenant 498 Label. It is implementation dependent and there is no restriction on 499 this assignment of labels. 501 The tenant gateway MAC differentiates inter-subnet Layer 3 traffic 502 or intra-subnet Layer 2 traffic on the egress RBridge. Each tenant 503 on a RBridge can use a different gateway MAC or same tenant gateway 504 MAC for inter-subnet traffic purposes. This is also implementation 505 dependant and there is no restriction on it. 507 When a local IP prefix is learned in a routing instance on an edge 508 RBridge, the edge RBridge should advertise the IP prefix information 509 for the routing instance so that other edge RBridges will generate 510 IP routing entries. If the ESs in a VN are spread over multiple 511 RBridges, these RBridges should advertise each local connecting end 512 station's IP address in the VN to other RBridges. If the ESs in a VN 513 are only connected to one edge RBridge, that RBridge only needs to 514 advertise the subnet corresponding to the VN to other RBridges. A 515 globally unique tenant ID is also carried in the advertisement to 516 differentiate IP prefixes between different tenants, because the IP 517 address space of different tenants can overlap (see Sections 7.3 and 518 7.4). 520 If a tenant is deleted on an edge Rbridge RB1, RB1 SHOULD re- 521 advertise the local tenant Label, tenant gateway MAC, and related IP 522 prefixes information of the rest tenants to other edge Rbridges. It 523 may take some time for the re-advertisement to reach all other 524 RBridges, so during this period of time there may be transient 525 routes inconsistency among the edge RBridges. If there are traffic 526 in flight during this time, it will be dropped at egress RBridge due 527 to local tenant deletion. In stable state, the traffic to the 528 deleted tenant will be dropped in ingress RBridge. Therefore the 529 transient routes consistency won't cause other issues other than 530 some unnecessary network bandwidth wasting. 532 If there is a new tenant which is created and the original's tenant 533 label is assigned to the new tenant immediately, it may cause a 534 security policy violation for the traffic in flight, because when 535 the egress Rbridge receives traffic from the old tenant, it will 536 forward it in the new tenant's routing instance and deliver it to 537 wrong destination. So tenant Label MUST NOT be re-allocated until a 538 reasonable amount of time, for example twice the IS-IS Holding Time 539 generally in use in the TRILL campus, has passed to allow any 540 traffic in flight to be discarded. 542 When the ARP entry in an edge Rbridge for an ES times out, it will 543 trigger an edge Rbridge LSP advertisement to other edge Rbridges 544 with the corresponding IP routing entry deleted. If the ES is an IP 545 router, the edge Rbridge also notifies other edge Rbridges that they 546 must delete the routing entries corresponding to the IP prefixes 547 accessible through that IP router. During the IP prefix deleting 548 process, if there is traffic in flight, the traffic will be 549 discarded at the egress Rbridge because there is no local IP routing 550 entry to the destination. 552 If an edge Rbridge changes its tenant gateway MAC, it will trigger 553 an edge Rbridge LSP advertisement to other edge Rbridges giving the 554 new gateway MAC to be used as Inner.MacDA for future traffic 555 destined to the edge Rbridge. During the gateway MAC changing 556 process, if there is traffic in flight using the old gateway MAC as 557 Inner.MacDA, the traffic will be discarded or be forwarded as layer 558 2 intra-subnet traffic on the edge Rbridge. If the inter-subnet 559 tenant Label is a unique Label that is different from any access 560 Labels, when the edge Rbridge receives the traffic whose Inner.MacDA 561 is different from local tenant gateway MAC, the traffic will be 562 discarded. If the edge RBridge uses one of the access Labels as 563 inter-subnet tenant Label, the traffic will be forwarded as layer 2 564 intra-subnet traffic unless a special traffic filtering policy is 565 enforced on the edge Rbridge. 567 If there are multiple nicknames owned by an edge RBridge, the edge 568 RBridge also can specify one nickname as the egress nickname for 569 inter-subnet traffic forwarding. A NickFlags APPsub-TLV with the SE- 570 flag set can be used for this purpose. If the edge RBridge doesn't 571 specify a nickname for this purpose, the ingress RBridge can use any 572 one of the nicknames owned by the egress as the egress nickname for 573 inter-subnet traffic forwarding. 575 TRILL E-L1FS FS-LSP [rfc7180bis] APPsub-TLVs are used for IP routing 576 information synchronization in each routing domain among edge 577 RBridges. Based on the synchronized information from other edge 578 RBridges, each edge RBridge generates routing entries in each 579 routing domain for remote IP addresses and subnets. 581 Through this solution, the intra-subnet forwarding function and 582 inter-subnet IP routing functions are integrated and network 583 management and deployment is simplified. 585 5.3. Active-active Access 587 TRILL active-active service provides end stations with flow level 588 load balance and resilience against link failures at the edge of 589 TRILL campuses as described in [RFC7379]. 591 If an ES is connected to two TRILL RBridges, say RB1 and RB2, in 592 active-active mode, RB1 and RB2 each act as a distributed layer 3 593 gateway for the ES. RB1 and RB2 each learn the ES's IP address 594 through the ARP/ND process and then they announce the IP address to 595 the TRILL campus independently. The remote ingress RBridge will 596 generate an IP routing entry corresponding with the IP address with 597 two IP next hops of RB1 and RB2. When the ingress RBridge receives 598 inter-subnet traffic from a local access network, the ingress 599 RBridge selects RB1 or RB2 as the IP next hop based on least cost or, 600 if costs are equal, the local load balancing algorithm. Then the 601 traffic will be transmitted to the selected next hop destination RB1 602 or RB2 through the TRILL campus. 604 5.4. Data Traffic Forwarding Process 606 After a Layer 2 connected ES1 in VLAN-x acquires its gateway's MAC, 607 it can start inter-subnet data traffic transmission to ES2 in VLAN-y. 609 When the edge RBridge attached to ES1 receives inter-subnet traffic 610 from ES1, that RBridge performs Layer 2 header termination, then, 611 using the local VRF corresponding to VLAN-x, it performs the IP 612 routing process in that VRF. 614 If destination ES2 is attached to the same edge RBridge, the traffic 615 will be locally forwarded to ES2 by that RBridge. Compared to the 616 centralized gateway solution, the forwarding path is optimal and a 617 traffic detour through the centralized gateway is avoided. 619 If ES2 is attached to a remote edge RBridge, the remote edge RBridge 620 is IP next hop and the inter-subnet traffic is forwarded to the IP 621 next hop through TRILL encapsulation. If there are multiple equal 622 cost shortest paths between ingress RBridge and egress RBridge, all 623 these paths can be used for inter-subnet traffic forwarding, so load 624 spreading can be achieved for inter-subnet traffic. 626 When the remote RBridge receives the inter-subnet TRILL encapsulated 627 traffic, the RBridge decapsulates the TRILL encapsulation and check 628 the Inner.MacDA. If that MAC address is the local gateway MAC 629 corresponding to the inner Label (VLAN or FGL), the inner Label will 630 be used to find the corresponding local VRF, then the IP routing 631 process in that VRF will be performed, and the traffic will be 632 locally forwarded to the destination ES2. 634 In summary, this solution avoids traffic detours through a central 635 gateway, both inter-subnet and intra-subnet traffic can be forwarded 636 along pair-wise shortest paths, and network bandwidth is conserved. 638 6. Distributed Layer 3 Gateway Process Example 640 This section gives a detailed description of a distributed layer 3 641 gateway solution example. 643 --------- --------- 644 | RB3 | | RB4 | 645 --------- --------- 646 # * # * 647 # ************************** 648 ########################### * 649 # * 650 # * 651 # * 652 --------- --------- 653 | RB1 | | RB2 | 654 --------- --------- 655 | | 656 ----- ----- 657 |ES1| |ES2| 658 ----- ----- 659 Figure 3. Distributed gateway scenario 661 In figure 3, RB1 and RB2 support the distribution gateway function, 662 ES1 connects to RB1, ES2 connects to RB2. ES1 and ES2 belong to 663 Tenant1, but are in different subnets. 665 The IP address, VLAN, and subnet information of ES1 and ES2 are as 666 follows: 668 +----+---------+----------------+---------------+----------+ 669 | ES | Tenant | IP Address | Subnet | VLAN | 670 +----+---------+----------------+---------------+----------+ 671 | ES1| Tenant1 | 192.0.2.2 | 192.0.2.0/32 | 10 | 672 +----+---------+----------------+---------------+----------+ 673 | ES2| Tenant1 |198.51.100.2 |198.51.100.0/32| 20 | 674 +----+---------+----------------+---------------+----------+ 675 Figure 4. ES information 677 The nickname, VRF, tenant Label, tenant gateway MAC for Tenant1 on 678 RB1 and RB2 are as follows: 679 +----+---------+----------+-------+--------------+--------------+ 680 | RB | Nickname| Tenant | VRF | Tenant Label | Gateway MAC | 681 +----+---------+----------+-------+--------------+--------------+ 682 | RB1| nick1 | Tenant1 | VRF1 | 100 | MAC1 | 683 +----+---------+----------+-------+--------------+--------------+ 684 | RB2| nick2 | Tenant1 | VRF2 | 100 | MAC2 | 685 +----+---------+----------+-------+--------------+--------------+ 686 Figure 5. RBridge information 688 6.1. Control plane process 690 RB1 advertises the following local routing information to the TRILL 691 campus: 693 Tenant ID: 1 695 Tenant gateway MAC: MAC1 697 Tenant Label for Tenant1: VLAN 100. 699 IP prefix in Tenant1: 192.0.2.2/32. 701 RB2 announces the following local routing information to TRILL 702 campus: 704 Tenant ID: 1 706 Tenant gateway MAC: MAC2 708 Tenant Label for Tenant1: VLAN 100. 710 IP prefix in Tenant1: 198.51.100.2/32. 712 Relying on the routing information from RB2, remote routing entries 713 on RB1 are generated as follows: 714 +---------------+-------------+--------------+----------------+ 715 | Prefix/Mask | Inner.MacDA | inner VLAN | egress nickname| 716 +---------------+-------------+--------------+----------------+ 717 |198.51.100.2/32| MAC2 | 100 | nick2 | 718 +---------------+-------------+--------------+----------------+ 719 Figure 6. Tenant 1 remote routing table on RB1 721 Similarly, relying on the routing information from RB1, remote 722 routing entries on RB2 are generated as follows: 724 +------------+-------------+-----------+---------------+ 725 |Prefix/Mask | Inner.MacDA |inner VLAN |egress nickname| 726 +------------+-------------+-----------+---------------+ 727 |192.0.2.2/32| MAC1 | 100 | nick1 | 728 +------------+-------------+-----------+---------------+ 729 Figure 7. Tenant 1 remote routing table on RB2 731 6.2. Data Plane Process 733 Assuming ES1 sends unicast inter-subnet traffic to ES2, the traffic 734 forwarding process is as follows: 736 1. ES1 sends unicast inter-subnet traffic to RB1 with RB1's 738 gateway's MAC as the destination MAC and VLAN as VLAN 10. 740 2. Ingress RBridge (RB1) forwarding process: 742 RB1 checks the destination MAC, if the destination MAC equals the 743 local gateway MAC, the gateway function will terminate the Layer 2 744 header and perform L3 routing. 746 RB1 looks up IP routing table information by destination IP and 747 Tenant ID to get IP next hop information, which includes the egress 748 RBridge's gateway MAC (MAC2), tenant Label (VLAN 100) and egress 749 nickname (nick2). Using this information, RB1 will perform inner 750 Ethernet header encapsulation and TRILL encapsulation. RB1 will use 751 MAC2 as the Inner.MacDA, MAC1 (RB1's own gateway MAC) as the 752 Inner.MacSA, VLAN 100 as the Inner.VLAN, nick2 as the egress 753 nickname and nick1 as the ingress nickname. 755 RB1 looks up TRILL forwarding information by egress nickname and 756 sends the traffic to the TRILL next hop as per [RFC6325]. The 757 traffic will be sent to RB3 or RB4 as a result of load balancing. 759 Assuming the traffic is forwarded to RB3, the following occurs: 761 3. Transit RBridge (RB3) forwarding process: 763 RB3 looks up TRILL forwarding information by egress nickname and 764 forwards the traffic to RB2 as per [RFC6325]. 766 4. Egress RBridge forwarding process: 768 As the egress nickname is RB2's own nickname, RB2 performs TRILL 769 decapsulation. Then it checks the Inner.MacDA and, because that MAC 770 is equal to the local gateway MAC, performs inner Ethernet header 771 termination. Using the inner VLAN, RB2 finds the local 772 corresponding VRF and looks up the packets destination IP address 773 in the VRF's IP routing table. The traffic is then be locally 774 forwarded to ES2 with VLAN 20. 776 7. TRILL Protocol Extensions 778 If an edge RBridge RB1 participates in the distributed gateway 779 function, it announces its tenant gateway MAC and tenant Data Label 780 to the TRILL campus through the tenant Label and gateway MAC APPsub- 781 TLV, it should announce its local IPv4 and IPv6 prefixs through the 782 IPv4 Prefix APPsub-TLV and the IPv6 Prefix APPsub-TLV respectively. 783 If RB1 has multiple nicknames, it can announce one nickname for 784 distributed gateway use using Nickname Flags APPsub-TLV with "SE" 785 Flag set to one. 787 The remote ingress RBridges belonging to the same routing domain use 788 this information to generate IP routing entries in that routing 789 domain. These RBridges use the nickname, tenant gateway MAC, and 790 tenant Label of RB1 to perform inter-subnet traffic TRILL 791 encapsulation when they receive inter-subnet traffic from a local ES. 792 The nickname is used as the egress nickname, the tenant gateway MAC 793 is used as the Inner.MacDA, and the tenant Data Label is used as the 794 Inner.Label. The following APPsub-TLVs MUST be included in a TRILL 795 GENINFO TLV in E-L1FS FS-LSPs [rfc7180bis]. 797 7.1. The Tenant Label and Gateway MAC APPsub-TLV 799 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 800 | Type | (2 bytes) 801 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 802 | Length | (2 bytes) 803 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 804 | Tenant ID (4 bytes) | 805 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 806 | Resv1 | Label1 | (2 bytes) 807 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 808 | Resv2 | Label2 | (2 bytes) 809 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+....-+-+-+-+-+-+-+-+-+-+ 810 | Tenant Gateway Mac (6 bytes) | 811 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+....-+-+-+-+-+-+-+-+-+-+ 813 o Type: Set to TENANT-LABEL sub-TLV type (TBD1). Two bytes, because 814 this APPsub-TLV appears in an extended TLV [RFC7356]. 816 o Length: If Label1 field is used to represent a VLAN, the value of 817 the length field is 12. If Label1 and Label2 field are used to 818 represent an FGL, the value of the length field is 14. 820 o Tenant ID: This identifies a global tenant ID. 822 o Resv1: 4 bits that MUST be sent as zero and ignored on receipt. 824 o Label1: If the value of the length field is 12, it identifies a 825 tenant Label corresponding to a VLAN ID. If the value of the length 826 field is 14, it identifies the higher 12 bits of a tenant Label 827 corresponding to a FGL. 829 o Resv2: 4 bits that MUST be sent as zero and ignored on receipt. 830 Only present if the length field is 14. 832 o Label2: This field has the lower 12 bits of tenant Label 833 corresponding to a FGL. Only present if the length field is 14. 835 o Tenant Gateway MAC: This identifies the local gateway MAC 836 corresponding to the tenant ID. The remote ingress RBridges uses 837 the Gateway MAC as Inner.MacDA. The advertising TRILL RBridge uses 838 the gateway MAC to differentiate layer 2 intra-subnet traffic and 839 layer 3 inter-subnet traffic in the egress direction. 841 7.2. "SE" Flag in NickFlags APPsub-TLV 843 The NickFlags APPsub-TLV is specified in [rfc7180bis] where the IN 844 flag is described. The SE Flag is assigned as follows: 845 +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ 846 | Nickname | 847 +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ 848 |IN|SE| RESV | 849 +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ 850 NICKFLAG RECORD 852 o SE. If the SE flag is one, it indicates that the advertising 853 RBridge suggests the nickname SHOULD be used as the Inter-Subnet 854 Egress nickname for inter-subnet traffic forwarding. If flag is 855 zero, that nickname SHOULD NOT be used for that purpose. 857 7.3. The IPv4 Prefix APPsub-TLV 858 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 859 | Type | (2 bytes) 860 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 861 | Total Length | (2 bytes) 862 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 863 | Tenant ID | (4 bytes) 864 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 865 |PrefixLength(1)| (1 byte) 866 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 867 | Prefix (1) | (variable) 868 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 869 | ..... | (1 byte) 870 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 871 | ..... | (variable) 872 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 873 |PrefixLength(N)| (1 byte) 874 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 875 | Prefix (N) | (variable) 876 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 878 o Type: Set to IPV4-PREFIX sub-TLV type (TBD2). Two bytes, because 879 this APPsub-TLV appears in an extended TLV [RFC7356]. 881 o Total Length: This 2-byte unsigned integer indicates the total 882 length of the Tenant ID, the Prefix Length, and the Prefix fields 883 in octets. A value of 0 indicates that no IPv4 prefix is being 884 advertised. 886 o Tenant ID: This identifies a global tenant ID. 888 o Prefix Length: The Prefix Length field indicates the length in bits 889 of the IPv4 address prefix. A length of zero indicates a prefix 890 that matches all IPv4 addresses (with prefix, itself, of zero 891 octets). 893 o Prefix: The Prefix field contains an IPv4 address prefix, followed 894 by enough trailing bits to make the end of the field fall on an 895 octet boundary. Note that the value of the trailing bits is 896 irrelevant. For example, if the Prefix Length is 12, indicating 12 897 bits, then the Prefix is 2 octets and the low order 4 bits of the 898 Prefix are irrelevant. 900 7.4. The IPv6 Prefix APPsub-TLV 901 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 902 | Type | (2 bytes) 903 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 904 | Total Length | (2 bytes) 905 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 906 | Tenant ID | (4 bytes) 907 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 908 |PrefixLength(1)| (1 byte) 909 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 910 | Prefix (1) | (variable) 911 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 912 | ..... | (1 byte) 913 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 914 | ..... | (variable) 915 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 916 |PrefixLength(N)| (1 byte) 917 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 918 | Prefix (N) | (variable) 919 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+...-+-+-+-+-+-+-+-+ 921 o Type: Set to IPV6-PREFIX sub-TLV type (TBD3). Two bytes, because 922 this APPsub-TLV appears in an extended TLV [RFC7356]. 924 o Total Length: This 2-byte unsigned integer indicates the total 925 length of the Tenant ID, the Prefix Length, and the Prefix fields 926 in octets. A value of 0 indicates that no IPv6 prefix is being 927 advertised. 929 o Tenant ID: This identifies a global tenant ID. 931 o Prefix Length: The Prefix Length field indicates the length in bits 932 of the IPv6 address prefix. A length of zero indicates a prefix 933 that matches all IPv6 addresses (with prefix, itself, of zero 934 octets). 936 o Prefix: The Prefix field contains an IPv6 address prefix, followed 937 by enough trailing bits to make the end of the field fall on an 938 octet boundary. Note that the value of the trailing bits is 939 irrelevant. For example, if the Prefix Length is 100, indicating 940 100 bits, then the Prefix is 13 octets and the low order 4 bits of 941 the Prefix are irrelevant. 943 8. Security Considerations 945 Correct configuration of the edge RBridges participating is 946 important to assure that data is not delivered to the wrong tenant, 947 which would violate security constrains. IS-IS security [RFC5310] 948 can be used to secure the information advertised by the edge 949 RBridges in LSPs and FS-LSPs. 951 See Section 5.2 for constraints on re-use of a tenant ID and on 952 tenant gateway MAC change to avoid the mishandling of data in flight. 954 Particularly sensitive data should be encrypted end-to-end, that is, 955 from the source end station to the destination end station. 957 For general TRILL Security Considerations, see [RFC6325]. 959 9. IANA Considerations 961 IANA is requested to assign three APPsub-TLV type numbers from the 962 range less than 255 and update the "TRILL APPsub-TLV Types under IS- 963 IS TLV 251 Application Identifier 1" registry as follows: 965 Type Name References 967 ---- ---------------- ------------ 969 TBD1 TENANT-GWMAC-LABEL [this document] 971 TBD2 IPV4-PREFIX [this document] 973 TBD3 IPV6-PREFIX [this document] 975 IANA is requested to assign a flag bit in the NickFlags APPsub-TLV 976 as described in Section 7.2 and update the ''Nick Flags'' registry, 977 created by [rfc7180bis], as follows: 979 Bit Mnemonic Description Reference 981 ----- -------- ------------------- ----------- 983 1 SE Inter-Subnet Egress [this document] 985 10. Normative References 987 [IS-IS] - ISO/IEC, "Intermediate system to Intermediate system 988 routeing information exchange protocol for use in conjunction with 989 the Protocol for providing the Connectionless-mode Network Service 990 (ISO 8473)", ISO/IEC 10589:2002. 992 [RFC2119] - Bradner, S., "Key words for use in RFCs to Indicate 993 Requirement Levels", BCP 14, RFC 2119, April 1997. 995 [RFC6325] - Perlman, R., Eastlake 3rd, D., Dutt, D., Gai, S., and 996 A.Ghanwani, "Routing Bridges (RBridges): Base Protocol 997 Specification", RFC 6325, July 2011. 999 [RFC7172] - Eastlake, D., M. Zhang, P. Agarwal, R. Perlman, D. Dutt, 1000 "TRILL (Transparent Interconnection of Lots of Links): Fine-Grained 1001 Labeling", RFC7172, May 2014. 1003 [RFC7176] - Eastlake, D., T. Senevirathne, A. Ghanwani, D. Dutt and 1004 A. Banerjee" Transparent Interconnection of Lots of Links (TRILL) Use 1005 of IS-IS", RFC7176, May 2014. 1007 [RFC7356] - Ginsberg, L., Previdi, S., and Y. Yang, "IS-IS Flooding 1008 Scope Link State PDUs (LSPs)", RFC 7356, September 2014, 1009 . 1011 [rfc7180bis] - Eastlake, D., et al, "TRILL: Clarifications, 1012 Corrections, and Updates", draft-ietf-trill-rfc7180bis, work in 1013 progress. 1015 11. Informative References 1017 [RFC826] - Plummer, D., "Ethernet Address Resolution Protocol: Or 1018 Converting Network Protocol Addresses to 48.bit Ethernet Address for 1019 Transmission on Ethernet Hardware", STD 37, RFC 826, November 1982, 1020 . 1022 [RFC4861] - Narten, T., Nordmark, E., Simpson, W., and H. Soliman, 1023 "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861, September 1024 2007, . 1026 [RFC5310] - Bhatia, M., Manral, V., Li, T., Atkinson, R., White, R., 1027 and M. Fanto, "IS-IS Generic Cryptographic Authentication", RFC 5310, 1028 February 2009. 1030 [RFC7379] - Li, Y., Hao, W., Perlman, R., Hudson, J., and H. Zhai, 1031 "Problem Statement and Goals for Active-Active Connection at the 1032 Transparent Interconnection of Lots of Links (TRILL) Edge", RFC 7379, 1033 October 2014, . 1035 Acknowledgments 1037 The authors wish to acknowledge the important contributions of 1038 Donald Eastlake, Gayle Noble, Muhammed Umair, Susan Hares, Guangrui Wu, 1039 Zhenbin Li, Zhibo Hu. 1041 Authors' Addresses 1043 Weiguo Hao 1044 Huawei Technologies 1045 101 Software Avenue, 1046 Nanjing 210012, China 1047 Phone: +86-25-56623144 1048 Email: haoweiguo@huawei.com 1050 Yizhou Li 1051 Huawei Technologies 1052 101 Software Avenue, 1053 Nanjing 210012, China 1055 Phone: +86-25-56625375 1056 Email: liyizhou@huawei.com 1058 Andrew Qu 1059 MediaTec 1060 Email: laodulaodu@gmail.com 1062 Muhammad Durrani 1063 Cisco 1064 Email: mdurrani@cisco.com 1066 Ponkarthick Sivamurugan 1067 Address: IP Infusion, 1068 RMZ Centennial 1069 Mahadevapura Post 1070 Bangalore - 560048 1071 Email: Ponkarthick.sivamurugan@ipinfusion.com 1073 Liang Xia (Frank) 1074 Huawei Technologies 1075 101 Software Avenue, 1076 Nanjing 210012, China 1078 Phone: +86-25-56624539 1079 Email: frank.xialiang@huawei.com