idnits 2.17.1 draft-matsushima-stateless-uplane-vepc-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 5 instances of too long lines in the document, the longest one being 1 character in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 758 has weird spacing: '... type fun...' -- The document date (March 22, 2016) is 2954 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-22) exists of draft-ietf-idr-flow-spec-v6-07 == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-01 ** Obsolete normative reference: RFC 5575 (Obsoleted by RFC 8955) == Outdated reference: A later version (-14) exists of draft-ietf-dmm-fpc-cpdp-01 Summary: 2 errors (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group S. Matsushima 3 Internet-Draft R. Wakikawa 4 Intended status: Informational SoftBank 5 Expires: September 23, 2016 March 22, 2016 7 Stateless user-plane architecture for virtualized EPC (vEPC) 8 draft-matsushima-stateless-uplane-vepc-06 10 Abstract 12 We envision a new mobile architecture for the future Evolved Packet 13 Core (EPC). The new architecture is designed to support the 14 virtualization scheme called NFV (Network Function Virtualization). 15 In our architecture, the user plane of EPC is decoupled from the 16 control-plane and uses routing information to forward packets of 17 mobile nodes. Although the EPC control plane will run on hypervisor, 18 our proposal does not modify the signaling of the EPC control plane. 19 The benefits of our architecture are 1) scalability, 2) flexibility 20 and 3) Manageability. How to run the EPC control plane on NFV is out 21 of our focus in this document. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at http://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on September 23, 2016. 40 Copyright Notice 42 Copyright (c) 2016 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (http://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 58 1.1. The Benefits of NFV . . . . . . . . . . . . . . . . . . . 3 59 2. Motivations and Requirements, - Why IETF? - . . . . . . . . . 4 60 2.1. Motivations . . . . . . . . . . . . . . . . . . . . . . . 4 61 2.2. Requirements . . . . . . . . . . . . . . . . . . . . . . 5 62 3. Stateless user-plane architecture for virtualized EPC . . . 8 63 3.1. Architecture Overview . . . . . . . . . . . . . . . . . . 8 64 3.2. Protocol Overview . . . . . . . . . . . . . . . . . . . . 10 65 3.2.1. Hand-over . . . . . . . . . . . . . . . . . . . . . . 13 66 3.2.2. Detaching UE . . . . . . . . . . . . . . . . . . . . 14 67 3.3. Control-plane awareness of stateless user-plane . . . . . 14 68 3.4. Routing mechanism . . . . . . . . . . . . . . . . . . . . 15 69 3.5. IPv4 Support . . . . . . . . . . . . . . . . . . . . . . 18 70 3.6. Interface between Control-plane and BGP Speaker . . . . . 19 71 4. Operational Considerations . . . . . . . . . . . . . . . . . 20 72 4.1. Scalability and Reliability . . . . . . . . . . . . . . . 20 73 4.2. Backward Compatibility . . . . . . . . . . . . . . . . . 22 74 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 75 6. Security Considerations . . . . . . . . . . . . . . . . . . . 22 76 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 77 7.1. Normative References . . . . . . . . . . . . . . . . . . 22 78 7.2. Informative References . . . . . . . . . . . . . . . . . 23 79 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 81 1. Introduction 83 3GPP introduces Evolved Packet Core (EPC) that is fully IP based 84 mobile system for LTE and -advanced in their Release-8 specification 85 and beyond. Operators are now deploying EPC for LTE services and 86 encounter rapid LTE traffic growth. There are various activities to 87 offload mobile traffic in 3GPP and IETF such as LIPA, SIPTO and DMM. 88 The concept is similar that traffic of OTT (Over The Top) application 89 is offloaded at entity that is closer to the mobile node (ex. eNodeB 90 or closer anchor). 92 Likewise, overload of signaling (control plane) is also increasing 93 day by day. Network operators expect recent innovation and trends of 94 NFV (Network Function Virtualization) to solve this overloaded 95 control plane. NFV is discussed at the ETSI NFV ISG and is 96 introduced in [NFV-WHITEPAPER]. Mobile operator's network is built 97 with variety of proprietary hardware appliances today. If we can get 98 rid of these physical appliances and could shift to a cloud-based 99 service, we will have a lot of benefits explained in the next 100 section. This document assumes that NFV will push networking 101 functions currently run on dedicated hardware onto a cloud network. 102 Expected network functions are Mobility Management Entity (MME), 103 Serving Gateway (SGW) PDN Gateway(PGW), etc. With NFV, EPC can be 104 operated onto servers/hyper-visors. We name it virtualized-EPC 105 (vEPC) in this document. 107 This document uses a lot of 3GPP specific terms. These terms can be 108 found mostly at [RFC6459]. 110 1.1. The Benefits of NFV 112 This section briefly explains the benefits of NFV. The detailed 113 benefits can be found in [NFV-WHITEPAPER]. Although today's eco- 114 system of EPC appliances might be affected, we believe there are 115 various approaches to enhance current eco-system and migrate to new 116 NFV approaches. For example, operators could pay monthly recurring 117 charges for the NFV services and operations to vendors, instead of 118 one-time purchase and a little maintenance cost. 120 o [Flexible Network Operations]: The control functions of EPC are no 121 longer in appliances deployed widely in operator's network and can 122 be run at hypervisor (cloud). It is easier to add and/ or delete 123 functions from the services, because no physical construction is 124 needed. Network operations will be much simpler and easier 125 because complications of today's network are pushed to NFV (i.e. 126 hypervisor). 128 o [Flexible Resource Managements]: The EPC functions can be run on 129 hypervisor and are now less dependent on proprietary hardware. 130 Adding additional resources is easier in hypervisor, while adding 131 or replacing physical appliances require installation, 132 construction, configuration, and even migration plan without 133 service cutoff. A hypervisor can be also shared across various 134 functions such as PGW, SGW and MME. NFV also brings multi-tenancy 135 and allows a single platform for different services and users. 136 The operator can optimize resources and costs to share a NFV 137 platform for multiple customers (ex. MVNO customers) and services 138 (ex. multiple APNs). 140 o [Faster Speed of Time to Market]: When an operator wants a new 141 function to its network and services, the operator needs to 142 negotiate appliance vendors to implement the new functions or to 143 find alternative equipment supporting the new function. It takes 144 a longer time to convince the vendors, or to replace existing 145 hardware. However, if functions can be implemented as a software, 146 it is much faster to implement the functions on NFV. Even the 147 operator may implement them and try the new functions by 148 themselves. Field trial is also getting easier because of no 149 physical installation or replacement. You may turn on a new 150 function in NFV and observe how the new function behaves in your 151 network. NFV can save preparation time and tuning time of the new 152 function. 154 o [Cost Optimization]: Last but not least, Cost is the most 155 important motivation for operators to realize NFV. Operators can 156 remove many of proprietary appliances from its network and replace 157 them with industry standard servers, switches and routers. In 158 addition, it is easy to scale up and down operator's services so 159 that resources can be always tuned to the size of services. In 160 addition, operational costs led by any physical hardware such as 161 power supply, maintenance, installation, construction and 162 replacement can be minimized or even removed. The network design 163 can be simpler, because complicated functions could be handled by 164 NFV. That simple operation may enable automatic configurations 165 and prevent unnecessary trouble-shooting. As a result, CAPEX and 166 OPEX can be always optimized and lowered. 168 2. Motivations and Requirements, - Why IETF? - 170 2.1. Motivations 172 What is a role of IETF to realize vEPC in the future? IETF is not 173 the right place to discuss, for instance, how to run MME on 174 hypervisor. An important IETF activity must be to decouple the 175 control- and user- planes of mobility protocols used in EPC.The 176 motivation of decoupling the user and control plane is discussed in 177 [I-D.wakikawa-req-mobile-cp-separation]. In doing so, NFV-enabled 178 solutions can be easily designed and implemented with 179 interoperability across multiple vendors and platforms. Otherwise, 180 NFV solutions can be easily fragmented due to many proprietary 181 solutions for the protocol separations. As stated in 182 [NFV-WHITEPAPER], interoperability is highly important. 184 In the past, IETF has developed tunnel based mechanisms for mobile 185 nodes such as Mobile IPv6 [RFC6275][RFC5555], Proxy Mobile IPv6 186 [RFC5213][RFC5844] and NEMO [RFC3963]. Similarly, 3GPP has developed 187 tunnel protocols called GPRS Tunneling Protocol (GTP). These tunnel- 188 based protocols establish a data path for a mobile node between the 189 mobile node and an anchor point (s). There is a case where an access 190 router terminates a tunnel instead of a mobile node (ex. Proxy 191 Mobile IP). In 3GPP, a tunnel is established between SGW and PGW per 192 a mobile node by either Proxy Mobile IPv6 or GTP. The control and 193 the user planes of these mobility protocols are tightly related and 194 cannot be decoupled. The signaling like Binding Update and user's 195 packets are routed along a same path in EPC. It might be necessary 196 to extend these mobility protocols for the user- and control- planes 197 separation. The protocol separation of Mobile IP is discussed in 198 [I-D.yokota-dmm-scenario]. 200 Alternatively, if vEPC was realized, we should have an opportunity to 201 re-visit the basic architecture of mobility system. Instead of 202 tunneling packets on today's EPC, why can't we just route packets to 203 a mobile node? Since a role of the user plane is "routing", BGP and 204 other routing protocols could be used to forward UE's traffic. This 205 document introduces a BGP-based solution. Software Defined 206 Networking (SDN) can be an alternative solution. Open Flow and other 207 relevant protocols can setup the forward path dynamically according 208 to UE's states available in the control plane. 210 We have to remember that there is a good reason of adapting tunneling 211 in Mobile IP based solutions, that is global mobility and signaling. 212 A mobile node should be able to move anywhere on the Internet and be 213 reachable from anyone on the Internet. There were routing based 214 global mobility solutions like Boeing global mobility [Boeing-BGP] 215 and WINMO [RFC6301]. In these proposals, BGP was used to propagate 216 forwarding information of mobile nodes to the Internet. Whenever a 217 mobile node changes its point of attachment, the route must be 218 updated. Due to scalability and stability issues of the Internet, 219 this solution was not recommended by IETF [Boeing-BGP]. However, as 220 Boeing showed, it is doable to support global mobility by using BGP 221 routing update. If scalability is not your concern, a routing based 222 approach becomes a candidate of the mobility solution. 224 While global mobility is important, the "reality" is that your cell 225 phones (i.e. UE/mobile node) are moving just within an operator's 226 network and fully controlled in your local EPC. If mobility is 227 limited within an operator, we believe a routing based approach is 228 feasible and practical for today's mobile system. Instead of 229 dedicated proprietary equipment like SGW and PGW to manage a tunnel 230 path for a mobile node, multiple industry standard routers and 231 switches are configured in the user plane. These switches and 232 routers receive mobile nodes' forwarding information from the control 233 plane of vEPC by routing update. 235 2.2. Requirements 237 Requirements of our stateless user plane for vEPC are followings. 239 NFV Support 240 The future EPC architecture must support NFV capability. The 241 control plane of EPC operated on NFV framework is named 242 "virtualized EPC (vEPC)" in this document. The control plane 243 of vEPC should keep backward compatibility with the today's 244 EPC's control plane. It means this document doesn't modify 245 the control plane at all. It only assumes software-based 246 MME, SGW, and PGW run on hypervisor. 248 Separation of Control- and User- Planes 249 Due to tight relationship of the control- and user- planes in 250 today's EPC, resource increase is always provisioned to both 251 planes at once. It prevents flexible resource arrangement 252 and introduces high capital investment and over-provisioned 253 resources to one of planes. If NFV is deployed, it is 254 expected that computing resources can be independently 255 allocated to the control planes of the vEPC in a flexible 256 manner. 258 _+---+_ _ _+---+_ _ _ 259 EPC / | S | | P | / 260 Control-plane / | G | | G | / 261 /_ _| W |_ _ _| W |__/ 262 +---+ +---+ | | | | 263 | | | e | | | | | 264 | U | | N | _| |_ _ _| |_ _ _ 265 EPC | E | | B | / | | | | / 266 User-plane +---+ +---+ / +---+ +---+ / 267 /_ _ _Existing EPC_ _/ 269 _+---+_ _ _+---+_ _ _ 270 vEPC / | S | | P | / 271 Control-plane / | G | | G | / 272 /_ _| W |_ _ _| W |__/ 273 | | | | 274 +---+ +---+ +---+ +---+ 275 | | | e | +----------+ 276 | U | | N | _|IP Routing|_ _ _ 277 Stateless | E | | B | / | Network | / 278 User-plane +---+ +---+ / +----------+ / 279 /_ _ _ _ _ _ _ _ _ / 281 NFV enabled EPC architecture 283 Figure 1 285 Figure 1 shows a possibility that the entities of EPC 286 Control- plane are virtualized in generic cloud environment, 287 however user packets won't go through those virtualized EPC 288 nodes. Decoupling User-plane from the Control-plane entities 289 will be made virtualized Control-plane nodes relax hyper- 290 visor data- path capacity requirements. On the other hand, 291 decoupled User-plane into IP routing network will be agnostic 292 from sessions and bearers states, of which are generated and 293 maintained in the Control-plane. In terms of IP routing, 294 forwarding packets through the networks is based on the 295 destination address of the packets evaluated with network 296 reachable information in the routing table that accommodated 297 in the routing nodes. To forward EPC User-plane packets 298 correctly, those states must be indicated by network 299 reachable information. 301 Flat Design for Distributed Operation 302 Today's 3GPP architecture introduces PDN gateway (PGW) as a 303 gateway to external networks like the Internet. PGW manages 304 all traffic from and to UEs and could be a bottleneck and 305 single point of failure of network connectivity. In 306 addition, due to recent rapid traffic increase, it is 307 important to perform traffic engineering and to offload 308 traffic to multiple locations (ex. SGW, PGW, eNodeB). For 309 enhancements of traffic engineering capability, more flat 310 design with multiple gateways is expected so that traffic can 311 be distributed to all these gateways. There were proposals 312 how to enable flat design to (Proxy) Mobile IP such as 313 [I-D.wakikawa-mext-haha-interop2008] in IETF. Distributed 314 Mobility Management (DMM) Working Group has also discussed 315 how to extend Mobile IP-based solutions to support traffic 316 distribution in an optimal way by removing centrally deployed 317 anchors that is like a Home Agent. 319 Stateless in User Plane 320 Ultimate goal of vEPC is to remove all mobility specific 321 states from the forwarding nodes in the user-plane of vEPC. 322 If we succeed in this, industry standard routers and switches 323 can be used to forward mobile nodes traffic in the user plane 324 of vEPC. A mobile node's specific states are kept in both an 325 IP header of the mobile node's packets and a routing entry of 326 the mobile node. The detail is described in Section 3.2 328 3. Stateless user-plane architecture for virtualized EPC 330 This section explains our solution that is the stateless user-plane 331 architecture for vEPC. This solution is basically a combination of 332 existing protocols defined in IETF. A minor extension might be 333 needed but it should be easily addressed in IETF. We first introduce 334 our architecture and then protocol overview. 336 3.1. Architecture Overview 338 Figure 2 shows the user plane of the current EPC architecture. A 339 tunnel is established between SGW and PGW by either Proxy Mobile IP 340 or GTP. PGW is an anchor point of UE for incoming packets. All the 341 packet destined to UE is routed first to PGW. The UE's packets are 342 intercepted by PGW and tunneled to SGW. SGW then forwards the packet 343 to UE via access points (i.e. eNodeB) over Radio Area Network (RAN). 345 .--. 346 _( `)_ 347 _( `) 348 +---+ +---+ ( EPC-NW ) +---+ 349 [UE]~~~~|RAN|====|SGW|===================|PGW| -> the Internet 350 +---+ +---+ `--(______)---' +---+ 351 <--------Tunneling------> 353 ~~~ Wireless Link 354 === GTP Tunnel (PMIP tunnel) 355 --- IP link 357 User plane of the current EPC 359 Figure 2 361 Figure 3 is our proposed user plane of vEPC. The control plane is 362 not shown in this figure. 364 [EPC-E anycast address] 365 | .--. 366 | _( `)_ 367 | +---+ _( `) 368 +---+ \ |EPC| ( IPv6 Core ) +---+ 369 [UE]~~~~|RAN|====| -E|-( ` NW )-|RTR| --> the Internet 370 +---+ +---+ ( ` ) ) +---+ 371 (eNodeB) `--(_____)--' 372 <--------Routing--------> 374 User plane of vEPC 376 Figure 3 378 We introduce two new entities such as 380 EPC Edge Router (EPC-E) 381 EPC-E is located at the same place of today's SGW and terminates 382 GTP tunnel established with eNodeB (RAN). EPC-E supports the user 383 plane functions of SGW and PGW. EPC-E is configured an anycast 384 address to the network interface facing to eNodeB. The eNodeB 385 establishes a GTP tunnel per UE with this anycast address. Thanks 386 for anycast address, UE's traffic forwarded by eNodeB is always 387 routed to the closest EPC-E of UE. EPC-E is a router and 388 maintains routing information of every UE that is notified by the 389 control plane. Detail of routing mechanism can be found in 390 Section 3.4. 392 Router (RTR) 393 It is a regular IP router. The control plane of vEPC distributes 394 routing information of every UE by a routing protocol like BGP. 395 Therefore any additional protocols other than routing protocols 396 are not needed for RTR. Multiple RTRs can be configured anywhere 397 in the user plane of vEPC. RTRs announce UE's routing information 398 to the external network (ex. The Internet). 400 As you see in Figure 3, we omit a tunneling mechanism originally 401 established between SGW and PGW for routing UE's packets in the user 402 plane. By removing this tunnel, UE's packets are forwarded to and 403 from the Internet according to routing tables on routers in the core 404 network. Note that, although we remove the tunnel for UE's traffic 405 in the user plane, the control-plane signaling stays same in the 406 control plane. If Proxy Mobile IP is used for this tunnel, Proxy 407 Binding Update and Acknowledgment are exchanged between PGW and SGW 408 that are managed by NFV on servers/hyper-visor. Instead of a tunnel 409 setup, states created by Proxy Mobile IP are distributed to all 410 routing entities (EPC-E and RTR) by a routing protocol. From the 411 user plane point of view, these states are just seen as routing 412 entries. EPC-E and RTR are not involved in any signaling of the 413 control plane. The control plane just injects routing information to 414 EPC-E and RTR to setup routing paths to and from UEs. 416 Although this architecture just uses IPv6 core network, it supports 417 both IPv4 and IPv6 packets. The detailed operation of IPv4 support 418 will be discussed in Section 3.5. 420 3.2. Protocol Overview 422 This section gives an example of protocols used for vEPC. Figure 4 423 is the procedure of the PDN connection setup in vEPC. This figure is 424 copied from the section 3 of [RFC6459]. All the steps from (1) to 425 (13) are same as the original except for NFV-based MME, SGW, PGW, 426 HSS, and AAA. 428 The vEPC introduces two new steps, (14) and (15), to setup paths in 429 the user-plane after finishing all the signaling on the control- 430 plane. (16) and (17) are the steps to assign IP address to the mobile 431 node. 433 vEPC(MME/SGW 434 UE eNodeB PGW/HSS/AAA) EPC-E RTR 435 | | | | | 436 |---------->|(1) | | | 437 | |---------->|(1) MME | | 438 |/---------------------\| | | 439 | Authentication |(2) AAA | | 440 |\---------------------/| | | 441 | | |--+SGW/PGW | | 442 | | | |(3)(4) |(4)SGW's TEID is configured 443 | | |<-+ | and notified to MME 444 | |<----------|(5) MME | | 445 |/---------\| | | | 446 | RB setup |(6) | | | 447 |\---------/| | | | 448 | |---------->|(7) MME | | 449 |---------->|(8) | | | 450 | |---------->|(9) MME |(9)eNodeB TEID is configured 451 | | | | and sent to MME 452 |= = = = = = = Uplink Data = = = = =>= = = = ==>|(10) 453 | The uplink is not yet built here | 454 | | | | | 455 | | |--+SGW/MME | | 456 | | | |(11,12) | | 457 | | |<-+ | | 458 |<= = = = = = Downlink Data = = = = <= = = = = =|(13) 459 | The downlink is not yet built here | 460 | | | | | 461 | | |---------->|(14) Route Update 462 | | | [Dst: UE-prefix, 463 | | | NxtHop: S1-U addr and TEID of enodeB 464 | | | | | 465 | | | |---------->|(15)Route update 466 | | | |[Dst: UE-prefix, 467 | | | | NxtHop: EPC-E address] 468 | | | | | 469 |---------RS or DHCP Request------->|(16) | 470 |<--------RA or DHCP Reply----------|(17) | 472 Extended PDN Connection Setup Procedure (copied Figure 8 of RFC6459) 474 Figure 4 476 In (14), vEPC advertises a routing information of UE to EPC-Es 477 immediately right after the control-plane signaling completion. The 478 routing information contain UE's prefix as destination, remote 479 endpoint of GTP-U tunnel as next-hop which is S1-U addresses and 480 TEIDs of serving eNodeB/EPC-E, and also QoS class applied to the UE. 482 In this document, the advertising entity is a BGP speaker so that the 483 BGP speaker is required to indicate those in BGP message. To achieve 484 that, the BGP speaker and EPC-E should be capable of (1) BGP Tunnel 485 Encaps Attribute [I-D.ietf-idr-tunnel-encaps] which specifies the 486 form to encode GTP-U endpoints, and (2) Dissemination of Flow 487 Specification Rule [RFC5575] with IPv6 amendment 488 [I-D.ietf-idr-flow-spec-v6] to indicate applied QoS class. 490 It is noted that the control-plane needs to expose user-plane 491 information of UEs to BGP speaker. The means of how the control- 492 plane and the BGP speaker deal with that is discussed in Section 3.6. 494 The EPC-E has a peering with the BGP speaker directly. It is thus 495 expected that there is no additional propagation delay of traversing 496 multiple BGP speakers between EPC-E and vEPC. Adding that kind of 497 surplus delay affects user-plane to be interrupted so that it should 498 be avoided as much as possible for user experience. 500 In step (15), the EPC-E advertises routes to upstream routers such as 501 the RTR. For scalable routing operation, UE's prefixes should be 502 aggregated into more shorter length prefixes. Due to that reason, 503 the EPC-E generates routing information and advertised it to the RTR 504 that includes aggregated prefix instead of UE's prefixes and EPC-E 505 address as the next-hop. 507 UE requests an IPv6 prefix for its address assignment in the step 508 (16). In our architecture, an IPv6 prefix is still assigned by vEPC 509 in the control plane, as PDN-GW does in the legacy EPC. However, 510 EPC-E is responsible to deliver the IPv6 prefix to UE by DHCP or 511 Stateless address autoconfiguration (SLAAC). 513 We now explain how EPC-E can know the prefix assigned to UE from vEPC 514 for address configuration steps (16 and 17). When (1) to (15) are 515 completed, vEPC has already advertised the UE's prefix as route 516 information to all the EPC-E. Therefore, when EPC-E receives a 517 packet of either Router Solicitation or DHCPv6 request message, it 518 just looks up the remote next-hop field of its routing information 519 base (RIB) with the source IP address and the TEID of the received 520 packet. A route entry matched for this search is the prefix 521 delegated to the requesting UE. Therefore, EPC-E simply uses the 522 prefix of the route entry as an assigned UE's prefix. 524 In (17), EPC-E returns the found prefix to UE by either Router 525 Advertisement or DHCPv6 reply message. UE now creates an address(es) 526 from the received prefix. It is important to highlight that UE can 527 obtain the same prefix information from any EPC-E all the time 528 because the same UE's route information is available on all the EPC- 529 E. 531 It would be convenient to use automatic UE's prefix creation rule or 532 algorithm for vEPC. There are various mechanisms to create UE's 533 prefix. As an example, Stateless IPv6 Prefix Delegation 534 [I-D.savolainen-stateless-pd] is introduced as an algorithm to create 535 UE's prefix in vEPC below. It important to mention that our 536 architecture of the stateless user plane does not rely on any 537 particular prefix creation mechanisms like 538 [I-D.savolainen-stateless-pd] and can be run with any of them. 540 In the case of an UE's prefix length is equal, or shorter than /64, 541 the generated prefix is consisted as shown in Figure 5. Each PDN is 542 assumed to have single or several prefixes (named PDN prefix) used to 543 generate UE's address. Followed by the PDN prefix, there is TEID 544 field assigned for a UE's session on S1-U interface of vEPC. TEID is 545 32 bits identifier in GTP header to distinguish each bearer. The 546 remaining bits are filled by subnet ID. 548 | n bits | o bits (=< 32)|64-n-o bits| 549 +---------------------------+---------------+-----------+ 550 | PDN Prefix | TEID | subnet ID | 551 +---------------------------+---------------+-----------+ 552 <---------------------------64bits----------------------> 554 Stateless-pd Prefix 556 Figure 5 558 3.2.1. Hand-over 560 When tunnel endpoint is updated by UE hand-over between eNodeBs, vEPC 561 must refresh the route of UE with the updated tunnel endpoint as new 562 remote next-hop. 564 Figure 6 shows vEPC that advertising updated route in (8) when UE 565 hand-over from source eNodeB to target eNodeB on simplified hand-over 566 procedure. The updated route that points to target eNodeB's S1-U 567 address and TEID as the next-hop should be immediately advertised to 568 all the EPC-Es right after the procedures (1) to (7) completed. 570 It is noted that RTR or any upstream routers of EPC-Es do not require 571 routing update for each of UE hand-over event. EPC-E is required to 572 just advertise once aggregate route during at least an UE route exist 573 so that EPC-E does not advertise hand-over UE route in Figure 6. 574 Operators require that their core network must be kept its routing 575 stable. This architecture prevents routing fluctuation in the 576 network that helps to fulfill that requirement consequently. 578 Source Target 579 UE eNodeB eNodeB vEPC EPC-E RTR 580 | | | | | | 581 | | | | | | 582 | |---Handover Required---->|(1) | | 583 | | |<--Handover---|(2) | | 584 | | | Request | | | 585 | | | | | | 586 | | |--Handover--->|(3) | | 587 | | | Acknoledge | | | 588 | | | | | | 589 |<---------|<----Handover Command----|(4) | | 590 |--Handover Confirm-->|(5) | | | 591 |= = = = = = = = = Uplink Data = = = = = = = = =>= = ==>|(6) 592 | | | | | | 593 | | |---Handover-->|(7) | | 594 | | | Notify | | | 595 | | | |-------->|(8) | 596 | | | | Route Update | 597 | | | |[Dst : UE-prefix, 598 | | | | NxtHop: Target eNode's 599 | | | | S1-U addr/TEID] 600 | | | | | | 601 |<== = = = = = = = Downlink Data = = = = = = =<== = = =|(9) 602 | | | | | | 604 Simplified Hand-over Procedure 606 Figure 6 608 3.2.2. Detaching UE 610 In the case of UE detachment, vEPC also advertises route update that 611 includes detached UE prefix as withdrawn route to delete the route of 612 the detached UE from EPC-Es. 614 3.3. Control-plane awareness of stateless user-plane 616 Nodes in the control-plane in vEPC must be aware that the anycast 617 address assigned to EPC-E is a S1-U address of vEPC. The vEPC must 618 use the anycast address in signaling between vEPC and RAN. By doing 619 this, packets from RAN are correctly forwarded to an appropriate EPC- 620 E. Due to anycast nature, it means there is no hand-off procedure 621 between SGWs because all eNodeB in the RAN send packets to the same 622 anycast address. 624 When an operator needs to increase virtualized instances to cope with 625 just signaling overload, the operator should use the existing S1-U 626 address (i.e. EPC-E anycast address) for the new instances. If the 627 operator would increase the capacity of the user plane, it can add 628 additional EPC-Es in the core network. The operator can group the 629 new EPC-Es as a set and increase scalability and performance of the 630 user plane. In this case, the operator uses a new anycast address to 631 the new set of EPC-E. We will discuss operational consideration in 632 detail in Section 4. 634 3.4. Routing mechanism 636 Figure 7 shows a packet forwarding mechanism of our stateless user 637 plane. As an example, there are four eNodeB (illustrated as eNB-x) , 638 three EPC-Edge routers(shown as EPC-Ex) and two routers (RTRx) in 639 Figure 7. UE is first connected to eNB-C and then moves to eNB-D. 640 The UE at the new location is illustrated as UE'. Routing entry for 641 UE is also illustrated at the right side in Figure 7. 643 EPC-E has two interfaces facing either RAN or CORE networks. An 644 anycast address (shown as X) is configured to the interface facing 645 RAN of all EPC-E. EPC-E assigns an individual IPv6 address to 646 another interface (illustrated "a" to "d" in the figure). It is 647 important to mention that the anycast address X can be treated as the 648 SGW's S1-U address. 650 Since RTRs are a gateway to the Internet, they advertise routes of an 651 operator's prefix to the Internet. After one of RTR receives a 652 packet of UE from the Internet, it needs to routing it to UE in the 653 user plane. RTR has a simple routing entry for PDN prefix whose next 654 hop points to the EPC-E. One of RTR (let's say RTR2 in this case) 655 looks up a routing table with UE's address and matched it with a 656 routing entry of PDN prefix. Since multiple EPC-Es advertise a route 657 for the same PDN prefix, RTR2 should forward the packet to one of 658 EPC-E according to the routing entry. This routing is known as hot- 659 potato routing. In this example, the RTR2 uses EPC-E2-b as a nexthop 660 of PDN prefix. 662 When the UE's packet is arrived at EPC-E2, EPC-E2 needs to forwards 663 them to UE via eNodeB to which UE is connecting by using GTP tunnel. 664 For this operation, EPC-E2 has a routing entry that destination is 665 UE's prefix and that next hop points to GTP tunnel between eNB-C and 666 the EPC-Es. In order to identify the GTP tunnel for UE, EPC-E needs 667 S1-U address and Tunnel Endpoint ID (TEID) of eNB-C that is eNB-C-3 668 in Figure 7. The eNB-C TEID for UE is illustrated as TEID[eNB-C]. 669 The SGW assigned TEID is utilized to generate the UE's prefix as we 670 explained in Section 3.2. These TEID are assigned per UE. The TEID 671 and S1-U address of eNodeB are retrieved from the next hop field of 672 the routing entry of the mobile node. By using the GTP information, 673 every EPC-E can now forward the UE's packets to right eNodeB. 675 Routing outgoing packets from UE is much simpler. The packets from 676 UE are always routed to the closest EPC-E to UE because of anycast 677 routing. In Figure 7, when UE sends a packet to a destination, the 678 packet is reached to eNB-C and tunneled to EPC-E's anycast address. 679 The GTP-tunneled packet is routed to the closest EPC-E that is EPC-E2 680 in this case. The packet is decapsulated by EPC-E2 and then 681 forwarded to one of RTR according to the routing table. Since the 682 decapsulated packet is regular IPv6 packet, no extra control other 683 than routing is necessary. 685 When UE moves to a new location (UE'), it updates its location on the 686 control plane. After signaling completion for location update, vEPC 687 needs to update the UE's routing entry of all EPC-E so that vEPC 688 advertises updated route with new location to all EPC-Es by a routing 689 protocol. The routing entry should be updated with the new eNodeB's 690 address that is eNB-D-4. During handover, there might be some 691 traffic arriving to the older eNodeB (eNB-C). These packets can be 692 re-routed to the new eNodeB (eNB-D) via X2-U interface in RAN. 694 The UE's address isn't changed when UE changes its attachment. In 695 our scenario, SGW run on hypervisor and is independent from network 696 topology. Therefore, logically we don't have handover across 697 different SGWs. UE can stay connected with the same SGW all the time 698 and can keep using the same TEID after handover. Thus, UE's address 699 is unchanged even after handover. 701 +--------------------+ 702 | Correspondent Node | 703 +--------------------+ 704 | 705 .--. [Route on the Internet] 706 _(. `) Destination: Operator's Prefix 707 _( `)_ NextHop: RTR-1/2 708 ( Internet `) 709 ( ` . ) ) 710 `--(_______)---' 711 / \ [Route at RTR] 712 +----+ +----+ Destination: PDN Prefix 713 |RTR1| |RTR2| NextHop: EPC-E2-b 714 +----+ +----+ 715 \ .--. / 716 _( IP `. 717 ( CORE NW ) 718 ___( ` . ) )__ 719 / `--(___.--' \ 720 / | \ 721 |a |b |c [Route at EPC-E] 722 +---+--+ +---+--+ +---+--+ Destination: UE's Prefix *1 723 |EPC-E1| |EPC-E2| |EPC-E3| NextHop: GTP tunnel (eNB-C) *2 724 +---+--+ +---+--+ +---+--+ 725 |X |X |X 726 \ .--. / 727 \_____( RAN `.____/ 728 ( ACCESS ) 729 ____( ` NW ) )_______ 730 / `--(___.--' \ 731 / | | \ 732 |1 |2 |3 |4 733 +--+--+ +--+--+ +--+--+ +--+--+ 734 |eNB-A| |eNB-B| |eNB-C| |eNB-D| 735 +--+--+ +--+--+ +--+--+ +--+--+ 736 : : : : 737 UE --> UE' 738 *1 TEID used at EPC-E for the UE is included in this UE's prefix. 739 see Figure 4. 740 *2 GTP tunnel state is stored in the next hop field. The state 741 information is the combination of eNB-C S1 address that is 742 eNB-C-3 and TEID(eNB-C) assigned for the UE. 744 Routing Mechanism Overview 746 Figure 7 748 3.5. IPv4 Support 750 Recent IPv6 transition mechanisms enable IPv6-only network to forward 751 IPv4 packet with encapsulation or translation techniques. By using 752 one of mechanisms, we can use IPv6 for our stateless user-plane 753 network for transporting both IPv4 and IPv6 packets. Figure 8 shows 754 available solutions of IPv4 support for each bearer type to deal with 755 that requirement. 757 Bearer UE EPC-E Gateway 758 type function function function 759 -------------------------------------------- 760 IPv4 - B4 AFTR 761 IPv4 - CLAT PLAT 762 IPv6 MAP-CE - MAP-BR 763 IPv6 B4 - AFTR 764 IPv6 CLAT - PLAT 766 Solutions and functions for IPv4 support 768 Figure 8 770 In the case of a UE only support IPv4 bearer, B4 function of DS-Lite 771 [RFC6333] or CLAT function of 464XLAT [RFC6877] may be implemented in 772 a EPC-E. Both functions are stateless therefore EPC-E isn't required 773 to maintain any tunneling or translation state. 775 Figure 9 shows how to support IPv4 on IPv6 core network in our vEPC. 776 Instead of using RTR as a gateway to the Internet, DS-LITE AFTR or 777 464XLAT PLAT is installed as a gateway to the IPv4 Internet. 779 .--. 780 EPC-E _( `)_ Gateway 781 +----+ _( `) +------+ 782 | B4 | ( IPv6 Core ) | AFTR | 783 [UE]==IPv4-only==| or |-( ` NW )-| or | --> Internet 784 bearer |CLAT| ( ` ) ) | PLAT | (IPv4) 785 +----+ `--(_____)--' +------+ 786 <-----IPv4 over IPv6-----> 787 or 788 v4v6 translation 790 IPv4 User plane of vEPC 792 Figure 9 794 If UE supports IPv6 capable bearer, IPv6 transition function may be 795 implemented in the UE such as MAP-CE [I-D.ietf-softwire-map], B4 or 796 CLAT. That means an EPC-E receives IPv6 packets from UE in this case 797 so that the EPC-E does not need to be involved in the part of IPv4 798 support functions. 800 3.6. Interface between Control-plane and BGP Speaker 802 In Section 3.2 described, mobility control-plane and BGP speakers 803 within a vEPC need an interface to export user-plane information from 804 the control-plane to the BGP speakers. Perhaps many solutions would 805 be developed proprietarily. However, adopting standardized interface 806 will be much appropriate. 808 Forwarding Policy Configuration Protocol (FPCP) 809 [I-D.ietf-dmm-fpc-cpdp] has been standardized in IETF for that 810 purpose. That provices client function to the mobility control-plane 811 to export user-plane information, and agent function which enables 812 the BGP speakers to receive the user-plane information when it is 813 implemented into them. 815 User-plane information contains UE's IP prefix, GTP-U tunnel 816 endpoints of serving eNodeB/EPC-E and applied QoS class. When those 817 information come into the BGP speaker, the agent renders it into BGP 818 attributes which are UE's IP prefix, GTP-U tunnel endpoints and QoS 819 class are indicated in NLRI, [I-D.ietf-idr-tunnel-encaps] and 820 [RFC5575] with [I-D.ietf-idr-flow-spec-v6] respectively. 822 The BGP speaker generates BGP UPDATE messages based on that and then 823 advertises it to EPC-E routers. Figure 10 depicts FPCP enabled vEPC 824 in which mobility control-plane and BGP speaker are interfaced 825 through FPCP client and agent functions. 827 +----------------------------------------------------+ 828 | vEPC | 829 | +----------------------------------+ | 830 | | Mobility Control-Plane | | 831 | | | | 832 | | +-----+ +-----+ +-----+ +------+ | | 833 | | | MME | | SGW | | PGW | | PCRF | | | 834 | | +-----+-+-----+-+-----+-+------+ | | 835 | | | Interface to BGP Speaker | | | 836 | | | (Client Function) | | | 837 | +-+--------------^---------------+-+ | 838 | | | 839 | | FPC Protocol | 840 | | | 841 | +-+--------------v----------------+-+ | 842 | | | Agent Function | | | 843 | | +-------------------------------+ | | 844 | | | | 845 | | BGP Speaker | | 846 | +----------------^------------------+ | 847 +-------------------------|--------------------------+ 848 | 849 | BGP Peering 850 | w/ Tunnel encaps & Flow-spec 851 | attributes 852 | 853 +--------------v----------------+ 854 | EPC-E Routers | 855 +-------------------------------+ 857 FPCP enabled vEPC 859 Figure 10 861 4. Operational Considerations 863 4.1. Scalability and Reliability 865 Virtualization allows vEPC to be elastic for steep demand of requests 866 to create and update for sessions. In our architecture, that makes 867 routing update fluctuation from vEPC to EPC-E. This is the reason 868 why we select BGP as a protocol between vEPC and EPC-E. BGP is 869 scalable and stable routing protocol today. 871 BGP is an incremental update protocol so that once BGP peer 872 established, millions of routes can be easily updated in stable 873 manners. Operators can appropriately design BGP peering between vEPC 874 and ECP-E to secure convergence time within appropriate period. 876 Granularity of the peering should be aware EPC-E capacity because it 877 is assumed that EPC-E has upper limit of routing entries. BGP 878 peering design should makes sure that total number of routes does not 879 exceed EPC-E capacity. 881 During the network planning, operators must understand EPC-E's 882 capacity such as # of routes, bandwidth, etc. An example of 883 estimation, if a EPC-E has 1Gbps throughput and each UE's bandwidth 884 consumption is 10Kbps in average, the EPC-E should have 100K routes 885 capacity. 887 This is an operational approach to minimize the risk of routing 888 update fluctuation. If it is hard to support all the UEs by a EPC-E 889 in an operators network, another EPC-E can be introduced and 890 configured as a set of EPC-Es. The UEs are distributed and handled 891 by the EPC-Es within the set. We don't need to support millions of 892 UEs by a single EPC-E. 894 EPC-E set is also useful to have EPC-E redundancy for reliable 895 operation. The nature of BGP makes easy to replicate UE routes to 896 multiple EPC-Es within a EPC-E set. In that EPC-E set, when an EPC-E 897 fall down to a failure, another EPC-E come out with same UE routes 898 that the fall-down EPC had and immediately re-converge to core 899 routing. That helps user-plane to minimize disruption during EPC-E 900 failure recovery. 902 These are another advantage of using routing mechanism in the user 903 plane. We already explain how to handle multiple EPC-Es and EPC-E 904 sets in our scheme in Section 3.3. 906 The notion of multiple EPC-E sets is easily fitted into our today's' 907 network. The operator's network is often separated into several 908 regional network for geographical scalability. Therefore, the 909 operator can assign different EPC-E set to different region for 910 better scalability. 912 In that network, when an UE hands over between two regions, the 913 session of the UE might be disconnected if the serving EPC-E doesn't 914 have reachability for those region access networks. For example, in 915 the case of regional access networks have duplicated IPv4 private 916 address space. To enable inter-region hand-over, it is recommended 917 that all of the access network, such as RAN, are IPv6 networks and 918 reachable each other. 920 In addition, routers and EPC-E in the IPv6 core network are required 921 to process just "route", they naturally aggregate those routing 922 entries. It helps limiting the total number of routing entries in 923 our core network. 925 4.2. Backward Compatibility 927 vEPC should be able to fall back to the legacy EPC based packet 928 forwarding to secure backward compatibility which is required to 929 connect existing system, or to connect roaming partners through 930 legacy S5/S8 interfaces. When fallback happened, all the packets are 931 not routed on our stateless user plane, but forwarded to vEPC (i.e. 932 SGW and PGW instances on hypervisor). vEPC must use a S1-U address 933 that is different from anycast address assigned to EPC-Es. This 934 address is assigned to SGW instances in vEPC and used to terminate 935 tunnels in vEPC servers (i.e. hypervisor). 937 5. IANA Considerations 939 This memo includes no request to IANA. 941 6. Security Considerations 943 There are no security considerations specific to this document at 944 this moment. 946 7. References 948 7.1. Normative References 950 [I-D.ietf-idr-flow-spec-v6] 951 McPherson, D., Raszuk, R., Pithawala, B., Andy, A., and S. 952 Hares, "Dissemination of Flow Specification Rules for 953 IPv6", draft-ietf-idr-flow-spec-v6-07 (work in progress), 954 March 2016. 956 [I-D.ietf-idr-tunnel-encaps] 957 Rosen, E., Patel, K., and G. Velde, "The BGP Tunnel 958 Encapsulation Attribute", draft-ietf-idr-tunnel-encaps-01 959 (work in progress), December 2015. 961 [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J., 962 and D. McPherson, "Dissemination of Flow Specification 963 Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009, 964 . 966 7.2. Informative References 968 [Boeing-BGP] 969 Andrew, , "Global IP Network Mobility using Border Gateway 970 Protocol (BGP)", IAB Plenary IAB Plenary of IETF 62nd, 971 March 2005. 973 [I-D.ietf-dmm-fpc-cpdp] 974 Liebsch, M., Matsushima, S., Gundavelli, S., and D. Moses, 975 "Protocol for Forwarding Policy Configuration (FPC) in 976 DMM", draft-ietf-dmm-fpc-cpdp-01 (work in progress), July 977 2015. 979 [I-D.ietf-softwire-map] 980 Troan, O., Dec, W., Li, X., Bao, C., Matsushima, S., 981 Murakami, T., and T. Taylor, "Mapping of Address and Port 982 with Encapsulation (MAP)", draft-ietf-softwire-map-13 983 (work in progress), March 2015. 985 [I-D.savolainen-stateless-pd] 986 Savolainen, T. and J. Korhonen, "Stateless IPv6 Prefix 987 Delegation for IPv6 enabled networks", draft-savolainen- 988 stateless-pd-01 (work in progress), February 2010. 990 [I-D.wakikawa-mext-haha-interop2008] 991 Wakikawa, R., Shima, K., and N. Shigechika, "The Global 992 HAHA Operation at the Interop Tokyo 2008", draft-wakikawa- 993 mext-haha-interop2008-00 (work in progress), July 2008. 995 [I-D.wakikawa-req-mobile-cp-separation] 996 Wakikawa, R., Matsushima, S., Patil, B., Chen, B., DJ, D., 997 and H. Deng, "Requirements and use cases for separating 998 control and user planes in mobile network architectures", 999 draft-wakikawa-req-mobile-cp-separation-00 (work in 1000 progress), November 2013. 1002 [I-D.yokota-dmm-scenario] 1003 Yokota, H., Seite, P., Demaria, E., and Z. Cao, "Use case 1004 scenarios for Distributed Mobility Management", draft- 1005 yokota-dmm-scenario-00 (work in progress), October 2010. 1007 [NFV-WHITEPAPER] 1008 Network Operators, , "Network Functions Virtualization, An 1009 Introduction, Benefits, Enablers, Challenges and Call for 1010 Action", SDN and OpenFlow SDN and OpenFlow World Congress, 1011 October 2012. 1013 [RFC3963] Devarapalli, V., Wakikawa, R., Petrescu, A., and P. 1014 Thubert, "Network Mobility (NEMO) Basic Support Protocol", 1015 RFC 3963, DOI 10.17487/RFC3963, January 2005, 1016 . 1018 [RFC5213] Gundavelli, S., Ed., Leung, K., Devarapalli, V., 1019 Chowdhury, K., and B. Patil, "Proxy Mobile IPv6", 1020 RFC 5213, DOI 10.17487/RFC5213, August 2008, 1021 . 1023 [RFC5555] Soliman, H., Ed., "Mobile IPv6 Support for Dual Stack 1024 Hosts and Routers", RFC 5555, DOI 10.17487/RFC5555, June 1025 2009, . 1027 [RFC5844] Wakikawa, R. and S. Gundavelli, "IPv4 Support for Proxy 1028 Mobile IPv6", RFC 5844, DOI 10.17487/RFC5844, May 2010, 1029 . 1031 [RFC6275] Perkins, C., Ed., Johnson, D., and J. Arkko, "Mobility 1032 Support in IPv6", RFC 6275, DOI 10.17487/RFC6275, July 1033 2011, . 1035 [RFC6301] Zhu, Z., Wakikawa, R., and L. Zhang, "A Survey of Mobility 1036 Support in the Internet", RFC 6301, DOI 10.17487/RFC6301, 1037 July 2011, . 1039 [RFC6333] Durand, A., Droms, R., Woodyatt, J., and Y. Lee, "Dual- 1040 Stack Lite Broadband Deployments Following IPv4 1041 Exhaustion", RFC 6333, DOI 10.17487/RFC6333, August 2011, 1042 . 1044 [RFC6459] Korhonen, J., Ed., Soininen, J., Patil, B., Savolainen, 1045 T., Bajko, G., and K. Iisakkila, "IPv6 in 3rd Generation 1046 Partnership Project (3GPP) Evolved Packet System (EPS)", 1047 RFC 6459, DOI 10.17487/RFC6459, January 2012, 1048 . 1050 [RFC6877] Mawatari, M., Kawashima, M., and C. Byrne, "464XLAT: 1051 Combination of Stateful and Stateless Translation", 1052 RFC 6877, DOI 10.17487/RFC6877, April 2013, 1053 . 1055 Authors' Addresses 1056 Satoru Matsushima 1057 SoftBank 1058 1-9-1,Higashi-Shimbashi,Minato-Ku 1059 Tokyo 105-7323 1060 Japan 1062 Email: satoru.matsushima@g.softbank.co.jp 1064 Ryuji Wakikawa 1065 SoftBank 1066 1-9-1,Higashi-Shimbashi,Minato-Ku 1067 Tokyo 105-7323 1068 Japan 1070 Email: ryuji.wakikawa@gmail.com