idnits 2.17.1 draft-matsushima-stateless-uplane-vepc-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 703 has weird spacing: '... type fun...' -- The document date (July 4, 2014) is 3556 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC5512' is defined on line 839, but no explicit reference was found in the text == Outdated reference: A later version (-09) exists of draft-vandevelde-idr-remote-next-hop-07 ** Obsolete normative reference: RFC 5512 (Obsoleted by RFC 9012) == Outdated reference: A later version (-13) exists of draft-ietf-softwire-map-10 Summary: 1 error (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group S. Matsushima 3 Internet-Draft Softbank Telecom 4 Intended status: Informational R. Wakikawa 5 Expires: January 5, 2015 Softbank Mobile 6 July 4, 2014 8 Stateless user-plane architecture for virtualized EPC (vEPC) 9 draft-matsushima-stateless-uplane-vepc-03 11 Abstract 13 We envision a new mobile architecture for the future Evolved Packet 14 Core (EPC). The new architecture is designed to support the 15 virtualization scheme called NFV (Network Function Virtualization). 16 In our architecture, the user plane of EPC is decoupled from the 17 control-plane and uses routing information to forward packets of 18 mobile nodes. Although the EPC control plane will run on hypervisor, 19 our proposal does not modify the signaling of the EPC control plane. 20 The benefits of our architecture are 1) scalability, 2) flexibility 21 and 3) Manageability. How to run the EPC control plane on NFV is out 22 of our focus in this document. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on January 5, 2015. 41 Copyright Notice 43 Copyright (c) 2014 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 59 1.1. The Benefits of NFV . . . . . . . . . . . . . . . . . . . 3 60 2. Motivations and Requirements, - Why IETF? - . . . . . . . . . 4 61 2.1. Motivations . . . . . . . . . . . . . . . . . . . . . . . 4 62 2.2. Requirements . . . . . . . . . . . . . . . . . . . . . . 5 63 3. Stateless user-plane architecture for virtualized EPC . . . 7 64 3.1. Architecture Overview . . . . . . . . . . . . . . . . . . 8 65 3.2. Protocol Overview . . . . . . . . . . . . . . . . . . . . 10 66 3.3. Control-plane awareness of stateless user-plane . . . . . 14 67 3.4. Routing mechanism . . . . . . . . . . . . . . . . . . . . 14 68 3.5. IPv4 Support . . . . . . . . . . . . . . . . . . . . . . 18 69 4. Operational Considerations . . . . . . . . . . . . . . . . . 19 70 4.1. Scalability and Reliability . . . . . . . . . . . . . . . 19 71 4.2. Backward Compatibility . . . . . . . . . . . . . . . . . 20 72 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 73 6. Security Considerations . . . . . . . . . . . . . . . . . . . 21 74 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 75 7.1. Normative References . . . . . . . . . . . . . . . . . . 21 76 7.2. Informative References . . . . . . . . . . . . . . . . . 21 77 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 79 1. Introduction 81 3GPP introduces Evolved Packet Core (EPC) that is fully IP based 82 mobile system for LTE and -advanced in their Release-8 specification 83 and beyond. Operators are now deploying EPC for LTE services and 84 encounter rapid LTE traffic growth. There are various activities to 85 offload mobile traffic in 3GPP and IETF such as LIPA, SIPTO and DMM. 86 The concept is similar that traffic of OTT (Over The Top) application 87 is offloaded at entity that is closer to the mobile node (ex. eNodeB 88 or closer anchor). 90 Likewise, overload of signaling (control plane) is also increasing 91 day by day. Network operators expect recent innovation and trends of 92 NFV (Network Function Virtualization) to solve this overloaded 93 control plane. NFV is discussed at the ETSI NFV ISG and is 94 introduced in [NFV-WHITEPAPER]. Mobile operator's network is built 95 with variety of proprietary hardware appliances today. If we can get 96 rid of these physical appliances and could shift to a cloud-based 97 service, we will have a lot of benefits explained in the next 98 section. This document assumes that NFV will push networking 99 functions currently run on dedicated hardware onto a cloud network. 100 Expected network functions are Mobility Management Entity (MME), 101 Serving Gateway (SGW) PDN Gateway(PGW), etc. With NFV, EPC can be 102 operated onto servers/hyper-visors. We name it virtualized-EPC 103 (vEPC) in this document. 105 This document uses a lot of 3GPP specific terms. These terms can be 106 found mostly at [RFC6459]. 108 1.1. The Benefits of NFV 110 This section briefly explains the benefits of NFV. The detailed 111 benefits can be found in [NFV-WHITEPAPER]. Although today's eco- 112 system of EPC appliances might be affected, we believe there are 113 various approaches to enhance current eco-system and migrate to new 114 NFV approaches. For example, operators could pay monthly recurring 115 charges for the NFV services and operations to vendors, instead of 116 one-time purchase and a little maintenance cost. 118 o [Flexible Network Operations]: The control functions of EPC are no 119 longer in appliances deployed widely in operator's network and can 120 be run at hypervisor (cloud). It is easier to add and/ or delete 121 functions from the services, because no physical construction is 122 needed. Network operations will be much simpler and easier 123 because complications of today's network are pushed to NFV (i.e. 124 hypervisor). 126 o [Flexible Resource Managements]: The EPC functions can be run on 127 hypervisor and are now less dependent on proprietary hardware. 128 Adding additional resources is easier in hypervisor, while adding 129 or replacing physical appliances require installation, 130 construction, configuration, and even migration plan without 131 service cutoff. A hypervisor can be also shared across various 132 functions such as PGW, SGW and MME. NFV also brings multi-tenancy 133 and allows a single platform for different services and users. 134 The operator can optimize resources and costs to share a NFV 135 platform for multiple customers (ex. MVNO customers) and services 136 (ex. multiple APNs). 138 o [Faster Speed of Time to Market]: When an operator wants a new 139 function to its network and services, the operator needs to 140 negotiate appliance vendors to implement the new functions or to 141 find alternative equipment supporting the new function. It takes 142 a longer time to convince the vendors, or to replace existing 143 hardware. However, if functions can be implemented as a software, 144 it is much faster to implement the functions on NFV. Even the 145 operator may implement them and try the new functions by 146 themselves. Field trial is also getting easier because of no 147 physical installation or replacement. You may turn on a new 148 function in NFV and observe how the new function behaves in your 149 network. NFV can save preparation time and tuning time of the new 150 function. 152 o [Cost Optimization]: Last but not least, Cost is the most 153 important motivation for operators to realize NFV. Operators can 154 remove many of proprietary appliances from its network and replace 155 them with industry standard servers, switches and routers. In 156 addition, it is easy to scale up and down operator's services so 157 that resources can be always tuned to the size of services. In 158 addition, operational costs led by any physical hardware such as 159 power supply, maintenance, installation, construction and 160 replacement can be minimized or even removed. The network design 161 can be simpler, because complicated functions could be handled by 162 NFV. That simple operation may enable automatic configurations 163 and prevent unnecessary trouble-shooting. As a result, CAPEX and 164 OPEX can be always optimized and lowered. 166 2. Motivations and Requirements, - Why IETF? - 168 2.1. Motivations 170 What is a role of IETF to realize vEPC in the future? IETF is not 171 the right place to discuss, for instance, how to run MME on 172 hypervisor. An important IETF activity must be to decouple the 173 control- and user- planes of mobility protocols used in EPC.The 174 motivation of decoupling the user and control plane is discussed in 175 [I-D.wakikawa-req-mobile-cp-separation]. In doing so, NFV-enabled 176 solutions can be easily designed and implemented with 177 interoperability across multiple vendors and platforms. Otherwise, 178 NFV solutions can be easily fragmented due to many proprietary 179 solutions for the protocol separations. As stated in 180 [NFV-WHITEPAPER], interoperability is highly important. 182 In the past, IETF has developed tunnel based mechanisms for mobile 183 nodes such as Mobile IPv6 [RFC6275][RFC5555], Proxy Mobile IPv6 184 [RFC5213][RFC5844] and NEMO [RFC3963]. Similarly, 3GPP has developed 185 tunnel protocols called GPRS Tunneling Protocol (GTP). These tunnel- 186 based protocols establish a data path for a mobile node between the 187 mobile node and an anchor point (s). There is a case where an access 188 router terminates a tunnel instead of a mobile node (ex. Proxy 189 Mobile IP). In 3GPP, a tunnel is established between SGW and PGW per 190 a mobile node by either Proxy Mobile IPv6 or GTP. The control and 191 the user planes of these mobility protocols are tightly related and 192 cannot be decoupled. The signaling like Binding Update and user's 193 packets are routed along a same path in EPC. It might be necessary 194 to extend these mobility protocols for the user- and control- planes 195 separation. The protocol separation of Mobile IP is discussed in 196 [I-D.yokota-dmm-scenario]. 198 Alternatively, if vEPC was realized, we should have an opportunity to 199 re-visit the basic architecture of mobility system. Instead of 200 tunneling packets on today's EPC, why can't we just route packets to 201 a mobile node? Since a role of the user plane is "routing", BGP and 202 other routing protocols could be used to forward UE's traffic. This 203 document introduces a BGP-based solution. Software Defined 204 Networking (SDN) can be an alternative solution. Open Flow and other 205 relevant protocols can setup the forward path dynamically according 206 to UE's states available in the control plane. 208 We have to remember that there is a good reason of adapting tunneling 209 in Mobile IP based solutions, that is global mobility and signaling. 210 A mobile node should be able to move anywhere on the Internet and be 211 reachable from anyone on the Internet. There were routing based 212 global mobility solutions like Boeing global mobility [Boeing-BGP] 213 and WINMO [RFC6301]. In these proposals, BGP was used to propagate 214 forwarding information of mobile nodes to the Internet. Whenever a 215 mobile node changes its point of attachment, the route must be 216 updated. Due to scalability and stability issues of the Internet, 217 this solution was not recommended by IETF [Boeing-BGP]. However, as 218 Boeing showed, it is doable to support global mobility by using BGP 219 routing update. If scalability is not your concern, a routing based 220 approach becomes a candidate of the mobility solution. 222 While global mobility is important, the "reality" is that your cell 223 phones (i.e. UE/mobile node) are moving just within an operator's 224 network and fully controlled in your local EPC. If mobility is 225 limited within an operator, we believe a routing based approach is 226 feasible and practical for today's mobile system. Instead of 227 dedicated proprietary equipment like SGW and PGW to manage a tunnel 228 path for a mobile node, multiple industry standard routers and 229 switches are configured in the user plane. These switches and 230 routers receive mobile nodes' forwarding information from the control 231 plane of vEPC by routing update. 233 2.2. Requirements 235 Requirements of our stateless user plane for vEPC are followings. 237 NFV Support 238 The future EPC architecture must support NFV capability. The 239 control plane of EPC operated on NFV framework is named 240 "virtualized EPC (vEPC)" in this document. The control plane 241 of vEPC should keep backward compatibility with the today's 242 EPC's control plane. It means this document doesn't modify 243 the control plane at all. It only assumes software-based 244 MME, SGW, and PGW run on hypervisor. 246 Separation of Control- and User- Planes 247 Due to tight relationship of the control- and user- planes in 248 today's EPC, resource increase is always provisioned to both 249 planes at once. It prevents flexible resource arrangement 250 and introduces high capital investment and over-provisioned 251 resources to one of planes. If NFV is deployed, it is 252 expected that computing resources can be independently 253 allocated to the control planes of the vEPC in a flexible 254 manner. 256 _+---+_ _ _+---+_ _ _ 257 EPC / | S | | P | / 258 Control-plane / | G | | G | / 259 /_ _| W |_ _ _| W |__/ 260 +---+ +---+ | | | | 261 | | | e | | | | | 262 | U | | N | _| |_ _ _| |_ _ _ 263 EPC | E | | B | / | | | | / 264 User-plane +---+ +---+ / +---+ +---+ / 265 /_ _ _Existing EPC_ _/ 267 _+---+_ _ _+---+_ _ _ 268 vEPC / | S | | P | / 269 Control-plane / | G | | G | / 270 /_ _| W |_ _ _| W |__/ 271 | | | | 272 +---+ +---+ +---+ +---+ 273 | | | e | +----------+ 274 | U | | N | _|IP Routing|_ _ _ 275 Stateless | E | | B | / | Network | / 276 User-plane +---+ +---+ / +----------+ / 277 /_ _ _ _ _ _ _ _ _ / 279 NFV enabled EPC architecture 281 Figure 1 283 Figure 1 shows a possibility that the entities of EPC 284 Control- plane are virtualized in generic cloud environment, 285 however user packets won't go through those virtualized EPC 286 nodes. Decoupling User-plane from the Control-plane entities 287 will be made virtualized Control-plane nodes relax hyper- 288 visor data- path capacity requirements. On the other hand, 289 decoupled User-plane into IP routing network will be agnostic 290 from sessions and bearers states, of which are generated and 291 maintained in the Control-plane. In terms of IP routing, 292 forwarding packets through the networks is based on the 293 destination address of the packets evaluated with network 294 reachable information in the routing table that accommodated 295 in the routing nodes. To forward EPC User-plane packets 296 correctly, those states must be indicated by network 297 reachable information. 299 Flat Design for Distributed Operation 300 Today's 3GPP architecture introduces PDN gateway (PGW) as a 301 gateway to external networks like the Internet. PGW manages 302 all traffic from and to UEs and could be a bottleneck and 303 single point of failure of network connectivity. In 304 addition, due to recent rapid traffic increase, it is 305 important to perform traffic engineering and to offload 306 traffic to multiple locations (ex. SGW, PGW, eNodeB). For 307 enhancements of traffic engineering capability, more flat 308 design with multiple gateways is expected so that traffic can 309 be distributed to all these gateways. There were proposals 310 how to enable flat design to (Proxy) Mobile IP such as 311 [I-D.wakikawa-mext-haha-interop2008] in IETF. Distributed 312 Mobility Management (DMM) Working Group has also discussed 313 how to extend Mobile IP-based solutions to support traffic 314 distribution in an optimal way by removing centrally deployed 315 anchors that is like a Home Agent. 317 Stateless in User Plane 318 Ultimate goal of vEPC is to remove all mobility specific 319 states from the forwarding nodes in the user-plane of vEPC. 320 If we succeed in this, industry standard routers and switches 321 can be used to forward mobile nodes traffic in the user plane 322 of vEPC. A mobile node's specific states are kept in both an 323 IP header of the mobile node's packets and a routing entry of 324 the mobile node. The detail is described in Section 3.2 326 3. Stateless user-plane architecture for virtualized EPC 328 This section explains our solution that is the stateless user-plane 329 architecture for vEPC. This solution is basically a combination of 330 existing protocols defined in IETF. A minor extension might be 331 needed but it should be easily addressed in IETF. We first introduce 332 our architecture and then protocol overview. 334 3.1. Architecture Overview 336 Figure 2 shows the user plane of the current EPC architecture. A 337 tunnel is established between SGW and PGW by either Proxy Mobile IP 338 or GTP. PGW is an anchor point of UE for incoming packets. All the 339 packet destined to UE is routed first to PGW. The UE's packets are 340 intercepted by PGW and tunneled to SGW. SGW then forwards the packet 341 to UE via access points (i.e. eNodeB) over Radio Area Network (RAN). 343 .--. 344 _( `)_ 345 _( `) 346 +---+ +---+ ( EPC-NW ) +---+ 347 [UE]~~~~|RAN|====|SGW|===================|PGW| -> the Internet 348 +---+ +---+ `--(______)---' +---+ 349 <--------Tunneling------> 351 ~~~ Wireless Link 352 === GTP Tunnel (PMIP tunnel) 353 --- IP link 355 User plane of the current EPC 357 Figure 2 359 Figure 3 is our proposed user plane of vEPC. The control plane is 360 not shown in this figure. 362 [EPC-E anycast address] 363 | .--. 364 | _( `)_ 365 | +---+ _( `) 366 +---+ \ |EPC| ( IPv6 Core ) +---+ 367 [UE]~~~~|RAN|====| -E|-( ` NW )-|RTR| --> the Internet 368 +---+ +---+ ( ` ) ) +---+ 369 (eNodeB) `--(_____)--' 370 <--------Routing--------> 372 User plane of vEPC 374 Figure 3 376 We introduce two new entities such as 378 EPC Edge Router (EPC-E) 379 EPC-E is located at the same place of today's SGW and terminates 380 GTP tunnel established with eNodeB (RAN). EPC-E supports the user 381 plane functions of SGW and PGW. EPC-E is configured an anycast 382 address to the network interface facing to eNodeB. The eNodeB 383 establishes a GTP tunnel per UE with this anycast address. Thanks 384 for anycast address, UE's traffic forwarded by eNodeB is always 385 routed to the closest EPC-E of UE. EPC-E is a router and 386 maintains routing information of every UE that is notified by the 387 control plane. Detail of routing mechanism can be found in 388 Section 3.4. 390 Router (RTR) 391 It is a regular IP router. The control plane of vEPC distributes 392 routing information of every UE by a routing protocol like BGP. 393 Therefore any additional protocols other than routing protocols 394 are not needed for RTR. Multiple RTRs can be configured anywhere 395 in the user plane of vEPC. RTRs announce UE's routing information 396 to the external network (ex. The Internet). 398 As you see in Figure 3, we omit a tunneling mechanism originally 399 established between SGW and PGW for routing UE's packets in the user 400 plane. By removing this tunnel, UE's packets are forwarded to and 401 from the Internet according to routing tables on routers in the core 402 network. Note that, although we remove the tunnel for UE's traffic 403 in the user plane, the control-plane signaling stays same in the 404 control plane. If Proxy Mobile IP is used for this tunnel, Proxy 405 Binding Update and Acknowledgment are exchanged between PGW and SGW 406 that are managed by NFV on servers/hyper-visor. Instead of a tunnel 407 setup, states created by Proxy Mobile IP are distributed to all 408 routing entities (EPC-E and RTR) by a routing protocol. From the 409 user plane point of view, these states are just seen as routing 410 entries. EPC-E and RTR are not involved in any signaling of the 411 control plane. The control plane just injects routing information to 412 EPC-E and RTR to setup routing paths to and from UEs. 414 Although this architecture just uses IPv6 core network, it supports 415 both IPv4 and IPv6 packets. The detailed operation of IPv4 support 416 will be discussed in Section 3.5. 418 3.2. Protocol Overview 420 This section gives an example of protocols used for vEPC. Figure 4 421 is the procedure of the PDN connection setup in vEPC. This figure is 422 copied from the section 3 of [RFC6459]. All the steps from (1) to 423 (13) are same as the original except for NFV-based MME, SGW, PGW, 424 HSS, and AAA. 426 The vEPC introduces two new steps, (14) and (15), to setup paths in 427 the user-plane after finishing all the signaling on the control- 428 plane. (16) and (17) are the steps to assign IP address to the mobile 429 node. 431 In (14), vEPC advertises a routing information of UE whose next hop 432 is set to GTP tunnel between eNodeB and UE in RAN as defined in 433 [I-D.vandevelde-idr-remote-next-hop]. The entity of advertising UE 434 route in vEPC must be a BGP speaker. In this document, it is a 435 further study item how the BGP speaker interfaces with the mobility 436 management entity to import UE route. 438 The EPC-E has a peering with the BGP speaker directly. It is thus 439 expected that there is no additional propagation delay of traversing 440 multiple BGP speakers between EPC-E and vEPC. Adding that kind of 441 surplus delay affects user-plane to be interrupted so that it should 442 be avoided as much as possible for user experience. 444 In step (15), the EPC-E advertises routes to upstream routers such as 445 the RTR. For scalable routing operation, UE's prefixes should be 446 aggregated into more shorter length prefixes. Due to that reason, 447 the EPC-E generates routing information and advertised it to the RTR 448 that includes aggregated prefix instead of UE's prefixes and EPC-E 449 address as the next-hop. 451 When tunnel endpoint is updated by UE hand-over between eNodeBs, vEPC 452 must refresh the route of UE with the updated tunnel endpoint as new 453 remote next-hop. The updated route should be immediately advertised 454 to all the EPC-Es. In the case of UE detachment, vEPC simply removes 455 the route of the detached UE. 457 Since the upstream routers do not require routing update for each of 458 UE hand-over event, the EPC-E is enough to advertise aggregate routes 459 once during the UE routes exist. The core network also needs to be 460 kept its routing stable. That means unnecessary routing update storm 461 should be prevented. So the EPC-E should suppress routing updates to 462 upstream routers for each received event from the vEPC except when 463 all UE routes has been gone from the EPC-E. 465 vEPC(MME/SGW 466 UE eNodeB PGW/HSS/AAA) EPC-E RTR 467 | | | | | 468 |---------->|(1) | | | 469 | |---------->|(1) MME | | 470 |/---------------------\| | | 471 | Authentication |(2) AAA | | 472 |\---------------------/| | | 473 | | |--+SGW/PGW | | 474 | | | |(3)(4) |(4)SGW's TEID is configured 475 | | |<-+ | and notified to MME 476 | |<----------|(5) MME | | 477 |/---------\| | | | 478 | RB setup |(6) | | | 479 |\---------/| | | | 480 | |---------->|(7) MME | | 481 |---------->|(8) | | | 482 | |---------->|(9) MME |(9)eNodeB TEID is configured 483 | | | | and sent to MME 484 |= = = = = = = Uplink Data = = = = =>= = = = ==>|(10) 485 | The uplink is not yet built here | 486 | | | | | 487 | | |--+SGW/MME | | 488 | | | |(11,12) | | 489 | | |<-+ | | 490 |<= = = = = = Downlink Data = = = = <= = = = = =|(13) 491 | The downlink is not yet built here | 492 | | | | | 493 | | |---------->|(14) Route Update 494 | | | [Dst: UE-prefix, 495 | | | NxtHop: S1-U addr and TEID of enodeB 496 | | | | | 497 | | | |---------->|(15)Route update 498 | | | |[Dst: UE-prefix, 499 | | | | NxtHop: EPC-E address] 500 | | | | | 501 |---------RS or DHCP Request------->|(16) | 502 |<--------RA or DHCP Reply----------|(17) | 504 Extended PDN Connection Setup Procedure (copied Figure 8 of RFC6459) 506 Figure 4 508 UE requests an IPv6 prefix for its address assignment in the step 509 (16). In our architecture, an IPv6 prefix is still assigned by vEPC 510 in the control plane, as PDN-GW does in the legacy EPC. However, 511 EPC-E is responsible to deliver the IPv6 prefix to UE by DHCP or 512 Stateless address autoconfiguration (SLAAC). 514 We now explain how EPC-E can know the prefix assigned to UE from vEPC 515 for address configuration steps (16 and 17). When (1) to (15) are 516 completed, vEPC has already advertised the UE's prefix as route 517 information to all the EPC-E. Therefore, when EPC-E receives a 518 packet of either Router Solicitation or DHCPv6 request message, it 519 just looks up the remote next-hop field of its routing information 520 base (RIB) with the source IP address and the TEID of the received 521 packet. A route entry matched for this search is the prefix 522 delegated to the requesting UE. Therefore, EPC-E simply uses the 523 prefix of the route entry as an assigned UE's prefix. 525 In (17), EPC-E returns the found prefix to UE by either Router 526 Advertisement or DHCPv6 reply message. UE now creates an address(es) 527 from the received prefix. It is important to highlight that UE can 528 obtain the same prefix information from any EPC-E all the time 529 because the same UE's route information is available on all the EPC- 530 E. 532 It would be convenient to use automatic UE's prefix creation rule or 533 algorithm for vEPC. There are various mechanisms to create UE's 534 prefix. As an example, Stateless IPv6 Prefix Delegation 535 [I-D.savolainen-stateless-pd] is introduced as an algorithm to create 536 UE's prefix in vEPC below. It important to mention that our 537 architecture of the stateless user plane does not rely on any 538 particular prefix creation mechanisms like 539 [I-D.savolainen-stateless-pd] and can be run with any of them. 541 In the case of an UE's prefix length is equal, or shorter than /64, 542 the generated prefix is consisted as shown in Figure 5. Each PDN is 543 assumed to have single or several prefixes (named PDN prefix) used to 544 generate UE's address. Followed by the PDN prefix, there is TEID 545 field assigned for a UE's session on S1-U interface of vEPC. TEID is 546 32 bits identifier in GTP header to distinguish each bearer. The 547 remaining bits are filled by subnet ID. 549 | n bits | o bits (=< 32)|64-n-o bits| 550 +---------------------------+---------------+-----------+ 551 | PDN Prefix | TEID | subnet ID | 552 +---------------------------+---------------+-----------+ 553 <---------------------------64bits----------------------> 555 Stateless-pd Prefix 557 Figure 5 559 3.3. Control-plane awareness of stateless user-plane 561 Nodes in the control-plane in vEPC must be aware that the anycast 562 address assigned to EPC-E is a S1-U address of vEPC. The vEPC must 563 use the anycast address in signaling between vEPC and RAN. By doing 564 this, packets from RAN are correctly forwarded to an appropriate EPC- 565 E. Due to anycast nature, it means there is no hand-off procedure 566 between SGWs because all eNodeB in the RAN send packets to the same 567 anycast address. 569 When an operator needs to increase virtualized instances to cope with 570 just signaling overload, the operator should use the existing S1-U 571 address (i.e. EPC-E anycast address) for the new instances. If the 572 operator would increase the capacity of the user plane, it can add 573 additional EPC-Es in the core network. The operator can group the 574 new EPC-Es as a set and increase scalability and performance of the 575 user plane. In this case, the operator uses a new anycast address to 576 the new set of EPC-E. We will discuss operational consideration in 577 detail in Section 4. 579 3.4. Routing mechanism 581 Figure 6 shows a packet forwarding mechanism of our stateless user 582 plane. As an example, there are four eNodeB (illustrated as eNB-x) , 583 three EPC-Edge routers(shown as EPC-Ex) and two routers (RTRx) in 584 Figure 6. UE is first connected to eNB-C and then moves to eNB-D. 585 The UE at the new location is illustrated as UE'. Routing entry for 586 UE is also illustrated at the right side in Figure 6. 588 EPC-E has two interfaces facing either RAN or CORE networks. An 589 anycast address (shown as X) is configured to the interface facing 590 RAN of all EPC-E. EPC-E assigns an individual IPv6 address to 591 another interface (illustrated "a" to "d" in the figure). It is 592 important to mention that the anycast address X can be treated as the 593 SGW's S1-U address. 595 Since RTRs are a gateway to the Internet, they advertise routes of an 596 operator's prefix to the Internet. After one of RTR receives a 597 packet of UE from the Internet, it needs to routing it to UE in the 598 user plane. RTR has a simple routing entry for PDN prefix whose next 599 hop points to the EPC-E. One of RTR (let's say RTR2 in this case) 600 looks up a routing table with UE's address and matched it with a 601 routing entry of PDN prefix. Since multiple EPC-Es advertise a route 602 for the same PDN prefix, RTR2 should forward the packet to one of 603 EPC-E according to the routing entry. This routing is known as hot- 604 potato routing. In this example, the RTR2 uses EPC-E2-b as a nexthop 605 of PDN prefix. 607 When the UE's packet is arrived at EPC-E2, EPC-E2 needs to forwards 608 them to UE via eNodeB to which UE is connecting by using GTP tunnel. 609 For this operation, EPC-E2 has a routing entry that destination is 610 UE's prefix and that next hop points to GTP tunnel between eNB-C and 611 the EPC-Es. In order to identify the GTP tunnel for UE, EPC-E needs 612 S1-U address and Tunnel Endpoint ID (TEID) of eNB-C that is eNB-C-3 613 in Figure 6. The eNB-C TEID for UE is illustrated as TEID[eNB-C]. 614 The SGW assigned TEID is utilized to generate the UE's prefix as we 615 explained in Section 3.2. These TEID are assigned per UE. The TEID 616 and S1-U address of eNodeB are retrieved from the next hop field of 617 the routing entry of the mobile node. By using the GTP information, 618 every EPC-E can now forward the UE's packets to right eNodeB. 620 Routing outgoing packets from UE is much simpler. The packets from 621 UE are always routed to the closest EPC-E to UE because of anycast 622 routing. In Figure 6, when UE sends a packet to a destination, the 623 packet is reached to eNB-C and tunneled to EPC-E's anycast address. 624 The GTP-tunneled packet is routed to the closest EPC-E that is EPC-E2 625 in this case. The packet is decapsulated by EPC-E2 and then 626 forwarded to one of RTR according to the routing table. Since the 627 decapsulated packet is regular IPv6 packet, no extra control other 628 than routing is necessary. 630 When UE moves to a new location (UE'), it updates its location on the 631 control plane. After signaling completion for location update, vEPC 632 needs to update the UE's routing entry of all EPC-E so that vEPC 633 advertises updated route with new location to all EPC-Es by a routing 634 protocol. The routing entry should be updated with the new eNodeB's 635 address that is eNB-D-4. During handover, there might be some 636 traffic arriving to the older eNodeB (eNB-C). These packets can be 637 re-routed to the new eNodeB (eNB-D) via X2-U interface in RAN. 639 The UE's address isn't changed when UE changes its attachment. In 640 our scenario, SGW run on hypervisor and is independent from network 641 topology. Therefore, logically we don't have handover across 642 different SGWs. UE can stay connected with the same SGW all the time 643 and can keep using the same TEID after handover. Thus, UE's address 644 is unchanged even after handover. 646 +--------------------+ 647 | Correspondent Node | 648 +--------------------+ 649 | 650 .--. [Route on the Internet] 651 _(. `) Destination: Operator's Prefix 652 _( `)_ NextHop: RTR-1/2 653 ( Internet `) 654 ( ` . ) ) 655 `--(_______)---' 656 / \ [Route at RTR] 657 +----+ +----+ Destination: PDN Prefix 658 |RTR1| |RTR2| NextHop: EPC-E2-b 659 +----+ +----+ 660 \ .--. / 661 _( IP `. 662 ( CORE NW ) 663 ___( ` . ) )__ 664 / `--(___.--' \ 665 / | \ 666 |a |b |c [Route at EPC-E] 667 +---+--+ +---+--+ +---+--+ Destination: UE's Prefix *1 668 |EPC-E1| |EPC-E2| |EPC-E3| NextHop: GTP tunnel (eNB-C) *2 669 +---+--+ +---+--+ +---+--+ 670 |X |X |X 671 \ .--. / 672 \_____( RAN `.____/ 673 ( ACCESS ) 674 ____( ` NW ) )_______ 675 / `--(___.--' \ 676 / | | \ 677 |1 |2 |3 |4 678 +--+--+ +--+--+ +--+--+ +--+--+ 679 |eNB-A| |eNB-B| |eNB-C| |eNB-D| 680 +--+--+ +--+--+ +--+--+ +--+--+ 681 : : : : 682 UE --> UE' 683 *1 TEID used at EPC-E for the UE is included in this UE's prefix. 684 see Figure 4. 685 *2 GTP tunnel state is stored in the next hop field. The state 686 information is the combination of eNB-C S1 address that is 687 eNB-C-3 and TEID(eNB-C) assigned for the UE. 689 Routing Mechanism Overview 691 Figure 6 693 3.5. IPv4 Support 695 Recent IPv6 transition mechanisms enable IPv6-only network to forward 696 IPv4 packet with encapsulation or translation techniques. By using 697 one of mechanisms, we can use IPv6 for our stateless user-plane 698 network for transporting both IPv4 and IPv6 packets. Figure 7 shows 699 available solutions of IPv4 support for each bearer type to deal with 700 that requirement. 702 Bearer UE EPC-E Gateway 703 type function function function 704 -------------------------------------------- 705 IPv4 - B4 AFTR 706 IPv4 - CLAT PLAT 707 IPv6 MAP-CE - MAP-BR 708 IPv6 B4 - AFTR 709 IPv6 CLAT - PLAT 711 Solutions and functions for IPv4 support 713 Figure 7 715 In the case of a UE only support IPv4 bearer, B4 function of DS-Lite 716 [RFC6333] or CLAT function of 464XLAT [RFC6877] may be implemented in 717 a EPC-E. Both functions are stateless therefore EPC-E isn't required 718 to maintain any tunneling or translation state. 720 Figure 8 shows how to support IPv4 on IPv6 core network in our vEPC. 721 Instead of using RqTR as a gateway to the Internet, DS-LITE AFTR or 722 464XLAT PLAT is installed as a gateway to the IPv4 Internet. 724 .--. 725 EPC-E _( `)_ Gateway 726 +----+ _( `) +------+ 727 | B4 | ( IPv6 Core ) | AFTR | 728 [UE]==IPv4-only==| or |-( ` NW )-| or | --> Internet 729 bearer |CLAT| ( ` ) ) | PLAT | (IPv4) 730 +----+ `--(_____)--' +------+ 731 <-----IPv4 over IPv6-----> 732 or 733 v4v6 translation 735 IPv4 User plane of vEPC 737 Figure 8 739 If UE supports IPv6 capable bearer, IPv6 transition function may be 740 implemented in the UE such as MAP-CE [I-D.ietf-softwire-map], B4 or 741 CLAT. That means an EPC-E receives IPv6 packets from UE in this case 742 so that the EPC-E does not need to be involved in the part of IPv4 743 support functions. 745 4. Operational Considerations 747 4.1. Scalability and Reliability 749 Virtualization allows vEPC to be elastic for steep demand of requests 750 to create and update for sessions. In our architecture, that makes 751 routing update fluctuation from vEPC to EPC-E. This is the reason 752 why we select BGP as a protocol between vEPC and EPC-E. BGP is 753 scalable and stable routing protocol today. 755 BGP is an incremental update protocol so that once BGP peer 756 established, millions of routes can be easily updated in stable 757 manners. Operators can appropriately design BGP peering between vEPC 758 and ECP-E to secure convergence time within appropriate period. 760 Granularity of the peering should be aware EPC-E capacity because it 761 is assumed that EPC-E has upper limit of routing entries. BGP 762 peering design should makes sure that total number of routes does not 763 exceed EPC-E capacity. 765 During the network planning, operators must understand EPC-E's 766 capacity such as # of routes, bandwidth, etc. An example of 767 estimation, if a EPC-E has 1Gbps throughput and each UE's bandwidth 768 consumption is 10Kbps in average, the EPC-E should have 100K routes 769 capacity. 771 This is an operational approach to minimize the risk of routing 772 update fluctuation. If it is hard to support all the UEs by a EPC-E 773 in an operators network, another EPC-E can be introduced and 774 configured as a set of EPC-Es. The UEs are distributed and handled 775 by the EPC-Es within the set. We don't need to support millions of 776 UEs by a single EPC-E. 778 EPC-E set is also useful to have EPC-E redundancy for reliable 779 operation. The nature of BGP makes easy to replicate UE routes to 780 multiple EPC-Es within a EPC-E set. In that EPC-E set, when an EPC-E 781 fall down to a failure, another EPC-E come out with same UE routes 782 that the fall-down EPC had and immediately re-converge to core 783 routing. That helps user-plane to minimize disruption during EPC-E 784 failure recovery. 786 These are another advantage of using routing mechanism in the user 787 plane. We already explain how to handle multiple EPC-Es and EPC-E 788 sets in our scheme in Section 3.3. 790 The notion of multiple EPC-E sets is easily fitted into our today's' 791 network. The operator's network is often separated into several 792 regional network for geographical scalability. Therefore, the 793 operator can assign different EPC-E set to different region for 794 better scalability. 796 In that network, when an UE hands over between two regions, the 797 session of the UE might be disconnected if the serving EPC-E doesn't 798 have reachability for those region access networks. For example, in 799 the case of regional access networks have duplicated IPv4 private 800 address space. To enable inter-region hand-over, it is recommended 801 that all of the access network, such as RAN, are IPv6 networks and 802 reachable each other. 804 In addition, routers and EPC-E in the IPv6 core network are required 805 to process just "route", they naturally aggregate those routing 806 entries. It helps limiting the total number of routing entries in 807 our core network. 809 4.2. Backward Compatibility 811 vEPC should be able to fall back to the legacy EPC based packet 812 forwarding to secure backward compatibility which is required to 813 connect existing system, or to connect roaming partners through 814 legacy S5/S8 interfaces. When fallback happened, all the packets are 815 not routed on our stateless user plane, but forwarded to vEPC (i.e. 816 SGW and PGW instances on hypervisor). vEPC must use a S1-U address 817 that is different from anycast address assigned to EPC-Es. This 818 address is assigned to SGW instances in vEPC and used to terminate 819 tunnels in vEPC servers (i.e. hypervisor). 821 5. IANA Considerations 823 This memo includes no request to IANA. 825 6. Security Considerations 827 There are no security considerations specific to this document at 828 this moment. 830 7. References 832 7.1. Normative References 834 [I-D.vandevelde-idr-remote-next-hop] 835 Velde, G., Patel, K., Rao, D., Raszuk, R., and R. Bush, 836 "BGP Remote-Next-Hop", draft-vandevelde-idr-remote-next- 837 hop-07 (work in progress), June 2014. 839 [RFC5512] Mohapatra, P. and E. Rosen, "The BGP Encapsulation 840 Subsequent Address Family Identifier (SAFI) and the BGP 841 Tunnel Encapsulation Attribute", RFC 5512, April 2009. 843 7.2. Informative References 845 [Boeing-BGP] 846 Andrew, , "Global IP Network Mobility using Border Gateway 847 Protocol (BGP)", IAB Plenary IAB Plenary of IETF 62nd, 848 March 2005. 850 [I-D.ietf-softwire-map] 851 Troan, O., Dec, W., Li, X., Bao, C., Matsushima, S., 852 Murakami, T., and T. Taylor, "Mapping of Address and Port 853 with Encapsulation (MAP)", draft-ietf-softwire-map-10 854 (work in progress), January 2014. 856 [I-D.savolainen-stateless-pd] 857 Savolainen, T. and J. Korhonen, "Stateless IPv6 Prefix 858 Delegation for IPv6 enabled networks", draft-savolainen- 859 stateless-pd-01 (work in progress), February 2010. 861 [I-D.wakikawa-mext-haha-interop2008] 862 Wakikawa, R., Shima, K., and N. Shigechika, "The Global 863 HAHA Operation at the Interop Tokyo 2008", draft-wakikawa- 864 mext-haha-interop2008-00 (work in progress), July 2008. 866 [I-D.wakikawa-req-mobile-cp-separation] 867 Wakikawa, R., Matsushima, S., Patil, B., Chen, B., DJ, D., 868 and H. Deng, "Requirements and use cases for separating 869 control and user planes in mobile network architectures", 870 draft-wakikawa-req-mobile-cp-separation-00 (work in 871 progress), November 2013. 873 [I-D.yokota-dmm-scenario] 874 Yokota, H., Seite, P., Demaria, E., and Z. Cao, "Use case 875 scenarios for Distributed Mobility Management", draft- 876 yokota-dmm-scenario-00 (work in progress), October 2010. 878 [NFV-WHITEPAPER] 879 Network Operators, , "Network Functions Virtualization, An 880 Introduction, Benefits, Enablers, Challenges and Call for 881 Action", SDN and OpenFlow SDN and OpenFlow World Congress, 882 October 2012. 884 [RFC3963] Devarapalli, V., Wakikawa, R., Petrescu, A., and P. 885 Thubert, "Network Mobility (NEMO) Basic Support Protocol", 886 RFC 3963, January 2005. 888 [RFC5213] Gundavelli, S., Leung, K., Devarapalli, V., Chowdhury, K., 889 and B. Patil, "Proxy Mobile IPv6", RFC 5213, August 2008. 891 [RFC5555] Soliman, H., "Mobile IPv6 Support for Dual Stack Hosts and 892 Routers", RFC 5555, June 2009. 894 [RFC5844] Wakikawa, R. and S. Gundavelli, "IPv4 Support for Proxy 895 Mobile IPv6", RFC 5844, May 2010. 897 [RFC6275] Perkins, C., Johnson, D., and J. Arkko, "Mobility Support 898 in IPv6", RFC 6275, July 2011. 900 [RFC6301] Zhu, Z., Wakikawa, R., and L. Zhang, "A Survey of Mobility 901 Support in the Internet", RFC 6301, July 2011. 903 [RFC6333] Durand, A., Droms, R., Woodyatt, J., and Y. Lee, "Dual- 904 Stack Lite Broadband Deployments Following IPv4 905 Exhaustion", RFC 6333, August 2011. 907 [RFC6459] Korhonen, J., Soininen, J., Patil, B., Savolainen, T., 908 Bajko, G., and K. Iisakkila, "IPv6 in 3rd Generation 909 Partnership Project (3GPP) Evolved Packet System (EPS)", 910 RFC 6459, January 2012. 912 [RFC6877] Mawatari, M., Kawashima, M., and C. Byrne, "464XLAT: 913 Combination of Stateful and Stateless Translation", RFC 914 6877, April 2013. 916 Authors' Addresses 918 Satoru Matsushima 919 Softbank Telecom 920 1-9-1,Higashi-Shimbashi,Minato-Ku 921 Tokyo 105-7322 922 Japan 924 Email: satoru.matsushima@g.softbank.co.jp 926 Ryuji Wakikawa 927 Softbank Mobile 928 1-9-1,Higashi-Shimbashi,Minato-Ku 929 Tokyo 105-7322 930 Japan 932 Email: ryuji.wakikawa@gmail.com