idnits 2.17.1 draft-matsushima-stateless-uplane-vepc-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 699 has weird spacing: '... type fun...' -- The document date (February 14, 2014) is 3717 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'I-D.vandevelde-idr-remote-next-hop' is defined on line 813, but no explicit reference was found in the text == Outdated reference: A later version (-09) exists of draft-vandevelde-idr-remote-next-hop-03 ** Obsolete normative reference: RFC 5512 (Obsoleted by RFC 9012) == Outdated reference: A later version (-13) exists of draft-ietf-softwire-map-07 Summary: 1 error (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Group S. Matsushima 3 Internet-Draft Softbank Telecom 4 Intended status: Informational R. Wakikawa 5 Expires: August 18, 2014 Softbank Mobile 6 February 14, 2014 8 Stateless user-plane architecture for virtualized EPC (vEPC) 9 draft-matsushima-stateless-uplane-vepc-02 11 Abstract 13 We envision a new mobile architecture for the future Evolved Packet 14 Core (EPC). The new architecture is designed to support the 15 virtualization scheme called NFV (Network Function Virtualization). 16 In our architecture, the user plane of EPC is decoupled from the 17 control-plane and uses routing information to forward packets of 18 mobile nodes. Although the EPC control plane will run on hypervisor, 19 our proposal does not modify the signaling of the EPC control plane. 20 The benefits of our architecture are 1) scalability, 2) flexibility 21 and 3) Manageability. How to run the EPC control plane on NFV is out 22 of our focus in this document. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on August 18, 2014. 41 Copyright Notice 43 Copyright (c) 2014 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 59 1.1. The Benefits of NFV . . . . . . . . . . . . . . . . . . . 3 60 2. Motivations and Requirements, - Why IETF? - . . . . . . . . . 4 61 2.1. Motivations . . . . . . . . . . . . . . . . . . . . . . . 4 62 2.2. Requirements . . . . . . . . . . . . . . . . . . . . . . 5 63 3. Stateless user-plane architecture for virtualized EPC . . . 8 64 3.1. Architecture Overview . . . . . . . . . . . . . . . . . . 8 65 3.2. Protocol Overview . . . . . . . . . . . . . . . . . . . . 10 66 3.3. Control-plane awareness of stateless user-plane . . . . . 13 67 3.4. Routing mechanism . . . . . . . . . . . . . . . . . . . . 13 68 3.5. IPv4 Support . . . . . . . . . . . . . . . . . . . . . . 16 69 4. Operational Considerations . . . . . . . . . . . . . . . . . 17 70 4.1. Scalability . . . . . . . . . . . . . . . . . . . . . . . 17 71 4.2. Backward Compatibility . . . . . . . . . . . . . . . . . 18 72 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 73 6. Security Considerations . . . . . . . . . . . . . . . . . . . 18 74 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 75 7.1. Normative References . . . . . . . . . . . . . . . . . . 18 76 7.2. Informative References . . . . . . . . . . . . . . . . . 19 77 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 79 1. Introduction 81 3GPP introduces Evolved Packet Core (EPC) that is fully IP based 82 mobile system for LTE and -advanced in their Release-8 specification 83 and beyond. Operators are now deploying EPC for LTE services and 84 encounter rapid LTE traffic growth. There are various activities to 85 offload mobile traffic in 3GPP and IETF such as LIPA, SIPTO and DMM. 86 The concept is similar that traffic of OTT (Over The Top) application 87 is offloaded at entity that is closer to the mobile node (ex. eNodeB 88 or closer anchor). 90 Likewise, overload of signaling (control plane) is also increasing 91 day by day. Network operators expect recent innovation and trends of 92 NFV (Network Function Virtualization) to solve this overloaded 93 control plane. NFV is discussed at the ETSI NFV ISG and is 94 introduced in [NFV-WHITEPAPER]. Mobile operator's network is built 95 with variety of proprietary hardware appliances today. If we can get 96 rid of these physical appliances and could shift to a cloud-based 97 service, we will have a lot of benefits explained in the next 98 section. This document assumes that NFV will push networking 99 functions currently run on dedicated hardware onto a cloud network. 100 Expected network functions are Mobility Management Entity (MME), 101 Serving Gateway (SGW) PDN Gateway(PGW), etc. With NFV, EPC can be 102 operated onto servers/hyper-visors. We name it virtualized-EPC 103 (vEPC) in this document. 105 This document uses a lot of 3GPP specific terms. These terms can be 106 found mostly at [RFC6459]. 108 1.1. The Benefits of NFV 110 This section briefly explains the benefits of NFV. The detailed 111 benefits can be found in [NFV-WHITEPAPER]. Although today's eco- 112 system of EPC appliances might be affected, we believe there are 113 various approaches to enhance current eco-system and migrate to new 114 NFV approaches. For example, operators could pay monthly recurring 115 charges for the NFV services and operations to vendors, instead of 116 one-time purchase and a little maintenance cost. 118 o [Flexible Network Operations]: The control functions of EPC are no 119 longer in appliances deployed widely in operator's network and can 120 be run at hypervisor (cloud). It is easier to add and/ or delete 121 functions from the services, because no physical construction is 122 needed. Network operations will be much simpler and easier 123 because complications of today's network are pushed to NFV (i.e. 124 hypervisor). 126 o [Flexible Resource Managements]: The EPC functions can be run on 127 hypervisor and are now less dependent on proprietary hardware. 128 Adding additional resources is easier in hypervisor, while adding 129 or replacing physical appliances require installation, 130 construction, configuration, and even migration plan without 131 service cutoff. A hypervisor can be also shared across various 132 functions such as PGW, SGW and MME. NFV also brings multi-tenancy 133 and allows a single platform for different services and users. 134 The operator can optimize resources and costs to share a NFV 135 platform for multiple customers (ex. MVNO customers) and services 136 (ex. multiple APNs). 138 o [Faster Speed of Time to Market]: When an operator wants a new 139 function to its network and services, the operator needs to 140 negotiate appliance vendors to implement the new functions or to 141 find alternative equipment supporting the new function. It takes 142 a longer time to convince the vendors, or to replace existing 143 hardware. However, if functions can be implemented as a software, 144 it is much faster to implement the functions on NFV. Even the 145 operator may implement them and try the new functions by 146 themselves. Field trial is also getting easier because of no 147 physical installation or replacement. You may turn on a new 148 function in NFV and observe how the new function behaves in your 149 network. NFV can save preparation time and tuning time of the new 150 function. 152 o [Cost Optimization]: Last but not least, Cost is the most 153 important motivation for operators to realize NFV. Operators can 154 remove many of proprietary appliances from its network and replace 155 them with industry standard servers, switches and routers. In 156 addition, it is easy to scale up and down operator's services so 157 that resources can be always tuned to the size of services. In 158 addition, operational costs led by any physical hardware such as 159 power supply, maintenance, installation, construction and 160 replacement can be minimized or even removed. The network design 161 can be simpler, because complicated functions could be handled by 162 NFV. That simple operation may enable automatic configurations 163 and prevent unnecessary trouble-shooting. As a result, CAPEX and 164 OPEX can be always optimized and lowered. 166 2. Motivations and Requirements, - Why IETF? - 168 2.1. Motivations 170 What is a role of IETF to realize vEPC in the future? IETF is not 171 the right place to discuss, for instance, how to run MME on 172 hypervisor. An important IETF activity must be to decouple the 173 control- and user- planes of mobility protocols used in EPC.The 174 motivation of decoupling the user and control plane is discussed in 175 [I-D.wakikawa-req-mobile-cp-separation]. In doing so, NFV-enabled 176 solutions can be easily designed and implemented with 177 interoperability across multiple vendors and platforms. Otherwise, 178 NFV solutions can be easily fragmented due to many proprietary 179 solutions for the protocol separations. As stated in 180 [NFV-WHITEPAPER], interoperability is highly important. 182 In the past, IETF has developed tunnel based mechanisms for mobile 183 nodes such as Mobile IPv6 [RFC6275][RFC5555], Proxy Mobile IPv6 184 [RFC5213][RFC5844] and NEMO [RFC3963]. Similarly, 3GPP has developed 185 tunnel protocols called GPRS Tunneling Protocol (GTP). These tunnel- 186 based protocols establish a data path for a mobile node between the 187 mobile node and an anchor point (s). There is a case where an access 188 router terminates a tunnel instead of a mobile node (ex. Proxy 189 Mobile IP). In 3GPP, a tunnel is established between SGW and PGW per 190 a mobile node by either Proxy Mobile IPv6 or GTP. The control and 191 the user planes of these mobility protocols are tightly related and 192 cannot be decoupled. The signaling like Binding Update and user's 193 packets are routed along a same path in EPC. It might be necessary 194 to extend these mobility protocols for the user- and control- planes 195 separation. The protocol separation of Mobile IP is discussed in 196 [I-D.yokota-dmm-scenario]. 198 Alternatively, if vEPC was realized, we should have an opportunity to 199 re-visit the basic architecture of mobility system. Instead of 200 tunneling packets on today's EPC, why can't we just route packets to 201 a mobile node? Since a role of the user plane is "routing", BGP and 202 other routing protocols could be used to forward UE's traffic. This 203 document introduces a BGP-based solution. Software Defined 204 Networking (SDN) can be an alternative solution. Open Flow and other 205 relevant protocols can setup the forward path dynamically according 206 to UE's states available in the control plane. 208 We have to remember that there is a good reason of adapting tunneling 209 in Mobile IP based solutions, that is global mobility and signaling. 210 A mobile node should be able to move anywhere on the Internet and be 211 reachable from anyone on the Internet. There were routing based 212 global mobility solutions like Boeing global mobility [Boeing-BGP] 213 and WINMO [RFC6301]. In these proposals, BGP was used to propagate 214 forwarding information of mobile nodes to the Internet. Whenever a 215 mobile node changes its point of attachment, the route must be 216 updated. Due to scalability and stability issues of the Internet, 217 this solution was not recommended by IETF [Boeing-BGP]. However, as 218 Boeing showed, it is doable to support global mobility by using BGP 219 routing update. If scalability is not your concern, a routing based 220 approach becomes a candidate of the mobility solution. 222 While global mobility is important, the "reality" is that your cell 223 phones (i.e. UE/mobile node) are moving just within an operator's 224 network and fully controlled in your local EPC. If mobility is 225 limited within an operator, we believe a routing based approach is 226 feasible and practical for today's mobile system. Instead of 227 dedicated proprietary equipment like SGW and PGW to manage a tunnel 228 path for a mobile node, multiple industry standard routers and 229 switches are configured in the user plane. These switches and 230 routers receive mobile nodes' forwarding information from the control 231 plane of vEPC by routing update. 233 2.2. Requirements 235 Requirements of our stateless user plane for vEPC are followings. 237 NFV Support 238 The future EPC architecture must support NFV capability. The 239 control plane of EPC operated on NFV framework is named 240 "virtualized EPC (vEPC)" in this document. The control plane 241 of vEPC should keep backward compatibility with the today's 242 EPC's control plane. It means this document doesn't modify 243 the control plane at all. It only assumes software-based 244 MME, SGW, and PGW run on hypervisor. 246 Separation of Control- and User- Planes 247 Due to tight relationship of the control- and user- planes in 248 today's EPC, resource increase is always provisioned to both 249 planes at once. It prevents flexible resource arrangement 250 and introduces high capital investment and over-provisioned 251 resources to one of planes. If NFV is deployed, it is 252 expected that computing resources can be independently 253 allocated to the control planes of the vEPC in a flexible 254 manner. 256 _+---+_ _ _+---+_ _ _ 257 EPC / | S | | P | / 258 Control-plane / | G | | G | / 259 /_ _| W |_ _ _| W |__/ 260 +---+ +---+ | | | | 261 | | | e | | | | | 262 | U | | N | _| |_ _ _| |_ _ _ 263 EPC | E | | B | / | | | | / 264 User-plane +---+ +---+ / +---+ +---+ / 265 /_ _ _Existing EPC_ _/ 267 _+---+_ _ _+---+_ _ _ 268 vEPC / | S | | P | / 269 Control-plane / | G | | G | / 270 /_ _| W |_ _ _| W |__/ 271 | | | | 272 +---+ +---+ +---+ +---+ 273 | | | e | +----------+ 274 | U | | N | _|IP Routing|_ _ _ 275 Stateless | E | | B | / | Network | / 276 User-plane +---+ +---+ / +----------+ / 277 /_ _ _ _ _ _ _ _ _ / 279 NFV enabled EPC architecture 281 Figure 1 283 Figure 1 shows a possibility that the entities of EPC 284 Control- plane are virtualized in generic cloud environment, 285 however user packets won't go through those virtualized EPC 286 nodes. Decoupling User-plane from the Control-plane entities 287 will be made virtualized Control-plane nodes relax hyper- 288 visor data- path capacity requirements. On the other hand, 289 decoupled User-plane into IP routing network will be agnostic 290 from sessions and bearers states, of which are generated and 291 maintained in the Control-plane. In terms of IP routing, 292 forwarding packets through the networks is based on the 293 destination address of the packets evaluated with network 294 reachable information in the routing table that accommodated 295 in the routing nodes. To forward EPC User-plane packets 296 correctly, those states must be indicated by network 297 reachable information. 299 Flat Design for Distributed Operation 300 Today's 3GPP architecture introduces PDN gateway (PGW) as a 301 gateway to external networks like the Internet. PGW manages 302 all traffic from and to UEs and could be a bottleneck and 303 single point of failure of network connectivity. In 304 addition, due to recent rapid traffic increase, it is 305 important to perform traffic engineering and to offload 306 traffic to multiple locations (ex. SGW, PGW, eNodeB). For 307 enhancements of traffic engineering capability, more flat 308 design with multiple gateways is expected so that traffic can 309 be distributed to all these gateways. There were proposals 310 how to enable flat design to (Proxy) Mobile IP such as 311 [I-D.wakikawa-mext-haha-interop2008] in IETF. Distributed 312 Mobility Management (DMM) Working Group has also discussed 313 how to extend Mobile IP-based solutions to support traffic 314 distribution in an optimal way by removing centrally deployed 315 anchors that is like a Home Agent. 317 Stateless in User Plane 318 Ultimate goal of vEPC is to remove all mobility specific 319 states from the forwarding nodes in the user-plane of vEPC. 320 If we succeed in this, industry standard routers and switches 321 can be used to forward mobile nodes traffic in the user plane 322 of vEPC. A mobile node's specific states are kept in both an 323 IP header of the mobile node's packets and a routing entry of 324 the mobile node. The detail is described in Section 3.2 326 3. Stateless user-plane architecture for virtualized EPC 328 This section explains our solution that is the stateless user-plane 329 architecture for vEPC. This solution is basically a combination of 330 existing protocols defined in IETF. A minor extension might be 331 needed but it should be easily addressed in IETF. We first introduce 332 our architecture and then protocol overview. 334 3.1. Architecture Overview 336 Figure 2 shows the user plane of the current EPC architecture. A 337 tunnel is established between SGW and PGW by either Proxy Mobile IP 338 or GTP. PGW is an anchor point of UE for incoming packets. All the 339 packet destined to UE is routed first to PGW. The UE's packets are 340 intercepted by PGW and tunneled to SGW. SGW then forwards the packet 341 to UE via access points (i.e. eNodeB) over Radio Area Network (RAN). 343 .--. 344 _( `)_ 345 _( `) 346 +---+ +---+ ( EPC-NW ) +---+ 347 [UE]~~~~|RAN|====|SGW|===================|PGW| -> the Internet 348 +---+ +---+ `--(______)---' +---+ 349 <--------Tunneling------> 351 ~~~ Wireless Link 352 === GTP Tunnel (PMIP tunnel) 353 --- IP link 355 User plane of the current EPC 357 Figure 2 359 Figure 3 is our proposed user plane of vEPC. The control plane is 360 not shown in this figure. 362 [EPC-E anycast address] 363 | .--. 364 | _( `)_ 365 | +---+ _( `) 366 +---+ \ |EPC| ( IPv6 Core ) +---+ 367 [UE]~~~~|RAN|====| -E|-( ` NW )-|RTR| --> the Internet 368 +---+ +---+ ( ` ) ) +---+ 369 (eNodeB) `--(_____)--' 370 <--------Routing--------> 372 User plane of vEPC 374 Figure 3 376 We introduce two new entities such as 378 EPC Edge Router (EPC-E) 379 EPC-E is located at the same place of today's SGW and terminates 380 GTP tunnel established with eNodeB (RAN). EPC-E supports the user 381 plane functions of SGW and PGW. EPC-E is configured an anycast 382 address to the network interface facing to eNodeB. The eNodeB 383 establishes a GTP tunnel per UE with this anycast address. Thanks 384 for anycast address, UE's traffic forwarded by eNodeB is always 385 routed to the closest EPC-E of UE. EPC-E is a router and 386 maintains routing information of every UE that is notified by the 387 control plane. Detail of routing mechanism can be found in 388 Section 3.4. 390 Router (RTR) 391 It is a regular IP router. The control plane of vEPC distributes 392 routing information of every UE by a routing protocol like BGP. 393 Therefore any additional protocols other than routing protocols 394 are not needed for RTR. Multiple RTRs can be configured anywhere 395 in the user plane of vEPC. RTRs announce UE's routing information 396 to the external network (ex. The Internet). 398 As you see in Figure 3, we omit a tunneling mechanism originally 399 established between SGW and PGW for routing UE's packets in the user 400 plane. By removing this tunnel, UE's packets are forwarded to and 401 from the Internet according to routing tables on routers in the core 402 network. Note that, although we remove the tunnel for UE's traffic 403 in the user plane, the control-plane signaling stays same in the 404 control plane. If Proxy Mobile IP is used for this tunnel, Proxy 405 Binding Update and Acknowledgment are exchanged between PGW and SGW 406 that are managed by NFV on servers/hyper-visor. Instead of a tunnel 407 setup, states created by Proxy Mobile IP are distributed to all 408 routing entities (EPC-E and RTR) by a routing protocol. From the 409 user plane point of view, these states are just seen as routing 410 entries. EPC-E and RTR are not involved in any signaling of the 411 control plane. The control plane just injects routing information to 412 EPC-E and RTR to setup routing paths to and from UEs. 414 Although this architecture just uses IPv6 core network, it supports 415 both IPv4 and IPv6 packets. The detailed operation of IPv4 support 416 will be discussed in Section 3.5. 418 3.2. Protocol Overview 420 This section gives an example of protocols used for vEPC. Figure 4 421 is the procedure of the PDN connection setup in vEPC. This figure is 422 copied from the section 3 of [RFC6459]. All the steps from (1) to 423 (13) are same as the original except for NFV-based MME, SGW, PGW, 424 HSS, and AAA. 426 The vEPC introduces two new steps, (14) and (15), to setup paths in 427 the user-plane after finishing all the signaling on the control- 428 plane. (16) and (17) are the steps to assign IP address to the mobile 429 node. 431 In (14), vEPC advertises a routing information of UE whose next hop 432 is set to GTP tunnel between eNodeB and UE in RAN. In order to 433 distribute a route entry of the UE's prefix from vEPC to all 434 EPC-E,[I-D.vandevelde-idr-remote-next-hop] enables BGP to carry the 435 tunnel information as Network Layer Reachability Information (NLRI) 436 within BGP route. In our document, we use GTP tunnel endpoint 437 information as a next-hop of the route entry of UE's prefix. The GTP 438 information is a combination of SI-U address and TEID of UE's 439 attaching eNodeB. 441 BGP has already been extended for BGP speakers to advertise tunneling 442 information to its peers [RFC5512], but [RFC5512] does not support 443 GTP as a tunnel type. Moreover, it assumes only a single tunnel 444 between a pair of BGP speakers although there are multiple tunnels 445 between RAN and vEPC in cellular system. To keep compatibility to 446 RAN architecture, it is not possible for eNodeB to act as a BGP 447 speaker. Some mechanism will thus be required to advertise and to 448 reflect tunnel endpoint routes to EPC-E instead of eNodeB. BGP 449 remote next-hop will be dealt with these issues. 451 In step (15), EPC-E can aggregate multiple UE's prefixes into less 452 BGP routes as a part of normal routing operation within operator's 453 network. 455 When tunnel endpoint is updated by UE hand-over between eNodeBs, vEPC 456 must refresh the route of UE with the updated tunnel endpoint as new 457 remote next-hop. The updated route should be immediately advertised 458 to all the EPC-Es. In the case of UE detachment, vEPC simply removes 459 the route of the detached UE. 461 vEPC(MME/SGW 462 UE eNodeB PGW/HSS/AAA) EPC-E RTR 463 | | | | | 464 |---------->|(1) | | | 465 | |---------->|(1) MME | | 466 |/---------------------\| | | 467 | Authentication |(2) AAA | | 468 |\---------------------/| | | 469 | | |--+SGW/PGW | | 470 | | | |(3)(4) |(4)SGW's TEID is configured 471 | | |<-+ | and notified to MME 472 | |<----------|(5) MME | | 473 |/---------\| | | | 474 | RB setup |(6) | | | 475 |\---------/| | | | 476 | |---------->|(7) MME | | 477 |---------->|(8) | | | 478 | |---------->|(9) MME |(9)eNodeB TEID is configured 479 | | | | and sent to MME 480 |= = = = = = = Uplink Data = = = = =>= = = = ==>|(10) 481 | The uplink is not yet built here | 482 | | | | | 483 | | |--+SGW/MME | | 484 | | | |(11,12) | | 485 | | |<-+ | | 486 |<= = = = = = Downlink Data = = = = <= = = = = =|(13) 487 | The downlink is not yet built here | 488 | | | | | 489 | | |---------->|(14) Route Update 490 | | | [Dst: UE-prefix, 491 | | | NxtHop: S1-U addr and TEID of enodeB 492 | | | | | 493 | | | |---------->|(15)Route update 494 | | | |[Dst: UE-prefix, 495 | | | | NxtHop: EPC-E address] 496 | | | | | 497 |---------RS or DHCP Request------->|(16) | 498 |<--------RA or DHCP Reply----------|(17) | 500 Extended PDN Connection Setup Procedure (copied Figure 8 of RFC6459) 502 Figure 4 504 UE requests an IPv6 prefix for its address assignment in the step 505 (16). In our architecture, an IPv6 prefix is still assigned by vEPC 506 in the control plane, as PDN-GW does in the legacy EPC. However, 507 EPC-E is responsible to deliver the IPv6 prefix to UE by DHCP or 508 Stateless address autoconfiguration (SLAAC). 510 We now explain how EPC-E can know the prefix assigned to UE from vEPC 511 for address configuration steps (16 and 17). When (1) to (15) are 512 completed, vEPC has already advertised the UE's prefix as route 513 information to all the EPC-E. Therefore, when EPC-E receives a 514 packet of either Router Solicitation or DHCPv6 request message, it 515 just looks up the remote next-hop field of its routing information 516 base (RIB) with the source IP address and the TEID of the received 517 packet. A route entry matched for this search is the prefix 518 delegated to the requesting UE. Therefore, EPC-E simply uses the 519 prefix of the route entry as an assigned UE's prefix. 521 In (17), EPC-E returns the found prefix to UE by either Router 522 Advertisement or DHCPv6 reply message. UE now creates an address(es) 523 from the received prefix. It is important to highlight that UE can 524 obtain the same prefix information from any EPC-E all the time 525 because the same UE's route information is available on all the 526 EPC-E. 528 It would be convenient to use automatic UE's prefix creation rule or 529 algorithm for vEPC. There are various mechanisms to create UE's 530 prefix. As an example, Stateless IPv6 Prefix Delegation 531 [I-D.savolainen-stateless-pd] is introduced as an algorithm to create 532 UE's prefix in vEPC below. It important to mention that our 533 architecture of the stateless user plane does not rely on any 534 particular prefix creation mechanisms like 535 [I-D.savolainen-stateless-pd] and can be run with any of them. 537 In the case of an UE's prefix length is equal, or shorter than /64, 538 the generated prefix is consisted as shown in Figure 5. Each PDN is 539 assumed to have single or several prefixes (named PDN prefix) used to 540 generate UE's address. Followed by the PDN prefix, there is TEID 541 field assigned for a UE's session on S1-U interface of vEPC. TEID is 542 32 bits identifier in GTP header to distinguish each bearer. The 543 remaining bits are filled by subnet ID. 545 | n bits | o bits (=< 32)|64-n-o bits| 546 +---------------------------+---------------+-----------+ 547 | PDN Prefix | TEID | subnet ID | 548 +---------------------------+---------------+-----------+ 549 <---------------------------64bits----------------------> 551 Stateless-pd Prefix 553 Figure 5 555 3.3. Control-plane awareness of stateless user-plane 557 Nodes in the control-plane in vEPC must be aware that the anycast 558 address assigned to EPC-E is a S1-U address of vEPC. The vEPC must 559 use the anycast address in signaling between vEPC and RAN. By doing 560 this, packets from RAN are correctly forwarded to an appropriate 561 EPC-E. Due to anycast nature, it means there is no hand-off procedure 562 between SGWs because all eNodeB in the RAN send packets to the same 563 anycast address. 565 When an operator needs to increase virtualized instances to cope with 566 just signaling overload, the operator should use the existing S1-U 567 address (i.e. EPC-E anycast address) for the new instances. If the 568 operator would increase the capacity of the user plane, it can add 569 additional EPC-Es in the core network. The operator can group the 570 new EPC-Es as a set and increase scalability and performance of the 571 user plane. In this case, the operator uses a new anycast address to 572 the new set of EPC-E. We will discuss operational consideration in 573 detail in Section 4. 575 3.4. Routing mechanism 577 Figure 6 shows a packet forwarding mechanism of our stateless user 578 plane. As an example, there are four eNodeB (illustrated as eNB-x) , 579 three EPC-Edge routers(shown as EPC-Ex) and two routers (RTRx) in 580 Figure 6. UE is first connected to eNB-C and then moves to eNB-D. 581 The UE at the new location is illustrated as UE'. Routing entry for 582 UE is also illustrated at the right side in Figure 6. 584 EPC-E has two interfaces facing either RAN or CORE networks. An 585 anycast address (shown as X) is configured to the interface facing 586 RAN of all EPC-E. EPC-E assigns an individual IPv6 address to 587 another interface (illustrated "a" to "d" in the figure). It is 588 important to mention that the anycast address X can be treated as the 589 SGW's S1-U address. 591 Since RTRs are a gateway to the Internet, they advertise routes of an 592 operator's prefix to the Internet. After one of RTR receives a 593 packet of UE from the Internet, it needs to routing it to UE in the 594 user plane. RTR has a simple routing entry for PDN prefix whose next 595 hop points to the EPC-E. One of RTR (let's say RTR2 in this case) 596 looks up a routing table with UE's address and matched it with a 597 routing entry of PDN prefix. Since multiple EPC-Es advertise a route 598 for the same PDN prefix, RTR2 should forward the packet to one of 599 EPC-E according to the routing entry. This routing is known as hot- 600 potato routing. In this example, the RTR2 uses EPC-E2-b as a nexthop 601 of PDN prefix. 603 When the UE's packet is arrived at EPC-E2, EPC-E2 needs to forwards 604 them to UE via eNodeB to which UE is connecting by using GTP tunnel. 605 For this operation, EPC-E2 has a routing entry that destination is 606 UE's prefix and that next hop points to GTP tunnel between eNB-C and 607 the EPC-Es. In order to identify the GTP tunnel for UE, EPC-E needs 608 S1-U address and Tunnel Endpoint ID (TEID) of eNB-C that is eNB-C-3 609 in Figure 6. The eNB-C TEID for UE is illustrated as TEID[eNB-C]. 610 The SGW assigned TEID is utilized to generate the UE's prefix as we 611 explained in Section 3.2. These TEID are assigned per UE. The TEID 612 and S1-U address of eNodeB are retrieved from the next hop field of 613 the routing entry of the mobile node. By using the GTP information, 614 every EPC-E can now forward the UE's packets to right eNodeB. 616 Routing outgoing packets from UE is much simpler. The packets from 617 UE are always routed to the closest EPC-E to UE because of anycast 618 routing. In Figure 6, when UE sends a packet to a destination, the 619 packet is reached to eNB-C and tunneled to EPC-E's anycast address. 620 The GTP-tunneled packet is routed to the closest EPC-E that is EPC-E2 621 in this case. The packet is decapsulated by EPC-E2 and then 622 forwarded to one of RTR according to the routing table. Since the 623 decapsulated packet is regular IPv6 packet, no extra control other 624 than routing is necessary. 626 When UE moves to a new location (UE'), it updates its location on the 627 control plane. After signaling completion for location update, vEPC 628 needs to update the UE's routing entry of all EPC-E so that vEPC 629 advertises updated route with new location to all EPC-Es by a routing 630 protocol. The routing entry should be updated with the new eNodeB's 631 address that is eNB-D-4. During handover, there might be some 632 traffic arriving to the older eNodeB (eNB-C). These packets can be 633 re-routed to the new eNodeB (eNB-D) via X2-U interface in RAN. 635 The UE's address isn't changed when UE changes its attachment. In 636 our scenario, SGW run on hypervisor and is independent from network 637 topology. Therefore, logically we don't have handover across 638 different SGWs. UE can stay connected with the same SGW all the time 639 and can keep using the same TEID after handover. Thus, UE's address 640 is unchanged even after handover. 642 +--------------------+ 643 | Correspondent Node | 644 +--------------------+ 645 | 646 .--. [Route on the Internet] 647 _(. `) Destination: Operator's Prefix 648 _( `)_ NextHop: RTR-1/2 649 ( Internet `) 650 ( ` . ) ) 651 `--(_______)---' 652 / \ [Route at RTR] 653 +----+ +----+ Destination: PDN Prefix 654 |RTR1| |RTR2| NextHop: EPC-E2-b 655 +----+ +----+ 656 \ .--. / 657 _( IP `. 658 ( CORE NW ) 659 ___( ` . ) )__ 660 / `--(___.--' \ 661 / | \ 662 |a |b |c [Route at EPC-E] 663 +---+--+ +---+--+ +---+--+ Destination: UE's Prefix *1 664 |EPC-E1| |EPC-E2| |EPC-E3| NextHop: GTP tunnel (eNB-C) *2 665 +---+--+ +---+--+ +---+--+ 666 |X |X |X 667 \ .--. / 668 \_____( RAN `.____/ 669 ( ACCESS ) 670 ____( ` NW ) )_______ 671 / `--(___.--' \ 672 / | | \ 673 |1 |2 |3 |4 674 +--+--+ +--+--+ +--+--+ +--+--+ 675 |eNB-A| |eNB-B| |eNB-C| |eNB-D| 676 +--+--+ +--+--+ +--+--+ +--+--+ 677 : : : : 678 UE --> UE' 679 *1 TEID used at EPC-E for the UE is included in this UE's prefix. 680 see Figure 4. 681 *2 GTP tunnel state is stored in the next hop field. The state 682 information is the combination of eNB-C S1 address that is 683 eNB-C-3 and TEID(eNB-C) assigned for the UE. 685 Routing Mechanism Overview 687 Figure 6 689 3.5. IPv4 Support 691 Recent IPv6 transition mechanisms enable IPv6-only network to forward 692 IPv4 packet with encapsulation or translation techniques. By using 693 one of mechanisms, we can use IPv6 for our stateless user-plane 694 network for transporting both IPv4 and IPv6 packets. Figure 7 shows 695 available solutions of IPv4 support for each bearer type to deal with 696 that requirement. 698 Bearer UE EPC-E Gateway 699 type function function function 700 -------------------------------------------- 701 IPv4 - B4 AFTR 702 IPv4 - CLAT PLAT 703 IPv6 MAP-CE - MAP-BR 704 IPv6 B4 - AFTR 705 IPv6 CLAT - PLAT 707 Solutions and functions for IPv4 support 709 Figure 7 711 In the case of a UE only support IPv4 bearer, B4 function of DS-Lite 712 [RFC6333] or CLAT function of 464XLAT [RFC6877] may be implemented in 713 a EPC-E. Both functions are stateless therefore EPC-E isn't required 714 to maintain any tunneling or translation state. 716 Figure 8 shows how to support IPv4 on IPv6 core network in our vEPC. 717 Instead of using RqTR as a gateway to the Internet, DS-LITE AFTR or 718 464XLAT PLAT is installed as a gateway to the IPv4 Internet. 720 .--. 721 EPC-E _( `)_ Gateway 722 +----+ _( `) +------+ 723 | B4 | ( IPv6 Core ) | AFTR | 724 [UE]==IPv4-only==| or |-( ` NW )-| or | --> Internet 725 bearer |CLAT| ( ` ) ) | PLAT | (IPv4) 726 +----+ `--(_____)--' +------+ 727 <-----IPv4 over IPv6-----> 728 or 729 v4v6 translation 731 IPv4 User plane of vEPC 733 Figure 8 735 If UE supports IPv6 capable bearer, IPv6 transition function may be 736 implemented in the UE such as MAP-CE [I-D.ietf-softwire-map], B4 or 737 CLAT. That means an EPC-E receives IPv6 packets from UE in this case 738 so that the EPC-E does not need to be involved in the part of IPv4 739 support functions. 741 4. Operational Considerations 743 4.1. Scalability 745 Virtualization allows vEPC to be elastic for steep demand of requests 746 to create and update for sessions. In our architecture, that makes 747 routing update fluctuation from vEPC to EPC-E. This is the reason why 748 we select BGP as a protocol between vEPC and EPC-E. BGP is scalable 749 and stable routing protocol today. 751 BGP is an incremental update protocol so that once BGP peer 752 established, millions of routes can be easily updated in stable 753 manners. Operators can appropriately design BGP peering between vEPC 754 and ECP-E to secure convergence time within appropriate period. 756 Granularity of the peering should be aware EPC-E capacity because it 757 is assumed that EPC-E has upper limit of routing entries. BGP 758 peering design should makes sure that total number of routes does not 759 exceed EPC-E capacity. 761 During the network planning, operators must understand EPC-E's 762 capacity such as # of routes, bandwidth, etc. An of example of 763 estimation, if a EPC-E has 1Gbps throughput and each UE's bandwidth 764 consumption is 10Kbps in average, the EPC-E should have 100K routes 765 capacity. 767 This is an operational approach to minimize the risk of routing 768 update fluctuation. If it is hard to support all the UEs by a single 769 set of EPC-E in an operators network, different set of EPC-E can be 770 introduced and configured in a network. The UEs are distributed to 771 one of the set and handled by the EPC-Es in the set. We don't need 772 to support millions of UEs by a single set of EPC-E. This is another 773 advantage of using routing mechanism in the user plane. We already 774 explain how to handle different set of EPC-E in our scheme in 775 Section 3.3. 777 The notion of multiple EPC-E sets is easily fitted into our today's' 778 network. The operator's network is often separated into several 779 regional network for geographical scalability. Therefore, the 780 operator can assign different EPC-E set to different region for 781 better scalability. 783 In addition, routers and EPC-E in the IPv6 core network are required 784 to process just "route", they naturally aggregate those routing 785 entries. It helps limiting the total number of routing entries in 786 our core network. 788 4.2. Backward Compatibility 790 vEPC should be able to fall back to the legacy EPC based packet 791 forwarding to secure backward compatibility which is required to 792 connect existing system, or to connect roaming partners through 793 legacy S5/S8 interfaces. When fallback happened, all the packets are 794 not routed on our stateless user plane, but forwarded to vEPC (i.e. 795 SGW and PGW instances on hypervisor). vEPC must use a S1-U address 796 that is different from anycast address assigned to EPC-Es. This 797 address is assigned to SGW instances in vEPC and used to terminate 798 tunnels in vEPC servers (i.e. hypervisor). 800 5. IANA Considerations 802 This memo includes no request to IANA. 804 6. Security Considerations 806 There are no security considerations specific to this document at 807 this moment. 809 7. References 811 7.1. Normative References 813 [I-D.vandevelde-idr-remote-next-hop] 814 Velde, G., Patel, K., Rao, D., Raszuk, R., and R. Bush, 815 "BGP Remote-Next-Hop", draft-vandevelde-idr-remote-next- 816 hop-03 (work in progress), October 2012. 818 [RFC5512] Mohapatra, P. and E. Rosen, "The BGP Encapsulation 819 Subsequent Address Family Identifier (SAFI) and the BGP 820 Tunnel Encapsulation Attribute", RFC 5512, April 2009. 822 7.2. Informative References 824 [Boeing-BGP] 825 Andrew, ., "Global IP Network Mobility using Border 826 Gateway Protocol (BGP)", IAB Plenary IAB Plenary of IETF 827 62nd, March 2005. 829 [I-D.ietf-softwire-map] 830 Troan, O., Dec, W., Li, X., Bao, C., Matsushima, S., 831 Murakami, T., and T. Taylor, "Mapping of Address and Port 832 with Encapsulation (MAP)", draft-ietf-softwire-map-07 833 (work in progress), May 2013. 835 [I-D.savolainen-stateless-pd] 836 Savolainen, T. and J. Korhonen, "Stateless IPv6 Prefix 837 Delegation for IPv6 enabled networks", draft-savolainen- 838 stateless-pd-01 (work in progress), February 2010. 840 [I-D.wakikawa-mext-haha-interop2008] 841 Wakikawa, R., Shima, K., and N. Shigechika, "The Global 842 HAHA Operation at the Interop Tokyo 2008", draft-wakikawa- 843 mext-haha-interop2008-00 (work in progress), July 2008. 845 [I-D.wakikawa-req-mobile-cp-separation] 846 Wakikawa, R., Matsushima, S., Patil, B., Chen, B., DJ, D., 847 and H. Deng, "Requirements and use cases for separating 848 control and user planes in mobile network architectures", 849 draft-wakikawa-req-mobile-cp-separation-00 (work in 850 progress), November 2013. 852 [I-D.yokota-dmm-scenario] 853 Yokota, H., Seite, P., Demaria, E., and Z. Cao, "Use case 854 scenarios for Distributed Mobility Management", draft- 855 yokota-dmm-scenario-00 (work in progress), October 2010. 857 [NFV-WHITEPAPER] 858 Network Operators, ., "Network Functions Virtualization, 859 An Introduction, Benefits, Enablers, Challenges and Call 860 for Action", SDN and OpenFlow SDN and OpenFlow World 861 Congress, October 2012. 863 [RFC3963] Devarapalli, V., Wakikawa, R., Petrescu, A., and P. 864 Thubert, "Network Mobility (NEMO) Basic Support Protocol", 865 RFC 3963, January 2005. 867 [RFC5213] Gundavelli, S., Leung, K., Devarapalli, V., Chowdhury, K., 868 and B. Patil, "Proxy Mobile IPv6", RFC 5213, August 2008. 870 [RFC5555] Soliman, H., "Mobile IPv6 Support for Dual Stack Hosts and 871 Routers", RFC 5555, June 2009. 873 [RFC5844] Wakikawa, R. and S. Gundavelli, "IPv4 Support for Proxy 874 Mobile IPv6", RFC 5844, May 2010. 876 [RFC6275] Perkins, C., Johnson, D., and J. Arkko, "Mobility Support 877 in IPv6", RFC 6275, July 2011. 879 [RFC6301] Zhu, Z., Wakikawa, R., and L. Zhang, "A Survey of Mobility 880 Support in the Internet", RFC 6301, July 2011. 882 [RFC6333] Durand, A., Droms, R., Woodyatt, J., and Y. Lee, "Dual- 883 Stack Lite Broadband Deployments Following IPv4 884 Exhaustion", RFC 6333, August 2011. 886 [RFC6459] Korhonen, J., Soininen, J., Patil, B., Savolainen, T., 887 Bajko, G., and K. Iisakkila, "IPv6 in 3rd Generation 888 Partnership Project (3GPP) Evolved Packet System (EPS)", 889 RFC 6459, January 2012. 891 [RFC6877] Mawatari, M., Kawashima, M., and C. Byrne, "464XLAT: 892 Combination of Stateful and Stateless Translation", RFC 893 6877, April 2013. 895 Authors' Addresses 897 Satoru Matsushima 898 Softbank Telecom 899 1-9-1,Higashi-Shimbashi,Minato-Ku 900 Tokyo 105-7322 901 Japan 903 Email: satoru.matsushima@g.softbank.co.jp 905 Ryuji Wakikawa 906 Softbank Mobile 907 1-9-1,Higashi-Shimbashi,Minato-Ku 908 Tokyo 105-7322 909 Japan 911 Email: ryuji.wakikawa@gmail.com