idnits 2.17.1 draft-matsushima-stateless-uplane-vepc-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 696 has weird spacing: '... type fun...' -- The document date (July 16, 2013) is 3929 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'I-D.vandevelde-idr-remote-next-hop' is defined on line 810, but no explicit reference was found in the text == Outdated reference: A later version (-09) exists of draft-vandevelde-idr-remote-next-hop-03 ** Obsolete normative reference: RFC 5512 (Obsoleted by RFC 9012) == Outdated reference: A later version (-13) exists of draft-ietf-softwire-map-07 Summary: 1 error (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Group S. Matsushima 3 Internet-Draft Softbank Telecom 4 Intended status: Informational R. Wakikawa 5 Expires: January 17, 2014 Softbank Mobile 6 July 16, 2013 8 Stateless user-plane architecture for virtualized EPC (vEPC) 9 draft-matsushima-stateless-uplane-vepc-01 11 Abstract 13 We envision a new mobile architecture for the future Evolved Packet 14 Core (EPC). The new architecture is designed to support the 15 virtualization scheme called NFV (Network Function Virtualization). 16 In our architecture, the user plane of EPC is decoupled from the 17 control-plane and uses routing information to forward packets of 18 mobile nodes. Although the EPC control plane will run on hypervisor, 19 our proposal does not modify the signaling of the EPC control plane. 20 The benefits of our architecture are 1) scalability, 2) flexibility 21 and 3) Manageability. How to run the EPC control plane on NFV is out 22 of our focus in this document. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on January 17, 2014. 41 Copyright Notice 43 Copyright (c) 2013 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 59 1.1. The Benefits of NFV . . . . . . . . . . . . . . . . . . . 3 60 2. Motivations and Requirements, - Why IETF? - . . . . . . . . . 4 61 2.1. Motivations . . . . . . . . . . . . . . . . . . . . . . . 4 62 2.2. Requirements . . . . . . . . . . . . . . . . . . . . . . 5 63 3. Stateless user-plane architecture for virtualized EPC . . . 7 64 3.1. Architecture Overview . . . . . . . . . . . . . . . . . . 8 65 3.2. Protocol Overview . . . . . . . . . . . . . . . . . . . . 9 66 3.3. Control-plane awareness of stateless user-plane . . . . . 13 67 3.4. Routing mechanism . . . . . . . . . . . . . . . . . . . . 13 68 3.5. IPv4 Support . . . . . . . . . . . . . . . . . . . . . . 16 69 4. Operational Considerations . . . . . . . . . . . . . . . . . 17 70 4.1. Scalability . . . . . . . . . . . . . . . . . . . . . . . 17 71 4.2. Backward Compatibility . . . . . . . . . . . . . . . . . 18 72 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 73 6. Security Considerations . . . . . . . . . . . . . . . . . . . 18 74 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 75 7.1. Normative References . . . . . . . . . . . . . . . . . . 18 76 7.2. Informative References . . . . . . . . . . . . . . . . . 19 77 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 79 1. Introduction 81 3GPP introduces Evolved Packet Core (EPC) that is fully IP based 82 mobile system for LTE and -advanced in their Release-8 specification 83 and beyond. Operators are now deploying EPC for LTE services and 84 encounter rapid LTE traffic growth. There are various activities to 85 offload mobile traffic in 3GPP and IETF such as LIPA, SIPTO and DMM. 86 The concept is similar that traffic of OTT (Over The Top) application 87 is offloaded at entity that is closer to the mobile node (ex. eNodeB 88 or closer anchor). 90 Likewise, overload of signaling (control plane) is also increasing 91 day by day. Network operators expect recent innovation and trends of 92 NFV (Network Function Virtualization) to solve this overloaded 93 control plane. NFV is discussed at the ETSI NFV ISG and is 94 introduced in [NFV-WHITEPAPER]. Mobile operator's network is built 95 with variety of proprietary hardware appliances today. If we can get 96 rid of these physical appliances and could shift to a cloud-based 97 service, we will have a lot of benefits explained in the next 98 section. This document assumes that NFV will push networking 99 functions currently run on dedicated hardware onto a cloud network. 100 Expected network functions are Mobility Management Entity (MME), 101 Serving Gateway (SGW) PDN Gateway(PGW), etc. With NFV, EPC can be 102 operated onto servers/hyper-visors. We name it virtualized-EPC 103 (vEPC) in this document. 105 This document uses a lot of 3GPP specific terms. These terms can be 106 found mostly at [RFC6459]. 108 1.1. The Benefits of NFV 110 This section briefly explains the benefits of NFV. The detailed 111 benefits can be found in [NFV-WHITEPAPER]. Although today's eco- 112 system of EPC appliances might be affected, we believe there are 113 various approaches to enhance current eco-system and migrate to new 114 NFV approaches. For example, operators could pay monthly recurring 115 charges for the NFV services and operations to vendors, instead of 116 one-time purchase and a little maintenance cost. 118 o [Flexible Network Operations]: The control functions of EPC are no 119 longer in appliances deployed widely in operator's network and can 120 be run at hypervisor (cloud). It is easier to add and/ or delete 121 functions from the services, because no physical construction is 122 needed. Network operations will be much simpler and easier 123 because complications of today's network are pushed to NFV (i.e. 124 hypervisor). 126 o [Flexible Resource Managements]: The EPC functions can be run on 127 hypervisor and are now less dependent on proprietary hardware. 128 Adding additional resources is easier in hypervisor, while adding 129 or replacing physical appliances require installation, 130 construction, configuration, and even migration plan without 131 service cutoff. A hypervisor can be also shared across various 132 functions such as PGW, SGW and MME. NFV also brings multi-tenancy 133 and allows a single platform for different services and users. 134 The operator can optimize resources and costs to share a NFV 135 platform for multiple customers (ex. MVNO customers) and services 136 (ex. multiple APNs). 138 o [Faster Speed of Time to Market]: When an operator wants a new 139 function to its network and services, the operator needs to 140 negotiate appliance vendors to implement the new functions or to 141 find alternative equipment supporting the new function. It takes 142 a longer time to convince the vendors, or to replace existing 143 hardware. However, if functions can be implemented as a software, 144 it is much faster to implement the functions on NFV. Even the 145 operator may implement them and try the new functions by 146 themselves. Field trial is also getting easier because of no 147 physical installation or replacement. You may turn on a new 148 function in NFV and observe how the new function behaves in your 149 network. NFV can save preparation time and tuning time of the new 150 function. 152 o [Cost Optimization]: Last but not least, Cost is the most 153 important motivation for operators to realize NFV. Operators can 154 remove many of proprietary appliances from its network and replace 155 them with industry standard servers, switches and routers. In 156 addition, it is easy to scale up and down operator's services so 157 that resources can be always tuned to the size of services. In 158 addition, operational costs led by any physical hardware such as 159 power supply, maintenance, installation, construction and 160 replacement can be minimized or even removed. The network design 161 can be simpler, because complicated functions could be handled by 162 NFV. That simple operation may enable automatic configurations 163 and prevent unnecessary trouble-shooting. As a result, CAPEX and 164 OPEX can be always optimized and lowered. 166 2. Motivations and Requirements, - Why IETF? - 168 2.1. Motivations 170 What is a role of IETF to realize vEPC in the future? IETF is not 171 the right place to discuss, for instance, how to run MME on 172 hypervisor. An important IETF activity must be to decouple the 173 control- and user- planes of mobility protocols used in EPC. In 174 doing so, NFV-enabled solutions can be easily designed and 175 implemented with interoperability across multiple vendors and 176 platforms. Otherwise, NFV solutions can be easily fragmented due to 177 many proprietary solutions for the protocol separations. As stated 178 in [NFV-WHITEPAPER], interoperability is highly important. 180 In the past, IETF has developed tunnel based mechanisms for mobile 181 nodes such as Mobile IPv6 [RFC6275][RFC5555], Proxy Mobile IPv6 182 [RFC5213][RFC5844] and NEMO [RFC3963]. Similarly, 3GPP has developed 183 tunnel protocols called GPRS Tunneling Protocol (GTP). These tunnel- 184 based protocols establish a data path for a mobile node between the 185 mobile node and an anchor point (s). There is a case where an access 186 router terminates a tunnel instead of a mobile node (ex. Proxy 187 Mobile IP). In 3GPP, a tunnel is established between SGW and PGW per 188 a mobile node by either Proxy Mobile IPv6 or GTP. The control and 189 the user planes of these mobility protocols are tightly related and 190 cannot be decoupled. The signaling like Binding Update and user's 191 packets are routed along a same path in EPC. It might be necessary 192 to extend these mobility protocols for the user- and control- planes 193 separation. The protocol separation of Mobile IP is discussed in 194 [I-D.yokota-dmm-scenario]. 196 Alternatively, if vEPC was realized, we should have an opportunity to 197 re-visit the basic architecture of mobility system. Instead of 198 tunneling packets on today's EPC, why can't we just route packets to 199 a mobile node? Since a role of the user plane is "routing", BGP and 200 other routing protocols could be used to forward UE's traffic. This 201 document introduces a BGP-based solution. Software Defined 202 Networking (SDN) can be an alternative solution. Open Flow and other 203 relevant protocols can setup the forward path dynamically according 204 to UE's states available in the control plane. 206 We have to remember that there is a good reason of adapting tunneling 207 in Mobile IP based solutions, that is global mobility and signaling. 208 A mobile node should be able to move anywhere on the Internet and be 209 reachable from anyone on the Internet. There were routing based 210 global mobility solutions like Boeing global mobility [Boeing-BGP] 211 and WINMO [RFC6301]. In these proposals, BGP was used to propagate 212 forwarding information of mobile nodes to the Internet. Whenever a 213 mobile node changes its point of attachment, the route must be 214 updated. Due to scalability and stability issues of the Internet, 215 this solution was not recommended by IETF [Boeing-BGP]. However, as 216 Boeing showed, it is doable to support global mobility by using BGP 217 routing update. If scalability is not your concern, a routing based 218 approach becomes a candidate of the mobility solution. 220 While global mobility is important, the "reality" is that your cell 221 phones (i.e. UE/mobile node) are moving just within an operator's 222 network and fully controlled in your local EPC. If mobility is 223 limited within an operator, we believe a routing based approach is 224 feasible and practical for today's mobile system. Instead of 225 dedicated proprietary equipment like SGW and PGW to manage a tunnel 226 path for a mobile node, multiple industry standard routers and 227 switches are configured in the user plane. These switches and 228 routers receive mobile nodes' forwarding information from the control 229 plane of vEPC by routing update. 231 2.2. Requirements 233 Requirements of our stateless user plane for vEPC are followings. 235 NFV Support 236 The future EPC architecture must support NFV capability. The 237 control plane of EPC operated on NFV framework is named 238 "virtualized EPC (vEPC)" in this document. The control plane 239 of vEPC should keep backward compatibility with the today's 240 EPC's control plane. It means this document doesn't modify 241 the control plane at all. It only assumes software-based 242 MME, SGW, and PGW run on hypervisor. 244 Separation of Control- and User- Planes 245 Due to tight relationship of the control- and user- planes in 246 today's EPC, resource increase is always provisioned to both 247 planes at once. It prevents flexible resource arrangement 248 and introduces high capital investment and over-provisioned 249 resources to one of planes. If NFV is deployed, it is 250 expected that computing resources can be independently 251 allocated to the control planes of the vEPC in a flexible 252 manner. 254 _+---+_ _ _+---+_ _ _ 255 EPC / | S | | P | / 256 Control-plane / | G | | G | / 257 /_ _| W |_ _ _| W |__/ 258 +---+ +---+ | | | | 259 | | | e | | | | | 260 | U | | N | _| |_ _ _| |_ _ _ 261 EPC | E | | B | / | | | | / 262 User-plane +---+ +---+ / +---+ +---+ / 263 /_ _ _Existing EPC_ _/ 265 _+---+_ _ _+---+_ _ _ 266 vEPC / | S | | P | / 267 Control-plane / | G | | G | / 268 /_ _| W |_ _ _| W |__/ 269 | | | | 270 +---+ +---+ +---+ +---+ 271 | | | e | +----------+ 272 | U | | N | _|IP Routing|_ _ _ 273 Stateless | E | | B | / | Network | / 274 User-plane +---+ +---+ / +----------+ / 275 /_ _ _ _ _ _ _ _ _ / 277 NFV enabled EPC architecture 279 Figure 1 281 Figure 1 shows a possibility that the entities of EPC 282 Control- plane are virtualized in generic cloud environment, 283 however user packets won't go through those virtualized EPC 284 nodes. Decoupling User-plane from the Control-plane entities 285 will be made virtualized Control-plane nodes relax hyper- 286 visor data- path capacity requirements. On the other hand, 287 decoupled User-plane into IP routing network will be agnostic 288 from sessions and bearers states, of which are generated and 289 maintained in the Control-plane. In terms of IP routing, 290 forwarding packets through the networks is based on the 291 destination address of the packets evaluated with network 292 reachable information in the routing table that accommodated 293 in the routing nodes. To forward EPC User-plane packets 294 correctly, those states must be indicated by network 295 reachable information. 297 Flat Design for Distributed Operation 298 Today's 3GPP architecture introduces PDN gateway (PGW) as a 299 gateway to external networks like the Internet. PGW manages 300 all traffic from and to UEs and could be a bottleneck and 301 single point of failure of network connectivity. In 302 addition, due to recent rapid traffic increase, it is 303 important to perform traffic engineering and to offload 304 traffic to multiple locations (ex. SGW, PGW, eNodeB). For 305 enhancements of traffic engineering capability, more flat 306 design with multiple gateways is expected so that traffic can 307 be distributed to all these gateways. There were proposals 308 how to enable flat design to (Proxy) Mobile IP such as 309 [I-D.wakikawa-mext-haha-interop2008] in IETF. Distributed 310 Mobility Management (DMM) Working Group has also discussed 311 how to extend Mobile IP-based solutions to support traffic 312 distribution in an optimal way by removing centrally deployed 313 anchors that is like a Home Agent. 315 Stateless in User Plane 316 Ultimate goal of vEPC is to remove all mobility specific 317 states from the forwarding nodes in the user-plane of vEPC. 318 If we succeed in this, industry standard routers and switches 319 can be used to forward mobile nodes traffic in the user plane 320 of vEPC. A mobile node's specific states are kept in both an 321 IP header of the mobile node's packets and a routing entry of 322 the mobile node. The detail is described in Section 3.2 324 3. Stateless user-plane architecture for virtualized EPC 326 This section explains our solution that is the stateless user-plane 327 architecture for vEPC. This solution is basically a combination of 328 existing protocols defined in IETF. A minor extension might be 329 needed but it should be easily addressed in IETF. We first introduce 330 our architecture and then protocol overview. 332 3.1. Architecture Overview 334 Figure 2 shows the user plane of the current EPC architecture. A 335 tunnel is established between SGW and PGW by either Proxy Mobile IP 336 or GTP. PGW is an anchor point of UE for incoming packets. All the 337 packet destined to UE is routed first to PGW. The UE's packets are 338 intercepted by PGW and tunneled to SGW. SGW then forwards the packet 339 to UE via access points (i.e. eNodeB) over Radio Area Network (RAN). 341 .--. 342 _( `)_ 343 _( `) 344 +---+ +---+ ( EPC-NW ) +---+ 345 [UE]~~~~|RAN|====|SGW|===================|PGW| -> the Internet 346 +---+ +---+ `--(______)---' +---+ 347 <--------Tunneling------> 349 ~~~ Wireless Link 350 === GTP Tunnel (PMIP tunnel) 351 --- IP link 353 User plane of the current EPC 355 Figure 2 357 Figure 3 is our proposed user plane of vEPC. The control plane is 358 not shown in this figure. 360 [EPC-E anycast address] 361 | .--. 362 | _( `)_ 363 | +---+ _( `) 364 +---+ \ |EPC| ( IPv6 Core ) +---+ 365 [UE]~~~~|RAN|====| -E|-( ` NW )-|RTR| --> the Internet 366 +---+ +---+ ( ` ) ) +---+ 367 (eNodeB) `--(_____)--' 368 <--------Routing--------> 370 User plane of vEPC 372 Figure 3 374 We introduce two new entities such as 376 EPC Edge Router (EPC-E) 377 EPC-E is located at the same place of today's SGW and terminates 378 GTP tunnel established with eNodeB (RAN). EPC-E supports the user 379 plane functions of SGW and PGW. EPC-E is configured an anycast 380 address to the network interface facing to eNodeB. The eNodeB 381 establishes a GTP tunnel per UE with this anycast address. Thanks 382 for anycast address, UE's traffic forwarded by eNodeB is always 383 routed to the closest EPC-E of UE. EPC-E is a router and 384 maintains routing information of every UE that is notified by the 385 control plane. Detail of routing mechanism can be found in 386 Section 3.4. 388 Router (RTR) 389 It is a regular IP router. The control plane of vEPC distributes 390 routing information of every UE by a routing protocol like BGP. 391 Therefore any additional protocols other than routing protocols 392 are not needed for RTR. Multiple RTRs can be configured anywhere 393 in the user plane of vEPC. RTRs announce UE's routing information 394 to the external network (ex. The Internet). 396 As you see in Figure 3, we omit a tunneling mechanism originally 397 established between SGW and PGW for routing UE's packets in the user 398 plane. By removing this tunnel, UE's packets are forwarded to and 399 from the Internet according to routing tables on routers in the core 400 network. Note that, although we remove the tunnel for UE's traffic 401 in the user plane, the control-plane signaling stays same in the 402 control plane. If Proxy Mobile IP is used for this tunnel, Proxy 403 Binding Update and Acknowledgment are exchanged between PGW and SGW 404 that are managed by NFV on servers/hyper-visor. Instead of a tunnel 405 setup, states created by Proxy Mobile IP are distributed to all 406 routing entities (EPC-E and RTR) by a routing protocol. From the 407 user plane point of view, these states are just seen as routing 408 entries. EPC-E and RTR are not involved in any signaling of the 409 control plane. The control plane just injects routing information to 410 EPC-E and RTR to setup routing paths to and from UEs. 412 Although this architecture just uses IPv6 core network, it supports 413 both IPv4 and IPv6 packets. The detailed operation of IPv4 support 414 will be discussed in Section 3.5. 416 3.2. Protocol Overview 417 This section gives an example of protocols used for vEPC. Figure 4 418 is the procedure of the PDN connection setup in vEPC. This figure is 419 copied from the section 3 of [RFC6459]. All the steps from (1) to 420 (13) are same as the original except for NFV-based MME, SGW, PGW, 421 HSS, and AAA. 423 The vEPC introduces two new steps, (14) and (15), to setup paths in 424 the user-plane after finishing all the signaling on the control- 425 plane. (16) and (17) are the steps to assign IP address to the mobile 426 node. 428 In (14), vEPC advertises a routing information of UE whose next hop 429 is set to GTP tunnel between eNodeB and UE in RAN. In order to 430 distribute a route entry of the UE's prefix from vEPC to all 431 EPC-E,[I-D.vandevelde-idr-remote-next-hop] enables BGP to carry the 432 tunnel information as Network Layer Reachability Information (NLRI) 433 within BGP route. In our document, we use GTP tunnel endpoint 434 information as a next-hop of the route entry of UE's prefix. The GTP 435 information is a combination of SI-U address and TEID of UE's 436 attaching eNodeB. 438 BGP has already been extended for BGP speakers to advertise tunneling 439 information to its peers [RFC5512], but [RFC5512] does not support 440 GTP as a tunnel type. Moreover, it assumes only a single tunnel 441 between a pair of BGP speakers although there are multiple tunnels 442 between RAN and vEPC in cellular system. To keep compatibility to 443 RAN architecture, it is not possible for eNodeB to act as a BGP 444 speaker. Some mechanism will thus be required to advertise and to 445 reflect tunnel endpoint routes to EPC-E instead of eNodeB. BGP 446 remote next-hop will be dealt with these issues. 448 In step (15), EPC-E can aggregate multiple UE's prefixes into less 449 BGP routes as a part of normal routing operation within operator's 450 network. 452 When tunnel endpoint is updated by UE hand-over between eNodeBs, vEPC 453 must refresh the route of UE with the updated tunnel endpoint as new 454 remote next-hop. The updated route should be immediately advertised 455 to all the EPC-Es. In the case of UE detachment, vEPC simply removes 456 the route of the detached UE. 458 vEPC(MME/SGW 459 UE eNodeB PGW/HSS/AAA) EPC-E RTR 460 | | | | | 461 |---------->|(1) | | | 462 | |---------->|(1) MME | | 463 |/---------------------\| | | 464 | Authentication |(2) AAA | | 465 |\---------------------/| | | 466 | | |--+SGW/PGW | | 467 | | | |(3)(4) |(4)SGW's TEID is configured 468 | | |<-+ | and notified to MME 469 | |<----------|(5) MME | | 470 |/---------\| | | | 471 | RB setup |(6) | | | 472 |\---------/| | | | 473 | |---------->|(7) MME | | 474 |---------->|(8) | | | 475 | |---------->|(9) MME |(9)eNodeB TEID is configured 476 | | | | and sent to MME 477 |= = = = = = = Uplink Data = = = = =>= = = = ==>|(10) 478 | The uplink is not yet built here | 479 | | | | | 480 | | |--+SGW/MME | | 481 | | | |(11,12) | | 482 | | |<-+ | | 483 |<= = = = = = Downlink Data = = = = <= = = = = =|(13) 484 | The downlink is not yet built here | 485 | | | | | 486 | | |---------->|(14) Route Update 487 | | | [Dst: UE-prefix, 488 | | | NxtHop: S1-U addr and TEID of enodeB 489 | | | | | 490 | | | |---------->|(15)Route update 491 | | | |[Dst: UE-prefix, 492 | | | | NxtHop: EPC-E address] 493 | | | | | 494 |---------RS or DHCP Request------->|(16) | 495 |<--------RA or DHCP Reply----------|(17) | 497 Extended PDN Connection Setup Procedure (copied Figure 8 of RFC6459) 499 Figure 4 501 UE requests an IPv6 prefix for its address assignment in the step 502 (16). In our architecture, an IPv6 prefix is still assigned by vEPC 503 in the control plane, as PDN-GW does in the legacy EPC. However, 504 EPC-E is responsible to deliver the IPv6 prefix to UE by DHCP or 505 Stateless address autoconfiguration (SLAAC). 507 We now explain how EPC-E can know the prefix assigned to UE from vEPC 508 for address configuration steps (16 and 17). When (1) to (15) are 509 completed, vEPC has already advertised the UE's prefix as route 510 information to all the EPC-E. Therefore, when EPC-E receives a 511 packet of either Router Solicitation or DHCPv6 request message, it 512 just looks up the remote next-hop field of its routing information 513 base (RIB) with the source IP address and the TEID of the received 514 packet. A route entry matched for this search is the prefix 515 delegated to the requesting UE. Therefore, EPC-E simply uses the 516 prefix of the route entry as an assigned UE's prefix. 518 In (17), EPC-E returns the found prefix to UE by either Router 519 Advertisement or DHCPv6 reply message. UE now creates an address(es) 520 from the received prefix. It is important to highlight that UE can 521 obtain the same prefix information from any EPC-E all the time 522 because the same UE's route information is available on all the 523 EPC-E. 525 It would be convenient to use automatic UE's prefix creation rule or 526 algorithm for vEPC. There are various mechanisms to create UE's 527 prefix. As an example, Stateless IPv6 Prefix Delegation 528 [I-D.savolainen-stateless-pd] is introduced as an algorithm to create 529 UE's prefix in vEPC below. It important to mention that our 530 architecture of the stateless user plane does not rely on any 531 particular prefix creation mechanisms like 532 [I-D.savolainen-stateless-pd] and can be run with any of them. 534 In the case of an UE's prefix length is equal, or shorter than /64, 535 the generated prefix is consisted as shown in Figure 5. Each PDN is 536 assumed to have single or several prefixes (named PDN prefix) used to 537 generate UE's address. Followed by the PDN prefix, there is TEID 538 field assigned for a UE's session on S1-U interface of vEPC. TEID is 539 32 bits identifier in GTP header to distinguish each bearer. The 540 remaining bits are filled by subnet ID. 542 | n bits | o bits (=< 32)|64-n-o bits| 543 +---------------------------+---------------+-----------+ 544 | PDN Prefix | TEID | subnet ID | 545 +---------------------------+---------------+-----------+ 546 <---------------------------64bits----------------------> 548 Stateless-pd Prefix 550 Figure 5 552 3.3. Control-plane awareness of stateless user-plane 554 Nodes in the control-plane in vEPC must be aware that the anycast 555 address assigned to EPC-E is a S1-U address of vEPC. The vEPC must 556 use the anycast address in signaling between vEPC and RAN. By doing 557 this, packets from RAN are correctly forwarded to an appropriate 558 EPC-E. Due to anycast nature, it means there is no hand-off procedure 559 between SGWs because all eNodeB in the RAN send packets to the same 560 anycast address. 562 When an operator needs to increase virtualized instances to cope with 563 just signaling overload, the operator should use the existing S1-U 564 address (i.e. EPC-E anycast address) for the new instances. If the 565 operator would increase the capacity of the user plane, it can add 566 additional EPC-Es in the core network. The operator can group the 567 new EPC-Es as a set and increase scalability and performance of the 568 user plane. In this case, the operator uses a new anycast address to 569 the new set of EPC-E. We will discuss operational consideration in 570 detail in Section 4. 572 3.4. Routing mechanism 574 Figure 6 shows a packet forwarding mechanism of our stateless user 575 plane. As an example, there are four eNodeB (illustrated as eNB-x) , 576 three EPC-Edge routers(shown as EPC-Ex) and two routers (RTRx) in 577 Figure 6. UE is first connected to eNB-C and then moves to eNB-D. 578 The UE at the new location is illustrated as UE'. Routing entry for 579 UE is also illustrated at the right side in Figure 6. 581 EPC-E has two interfaces facing either RAN or CORE networks. An 582 anycast address (shown as X) is configured to the interface facing 583 RAN of all EPC-E. EPC-E assigns an individual IPv6 address to 584 another interface (illustrated "a" to "d" in the figure). It is 585 important to mention that the anycast address X can be treated as the 586 SGW's S1-U address. 588 Since RTRs are a gateway to the Internet, they advertise routes of an 589 operator's prefix to the Internet. After one of RTR receives a 590 packet of UE from the Internet, it needs to routing it to UE in the 591 user plane. RTR has a simple routing entry for PDN prefix whose next 592 hop points to the EPC-E. One of RTR (let's say RTR2 in this case) 593 looks up a routing table with UE's address and matched it with a 594 routing entry of PDN prefix. Since multiple EPC-Es advertise a route 595 for the same PDN prefix, RTR2 should forward the packet to one of 596 EPC-E according to the routing entry. This routing is known as hot- 597 potato routing. In this example, the RTR2 uses EPC-E2-b as a nexthop 598 of PDN prefix. 600 When the UE's packet is arrived at EPC-E2, EPC-E2 needs to forwards 601 them to UE via eNodeB to which UE is connecting by using GTP tunnel. 602 For this operation, EPC-E2 has a routing entry that destination is 603 UE's prefix and that next hop points to GTP tunnel between eNB-C and 604 the EPC-Es. In order to identify the GTP tunnel for UE, EPC-E needs 605 S1-U address and Tunnel Endpoint ID (TEID) of eNB-C that is eNB-C-3 606 in Figure 6. The eNB-C TEID for UE is illustrated as TEID[eNB-C]. 607 The SGW assigned TEID is utilized to generate the UE's prefix as we 608 explained in Section 3.2. These TEID are assigned per UE. The TEID 609 and S1-U address of eNodeB are retrieved from the next hop field of 610 the routing entry of the mobile node. By using the GTP information, 611 every EPC-E can now forward the UE's packets to right eNodeB. 613 Routing outgoing packets from UE is much simpler. The packets from 614 UE are always routed to the closest EPC-E to UE because of anycast 615 routing. In Figure 6, when UE sends a packet to a destination, the 616 packet is reached to eNB-C and tunneled to EPC-E's anycast address. 617 The GTP-tunneled packet is routed to the closest EPC-E that is EPC-E2 618 in this case. The packet is decapsulated by EPC-E2 and then 619 forwarded to one of RTR according to the routing table. Since the 620 decapsulated packet is regular IPv6 packet, no extra control other 621 than routing is necessary. 623 When UE moves to a new location (UE'), it updates its location on the 624 control plane. After signaling completion for location update, vEPC 625 needs to update the UE's routing entry of all EPC-E so that vEPC 626 advertises updated route with new location to all EPC-Es by a routing 627 protocol. The routing entry should be updated with the new eNodeB's 628 address that is eNB-D-4. During handover, there might be some 629 traffic arriving to the older eNodeB (eNB-C). These packets can be 630 re-routed to the new eNodeB (eNB-D) via X2-U interface in RAN. 632 The UE's address isn't changed when UE changes its attachment. In 633 our scenario, SGW run on hypervisor and is independent from network 634 topology. Therefore, logically we don't have handover across 635 different SGWs. UE can stay connected with the same SGW all the time 636 and can keep using the same TEID after handover. Thus, UE's address 637 is unchanged even after handover. 639 +--------------------+ 640 | Correspondent Node | 641 +--------------------+ 642 | 643 .--. [Route on the Internet] 644 _(. `) Destination: Operator's Prefix 645 _( `)_ NextHop: RTR-1/2 646 ( Internet `) 647 ( ` . ) ) 648 `--(_______)---' 649 / \ [Route at RTR] 650 +----+ +----+ Destination: PDN Prefix 651 |RTR1| |RTR2| NextHop: EPC-E2-b 652 +----+ +----+ 653 \ .--. / 654 _( IP `. 655 ( CORE NW ) 656 ___( ` . ) )__ 657 / `--(___.--' \ 658 / | \ 659 |a |b |c [Route at EPC-E] 660 +---+--+ +---+--+ +---+--+ Destination: UE's Prefix *1 661 |EPC-E1| |EPC-E2| |EPC-E3| NextHop: GTP tunnel (eNB-C) *2 662 +---+--+ +---+--+ +---+--+ 663 |X |X |X 664 \ .--. / 665 \_____( RAN `.____/ 666 ( ACCESS ) 667 ____( ` NW ) )_______ 668 / `--(___.--' \ 669 / | | \ 670 |1 |2 |3 |4 671 +--+--+ +--+--+ +--+--+ +--+--+ 672 |eNB-A| |eNB-B| |eNB-C| |eNB-D| 673 +--+--+ +--+--+ +--+--+ +--+--+ 674 : : : : 675 UE --> UE' 676 *1 TEID used at EPC-E for the UE is included in this UE's prefix. 677 see Figure 4. 678 *2 GTP tunnel state is stored in the next hop field. The state 679 information is the combination of eNB-C S1 address that is 680 eNB-C-3 and TEID(eNB-C) assigned for the UE. 682 Routing Mechanism Overview 684 Figure 6 686 3.5. IPv4 Support 688 Recent IPv6 transition mechanisms enable IPv6-only network to forward 689 IPv4 packet with encapsulation or translation techniques. By using 690 one of mechanisms, we can use IPv6 for our stateless user-plane 691 network for transporting both IPv4 and IPv6 packets. Figure 7 shows 692 available solutions of IPv4 support for each bearer type to deal with 693 that requirement. 695 Bearer UE EPC-E Gateway 696 type function function function 697 -------------------------------------------- 698 IPv4 - B4 AFTR 699 IPv4 - CLAT PLAT 700 IPv6 MAP-CE - MAP-BR 701 IPv6 B4 - AFTR 702 IPv6 CLAT - PLAT 704 Solutions and functions for IPv4 support 706 Figure 7 708 In the case of a UE only support IPv4 bearer, B4 function of DS-Lite 709 [RFC6333] or CLAT function of 464XLAT [RFC6877] may be implemented in 710 a EPC-E. Both functions are stateless therefore EPC-E isn't required 711 to maintain any tunneling or translation state. 713 Figure 8 shows how to support IPv4 on IPv6 core network in our vEPC. 714 Instead of using RqTR as a gateway to the Internet, DS-LITE AFTR or 715 464XLAT PLAT is installed as a gateway to the IPv4 Internet. 717 .--. 718 EPC-E _( `)_ Gateway 719 +----+ _( `) +------+ 720 | B4 | ( IPv6 Core ) | AFTR | 721 [UE]==IPv4-only==| or |-( ` NW )-| or | --> Internet 722 bearer |CLAT| ( ` ) ) | PLAT | (IPv4) 723 +----+ `--(_____)--' +------+ 724 <-----IPv4 over IPv6-----> 725 or 726 v4v6 translation 728 IPv4 User plane of vEPC 730 Figure 8 732 If UE supports IPv6 capable bearer, IPv6 transition function may be 733 implemented in the UE such as MAP-CE [I-D.ietf-softwire-map], B4 or 734 CLAT. That means an EPC-E receives IPv6 packets from UE in this case 735 so that the EPC-E does not need to be involved in the part of IPv4 736 support functions. 738 4. Operational Considerations 740 4.1. Scalability 742 Virtualization allows vEPC to be elastic for steep demand of requests 743 to create and update for sessions. In our architecture, that makes 744 routing update fluctuation from vEPC to EPC-E. This is the reason why 745 we select BGP as a protocol between vEPC and EPC-E. BGP is scalable 746 and stable routing protocol today. 748 BGP is an incremental update protocol so that once BGP peer 749 established, millions of routes can be easily updated in stable 750 manners. Operators can appropriately design BGP peering between vEPC 751 and ECP-E to secure convergence time within appropriate period. 753 Granularity of the peering should be aware EPC-E capacity because it 754 is assumed that EPC-E has upper limit of routing entries. BGP 755 peering design should makes sure that total number of routes does not 756 exceed EPC-E capacity. 758 During the network planning, operators must understand EPC-E's 759 capacity such as # of routes, bandwidth, etc. An of example of 760 estimation, if a EPC-E has 1Gbps throughput and each UE's bandwidth 761 consumption is 10Kbps in average, the EPC-E should have 100K routes 762 capacity. 764 This is an operational approach to minimize the risk of routing 765 update fluctuation. If it is hard to support all the UEs by a single 766 set of EPC-E in an operators network, different set of EPC-E can be 767 introduced and configured in a network. The UEs are distributed to 768 one of the set and handled by the EPC-Es in the set. We don't need 769 to support millions of UEs by a single set of EPC-E. This is another 770 advantage of using routing mechanism in the user plane. We already 771 explain how to handle different set of EPC-E in our scheme in 772 Section 3.3. 774 The notion of multiple EPC-E sets is easily fitted into our today's' 775 network. The operator's network is often separated into several 776 regional network for geographical scalability. Therefore, the 777 operator can assign different EPC-E set to different region for 778 better scalability. 780 In addition, routers and EPC-E in the IPv6 core network are required 781 to process just "route", they naturally aggregate those routing 782 entries. It helps limiting the total number of routing entries in 783 our core network. 785 4.2. Backward Compatibility 787 vEPC should be able to fall back to the legacy EPC based packet 788 forwarding to secure backward compatibility which is required to 789 connect existing system, or to connect roaming partners through 790 legacy S5/S8 interfaces. When fallback happened, all the packets are 791 not routed on our stateless user plane, but forwarded to vEPC (i.e. 792 SGW and PGW instances on hypervisor). vEPC must use a S1-U address 793 that is different from anycast address assigned to EPC-Es. This 794 address is assigned to SGW instances in vEPC and used to terminate 795 tunnels in vEPC servers (i.e. hypervisor). 797 5. IANA Considerations 799 This memo includes no request to IANA. 801 6. Security Considerations 803 There are no security considerations specific to this document at 804 this moment. 806 7. References 808 7.1. Normative References 810 [I-D.vandevelde-idr-remote-next-hop] 811 Velde, G., Patel, K., Rao, D., Raszuk, R., and R. Bush, 812 "BGP Remote-Next-Hop", draft-vandevelde-idr-remote-next- 813 hop-03 (work in progress), October 2012. 815 [RFC5512] Mohapatra, P. and E. Rosen, "The BGP Encapsulation 816 Subsequent Address Family Identifier (SAFI) and the BGP 817 Tunnel Encapsulation Attribute", RFC 5512, April 2009. 819 7.2. Informative References 821 [Boeing-BGP] 822 Andrew, ., "Global IP Network Mobility using Border 823 Gateway Protocol (BGP)", IAB Plenary IAB Plenary of IETF 824 62nd, March 2005. 826 [I-D.ietf-softwire-map] 827 Troan, O., Dec, W., Li, X., Bao, C., Matsushima, S., 828 Murakami, T., and T. Taylor, "Mapping of Address and Port 829 with Encapsulation (MAP)", draft-ietf-softwire-map-07 830 (work in progress), May 2013. 832 [I-D.savolainen-stateless-pd] 833 Savolainen, T. and J. Korhonen, "Stateless IPv6 Prefix 834 Delegation for IPv6 enabled networks", draft-savolainen- 835 stateless-pd-01 (work in progress), February 2010. 837 [I-D.wakikawa-mext-haha-interop2008] 838 Wakikawa, R., Shima, K., and N. Shigechika, "The Global 839 HAHA Operation at the Interop Tokyo 2008", draft-wakikawa- 840 mext-haha-interop2008-00 (work in progress), July 2008. 842 [I-D.yokota-dmm-scenario] 843 Yokota, H., Seite, P., Demaria, E., and Z. Cao, "Use case 844 scenarios for Distributed Mobility Management", draft- 845 yokota-dmm-scenario-00 (work in progress), October 2010. 847 [NFV-WHITEPAPER] 848 Network Operators, ., "Network Functions Virtualization, 849 An Introduction, Benefits, Enablers, Challenges and Call 850 for Action", SDN and OpenFlow SDN and OpenFlow World 851 Congress, October 2012. 853 [RFC3963] Devarapalli, V., Wakikawa, R., Petrescu, A., and P. 854 Thubert, "Network Mobility (NEMO) Basic Support Protocol", 855 RFC 3963, January 2005. 857 [RFC5213] Gundavelli, S., Leung, K., Devarapalli, V., Chowdhury, K., 858 and B. Patil, "Proxy Mobile IPv6", RFC 5213, August 2008. 860 [RFC5555] Soliman, H., "Mobile IPv6 Support for Dual Stack Hosts and 861 Routers", RFC 5555, June 2009. 863 [RFC5844] Wakikawa, R. and S. Gundavelli, "IPv4 Support for Proxy 864 Mobile IPv6", RFC 5844, May 2010. 866 [RFC6275] Perkins, C., Johnson, D., and J. Arkko, "Mobility Support 867 in IPv6", RFC 6275, July 2011. 869 [RFC6301] Zhu, Z., Wakikawa, R., and L. Zhang, "A Survey of Mobility 870 Support in the Internet", RFC 6301, July 2011. 872 [RFC6333] Durand, A., Droms, R., Woodyatt, J., and Y. Lee, "Dual- 873 Stack Lite Broadband Deployments Following IPv4 874 Exhaustion", RFC 6333, August 2011. 876 [RFC6459] Korhonen, J., Soininen, J., Patil, B., Savolainen, T., 877 Bajko, G., and K. Iisakkila, "IPv6 in 3rd Generation 878 Partnership Project (3GPP) Evolved Packet System (EPS)", 879 RFC 6459, January 2012. 881 [RFC6877] Mawatari, M., Kawashima, M., and C. Byrne, "464XLAT: 882 Combination of Stateful and Stateless Translation", RFC 883 6877, April 2013. 885 Authors' Addresses 887 Satoru Matsushima 888 Softbank Telecom 889 1-9-1,Higashi-Shimbashi,Minato-Ku 890 Tokyo 105-7322 891 Japan 893 Email: satoru.matsushima@g.softbank.co.jp 895 Ryuji Wakikawa 896 Softbank Mobile 897 1-9-1,Higashi-Shimbashi,Minato-Ku 898 Tokyo 105-7322 899 Japan 901 Email: ryuji.wakikawa@gmail.com