idnits 2.17.1 draft-dunbar-idr-5g-edge-compute-app-meta-data-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 17 characters in excess of 72. ** The abstract seems to contain references ([RFC9012]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (December 13, 2021) is 859 days in the past. Is this intentional? Checking references for intended status: Full Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC9012' is mentioned on line 605, but not defined == Missing Reference: '5GC' is mentioned on line 138, but not defined == Missing Reference: '5g-edge-compute-sticky-service' is mentioned on line 280, but not defined == Missing Reference: 'A-ER' is mentioned on line 429, but not defined == Missing Reference: '5G-Sticky-Service' is mentioned on line 559, but not defined == Missing Reference: 'Section 5' is mentioned on line 682, but not defined == Unused Reference: 'RFC4364' is defined on line 831, but no explicit reference was found in the text == Unused Reference: 'RFC8200' is defined on line 839, but no explicit reference was found in the text == Unused Reference: '3GPP-EdgeComputing' is defined on line 844, but no explicit reference was found in the text == Unused Reference: 'RFC5521' is defined on line 860, but no explicit reference was found in the text == Unused Reference: 'BGP-SDWAN-Port' is defined on line 864, but no explicit reference was found in the text == Unused Reference: 'SDWAN-EDGE-Discovery' is defined on line 870, but no explicit reference was found in the text == Unused Reference: 'Tunnel-Encap' is defined on line 875, but no explicit reference was found in the text ** Downref: Normative reference to an Proposed Standard RFC: RFC 4364 == Outdated reference: A later version (-02) exists of draft-dunbar-ippm-5g-edge-compute-ip-layer-metrics-00 -- No information found for draft-dunbar-6man-5g-ec-sticky-service - is the name correct? == Outdated reference: A later version (-04) exists of draft-dunbar-idr-sdwan-edge-discovery-00 == Outdated reference: A later version (-22) exists of draft-ietf-idr-tunnel-encaps-10 Summary: 3 errors (**), 0 flaws (~~), 18 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft Futurewei 3 Intended status: Standard K. Majumdar 4 Expires: June 13, 2022 CommScope 5 H. Wang 6 Huawei 7 G. Mishra 8 Verizon 9 December 13, 2021 11 BGP Update for 5G Edge Computing Service Metadata 12 draft-dunbar-idr-5g-edge-compute-app-meta-data-05 14 Abstract 15 This draft describes a new AppMetaData subTLV carried by 16 Tunnel Encap[RFC9012] Path Attribute for egress router to 17 advertise the running status and environment for the directly 18 attached 5G Edge Computing (EC) servers. The AppMetaData can 19 be used by the ingress routers in the 5G Local Data Network to 20 make path selection not only based on the routing distance but 21 also the running environment of the destinations. The goal is 22 to improve latency and performance for 5G EC services. 24 The extension enables an EC server at one specific location to 25 be more preferred than the others with the same IP address to 26 receive data flows from a specific source (UE). 28 Status of this Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. This document may not be 35 modified, and derivative works of it may not be created, 36 except to publish it as an RFC and to translate it into 37 languages other than English. 39 Internet-Drafts are working documents of the Internet 40 Engineering Task Force (IETF), its areas, and its working 41 groups. Note that other groups may also distribute working 42 documents as Internet-Drafts. 44 Internet-Draft AppMetaData for 5G EC Service 46 Internet-Drafts are draft documents valid for a maximum of six 47 months and may be updated, replaced, or obsoleted by other 48 documents at any time. It is inappropriate to use Internet- 49 Drafts as reference material or to cite them other than as 50 "work in progress." 52 The list of current Internet-Drafts can be accessed at 53 http://www.ietf.org/ietf/1id-abstracts.txt 55 The list of Internet-Draft Shadow Directories can be accessed 56 at http://www.ietf.org/shadow.html 58 This Internet-Draft will expire on April 7, 2021. 60 Copyright Notice 62 Copyright (c) 2021 IETF Trust and the persons identified as 63 the document authors. All rights reserved. 65 This document is subject to BCP 78 and the IETF Trust's Legal 66 Provisions Relating to IETF Documents 67 (http://trustee.ietf.org/license-info) in effect on the date 68 of publication of this document. Please review these documents 69 carefully, as they describe your rights and restrictions with 70 respect to this document. Code Components extracted from this 71 document must include Simplified BSD License text as described 72 in Section 4.e of the Trust Legal Provisions and are provided 73 without warranty as described in the Simplified BSD License. 75 Table of Contents 77 1. Introduction.............................................. 3 78 1.1. 5G Edge Computing Background......................... 3 79 1.2. 5G Edge Computing Network Properties................. 4 80 1.3. Problem#1: ANYCAST in 5G EC Environment.............. 6 81 1.4. Problem #2: Unbalanced Anycast Distribution due to UE 82 Mobility.................................................. 7 83 1.5. Problem 3: Application Server Relocation............. 7 85 Internet-Draft AppMetaData for 5G EC Service 87 2. Conventions used in this document......................... 8 88 3. Usage of App-Meta-Data for 5G Edge Computing.............. 9 89 3.1. Assumptions.......................................... 9 90 3.2. IP Layer Metrics to Gauge Application Behavior...... 10 91 3.3. AppMetaData Constrained Optimal Path Selection...... 11 92 3.4. BGP Protocol Extension to advertise Load & Capacity. 12 93 3.5. Ingress Node BGP Path Selection Behavior............ 12 94 3.5.1. AppMetaData Influenced BGP Path Selection...... 12 95 3.5.2. Forwarding Behavior............................ 12 96 3.5.3. Forwarding Behavior after a UE moving to a new 5G 97 Site.................................................. 14 98 4. The Sub-TLVs for App-Meta-Data........................... 14 99 4.1. Load Measurement sub-TLV format..................... 15 100 4.2. Capacity Index sub-TLV format....................... 16 101 4.3. The Site Preference Index sub-TLV format............ 16 102 5. AppMetaData Propagation Scope............................ 17 103 6. Metrics Change Rate Consideration........................ 17 104 7. Soft Anchoring of an ANYCAST Flow........................ 17 105 8. Manageability Considerations............................. 19 106 9. Security Considerations.................................. 19 107 10. IANA Considerations..................................... 19 108 11. References.............................................. 19 109 11.1. Normative References............................... 20 110 11.2. Informative References............................. 20 111 12. Acknowledgments......................................... 21 113 1. Introduction 115 This document describes a new subTLV, AppMetaData, for egress 116 routers to advertise the running status and environment for 117 the directly attached Edge Computing (EC) servers. The 118 AppMetaData can be used by the ingress routers in the 5G Local 119 Data Network to make path selection not only based on the 120 routing distance but also the running environment of the 121 destinations. The goal is to improve latency and performance 122 for 5G Edge Computing services. 124 1.1. 5G Edge Computing Background 126 In 5G Edge Computing (EC), one Application can be hosted on 127 multiple Application Servers in different EC data centers that 128 are close in proximity. The network connecting the EC data 129 centers with the 5G Base stations consists of small number of 130 routers dedicated for the 5G Local Data Network (LDN), to 131 minimize latency and optimize the user experience. 133 Internet-Draft AppMetaData for 5G EC Service 135 When a User Equipment (UE) initiates application packets using 136 the destination address from a DNS reply or its cache, the 137 packets from the UE are carried in a PDU session through 5G 138 Core [5GC] to the 5G UPF-PSA (User Plan Function - PDU Session 139 Anchor). The UPF-PSA decapsulates the 5G GTP outer header and 140 forwards the packets from the UEs to its directly connected 141 Ingress router of the 5G LDN. The LDN for 5G EC, which is the 142 IP Networks from the 5GC perspective, is responsible for 143 forwarding the packets to the intended destinations. 145 When the UE moves out of coverage of its current gNB (next- 146 generation Node B)and anchors to a new gNB, the 5G SMF 147 (Session Management Function) could select the same UPF or a 148 new UPF for the UE per standard handover procedures described 149 in 3GPP TS 23.501 and TS 23.502. If the UE is anchored to a 150 new UPF-PSA when the handover process is complete, the packets 151 to/from the UE is carried by a GTP tunnel to the new UPF-PSA. 152 Per TS 23.501-h20 Section 5.8.2, the UE may maintain its IP 153 address when anchored to a new UPF-PSA unless the new UFP-PSA 154 belongs to different mobile operators. 5GC may maintain a path 155 from the old UPF to the new UPF for a short time for the SSC 156 [Session and Service Continuity] mode 3 to make the handover 157 process more seamless. 159 1.2. 5G Edge Computing Network Properties 161 In this document, 5G Edge Computing Network refers to multiple 162 Local IP Data Networks (LDN) in one region that interconnect 163 the Edge Computing data centers. Those IP LDN networks are the 164 N6 interfaces from 3GPP 5G perspective. 166 The ingress routers to the 5G Edge Computing Network are the 167 routers directly connected to 5G UPFs. The egress routers to 168 the 5G Edge Computing Network are the routers that have a 169 direct link to the Edge Computing servers. The servers and the 170 egress routers are co-located. Some of those Edge Computing 171 Data centers may have Virtual switches or Top of Rack switches 172 between the egress routers and the servers. But transmission 173 delay between the egress routers and the Edge Computing 174 servers is too small to be considered in this document. 176 Internet-Draft AppMetaData for 5G EC Service 178 When one EC data center has multiple EC Servers attached to 179 one App Layer Load Balancer, only the App Layer Load Balancer 180 is visible to the 5G Edge Computing Network. How the App Layer 181 Load balancer manages the individual servers is out of the 182 scope of the network layer. 184 The 5G EC Services are specially managed services optimized by 185 utilizing the network topology and multiple servers with the 186 same IP address (ANYCAST) in multiple EC Data Centers. Many 187 services by the UEs are not part of the registered 5G EC 188 Services. 190 +--+ 191 |UE|---\+---------+ +------------------+ 192 +--+ | 5G | +--------+ | S1: aa08::4450 | 193 +--+ | Site +--+-+---+ +----+ | 194 |UE|----| A |PSA1| Ra| | R1 | S2: aa08::4460 | 195 +--+ | +----+---+ +----+ | 196 +---+ | | | | | S3: aa08::4470 | 197 |UE1|---/+---------+ | | +------------------+ 198 +---+ |IP Network | L-DN1 199 |(3GPP N6) | 200 | | | +------------------+ 201 | UE1 | | | S1: aa08::4450 | 202 | moves to | +----+ | 203 | Site B | | R3 | S2: aa08::4460 | 204 v | +----+ | 205 | | | S3: aa08::4470 | 206 | | +------------------+ 207 | | L-DN3 208 +--+ | | 209 |UE|---\+---------+ | | +------------------+ 210 +--+ | 5G | | | | S1: aa08::4450 | 211 +--+ | Site +--+--+---+ +----+ | 212 |UE|----| B |PSA2| Rb | | R2 | S2: aa08::4460 | 213 +--+ | +--+-+----+ +----+ | 214 +--+ | | +-----------+ | S3: aa08::4470 | 215 |UE|---/+---------+ +------------------+ 216 +--+ L-DN2 217 Figure 1: App Servers in different edge DCs 219 Internet-Draft AppMetaData for 5G EC Service 221 1.3. Problem#1: ANYCAST in 5G EC Environment 223 Increasingly, Anycast is used by various application providers 224 and CDNs because Anycast provides better and faster resiliency 225 to failover events than GEO database DNS-based load balancing, 226 which relies on DNS to provide a different IP based on source 227 address. 229 Anycast address leverages the proximity information present in 230 the network (routing) layer. It eliminates the single point of 231 failure and bottleneck at the DNS resolvers and application 232 layer load balancers. Another benefit of using the ANYCAST 233 address is removing the dependency on UEs. Some UEs (or 234 clients) might use their cached IP addresses for an extended 235 period instead of querying DNS. 237 Client using Virtual IP address is a common practice in Cloud 238 Native networking, e.g., Kubernetes, to scale dynamic changes 239 of app servers' instantiations. However, Virtual IP requires 240 the destination node to perform address translation for return 241 traffic, which is unsuitable for underlay network nodes with 242 millions of flows passing by. 244 Having multiple locations of the same ANYCAST address in the 245 5G EC LDC can be problematic if path selection is solely based 246 on routing cost as the difference in the routing cost to reach 247 the Application Servers attached to different egress routers 248 can be very small. This list elaborates the issues in detail: 250 a) Path Selection: When a new flow comes to an ingress node 251 (Ra), how to avoid instability with Anycast flipping 252 between paths to the same address. The problem is more 253 so with BGP multipath and picking the optimal path 254 depending on close metrics. 256 b) Ingress node forwards the packets from one flow to the 257 same ANYCAST server. 259 a.k.a. Flow Affinity, or Flow-based load balancing. 260 Almost all vendors have supported flow or session based 261 ECMP load balancing and not per packet to avoid out of 262 order packets for decades. When a flow is hashed to an 264 Internet-Draft AppMetaData for 5G EC Service 266 ECMP path, the flow remains on that path for the life of 267 the flow until the flow ends. 269 The ingress node, (Ra/Rb), can use Flow ID (in IPv6 270 header) or UDP/TCP port number combined with the source 271 address to enforce packets in one flow being placed in 272 one tunnel to one Egress router. No new features are 273 needed. 275 c) When a UE moves to a new Cell Tower, a method is needed 276 to stick the flow to the same ANYCAST server, which is 277 required by 5G Edge Computing: 3GPP TR 23.748. 279 Soft-Anchoring in Section 7 describes one method to 280 achieve stickiness. [5g-edge-compute-sticky-service] 281 describes several approaches to achieve stickiness in the 282 IPv6 domain. 284 From BGP perspective, the multiple servers with the same IP 285 address (ANYCAST)attached to different egress routers is the 286 same as multiple next hops for the IP address. 288 This draft describes the BGP UPDATE to enable ingress routers 289 to take the App Server load, the capacity index, and the 290 location preference into consideration when computing the 291 optimal path to egress routers. 293 1.4. Problem #2: Unbalanced Anycast Distribution due to UE 294 Mobility 296 UEs frequent moving from one 5G site to another can make it 297 difficult to plan where and how many to deploy the App 298 servers. When one App server is heavily utilized, other 299 servers of the same App close-by can be very underutilized. 300 Since the condition can be short-lived, it is difficult for 301 the application controller to anticipate the move and adjust. 303 1.5. Problem 3: Application Server Relocation 305 When an Application Server is added to, moved, or deleted from 306 a 5G EC Data Center, the routing protocol needs to propagate 308 Internet-Draft AppMetaData for 5G EC Service 310 the changes to 5G PSA or the PSA adjacent routers. After the 311 change, the cost associated with the site might change as 312 well. 314 Note: for ease of description, the Edge Application Server and 315 Application Server are used interchangeably throughout this 316 document. 318 2. Conventions used in this document 320 A-ER: Egress Router to an Application Server, [A-ER] is 321 used to describe the last router that the 322 Application Server is attached. For a 5G EC 323 environment, the A-ER can be the gateway router to 324 a (mini) Edge Computing Data Center. 326 Application Server: An application server is a physical or 327 virtual server that hosts the software system for 328 the application. 330 Application Server Location: Represent a cluster of servers at 331 one location serving the same Application. One 332 application may have a Layer 7 Load balancer, 333 whose address(es) are reachable from an external 334 IP network, in front of a set of application 335 servers. From an IP network perspective, this 336 whole group of servers is considered as the 337 Application server at the location. 339 Edge Application Server: used interchangeably with Application 340 Server throughout this document. 342 EC: Edge Computing 344 Edge Hosting Environment: An environment providing the support 345 required for Edge Application Server's execution. 347 NOTE: The above terminologies are the same as 348 those used in 3GPP TR 23.758 350 Edge DC: Edge Data Center, which provides the Edge 351 Computing Hosting Environment. An Edge DC might 353 Internet-Draft AppMetaData for 5G EC Service 355 host 5G core functions in addition to the 356 frequently used application servers. 358 gNB next generation Node B 360 L-DN: Local Data Network 362 PSA: PDU Session Anchor (UPF) 364 SSC: Session and Service Continuity 366 UE: User Equipment 368 UPF: User Plane Function 370 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL 371 NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT 372 RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be 373 interpreted as described in BCP 14 [RFC2119] [RFC8174] when, 374 and only when, they appear in all capitals, as shown here. 376 3. Usage of App-Meta-Data for 5G Edge Computing 378 3.1. Assumptions 380 From IP Layer, the Application servers are identified by their 381 IP (ANYCAST) addresses. Here are some assumptions about the 5G 382 EC services: 383 - Only the registered EC services, which are only a small 384 portion of the services, need to include the AppMetadata 385 in path selection. 386 - The 5G EC controller or management system push down the 387 policies (e.g., ACLs) on the relevant routers to filter 388 out those registered EC services. 389 - The ingress routers' local BGP path compute algorithm 390 includes a special plugin that can compute the path to 391 the optimal Next Hop (egress router) based on the BGP 392 AppMetaData TLV received for the registered EC services. 394 The proposed solution is for the egress routers, i.e. A-ER, 395 that have direct links to the Application Servers to collect 396 various measurements about the Servers' running status [5G-EC- 398 Internet-Draft AppMetaData for 5G EC Service 400 Metrics] and advertise the metrics to other routers in 5G EC 401 LDN (Local Data Network). 403 3.2. IP Layer Metrics to Gauge Application Behavior 405 [5G-EC-Metrics] describes the IP Layer Metrics that can gauge 406 the application servers running status and environment: 408 - IP-Layer Metric for App Server Load Measurement: 409 The Load Measurement to an App Server is a weighted 410 combination of the number of packets/bytes to the App Server 411 and the number of packets/bytes from the App Server which 412 are collected by the A-ER to which the App Server is 413 directly attached. 414 The A-ER is configured with an ACL that can filter out the 415 packets for the Application Server. 416 - Capacity Index 417 a numeric number, configured on all A-ERs in the domain 418 consistently, is used to represent the capacity of the 419 application server attached to an A-ER. At some sites, the 420 IP address exposed to the A-ER is the App Layer Load 421 balancer that have many instances attached. At other sites, 422 the IP address exposed is the server instance itself. 423 - Site preference index: 424 is used to describe some sites are more preferred than 425 others. For example, a site with higher bandwidth has a 426 higher preference number than other. 428 In this document, the term "Application Server Egress Router" 429 [A-ER] is used to describe the last router that an Application 430 Server is attached. For the 5G EC environment, the A-ER can be 431 the gateway router to the EC DC where multiple Application 432 servers are hosted. 434 From IP Layer, an Application Server is identified by its IP 435 (ANYCAST) Address. Those IP addresses are called the 436 Application Server IDs throughout this document. 438 Internet-Draft AppMetaData for 5G EC Service 440 3.3. AppMetaData Constrained Optimal Path Selection 442 The main benefit of using ANYCAST is to leverage the network 443 layer information to select an optimal path among multiple 444 application Server locations of the same application 445 identified by its ANYCAST addresses. 447 For the 5G EC environment, the ingress routers to the LDN need 448 to be notified of the Load Index and Capacity Index of the App 449 Servers at different EC data centers to make the intelligent 450 decision on where to forward the traffic for the application 451 from UEs. 453 Here is an algorithm that computes the cost to reach the App 454 Servers attached to Site-i relative to another site, say Site- 455 b. When the reference site, Site-b, is plugged in the formula, 456 the cost is 1. So, if the formula returns a value less than 1, 457 the cost to reach Site-i is less than reaching Site-b. 459 CP-b * Load-i Pref-b * Network-Delay-i 460 Cost-i= (w *(----------------) + (1-w) *(-------------------------)) 461 CP-i * Load-b Pref-i * Network-Delay-b 463 Load-i: Load Index at Site-i, it is the weighted 464 combination of the total packets or/and bytes sent to and 465 received from the Application Server at Site-i during a 466 fixed time period. 468 CP-i: capacity index at Site-i, a higher value means higher 469 capacity. 471 Delay-i: Network latency measurement (RTT) to the A-ER that 472 has the Application Server attached at the site-i. 474 Pref-i: Preference index for the Site-i, a higher value 475 means higher preference. 477 w: Weight for load and site information, which is a value 478 between 0 and 1. If smaller than 0.5, Network latency and 479 the site Preference have more influence; otherwise, Server 480 load and its capacity have more influence. 482 Internet-Draft AppMetaData for 5G EC Service 484 3.4. BGP Protocol Extension to advertise Load & Capacity 486 The goal of the protocol extension: 487 - Propagate the Load Measurement Index for the attached App 488 Servers to other routers in the LDN. 490 - Propagate the Capacity Index, and 492 - Propagate Site Preference Index. 494 The BGP extension is to include the Load Index Sub-TLV, 495 Capacity Sub-TLV, and the Site Preference Sub-TLV in the 496 Tunnel Encap Path Attribute associated with the routes. 498 3.5. Ingress Node BGP Path Selection Behavior 500 3.5.1. AppMetaData Influenced BGP Path Selection 502 In this scenario, an ingress router will receive one ANYCAST 503 address's multiple routes from different egress routers that 504 have the direct links to the ANYCAST servers. The ingress 505 router's BGP engine will do path selection, select the best 506 route, and download to FIB. And BGP engine will also download 507 the other paths to FIB that with the AppMetaData taken into 508 the consideration. 510 Assume that both Ra and Rb in Figure-1 have BGP Multipath 511 enabled. As a result, Dst Address: S1:aa08::4450 is resolved 512 via multiple NextHop: R1, R2, R3. 514 Suppose the local BGP special Plugin for AppMetaData finds R1 515 is the best for the flow towards S1:aa08::4450. Then this 516 special Plugin can insert a higher weight for the path R1 so 517 that BGP Best Path is locally influenced by the weight 518 parameter based on the local decision. 520 3.5.2. Forwarding Behavior 522 When the ingress router receives a packet and lookup the FIB, 523 it gets the destination prefix's whole path and AppMetaData. 524 The Forwarding Plane will do computing for the packet and 525 choose the suitable path as the result of the computing. Then 526 the Forwarding Plane encapsulates the packet destined towards 527 the optimal egress node. 529 Internet-Draft AppMetaData for 5G EC Service 531 For subsequent packets belonging to the same flow, the ingress 532 router needs to forward them to the same egress router unless 533 the selected egress router is no longer reachable. Keeping 534 packets from one flow to the same egress router, a.k.a. Flow 535 Affinity, is supported by many commercial routers. 537 How Flow Affinity is implemented is out of the scope for this 538 document. Here is one example to illustrate how Flow Affinity 539 can be achieved. This illustration is not to be standardized. 541 For the registered EC services, the ingress node keeps a 542 table of 543 - Service ID (i.e., ANYCAST address) 544 - Flow-ID 545 - Sticky Egress ID (egress router loopback address) 546 - A timer 548 The Flow-ID in this table is to identify a flow, initialized 549 to NULL. How Flow-ID is constructed is out of the scope for 550 this document. Here is one example of constructing the Flow- 551 ID: 552 - For IPv6, the Flow-ID can be the Flow-ID extracted from 553 the IPv6 packet header with or without the source 554 address. 555 - For IPv4, the Flow-ID can be the combination of the 556 Source Address with or without the TCP/UDP Port number. 558 The Sticky Egress ID is the egress node address for the same 559 flow. [5G-Sticky-Service] describes several methods to 560 derive the Sticky Egress ID. 562 The Timer is always refreshed when a packet with the 563 matching EC Service ID (ANYCAST address) is received by the 564 node. 566 If there is no Stick Egress ID present in the table for the 567 EC Service ID, the forwarding plane computes the optimal 568 path to an egress (NextHop) with the AppMetaData taken into 569 consideration. The forwarding plane encapsulates the packet 570 with a tunnel to the chosen egress (NextHop). The chosen 571 NextHop and the Flow ID are recorded in the table entry of 572 the EC Service ID. 574 When the selected optimal egress router is no longer 575 reachable, refer to Section 6 Soft Anchoring on how another 576 path is selected. 578 Internet-Draft AppMetaData for 5G EC Service 580 3.5.3. Forwarding Behavior after a UE moving to a new 5G Site 582 When a UE moves to a new 5G eNB which is anchored to the same 583 UPF, the packets from the UE traverse to the same ingress 584 router. Path selection and forwarding behavior are same as 585 before. 587 When the new eNB is anchored to a different UPF, the packets 588 from the UE traverse a different ingress router. If the UE 589 source IP address has been changed, indicating the new UPF 590 might belongs to a different administrative domain, the new 591 ingress router treats the packets from the UE as a new flow 592 and select the optimal path based on the configured policies. 593 If the UE maintains the same IP address when anchored to a new 594 UPF, the directly connected ingress router might use the pre- 595 computed Egress Router which is passed from the neighboring 596 router. [5G-Edge-Sticky] describes the method for the ingress 597 router connected to the UPF in the new site to take into 598 consideration the information passed from other ingress 599 routers in selecting the optimal paths. The detailed algorithm 600 is out of the scope of this document. 602 4. The Sub-TLVs for App-Meta-Data 604 The App-Meta-Data attribute is encoded in an optional subTLV 605 within the Tunnel Encap [RFC9012] Path Attribute. 607 All values in the Sub-TLVs are unsigned 32 bits integers. 609 Internet-Draft AppMetaData for 5G EC Service 611 4.1. Load Measurement sub-TLV format 613 Two types of Load Measurement Sub-TLVs are specified. One is 614 to carry the aggregated cost Index based on a weighted 615 combination of the collected measurements; another one is to 616 carry the raw measurements of packets/bytes to/from the App 617 Server address. The raw measurement is useful when ingress 618 routers have embedded analytics relying on the raw 619 measurements. 621 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 622 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 623 | Type (TBD1) | Length | 624 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 625 | Measurement Period | 626 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 627 | Aggregated Load Index to reach the App Server | 628 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 629 Figure 2: Aggregated Load Index Sub-TLV 631 Raw Load Measurement sub-TLV has the following format: 633 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 634 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 635 | Type (TBD2) | Length | 636 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 637 | Measurement Period | 638 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 639 | total number of packets to the AppServer | 640 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 641 | total number of packets from the AppServer | 642 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 643 | total number of bytes to the AppServer | 644 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 645 | total number of bytes from the AppServer | 646 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 647 Figure 5: Raw Load Measurement Sub-TLV 649 Type =TBD1: Aggregated Load Measurement Index derived from 650 the Weighted combination of bytes/packets sent to/received 651 from the App server: 653 Index=w1*ToPackets+w2*FromPackes+w3*ToBytes+w4*FromBytes 655 Where wi is a value between 0 and 1; w1+ w2+ w3+ w4 = 1. 657 Internet-Draft AppMetaData for 5G EC Service 659 Type= TBD2: Raw measurements of packets/bytes to/from the 660 App Server address. 662 Measure Period: BGP Update period or user-specified period. 664 4.2. Capacity Index sub-TLV format 666 The Capacity Index sub-TLV has the following format: 668 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 669 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 670 | Type (TBD3) | Length | 671 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 672 | Capacity Index | 673 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 675 Note: "Capacity Index" can be more stable for each site. If 676 those values are configured to nodes, they might not need to 677 be included in every BGP UPDATE. 679 4.3. The Site Preference Index sub-TLV format 681 The site Preference Index is used to achieve Soft Anchoring 682 [Section 5] an application flow from a UE to a specific 683 location when the UE moves from one 5G site to another. 685 The Preference Index sub-TLV has the following format: 687 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 688 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 689 | Type (TBD4) | Length | 690 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 691 | Preference Index | 692 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 694 Note: "Site Preference Index" can be more stable for each 695 site. If those values are configured to nodes, they might not 696 need to be included in every BGP UPDATE. 698 Internet-Draft AppMetaData for 5G EC Service 700 5. AppMetaData Propagation Scope 702 AppMetaData is only to be distributed to the relevant ingress 703 nodes of the 5G EC local data networks. Only the ingress 704 routers that are configured with the 5G EC services ACLs need 705 to receive the AppMetaData for specific services. 707 For each registered EC service, a corresponding filter group 708 can be formed on RR to represent the interested ingress 709 routers that are interested in receiving the corresponding 710 AppMetaData information. 712 6. Metrics Change Rate Consideration 714 As the metrics change can impact the path selection, it is 715 recommended to control the update frequency to avoid rapid 716 route oscillations. 718 7. Soft Anchoring of an ANYCAST Flow 719 "Sticky Service" in the 3GPP Edge Computing specification 720 (3GPP TR 23.748) is about flows from a UE sticking to a 721 specific ANYCAST location when the UE moves from one 5G Site 722 to another. 724 "Soft Anchoring" is a mechanism for ingress routers to apply 725 preference to the path towards the previous server location 726 when the UE is anchored to a new UPF and continue using its 727 cached IP for the App server. 729 Let's assume one application "App.net" is instantiated on 730 four servers that are attached to four different routers R1, 731 R2, R3, and R4 respectively. It is desired for packets to the 732 "App.net" from UE-1 to stick with one server, say the App 733 Server attached to R1, even when the UE moves from one 5G 734 site to another. However, when there is a failure reaching R1 735 or the Application Server attached to R1, the packets of the 736 flow "App.net" from UE-1 need to be forwarded to the 737 Application Server attached to R2, R3, or R4. 739 We call this kind of sticky service "Soft Anchoring", meaning 740 that anchoring to the site of R1 is preferred, but other 741 sites can be chosen when the preferred site encounters a 742 failure. 744 Internet-Draft AppMetaData for 5G EC Service 746 Here are the detailed steps: 748 - Assign a group of ANYCAST addresses to one application. 749 For example, "App.net" is assigned with 4 ANYCAST 750 addresses, L1, L2, L3, and L4. L1/L2/L3/L4 represents 751 the location preferred ANYCAST addresses. 752 - For the App.net Server attached to a router, the router 753 has four Stub links to the same Server, L1, L2, L3, and 754 L4 respectively. The cost to L1, L2, L3, and L4 is 755 assigned differently for different egress routers. For 756 example, 757 o When attached to R1, the L1 has the lowest cost, 758 say 10, when attached to R2, R3, and R4, the L1 can 759 have a higher cost, say 30. 760 o ANYCAST L2 has the lowest cost when attached to R2, 761 higher cost when attached to R1, R3, R4 762 respectively. 763 o ANYCAST L3 has the lowest cost when attached to R3, 764 higher cost when attached to R1, R2, R4 765 respectively, and 766 o ANYCAST L4 has the lowest cost when attached to R4, 767 higher cost when attached to R1, R2, R3 768 respectively 769 - When a UE queries for the "App.net" for the first time, 770 the DNS reply has the location preferred ANYCAST 771 address, say L1, based on where the query is initiated. 772 - When the UE moves from one 5G site-A to Site-B, UE 773 continues sending packets of the "App.net" to ANYCAST 774 address L1. The routers will continue sending packets to 775 R1 because the total cost for the App.net instance for 776 ANYCAST L1 is lowest at R1. If any failure occurs making 777 R1 not reachable, the packets of the "App.net" from UE-1 778 will be sent to R2, R3, or R4 (depending on the total 779 cost to reach each of them). 781 If the Application Server supports the HTTP redirect, more 782 optimal forwarding can be achieved. 784 Internet-Draft AppMetaData for 5G EC Service 786 - When a UE queries for the "App.net" for the first time, 787 the global DNS reply has the ANYCAST address G1, which 788 has the same cost regardless of where the Application 789 servers are attached. 790 - When the UE initiates the communication to G1, the 791 packets from the UE will be sent to the Application 792 Server that has the lowest cost, say the Server attached 793 to R1. The Application server is instructed with HTTPs 794 Redirect to reply with a location-specific URL, say 795 App.net-Loc1. The client on the UE will query the DNS 796 for App.net-Loc1 and get the response of ANYCAST L1. The 797 subsequent packets from the UE-1 for App.net are sent to 798 L1. 800 8. Manageability Considerations 802 To be added. 804 9. Security Considerations 806 To be added. 808 10. IANA Considerations 810 Here are new Sub-TLV types requiring IANA registration: 812 Type = TBD1: Aggregated Load Measurement Index derived from 813 the Weighted combination of bytes/packets sent to/received 814 from the App server. 816 Type = TBD2: Raw measurements of packets/bytes to/from the 817 App Server address. 819 Type = TBD3: Capacity value sub-TLV 821 Type = TBD4: Site preference value sub-TLV 823 11. References 824 Internet-Draft AppMetaData for 5G EC Service 826 11.1. Normative References 828 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 829 Requirement Levels", BCP 14, RFC 2119, March 1997. 831 [RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private 832 networks (VPNs)", Feb 2006. 834 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in 835 RFC 2119 Key Words", BCP 14, RFC 8174, DOI 836 10.17487/RFC8174, May 2017, . 839 [RFC8200] s. Deering R. Hinden, "Internet Protocol, Version 6 840 (IPv6) Specification", July 2017 842 11.2. Informative References 844 [3GPP-EdgeComputing] 3GPP TR 23.748, "3rd Generation 845 Partnership Project; Technical Specification Group 846 Services and System Aspects; Study on enhancement of 847 support for Edge Computing in 5G Core network 848 (5GC)", Release 17 work in progress, Aug 2020. 850 [5G-EC-Metrics] L. Dunbar, H. Song, J. Kaippallimalil, "IP 851 Layer Metrics for 5G Edge Computing Service", draft- 852 dunbar-ippm-5g-edge-compute-ip-layer-metrics-00, 853 work-in-progress, Oct 2020. 855 [5G-Edge-Sticky] L. Dunbar, J. Kaippallimalil, "IPv6 Solution 856 for 5G Edge Computing Sticky Service", draft-dunbar- 857 6man-5g-ec-sticky-service-00, work-in-progress, Oct 858 2020. 860 [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation 861 Subsequent Address Family Identifier (SAFI) and the 862 BGP Tunnel Encapsulation Attribute", April 2009. 864 [BGP-SDWAN-Port] L. Dunbar, H. Wang, W. Hao, "BGP Extension 865 for SDWAN Overlay Networks", draft-dunbar-idr-bgp- 866 sdwan-overlay-ext-03, work-in-progress, Nov 2018. 868 Internet-Draft AppMetaData for 5G EC Service 870 [SDWAN-EDGE-Discovery] L. Dunbar, S. Hares, R. Raszuk, K. 871 Majumdar, "BGP UPDATE for SDWAN Edge Discovery", 872 draft-dunbar-idr-sdwan-edge-discovery-00, work-in- 873 progress, July 2020. 875 [Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation 876 Attribute", draft-ietf-idr-tunnel-encaps-10, Aug 877 2018. 879 12. Acknowledgments 881 Acknowledgements to Donald Eastlake for their review and 882 contributions. 884 This document was prepared using 2-Word-v2.0.template.dot. 886 Authors' Addresses 888 Linda Dunbar 889 Futurewei 890 Email: ldunbar@futurewei.com 892 Kausik Majumdar 893 CommScope 894 350 W Java Drive, Sunnyvale, CA 94089 895 Email: kausik.majumdar@commscope.com 897 Haibo Wang 898 Huawei 899 Email: rainsword.wang@huawei.com 901 Gyan Mishra 902 Verizon 903 Email: gyan.s.mishra@verizon.com