Network Working Group                                   L. Dunbar
Internet Draft                                            H. Song
                                                J. Kaippallimalil
Intended status: Standard                               Futurewei
Expires: April 26, 2021

                                                 October 26, 2020



           IP Layer Metrics for 5G Edge Computing Service
       draft-dunbar-ippm-5g-edge-compute-ip-layer-metrics-00

Abstract

   This draft describes the IP Layer metrics and methods to
   measure the Edge Computing running status in order for IP
   networks to select the optimal application server location in
   5G Edge Computing (EC) environment. Those measurements are
   for network to dynamically optimize the forwarding of 5G edge
   computing service without any knowledge above IP layer.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79. This document may not be
   modified, and derivative works of it may not be created,
   except to publish it as an RFC and to translate it into
   languages other than English.

   Internet-Drafts are working documents of the Internet
   Engineering Task Force (IETF), its areas, and its working
   groups.  Note that other groups may also distribute working
   documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of
   six months and may be updated, replaced, or obsoleted by
   other documents at any time.  It is inappropriate to use
   Internet-Drafts as reference material or to cite them other
   than as "work in progress."



xxx, et al.             Expires April 26, 2021           [Page 1]

         IP Layer Metrics for 5G Edge Computing Services

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed
   at http://www.ietf.org/shadow.html

   This Internet-Draft will expire on April 7, 2021.

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as
   the document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date
   of publication of this document. Please review these
   documents carefully, as they describe your rights and
   restrictions with respect to this document. Code Components
   extracted from this document must include Simplified BSD
   License text as described in Section 4.e of the Trust Legal
   Provisions and are provided without warranty as described in
   the Simplified BSD License.

Table of Contents


   1. Introduction..............................................3
      1.1. 5G Edge Computing Background.........................3
      1.2. Problem 1: Selecting 5G Edge Application Location....4
      1.3. Problem 2: UE mobility creates unbalanced anycast
      distribution..............................................5
   2. Conventions used in this document.........................7
   3. IP-Layer Metrics Definitions for 5G EC services...........9
      3.1. IP-Layer Application ID..............................9
      3.2. IP-Layer metric for App Server Load Measurement......9
      3.3. Capacity Index in the overall cost..................11
      3.4. Site Preference Index in the overall cost...........11
      3.5. RTT to an ANYCAST Address in 5G EC..................12
   4. Algorithm in Selecting the optimal Target Location.......13
   5. Scope of IP Layer Metrics Advertisement..................14
   6. Manageability Considerations.............................14
   7. Security Considerations..................................14
   8. IANA Considerations......................................15


Dunbar, et al.          Expires April 26, 2021           [Page 2]

         IP Layer Metrics for 5G Edge Computing Services

   9. References...............................................15
      9.1. Normative References................................15
      9.2. Informative References..............................15
   10. Acknowledgments.........................................16

1. Introduction
1.1. 5G Edge Computing Background

   In 5G Edge Computing environment [3GPP-EdgeComputing], one
   Application can have multiple Application Servers hosted in
   different Edge Computing data centers that are close in
   proximity. Those Edge Computing (mini) data centers are
   usually very close to, or co-located with, 5G base stations,
   with the goal to minimize latency and optimize the user
   experience.

   When a UE (User Equipment) initiates application packets
   using the destination address from a DNS reply or from its
   own cache, the packets from the UE are carried in a PDU
   session through 5G Core [5GC] to the 5G UPF-PSA (User Plan
   Function - PDU Session Anchor). The UPF-PSA decapsulate the
   5G GTP outer header and forwards the packets from the UEs to
   the Ingress router of the Edge Computing (EC) Local Data
   Network (LDN). The LDN for 5G EC, which is the IP Networks
   from 5GC perspective, is responsible for forwarding the
   packets to the intended destinations.

   Routers in the local IP network should be able to select the
   "best" or "closest" application server location out of many.
   However, simply using distance alone as a metric may not be
   sufficient as there may be many locations in close proximity.
   Moreover, one of the main aims of locating application
   servers so close to the user is to provide lower latency.
   When a user moves and attaches from another access router
   (UPF), the local IP network should be able to continue
   routing to the established application server. As a user
   keeps moving further away, a closer application server maybe
   able to serve the user better. Network measurements,
   including latency of various paths are provided to the
   application domain to assist in reselection. These problems
   are outlined in sections 1.2, and 1.3.



Dunbar, et al.          Expires April 26, 2021           [Page 3]

         IP Layer Metrics for 5G Edge Computing Services


1.2. Problem 1: Selecting 5G Edge Application Location

   Having multiple locations closer to UEs to host one
   Application server can greatly improve the user experience.
   But selecting an optimal location for the application traffic
   from a UE may not be that simple.

   Using DNS to reply with the address of the Application Server
   location closest to the requesting UE can encounter issues
   like:
     - UE can cache results indefinitely, when the UE moves to a
       5G cell site very far away, the cached address may still
       be used, which can incur large network delay.
     - The Application Server at a specific location whose
       address replied by the DNS might be heavily loaded
       causing slow or no response, when there are available low
       utilized Application Server, for the same application, at
       different locations very close in proximity.
     - No inherent leverage of proximity information present in
       the network (routing) layer, resulting in loss of
       performance
     - Local DNS resolver become the unit of traffic management


   Increasingly, Anycast is used extensively by various
   application providers and CDNs because ANYCAST makes it
   possible to dynamically load balance across locations that
   host the Application server based on network conditions.
   Application server location selection using Anycast address
   leverages the proximity information present in the network
   (routing) layer and eliminates the single point of failure
   and bottleneck at the DNS resolvers and application layer
   load balancers. Another benefit of using ANYCAST address is
   removing the dependency on UEs that use their cached
   destination IP addresses for extended period.

   But selection of an ANYCAST location purely based on the
   network condition can encounter issue of the location
   selected by network routing information being overutilized
   while there are available underutilized locations close by.



Dunbar, et al.          Expires April 26, 2021           [Page 4]

         IP Layer Metrics for 5G Edge Computing Services



1.3. Problem 2: UE mobility creates unbalanced anycast
   distribution

   Another problem of using ANYCAST address for multiple
   locations of an Application Server in 5G environment is that
   UEs' frequent moving from one 5G site to another. The
   frequent move of UEs can make it difficult to plan where
   Application server should be hosted. When a large number of
   UEs using a particular Application congregate together
   unpredictably, the ANYCAST location selected based on routing
   distance can be heavily utilized, while the same Application
   Server at other locations close-by are underutilized.




































Dunbar, et al.          Expires April 26, 2021           [Page 5]

         IP Layer Metrics for 5G Edge Computing Services

   +--+
   |UE|---\+---------+                 +------------------+
   +--+    |  5G     |  +-----------+  |  S1: aa08::4450  |
   +--+    | Site A +----+         +----+                 |
   |UE|----|        | Ra |         | R1 | S2: aa08::4460  |
   +--+    |        +----+         +----+                 |
  +---+    |         |  |           |  |  S3: aa08::4470  |
  |UE1|---/+---------+  |           |  +------------------+
  +---+                 |IP Network |       L-DN1
                        |(3GPP N6)  |
     |                  |           |  +------------------+
     |                  |           |  |  S1: aa08::4450  |
     |                  |          +----+                 |
     |                  |          | R3 | S2: aa08::4460  |
     v                  |          +----+                 |
                        |           |  |  S3: aa08::4470  |
                        |           |  +------------------+
                        |           |      L-DN3
   +--+                 |           |
   |UE|---\+---------+  |           |  +------------------+
   +--+    |  5G     |  |           |  |  S1: aa08::4450  |
   +--+    | Site B +----+         +----+                 |
   |UE|----|        | Rb |         | R2 | S2: aa08::4460  |
   +--+    |        +----+         +----+                 |
   +--+    |         |  +-----------+  |  S3: aa08::4470  |
   |UE|---/+---------+                 +------------------+
   +--+                                     L-DN2
     Figure 1: multiple ANYCAST instances in different edge DCs




This document describes the measurements at IP Layer that can
reflect the application server running status and environment at
the specific locations. This document also describes the method
of incorporating those measurements with network routing cost to
come up with a more optimal criteria in selecting ANYCAST
locations.


Note: for the ease of description, the Edge Application Server
and Application Server are used interchangeably throughout this
document.


Dunbar, et al.          Expires April 26, 2021           [Page 6]

         IP Layer Metrics for 5G Edge Computing Services





2. Conventions used in this document


   A-ER:       Egress Router to an Application Server Instance,
               [A-ER] is used to describe the last router that
               the application instance is attached. For 5G EC
               environment, the A-ER can be the gateway router
               to the Edge Computing Data Center.

   ANYCAST Instance: refer to the Application Server Gateway at
               a specific location which is reachable by the
               ANYCAST address.

   Application Server: An application server is a physical or
               virtual server that host the software system for
               the application.

   Application Server Location: Represent a cluster of servers
               at one location serving the same Application. One
               application may have a Layer 7 Load balancer,
               whose address(es) are reachable from external IP
               network, in front of a set of application
               servers. From IP network perspective, this whole
               group of servers are considered as the
               Application server at the location.

   EC:         Edge Computing

   Edge Application Server: used interchangeably with
               Application Server throughout this document.

   Edge Computing Hosting Environment: An environment, such as
               psychical or virtual machines, providing support
               required for Edge Application Server's execution.

               NOTE: The above terminologies are the same as
               those used in 3GPP TR 23.758



Dunbar, et al.          Expires April 26, 2021           [Page 7]

         IP Layer Metrics for 5G Edge Computing Services

   Edge DC:    Edge Data Center, which provides the Edge Hosting
               Environment. It might be co-located with 5G Base
               Station and not only host 5G core functions, but
               also host frequently used Edge server instances.



   gNB         next generation Node B

   Instance:   the term "Instance" if used in the context of
               Application Server, is referring to one location
               of an application server; if used in the ANYCAST
               context, is referring to one location of the
               Application server with the same ANYCAST address.



   L-DN:       Local Data Network

   PSA:        PDU Session Anchor (UPF)

   RTT:        Round Trip Time

   RTT-ANYCAST:   A list of Round trip times to a group of
               routers that have the ANYCAST instances directly
               attached.

   SSC:        Session and Service Continuity

   UE:         User Equipment

   UPF:        User Plane Function


   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
   NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
   RECOMMENDED", "MAY", and "OPTIONAL" in this document are to
   be interpreted as described in BCP 14 [RFC2119] [RFC8174]
   when, and only when, they appear in all capitals, as shown
   here.





Dunbar, et al.          Expires April 26, 2021           [Page 8]

         IP Layer Metrics for 5G Edge Computing Services

3. IP-Layer Metrics Definitions for 5G EC services

 3.1. IP-Layer Application ID

   From network perspective, an application server has an
   Identifier, or IP Layer Application Server ID, which is
   usually an ANYCAST address that can represent multiple
   locations that host the application server.



 3.2. IP-Layer metric for App Server Load Measurement

   There are many network techniques and protocols to optimize
   forwarding and ensuring QoS for applications, such as
   DSCP/DiffServ, Traffic Engineered (TE) solutions, Segment
   Routing, etc. But the reality is that most application
   servers don't expose their internal logics to network
   operators. Their communications are generally encrypted. Most
   of them do not even respond to PING or ICMP messages
   initiated by routers or network gears.

   Even without knowledge of application internal logics,
   network layer or IP Layer can monitor the traffic patterns
   to/from the Application Server at each location to gauge the
   running status of the application server at the location.

   First, the network needs to discover which router(s) has the
   application server attached. Those routers are called
   Application Server Egress Router, or A-ER for short. A-ER is
   usually the Gateway Router to an Edge Computing Data Center.
   To discover if a (Gateway) router is the A-ER for a specific
   Edge Application Server, the (Gateway) router can
   periodically send reverse ARP (IPv4) or Neighbor Discovery
   scan with the address of the Application Server to discover
   if the Application Servers are hosted in its edge computing
   data center. If yes, the router or routers are identified as
   the A-ER for the Application Server. For one Application
   Server, there can be many A-ERs at different EC Data Centers.

   For an Application Server at a specific location, which is
   identified by the address of the application server at the IP
   layer, the A-ER can measure the amount of traffic destined
   towards the address & the amount of the traffic from the
   specific address, such as:





Dunbar, et al.          Expires April 26, 2021           [Page 9]

         IP Layer Metrics for 5G Edge Computing Services

     - Total number of packets to the attached App Server
       (ToPackets);
     - Total number of packets from the attached App Server
       (FromPackets);
     - Total number of Bytes to the attached App Server
       (ToBytes);
     - Total number of bytes from the attached App Server
       (FromBytes);


   The A-ER can advertise those measurements periodically, by BGP
   UPDATE messages or OSPF/ISIS Link statement Advertisements, to
   a group of routers that have traffic destined towards the
   ANYCAST addresses of those application servers.

   The actual load measurement to the App Server attached to an A-
   ER can be based on one of the metrics above or including all
   four metrics with different weights applied to each, such as:

   LoadIndex =
   w1*ToPackets+w2*FromPackes+w3*ToBytes+w4*FromBytes

          Where 1<wi<1 and w1+ w2+ w3+ w4 = 1.

   The weights of each metric contributing to the load index of
   the App Server attached to an A-ER can be configured or learned
   by self-adjusting based on user feedbacks.

   Even though it would be better to have applications or their
   controllers directly reporting their own workload running
   status  to network, it is not easy to have third party
   applications to provide information to network operators in
   addition to that applications servers can be running anywhere
   and.

   The IP-Layer Load Measurements provides an intelligent
   estimate of the application server running status at a
   specific location without requiring cooperation from third
   party Applications or Application controllers.





Dunbar, et al.          Expires April 26, 2021          [Page 10]

         IP Layer Metrics for 5G Edge Computing Services



 3.3. Capacity Index in the overall cost

   Given that different Edge Computing DCs may have different
   capacity, the following metrics need to be included to gauge
   an application Server's running status at a specific
   location:

   - Capacity Index:
     Capacity Index is used to differentiate the running
     environment of the application server. Some data centers
     can have hundreds, or thousands, of servers behind an
     Application Server's App Layer Load Balancer that is
     reachable from external world. Other data centers can have
     very small number of servers for the application server.
     "Capacity Index", which is a numeric number, is used to
     represent the capacity of the application server in a
     specific location.

     "Capacity Index" can be a configured value indicating the
     capacity of a specific Application Server at a specific
     location, e.g. an Edge Computing DC. The higher value means
     the higher capacity.  The Capacity Index is Application
     Server specific, meaning at one location, one Application
     Server can have the Capacity Index to be 10 and another
     server can be 2.

     If the Application Server capacity configuration is not
     available, a network analytics tool can use the historic
     measurements as the basis to estimate the site capacity. If
     an Application Server at a specific site has high volume
     for extended period historically, the site capacity can be
     considered as higher than the other site with historic low
     volume. This is under the assumption that application
     controllers monitor utilization of the application servers
     at different locations. If an application server has
     prolonged over-utilized servers at some locations, the
     application controller will trigger manual intervention to
     increase the computing powers at those locations. However,
     the manual intervention cycles can be in weeks/months. That
     is why the IP layer metrics and algorithms that can change
     flows direction in minutes become very essential.



 3.4. Site Preference Index in the overall cost



Dunbar, et al.          Expires April 26, 2021          [Page 11]

         IP Layer Metrics for 5G Edge Computing Services

     As described in [IPv6-StickyService] and [ISPF-EXT-EC], an
     EC sticky service needs to connect a UE to the application
     server that has been serving the UE before the UE moves to
     a new 5G Site, unless there is failure to that location.

     To achieve the goal of sticking a flow from one specific UE
     to a specific site, a "site Preference Index" is created.
     The value of the Site Preference Index can be manipulated
     for packets of some flows to be steered towards an
     application server location farther away in routing
     distance. The "Site Preference Index" enables some sites to
     be more preferred for handling the UE traffic to a
     application server than others.

 3.5. RTT to an ANYCAST Address in 5G EC

     ANYCAST used in 5G Edge computing environment is slightly
     different from the typical ANYCAST address being deployed.
     Typical ANYCAST address is used to represent instances in
     vast different geographical locations, such as different
     continents. ANCAST address for "app.net" for Asia lead
     packets to a server instance of "app.net" hosted in Asia.
     Therefore, the RTT for "app.net" in Asia, is a single value
     that represent the round time trip to the server in Asia
     that host the "app.net".

     5G Edge Computing environment can have one Application
     server hosted in multiple Edge Computing DCs close in
     proximity. Routers, i.e. the ingress router to 5G LDN
     (Local Data Network), can forward packets for the ANYCAST
     address of "app.net" to different egress routers that have
     "app.net" servers attached.

     If "app.net" is hosted in four different 5G Edge Computing
     Data Centers. All those DCs have the same ANYCAST address
     for the "app.net". The RTT to "app.net" ANYCAST address
     need to be a group of values (instead of one RTT value to a
     unicast address). The RTT group value should include the A-
     ER router's specific unicast address (e.g. the loopback
     address) to which the Application Server is attached.

     RTT to "app.net" ANYCAST Address is represented as:

         List of {Egress Router address, RTT value}

     This list is called "RTT-ANYCAST".




Dunbar, et al.          Expires April 26, 2021          [Page 12]

         IP Layer Metrics for 5G Edge Computing Services

     In order to better optimize the ANYCAST traffic, each
     router adjacent to 5G PSA needs to periodically measure RTT
     to a list of A-ER routers that advertise the ANYCAST
     address. The RTT to egress router at Site-i is considered
     as the RTT to the ANYCAST instance at the Site-i.



4. Algorithm in Selecting the optimal Target Location

     The goal of the algorithm is to equalize the traffic among
     multiple locations of the same ANYCAST address.

     The main benefit of using ANYCAST is to leverage the IP-
     layer information to equalize the traffic among multiple
     locations of the same Application Server, usually
     identified by one or a group of ANYCAST addresses.

     For 5G Edge Computing environment, the ingress router to
     each LDN needs to be notified of the Load Index and
     Capacity Index of the Application Servers at different EC
     site to make the intelligent decision on where to forward
     the traffic from UEs for an application.

     The Algorithm needs to take the following attributes into
     consideration:

           - Load Measurement Index [Section 3.2],
           - capacity index [Section 3.3],
           - Preference Index [Section 3.4], and
           - network delay [Section 3.5].

     Here is an algorithm for a router, e.g. the router directly
     attached to the 5G PSA, to compare the cost to reach the
     Application Server between the Site-i or Site-j:

            alpha*(LoadIndex-i*Beta-i)    (1-alpha)*(Delay-i * gamma-i)
  Cost=min( --------------- ----------  + ---------------------------- )
               (LoadIndex-j * Beta-j)          ( Delay-j *gamma-j)


     LoadIndex-i: weighted combination of the total bytes
     (or/and packets) sent to/received from the Application
     Server at Site-i during a fixed time period.

     Beta-i (larger value means higher capacity): capacity index
     at the site i.


Dunbar, et al.          Expires April 26, 2021          [Page 13]

         IP Layer Metrics for 5G Edge Computing Services

     Delay-i: Network latency measurement (RTT) to the A-ER that
     has the Application Server attached at the site-i.

     gamma (larger value means higher preference): Network
     Preference index for the site-I.

     alpha (a value between 0 and 1: Weight for load & site
     Index. If smaller than 0.5, Network latency has more
     influence; otherwise, Server load has more influence).

5. Scope of IP Layer Metrics Advertisement

   Each Application Server might be used by a small group of
   UEs. Therefore, it is not necessary for A-ER router to
   advertise the IP layer metrics to all other routers in the 5G
   LDN. Likewise, each EC Data Center may only host a small
   number of application servers.

   "Application Bound Group Routers" is used to refer a group of
   routers that are interested in a group of specific ANYCAST
   addresses. The IP Layer Metrics for an Application Server
   should be advertised among the routers in the "Application
   bound Group Routers".

   BGP RT Constrained Distribution [RFC4684] can be used to form
   the "Application Bound Group Routers".

   Since there are much more Application Servers than the number
   of routers in 5G LDN, a more practical way to form the
   "Application Bound Group of Routers" is for each ingress
   router to query a network controller upon receiving the first
   packet to a specific ANYCAST address to be included in the
   "Application Bound Group Routers". There should be a timer
   associated with Ingress router, as the UE that uses the
   Application Server might move away. Upon timer expires, the
   Ingress Router is removed from the "Application Bound Group
   of Routers".



6. Manageability Considerations

     To be added.

7. Security Considerations




Dunbar, et al.          Expires April 26, 2021          [Page 14]

         IP Layer Metrics for 5G Edge Computing Services

   To be added.

8. IANA Considerations

       To be added.

9. References


 9.1. Normative References

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private
             networks (VPNs)", Feb 2006.

   [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in
             RFC 2119 Key Words", BCP 14, RFC 8174, DOI
             10.17487/RFC8174, May 2017, <https://www.rfc-
             editor.org/info/rfc8174>.

   [RFC8200] s. Deering R. Hinden, "Internet Protocol, Version 6
             (IPv6) Specification", July 2017


9.2. Informative References

   [3GPP-EdgeComputing] 3GPP TR 23.748, "3rd Generation
             Partnership Project; Technical Specification Group
             Services and System Aspects; Study on enhancement
             of support for Edge Computing in 5G Core network
             (5GC)", Release 17 work in progress, Aug 2020.

   [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation
             Subsequent Address Family Identifier (SAFI) and the
             BGP Tunnel Encapsulation Attribute", April 2009.

   [BGP-SDWAN-Port] L. Dunbar, H. Wang, W. Hao, "BGP Extension
             for SDWAN Overlay Networks", draft-dunbar-idr-bgp-
             sdwan-overlay-ext-03, work-in-progress, Nov 2018.




Dunbar, et al.          Expires April 26, 2021          [Page 15]

         IP Layer Metrics for 5G Edge Computing Services

   [SDWAN-EDGE-Discovery] L. Dunbar, S. Hares, R. Raszuk, K.
             Majumdar, "BGP UPDATE for SDWAN Edge Discovery",
             draft-dunbar-idr-sdwan-edge-discovery-00, work-in-
             progress, July 2020.

   [Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation
             Attribute", draft-ietf-idr-tunnel-encaps-10, Aug
             2018.



10. Acknowledgments

   Acknowledgements to Donald Eastlake for their review and
   contributions.

   This document was prepared using 2-Word-v2.0.template.dot.






























Dunbar, et al.          Expires April 26, 2021          [Page 16]

         IP Layer Metrics for 5G Edge Computing Services

Authors' Addresses


   Linda Dunbar
   Futurewei
   Email: ldunbar@futurewei.com

   HaoYu Song
   Futurewei
   Email: haoyu.song@futurewei.com

   John Kaippallimalil
   Futurewei
   Email: john.kaippallimalil@futurewei.com


































Dunbar, et al.          Expires April 26, 2021          [Page 17]