DETNET Q. Xiong Internet-Draft ZTE Corporation Intended status: Standards Track Z. Du Expires: 28 August 2022 China Mobile February 2022 DetNet Enhancements for Large-Scale Deterministic Networks draft-xiong-detnet-large-scale-enhancements-00 Abstract This document describes enhancements to DetNet to achieve the differentiated DetNet QoS in large-scale deterministic networks including the overall requirements and solutions with deterministic resources, routes and services. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 5 August 2022. Copyright Notice Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Xiong & Du Expires 28 August 2022 [Page 1] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Conventions used in this document . . . . . . . . . . . . . . 3 2.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 2.2. Requirements Language . . . . . . . . . . . . . . . . . . 3 3. DetNet Applicability for Large-Scale Deterministic Networks . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Overall Requirements of Large-Scale Deterministic Networks . 5 4.1. Service Requirements . . . . . . . . . . . . . . . . . . 5 4.1.1. Support the Differentiated DetNet QoS of Multiple Services . . . . . . . . . . . . . . . . . . . . . . 5 4.1.2. Guarantees of Multiple Dynamic Deterministic Flows . 7 4.2. Route Requirements . . . . . . . . . . . . . . . . . . . 7 4.2.1. Support the Distributed Deterministic Routes . . . . 8 4.2.2. Support the Inter-domain Deterministic Routes . . . . 8 4.2.3. Support the Replication and Elimination Deterministic Routes . . . . . . . . . . . . . . . . . . . . . . . 8 4.3. Resource Requirements . . . . . . . . . . . . . . . . . . 8 4.3.1. Management and Scheduling of the Network Resources . 8 4.3.2. Support the Utilization of Heterogeneous Resources . 9 5. Solutions of Large-Scale Deterministic Networks . . . . . . . 9 5.1. Enhanced Layering Model . . . . . . . . . . . . . . . . . 9 5.2. Mechanisms to Achieve Differentiated DetNet QoS . . . . . 10 5.2.1. Deterministic Resources . . . . . . . . . . . . . . . 10 5.2.2. Deterministic Routes . . . . . . . . . . . . . . . . 11 5.2.3. Deterministic Services . . . . . . . . . . . . . . . 11 6. Security Considerations . . . . . . . . . . . . . . . . . . . 12 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 9. Normative References . . . . . . . . . . . . . . . . . . . . 12 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12 1. Introduction 5G network is oriented to the internet of everything. In addition to the Enhanced Mobile Broadband (eMBB) and Massive Machine Type Communications(mMTC) services, it also supports the Ultra-reliable Low Latency Communications (uRLLC) services. The uRLLC services demand SLA guarantees such as low latency and high reliability and other deterministic and precise properties especially in Wide Area Network (WAN) applications. The uRLLC services should be provided in large-scale networks which cover the industries such as intelligent electrical network, intelligent factory, internet of vehicles, industry automation and other industrial internet scenarios. The industrial internet is the key infrastructure that coordinate various units of work over various Xiong & Du Expires 28 August 2022 [Page 2] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 system components, e.g. people, machines and things in the industrial environment including big data, cloud computing, Internet of Things (IOT), Augment Reality (AR), industrial robots, Artificial Intelligence (AI) and other basic technologies. For the intelligent electrical network, there are deterministic requirements for communication delay, jitter and packet loss rate. For example, in the electrical current difference model, a delay of 3~10ms and a jitter variation is no more than 100us are required. For the automation control, it is one of the basic application and the the core is closed-loop control system. The control process cycle is as low as millisecond level, so the system communication delay needs to reach millisecond level or even lower to ensure the realization of precise control. There are three levels of real-time requirements for industrial interconnection: factory level is about 1s, and process level is 10~100ms, and the highest real-time requirement is motion control, which requires less than 1ms. According to [RFC8655], Deterministic Networking (DetNet) operates at the IP layer and delivers service which provides extremely low data loss rates and bounded latency within a network domain. The applications in 5G networks demand much more deterministic and precise properties in WAN. The existing deterministic technologies are facing large-scale number of nodes and long-distance transmission, traffic scheduling, dynamic flows, and other controversial issues in large-scale networks. This document describes enhancements to DetNet to achieve the differentiated DetNet QoS in large-scale deterministic networks including the overall requirements and solutions with deterministic resources, routes and services. 2. Conventions used in this document 2.1. Terminology The terminology is defined as [RFC8655]. 2.2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. Xiong & Du Expires 28 August 2022 [Page 3] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 3. DetNet Applicability for Large-Scale Deterministic Networks As per [RFC8655], it defined the overall architecture for DetNet, which provides a capability for real-time applications with extremely low data loss rates and bounded latency within a network domain. It has three goals: minimum and maximum end-to-end latency from source to destination, bounded jitter (packet delay variation), packet loss ratio and upper bound on out-of-order packet delivery. To achieve the above objectives, multiple techniques need to be used in combination, including explicit routes, service protection and resource allocation defined by DetNet. And the DetNet functionality is implemented at DetNet service sub-layer and DetNet forwarding sub- layer. It is required to analyse the applicability in DetNet for large-scale deterministic networks. From the perspective of services requirements discussed in section 4.1, a large-scale network needs to provide the deterministic service for various applications. And the deterministic service may demand different deterministic QoS requirements according to different application scenarios. The service protection in service sub-layer is not sufficient to meet the services requirements of large-scale networks, it should provide unified planning and scheduling mechanisms for service flows to perform end-to-end delay and jitter control and achieve differentiated DetNet QoS of multiple services. The large-scale deterministic networks have a large number of hops and high link delay, which makes it difficult to achieve network-wide precise time synchronization. It may across multiple IP domains, or there may be different heterogeneous forwarding plane transport technologies. It is required to consider the efficiency of resources utilization and routes steering. From the perspective of routes requirements discussed in section 4.3, a large-scale network should provide the deterministic paths for the services in large-scale networks. The deterministic routes should be calculated based on the deterministic metrics such as the end-to-end bounded latency and jitter. The forwarding sub-layer should establish the deterministic routes with SLA guarantees based on the deterministic resources. Moreover, other than explicit routes in centralized control scenarios, the distributed routes when the DetNet deployed with no controller may be more important for large-scale networks. From the perspective of resources requirements discussed in section 4.3, a large-scale network should utilize the bandwidth, nodes, links, jitter resource, and queue scheduling resource and the other heterogeneous resources to establish the deterministic links which could provide SLA guarantees for the deterministic forwarding Xiong & Du Expires 28 August 2022 [Page 4] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 capabilities at different levels. Other than resource allocation, the forwarding sub-layer should support the unified and simplified scheduling and management mechanism for resources. For example, resource modeling, isolation and reservation should be considered to guarantee the deterministic transmission. It is required to provide mechanisms within DetNet service and forwarding sub-layers to meet the requirements of large-scale deterministic networks. This document describes enhancements to DetNet to achieve the differentiated DetNet QoS in large-scale deterministic networks including the overall requirements and solutions with deterministic resources, routes and services. 4. Overall Requirements of Large-Scale Deterministic Networks As per [draft-liu-detnet-large-scale-requirements], the technical and operational requirements have been specified for large-scale deterministic networks. For DetNet architecture to support deterministic service in a large-scale network, the requirements from services, routes and resources also need to be considered. 4.1. Service Requirements 4.1.1. Support the Differentiated DetNet QoS of Multiple Services As defined in [RFC8655], the DetNet QoS can be expressed in terms of : Minimum and maximum end-to-end latency, bounded jitter (packet delay variation), packet loss ratio and an upper bound on out-of- order packet delivery. As described in [RFC8578], DetNet applications differ in their network topologies and specific desired behavior and different services requires differentiated DetNet QoS. In the large-scale networks, multiple services with differentiated DetNet QoS is co-existed in the same DetNet network. The classification of the deterministic flows within different levels is should be taken into considerations. It is required to provide Latency, bounded jitter and packet loss dynamically and flexibly in all scenarios for each characterized flow. As the Figure 1 shows, the services can be divided into 5 levels and level 2~5 is the DetNet flows and level-1 is non-DetNet flow. DetNet applications and DetNet QoS is differentiated within each level. Xiong & Du Expires 28 August 2022 [Page 5] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 +-------------+-----------+----------+----------+----------+-----------+ | Item | Level-1 | Level-2 | Level-3 | Level-4 | Level-5 | +-------------+-----------+----------+----------+----------+-----------+ | Applications| Broadcast | Voice | Audio and| AR/VR | Industrial| | Examples | | | Video | | | +-------------+-----------+----------+----------+----------+-----------+ | DetNet QoS | Bandwidth | Jitter | Latency | Low | Ultra-low | | | Guarantee | Guarantee| Guarantee| latency |latency and| | | | | |and jitter| jitter | +-------------+-----------+----------+----------+----------+-----------+ Figure 1: Figure 1: The classification of multiple services From the perspective of deterministic service requirements, deterministic Quality of Service (QoS) in the network can be divided into five types or levels: Level-1: bandwidth guarantee. The indicator requirements include basic bandwidth guarantee and certain packet loss tolerance. There is no requirement for the upper bound of the latency, and no requirement for the jitter. Typical services include download and FTP services. Level-2: jitter guarantee. The indicator requirements include: jitter 50ms, delay 300ms. Typical services include synchronous voice services, such as voice call. Level-3: Latency guarantee. The indicator requirements include: delay 50ms, jitter 50ms. Typical services include real-time communication services, such as video, production monitoring, and communication services. Level-4: low delay and low jitter guarantee. The indicator requirements include: delay 20ms, jitter 5ms. Typical services include video interaction services, such as AR/VR, holographic communication, cloud video and cloud games. Level-5: ultra-low delay and jitter guarantee. The indicator requirements include: delay 10ms, jitter 100us. Typical services include production control services, such as power protection and remote control. Moreover, different DetNet services is required to tolerate different percentage of packet loss ratio such as 99.9%, 99.99%, 99.999%, and so on. It is also required to provide service isolation. In some scenarios, such as intelligent electrical network, the isolation Xiong & Du Expires 28 August 2022 [Page 6] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 requirements are very important. For example, the automatic operation or control of a process or isochronous data and service with different priorities need to meet the requirements of hard isolation. In addition to the requirements of delay and jitter, the differential protection (DP) service needs to be isolated from other services and hard isolated tunnel is required. 4.1.2. Guarantees of Multiple Dynamic Deterministic Flows As described in [RFC8557], deterministic forwarding can only apply to flows with such well-defined characteristics as periodicity and burstiness. As defined in DetNet architecture [RFC8655], the traffic characteristics of an App-flow can be CBR (constant bit rate) or VBR (variable bit rate) of L1, L2 and L3 layers (VBR takes the maximum value when reserving resources). But the current scenarios and technical solutions only consider CBR flow, without considering the coexistence of VBR and CBR, the burst and aperiodicity of flows. The operations such as shaping or scheduling have not been specified. Even TSN mechanisms are based on a constant and forecastable traffic characteristics. It will be more complicated in WAN applications where much more flows coexist and the traffic characteristics is more dynamic. A huge number of flows with different DetNet QoS requirements is dynamically concurrent and the state of each flow cannot be maintained. It is required to offer reliable delivery and SLA guarantee for dynamic flows. For example, periodic flow and aperiodic flow (including micro burst flow, etc.), CBR and VBR flow, flow with different periods or phases, etc. When the network needs to forward these deterministic flows at the same time, it must solve the problems of time micro bursts, queue processing and aggregation of multiple flows. It is required to guarantee the deterministic QoS of multiple dynamic flows. Flow shaping and concurrent and micro-burst control should be provided. 4.2. Route Requirements Traditional routes only have reachability. Deterministic requirements such as delay and jitter are only used as path computation constraints. The paths vary with the real-time change of the network topology. They do not have Service Level Agreement (SLA) capability, and cannot meet the deterministic requirements at different levels. On the basic of the resources, the steering path and routes for deterministic flows should be programmed before the flows coming and able to provide SLA capability. And the routes should be considered to be established in distributed and centralized control Plane. Xiong & Du Expires 28 August 2022 [Page 7] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 4.2.1. Support the Distributed Deterministic Routes In large-scale deterministic networks, the distributed scenario with no controller should be taken into consideration. It is required to support the distributed deterministic routes which are established by distributed protocols such as IGP. 4.2.2. Support the Inter-domain Deterministic Routes In large-scale deterministic networks, it may across multiple network domains, it is required to support the inter-domain deterministic routes to achieve the end-to-end latency, bounded jitter. And the deadline of latency and jitter of each domain and segment should be determined and controlled. The inter-domain mechanism MUST be considered at the boundary nodes such as BGP configurations. 4.2.3. Support the Replication and Elimination Deterministic Routes As described in [RFC8557], the packet replication and elimination service protection should be provided to achieve the low packet loss ratio. It will copy the flows and spread the data over multiple disjoint forwarding paths. The bounded latency and jitter of each path should be meet service deterministic requirement. And the difference of latency within these paths should be limited. So the replication and elimination deterministic routes with configured latency and jitter policy should be supported. 4.3. Resource Requirements 4.3.1. Management and Scheduling of the Network Resources Traditional Ethernet, IP and MPLS networks which is based on statistical multiplexing provides best-effort packet service and offers no delivery and SLA guarantee. As described in [RFC8655], the primary technique by which DetNet achieves its QoS is to allocate sufficient resources. But it can not be achieved by not sufficient resource which can be allocated due to practical and cost reason. So it is required to achieve the high-efficiency of resources utilization when provide the DetNet service. Network resources include nodes, links, ports, bandwidth, queues, etc. The congestion control, shaping and queue scheduling and other traffic mechanisms which have been proposed in IEEE 802.1 TSN such as IEEE802.1Qbv, IEEE802.1Qch, IEEE802.1Qav, IEEE802.1Qcr and so on. Resource classification and modeling is required along with the explicit path with more SLA guarantee parameters like bandwidth, latency, jitter, packet loss and so on. Xiong & Du Expires 28 August 2022 [Page 8] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 4.3.2. Support the Utilization of Heterogeneous Resources In large-scale application, a large-scale number of nodes and long- distance transmission in the network will lead to latency and jitter, such as increasing transmission latency, jitter and packet loss. It is required to reduce the scale of the network topology by establishing cut-through channels. The existing technologies such as FlexE and SR tunnels should be taken into consideration. And multiple capabilities is also provided by the nodes and links within the network topology such as FlexE tunnels, TSN sub-network and IP/MPLS/SRv6 tunnels. It is required to integrate the multi- capability resources to achieve the optimal DetNet QoS. Heterogeneous resource should be used and unified and simplified resources mechanism under the selection of existing multiple technical methods to realize the elastic of deterministic capability. 5. Solutions of Large-Scale Deterministic Networks 5.1. Enhanced Layering Model The large-scale IP network can provide three levels of determinism, deterministic resources, deterministic routes and deterministic services, to establish a unified large-scale deterministic IP network architecture. The deterministic resources maintains the resources of the entire network, and performs unified modeling for deterministic resources to form deterministic links to shield the differences in heterogeneous resource capabilities. The deterministic routes computes routes based on the deterministic links modeled at the resource layer to provide deterministic transport capabilities. The deterministic services performs traffic monitoring on ingress nodes by planning the traffic characteristics of service flows, and maps them to deterministic routes to meet the time requirements of different types and levels of services. Xiong & Du Expires 28 August 2022 [Page 9] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 +-----------------------------------------+ | Service sub-layer | +-----------------------------------------+ | Differentiated DetNet QoS for Services | +-----------------------------------------+ | Forwarding sub-layer | +-----------------------------------------+ | Routes with Deterministic metrics | | Distributed Deterministic Routes | | Inter-domain Deterministic Routes | | Replication and Elimination Routes | +-----------------------------------------+ | Resource Modeling | | Resource Reservation | | Resource Isolation | +-----------------------------------------+ Figure 2: Figure 2: The Enhanced Layering Model of Large-Scale Deterministic Networks 5.2. Mechanisms to Achieve Differentiated DetNet QoS 5.2.1. Deterministic Resources Differentiated deterministic service requirements require the networks to provide different deterministic capabilities. The resources related to deterministic capabilities are also differentiated. The networks need to shield the differences between network capabilities. Deterministic resource is the basis for providing deterministic network services. It refers to the resources that meet the deterministic indicators of a node and link processing as well as the corresponding resource processing mechanisms (such as link bandwidth, queues, and scheduling algorithms). It is necessary to make overall resource planning for the network and make unified modeling for heterogeneous deterministic resources to form unified deterministic links to provide guarantee for the deterministic forwarding capabilities at different levels. A deterministic link can be a sub-network that provides deterministic transmission or a Point-to-Point (P2P) link. When the existing resources in the network are insufficient to meet the SLA requirements, virtual networks need to be reconstructed. Xiong & Du Expires 28 August 2022 [Page 10] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 5.2.2. Deterministic Routes To meet the requirements of different types and levels of deterministic services, deterministic route is to create deterministic routes with different SLA levels based on the deterministic link resources after unified modeling. Deterministic routes can be based on strict explicit paths or loose routes. The former is applicable to centralized scenarios with controllers, and the latter is applicable to distributed scenarios without controllers. In the centralized scenario, when the source and sink PEs of a deterministic service are located at the two ends of a WAN with a limited physical range, one controller (single domain) or multiple controllers (cross domain) compute one or more paths with deterministic SLA in advance according to the typical Traffic Specification (T-SPEC) based on the collected deterministic resources, or compute dynamically according to the service T-SPEC as required by the services. It is suggested to generate two non- intersecting paths with very close delay to form 1+1 protection and perform concurrent transmission and dual reception, and make replication and elimination on the egress PE. In the distributed scenario, intrinsic deterministic loose routes are computed on the device side through routing protocols. Interior Gateway Protocol (IGP) is used to compute deterministic routes based on deterministic- delay inside a domain, and Border Gateway Protocol (BGP) is used to compute deterministic routes based on accurate delay/jitter across domains. 5.2.3. Deterministic Services Deterministic services provide unified planning and scheduling mechanisms for service flows and perform end-to-end delay and jitter control. It is necessary to implement admission control and traffic policing at the ingress PE node based on the SLA of deterministic service flows, and map the service flows to deterministic routes to achieve the final goal of deterministic QoS. Deterministic services support that the end-to-end delay/jitter of the traffic with a specific T-SPEC in the network will be strictly limited within a bounded range on the basis of deterministic resource and route . As different service levels have different requirements for delay and jitter, the resources and routing mechanisms used for mapping services to deterministic routes are also different. For example, the extremely low delay and jitter can be guaranteed by multiplexing the rigid pipes at L1, so as to avoid the excessive intra-node delay contributed by too many hops of intermediate nodes at L3. Or in the customized virtual network, the bounded delay and jitter can be guaranteed by forwarding along the paths composed of Xiong & Du Expires 28 August 2022 [Page 11] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 links based on the ATS or CQF scheduling algorithm. Traffic policing on the ingress PE ensures that the service traffic does not exceed the reserved bandwidth, and then performs traffic shaping on the egress node. Different scheduling algorithms have different shaping effects. 6. Security Considerations TBA 7. Acknowledgements The authors would like to thank Peng Liu, Bin Tan, Aihua Liu Shaofu Peng for their review, suggestions and comments to this document. 8. IANA Considerations TBA 9. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8557] Finn, N. and P. Thubert, "Deterministic Networking Problem Statement", RFC 8557, DOI 10.17487/RFC8557, May 2019, . [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", RFC 8578, DOI 10.17487/RFC8578, May 2019, . [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", RFC 8655, DOI 10.17487/RFC8655, October 2019, . Authors' Addresses Quan Xiong ZTE Corporation No.6 Huashi Park Rd Xiong & Du Expires 28 August 2022 [Page 12] Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022 Wuhan Hubei, 430223 China Email: xiong.quan@zte.com.cn ZongPeng Du China Mobile Beijing China Email: duzongpeng@chinamobile.com Xiong & Du Expires 28 August 2022 [Page 13]