Network Working Group Shaofu. Peng Internet-Draft ZTE Corporation Intended status: Standards Track Tony. Li Expires: August 31, 2022 Juniper Networks February 27, 2022 IGP Flexible Algorithm with Deterministic Routing draft-peng-lsr-flex-algo-deterministic-routing-02 Abstract IGP Flex Algorithm proposes a solution that allows IGPs themselves to compute constraint based paths over the network, and it also specifies a way of using Segment Routing (SR) Prefix-SIDs and SRv6 locators, or pure IP prefix to steer packets along the constraint- based paths. This document describes how to compute deterministic delay paths within Flex-algo plane. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on August 31, 2022. Copyright Notice Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of Peng & Li Expires August 31, 2022 [Page 1] Internet-Draft flex-algo deterministic February 2022 the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 4 3. Determinisitc Links . . . . . . . . . . . . . . . . . . . . . 4 3.1. Deterministic Link Bound with CQF . . . . . . . . . . . . 5 3.2. Deterministic Link Bound with Deadline . . . . . . . . . 6 4. Deterministic Delay Metric Extension to ISIS . . . . . . . . 7 4.1. CQF Scheduding Delay Intra Node Sub-Sub-TLV . . . . . . . 8 4.2. Deadline Scheduding Delay Intra Node Sub-Sub-TLV . . . . 10 4.2.1. Another Simplified Extension . . . . . . . . . . . . 11 5. Deterministic Delay Metric Extension to OSPF . . . . . . . . 12 6. Announcement Suppression . . . . . . . . . . . . . . . . . . 12 7. Deterministic Routes Computation . . . . . . . . . . . . . . 12 7.1. Bind CQF Scheduling parameters with Flex-Algo . . . . . . 13 7.1.1. ISIS Advertisement of Flex-algo Binding CQF . . . . . 13 7.1.2. OSPF Advertisement of Flex-algo Binding CQF . . . . . 14 7.2. Bind Deadline Scheduling parameters with Flex-Algo . . . 14 7.2.1. ISIS Advertisement of Flex-algo Binding Deadline . . 14 7.2.2. OSPF Advertisement of Flex-algo Binding Deadline . . 15 7.3. CQF based Deterministic Routes Computation . . . . . . . 16 7.4. Deadline based Deterministic Routes Computation . . . . . 16 8. Routing Convergence and Redundance Considerations . . . . . . 18 9. Examples of Deterministic delay SPF . . . . . . . . . . . . . 20 9.1. CQF Based Deterministic Delay SPF Path Example . . . . . 20 9.2. Deadline Based Deterministic Delay SPF Path Example . . . 21 10. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 23 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24 11.1. ISIS Deterministic Delay Metric Sub-TLV . . . . . . . . 24 11.2. Sub-Sub-TLVs for ISIS Deterministic Delay Metric Sub-TLV 24 11.3. IGP Metric-Type Registry . . . . . . . . . . . . . . . . 24 11.4. ISIS Sub-Sub-TLVs for Flexible Algorithm Definition Sub- TLV . . . . . . . . . . . . . . . . . . . . . . . . . . 25 11.5. OSPF IANA considerations . . . . . . . . . . . . . . . . 25 12. Security Considerations . . . . . . . . . . . . . . . . . . . 25 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 25 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 25 14.1. Normative References . . . . . . . . . . . . . . . . . . 25 14.2. Informative References . . . . . . . . . . . . . . . . . 26 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 Peng & Li Expires August 31, 2022 [Page 2] Internet-Draft flex-algo deterministic February 2022 1. Introduction IGP Flex Algorithm [I-D.ietf-lsr-flex-algo] proposes a solution that allows IGPs themselves to compute constraint based paths over the network, and it also specifies a way of using Segment Routing [RFC8402] Prefix-SIDs and SRv6 locators, or pure IP prefix [I-D.ietf-lsr-ip-flexalgo] to steer packets along the constraint- based paths. It specifies a set of extensions to ISIS, OSPFv2 and OSPFv3 that enable a router to send TLVs that identify (a) calculation-type, (b) specify a metric-type, and (c )describe a set of constraints on the topology, that are to be used to compute the best paths along the constrained topology. A given combination of calculation-type, metric-type, and constraints is known as an FAD (Flexible Algorithm Definition). [RFC8655] describes the architecture of deterministic network and defines the QoS goals of deterministic forwarding: Minimum and maximum end-to-end latency from source to destination, timely delivery, and bounded jitter (packet delay variation); packet loss ratio under various assumptions as to the operational states of the nodes and links; an upper bound on out-of-order packet delivery. In order to achieve these goals, deterministic networks use resource reservation, explicit routing, service protection and other means. A deterministic path is typically (but not necessarily) explicit routes so that it does not normally suffer temporary interruptions caused by the convergence of routing or bridging protocols. IGP Flex-algo has the characteristic mentioned in [RFC8655]: under a single administrative control or within a closed group of administrative control. IGP Flex-algo supports Min Unidirectional Link Delay (defined in [RFC8570]) metric type to compute shortest paths with minimum delay, however, the cumulative delay is essentially the accumulation of transmission delay of all links, excluding node delay. In order to make up for this gap, it is necessary to enhance IGP flex-algo to compute the path with deterministic delay, i.e., including deterministic node delay and link transmission delay. This document describes how to compute distributed shortest paths with deterministic delay metric within Flex-algo plane, as the basis of the whole distributed deterministic scheme. It should be noted that relying on this enhancement alone does not guarantee complete determinacy, it needs to be used in conjunction with other tools, such as creating additional redundant deterministic delay path with consistent delay metric for PREOF (Packet Replication, Elimination, and Ordering Functions), smoothing the delay jitter during route convergence, providing deterministic forwarding mechanism, admission control, etc. Peng & Li Expires August 31, 2022 [Page 3] Internet-Draft flex-algo deterministic February 2022 2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 3. Determinisitc Links When a packet is forwarded to a link, the delay produced includes two parts: the first part is the dwell delay of the packet in the node, and the second part is the transmission delay of the packet on the link. In packet switching networks, priority based queuing scheme is generally used. It may give better average latency, but may have worst case latency. [SP-LATENCY] analyzes the guaranteed latency with the traditional strict priority scheme, and shows that low bounded latency is achievable when high priority traffic is constrained in low utilization, but deteriorates quickly with increasing utilization of high priority traffics. DiffServ [RFC2475] with strict priority has been widely deployed in the network, the existing non-deterministic service flow may set the highest priority, so it is difficult to support deterministic services based on it without any modification. We call those links bound with a queue mechanism that can not guarantee node delay are non-determinisitc links. On the contrary, those links bound with a queue mechanism that can provide deterministic node delay are called deterministic links. Therefore, other new scheduling mechanisms need to be introduced, and their scheduling priority is higher than that of the traditional strict priority queue. The typical queue mechanisms are as follows: o IEEE 802.1 WG has specified IEEE802.1Qav [CBS] which uses credit- based shaper mechanism to assign packets to different queues by giving a credit value which is related with reserved bandwidth. The credit values of different transmission queues will automatically change with the packet transmission process, which will ensure that the packet with lower priority will also get transmission. CBS shaper is similar to Weighted Fair Queuing (WFQ), and they all control the sending of packets based on reserved bandwidth. The worst-case delay calculation of class A of CBS is relatively simple, but other classes are complex. For class A traffic, the queuing delay equals to the maximum size of the interference frame (such as 2000 octes) divided by the port bandwidth. Peng & Li Expires August 31, 2022 [Page 4] Internet-Draft flex-algo deterministic February 2022 o IEEE 802.1 WG has specified IEEE802.1Qch [CQF] which uses cyclic queuing and forwarding (CQF) mechanism and relies on time synchronization. According to CQF, the maximum delay experienced by a given packet is (H+1)*D, the minimum delay experienced by a given packet is (H-1)*D, and the delay jitter is 2*D, where H is the number of hops and D is cycle duration. Other variants based on CQF can avoid relying on time synchronization, but only the same cycle duration for all nodes. Basically, the packet received in the current sending window (i.e., cycle) will ensure that it can be sent in the next sending window, then the deterministic node delay, on average, is one cycle duration, or seveval cycle durations if the forwarding delay intra node (from incoming port to outgoing port) can't be ignored. o [I-D.peng-detnet-deadline-based-forwarding] introduced a deadline based forwarding mechanism that allow packet to control its expected dwell time in the node according to the planned deadline. There are two policies for deadline queue to schedule packets. For in-time policy, the end-to-end delay is H*(F~D), jitter is H*Q, where, H is the number of hops, F is the forwarding delay intra node, D is the planned deadline, and Q is the scheduling delay; For on-time policy, the end-to-end delay is H*D, jitter is 0 (however there may be one authorization time due to the granularity of queue scheduling). That is, the packet received at any time will ensure that it can be sent in offset time F~D or D respectively for these two policies. This document mainly describes the deterministic link based on CQF or Deadline algorithm. Other algorithms will be described in the future. 3.1. Deterministic Link Bound with CQF A node may configure the CQF based packet scheduling parameter information for its local link, including CQF scheduling enable/ disable, one or more cycle durations. Accordingly, for each cycle duration, the node delay/jitter attributes of the link will be obtained. The meanings of these parameters or attributes of the link are as follows: o CQF scheduling enable/disable: the CQF scheduling algorithm can be enabled for a link, then the packets sent to that link will be scheduled by the CQF scheduling algorithm. o Cycle duration: the duration of the cycle of CQF, which is also called cycle_size. One or more cycle_size with different lengths can be configured for a link, such as 10us, 20us, 30us, and so on. Peng & Li Expires August 31, 2022 [Page 5] Internet-Draft flex-algo deterministic February 2022 o Node delay/jitter: * According to classical TSN CQF, for a given cycle_size, it can be deduced that the minimum delay in the node of the packet is 0, the maximum delay in the node is 2*cycle_size, the average delay in the node is one cycle_size, and the delay jitter in the node is 2*cycle_size. The detailed reasons for these data are as follows: if a node receives a packet at the tail end of cycle i and sends that packet at the head end of cycle i+1, the resulting node delay, i.e., the minimum node delay, is 0; if a node receives a packet at the head end of cycle i and sends that packet at the tail end of cycle i+1, the resulting node delay, i.e., the maximum node delay, is 2*cycle_ size; the average node delay is one cycle_size, and the node delay jitter is 2*cycle_size. Each cycle_size corresponds to a different set of delay/jitter attributes. * However, for some variants based on TSN CQF, if the forwarding delay intra node can't be ignored, e.g, wasting 2 cycle duration, then the minimum node delay, the maximum node delay, and the average node delay need to add 2 cycle_size respectively, but the node delay jitter is still 2*cycle_size. 3.2. Deterministic Link Bound with Deadline A node may configure the deadline based packet scheduling parameter information for its local link, including deadline scheduling enable/ disable, one or more deadline scheduling delays, and the scheduling policy supported for each deadline scheduling delay. Accordingly, for each deadline scheduling delay, the node delay/jitter attributes of the link will be obtained. The meanings of these parameters or attributes of the link are as follows: o Deadline scheduling enable/disable: the deadline scheduling algorithm can be enabled for a link, then the packet forwarded to the link will be scheduled by the deadline based packet scheduling algorithm. The dwell time of the packet in the node does not exceed the maximum allowable dwell time D, where, D = forwarding delay intra node (F) + specific deadline scheduling delay (Q). o Supported deadline scheduling delay set: the set composed of one or more deadline scheduling delays , assuming that Q1 is the minimum and Qn is the maximum in the set. Generally, the difference between two adjacent elements in the set is fixed, for example, a fixed interval (I). o Scheduling policy: for each scheduling delay Q, there are two scheduling policies: in-time policy and on-time policy. In case Peng & Li Expires August 31, 2022 [Page 6] Internet-Draft flex-algo deterministic February 2022 of in-time policy, the scheduling delay of the packet may be sent to the outgoing port when it does not reach Q; In the on-time policy, the packet is sent to the outgoing port only when the scheduling delay of the packet is equal to Q. Therefore, for in- time policy, the actual dwell time of the packet in the node is within the range [F, F+Q], i.e., the minimum node delay is F, the maximum node delay is F+Q, and the node delay jitter is Q; For on- time policy, the actual dwell time of the packet in the node is equal to F+Q, i.e., the minimum node delay is F+Q, the maximum node delay is also F+Q, and the node delay jitter is 0 (however, there may be one authorization time due to the granularity of queue scheduling). 4. Deterministic Delay Metric Extension to ISIS This document registers a new sub-TLV, Deterministic Delay Metric, in the "Sub-TLVs for IS-IS Sub-TLVs for TLVs Advertising Neighbor Information" registry, to distribute deterministic delay information. The deterministic delay advertised by this sub-TLV MUST be the delay from the local neighbor to the remote neighbor (i.e., the forward- path latency). The format of this sub-TLV is shown in the following diagram: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |A| RESERVED | Link transmission delay | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Forwarding Delay Intra Node | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // sub-sub-TLVs for Scheduding Delay Intra Node // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 1 where: Type: TBD Length: Variable. A bit: This field represents the Anomalous (A) bit. The A bit is set when one or more measured values of link transmission delay exceed a configured maximum threshold. The A bit is cleared when the measured value falls below its configured reuse threshold. If Peng & Li Expires August 31, 2022 [Page 7] Internet-Draft flex-algo deterministic February 2022 the A bit is cleared, the sub-TLV represents steady-state link transmission delay. RESERVED: This field is reserved for future use. It MUST be set to 0 when sent and MUST be ignored when received. Link transmission delay: This 24-bit field carries the average measured link transmission delay value (in microseconds) over a configurable interval, encoded as an integer value. Implementations MAY also permit the configuration of an offset value (in microseconds) to be added to the measured delay value, to facilitate the communication of operator-specific delay constraints. When the delay value is set to the maximum value 16,777,215 (16.777215 seconds), then the delay is at least that value and may be larger. Forwarding Delay Intra Node: This 16-bit field carries the forwarding delay value (in microseconds) intra node. It represents the latency of packet from the incoming port (or generated from control plane) to the outgoing port. If the forwarding delay can be ignored, it is set to 0. NOTE: for all links of a specific node, it may be possible that they have the same forwarding delay, therefore the forwarding delay can also be advertised by a unified node attribute. This would be considered in future versions. sub-sub-TLVs for Scheduding Delay Intra Node: Optional sub-sub- TLVs are contained to indicate the scheduling delay that is related to the specific scheduling algorithm such as CQF, deadline, etc. If this field is absent, the scheduling delay is unknown. Typically, a link may enable a single scheduling algorithm to get deterministic scheduling delay, so that a single sub-sub-TLV is included. However, it is possible for a link to enable multiple different scheduling algorithms, as long as these algorithms can coordinate the forwarding resources, in this case, multiple sub-sub-TLVs are included. Supported Sub-sub-TLVs are specified in the following sub-sections. 4.1. CQF Scheduding Delay Intra Node Sub-Sub-TLV CQF Scheduding Delay Intra Node Sub-Sub-TLV is an optional Sub-Sub- TLV of Deterministic Delay Metric Sub-TLV. At most only one CQF Scheduding Delay Intra Node Sub-Sub-TLV can be included. The following format is defined for the CQF Scheduding Delay Intra Node Sub-Sub-TLV: Peng & Li Expires August 31, 2022 [Page 8] Internet-Draft flex-algo deterministic February 2022 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Length | Cycle_size 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Cycle_size 2 | ... ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Cycle_size N | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 2 where: Type: TBD Length: 2*N, depending on the count of the cycle_size. Cycle_size: The length of cycle duration, in units of microseconds. A link can support multiple cycle durations, for example, 10us, 20us, 30us, etc, each for a specific service requirement. Only those links that enable CQF scheduling algorithm need to advertise the CQF Scheduding Delay Intra Node Sub-Sub-TLV, otherwise there is no need to advertise. Note that the advertised cycle_size must be consistent with the CQF queue scheduling mechanism actually instantiated by the link in the forwarding plane. If the forwarding plane does not instantiate a CQF queue scheduling supporting a certain cycle_size, which is however advertised in the CQF Scheduding Delay Intra Node Sub-Sub-TLV, the subsequent route computation may get wrong results. For a given cycle_size, it can deduce the corresponding node delay and jitter attributes, so these attributes can no longer be explicitly included in the CQF Scheduding Delay Intra Node Sub-Sub- TLV. As mentioned earlier, if the forwarding delay intra node (assuming F) is not 0, the minimum node delay, the maximum node delay, and the average node delay need to take F into account respectively. F is replaced by ((F/cycle_size)+1)*cycle_size for deducing. That is: o If F is 0, for a given cycle_size, the minimum node delay is 0, the maximum node delay is 2*cycle_size, the average node delay is cycle_size, and the node delay jitter is 2*cycle_size. Peng & Li Expires August 31, 2022 [Page 9] Internet-Draft flex-algo deterministic February 2022 o If F is not 0, for a given cycle_size, the minimum node delay is ((F/cycle_size)+1)*cycle_size, the maximum node delay is ((F/ cycle_size)+3)*cycle_size, the average node delay is ((F/ cycle_size)+2)*cycle_size, and the node delay jitter is 2*cycle_size. 4.2. Deadline Scheduding Delay Intra Node Sub-Sub-TLV Deadline Scheduding Delay Intra Node Sub-Sub-TLV is an optional Sub- Sub-TLV of Deterministic Delay Metric Sub-TLV. At most only one Deadline Scheduding Delay Intra Node Sub-Sub-TLV can be included. The following format is defined for the Deadline Scheduding Delay Intra Node Sub-Sub-TLV: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Length | P | Q1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | P | Q2 | P | Q3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... ... | P | Qn | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3 where: Type: TBD Length: 2*N, depending on the count of the supported deadline scheduling delay. Q: Indicates the scheduling delay set, , supported by the link, in units of microseconds. For each supported scheduling delay, the highest two bits represent the scheduling policy P. The value of scheduling policy P can be: 0, not defined yet; 1, indicates that it supports the in-time policy; 2, indicates that it supports the on-time policy; 3, indicates that it supports both in-time policy and on-time policy. Peng & Li Expires August 31, 2022 [Page 10] Internet-Draft flex-algo deterministic February 2022 As mentioned earlier, given the scheduling delay Q and its scheduling policy, combined with the forwarding delay F intra node, the corresponding delay and jitter attributes in the node can be derived. Therefore, these attributes can no longer be explicitly included in the Deadline Scheduding Delay Intra Node Sub-Sub-TLV. Note that the scheduling delay Q advertised in the Deadline Scheduding Delay Intra Node Sub-Sub-TLV must be consistent with the deadline queue scheduling mechanism actually instantiated by the link in the forwarding plane. If the forwarding plane does not instantiate the deadline queue scheduling supporting a certain scheduling delay Q, which is however advertised in the Deadline Scheduding Delay Intra Node Sub-Sub-TLV, the subsequent route computation may get wrong results. 4.2.1. Another Simplified Extension If the set to be advertised contains too many elements, and the difference between two adjacent elements in the set is a fixed interval (I), and the scheduling policy supported for all elements are same, another more simplified extension, the Simplified Deadline Scheduding Delay Intra Node Sub-Sub-TLV, can be defined as below. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Length | Q1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Qn | P | I | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4 where: Type: TBD Length: 6. Q1: the minimum scheduling delay, in units of microseconds. Qn: the maximum scheduling delay, in units of microseconds. I: the fixed interval between any two adjacent elements in the set, in units of microseconds. The highest two bits represent the scheduling policy P. The value of scheduling policy P can be: Peng & Li Expires August 31, 2022 [Page 11] Internet-Draft flex-algo deterministic February 2022 0, not defined yet; 1, indicates that it supports the in-time policy; 2, indicates that it supports the on-time policy; 3, indicates that it supports both in-time policy and on-time policy. 5. Deterministic Delay Metric Extension to OSPF To be defined in next version. 6. Announcement Suppression The value of Deterministic Delay Metric defined in this document contains node delay provided by instantiated scheduling algorithm and link transmission delay provided by some measure mechanisms. For the announcement of the node delay part, it is constant and depend on the capability of instantiating the scheduling algorithm. However, for the announcement of the link transmission delay part, a measure mechanism may frequently produce different measurements. Please refer to [RFC8570] section 6 for the same principle of announcement suppression. 7. Deterministic Routes Computation In order to use the deterministic link resources in the network to compute a deterministic delay SPF path, corresponding Flex-algo plane need to be created. To dinstinguish between traditional low latency SPF path (based on metric type "Min Unidirectional Link Delay") and deterministic low latency SPF path introduced in this document, new metric type, i.e., Deterministic Delay Metric, will be defined and used in Flexible Algorithm Definition (FAD). o Metric-Type: TBD, to be used during the calculation of deterministic low latency SPF path. Additional FAD constraints are also necessary, to bind individual item from the scheduling delay set. It is possible to create multiple flex-algo instances each binding to different scheduling delay for different service requirements. Note that sometimes from the perspective of the end-to-end delay requirements of the service flow, the node delay of the ingress PE node can be ignored and regarded as 0. However, this has no implication for the rules of deterministic low latency SPF path Peng & Li Expires August 31, 2022 [Page 12] Internet-Draft flex-algo deterministic February 2022 calculation. During the deterministic low latency SPF path calculation, we must always take the node delay into account, because any node may be the intermediate node of an SPF path and its node delay is taken into account. But if the node takes itself as the root and does not consider its node delay when calculating the deterministic low latency SPF path, the calculation results may be inconsistent, resulting in the loop. 7.1. Bind CQF Scheduling parameters with Flex-Algo The binding relationship can be configured on one or more nodes participating in the same IGP Flex-algo plane, and then advertised in the IGP domain. If there are multiple binding relationship advertised for the same algorithm, it should choose to use the binding cycle_size contained in the FAD with the highest priority. If a Flex-algo plane eventually uses a binding cycle_size, all links participated to the Flex-algo plane must be configured with CQF scheduling enabled and corresponding cycle_size, otherwise, links that do not meet the conditions must be excluded from the Flex-algo plane. 7.1.1. ISIS Advertisement of Flex-algo Binding CQF The Flexible Algorithm definition can specify the binding cycle_size that are used to determine the deterministic delay metric for the computed path within the Flex-algo plane. A new IS-IS sub-TLV is defined: the FAD Binding CQF Sub-Sub-TLV, which is advertised within IS-IS Flexible Algorithm Definition Sub- TLV. At most only one FAD Binding CQF Sub-Sub-TLV can be included. The following format is defined for the FAD Binding CQF Sub-Sub-TLV: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Length | Binding Cycle_size | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5 where: Type: TBD Length: 2 Peng & Li Expires August 31, 2022 [Page 13] Internet-Draft flex-algo deterministic February 2022 Binding Cycle_size: Cycle_size of CQF scheduling bound by Flex- algo, in units of microseconds. The binding cycle_size contained in the FAD with the highest priority will take effect. If the FAD with the highest priority does not contain the FAD Binding CQF Sub-Sub-TLV, the traditional path considering only link transmission delay will be calculated (i.e., degenerating into the calculation result similar as based on Min Unidirectional Link Delay metric type), otherwise, the path will consider both node delay and link delay. 7.1.2. OSPF Advertisement of Flex-algo Binding CQF To be defined in next version. 7.2. Bind Deadline Scheduling parameters with Flex-Algo The binding relationship can be configured on one or more nodes participating in the same IGP Flex-algo plane, and then advertised in the IGP domain. If there are multiple binding relationship advertised for the same algorithm, it should choose to use the binding scheduling delay and scheduling policy contained in the FAD with the highest priority. If a Flex-algo plane eventually uses a binding deadline parameter, all links participated to the Flex-algo plane must be configured with deadline scheduling enabled and corresponding scheduling delay and scheduling policy, otherwise, links that do not meet the conditions must be excluded from the Flex-algo plane. 7.2.1. ISIS Advertisement of Flex-algo Binding Deadline The Flexible Algorithm definition can specify the binding deadline scheduling delay and scheduling policy that are used to determine the deterministic delay metric for the computed path within the Flex-algo plane. A new IS-IS sub-TLV is defined: the FAD Binding Deadline Sub-Sub-TLV, which is advertised within IS-IS Flexible Algorithm Definition Sub- TLV. At most only one FAD Binding Deadline Sub-Sub-TLV can be included. The following format is defined for the FAD Binding Deadline Sub-Sub- TLV: Peng & Li Expires August 31, 2022 [Page 14] Internet-Draft flex-algo deterministic February 2022 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Length | Flags |U| P | Q // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // Q | +-+-+-+-+-+-+-+-+ Figure 6 where: Type: TBD Length: 3 Flags: Two flags are currently defined. U-flag: 1 bit, indicating whether the value of scheduling delay Q is known or unknown. 0 indicates known and 1 indicates unknown. P-flag: 2 bits, indicating scheduling policy. The value can be: 0, not defined yet; 1, indicates the in-time policy; 2, indicates the on-time policy; 3, not defined yet. Q: Indicates the deadline scheduling delay Q bound by flex algorithm, in units of microseconds. Note that if the U-flag is 1, the value of Q must be ignored and set to 0. The binding deadline parameter contained in the FAD with the highest priority will take effect. If the FAD with the highest priority does not contain the FAD Binding Deadline Sub-Sub-TLV, the traditional path considering only link transmission delay will be calculated (i.e., degenerating into the calculation result similar as based on Min Unidirectional Link Delay metric type), otherwise, the path will consider both node delay and link delay. Note that the FAD Binding CQF Sub-Sub-TLV and the FAD Binding Deadline Sub-Sub-TLV MUST not appear in FAD at the same time, otherwise, the first one is selected. 7.2.2. OSPF Advertisement of Flex-algo Binding Deadline To be defined in next version. Peng & Li Expires August 31, 2022 [Page 15] Internet-Draft flex-algo deterministic February 2022 7.3. CQF based Deterministic Routes Computation This document use the new Metric-Type, Deterministic Delay, combined with the FAD Binding CQF Sub-Sub-TLV, to compute CQF based shortest path with minimum deterministic end-to-end delay, which contains accumulated node delay provided by CQF and accumulated link transmission delay. For a Flex-algo plane that bound to a specific CQF cycle_size, the delay metric of a candidate path within the Flex-algo plane equals: H * node delay, where H is the number of hops, and node delay can be deduced by the cycle_size and forwarding delay intra node as described in Section 4.1; plus Accumulated link transmission delay; From the source node to the destination node, the candidate path with minimum deterministic delay metric is the best one. This calculation result may be different from the traditional calculation result considering only link transmission delay, depending on the proportion of node delay. If the number of intermediate nodes included in the two candidate paths is different, the node delay may be different. For example, a traditional optimal low latency path only considering the link transmission delay may contain more hops, resulting in not being recognized as the optimal deterministic latency path. The deterministic delay jitter of a candidate path within the Flex- algo plane equals: node delay jitter, which is 2*cycle_size; plus Accumulated link delay jitter, which is almost 0; 7.4. Deadline based Deterministic Routes Computation This document use the new Metric-Type, Deterministic Delay, combined with the FAD Binding Deadline Sub-Sub-TLV, to compute deadline based shortest path with minimum deterministic end-to-end delay, which contains accumulated node delay provided by deadline and accumulated link transmission delay. For a Flex-algo plane that bound to a specific deadline scheduling parameter, the delay metric of a candidate path within the Flex-algo plane equals: Peng & Li Expires August 31, 2022 [Page 16] Internet-Draft flex-algo deterministic February 2022 H * node delay, where H is the number of hops, and node delay can be deduced by the scheduling delay, scheduling policy and forwarding delay intra node as described in Section 4.2; plus Accumulated link transmission delay; Assuming that the bound scheduling delay Q and scheduling policy P are obtained from the FAD Binding Deadline Sub-Sub-TLV (note that if the bound scheduling delay Q is an unknown value, the scheduling delay Q is temporarily replaced by 0 during path compuation), the node delay contributed by any intermediate node i in the candidate path is: o For in-time policy, the node delay is in the range of [F(i), F(i)+Q], where F(i) represents the forwarding delay intra node i. Because the node delay value in this case is a range, and we need to get a specific value for SPF computation, thus there are several options to select a specific value as node delay, i.e., select F(i), or F(i)+Q, or the average of F(i) and F(i)+Q. This document take F(i)+Q as the default option. o For on-time policy, the node delay is equal to F(i)+Q. It should be noted that the above calculation process is used to select the optimal deterministic delay path from multiple candidate paths. However, once the deterministic delay SPF path is obtained, the deterministic delay metric of the deterministic delay SPF path should reflect the actual delay. Especially: o When the bound scheduling delay Q is an unknown value, the deterministic delay metric of the deterministic delay SPF path is an formula containing variable quantity Q. In this case, the value of scheduling delay Q needs to be given through other methods, such as carried in the forwarded data packet. This means that the same path can provide different delays for different services. o For in-time policy, the min delay of the SPF path is H*F, which is different with the max delay of SPF path is H*(F+Q), so that delay jitter is H*Q. The deterministic delay jitter of a candidate path within the Flex- algo plane equals: o Accumulated node delay jitter, which is H*Q for in-time policy and 0 for on-time policy; plus o Accumulated link delay jitter, which is almost 0; Peng & Li Expires August 31, 2022 [Page 17] Internet-Draft flex-algo deterministic February 2022 8. Routing Convergence and Redundance Considerations As described in [I-D.ietf-lsr-flex-algo], Loop Free Alternate (LFA) paths for a given Flex-Algorithm MUST be computed using the same constraints as the calculation of the primary paths for that Flex- Algorithm. Within the Segment Routing framework, [I-D.ietf-rtgwg-segment-routing-ti-lfa] can provide TI-LFA path, as the expected post-convergence paths from the point of local repair, in any two connected network using a link-state IGP. However, ordinary IGP convergence and FRR protection may not meet the needs of deterministic services. The main reasons include: o IGP convergence may cause considerable packet loss rate, even if FRR switching is implemented on the basis of rapid fault detection. o The cumulative deterministic delay of the LFA path may be very different from that of the primary path, which does not meet the strict requirements for delay jitter. Thus, according to Service Protection function defined in [RFC8655], packets can be spreaded over multiple disjoint forwarding paths to mitigate or eliminate the packet loss rate. In the context of Flex- algo, an additional redundant deterministic delay path different from FRR path need to be created, when if PLR enable Packet Replication Function (PRF) and the destination enable Packet Elimination Function (PEF). In this case, the data packets are sent along the primary deterministic delay SPF path and the redundant deterministic delay path at the same time, with almost the same cumulative delay. The additional redundant deterministic delay path within the Flex- algo plane is often a traffic engineering path that is calculated by PLR based on the constraints contained in FAD and the following constraints: o The number of nodes intersecting the primary and redundant deterministic delay paths shall be minimized; o The difference between the number of hops of the primary and redundant deterministic delay paths shall be minimized; o The difference between the cumulative link transmission delay of the primary and redundant deterministic delay paths shall be minimized. Unlike LFA FRR path, more scheduling parameters read from link-state database can be attempted to used in the redundant deterministic delay path, to obtain the delay equal or close to the primary path. Peng & Li Expires August 31, 2022 [Page 18] Internet-Draft flex-algo deterministic February 2022 Take deadline based path as an example, suppose that the number of hops in the primary deterministic delay SPF path is m, and the intermediate nodes passing through are A1, A2, ..., Am, the forwarding delay intra node for each hop is Fa, the scheduling delay intra node for each hop is Qa, and the cumulative link transmission delay is La, then the cumulative deterministic delay of the primary deterministic delay SPF path is the following formula: o Delay(primary) = m*Fa + m*Qa + La Similarly, suppose that the number of hops in the redundant deterministic delay path is n, and the intermediate nodes passing through are B1, B2, ..., Bn, the forwarding delay intra node for each hop is Fb, the scheduling delay intra node for each hop is Qb, and the cumulative link transmission delay is Lb, then the cumulative deterministic delay of the redundant deterministic delay path is the following formula: o Delay(redundant) = n*Fb + n*Qb + Lb The value of Delay(primary) can be calculated based on the known value of Qa that is bound to the flex-algo. Then, an appropriate Qb is slelected to make Delay(redundant) equal to Delay(primary). Qb, that is likely to be different from the bound value Qa, SHOULD be carried in the packets sent along the redundant deterministic delay path to get the expected latency. If the value of Qa bound to the Flex-algo is unknown, states per service should be maintained at the ingress node, to determine the specific value of Qa according to SLA of the service sent along the primary deterministic delay SPF path. On this basis, Qb is then calculated. In this case, both Qa and Qb SHOULD be carried in the packets to get the expected latency. If the further packet replication function continues to be implemented on an intermediate node of the network, the intermediate node only needs to regard itself as the head node of the new protection sub-domain, and can still adopt the above scheme. The intermediate node can also get the value of Delay(primary) based on the bound known Qa (or get from packets), On this basis, Qb is then calculated. It should be noted that both packets sent along primary deterministic delay SPF path and redundant deterministic delay path within a flex- algo plane MUST use SIDs or prefix related with that algortihm. The FIB entries within the flex-algo plane, such as SID entries, contain specific deterministic scheduling parameters to enable the Peng & Li Expires August 31, 2022 [Page 19] Internet-Draft flex-algo deterministic February 2022 packet to excecute corresponding scheduling function. However, if the packet also carries scheduling parameters, the one in the packet must be preferred. 9. Examples of Deterministic delay SPF As shown in Figure 7, the IGP flex-algo 128 plane contains five nodes, of which each link is a bidirectional link. The figure shows the transmission delay parameters of each lin, e.g, the transmission delay of the link between node R1 and node R2 is 10us. 20us 30us +----- R3 -----+ | | 10us | | R1 ----- R2 R5 | | | 10us 20us | +----- R4 -----+ Figure 7 9.1. CQF Based Deterministic Delay SPF Path Example It is assumed that the links of all nodes in the network are configured with consistent CQF scheduling parameters and have consistent node delay and delay jitter attributes, as follows: Forwarding delay intra node = 0us CQF enable/disable = ON Supported cycle_size set = <10 us, 20 us> Configure FAD of IGP flex-algo 128, set metric-type to Deterministic Delay, and set bound CQF scheduling parameters (cycle_size = 10us). Suppose that FAD is optimal after negotiation. Taking node R1 as an example, it takes itself as the root to calculate the deterministic delay SPT as shown in Figure 8. In the figure, the sum of the node delay and the link transmssion delay is marked on each link. For example, the delay of link from node R2 to R3 is 10+20, where 10 is node delay and 20 is link transmission delay. Peng & Li Expires August 31, 2022 [Page 20] Internet-Draft flex-algo deterministic February 2022 R1 10+10 / R2 10+20 / \ 10+10 R3 R4 \ 10+20 R5 Figure 8 Therefore, with R1 as the source node and R5 as the destination node, the cumulative deterministic delay of CQF based SPF path (R1-R2-R4-R5) is 70us, The cumulative deterministic delay jitter is 20us. Assuming that node R5 advertised SID-R5 that belongs to the flex-algo 128 plane, the following deterministic SPF FIB entry will be created on node R1. KEY: SID-R5 Forwarding information: next_hop = R2 interface = link(R1-R2) metric_type = Deterministic Delay scheduling algorithm = CQF with cycle_size 10 us total_metric = 70 us total_metric_variation = 20 us 9.2. Deadline Based Deterministic Delay SPF Path Example It is assumed that the links of all nodes in the network are configured with consistent deadline scheduling parameters, as follows: Forwarding delay intra node = 5us Deadline enable/disable = ON Peng & Li Expires August 31, 2022 [Page 21] Internet-Draft flex-algo deterministic February 2022 Supported scheduling delay set = <10us, 20us, 30us, 40us, 50us, 60us>, each item in the set support both in-time and on-time policy Configure FAD of IGP flex-algo 128, set metric-type to Deterministic Delay, and set bound Deadline scheduling parameters (Q = 10us, with in-time policy). Suppose that FAD is optimal after negotiation. Taking R1 node as an example, it takes itself as the root to calculate the deterministic delay SPT as shown in Figure 9. In the figure, the sum of the node delay and the link transmssion delay is marked on each link. Note that this document suggest to take F+Q as node delay during calculation even for in-time policy. For example, the delay of link from node R2 to R3 is 15+20, where 15 is node delay and 20 is link transmission delay. R1 15+10 / R2 15+20 / \ 15+10 R3 R4 \ 15+20 R5 Figure 9 Therefore, with R1 as the source node and R5 as the destination node, the cumulative deterministic delay of Deadline based SPF path (R1-R2-R4-R5) is 85us, and the cumulative deterministic delay jitter is 30us. Assuming that node R5 advertised SID-R5 that belongs to the fle-algo 128 plane, the following deterministic SPF FIB entry will be created on node R1. KEY: SID-R5 Forwarding information: next_hop = R2 interface = link(R1-R2) metric_type = Deterministic Delay scheduling algorithm = Deadline with Q=10 us with in-time policy Peng & Li Expires August 31, 2022 [Page 22] Internet-Draft flex-algo deterministic February 2022 total_metric = 85 us total_metric_variation = 30 us Similarly, if on-time policy is bound to the flex-algo, the cumulative deterministic delay of Deadline based SPF path (R1-R2-R4-R5) is 85us, but the cumulative deterministic delay jitter is 0. The deterministic SPF FIB entry created on node R1 is changed to: KEY: SID-R5 Forwarding information: next_hop = R2 interface = link(R1-R2) metric_type = Deterministic Delay scheduling algorithm = Deadline with Q=10 us with on-time policy total_metric = 85 us total_metric_variation = 0 10. Use Cases [RFC8578] described various deterministic routing use cases from multiple industries, including: Pro Audio and Video, Electrical Utilities, Building Automation Systems, Wireless for Industrial Applications, Cellular Radio, Industrial Machine to Machine (M2M), Mining Industry, Private Blockchain, Network Slicing, etc. Among them, some industries are now transitioning to packet based infrastructure, and some industries have already linked their different subsystems through networks (intra-domain or inter-domain). These industries have put forward the requirements of delay and delay jitter with different indicators, such as BAS requires low delay (10ms ~ 100ms) and low jitter (1ms); M2M requires that the underlying network infrastructure must ensure that the maximum end-to-end message delivery time is between 100 us and 50 ms; Mining industry requires predictable time delay to realize real-time monitoring. The deterministic paths can be centralized centralized computing, or distributed computing when there is a lack of controller. The mechanism introduced in this document can get a SPF path with determinstic delay metric, but more importantly, with deterministic Peng & Li Expires August 31, 2022 [Page 23] Internet-Draft flex-algo deterministic February 2022 dealy jitter. The determinsitic delay metric of the path actually depends on the network scale. It can be large or small, but it can be guaranteed to be the smallest of all candidate paths. The determinsitic delay jitter is also bounded and may be a cumulative value related to the number of hops or a value independent of the number of hops. SPF Paths with such characteristics will benefit multiple applications as mentioned above. 11. IANA Considerations 11.1. ISIS Deterministic Delay Metric Sub-TLV This document registers the following Sub-TLV in the "Sub-TLVs for IS-IS Sub-TLVs for TLVs Advertising Neighbor Information" registry: +------+--------------------+----+----+----+-----+-----+-----+ | Type | Description | 22 | 23 | 25 | 141 | 222 | 223 | +======+====================+====+====+====+=====+=====+=====+ | | Deterministic Delay| | | | | | | | TBA1 | Metric | y | y | y | y | y | y | +------+--------------------+----+----+----+-----+-----+-----+ 11.2. Sub-Sub-TLVs for ISIS Deterministic Delay Metric Sub-TLV This document registers the following Sub-TLV in the "Sub-TLVs for IS-IS Sub-TLVs for TLVs Advertising Neighbor Information" registry: +------+----------------------+---------------------------+ | Type | Description | Reference | +======+======================+===========================+ | | CQF Scheduding Delay | This document Section 4.1 | | TBA2 | Intra Node | | +------+----------------------+---------------------------+ | | Deadline Scheduding | This document Section 4.2 | | TBA3 | Delay Intra Node | | +------+----------------------+---------------------------+ 11.3. IGP Metric-Type Registry This document registers the following values in the "IGP Metric-Type Registry" for FAD: Type: TBA4 (suggested 4) Description: Deterministic Delay Metric as defined in this document Reference: This document (Section 4) Peng & Li Expires August 31, 2022 [Page 24] Internet-Draft flex-algo deterministic February 2022 11.4. ISIS Sub-Sub-TLVs for Flexible Algorithm Definition Sub-TLV This document defines the following Sub-Sub-TLVs in the "Sub-Sub-TLVs for Flexible Algorithm Definition Sub-TLV" registry: +------+----------------------+-----------------------------+ | Type | Description | Reference | +======+======================+=============================+ | TBA5 | FAD Binding CQF | This document Section 7.1.1 | +------+----------------------+-----------------------------+ | TBA6 | FAD Binding Deadline | This document Section 7.2.1 | +------+----------------------+-----------------------------+ 11.5. OSPF IANA considerations TBD. 12. Security Considerations TBD. 13. Acknowledgements The authors would like to acknowledge the review and inputs from Peter Psenak, Bin Tan, Quan Xiong. 14. References 14.1. Normative References [I-D.ietf-lsr-flex-algo] Psenak, P., Hegde, S., Filsfils, C., Talaulikar, K., and A. Gulko, "IGP Flexible Algorithm", draft-ietf-lsr-flex- algo-18 (work in progress), October 2021. [I-D.ietf-lsr-ip-flexalgo] Britto, W., Hegde, S., Kaneriya, P., Shetty, R., Bonica, R., and P. Psenak, "IGP Flexible Algorithms (Flex- Algorithm) In IP Networks", draft-ietf-lsr-ip-flexalgo-04 (work in progress), December 2021. [I-D.ietf-rtgwg-segment-routing-ti-lfa] Litkowski, S., Bashandy, A., Filsfils, C., Francois, P., Decraene, B., and D. Voyer, "Topology Independent Fast Reroute using Segment Routing", draft-ietf-rtgwg-segment- routing-ti-lfa-08 (work in progress), January 2022. Peng & Li Expires August 31, 2022 [Page 25] Internet-Draft flex-algo deterministic February 2022 [I-D.peng-detnet-deadline-based-forwarding] Peng, S. and B. Tan, "Deadline Based Deterministic Forwarding", draft-peng-detnet-deadline-based- forwarding-00 (work in progress), January 2022. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, July 2018, . [RFC8570] Ginsberg, L., Ed., Previdi, S., Ed., Giacalone, S., Ward, D., Drake, J., and Q. Wu, "IS-IS Traffic Engineering (TE) Metric Extensions", RFC 8570, DOI 10.17487/RFC8570, March 2019, . [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", RFC 8578, DOI 10.17487/RFC8578, May 2019, . [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", RFC 8655, DOI 10.17487/RFC8655, October 2019, . 14.2. Informative References [CBS] "IEEE802.1Qav", 2009, . [CQF] "IEEE802.1Qch", 2017, . Peng & Li Expires August 31, 2022 [Page 26] Internet-Draft flex-algo deterministic February 2022 [SP-LATENCY] "Guaranteed Latency with SP", 2020, . Authors' Addresses Shaofu Peng ZTE Corporation China Email: peng.shaofu@zte.com.cn Tony Li Juniper Networks United States of America Email: tony.li@tony.li Peng & Li Expires August 31, 2022 [Page 27]