idnits 2.17.1 draft-liu-detnet-large-scale-requirements-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (11 April 2022) is 746 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'I-D.du-detnet-layer3-low-latency' is defined on line 593, but no explicit reference was found in the text == Unused Reference: 'I-D.eckert-detnet-bounded-latency-problems' is defined on line 600, but no explicit reference was found in the text == Unused Reference: 'I-D.geng-detnet-requirements-bounded-latency' is defined on line 608, but no explicit reference was found in the text == Unused Reference: 'I-D.qiang-detnet-large-scale-detnet' is defined on line 616, but no explicit reference was found in the text == Outdated reference: A later version (-05) exists of draft-du-detnet-layer3-low-latency-04 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Deterministic Networking Working Group P. Liu 3 Internet-Draft China Mobile 4 Intended status: Informational Y. Li 5 Expires: 13 October 2022 Huawei 6 T. Eckert 7 Futurewei Technologies USA 8 Q. Xiong 9 ZTE Corporation 10 J. Ryoo 11 ETRI 12 11 April 2022 14 Requirements for Large-Scale Deterministic Networks 15 draft-liu-detnet-large-scale-requirements-02 17 Abstract 19 Aiming at the large-scale deterministic network, this document 20 describes the technical and operational requirements when the 21 different deterministic levels of applications co-exist and are 22 transported over a wide area. This document also describes the 23 corresponding Deterministic Networking (DetNet) data plane 24 enhancement requirements. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at https://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on 13 October 2022. 43 Copyright Notice 45 Copyright (c) 2022 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 50 license-info) in effect on the date of publication of this document. 51 Please review these documents carefully, as they describe your rights 52 and restrictions with respect to this document. Code Components 53 extracted from this document must include Revised BSD License text as 54 described in Section 4.e of the Trust Legal Provisions and are 55 provided without warranty as described in the Revised BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 2. Conventions Used in This Document . . . . . . . . . . . . . . 4 61 3. The Overall Characteristics of Large-Scale Deterministic 62 Networks . . . . . . . . . . . . . . . . . . . . . . . . 4 63 4. Technical Requirements in Large-Scale Deterministic 64 Networks . . . . . . . . . . . . . . . . . . . . . . . . 6 65 4.1. Tolerate Time Asynchrony . . . . . . . . . . . . . . . . 6 66 4.1.1. Support Asynchronous Clocks Across Domains . . . . . 6 67 4.1.2. Tolerate Clock Jitter & Wander within a Clock 68 Synchronous Domain . . . . . . . . . . . . . . . . . 7 69 4.1.3. Provide Mechanisms not Requiring Full Time 70 Synchronization . . . . . . . . . . . . . . . . . . . 7 71 4.1.4. Support Asynchronization based Methods . . . . . . . 7 72 4.2. Support Large Single-hop Propagation Latency . . . . . . 7 73 4.3. Accommodate the Higher Link Speed . . . . . . . . . . . . 8 74 4.4. Be Scalable to Numerous Network Devices and Massive Traffic 75 Flows . . . . . . . . . . . . . . . . . . . . . . . . . . 9 76 4.5. Tolerate Failures of Links or Nodes and Topology 77 Changes . . . . . . . . . . . . . . . . . . . . . . . . . 10 78 4.6. Support Configuration of Multiple Queueing Mechanisms . . 10 79 4.7. Support Queueing Mechanisms Switchover Crossing 80 Multi-domains . . . . . . . . . . . . . . . . . . . . . . 11 81 5. Data Plane Enhancement Requirements . . . . . . . . . . . . . 11 82 5.1. Support Aggregated Flow Identification . . . . . . . . . 11 83 5.2. Support Queuing Related Information . . . . . . . . . . . 12 84 5.3. Support Redundancy Related Fields . . . . . . . . . . . . 12 85 5.4. Support Explicit Path Selection . . . . . . . . . . . . . 12 86 6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 12 87 7. Security Considerations . . . . . . . . . . . . . . . . . . . 12 88 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 89 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 13 90 10. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 13 91 11. Normative References . . . . . . . . . . . . . . . . . . . . 13 92 Appendix A. Examples of Large-Scale Deterministic Network 93 Trials . . . . . . . . . . . . . . . . . . . . . . . . . 15 94 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 16 96 1. Introduction 98 Packet networks are evolving from bandwidth-guaranteed Quality of 99 Service (QoS) to latency-guaranteed QoS that guarantees bounded 100 latency and definite latency. Bounded latency and definite latency 101 can be further understood as in-time delivery, in which a packet 102 arrives without exceeding a predetermined time, and on-time delivery, 103 in which a packet arrives at a predetermined time, respectively. In 104 addition, network survivability, which typically guarantees traffic 105 recovery within 50 ms in the event of a network failure, is evolving 106 to a level that guarantees lossless recovery. In order to realize 107 the evolution of QoS and network survivability of these networks, 108 Time-Sensitive Networking (TSN) technology and Deterministic 109 Networking (DetNet) technology are considered to be essential. 111 TSN is a set of standards developed by the IEEE 802.1 TSN Task Group 112 (TG) [IEEE802.1TSN] and specifies mechanisms and protocols necessary 113 to realize highly available IEEE 802.1 networks with bounded latency 114 to carry time-sensitive, real-time application traffic. 116 DetNet, of which architecture is defined in RFC 8655 [RFC8655], 117 provides a capability to carry specified unicast or multicast data 118 flows for real-time applications with extremely low data loss rates 119 and bounded latency within a network domain. The overall framework 120 for DetNet data plane is provided in [RFC8938], and various documents 121 on different data plane technologies and their interworking 122 technologies to extend the service range of data that TSN intends to 123 deliver to the IP (Internet Protocol) and MPLS (Multi-Protocol Label 124 Switching) networks have been standardized. 126 Since TSN and DetNet were proposed, application use cases have always 127 been one of the hottest topics. As documented in RFC 8578 [RFC8578], 128 the scope of networks addressed by the current DetNet is limited to 129 networks that can be centrally controlled, i.e., an "enterprise" (aka 130 "corporate") network, excluding "the open Internet," explicitly. 131 After years of development, TSN has been used in several industries, 132 and has enough public awareness of the industry for its scope. 133 DetNet also has done a lot of work and the standards are mature, and 134 people become concerned about how to meet deterministic service 135 demand in large-scale networks. The current DetNet is limited to a 136 single administrative domain network, and there are technical 137 elements necessary for application to a large-scale network spanning 138 multiple domains. 140 This document describes requirements for large-scale deterministic 141 networks where different deterministic levels of applications co- 142 exist and large-scale deterministic networking across multiple 143 administrative domains is possible. This document also describes the 144 requirements for enhancing the DetNet data plane defined prior to 145 this document. 147 2. Conventions Used in This Document 149 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 150 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 151 "OPTIONAL" in this document are to be interpreted as described in BCP 152 14 [RFC2119][RFC8174] when, and only when, they appear in all 153 capitals, as shown here. 155 While [RFC2119] and [RFC8174] describe interpretations of these key 156 words in terms of protocol specifications and implementations, they 157 are used in this document to describe technical and operational 158 requirements to realize large-scale deterministic networks. 160 3. The Overall Characteristics of Large-Scale Deterministic Networks 162 When deterministic network services are introduced, network providers 163 always face the problem of how to match application needs to the 164 technology, so more works are needed for network service providers to 165 successfully sell DetNet type services to customers. The providers 166 are in need of the following: 168 Service level objective definitions, considering absolute or relative 169 latency and jitter bounds, flows types and physical network scale 171 Suitable queuing mechanisms, considering more options for queuing 172 mechanisms for different service level, and 174 Deployment strategies, considering how to integrate into existing 175 networks, service, and control plane. 177 [RFC8578]provides various use cases and their requirements in the 178 areas of industry, electricity, buildings, etc. Some of them clearly 179 specify the requirements for latency and jitter, while some others do 180 not for the jitter. Different types of users have different demands, 181 just as a network provider provides different network services for 182 personal business or enterprise business. 184 One kind has critical SLA requirement, such as remote control or 185 cloud Programmable Logic Controller (PLC) of manufacturing and 186 differential protection of electricity. If these services exceed the 187 boundaries of latency and jitter, it will bring property losses and 188 security risks, so they cannot tolerate with any non-deterministic 189 situation and can pay more on the network service. 191 Another kind has relatively losse levels of SLA requirement, such as 192 cloud gaming, cloud VR and online meeting for "consumer" networks. 193 The users of these applications hope to have a better network 194 experience, but they can tolerate it to a certain extent. If the 195 network quality is not good sometime, they might be willing to spend 196 more money for high-quality network services. In some aspects, 197 because such services have no industry barriers and can tolerate 198 exceeding the upper boundary of latency within a small probability, 199 they have relatively lower requirements for the network and may be 200 easier to deploy. 202 Different application demands are actually related to cost. For 203 strict deterministic services, strict technologies need to be used, 204 and all network devices may need to be upgraded. For non-strict 205 deterministic services, it may only be necessary to upgrade some 206 network devices (maybe edge nodes) or share corresponding network 207 resources. From the perspective of deployment, it is helpful if 208 there is a clear classification of application demands, including 209 latency, jitter, reliability, etc. In this way, the corresponding 210 technology to implement could be chosen, taking into account both 211 performance and cost, but how to make choice is not within the scope 212 of this document. 214 Critical latency requirements: 216 | <->| Industrial, tight jitter, hard latency limit 217 |<------->| Industrial, hard latency limit 218 | 219 |<-------------.....> Relatively lower latency requirements 220 | 221 |<-------------........................> Best effort 222 | 223 +----------------------------------------------------------> 224 latency 226 Figure 1: Figure 1: Different levels of application requirements 228 4. Technical Requirements in Large-Scale Deterministic Networks 230 Due to the different kinds of application requirements in large-scale 231 networks, the corresponding technical requirements should be 232 considered. 234 4.1. Tolerate Time Asynchrony 236 4.1.1. Support Asynchronous Clocks Across Domains 238 A large-scale network may span over multiple networks with one or 239 more administrative domains. One of DetNet's objectives is to stitch 240 TSN islands together. All devices inside a TSN domain are time- 241 synchronized, and most of TSN technologies rely on precise time 242 synchronization[IEEE802.1Qbv][IEEE802.1Qch][IEEE802.1Qav]However, 243 different TSN islands may have different clocks which are not 244 synchronized as shown in Figure 2, where the time difference of two 245 TSN domains is D. DetNet needs to connect these two TSN domains 246 together and provide end-to-end deterministic latency service. The 247 mechanism adopted by a large-scale deterministic network MUST support 248 the interaction across time domains, so that time domains are 249 synchronized. This can be done, for example, by putting extra buffer 250 space at the ingress of a new domain, increasing the dead time as a 251 guard band, or using some timing compensation mechanism. This 252 document does not intend to list all the potential ways. 254 +--------------+ +--------------+ 255 | | DetNet Connection | | 256 | TSN Domain I +-----------------------------+ TSN Domain II| 257 | | | | 258 +--------------+ +--------------+ 259 | | | | | 260 Clock of TSN +--------+--------+--------+--------+ 261 Domain I = 262 = 263 = | | | | | 264 Clock of TSN = +--------+--------+--------+--------+ 265 Domain II = = 266 =<==D==>= 267 = = 269 Figure 2: Figure 2: Clock asynchrony between two TSN islands 271 4.1.2. Tolerate Clock Jitter & Wander within a Clock Synchronous Domain 273 Within a single time synchronization domain, different clock accuracy 274 is expected, for example the crystal oscillator in Ethernet is 275 specified at 100 ppm[Fast-Ethernet-MII-clock], Synchronous Ethernet 276 (SyncE) can achieve 50 ppb[G.8262], and more precise time 277 synchronization[G.8273] is expected in 5G mobile backhaul. The 278 clocks experience different jitter and wander. It may cause 279 different level of asymmetry of the path. The large-scale networks 280 SHOULD be able to recover or absorb such time variance within a 281 domain and across multiple domains. 283 4.1.3. Provide Mechanisms not Requiring Full Time Synchronization 285 Some networks like mobile backhaul use frequency synchronization, 286 such as SyncE, instead of the strict time synchronization. It is 287 usually hard to achieve the full time synchronization in large-scale 288 networks when considering the size of the network topology. It is 289 desired that the same deterministic performance in term of the 290 bounded latency and jitter SHOULD be achieved when full time 291 synchronization is not available, that is to say, when only partial 292 synchronization (SyncE is one of the examples) is in use. 294 4.1.4. Support Asynchronization based Methods 296 There are a large number of traffic flows in a large-scale network 297 and some of them are acyclic. Asynchronization based methods can 298 meet the requirements of those traffic flows. Moreover, The 299 mechanisms not requiring the time and/or frequency synchronization 300 eliminate the hardware cost and difficulty at the network 301 nodes.[IEEE802.1Qcr] conceptually uses per-flow based asynchronous 302 shaper to achieve bounded latency. The formula proof shows its 303 effectiveness. It can naturally tolerate the time variance, but it 304 exhibits the concerns of per-flow state buffer management as shown 305 in[I-D.eckert-detnet-bounded-latency-problems]When it is in use, the 306 requirement in Section 4.3 SHOULD be carefully met. 308 4.2. Support Large Single-hop Propagation Latency 310 In a large-scale network, a single hop distance is enough to generate 311 large latency. The speed of optical transmission in fiber is 200 km/ 312 ms. Thus, the propagation delay of a single hop can be in the order 313 of a few milliseconds. It is much greater than that of a LAN, and 314 introduces impacts on queuing mechanisms, such as cyclic or time 315 aware scheduling method. 317 For a cyclic based method, suppose a large-scale network wants to 318 keep using the simple cycle mapping relationship, however the link 319 distance between two nodes is longer. Moreover, a downstream node 320 may have many upstream nodes each with different link propagation 321 delays (e.g., 9 us, 10 us, 11 us, 15 us and 20 us). In order to 322 absorb the longest link propagation delay, the length of cycle must 323 be set to at least 20 us. However, since packet's arrival time 324 varies within the receiving cycle, larger cycle length means larger 325 delay variance. 327 Upstream Node X |sending cycle | | 328 +--"------------+------------+ 329 = "\ = = 330 = " \ = = 331 = " \ = = 332 = " \ = = 333 = " V = = 334 Downstream Node Y |receiving cycle| | 335 +--"----"-------+----\-------+ 336 = " " = \ = 337 = " " = V resent out 338 = " " = = 339 Time Line -=--"----"-------=------------=-----> 340 (in us) 0 | | 10 20 341 v v 342 Transmission Latency 344 Figure 3: Figure 3: The influence of transmission latency on a 345 cyclic method 347 4.3. Accommodate the Higher Link Speed 349 A large-scale network normally uses higher speed links, especially 350 for its backbone. Current deterministic mechanisms used in a local 351 network is usually deployed in link speed of 10 Mbps or 1 Gbps, or 352 possibly 10 Gbps. The data rate of 10G, 100G, 400G and even higher 353 is commonly used in wide area networks. With the increasing of the 354 data rate, the network scheduling cycle can be reduced if the same 355 amount of the data is required to be sent each cycle for each 356 application. Or more data can be sent if the network cycle time 357 remains the same. For the former, it requires the more precise time 358 control (e.g. cycle in the order of a few microseconds or sub- 359 microseconds) for the input stream gate and the timed output buffer. 360 For the latter, more buffer space is required which imposes more 361 complex buffer or queue management and larger memory consumption. 363 Another aspect to consider is the aggregation of the flows. In the 364 large-scale network, the number of flows can be hundreds or tens of 365 thousands. They can be aggregated into a small number of 366 deterministic path or tunnels. It is practical to have a few flow- 367 based or aggregated-flow based status in the local network. But in 368 higher speed and larger scale networks, it is hardly feasible. 369 If[IEEE802.1Qcr]is in use, it requires more buffers comparing to the 370 other full/partial time synchronized mechanisms. Therefore, it 371 requires optimizations to support higher link speeds. 373 4.4. Be Scalable to Numerous Network Devices and Massive Traffic Flows 375 Comparing to a LAN, a large-scale network may have more network 376 devices and traffic flows, and there is a greater possibility of 377 adding or removing network devices and traffic flows. The 378 deterministic latency forwarding mechanisms MUST scale to networks of 379 significant size with numerous network devices and a massive traffic 380 flows. 382 The increase or decrease of network devices in large-scale networks 383 is more frequent than that in LANs. The change of the number of 384 devices may affect the implementation and adjustment of deterministic 385 network mechanism, such as the topology discovery, queuing mechanism 386 and packet replication and elimination. A simple use case to 387 understand is ultra-low-latency (public) 5G transport networks, which 388 would require DetNet extend to every 5G base station. For some 389 network operators, their networks may need to connect to ~100 K base 390 stations (serving multiple mobile networks operators), and this 391 number will only increase with 5G. 393 It is almost impossible to identify individual IP flows at the DetNet 394 data plane because of the large overhead and resource reservation for 395 a massive number of flows. DetNet allows the leverage of the flow 396 aggregation. With the large scaling of the network, proper provision 397 at the control plane to accommodate such higher aggregation is 398 required. Individual flows may join and exit the aggregated flow 399 rapidly which causes the dynamic in identification of the aggregated 400 DetNet flow. The wildcards and value ranges used in the 401 identification may have to change in order to ensure the aggregated 402 flows have compatible deterministic characteristics. 404 The micro-burst will happen more often due to the massive traffic 405 flows, so some methods to decrease it are 406 needed.[I-D.du-detnet-layer3-low-latency]introduces a reference 407 method requiring a scalable buffer to adjust the speed of sending the 408 packets, so as to keep a uniform transmission rate, and it also 409 support the flow aggregation. 411 4.5. Tolerate Failures of Links or Nodes and Topology Changes 413 Network link failures are more common in large-scale networks. Path 414 switching or re-convergence of routing will cause high latency of 415 packet loss and retransmission, which is usually in seconds before 416 the network becomes stable again. It is necessary to support certain 417 mechanisms to adapt to failures of links or nodes and topology 418 changes. 420 The change of path or topology poses a higher challenge to packet 421 replication and elimination. The full disjoint paths when 422 implementing the Packet Replication, Elimination, and Ordering 423 Functions (PREOF) gives a better chance of survival when one of the 424 nodes or links in the path fails. At the same time, it brings the 425 challenges of finding paths with similar distance and/or number of 426 hops so that there is enough buffer space to absorb the latency 427 difference caused by different paths when the scale is large. 429 4.6. Support Configuration of Multiple Queueing Mechanisms 431 It is required to provide diversified deterministic service for 432 various applications in a large-scale network and to support the 433 corresponding diversified queueing mechanisms (possibly at multiple 434 DetNet QoS levels). Different queueing mechanisms can provide 435 different levels of latency, jitter and other guarantees, and there 436 may be situations where a network device provides multiple queueing 437 mechanisms at the same time. For example, a network aggregation 438 device may use the mechanisms specified in [IEEE802.1Qbv] and 439 [IEEE802.1Qcr], and other mechanisms to forward traffic to different 440 paths at the same time. By providing a variety of queueing 441 mechanisms to meet diversified deterministic service Requirements, 442 compared with LAN environment, this demand is particularly prominent 443 in large-scale networks. There are usually eight traffic classes in 444 TSN enabled networks. The different queueing mechanisms can be 445 employed to the queues of one or more of those traffic class. In 446 practice, there may be more than eight queues or sub-queues to 447 support more complicated queueing mechanisms. 449 Accordingly, the configuration for multiple queueing mechanisms is 450 complicated in large-scale deterministic networks and MUST support 451 the unified or simplified scheduling and management of multiple queue 452 mechanisms. For example, in the distributed scenario where there is 453 no controller, flooding the related information of the queue 454 mechanism, including the types and related algorithms, queue 455 forwarding capability, etc. In the centralized scenario, the 456 queueing mechanisms and other information could be reported to the 457 controller to build a deterministic network resource topology pool 458 for path calculation. 460 4.7. Support Queueing Mechanisms Switchover Crossing Multi-domains 462 In large-scale deterministic networks, it may across multiple network 463 domains and adopt a variety of different queueing mechanisms within 464 each domain. It is required to support the inter-domain 465 deterministic mechanism at the inter-domain boundary nodes such as 466 the priority redefinition and rescheduling of queues to achieve the 467 end-to-end latency, bounded jitter and packet loss ratio. 469 Moreover, changing from one queueing mechanism to another may 470 generate additional end-to-end latency and/or jitter which should be 471 taken into consideration. For example, when a flow is forwarded 472 across multiple network domains based on different queueing 473 mechanisms, such as a time synchronous Qbv mechanism[IEEE802.1Qbv] 474 and an asynchronous Qcr mechanism [IEEE802.1Qcr], a collaboration 475 mechanism crossing multi-domains MUST be considered, such as 476 increasing the buffer of inter-domain devices to provide enough 477 adjustment space for the flow to cross different queueing mechanisms, 478 so as to provide end-to-end deterministic services across multiple 479 network domains. 481 5. Data Plane Enhancement Requirements 483 According to[RFC8938], the DetNet data plane can provide or carry two 484 metadata in MPLS and IP data planes: Flow-ID and sequence number. 485 The Flow-ID could be used for identification of the DetNet flow or 486 aggregate flow, and the sequence number could be used for PREOF for 487 each DetNet flow. The Flow-ID is used by both the service and 488 forwarding sub-layers, but the sequence number is only used by the 489 service layer. Metadata can also be used for OAM indications and 490 instrumentation of DetNet data plane operation. 492 Generally speaking, more data plane metadata and related processing 493 SHOULD be supported in the large-scale networks. Native IPv6 data 494 plane should be supported. This section lists the data plane 495 enhancement requirements based on but not limited to the technical 496 requirements in Section 4. 498 5.1. Support Aggregated Flow Identification 500 Current IPv6 aggregated flow identification is generally based on 5 501 or 6 tuples, IP prefixes, or wildcards as indicated in [RFC8938]. 502 However, in large-scale deterministic networks the number of 503 individual flows is huge, and they may randomly join and leave the 504 aggregated flow at each hop. Such behaviours lead to the difficulty 505 in identifying aggregated flows by relying on the prefixes or 506 wildcards. 508 In addition, flow identification is also used to quickly push a 509 packet to a suitable queue. In a large-scale network, there are mix 510 of flows requiring deterministic latency service and normal 511 forwarding service. Explicit flow identification makes it easier to 512 quickly distinguish the DetNet flows without requiring the longest 513 match rule on multiple tuples in IP data plane. Therefore, explicit 514 aggregated flow identification SHOULD be supported. 516 5.2. Support Queuing Related Information 518 According to Section 4.1, a large-scale network should support 519 synchronized or asynchronized queuing mechanisms. Different queueing 520 mechanisms require different metadata to be defined to help 521 regulation and queue management. For instance, the data plane MUST 522 support the identification of cycle for cyclic queuing or the timing 523 related information for time based queuing. 525 5.3. Support Redundancy Related Fields 527 Sequence number is the only metadata currently defined for redundancy 528 feature of Detnet. MPLS data plane uses Detnet-over-MPLS label stack 529 to carry it. At the same time, native IPv6 data plane should be able 530 to carry this information too. If specific IP encapsulation or 531 tunnel is in use, this meta data should be defined explicitly for 532 that data plane. 534 5.4. Support Explicit Path Selection 536 Explicit route at the control plane and/or management is required so 537 that the "best" path can be selected to meet the latency requirement 538 for DetNet flows. At the data planes, MPLS label stack can be used 539 for this purpose. IP data plane enhancement is required to support 540 the explicit path selection based on IP source routing or SRv6. 542 6. Conclusion 544 This document specifies the technical requirements when ensuring the 545 deterministic features in the large-scale networks, and the 546 corresponding data plane enhancement requirements to support the 547 them. Some of the proposed queueing mechanisms and trials are cited 548 and the authors of the document think those proposals give reasonably 549 sound insights to enhancement the current queueing mechanisms to meet 550 the deterministic requirements of the large-scale networks. 552 7. Security Considerations 554 There are no IANA actions required by this document. 556 8. IANA Considerations 558 This section will be described later. 560 9. Acknowledgements 562 The authors would like to thank Yaakov Stein for helpful suggestions. 563 The authors also would like to thank Liang Geng, Peter Willis, 564 Shunsuke Homma and Li Qiang for their previous works. 566 10. Contributors 568 The following people have substantially contributed to this document: 570 Zongpeng Du 571 China Mobile 572 EMail: duzongpeng@chinamobile.com 574 11. Normative References 576 [Fast-Ethernet-MII-clock] 577 "Fast Ethernet MII clock". 579 [G.8262] International Telecommunication Union, "Timing 580 characteristics of a synchronous equipment slave clock", 581 ITU-T Recommendation G.8262, November 2018. 583 [G.8273] International Telecommunication Union, "Framework of phase 584 and time clocks", ITU-T Recommendation G.8273, March 2018. 586 [I-D.dang-queuing-with-multiple-cyclic-buffers] 587 Liu, B. and J. Dang, "A Queuing Mechanism with Multiple 588 Cyclic Buffers", Work in Progress, Internet-Draft, draft- 589 dang-queuing-with-multiple-cyclic-buffers-00, 22 February 590 2021, . 593 [I-D.du-detnet-layer3-low-latency] 594 Du, Z. and P. Liu, "Micro-burst Decreasing in Layer3 595 Network for Low-Latency Traffic", Work in Progress, 596 Internet-Draft, draft-du-detnet-layer3-low-latency-04, 25 597 October 2021, . 600 [I-D.eckert-detnet-bounded-latency-problems] 601 Eckert, T. and S. Bryant, "Problems with existing DetNet 602 bounded latency queuing mechanisms", Work in Progress, 603 Internet-Draft, draft-eckert-detnet-bounded-latency- 604 problems-00, 12 July 2021, 605 . 608 [I-D.geng-detnet-requirements-bounded-latency] 609 Geng, L., Willis, P., Homma, S., and L. Qiang, 610 "Requirements of Layer 3 Deterministic Latency Service", 611 Work in Progress, Internet-Draft, draft-geng-detnet- 612 requirements-bounded-latency-03, 7 July 2019, 613 . 616 [I-D.qiang-detnet-large-scale-detnet] 617 Qiang, L., Geng, X., Liu, B., Eckert, T., Geng, L., and G. 618 Li, "Large-Scale Deterministic IP Network", Work in 619 Progress, Internet-Draft, draft-qiang-detnet-large-scale- 620 detnet-05, 2 September 2019, 621 . 624 [IEEE802.1Qav] 625 IEEE, "IEEE Standard for Local and metropolitan area 626 networks -- Virtual Bridged Local Area Networks - 627 Amendment 12: Forwarding and Queuing Enhancements for 628 Time-Sensitive Streams", IEEE 802.1Qav-2009, 629 DOI 10.1109/IEEESTD.2010.8684664, 5 January 2010, 630 . 632 [IEEE802.1Qbv] 633 IEEE, "IEEE Standard for Local and metropolitan area 634 networks -- Bridges and Bridged Networks - Amendment 25: 635 Enhancements for Scheduled Traffic", IEEE 802.1Qbv-2015, 636 DOI 10.1109/IEEESTD.2016.8613095, 18 March 2016, 637 . 639 [IEEE802.1Qch] 640 IEEE, "IEEE Standard for Local and metropolitan area 641 networks -- Bridges and Bridged Networks - Amendment 29: 642 Cyclic Queuing and Forwarding", IEEE 802.1Qch-2017, 643 DOI 10.1109/IEEESTD.2017.7961303, 28 June 2017, 644 . 646 [IEEE802.1Qcr] 647 IEEE, "IEEE Standard for Local and Metropolitan Area 648 Networks -- Bridges and Bridged Networks - Amendment 34: 649 Asynchronous Traffic Shaping", IEEE 802.1Qcr-2020, 650 DOI 10.1109/IEEESTD.2020.9253013, 6 November 2020, 651 . 653 [IEEE802.1TSN] 654 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 655 Networking Task Group", 656 . 658 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 659 Requirement Levels", BCP 14, RFC 2119, 660 DOI 10.17487/RFC2119, March 1997, 661 . 663 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 664 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 665 May 2017, . 667 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 668 RFC 8578, DOI 10.17487/RFC8578, May 2019, 669 . 671 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 672 "Deterministic Networking Architecture", RFC 8655, 673 DOI 10.17487/RFC8655, October 2019, 674 . 676 [RFC8938] Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S. 677 Bryant, "Deterministic Networking (DetNet) Data Plane 678 Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020, 679 . 681 Appendix A. Examples of Large-Scale Deterministic Network Trials 683 Some trials have been carried out to verify the concept of large- 684 scale deterministic networks. 686 In order to verify the deterministic technology of large-scale 687 networks, a trial of Deterministic IP on China Environment for 688 Network Innovations (CENI), which is a network built for new network 689 technology trial, was deployed. A network with a distance of 3,000 690 km over 13 hops was tested, and the jitter was controlled within 691 100us. 693 In order to verify the remote control on Deterministic IP, which 694 required that the latency should be controlled within 4 ms and jitter 695 should be controlled within 20 us. A trial cooperated with Baosteel 696 spanned 600 km was deployed. Baosteel is a Chinese steel company and 697 put forward this demand. Both of the first and second trials are 698 based on a frequency synchronization solution. The mechanism details 699 could be found in . [I-D.dang-queuing-with-multiple-cyclic-buffers][I 700 -D.qiang-detnet-large-scale-detnet]. 702 In order to realize multi flows synchronization on an inter- 703 provincial network in an exhibition, Emergen proposed the requirement 704 that two flows of video and virtual reality (VR) were sent from 705 province A, and arrived at province B together, so people can see the 706 synchronization of video collected by camera and the VR model. This 707 requirement was proposed to facilitate the virtual industry product 708 deployment. Due to time and other problems, it was realized by the 709 edge network device for a relatively lower levels of service level 710 agreement (SLA). 712 Teaming up with a smart factory operator, network operators, 713 equipment companies, and universities, ETRI demonstrated an ultra-low 714 latency, high-reliability 5G wired and wireless network-based remote 715 industrial Internet of Things (IIoT) service by connecting a control 716 center and a smart factory through three different operators' 717 networks at a distance of 280 km. In this trail, it was demonstrated 718 that real-time remote smart manufacturing service is possible by 719 making round-trip delay below 3 ms within a smart factory and below 720 10 ms between remote 5G industrial devices. In the future, the team 721 plans to examine feasibility of large-scale deterministic networking 722 by connecting smart factories in Gyeongsan, South Korea and Oulu, 723 Finland. 725 These trials show that both operators and enterprise users begin to 726 put forward requirements for the certainty of large-scale networks, 727 but the implementation technologies are not exactly the same. 729 Authors' Addresses 731 Peng Liu 732 China Mobile 733 Beijing 734 100053 735 China 736 Email: liupengyjy@chinamobile.com 738 Yizhou Li 739 Huawei 740 Nanjing 741 210012 742 China 743 Email: liyizhou@huawei.com 744 Toerless Eckert 745 Futurewei Technologies USA 746 Santa Clara, 95014 747 United States of America 748 Email: tte@cs.fau.de 750 Quan Xiong 751 ZTE Corporation 752 Wuhan 753 430223 754 China 755 Email: xiong.quan@zte.com.cn 757 Jeong-dong Ryoo 758 ETRI 759 Daejeon 760 34129 761 South Korea 762 Email: ryoo@etri.re.kr