idnits 2.17.1 draft-liu-detnet-large-scale-requirements-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (11 February 2022) is 805 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'I-D.du-detnet-layer3-low-latency' is defined on line 525, but no explicit reference was found in the text == Unused Reference: 'I-D.eckert-detnet-bounded-latency-problems' is defined on line 532, but no explicit reference was found in the text == Unused Reference: 'I-D.geng-detnet-requirements-bounded-latency' is defined on line 540, but no explicit reference was found in the text == Unused Reference: 'I-D.qiang-detnet-large-scale-detnet' is defined on line 548, but no explicit reference was found in the text == Outdated reference: A later version (-05) exists of draft-du-detnet-layer3-low-latency-04 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Deterministic Networking Working Group P. Liu 3 Internet-Draft China Mobile 4 Intended status: Informational Y. Li 5 Expires: 15 August 2022 Huawei 6 T. Eckert 7 Futurewei Technologies USA 8 Q. Xiong 9 ZTE Corporation 10 J. Ryoo 11 ETRI 12 11 February 2022 14 Requirements for Large-Scale Deterministic Networks 15 draft-liu-detnet-large-scale-requirements-01 17 Abstract 19 Aiming at the large-scale deterministic network, this document 20 specifies the technical and operational requirements when the 21 different deterministic levels of applications co-exist and are 22 transported over a wide area. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on 15 August 2022. 41 Copyright Notice 43 Copyright (c) 2022 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 48 license-info) in effect on the date of publication of this document. 49 Please review these documents carefully, as they describe your rights 50 and restrictions with respect to this document. Code Components 51 extracted from this document must include Revised BSD License text as 52 described in Section 4.e of the Trust Legal Provisions and are 53 provided without warranty as described in the Revised BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 2. Conventions Used in This Document . . . . . . . . . . . . . . 4 59 3. The Overall Characteristics of Large-Scale Deterministic 60 Networks . . . . . . . . . . . . . . . . . . . . . . . . 4 61 4. Technical Requirements in Large-Scale Deterministic 62 Networks . . . . . . . . . . . . . . . . . . . . . . . . 5 63 4.1. Tolerate Time Asynchrony . . . . . . . . . . . . . . . . 5 64 4.1.1. Support Asynchronous Clocks Across Domains . . . . . 6 65 4.1.2. Tolerate Clock Jitter & Wander within a Clock 66 Synchronous Domain . . . . . . . . . . . . . . . . . 6 67 4.1.3. Provide Mechanisms not Requiring Full Time 68 Synchronization . . . . . . . . . . . . . . . . . . . 7 69 4.1.4. Support Asynchronization based Methods . . . . . . . 7 70 4.2. Support Large Single-hop Propagation Latency . . . . . . 7 71 4.3. Accommodate the Higher Link Speed . . . . . . . . . . . . 8 72 4.4. Be Scalable . . . . . . . . . . . . . . . . . . . . . . . 9 73 4.4.1. Be Scalable to Numerous Network Devices . . . . . . . 9 74 4.4.2. Be Scalable to Massive Traffic Flows . . . . . . . . 9 75 4.5. Tolerate Failures of Links or Nodes and Topology 76 Changes . . . . . . . . . . . . . . . . . . . . . . . . . 10 77 4.6. Support Configuration of Multiple Queueing Mechanisms . . 10 78 4.7. Support Queueing Mechanisms Switchover Crossing 79 Multi-domains . . . . . . . . . . . . . . . . . . . . . . 11 80 5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 11 81 6. Security Considerations . . . . . . . . . . . . . . . . . . . 11 82 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 83 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 11 84 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 11 85 10. Normative References . . . . . . . . . . . . . . . . . . . . 12 86 Appendix A. Examples of Large-Scale Deterministic Network 87 Trials . . . . . . . . . . . . . . . . . . . . . . . . . 14 88 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 15 90 1. Introduction 92 Packet networks are evolving from bandwidth-guaranteed Quality of 93 Service (QoS) to latency-guaranteed QoS that guarantees bounded 94 latency and definite latency. Bounded latency and definite latency 95 can be further understood as in-time delivery, in which a packet 96 arrives without exceeding a predetermined time, and on-time delivery, 97 in which a packet arrives at a predetermined time, respectively. In 98 addition, network survivability, which typically guarantees traffic 99 recovery within 50 ms in the event of a network failure, is evolving 100 to a level that guarantees lossless recovery. In order to realize 101 the evolution of QoS and network survivability of these networks, 102 Time-Sensitive Networking (TSN) technology and Deterministic 103 Networking (DetNet) technology are considered to be essential. 105 TSN is a set of standards developed by the IEEE 802.1 TSN Task Group 106 (TG) [IEEE802.1TSN] and specifies mechanisms and protocols necessary 107 to realize highly available IEEE 802.1 networks with bounded latency 108 to carry time-sensitive, real-time application traffic. 110 DetNet, of which architecture is defined in RFC 8655 [RFC8655], 111 provides a capability to carry specified unicast or multicast data 112 flows for real-time applications with extremely low data loss rates 113 and bounded latency within a network domain. Various documents on 114 data planes and their interworking technologies to extend the service 115 range of data that TSN intends to deliver to the IP (Internet 116 Protocol) and MPLS (Multi-Protocol Label Switching) networks have 117 been standardized. 119 Since TSN and DetNet were proposed, application use cases have always 120 been one of the hottest topics. As documented in RFC 8578 [RFC8578], 121 the scope of networks addressed by the current DetNet is limited to 122 networks that can be centrally controlled, i.e., an "enterprise" (aka 123 "corporate") network, excluding "the open Internet," explicitly. 124 After years of development, TSN has been used in several industries, 125 and has enough public awareness of the industry for its scope. 126 DetNet also has done a lot of work and the standards are mature, and 127 people become concerned about how to meet deterministic service 128 demand in large-scale networks. The current DetNet is limited to a 129 single administrative domain network, and there are technical 130 elements necessary for application to a large-scale network spanning 131 multiple domains. 133 This document describes requirements for large-scale deterministic 134 networks where different deterministic levels of applications co- 135 exist and large-scale deterministic networking across multiple 136 administrative domains is possible. 138 2. Conventions Used in This Document 140 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 141 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 142 "OPTIONAL" in this document are to be interpreted as described in BCP 143 14 [RFC2119][RFC8174] when, and only when, they appear in all 144 capitals, as shown here. 146 While [RFC2119] and [RFC8174] describe interpretations of these key 147 words in terms of protocol specifications and implementations, they 148 are used in this document to describe technical and operational 149 requirements to realize large-scale deterministic networks. 151 3. The Overall Characteristics of Large-Scale Deterministic Networks 153 When deterministic network services are introduced, network providers 154 always face the problem of how to match application needs to the 155 technology, so more works are needed for network service providers to 156 successfully sell DetNet type services to customers. The providers 157 are in need of the following: 159 Service level objective definitions, considering absolute or relative 160 latency and jitter bounds, flows types and physical network scale 162 Suitable queuing mechanisms, considering more options for queuing 163 mechanisms for different service level, and 165 Deployment strategies, considering how to integrate into existing 166 networks, service, and control plane. 168 [RFC8578]provides various use cases and their requirements in the 169 areas of industry, electricity, buildings, etc. Some of them clearly 170 specify the requirements for latency and jitter, while some others do 171 not for the jitter. Different types of users have different demands, 172 just as a network provider provides different network services for 173 personal business or enterprise business. 175 One kind has critical SLA requirement, such as remote control or 176 cloud Programmable Logic Controller (PLC) of manufacturing and 177 differential protection of electricity. If these services exceed the 178 boundaries of latency and jitter, it will bring property losses and 179 security risks, so they cannot tolerate with any non-deterministic 180 situation and can pay more on the network service. 182 Another kind has relatively lower levels of SLA requirement, such as 183 cloud gaming, cloud VR and online meeting for "consumer" networks. 184 The users of these applications hope to have a better network 185 experience, but they can tolerate it to a certain extent. If the 186 network quality is not good sometime, they might be willing to spend 187 more money for high-quality network services. In some aspects, 188 because such services have no industry barriers and can tolerate 189 exceeding the upper boundary of latency within a small probability, 190 they have relatively lower requirements for the network and may be 191 easier to deploy. 193 Different application demands are actually related to cost. For 194 strict deterministic services, strict technologies need to be used, 195 and all network devices may need to be upgraded. For non-strict 196 deterministic services, it may only be necessary to upgrade some 197 network devices (maybe edge nodes) or share corresponding network 198 resources. From the perspective of deployment, it is helpful if 199 there is a clear classification of application demands, including 200 latency, jitter, reliability, etc. In this way, the corresponding 201 technology to implement could be chosen, taking into account both 202 performance and cost, but how to make choice is not within the scope 203 of this document. 205 Critical latency requirements: 207 | <->| Industrial, tight jitter, hard latency limit 208 |<------->| Industrial, hard latency limit 209 | 210 |<-------------.....> Relatively lower latency requirements 211 | 212 |<-------------........................> Best effort 213 | 214 +----------------------------------------------------------> 215 latency 217 Figure 1: Figure 1: Different levels of application requirements 219 4. Technical Requirements in Large-Scale Deterministic Networks 221 Due to the different kinds of application requirements in large-scale 222 networks, the corresponding technical requirements should be 223 considered. 225 4.1. Tolerate Time Asynchrony 226 4.1.1. Support Asynchronous Clocks Across Domains 228 A large-scale network may span over multiple networks with one or 229 more administrators. One of DetNet's objectives is to stitch TSN 230 islands together. All devices inside a TSN domain are time- 231 synchronized, and most of TSN technologies rely on precise time 232 synchronization[IEEE802.1Qbv][IEEE802.1Qch][IEEE802.1Qav]However, 233 different TSN islands may have different clocks which are not 234 synchronized as shown in Figure 2, where the time difference of two 235 TSN domains is D. DetNet needs to connect these two TSN domains 236 together and provide end-to-end deterministic latency service. The 237 mechanism adopted by a large-scale deterministic network MUST support 238 the interaction across time domains, so that time domains are 239 synchronized. This can be done, for example, by putting extra buffer 240 space at the ingress of a new domain, increasing the dead time as a 241 guard band, or using some timing compensation mechanism. This 242 document does not intend to list all the potential ways. 244 +--------------+ +--------------+ 245 | | DetNet Connection | | 246 | TSN Domain I +-----------------------------+ TSN Domain II| 247 | | | | 248 +--------------+ +--------------+ 249 | | | | | 250 Clock of TSN +--------+--------+--------+--------+ 251 Domain I = 252 = 253 = | | | | | 254 Clock of TSN = +--------+--------+--------+--------+ 255 Domain II = = 256 =<==D==>= 257 = = 259 Figure 2: Figure 2: Clock asynchrony between two TSN islands 261 4.1.2. Tolerate Clock Jitter & Wander within a Clock Synchronous Domain 263 Within a single time synchronization domain, different clock accuracy 264 is expected, for example the crystal oscillator in Ethernet is 265 specified at 100 ppm[Fast-Ethernet-MII-clock], Synchronous Ethernet 266 (SyncE) can achieve 50 ppb[G.8262], and more precise time 267 synchronization[G.8273] is expected in 5G mobile backhaul. The 268 clocks experience different jitter and wander. It may cause 269 different level of asymmetry of the path. The large-scale networks 270 SHOULD be able to recover or absorb such time variance within a 271 domain and across multiple domains. 273 4.1.3. Provide Mechanisms not Requiring Full Time Synchronization 275 Some networks like mobile backhaul use frequency synchronization, 276 such as SyncE, instead of the strict time synchronization. It is 277 usually hard to achieve the full time synchronization in large-scale 278 networks when considering the size of the network topology. It is 279 desired that the same deterministic performance in term of the 280 bounded latency and jitter SHOULD be achieved when full time 281 synchronization is not available, that is to say, when only partial 282 synchronization (SyncE is one of the examples) is in use. 284 4.1.4. Support Asynchronization based Methods 286 There are a large number of traffic flows in a large-scale network 287 and some of them are acyclic. Asynchronization based methods can 288 meet the requirements of those traffic flows. Moreover, The 289 mechanisms not requiring the time and/or frequency synchronization 290 eliminate the hardware cost and difficulty at the network 291 nodes.[IEEE802.1Qcr] conceptually uses per-flow based asynchronous 292 shaper to achieve bounded latency. The formula proof shows its 293 effectiveness. It can naturally tolerate the time variance, but it 294 exhibits the concerns of per-flow state buffer management as shown 295 in[I-D.eckert-detnet-bounded-latency-problems]When it is in use, the 296 requirement in Section 4.3 SHOULD be carefully met. 298 4.2. Support Large Single-hop Propagation Latency 300 In a large-scale network, a single hop distance is enough to generate 301 large latency. The speed of optical transmission in fiber is 200 km/ 302 ms. Thus, the propagation delay of a single hop can be in the order 303 of a few milliseconds. It is much greater than that of a LAN, and 304 introduces impacts on queuing mechanisms, such as cyclic or time 305 aware scheduling method. 307 For a cyclic based method, suppose a large-scale network wants to 308 keep using the simple cycle mapping relationship, however the link 309 distance between two nodes is longer. Moreover, a downstream node 310 may have many upstream nodes each with different link propagation 311 delays (e.g., 9 us, 10 us, 11 us, 15 us and 20 us). In order to 312 absorb the longest link propagation delay, the length of cycle must 313 be set to at least 20 us. However, since packet's arrival time 314 varies within the receiving cycle, larger cycle length means larger 315 delay variance. 317 Upstream Node X |sending cycle | | 318 +--"------------+------------+ 319 = "\ = = 320 = " \ = = 321 = " \ = = 322 = " \ = = 323 = " V = = 324 Downstream Node Y |receiving cycle| | 325 +--"----"-------+----\-------+ 326 = " " = \ = 327 = " " = V resent out 328 = " " = = 329 Time Line -=--"----"-------=------------=-----> 330 (in us) 0 | | 10 20 331 v v 332 Transmission Latency 334 Figure 3: Figure 3: The influence of transmission latency on a 335 cyclic method 337 4.3. Accommodate the Higher Link Speed 339 A large-scale network normally uses higher speed links, especially 340 for its backbone. Current deterministic mechanisms used in a local 341 network is usually deployed in link speed of 10 Mbps or 1 Gbps, or 342 possibly 10 Gbps. The data rate of 10G, 100G, 400G and even higher 343 is commonly used in wide area networks. With the increasing of the 344 data rate, the network scheduling cycle can be reduced if the same 345 amount of the data is required to be sent each cycle for each 346 application. Or more data can be sent if the network cycle time 347 remains the same. For the former, it requires the more precise time 348 control (e.g. cycle in the order of a few microseconds or sub- 349 microseconds) for the input stream gate and the timed output buffer. 350 For the latter, more buffer space is required which imposes more 351 complex buffer or queue management and larger memory consumption. 353 Another aspect to consider is the aggregation of the flows. In the 354 large-scale network, the number of flows can be hundreds or tens of 355 thousands. They can be aggregated into a small number of 356 deterministic path or tunnels. It is practical to have a few flow- 357 based or aggregated-flow based status in the local network. But in 358 higher speed and larger scale networks, it is hardly feasible. 359 If[IEEE802.1Qcr]is in use, it requires more buffers comparing to the 360 other full/partial time synchronized mechanisms. Therefore, it 361 requires optimizations to support higher link speeds. 363 4.4. Be Scalable 365 Comparing to a LAN, a large-scale network may have more network 366 devices and traffic flows, and there is a greater possibility of 367 adding or removing network devices and traffic flows. The 368 deterministic latency forwarding mechanisms MUST scale to networks of 369 significant size with numerous network devices and a massive traffic 370 flows. 372 4.4.1. Be Scalable to Numerous Network Devices 374 The increase or decrease of network devices in large-scale networks 375 is more frequent than that in LANs. The change of the number of 376 devices may affect the implementation and adjustment of deterministic 377 network mechanism, such as the topology discovery, queuing mechanism 378 and packet replication and elimination. A simple use case to 379 understand is ultra-low-latency (public) 5G transport networks, which 380 would require DetNet extend to every 5G base station. For some 381 network operators, their networks may need to connect to ~100 K base 382 stations (serving multiple mobile networks operators), and this 383 number will only increase with 5G. 385 4.4.2. Be Scalable to Massive Traffic Flows 387 It is almost impossible to identify individual IP flows at the DetNet 388 data plane because of the large overhead and resource reservation for 389 a massive number of flows. DetNet allows the leverage of the flow 390 aggregation. With the large scaling of the network, proper provision 391 at the control plane to accommodate such higher aggregation is 392 required. Individual flows may join and exit the aggregated flow 393 rapidly which causes the dynamic in identification of the aggregated 394 DetNet flow. The wildcards and value ranges used in the 395 identification may have to change in order to ensure the aggregated 396 flows have compatible deterministic characteristics. 398 The micro-burst will happen more often due to the massive traffic 399 flows, so some methods to decrease it are 400 needed.[I-D.du-detnet-layer3-low-latency]introduces a reference 401 method requiring a scalable buffer to adjust the speed of sending the 402 packets, so as to keep a uniform transmission rate, and it also 403 support the flow aggregation. 405 4.5. Tolerate Failures of Links or Nodes and Topology Changes 407 Network link failures are more common in large-scale networks. Path 408 switching or re-convergence of routing will cause high latency of 409 packet loss and retransmission, which is usually in seconds before 410 the network becomes stable again. It is necessary to support certain 411 mechanisms to adapt to failures of links or nodes and topology 412 changes. 414 The change of path or topology poses a higher challenge to packet 415 replication and elimination. The full disjoint paths when 416 implementing the Packet Replication, Elimination, and Ordering 417 Functions (PREOF) gives a better chance of survival when one of the 418 nodes or links in the path fails. At the same time, it brings the 419 challenges of finding paths with similar distance and/or number of 420 hops so that there is enough buffer space to absorb the latency 421 difference caused by different paths when the scale is large. 423 4.6. Support Configuration of Multiple Queueing Mechanisms 425 It is required to provide diversified deterministic service for 426 various applications in a large-scale network and to support the 427 corresponding diversified queueing mechanisms (possibly at multiple 428 DetNet QoS levels). Different queueing mechanisms can provide 429 different levels of latency, jitter and other guarantees, and there 430 may be situations where a network device provides multiple queueing 431 mechanisms at the same time. For example, a network aggregation 432 device may use the mechanisms specified in [IEEE802.1Qbv] and 433 [IEEE802.1Qcr], and other mechanisms to forward traffic to different 434 paths at the same time. By providing a variety of queueing 435 mechanisms to meet diversified deterministic service Requirements, 436 compared with LAN environment, this demand is particularly prominent 437 in large-scale networks. There are usually eight traffic classes in 438 TSN enabled networks. The different queueing mechanisms can be 439 employed to the queues of one or more of those traffic class. In 440 practice, there may be more than eight queues or sub-queues to 441 support more complicated queueing mechanisms. 443 Accordingly, the configuration for multiple queueing mechanisms is 444 complicated in large-scale deterministic networks and MUST support 445 the unified or simplified scheduling and management of multiple queue 446 mechanisms. For example, in the distributed scenario where there is 447 no controller, flooding the related information of the queue 448 mechanism, including the types and related algorithms, queue 449 forwarding capability, etc. In the centralized scenario, the 450 queueing mechanisms and other information could be reported to the 451 controller to build a deterministic network resource topology pool 452 for path calculation. 454 4.7. Support Queueing Mechanisms Switchover Crossing Multi-domains 456 In large-scale deterministic networks, it may across multiple network 457 domains and adopt a variety of different queueing mechanisms within 458 each domain. It is required to support the inter-domain 459 deterministic mechanism at the inter-domain boundary nodes such as 460 the priority redefinition and rescheduling of queues to achieve the 461 end-to-end latency, bounded jitter and packet loss ratio. 463 Moreover, changing from one queueing mechanism to another may 464 generate additional end-to-end latency and/or jitter which should be 465 taken into consideration. For example, when a flow is forwarded 466 across multiple network domains based on different queueing 467 mechanisms, such as a time synchronous Qbv mechanism[IEEE802.1Qbv] 468 and an asynchronous Qcr mechanism [IEEE802.1Qcr], a collaboration 469 mechanism crossing multi-domains MUST be considered, such as 470 increasing the buffer of inter-domain devices to provide enough 471 adjustment space for the flow to cross different queueing mechanisms, 472 so as to provide end-to-end deterministic services across multiple 473 network domains. 475 5. Conclusion 477 This document specifies the technical requirements when ensuring the 478 deterministic features in the large-scale networks. Some of the 479 proposed queueing mechanisms are analyzed and the authors of the 480 document think those proposals give reasonably sound insights to 481 enhancement the current queueing mechanisms to meet the deterministic 482 requirements of the large-scale networks. 484 6. Security Considerations 486 There are no IANA actions required by this document. 488 7. IANA Considerations 490 This section will be described later. 492 8. Acknowledgements 494 The authors would like to thank Yaakov Stein for helpful suggestions. 495 The authors also would like to thank Liang Geng, Peter Willis, 496 Shunsuke Homma and Li Qiang for their previous works. 498 9. Contributors 500 The following people have substantially contributed to this document: 502 Zongpeng Du 503 China Mobile 504 EMail: duzongpeng@chinamobile.com 506 10. Normative References 508 [Fast-Ethernet-MII-clock] 509 "Fast Ethernet MII clock". 511 [G.8262] International Telecommunication Union, "Timing 512 characteristics of a synchronous equipment slave clock", 513 ITU-T Recommendation G.8262, November 2018. 515 [G.8273] International Telecommunication Union, "Framework of phase 516 and time clocks", ITU-T Recommendation G.8273, March 2018. 518 [I-D.dang-queuing-with-multiple-cyclic-buffers] 519 Liu, B. and J. Dang, "A Queuing Mechanism with Multiple 520 Cyclic Buffers", Work in Progress, Internet-Draft, draft- 521 dang-queuing-with-multiple-cyclic-buffers-00, 22 February 522 2021, . 525 [I-D.du-detnet-layer3-low-latency] 526 Du, Z. and P. Liu, "Micro-burst Decreasing in Layer3 527 Network for Low-Latency Traffic", Work in Progress, 528 Internet-Draft, draft-du-detnet-layer3-low-latency-04, 25 529 October 2021, . 532 [I-D.eckert-detnet-bounded-latency-problems] 533 Eckert, T. and S. Bryant, "Problems with existing DetNet 534 bounded latency queuing mechanisms", Work in Progress, 535 Internet-Draft, draft-eckert-detnet-bounded-latency- 536 problems-00, 12 July 2021, 537 . 540 [I-D.geng-detnet-requirements-bounded-latency] 541 Geng, L., Willis, P., Homma, S., and L. Qiang, 542 "Requirements of Layer 3 Deterministic Latency Service", 543 Work in Progress, Internet-Draft, draft-geng-detnet- 544 requirements-bounded-latency-03, 7 July 2019, 545 . 548 [I-D.qiang-detnet-large-scale-detnet] 549 Qiang, L., Geng, X., Liu, B., Eckert, T., Geng, L., and G. 550 Li, "Large-Scale Deterministic IP Network", Work in 551 Progress, Internet-Draft, draft-qiang-detnet-large-scale- 552 detnet-05, 2 September 2019, 553 . 556 [IEEE802.1Qav] 557 IEEE, "IEEE Standard for Local and metropolitan area 558 networks -- Virtual Bridged Local Area Networks - 559 Amendment 12: Forwarding and Queuing Enhancements for 560 Time-Sensitive Streams", IEEE 802.1Qav-2009, 561 DOI 10.1109/IEEESTD.2010.8684664, 5 January 2010, 562 . 564 [IEEE802.1Qbv] 565 IEEE, "IEEE Standard for Local and metropolitan area 566 networks -- Bridges and Bridged Networks - Amendment 25: 567 Enhancements for Scheduled Traffic", IEEE 802.1Qbv-2015, 568 DOI 10.1109/IEEESTD.2016.8613095, 18 March 2016, 569 . 571 [IEEE802.1Qch] 572 IEEE, "IEEE Standard for Local and metropolitan area 573 networks -- Bridges and Bridged Networks - Amendment 29: 574 Cyclic Queuing and Forwarding", IEEE 802.1Qch-2017, 575 DOI 10.1109/IEEESTD.2017.7961303, 28 June 2017, 576 . 578 [IEEE802.1Qcr] 579 IEEE, "IEEE Standard for Local and Metropolitan Area 580 Networks -- Bridges and Bridged Networks - Amendment 34: 581 Asynchronous Traffic Shaping", IEEE 802.1Qcr-2020, 582 DOI 10.1109/IEEESTD.2020.9253013, 6 November 2020, 583 . 585 [IEEE802.1TSN] 586 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 587 Networking Task Group", 588 . 590 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 591 Requirement Levels", BCP 14, RFC 2119, 592 DOI 10.17487/RFC2119, March 1997, 593 . 595 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 596 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 597 May 2017, . 599 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 600 RFC 8578, DOI 10.17487/RFC8578, May 2019, 601 . 603 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 604 "Deterministic Networking Architecture", RFC 8655, 605 DOI 10.17487/RFC8655, October 2019, 606 . 608 Appendix A. Examples of Large-Scale Deterministic Network Trials 610 Some trials have been carried out to verify the concept of large- 611 scale deterministic networks. 613 In order to verify the deterministic technology of large-scale 614 networks, a trial of Deterministic IP on China Environment for 615 Network Innovations (CENI), which is a network built for new network 616 technology trial, was deployed. A network with a distance of 3,000 617 km over 13 hops was tested, and the jitter was controlled within 618 100us. 620 In order to verify the remote control on Deterministic IP, which 621 required that the latency should be controlled within 4 ms and jitter 622 should be controlled within 20 us. A trial cooperated with Baosteel 623 spanned 600 km was deployed. Baosteel is a Chinese steel company and 624 put forward this demand. Both of the first and second trials are 625 based on a frequency synchronization solution. The mechanism details 626 could be found in . [I-D.dang-queuing-with-multiple-cyclic-buffers][I 627 -D.qiang-detnet-large-scale-detnet]. 629 In order to realize multi flows synchronization on an inter- 630 provincial network in an exhibition, Emergen proposed the requirement 631 that two flows of video and virtual reality (VR) were sent from 632 province A, and arrived at province B together, so people can see the 633 synchronization of video collected by camera and the VR model. This 634 requirement was proposed to facilitate the virtual industry product 635 deployment. Due to time and other problems, it was realized by the 636 edge network device for a relatively lower levels of service level 637 agreement (SLA). 639 Teaming up with a smart factory operator, network operators, 640 equipment companies, and universities, ETRI demonstrated an ultra-low 641 latency, high-reliability 5G wired and wireless network-based remote 642 industrial Internet of Things (IIoT) service by connecting a control 643 center and a smart factory through three different operators' 644 networks at a distance of 280 km. In this trail, it was demonstrated 645 that real-time remote smart manufacturing service is possible by 646 making round-trip delay below 3 ms within a smart factory and below 647 10 ms between remote 5G industrial devices. In the future, the team 648 plans to examine feasibility of large-scale deterministic networking 649 by connecting smart factories in Gyeongsan, South Korea and Oulu, 650 Finland. 652 These trials show that both operators and enterprise users begin to 653 put forward requirements for the certainty of large-scale networks, 654 but the implementation technologies are not exactly the same. 656 Authors' Addresses 658 Peng Liu 659 China Mobile 660 Beijing 661 100053 662 China 664 Email: liupengyjy@chinamobile.com 666 Yizhou Li 667 Huawei 668 Nanjing 669 210012 670 China 672 Email: liyizhou@huawei.com 674 Toerless Eckert 675 Futurewei Technologies USA 676 Santa Clara, 95014 677 United States of America 679 Email: tte@cs.fau.de 681 Quan Xiong 682 ZTE Corporation 683 Wuhan 684 430223 685 China 687 Email: xiong.quan@zte.com.cn 688 Jeong-dong Ryoo 689 ETRI 690 Daejeon 691 34129 692 South Korea 694 Email: ryoo@etri.re.kr