idnits 2.17.1 draft-ietf-detnet-architecture-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 639 has weird spacing: '...e stack v ...' == Line 1141 has weird spacing: '...mediate inter...' == Line 1144 has weird spacing: '...mediate inter...' -- The document date (May 1, 2018) is 2180 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Network' is mentioned on line 775, but not defined == Unused Reference: 'AVnu' is defined on line 1623, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-architecture' is defined on line 1647, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-tsch' is defined on line 1652, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-roll-rpl-industrial-applicability' is defined on line 1667, but no explicit reference was found in the text == Unused Reference: 'I-D.svshah-tsvwg-deterministic-forwarding' is defined on line 1673, but no explicit reference was found in the text == Unused Reference: 'I-D.varga-detnet-service-model' is defined on line 1678, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1AS-2011' is defined on line 1683, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1TSNTG' is defined on line 1728, but no explicit reference was found in the text == Unused Reference: 'IEEE802.3-2015' is defined on line 1733, but no explicit reference was found in the text == Unused Reference: 'ISA95' is defined on line 1743, but no explicit reference was found in the text == Unused Reference: 'ODVA' is defined on line 1747, but no explicit reference was found in the text == Unused Reference: 'Profinet' is defined on line 1754, but no explicit reference was found in the text == Unused Reference: 'RFC5673' is defined on line 1810, but no explicit reference was found in the text == Outdated reference: A later version (-30) exists of draft-ietf-6tisch-architecture-13 == Outdated reference: A later version (-09) exists of draft-ietf-detnet-problem-statement-03 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-15 -- Obsolete informational reference (is this intentional?): RFC 5316 (Obsoleted by RFC 9346) Summary: 0 errors (**), 0 flaws (~~), 21 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft Huawei 4 Intended status: Standards Track P. Thubert 5 Expires: November 2, 2018 Cisco 6 B. Varga 7 J. Farkas 8 Ericsson 9 May 1, 2018 11 Deterministic Networking Architecture 12 draft-ietf-detnet-architecture-05 14 Abstract 16 Deterministic Networking (DetNet) provides a capability to carry 17 specified unicast or multicast data flows for real-time applications 18 with extremely low data loss rates and bounded latency. Techniques 19 used include: 1) reserving data plane resources for individual (or 20 aggregated) DetNet flows in some or all of the intermediate nodes 21 (e.g. bridges or routers) along the path of the flow; 2) providing 22 explicit routes for DetNet flows that do not rapidly change with the 23 network topology; and 3) distributing data from DetNet flow packets 24 over time and/or space to ensure delivery of each packet's data' in 25 spite of the loss of a path. The capabilities can be managed by 26 configuration, or by manual or automatic network management. 28 Status of This Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at https://datatracker.ietf.org/drafts/current/. 38 Internet-Drafts are draft documents valid for a maximum of six months 39 and may be updated, replaced, or obsoleted by other documents at any 40 time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress." 43 This Internet-Draft will expire on November 2, 2018. 45 Copyright Notice 47 Copyright (c) 2018 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (https://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with respect 55 to this document. Code Components extracted from this document must 56 include Simplified BSD License text as described in Section 4.e of 57 the Trust Legal Provisions and are provided without warranty as 58 described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 63 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 64 2.1. Terms used in this document . . . . . . . . . . . . . . . 4 65 2.2. IEEE 802 TSN to DetNet dictionary . . . . . . . . . . . . 6 66 3. Providing the DetNet Quality of Service . . . . . . . . . . . 7 67 3.1. Primary goals defining the DetNet QoS . . . . . . . . . . 7 68 3.2. Mechanisms to achieve DetNet Qos . . . . . . . . . . . . 9 69 3.2.1. Congestion protection . . . . . . . . . . . . . . . . 9 70 3.2.2. Explicit routes . . . . . . . . . . . . . . . . . . . 9 71 3.2.3. Jitter Reduction . . . . . . . . . . . . . . . . . . 10 72 3.2.4. Packet Replication and Elimination . . . . . . . . . 11 73 3.2.5. Packet encoding for service protection . . . . . . . 12 74 3.3. Secondary goals for DetNet . . . . . . . . . . . . . . . 13 75 3.3.1. Coexistence with normal traffic . . . . . . . . . . . 13 76 3.3.2. Fault Mitigation . . . . . . . . . . . . . . . . . . 13 77 4. DetNet Architecture . . . . . . . . . . . . . . . . . . . . . 14 78 4.1. DetNet stack model . . . . . . . . . . . . . . . . . . . 14 79 4.1.1. Representative Protocol Stack Model . . . . . . . . . 14 80 4.1.2. DetNet Data Plane Overview . . . . . . . . . . . . . 16 81 4.1.3. Network reference model . . . . . . . . . . . . . . . 18 82 4.2. DetNet systems . . . . . . . . . . . . . . . . . . . . . 19 83 4.2.1. End system . . . . . . . . . . . . . . . . . . . . . 19 84 4.2.2. DetNet edge, relay, and transit nodes . . . . . . . . 20 85 4.3. DetNet flows . . . . . . . . . . . . . . . . . . . . . . 21 86 4.3.1. DetNet flow types . . . . . . . . . . . . . . . . . . 21 87 4.3.2. Source guarantees . . . . . . . . . . . . . . . . . . 21 88 4.3.3. Incomplete Networks . . . . . . . . . . . . . . . . . 23 89 4.4. Traffic Engineering for DetNet . . . . . . . . . . . . . 23 90 4.4.1. The Application Plane . . . . . . . . . . . . . . . . 23 91 4.4.2. The Controller Plane . . . . . . . . . . . . . . . . 24 92 4.4.3. The Network Plane . . . . . . . . . . . . . . . . . . 24 94 4.5. Queuing, Shaping, Scheduling, and Preemption . . . . . . 25 95 4.6. Service instance . . . . . . . . . . . . . . . . . . . . 26 96 4.7. Flow identification at technology borders . . . . . . . . 27 97 4.7.1. Exporting flow identification . . . . . . . . . . . . 27 98 4.7.2. Flow attribute mapping between layers . . . . . . . . 29 99 4.7.3. Flow-ID mapping examples . . . . . . . . . . . . . . 30 100 4.8. Advertising resources, capabilities and adjacencies . . . 32 101 4.9. Provisioning model . . . . . . . . . . . . . . . . . . . 32 102 4.9.1. Centralized Path Computation and Installation . . . . 32 103 4.9.2. Distributed Path Setup . . . . . . . . . . . . . . . 32 104 4.10. Scaling to larger networks . . . . . . . . . . . . . . . 33 105 4.11. Connected islands vs. networks . . . . . . . . . . . . . 33 106 4.12. Compatibility with Layer-2 . . . . . . . . . . . . . . . 33 107 5. Security Considerations . . . . . . . . . . . . . . . . . . . 34 108 6. Privacy Considerations . . . . . . . . . . . . . . . . . . . 34 109 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 110 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 35 111 9. Access to IEEE 802.1 documents . . . . . . . . . . . . . . . 35 112 10. Informative References . . . . . . . . . . . . . . . . . . . 35 113 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 40 115 1. Introduction 117 Deterministic Networking (DetNet) is a service that can be offered by 118 a network to DetNet flows. DetNet provides these flows extremely low 119 packet loss rates and assured maximum end-to-end delivery latency. 120 This is accomplished by dedicating network resources such as link 121 bandwidth and buffer space to DetNet flows and/or classes of DetNet 122 flows, and by replicating packets along multiple paths. Unused 123 reserved resources are available to non-DetNet packets. 125 The Deterministic Networking Problem Statement 126 [I-D.ietf-detnet-problem-statement] introduces Deterministic 127 Networking, and Deterministic Networking Use Cases 128 [I-D.ietf-detnet-use-cases] summarizes the need for it. See 129 [I-D.dt-detnet-dp-alt] for a discussion of specific techniques that 130 can be used to identify DetNet Flows and assign them to specific 131 paths through a network. 133 A goal of DetNet is a converged network in all respects. That is, 134 the presence of DetNet flows does not preclude non-DetNet flows, and 135 the benefits offered DetNet flows should not, except in extreme 136 cases, prevent existing QoS mechanisms from operating in a normal 137 fashion, subject to the bandwidth required for the DetNet flows. A 138 single source-destination pair can trade both DetNet and non-DetNet 139 flows. End systems and applications need not instantiate special 140 interfaces for DetNet flows. Networks are not restricted to certain 141 topologies; connectivity is not restricted. Any application that 142 generates a data flow that can be usefully characterized as having a 143 maximum bandwidth should be able to take advantage of DetNet, as long 144 as the necessary resources can be reserved. Reservations can be made 145 by the application itself, via network management, by an applications 146 controller, or by other means. 148 Many applications of interest to Deterministic Networking require the 149 ability to synchronize the clocks in end systems to a sub-microsecond 150 accuracy. Some of the queue control techniques defined in 151 Section 4.5 also require time synchronization among relay and transit 152 nodes. The means used to achieve time synchronization are not 153 addressed in this document. DetNet should accommodate various 154 synchronization techniques and profiles that are defined elsewhere to 155 solve exchange time in different market segments. 157 Wired and wireless media differ greatly in a number of ways, 158 including connectivity possibilities and the reliability of packet 159 transmission. While some of the techniques described in this 160 document may be applicable to wireless media, the DetNet architecture 161 assumes the use of links with characteristics typical of wired, and 162 not wireless, media. 164 2. Terminology 166 2.1. Terms used in this document 168 The following special terms are used in this document in order to 169 avoid the assumption that a given element in the architecture does or 170 does not have Internet Protocol stack, functions as a router, bridge, 171 firewall, or otherwise plays a particular role at Layer-2 or higher. 173 App-flow 174 The native format of a DetNet flow. 176 destination 177 An end system capable of receiving a DetNet flow. 179 DetNet domain 180 The portion of a network that is DetNet aware. It includes 181 end systems and other DetNet nodes. 183 DetNet flow 184 A DetNet flow is a sequence of packets to which the DetNet 185 service is to be provided. 187 DetNet compound flow and DetNet member flow 188 A DetNet compound flow is a DetNet flow that has been 189 separated into multiple duplicate DetNet member flows, which 190 are eventually merged back into a single DetNet compound 191 flow, at the DetNet transport layer. "Compound" and "member" 192 are strictly relative to each other, not absolutes; a DetNet 193 compound flow comprising multiple DetNet member flows can, in 194 turn, be a member of a higher-order compound. 196 DetNet intermediate node 197 A DetNet relay node or transit node. 199 DetNet edge node 200 An instance of a DetNet relay node that includes either a 201 DetNet service layer proxy function for DetNet service 202 protection (e.g. the addition or removal of packet sequencing 203 information) for one or more end systems, or starts or 204 terminates congestion protection at the DetNet transport 205 layer, analogous to a Label Edge Router (LER). 207 DetNet-UNI 208 User-to-Network Interface with DetNet specific 209 functionalities. It is a packet-based reference point and 210 may provide multiple functions like encapsulation, status, 211 synchronization, etc. 213 end system 214 Commonly called a "host" or "node" in IETF documents, and an 215 "end station" is IEEE 802 documents. End systems of interest 216 to this document are either sources or destinations of DetNet 217 flows. And end system may or may not be DetNet transport 218 layer aware or DetNet service layer aware. 220 link 221 A connection between two DetNet nodes. It may be composed of 222 a physical link or a sub-network technology that can provide 223 appropriate traffic delivery for DetNet flows. 225 DetNet node 226 A DetNet aware end system, transit node, or relay node. 227 "DetNet" may be omitted in some text. 229 Detnet relay node 230 A DetNet node including a service layer function that 231 interconnects different DetNet transport layer paths to 232 provide service protection. A DetNet relay node can be a 233 bridge, a router, a firewall, or any other system that 234 participates in the DetNet service layer. It typically 235 incorporates DetNet transport layer functions as well, in 236 which case it is collocated with a transit node. 238 reservation 239 A trail of configuration between source to destination(s) 240 through transit nodes and subnets associated with a DetNet 241 flow, to provide congestion protection. 243 DetNet service layer 244 The layer at which service protection is provided, either 245 packet sequencing, replication, and elimination 246 (Section 3.2.4) or packet encoding (Section 3.2.5). 248 source 249 An end system capable of sourcing a DetNet flow. 251 DetNet transit node 252 A node operating at the DetNet transport layer, that utilizes 253 link layer and/or network layer switching across multiple 254 links and/or sub-networks to provide paths for DetNet service 255 layer functions. Optionally provides congestion protection 256 over those paths. An MPLS LSR is an example of a DetNet 257 transit node. 259 DetNet transport layer 260 The layer that optionally provides congestion protection for 261 DetNet flows over paths provided by the underlying network. 263 TSN 264 Time-Sensitive Networking, TSN is a Task Group of the IEEE 265 802.1 Working Group. 267 2.2. IEEE 802 TSN to DetNet dictionary 269 This section also serves as a dictionary for translating from the 270 terms used by the IEEE 802 Time-Sensitive Networking (TSN) Task Group 271 to those of the DetNet WG. 273 Listener 274 The IEEE 802 term for a destination of a DetNet flow. 276 relay system 277 The IEEE 802 term for a DetNet intermediate node. 279 Stream 280 The IEEE 802 term for a DetNet flow. 282 Talker 283 The IEEE 802 term for the source of a DetNet flow. 285 3. Providing the DetNet Quality of Service 287 3.1. Primary goals defining the DetNet QoS 289 The DetNet Quality of Service can be expressed in terms of: 291 o Minimum and maximum end-to-end latency from source to destination; 292 timely delivery and jitter avoidance derive from these constraints 294 o Probability of loss of a packet, under various assumptions as to 295 the operational states of the nodes and links. A derived property 296 is whether it is acceptable to deliver a duplicate packet, which 297 is an inherent risk in highly reliable and/or broadcast 298 transmissions 300 It is a distinction of DetNet that it is concerned solely with worst- 301 case values for the end-to-end latency. Average, mean, or typical 302 values are of no interest, because they do not affect the ability of 303 a real-time system to perform its tasks. In general, a trivial 304 priority-based queuing scheme will give better average latency to a 305 data flow than DetNet, but of course, the worst-case latency can be 306 essentially unbounded. 308 Three techniques are used by DetNet to provide these qualities of 309 service: 311 o Congestion protection (Section 3.2.1). 313 o Explicit routes (Section 3.2.2). 315 o Service protection (Section 3.2.4). 317 Congestion protection operates by reserving resources along the path 318 of a DetNet Flow, e.g. buffer space or link bandwidth. Congestion 319 protection greatly reduces, or even eliminates entirely, packet loss 320 due to output packet congestion within the network, but it can only 321 be supplied to a DetNet flow that is limited at the source to a 322 maximum packet size and transmission rate. 324 Congestion protection addresses both of the DetNet QoS requirements 325 (latency and packet loss). Given that DetNet nodes have a finite 326 amount of buffer space, congestion protection necessarily results in 327 a maximum end-to-end latency. It also addresses the largest 328 contribution to packet loss, which is buffer congestion. 330 After congestion, the most important contributions to packet loss are 331 typically from random media errors and equipment failures. Service 332 protection is the name for the mechanisms used by DetNet to address 333 these losses. The mechanisms employed are constrained by the 334 requirement to meet the users' latency requirements. Packet 335 replication and elimination (Section 3.2.4) and packet encoding 336 (Section 3.2.5) are described in this document to provide service 337 protection; others may be found. This mechanism distributes the 338 contents of DetNet flows over multiple paths in time and/or space, so 339 that the loss of some of the paths does need not cause the loss of 340 any packets. The paths are typically (but not necessarily) explicit 341 routes, so that they cannot suffer temporary interruptions caused by 342 the reconvergence of routing or bridging protocols. 344 These three techniques can be applied independently, giving eight 345 possible combinations, including none (no DetNet), although some 346 combinations are of wider utility than others. This separation keeps 347 the protocol stack coherent and maximizes interoperability with 348 existing and developing standards in this (IETF) and other Standards 349 Development Organizations. Some examples of typical expected 350 combinations: 352 o Explicit routes plus service protection are exactly the techniques 353 employed by [HSR-PRP]. Explicit routes are achieved by limiting 354 the physical topology of the network, and the sequentialization, 355 replication, and duplicate elimination are facilitated by packet 356 tags added at the front or the end of Ethernet frames. 358 o Congestion protection alone is is offered by IEEE 802.1 Audio 359 Video bridging [IEEE802.1BA-2011]. As long as the network suffers 360 no failures, zero congestion loss can be achieved through the use 361 of a reservation protocol (MSRP), shapers in every bridge, and a 362 bit of network calculus. 364 o Using all three together gives maximum protection. 366 There are, of course, simpler methods available (and employed, today) 367 to achieve levels of latency and packet loss that are satisfactory 368 for many applications. Prioritization and over-provisioning is one 369 such technique. However, these methods generally work best in the 370 absence of any significant amount of non-critical traffic in the 371 network (if, indeed, such traffic is supported at all), or work only 372 if the critical traffic constitutes only a small portion of the 373 network's theoretical capacity, or work only if all systems are 374 functioning properly, or in the absence of actions by end systems 375 that disrupt the network's operations. 377 There are any number of methods in use, defined, or in progress for 378 accomplishing each of the above techniques. It is expected that this 379 DetNet Architecture will assist various vendors, users, and/or 380 "vertical" Standards Development Organizations (dedicated to a single 381 industry) to make selections among the available means of 382 implementing DetNet networks. 384 3.2. Mechanisms to achieve DetNet Qos 386 3.2.1. Congestion protection 388 The primary means by which DetNet achieves its QoS assurances is to 389 reduce, or even completely eliminate, congestion at an output port as 390 a cause of packet loss. Given that a DetNet flow cannot be 391 throttled, this can be achieved only by the provision of sufficient 392 buffer storage at each hop through the network to ensure that no 393 packets are dropped due to a lack of buffer storage. 395 Ensuring adequate buffering requires, in turn, that the source, and 396 every intermediate node along the path to the destination (or nearly 397 every node -- see Section 4.3.3) be careful to regulate its output to 398 not exceed the data rate for any DetNet flow, except for brief 399 periods when making up for interfering traffic. Any packet sent 400 ahead of its time potentially adds to the number of buffers required 401 by the next hop, and may thus exceed the resources allocated for a 402 particular DetNet flow. 404 The low-level mechanisms described in Section 4.5 provide the 405 necessary regulation of transmissions by an end system or 406 intermediate node to provide congestion protection. The reservation 407 of the bandwidth and buffers for a DetNet flow requires the 408 provisioning described in Section 4.9. A DetNet node may have other 409 resources requiring allocation and/or scheduling, that might 410 otherwise be over-subscribed and trigger the rejection of a 411 reservation. 413 3.2.2. Explicit routes 415 In networks controlled by typical peer-to-peer protocols such as IEEE 416 802.1 ISIS bridged networks or IETF OSPF routed networks, a network 417 topology event in one part of the network can impact, at least 418 briefly, the delivery of data in parts of the network remote from the 419 failure or recovery event. Thus, even redundant paths through a 420 network, if controlled by the typical peer-to-peer protocols, do not 421 eliminate the chances of brief losses of contact. 423 Many real-time networks rely on physical rings or chains of two-port 424 devices, with a relatively simple ring control protocol. This 425 supports redundant paths for service protection with a minimum of 426 wiring. As an additional benefit, ring topologies can often utilize 427 different topology management protocols than those used for a mesh 428 network, with a consequent reduction in the response time to topology 429 changes. Of course, this comes at some cost in terms of increased 430 hop count, and thus latency, for the typical path. 432 In order to get the advantages of low hop count and still ensure 433 against even very brief losses of connectivity, DetNet employs 434 explicit routes, where the path taken by a given DetNet flow does not 435 change, at least immediately, and likely not at all, in response to 436 network topology events. Service protection (Section 3.2.4 or 437 Section 3.2.5) over explicit routes provides a high likelihood of 438 continuous connectivity. Explicit routes are commonly used in MPLS 439 TE LSPs. 441 3.2.3. Jitter Reduction 443 A core objective of DetNet is to enable the convergence of Non-IP 444 networks onto a common network infrastructure. This requires the 445 accurate emulation of currently deployed mission-specific networks, 446 which typically rely on point-to-point analog (e.g. 4-20mA 447 modulation) and serial-digital cables (or buses) for highly reliable, 448 synchronized and jitter-free communications. While the latency of 449 analog transmissions is basically the speed of light, legacy serial 450 links are usually slow (in the order of Kbps) compared to, say, GigE, 451 and some latency is usually acceptable. What is not acceptable is 452 the introduction of excessive jitter, which may, for instance, affect 453 the stability of control systems. 455 Applications that are designed to operate on serial links usually do 456 not provide services to recover the jitter, because jitter simply 457 does not exists there. Streams of information are expected to be 458 delivered in-order and the precise time of reception influences the 459 processes. In order to converge such existing applications, there is 460 a desire to emulate all properties of the serial cable, such as clock 461 transportation, perfect flow isolation and fixed latency. While 462 minimal jitter (in the form of specifying minimum, as well as 463 maximum, end-to-end latency) is supported by DetNet, there are 464 practical limitations on packet-based networks in this regard. In 465 general, users are encouraged to use, instead of, "do this when you 466 get the packet," a combination of: 468 o Sub-microsecond time synchronization among all source and 469 destination end systems, and 471 o Time-of-execution fields in the application packets. 473 Jitter reduction is provided by the mechanisms described in 474 Section 4.5 that also provide congestion protection. 476 3.2.4. Packet Replication and Elimination 478 After congestion loss has been eliminated, the most important causes 479 of packet loss are random media and/or memory faults, and equipment 480 failures. Both causes of packet loss can be greatly reduced by 481 spreading the data in a packet over multiple transmissions. One such 482 method for service protection is described in this section, which 483 sends the same packets over multiple paths. 485 Packet replication and elimination, also known as seamless redundancy 486 [HSR-PRP], or 1+1 hitless protection, is a function of the DetNet 487 service layer. It involves three capabilities: 489 o Providing sequencing information, once, at or near the source, to 490 the packets of a DetNet compound flow. This may be done by adding 491 a sequence number or time stamp as part of DetNet, or may be 492 inherent in the packet, e.g. in a transport protocol, or 493 associated to other physical properties such as the precise time 494 (and radio channel) of reception of the packet. Section 3.2.2. 496 o Replicating these packets into multiple DetNet member flows and, 497 typically, sending them along at least two different paths to the 498 destination(s), e.g. over the explicit routes of 500 o Eliminating duplicated packets. This may be done at any step 501 along the path to save network resources further down, in 502 particular if multiple Replication points exist. But the most 503 common case is to perform this operation at the very edge of the 504 DetNet network, preferably in or near the receiver. 506 This function is a "hitless" version of, e.g., the 1+1 linear 507 protection in [RFC6372]. That is, instead of switching from one flow 508 to the other when a failure of a flow is detected, DetNet combines 509 both flows, and performs a packet-by-packet selection of which to 510 discard, based on sequence number. 512 In the simplest case, this amounts to replicating each packet in a 513 source that has two interfaces, and conveying them through the 514 network, along separate paths, to the similarly dual-homed 515 destinations, that discard the extras. This ensures that one path 516 (with zero congestion loss) remains, even if some intermediate node 517 fails. The sequence numbers can also be used for loss detection and 518 for re-ordering. 520 Detnet relay nodes in the network can provide replication and 521 elimination facilities at various points in the network, so that 522 multiple failures can be accommodated. 524 This is shown in the following figure, where the two relay nodes each 525 replicate (R) the DetNet flow on input, sending the DetNet member 526 flows to both the other relay node and to the end system, and 527 eliminate duplicates (E) on the output interface to the right-hand 528 end system. Any one link in the network can fail, and the Detnet 529 compound flow can still get through. Furthermore, two links can 530 fail, as long as they are in different segments of the network. 532 Packet replication and elimination 534 > > > > > > > > > relay > > > > > > > > 535 > /------------+ R node E +------------\ > 536 > / v + ^ \ > 537 end R + v | ^ + E end 538 system + v | ^ + system 539 > \ v + ^ / > 540 > \------------+ R relay E +-----------/ > 541 > > > > > > > > > node > > > > > > > > 543 Figure 1 545 Packet replication and elimination does not react to and correct 546 failures; it is entirely passive. Thus, intermittent failures, 547 mistakenly created packet filters, or misrouted data is handled just 548 the same as the equipment failures that are detected handled by 549 typical routing and bridging protocols. 551 If packet replication and elimination is used over paths providing 552 congestion protection (Section 3.2.1), and member flows that take 553 different-length paths through the network are combined, a merge 554 point may require extra buffering to equalize the delays over the 555 different paths. This equalization ensures that the resultant 556 compound flow will not exceed its contracted bandwidth even after one 557 or the other of the paths is restored after a failure. 559 3.2.5. Packet encoding for service protection 561 There are methods for using multiple paths to provide service 562 protection that involve encoding the information in a packet 563 belonging to a DetNet flow into multiple transmission units, 564 combining information from multiple packets into any given 565 transmission unit. Such techniques, also known as "network coding", 566 can be used as a DetNet service protection technique. 568 3.3. Secondary goals for DetNet 570 Many applications require DetNet to provide additional services, 571 including coesistence with other QoS mechanisms Section 3.3.1 and 572 protection against misbehaving transmitters Section 3.3.2. 574 3.3.1. Coexistence with normal traffic 576 A DetNet network supports the dedication of a high proportion (e.g. 577 75%) of the network bandwidth to DetNet flows. But, no matter how 578 much is dedicated for DetNet flows, it is a goal of DetNet to coexist 579 with existing Class of Service schemes (e.g., DiffServ). It is also 580 important that non-DetNet traffic not disrupt the DetNet flow, of 581 course (see Section 3.3.2 and Section 5). For these reasons: 583 o Bandwidth (transmission opportunities) not utilized by a DetNet 584 flow are available to non-DetNet packets (though not to other 585 DetNet flows). 587 o DetNet flows can be shaped or scheduled, in order to ensure that 588 the highest-priority non-DetNet packet also is ensured a worst- 589 case latency (at any given hop). 591 o When transmission opportunities for DetNet flows are scheduled in 592 detail, then the algorithm constructing the schedule should leave 593 sufficient opportunities for non-DetNet packets to satisfy the 594 needs of the users of the network. Detailed scheduling can also 595 permit the time-shared use of buffer resources by different DetNet 596 flows. 598 Ideally, the net effect of the presence of DetNet flows in a network 599 on the non-DetNet packets is primarily a reduction in the available 600 bandwidth. 602 3.3.2. Fault Mitigation 604 One key to building robust real-time systems is to reduce the 605 infinite variety of possible failures to a number that can be 606 analyzed with reasonable confidence. DetNet aids in the process by 607 providing filters and policers to detect DetNet packets received on 608 the wrong interface, or at the wrong time, or in too great a volume, 609 and to then take actions such as discarding the offending packet, 610 shutting down the offending DetNet flow, or shutting down the 611 offending interface. 613 It is also essential that filters and service remarking be employed 614 at the network edge to prevent non-DetNet packets from being mistaken 615 for DetNet packets, and thus impinging on the resources allocated to 616 DetNet packets. 618 There exist techniques, at present and/or in various stages of 619 standardization, that can perform these fault mitigation tasks that 620 deliver a high probability that misbehaving systems will have zero 621 impact on well-behaved DetNet flows, except of course, for the 622 receiving interface(s) immediately downstream of the misbehaving 623 device. Examples of such techniques include traffic policing 624 functions (e.g. [RFC2475]) and separating flows into per-flow rate- 625 limited queues. 627 4. DetNet Architecture 629 4.1. DetNet stack model 631 4.1.1. Representative Protocol Stack Model 633 Figure 2 illustrates a conceptual DetNet data plane layering model. 634 One may compare it to that in [IEEE802.1CB], Annex C. 636 DetNet data plane protocol stack 638 | packets going | ^ packets coming ^ 639 v down the stack v | up the stack | 640 +----------------------+ +-----------------------+ 641 | Source | | Destination | 642 +----------------------+ +-----------------------+ 643 | Service layer | | Service layer | 644 | Packet sequencing | | Duplicate elimination | 645 | Flow duplication | | Flow merging | 646 | Packet encoding | | Packet decoding | 647 +----------------------+ +-----------------------+ 648 | Transport layer | | Transport layer | 649 | Congestion prot. | | Congestion prot. | 650 +----------------------+ +-----------------------+ 651 | Lower layers | | Lower layers | 652 +----------------------+ +-----------------------+ 653 v ^ 654 \_________________________/ 656 Figure 2 658 Not all layers are required for any given application, or even for 659 any given network. The layers are, from top to bottom: 661 Application 662 Shown as "source" and "destination" in the diagram. 664 OAM 665 Operations, Administration, and Maintenance leverages in-band 666 and out-of-and signaling that validates whether the service 667 is effectively obtained within QoS constraints. OAM is not 668 shown in Figure 2; it may reside in any number of the layers. 669 OAM can involve specific tagging added in the packets for 670 tracing implementation or network configuration errors; 671 traceability enables to find whether a packet is a replica, 672 which relay node performed the replication, and which segment 673 was intended for the replica. 675 Packet sequencing 676 As part of DetNet service protection, supplies the sequence 677 number for packet replication and elimination 678 (Section 3.2.4). Peers with Duplicate elimination. This 679 layer is not needed if a higher-layer transport protocol is 680 expected to perform any packet sequencing and duplicate 681 elimination required by the DetNet flow duplication. 683 Duplicate elimination 684 As part of the DetNet service layer, based on the sequenced 685 number supplied by its peer, packet sequencing, Duplicate 686 elimination discards any duplicate packets generated by 687 DetNet flow duplication. It can operate on member flows, 688 compound flows, or both. The duplication may also be 689 inferred from other information such as the precise time of 690 reception in a scheduled network. The duplicate elimination 691 layer may also perform resequencing of packets to restore 692 packet order in a flow that was disrupted by the loss of 693 packets on one or another of the multiple paths taken. 695 Flow duplication 696 As part of DetNet service protection, packets that belong to 697 a DetNet compound flow are replicated into two or more DetNet 698 member flows. This function is separate from packet 699 sequencing. Flow duplication can be an explicit duplication 700 and remarking of packets, or can be performed by, for 701 example, techniques similar to ordinary multicast 702 replication. Peers with DetNet flow merging. 704 Network flow merging 705 As part of DetNet service protection, merges DetNet member 706 flows together for packets coming up the stack belonging to a 707 specific DetNet compound flow. Peers with DetNet flow 708 duplication. DetNet flow merging, together with packet 709 sequencing, duplicate elimination, and DetNet flow 710 duplication, performs packet replication and elimination 711 (Section 3.2.4). 713 Packet encoding 714 As part of DetNet service protection, as an alternative to 715 packet sequencing and flow duplication, packet encoding 716 combines the information in multiple DetNet packets, perhaps 717 from different DetNet compound flows, and transmits that 718 information in packets on different DetNet member Flows. 719 Peers with Packet decoding. 721 Packet decoding 722 As part of DetNet service protection, as an alternative to 723 flow merging and duplicate elimination, packet decoding takes 724 packets from different DetNet member flows, and computes from 725 those packets the original DetNet packets from the compound 726 flows input to packet encoding. Peers with Packet encoding. 728 Congestion protection 729 The DetNet transport layer provides congestion protection. 730 See Section 4.5. The actual queuing and shaping mechanisms 731 are typically provided by underlying subnet layers, but since 732 these are can be closely associated with the means of 733 providing paths for DetNet flows (e.g. MPLS LSPs or {VLAN, 734 multicast destination MAC address} pairs), the path and the 735 congestion protection are conflated in this figure. 737 The packet sequencing and duplication elimination functions at the 738 source and destination ends of a DetNet compound flow may be 739 performed either in the end system or in a DetNet edge node. The 740 reader must not confuse a DetNet edge function with other kinds of 741 edge functions, e.g. an Label Edge Router, although the two functions 742 may be performed together. The DetNet edge function is concerned 743 with sequencing packets belonging to DetNet flows. The LER with 744 encapsulating/decapsulating packets for transport, and is considered 745 part of the network underlying the DetNet transport layer. 747 4.1.2. DetNet Data Plane Overview 749 A "Deterministic Network" will be composed of DetNet enabled nodes 750 i.e., End Systems, Edge Nodes, Relay Nodes and collectively deliver 751 DetNet services. DetNet enabled nodes are interconnected via Transit 752 Nodes (i.e., routers) which support DetNet, but are not DetNet 753 service aware. Transit nodes see DetNet nodes as end points. All 754 DetNet enabled nodes are connect to sub-networks, where a point-to- 755 point link is also considered as a simple sub-network. These sub- 756 networks will provide DetNet compatible service for support of DetNet 757 traffic. Examples of sub-networks include IEEE 802.1 TSN and OTN. 758 Of course, multi-layer DetNet systems may also be possible, where one 759 DetNet appears as a sub-network, and provides service to, a higher 760 layer DetNet system. A simple DetNet concept network is shown in 761 Figure 3. 763 TSN Edge Transit Relay DetNet 764 End System Node Node Node End System 766 +---------+ +.........+ +---------+ 767 | Appl. |<---:Svc Proxy:-- End to End Service ---------->| Appl. | 768 +---------+ +---------+ +---------+ +---------+ 769 | TSN | |TSN| |Svc|<-- DetNet flow ---: Service :-->| Service | 770 +---------+ +---+ +---+ +---------+ +---------+ +---------+ 771 |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| 772 +-------.-+ +-.-+ +-.-+ +--.----.-+ +-.-+ +-.-+ +---.-----+ 773 : Link : / ,-----. \ : Link : / ,-----. \ 774 +........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ 775 [Network] [Network] 776 `-----' `-----' 778 Figure 3: A Simple DetNet Enabled Network 780 Distinguishing the function of these two DetNet data plane layers, 781 the DetNet service layer and the DetNet transport layer, helps to 782 explore and evaluate various combinations of the data plane solutions 783 available. This separation of DetNet layers, while helpful, should 784 not be considered as formal requirement. For example, some 785 technologies may violate these strict layers and still be able to 786 deliver a DetNet service. 788 . 789 . 790 +-----------+ 791 | Service | PW, RTP/(UDP), GRE 792 +-----------+ 793 | Transport | (UDP)/IPv6, (UDP)/IPv4, MPLS LSPs, BIER 794 +-----------+ 795 . 796 . 798 Figure 4: DetNet adaptation to data plane 800 In some networking scenarios, the end system initially provides a 801 DetNet flow encapsulation, which contains all information needed by 802 DetNet nodes (e.g., Real-time Transport Protocol (RTP) [RFC3550] 803 based DetNet flow transported over a native UDP/IP network or 804 PseudoWire). In other scenarios, the encapsulation formats might 805 differ significantly. As an example, a CPRI "application's" I/Q data 806 mapped directly to Ethernet frames may have to be transported over an 807 MPLS-based packet switched network (PSN). 809 There are many valid options to create a data plane solution for 810 DetNet traffic by selecting a technology approach for the DetNet 811 service layer and also selecting a technology approach for the DetNet 812 transport layer. There are a high number of valid combinations. 814 One of the most fundamental differences between different potential 815 data plane options is the basic addressing and headers used by DetNet 816 end systems. For example, is the basic service a Layer 2 (e.g., 817 Ethernet) or Layer 3 (i.e., IP) service. This decision impacts how 818 DetNet end systems are addressed, and the basic forwarding logic for 819 the DetNet service layer. 821 4.1.3. Network reference model 823 The figure below shows another view of the DetNet service related 824 reference points and main components (Figure 5). 826 DetNet DetNet 827 end system end system 828 _ _ 829 / \ +----DetNet-UNI (U) / \ 830 /App\ | /App\ 831 /-----\ | /-----\ 832 | NIC | v ________ | NIC | 833 +--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+ 834 | / \__/ \ | | 835 | / +----+ +----+ \_____ | | 836 | / | | | | \_______ | | 837 +------U PE +----+ P +----+ \ _ v | 838 | | | | | | | ___/ \ | 839 | +--+-+ +----+ | +----+ | / \_ | 840 \ | | | | | / \ | 841 \ | +----+ +--+-+ +--+PE |-------- U------+ 842 \ | | | | | | | | | \_ _/ 843 \ +---+ P +----+ P +--+ +----+ | \____/ 844 \___ | | | | / 845 \ +----+__ +----+ DetNet-1 DetNet-2 846 | \_____/ \___________/ | 847 | | 848 | | End-to-End-Service | | | | 849 <----------------------------------------------------------------> 850 | | DetNet-Service | | | | 851 | <--------------------------------------------------> | 852 | | | | | | 854 Figure 5: DetNet Service Reference Model (multi-domain) 856 DetNet-UNIs ("U" in Figure 5) are assumed in this document to be 857 packet-based reference points and provide connectivity over the 858 packet network. A DetNet-UNI may provide multiple functions, e.g., 859 it may add networking technology specific encapsulation to the DetNet 860 flows if necessary; it may provide status of the availability of the 861 connection associated to a reservation; it may provide a 862 synchronization service for the end system; it may carry enough 863 signaling to place the reservation in a network without a controller, 864 or if the controller only deals with the network but not the end 865 points. Internal reference points of end systems (between the 866 application and the NIC) are more challenging from control 867 perspective and they may have extra requirements (e.g., in-order 868 delivery is expected in end system internal reference points, whereas 869 it is considered optional over the DetNet-UNI), therefore not covered 870 in this document. 872 4.2. DetNet systems 874 4.2.1. End system 876 The native data flow between the source/destination end systems is 877 referred to as application-flow (App-flow). The traffic 878 characteristics of an App-flow can be CBR (constant bit rate) or VBR 879 (variable bit rate) and can have L1 or L2 or L3 encapsulation (e.g., 880 TDM (time-division multiplexing), Ethernet, IP). These 881 characteristics are considered as input for resource reservation and 882 might be simplified to ensure determinism during transport (e.g., 883 making reservations for the peak rate of VBR traffic, etc.). 885 An end system may or may not be DetNet transport layer aware or 886 DetNet service layer aware. That is, an end system may or may not 887 contain DetNet specific functionality. End systems with DetNet 888 functionalities may have the same or different transport layer as the 889 connected DetNet domain. Grouping of end systems are shown in 890 Figure 6. 892 End system 893 | 894 | 895 | DetNet aware ? 896 / \ 897 +------< >------+ 898 NO | \ / | YES 899 | v | 900 DetNet unaware | 901 End system | 902 | Service/ 903 | Transport 904 / \ aware ? 905 +--------< >-------------+ 906 t-aware | \ / | s-aware 907 | v | 908 | | both | 909 | | | 910 DetNet t-aware | DetNet s-aware 911 End system | End system 912 v 913 DetNet st-aware 914 End system 916 Figure 6: Grouping of end systems 918 Note some known use cases for end systems: 920 o DetNet unaware: The classic case requiring network proxies. 922 o DetNet t-aware: An extant TSN system. It knows about some TSN 923 functions (e.g., reservation), but not about replication/ 924 elimination. 926 o DetNet s-aware: An extant IEC 62439-3 system. It supplies 927 sequence numbers, but doesn't know about zero congestion loss. 929 o DetNet st-aware: A full functioning DetNet end station, it has 930 DetNet functionalities and usually the same forwarding paradigm as 931 the connected DetNet domain. It can be treated as an integral 932 part of the DetNet domain . 934 4.2.2. DetNet edge, relay, and transit nodes 936 As shown in Figure 3, DetNet edge nodes providing proxy service and 937 DetNet relay nodes providing the DetNet service layer are DetNet- 938 aware, and DetNet transit nodes need only be aware of the DetNet 939 transport layer. 941 In general, if a DetNet flow passes through one or more DetNet- 942 unaware network node between two DetNet nodes providing the DetNet 943 transport layer for that flow, there is a potential for disruption or 944 failure of the DetNet QoS. A network administrator needs to ensure 945 that the DetNet-unaware network nodes are configured to minimize the 946 chances of packet loss and delay, and provision enough exra buffer 947 space in the DetNet transit node following the DetNet-unaware network 948 nodes to absorb the induced latency variations. 950 4.3. DetNet flows 952 4.3.1. DetNet flow types 954 A DetNet flow can have different formats during while it is 955 transported between the peer end systems. Therefore, the following 956 possible types / formats of a DetNet flow are distinguished in this 957 document: 959 o App-flow: native format of a DetNet flow. It does not contain any 960 DetNet related attributes. 962 o DetNet-t-flow: specific format of a DetNet flow. Only requires 963 the congestion / latency features provided by the Detnet transport 964 layer. 966 o DetNet-s-flow: specific format of a DetNet flow. Only requires 967 the replication/elimination feature ensured by the DetNet service 968 layer. 970 o DetNet-st-flow: specific format of a DetNet flow. It requires 971 both DetNet service layer and DetNet transport layer functions 972 during forwarding. 974 4.3.2. Source guarantees 976 For the purposes of congestion protection, DetNet flows can be 977 synchronous or asynchronous. In synchronous DetNet flows, at least 978 the intermediate nodes (and possibly the end systems) are closely 979 time synchronized, typically to better than 1 microsecond. By 980 transmitting packets from different DetNet flows or classes of DetNet 981 flows at different times, using repeating schedules synchronized 982 among the intermediate nodes, resources such as buffers and link 983 bandwidth can be shared over the time domain among different DetNet 984 flows. There is a tradeoff among techniques for synchronous DetNet 985 flows between the burden of fine-grained scheduling and the benefit 986 of reducing the required resources, especially buffer space. 988 In contrast, asynchronous DetNet flows are not coordinated with a 989 fine-grained schedule, so relay and end systems must assume worst- 990 case interference among DetNet flows contending for buffer resources. 991 Asynchronous DetNet flows are characterized by: 993 o A maximum packet size; 995 o An observation interval; and 997 o A maximum number of transmissions during that observation 998 interval. 1000 These parameters, together with knowledge of the protocol stack used 1001 (and thus the size of the various headers added to a packet), limit 1002 the number of bit times per observation interval that the DetNet flow 1003 can occupy the physical medium. 1005 The source promises that these limits will not be exceeded. If the 1006 source transmits less data than this limit allows, the unused 1007 resources such as link bandwidth can be made available by the system 1008 to non-DetNet packets. However, making those resources available to 1009 DetNet packets in other DetNet flows would serve no purpose. Those 1010 other DetNet flows have their own dedicated resources, on the 1011 assumption that all DetNet flows can use all of their resources over 1012 a long period of time. 1014 There is no provision in DetNet for throttling DetNet flows (reducing 1015 the transmission rate via feedback); the assumption is that a DetNet 1016 flow, to be useful, must be delivered in its entirety. That is, 1017 while any useful application is written to expect a certain number of 1018 lost packets, the real-time applications of interest to DetNet demand 1019 that the loss of data due to the network is extraordinarily 1020 infrequent. 1022 Although DetNet strives to minimize the changes required of an 1023 application to allow it to shift from a special-purpose digital 1024 network to an Internet Protocol network, one fundamental shift in the 1025 behavior of network applications is impossible to avoid: the 1026 reservation of resources before the application starts. In the first 1027 place, a network cannot deliver finite latency and practically zero 1028 packet loss to an arbitrarily high offered load. Secondly, achieving 1029 practically zero packet loss for unthrottled (though bandwidth 1030 limited) DetNet flows means that bridges and routers have to dedicate 1031 buffer resources to specific DetNet flows or to classes of DetNet 1032 flows. The requirements of each reservation have to be translated 1033 into the parameters that control each system's queuing, shaping, and 1034 scheduling functions and delivered to the hosts, bridges, and 1035 routers. 1037 4.3.3. Incomplete Networks 1039 The presence in the network of transit nodes or subnets that are not 1040 fully capable of offering DetNet services complicates the ability of 1041 the intermediate nodes and/or controller to allocate resources, as 1042 extra buffering, and thus extra latency, must be allocated at points 1043 downstream from the non-DetNet intermediate node for a DetNet flow. 1045 4.4. Traffic Engineering for DetNet 1047 Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines 1048 traffic-engineering architectures for generic applicability across 1049 packet and non-packet networks. From TEAS perspective, Traffic 1050 Engineering (TE) refers to techniques that enable operators to 1051 control how specific traffic flows are treated within their networks. 1053 Because if its very nature of establishing explicit optimized paths, 1054 Deterministic Networking can be seen as a new, specialized branch of 1055 Traffic Engineering, and inherits its architecture with a separation 1056 into planes. 1058 The Deterministic Networking architecture is thus composed of three 1059 planes, a (User) Application Plane, a Controller Plane, and a Network 1060 Plane, which echoes that of Figure 1 of Software-Defined Networking 1061 (SDN): Layers and Architecture Terminology [RFC7426].: 1063 4.4.1. The Application Plane 1065 Per [RFC7426], the Application Plane includes both applications and 1066 services. In particular, the Application Plane incorporates the User 1067 Agent, a specialized application that interacts with the end user / 1068 operator and performs requests for Deterministic Networking services 1069 via an abstract Flow Management Entity, (FME) which may or may not be 1070 collocated with (one of) the end systems. 1072 At the Application Plane, a management interface enables the 1073 negotiation of flows between end systems. An abstraction of the flow 1074 called a Traffic Specification (TSpec) provides the representation. 1075 This abstraction is used to place a reservation over the (Northbound) 1076 Service Interface and within the Application plane. It is associated 1077 with an abstraction of location, such as IP addresses and DNS names, 1078 to identify the end systems and eventually specify intermediate 1079 nodes. 1081 4.4.2. The Controller Plane 1083 The Controller Plane corresponds to the aggregation of the Control 1084 and Management Planes in [RFC7426], though Common Control and 1085 Measurement Plane (CCAMP) [CCAMP] makes an additional distinction 1086 between management and measurement. When the logical separation of 1087 the Control, Measurement and other Management entities is not 1088 relevant, the term Controller Plane is used for simplicity to 1089 represent them all, and the term controller refers to any device 1090 operating in that plane, whether is it a Path Computation entity or a 1091 Network Management entity (NME). The Path Computation Element (PCE) 1092 [PCE] is a core element of a controller, in charge of computing 1093 Deterministic paths to be applied in the Network Plane. 1095 A (Northbound) Service Interface enables applications in the 1096 Application Plane to communicate with the entities in the Controller 1097 Plane. 1099 One or more PCE(s) collaborate to implement the requests from the FME 1100 as Per-Flow Per-Hop Behaviors installed in the intermediate nodes for 1101 each individual flow. The PCEs place each flow along a deterministic 1102 sequence of intermediate nodes so as to respect per-flow constraints 1103 such as security and latency, and optimize the overall result for 1104 metrics such as an abstract aggregated cost. The deterministic 1105 sequence can typically be more complex than a direct sequence and 1106 include redundancy path, with one or more packet replication and 1107 elimination points. 1109 4.4.3. The Network Plane 1111 The Network Plane represents the network devices and protocols as a 1112 whole, regardless of the Layer at which the network devices operate. 1113 It includes Forwarding Plane (data plane), Application, and 1114 Operational Plane (control plane) aspects. 1116 The network Plane comprises the Network Interface Cards (NIC) in the 1117 end systems, which are typically IP hosts, and intermediate nodes, 1118 which are typically IP routers and switches. Network-to-Network 1119 Interfaces such as used for Traffic Engineering path reservation in 1120 [RFC5921], as well as User-to-Network Interfaces (UNI) such as 1121 provided by the Local Management Interface (LMI) between network and 1122 end systems, are both part of the Network Plane, both in the control 1123 plane and the data plane. 1125 A Southbound (Network) Interface enables the entities in the 1126 Controller Plane to communicate with devices in the Network Plane. 1127 This interface leverages and extends TEAS to describe the physical 1128 topology and resources in the Network Plane. 1130 Flow Management Entity 1132 End End 1133 System System 1135 -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 1137 PCE PCE PCE PCE 1139 -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 1141 intermediate intermed. intermed. intermed. 1142 Node Node Node Node 1143 NIC NIC 1144 intermediate intermed. intermed. intermed. 1145 Node Node Node Node 1147 Figure 7 1149 The intermediate nodes (and eventually the end systems NIC) expose 1150 their capabilities and physical resources to the controller (the 1151 PCE), and update the PCE with their dynamic perception of the 1152 topology, across the Southbound Interface. In return, the PCE(s) set 1153 the per-flow paths up, providing a Flow Characterization that is more 1154 tightly coupled to the intermediate node Operation than a TSpec. 1156 At the Network plane, intermediate nodes may exchange information 1157 regarding the state of the paths, between adjacent systems and 1158 eventually with the end systems, and forward packets within 1159 constraints associated to each flow, or, when unable to do so, 1160 perform a last resort operation such as drop or declassify. 1162 This specification focuses on the Southbound interface and the 1163 operation of the Network Plane. 1165 4.5. Queuing, Shaping, Scheduling, and Preemption 1167 DetNet achieves congestion protection and bounded delivery latency by 1168 reserving bandwidth and buffer resources at every hop along the path 1169 of the DetNet flow. The reservation itself is not sufficient, 1170 however. Implementors and users of a number of proprietary and 1171 standard real-time networks have found that standards for specific 1172 data plane techniques are required to enable these assurances to be 1173 made in a multi-vendor network. The fundamental reason is that 1174 latency variation in one system results in the need for extra buffer 1175 space in the next-hop system(s), which in turn, increases the worst- 1176 case per-hop latency. 1178 Standard queuing and transmission selection algorithms allow a 1179 central controller to compute the latency contribution of each 1180 transit node to the end-to-end latency, to compute the amount of 1181 buffer space required in each transit node for each incremental 1182 DetNet flow, and most importantly, to translate from a flow 1183 specification to a set of values for the managed objects that control 1184 each relay or end system. The IEEE 802 has specified (and is 1185 specifying) a set of queuing, shaping, and scheduling algorithms that 1186 enable each transit node (bridge or router), and/or a central 1187 controller, to compute these values. These algorithms include: 1189 o A credit-based shaper [IEEE802.1Q-2014] Clause 34. 1191 o Time-gated queues governed by a rotating time schedule, 1192 synchronized among all transit nodes [IEEE802.1Qbv]. 1194 o Synchronized double (or triple) buffers driven by synchronized 1195 time ticks. [IEEE802.1Qch]. 1197 o Pre-emption of an Ethernet packet in transmission by a packet with 1198 a more stringent latency requirement, followed by the resumption 1199 of the preempted packet [IEEE802.1Qbu], [IEEE802.3br]. 1201 While these techniques are currently embedded in Ethernet and 1202 bridging standards, we can note that they are all, except perhaps for 1203 packet preemption, equally applicable to other media than Ethernet, 1204 and to routers as well as bridges. 1206 4.6. Service instance 1208 A Service instance represents all the functions required on a node to 1209 allow the end-to-end service between the UNIs. 1211 The DetNet network reference model is shown in Figure 8 for a DetNet- 1212 Service scenario (i.e. between two DetNet-UNIs). In this figure, the 1213 end systems ("A" and "B") are connected directly to the edge nodes of 1214 the IP/MPLS network ("PE1" and "PE2"). End-systems participating 1215 DetNet communication may require connectivity before setting up an 1216 App-flow that requires the DetNet service. Such a connectivity 1217 related service instance and the one dedicated for DetNet service 1218 share the same access. Packets belonging to a DetNet flow are 1219 selected by a filter configured on the access ("F1" and "F2"). As a 1220 result, data flow specific access ("access-A + F1" and "access-B + 1221 F2") are terminated in the flow specific service instance ("SI-1" and 1222 "SI-2"). A tunnel is used to provide connectivity between the 1223 service instances. 1225 The tunnel is used to transport exclusively the packets of the DetNet 1226 flow between "SI-1" and "SI-2". The service instances are configured 1227 to implement DetNet functions and a flow specific routing or bridging 1228 function depending on what connectivity the participating end systems 1229 require (L3 or L2). The service instance and the tunnel may or may 1230 not be shared by multiple DetNet flows. Sharing the service instance 1231 by multiple DetNet flows requires properly populated forwarding 1232 tables of the service instance. 1234 access-A access-B 1235 <-----> <---------- tunnel ----------> <-----> 1237 +---------+ ___ _ +---------+ 1238 End system | +----+ | / \/ \_ | +----+ | End system 1239 "A" -------F1+ | | / \ | | +F2----- "B" 1240 | | +==========+ IP/MPLS +========+ | | 1241 | |SI-1| | \__ Net._/ | |SI-2| | 1242 | +----+ | \____/ | +----+ | 1243 |PE1 | | PE2| 1244 +---------+ +---------+ 1246 Figure 8: DetNet network reference model 1248 The tunnel between the service instances may have some special 1249 characteristics. For example, in case of a "packet PW" based tunnel, 1250 there are differences in the usage of the packet PW for DetNet 1251 traffic compared to the network model described in [RFC6658]. In the 1252 DetNet scenario, the packet PW is used exclusively by the DetNet 1253 flow, whereas [RFC6658] states: "The packet PW appears as a single 1254 point-to-point link to the client layer. Network-layer adjacency 1255 formation and maintenance between the client equipments will follow 1256 the normal practice needed to support the required relationship in 1257 the client layer ... This packet pseudowire is used to transport all 1258 of the required layer 2 and layer 3 protocols between LSR1 and LSR2". 1260 4.7. Flow identification at technology borders 1262 4.7.1. Exporting flow identification 1264 An interesting feature of DetNet, and one that invites 1265 implementations that can be accused of "layering violations", is the 1266 need for lower layers to be aware of specific flows at higher layers, 1267 in order to provide specific queuing and shaping services for 1268 specific flows. For example: 1270 o A non-IP, strictly L2 source end system X may be sending multiple 1271 flows to the same L2 destination end system Y. Those flows may 1272 include DetNet flows with different QoS requirements, and may 1273 include non-DetNet flows. 1275 o A router may be sending any number of flows to another router. 1276 Again, those flows may include DetNet flows with different QoS 1277 requirements, and may include non-DetNet flows. 1279 o Two routers may be separated by bridges. For these bridges to 1280 perform any required per-flow queuing and shaping, they must be 1281 able to identify the individual flows. 1283 o A Label Edge Router (LERs) may have a Label Switched Path (LSP) 1284 set up for handling traffic destined for a particular IP address 1285 carrying only non-DetNet flows. If a DetNet flow to that same 1286 address is requested, a separate LSP may be needed, in order that 1287 all of the Label Switch Routers (LSRs) along the path to the 1288 destination give that flow special queuing and shaping. 1290 The need for a lower-level DetNet node to be aware of individual 1291 higher-layer flows is not unique to DetNet. But, given the endless 1292 complexity of layering and relayering over tunnels that is available 1293 to network designers, DetNet needs to provide a model for flow 1294 identification that is at least somewhat better than packet 1295 inspection. That is not to say that packet inspection to layer 4 or 1296 5 addresses will not be used, or the capability standardized; but, 1297 there are alternatives. 1299 A DetNet relay node can connect DetNet flows on different paths using 1300 different flow identification methods. For example: 1302 o A single unicast DetNet flow passing from router A through a 1303 bridged network to router B may be assigned a {VLAN, multicast 1304 destination MAC address} pair that is unique within that bridged 1305 network. The bridges can then identify the flow without accessing 1306 higher-layer headers. Of course, the receiving router must 1307 recognize and accept that multicast MAC address. 1309 o A DetNet flow passing from LSR A to LSR B may be assigned a 1310 different label than that used for other flows to the same IP 1311 destination. 1313 In any of the above cases, it is possible that an existing DetNet 1314 flow can be used as a carrier for multiple DetNet sub-flows. (Not to 1315 be confused with DetNet compound vs. member flows.) Of course, this 1316 requires that the aggregate DetNet flow be provisioned properly to 1317 carry the sub-flows. 1319 Thus, rather than packet inspection, there is the option to export 1320 higher-layer information to the lower layer. The requirement to 1321 support one or the other method for flow identification (or both) is 1322 the essential complexity that DetNet brings to existing control plane 1323 models. 1325 4.7.2. Flow attribute mapping between layers 1327 Transport of DetNet flows over multiple technology domains may 1328 require that lower layers are aware of specific flows of higher 1329 layers. Such an "exporting of flow identification" is needed each 1330 time when the forwarding paradigm is changed on the transport path 1331 (e.g., two LSRs are interconnected by a L2 bridged domain, etc.). 1332 The three main forwarding methods considered for deterministic 1333 networking are: 1335 o IP routing 1337 o MPLS label switching 1339 o Ethernet bridging 1341 add/remove add/remove 1342 Eth Flow-ID IP Flow-ID 1343 | | 1344 v v 1345 +-----------------------------------------------------------+ 1346 | | | | | 1347 | Eth | MPLS | IP | Application data | 1348 | | | | | 1349 +-----------------------------------------------------------+ 1350 ^ 1351 | 1352 add/remove 1353 MPLS Flow-ID 1355 Figure 9: Packet with multiple Flow-IDs 1357 The additional (domain specific) Flow-ID can be 1359 o created by a domain specific function or 1361 o derived from the Flow-ID added to the App-flow, 1363 so that it must be unique inside the given domain. Note that the 1364 Flow-ID added to the App-flow is still present in the packet, but 1365 transport nodes may lack the function to recognize it; that's why the 1366 additional Flow-ID is added (pushed). 1368 4.7.3. Flow-ID mapping examples 1370 IP nodes and MPLS nodes are assumed to be configured to push such an 1371 additional (domain specific) Flow-ID when sending traffic to an 1372 Ethernet switch (as shown in the examples below). 1374 Figure 10 shows a scenario where an IP end system ("IP-A") is 1375 connected via two Ethernet switches ("ETH-n") to an IP router ("IP- 1376 1"). 1378 IP domain 1379 <----------------------------------------------- 1381 +======+ +======+ 1382 |L3-ID | |L3-ID | 1383 +======+ /\ +-----+ +======+ 1384 / \ Forward as | | 1385 /IP-A\ per ETH-ID |IP-1 | Recognize 1386 Push ------> +-+----+ | +---+-+ <----- ETH-ID 1387 ETH-ID | +----+-----+ | 1388 | v v | 1389 | +-----+ +-----+ | 1390 +------+ | | +---------+ 1391 +......+ |ETH-1+----+ETH-2| +======+ 1392 .L3-ID . +-----+ +-----+ |L3-ID | 1393 +======+ +......+ +======+ 1394 |ETH-ID| .L3-ID . |ETH-ID| 1395 +======+ +======+ +------+ 1396 |ETH-ID| 1397 +======+ 1399 Ethernet domain 1400 <----------------> 1402 Figure 10: IP nodes interconnected by an Ethernet domain 1404 End system "IP-A" uses the original App-flow specific ID ("L3-ID"), 1405 but as it is connected to an Ethernet domain it has to push an 1406 Ethernet-domain specific flow-ID ("VID + multicast MAC address", 1407 referred as "ETH-ID") before sending the packet to "ETH-1" node. 1408 Ethernet switch "ETH-1" can recognize the data flow based on the 1409 "ETH-ID" and it does forwarding toward "ETH-2". "ETH-2" switches the 1410 packet toward the IP router. "IP-1" must be configured to receive 1411 the Ethernet Flow-ID specific multicast stream, but (as it is an L3 1412 node) it decodes the data flow ID based on the "L3-ID" fields of the 1413 received packet. 1415 Figure 11 shows a scenario where MPLS domain nodes ("PE-n" and "P-m") 1416 are connected via two Ethernet switches ("ETH-n"). 1418 MPLS domain 1419 <-----------------------------------------------> 1421 +=======+ +=======+ 1422 |MPLS-ID| |MPLS-ID| 1423 +=======+ +-----+ +-----+ +=======+ +-----+ 1424 | | Forward as | | | | 1425 |PE-1 | per ETH-ID | P-2 +-----------+ PE-2| 1426 Push -----> +-+---+ | +---+-+ +-----+ 1427 ETH-ID | +-----+----+ | \ Recognize 1428 | v v | +-- ETH-ID 1429 | +-----+ +-----+ | 1430 +---+ | | +----+ 1431 +.......+ |ETH-1+----+ETH-2| +=======+ 1432 .MPLS-ID. +-----+ +-----+ |MPLS-ID| 1433 +=======+ +=======+ 1434 |ETH-ID | +.......+ |ETH-ID | 1435 +=======+ .MPLS-ID. +-------+ 1436 +=======+ 1437 |ETH-ID | 1438 +=======+ 1439 Ethernet domain 1440 <----------------> 1442 Figure 11: MPLS nodes interconnected by an Ethernet domain 1444 "PE-1" uses the MPLS specific ID ("MPLS-ID"), but as it is connected 1445 to an Ethernet domain it has to push an Ethernet-domain specific 1446 flow-ID ("VID + multicast MAC address", referred as "ETH-ID") before 1447 sending the packet to "ETH-1". Ethernet switch "ETH-1" can recognize 1448 the data flow based on the "ETH-ID" and it does forwarding toward 1449 "ETH-2". "ETH-2" switches the packet toward the MPLS node ("P-2"). 1450 "P-2" must be configured to receive the Ethernet Flow-ID specific 1451 multicast stream, but (as it is an MPLS node) it decodes the data 1452 flow ID based on the "MPLS-ID" fields of the received packet. 1454 One can appreciate from the above example that, when the means used 1455 for DetNet flow identifcation is altered or exported, the means for 1456 encoding the sequence number information must similarly be altered or 1457 exported. 1459 4.8. Advertising resources, capabilities and adjacencies 1461 There are three classes of information that a central controller or 1462 decentralized control plane needs to know that can only be obtained 1463 from the end systems and/or transit nodes in the network. When using 1464 a peer-to-peer control plane, some of this information may be 1465 required by a system's neighbors in the network. 1467 o Details of the system's capabilities that are required in order to 1468 accurately allocate that system's resources, as well as other 1469 systems' resources. This includes, for example, which specific 1470 queuing and shaping algorithms are implemented (Section 4.5), the 1471 number of buffers dedicated for DetNet allocation, and the worst- 1472 case forwarding delay. 1474 o The dynamic state of an end or transit node's DetNet resources. 1476 o The identity of the system's neighbors, and the characteristics of 1477 the link(s) between the systems, including the length (in 1478 nanoseconds) of the link(s). 1480 4.9. Provisioning model 1482 4.9.1. Centralized Path Computation and Installation 1484 A centralized routing model, such as provided with a PCE (RFC 4655 1485 [RFC4655]), enables global and per-flow optimizations. (See 1486 Section 4.4.) The model is attractive but a number of issues are 1487 left to be solved. In particular: 1489 o Whether and how the path computation can be installed by 1) an end 1490 device or 2) a Network Management entity, 1492 o And how the path is set up, either by installing state at each hop 1493 with a direct interaction between the forwarding device and the 1494 PCE, or along a path by injecting a source-routed request at one 1495 end of the path. 1497 4.9.2. Distributed Path Setup 1499 Significant work on distributed path setup can be leveraged from MPLS 1500 Traffic Engineering, in both its GMPLS and non-GMPLS forms. The 1501 protocols within scope are Resource ReSerVation Protocol [RFC3209] 1502 [RFC3473](RSVP-TE), OSPF-TE [RFC4203] [RFC5392] and ISIS-TE [RFC5307] 1503 [RFC5316]. These should be viewed as starting points as there are 1504 feature specific extensions defined that may be applicable to DetNet. 1506 In a Layer-2 only environment, or as part of a layered approach to a 1507 mixed environment, IEEE 802.1 also has work, either completed or in 1508 progress. [IEEE802.1Q-2014] Clause 35 describes SRP, a peer-to-peer 1509 protocol for Layer-2 roughly analogous to RSVP [RFC2205]. 1510 [IEEE802.1Qca] defines how ISIS can provide multiple disjoint paths 1511 or distribution trees. Also in progress is [IEEE802.1Qcc], which 1512 expands the capabilities of SRP. 1514 The integration/interaction of the DetNet control layer with an 1515 underlying IEEE 802.1 sub-network control layer will need to be 1516 defined. 1518 4.10. Scaling to larger networks 1520 Reservations for individual DetNet flows require considerable state 1521 information in each transit node, especially when adequate fault 1522 mitigation (Section 3.3.2) is required. The DetNet data plane, in 1523 order to support larger numbers of DetNet flows, must support the 1524 aggregation of DetNet flows into tunnels, which themselves can be 1525 viewed by the transit nodes' data planes largely as individual DetNet 1526 flows. Without such aggregation, the per-relay system may limit the 1527 scale of DetNet networks. 1529 4.11. Connected islands vs. networks 1531 Given that users have deployed examples of the IEEE 802.1 TSN TG 1532 standards, which provide capabilities similar to DetNet, it is 1533 obvious to ask whether the IETF DetNet effort can be limited to 1534 providing Layer-2 connections (VPNs) between islands of bridged TSN 1535 networks. While this capability is certainly useful to some 1536 applications, and must not be precluded by DetNet, tunneling alone is 1537 not a sufficient goal for the DetNet WG. As shown in the 1538 Deterministic Networking Use Cases draft [I-D.ietf-detnet-use-cases], 1539 there are already deployments of Layer-2 TSN networks that are 1540 encountering the well-known problems of over-large broadcast domains. 1541 Routed solutions, and combinations routed/bridged solutions, are both 1542 required. 1544 4.12. Compatibility with Layer-2 1546 Standards providing similar capabilities for bridged networks (only) 1547 have been and are being generated in the IEEE 802 LAN/MAN Standards 1548 Committee. The present architecture describes an abstract model that 1549 can be applicable both at Layer-2 and Layer-3, and over links not 1550 defined by IEEE 802. It is the intention of the authors (and 1551 hopefully, as this draft progresses, of the DetNet Working Group) 1552 that IETF and IEEE 802 will coordinate their work, via the 1553 participation of common individuals, liaisons, and other means, to 1554 maximize the compatibility of their outputs. 1556 DetNet enabled end systems and intermediate nodes can be 1557 interconnected by sub-networks, i.e., Layer-2 technologies. These 1558 sub-networks will provide DetNet compatible service for support of 1559 DetNet traffic. Examples of sub-networks include 802.1TSN and a 1560 point-to-point OTN link. Of course, multi-layer DetNet systems may 1561 be possible too, where one DetNet appears as a sub-network, and 1562 provides service to, a higher layer DetNet system. 1564 5. Security Considerations 1566 Security in the context of Deterministic Networking has an added 1567 dimension; the time of delivery of a packet can be just as important 1568 as the contents of the packet, itself. A man-in-the-middle attack, 1569 for example, can impose, and then systematically adjust, additional 1570 delays into a link, and thus disrupt or subvert a real-time 1571 application without having to crack any encryption methods employed. 1572 See [RFC7384] for an exploration of this issue in a related context. 1574 Furthermore, in a control system where millions of dollars of 1575 equipment, or even human lives, can be lost if the DetNet QoS is not 1576 delivered, one must consider not only simple equipment failures, 1577 where the box or wire instantly becomes perfectly silent, but bizarre 1578 errors such as can be caused by software failures. Because there is 1579 essential no limit to the kinds of failures that can occur, 1580 protecting against realistic equipment failures is indistinguishable, 1581 in most cases, from protecting against malicious behavior, whether 1582 accidental or intentional. See also Section 3.3.2. 1584 Security must cover: 1586 o the protection of the signaling protocol 1588 o the authentication and authorization of the controlling systems 1590 o the identification and shaping of the DetNet flows 1592 6. Privacy Considerations 1594 DetNet is provides a Quality of Service (QoS), and as such, does not 1595 directly raise any new privacy considerations. 1597 However, the requirement for every (or almost every) node along the 1598 path of a DetNet flow to identify DetNet flows may present an 1599 additional attack surface for privacy, should the DetNet paradigm be 1600 found useful in broader environments. 1602 7. IANA Considerations 1604 This document does not require an action from IANA. 1606 8. Acknowledgements 1608 The authors wish to thank Jouni Korhonen, Erik Nordmark, George 1609 Swallow, Rudy Klecka, Anca Zamfir, David Black, Thomas Watteyne, 1610 Shitanshu Shah, Craig Gunther, Rodney Cummings, Balazs Varga, 1611 Wilfried Steiner, Marcel Kiessling, Karl Weber, Janos Farkas, Ethan 1612 Grossman, Pat Thaler, Lou Berger, and especially Michael Johas 1613 Teener, for their various contribution with this work. 1615 9. Access to IEEE 802.1 documents 1617 To access password protected IEEE 802.1 drafts, see the IETF IEEE 1618 802.1 information page at https://www.ietf.org/proceedings/52/slides/ 1619 bridge-0/tsld003.htm. 1621 10. Informative References 1623 [AVnu] http://www.avnu.org/, "The AVnu Alliance tests and 1624 certifies devices for interoperability, providing a simple 1625 and reliable networking solution for AV network 1626 implementation based on the Audio Video Bridging (AVB) 1627 standards.". 1629 [CCAMP] IETF, "Common Control and Measurement Plane", 1630 . 1632 [HSR-PRP] IEC, "High availability seamless redundancy (HSR) is a 1633 further development of the PRP approach, although HSR 1634 functions primarily as a protocol for creating media 1635 redundancy while PRP, as described in the previous 1636 section, creates network redundancy. PRP and HSR are both 1637 described in the IEC 62439 3 standard.", 1638 . 1641 [I-D.dt-detnet-dp-alt] 1642 Korhonen, J., Farkas, J., Mirsky, G., Thubert, P., 1643 Zhuangyan, Z., and L. Berger, "DetNet Data Plane Protocol 1644 and Solution Alternatives", draft-dt-detnet-dp-alt-04 1645 (work in progress), September 2016. 1647 [I-D.ietf-6tisch-architecture] 1648 Thubert, P., "An Architecture for IPv6 over the TSCH mode 1649 of IEEE 802.15.4", draft-ietf-6tisch-architecture-13 (work 1650 in progress), November 2017. 1652 [I-D.ietf-6tisch-tsch] 1653 Watteyne, T., Palattella, M., and L. Grieco, "Using 1654 IEEE802.15.4e TSCH in an IoT context: Overview, Problem 1655 Statement and Goals", draft-ietf-6tisch-tsch-06 (work in 1656 progress), March 2015. 1658 [I-D.ietf-detnet-problem-statement] 1659 Finn, N. and P. Thubert, "Deterministic Networking Problem 1660 Statement", draft-ietf-detnet-problem-statement-03 (work 1661 in progress), March 2018. 1663 [I-D.ietf-detnet-use-cases] 1664 Grossman, E., "Deterministic Networking Use Cases", draft- 1665 ietf-detnet-use-cases-15 (work in progress), April 2018. 1667 [I-D.ietf-roll-rpl-industrial-applicability] 1668 Phinney, T., Thubert, P., and R. Assimiti, "RPL 1669 applicability in industrial networks", draft-ietf-roll- 1670 rpl-industrial-applicability-02 (work in progress), 1671 October 2013. 1673 [I-D.svshah-tsvwg-deterministic-forwarding] 1674 Shah, S. and P. Thubert, "Deterministic Forwarding PHB", 1675 draft-svshah-tsvwg-deterministic-forwarding-04 (work in 1676 progress), August 2015. 1678 [I-D.varga-detnet-service-model] 1679 Varga, B. and J. Farkas, "DetNet Service Model", draft- 1680 varga-detnet-service-model-02 (work in progress), May 1681 2017. 1683 [IEEE802.1AS-2011] 1684 IEEE, "IEEE Std 802.1AS Timing and Synchronization for 1685 Time-Sensitive Applications in Bridged Local Area 1686 Networks", 2011, 1687 . 1689 [IEEE802.1BA-2011] 1690 IEEE, "IEEE Std 802.1BA Audio Video Bridging (AVB) 1691 Systems", 2011, 1692 . 1694 [IEEE802.1CB] 1695 IEEE, "IEEE Std 802.1CB Frame Replication and Elimination 1696 for Reliability", 2017, 1697 . 1699 [IEEE802.1Q-2014] 1700 IEEE, "IEEE Std 802.1Q Bridges and Bridged Networks", 1701 2014, . 1703 [IEEE802.1Qbu] 1704 IEEE, "IEEEE Std 802.1Qbu Bridges and Bridged Networks - 1705 Amendment 26: Frame Preemption", 2016, 1706 . 1708 [IEEE802.1Qbv] 1709 IEEE, "IEEEE Std 802.1Qbu Bridges and Bridged Networks - 1710 Amendment 25: Enhancements for Scheduled Traffic", 2015, 1711 . 1713 [IEEE802.1Qca] 1714 IEEE, "IEEE Std 802.1Qca Bridges and Bridged Networks - 1715 Amendment 24: Path Control and Reservation", June 2015, 1716 . 1718 [IEEE802.1Qcc] 1719 IEEE, "Stream Reservation Protocol (SRP) Enhancements and 1720 Performance Improvements (IEEE Draft P802.1Qcc)", 2017, 1721 . 1723 [IEEE802.1Qch] 1724 IEEE, "Cyclic Queuing and Forwarding (IEEE Draft 1725 P802.1Qch)", 2017, 1726 . 1728 [IEEE802.1TSNTG] 1729 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1730 Networks Task Group", 2013, 1731 . 1733 [IEEE802.3-2015] 1734 IEEE, "IEEE Std 802.3 Standard for Ethernet", 2015, 1735 . 1737 [IEEE802.3br] 1738 IEEE, "IEEE Std 802.3br Standard for Ethernet Amendment 5: 1739 Specification and Management Parameters for Interspersing 1740 Express Traffic", 2016, 1741 . 1743 [ISA95] ANSI/ISA, "Enterprise-Control System Integration Part 1: 1744 Models and Terminology", 2000, 1745 . 1747 [ODVA] http://www.odva.org/, "The organization that supports 1748 network technologies built on the Common Industrial 1749 Protocol (CIP) including EtherNet/IP.". 1751 [PCE] IETF, "Path Computation Element", 1752 . 1754 [Profinet] 1755 http://us.profinet.com/technology/profinet/, "PROFINET is 1756 a standard for industrial networking in automation.", 1757 . 1759 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1760 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1761 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1762 September 1997, . 1764 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1765 and W. Weiss, "An Architecture for Differentiated 1766 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1767 . 1769 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1770 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1771 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1772 . 1774 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 1775 Switching (GMPLS) Signaling Resource ReserVation Protocol- 1776 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 1777 DOI 10.17487/RFC3473, January 2003, 1778 . 1780 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1781 Jacobson, "RTP: A Transport Protocol for Real-Time 1782 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 1783 July 2003, . 1785 [RFC4203] Kompella, K., Ed. and Y. Rekhter, Ed., "OSPF Extensions in 1786 Support of Generalized Multi-Protocol Label Switching 1787 (GMPLS)", RFC 4203, DOI 10.17487/RFC4203, October 2005, 1788 . 1790 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 1791 Element (PCE)-Based Architecture", RFC 4655, 1792 DOI 10.17487/RFC4655, August 2006, 1793 . 1795 [RFC5307] Kompella, K., Ed. and Y. Rekhter, Ed., "IS-IS Extensions 1796 in Support of Generalized Multi-Protocol Label Switching 1797 (GMPLS)", RFC 5307, DOI 10.17487/RFC5307, October 2008, 1798 . 1800 [RFC5316] Chen, M., Zhang, R., and X. Duan, "ISIS Extensions in 1801 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1802 Traffic Engineering", RFC 5316, DOI 10.17487/RFC5316, 1803 December 2008, . 1805 [RFC5392] Chen, M., Zhang, R., and X. Duan, "OSPF Extensions in 1806 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1807 Traffic Engineering", RFC 5392, DOI 10.17487/RFC5392, 1808 January 2009, . 1810 [RFC5673] Pister, K., Ed., Thubert, P., Ed., Dwars, S., and T. 1811 Phinney, "Industrial Routing Requirements in Low-Power and 1812 Lossy Networks", RFC 5673, DOI 10.17487/RFC5673, October 1813 2009, . 1815 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 1816 L., and L. Berger, "A Framework for MPLS in Transport 1817 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 1818 . 1820 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 1821 Profile (MPLS-TP) Survivability Framework", RFC 6372, 1822 DOI 10.17487/RFC6372, September 2011, 1823 . 1825 [RFC6658] Bryant, S., Ed., Martini, L., Swallow, G., and A. Malis, 1826 "Packet Pseudowire Encapsulation over an MPLS PSN", 1827 RFC 6658, DOI 10.17487/RFC6658, July 2012, 1828 . 1830 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1831 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1832 October 2014, . 1834 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1835 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1836 Defined Networking (SDN): Layers and Architecture 1837 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1838 2015, . 1840 [TEAS] IETF, "Traffic Engineering Architecture and Signaling", 1841 . 1843 Authors' Addresses 1845 Norman Finn 1846 Huawei 1847 3755 Avocado Blvd. 1848 PMB 436 1849 La Mesa, California 91941 1850 US 1852 Phone: +1 925 980 6430 1853 Email: norman.finn@mail01.huawei.com 1855 Pascal Thubert 1856 Cisco Systems 1857 Village d'Entreprises Green Side 1858 400, Avenue de Roumanille 1859 Batiment T3 1860 Biot - Sophia Antipolis 06410 1861 FRANCE 1863 Phone: +33 4 97 23 26 34 1864 Email: pthubert@cisco.com 1866 Balazs Varga 1867 Ericsson 1868 Konyves Kalman krt. 11/B 1869 Budapest 1097 1870 Hungary 1872 Email: balazs.a.varga@ericsson.com 1873 Janos Farkas 1874 Ericsson 1875 Konyves Kalman krt. 11/B 1876 Budapest 1097 1877 Hungary 1879 Email: janos.farkas@ericsson.com