idnits 2.17.1 draft-ietf-detnet-architecture-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 732 has weird spacing: '...e stack v ...' == Line 1234 has weird spacing: '...mediate inter...' == Line 1237 has weird spacing: '...mediate inter...' -- The document date (August 3, 2018) is 2085 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Network' is mentioned on line 869, but not defined == Outdated reference: A later version (-30) exists of draft-ietf-6tisch-architecture-14 == Outdated reference: A later version (-02) exists of draft-ietf-detnet-dp-sol-ip-00 == Outdated reference: A later version (-02) exists of draft-ietf-detnet-dp-sol-mpls-00 == Outdated reference: A later version (-09) exists of draft-ietf-detnet-problem-statement-06 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-17 Summary: 0 errors (**), 0 flaws (~~), 10 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft Huawei 4 Intended status: Standards Track P. Thubert 5 Expires: February 4, 2019 Cisco 6 B. Varga 7 J. Farkas 8 Ericsson 9 August 3, 2018 11 Deterministic Networking Architecture 12 draft-ietf-detnet-architecture-07 14 Abstract 16 Deterministic Networking (DetNet) provides a capability to carry 17 specified unicast or multicast data flows for real-time applications 18 with extremely low data loss rates and bounded latency. Techniques 19 used include: 1) reserving data plane resources for individual (or 20 aggregated) DetNet flows in some or all of the intermediate nodes 21 (e.g., bridges or routers) along the path of the flow; 2) providing 22 explicit routes for DetNet flows that do not immediately change with 23 the network topology; and 3) distributing data from DetNet flow 24 packets over time and/or space to ensure delivery of each packet's 25 data in spite of the loss of a path. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at https://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on February 4, 2019. 44 Copyright Notice 46 Copyright (c) 2018 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (https://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 62 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 63 2.1. Terms used in this document . . . . . . . . . . . . . . . 4 64 2.2. IEEE 802.1 TSN to DetNet dictionary . . . . . . . . . . . 6 65 3. Providing the DetNet Quality of Service . . . . . . . . . . . 7 66 3.1. Primary goals defining the DetNet QoS . . . . . . . . . . 7 67 3.2. Mechanisms to achieve DetNet QoS . . . . . . . . . . . . 9 68 3.2.1. Congestion protection . . . . . . . . . . . . . . . . 9 69 3.2.1.1. Eliminate congestion loss . . . . . . . . . . . . 9 70 3.2.1.2. Jitter Reduction . . . . . . . . . . . . . . . . 10 71 3.2.2. Service Protection . . . . . . . . . . . . . . . . . 11 72 3.2.2.1. In-Order Delivery . . . . . . . . . . . . . . . . 11 73 3.2.2.2. Packet Replication and Elimination . . . . . . . 11 74 3.2.2.3. Packet encoding for service protection . . . . . 13 75 3.2.3. Explicit routes . . . . . . . . . . . . . . . . . . . 13 76 3.3. Secondary goals for DetNet . . . . . . . . . . . . . . . 14 77 3.3.1. Coexistence with normal traffic . . . . . . . . . . . 14 78 3.3.2. Fault Mitigation . . . . . . . . . . . . . . . . . . 15 79 4. DetNet Architecture . . . . . . . . . . . . . . . . . . . . . 16 80 4.1. DetNet stack model . . . . . . . . . . . . . . . . . . . 16 81 4.1.1. Representative Protocol Stack Model . . . . . . . . . 16 82 4.1.2. DetNet Data Plane Overview . . . . . . . . . . . . . 18 83 4.1.3. Network reference model . . . . . . . . . . . . . . . 20 84 4.2. DetNet systems . . . . . . . . . . . . . . . . . . . . . 21 85 4.2.1. End system . . . . . . . . . . . . . . . . . . . . . 21 86 4.2.2. DetNet edge, relay, and transit nodes . . . . . . . . 22 87 4.3. DetNet flows . . . . . . . . . . . . . . . . . . . . . . 23 88 4.3.1. DetNet flow types . . . . . . . . . . . . . . . . . . 23 89 4.3.2. Source transmission behavior . . . . . . . . . . . . 23 90 4.3.3. Incomplete Networks . . . . . . . . . . . . . . . . . 25 91 4.4. Traffic Engineering for DetNet . . . . . . . . . . . . . 25 92 4.4.1. The Application Plane . . . . . . . . . . . . . . . . 25 93 4.4.2. The Controller Plane . . . . . . . . . . . . . . . . 26 94 4.4.3. The Network Plane . . . . . . . . . . . . . . . . . . 26 95 4.5. Queuing, Shaping, Scheduling, and Preemption . . . . . . 27 96 4.6. Service instance . . . . . . . . . . . . . . . . . . . . 28 97 4.7. Flow identification at technology borders . . . . . . . . 30 98 4.7.1. Exporting flow identification . . . . . . . . . . . . 30 99 4.7.2. Flow attribute mapping between layers . . . . . . . . 31 100 4.7.3. Flow-ID mapping examples . . . . . . . . . . . . . . 32 101 4.8. Advertising resources, capabilities and adjacencies . . . 34 102 4.9. Scaling to larger networks . . . . . . . . . . . . . . . 35 103 4.10. Compatibility with Layer-2 . . . . . . . . . . . . . . . 35 104 5. Security Considerations . . . . . . . . . . . . . . . . . . . 35 105 6. Privacy Considerations . . . . . . . . . . . . . . . . . . . 36 106 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 36 107 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 36 108 9. Informative References . . . . . . . . . . . . . . . . . . . 37 109 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 40 111 1. Introduction 113 Deterministic Networking (DetNet) is a service that can be offered by 114 a network to DetNet flows. DetNet provides these flows with 115 extremely low packet loss rates and assured maximum end-to-end 116 delivery latency. This is accomplished by dedicating network 117 resources such as link bandwidth and buffer space to DetNet flows 118 and/or classes of DetNet flows, and by replicating packets along 119 multiple paths. Unused reserved resources are available to non- 120 DetNet packets. 122 The Deterministic Networking Problem Statement 123 [I-D.ietf-detnet-problem-statement] introduces Deterministic 124 Networking, and Deterministic Networking Use Cases 125 [I-D.ietf-detnet-use-cases] summarizes the need for it. See 126 [I-D.ietf-detnet-dp-sol-mpls] and [I-D.ietf-detnet-dp-sol-ip] for 127 specific techniques that can be used to identify DetNet flows and 128 assign them to specific paths through a network. 130 A goal of DetNet is a converged network in all respects. That is, 131 the presence of DetNet flows does not preclude non-DetNet flows, and 132 the benefits offered DetNet flows should not, except in extreme 133 cases, prevent existing QoS mechanisms from operating in a normal 134 fashion, subject to the bandwidth required for the DetNet flows. A 135 single source-destination pair can trade both DetNet and non-DetNet 136 flows. End systems and applications need not instantiate special 137 interfaces for DetNet flows. Networks are not restricted to certain 138 topologies; connectivity is not restricted. Any application that 139 generates a data flow that can be usefully characterized as having a 140 maximum bandwidth should be able to take advantage of DetNet, as long 141 as the necessary resources can be reserved. Reservations can be made 142 by the application itself, via network management, by an 143 application's controller, or by other means, e.g., a dynamic control 144 plane (e.g., [RFC2205]). 146 Many applications that are intended to be served by Deterministic 147 Networking require the ability to synchronize the clocks in end 148 systems to a sub-microsecond accuracy. Some of the queue control 149 techniques defined in Section 4.5 also require time synchronization 150 among relay and transit nodes. The means used to achieve time 151 synchronization are not addressed in this document. DetNet can 152 accommodate various time synchronization techniques and profiles that 153 are defined elsewhere to address the needs of different market 154 segments. 156 2. Terminology 158 2.1. Terms used in this document 160 The following terms are used in the context of DetNet in this 161 document: 163 allocation 164 Resources are dedicated to support a DetNet flow. Depending 165 on an implementation, the resource may be reused by non- 166 DetNet flows when it is not used by the DetNet flow. 168 App-flow 169 The native format of a DetNet flow. 171 DetNet destination 172 An end system capable of terminating a DetNet flow. 174 DetNet domain 175 The portion of a network that is DetNet aware. It includes 176 end systems and other DetNet nodes. 178 DetNet flow 179 A DetNet flow is a sequence of packets to which the DetNet 180 service is to be provided. 182 DetNet compound flow and DetNet member flow 183 A DetNet compound flow is a DetNet flow that has been 184 separated into multiple duplicate DetNet member flows for 185 service protection at the DetNet service layer. Member flows 186 are merged back into a single DetNet compound flow such that 187 there are no duplicate packets. "Compound" and "member" are 188 strictly relative to each other, not absolutes; a DetNet 189 compound flow comprising multiple DetNet member flows can, in 190 turn, be a member of a higher-order compound. 192 DetNet intermediate node 193 A DetNet relay node or transit node. 195 DetNet edge node 196 An instance of a DetNet relay node that acts as a source and/ 197 or destination at the DetNet service layer. For example, it 198 can include a DetNet service layer proxy function for DetNet 199 service protection (e.g., the addition or removal of packet 200 sequencing information) for one or more end systems, or 201 starts or terminates congestion protection at the DetNet 202 transport layer, or aggregates DetNet services into new 203 DetNet flows. It is analogous to a Label Edge Router (LER) 204 or a Provider Edge (PE) router. 206 DetNet-UNI 207 User-to-Network Interface with DetNet specific 208 functionalities. It is a packet-based reference point and 209 may provide multiple functions like encapsulation, status, 210 synchronization, etc. 212 end system 213 Commonly called a "host" in IETF documents, and an "end 214 station" is IEEE 802 documents. End systems of interest to 215 this document are either sources or destinations of DetNet 216 flows. And end system may or may not be DetNet transport 217 layer aware or DetNet service layer aware. 219 link 220 A connection between two DetNet nodes. It may be composed of 221 a physical link or a sub-network technology that can provide 222 appropriate traffic delivery for DetNet flows. 224 DetNet system 225 A DetNet aware end system, transit node, or relay node. 226 "DetNet" may be omitted in some text. 228 DetNet relay node 229 A DetNet node including a service layer function that 230 interconnects different DetNet transport layer paths to 231 provide service protection. A DetNet relay node can be a 232 bridge, a router, a firewall, or any other system that 233 participates in the DetNet service layer. It typically 234 incorporates DetNet transport layer functions as well, in 235 which case it is collocated with a transit node. 237 PEF A Packet Elimination Function (PEF) eliminates duplicate 238 copies of packets to prevent excess packets flooding the 239 network or duplicate packets being sent out of the DetNet 240 domain. PEF can be implemented by an edge node, a relay 241 node, or an end system. 243 PRF A Packet Replication Function (PRF) replicates DetNet flow 244 packets and forwards them to one or more next hops in the 245 DetNet domain. The number of packet copies sent to each next 246 hop is a DetNet flow specific parameter at the node doing the 247 replication. PRF can be implemented by an edge node, a relay 248 node, or an end system. 250 PREOF Collective name for Packet Replication, Elimination, and 251 Ordering Functions. 253 POF A Packet Ordering Function (POF) re-orders packets within a 254 DetNet flow that are received out of order. This function 255 can be implemented by an edge node, a relay node, or an end 256 system. 258 reservation 259 The set of resources allocated between a source and one or 260 more destinations through transit nodes and subnets 261 associated with a DetNet flow, to provide the provisioned 262 DetNet service. 264 DetNet service layer 265 The layer at which A DetNet service, e.g., service protection 266 is provided. 268 DetNet service proxy 269 Maps between App-flows and DetNet flows. 271 DetNet source 272 An end system capable of originating a DetNet flow. 274 DetNet transit node 275 A node operating at the DetNet transport layer, that utilizes 276 link layer and/or network layer switching across multiple 277 links and/or sub-networks to provide paths for DetNet service 278 layer functions. Typically provides congestion protection 279 over those paths. An MPLS LSR is an example of a DetNet 280 transit node. 282 DetNet transport layer 283 The layer that optionally provides congestion protection for 284 DetNet flows over paths provided by the underlying network. 286 2.2. IEEE 802.1 TSN to DetNet dictionary 288 This section also serves as a dictionary for translating from the 289 terms used by the Time-Sensitive Networking (TSN) Task Group 290 [IEEE802.1TSNTG] of the IEEE 802.1 WG to those of the DetNet WG. 292 Listener 293 The IEEE 802.1 term for a destination of a DetNet flow. 295 relay system 296 The IEEE 802.1 term for a DetNet intermediate node. 298 Stream 299 The IEEE 802.1 term for a DetNet flow. 301 Talker 302 The IEEE 802.1 term for the source of a DetNet flow. 304 3. Providing the DetNet Quality of Service 306 3.1. Primary goals defining the DetNet QoS 308 The DetNet Quality of Service can be expressed in terms of: 310 o Minimum and maximum end-to-end latency from source to destination; 311 timely delivery, and bounded jitter (packet delay variation) 312 derived from these constraints. 314 o Packet loss ratio, under various assumptions as to the operational 315 states of the nodes and links. 317 o An upper bound on out-of-order packet delivery. It is worth 318 noting that some DetNet applications are unable to tolerate any 319 out-of-order delivery. 321 It is a distinction of DetNet that it is concerned solely with worst- 322 case values for the end-to-end latency, jitter, and misordering. 323 Average, mean, or typical values are of little interest, because they 324 do not affect the ability of a real-time system to perform its tasks. 325 In general, a trivial priority-based queuing scheme will give better 326 average latency to a data flow than DetNet, but of course, the worst- 327 case latency can be essentially unbounded. 329 Three techniques are used by DetNet to provide these qualities of 330 service: 332 o Congestion protection (Section 3.2.1). 334 o Service protection (Section 3.2.2). 336 o Explicit routes (Section 3.2.3). 338 Congestion protection operates by allocating resources along the path 339 of a DetNet flow, e.g., buffer space or link bandwidth. Congestion 340 protection greatly reduces, or even eliminates entirely, packet loss 341 due to output packet congestion within the network, but it can only 342 be supplied to a DetNet flow that is limited at the source to a 343 maximum packet size and transmission rate. Note that congestion 344 protection provided via congestion detection and notification is 345 explicitly excluded from consideration in DetNet, as it serves a 346 different set of applications. 348 Congestion protection addresses two of the DetNet QoS requirements: 349 latency and packet loss. Given that DetNet nodes have a finite 350 amount of buffer space, congestion protection necessarily results in 351 a maximum end-to-end latency. It also addresses the largest 352 contribution to packet loss, which is buffer congestion. 354 After congestion, the most important contributions to packet loss are 355 typically from random media errors and equipment failures. Service 356 protection is the name for the mechanisms used by DetNet to address 357 these losses. The mechanisms employed are constrained by the 358 requirement to meet the users' latency requirements. Packet 359 replication and elimination (Section 3.2.2) and packet encoding 360 (Section 3.2.2.3) are described in this document to provide service 361 protection; others may be found. For instance, packet encoding can 362 be used to provide service protection against random media errors, 363 packet replication and elimination can be used to provide service 364 protection against equipment failures. This mechanism distributes 365 the contents of DetNet flows over multiple paths in time and/or 366 space, so that the loss of some of the paths does need not cause the 367 loss of any packets. 369 The paths are typically (but not necessarily) explicit routes, so 370 that they do not normally suffer temporary interruptions caused by 371 the convergence of routing or bridging protocols. 373 These three techniques can be applied independently, giving eight 374 possible combinations, including none (no DetNet), although some 375 combinations are of wider utility than others. This separation keeps 376 the protocol stack coherent and maximizes interoperability with 377 existing and developing standards in this (IETF) and other Standards 378 Development Organizations. Some examples of typical expected 379 combinations: 381 o Explicit routes plus service protection are exactly the techniques 382 employed by seamless redundancy mechanisms applied on a ring 383 topology as described, e.g., in [IEC62439-3-2016]. In this 384 example, explicit routes are achieved by limiting the physical 385 topology of the network to a ring. Sequentialization, 386 replication, and duplicate elimination are facilitated by packet 387 tags added at the front or the end of Ethernet frames. [RFC8227] 388 provides another example in the context of MPLS. 390 o Congestion protection alone is offered by IEEE 802.1 Audio Video 391 bridging [IEEE802.1BA]. As long as the network suffers no 392 failures, zero congestion loss can be achieved through the use of 393 a reservation protocol (MSRP [IEEE802.1Q-2018]), shapers in every 394 bridge, and proper dimensioning. 396 o Using all three together gives maximum protection. 398 There are, of course, simpler methods available (and employed, today) 399 to achieve levels of latency and packet loss that are satisfactory 400 for many applications. Prioritization and over-provisioning is one 401 such technique. However, these methods generally work best in the 402 absence of any significant amount of non-critical traffic in the 403 network (if, indeed, such traffic is supported at all), or work only 404 if the critical traffic constitutes only a small portion of the 405 network's theoretical capacity, or work only if all systems are 406 functioning properly, or in the absence of actions by end systems 407 that disrupt the network's operations. 409 There are any number of methods in use, defined, or in progress for 410 accomplishing each of the above techniques. It is expected that this 411 DetNet Architecture will assist various vendors, users, and/or 412 "vertical" Standards Development Organizations (dedicated to a single 413 industry) to make selections among the available means of 414 implementing DetNet networks. 416 3.2. Mechanisms to achieve DetNet QoS 418 3.2.1. Congestion protection 420 3.2.1.1. Eliminate congestion loss 422 The primary means by which DetNet achieves its QoS assurances is to 423 reduce, or even completely eliminate, congestion within a node as a 424 cause of packet loss. Given that a DetNet flow cannot be throttled, 425 this can be achieved only by the provision of sufficient buffer 426 storage at each hop through the network to ensure that no packets are 427 dropped due to a lack of buffer storage. 429 Ensuring adequate buffering requires, in turn, that the source, and 430 every intermediate node along the path to the destination (or nearly 431 every node, see Section 4.3.3) be careful to regulate its output to 432 not exceed the data rate for any DetNet flow, except for brief 433 periods when making up for interfering traffic. Any packet sent 434 ahead of its time potentially adds to the number of buffers required 435 by the next hop and may thus exceed the resources allocated for a 436 particular DetNet flow. 438 The low-level mechanisms described in Section 4.5 provide the 439 necessary regulation of transmissions by an end system or 440 intermediate node to provide congestion protection. The allocation 441 of the bandwidth and buffers for a DetNet flow requires provisioning 442 A DetNet node may have other resources requiring allocation and/or 443 scheduling, that might otherwise be over-subscribed and trigger the 444 rejection of a reservation. 446 3.2.1.2. Jitter Reduction 448 A core objective of DetNet is to enable the convergence of sensitive 449 non-IP networks onto a common network infrastructure. This requires 450 the accurate emulation of currently deployed mission-specific 451 networks, which for example rely on point-to-point analog (e.g., 452 4-20mA modulation) and serial-digital cables (or buses) for highly 453 reliable, synchronized and jitter-free communications. While the 454 latency of analog transmissions is basically the speed of light, 455 legacy serial links are usually slow (in the order of Kbps) compared 456 to, say, GigE, and some latency is usually acceptable. What is not 457 acceptable is the introduction of excessive jitter, which may, for 458 instance, affect the stability of control systems. 460 Applications that are designed to operate on serial links usually do 461 not provide services to recover the jitter, because jitter simply 462 does not exist there. DetNet flows are generally expected to be 463 delivered in-order and the precise time of reception influences the 464 processes. In order to converge such existing applications, there is 465 a desire to emulate all properties of the serial cable, such as clock 466 transportation, perfect flow isolation and fixed latency. While 467 minimal jitter (in the form of specifying minimum, as well as 468 maximum, end-to-end latency) is supported by DetNet, there are 469 practical limitations on packet-based networks in this regard. In 470 general, users are encouraged to use, instead of, "do this when you 471 get the packet," a combination of: 473 o Sub-microsecond time synchronization among all source and 474 destination end systems, and 476 o Time-of-execution fields in the application packets. 478 Jitter reduction is provided by the mechanisms described in 479 Section 4.5 that also provide congestion protection. 481 3.2.2. Service Protection 483 Service protection aims to mitigate or eliminate packet loss due to 484 equipment failures, random media and/or memory faults. These types 485 of packet loss can be greatly reduced by spreading the data over 486 multiple disjoint forwarding paths. Various service protection 487 methods are described in [RFC6372], e.g., 1+1 linear protection. 488 This section describes the functional details of an additional method 489 in Section 3.2.2.2, which can be implemented as described in 490 Section 3.2.2.3 or as specified in [I-D.ietf-detnet-dp-sol-mpls] in 491 order to provide 1+n hitless protection. The appropriate service 492 protection mechanism depends on the scenario and the requirements. 494 3.2.2.1. In-Order Delivery 496 Out-of-order packet delivery can be a side effect of service 497 protection. Packets delivered out-of-order impact the amount of 498 buffering needed at the destination to properly process the received 499 data. Such packets also influence the jitter of a flow. The DetNet 500 service includes maximum allowed misordering as a constraint. Zero 501 misordering would be a valid service constraint to reflect that the 502 end system(s) of the flow cannot tolerate any out-of-order delivery. 503 DetNet Packet Ordering Functionality (POF) (Section 3.2.2.2) can be 504 used to provide in-order delivery. 506 3.2.2.2. Packet Replication and Elimination 508 This section describes a service protection method that sends copies 509 of the same packets over multiple paths. 511 The DetNet service layer includes the packet replication (PRF), the 512 packet elimination (PEF), and the packet ordering functionality (POF) 513 for use in DetNet edge, relay node, and end system packet processing. 514 Either of these functions can be enabled in a DetNet edge node, relay 515 node or end system. The collective name for all three functions is 516 PREOF. The packet replication and elimination service protection 517 method altogether involves four capabilities: 519 o Providing sequencing information to the packets of a DetNet 520 compound flow. This may be done by adding a sequence number or 521 time stamp as part of DetNet, or may be inherent in the packet, 522 e.g., in a transport protocol, or associated to other physical 523 properties such as the precise time (and radio channel) of 524 reception of the packet. This is typically done once, at or near 525 the source. 527 o The Packet Replication Function (PRF) replicates these packets 528 into multiple DetNet member flows and typically sends them along 529 multiple different paths to the destination(s), e.g., over the 530 explicit routes of Section 3.2.3. The location within a node, and 531 the mechanism used for the PRF is implementation specific. 533 o The Packet Elimination Function (PEF) eliminates duplicate packets 534 of a DetNet flow based on the sequencing information and a history 535 of received packets. The output of the PEF is always a single 536 packet. This may be done at any node along the path to save 537 network resources further downstream, in particular if multiple 538 Replication points exist. But the most common case is to perform 539 this operation at the very edge of the DetNet network, preferably 540 in or near the receiver. The location within a node, and 541 mechanism used for the PEF is implementation specific. 543 o The Packet Ordering Function (POF) uses the sequencing information 544 to re-order a DetNet flow's packets that are received out of 545 order. 547 The order in which a node applies PEF, POF, and PRF to a DetNet flow 548 is implementation specific. 550 Some service protection mechanisms rely on switching from one flow to 551 another when a failure of a flow is detected. Contrarily, packet 552 replication and elimination combines the DetNet member flows sent 553 along multiple different paths, and performs a packet-by-packet 554 selection of which to discard, e.g., based on sequencing information. 556 In the simplest case, this amounts to replicating each packet in a 557 source that has two interfaces, and conveying them through the 558 network, along separate (disjoint non-SRLG) paths, to the similarly 559 dual-homed destinations, that discard the extras. This ensures that 560 one path (with zero congestion loss) remains, even if some 561 intermediate node fails. The sequencing information can also be used 562 for loss detection and for re-ordering. 564 DetNet relay nodes in the network can provide replication and 565 elimination facilities at various points in the network, so that 566 multiple failures can be accommodated. 568 This is shown in Figure 1, where the two relay nodes each replicate 569 (R) the DetNet flow on input, sending the DetNet member flows to both 570 the other relay node and to the end system, and eliminate duplicates 571 (E) on the output interface to the right-hand end system. Any one 572 link in the network can fail, and the DetNet compound flow can still 573 get through. Furthermore, two links can fail, as long as they are in 574 different segments of the network. 576 > > > > > > > > > relay > > > > > > > > 577 > /------------+ R node E +------------\ > 578 > / v + ^ \ > 579 end R + v | ^ + E end 580 system + v | ^ + system 581 > \ v + ^ / > 582 > \------------+ R relay E +-----------/ > 583 > > > > > > > > > node > > > > > > > > 585 Figure 1: Packet replication and elimination 587 Packet replication and elimination does not react to and correct 588 failures; it is entirely passive. Thus, intermittent failures, 589 mistakenly created packet filters, or misrouted data is handled just 590 the same as the equipment failures that are handled by typical 591 routing and bridging protocols. 593 If packet replication and elimination is used over paths providing 594 congestion protection (Section 3.2.1), and member flows that take 595 different-length paths through the network are combined, a merge 596 point may require extra buffering to equalize the delays over the 597 different paths. This equalization ensures that the resultant 598 compound flow will not exceed its contracted bandwidth even after one 599 or the other of the paths is restored after a failure. The extra 600 buffering can be also used to provide in-order delivery. 602 3.2.2.3. Packet encoding for service protection 604 There are methods for using multiple paths to provide service 605 protection that involve encoding the information in a packet 606 belonging to a DetNet flow into multiple transmission units, 607 combining information from multiple packets into any given 608 transmission unit. Such techniques, also known as "network coding", 609 can be used as a DetNet service protection technique. 611 3.2.3. Explicit routes 613 In networks controlled by typical dynamic control protocols such as 614 IS-IS or OSPF, a network topology event in one part of the network 615 can impact, at least briefly, the delivery of data in parts of the 616 network remote from the failure or recovery event. Even the use of 617 redundant paths through a network defined, e.g., as defined by 618 [RFC6372] do not eliminate the chances of packet loss. Furthermore, 619 out-of-order packet delivery can be a side effect of route changes. 621 Many real-time networks rely on physical rings of two-port devices, 622 with a relatively simple ring control protocol. This supports 623 redundant paths for service protection with a minimum of wiring. As 624 an additional benefit, ring topologies can often utilize different 625 topology management protocols than those used for a mesh network, 626 with a consequent reduction in the response time to topology changes. 627 Of course, this comes at some cost in terms of increased hop count, 628 and thus latency, for the typical path. 630 In order to get the advantages of low hop count and still ensure 631 against even very brief losses of connectivity, DetNet employs 632 explicit routes, where the path taken by a given DetNet flow does not 633 change, at least immediately, and likely not at all, in response to 634 network topology events. Service protection (Section 3.2.2 or 635 Section 3.2.2.3) over explicit routes provides a high likelihood of 636 continuous connectivity. Explicit routes can be established in 637 various ways, e.g., with RSVP-TE [RFC3209], with Segment Routing (SR) 638 [RFC8402], via a Software Defined Networking approach [RFC7426], with 639 IS-IS [RFC7813], etc. Explicit routes are typically used in MPLS TE 640 LSPs. 642 Out-of-order packet delivery can be a side effect of distributing a 643 single flow over multiple paths especially when there is a change 644 from one path to another when combining the flow. This is 645 irrespective of the distribution method used, and also applies to 646 service protection over explicit routes. As described in 647 Section 3.2.2.1, out-of-order packets influence the jitter of a flow 648 and impact the amount of buffering needed to process the data; 649 therefore, DetNet service includes maximum allowed misordering as a 650 constraint. The use of explicit routes helps to provide in-order 651 delivery because there is no immediate route change with the network 652 topology, but the changes are plannable as they are between the 653 different explicit routes. 655 3.3. Secondary goals for DetNet 657 Many applications require DetNet to provide additional services, 658 including coexistence with other QoS mechanisms Section 3.3.1 and 659 protection against misbehaving transmitters Section 3.3.2. 661 3.3.1. Coexistence with normal traffic 663 A DetNet network supports the dedication of a high proportion (e.g. 664 75%) of the network bandwidth to DetNet flows. But, no matter how 665 much is dedicated for DetNet flows, it is a goal of DetNet to coexist 666 with existing Class of Service schemes (e.g., DiffServ). It is also 667 important that non-DetNet traffic not disrupt the DetNet flow, of 668 course (see Section 3.3.2 and Section 5). For these reasons: 670 o Bandwidth (transmission opportunities) not utilized by a DetNet 671 flow is available to non-DetNet packets (though not to other 672 DetNet flows). 674 o DetNet flows can be shaped or scheduled, in order to ensure that 675 the highest-priority non-DetNet packet is also ensured a worst- 676 case latency (at any given hop). 678 o When transmission opportunities for DetNet flows are scheduled in 679 detail, then the algorithm constructing the schedule should leave 680 sufficient opportunities for non-DetNet packets to satisfy the 681 needs of the users of the network. Detailed scheduling can also 682 permit the time-shared use of buffer resources by different DetNet 683 flows. 685 Ideally, the net effect of the presence of DetNet flows in a network 686 on the non-DetNet packets is primarily a reduction in the available 687 bandwidth. 689 3.3.2. Fault Mitigation 691 One key to building robust real-time systems is to reduce the 692 infinite variety of possible failures to a number that can be 693 analyzed with reasonable confidence. DetNet aids in the process by 694 allowing for filters and policers to detect DetNet packets received 695 on the wrong interface, or at the wrong time, or in too great a 696 volume, and to then take actions such as discarding the offending 697 packet, shutting down the offending DetNet flow, or shutting down the 698 offending interface. 700 It is also essential that filters and service remarking be employed 701 at the network edge to prevent non-DetNet packets from being mistaken 702 for DetNet packets, and thus impinging on the resources allocated to 703 DetNet packets. 705 There exist techniques, at present and/or in various stages of 706 standardization, that can perform these fault mitigation tasks that 707 deliver a high probability that misbehaving systems will have zero 708 impact on well-behaved DetNet flows, except of course, for the 709 receiving interface(s) immediately downstream of the misbehaving 710 device. Examples of such techniques include traffic policing 711 functions (e.g. [RFC2475]) and separating flows into per-flow rate- 712 limited queues. 714 4. DetNet Architecture 716 4.1. DetNet stack model 718 DetNet functionality (Section 3) is implemented in two adjacent 719 layers in the protocol stack: the DetNet service layer and the DetNet 720 transport layer. The DetNet service layer provides DetNet service, 721 e.g., service protection, to higher layers in the protocol stack and 722 applications. The DetNet transport layer supports DetNet service in 723 the underlying network, e.g., by providing explicit routes and 724 congestion protection to DetNet flows. 726 4.1.1. Representative Protocol Stack Model 728 Figure 2 illustrates a conceptual DetNet data plane layering model. 729 One may compare it to that in [IEEE802.1CB], Annex C. 731 | packets going | ^ packets coming ^ 732 v down the stack v | up the stack | 733 +----------------------+ +-----------------------+ 734 | Source | | Destination | 735 +----------------------+ +-----------------------+ 736 | Service layer: | | Service layer: | 737 | Packet sequencing | | Duplicate elimination | 738 | Flow replication | | Flow merging | 739 | Packet encoding | | Packet decoding | 740 +----------------------+ +-----------------------+ 741 | Transport layer: | | Transport layer: | 742 | Congestion prot. | | Congestion prot. | 743 | Explicit routes | | Explicit routes | 744 +----------------------+ +-----------------------+ 745 | Lower layers | | Lower layers | 746 +----------------------+ +-----------------------+ 747 v ^ 748 \_________________________/ 750 Figure 2: DetNet data plane protocol stack 752 Not all layers are required for any given application, or even for 753 any given network. The functionality shown in Figure 2 is: 755 Application 756 Shown as "source" and "destination" in the diagram. 758 Packet sequencing 759 As part of DetNet service protection, supplies the sequence 760 number for packet replication and elimination 761 (Section 3.2.2). Peers with Duplicate elimination. This 762 layer is not needed if a higher-layer transport protocol is 763 expected to perform any packet sequencing and duplicate 764 elimination required by the DetNet flow replication. 766 Duplicate elimination 767 As part of the DetNet service layer, based on the sequenced 768 number supplied by its peer, packet sequencing, Duplicate 769 elimination discards any duplicate packets generated by 770 DetNet flow replication. It can operate on member flows, 771 compound flows, or both. The replication may also be 772 inferred from other information such as the precise time of 773 reception in a scheduled network. The duplicate elimination 774 layer may also perform resequencing of packets to restore 775 packet order in a flow that was disrupted by the loss of 776 packets on one or another of the multiple paths taken. 778 Flow replication 779 As part of DetNet service protection, packets that belong to 780 a DetNet compound flow are replicated into two or more DetNet 781 member flows. This function is separate from packet 782 sequencing. Flow replication can be an explicit replication 783 and remarking of packets, or can be performed by, for 784 example, techniques similar to ordinary multicast 785 replication, albeit with resource allocation implications. 786 Peers with DetNet flow merging. 788 Flow merging 789 As part of DetNet service protection, merges DetNet member 790 flows together for packets coming up the stack belonging to a 791 specific DetNet compound flow. Peers with DetNet flow 792 replication. DetNet flow merging, together with packet 793 sequencing, duplicate elimination, and DetNet flow 794 replication perform packet replication and elimination 795 (Section 3.2.2). 797 Packet encoding 798 As part of DetNet service protection, as an alternative to 799 packet sequencing and flow replication, packet encoding 800 combines the information in multiple DetNet packets, perhaps 801 from different DetNet compound flows, and transmits that 802 information in packets on different DetNet member Flows. 803 Peers with Packet decoding. 805 Packet decoding 806 As part of DetNet service protection, as an alternative to 807 flow merging and duplicate elimination, packet decoding takes 808 packets from different DetNet member flows, and computes from 809 those packets the original DetNet packets from the compound 810 flows input to packet encoding. Peers with Packet encoding. 812 Congestion protection 813 The DetNet transport layer provides congestion protection. 814 See Section 4.5. The actual queuing and shaping mechanisms 815 are typically provided by underlying subnet layers, these can 816 be closely associated with the means of providing paths for 817 DetNet flows, the path and the congestion protection are 818 conflated in this figure. 820 Explicit routes 821 The DetNet transport layer provides mechanisms to ensure that 822 fixed paths are provided for DetNet flows. These explicit 823 paths avoid the impact of network convergence. 825 Operations, Administration, and Maintenance (OAM) leverages in-band 826 and out-of-band signaling that validates whether the service is 827 effectively obtained within QoS constraints. OAM is not shown in 828 Figure 2; it may reside in any number of the layers. OAM can involve 829 specific tagging added in the packets for tracing implementation or 830 network configuration errors; traceability enables to find whether a 831 packet is a replica, which relay node performed the replication, and 832 which segment was intended for the replica. Active and hybrid OAM 833 methods require additional bandwidth to perform fault management and 834 performance monitoring of the DetNet domain. OAM may, for instance, 835 generate special test probes or add OAM information into the data 836 packet. 838 The packet sequencing and replication elimination functions at the 839 source and destination ends of a DetNet compound flow may be 840 performed either in the end system or in a DetNet relay node. 842 4.1.2. DetNet Data Plane Overview 844 A "Deterministic Network" will be composed of DetNet enabled end 845 systems and nodes, i.e., edge nodes, relay nodes and collectively 846 deliver DetNet services. DetNet enabled nodes are interconnected via 847 transit nodes (e.g., LSRs) which support DetNet, but are not DetNet 848 service aware. All DetNet enabled nodes are connected to sub- 849 networks, where a point-to-point link is also considered as a simple 850 sub-network. These sub-networks will provide DetNet compatible 851 service for support of DetNet traffic. Examples of sub-networks 852 include MPLS TE, IEEE 802.1 TSN and OTN. Of course, multi-layer 853 DetNet systems may also be possible, where one DetNet appears as a 854 sub-network, and provides service to, a higher layer DetNet system. 855 A simple DetNet concept network is shown in Figure 3. 857 TSN Edge Transit Relay DetNet 858 End System Node Node Node End System 860 +---------+ +.........+ +---------+ 861 | Appl. |<--:Svc Proxy:-- End to End Service ---------->| Appl. | 862 +---------+ +---------+ +---------+ +---------+ 863 | TSN | |TSN| |Svc|<-- DetNet flow ---: Service :-->| Service | 864 +---------+ +---+ +---+ +---------+ +---------+ +---------+ 865 |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| 866 +-------.-+ +-.-+ +-.-+ +--.----.-+ +-.-+ +-.-+ +---.-----+ 867 : Link : / ,-----. \ : Link : / ,-----. \ 868 +.......+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ 869 [Network] [Network] 870 `-----' `-----' 872 Figure 3: A Simple DetNet Enabled Network 874 Distinguishing the function of two DetNet data plane layers, the 875 DetNet service layer and the DetNet transport layer, helps to explore 876 and evaluate various combinations of the data plane solutions 877 available, some are illustrated in Figure 4. This separation of 878 DetNet layers, while helpful, should not be considered as formal 879 requirement. For example, some technologies may violate these strict 880 layers and still be able to deliver a DetNet service. 882 . 883 . 884 +-----------+ 885 | Service | PW, UDP, GRE 886 +-----------+ 887 | Transport | IPv6, IPv4, MPLS TE LSPs, MPLS SR 888 +-----------+ 889 . 890 . 892 Figure 4: DetNet adaptation to data plane 894 In some networking scenarios, the end system initially provides a 895 DetNet flow encapsulation, which contains all information needed by 896 DetNet nodes (e.g., Real-time Transport Protocol (RTP) [RFC3550] 897 based DetNet flow transported over a native UDP/IP network or 898 PseudoWire). In other scenarios, the encapsulation formats might 899 differ significantly. 901 There are many valid options to create a data plane solution for 902 DetNet traffic by selecting a technology approach for the DetNet 903 service layer and also selecting a technology approach for the DetNet 904 transport layer. There are a high number of valid combinations. 906 One of the most fundamental differences between different potential 907 data plane options is the basic headers used by DetNet nodes. For 908 example, the basic service can be delivered based on an MPLS label or 909 an IP header. This decision impacts the basic forwarding logic for 910 the DetNet service layer. Note that in both cases, IP addresses are 911 used to address DetNet nodes. The selected DetNet transport layer 912 technology also needs to be mapped to the sub-net technology used to 913 interconnect DetNet nodes. For example, DetNet flows will need to be 914 mapped to TSN Streams. 916 4.1.3. Network reference model 918 Figure 5 shows another view of the DetNet service related reference 919 points and main components. 921 DetNet DetNet 922 end system end system 923 _ _ 924 / \ +----DetNet-UNI (U) / \ 925 /App\ | /App\ 926 /-----\ | /-----\ 927 | NIC | v ________ | NIC | 928 +--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+ 929 | / \__/ \ | | 930 | / +----+ +----+ \_____ | | 931 | / | | | | \_______ | | 932 +------U PE +----+ P +----+ \ _ v | 933 | | | | | | | ___/ \ | 934 | +--+-+ +----+ | +----+ | / \_ | 935 \ | | | | | / \ | 936 \ | +----+ +--+-+ +--+PE |------ U-----+ 937 \ | | | | | | | | | \_ _/ 938 \ +---+ P +----+ P +--+ +----+ | \____/ 939 \___ | | | | / 940 \ +----+__ +----+ DetNet-1 DetNet-2 941 | \_____/ \___________/ | 942 | | 943 | | End-to-End service | | | | 944 <-------------------------------------------------------------> 945 | | DetNet service | | | | 946 | <------------------------------------------------> | 947 | | | | | | 949 Figure 5: DetNet Service Reference Model (multi-domain) 951 DetNet-UNIs ("U" in Figure 5) are assumed in this document to be 952 packet-based reference points and provide connectivity over the 953 packet network. A DetNet-UNI may provide multiple functions, e.g., 954 it may add networking technology specific encapsulation to the DetNet 955 flows if necessary; it may provide status of the availability of the 956 resources associated with a reservation; it may provide a 957 synchronization service for the end system; it may carry enough 958 signaling to place the reservation in a network without a controller, 959 or if the controller only deals with the network but not the end 960 systems. Internal reference points of end systems (between the 961 application and the NIC) are more challenging from control 962 perspective and they may have extra requirements (e.g., in-order 963 delivery is expected in end system internal reference points, whereas 964 it is considered optional over the DetNet-UNI). 966 4.2. DetNet systems 968 4.2.1. End system 970 The native data flow between the source/destination end systems is 971 referred to as application-flow (App-flow). The traffic 972 characteristics of an App-flow can be CBR (constant bit rate) or VBR 973 (variable bit rate) and can have L1 or L2 or L3 encapsulation (e.g., 974 TDM (time-division multiplexing), Ethernet, IP). These 975 characteristics are considered as input for resource reservation and 976 might be simplified to ensure determinism during transport (e.g., 977 making reservations for the peak rate of VBR traffic, etc.). 979 An end system may or may not be DetNet transport layer aware or 980 DetNet service layer aware. That is, an end system may or may not 981 contain DetNet specific functionality. End systems with DetNet 982 functionalities may have the same or different transport layer as the 983 connected DetNet domain. Categorization of end systems are shown in 984 Figure 6. 986 End system 987 | 988 | 989 | DetNet aware ? 990 / \ 991 +------< >------+ 992 NO | \ / | YES 993 | v | 994 DetNet unaware | 995 End system | 996 | Service/ 997 | Transport 998 / \ aware ? 999 +--------< >-------------+ 1000 t-aware | \ / | s-aware 1001 | v | 1002 | | both | 1003 | | | 1004 DetNet t-aware | DetNet s-aware 1005 End system | End system 1006 v 1007 DetNet st-aware 1008 End system 1010 Figure 6: Categorization of end systems 1012 Note some known use case examples for end systems: 1014 o DetNet unaware: The classic case requiring service proxies. 1016 o DetNet t-aware: A DetNet transport-aware system. It knows about 1017 some TSN functions (e.g., reservation), but not about service 1018 protection. 1020 o DetNet s-aware: A DetNet service-aware system. It supplies 1021 sequence numbers, but doesn't know about zero congestion loss. 1023 o DetNet st-aware: A full functioning DetNet end system, it has 1024 DetNet functionalities and usually the same forwarding paradigm as 1025 the connected DetNet domain. It can be treated as an integral 1026 part of the DetNet domain. 1028 4.2.2. DetNet edge, relay, and transit nodes 1030 As shown in Figure 3, DetNet edge nodes providing proxy service and 1031 DetNet relay nodes providing the DetNet service layer are DetNet- 1032 aware, and DetNet transit nodes need only be aware of the DetNet 1033 transport layer. 1035 In general, if a DetNet flow passes through one or more DetNet- 1036 unaware network nodes between two DetNet nodes providing the DetNet 1037 transport layer for that flow, there is a potential for disruption or 1038 failure of the DetNet QoS. A network administrator needs to ensure 1039 that the DetNet-unaware network nodes are configured to minimize the 1040 chances of packet loss and delay, and provision enough extra buffer 1041 space in the DetNet transit node following the DetNet-unaware network 1042 nodes to absorb the induced latency variations. 1044 4.3. DetNet flows 1046 4.3.1. DetNet flow types 1048 A DetNet flow can have different formats while it is transported 1049 between the peer end systems. Therefore, the following possible 1050 types / formats of a DetNet flow are distinguished in this document: 1052 o App-flow: native format of the data carried over a DetNet flow. 1053 It does not contain any DetNet related attributes. 1055 o DetNet-t-flow: specific format of a DetNet flow. Only requires 1056 the congestion / latency features provided by the DetNet transport 1057 layer. 1059 o DetNet-s-flow: specific format of a DetNet flow. Only requires 1060 the service protection feature ensured by the DetNet service 1061 layer. 1063 o DetNet-st-flow: specific format of a DetNet flow. It requires 1064 both DetNet service layer and DetNet transport layer functions 1065 during forwarding. 1067 4.3.2. Source transmission behavior 1069 For the purposes of congestion protection, DetNet flows can be 1070 synchronous or asynchronous. In synchronous DetNet flows, at least 1071 the intermediate nodes (and possibly the end systems) are closely 1072 time synchronized, typically to better than 1 microsecond. By 1073 transmitting packets from different DetNet flows or classes of DetNet 1074 flows at different times, using repeating schedules synchronized 1075 among the intermediate nodes, resources such as buffers and link 1076 bandwidth can be shared over the time domain among different DetNet 1077 flows. There is a tradeoff among techniques for synchronous DetNet 1078 flows between the burden of fine-grained scheduling and the benefit 1079 of reducing the required resources, especially buffer space. 1081 In contrast, asynchronous DetNet flows are not coordinated with a 1082 fine-grained schedule, so relay and end systems must assume worst- 1083 case interference among DetNet flows contending for buffer resources. 1084 Asynchronous DetNet flows are characterized by: 1086 o A maximum packet size; 1088 o An observation interval; and 1090 o A maximum number of transmissions during that observation 1091 interval. 1093 These parameters, together with knowledge of the protocol stack used 1094 (and thus the size of the various headers added to a packet), limit 1095 the number of bit times per observation interval that the DetNet flow 1096 can occupy the physical medium. 1098 The source is required not to exceed these limits in order to obtain 1099 DetNet service. If the source transmits less data than this limit 1100 allows, the unused resource such as link bandwidth can be made 1101 available by the system to non-DetNet packets. However, making those 1102 resources available to DetNet packets in other DetNet flows would 1103 serve no purpose. Those other DetNet flows have their own dedicated 1104 resources, on the assumption that all DetNet flows can use all of 1105 their resources over a long period of time. 1107 There is no provision in DetNet for throttling DetNet flows (reducing 1108 end-to-end transmission rate via any explicit congestion 1109 notification); the assumption is that a DetNet flow, to be useful, 1110 must be delivered in its entirety. That is, while any useful 1111 application is written to expect a certain number of lost packets, 1112 the real-time applications of interest to DetNet demand that the loss 1113 of data due to the network is an extraordinarily event. 1115 Although DetNet strives to minimize the changes required of an 1116 application to allow it to shift from a special-purpose digital 1117 network to an Internet Protocol network, one fundamental shift in the 1118 behavior of network applications is impossible to avoid: the 1119 reservation of resources before the application starts. In the first 1120 place, a network cannot deliver finite latency and practically zero 1121 packet loss to an arbitrarily high offered load. Secondly, achieving 1122 practically zero packet loss for unthrottled (though bandwidth 1123 limited) DetNet flows means that bridges and routers have to dedicate 1124 buffer resources to specific DetNet flows or to classes of DetNet 1125 flows. The requirements of each reservation have to be translated 1126 into the parameters that control each system's queuing, shaping, and 1127 scheduling functions and delivered to the hosts, bridges, and 1128 routers. 1130 4.3.3. Incomplete Networks 1132 The presence in the network of transit nodes or subnets that are not 1133 fully capable of offering DetNet services complicates the ability of 1134 the intermediate nodes and/or controller to allocate resources, as 1135 extra buffering must be allocated at points downstream from the non- 1136 DetNet intermediate node for a DetNet flow. This extra buffering may 1137 increase latency and/or jitter. 1139 4.4. Traffic Engineering for DetNet 1141 Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines 1142 traffic-engineering architectures for generic applicability across 1143 packet and non-packet networks. From a TEAS perspective, Traffic 1144 Engineering (TE) refers to techniques that enable operators to 1145 control how specific traffic flows are treated within their networks. 1147 Because if its very nature of establishing explicit optimized paths, 1148 Deterministic Networking can be seen as a new, specialized branch of 1149 Traffic Engineering, and inherits its architecture with a separation 1150 into planes. 1152 The Deterministic Networking architecture is thus composed of three 1153 planes, a (User) Application Plane, a Controller Plane, and a Network 1154 Plane, which echoes that of Figure 1 of Software-Defined Networking 1155 (SDN): Layers and Architecture Terminology [RFC7426].: 1157 4.4.1. The Application Plane 1159 Per [RFC7426], the Application Plane includes both applications and 1160 services. In particular, the Application Plane incorporates the User 1161 Agent, a specialized application that interacts with the end user / 1162 operator and performs requests for Deterministic Networking services 1163 via an abstract Flow Management Entity, (FME) which may or may not be 1164 collocated with (one of) the end systems. 1166 At the Application Plane, a management interface enables the 1167 negotiation of flows between end systems. An abstraction of the flow 1168 called a Traffic Specification (TSpec) provides the representation. 1169 This abstraction is used to place a reservation over the (Northbound) 1170 Service Interface and within the Application plane. It is associated 1171 with an abstraction of location, such as IP addresses and DNS names, 1172 to identify the end systems and eventually specify intermediate 1173 nodes. 1175 4.4.2. The Controller Plane 1177 The Controller Plane corresponds to the aggregation of the Control 1178 and Management Planes in [RFC7426], though Common Control and 1179 Measurement Plane (CCAMP) [CCAMP] makes an additional distinction 1180 between management and measurement. When the logical separation of 1181 the Control, Measurement and other Management entities is not 1182 relevant, the term Controller Plane is used for simplicity to 1183 represent them all, and the term Controller Plane Function (CPF) 1184 refers to any device operating in that plane, whether is it a Path 1185 Computation Element (PCE) [RFC4655], or a Network Management entity 1186 (NME), or a distributed control plane. The CPF is a core element of 1187 a controller, in charge of computing Deterministic paths to be 1188 applied in the Network Plane. 1190 A (Northbound) Service Interface enables applications in the 1191 Application Plane to communicate with the entities in the Controller 1192 Plane as illustrated in Figure 7. 1194 One or more CPF(s) collaborate to implement the requests from the FME 1195 as Per-Flow Per-Hop Behaviors installed in the intermediate nodes for 1196 each individual flow. The CPFs place each flow along a deterministic 1197 sequence of intermediate nodes so as to respect per-flow constraints 1198 such as security and latency, and optimize the overall result for 1199 metrics such as an abstract aggregated cost. The deterministic 1200 sequence can typically be more complex than a direct sequence and 1201 include redundancy path, with one or more packet replication and 1202 elimination points. 1204 4.4.3. The Network Plane 1206 The Network Plane represents the network devices and protocols as a 1207 whole, regardless of the Layer at which the network devices operate. 1208 It includes Forwarding Plane (data plane), Application, and 1209 Operational Plane (e.g., OAM) aspects. 1211 The network Plane comprises the Network Interface Cards (NIC) in the 1212 end systems, which are typically IP hosts, and intermediate nodes, 1213 which are typically IP routers and switches. Network-to-Network 1214 Interfaces such as used for Traffic Engineering path reservation in 1215 [RFC5921], as well as User-to-Network Interfaces (UNI) such as 1216 provided by the Local Management Interface (LMI) between network and 1217 end systems, are both part of the Network Plane, both in the control 1218 plane and the data plane. 1220 A Southbound (Network) Interface enables the entities in the 1221 Controller Plane to communicate with devices in the Network Plane as 1222 illustrated in Figure 7. This interface leverages and extends TEAS 1223 to describe the physical topology and resources in the Network Plane. 1225 End End 1226 System System 1228 -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 1230 CPF CPF CPF CPF 1232 -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 1234 intermediate intermed. intermed. intermed. 1235 Node Node Node Node 1236 NIC NIC 1237 intermediate intermed. intermed. intermed. 1238 Node Node Node Node 1240 Figure 7: Northbound and Southbound interfaces 1242 The intermediate nodes (and eventually the end systems NIC) expose 1243 their capabilities and physical resources to the controller (the 1244 CPF), and update the CPFs with their dynamic perception of the 1245 topology, across the Southbound Interface. In return, the CPFs set 1246 the per-flow paths up, providing a Flow Characterization that is more 1247 tightly coupled to the intermediate node Operation than a TSpec. 1249 At the Network plane, intermediate nodes may exchange information 1250 regarding the state of the paths, between adjacent systems and 1251 eventually with the end systems, and forward packets within 1252 constraints associated to each flow, or, when unable to do so, 1253 perform a last resort operation such as drop or declassify. 1255 This document focuses on the Southbound interface and the operation 1256 of the Network Plane. 1258 4.5. Queuing, Shaping, Scheduling, and Preemption 1260 DetNet achieves congestion protection and bounded delivery latency by 1261 reserving bandwidth and buffer resources at every hop along the path 1262 of the DetNet flow. The reservation itself is not sufficient, 1263 however. Implementors and users of a number of proprietary and 1264 standard real-time networks have found that standards for specific 1265 data plane techniques are required to enable these assurances to be 1266 made in a multi-vendor network. The fundamental reason is that 1267 latency variation in one system results in the need for extra buffer 1268 space in the next-hop system(s), which in turn, increases the worst- 1269 case per-hop latency. 1271 Standard queuing and transmission selection algorithms allow a 1272 central controller to compute the latency contribution of each 1273 transit node to the end-to-end latency, to compute the amount of 1274 buffer space required in each transit node for each incremental 1275 DetNet flow, and most importantly, to translate from a flow 1276 specification to a set of values for the managed objects that control 1277 each relay or end system. For example, the IEEE 802.1 WG has 1278 specified (and is specifying) a set of queuing, shaping, and 1279 scheduling algorithms that enable each transit node (bridge or 1280 router), and/or a central controller, to compute these values. These 1281 algorithms include: 1283 o A credit-based shaper [IEEE802.1Qav] (superseded by 1284 [IEEE802.1Q-2018]). 1286 o Time-gated queues governed by a rotating time schedule, 1287 synchronized among all transit nodes [IEEE802.1Qbv] (superseded by 1288 [IEEE802.1Q-2018]). 1290 o Synchronized double (or triple) buffers driven by synchronized 1291 time ticks. [IEEE802.1Qch] (superseded by [IEEE802.1Q-2018]). 1293 o Pre-emption of an Ethernet packet in transmission by a packet with 1294 a more stringent latency requirement, followed by the resumption 1295 of the preempted packet [IEEE802.1Qbu] (superseded by 1296 [IEEE802.1Q-2018]), [IEEE802.3br] (superseded by 1297 [IEEE802.3-2018]). 1299 While these techniques are currently embedded in Ethernet 1300 [IEEE802.3-2018] and bridging standards, we can note that they are 1301 all, except perhaps for packet preemption, equally applicable to 1302 other media than Ethernet, and to routers as well as bridges. Other 1303 media may have its own methods, see, e.g., 1304 [I-D.ietf-6tisch-architecture], [RFC7554]. DetNet may include such 1305 definitions in the future, or may define how these techniques can be 1306 used by DetNet nodes. 1308 4.6. Service instance 1310 A Service instance represents all the functions required on a node to 1311 allow the end-to-end service between the UNIs. 1313 The DetNet network general reference model is shown in Figure 8 for a 1314 DetNet service scenario (i.e., between two DetNet-UNIs). In this 1315 figure, end systems ("A" and "B") are connected directly to the edge 1316 nodes of an IP/MPLS network ("PE1" and "PE2"). End systems 1317 participating in DetNet communication may require connectivity before 1318 setting up an App-flow that requires the DetNet service. Such a 1319 connectivity related service instance and the one dedicated for 1320 DetNet service share the same access. Packets belonging to a DetNet 1321 flow are selected by a filter configured on the access ("F1" and 1322 "F2"). As a result, data flow specific access ("access-A + F1" and 1323 "access-B + F2") are terminated in the flow specific service instance 1324 ("SI-1" and "SI-2"). A tunnel is used to provide connectivity 1325 between the service instances. 1327 The tunnel is used to transport exclusively the packets of the DetNet 1328 flow between "SI-1" and "SI-2". The service instances are configured 1329 to implement DetNet functions and a flow specific DetNet transport. 1330 The service instance and the tunnel may or may not be shared by 1331 multiple DetNet flows. Sharing the service instance by multiple 1332 DetNet flows requires properly populated forwarding tables of the 1333 service instance. 1335 access-A access-B 1336 <-----> <-------- tunnel ----------> <-----> 1338 +---------+ ___ _ +---------+ 1339 End system | +----+ | / \/ \_ | +----+ | End system 1340 "A" -------F1+ | | / \ | | +F2----- "B" 1341 | | +========+ IP/MPLS +=======+ | | 1342 | |SI-1| | \__ Net._/ | |SI-2| | 1343 | +----+ | \____/ | +----+ | 1344 |PE1 | | PE2| 1345 +---------+ +---------+ 1347 Figure 8: DetNet network general reference model 1349 The tunnel between the service instances may have some special 1350 characteristics. For example, in case of a DetNet L3 service, there 1351 are differences in the usage of the PW for DetNet traffic compared to 1352 the network model described in [RFC6658]. In the DetNet scenario, 1353 the PW is likely to be used exclusively by the DetNet flow, whereas 1354 [RFC6658] states: "The packet PW appears as a single point-to-point 1355 link to the client layer. Network-layer adjacency formation and 1356 maintenance between the client equipment will follow the normal 1357 practice needed to support the required relationship in the client 1358 layer ... This packet PseudoWire is used to transport all of the 1359 required Layer-2 and Layer-3 protocols between LSR1 and LSR2". 1360 Further details are network technology specific and can be found in 1361 [I-D.ietf-detnet-dp-sol-mpls] and [I-D.ietf-detnet-dp-sol-ip]. 1363 4.7. Flow identification at technology borders 1365 4.7.1. Exporting flow identification 1367 A DetNet node may need to map specific flows to lower layer flows (or 1368 Streams) in order to provide specific queuing and shaping services 1369 for specific flows. For example: 1371 o A non-IP, strictly L2 source end system X may be sending multiple 1372 flows to the same L2 destination end system Y. Those flows may 1373 include DetNet flows with different QoS requirements, and may 1374 include non-DetNet flows. 1376 o A router may be sending any number of flows to another router. 1377 Again, those flows may include DetNet flows with different QoS 1378 requirements, and may include non-DetNet flows. 1380 o Two routers may be separated by bridges. For these bridges to 1381 perform any required per-flow queuing and shaping, they must be 1382 able to identify the individual flows. 1384 o A Label Edge Router (LER) may have a Label Switched Path (LSP) set 1385 up for handling traffic destined for a particular IP address 1386 carrying only non-DetNet flows. If a DetNet flow to that same 1387 address is requested, a separate LSP may be needed, in order that 1388 all of the Label Switch Routers (LSRs) along the path to the 1389 destination give that flow special queuing and shaping. 1391 The need for a lower-layer node to be aware of individual higher- 1392 layer flows is not unique to DetNet. But, given the endless 1393 complexity of layering and relayering over tunnels that is available 1394 to network designers, DetNet needs to provide a model for flow 1395 identification that is better than packet inspection. That is not to 1396 say that packet inspection to layer 4 or 5 addresses will not be 1397 used, or the capability standardized; but, there are alternatives. 1399 A DetNet relay node can connect DetNet flows on different paths using 1400 different flow identification methods. For example: 1402 o A single unicast DetNet flow passing from router A through a 1403 bridged network to router B may be assigned a TSN Stream 1404 identifier that is unique within that bridged network. The 1405 bridges can then identify the flow without accessing higher-layer 1406 headers. Of course, the receiving router must recognize and 1407 accept that TSN Stream. 1409 o A DetNet flow passing from LSR A to LSR B may be assigned a 1410 different label than that used for other flows to the same IP 1411 destination. 1413 In any of the above cases, it is possible that an existing DetNet 1414 flow can be an aggregate carrying multiple other DetNet flows. (Not 1415 to be confused with DetNet compound vs. member flows.) Of course, 1416 this requires that the aggregate DetNet flow be provisioned properly 1417 to carry the aggregated flows. 1419 Thus, rather than packet inspection, there is the option to export 1420 higher-layer information to the lower layer. The requirement to 1421 support one or the other method for flow identification (or both) is 1422 a complexity that is part of DetNet control models. 1424 4.7.2. Flow attribute mapping between layers 1426 Transport of DetNet flows over multiple technology domains may 1427 require that lower layers are aware of specific flows of higher 1428 layers. Such an "exporting of flow identification" is needed each 1429 time when the forwarding paradigm is changed on the transport path 1430 (e.g., two LSRs are interconnected by a L2 bridged domain, etc.). 1431 The three representative forwarding methods considered for 1432 deterministic networking are: 1434 o IP routing 1436 o MPLS label switching 1438 o Ethernet bridging 1440 A packet with corresponding Flow-IDs is illustrated in Figure 9, 1441 which also indicates where each Flow-ID can be added or removed. 1443 add/remove add/remove 1444 Eth Flow-ID IP Flow-ID 1445 | | 1446 v v 1447 +-----------------------------------------------------------+ 1448 | | | | | 1449 | Eth | MPLS | IP | Application data | 1450 | | | | | 1451 +-----------------------------------------------------------+ 1452 ^ 1453 | 1454 add/remove 1455 MPLS Flow-ID 1457 Figure 9: Packet with multiple Flow-IDs 1459 The additional (domain specific) Flow-ID can be 1461 o created by a domain specific function or 1463 o derived from the Flow-ID added to the App-flow. 1465 The Flow-ID must be unique inside a given domain. Note that the 1466 Flow-ID added to the App-flow is still present in the packet, but 1467 transport nodes may lack the function to recognize it; that's why the 1468 additional Flow-ID is added. 1470 4.7.3. Flow-ID mapping examples 1472 IP nodes and MPLS nodes are assumed to be configured to push such an 1473 additional (domain specific) Flow-ID when sending traffic to an 1474 Ethernet switch (as shown in the examples below). 1476 Figure 10 shows a scenario where an IP end system ("IP-A") is 1477 connected via two Ethernet switches ("ETH-n") to an IP router ("IP- 1478 1"). 1480 IP domain 1481 <----------------------------------------------- 1483 +======+ +======+ 1484 |L3-ID | |L3-ID | 1485 +======+ /\ +-----+ +======+ 1486 / \ Forward as | | 1487 /IP-A\ per ETH-ID |IP-1 | Recognize 1488 Push ------> +-+----+ | +---+-+ <----- ETH-ID 1489 ETH-ID | +----+-----+ | 1490 | v v | 1491 | +-----+ +-----+ | 1492 +------+ | | +---------+ 1493 +......+ |ETH-1+----+ETH-2| +======+ 1494 .L3-ID . +-----+ +-----+ |L3-ID | 1495 +======+ +......+ +======+ 1496 |ETH-ID| .L3-ID . |ETH-ID| 1497 +======+ +======+ +------+ 1498 |ETH-ID| 1499 +======+ 1501 Ethernet domain 1502 <----------------> 1504 Figure 10: IP nodes interconnected by an Ethernet domain 1506 End system "IP-A" uses the original App-flow specific ID ("L3-ID"), 1507 but as it is connected to an Ethernet domain it has to push an 1508 Ethernet-domain specific flow-ID ("VID + multicast MAC address", 1509 referred as "ETH-ID") before sending the packet to "ETH-1" node. 1510 Ethernet switch "ETH-1" can recognize the data flow based on the 1511 "ETH-ID" and it does forwarding toward "ETH-2". "ETH-2" switches the 1512 packet toward the IP router. "IP-1" must be configured to receive 1513 the Ethernet Flow-ID specific multicast flow, but (as it is an L3 1514 node) it decodes the data flow ID based on the "L3-ID" fields of the 1515 received packet. 1517 Figure 11 shows a scenario where MPLS domain nodes ("PE-n" and "P-m") 1518 are connected via two Ethernet switches ("ETH-n"). 1520 MPLS domain 1521 <-----------------------------------------------> 1523 +=======+ +=======+ 1524 |MPLS-ID| |MPLS-ID| 1525 +=======+ +-----+ +-----+ +=======+ +-----+ 1526 | | Forward as | | | | 1527 |PE-1 | per ETH-ID | P-2 +-----------+ PE-2| 1528 Push -----> +-+---+ | +---+-+ +-----+ 1529 ETH-ID | +-----+----+ | \ Recognize 1530 | v v | +-- ETH-ID 1531 | +-----+ +-----+ | 1532 +---+ | | +----+ 1533 +.......+ |ETH-1+----+ETH-2| +=======+ 1534 .MPLS-ID. +-----+ +-----+ |MPLS-ID| 1535 +=======+ +=======+ 1536 |ETH-ID | +.......+ |ETH-ID | 1537 +=======+ .MPLS-ID. +-------+ 1538 +=======+ 1539 |ETH-ID | 1540 +=======+ 1541 Ethernet domain 1542 <----------------> 1544 Figure 11: MPLS nodes interconnected by an Ethernet domain 1546 "PE-1" uses the MPLS specific ID ("MPLS-ID"), but as it is connected 1547 to an Ethernet domain it has to push an Ethernet-domain specific 1548 flow-ID ("VID + multicast MAC address", referred as "ETH-ID") before 1549 sending the packet to "ETH-1". Ethernet switch "ETH-1" can recognize 1550 the data flow based on the "ETH-ID" and it does forwarding toward 1551 "ETH-2". "ETH-2" switches the packet toward the MPLS node ("P-2"). 1552 "P-2" must be configured to receive the Ethernet Flow-ID specific 1553 multicast flow, but (as it is an MPLS node) it decodes the data flow 1554 ID based on the "MPLS-ID" fields of the received packet. 1556 One can appreciate from the above example that, when the means used 1557 for DetNet flow identification is altered or exported, the means for 1558 encoding the sequence number information must similarly be altered or 1559 exported. 1561 4.8. Advertising resources, capabilities and adjacencies 1563 There are three classes of information that a central controller or 1564 distributed control plane needs to know that can only be obtained 1565 from the end systems and/or nodes in the network. When using a peer- 1566 to-peer control plane, some of this information may be required by a 1567 system's neighbors in the network. 1569 o Details of the system's capabilities that are required in order to 1570 accurately allocate that system's resources, as well as other 1571 systems' resources. This includes, for example, which specific 1572 queuing and shaping algorithms are implemented (Section 4.5), the 1573 number of buffers dedicated for DetNet allocation, and the worst- 1574 case forwarding delay and misordering. 1576 o The dynamic state of a node's DetNet resources. 1578 o The identity of the system's neighbors, and the characteristics of 1579 the link(s) between the systems, including the length (in 1580 nanoseconds) of the link(s). 1582 4.9. Scaling to larger networks 1584 Reservations for individual DetNet flows require considerable state 1585 information in each transit node, especially when adequate fault 1586 mitigation (Section 3.3.2) is required. The DetNet data plane, in 1587 order to support larger numbers of DetNet flows, must support the 1588 aggregation of DetNet flows. Such aggregated flows can be viewed by 1589 the transit nodes' data plane largely as individual DetNet flows. 1590 Without such aggregation, the per-relay system may limit the scale of 1591 DetNet networks. Example techniques that may be used include MPLS 1592 hierarchy and IP DiffServ Code Points (DSCPs). 1594 4.10. Compatibility with Layer-2 1596 Standards providing similar capabilities for bridged networks (only) 1597 have been and are being generated in the IEEE 802 LAN/MAN Standards 1598 Committee. The present architecture describes an abstract model that 1599 can be applicable both at Layer-2 and Layer-3, and over links not 1600 defined by IEEE 802. 1602 DetNet enabled end systems and intermediate nodes can be 1603 interconnected by sub-networks, i.e., Layer-2 technologies. These 1604 sub-networks will provide DetNet compatible service for support of 1605 DetNet traffic. Examples of sub-networks include MPLS TE, 802.1 TSN, 1606 and a point-to-point OTN link. Of course, multi-layer DetNet systems 1607 may be possible too, where one DetNet appears as a sub-network, and 1608 provides service to, a higher layer DetNet system. 1610 5. Security Considerations 1612 Security in the context of Deterministic Networking has an added 1613 dimension; the time of delivery of a packet can be just as important 1614 as the contents of the packet, itself. A man-in-the-middle attack, 1615 for example, can impose, and then systematically adjust, additional 1616 delays into a link, and thus disrupt or subvert a real-time 1617 application without having to crack any encryption methods employed. 1618 See [RFC7384] for an exploration of this issue in a related context. 1620 Furthermore, in a control system where millions of dollars of 1621 equipment, or even human lives, can be lost if the DetNet QoS is not 1622 delivered, one must consider not only simple equipment failures, 1623 where the box or wire instantly becomes perfectly silent, but complex 1624 errors such as can be caused by software failures. Because there is 1625 essential no limit to the kinds of failures that can occur, 1626 protecting against realistic equipment failures is indistinguishable, 1627 in most cases, from protecting against malicious behavior, whether 1628 accidental or intentional. See also Section 3.3.2. 1630 Security must cover: 1632 o the protection of the signaling protocol 1634 o the authentication and authorization of the controlling systems 1636 o the identification and shaping of the DetNet flows 1638 6. Privacy Considerations 1640 DetNet is provides a Quality of Service (QoS), and as such, does not 1641 directly raise any new privacy considerations. 1643 However, the requirement for every (or almost every) node along the 1644 path of a DetNet flow to identify DetNet flows may present an 1645 additional attack surface for privacy, should the DetNet paradigm be 1646 found useful in broader environments. 1648 7. IANA Considerations 1650 This document does not require an action from IANA. 1652 8. Acknowledgements 1654 The authors wish to thank Lou Berger, David Black, Stewart Bryant, 1655 Rodney Cummings, Ethan Grossman, Craig Gunther, Marcel Kiessling, 1656 Rudy Klecka, Jouni Korhonen, Erik Nordmark, Shitanshu Shah, Wilfried 1657 Steiner, George Swallow, Michael Johas Teener, Pat Thaler, Thomas 1658 Watteyne, Patrick Wetterwald, Karl Weber, Anca Zamfir, for their 1659 various contribution with this work. 1661 9. Informative References 1663 [CCAMP] IETF, "Common Control and Measurement Plane Working 1664 Group", 1665 . 1667 [I-D.ietf-6tisch-architecture] 1668 Thubert, P., "An Architecture for IPv6 over the TSCH mode 1669 of IEEE 802.15.4", draft-ietf-6tisch-architecture-14 (work 1670 in progress), April 2018. 1672 [I-D.ietf-detnet-dp-sol-ip] 1673 Korhonen, J. and B. Varga, "DetNet IP Data Plane 1674 Encapsulation", draft-ietf-detnet-dp-sol-ip-00 (work in 1675 progress), July 2018. 1677 [I-D.ietf-detnet-dp-sol-mpls] 1678 Korhonen, J. and B. Varga, "DetNet MPLS Data Plane 1679 Encapsulation", draft-ietf-detnet-dp-sol-mpls-00 (work in 1680 progress), July 2018. 1682 [I-D.ietf-detnet-problem-statement] 1683 Finn, N. and P. Thubert, "Deterministic Networking Problem 1684 Statement", draft-ietf-detnet-problem-statement-06 (work 1685 in progress), July 2018. 1687 [I-D.ietf-detnet-use-cases] 1688 Grossman, E., "Deterministic Networking Use Cases", draft- 1689 ietf-detnet-use-cases-17 (work in progress), June 2018. 1691 [IEC62439-3-2016] 1692 International Electrotechnical Commission (IEC) TC 65/SC 1693 65C - Industrial networks, "IEC 62439-3:2016 Industrial 1694 communication networks - High availability automation 1695 networks - Part 3: Parallel Redundancy Protocol (PRP) and 1696 High-availability Seamless Redundancy (HSR)", 2016, 1697 . 1699 [IEEE802.1BA] 1700 IEEE Standards Association, "IEEE Std 802.1BA-2011 Audio 1701 Video Bridging (AVB) Systems", 2011, 1702 . 1704 [IEEE802.1CB] 1705 IEEE Standards Association, "IEEE Std 802.1CB-2017 Frame 1706 Replication and Elimination for Reliability", 2017, 1707 . 1709 [IEEE802.1Q-2018] 1710 IEEE Standards Association, "IEEE Std 802.1Q-2018 Bridges 1711 and Bridged Networks", 2018, 1712 . 1715 [IEEE802.1Qav] 1716 IEEE Standards Association, "IEEE Std 802.1Qav-2009 1717 Bridges and Bridged Networks - Amendment 12: Forwarding 1718 and Queuing Enhancements for Time-Sensitive Streams", 1719 2009, . 1721 [IEEE802.1Qbu] 1722 IEEE Standards Association, "IEEE Std 802.1Qbu-2016 1723 Bridges and Bridged Networks - Amendment 26: Frame 1724 Preemption", 2016, 1725 . 1727 [IEEE802.1Qbv] 1728 IEEE Standards Association, "IEEE Std 802.1Qbv-2015 1729 Bridges and Bridged Networks - Amendment 25: Enhancements 1730 for Scheduled Traffic", 2015, 1731 . 1733 [IEEE802.1Qch] 1734 IEEE Standards Association, "IEEE Std 802.1Qch-2017 1735 Bridges and Bridged Networks - Amendment 29: Cyclic 1736 Queuing and Forwarding", 2017, 1737 . 1739 [IEEE802.1TSNTG] 1740 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1741 Networking Task Group", 2013, 1742 . 1744 [IEEE802.3-2018] 1745 IEEE Standards Association, "IEEE Std 802.3-2018 Standard 1746 for Ethernet", 2018, . 1749 [IEEE802.3br] 1750 IEEE Standards Association, "IEEE Std 802.3br-2016 1751 Standard for Ethernet Amendment 5: Specification and 1752 Management Parameters for Interspersing Express Traffic", 1753 2016, . 1755 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1756 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1757 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1758 September 1997, . 1760 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1761 and W. Weiss, "An Architecture for Differentiated 1762 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1763 . 1765 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1766 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1767 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1768 . 1770 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1771 Jacobson, "RTP: A Transport Protocol for Real-Time 1772 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 1773 July 2003, . 1775 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 1776 Element (PCE)-Based Architecture", RFC 4655, 1777 DOI 10.17487/RFC4655, August 2006, 1778 . 1780 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 1781 L., and L. Berger, "A Framework for MPLS in Transport 1782 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 1783 . 1785 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 1786 Profile (MPLS-TP) Survivability Framework", RFC 6372, 1787 DOI 10.17487/RFC6372, September 2011, 1788 . 1790 [RFC6658] Bryant, S., Ed., Martini, L., Swallow, G., and A. Malis, 1791 "Packet Pseudowire Encapsulation over an MPLS PSN", 1792 RFC 6658, DOI 10.17487/RFC6658, July 2012, 1793 . 1795 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1796 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1797 October 2014, . 1799 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1800 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1801 Defined Networking (SDN): Layers and Architecture 1802 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1803 2015, . 1805 [RFC7554] Watteyne, T., Ed., Palattella, M., and L. Grieco, "Using 1806 IEEE 802.15.4e Time-Slotted Channel Hopping (TSCH) in the 1807 Internet of Things (IoT): Problem Statement", RFC 7554, 1808 DOI 10.17487/RFC7554, May 2015, 1809 . 1811 [RFC7813] Farkas, J., Ed., Bragg, N., Unbehagen, P., Parsons, G., 1812 Ashwood-Smith, P., and C. Bowers, "IS-IS Path Control and 1813 Reservation", RFC 7813, DOI 10.17487/RFC7813, June 2016, 1814 . 1816 [RFC8227] Cheng, W., Wang, L., Li, H., van Helvoort, H., and J. 1817 Dong, "MPLS-TP Shared-Ring Protection (MSRP) Mechanism for 1818 Ring Topology", RFC 8227, DOI 10.17487/RFC8227, August 1819 2017, . 1821 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1822 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1823 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1824 July 2018, . 1826 [TEAS] IETF, "Traffic Engineering Architecture and Signaling 1827 Working Group", 1828 . 1830 Authors' Addresses 1832 Norman Finn 1833 Huawei 1834 3755 Avocado Blvd. 1835 PMB 436 1836 La Mesa, California 91941 1837 US 1839 Phone: +1 925 980 6430 1840 Email: norman.finn@mail01.huawei.com 1841 Pascal Thubert 1842 Cisco Systems 1843 Village d'Entreprises Green Side 1844 400, Avenue de Roumanille 1845 Batiment T3 1846 Biot - Sophia Antipolis 06410 1847 FRANCE 1849 Phone: +33 4 97 23 26 34 1850 Email: pthubert@cisco.com 1852 Balazs Varga 1853 Ericsson 1854 Magyar tudosok korutja 11 1855 Budapest 1117 1856 Hungary 1858 Email: balazs.a.varga@ericsson.com 1860 Janos Farkas 1861 Ericsson 1862 Magyar tudosok korutja 11 1863 Budapest 1117 1864 Hungary 1866 Email: janos.farkas@ericsson.com