idnits 2.17.1 draft-ietf-detnet-architecture-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 757 has weird spacing: '...e stack v ...' -- The document date (December 19, 2018) is 1926 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Network' is mentioned on line 899, but not defined == Outdated reference: A later version (-30) exists of draft-ietf-6tisch-architecture-19 == Outdated reference: A later version (-02) exists of draft-ietf-detnet-dp-sol-ip-01 == Outdated reference: A later version (-02) exists of draft-ietf-detnet-dp-sol-mpls-01 == Outdated reference: A later version (-09) exists of draft-ietf-detnet-problem-statement-08 == Outdated reference: A later version (-16) exists of draft-ietf-detnet-security-03 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-19 Summary: 0 errors (**), 0 flaws (~~), 9 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft Huawei 4 Intended status: Standards Track P. Thubert 5 Expires: June 22, 2019 Cisco 6 B. Varga 7 J. Farkas 8 Ericsson 9 December 19, 2018 11 Deterministic Networking Architecture 12 draft-ietf-detnet-architecture-10 14 Abstract 16 This document provides the overall architecture for Deterministic 17 Networking (DetNet), which provides a capability to carry specified 18 unicast or multicast data flows for real-time applications with 19 extremely low data loss rates and bounded latency within a network 20 domain. Techniques used include: 1) reserving data plane resources 21 for individual (or aggregated) DetNet flows in some or all of the 22 intermediate nodes along the path of the flow; 2) providing explicit 23 routes for DetNet flows that do not immediately change with the 24 network topology; and 3) distributing data from DetNet flow packets 25 over time and/or space to ensure delivery of each packet's data in 26 spite of the loss of a path. DetNet operates at the IP layer and 27 delivers service over sub-network technologies such as MPLS and IEEE 28 802.1 Time-Sensitive Networking (TSN). 30 Status of This Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF). Note that other groups may also distribute 37 working documents as Internet-Drafts. The list of current Internet- 38 Drafts is at https://datatracker.ietf.org/drafts/current/. 40 Internet-Drafts are draft documents valid for a maximum of six months 41 and may be updated, replaced, or obsoleted by other documents at any 42 time. It is inappropriate to use Internet-Drafts as reference 43 material or to cite them other than as "work in progress." 45 This Internet-Draft will expire on June 22, 2019. 47 Copyright Notice 49 Copyright (c) 2018 IETF Trust and the persons identified as the 50 document authors. All rights reserved. 52 This document is subject to BCP 78 and the IETF Trust's Legal 53 Provisions Relating to IETF Documents 54 (https://trustee.ietf.org/license-info) in effect on the date of 55 publication of this document. Please review these documents 56 carefully, as they describe your rights and restrictions with respect 57 to this document. Code Components extracted from this document must 58 include Simplified BSD License text as described in Section 4.e of 59 the Trust Legal Provisions and are provided without warranty as 60 described in the Simplified BSD License. 62 Table of Contents 64 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 65 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 66 2.1. Terms used in this document . . . . . . . . . . . . . . . 4 67 2.2. IEEE 802.1 TSN to DetNet dictionary . . . . . . . . . . . 7 68 3. Providing the DetNet Quality of Service . . . . . . . . . . . 7 69 3.1. Primary goals defining the DetNet QoS . . . . . . . . . . 7 70 3.2. Mechanisms to achieve DetNet QoS . . . . . . . . . . . . 10 71 3.2.1. Resource allocation . . . . . . . . . . . . . . . . . 10 72 3.2.1.1. Eliminate contention loss . . . . . . . . . . . . 10 73 3.2.1.2. Jitter Reduction . . . . . . . . . . . . . . . . 10 74 3.2.2. Service Protection . . . . . . . . . . . . . . . . . 11 75 3.2.2.1. In-Order Delivery . . . . . . . . . . . . . . . . 11 76 3.2.2.2. Packet Replication and Elimination . . . . . . . 12 77 3.2.2.3. Packet encoding for service protection . . . . . 14 78 3.2.3. Explicit routes . . . . . . . . . . . . . . . . . . . 14 79 3.3. Secondary goals for DetNet . . . . . . . . . . . . . . . 15 80 3.3.1. Coexistence with normal traffic . . . . . . . . . . . 15 81 3.3.2. Fault Mitigation . . . . . . . . . . . . . . . . . . 16 82 4. DetNet Architecture . . . . . . . . . . . . . . . . . . . . . 16 83 4.1. DetNet stack model . . . . . . . . . . . . . . . . . . . 16 84 4.1.1. Representative Protocol Stack Model . . . . . . . . . 16 85 4.1.2. DetNet Data Plane Overview . . . . . . . . . . . . . 19 86 4.1.3. Network reference model . . . . . . . . . . . . . . . 21 87 4.2. DetNet systems . . . . . . . . . . . . . . . . . . . . . 22 88 4.2.1. End system . . . . . . . . . . . . . . . . . . . . . 22 89 4.2.2. DetNet edge, relay, and transit nodes . . . . . . . . 23 90 4.3. DetNet flows . . . . . . . . . . . . . . . . . . . . . . 24 91 4.3.1. DetNet flow types . . . . . . . . . . . . . . . . . . 24 92 4.3.2. Source transmission behavior . . . . . . . . . . . . 24 93 4.3.3. Incomplete Networks . . . . . . . . . . . . . . . . . 26 94 4.4. Traffic Engineering for DetNet . . . . . . . . . . . . . 26 95 4.4.1. The Application Plane . . . . . . . . . . . . . . . . 27 96 4.4.2. The Controller Plane . . . . . . . . . . . . . . . . 27 97 4.4.3. The Network Plane . . . . . . . . . . . . . . . . . . 28 98 4.5. Queuing, Shaping, Scheduling, and Preemption . . . . . . 29 99 4.6. Service instance . . . . . . . . . . . . . . . . . . . . 30 100 4.7. Flow identification at technology borders . . . . . . . . 31 101 4.7.1. Exporting flow identification . . . . . . . . . . . . 31 102 4.7.2. Flow attribute mapping between layers . . . . . . . . 33 103 4.7.3. Flow-ID mapping examples . . . . . . . . . . . . . . 34 104 4.8. Advertising resources, capabilities and adjacencies . . . 35 105 4.9. Scaling to larger networks . . . . . . . . . . . . . . . 36 106 4.10. Compatibility with Layer-2 . . . . . . . . . . . . . . . 36 107 5. Security Considerations . . . . . . . . . . . . . . . . . . . 36 108 6. Privacy Considerations . . . . . . . . . . . . . . . . . . . 37 109 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 37 110 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 37 111 9. Informative References . . . . . . . . . . . . . . . . . . . 38 112 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 42 114 1. Introduction 116 This document provides the overall architecture for Deterministic 117 Networking (DetNet), which provides a capability for the delivery of 118 data flows with extremely low packet loss rates and bounded end-to- 119 end delivery latency. DetNet is for networks that are under a single 120 administrative control or within a closed group of administrative 121 control; these include campus-wide networks and private WANs. DetNet 122 is not for large groups of domains such as the Internet. 124 DetNet operates at the IP layer and delivers service over sub-network 125 technologies such as MPLS and IEEE 802.1 Time-Sensitive Networking 126 (TSN). DetNet accomplishes these goals by dedicating network 127 resources such as link bandwidth and buffer space to DetNet flows 128 and/or classes of DetNet flows, and by replicating packets along 129 multiple paths. Unused reserved resources are available to non- 130 DetNet packets as long as all guarantees are fulfilled. 132 The Deterministic Networking Problem Statement 133 [I-D.ietf-detnet-problem-statement] introduces Deterministic 134 Networking, and Deterministic Networking Use Cases 135 [I-D.ietf-detnet-use-cases] summarizes the need for it. See 136 [I-D.ietf-detnet-dp-sol-mpls] and [I-D.ietf-detnet-dp-sol-ip] for 137 specific techniques that can be used to identify DetNet flows and 138 assign them to specific paths through a network. 140 A goal of DetNet is a converged network in all respects. That is, 141 the presence of DetNet flows does not preclude non-DetNet flows, and 142 the benefits offered DetNet flows should not, except in extreme 143 cases, prevent existing Quality of Service (QoS) mechanisms from 144 operating in a normal fashion, subject to the bandwidth required for 145 the DetNet flows. A single source-destination pair can trade both 146 DetNet and non-DetNet flows. End systems and applications need not 147 instantiate special interfaces for DetNet flows. Networks are not 148 restricted to certain topologies; connectivity is not restricted. 149 Any application that generates a data flow that can be usefully 150 characterized as having a maximum bandwidth should be able to take 151 advantage of DetNet, as long as the necessary resources can be 152 reserved. Reservations can be made by the application itself, via 153 network management, by an application's controller, or by other 154 means, e.g., a dynamic control plane (e.g., [RFC2205]). QoS 155 requirements of DetNet flows can be met if all network nodes in a 156 DetNet domain implement DetNet capabilities. DetNet nodes can be 157 interconnected with different sub-network technologies 158 (Section 4.1.2), where the nodes of the subnet are not DetNet aware 159 (Section 4.1.3). 161 Many applications that are intended to be served by Deterministic 162 Networking require the ability to synchronize the clocks in end 163 systems to a sub-microsecond accuracy. Some of the queue control 164 techniques defined in Section 4.5 also require time synchronization 165 among network nodes. The means used to achieve time synchronization 166 are not addressed in this document. DetNet can accommodate various 167 time synchronization techniques and profiles that are defined 168 elsewhere to address the needs of different market segments. 170 2. Terminology 172 2.1. Terms used in this document 174 The following terms are used in the context of DetNet in this 175 document: 177 allocation 178 Resources are dedicated to support a DetNet flow. Depending 179 on an implementation, the resource may be reused by non- 180 DetNet flows when it is not used by the DetNet flow. 182 App-flow 183 The native format of a DetNet flow. 185 DetNet compound flow and DetNet member flow 186 A DetNet compound flow is a DetNet flow that has been 187 separated into multiple duplicate DetNet member flows for 188 service protection at the DetNet service sub-layer. Member 189 flows are merged back into a single DetNet compound flow such 190 that there are no duplicate packets. "Compound" and "member" 191 are strictly relative to each other, not absolutes; a DetNet 192 compound flow comprising multiple DetNet member flows can, in 193 turn, be a member of a higher-order compound. 195 DetNet destination 196 An end system capable of terminating a DetNet flow. 198 DetNet domain 199 The portion of a network that is DetNet aware. It includes 200 end systems and DetNet nodes. 202 DetNet edge node 203 An instance of a DetNet relay node that acts as a source and/ 204 or destination at the DetNet service sub-layer. For example, 205 it can include a DetNet service sub-layer proxy function for 206 DetNet service protection (e.g., the addition or removal of 207 packet sequencing information) for one or more end systems, 208 or starts or terminates resource allocation at the DetNet 209 forwarding sub-layer, or aggregates DetNet services into new 210 DetNet flows. It is analogous to a Label Edge Router (LER) 211 or a Provider Edge (PE) router. 213 DetNet flow 214 A DetNet flow is a sequence of packets from one source to one 215 or more destinations, which conform uniquely to a flow 216 identifier, and to which the DetNet service is to be 217 provided. 219 DetNet forwarding sub-layer 220 The DetNet layer that optionally provides resource allocation 221 for DetNet flows over paths provided by the underlying 222 network. 224 DetNet intermediate node 225 A DetNet relay node or DetNet transit node. 227 DetNet node 228 A DetNet edge node, a DetNet relay node, or a DetNet transit 229 node. 231 DetNet relay node 232 A DetNet node including a service sub-layer function that 233 interconnects different DetNet forwarding sub-layer paths to 234 provide service protection. A DetNet relay node participates 235 in the DetNet service sub-layer. It typically incorporates 236 DetNet forwarding sub-layer functions as well, in which case 237 it is collocated with a transit node. 239 DetNet service sub-layer 240 The DetNet sub-layer at which A DetNet service, e.g., service 241 protection is provided. 243 DetNet service proxy 244 Maps between App-flows and DetNet flows. 246 DetNet source 247 An end system capable of originating a DetNet flow. 249 DetNet system 250 A DetNet aware end system, transit node, or relay node. 251 "DetNet" may be omitted in some text. 253 DetNet transit node 254 A DetNet node operating at the DetNet forwarding sub-layer, 255 that utilizes link layer and/or network layer switching 256 across multiple links and/or sub-networks to provide paths 257 for DetNet service sub-layer functions. Typically provides 258 resource allocation over those paths. An MPLS LSR is an 259 example of a DetNet transit node. 261 DetNet-UNI 262 User-to-Network Interface with DetNet specific 263 functionalities. It is a packet-based reference point and 264 may provide multiple functions like encapsulation, status, 265 synchronization, etc. 267 end system 268 Commonly called a "host" in IETF documents, and an "end 269 station" is IEEE 802 documents. End systems of interest to 270 this document are either sources or destinations of DetNet 271 flows. And end system may or may not be DetNet forwarding 272 sub-layer aware or DetNet service sub-layer aware. 274 link 275 A connection between two DetNet nodes. It may be composed of 276 a physical link or a sub-network technology that can provide 277 appropriate traffic delivery for DetNet flows. 279 PEF A Packet Elimination Function (PEF) eliminates duplicate 280 copies of packets to prevent excess packets flooding the 281 network or duplicate packets being sent out of the DetNet 282 domain. PEF can be implemented by a DetNet edge node, a 283 DetNet relay node, or an end system. 285 PRF A Packet Replication Function (PRF) replicates DetNet flow 286 packets and forwards them to one or more next hops in the 287 DetNet domain. The number of packet copies sent to the next 288 hops is a DetNet flow specific parameter at the point of 289 replication. PRF can be implemented by a DetNet edge node, a 290 DetNet relay node, or an end system. 292 PREOF Collective name for Packet Replication, Elimination, and 293 Ordering Functions. 295 POF A Packet Ordering Function (POF) re-orders packets within a 296 DetNet flow that are received out of order. This function 297 can be implemented by a DetNet edge node, a DetNet relay 298 node, or an end system. 300 reservation 301 The set of resources allocated between a source and one or 302 more destinations through DetNet nodes and subnets associated 303 with a DetNet flow, to provide the provisioned DetNet 304 service. 306 2.2. IEEE 802.1 TSN to DetNet dictionary 308 This section also serves as a dictionary for translating from the 309 terms used by the Time-Sensitive Networking (TSN) Task Group 310 [IEEE802.1TSNTG] of the IEEE 802.1 WG to those of the DetNet WG. 312 Listener 313 The IEEE 802.1 term for a destination of a DetNet flow. 315 relay system 316 The IEEE 802.1 term for a DetNet intermediate node. 318 Stream 319 The IEEE 802.1 term for a DetNet flow. 321 Talker 322 The IEEE 802.1 term for the source of a DetNet flow. 324 3. Providing the DetNet Quality of Service 326 3.1. Primary goals defining the DetNet QoS 328 The DetNet Quality of Service can be expressed in terms of: 330 o Minimum and maximum end-to-end latency from source to destination; 331 timely delivery, and bounded jitter (packet delay variation) 332 derived from these constraints. 334 o Packet loss ratio, under various assumptions as to the operational 335 states of the nodes and links. 337 o An upper bound on out-of-order packet delivery. It is worth 338 noting that some DetNet applications are unable to tolerate any 339 out-of-order delivery. 341 It is a distinction of DetNet that it is concerned solely with worst- 342 case values for the end-to-end latency, jitter, and misordering. 343 Average, mean, or typical values are of little interest, because they 344 do not affect the ability of a real-time system to perform its tasks. 345 In general, a trivial priority-based queuing scheme will give better 346 average latency to a data flow than DetNet; however, it may not be a 347 suitable option for DetNet because of its worst-case latency. 349 Three techniques are used by DetNet to provide these qualities of 350 service: 352 o Resource allocation (Section 3.2.1). 354 o Service protection (Section 3.2.2). 356 o Explicit routes (Section 3.2.3). 358 Resource allocation operates by assigning resources, e.g., buffer 359 space or link bandwidth, to a DetNet flow (or flow aggregate) along 360 its path. Resource allocation greatly reduces, or even eliminates 361 entirely, packet loss due to output packet contention within the 362 network, but it can only be supplied to a DetNet flow that is limited 363 at the source to a maximum packet size and transmission rate. Note 364 that congestion control provided via congestion detection and 365 notification [RFC3168] is explicitly excluded from consideration in 366 DetNet, as it serves a different set of applications. 368 Resource allocation addresses two of the DetNet QoS requirements: 369 latency and packet loss. Given that DetNet nodes have a finite 370 amount of buffer space, resource allocation necessarily results in a 371 maximum end-to-end latency. It also addresses contention related 372 packet loss. 374 Other important contribution to packet loss are random media errors 375 and equipment failures. Service protection is the name for the 376 mechanisms used by DetNet to address these losses. The mechanisms 377 employed are constrained by the requirement to meet the users' 378 latency requirements. Packet replication and elimination 379 (Section 3.2.2) and packet encoding (Section 3.2.2.3) are described 380 in this document to provide service protection; others may be found. 381 For instance, packet encoding can be used to provide service 382 protection against random media errors, packet replication and 383 elimination can be used to provide service protection against 384 equipment failures. This mechanism distributes the contents of 385 DetNet flows over multiple paths in time and/or space, so that the 386 loss of some of the paths does need not cause the loss of any 387 packets. 389 The paths are typically (but not necessarily) explicit routes, so 390 that they do not normally suffer temporary interruptions caused by 391 the convergence of routing or bridging protocols. 393 These three techniques can be applied independently, giving eight 394 possible combinations, including none (no DetNet), although some 395 combinations are of wider utility than others. This separation keeps 396 the protocol stack coherent and maximizes interoperability with 397 existing and developing standards in this (IETF) and other Standards 398 Development Organizations. Some examples of typical expected 399 combinations: 401 o Explicit routes plus service protection are exactly the techniques 402 employed by seamless redundancy mechanisms applied on a ring 403 topology as described, e.g., in [IEC62439-3-2016]. In this 404 example, explicit routes are achieved by limiting the physical 405 topology of the network to a ring. Sequentialization, 406 replication, and duplicate elimination are facilitated by packet 407 tags added at the front or the end of Ethernet frames. [RFC8227] 408 provides another example in the context of MPLS. 410 o Resource allocation alone was originally offered by IEEE 802.1 411 Audio Video bridging [IEEE802.1BA]. As long as the network 412 suffers no failures, packet loss due to output packet contention 413 can be eliminated through the use of a reservation protocol (e.g., 414 Multiple Stream Registration Protocol [IEEE802.1Q-2018]), shapers 415 in every bridge, and proper dimensioning. 417 o Using all three together gives maximum protection. 419 There are, of course, simpler methods available (and employed, today) 420 to achieve levels of latency and packet loss that are satisfactory 421 for many applications. Prioritization and over-provisioning is one 422 such technique. However, these methods generally work best in the 423 absence of any significant amount of non-critical traffic in the 424 network (if, indeed, such traffic is supported at all), or work only 425 if the critical traffic constitutes only a small portion of the 426 network's theoretical capacity, or work only if all systems are 427 functioning properly, or in the absence of actions by end systems 428 that disrupt the network's operations. 430 There are any number of methods in use, defined, or in progress for 431 accomplishing each of the above techniques. It is expected that this 432 DetNet Architecture will assist various vendors, users, and/or 433 "vertical" Standards Development Organizations (dedicated to a single 434 industry) to make selections among the available means of 435 implementing DetNet networks. 437 3.2. Mechanisms to achieve DetNet QoS 439 3.2.1. Resource allocation 441 3.2.1.1. Eliminate contention loss 443 The primary means by which DetNet achieves its QoS assurances is to 444 reduce, or even completely eliminate packet loss due to output packet 445 contention within a DetNet node as a cause of packet loss. This can 446 be achieved only by the provision of sufficient buffer storage at 447 each node through the network to ensure that no packets are dropped 448 due to a lack of buffer storage. Note that a DetNet flow cannot be 449 throttled, i.e., its transmission rate cannot be reduced via explicit 450 congestion notification [RFC3168]. 452 Ensuring adequate buffering requires, in turn, that the source, and 453 every DetNet node along the path to the destination (or nearly every 454 node, see Section 4.3.3) be careful to regulate its output to not 455 exceed the data rate for any DetNet flow, except for brief periods 456 when making up for interfering traffic. Any packet sent ahead of its 457 time potentially adds to the number of buffers required by the next 458 hop DetNet node and may thus exceed the resources allocated for a 459 particular DetNet flow. 461 The low-level mechanisms described in Section 4.5 provide the 462 necessary regulation of transmissions by an end system or DetNet node 463 to provide resource allocation. The allocation of the bandwidth and 464 buffers for a DetNet flow requires provisioning. A DetNet node may 465 have other resources requiring allocation and/or scheduling, that 466 might otherwise be over-subscribed and trigger the rejection of a 467 reservation. 469 3.2.1.2. Jitter Reduction 471 A core objective of DetNet is to enable the convergence of sensitive 472 non-IP networks onto a common network infrastructure. This requires 473 the accurate emulation of currently deployed mission-specific 474 networks, which for example rely on point-to-point analog (e.g., 475 4-20mA modulation) and serial-digital cables (or buses) for highly 476 reliable, synchronized and jitter-free communications. While the 477 latency of analog transmissions is basically the speed of light, 478 legacy serial links are usually slow (in the order of Kbps) compared 479 to, say, GigE, and some latency is usually acceptable. What is not 480 acceptable is the introduction of excessive jitter, which may, for 481 instance, affect the stability of control systems. 483 Applications that are designed to operate on serial links usually do 484 not provide services to recover the jitter, because jitter simply 485 does not exist there. DetNet flows are generally expected to be 486 delivered in-order and the precise time of reception influences the 487 processes. In order to converge such existing applications, there is 488 a desire to emulate all properties of the serial cable, such as clock 489 transportation, perfect flow isolation and fixed latency. While 490 minimal jitter (in the form of specifying minimum, as well as 491 maximum, end-to-end latency) is supported by DetNet, there are 492 practical limitations on packet-based networks in this regard. In 493 general, users are encouraged to use, instead of, "do this when you 494 get the packet," a combination of: 496 o Sub-microsecond time synchronization among all source and 497 destination end systems, and 499 o Time-of-execution fields in the application packets. 501 Jitter reduction is provided by the mechanisms described in 502 Section 4.5 that also provide resource allocation. 504 3.2.2. Service Protection 506 Service protection aims to mitigate or eliminate packet loss due to 507 equipment failures, random media and/or memory faults. These types 508 of packet loss can be greatly reduced by spreading the data over 509 multiple disjoint forwarding paths. Various service protection 510 methods are described in [RFC6372], e.g., 1+1 linear protection. 511 This section describes the functional details of an additional method 512 in Section 3.2.2.2, which can be implemented as described in 513 Section 3.2.2.3 or as specified in [I-D.ietf-detnet-dp-sol-mpls] in 514 order to provide 1+n hitless protection. The appropriate service 515 protection mechanism depends on the scenario and the requirements. 517 3.2.2.1. In-Order Delivery 519 Out-of-order packet delivery can be a side effect of service 520 protection. Packets delivered out-of-order impact the amount of 521 buffering needed at the destination to properly process the received 522 data. Such packets also influence the jitter of a flow. The DetNet 523 service includes maximum allowed misordering as a constraint. Zero 524 misordering would be a valid service constraint to reflect that the 525 end system(s) of the flow cannot tolerate any out-of-order delivery. 527 DetNet Packet Ordering Functionality (POF) (Section 3.2.2.2) can be 528 used to provide in-order delivery. 530 3.2.2.2. Packet Replication and Elimination 532 This section describes a service protection method that sends copies 533 of the same packets over multiple paths. 535 The DetNet service sub-layer includes the packet replication (PRF), 536 the packet elimination (PEF), and the packet ordering functionality 537 (POF) for use in DetNet edge, relay node, and end system packet 538 processing. Either of these functions can be enabled in a DetNet 539 edge node, relay node or end system. The collective name for all 540 three functions is PREOF. The packet replication and elimination 541 service protection method altogether involves four capabilities: 543 o Providing sequencing information to the packets of a DetNet 544 compound flow. This may be done by adding a sequence number or 545 time stamp as part of DetNet, or may be inherent in the packet, 546 e.g., in a higher layer protocol, or associated to other physical 547 properties such as the precise time (and radio channel) of 548 reception of the packet. This is typically done once, at or near 549 the source. 551 o The Packet Replication Function (PRF) replicates these packets 552 into multiple DetNet member flows and typically sends them along 553 multiple different paths to the destination(s), e.g., over the 554 explicit routes of Section 3.2.3. The location within a DetNet 555 node, and the mechanism used for the PRF is implementation 556 specific. 558 o The Packet Elimination Function (PEF) eliminates duplicate packets 559 of a DetNet flow based on the sequencing information and a history 560 of received packets. The output of the PEF is always a single 561 packet. This may be done at any DetNet node along the path to 562 save network resources further downstream, in particular if 563 multiple Replication points exist. But the most common case is to 564 perform this operation at the very edge of the DetNet network, 565 preferably in or near the receiver. The location within a DetNet 566 node, and mechanism used for the PEF is implementation specific. 568 o The Packet Ordering Function (POF) uses the sequencing information 569 to re-order a DetNet flow's packets that are received out of 570 order. 572 The order in which a DetNet node applies PEF, POF, and PRF to a 573 DetNet flow is implementation specific. 575 Some service protection mechanisms rely on switching from one flow to 576 another when a failure of a flow is detected. Contrarily, packet 577 replication and elimination combines the DetNet member flows sent 578 along multiple different paths, and performs a packet-by-packet 579 selection of which to discard, e.g., based on sequencing information. 581 In the simplest case, this amounts to replicating each packet in a 582 source that has two interfaces, and conveying them through the 583 network, along separate (SRLG disjoint) paths, to the similarly dual- 584 homed destinations, that discard the extras. This ensures that one 585 path remains, even if some DetNet intermediate node fails. The 586 sequencing information can also be used for loss detection and for 587 re-ordering. 589 DetNet relay nodes in the network can provide replication and 590 elimination facilities at various points in the network, so that 591 multiple failures can be accommodated. 593 This is shown in Figure 1, where the two relay nodes each replicate 594 (R) the DetNet flow on input, sending the DetNet member flows to both 595 the other relay node and to the end system, and eliminate duplicates 596 (E) on the output interface to the right-hand end system. Any one 597 link in the network can fail, and the DetNet compound flow can still 598 get through. Furthermore, two links can fail, as long as they are in 599 different segments of the network. 601 > > > > > > > > > relay > > > > > > > > 602 > /------------+ R node E +------------\ > 603 > / v + ^ \ > 604 end R + v | ^ + E end 605 system + v | ^ + system 606 > \ v + ^ / > 607 > \------------+ R relay E +-----------/ > 608 > > > > > > > > > node > > > > > > > > 610 Figure 1: Packet replication and elimination 612 Packet replication and elimination does not react to and correct 613 failures; it is entirely passive. Thus, intermittent failures, 614 mistakenly created packet filters, or misrouted data is handled just 615 the same as the equipment failures that are handled by typical 616 routing and bridging protocols. 618 If packet replication and elimination is used over paths with 619 resource allocation (Section 3.2.1), and member flows that take 620 different-length paths through the network are combined, a merge 621 point may require extra buffering to equalize the delays over the 622 different paths. This equalization ensures that the resultant 623 compound flow will not exceed its contracted bandwidth even after one 624 or the other of the paths is restored after a failure. The extra 625 buffering can be also used to provide in-order delivery. 627 3.2.2.3. Packet encoding for service protection 629 There are methods for using multiple paths to provide service 630 protection that involve encoding the information in a packet 631 belonging to a DetNet flow into multiple transmission units, 632 combining information from multiple packets into any given 633 transmission unit. Such techniques, also known as "network coding", 634 can be used as a DetNet service protection technique. 636 3.2.3. Explicit routes 638 In networks controlled by typical dynamic control protocols such as 639 IS-IS or OSPF, a network topology event in one part of the network 640 can impact, at least briefly, the delivery of data in parts of the 641 network remote from the failure or recovery event. Even the use of 642 redundant paths through a network, e.g., as defined by [RFC6372] do 643 not eliminate the chances of packet loss. Furthermore, out-of-order 644 packet delivery can be a side effect of route changes. 646 Many real-time networks rely on physical rings of two-port devices, 647 with a relatively simple ring control protocol. This supports 648 redundant paths for service protection with a minimum of wiring. As 649 an additional benefit, ring topologies can often utilize different 650 topology management protocols than those used for a mesh network, 651 with a consequent reduction in the response time to topology changes. 652 Of course, this comes at some cost in terms of increased hop count, 653 and thus latency, for the typical path. 655 In order to get the advantages of low hop count and still ensure 656 against even very brief losses of connectivity, DetNet employs 657 explicit routes, where the path taken by a given DetNet flow does not 658 change, at least immediately, and likely not at all, in response to 659 network topology events. Service protection (Section 3.2.2 or 660 Section 3.2.2.3) over explicit routes provides a high likelihood of 661 continuous connectivity. Explicit routes can be established in 662 various ways, e.g., with RSVP-TE [RFC3209], with Segment Routing (SR) 663 [RFC8402], via a Software Defined Networking approach [RFC7426], 664 [RFC8453], and [RFC8453], with IS-IS [RFC7813], etc. Explicit routes 665 are typically used in MPLS TE LSPs. 667 Out-of-order packet delivery can be a side effect of distributing a 668 single flow over multiple paths especially when there is a change 669 from one path to another when combining the flow. This is 670 irrespective of the distribution method used, and also applies to 671 service protection over explicit routes. As described in 672 Section 3.2.2.1, out-of-order packets influence the jitter of a flow 673 and impact the amount of buffering needed to process the data; 674 therefore, DetNet service includes maximum allowed misordering as a 675 constraint. The use of explicit routes helps to provide in-order 676 delivery because there is no immediate route change with the network 677 topology, but the changes are plannable as they are between the 678 different explicit routes. 680 3.3. Secondary goals for DetNet 682 Many applications require DetNet to provide additional services, 683 including coexistence with other QoS mechanisms Section 3.3.1 and 684 protection against misbehaving transmitters Section 3.3.2. 686 3.3.1. Coexistence with normal traffic 688 A DetNet network supports the dedication of a high proportion of the 689 network bandwidth to DetNet flows. But, no matter how much is 690 dedicated for DetNet flows, it is a goal of DetNet to coexist with 691 existing Class of Service schemes (e.g., DiffServ). It is also 692 important that non-DetNet traffic not disrupt the DetNet flow, of 693 course (see Section 3.3.2 and Section 5). For these reasons: 695 o Bandwidth (transmission opportunities) not utilized by a DetNet 696 flow is available to non-DetNet packets (though not to other 697 DetNet flows). 699 o DetNet flows can be shaped or scheduled, in order to ensure that 700 the highest-priority non-DetNet packet is also ensured a worst- 701 case latency. 703 o When transmission opportunities for DetNet flows are scheduled in 704 detail, then the algorithm constructing the schedule should leave 705 sufficient opportunities for non-DetNet packets to satisfy the 706 needs of the users of the network. Detailed scheduling can also 707 permit the time-shared use of buffer resources by different DetNet 708 flows. 710 Starvation of non-DetNet traffic must be avoided, e.g., by traffic 711 policing functions (e.g., [RFC2475]). Thus, the net effect of the 712 presence of DetNet flows in a network on the non-DetNet flows is 713 primarily a reduction in the available bandwidth. 715 3.3.2. Fault Mitigation 717 Robust real-time systems require to reduce the number of possible 718 failures. Filters and policers should be used in a DetNet network to 719 detect if DetNet packets are received on the wrong interface, or at 720 the wrong time, or in too great a volume. Furthermore, filters and 721 policers can take actions to discard the offending packets or flows, 722 or trigger shutting down the offending flow or the offending 723 interface. 725 It is also essential that filters and service remarking be employed 726 at the network edge to prevent non-DetNet packets from being mistaken 727 for DetNet packets, and thus impinging on the resources allocated to 728 DetNet packets. 730 There exist techniques, at present and/or in various stages of 731 standardization, that can perform these fault mitigation tasks that 732 deliver a high probability that misbehaving systems will have zero 733 impact on well-behaved DetNet flows, except of course, for the 734 receiving interface(s) immediately downstream of the misbehaving 735 device. Examples of such techniques include traffic policing 736 functions (e.g., [RFC2475]) and separating flows into per-flow rate- 737 limited queues. 739 4. DetNet Architecture 741 4.1. DetNet stack model 743 DetNet functionality (Section 3) is implemented in two adjacent sub- 744 layers in the protocol stack: the DetNet service sub-layer and the 745 DetNet forwarding sub-layer. The DetNet service sub-layer provides 746 DetNet service, e.g., service protection, to higher layers in the 747 protocol stack and applications. The DetNet forwarding sub-layer 748 supports DetNet service in the underlying network, e.g., by providing 749 explicit routes and resource allocation to DetNet flows. 751 4.1.1. Representative Protocol Stack Model 753 Figure 2 illustrates a conceptual DetNet data plane layering model. 754 One may compare it to that in [IEEE802.1CB], Annex C. 756 | packets going | ^ packets coming ^ 757 v down the stack v | up the stack | 758 +-----------------------+ +-----------------------+ 759 | Source | | Destination | 760 +-----------------------+ +-----------------------+ 761 | Service sub-layer: | | Service sub-layer: | 762 | Packet sequencing | | Duplicate elimination | 763 | Flow replication | | Flow merging | 764 | Packet encoding | | Packet decoding | 765 +-----------------------+ +-----------------------+ 766 | Forwarding sub-layer: | | Forwarding sub-layer: | 767 | Resource allocation | | Resource allocation | 768 | Explicit routes | | Explicit routes | 769 +-----------------------+ +-----------------------+ 770 | Lower layers | | Lower layers | 771 +-----------------------+ +-----------------------+ 772 v ^ 773 \_________________________/ 775 Figure 2: DetNet data plane protocol stack 777 Not all sub-layers are required for any given application, or even 778 for any given network. The functionality shown in Figure 2 is: 780 Application 781 Shown as "source" and "destination" in the diagram. 783 Packet sequencing 784 As part of DetNet service protection, supplies the sequence 785 number for packet replication and elimination 786 (Section 3.2.2). Peers with Duplicate elimination. This 787 sub-layer is not needed if a higher layer protocol is 788 expected to perform any packet sequencing and duplicate 789 elimination required by the DetNet flow replication. 791 Duplicate elimination 792 As part of the DetNet service sub-layer, based on the 793 sequenced number supplied by its peer, packet sequencing, 794 Duplicate elimination discards any duplicate packets 795 generated by DetNet flow replication. It can operate on 796 member flows, compound flows, or both. The replication may 797 also be inferred from other information such as the precise 798 time of reception in a scheduled network. The duplicate 799 elimination sub-layer may also perform resequencing of 800 packets to restore packet order in a flow that was disrupted 801 by the loss of packets on one or another of the multiple 802 paths taken. 804 Flow replication 805 As part of DetNet service protection, packets that belong to 806 a DetNet compound flow are replicated into two or more DetNet 807 member flows. This function is separate from packet 808 sequencing. Flow replication can be an explicit replication 809 and remarking of packets, or can be performed by, for 810 example, techniques similar to ordinary multicast 811 replication, albeit with resource allocation implications. 812 Peers with DetNet flow merging. 814 Flow merging 815 As part of DetNet service protection, merges DetNet member 816 flows together for packets coming up the stack belonging to a 817 specific DetNet compound flow. Peers with DetNet flow 818 replication. DetNet flow merging, together with packet 819 sequencing, duplicate elimination, and DetNet flow 820 replication perform packet replication and elimination 821 (Section 3.2.2). 823 Packet encoding 824 As part of DetNet service protection, as an alternative to 825 packet sequencing and flow replication, packet encoding 826 combines the information in multiple DetNet packets, perhaps 827 from different DetNet compound flows, and transmits that 828 information in packets on different DetNet member Flows. 829 Peers with Packet decoding. 831 Packet decoding 832 As part of DetNet service protection, as an alternative to 833 flow merging and duplicate elimination, packet decoding takes 834 packets from different DetNet member flows, and computes from 835 those packets the original DetNet packets from the compound 836 flows input to packet encoding. Peers with Packet encoding. 838 Resource allocation 839 The DetNet forwarding sub-layer provides resource allocation. 840 See Section 4.5. The actual queuing and shaping mechanisms 841 are typically provided by underlying subnet, these can be 842 closely associated with the means of providing paths for 843 DetNet flows, the path and the resource allocation are 844 conflated in this figure. 846 Explicit routes 847 The DetNet forwarding sub-layer provides mechanisms to ensure 848 that fixed paths are provided for DetNet flows. These 849 explicit paths avoid the impact of network convergence. 851 Operations, Administration, and Maintenance (OAM) leverages in-band 852 and out-of-band signaling that validates whether the service is 853 effectively obtained within QoS constraints. OAM is not shown in 854 Figure 2; it may reside in any number of the layers. OAM can involve 855 specific tagging added in the packets for tracing implementation or 856 network configuration errors; traceability enables to find whether a 857 packet is a replica, which DetNet relay node performed the 858 replication, and which segment was intended for the replica. Active 859 and hybrid OAM methods require additional bandwidth to perform fault 860 management and performance monitoring of the DetNet domain. OAM may, 861 for instance, generate special test probes or add OAM information 862 into the data packet. 864 The packet sequencing and replication elimination functions at the 865 source and destination ends of a DetNet compound flow may be 866 performed either in the end system or in a DetNet relay node. 868 4.1.2. DetNet Data Plane Overview 870 A "Deterministic Network" will be composed of DetNet enabled end 871 systems, DetNet edge nodes, DetNet relay nodes and collectively 872 deliver DetNet services. DetNet relay and edge nodes are 873 interconnected via DetNet transit nodes (e.g., LSRs) which support 874 DetNet, but are not DetNet service aware. All DetNet nodes are 875 connected to sub-networks, where a point-to-point link is also 876 considered as a simple sub-network. These sub-networks will provide 877 DetNet compatible service for support of DetNet traffic. Examples of 878 sub-networks include MPLS TE, IEEE 802.1 TSN and OTN. Of course, 879 multi-layer DetNet systems may also be possible, where one DetNet 880 appears as a sub-network, and provides service to, a higher layer 881 DetNet system. A simple DetNet concept network is shown in Figure 3. 882 Note that in this and following figures "Forwarding" and "Fwd" refer 883 to the DetNet forwarding sub-layer, "Service" and "Svc" refer to the 884 DetNet service sub-layer, which are described in detail in 885 Section 4.1. 887 TSN Edge Transit Relay DetNet 888 End System Node Node Node End System 890 +----------+ +.........+ +----------+ 891 | Appl. |<--:Svc Proxy:-- End to End Service -------->| Appl. | 892 +----------+ +---------+ +---------+ +----------+ 893 | TSN | |TSN| |Svc|<- DetNet flow --: Service :-->| Service | 894 +----------+ +---+ +---+ +--------+ +---------+ +----------+ 895 |Forwarding| |Fwd| |Fwd| | Fwd | |Fwd| |Fwd| |Forwarding| 896 +-------.--+ +-.-+ +-.-+ +--.----.+ +-.-+ +-.-+ +---.------+ 897 : Link : / ,-----. \ : Link : / ,-----. \ 898 +........+ +-[ Sub ]-+ +.......+ +-[ Sub ]-+ 899 [Network] [Network] 900 `-----' `-----' 902 Figure 3: A Simple DetNet Enabled Network 904 Distinguishing the function of two DetNet data plane sub-layers, the 905 DetNet service sub-layer and the DetNet forwarding sub-layer, helps 906 to explore and evaluate various combinations of the data plane 907 solutions available, some are illustrated in Figure 4. This 908 separation of DetNet sub-layers, while helpful, should not be 909 considered as formal requirement. For example, some technologies may 910 violate these strict sub-layers and still be able to deliver a DetNet 911 service. 913 . 914 . 915 +-----------------------------+ 916 | DetNet Service sub-layer | PW, UDP, GRE 917 +-----------------------------+ 918 | DetNet Forwarding sub-layer | IPv6, IPv4, MPLS TE LSPs, MPLS SR 919 +-----------------------------+ 920 . 921 . 923 Figure 4: DetNet adaptation to data plane 925 In some networking scenarios, the end system initially provides a 926 DetNet flow encapsulation, which contains all information needed by 927 DetNet nodes (e.g., Real-time Transport Protocol (RTP) [RFC3550] 928 based DetNet flow carried over a native UDP/IP network or 929 PseudoWire). In other scenarios, the encapsulation formats might 930 differ significantly. 932 There are many valid options to create a data plane solution for 933 DetNet traffic by selecting a technology approach for the DetNet 934 service sub-layer and also selecting a technology approach for the 935 DetNet forwarding sub-layer. There are a high number of valid 936 combinations. 938 One of the most fundamental differences between different potential 939 data plane options is the basic headers used by DetNet nodes. For 940 example, the basic service can be delivered based on an MPLS label or 941 an IP header. This decision impacts the basic forwarding logic for 942 the DetNet service sub-layer. Note that in both cases, IP addresses 943 are used to address DetNet nodes. The selected DetNet forwarding 944 sub-layer technology also needs to be mapped to the sub-net 945 technology used to interconnect DetNet nodes. For example, DetNet 946 flows will need to be mapped to TSN Streams. 948 4.1.3. Network reference model 950 Figure 5 shows another view of the DetNet service related reference 951 points and main components. 953 DetNet DetNet 954 end system end system 955 _ _ 956 / \ +----DetNet-UNI (U) / \ 957 /App\ | /App\ 958 /-----\ | /-----\ 959 | NIC | v ________ | NIC | 960 +--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+ 961 | / \__/ \ | | 962 | / +----+ +----+ \_____ | | 963 | / | | | | \_______ | | 964 +------U PE +----+ P +----+ \ _ v | 965 | | | | | | | ___/ \ | 966 | +--+-+ +----+ | +----+ | / \_ | 967 \ | | | | | / \ | 968 \ | +----+ +--+-+ +--+PE |------ U-----+ 969 \ | | | | | | | | | \_ _/ 970 \ +---+ P +----+ P +--+ +----+ | \____/ 971 \___ | | | | / 972 \ +----+__ +----+ DetNet-1 DetNet-2 973 | \_____/ \___________/ | 974 | | 975 | | End-to-End service | | | | 976 <-------------------------------------------------------------> 977 | | DetNet service | | | | 978 | <------------------------------------------------> | 979 | | | | | | 981 Figure 5: DetNet Service Reference Model (multi-domain) 983 DetNet-UNIs ("U" in Figure 5) are assumed in this document to be 984 packet-based reference points and provide connectivity over the 985 packet network. A DetNet-UNI may provide multiple functions, e.g., 986 it may add networking technology specific encapsulation to the DetNet 987 flows if necessary; it may provide status of the availability of the 988 resources associated with a reservation; it may provide a 989 synchronization service for the end system; it may carry enough 990 signaling to place the reservation in a network without a controller, 991 or if the controller only deals with the network but not the end 992 systems. Internal reference points of end systems (between the 993 application and the NIC) are more challenging from control 994 perspective and they may have extra requirements (e.g., in-order 995 delivery is expected in end system internal reference points, whereas 996 it is considered optional over the DetNet-UNI). 998 4.2. DetNet systems 1000 4.2.1. End system 1002 The native data flow between the source/destination end systems is 1003 referred to as application-flow (App-flow). The traffic 1004 characteristics of an App-flow can be CBR (constant bit rate) or VBR 1005 (variable bit rate) and can have L1 or L2 or L3 encapsulation (e.g., 1006 TDM (time-division multiplexing), Ethernet, IP). These 1007 characteristics are considered as input for resource reservation and 1008 might be simplified to ensure determinism during packet forwarding 1009 (e.g., making reservations for the peak rate of VBR traffic, etc.). 1011 An end system may or may not be DetNet forwarding sub-layer aware or 1012 DetNet service sub-layer aware. That is, an end system may or may 1013 not contain DetNet specific functionality. End systems with DetNet 1014 functionalities may have the same or different forwarding sub-layer 1015 as the connected DetNet domain. Categorization of end systems are 1016 shown in Figure 6. 1018 End system 1019 | 1020 | 1021 | DetNet aware ? 1022 / \ 1023 +------< >------+ 1024 NO | \ / | YES 1025 | v | 1026 DetNet unaware | 1027 End system | 1028 | Service/Forwarding 1029 | sub-layer 1030 / \ aware ? 1031 +--------< >-------------+ 1032 f-aware | \ / | s-aware 1033 | v | 1034 | | both | 1035 | | | 1036 DetNet f-aware | DetNet s-aware 1037 End system | End system 1038 v 1039 DetNet sf-aware 1040 End system 1042 Figure 6: Categorization of end systems 1044 Note some known use case examples for end systems: 1046 o DetNet unaware: The classic case requiring service proxies. 1048 o DetNet f-aware: A DetNet forwarding sub-layer aware system. It 1049 knows about some TSN functions (e.g., reservation), but not about 1050 service protection. 1052 o DetNet s-aware: A DetNet service sub-layer aware system. It 1053 supplies sequence numbers, but doesn't know about resource 1054 allocation. 1056 o DetNet sf-aware: A full functioning DetNet end system, it has 1057 DetNet functionalities and usually the same forwarding paradigm as 1058 the connected DetNet domain. It can be treated as an integral 1059 part of the DetNet domain. 1061 4.2.2. DetNet edge, relay, and transit nodes 1063 As shown in Figure 3, DetNet edge nodes providing proxy service and 1064 DetNet relay nodes providing the DetNet service sub-layer are DetNet- 1065 aware, and DetNet transit nodes need only be aware of the DetNet 1066 forwarding sub-layer. 1068 In general, if a DetNet flow passes through one or more DetNet- 1069 unaware network nodes between two DetNet nodes providing the DetNet 1070 forwarding sub-layer for that flow, there is a potential for 1071 disruption or failure of the DetNet QoS. A network administrator 1072 needs to ensure that the DetNet-unaware network nodes are configured 1073 to minimize the chances of packet loss and delay, and provision 1074 enough extra buffer space in the DetNet transit node following the 1075 DetNet-unaware network nodes to absorb the induced latency 1076 variations. 1078 4.3. DetNet flows 1080 4.3.1. DetNet flow types 1082 A DetNet flow can have different formats while its packets are 1083 forwarded between the peer end systems. Therefore, the following 1084 possible types / formats of a DetNet flow are distinguished in this 1085 document: 1087 o App-flow: native format of the data carried over a DetNet flow. 1088 It does not contain any DetNet related attributes. 1090 o DetNet-f-flow: specific format of a DetNet flow. It only requires 1091 the resource allocation features provided by the DetNet forwarding 1092 sub-layer. 1094 o DetNet-s-flow: specific format of a DetNet flow. It only requires 1095 the service protection feature ensured by the DetNet service sub- 1096 layer. 1098 o DetNet-sf-flow: specific format of a DetNet flow. It requires 1099 both DetNet service sub-layer and DetNet forwarding sub-layer 1100 functions during forwarding. 1102 4.3.2. Source transmission behavior 1104 For the purposes of resource allocation, DetNet flows can be 1105 synchronous or asynchronous. In synchronous DetNet flows, at least 1106 the DetNet nodes (and possibly the end systems) are closely time 1107 synchronized, typically to better than 1 microsecond. By 1108 transmitting packets from different DetNet flows or classes of DetNet 1109 flows at different times, using repeating schedules synchronized 1110 among the DetNet nodes, resources such as buffers and link bandwidth 1111 can be shared over the time domain among different DetNet flows. 1112 There is a tradeoff among techniques for synchronous DetNet flows 1113 between the burden of fine-grained scheduling and the benefit of 1114 reducing the required resources, especially buffer space. 1116 In contrast, asynchronous DetNet flows are not coordinated with a 1117 fine-grained schedule, so relay and end systems must assume worst- 1118 case interference among DetNet flows contending for buffer resources. 1119 Asynchronous DetNet flows are characterized by: 1121 o A maximum packet size; 1123 o An observation interval; and 1125 o A maximum number of transmissions during that observation 1126 interval. 1128 These parameters, together with knowledge of the protocol stack used 1129 (and thus the size of the various headers added to a packet), limit 1130 the number of bit times per observation interval that the DetNet flow 1131 can occupy the physical medium. 1133 The source is required not to exceed these limits in order to obtain 1134 DetNet service. If the source transmits less data than this limit 1135 allows, the unused resource such as link bandwidth can be made 1136 available by the DetNet system to non-DetNet packets as long as all 1137 guarantees are fulfilled. However, making those resources available 1138 to DetNet packets in other DetNet flows would serve no purpose. 1139 Those other DetNet flows have their own dedicated resources, on the 1140 assumption that all DetNet flows can use all of their resources over 1141 a long period of time. 1143 There is no provision in DetNet for throttling DetNet flows, i.e., 1144 the transmission rate cannot be reduced via explicit congestion 1145 notification [RFC3168]. The assumption is that a DetNet flow, to be 1146 useful, must be delivered in its entirety. That is, while any useful 1147 application is written to expect a certain number of lost packets, 1148 the real-time applications of interest to DetNet demand that the loss 1149 of data due to the network is a rare event. 1151 Although DetNet strives to minimize the changes required of an 1152 application to allow it to shift from a special-purpose digital 1153 network to an Internet Protocol network, one fundamental shift in the 1154 behavior of network applications is impossible to avoid: the 1155 reservation of resources before the application starts. In the first 1156 place, a network cannot deliver finite latency and practically zero 1157 packet loss to an arbitrarily high offered load. Secondly, achieving 1158 practically zero packet loss for unthrottled (though bandwidth 1159 limited) DetNet flows means that DetNet nodes have to dedicate buffer 1160 resources to specific DetNet flows or to classes of DetNet flows. 1162 The requirements of each reservation have to be translated into the 1163 parameters that control each DetNet system's queuing, shaping, and 1164 scheduling functions and delivered to the DetNet nodes and end 1165 systems. 1167 All nodes in a DetNet domain are expected to support the data 1168 behavior required to deliver a particular DetNet service. If a node 1169 itself is not DetNet service aware, the DetNet nodes that are 1170 adjacent to such non-DetNet aware nodes must ensure that the non- 1171 DetNet aware node is provisioned to appropriately support the DetNet 1172 service. For example, an IEEE 802.1 TSN node may be used to 1173 interconnect DetNet aware nodes, and these DetNet nodes can map 1174 DetNet flows to 802.1 TSN flows. Another example, an MPLS-TE or TP 1175 domain may be used to interconnect DetNet aware nodes, and these 1176 DetNet nodes can map DetNet flows to TE LSPs which can provide the 1177 QoS requirements of the DetNet service. 1179 4.3.3. Incomplete Networks 1181 The presence in the network of intermediate nodes or subnets that are 1182 not fully capable of offering DetNet services complicates the ability 1183 of the intermediate nodes and/or controller to allocate resources, as 1184 extra buffering must be allocated at points downstream from the non- 1185 DetNet intermediate node for a DetNet flow. This extra buffering may 1186 increase latency and/or jitter. 1188 4.4. Traffic Engineering for DetNet 1190 Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines 1191 traffic-engineering architectures for generic applicability across 1192 packet and non-packet networks. From a TEAS perspective, Traffic 1193 Engineering (TE) refers to techniques that enable operators to 1194 control how specific traffic flows are treated within their networks. 1196 Because if its very nature of establishing explicit optimized paths, 1197 Deterministic Networking can be seen as a new, specialized branch of 1198 Traffic Engineering, and inherits its architecture with a separation 1199 into planes. 1201 The Deterministic Networking architecture is thus composed of three 1202 planes, a (User) Application Plane, a Controller Plane, and a Network 1203 Plane, which echoes that of Figure 1 of Software-Defined Networking 1204 (SDN): Layers and Architecture Terminology [RFC7426], and the 1205 Controllers identified in [RFC8453] and [RFC7149]. 1207 4.4.1. The Application Plane 1209 Per [RFC7426], the Application Plane includes both applications and 1210 services. In particular, the Application Plane incorporates the User 1211 Agent, a specialized application that interacts with the end user / 1212 operator and performs requests for Deterministic Networking services 1213 via an abstract Flow Management Entity, (FME) which may or may not be 1214 collocated with (one of) the end systems. 1216 At the Application Plane, a management interface enables the 1217 negotiation of flows between end systems. An abstraction of the flow 1218 called a Traffic Specification (TSpec) provides the representation. 1219 This abstraction is used to place a reservation over the (Northbound) 1220 Service Interface and within the Application plane. It is associated 1221 with an abstraction of location, such as IP addresses and DNS names, 1222 to identify the end systems and possibly specify DetNet nodes. 1224 4.4.2. The Controller Plane 1226 The Controller Plane corresponds to the aggregation of the Control 1227 and Management Planes in [RFC7426], though Common Control and 1228 Measurement Plane (CCAMP) [CCAMP] makes an additional distinction 1229 between management and measurement. When the logical separation of 1230 the Control, Measurement and other Management entities is not 1231 relevant, the term Controller Plane is used for simplicity to 1232 represent them all, and the term Controller Plane Function (CPF) 1233 refers to any device operating in that plane, whether is it a Path 1234 Computation Element (PCE) [RFC4655], or a Network Management entity 1235 (NME), or a distributed control plane. The CPF is a core element of 1236 a controller, in charge of computing Deterministic paths to be 1237 applied in the Network Plane. 1239 A (Northbound) Service Interface enables applications in the 1240 Application Plane to communicate with the entities in the Controller 1241 Plane as illustrated in Figure 7. 1243 One or more CPF(s) collaborate to implement the requests from the FME 1244 as Per-Flow Per-Hop Behaviors installed in the DetNet nodes for each 1245 individual flow. The CPFs place each flow along a deterministic 1246 sequence of DetNet nodes so as to respect per-flow constraints such 1247 as security and latency, and optimize the overall result for metrics 1248 such as an abstract aggregated cost. The deterministic sequence can 1249 typically be more complex than a direct sequence and include 1250 redundancy path, with one or more packet replication and elimination 1251 points. Scaling to larger networks is discussed in Section 4.9. 1253 4.4.3. The Network Plane 1255 The Network Plane represents the network devices and protocols as a 1256 whole, regardless of the Layer at which the network devices operate. 1257 It includes Forwarding Plane (data plane), Application, and 1258 Operational Plane (e.g., OAM) aspects. 1260 The network Plane comprises the Network Interface Cards (NIC) in the 1261 end systems, which are typically IP hosts, and DetNet nodes, which 1262 are typically IP routers and MPLS switches. Network-to-Network 1263 Interfaces such as used for Traffic Engineering path reservation in 1264 [RFC5921], as well as User-to-Network Interfaces (UNI) such as 1265 provided by the Local Management Interface (LMI) between network and 1266 end systems, are both part of the Network Plane, both in the control 1267 plane and the data plane. 1269 A Southbound (Network) Interface enables the entities in the 1270 Controller Plane to communicate with devices in the Network Plane as 1271 illustrated in Figure 7. This interface leverages and extends TEAS 1272 to describe the physical topology and resources in the Network Plane. 1274 End End 1275 System System 1277 -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 1279 CPF CPF CPF CPF 1281 -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 1283 DetNet DetNet DetNet DetNet 1284 Node Node Node Node 1285 NIC NIC 1286 DetNet DetNet DetNet DetNet 1287 Node Node Node Node 1289 Figure 7: Northbound and Southbound interfaces 1291 The DetNet nodes (and possibly the end systems NIC) expose their 1292 capabilities and physical resources to the controller (the CPF), and 1293 update the CPFs with their dynamic perception of the topology, across 1294 the Southbound Interface. In return, the CPFs set the per-flow paths 1295 up, providing a Flow Characterization that is more tightly coupled to 1296 the DetNet node Operation than a TSpec. 1298 At the Network plane, DetNet nodes may exchange information regarding 1299 the state of the paths, between adjacent DetNet nodes and possibly 1300 with the end systems, and forward packets within constraints 1301 associated to each flow, or, when unable to do so, perform a last 1302 resort operation such as drop or declassify. 1304 This document focuses on the Southbound interface and the operation 1305 of the Network Plane. 1307 4.5. Queuing, Shaping, Scheduling, and Preemption 1309 DetNet achieves bounded delivery latency by reserving bandwidth and 1310 buffer resources at each DetNet node along the path of the DetNet 1311 flow. The reservation itself is not sufficient, however. 1312 Implementors and users of a number of proprietary and standard real- 1313 time networks have found that standards for specific data plane 1314 techniques are required to enable these assurances to be made in a 1315 multi-vendor network. The fundamental reason is that latency 1316 variation in one DetNet system results in the need for extra buffer 1317 space in the next-hop DetNet system(s), which in turn, increases the 1318 worst-case per-hop latency. 1320 Standard queuing and transmission selection algorithms allow traffic 1321 engineering Section 4.4 to compute the latency contribution of each 1322 DetNet node to the end-to-end latency, to compute the amount of 1323 buffer space required in each DetNet node for each incremental DetNet 1324 flow, and most importantly, to translate from a flow specification to 1325 a set of values for the managed objects that control each relay or 1326 end system. For example, the IEEE 802.1 WG has specified (and is 1327 specifying) a set of queuing, shaping, and scheduling algorithms that 1328 enable each DetNet node, and/or a central controller, to compute 1329 these values. These algorithms include: 1331 o A credit-based shaper [IEEE802.1Qav] (superseded by 1332 [IEEE802.1Q-2018]). 1334 o Time-gated queues governed by a rotating time schedule based on 1335 synchronized time [IEEE802.1Qbv] (superseded by 1336 [IEEE802.1Q-2018]). 1338 o Synchronized double (or triple) buffers driven by synchronized 1339 time ticks. [IEEE802.1Qch] (superseded by [IEEE802.1Q-2018]). 1341 o Pre-emption of an Ethernet packet in transmission by a packet with 1342 a more stringent latency requirement, followed by the resumption 1343 of the preempted packet [IEEE802.1Qbu] (superseded by 1344 [IEEE802.1Q-2018]), [IEEE802.3br] (superseded by 1345 [IEEE802.3-2018]). 1347 While these techniques are currently embedded in Ethernet 1348 [IEEE802.3-2018] and bridging standards, we can note that they are 1349 all, except perhaps for packet preemption, equally applicable to 1350 other media than Ethernet, and to routers as well as bridges. Other 1351 media may have its own methods, see, e.g., 1352 [I-D.ietf-6tisch-architecture], [RFC7554]. DetNet may include such 1353 definitions in the future, or may define how these techniques can be 1354 used by DetNet nodes. 1356 4.6. Service instance 1358 A Service instance represents all the functions required on a DetNet 1359 node to allow the end-to-end service between the UNIs. 1361 The DetNet network general reference model is shown in Figure 8 for a 1362 DetNet service scenario (i.e., between two DetNet-UNIs). In this 1363 figure, end systems ("A" and "B") are connected directly to the edge 1364 nodes of an IP/MPLS network ("PE1" and "PE2"). End systems 1365 participating in DetNet communication may require connectivity before 1366 setting up an App-flow that requires the DetNet service. Such a 1367 connectivity related service instance and the one dedicated for 1368 DetNet service share the same access. Packets belonging to a DetNet 1369 flow are selected by a filter configured on the access ("F1" and 1370 "F2"). As a result, data flow specific access ("access-A + F1" and 1371 "access-B + F2") are terminated in the flow specific service instance 1372 ("SI-1" and "SI-2"). A tunnel is used to provide connectivity 1373 between the service instances. 1375 The tunnel is exclusively used for the packets of the DetNet flow 1376 between "SI-1" and "SI-2". The service instances are configured to 1377 implement DetNet functions and a flow specific DetNet forwarding. 1378 The service instance and the tunnel may or may not be shared by 1379 multiple DetNet flows. Sharing the service instance by multiple 1380 DetNet flows requires properly populated forwarding tables of the 1381 service instance. 1383 access-A access-B 1384 <-----> <-------- tunnel ----------> <-----> 1386 +---------+ ___ _ +---------+ 1387 End system | +----+ | / \/ \_ | +----+ | End system 1388 "A" -------F1+ | | / \ | | +F2----- "B" 1389 | | +========+ IP/MPLS +=======+ | | 1390 | |SI-1| | \__ Net._/ | |SI-2| | 1391 | +----+ | \____/ | +----+ | 1392 |PE1 | | PE2| 1393 +---------+ +---------+ 1395 Figure 8: DetNet network general reference model 1397 The tunnel between the service instances may have some special 1398 characteristics. For example, in case of a DetNet L3 service, there 1399 are differences in the usage of the PW for DetNet traffic compared to 1400 the network model described in [RFC6658]. In the DetNet scenario, 1401 the PW is likely to be used exclusively by the DetNet flow, whereas 1402 [RFC6658] states: "The packet PW appears as a single point-to-point 1403 link to the client layer. Network-layer adjacency formation and 1404 maintenance between the client equipment will follow the normal 1405 practice needed to support the required relationship in the client 1406 layer ... This packet PseudoWire is used to transport all of the 1407 required Layer-2 and Layer-3 protocols between LSR1 and LSR2". 1408 Further details are network technology specific and can be found in 1409 [I-D.ietf-detnet-dp-sol-mpls] and [I-D.ietf-detnet-dp-sol-ip]. 1411 4.7. Flow identification at technology borders 1413 4.7.1. Exporting flow identification 1415 A DetNet node may need to map specific flows to lower layer flows (or 1416 Streams) in order to provide specific queuing and shaping services 1417 for specific flows. For example: 1419 o A non-IP, strictly L2 source end system X may be sending multiple 1420 flows to the same L2 destination end system Y. Those flows may 1421 include DetNet flows with different QoS requirements, and may 1422 include non-DetNet flows. 1424 o A router may be sending any number of flows to another router. 1425 Again, those flows may include DetNet flows with different QoS 1426 requirements, and may include non-DetNet flows. 1428 o Two routers may be separated by bridges. For these bridges to 1429 perform any required per-flow queuing and shaping, they must be 1430 able to identify the individual flows. 1432 o A Label Edge Router (LER) may have a Label Switched Path (LSP) set 1433 up for handling traffic destined for a particular IP address 1434 carrying only non-DetNet flows. If a DetNet flow to that same 1435 address is requested, a separate LSP may be needed, in order that 1436 all of the Label Switch Routers (LSRs) along the path to the 1437 destination give that flow special queuing and shaping. 1439 The need for a lower-layer node to be aware of individual higher- 1440 layer flows is not unique to DetNet. But, given the endless 1441 complexity of layering and relayering over tunnels that is available 1442 to network designers, DetNet needs to provide a model for flow 1443 identification that is better than packet inspection. That is not to 1444 say that packet inspection to Layer-4 or Layer-5 addresses will not 1445 be used, or the capability standardized; but, there are alternatives. 1447 A DetNet relay node can connect DetNet flows on different paths using 1448 different flow identification methods. For example: 1450 o A single unicast DetNet flow passing from router A through a 1451 bridged network to router B may be assigned a TSN Stream 1452 identifier that is unique within that bridged network. The 1453 bridges can then identify the flow without accessing higher-layer 1454 headers. Of course, the receiving router must recognize and 1455 accept that TSN Stream. 1457 o A DetNet flow passing from LSR A to LSR B may be assigned a 1458 different label than that used for other flows to the same IP 1459 destination. 1461 In any of the above cases, it is possible that an existing DetNet 1462 flow can be an aggregate carrying multiple other DetNet flows. (Not 1463 to be confused with DetNet compound vs. member flows.) Of course, 1464 this requires that the aggregate DetNet flow be provisioned properly 1465 to carry the aggregated flows. 1467 Thus, rather than packet inspection, there is the option to export 1468 higher-layer information to the lower layer. The requirement to 1469 support one or the other method for flow identification (or both) is 1470 a complexity that is part of DetNet control models. 1472 4.7.2. Flow attribute mapping between layers 1474 Forwarding of packets of DetNet flows over multiple technology 1475 domains may require that lower layers are aware of specific flows of 1476 higher layers. Such an "exporting of flow identification" is needed 1477 each time when the forwarding paradigm is changed on the forwarding 1478 path (e.g., two LSRs are interconnected by a L2 bridged domain, 1479 etc.). The three representative forwarding methods considered for 1480 deterministic networking are: 1482 o IP routing 1484 o MPLS label switching 1486 o Ethernet bridging 1488 A packet with corresponding Flow-IDs is illustrated in Figure 9, 1489 which also indicates where each Flow-ID can be added or removed. 1491 add/remove add/remove 1492 Eth Flow-ID IP Flow-ID 1493 | | 1494 v v 1495 +-----------------------------------------------------------+ 1496 | | | | | 1497 | Eth | MPLS | IP | Application data | 1498 | | | | | 1499 +-----------------------------------------------------------+ 1500 ^ 1501 | 1502 add/remove 1503 MPLS Flow-ID 1505 Figure 9: Packet with multiple Flow-IDs 1507 The additional (domain specific) Flow-ID can be 1509 o created by a domain specific function or 1511 o derived from the Flow-ID added to the App-flow. 1513 The Flow-ID must be unique inside a given domain. Note that the 1514 Flow-ID added to the App-flow is still present in the packet, but 1515 some nodes may lack the function to recognize it; that's why the 1516 additional Flow-ID is added. 1518 4.7.3. Flow-ID mapping examples 1520 IP nodes and MPLS nodes are assumed to be configured to push such an 1521 additional (domain specific) Flow-ID when sending traffic to an 1522 Ethernet switch (as shown in the examples below). 1524 Figure 10 shows a scenario where an IP end system ("IP-A") is 1525 connected via two Ethernet switches ("ETH-n") to an IP router ("IP- 1526 1"). 1528 IP domain 1529 <----------------------------------------------- 1531 +======+ +======+ 1532 |L3-ID | |L3-ID | 1533 +======+ /\ +-----+ +======+ 1534 / \ Forward as | | 1535 /IP-A\ per ETH-ID |IP-1 | Recognize 1536 Push ------> +-+----+ | +---+-+ <----- ETH-ID 1537 ETH-ID | +----+-----+ | 1538 | v v | 1539 | +-----+ +-----+ | 1540 +------+ | | +---------+ 1541 +......+ |ETH-1+----+ETH-2| +======+ 1542 .L3-ID . +-----+ +-----+ |L3-ID | 1543 +======+ +......+ +======+ 1544 |ETH-ID| .L3-ID . |ETH-ID| 1545 +======+ +======+ +------+ 1546 |ETH-ID| 1547 +======+ 1549 Ethernet domain 1550 <----------------> 1552 Figure 10: IP nodes interconnected by an Ethernet domain 1554 End system "IP-A" uses the original App-flow specific ID ("L3-ID"), 1555 but as it is connected to an Ethernet domain it has to push an 1556 Ethernet-domain specific flow-ID ("ETH-ID") before sending the packet 1557 to "ETH-1" node. Ethernet switch "ETH-1" can recognize the data flow 1558 based on the "ETH-ID" and it does forwarding toward "ETH-2". "ETH-2" 1559 switches the packet toward the IP router. "IP-1" must be configured 1560 to receive the Ethernet Flow-ID specific multicast flow, but (as it 1561 is an L3 node) it decodes the data flow ID based on the "L3-ID" 1562 fields of the received packet. 1564 Figure 11 shows a scenario where MPLS domain nodes ("PE-n" and "P-m") 1565 are connected via two Ethernet switches ("ETH-n"). 1567 MPLS domain 1568 <-----------------------------------------------> 1570 +=======+ +=======+ 1571 |MPLS-ID| |MPLS-ID| 1572 +=======+ +-----+ +-----+ +=======+ +-----+ 1573 | | Forward as | | | | 1574 |PE-1 | per ETH-ID | P-2 +-----------+ PE-2| 1575 Push -----> +-+---+ | +---+-+ +-----+ 1576 ETH-ID | +-----+----+ | \ Recognize 1577 | v v | +-- ETH-ID 1578 | +-----+ +-----+ | 1579 +---+ | | +----+ 1580 +.......+ |ETH-1+----+ETH-2| +=======+ 1581 .MPLS-ID. +-----+ +-----+ |MPLS-ID| 1582 +=======+ +=======+ 1583 |ETH-ID | +.......+ |ETH-ID | 1584 +=======+ .MPLS-ID. +-------+ 1585 +=======+ 1586 |ETH-ID | 1587 +=======+ 1588 Ethernet domain 1589 <----------------> 1591 Figure 11: MPLS nodes interconnected by an Ethernet domain 1593 "PE-1" uses the MPLS specific ID ("MPLS-ID"), but as it is connected 1594 to an Ethernet domain it has to push an Ethernet-domain specific 1595 flow-ID ("ETH-ID") before sending the packet to "ETH-1". Ethernet 1596 switch "ETH-1" can recognize the data flow based on the "ETH-ID" and 1597 it does forwarding toward "ETH-2". "ETH-2" switches the packet 1598 toward the MPLS node ("P-2"). "P-2" must be configured to receive 1599 the Ethernet Flow-ID specific multicast flow, but (as it is an MPLS 1600 node) it decodes the data flow ID based on the "MPLS-ID" fields of 1601 the received packet. 1603 One can appreciate from the above example that, when the means used 1604 for DetNet flow identification is altered or exported, the means for 1605 encoding the sequence number information must similarly be altered or 1606 exported. 1608 4.8. Advertising resources, capabilities and adjacencies 1610 Provisioning of DetNet requires knowledge about: 1612 o Details of the DetNet system's capabilities that are required in 1613 order to accurately allocate that DetNet system's resources, as 1614 well as other DetNet systems' resources. This includes, for 1615 example, which specific queuing and shaping algorithms are 1616 implemented (Section 4.5), the number of buffers dedicated for 1617 DetNet allocation, and the worst-case forwarding delay and 1618 misordering. 1620 o The dynamic state of a DetNet node's DetNet resources. 1622 o The identity of the DetNet system's neighbors, and the 1623 characteristics of the link(s) between the DetNet systems, 1624 including the latency of the links (in nanoseconds). 1626 4.9. Scaling to larger networks 1628 Reservations for individual DetNet flows require considerable state 1629 information in each DetNet node, especially when adequate fault 1630 mitigation (Section 3.3.2) is required. The DetNet data plane, in 1631 order to support larger numbers of DetNet flows, must support the 1632 aggregation of DetNet flows. Such aggregated flows can be viewed by 1633 the DetNet nodes' data plane largely as individual DetNet flows. 1634 Without such aggregation, the per-relay system may limit the scale of 1635 DetNet networks. Example techniques that may be used include MPLS 1636 hierarchy and IP DiffServ Code Points (DSCPs). 1638 4.10. Compatibility with Layer-2 1640 Standards providing similar capabilities for bridged networks (only) 1641 have been and are being generated in the IEEE 802 LAN/MAN Standards 1642 Committee. The present architecture describes an abstract model that 1643 can be applicable both at Layer-2 and Layer-3, and over links not 1644 defined by IEEE 802. 1646 DetNet enabled end systems and DetNet nodes can be interconnected by 1647 sub-networks, i.e., Layer-2 technologies. These sub-networks will 1648 provide DetNet compatible service for support of DetNet traffic. 1649 Examples of sub-networks include MPLS TE, 802.1 TSN, and a point-to- 1650 point OTN link. Of course, multi-layer DetNet systems may be 1651 possible too, where one DetNet appears as a sub-network, and provides 1652 service to, a higher layer DetNet system. 1654 5. Security Considerations 1656 Security in the context of Deterministic Networking has an added 1657 dimension; the time of delivery of a packet can be just as important 1658 as the contents of the packet, itself. A man-in-the-middle attack, 1659 for example, can impose, and then systematically adjust, additional 1660 delays into a link, and thus disrupt or subvert a real-time 1661 application without having to crack any encryption methods employed. 1662 See [RFC7384] for an exploration of this issue in a related context. 1664 Furthermore, in a control system where millions of dollars of 1665 equipment, or even human lives, can be lost if the DetNet QoS is not 1666 delivered, one must consider not only simple equipment failures, 1667 where the box or wire instantly becomes perfectly silent, but complex 1668 errors such as can be caused by software failures. Because there is 1669 essential no limit to the kinds of failures that can occur, 1670 protecting against realistic equipment failures is indistinguishable, 1671 in most cases, from protecting against malicious behavior, whether 1672 accidental or intentional. See also Section 3.3.2. 1674 Security must cover: 1676 o the protection of the signaling protocol 1678 o the authentication and authorization of the controlling systems 1680 o the identification and shaping of the DetNet flows 1682 Security considerations for DetNet are described in detail in 1683 [I-D.ietf-detnet-security]. 1685 6. Privacy Considerations 1687 DetNet is provides a Quality of Service (QoS), and as such, does not 1688 directly raise any new privacy considerations. 1690 However, the requirement for every (or almost every) node along the 1691 path of a DetNet flow to identify DetNet flows may present an 1692 additional attack surface for privacy, should the DetNet paradigm be 1693 found useful in broader environments. 1695 7. IANA Considerations 1697 This document does not require an action from IANA. 1699 8. Acknowledgements 1701 The authors wish to thank Lou Berger, David Black, Stewart Bryant, 1702 Rodney Cummings, Ethan Grossman, Craig Gunther, Marcel Kiessling, 1703 Rudy Klecka, Jouni Korhonen, Erik Nordmark, Shitanshu Shah, Wilfried 1704 Steiner, George Swallow, Michael Johas Teener, Pat Thaler, Thomas 1705 Watteyne, Patrick Wetterwald, Karl Weber, Anca Zamfir, for their 1706 various contribution with this work. 1708 9. Informative References 1710 [CCAMP] IETF, "Common Control and Measurement Plane Working 1711 Group", 1712 . 1714 [I-D.ietf-6tisch-architecture] 1715 Thubert, P., "An Architecture for IPv6 over the TSCH mode 1716 of IEEE 802.15.4", draft-ietf-6tisch-architecture-19 (work 1717 in progress), December 2018. 1719 [I-D.ietf-detnet-dp-sol-ip] 1720 Korhonen, J. and B. Varga, "DetNet IP Data Plane 1721 Encapsulation", draft-ietf-detnet-dp-sol-ip-01 (work in 1722 progress), October 2018. 1724 [I-D.ietf-detnet-dp-sol-mpls] 1725 Korhonen, J. and B. Varga, "DetNet MPLS Data Plane 1726 Encapsulation", draft-ietf-detnet-dp-sol-mpls-01 (work in 1727 progress), October 2018. 1729 [I-D.ietf-detnet-problem-statement] 1730 Finn, N. and P. Thubert, "Deterministic Networking Problem 1731 Statement", draft-ietf-detnet-problem-statement-08 (work 1732 in progress), December 2018. 1734 [I-D.ietf-detnet-security] 1735 Mizrahi, T., Grossman, E., Hacker, A., Das, S., Dowdell, 1736 J., Austad, H., Stanton, K., and N. Finn, "Deterministic 1737 Networking (DetNet) Security Considerations", draft-ietf- 1738 detnet-security-03 (work in progress), October 2018. 1740 [I-D.ietf-detnet-use-cases] 1741 Grossman, E., "Deterministic Networking Use Cases", draft- 1742 ietf-detnet-use-cases-19 (work in progress), October 2018. 1744 [IEC62439-3-2016] 1745 International Electrotechnical Commission (IEC) TC 65/SC 1746 65C - Industrial networks, "IEC 62439-3:2016 Industrial 1747 communication networks - High availability automation 1748 networks - Part 3: Parallel Redundancy Protocol (PRP) and 1749 High-availability Seamless Redundancy (HSR)", 2016, 1750 . 1752 [IEEE802.1BA] 1753 IEEE Standards Association, "IEEE Std 802.1BA-2011 Audio 1754 Video Bridging (AVB) Systems", 2011, 1755 . 1757 [IEEE802.1CB] 1758 IEEE Standards Association, "IEEE Std 802.1CB-2017 Frame 1759 Replication and Elimination for Reliability", 2017, 1760 . 1762 [IEEE802.1Q-2018] 1763 IEEE Standards Association, "IEEE Std 802.1Q-2018 Bridges 1764 and Bridged Networks", 2018, 1765 . 1767 [IEEE802.1Qav] 1768 IEEE Standards Association, "IEEE Std 802.1Qav-2009 1769 Bridges and Bridged Networks - Amendment 12: Forwarding 1770 and Queuing Enhancements for Time-Sensitive Streams", 1771 2009, . 1773 [IEEE802.1Qbu] 1774 IEEE Standards Association, "IEEE Std 802.1Qbu-2016 1775 Bridges and Bridged Networks - Amendment 26: Frame 1776 Preemption", 2016, 1777 . 1779 [IEEE802.1Qbv] 1780 IEEE Standards Association, "IEEE Std 802.1Qbv-2015 1781 Bridges and Bridged Networks - Amendment 25: Enhancements 1782 for Scheduled Traffic", 2015, 1783 . 1785 [IEEE802.1Qch] 1786 IEEE Standards Association, "IEEE Std 802.1Qch-2017 1787 Bridges and Bridged Networks - Amendment 29: Cyclic 1788 Queuing and Forwarding", 2017, 1789 . 1791 [IEEE802.1TSNTG] 1792 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1793 Networking Task Group", 2013, 1794 . 1796 [IEEE802.3-2018] 1797 IEEE Standards Association, "IEEE Std 802.3-2018 Standard 1798 for Ethernet", 2018, 1799 . 1801 [IEEE802.3br] 1802 IEEE Standards Association, "IEEE Std 802.3br-2016 1803 Standard for Ethernet Amendment 5: Specification and 1804 Management Parameters for Interspersing Express Traffic", 1805 2016, . 1807 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1808 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1809 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1810 September 1997, . 1812 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1813 and W. Weiss, "An Architecture for Differentiated 1814 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1815 . 1817 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1818 of Explicit Congestion Notification (ECN) to IP", 1819 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1820 . 1822 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1823 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1824 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1825 . 1827 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1828 Jacobson, "RTP: A Transport Protocol for Real-Time 1829 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 1830 July 2003, . 1832 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 1833 Element (PCE)-Based Architecture", RFC 4655, 1834 DOI 10.17487/RFC4655, August 2006, 1835 . 1837 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 1838 L., and L. Berger, "A Framework for MPLS in Transport 1839 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 1840 . 1842 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 1843 Profile (MPLS-TP) Survivability Framework", RFC 6372, 1844 DOI 10.17487/RFC6372, September 2011, 1845 . 1847 [RFC6658] Bryant, S., Ed., Martini, L., Swallow, G., and A. Malis, 1848 "Packet Pseudowire Encapsulation over an MPLS PSN", 1849 RFC 6658, DOI 10.17487/RFC6658, July 2012, 1850 . 1852 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 1853 Networking: A Perspective from within a Service Provider 1854 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 1855 . 1857 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1858 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1859 October 2014, . 1861 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1862 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1863 Defined Networking (SDN): Layers and Architecture 1864 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1865 2015, . 1867 [RFC7554] Watteyne, T., Ed., Palattella, M., and L. Grieco, "Using 1868 IEEE 802.15.4e Time-Slotted Channel Hopping (TSCH) in the 1869 Internet of Things (IoT): Problem Statement", RFC 7554, 1870 DOI 10.17487/RFC7554, May 2015, 1871 . 1873 [RFC7813] Farkas, J., Ed., Bragg, N., Unbehagen, P., Parsons, G., 1874 Ashwood-Smith, P., and C. Bowers, "IS-IS Path Control and 1875 Reservation", RFC 7813, DOI 10.17487/RFC7813, June 2016, 1876 . 1878 [RFC8227] Cheng, W., Wang, L., Li, H., van Helvoort, H., and J. 1879 Dong, "MPLS-TP Shared-Ring Protection (MSRP) Mechanism for 1880 Ring Topology", RFC 8227, DOI 10.17487/RFC8227, August 1881 2017, . 1883 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1884 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1885 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1886 July 2018, . 1888 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1889 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1890 DOI 10.17487/RFC8453, August 2018, 1891 . 1893 [TEAS] IETF, "Traffic Engineering Architecture and Signaling 1894 Working Group", 1895 . 1897 Authors' Addresses 1899 Norman Finn 1900 Huawei 1901 3101 Rio Way 1902 Spring Valley, California 91977 1903 US 1905 Phone: +1 925 980 6430 1906 Email: norman.finn@mail01.huawei.com 1908 Pascal Thubert 1909 Cisco Systems 1910 Village d'Entreprises Green Side 1911 400, Avenue de Roumanille 1912 Batiment T3 1913 Biot - Sophia Antipolis 06410 1914 FRANCE 1916 Phone: +33 4 97 23 26 34 1917 Email: pthubert@cisco.com 1919 Balazs Varga 1920 Ericsson 1921 Magyar tudosok korutja 11 1922 Budapest 1117 1923 Hungary 1925 Email: balazs.a.varga@ericsson.com 1927 Janos Farkas 1928 Ericsson 1929 Magyar tudosok korutja 11 1930 Budapest 1117 1931 Hungary 1933 Email: janos.farkas@ericsson.com