idnits 2.17.1 draft-ietf-detnet-architecture-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 729 has weird spacing: '...e stack v ...' == Line 1226 has weird spacing: '...mediate inter...' == Line 1229 has weird spacing: '...mediate inter...' -- The document date (June 28, 2018) is 2122 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Network' is mentioned on line 862, but not defined == Outdated reference: A later version (-30) exists of draft-ietf-6tisch-architecture-14 == Outdated reference: A later version (-09) exists of draft-ietf-detnet-problem-statement-05 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-17 Summary: 0 errors (**), 0 flaws (~~), 8 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft Huawei 4 Intended status: Standards Track P. Thubert 5 Expires: December 30, 2018 Cisco 6 B. Varga 7 J. Farkas 8 Ericsson 9 June 28, 2018 11 Deterministic Networking Architecture 12 draft-ietf-detnet-architecture-06 14 Abstract 16 Deterministic Networking (DetNet) provides a capability to carry 17 specified unicast or multicast data flows for real-time applications 18 with extremely low data loss rates and bounded latency. Techniques 19 used include: 1) reserving data plane resources for individual (or 20 aggregated) DetNet flows in some or all of the intermediate nodes 21 (e.g., bridges or routers) along the path of the flow; 2) providing 22 explicit routes for DetNet flows that do not immediately change with 23 the network topology; and 3) distributing data from DetNet flow 24 packets over time and/or space to ensure delivery of each packet's 25 data' in spite of the loss of a path. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at https://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on December 30, 2018. 44 Copyright Notice 46 Copyright (c) 2018 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (https://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 62 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 63 2.1. Terms used in this document . . . . . . . . . . . . . . . 4 64 2.2. IEEE 802.1 TSN to DetNet dictionary . . . . . . . . . . . 6 65 3. Providing the DetNet Quality of Service . . . . . . . . . . . 7 66 3.1. Primary goals defining the DetNet QoS . . . . . . . . . . 7 67 3.2. Mechanisms to achieve DetNet QoS . . . . . . . . . . . . 9 68 3.2.1. Congestion protection . . . . . . . . . . . . . . . . 9 69 3.2.1.1. Eliminate congestion loss . . . . . . . . . . . . 9 70 3.2.1.2. Jitter Reduction . . . . . . . . . . . . . . . . 10 71 3.2.2. Service Protection . . . . . . . . . . . . . . . . . 11 72 3.2.2.1. In-Order Delivery . . . . . . . . . . . . . . . . 11 73 3.2.2.2. Packet Replication and Elimination . . . . . . . 11 74 3.2.2.3. Packet encoding for service protection . . . . . 13 75 3.2.3. Explicit routes . . . . . . . . . . . . . . . . . . . 13 76 3.3. Secondary goals for DetNet . . . . . . . . . . . . . . . 14 77 3.3.1. Coexistence with normal traffic . . . . . . . . . . . 14 78 3.3.2. Fault Mitigation . . . . . . . . . . . . . . . . . . 15 79 4. DetNet Architecture . . . . . . . . . . . . . . . . . . . . . 16 80 4.1. DetNet stack model . . . . . . . . . . . . . . . . . . . 16 81 4.1.1. Representative Protocol Stack Model . . . . . . . . . 16 82 4.1.2. DetNet Data Plane Overview . . . . . . . . . . . . . 18 83 4.1.3. Network reference model . . . . . . . . . . . . . . . 20 84 4.2. DetNet systems . . . . . . . . . . . . . . . . . . . . . 21 85 4.2.1. End system . . . . . . . . . . . . . . . . . . . . . 21 86 4.2.2. DetNet edge, relay, and transit nodes . . . . . . . . 22 87 4.3. DetNet flows . . . . . . . . . . . . . . . . . . . . . . 23 88 4.3.1. DetNet flow types . . . . . . . . . . . . . . . . . . 23 89 4.3.2. Source transmission behavior . . . . . . . . . . . . 23 90 4.3.3. Incomplete Networks . . . . . . . . . . . . . . . . . 25 91 4.4. Traffic Engineering for DetNet . . . . . . . . . . . . . 25 92 4.4.1. The Application Plane . . . . . . . . . . . . . . . . 25 93 4.4.2. The Controller Plane . . . . . . . . . . . . . . . . 26 94 4.4.3. The Network Plane . . . . . . . . . . . . . . . . . . 26 95 4.5. Queuing, Shaping, Scheduling, and Preemption . . . . . . 27 96 4.6. Service instance . . . . . . . . . . . . . . . . . . . . 28 97 4.7. Flow identification at technology borders . . . . . . . . 29 98 4.7.1. Exporting flow identification . . . . . . . . . . . . 29 99 4.7.2. Flow attribute mapping between layers . . . . . . . . 31 100 4.7.3. Flow-ID mapping examples . . . . . . . . . . . . . . 32 101 4.8. Advertising resources, capabilities and adjacencies . . . 34 102 4.9. Scaling to larger networks . . . . . . . . . . . . . . . 34 103 4.10. Compatibility with Layer-2 . . . . . . . . . . . . . . . 34 104 5. Security Considerations . . . . . . . . . . . . . . . . . . . 35 105 6. Privacy Considerations . . . . . . . . . . . . . . . . . . . 35 106 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 107 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 35 108 9. Informative References . . . . . . . . . . . . . . . . . . . 36 109 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 111 1. Introduction 113 Deterministic Networking (DetNet) is a service that can be offered by 114 a network to DetNet flows. DetNet provides these flows with 115 extremely low packet loss rates and assured maximum end-to-end 116 delivery latency. This is accomplished by dedicating network 117 resources such as link bandwidth and buffer space to DetNet flows 118 and/or classes of DetNet flows, and by replicating packets along 119 multiple paths. Unused reserved resources are available to non- 120 DetNet packets. 122 The Deterministic Networking Problem Statement 123 [I-D.ietf-detnet-problem-statement] introduces Deterministic 124 Networking, and Deterministic Networking Use Cases 125 [I-D.ietf-detnet-use-cases] summarizes the need for it. See 126 [I-D.ietf-detnet-dp-sol-mpls] and [I-D.ietf-detnet-dp-sol-ip] for 127 specific techniques that can be used to identify DetNet Flows and 128 assign them to specific paths through a network. 130 A goal of DetNet is a converged network in all respects. That is, 131 the presence of DetNet flows does not preclude non-DetNet flows, and 132 the benefits offered DetNet flows should not, except in extreme 133 cases, prevent existing QoS mechanisms from operating in a normal 134 fashion, subject to the bandwidth required for the DetNet flows. A 135 single source-destination pair can trade both DetNet and non-DetNet 136 flows. End systems and applications need not instantiate special 137 interfaces for DetNet flows. Networks are not restricted to certain 138 topologies; connectivity is not restricted. Any application that 139 generates a data flow that can be usefully characterized as having a 140 maximum bandwidth should be able to take advantage of DetNet, as long 141 as the necessary resources can be reserved. Reservations can be made 142 by the application itself, via network management, by an applications 143 controller, or by other means, e.g., a dynamic control plane (e.g., 144 [RFC2205]). 146 Many applications, that are intended to be served by Deterministic 147 Networking, require the ability to synchronize the clocks in end 148 systems to a sub-microsecond accuracy. Some of the queue control 149 techniques defined in Section 4.5 also require time synchronization 150 among relay and transit nodes. The means used to achieve time 151 synchronization are not addressed in this document. DetNet should 152 accommodate various synchronization techniques and profiles that are 153 defined elsewhere to solve exchange time in different market 154 segments. 156 2. Terminology 158 2.1. Terms used in this document 160 The following terms are used in the context of DetNet in this 161 document: 163 allocation 164 Resources are dedicated to support a DetNet flow. Depending 165 on an implementation, the resource may be reused by non- 166 DetNet flows when it is not used by the DetNet flow. 168 App-flow 169 The native format of a DetNet flow. 171 DetNet destination 172 An end system capable of terminating a DetNet flow. 174 DetNet domain 175 The portion of a network that is DetNet aware. It includes 176 end systems and other DetNet nodes. 178 DetNet flow 179 A DetNet flow is a sequence of packets to which the DetNet 180 service is to be provided. 182 DetNet compound flow and DetNet member flow 183 A DetNet compound flow is a DetNet flow that has been 184 separated into multiple duplicate DetNet member flows for 185 service protection at the DetNet service layer. Member flows 186 are merged back into a single DetNet compound flow such that 187 there are no duplicate packets. "Compound" and "member" are 188 strictly relative to each other, not absolutes; a DetNet 189 compound flow comprising multiple DetNet member flows can, in 190 turn, be a member of a higher-order compound. 192 DetNet intermediate node 193 A DetNet relay node or transit node. 195 DetNet edge node 196 An instance of a DetNet relay node that acts as a source and/ 197 or destination at the DetNet service layer. For example, it 198 can include a DetNet service layer proxy function for DetNet 199 service protection (e.g., the addition or removal of packet 200 sequencing information) for one or more end systems, or 201 starts or terminates congestion protection at the DetNet 202 transport layer, or aggregates DetNet services into new 203 DetNet flows. It is analogous to a Label Edge Router (LER) 204 or a Provider Edge (PE) router. 206 DetNet-UNI 207 User-to-Network Interface with DetNet specific 208 functionalities. It is a packet-based reference point and 209 may provide multiple functions like encapsulation, status, 210 synchronization, etc. 212 end system 213 Commonly called a "host" in IETF documents, and an "end 214 station" is IEEE 802 documents. End systems of interest to 215 this document are either sources or destinations of DetNet 216 flows. And end system may or may not be DetNet transport 217 layer aware or DetNet service layer aware. 219 link 220 A connection between two DetNet nodes. It may be composed of 221 a physical link or a sub-network technology that can provide 222 appropriate traffic delivery for DetNet flows. 224 DetNet system 225 A DetNet aware end system, transit node, or relay node. 226 "DetNet" may be omitted in some text. 228 DetNet relay node 229 A DetNet node including a service layer function that 230 interconnects different DetNet transport layer paths to 231 provide service protection. A DetNet relay node can be a 232 bridge, a router, a firewall, or any other system that 233 participates in the DetNet service layer. It typically 234 incorporates DetNet transport layer functions as well, in 235 which case it is collocated with a transit node. 237 PEF A Packet Elimination Function (PEF) eliminates duplicate 238 copies of packets to prevent excess packets flooding the 239 network or duplicate packets being sent out of the DetNet 240 domain. PEF can be implemented by an edge node, a relay 241 node, or an end system. 243 PRF A Packet Replication Function (PRF) replicates DetNet flow 244 packets and forwards them to one or more next hops in the 245 DetNet domain. The number of packet copies sent to each next 246 hop is a DetNet flow specific parameter at the node doing the 247 replication. PRF can be implemented by an edge node, a relay 248 node, or an end system. 250 PREOF Collective name for Packet Replication, Elimination, and 251 Ordering Functions. 253 POF A Packet Ordering Function (POF) re-orders packets within a 254 DetNet flow that are received out of order. This function 255 can be implemented by an edge node, a relay node, or an end 256 system. 258 reservation 259 The set of resources allocated between a source and one or 260 more destinations through transit nodes and subnets 261 associated with a DetNet flow, to provide the expected DetNet 262 Service. 264 DetNet service layer 265 The layer at which A DetNet Service, such as congestion or 266 service protection is provided. 268 DetNet service proxy 269 Maps between App-flows and DetNet flows. 271 DetNet source 272 An end system capable of originating a DetNet flow. 274 DetNet transit node 275 A node operating at the DetNet transport layer, that utilizes 276 link layer and/or network layer switching across multiple 277 links and/or sub-networks to provide paths for DetNet service 278 layer functions. Typically provides congestion protection 279 over those paths. An MPLS LSR is an example of a DetNet 280 transit node. 282 DetNet transport layer 283 The layer that optionally provides congestion protection for 284 DetNet flows over paths provided by the underlying network. 286 2.2. IEEE 802.1 TSN to DetNet dictionary 288 This section also serves as a dictionary for translating from the 289 terms used by the Time-Sensitive Networking (TSN) Task Group 290 [IEEE802.1TSNTG] of the IEEE 802.1 WG to those of the DetNet WG. 292 Listener 293 The IEEE 802.1 term for a destination of a DetNet flow. 295 relay system 296 The IEEE 802.1 term for a DetNet intermediate node. 298 Stream 299 The IEEE 802.1 term for a DetNet flow. 301 Talker 302 The IEEE 802.1 term for the source of a DetNet flow. 304 bridged path 305 A VLAN bridge uses the VLAN ID and the destination MAC 306 address to select the outbound port hence the path for a 307 frame. 309 3. Providing the DetNet Quality of Service 311 3.1. Primary goals defining the DetNet QoS 313 The DetNet Quality of Service can be expressed in terms of: 315 o Minimum and maximum end-to-end latency from source to destination; 316 timely delivery, and bounded jitter (packet delay variation) 317 derived from these constraints. 319 o Probability of loss of a packet, under various assumptions as to 320 the operational states of the nodes and links. If packet 321 replication is used to reduce the probability of packet loss, then 322 a related property is the probability (may be zero) of delivery of 323 a duplicate packet. Duplicate packet delivery is an inherent risk 324 in highly reliable and/or broadcast transmissions. 326 o An upper bound on out-of-order packet delivery. It is worth 327 noting that some DetNet applications are unable to tolerate any 328 out-of-order delivery. 330 It is a distinction of DetNet that it is concerned solely with worst- 331 case values for the end-to-end latency, jitter, and misordering. 332 Average, mean, or typical values are of little interest, because they 333 do not affect the ability of a real-time system to perform its tasks. 334 In general, a trivial priority-based queuing scheme will give better 335 average latency to a data flow than DetNet, but of course, the worst- 336 case latency can be essentially unbounded. 338 Three techniques are used by DetNet to provide these qualities of 339 service: 341 o Congestion protection (Section 3.2.1). 343 o Service protection (Section 3.2.2). 345 o Explicit routes (Section 3.2.3). 347 Congestion protection operates by allocating resources along the path 348 of a DetNet Flow, e.g., buffer space or link bandwidth. Congestion 349 protection greatly reduces, or even eliminates entirely, packet loss 350 due to output packet congestion within the network, but it can only 351 be supplied to a DetNet flow that is limited at the source to a 352 maximum packet size and transmission rate. 354 Congestion protection addresses two of the DetNet QoS requirements: 355 latency and packet loss. Given that DetNet nodes have a finite 356 amount of buffer space, congestion protection necessarily results in 357 a maximum end-to-end latency. It also addresses the largest 358 contribution to packet loss, which is buffer congestion. 360 After congestion, the most important contributions to packet loss are 361 typically from random media errors and equipment failures. Service 362 protection is the name for the mechanisms used by DetNet to address 363 these losses. The mechanisms employed are constrained by the 364 requirement to meet the users' latency requirements. Packet 365 replication and elimination (Section 3.2.2) and packet encoding 366 (Section 3.2.2.3) are described in this document to provide service 367 protection; others may be found. For instance, packet encoding can 368 be used to provide service protection against random media errors, 369 packet replication and elimination can be used to provide service 370 protection against equipment failures. This mechanism distributes 371 the contents of DetNet flows over multiple paths in time and/or 372 space, so that the loss of some of the paths does need not cause the 373 loss of any packets. 375 The paths are typically (but not necessarily) explicit routes, so 376 that they do not normally suffer temporary interruptions caused by 377 the convergence of routing or bridging protocols. 379 These three techniques can be applied independently, giving eight 380 possible combinations, including none (no DetNet), although some 381 combinations are of wider utility than others. This separation keeps 382 the protocol stack coherent and maximizes interoperability with 383 existing and developing standards in this (IETF) and other Standards 384 Development Organizations. Some examples of typical expected 385 combinations: 387 o Explicit routes plus service protection are exactly the techniques 388 employed by seamless redundancy mechanisms applied on a ring 389 topology as described, e.g., in [IEEE802.1CB]. In this case, 390 explicit routes are achieved by limiting the physical topology of 391 the network to a ring. Sequentialization, replication, and 392 duplicate elimination are facilitated by packet tags added at the 393 front or the end of Ethernet frames. 395 o Congestion protection alone is offered by IEEE 802.1 Audio Video 396 bridging [IEEE802.1BA]. As long as the network suffers no 397 failures, zero congestion loss can be achieved through the use of 398 a reservation protocol (MSRP [IEEE802.1Q]), shapers in every 399 bridge, and proper dimensioning. 401 o Using all three together gives maximum protection. 403 There are, of course, simpler methods available (and employed, today) 404 to achieve levels of latency and packet loss that are satisfactory 405 for many applications. Prioritization and over-provisioning is one 406 such technique. However, these methods generally work best in the 407 absence of any significant amount of non-critical traffic in the 408 network (if, indeed, such traffic is supported at all), or work only 409 if the critical traffic constitutes only a small portion of the 410 network's theoretical capacity, or work only if all systems are 411 functioning properly, or in the absence of actions by end systems 412 that disrupt the network's operations. 414 There are any number of methods in use, defined, or in progress for 415 accomplishing each of the above techniques. It is expected that this 416 DetNet Architecture will assist various vendors, users, and/or 417 "vertical" Standards Development Organizations (dedicated to a single 418 industry) to make selections among the available means of 419 implementing DetNet networks. 421 3.2. Mechanisms to achieve DetNet QoS 423 3.2.1. Congestion protection 425 3.2.1.1. Eliminate congestion loss 427 The primary means by which DetNet achieves its QoS assurances is to 428 reduce, or even completely eliminate, congestion within a node as a 429 cause of packet loss. Given that a DetNet flow cannot be throttled, 430 this can be achieved only by the provision of sufficient buffer 431 storage at each hop through the network to ensure that no packets are 432 dropped due to a lack of buffer storage. 434 Ensuring adequate buffering requires, in turn, that the source, and 435 every intermediate node along the path to the destination (or nearly 436 every node, see Section 4.3.3) be careful to regulate its output to 437 not exceed the data rate for any DetNet flow, except for brief 438 periods when making up for interfering traffic. Any packet sent 439 ahead of its time potentially adds to the number of buffers required 440 by the next hop and may thus exceed the resources allocated for a 441 particular DetNet flow. 443 The low-level mechanisms described in Section 4.5 provide the 444 necessary regulation of transmissions by an end system or 445 intermediate node to provide congestion protection. The allocation 446 of the bandwidth and buffers for a DetNet flow requires provisioning 447 A DetNet node may have other resources requiring allocation and/or 448 scheduling, that might otherwise be over-subscribed and trigger the 449 rejection of a reservation. 451 3.2.1.2. Jitter Reduction 453 A core objective of DetNet is to enable the convergence of sensitive 454 non-IP networks onto a common network infrastructure. This requires 455 the accurate emulation of currently deployed mission-specific 456 networks, which for example rely on point-to-point analog (e.g., 457 4-20mA modulation) and serial-digital cables (or buses) for highly 458 reliable, synchronized and jitter-free communications. While the 459 latency of analog transmissions is basically the speed of light, 460 legacy serial links are usually slow (in the order of Kbps) compared 461 to, say, GigE, and some latency is usually acceptable. What is not 462 acceptable is the introduction of excessive jitter, which may, for 463 instance, affect the stability of control systems. 465 Applications that are designed to operate on serial links usually do 466 not provide services to recover the jitter, because jitter simply 467 does not exist there. DetNet flows are generally expected to be 468 delivered in-order and the precise time of reception influences the 469 processes. In order to converge such existing applications, there is 470 a desire to emulate all properties of the serial cable, such as clock 471 transportation, perfect flow isolation and fixed latency. While 472 minimal jitter (in the form of specifying minimum, as well as 473 maximum, end-to-end latency) is supported by DetNet, there are 474 practical limitations on packet-based networks in this regard. In 475 general, users are encouraged to use, instead of, "do this when you 476 get the packet," a combination of: 478 o Sub-microsecond time synchronization among all source and 479 destination end systems, and 481 o Time-of-execution fields in the application packets. 483 Jitter reduction is provided by the mechanisms described in 484 Section 4.5 that also provide congestion protection. 486 3.2.2. Service Protection 488 Service protection aims to mitigate or eliminate packet loss due to 489 equipment failures, random media and/or memory faults. These types 490 of packet loss can be greatly reduced by spreading the data over 491 multiple disjoint forwarding paths. Various service protection 492 methods are described in [RFC6372], e.g., 1+1 linear protection. 493 This section describes the functional details of an additional method 494 in Section 3.2.2.2, which can be implemented as described in 495 Section 3.2.2.3 or as specified in [I-D.ietf-detnet-dp-sol-mpls] in 496 order to provide 1+n hitless protection. The appropriate service 497 protection mechanism depends on the scenario and the requirements. 499 3.2.2.1. In-Order Delivery 501 Out-of-order packet delivery can be a side effect of service 502 protection. Packets delivered out-of-order impact the amount of 503 buffering needed at the destination to properly process the received 504 data. Such packets also influence the jitter of a flow. The DetNet 505 service includes maximum allowed misordering as a constraint. Zero 506 misordering would be a valid service constraint to reflect that the 507 end system(s) of the flow cannot tolerate any out-of-order delivery. 508 Service protection may provide a mechanism to support in-order 509 delivery. 511 3.2.2.2. Packet Replication and Elimination 513 This section describes a service protection method that sends copies 514 of the same packets over multiple paths. 516 The DetNet service layer includes the packet replication (PRF), the 517 packet elimination (PEF), and the packet ordering functionality (POF) 518 for use in DetNet edge, relay node, and end system packet processing. 519 Either of these functions can be enabled in a DetNet edge node, relay 520 node or end system. The collective name for all three functions is 521 PREOF. The packet replication and elimination service protection 522 method altogether involves four capabilities: 524 o Providing sequencing information to the packets of a DetNet 525 compound flow. This may be done by adding a sequence number or 526 time stamp as part of DetNet, or may be inherent in the packet, 527 e.g., in a transport protocol, or associated to other physical 528 properties such as the precise time (and radio channel) of 529 reception of the packet. This is typically done once, at or near 530 the source. 532 o The Packet Replication Function (PRF) replicates these packets 533 into multiple DetNet member flows and typically sends them along 534 multiple different paths to the destination(s), e.g., over the 535 explicit routes of Section 3.2.3. The location within a node, and 536 the mechanism used for the PRF is implementation specific. 538 o The Packet Elimination Function (PEF) eliminates duplicate packets 539 of a DetNet flow based on the sequencing information and a history 540 of received packets. The output of the PEF is always a single 541 packet. This may be done at any node along the path to save 542 network resources further downstream, in particular if multiple 543 Replication points exist. But the most common case is to perform 544 this operation at the very edge of the DetNet network, preferably 545 in or near the receiver. The location within a node, and 546 mechanism used for the PEF is implementation specific. 548 o The Packet Ordering Function (POF) uses the sequencing information 549 to re-order a DetNet flow's packets that are received out of 550 order. 552 The order in which a node applies PEF, POF, and PRF to a DetNet flow 553 is implementation specific. 555 Some service protection mechanisms rely on switching from one flow to 556 another when a failure of a flow is detected. Contrarily, packet 557 replication and elimination combines the DetNet member flows sent 558 along multiple different paths, and performs a packet-by-packet 559 selection of which to discard, e.g., based on sequencing information. 561 In the simplest case, this amounts to replicating each packet in a 562 source that has two interfaces, and conveying them through the 563 network, along separate (disjoint non-SRLG) paths, to the similarly 564 dual-homed destinations, that discard the extras. This ensures that 565 one path (with zero congestion loss) remains, even if some 566 intermediate node fails. The sequencing information can also be used 567 for loss detection and for re-ordering. 569 DetNet relay nodes in the network can provide replication and 570 elimination facilities at various points in the network, so that 571 multiple failures can be accommodated. 573 This is shown in Figure 1, where the two relay nodes each replicate 574 (R) the DetNet flow on input, sending the DetNet member flows to both 575 the other relay node and to the end system, and eliminate duplicates 576 (E) on the output interface to the right-hand end system. Any one 577 link in the network can fail, and the DetNet compound flow can still 578 get through. Furthermore, two links can fail, as long as they are in 579 different segments of the network. 581 > > > > > > > > > relay > > > > > > > > 582 > /------------+ R node E +------------\ > 583 > / v + ^ \ > 584 end R + v | ^ + E end 585 system + v | ^ + system 586 > \ v + ^ / > 587 > \------------+ R relay E +-----------/ > 588 > > > > > > > > > node > > > > > > > > 590 Figure 1: Packet replication and elimination 592 Packet replication and elimination does not react to and correct 593 failures; it is entirely passive. Thus, intermittent failures, 594 mistakenly created packet filters, or misrouted data is handled just 595 the same as the equipment failures that are handled by typical 596 routing and bridging protocols. 598 If packet replication and elimination is used over paths providing 599 congestion protection (Section 3.2.1), and member flows that take 600 different-length paths through the network are combined, a merge 601 point may require extra buffering to equalize the delays over the 602 different paths. This equalization ensures that the resultant 603 compound flow will not exceed its contracted bandwidth even after one 604 or the other of the paths is restored after a failure. The extra 605 buffering can be also used to provide in-order delivery. 607 3.2.2.3. Packet encoding for service protection 609 There are methods for using multiple paths to provide service 610 protection that involve encoding the information in a packet 611 belonging to a DetNet flow into multiple transmission units, 612 combining information from multiple packets into any given 613 transmission unit. Such techniques, also known as "network coding", 614 can be used as a DetNet service protection technique. 616 3.2.3. Explicit routes 618 In networks controlled by typical dynamic control protocols such as 619 IS-IS or OSPF, a network topology event in one part of the network 620 can impact, at least briefly, the delivery of data in parts of the 621 network remote from the failure or recovery event. Even the use of 622 redundant paths through a network defined, e.g., by [RFC6372] do not 623 eliminate the chances of packet loss. Furthermore, out-of-order 624 packet delivery can be a side effect of route changes. 626 Many real-time networks rely on physical rings or chains of two-port 627 devices, with a relatively simple ring control protocol. This 628 supports redundant paths for service protection with a minimum of 629 wiring. As an additional benefit, ring topologies can often utilize 630 different topology management protocols than those used for a mesh 631 network, with a consequent reduction in the response time to topology 632 changes. Of course, this comes at some cost in terms of increased 633 hop count, and thus latency, for the typical path. 635 In order to get the advantages of low hop count and still ensure 636 against even very brief losses of connectivity, DetNet employs 637 explicit routes, where the path taken by a given DetNet flow does not 638 change, at least immediately, and likely not at all, in response to 639 network topology events. Service protection (Section 3.2.2 or 640 Section 3.2.2.3) over explicit routes provides a high likelihood of 641 continuous connectivity. Explicit routes can be established various 642 ways, e.g., with RSVP-TE [RFC3209], with Segment Routing (SR) 643 [I-D.ietf-spring-segment-routing], via a Software Defined Networking 644 approach [RFC7426], with IS-IS [RFC7813], etc. Explicit routes are 645 typically used in MPLS TE LSPs. 647 Out-of-order packet delivery can be a side effect of distributing a 648 single flow over multiple paths especially when there is a change 649 from one path to another when combining the flow. This is 650 irrespective of the distribution method used, also applies to service 651 protection over explicit routes. As described in Section 3.2.2.1, 652 out-of-order packets influence the jitter of a flow and impact the 653 amount of buffering needed to process the data; therefore, DetNet 654 service includes maximum allowed misordering as a constraint. The 655 use of explicit routes helps to provide in-order delivery because 656 there is no immediate route change with the network topology, but the 657 changes are plannable as they are between the different explicit 658 routes. 660 3.3. Secondary goals for DetNet 662 Many applications require DetNet to provide additional services, 663 including coexistence with other QoS mechanisms Section 3.3.1 and 664 protection against misbehaving transmitters Section 3.3.2. 666 3.3.1. Coexistence with normal traffic 668 A DetNet network supports the dedication of a high proportion (e.g. 669 75%) of the network bandwidth to DetNet flows. But, no matter how 670 much is dedicated for DetNet flows, it is a goal of DetNet to coexist 671 with existing Class of Service schemes (e.g., DiffServ). It is also 672 important that non-DetNet traffic not disrupt the DetNet flow, of 673 course (see Section 3.3.2 and Section 5). For these reasons: 675 o Bandwidth (transmission opportunities) not utilized by a DetNet 676 flow are available to non-DetNet packets (though not to other 677 DetNet flows). 679 o DetNet flows can be shaped or scheduled, in order to ensure that 680 the highest-priority non-DetNet packet is also ensured a worst- 681 case latency (at any given hop). 683 o When transmission opportunities for DetNet flows are scheduled in 684 detail, then the algorithm constructing the schedule should leave 685 sufficient opportunities for non-DetNet packets to satisfy the 686 needs of the users of the network. Detailed scheduling can also 687 permit the time-shared use of buffer resources by different DetNet 688 flows. 690 Ideally, the net effect of the presence of DetNet flows in a network 691 on the non-DetNet packets is primarily a reduction in the available 692 bandwidth. 694 3.3.2. Fault Mitigation 696 One key to building robust real-time systems is to reduce the 697 infinite variety of possible failures to a number that can be 698 analyzed with reasonable confidence. DetNet aids in the process by 699 allowing for filters and policers to detect DetNet packets received 700 on the wrong interface, or at the wrong time, or in too great a 701 volume, and to then take actions such as discarding the offending 702 packet, shutting down the offending DetNet flow, or shutting down the 703 offending interface. 705 It is also essential that filters and service remarking be employed 706 at the network edge to prevent non-DetNet packets from being mistaken 707 for DetNet packets, and thus impinging on the resources allocated to 708 DetNet packets. 710 There exist techniques, at present and/or in various stages of 711 standardization, that can perform these fault mitigation tasks that 712 deliver a high probability that misbehaving systems will have zero 713 impact on well-behaved DetNet flows, except of course, for the 714 receiving interface(s) immediately downstream of the misbehaving 715 device. Examples of such techniques include traffic policing 716 functions (e.g. [RFC2475]) and separating flows into per-flow rate- 717 limited queues. 719 4. DetNet Architecture 721 4.1. DetNet stack model 723 4.1.1. Representative Protocol Stack Model 725 Figure 2 illustrates a conceptual DetNet data plane layering model. 726 One may compare it to that in [IEEE802.1CB], Annex C. 728 | packets going | ^ packets coming ^ 729 v down the stack v | up the stack | 730 +----------------------+ +-----------------------+ 731 | Source | | Destination | 732 +----------------------+ +-----------------------+ 733 | Service layer: | | Service layer: | 734 | Packet sequencing | | Duplicate elimination | 735 | Flow replication | | Flow merging | 736 | Packet encoding | | Packet decoding | 737 +----------------------+ +-----------------------+ 738 | Transport layer: | | Transport layer: | 739 | Congestion prot. | | Congestion prot. | 740 | Explicit routes | | Explicit routes | 741 +----------------------+ +-----------------------+ 742 | Lower layers | | Lower layers | 743 +----------------------+ +-----------------------+ 744 v ^ 745 \_________________________/ 747 Figure 2: DetNet data plane protocol stack 749 Not all layers are required for any given application, or even for 750 any given network. The functionality shown in Figure 2 is: 752 Application 753 Shown as "source" and "destination" in the diagram. 755 Packet sequencing 756 As part of DetNet service protection, supplies the sequence 757 number for packet replication and elimination 758 (Section 3.2.2). Peers with Duplicate elimination. This 759 layer is not needed if a higher-layer transport protocol is 760 expected to perform any packet sequencing and duplicate 761 elimination required by the DetNet flow replication. 763 Duplicate elimination 764 As part of the DetNet service layer, based on the sequenced 765 number supplied by its peer, packet sequencing, Duplicate 766 elimination discards any duplicate packets generated by 767 DetNet flow replication. It can operate on member flows, 768 compound flows, or both. The replication may also be 769 inferred from other information such as the precise time of 770 reception in a scheduled network. The duplicate elimination 771 layer may also perform resequencing of packets to restore 772 packet order in a flow that was disrupted by the loss of 773 packets on one or another of the multiple paths taken. 775 Flow replication 776 As part of DetNet service protection, packets that belong to 777 a DetNet compound flow are replicated into two or more DetNet 778 member flows. This function is separate from packet 779 sequencing. Flow replication can be an explicit replication 780 and remarking of packets, or can be performed by, for 781 example, techniques similar to ordinary multicast 782 replication, albeit with resource allocation implications. 783 Peers with DetNet flow merging. 785 Flow merging 786 As part of DetNet service protection, merges DetNet member 787 flows together for packets coming up the stack belonging to a 788 specific DetNet compound flow. Peers with DetNet flow 789 replication. DetNet flow merging, together with packet 790 sequencing, duplicate elimination, and DetNet flow 791 replication perform packet replication and elimination 792 (Section 3.2.2). 794 Packet encoding 795 As part of DetNet service protection, as an alternative to 796 packet sequencing and flow replication, packet encoding 797 combines the information in multiple DetNet packets, perhaps 798 from different DetNet compound flows, and transmits that 799 information in packets on different DetNet member Flows. 800 Peers with Packet decoding. 802 Packet decoding 803 As part of DetNet service protection, as an alternative to 804 flow merging and duplicate elimination, packet decoding takes 805 packets from different DetNet member flows, and computes from 806 those packets the original DetNet packets from the compound 807 flows input to packet encoding. Peers with Packet encoding. 809 Congestion protection 810 The DetNet transport layer provides congestion protection. 811 See Section 4.5. The actual queuing and shaping mechanisms 812 are typically provided by underlying subnet layers, these can 813 be closely associated with the means of providing paths for 814 DetNet flows (e.g., MPLS LSPs or bridged paths), the path and 815 the congestion protection are conflated in this figure. 817 Explicit routes 818 The DetNet transport layer provides mechanisms to ensure that 819 fixed paths are provided for DetNet flows. These explicit 820 paths avoid the impact of network convergence. 822 Operations, Administration, and Maintenance (OAM) leverages in-band 823 and out-of-band signaling that validates whether the service is 824 effectively obtained within QoS constraints. OAM is not shown in 825 Figure 2; it may reside in any number of the layers. OAM can involve 826 specific tagging added in the packets for tracing implementation or 827 network configuration errors; traceability enables to find whether a 828 packet is a replica, which relay node performed the replication, and 829 which segment was intended for the replica. 831 The packet sequencing and replication elimination functions at the 832 source and destination ends of a DetNet compound flow may be 833 performed either in the end system or in a DetNet relay node. 835 4.1.2. DetNet Data Plane Overview 837 A "Deterministic Network" will be composed of DetNet enabled end 838 systems and nodes, i.e., edge nodes, relay nodes and collectively 839 deliver DetNet services. DetNet enabled nodes are interconnected via 840 transit nodes (e.g., LSRs) which support DetNet, but are not DetNet 841 service aware. All DetNet enabled nodes are connected to sub- 842 networks, where a point-to-point link is also considered as a simple 843 sub-network. These sub-networks will provide DetNet compatible 844 service for support of DetNet traffic. Examples of sub-networks 845 include MPLS TE, IEEE 802.1 TSN and OTN. Of course, multi-layer 846 DetNet systems may also be possible, where one DetNet appears as a 847 sub-network, and provides service to, a higher layer DetNet system. 848 A simple DetNet concept network is shown in Figure 3. 850 TSN Edge Transit Relay DetNet 851 End System Node Node Node End System 853 +---------+ +.........+ +---------+ 854 | Appl. |<--:Svc Proxy:-- End to End Service ---------->| Appl. | 855 +---------+ +---------+ +---------+ +---------+ 856 | TSN | |TSN| |Svc|<-- DetNet flow ---: Service :-->| Service | 857 +---------+ +---+ +---+ +---------+ +---------+ +---------+ 858 |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| 859 +-------.-+ +-.-+ +-.-+ +--.----.-+ +-.-+ +-.-+ +---.-----+ 860 : Link : / ,-----. \ : Link : / ,-----. \ 861 +.......+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ 862 [Network] [Network] 863 `-----' `-----' 865 Figure 3: A Simple DetNet Enabled Network 867 Distinguishing the function of two DetNet data plane layers, the 868 DetNet service layer and the DetNet transport layer, helps to explore 869 and evaluate various combinations of the data plane solutions 870 available, some are illustrated in Figure 4. This separation of 871 DetNet layers, while helpful, should not be considered as formal 872 requirement. For example, some technologies may violate these strict 873 layers and still be able to deliver a DetNet service. 875 . 876 . 877 +-----------+ 878 | Service | PW, UDP, GRE 879 +-----------+ 880 | Transport | IPv6, IPv4, MPLS TE LSPs, MPLS SR 881 +-----------+ 882 . 883 . 885 Figure 4: DetNet adaptation to data plane 887 In some networking scenarios, the end system initially provides a 888 DetNet flow encapsulation, which contains all information needed by 889 DetNet nodes (e.g., Real-time Transport Protocol (RTP) [RFC3550] 890 based DetNet flow transported over a native UDP/IP network or 891 PseudoWire). In other scenarios, the encapsulation formats might 892 differ significantly. 894 There are many valid options to create a data plane solution for 895 DetNet traffic by selecting a technology approach for the DetNet 896 service layer and also selecting a technology approach for the DetNet 897 transport layer. There are a high number of valid combinations. 899 One of the most fundamental differences between different potential 900 data plane options is the basic headers used by DetNet nodes. For 901 example, the basic service can be delivered based on an MPLS label or 902 an IP header. This decision impacts the basic forwarding logic for 903 the DetNet service layer. Note that in both cases, IP addresses are 904 used to address DetNet nodes. The selected DetNet transport layer 905 technology also needs to be mapped to the sub-net technology used to 906 interconnect DetNet nodes. For example, DetNet flows will need to be 907 mapped to TSN Streams. 909 4.1.3. Network reference model 911 Figure 5 shows another view of the DetNet service related reference 912 points and main components. 914 DetNet DetNet 915 end system end system 916 _ _ 917 / \ +----DetNet-UNI (U) / \ 918 /App\ | /App\ 919 /-----\ | /-----\ 920 | NIC | v ________ | NIC | 921 +--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+ 922 | / \__/ \ | | 923 | / +----+ +----+ \_____ | | 924 | / | | | | \_______ | | 925 +------U PE +----+ P +----+ \ _ v | 926 | | | | | | | ___/ \ | 927 | +--+-+ +----+ | +----+ | / \_ | 928 \ | | | | | / \ | 929 \ | +----+ +--+-+ +--+PE |------ U-----+ 930 \ | | | | | | | | | \_ _/ 931 \ +---+ P +----+ P +--+ +----+ | \____/ 932 \___ | | | | / 933 \ +----+__ +----+ DetNet-1 DetNet-2 934 | \_____/ \___________/ | 935 | | 936 | | End-to-End-Service | | | | 937 <-------------------------------------------------------------> 938 | | DetNet-Service | | | | 939 | <------------------------------------------------> | 940 | | | | | | 942 Figure 5: DetNet Service Reference Model (multi-domain) 944 DetNet-UNIs ("U" in Figure 5) are assumed in this document to be 945 packet-based reference points and provide connectivity over the 946 packet network. A DetNet-UNI may provide multiple functions, e.g., 947 it may add networking technology specific encapsulation to the DetNet 948 flows if necessary; it may provide status of the availability of the 949 resources associated with a reservation; it may provide a 950 synchronization service for the end system; it may carry enough 951 signaling to place the reservation in a network without a controller, 952 or if the controller only deals with the network but not the end 953 systems. Internal reference points of end systems (between the 954 application and the NIC) are more challenging from control 955 perspective and they may have extra requirements (e.g., in-order 956 delivery is expected in end system internal reference points, whereas 957 it is considered optional over the DetNet-UNI). 959 4.2. DetNet systems 961 4.2.1. End system 963 The native data flow between the source/destination end systems is 964 referred to as application-flow (App-flow). The traffic 965 characteristics of an App-flow can be CBR (constant bit rate) or VBR 966 (variable bit rate) and can have L1 or L2 or L3 encapsulation (e.g., 967 TDM (time-division multiplexing), Ethernet, IP). These 968 characteristics are considered as input for resource reservation and 969 might be simplified to ensure determinism during transport (e.g., 970 making reservations for the peak rate of VBR traffic, etc.). 972 An end system may or may not be DetNet transport layer aware or 973 DetNet service layer aware. That is, an end system may or may not 974 contain DetNet specific functionality. End systems with DetNet 975 functionalities may have the same or different transport layer as the 976 connected DetNet domain. Categorization of end systems are shown in 977 Figure 6. 979 End system 980 | 981 | 982 | DetNet aware ? 983 / \ 984 +------< >------+ 985 NO | \ / | YES 986 | v | 987 DetNet unaware | 988 End system | 989 | Service/ 990 | Transport 991 / \ aware ? 992 +--------< >-------------+ 993 t-aware | \ / | s-aware 994 | v | 995 | | both | 996 | | | 997 DetNet t-aware | DetNet s-aware 998 End system | End system 999 v 1000 DetNet st-aware 1001 End system 1003 Figure 6: Categorization of end systems 1005 Note some known use case examples for end systems: 1007 o DetNet unaware: The classic case requiring service proxies. 1009 o DetNet t-aware: An extant TSN system. It knows about some TSN 1010 functions (e.g., reservation), but not about service protection. 1012 o DetNet s-aware: An extant IEC 62439-3 system. It supplies 1013 sequence numbers, but doesn't know about zero congestion loss. 1015 o DetNet st-aware: A full functioning DetNet end system, it has 1016 DetNet functionalities and usually the same forwarding paradigm as 1017 the connected DetNet domain. It can be treated as an integral 1018 part of the DetNet domain. 1020 4.2.2. DetNet edge, relay, and transit nodes 1022 As shown in Figure 3, DetNet edge nodes providing proxy service and 1023 DetNet relay nodes providing the DetNet service layer are DetNet- 1024 aware, and DetNet transit nodes need only be aware of the DetNet 1025 transport layer. 1027 In general, if a DetNet flow passes through one or more DetNet- 1028 unaware network nodes between two DetNet nodes providing the DetNet 1029 transport layer for that flow, there is a potential for disruption or 1030 failure of the DetNet QoS. A network administrator needs to ensure 1031 that the DetNet-unaware network nodes are configured to minimize the 1032 chances of packet loss and delay, and provision enough extra buffer 1033 space in the DetNet transit node following the DetNet-unaware network 1034 nodes to absorb the induced latency variations. 1036 4.3. DetNet flows 1038 4.3.1. DetNet flow types 1040 A DetNet flow can have different formats while it is transported 1041 between the peer end systems. Therefore, the following possible 1042 types / formats of a DetNet flow are distinguished in this document: 1044 o App-flow: native format of the data carried over a DetNet flow. 1045 It does not contain any DetNet related attributes. 1047 o DetNet-t-flow: specific format of a DetNet flow. Only requires 1048 the congestion / latency features provided by the DetNet transport 1049 layer. 1051 o DetNet-s-flow: specific format of a DetNet flow. Only requires 1052 the service protection feature ensured by the DetNet service 1053 layer. 1055 o DetNet-st-flow: specific format of a DetNet flow. It requires 1056 both DetNet service layer and DetNet transport layer functions 1057 during forwarding. 1059 4.3.2. Source transmission behavior 1061 For the purposes of congestion protection, DetNet flows can be 1062 synchronous or asynchronous. In synchronous DetNet flows, at least 1063 the intermediate nodes (and possibly the end systems) are closely 1064 time synchronized, typically to better than 1 microsecond. By 1065 transmitting packets from different DetNet flows or classes of DetNet 1066 flows at different times, using repeating schedules synchronized 1067 among the intermediate nodes, resources such as buffers and link 1068 bandwidth can be shared over the time domain among different DetNet 1069 flows. There is a tradeoff among techniques for synchronous DetNet 1070 flows between the burden of fine-grained scheduling and the benefit 1071 of reducing the required resources, especially buffer space. 1073 In contrast, asynchronous DetNet flows are not coordinated with a 1074 fine-grained schedule, so relay and end systems must assume worst- 1075 case interference among DetNet flows contending for buffer resources. 1076 Asynchronous DetNet flows are characterized by: 1078 o A maximum packet size; 1080 o An observation interval; and 1082 o A maximum number of transmissions during that observation 1083 interval. 1085 These parameters, together with knowledge of the protocol stack used 1086 (and thus the size of the various headers added to a packet), limit 1087 the number of bit times per observation interval that the DetNet flow 1088 can occupy the physical medium. 1090 The source is required not to exceed these limits in order to obtain 1091 DetNet service. If the source transmits less data than this limit 1092 allows, the unused resource such as link bandwidth can be made 1093 available by the system to non-DetNet packets. However, making those 1094 resources available to DetNet packets in other DetNet flows would 1095 serve no purpose. Those other DetNet flows have their own dedicated 1096 resources, on the assumption that all DetNet flows can use all of 1097 their resources over a long period of time. 1099 There is no provision in DetNet for throttling DetNet flows (reducing 1100 end-to-end transmission rate via any explicit congestion 1101 notification); the assumption is that a DetNet flow, to be useful, 1102 must be delivered in its entirety. That is, while any useful 1103 application is written to expect a certain number of lost packets, 1104 the real-time applications of interest to DetNet demand that the loss 1105 of data due to the network is an extraordinarily event. 1107 Although DetNet strives to minimize the changes required of an 1108 application to allow it to shift from a special-purpose digital 1109 network to an Internet Protocol network, one fundamental shift in the 1110 behavior of network applications is impossible to avoid: the 1111 reservation of resources before the application starts. In the first 1112 place, a network cannot deliver finite latency and practically zero 1113 packet loss to an arbitrarily high offered load. Secondly, achieving 1114 practically zero packet loss for unthrottled (though bandwidth 1115 limited) DetNet flows means that bridges and routers have to dedicate 1116 buffer resources to specific DetNet flows or to classes of DetNet 1117 flows. The requirements of each reservation have to be translated 1118 into the parameters that control each system's queuing, shaping, and 1119 scheduling functions and delivered to the hosts, bridges, and 1120 routers. 1122 4.3.3. Incomplete Networks 1124 The presence in the network of transit nodes or subnets that are not 1125 fully capable of offering DetNet services complicates the ability of 1126 the intermediate nodes and/or controller to allocate resources, as 1127 extra buffering must be allocated at points downstream from the non- 1128 DetNet intermediate node for a DetNet flow. This extra buffering may 1129 increase latency and/or jitter. 1131 4.4. Traffic Engineering for DetNet 1133 Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines 1134 traffic-engineering architectures for generic applicability across 1135 packet and non-packet networks. From a TEAS perspective, Traffic 1136 Engineering (TE) refers to techniques that enable operators to 1137 control how specific traffic flows are treated within their networks. 1139 Because if its very nature of establishing explicit optimized paths, 1140 Deterministic Networking can be seen as a new, specialized branch of 1141 Traffic Engineering, and inherits its architecture with a separation 1142 into planes. 1144 The Deterministic Networking architecture is thus composed of three 1145 planes, a (User) Application Plane, a Controller Plane, and a Network 1146 Plane, which echoes that of Figure 1 of Software-Defined Networking 1147 (SDN): Layers and Architecture Terminology [RFC7426].: 1149 4.4.1. The Application Plane 1151 Per [RFC7426], the Application Plane includes both applications and 1152 services. In particular, the Application Plane incorporates the User 1153 Agent, a specialized application that interacts with the end user / 1154 operator and performs requests for Deterministic Networking services 1155 via an abstract Flow Management Entity, (FME) which may or may not be 1156 collocated with (one of) the end systems. 1158 At the Application Plane, a management interface enables the 1159 negotiation of flows between end systems. An abstraction of the flow 1160 called a Traffic Specification (TSpec) provides the representation. 1161 This abstraction is used to place a reservation over the (Northbound) 1162 Service Interface and within the Application plane. It is associated 1163 with an abstraction of location, such as IP addresses and DNS names, 1164 to identify the end systems and eventually specify intermediate 1165 nodes. 1167 4.4.2. The Controller Plane 1169 The Controller Plane corresponds to the aggregation of the Control 1170 and Management Planes in [RFC7426], though Common Control and 1171 Measurement Plane (CCAMP) [CCAMP] makes an additional distinction 1172 between management and measurement. When the logical separation of 1173 the Control, Measurement and other Management entities is not 1174 relevant, the term Controller Plane is used for simplicity to 1175 represent them all, and the term controller plane entity (CPE) refers 1176 to any device operating in that plane, whether is it a Path 1177 Computation entity, or a Network Management entity (NME)), or a 1178 distributed control plane. The Path Computation Element (PCE) 1179 [RFC4655] is a core element of a controller, in charge of computing 1180 Deterministic paths to be applied in the Network Plane. 1182 A (Northbound) Service Interface enables applications in the 1183 Application Plane to communicate with the entities in the Controller 1184 Plane as illustrated in Figure 7. 1186 One or more PCE(s) collaborate to implement the requests from the FME 1187 as Per-Flow Per-Hop Behaviors installed in the intermediate nodes for 1188 each individual flow. The PCEs place each flow along a deterministic 1189 sequence of intermediate nodes so as to respect per-flow constraints 1190 such as security and latency, and optimize the overall result for 1191 metrics such as an abstract aggregated cost. The deterministic 1192 sequence can typically be more complex than a direct sequence and 1193 include redundancy path, with one or more packet replication and 1194 elimination points. 1196 4.4.3. The Network Plane 1198 The Network Plane represents the network devices and protocols as a 1199 whole, regardless of the Layer at which the network devices operate. 1200 It includes Forwarding Plane (data plane), Application, and 1201 Operational Plane (control plane) aspects. 1203 The network Plane comprises the Network Interface Cards (NIC) in the 1204 end systems, which are typically IP hosts, and intermediate nodes, 1205 which are typically IP routers and switches. Network-to-Network 1206 Interfaces such as used for Traffic Engineering path reservation in 1207 [RFC5921], as well as User-to-Network Interfaces (UNI) such as 1208 provided by the Local Management Interface (LMI) between network and 1209 end systems, are both part of the Network Plane, both in the control 1210 plane and the data plane. 1212 A Southbound (Network) Interface enables the entities in the 1213 Controller Plane to communicate with devices in the Network Plane as 1214 illustrated in Figure 7. This interface leverages and extends TEAS 1215 to describe the physical topology and resources in the Network Plane. 1217 End End 1218 System System 1220 -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 1222 CPE CPE CPE CPE 1224 -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 1226 intermediate intermed. intermed. intermed. 1227 Node Node Node Node 1228 NIC NIC 1229 intermediate intermed. intermed. intermed. 1230 Node Node Node Node 1232 Figure 7: Northbound and Southbound interfaces 1234 The intermediate nodes (and eventually the end systems NIC) expose 1235 their capabilities and physical resources to the controller (the 1236 CPE), and update the CPEs with their dynamic perception of the 1237 topology, across the Southbound Interface. In return, the CPEs set 1238 the per-flow paths up, providing a Flow Characterization that is more 1239 tightly coupled to the intermediate node Operation than a TSpec. 1241 At the Network plane, intermediate nodes may exchange information 1242 regarding the state of the paths, between adjacent systems and 1243 eventually with the end systems, and forward packets within 1244 constraints associated to each flow, or, when unable to do so, 1245 perform a last resort operation such as drop or declassify. 1247 This document focuses on the Southbound interface and the operation 1248 of the Network Plane. 1250 4.5. Queuing, Shaping, Scheduling, and Preemption 1252 DetNet achieves congestion protection and bounded delivery latency by 1253 reserving bandwidth and buffer resources at every hop along the path 1254 of the DetNet flow. The reservation itself is not sufficient, 1255 however. Implementors and users of a number of proprietary and 1256 standard real-time networks have found that standards for specific 1257 data plane techniques are required to enable these assurances to be 1258 made in a multi-vendor network. The fundamental reason is that 1259 latency variation in one system results in the need for extra buffer 1260 space in the next-hop system(s), which in turn, increases the worst- 1261 case per-hop latency. 1263 Standard queuing and transmission selection algorithms allow a 1264 central controller to compute the latency contribution of each 1265 transit node to the end-to-end latency, to compute the amount of 1266 buffer space required in each transit node for each incremental 1267 DetNet flow, and most importantly, to translate from a flow 1268 specification to a set of values for the managed objects that control 1269 each relay or end system. For example, the IEEE 802.1 WG has 1270 specified (and is specifying) a set of queuing, shaping, and 1271 scheduling algorithms that enable each transit node (bridge or 1272 router), and/or a central controller, to compute these values. These 1273 algorithms include: 1275 o A credit-based shaper [IEEE802.1Q] Clause 34. 1277 o Time-gated queues governed by a rotating time schedule, 1278 synchronized among all transit nodes [IEEE802.1Qbv]. 1280 o Synchronized double (or triple) buffers driven by synchronized 1281 time ticks. [IEEE802.1Qch]. 1283 o Pre-emption of an Ethernet packet in transmission by a packet with 1284 a more stringent latency requirement, followed by the resumption 1285 of the preempted packet [IEEE802.1Qbu], [IEEE802.3br]. 1287 While these techniques are currently embedded in Ethernet [IEEE802.3] 1288 and bridging standards, we can note that they are all, except perhaps 1289 for packet preemption, equally applicable to other media than 1290 Ethernet, and to routers as well as bridges. Other media may have 1291 its own methods, see, e.g., [I-D.ietf-6tisch-architecture], 1292 [RFC7554]. DetNet may include such definitions in the future, or may 1293 define how these techniques can be used by DetNet nodes. 1295 4.6. Service instance 1297 A Service instance represents all the functions required on a node to 1298 allow the end-to-end service between the UNIs. 1300 The DetNet network general reference model is shown in Figure 8 for a 1301 DetNet-Service scenario (i.e., between two DetNet-UNIs). In this 1302 figure, end systems ("A" and "B") are connected directly to the edge 1303 nodes of an IP/MPLS network ("PE1" and "PE2"). End systems 1304 participating in DetNet communication may require connectivity before 1305 setting up an App-flow that requires the DetNet service. Such a 1306 connectivity related service instance and the one dedicated for 1307 DetNet service share the same access. Packets belonging to a DetNet 1308 flow are selected by a filter configured on the access ("F1" and 1309 "F2"). As a result, data flow specific access ("access-A + F1" and 1310 "access-B + F2") are terminated in the flow specific service instance 1311 ("SI-1" and "SI-2"). A tunnel is used to provide connectivity 1312 between the service instances. 1314 The tunnel is used to transport exclusively the packets of the DetNet 1315 flow between "SI-1" and "SI-2". The service instances are configured 1316 to implement DetNet functions and a flow specific DetNet transport. 1317 The service instance and the tunnel may or may not be shared by 1318 multiple DetNet flows. Sharing the service instance by multiple 1319 DetNet flows requires properly populated forwarding tables of the 1320 service instance. 1322 access-A access-B 1323 <-----> <-------- tunnel ----------> <-----> 1325 +---------+ ___ _ +---------+ 1326 End system | +----+ | / \/ \_ | +----+ | End system 1327 "A" -------F1+ | | / \ | | +F2----- "B" 1328 | | +========+ IP/MPLS +=======+ | | 1329 | |SI-1| | \__ Net._/ | |SI-2| | 1330 | +----+ | \____/ | +----+ | 1331 |PE1 | | PE2| 1332 +---------+ +---------+ 1334 Figure 8: DetNet network general reference model 1336 The tunnel between the service instances may have some special 1337 characteristics. For example, in case of a DetNet L3 service, there 1338 are differences in the usage of the PW for DetNet traffic compared to 1339 the network model described in [RFC6658]. In the DetNet scenario, 1340 the PW is likely to be used exclusively by the DetNet flow, whereas 1341 [RFC6658] states: "The packet PW appears as a single point-to-point 1342 link to the client layer. Network-layer adjacency formation and 1343 maintenance between the client equipment will follow the normal 1344 practice needed to support the required relationship in the client 1345 layer ... This packet pseudowire is used to transport all of the 1346 required Layer-2 and Layer-3 protocols between LSR1 and LSR2". 1347 Further details are network technology specific and can be found in 1348 [I-D.ietf-detnet-dp-sol-mpls] and [I-D.ietf-detnet-dp-sol-ip]. 1350 4.7. Flow identification at technology borders 1352 4.7.1. Exporting flow identification 1354 A DetNet node may need to map specific flows to lower layer flows (or 1355 Streams) in order to provide specific queuing and shaping services 1356 for specific flows. For example: 1358 o A non-IP, strictly L2 source end system X may be sending multiple 1359 flows to the same L2 destination end system Y. Those flows may 1360 include DetNet flows with different QoS requirements, and may 1361 include non-DetNet flows. 1363 o A router may be sending any number of flows to another router. 1364 Again, those flows may include DetNet flows with different QoS 1365 requirements, and may include non-DetNet flows. 1367 o Two routers may be separated by bridges. For these bridges to 1368 perform any required per-flow queuing and shaping, they must be 1369 able to identify the individual flows. 1371 o A Label Edge Router (LER) may have a Label Switched Path (LSP) set 1372 up for handling traffic destined for a particular IP address 1373 carrying only non-DetNet flows. If a DetNet flow to that same 1374 address is requested, a separate LSP may be needed, in order that 1375 all of the Label Switch Routers (LSRs) along the path to the 1376 destination give that flow special queuing and shaping. 1378 The need for a lower-layer node to be aware of individual higher- 1379 layer flows is not unique to DetNet. But, given the endless 1380 complexity of layering and relayering over tunnels that is available 1381 to network designers, DetNet needs to provide a model for flow 1382 identification that is better than packet inspection. That is not to 1383 say that packet inspection to layer 4 or 5 addresses will not be 1384 used, or the capability standardized; but, there are alternatives. 1386 A DetNet relay node can connect DetNet flows on different paths using 1387 different flow identification methods. For example: 1389 o A single unicast DetNet flow passing from router A through a 1390 bridged network to router B may be assigned a TSN Stream 1391 identifier that is unique within that bridged network. The 1392 bridges can then identify the flow without accessing higher-layer 1393 headers. Of course, the receiving router must recognize and 1394 accept that TSN Stream. 1396 o A DetNet flow passing from LSR A to LSR B may be assigned a 1397 different label than that used for other flows to the same IP 1398 destination. 1400 In any of the above cases, it is possible that an existing DetNet 1401 flow can be an aggregate carrying multiple other DetNet flows. (Not 1402 to be confused with DetNet compound vs. member flows.) Of course, 1403 this requires that the aggregate DetNet flow be provisioned properly 1404 to carry the aggregated flows. 1406 Thus, rather than packet inspection, there is the option to export 1407 higher-layer information to the lower layer. The requirement to 1408 support one or the other method for flow identification (or both) is 1409 a complexity that is part of DetNet control models. 1411 4.7.2. Flow attribute mapping between layers 1413 Transport of DetNet flows over multiple technology domains may 1414 require that lower layers are aware of specific flows of higher 1415 layers. Such an "exporting of flow identification" is needed each 1416 time when the forwarding paradigm is changed on the transport path 1417 (e.g., two LSRs are interconnected by a L2 bridged domain, etc.). 1418 The three representative forwarding methods considered for 1419 deterministic networking are: 1421 o IP routing 1423 o MPLS label switching 1425 o Ethernet bridging 1427 A packet with corresponding Flow-IDs is illustrated in Figure 9. 1429 add/remove add/remove 1430 Eth Flow-ID IP Flow-ID 1431 | | 1432 v v 1433 +-----------------------------------------------------------+ 1434 | | | | | 1435 | Eth | MPLS | IP | Application data | 1436 | | | | | 1437 +-----------------------------------------------------------+ 1438 ^ 1439 | 1440 add/remove 1441 MPLS Flow-ID 1443 Figure 9: Packet with multiple Flow-IDs 1445 The additional (domain specific) Flow-ID can be 1447 o created by a domain specific function or 1449 o derived from the Flow-ID added to the App-flow. 1451 The Flow-ID must be unique inside a given domain. Note that the 1452 Flow-ID added to the App-flow is still present in the packet, but 1453 transport nodes may lack the function to recognize it; that's why the 1454 additional Flow-ID is added. 1456 4.7.3. Flow-ID mapping examples 1458 IP nodes and MPLS nodes are assumed to be configured to push such an 1459 additional (domain specific) Flow-ID when sending traffic to an 1460 Ethernet switch (as shown in the examples below). 1462 Figure 10 shows a scenario where an IP end system ("IP-A") is 1463 connected via two Ethernet switches ("ETH-n") to an IP router ("IP- 1464 1"). 1466 IP domain 1467 <----------------------------------------------- 1469 +======+ +======+ 1470 |L3-ID | |L3-ID | 1471 +======+ /\ +-----+ +======+ 1472 / \ Forward as | | 1473 /IP-A\ per ETH-ID |IP-1 | Recognize 1474 Push ------> +-+----+ | +---+-+ <----- ETH-ID 1475 ETH-ID | +----+-----+ | 1476 | v v | 1477 | +-----+ +-----+ | 1478 +------+ | | +---------+ 1479 +......+ |ETH-1+----+ETH-2| +======+ 1480 .L3-ID . +-----+ +-----+ |L3-ID | 1481 +======+ +......+ +======+ 1482 |ETH-ID| .L3-ID . |ETH-ID| 1483 +======+ +======+ +------+ 1484 |ETH-ID| 1485 +======+ 1487 Ethernet domain 1488 <----------------> 1490 Figure 10: IP nodes interconnected by an Ethernet domain 1492 End system "IP-A" uses the original App-flow specific ID ("L3-ID"), 1493 but as it is connected to an Ethernet domain it has to push an 1494 Ethernet-domain specific flow-ID ("VID + multicast MAC address", 1495 referred as "ETH-ID") before sending the packet to "ETH-1" node. 1496 Ethernet switch "ETH-1" can recognize the data flow based on the 1497 "ETH-ID" and it does forwarding toward "ETH-2". "ETH-2" switches the 1498 packet toward the IP router. "IP-1" must be configured to receive 1499 the Ethernet Flow-ID specific multicast flow, but (as it is an L3 1500 node) it decodes the data flow ID based on the "L3-ID" fields of the 1501 received packet. 1503 Figure 11 shows a scenario where MPLS domain nodes ("PE-n" and "P-m") 1504 are connected via two Ethernet switches ("ETH-n"). 1506 MPLS domain 1507 <-----------------------------------------------> 1509 +=======+ +=======+ 1510 |MPLS-ID| |MPLS-ID| 1511 +=======+ +-----+ +-----+ +=======+ +-----+ 1512 | | Forward as | | | | 1513 |PE-1 | per ETH-ID | P-2 +-----------+ PE-2| 1514 Push -----> +-+---+ | +---+-+ +-----+ 1515 ETH-ID | +-----+----+ | \ Recognize 1516 | v v | +-- ETH-ID 1517 | +-----+ +-----+ | 1518 +---+ | | +----+ 1519 +.......+ |ETH-1+----+ETH-2| +=======+ 1520 .MPLS-ID. +-----+ +-----+ |MPLS-ID| 1521 +=======+ +=======+ 1522 |ETH-ID | +.......+ |ETH-ID | 1523 +=======+ .MPLS-ID. +-------+ 1524 +=======+ 1525 |ETH-ID | 1526 +=======+ 1527 Ethernet domain 1528 <----------------> 1530 Figure 11: MPLS nodes interconnected by an Ethernet domain 1532 "PE-1" uses the MPLS specific ID ("MPLS-ID"), but as it is connected 1533 to an Ethernet domain it has to push an Ethernet-domain specific 1534 flow-ID ("VID + multicast MAC address", referred as "ETH-ID") before 1535 sending the packet to "ETH-1". Ethernet switch "ETH-1" can recognize 1536 the data flow based on the "ETH-ID" and it does forwarding toward 1537 "ETH-2". "ETH-2" switches the packet toward the MPLS node ("P-2"). 1538 "P-2" must be configured to receive the Ethernet Flow-ID specific 1539 multicast flow, but (as it is an MPLS node) it decodes the data flow 1540 ID based on the "MPLS-ID" fields of the received packet. 1542 One can appreciate from the above example that, when the means used 1543 for DetNet flow identification is altered or exported, the means for 1544 encoding the sequence number information must similarly be altered or 1545 exported. 1547 4.8. Advertising resources, capabilities and adjacencies 1549 There are three classes of information that a central controller or 1550 distributed control plane needs to know that can only be obtained 1551 from the end systems and/or nodes in the network. When using a peer- 1552 to-peer control plane, some of this information may be required by a 1553 system's neighbors in the network. 1555 o Details of the system's capabilities that are required in order to 1556 accurately allocate that system's resources, as well as other 1557 systems' resources. This includes, for example, which specific 1558 queuing and shaping algorithms are implemented (Section 4.5), the 1559 number of buffers dedicated for DetNet allocation, and the worst- 1560 case forwarding delay and misordering. 1562 o The dynamic state of a node's DetNet resources. 1564 o The identity of the system's neighbors, and the characteristics of 1565 the link(s) between the systems, including the length (in 1566 nanoseconds) of the link(s). 1568 4.9. Scaling to larger networks 1570 Reservations for individual DetNet flows require considerable state 1571 information in each transit node, especially when adequate fault 1572 mitigation (Section 3.3.2) is required. The DetNet data plane, in 1573 order to support larger numbers of DetNet flows, must support the 1574 aggregation of DetNet flows. Such aggregated flows can be viewed by 1575 the transit nodes' data plane largely as individual DetNet flows. 1576 Without such aggregation, the per-relay system may limit the scale of 1577 DetNet networks. Example techniques that may be used include MPLS 1578 hierarchy and IP DiffServ Code Points (DSCPs). 1580 4.10. Compatibility with Layer-2 1582 Standards providing similar capabilities for bridged networks (only) 1583 have been and are being generated in the IEEE 802 LAN/MAN Standards 1584 Committee. The present architecture describes an abstract model that 1585 can be applicable both at Layer-2 and Layer-3, and over links not 1586 defined by IEEE 802. 1588 DetNet enabled end systems and intermediate nodes can be 1589 interconnected by sub-networks, i.e., Layer-2 technologies. These 1590 sub-networks will provide DetNet compatible service for support of 1591 DetNet traffic. Examples of sub-networks include MPLS TE, 802.1 TSN, 1592 and a point-to-point OTN link. Of course, multi-layer DetNet systems 1593 may be possible too, where one DetNet appears as a sub-network, and 1594 provides service to, a higher layer DetNet system. 1596 5. Security Considerations 1598 Security in the context of Deterministic Networking has an added 1599 dimension; the time of delivery of a packet can be just as important 1600 as the contents of the packet, itself. A man-in-the-middle attack, 1601 for example, can impose, and then systematically adjust, additional 1602 delays into a link, and thus disrupt or subvert a real-time 1603 application without having to crack any encryption methods employed. 1604 See [RFC7384] for an exploration of this issue in a related context. 1606 Furthermore, in a control system where millions of dollars of 1607 equipment, or even human lives, can be lost if the DetNet QoS is not 1608 delivered, one must consider not only simple equipment failures, 1609 where the box or wire instantly becomes perfectly silent, but complex 1610 errors such as can be caused by software failures. Because there is 1611 essential no limit to the kinds of failures that can occur, 1612 protecting against realistic equipment failures is indistinguishable, 1613 in most cases, from protecting against malicious behavior, whether 1614 accidental or intentional. See also Section 3.3.2. 1616 Security must cover: 1618 o the protection of the signaling protocol 1620 o the authentication and authorization of the controlling systems 1622 o the identification and shaping of the DetNet flows 1624 6. Privacy Considerations 1626 DetNet is provides a Quality of Service (QoS), and as such, does not 1627 directly raise any new privacy considerations. 1629 However, the requirement for every (or almost every) node along the 1630 path of a DetNet flow to identify DetNet flows may present an 1631 additional attack surface for privacy, should the DetNet paradigm be 1632 found useful in broader environments. 1634 7. IANA Considerations 1636 This document does not require an action from IANA. 1638 8. Acknowledgements 1640 The authors wish to thank Lou Berger, David Black, Stewart Bryant, 1641 Rodney Cummings, Ethan Grossman, Craig Gunther, Marcel Kiessling, 1642 Rudy Klecka, Jouni Korhonen, Erik Nordmark, Shitanshu Shah, Wilfried 1643 Steiner, George Swallow, Michael Johas Teener, Pat Thaler, Thomas 1644 Watteyne, Patrick Wetterwald, Karl Weber, Anca Zamfir, for their 1645 various contribution with this work. 1647 9. Informative References 1649 [CCAMP] IETF, "Common Control and Measurement Plane Working 1650 Group", 1651 . 1653 [I-D.ietf-6tisch-architecture] 1654 Thubert, P., "An Architecture for IPv6 over the TSCH mode 1655 of IEEE 802.15.4", draft-ietf-6tisch-architecture-14 (work 1656 in progress), April 2018. 1658 [I-D.ietf-detnet-dp-sol-ip] 1659 IETF, "DetNet IP Data Plane Encapsulation", July 2018, 1660 . 1663 [I-D.ietf-detnet-dp-sol-mpls] 1664 IETF, "DetNet MPLS Data Plane Encapsulation", July 2018, 1665 . 1668 [I-D.ietf-detnet-problem-statement] 1669 Finn, N. and P. Thubert, "Deterministic Networking Problem 1670 Statement", draft-ietf-detnet-problem-statement-05 (work 1671 in progress), June 2018. 1673 [I-D.ietf-detnet-use-cases] 1674 Grossman, E., "Deterministic Networking Use Cases", draft- 1675 ietf-detnet-use-cases-17 (work in progress), June 2018. 1677 [I-D.ietf-spring-segment-routing] 1678 Filsfils, C., Previdi, S., Ginsberg, L., Decraene, B., 1679 Litkowski, S., and R. Shakir, "Segment Routing 1680 Architecture", draft-ietf-spring-segment-routing-15 (work 1681 in progress), January 2018. 1683 [IEEE802.1BA] 1684 IEEE Standards Association, "IEEE Std 802.1BA-2011 Audio 1685 Video Bridging (AVB) Systems", 2011, 1686 . 1688 [IEEE802.1CB] 1689 IEEE Standards Association, "IEEE Std 802.1CB Frame 1690 Replication and Elimination for Reliability", 2017, 1691 . 1693 [IEEE802.1Q] 1694 IEEE Standards Association, "IEEE Std 802.1Q-2018 Bridges 1695 and Bridged Networks", 2018, 1696 . 1699 [IEEE802.1Qbu] 1700 IEEE Standards Association, "IEEE Std 802.1Qbu-2016 1701 Bridges and Bridged Networks - Amendment 26: Frame 1702 Preemption", 2016, 1703 . 1705 [IEEE802.1Qbv] 1706 IEEE Standards Association, "IEEE Std 802.1Qbv-2015 1707 Bridges and Bridged Networks - Amendment 25: Enhancements 1708 for Scheduled Traffic", 2015, 1709 . 1711 [IEEE802.1Qch] 1712 IEEE Standards Association, "IEEE Std 802.1Qbv-2015 1713 Bridges and Bridged Networks - Amendment 29: Cyclic 1714 Queuing and Forwarding", 2017, 1715 . 1718 [IEEE802.1TSNTG] 1719 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1720 Networking Task Group", 2013, 1721 . 1723 [IEEE802.3] 1724 IEEE Standards Association, "IEEE Std 802.3-2015 Standard 1725 for Ethernet", 2015, 1726 . 1728 [IEEE802.3br] 1729 IEEE Standards Association, "IEEE Std 802.3br-2016 1730 Standard for Ethernet Amendment 5: Specification and 1731 Management Parameters for Interspersing Express Traffic", 1732 2016, . 1734 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1735 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1736 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1737 September 1997, . 1739 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1740 and W. Weiss, "An Architecture for Differentiated 1741 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1742 . 1744 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1745 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1746 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1747 . 1749 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1750 Jacobson, "RTP: A Transport Protocol for Real-Time 1751 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 1752 July 2003, . 1754 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 1755 Element (PCE)-Based Architecture", RFC 4655, 1756 DOI 10.17487/RFC4655, August 2006, 1757 . 1759 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 1760 L., and L. Berger, "A Framework for MPLS in Transport 1761 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 1762 . 1764 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 1765 Profile (MPLS-TP) Survivability Framework", RFC 6372, 1766 DOI 10.17487/RFC6372, September 2011, 1767 . 1769 [RFC6658] Bryant, S., Ed., Martini, L., Swallow, G., and A. Malis, 1770 "Packet Pseudowire Encapsulation over an MPLS PSN", 1771 RFC 6658, DOI 10.17487/RFC6658, July 2012, 1772 . 1774 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1775 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1776 October 2014, . 1778 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1779 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1780 Defined Networking (SDN): Layers and Architecture 1781 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1782 2015, . 1784 [RFC7554] Watteyne, T., Ed., Palattella, M., and L. Grieco, "Using 1785 IEEE 802.15.4e Time-Slotted Channel Hopping (TSCH) in the 1786 Internet of Things (IoT): Problem Statement", RFC 7554, 1787 DOI 10.17487/RFC7554, May 2015, 1788 . 1790 [RFC7813] Farkas, J., Ed., Bragg, N., Unbehagen, P., Parsons, G., 1791 Ashwood-Smith, P., and C. Bowers, "IS-IS Path Control and 1792 Reservation", RFC 7813, DOI 10.17487/RFC7813, June 2016, 1793 . 1795 [TEAS] IETF, "Traffic Engineering Architecture and Signaling 1796 Working Group", 1797 . 1799 Authors' Addresses 1801 Norman Finn 1802 Huawei 1803 3755 Avocado Blvd. 1804 PMB 436 1805 La Mesa, California 91941 1806 US 1808 Phone: +1 925 980 6430 1809 Email: norman.finn@mail01.huawei.com 1811 Pascal Thubert 1812 Cisco Systems 1813 Village d'Entreprises Green Side 1814 400, Avenue de Roumanille 1815 Batiment T3 1816 Biot - Sophia Antipolis 06410 1817 FRANCE 1819 Phone: +33 4 97 23 26 34 1820 Email: pthubert@cisco.com 1822 Balazs Varga 1823 Ericsson 1824 Magyar tudosok korutja 11 1825 Budapest 1117 1826 Hungary 1828 Email: balazs.a.varga@ericsson.com 1829 Janos Farkas 1830 Ericsson 1831 Magyar tudosok korutja 11 1832 Budapest 1117 1833 Hungary 1835 Email: janos.farkas@ericsson.com