idnits 2.17.1 draft-finn-detnet-architecture-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 637 has weird spacing: '...mediate inter...' == Line 640 has weird spacing: '...mediate inter...' == Line 836 has weird spacing: '...e stack v ...' -- The document date (July 25, 2016) is 2824 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'AVnu' is defined on line 1233, but no explicit reference was found in the text == Unused Reference: 'HART' is defined on line 1242, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-architecture' is defined on line 1261, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-tsch' is defined on line 1266, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-roll-rpl-industrial-applicability' is defined on line 1284, but no explicit reference was found in the text == Unused Reference: 'I-D.svshah-tsvwg-deterministic-forwarding' is defined on line 1290, but no explicit reference was found in the text == Unused Reference: 'IEEE802.11-2012' is defined on line 1295, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1AS-2011' is defined on line 1301, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1TSNTG' is defined on line 1345, but no explicit reference was found in the text == Unused Reference: 'IEEE802154' is defined on line 1359, but no explicit reference was found in the text == Unused Reference: 'IEEE802154e' is defined on line 1365, but no explicit reference was found in the text == Unused Reference: 'ISA95' is defined on line 1377, but no explicit reference was found in the text == Unused Reference: 'ODVA' is defined on line 1381, but no explicit reference was found in the text == Unused Reference: 'Profinet' is defined on line 1388, but no explicit reference was found in the text == Unused Reference: 'RFC5673' is defined on line 1439, but no explicit reference was found in the text == Unused Reference: 'WirelessHART' is defined on line 1467, but no explicit reference was found in the text == Outdated reference: A later version (-04) exists of draft-dt-detnet-dp-alt-01 == Outdated reference: A later version (-30) exists of draft-ietf-6tisch-architecture-10 == Outdated reference: A later version (-09) exists of draft-ietf-detnet-problem-statement-00 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-10 -- Obsolete informational reference (is this intentional?): RFC 5316 (Obsoleted by RFC 9346) Summary: 0 errors (**), 0 flaws (~~), 24 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft P. Thubert 4 Intended status: Standards Track Cisco 5 Expires: January 26, 2017 M. Johas Teener 6 Broadcom 7 July 25, 2016 9 Deterministic Networking Architecture 10 draft-finn-detnet-architecture-07 12 Abstract 14 Deterministic Networking (DetNet) provides a capability to carry 15 specified unicast or multicast data flows for real-time applications 16 with extremely low data loss rates and bounded latency. Techniques 17 used include: 1) reserving data plane resources for individual (or 18 aggregated) DetNet flows in some or all of the intermediate nodes 19 (e.g. bridges or routers) along the path of the flow; 2) providing 20 explicit routes for DetNet flows that do not rapidly change with the 21 network topology; and 3) distributing data from DetNet flow packets 22 over time and/or space to ensure delivery of each packet's data' in 23 spite of the loss of a path. The capabilities can be managed by 24 configuration, or by manual or automatic network management. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on January 26, 2017. 43 Copyright Notice 45 Copyright (c) 2016 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 2.1. Terms used in this document . . . . . . . . . . . . . . . 4 63 2.2. IEEE 802 TSN to DetNet dictionary . . . . . . . . . . . . 6 64 3. Providing the DetNet Quality of Service . . . . . . . . . . . 6 65 3.1. Congestion protection . . . . . . . . . . . . . . . . . . 8 66 3.2. Explicit routes . . . . . . . . . . . . . . . . . . . . . 9 67 3.3. Jitter Reduction . . . . . . . . . . . . . . . . . . . . 9 68 3.4. Packet Replication and Elimination . . . . . . . . . . . 10 69 3.5. Packet encoding for service protection . . . . . . . . . 12 70 4. DetNet Architecture . . . . . . . . . . . . . . . . . . . . . 12 71 4.1. Traffic Engineering for DetNet . . . . . . . . . . . . . 12 72 4.1.1. The Application Plane . . . . . . . . . . . . . . . . 12 73 4.1.2. The Controller Plane . . . . . . . . . . . . . . . . 13 74 4.1.3. The Network Plane . . . . . . . . . . . . . . . . . . 13 75 4.2. DetNet flows . . . . . . . . . . . . . . . . . . . . . . 14 76 4.2.1. Source guarantees . . . . . . . . . . . . . . . . . . 15 77 4.2.2. Incomplete Networks . . . . . . . . . . . . . . . . . 16 78 4.3. Queuing, Shaping, Scheduling, and Preemption . . . . . . 16 79 4.4. Coexistence with normal traffic . . . . . . . . . . . . . 17 80 4.5. Fault Mitigation . . . . . . . . . . . . . . . . . . . . 17 81 4.6. Representative Protocol Stack Model . . . . . . . . . . . 18 82 4.7. Exporting flow identification . . . . . . . . . . . . . . 20 83 4.8. Advertising resources, capabilities and adjacencies . . . 22 84 4.9. Provisioning model . . . . . . . . . . . . . . . . . . . 22 85 4.9.1. Centralized Path Computation and Installation . . . . 22 86 4.9.2. Distributed Path Setup . . . . . . . . . . . . . . . 23 87 4.10. Scaling to larger networks . . . . . . . . . . . . . . . 23 88 4.11. Connected islands vs. networks . . . . . . . . . . . . . 23 89 5. Compatibility with Layer-2 . . . . . . . . . . . . . . . . . 24 90 6. Open Questions . . . . . . . . . . . . . . . . . . . . . . . 24 91 6.1. Flat vs. hierarchical control . . . . . . . . . . . . . . 24 92 6.2. Peer-to-peer reservation protocol . . . . . . . . . . . . 25 93 6.3. Wireless media interactions . . . . . . . . . . . . . . . 25 94 7. Security Considerations . . . . . . . . . . . . . . . . . . . 25 95 8. Privacy Considerations . . . . . . . . . . . . . . . . . . . 26 96 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 97 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 26 98 11. Access to IEEE 802.1 documents . . . . . . . . . . . . . . . 26 99 12. Informative References . . . . . . . . . . . . . . . . . . . 27 100 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32 102 1. Introduction 104 Deterministic Networking (DetNet) is a service that can be offered by 105 a network to DetNet flows. DetNet provides these flows extremely low 106 packet loss rates and assured maximum end-to-end delivery latency. 107 This is accomplished by dedicating network resources such as link 108 bandwidth and buffer space to DetNet flows and/or classes of DetNet 109 flows, and by replicating packets along multiple paths. Unused 110 reserved resources are available to non-DetNet packets. 112 The Deterministic Networking Problem Statement 113 [I-D.ietf-detnet-problem-statement] introduces Deterministic 114 Networking, and Deterministic Networking Use Cases 115 [I-D.ietf-detnet-use-cases] summarizes the need for it. See 116 [I-D.dt-detnet-dp-alt] for a discussion of specific techniques that 117 can be used to identify DetNet Flows and assign them to specific 118 paths through a network. 120 A goal of DetNet is a converged network in all respects. That is, 121 the presence of DetNet flows does not preclude non-DetNet flows, and 122 the benefits offered DetNet flows should not, except in extreme 123 cases, prevent existing QoS mechanisms from operating in a normal 124 fashion, subject to the bandwidth required for the DetNet flows. A 125 single source-destination pair can trade both DetNet and non-DetNet 126 flows. End systems and applications need not instantiate special 127 interfaces for DetNet flows. Networks are not restricted to certain 128 topologies; connectivity is not restricted. Any application that 129 generates a data flow that can be usefully characterized as having a 130 maximum bandwidth should be able to take advantage of DetNet, as long 131 as the necessary resources can be reserved. Reservations can be made 132 by the application itself, via network management, by an applications 133 controller, or by other means. 135 Many applications of interest to Deterministic Networking require the 136 ability to synchronize the clocks in end systems to a sub-microsecond 137 accuracy. Some of the queue control techniques defined in 138 Section 4.3 also require time synchronization among relay and transit 139 nodes. The means used to achieve time synchronization are not 140 addressed in this document. DetNet should accommodate various 141 synchronization techniques and profiles that are defined elsewhere to 142 solve exchange time in different market segments. 144 The present document is an individual contribution, but it is 145 intended by the authors for adoption by the DetNet working group. 147 2. Terminology 149 2.1. Terms used in this document 151 The following special terms are used in this document in order to 152 avoid the assumption that a given element in the architecture does or 153 does not have Internet Protocol stack, functions as a router, bridge, 154 firewall, or otherwise plays a particular role at Layer-2 or higher. 156 destination 157 An end system capable of receiving a DetNet flow. 159 DetNet domain 160 The portion of a network that is DetNet aware. It includes 161 end systems and other DetNet nodes. 163 DetNet flow 164 A DetNet flow is a sequence of packets to which the DetNet 165 service is to be provided. 167 DetNet compound flow and DetNet member flow 168 A DetNet compound flow is a DetNet flow that has been 169 separated into multiple duplicate DetNet member flows, which 170 are eventually merged back into a single DetNet compound 171 flow, at the DetNet transport layer. "Compound" and "member" 172 are strictly relative to each other, not absolutes; a DetNet 173 compound flow comprising multiple DetNet member flows can, in 174 turn, be a member of a higher-order compound. 176 DetNet intermediate node 177 A DetNet relay node or transit node. 179 DetNet edge node 180 An instance of a DetNet relay node that includes either a 181 DetNet service layer proxy function for DetNet service 182 protection (e.g. the addition or removal of packet sequencing 183 information) for one or more end systems, or starts or 184 terminates congestion protection at the DetNet transport 185 layer, analogous to a Label Edge Router (LER). 187 end system 188 Commonly called a "host" or "node" in IETF documents, and an 189 "end station" is IEEE 802 documents. End systems of interest 190 to this document are either sources or destinations of DetNet 191 flows. And end system may or may not be DetNet transport 192 layer aware or DetNet service layer aware. 194 link 195 A connection between two DetNet nodes. It may be composed of 196 a physical link or a sub-network technology that can provide 197 appropriate traffic delivery for DetNet flows. 199 DetNet node 200 A DetNet aware end system, transit node, or relay node. 201 "DetNet" may be omitted in some text. 203 Detnet relay node 204 A DetNet node including a service layer function that 205 interconnects different DetNet transport layer paths to 206 provide service protection. A DetNet relay node can be a 207 bridge, a router, a firewall, or any other system that 208 participates in the DetNet service layer. It typically 209 incorporates DetNet transport layer functions as well, in 210 which case it is collocated with a transit node. 212 reservation 213 A trail of configuration between source to destination(s) 214 through transit nodes and subnets associated with a DetNet 215 flow, to provide congestion protection. 217 DetNet service layer 218 The layer at which service protection is provided, either 219 packet sequencing, replication, and elimination (Section 3.4) 220 or network coding (Section 3.5). 222 source 223 An end system capable of sourcing a DetNet flow. 225 DetNet transit node 226 A node operating at the DetNet transport layer, that utilizes 227 link layer and/or network layer switching across multiple 228 links and/or sub-networks to provide paths for DetNet service 229 layer functions. Optionally provides congestion protection 230 over those paths. An MPLS LSR is an example of a DetNet 231 transit node. 233 DetNet transport layer 234 The layer that optionally provides congestion protection for 235 DetNet flows over paths provided by the underlying network. 237 2.2. IEEE 802 TSN to DetNet dictionary 239 This section also serves as a dictionary for translating from the 240 terms used by the IEEE 802 Time-Sensitive Networking (TSN) Task Group 241 to those of the DetNet WG. 243 Listener 244 The IEEE 802 term for a destination of a DetNet flow. 246 relay system 247 The IEEE 802 term for a DetNet intermediate node. 249 Stream 250 The IEEE 802 term for a DetNet flow. 252 Talker 253 The IEEE 802 term for the source of a DetNet flow. 255 3. Providing the DetNet Quality of Service 257 The DetNet Quality of Service can be expressed in terms of: 259 o Minimum and maximum end-to-end latency from source to destination; 260 timely delivery and jitter avoidance derive from these constraints 262 o Probability of loss of a packet, under various assumptions as to 263 the operational states of the nodes and links. A derived property 264 is whether it is acceptable to deliver a duplicate packet, which 265 is an inherent risk in highly reliable and/or broadcast 266 transmissions 268 It is a distinction of DetNet that it is concerned solely with worst- 269 case values for the end-to-end latency. Average, mean, or typical 270 values are of no interest, because they do not affect the ability of 271 a real-time system to perform its tasks. In general, a trivial 272 priority-based queuing scheme will give better average latency to a 273 data flow than DetNet, but of course, the worst-case latency can be 274 essentially unbounded. 276 Three techniques are used by DetNet to provide these qualities of 277 service: 279 o Congestion protection (Section 3.1). 281 o Explicit routes (Section 3.2). 283 o Service protection. 285 Congestion protection operates by reserving resources along the path 286 of a DetNet Flow, e.g. buffer space or link bandwidth. Congestion 287 protection greatly reduces, or even eliminates entirely, packet loss 288 due to output packet congestion within the network, but it can only 289 be supplied to a DetNet flow that is limited at the source to a 290 maximum packet size and transmission rate. 292 Congestion protection addresses both of the DetNet QoS requirements 293 (latency and packet loss). Given that DetNet nodes have a finite 294 amount of buffer space, congestion protection necessarily results in 295 a maximum end-to-end latency. It also addresses the largest 296 contribution to packet loss, which is buffer congestion. 298 After congestion, the most important contributions to packet loss are 299 typically from random media errors and equipment failures. Service 300 protection is the name for the mechanisms used by DetNet to address 301 these losses. The mechanisms employed are constrained by the 302 requirement to meet the users' latency requirements. Packet 303 replication and elimination (Section 3.4) packet encoding Section 3.5 304 are described in this document to provide service protection; others 305 may be found. Both mechanisms distribute the contents of DetNet 306 flows over multiple paths in time and/or space, so that the loss of 307 some of the paths does need not cause the loss of any packets. The 308 paths are typically (but not necessarily) explicit routes, so that 309 they cannot suffer temporary interruptions caused by the 310 reconvergence of routing or bridging protocols. 312 These three techniques can be applied independently, giving eight 313 possible combinations, including none (no DetNet), although some 314 combinations are of wider utility than others. This separation keeps 315 the protocol stack coherent and maximizes interoperability with 316 existing and developing standards in this (IETF) and other Standards 317 Development Organizations. Some examples of typical expected 318 combinations: 320 o Explicit routes plus service protection are exactly the techniques 321 employed by [HSR-PRP]. Explicit routes are achieved by limiting 322 the physical topology of the network, and the sequentialization, 323 replication, and duplicate elimination are facilitated by packet 324 tags added at the front or the end of Ethernet frames. 326 o Congestion protection alone is is offered by IEEE 802.1 Audio 327 Video bridging [IEEE802.1BA-2011]. As long as the network suffers 328 no failures, zero congestion loss can be achieved through the use 329 of a reservation protocol (MSRP), shapers in every bridge, and a 330 bit of network calculus. 332 o Using all three together gives maximum protection. 334 There are, of course, simpler methods available (and employed, today) 335 to achieve levels of latency and packet loss that are satisfactory 336 for many applications. Prioritization and over-provisioning is one 337 such technique. However, these methods generally work best in the 338 absence of any significant amount of non-critical traffic in the 339 network (if, indeed, such traffic is supported at all), or work only 340 if the critical traffic constitutes only a small portion of the 341 network's theoretical capacity, or work only if all systems are 342 functioning properly, or in the absence of actions by end systems 343 that disrupt the network's operations. 345 There are any number of methods in use, defined, or in progress for 346 accomplishing each of the above techniques. It is expected that this 347 DetNet Architecture will assist various vendors, users, and/or 348 "vertical" Standards Development Organizations (dedicated to a single 349 industry) to make selections among the available means of 350 implementing DetNet networks. 352 3.1. Congestion protection 354 The primary means by which DetNet achieves its QoS assurances is to 355 reduce, or even completely eliminate, congestion at an output port as 356 a cause of packet loss. Given that a DetNet flow cannot be 357 throttled, this can be achieved only by the provision of sufficient 358 buffer storage at each hop through the network to ensure that no 359 packets are dropped due to a lack of buffer storage. 361 Ensuring adequate buffering requires, in turn, that the source, and 362 every intermediate node along the path to the destination (or nearly 363 every node -- see Section 4.2.2) be careful to regulate its output to 364 not exceed the data rate for any DetNet flow, except for brief 365 periods when making up for interfering traffic. Any packet sent 366 ahead of its time potentially adds to the number of buffers required 367 by the next hop, and may thus exceed the resources allocated for a 368 particular DetNet flow. 370 The low-level mechanisms described in Section 4.3 provide the 371 necessary regulation of transmissions by an end system or 372 intermediate node to provide congestion protection. The reservation 373 of the bandwidth and buffers for a DetNet flow requires the 374 provisioning described in Section 4.9. A DetNet node may have other 375 resources requiring allocation and/or scheduling, that might 376 otherwise be over-subscribed and trigger the rejection of a 377 reservation. 379 3.2. Explicit routes 381 In networks controlled by typical peer-to-peer protocols such as IEEE 382 802.1 ISIS bridged networks or IETF OSPF routed networks, a network 383 topology event in one part of the network can impact, at least 384 briefly, the delivery of data in parts of the network remote from the 385 failure or recovery event. Thus, even redundant paths through a 386 network, if controlled by the typical peer-to-peer protocols, do not 387 eliminate the chances of brief losses of contact. 389 Many real-time networks rely on physical rings or chains of two-port 390 devices, with a relatively simple ring control protocol. This 391 supports redundant paths for service protection with a minimum of 392 wiring. As an additional benefit, ring topologies can often utilize 393 different topology management protocols than those used for a mesh 394 network, with a consequent reduction in the response time to topology 395 changes. Of course, this comes at some cost in terms of increased 396 hop count, and thus latency, for the typical path. 398 In order to get the advantages of low hop count and still ensure 399 against even very brief losses of connectivity, DetNet employs 400 explicit routes, where the path taken by a given DetNet flow does not 401 change, at least immediately, and likely not at all, in response to 402 network topology events. Service protection (Section 3.4 or 403 Section 3.5) over explicit routes provides a high likelihood of 404 continuous connectivity. Explicit routes are commonly used in MPLS 405 TE LSPs. 407 3.3. Jitter Reduction 409 A core objective of DetNet is to enable the convergence of Non-IP 410 networks onto a common network infrastructure. This requires the 411 accurate emulation of currently deployed mission-specific networks, 412 which typically rely on point-to-point analog (e.g. 4-20mA 413 modulation) and serial-digital cables (or buses) for highly reliable, 414 synchronized and jitter-free communications. While the latency of 415 analog transmissions is basically the speed of light, legacy serial 416 links are usually slow (in the order of Kbps) compared to, say, GigE, 417 and some latency is usually acceptable. What is not acceptable is 418 the introduction of excessive jitter, which may, for instance, affect 419 the stability of control systems. 421 Applications that are designed to operate on serial links usually do 422 not provide services to recover the jitter, because jitter simply 423 does not exists there. Streams of information are expected to be 424 delivered in-order and the precise time of reception influences the 425 processes. In order to converge such existing applications, there is 426 a desire to emulate all properties of the serial cable, such as clock 427 transportation, perfect flow isolation and fixed latency. While 428 minimal jitter (in the form of specifying minimum, as well as 429 maximum, end-to-end latency) is supported by DetNet, there are 430 practical limitations on packet-based networks in this regard. In 431 general, users are encouraged to use, instead of, "do this when you 432 get the packet," a combination of: 434 o Sub-microsecond time synchronization among all source and 435 destination end systems, and 437 o Time-of-execution fields in the application packets. 439 Jitter reduction is provided by the mechanisms described in 440 Section 4.3 that also provide congestion protection. 442 3.4. Packet Replication and Elimination 444 After congestion loss has been eliminated, the most important causes 445 of packet loss are random media and/or memory faults, and equipment 446 failures. Both causes of packet loss can be greatly reduced by 447 spreading the data in a packet over multiple transmissions. One such 448 method for service protection is described in this section, which 449 sends the same packets over multiple paths. See also Section 3.5. 451 Packet replication and elimination, also known as seamless redundancy 452 [HSR-PRP], or 1+1 hitless protection, is a function of the DetNet 453 service layer. It involves three capabilities: 455 o Providing sequencing information, once, at or near the source, to 456 the packets of a DetNet compound flow. This may be done by adding 457 a sequence number or time stamp as part of DetNet, or may be 458 inherent in the packet, e.g. in a transport protocol, or 459 associated to other physical properties such as the precise time 460 (and radio channel) of reception of the packet. Section 3.2. 462 o Replicating these packets into multiple DetNet member flows and, 463 typically, sending them along at least two different paths to the 464 destination(s), e.g. over the explicit routes of 466 o Eliminating duplicated packets. This may be done at any step 467 along the path to save network resources further down, in 468 particular if multiple Replication points exist. But the most 469 common case is to perform this operation at the very edge of the 470 DetNet network, preferably in or near the receiver. 472 This function is a "hitless" version of, e.g., the 1+1 linear 473 protection in [RFC6372]. That is, instead of switching from one flow 474 to the other when a failure of a flow is detected, DetNet combines 475 both flows, and performs a packet-by-packet selection of which to 476 discard, based on sequence number. 478 In the simplest case, this amounts to replicating each packet in a 479 source that has two interfaces, and conveying them through the 480 network, along separate paths, to the similarly dual-homed 481 destinations, that discard the extras. This ensures that one path 482 (with zero congestion loss) remains, even if some intermediate node 483 fails. The sequence numbers can also be used for loss detection and 484 for re-ordering. 486 Detnet relay nodes in the network can provide replication and 487 elimination facilities at various points in the network, so that 488 multiple failures can be accommodated. 490 This is shown in the following figure, where the two relay nodes each 491 replicate (R) the DetNet flow on input, sending the DetNet member 492 flows to both the other relay node and to the end system, and 493 eliminate duplicates (E) on the output interface to the right-hand 494 end system. Any one link in the network can fail, and the Detnet 495 compound flow can still get through. Furthermore, two links can 496 fail, as long as they are in different segments of the network. 498 Packet replication and elimination 500 > > > > > > > > > relay > > > > > > > > 501 > /------------+ R node E +------------\ > 502 > / v + ^ \ > 503 end R + v | ^ + E end 504 system + v | ^ + system 505 > \ v + ^ / > 506 > \------------+ R relay E +-----------/ > 507 > > > > > > > > > node > > > > > > > > 509 Figure 1 511 Note that packet replication and elimination does not react to and 512 correct failures; it is entirely passive. Thus, intermittent 513 failures, mistakenly created packet filters, or misrouted data is 514 handled just the same as the equipment failures that are detected 515 handled by typical routing and bridging protocols. 517 If packet replication and elimination is used over paths providing 518 congestion protection (Section 3.1), and member flows that take 519 different-length paths through the network are combined, a merge 520 point may require extra buffering to equalize the delays over the 521 different paths. This equalization ensures that the resultant 522 compound flow will not exceed its contracted bandwidth even after one 523 or the other of the paths is restored after a failure. 525 3.5. Packet encoding for service protection 527 There are methods for using multiple paths to provide service 528 protection that involve encoding the information in a packet 529 belonging to a DetNet flow into multiple transmission units, 530 typically combining information from multiple packets into any given 531 transmission unit. Such techniques may be applicable for use as a 532 DetNet service protection technique, assuming that the DetNet users' 533 needs for timeliness of delivery and freedom from interference with 534 misbehaving DetNet flows can be met. 536 No specific mechanisms are defined here, at this time. This section 537 will either be enhanced or removed. Contributions are invited. 539 4. DetNet Architecture 541 4.1. Traffic Engineering for DetNet 543 Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines 544 traffic-engineering architectures for generic applicability across 545 packet and non-packet networks. From TEAS perspective, Traffic 546 Engineering (TE) refers to techniques that enable operators to 547 control how specific traffic flows are treated within their networks. 549 Because if its very nature of establishing explicit optimized paths, 550 Deterministic Networking can be seen as a new, specialized branch of 551 Traffic Engineering, and inherits its architecture with a separation 552 into planes. 554 The Deterministic Networking architecture is thus composed of three 555 planes, a (User) Application Plane, a Controller Plane, and a Network 556 Plane, which echoes that of Figure 1 of Software-Defined Networking 557 (SDN): Layers and Architecture Terminology [RFC7426].: 559 4.1.1. The Application Plane 561 Per [RFC7426], the Application Plane includes both applications and 562 services. In particular, the Application Plane incorporates the User 563 Agent, a specialized application that interacts with the end user / 564 operator and performs requests for Deterministic Networking services 565 via an abstract Flow Management Entity, (FME) which may or may not be 566 collocated with (one of) the end systems. 568 At the Application Plane, a management interface enables the 569 negotiation of flows between end systems. An abstraction of the flow 570 called a Traffic Specification (TSpec) provides the representation. 571 This abstraction is used to place a reservation over the (Northbound) 572 Service Interface and within the Application plane. It is associated 573 with an abstraction of location, such as IP addresses and DNS names, 574 to identify the end systems and eventually specify intermediate 575 nodes. 577 4.1.2. The Controller Plane 579 The Controller Plane corresponds to the aggregation of the Control 580 and Management Planes in [RFC7426], though Common Control and 581 Measurement Plane (CCAMP) [CCAMP] makes an additional distinction 582 between management and measurement. When the logical separation of 583 the Control, Measurement and other Management entities is not 584 relevant, the term Controller Plane is used for simplicity to 585 represent them all, and the term controller refers to any device 586 operating in that plane, whether is it a Path Computation entity or a 587 Network Management entity (NME). The Path Computation Element (PCE) 588 [PCE] is a core element of a controller, in charge of computing 589 Deterministic paths to be applied in the Network Plane. 591 A (Northbound) Service Interface enables applications in the 592 Application Plane to communicate with the entities in the Controller 593 Plane. 595 One or more PCE(s) collaborate to implement the requests from the FME 596 as Per-Flow Per-Hop Behaviors installed in the intermediate nodes for 597 each individual flow. The PCEs place each flow along a deterministic 598 sequence of intermediate nodes so as to respect per-flow constraints 599 such as security and latency, and optimize the overall result for 600 metrics such as an abstract aggregated cost. The deterministic 601 sequence can typically be more complex than a direct sequence and 602 include redundancy path, with one or more packet replication and 603 elimination points. 605 4.1.3. The Network Plane 607 The Network Plane represents the network devices and protocols as a 608 whole, regardless of the Layer at which the network devices operate. 609 It includes Forwarding Plane (data plane), Application, and 610 Operational Plane (control plane) aspects. 612 The network Plane comprises the Network Interface Cards (NIC) in the 613 end systems, which are typically IP hosts, and intermediate nodes, 614 which are typically IP routers and switches. Network-to-Network 615 Interfaces such as used for Traffic Engineering path reservation in 616 [RFC5921], as well as User-to-Network Interfaces (UNI) such as 617 provided by the Local Management Interface (LMI) between network and 618 end systems, are both part of the Network Plane, both in the control 619 plane and the data plane. 621 A Southbound (Network) Interface enables the entities in the 622 Controller Plane to communicate with devices in the Network Plane. 623 This interface leverages and extends TEAS to describe the physical 624 topology and resources in the Network Plane. 626 Flow Management Entity 628 End End 629 System System 631 -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 633 PCE PCE PCE PCE 635 -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 637 intermediate intermed. intermed. intermed. 638 Node Node Node Node 639 NIC NIC 640 intermediate intermed. intermed. intermed. 641 Node Node Node Node 643 Figure 2 645 The intermediate nodes (and eventually the end systems NIC) expose 646 their capabilities and physical resources to the controller (the 647 PCE), and update the PCE with their dynamic perception of the 648 topology, across the Southbound Interface. In return, the PCE(s) set 649 the per-flow paths up, providing a Flow Characterization that is more 650 tightly coupled to the intermediate node Operation than a TSpec. 652 At the Network plane, intermediate nodes may exchange information 653 regarding the state of the paths, between adjacent systems and 654 eventually with the end systems, and forward packets within 655 constraints associated to each flow, or, when unable to do so, 656 perform a last resort operation such as drop or declassify. 658 This specification focuses on the Southbound interface and the 659 operation of the Network Plane. 661 4.2. DetNet flows 662 4.2.1. Source guarantees 664 For the purposes of congestion protection, DetNet flows can be 665 synchronous or asynchronous. In synchronous DetNet flows, at least 666 the intermediate nodes (and possibly the end systems) are closely 667 time synchronized, typically to better than 1 microsecond. By 668 transmitting packets from different DetNet flows or classes of DetNet 669 flows at different times, using repeating schedules synchronized 670 among the intermediate nodes, resources such as buffers and link 671 bandwidth can be shared over the time domain among different DetNet 672 flows. There is a tradeoff among techniques for synchronous DetNet 673 flows between the burden of fine-grained scheduling and the benefit 674 of reducing the required resources, especially buffer space. 676 In contrast, asynchronous DetNet flows are not coordinated with a 677 fine-grained schedule, so relay and end systems must assume worst- 678 case interference among DetNet flows contending for buffer resources. 679 Asynchronous DetNet flows are characterized by: 681 o A maximum packet size; 683 o An observation interval; and 685 o A maximum number of transmissions during that observation 686 interval. 688 These parameters, together with knowledge of the protocol stack used 689 (and thus the size of the various headers added to a packet), limit 690 the number of bit times per observation interval that the DetNet flow 691 can occupy the physical medium. 693 The source promises that these limits will not be exceeded. If the 694 source transmits less data than this limit allows, the unused 695 resources such as link bandwidth can be made available by the system 696 to non-DetNet packets. However, making those resources available to 697 DetNet packets in other DetNet flows would serve no purpose. Those 698 other DetNet flows have their own dedicated resources, on the 699 assumption that all DetNet flows can use all of their resources over 700 a long period of time. 702 Note that there is no provision in DetNet for throttling DetNet flows 703 (reducing the transmission rate via feedback); the assumption is that 704 a DetNet flow, to be useful, must be delivered in its entirety. That 705 is, while any useful application is written to expect a certain 706 number of lost packets, the real-time applications of interest to 707 DetNet demand that the loss of data due to the network is 708 extraordinarily infrequent. 710 Although DetNet strives to minimize the changes required of an 711 application to allow it to shift from a special-purpose digital 712 network to an Internet Protocol network, one fundamental shift in the 713 behavior of network applications is impossible to avoid: the 714 reservation of resources before the application starts. In the first 715 place, a network cannot deliver finite latency and practically zero 716 packet loss to an arbitrarily high offered load. Secondly, achieving 717 practically zero packet loss for unthrottled (though bandwidth 718 limited) DetNet flows means that bridges and routers have to dedicate 719 buffer resources to specific DetNet flows or to classes of DetNet 720 flows. The requirements of each reservation have to be translated 721 into the parameters that control each system's queuing, shaping, and 722 scheduling functions and delivered to the hosts, bridges, and 723 routers. 725 4.2.2. Incomplete Networks 727 The presence in the network of transit nodes or subnets that are not 728 fully capable of offering DetNet services complicates the ability of 729 the intermediate nodes and/or controller to allocate resources, as 730 extra buffering, and thus extra latency, must be allocated at points 731 downstream from the non-DetNet intermediate node for a DetNet flow. 733 4.3. Queuing, Shaping, Scheduling, and Preemption 735 DetNet achieves congestion protection and bounded delivery latency by 736 reserving bandwidth and buffer resources at every hop along the path 737 of the DetNet flow. The reservation itself is not sufficient, 738 however. Implementors and users of a number of proprietary and 739 standard real-time networks have found that standards for specific 740 data plane techniques are required to enable these assurances to be 741 made in a multi-vendor network. The fundamental reason is that 742 latency variation in one system results in the need for extra buffer 743 space in the next-hop system(s), which in turn, increases the worst- 744 case per-hop latency. 746 Standard queuing and transmission selection algorithms allow a 747 central controller to compute the latency contribution of each 748 transit node to the end-to-end latency, to compute the amount of 749 buffer space required in each transit node for each incremental 750 DetNet flow, and most importantly, to translate from a flow 751 specification to a set of values for the managed objects that control 752 each relay or end system. The IEEE 802 has specified (and is 753 specifying) a set of queuing, shaping, and scheduling algorithms that 754 enable each transit node (bridge or router), and/or a central 755 controller, to compute these values. These algorithms include: 757 o A credit-based shaper [IEEE802.1Q-2014] Clause 34. 759 o Time-gated queues governed by a rotating time schedule, 760 synchronized among all transit nodes [IEEE802.1Qbv]. 762 o Synchronized double (or triple) buffers driven by synchronized 763 time ticks. [IEEE802.1Qch]. 765 o Pre-emption of an Ethernet packet in transmission by a packet with 766 a more stringent latency requirement, followed by the resumption 767 of the preempted packet [IEEE802.1Qbu], [IEEE802.3br]. 769 While these techniques are currently embedded in Ethernet and 770 bridging standards, we can note that they are all, except perhaps for 771 packet preemption, equally applicable to other media than Ethernet, 772 and to routers as well as bridges. 774 4.4. Coexistence with normal traffic 776 A DetNet network supports the dedication of a high proportion (e.g. 777 75%) of the network bandwidth to DetNet flows. But, no matter how 778 much is dedicated for DetNet flows, it is a goal of DetNet to coexist 779 with existing Class of Service schemes (e.g., DiffServ). It is also 780 important that non-DetNet traffic not disrupt the DetNet flow, of 781 course (see Section 4.5 and Section 7). For these reasons: 783 o Bandwidth (transmission opportunities) not utilized by a DetNet 784 flow are available to non-DetNet packets (though not to other 785 DetNet flows). 787 o DetNet flows can be shaped or scheduled, in order to ensure that 788 the highest-priority non-DetNet packet also is ensured a worst- 789 case latency (at any given hop). 791 o When transmission opportunities for DetNet flows are scheduled in 792 detail, then the algorithm constructing the schedule should leave 793 sufficient opportunities for non-DetNet packets to satisfy the 794 needs of the users of the network. Detailed scheduling can also 795 permit the time-shared use of buffer resources by different DetNet 796 flows. 798 Ideally, the net effect of the presence of DetNet flows in a network 799 on the non-DetNet packets is primarily a reduction in the available 800 bandwidth. 802 4.5. Fault Mitigation 804 One key to building robust real-time systems is to reduce the 805 infinite variety of possible failures to a number that can be 806 analyzed with reasonable confidence. DetNet aids in the process by 807 providing filters and policers to detect DetNet packets received on 808 the wrong interface, or at the wrong time, or in too great a volume, 809 and to then take actions such as discarding the offending packet, 810 shutting down the offending DetNet flow, or shutting down the 811 offending interface. 813 It is also essential that filters and service remarking be employed 814 at the network edge to prevent non-DetNet packets from being mistaken 815 for DetNet packets, and thus impinging on the resources allocated to 816 DetNet packets. 818 There exist techniques, at present and/or in various stages of 819 standardization, that can perform these fault mitigation tasks that 820 deliver a high probability that misbehaving systems will have zero 821 impact on well-behaved DetNet flows, except of course, for the 822 receiving interface(s) immediately downstream of the misbehaving 823 device. Examples of such techniques include traffic policing 824 functions (e.g. [RFC2475]) and separating flows into per-flow rate- 825 limited queues. 827 4.6. Representative Protocol Stack Model 829 Figure 3 illustrates a conceptual DetNet data plane layering model. 830 One may compare it to that in [IEEE802.1CB], Annex C, a work in 831 progress. 833 DetNet data plane protocol stack 835 | packets going | ^ packets coming ^ 836 v down the stack v | up the stack | 837 +----------------------+ +-----------------------+ 838 | Source | | Destination | 839 +----------------------+ +-----------------------+ 840 | Service layer | | Service layer | 841 | Packet sequencing | | Duplicate elimination | 842 | Flow duplication | | Flow merging | 843 | Packet encoding | | Packet decoding | 844 +----------------------+ +-----------------------+ 845 | Transport layer | | Transport layer | 846 | Congestion prot. | | Congestion prot. | 847 +----------------------+ +-----------------------+ 848 | Lower layers | | Lower layers | 849 +----------------------+ +-----------------------+ 850 v ^ 851 \_________________________/ 853 Figure 3 855 Not all layers are required for any given application, or even for 856 any given network. The layers are, from top to bottom: 858 Application 859 Shown as "source" and "destination" in the diagram. 861 OAM 862 Operations, Administration, and Maintenance leverages in-band 863 and out-of-and signaling that validates whether the service 864 is effectively obtained within QoS constraints. OAM is not 865 shown in Figure 3; it may reside in any number of the layers. 866 OAM can involve specific tagging added in the packets for 867 tracing implementation or network configuration errors; 868 traceability enables to find whether a packet is a replica, 869 which relay node performed the replication, and which segment 870 was intended for the replica. 872 Packet sequencing 873 As part of DetNet service protection, supplies the sequence 874 number for packet replication and elimination (Section 3.4). 875 Peers with Duplicate elimination. This layer is not needed 876 if a higher-layer transport protocol is expected to perform 877 any packet sequencing and duplicate elimination required by 878 the DetNet flow duplication. 880 Duplicate elimination 881 As part of the DetNet service layer, based on the sequenced 882 number supplied by its peer, packet sequencing, Duplicate 883 elimination discards any duplicate packets generated by 884 DetNet flow duplication. It can operate on member flows, 885 compound flows, or both. The duplication may also be 886 inferred from other information such as the precise time of 887 reception in a scheduled network. The duplicate elimination 888 layer may also perform resequencing of packets to restore 889 packet order in a flow that was disrupted by the loss of 890 packets on one or another of the multiple paths taken. 892 Flow duplication 893 As part of DetNet service protection, replicates packets that 894 belong to a DetNet compound flow into two or more DetNet 895 member flows. Note that this function is separate from 896 packet sequencing. Flow duplication can be an explicit 897 duplication and remarking of packets, or can be performed by, 898 for example, techniques similar to ordinary multicast 899 replication. Peers with DetNet flow merging. 901 Network flow merging 902 As part of DetNet service protection, merges DetNet member 903 flows together for packets coming up the stack belonging to a 904 specific DetNet compound flow. Peers with DetNet flow 905 duplication. DetNet flow merging, together with packet 906 sequencing, duplicate elimination, and DetNet flow 907 duplication, performs packet replication and elimination 908 (Section 3.4). 910 Packet encoding 911 As part of DetNet service protection, as an alternative to 912 packet sequencing and flow duplication, packet encoding 913 combines the information in multiple DetNet packets, perhaps 914 from different DetNet compound flows, and transmits that 915 information in packets on different DetNet member Flows. 916 Peers with Packet decoding. 918 Packet decoding 919 As part of DetNet service protection, as an alternative to 920 flow merging and duplicate elimination, packet decoding takes 921 packets from different DetNet member flows, and computes from 922 those packets the original DetNet packets from the compound 923 flows input to packet encoding. Peers with Packet encoding. 925 Congestio protection 926 The DetNet transport layer provides congestion protection. 927 See Section 4.3. The actual queuing and shaping mechaniss 928 are typically provided by underlying subnet layers, but since 929 these are can be closely associated with the means of 930 providing paths for DetNet flows (e.g. MPLS LSPs or {VLAN, 931 multicast destination MAC address} pairs), the path and the 932 congestion protection are conflated in this figure. 934 Note that the packet sequencing and duplication elimination functions 935 at the source and destination ends of a DetNet compound flow may be 936 performed either in the end system or in a DetNet edge node. The 937 reader must not confuse a DetNet edge function with other kinds of 938 edge functions, e.g. an Label Edge Router, although the two functions 939 may be performed together. The DetNet edge function is concerned 940 with sequencing packets belonging to DetNet flows. The LER with 941 encapsulating/decapsulating packets for transport, and is considered 942 part of the network underlying the DetNet transport layer. 944 4.7. Exporting flow identification 946 An interesting feature of DetNet, and one that invites 947 implementations that can be accused of "layering violations", is the 948 need for lower layers to be aware of specific flows at higher layers, 949 in order to provide specific queuing and shaping services for 950 specific flows. For example: 952 o A non-IP, strictly L2 source end system X may be sending multiple 953 flows to the same L2 destination end system Y. Those flows may 954 include DetNet flows with different QoS requirements, and may 955 include non-DetNet flows. 957 o A router may be sending any number of flows to another router. 958 Again, those flows may include DetNet flows with different QoS 959 requirements, and may include non-DetNet flows. 961 o Two routers may be separated by bridges. For these bridges to 962 perform any required per-flow queuing and shaping, they must be 963 able to identify the individual flows. 965 o A Label Edge Router (LERs) may have a Label Switched Path (LSP) 966 set up for handling traffic destined for a particular IP address 967 carrying only non-DetNet flows. If a DetNet flow to that same 968 address is requested, a separate LSP may be needed, in order that 969 all of the Label Switch Routers (LSRs) along the path to the 970 destination give that flow special queuing and shaping. 972 The need for a lower-level DetNet node to be aware of individual 973 higher-layer flows is not unique to DetNet. But, given the endless 974 complexity of layering and relayering over tunnels that is available 975 to network designers, DetNet needs to provide a model for flow 976 identification that is at least somewhat better than packet 977 inspection. That is not to say that packet inspection to layer 4 or 978 5 addresses will not be used, or the capability standardized; but, 979 there are alternatives. 981 A DetNet relay node can connect DetNet flows on different paths using 982 different flow identification methods. For example: 984 o A single unicast DetNet flow passing from router A through a 985 bridged network to router B may be assigned a {VLAN, multicast 986 destination MAC address} pair that is unique within that bridged 987 network. The bridges can then identify the flow without accessing 988 higher-layer headers. Of course, the receiving router must 989 recognize and accept that multicast MAC address. 991 o A DetNet flow passing from LSR A to LSR B may be assigned a 992 different label than that used for other flows to the same IP 993 destination. 995 In any of the above cases, it is possible that an existing DetNet 996 flow can be used as a carrier for multiple DetNet sub-flows. (Not to 997 be confused with DetNet compound vs. member flows.) Of course, this 998 requires that the aggregate DetNet flow be provisioned properly to 999 carry the sub-flows. 1001 Thus, rather than packet inspection, there is the option to export 1002 higher-layer information to the lower layer. The requirement to 1003 support one or the other method for flow identification (or both) is 1004 the essential complexity that DetNet brings to existing control plane 1005 models. 1007 4.8. Advertising resources, capabilities and adjacencies 1009 There are three classes of information that a central controller or 1010 decentralized control plane needs to know that can only be obtained 1011 from the end systems and/or transit nodes in the network. When using 1012 a peer-to-peer control plane, some of this information may be 1013 required by a system's neighbors in the network. 1015 o Details of the system's capabilities that are required in order to 1016 accurately allocate that system's resources, as well as other 1017 systems' resources. This includes, for example, which specific 1018 queuing and shaping algorithms are implemented (Section 4.3), the 1019 number of buffers dedicated for DetNet allocation, and the worst- 1020 case forwarding delay. 1022 o The dynamic state of an end or transit node's DetNet resources. 1024 o The identity of the system's neighbors, and the characteristics of 1025 the link(s) between the systems, including the length (in 1026 nanoseconds) of the link(s). 1028 4.9. Provisioning model 1030 4.9.1. Centralized Path Computation and Installation 1032 A centralized routing model, such as provided with a PCE (RFC 4655 1033 [RFC4655]), enables global and per-flow optimizations. (See 1034 Section 4.1.) The model is attractive but a number of issues are 1035 left to be solved. In particular: 1037 o Whether and how the path computation can be installed by 1) an end 1038 device or 2) a Network Management entity, 1040 o And how the path is set up, either by installing state at each hop 1041 with a direct interaction between the forwarding device and the 1042 PCE, or along a path by injecting a source-routed request at one 1043 end of the path. 1045 4.9.2. Distributed Path Setup 1047 Significant work on distributed path setup can be leveraged from MPLS 1048 Traffic Engineering, in both its GMPLS and non-GMPLS forms. The 1049 protocols within scope are Resource ReSerVation Protocol [RFC3209] 1050 [RFC3473](RSVP-TE), OSPF-TE [RFC4203] [RFC5392] and ISIS-TE [RFC5307] 1051 [RFC5316]. These should be viewed as starting points as there are 1052 feature specific extensions defined that may be applicable to DetNet. 1054 In a Layer-2 only environment, or as part of a layered approach to a 1055 mixed environment, IEEE 802.1 also has work, either completed or in 1056 progress. [IEEE802.1Q-2014] Clause 35 describes SRP, a peer-to-peer 1057 protocol for Layer-2 roughly analogous to RSVP [RFC2205]. 1058 [IEEE802.1Qca] defines how ISIS can provide multiple disjoint paths 1059 or distribution trees. Also in progress is [IEEE802.1Qcc], which 1060 expands the capabilities of SRP. 1062 The integration/interaction of the DetNet control layer with an 1063 underlying IEEE 802.1 sub-network control layer will need to be 1064 defined. 1066 4.10. Scaling to larger networks 1068 Reservations for individual DetNet flows require considerable state 1069 information in each transit node, especially when adequate fault 1070 mitigation (Section 4.5) is required. The DetNet data plane, in 1071 order to support larger numbers of DetNet flows, must support the 1072 aggregation of DetNet flows into tunnels, which themselves can be 1073 viewed by the transit nodes' data planes largely as individual DetNet 1074 flows. Without such aggregation, the per-relay system may limit the 1075 scale of DetNet networks. 1077 4.11. Connected islands vs. networks 1079 Given that users have deployed examples of the IEEE 802.1 TSN TG 1080 standards, which provide capabilities similar to DetNet, it is 1081 obvious to ask whether the IETF DetNet effort can be limited to 1082 providing Layer-2 connections (VPNs) between islands of bridged TSN 1083 networks. While this capability is certainly useful to some 1084 applications, and must not be precluded by DetNet, tunneling alone is 1085 not a sufficient goal for the DetNet WG. As shown in the 1086 Deterministic Networking Use Cases draft [I-D.ietf-detnet-use-cases], 1087 there are already deployments of Layer-2 TSN networks that are 1088 encountering the well-known problems of over-large broadcast domains. 1089 Routed solutions, and combinations routed/bridged solutions, are both 1090 required. 1092 5. Compatibility with Layer-2 1094 Standards providing similar capabilities for bridged networks (only) 1095 have been and are being generated in the IEEE 802 LAN/MAN Standards 1096 Committee. The present architecture describes an abstract model that 1097 can be applicable both at Layer-2 and Layer-3, and over links not 1098 defined by IEEE 802. It is the intention of the authors (and 1099 hopefully, as this draft progresses, of the DetNet Working Group) 1100 that IETF and IEEE 802 will coordinate their work, via the 1101 participation of common individuals, liaisons, and other means, to 1102 maximize the compatibility of their outputs. 1104 DetNet enabled end systems and intermediate nodes can be 1105 interconnected by sub-networks, i.e., Layer-2 technologies. These 1106 sub-networks will provide DetNet compatible service for support of 1107 DetNet traffic. Examples of sub-networks include 802.1TSN and a 1108 point-to-point OTN link. Of course, multi-layer DetNet systems may 1109 be possible too, where one DetNet appears as a sub-network, and 1110 provides service to, a higher layer DetNet system. 1112 6. Open Questions 1114 There are a number of architectural questions that will have to be 1115 resolved before this document can be submitted for publication. 1116 Aside from the obvious fact that this present draft is subject to 1117 change, there are specific questions to which the authors wish to 1118 direct the readers' attention. 1120 6.1. Flat vs. hierarchical control 1122 Boxes that are solely routers or solely bridges are rare in today's 1123 market. In a multi-tenant data center, multiple users' virtual 1124 Layer-2/Layer-3 topologies exist simultaneously, implemented on a 1125 network whose physical topology bears only accidental resemblance to 1126 the virtual topologies. 1128 While the forwarding topology (the bridges and routers) are an 1129 important consideration for a DetNet Flow Management Entity 1130 (Section 4.1.1), so is the purely physical topology. Ultimately, the 1131 model used by the management entities is based on boxes, queues, and 1132 links. The authors hope that the work of the TEAS WG will help to 1133 clarify exactly what model parameters need to be traded between the 1134 intermediate nodes and the controller(s). 1136 6.2. Peer-to-peer reservation protocol 1138 As described in Section 4.9.2, the DetNet WG needs to decide whether 1139 to support a peer-to-peer protocol for a source and a destination to 1140 reserve resources for a DetNet stream. Assuming that enabling the 1141 involvement of the source and/or destination is desirable (see 1142 Deterministic Networking Use Cases [I-D.ietf-detnet-use-cases]), it 1143 remains to decide whether the DetNet WG will make it possible to 1144 deploy at least some DetNet capabilities in a network using only a 1145 peer-to-peer protocol, without a central controller. 1147 (Note that a UNI (see Section 4.1.3) between an end system and a 1148 DetNet edge node, for sources and/or listeners to request DetNet 1149 services, can be either the first hop of a per-to-peer reservation 1150 protocol, or can be deflected by the DetNet edge node to a central 1151 controller for resolution. Similarly, a decision by a central 1152 controller can be effected by the controller instructing the end 1153 system or DetNet edge node to initiate a per-to-peer protocol 1154 activity.) 1156 6.3. Wireless media interactions 1158 Deterministic Networking Use Cases [I-D.ietf-detnet-use-cases] 1159 illustrates cases where wireless media are needed in a DetNet 1160 network. Some wireless media in general use, such as IEEE 802.11 1161 [IEEE802.1Q-2014], have significantly higher packet loss rates than 1162 typical wired media, such as Ethernet [IEEE802.3-2012]. IEEE 802.11 1163 includes support for such features as MAC-layer acknowledgements and 1164 retransmissions. 1166 The techniques described in Section 3 are likely to improve the 1167 ability of a mixed wired/wireless network to offer the DetNet QoS 1168 features. The interaction of these techniques with the features of 1169 specific wireless media, although they may be significant, cannot be 1170 addressed in this document. It remains to be decided to what extent 1171 the DetNet WG will address them, and to what extent other WGs, e.g. 1172 6TiSCH, will do so. 1174 7. Security Considerations 1176 Security in the context of Deterministic Networking has an added 1177 dimension; the time of delivery of a packet can be just as important 1178 as the contents of the packet, itself. A man-in-the-middle attack, 1179 for example, can impose, and then systematically adjust, additional 1180 delays into a link, and thus disrupt or subvert a real-time 1181 application without having to crack any encryption methods employed. 1182 See [RFC7384] for an exploration of this issue in a related context. 1184 Furthermore, in a control system where millions of dollars of 1185 equipment, or even human lives, can be lost if the DetNet QoS is not 1186 delivered, one must consider not only simple equipment failures, 1187 where the box or wire instantly becomes perfectly silent, but bizarre 1188 errors such as can be caused by software failures. Because there is 1189 essential no limit to the kinds of failures that can occur, 1190 protecting against realistic equipment failures is indistinguishable, 1191 in most cases, from protecting against malicious behavior, whether 1192 accidental or intentional. See also Section 4.5. 1194 Security must cover: 1196 o the protection of the signaling protocol 1198 o the authentication and authorization of the controlling systems 1200 o the identification and shaping of the DetNet flows 1202 8. Privacy Considerations 1204 DetNet is provides a Quality of Service (QoS), and as such, does not 1205 directly raise any new privacy considerations. 1207 However, the requirement for every (or almost every) node along the 1208 path of a DetNet flow to identify DetNet flows may present an 1209 additional attack surface for privacy, should the DetNet paradigm be 1210 found useful in broader environments. 1212 9. IANA Considerations 1214 This document does not require an action from IANA. 1216 10. Acknowledgements 1218 The authors wish to thank Jouni Korhonen, Erik Nordmark, George 1219 Swallow, Rudy Klecka, Anca Zamfir, David Black, Thomas Watteyne, 1220 Shitanshu Shah, Craig Gunther, Rodney Cummings, Balazs Varga, 1221 Wilfried Steiner, Marcel Kiessling, Karl Weber, Janos Farkas, Ethan 1222 Grossman, Pat Thaler, and Lou Berger for their various contribution 1223 with this work. 1225 11. Access to IEEE 802.1 documents 1227 To access password protected IEEE 802.1 drafts, see the IETF IEEE 1228 802.1 information page at https://www.ietf.org/proceedings/52/slides/ 1229 bridge-0/tsld003.htm. 1231 12. Informative References 1233 [AVnu] http://www.avnu.org/, "The AVnu Alliance tests and 1234 certifies devices for interoperability, providing a simple 1235 and reliable networking solution for AV network 1236 implementation based on the Audio Video Bridging (AVB) 1237 standards.". 1239 [CCAMP] IETF, "Common Control and Measurement Plane", 1240 . 1242 [HART] www.hartcomm.org, "Highway Addressable Remote Transducer, 1243 a group of specifications for industrial process and 1244 control devices administered by the HART Foundation". 1246 [HSR-PRP] IEC, "High availability seamless redundancy (HSR) is a 1247 further development of the PRP approach, although HSR 1248 functions primarily as a protocol for creating media 1249 redundancy while PRP, as described in the previous 1250 section, creates network redundancy. PRP and HSR are both 1251 described in the IEC 62439 3 standard.", 1252 . 1255 [I-D.dt-detnet-dp-alt] 1256 Korhonen, J., Farkas, J., Mirsky, G., Thubert, P., 1257 Zhuangyan, Z., and L. Berger, "DetNet Data Plane Protocol 1258 and Solution Alternatives", draft-dt-detnet-dp-alt-01 1259 (work in progress), July 2016. 1261 [I-D.ietf-6tisch-architecture] 1262 Thubert, P., "An Architecture for IPv6 over the TSCH mode 1263 of IEEE 802.15.4", draft-ietf-6tisch-architecture-10 (work 1264 in progress), June 2016. 1266 [I-D.ietf-6tisch-tsch] 1267 Watteyne, T., Palattella, M., and L. Grieco, "Using 1268 IEEE802.15.4e TSCH in an IoT context: Overview, Problem 1269 Statement and Goals", draft-ietf-6tisch-tsch-06 (work in 1270 progress), March 2015. 1272 [I-D.ietf-detnet-problem-statement] 1273 Finn, N. and P. Thubert, "Deterministic Networking Problem 1274 Statement", draft-ietf-detnet-problem-statement-00 (work 1275 in progress), April 2016. 1277 [I-D.ietf-detnet-use-cases] 1278 Grossman, E., Gunther, C., Thubert, P., Wetterwald, P., 1279 Raymond, J., Korhonen, J., Kaneko, Y., Das, S., Zha, Y., 1280 Varga, B., Farkas, J., Goetz, F., and J. Schmitt, 1281 "Deterministic Networking Use Cases", draft-ietf-detnet- 1282 use-cases-10 (work in progress), July 2016. 1284 [I-D.ietf-roll-rpl-industrial-applicability] 1285 Phinney, T., Thubert, P., and R. Assimiti, "RPL 1286 applicability in industrial networks", draft-ietf-roll- 1287 rpl-industrial-applicability-02 (work in progress), 1288 October 2013. 1290 [I-D.svshah-tsvwg-deterministic-forwarding] 1291 Shah, S. and P. Thubert, "Deterministic Forwarding PHB", 1292 draft-svshah-tsvwg-deterministic-forwarding-04 (work in 1293 progress), August 2015. 1295 [IEEE802.11-2012] 1296 IEEE, "Wireless LAN Medium Access Control (MAC) and 1297 Physical Layer (PHY) Specifications", 2012, 1298 . 1301 [IEEE802.1AS-2011] 1302 IEEE, "Timing and Synchronizations (IEEE 802.1AS-2011)", 1303 2011, . 1306 [IEEE802.1BA-2011] 1307 IEEE, "AVB Systems (IEEE 802.1BA-2011)", 2011, 1308 . 1311 [IEEE802.1CB] 1312 IEEE, "Frame Replication and Elimination for Reliability 1313 (IEEE Draft P802.1CB)", 2016, 1314 . 1316 [IEEE802.1Q-2014] 1317 IEEE, "MAC Bridges and VLANs (IEEE 802.1Q-2014", 2014, 1318 . 1321 [IEEE802.1Qbu] 1322 IEEE, "Frame Preemption", 2016, 1323 . 1325 [IEEE802.1Qbv] 1326 IEEE, "Enhancements for Scheduled Traffic", 2016, 1327 . 1329 [IEEE802.1Qca] 1330 IEEE 802.1, "IEEE 802.1Qca Bridges and Bridged Networks - 1331 Amendment 24: Path Control and Reservation", IEEE 1332 P802.1Qca/D2.1 P802.1Qca, June 2015, 1333 . 1336 [IEEE802.1Qcc] 1337 IEEE, "Stream Reservation Protocol (SRP) Enhancements and 1338 Performance Improvements", 2016, 1339 . 1341 [IEEE802.1Qch] 1342 IEEE, "Cyclic Queuing and Forwarding", 2016, 1343 . 1345 [IEEE802.1TSNTG] 1346 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1347 Networks Task Group", 2013, 1348 . 1350 [IEEE802.3-2012] 1351 IEEE, "IEEE Stabdard for Ethernet", 2012, 1352 . 1355 [IEEE802.3br] 1356 IEEE, "Interspersed Express Traffic", 2016, 1357 . 1359 [IEEE802154] 1360 IEEE standard for Information Technology, "IEEE std. 1361 802.15.4, Part. 15.4: Wireless Medium Access Control (MAC) 1362 and Physical Layer (PHY) Specifications for Low-Rate 1363 Wireless Personal Area Networks", June 2011. 1365 [IEEE802154e] 1366 IEEE standard for Information Technology, "IEEE std. 1367 802.15.4e, Part. 15.4: Low-Rate Wireless Personal Area 1368 Networks (LR-WPANs) Amendment 1: MAC sublayer", April 1369 2012. 1371 [ISA100.11a] 1372 ISA/IEC, "ISA100.11a, Wireless Systems for Automation, 1373 also IEC 62734", 2011, < http://www.isa100wci.org/en- 1374 US/Documents/PDF/3405-ISA100-WirelessSystems-Future-broch- 1375 WEB-ETSI.aspx>. 1377 [ISA95] ANSI/ISA, "Enterprise-Control System Integration Part 1: 1378 Models and Terminology", 2000, . 1381 [ODVA] http://www.odva.org/, "The organization that supports 1382 network technologies built on the Common Industrial 1383 Protocol (CIP) including EtherNet/IP.". 1385 [PCE] IETF, "Path Computation Element", 1386 . 1388 [Profinet] 1389 http://us.profinet.com/technology/profinet/, "PROFINET is 1390 a standard for industrial networking in automation.", 1391 . 1393 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1394 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1395 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1396 September 1997, . 1398 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1399 and W. Weiss, "An Architecture for Differentiated 1400 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1401 . 1403 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1404 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1405 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1406 . 1408 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 1409 Switching (GMPLS) Signaling Resource ReserVation Protocol- 1410 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 1411 DOI 10.17487/RFC3473, January 2003, 1412 . 1414 [RFC4203] Kompella, K., Ed. and Y. Rekhter, Ed., "OSPF Extensions in 1415 Support of Generalized Multi-Protocol Label Switching 1416 (GMPLS)", RFC 4203, DOI 10.17487/RFC4203, October 2005, 1417 . 1419 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 1420 Element (PCE)-Based Architecture", RFC 4655, 1421 DOI 10.17487/RFC4655, August 2006, 1422 . 1424 [RFC5307] Kompella, K., Ed. and Y. Rekhter, Ed., "IS-IS Extensions 1425 in Support of Generalized Multi-Protocol Label Switching 1426 (GMPLS)", RFC 5307, DOI 10.17487/RFC5307, October 2008, 1427 . 1429 [RFC5316] Chen, M., Zhang, R., and X. Duan, "ISIS Extensions in 1430 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1431 Traffic Engineering", RFC 5316, DOI 10.17487/RFC5316, 1432 December 2008, . 1434 [RFC5392] Chen, M., Zhang, R., and X. Duan, "OSPF Extensions in 1435 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1436 Traffic Engineering", RFC 5392, DOI 10.17487/RFC5392, 1437 January 2009, . 1439 [RFC5673] Pister, K., Ed., Thubert, P., Ed., Dwars, S., and T. 1440 Phinney, "Industrial Routing Requirements in Low-Power and 1441 Lossy Networks", RFC 5673, DOI 10.17487/RFC5673, October 1442 2009, . 1444 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 1445 L., and L. Berger, "A Framework for MPLS in Transport 1446 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 1447 . 1449 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 1450 Profile (MPLS-TP) Survivability Framework", RFC 6372, 1451 DOI 10.17487/RFC6372, September 2011, 1452 . 1454 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1455 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1456 October 2014, . 1458 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1459 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1460 Defined Networking (SDN): Layers and Architecture 1461 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1462 2015, . 1464 [TEAS] IETF, "Traffic Engineering Architecture and Signaling", 1465 . 1467 [WirelessHART] 1468 www.hartcomm.org, "Industrial Communication Networks - 1469 Wireless Communication Network and Communication Profiles 1470 - WirelessHART - IEC 62591", 2010. 1472 Authors' Addresses 1474 Norman Finn 1475 Cisco Systems 1476 170 W Tasman Dr. 1477 San Jose, California 95134 1478 USA 1480 Phone: +1 408 526 4495 1481 Email: nfinn@cisco.com 1483 Pascal Thubert 1484 Cisco Systems 1485 Village d'Entreprises Green Side 1486 400, Avenue de Roumanille 1487 Batiment T3 1488 Biot - Sophia Antipolis 06410 1489 FRANCE 1491 Phone: +33 4 97 23 26 34 1492 Email: pthubert@cisco.com 1494 Michael Johas Teener 1495 Broadcom Corp. 1496 3151 Zanker Rd. 1497 San Jose, California 95134 1498 USA 1500 Phone: +1 831 824 4228 1501 Email: MikeJT@broadcom.com