idnits 2.17.1 draft-finn-detnet-architecture-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 635 has weird spacing: '...mediate inter...' == Line 638 has weird spacing: '...mediate inter...' == Line 835 has weird spacing: '...e stack v ...' -- The document date (August 18, 2016) is 2808 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'AVnu' is defined on line 1232, but no explicit reference was found in the text == Unused Reference: 'HART' is defined on line 1241, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-architecture' is defined on line 1260, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-tsch' is defined on line 1265, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-roll-rpl-industrial-applicability' is defined on line 1283, but no explicit reference was found in the text == Unused Reference: 'I-D.svshah-tsvwg-deterministic-forwarding' is defined on line 1289, but no explicit reference was found in the text == Unused Reference: 'IEEE802.11-2012' is defined on line 1294, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1AS-2011' is defined on line 1300, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1TSNTG' is defined on line 1344, but no explicit reference was found in the text == Unused Reference: 'IEEE802154' is defined on line 1358, but no explicit reference was found in the text == Unused Reference: 'IEEE802154e' is defined on line 1364, but no explicit reference was found in the text == Unused Reference: 'ISA95' is defined on line 1376, but no explicit reference was found in the text == Unused Reference: 'ODVA' is defined on line 1380, but no explicit reference was found in the text == Unused Reference: 'Profinet' is defined on line 1387, but no explicit reference was found in the text == Unused Reference: 'RFC5673' is defined on line 1438, but no explicit reference was found in the text == Unused Reference: 'WirelessHART' is defined on line 1466, but no explicit reference was found in the text == Outdated reference: A later version (-04) exists of draft-dt-detnet-dp-alt-03 == Outdated reference: A later version (-30) exists of draft-ietf-6tisch-architecture-10 == Outdated reference: A later version (-09) exists of draft-ietf-detnet-problem-statement-00 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-10 -- Obsolete informational reference (is this intentional?): RFC 5316 (Obsoleted by RFC 9346) Summary: 0 errors (**), 0 flaws (~~), 24 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft P. Thubert 4 Intended status: Standards Track Cisco 5 Expires: February 19, 2017 August 18, 2016 7 Deterministic Networking Architecture 8 draft-finn-detnet-architecture-08 10 Abstract 12 Deterministic Networking (DetNet) provides a capability to carry 13 specified unicast or multicast data flows for real-time applications 14 with extremely low data loss rates and bounded latency. Techniques 15 used include: 1) reserving data plane resources for individual (or 16 aggregated) DetNet flows in some or all of the intermediate nodes 17 (e.g. bridges or routers) along the path of the flow; 2) providing 18 explicit routes for DetNet flows that do not rapidly change with the 19 network topology; and 3) distributing data from DetNet flow packets 20 over time and/or space to ensure delivery of each packet's data' in 21 spite of the loss of a path. The capabilities can be managed by 22 configuration, or by manual or automatic network management. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on February 19, 2017. 41 Copyright Notice 43 Copyright (c) 2016 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 60 2.1. Terms used in this document . . . . . . . . . . . . . . . 4 61 2.2. IEEE 802 TSN to DetNet dictionary . . . . . . . . . . . . 5 62 3. Providing the DetNet Quality of Service . . . . . . . . . . . 6 63 3.1. Congestion protection . . . . . . . . . . . . . . . . . . 8 64 3.2. Explicit routes . . . . . . . . . . . . . . . . . . . . . 8 65 3.3. Jitter Reduction . . . . . . . . . . . . . . . . . . . . 9 66 3.4. Packet Replication and Elimination . . . . . . . . . . . 10 67 3.5. Packet encoding for service protection . . . . . . . . . 11 68 4. DetNet Architecture . . . . . . . . . . . . . . . . . . . . . 12 69 4.1. Traffic Engineering for DetNet . . . . . . . . . . . . . 12 70 4.1.1. The Application Plane . . . . . . . . . . . . . . . . 12 71 4.1.2. The Controller Plane . . . . . . . . . . . . . . . . 13 72 4.1.3. The Network Plane . . . . . . . . . . . . . . . . . . 13 73 4.2. DetNet flows . . . . . . . . . . . . . . . . . . . . . . 14 74 4.2.1. Source guarantees . . . . . . . . . . . . . . . . . . 14 75 4.2.2. Incomplete Networks . . . . . . . . . . . . . . . . . 16 76 4.3. Queuing, Shaping, Scheduling, and Preemption . . . . . . 16 77 4.4. Coexistence with normal traffic . . . . . . . . . . . . . 17 78 4.5. Fault Mitigation . . . . . . . . . . . . . . . . . . . . 17 79 4.6. Representative Protocol Stack Model . . . . . . . . . . . 18 80 4.7. Exporting flow identification . . . . . . . . . . . . . . 20 81 4.8. Advertising resources, capabilities and adjacencies . . . 22 82 4.9. Provisioning model . . . . . . . . . . . . . . . . . . . 22 83 4.9.1. Centralized Path Computation and Installation . . . . 22 84 4.9.2. Distributed Path Setup . . . . . . . . . . . . . . . 22 85 4.10. Scaling to larger networks . . . . . . . . . . . . . . . 23 86 4.11. Connected islands vs. networks . . . . . . . . . . . . . 23 87 5. Compatibility with Layer-2 . . . . . . . . . . . . . . . . . 23 88 6. Open Questions . . . . . . . . . . . . . . . . . . . . . . . 24 89 6.1. Flat vs. hierarchical control . . . . . . . . . . . . . . 24 90 6.2. Peer-to-peer reservation protocol . . . . . . . . . . . . 24 91 6.3. Wireless media interactions . . . . . . . . . . . . . . . 25 92 7. Security Considerations . . . . . . . . . . . . . . . . . . . 25 93 8. Privacy Considerations . . . . . . . . . . . . . . . . . . . 26 94 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 95 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 26 96 11. Access to IEEE 802.1 documents . . . . . . . . . . . . . . . 26 97 12. Informative References . . . . . . . . . . . . . . . . . . . 26 98 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 100 1. Introduction 102 Deterministic Networking (DetNet) is a service that can be offered by 103 a network to DetNet flows. DetNet provides these flows extremely low 104 packet loss rates and assured maximum end-to-end delivery latency. 105 This is accomplished by dedicating network resources such as link 106 bandwidth and buffer space to DetNet flows and/or classes of DetNet 107 flows, and by replicating packets along multiple paths. Unused 108 reserved resources are available to non-DetNet packets. 110 The Deterministic Networking Problem Statement 111 [I-D.ietf-detnet-problem-statement] introduces Deterministic 112 Networking, and Deterministic Networking Use Cases 113 [I-D.ietf-detnet-use-cases] summarizes the need for it. See 114 [I-D.dt-detnet-dp-alt] for a discussion of specific techniques that 115 can be used to identify DetNet Flows and assign them to specific 116 paths through a network. 118 A goal of DetNet is a converged network in all respects. That is, 119 the presence of DetNet flows does not preclude non-DetNet flows, and 120 the benefits offered DetNet flows should not, except in extreme 121 cases, prevent existing QoS mechanisms from operating in a normal 122 fashion, subject to the bandwidth required for the DetNet flows. A 123 single source-destination pair can trade both DetNet and non-DetNet 124 flows. End systems and applications need not instantiate special 125 interfaces for DetNet flows. Networks are not restricted to certain 126 topologies; connectivity is not restricted. Any application that 127 generates a data flow that can be usefully characterized as having a 128 maximum bandwidth should be able to take advantage of DetNet, as long 129 as the necessary resources can be reserved. Reservations can be made 130 by the application itself, via network management, by an applications 131 controller, or by other means. 133 Many applications of interest to Deterministic Networking require the 134 ability to synchronize the clocks in end systems to a sub-microsecond 135 accuracy. Some of the queue control techniques defined in 136 Section 4.3 also require time synchronization among relay and transit 137 nodes. The means used to achieve time synchronization are not 138 addressed in this document. DetNet should accommodate various 139 synchronization techniques and profiles that are defined elsewhere to 140 solve exchange time in different market segments. 142 The present document is an individual contribution, but it is 143 intended by the authors for adoption by the DetNet working group. 145 2. Terminology 147 2.1. Terms used in this document 149 The following special terms are used in this document in order to 150 avoid the assumption that a given element in the architecture does or 151 does not have Internet Protocol stack, functions as a router, bridge, 152 firewall, or otherwise plays a particular role at Layer-2 or higher. 154 destination 155 An end system capable of receiving a DetNet flow. 157 DetNet domain 158 The portion of a network that is DetNet aware. It includes 159 end systems and other DetNet nodes. 161 DetNet flow 162 A DetNet flow is a sequence of packets to which the DetNet 163 service is to be provided. 165 DetNet compound flow and DetNet member flow 166 A DetNet compound flow is a DetNet flow that has been 167 separated into multiple duplicate DetNet member flows, which 168 are eventually merged back into a single DetNet compound 169 flow, at the DetNet transport layer. "Compound" and "member" 170 are strictly relative to each other, not absolutes; a DetNet 171 compound flow comprising multiple DetNet member flows can, in 172 turn, be a member of a higher-order compound. 174 DetNet intermediate node 175 A DetNet relay node or transit node. 177 DetNet edge node 178 An instance of a DetNet relay node that includes either a 179 DetNet service layer proxy function for DetNet service 180 protection (e.g. the addition or removal of packet sequencing 181 information) for one or more end systems, or starts or 182 terminates congestion protection at the DetNet transport 183 layer, analogous to a Label Edge Router (LER). 185 end system 186 Commonly called a "host" or "node" in IETF documents, and an 187 "end station" is IEEE 802 documents. End systems of interest 188 to this document are either sources or destinations of DetNet 189 flows. And end system may or may not be DetNet transport 190 layer aware or DetNet service layer aware. 192 link 193 A connection between two DetNet nodes. It may be composed of 194 a physical link or a sub-network technology that can provide 195 appropriate traffic delivery for DetNet flows. 197 DetNet node 198 A DetNet aware end system, transit node, or relay node. 199 "DetNet" may be omitted in some text. 201 Detnet relay node 202 A DetNet node including a service layer function that 203 interconnects different DetNet transport layer paths to 204 provide service protection. A DetNet relay node can be a 205 bridge, a router, a firewall, or any other system that 206 participates in the DetNet service layer. It typically 207 incorporates DetNet transport layer functions as well, in 208 which case it is collocated with a transit node. 210 reservation 211 A trail of configuration between source to destination(s) 212 through transit nodes and subnets associated with a DetNet 213 flow, to provide congestion protection. 215 DetNet service layer 216 The layer at which service protection is provided, either 217 packet sequencing, replication, and elimination (Section 3.4) 218 or network coding (Section 3.5). 220 source 221 An end system capable of sourcing a DetNet flow. 223 DetNet transit node 224 A node operating at the DetNet transport layer, that utilizes 225 link layer and/or network layer switching across multiple 226 links and/or sub-networks to provide paths for DetNet service 227 layer functions. Optionally provides congestion protection 228 over those paths. An MPLS LSR is an example of a DetNet 229 transit node. 231 DetNet transport layer 232 The layer that optionally provides congestion protection for 233 DetNet flows over paths provided by the underlying network. 235 2.2. IEEE 802 TSN to DetNet dictionary 237 This section also serves as a dictionary for translating from the 238 terms used by the IEEE 802 Time-Sensitive Networking (TSN) Task Group 239 to those of the DetNet WG. 241 Listener 242 The IEEE 802 term for a destination of a DetNet flow. 244 relay system 245 The IEEE 802 term for a DetNet intermediate node. 247 Stream 248 The IEEE 802 term for a DetNet flow. 250 Talker 251 The IEEE 802 term for the source of a DetNet flow. 253 3. Providing the DetNet Quality of Service 255 The DetNet Quality of Service can be expressed in terms of: 257 o Minimum and maximum end-to-end latency from source to destination; 258 timely delivery and jitter avoidance derive from these constraints 260 o Probability of loss of a packet, under various assumptions as to 261 the operational states of the nodes and links. A derived property 262 is whether it is acceptable to deliver a duplicate packet, which 263 is an inherent risk in highly reliable and/or broadcast 264 transmissions 266 It is a distinction of DetNet that it is concerned solely with worst- 267 case values for the end-to-end latency. Average, mean, or typical 268 values are of no interest, because they do not affect the ability of 269 a real-time system to perform its tasks. In general, a trivial 270 priority-based queuing scheme will give better average latency to a 271 data flow than DetNet, but of course, the worst-case latency can be 272 essentially unbounded. 274 Three techniques are used by DetNet to provide these qualities of 275 service: 277 o Congestion protection (Section 3.1). 279 o Explicit routes (Section 3.2). 281 o Service protection. 283 Congestion protection operates by reserving resources along the path 284 of a DetNet Flow, e.g. buffer space or link bandwidth. Congestion 285 protection greatly reduces, or even eliminates entirely, packet loss 286 due to output packet congestion within the network, but it can only 287 be supplied to a DetNet flow that is limited at the source to a 288 maximum packet size and transmission rate. 290 Congestion protection addresses both of the DetNet QoS requirements 291 (latency and packet loss). Given that DetNet nodes have a finite 292 amount of buffer space, congestion protection necessarily results in 293 a maximum end-to-end latency. It also addresses the largest 294 contribution to packet loss, which is buffer congestion. 296 After congestion, the most important contributions to packet loss are 297 typically from random media errors and equipment failures. Service 298 protection is the name for the mechanisms used by DetNet to address 299 these losses. The mechanisms employed are constrained by the 300 requirement to meet the users' latency requirements. Packet 301 replication and elimination (Section 3.4) packet encoding Section 3.5 302 are described in this document to provide service protection; others 303 may be found. Both mechanisms distribute the contents of DetNet 304 flows over multiple paths in time and/or space, so that the loss of 305 some of the paths does need not cause the loss of any packets. The 306 paths are typically (but not necessarily) explicit routes, so that 307 they cannot suffer temporary interruptions caused by the 308 reconvergence of routing or bridging protocols. 310 These three techniques can be applied independently, giving eight 311 possible combinations, including none (no DetNet), although some 312 combinations are of wider utility than others. This separation keeps 313 the protocol stack coherent and maximizes interoperability with 314 existing and developing standards in this (IETF) and other Standards 315 Development Organizations. Some examples of typical expected 316 combinations: 318 o Explicit routes plus service protection are exactly the techniques 319 employed by [HSR-PRP]. Explicit routes are achieved by limiting 320 the physical topology of the network, and the sequentialization, 321 replication, and duplicate elimination are facilitated by packet 322 tags added at the front or the end of Ethernet frames. 324 o Congestion protection alone is is offered by IEEE 802.1 Audio 325 Video bridging [IEEE802.1BA-2011]. As long as the network suffers 326 no failures, zero congestion loss can be achieved through the use 327 of a reservation protocol (MSRP), shapers in every bridge, and a 328 bit of network calculus. 330 o Using all three together gives maximum protection. 332 There are, of course, simpler methods available (and employed, today) 333 to achieve levels of latency and packet loss that are satisfactory 334 for many applications. Prioritization and over-provisioning is one 335 such technique. However, these methods generally work best in the 336 absence of any significant amount of non-critical traffic in the 337 network (if, indeed, such traffic is supported at all), or work only 338 if the critical traffic constitutes only a small portion of the 339 network's theoretical capacity, or work only if all systems are 340 functioning properly, or in the absence of actions by end systems 341 that disrupt the network's operations. 343 There are any number of methods in use, defined, or in progress for 344 accomplishing each of the above techniques. It is expected that this 345 DetNet Architecture will assist various vendors, users, and/or 346 "vertical" Standards Development Organizations (dedicated to a single 347 industry) to make selections among the available means of 348 implementing DetNet networks. 350 3.1. Congestion protection 352 The primary means by which DetNet achieves its QoS assurances is to 353 reduce, or even completely eliminate, congestion at an output port as 354 a cause of packet loss. Given that a DetNet flow cannot be 355 throttled, this can be achieved only by the provision of sufficient 356 buffer storage at each hop through the network to ensure that no 357 packets are dropped due to a lack of buffer storage. 359 Ensuring adequate buffering requires, in turn, that the source, and 360 every intermediate node along the path to the destination (or nearly 361 every node -- see Section 4.2.2) be careful to regulate its output to 362 not exceed the data rate for any DetNet flow, except for brief 363 periods when making up for interfering traffic. Any packet sent 364 ahead of its time potentially adds to the number of buffers required 365 by the next hop, and may thus exceed the resources allocated for a 366 particular DetNet flow. 368 The low-level mechanisms described in Section 4.3 provide the 369 necessary regulation of transmissions by an end system or 370 intermediate node to provide congestion protection. The reservation 371 of the bandwidth and buffers for a DetNet flow requires the 372 provisioning described in Section 4.9. A DetNet node may have other 373 resources requiring allocation and/or scheduling, that might 374 otherwise be over-subscribed and trigger the rejection of a 375 reservation. 377 3.2. Explicit routes 379 In networks controlled by typical peer-to-peer protocols such as IEEE 380 802.1 ISIS bridged networks or IETF OSPF routed networks, a network 381 topology event in one part of the network can impact, at least 382 briefly, the delivery of data in parts of the network remote from the 383 failure or recovery event. Thus, even redundant paths through a 384 network, if controlled by the typical peer-to-peer protocols, do not 385 eliminate the chances of brief losses of contact. 387 Many real-time networks rely on physical rings or chains of two-port 388 devices, with a relatively simple ring control protocol. This 389 supports redundant paths for service protection with a minimum of 390 wiring. As an additional benefit, ring topologies can often utilize 391 different topology management protocols than those used for a mesh 392 network, with a consequent reduction in the response time to topology 393 changes. Of course, this comes at some cost in terms of increased 394 hop count, and thus latency, for the typical path. 396 In order to get the advantages of low hop count and still ensure 397 against even very brief losses of connectivity, DetNet employs 398 explicit routes, where the path taken by a given DetNet flow does not 399 change, at least immediately, and likely not at all, in response to 400 network topology events. Service protection (Section 3.4 or 401 Section 3.5) over explicit routes provides a high likelihood of 402 continuous connectivity. Explicit routes are commonly used in MPLS 403 TE LSPs. 405 3.3. Jitter Reduction 407 A core objective of DetNet is to enable the convergence of Non-IP 408 networks onto a common network infrastructure. This requires the 409 accurate emulation of currently deployed mission-specific networks, 410 which typically rely on point-to-point analog (e.g. 4-20mA 411 modulation) and serial-digital cables (or buses) for highly reliable, 412 synchronized and jitter-free communications. While the latency of 413 analog transmissions is basically the speed of light, legacy serial 414 links are usually slow (in the order of Kbps) compared to, say, GigE, 415 and some latency is usually acceptable. What is not acceptable is 416 the introduction of excessive jitter, which may, for instance, affect 417 the stability of control systems. 419 Applications that are designed to operate on serial links usually do 420 not provide services to recover the jitter, because jitter simply 421 does not exists there. Streams of information are expected to be 422 delivered in-order and the precise time of reception influences the 423 processes. In order to converge such existing applications, there is 424 a desire to emulate all properties of the serial cable, such as clock 425 transportation, perfect flow isolation and fixed latency. While 426 minimal jitter (in the form of specifying minimum, as well as 427 maximum, end-to-end latency) is supported by DetNet, there are 428 practical limitations on packet-based networks in this regard. In 429 general, users are encouraged to use, instead of, "do this when you 430 get the packet," a combination of: 432 o Sub-microsecond time synchronization among all source and 433 destination end systems, and 435 o Time-of-execution fields in the application packets. 437 Jitter reduction is provided by the mechanisms described in 438 Section 4.3 that also provide congestion protection. 440 3.4. Packet Replication and Elimination 442 After congestion loss has been eliminated, the most important causes 443 of packet loss are random media and/or memory faults, and equipment 444 failures. Both causes of packet loss can be greatly reduced by 445 spreading the data in a packet over multiple transmissions. One such 446 method for service protection is described in this section, which 447 sends the same packets over multiple paths. See also Section 3.5. 449 Packet replication and elimination, also known as seamless redundancy 450 [HSR-PRP], or 1+1 hitless protection, is a function of the DetNet 451 service layer. It involves three capabilities: 453 o Providing sequencing information, once, at or near the source, to 454 the packets of a DetNet compound flow. This may be done by adding 455 a sequence number or time stamp as part of DetNet, or may be 456 inherent in the packet, e.g. in a transport protocol, or 457 associated to other physical properties such as the precise time 458 (and radio channel) of reception of the packet. Section 3.2. 460 o Replicating these packets into multiple DetNet member flows and, 461 typically, sending them along at least two different paths to the 462 destination(s), e.g. over the explicit routes of 464 o Eliminating duplicated packets. This may be done at any step 465 along the path to save network resources further down, in 466 particular if multiple Replication points exist. But the most 467 common case is to perform this operation at the very edge of the 468 DetNet network, preferably in or near the receiver. 470 This function is a "hitless" version of, e.g., the 1+1 linear 471 protection in [RFC6372]. That is, instead of switching from one flow 472 to the other when a failure of a flow is detected, DetNet combines 473 both flows, and performs a packet-by-packet selection of which to 474 discard, based on sequence number. 476 In the simplest case, this amounts to replicating each packet in a 477 source that has two interfaces, and conveying them through the 478 network, along separate paths, to the similarly dual-homed 479 destinations, that discard the extras. This ensures that one path 480 (with zero congestion loss) remains, even if some intermediate node 481 fails. The sequence numbers can also be used for loss detection and 482 for re-ordering. 484 Detnet relay nodes in the network can provide replication and 485 elimination facilities at various points in the network, so that 486 multiple failures can be accommodated. 488 This is shown in the following figure, where the two relay nodes each 489 replicate (R) the DetNet flow on input, sending the DetNet member 490 flows to both the other relay node and to the end system, and 491 eliminate duplicates (E) on the output interface to the right-hand 492 end system. Any one link in the network can fail, and the Detnet 493 compound flow can still get through. Furthermore, two links can 494 fail, as long as they are in different segments of the network. 496 Packet replication and elimination 498 > > > > > > > > > relay > > > > > > > > 499 > /------------+ R node E +------------\ > 500 > / v + ^ \ > 501 end R + v | ^ + E end 502 system + v | ^ + system 503 > \ v + ^ / > 504 > \------------+ R relay E +-----------/ > 505 > > > > > > > > > node > > > > > > > > 507 Figure 1 509 Note that packet replication and elimination does not react to and 510 correct failures; it is entirely passive. Thus, intermittent 511 failures, mistakenly created packet filters, or misrouted data is 512 handled just the same as the equipment failures that are detected 513 handled by typical routing and bridging protocols. 515 If packet replication and elimination is used over paths providing 516 congestion protection (Section 3.1), and member flows that take 517 different-length paths through the network are combined, a merge 518 point may require extra buffering to equalize the delays over the 519 different paths. This equalization ensures that the resultant 520 compound flow will not exceed its contracted bandwidth even after one 521 or the other of the paths is restored after a failure. 523 3.5. Packet encoding for service protection 525 There are methods for using multiple paths to provide service 526 protection that involve encoding the information in a packet 527 belonging to a DetNet flow into multiple transmission units, 528 typically combining information from multiple packets into any given 529 transmission unit. Such techniques may be applicable for use as a 530 DetNet service protection technique, assuming that the DetNet users' 531 needs for timeliness of delivery and freedom from interference with 532 misbehaving DetNet flows can be met. 534 No specific mechanisms are defined here, at this time. This section 535 will either be enhanced or removed. Contributions are invited. 537 4. DetNet Architecture 539 4.1. Traffic Engineering for DetNet 541 Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines 542 traffic-engineering architectures for generic applicability across 543 packet and non-packet networks. From TEAS perspective, Traffic 544 Engineering (TE) refers to techniques that enable operators to 545 control how specific traffic flows are treated within their networks. 547 Because if its very nature of establishing explicit optimized paths, 548 Deterministic Networking can be seen as a new, specialized branch of 549 Traffic Engineering, and inherits its architecture with a separation 550 into planes. 552 The Deterministic Networking architecture is thus composed of three 553 planes, a (User) Application Plane, a Controller Plane, and a Network 554 Plane, which echoes that of Figure 1 of Software-Defined Networking 555 (SDN): Layers and Architecture Terminology [RFC7426].: 557 4.1.1. The Application Plane 559 Per [RFC7426], the Application Plane includes both applications and 560 services. In particular, the Application Plane incorporates the User 561 Agent, a specialized application that interacts with the end user / 562 operator and performs requests for Deterministic Networking services 563 via an abstract Flow Management Entity, (FME) which may or may not be 564 collocated with (one of) the end systems. 566 At the Application Plane, a management interface enables the 567 negotiation of flows between end systems. An abstraction of the flow 568 called a Traffic Specification (TSpec) provides the representation. 569 This abstraction is used to place a reservation over the (Northbound) 570 Service Interface and within the Application plane. It is associated 571 with an abstraction of location, such as IP addresses and DNS names, 572 to identify the end systems and eventually specify intermediate 573 nodes. 575 4.1.2. The Controller Plane 577 The Controller Plane corresponds to the aggregation of the Control 578 and Management Planes in [RFC7426], though Common Control and 579 Measurement Plane (CCAMP) [CCAMP] makes an additional distinction 580 between management and measurement. When the logical separation of 581 the Control, Measurement and other Management entities is not 582 relevant, the term Controller Plane is used for simplicity to 583 represent them all, and the term controller refers to any device 584 operating in that plane, whether is it a Path Computation entity or a 585 Network Management entity (NME). The Path Computation Element (PCE) 586 [PCE] is a core element of a controller, in charge of computing 587 Deterministic paths to be applied in the Network Plane. 589 A (Northbound) Service Interface enables applications in the 590 Application Plane to communicate with the entities in the Controller 591 Plane. 593 One or more PCE(s) collaborate to implement the requests from the FME 594 as Per-Flow Per-Hop Behaviors installed in the intermediate nodes for 595 each individual flow. The PCEs place each flow along a deterministic 596 sequence of intermediate nodes so as to respect per-flow constraints 597 such as security and latency, and optimize the overall result for 598 metrics such as an abstract aggregated cost. The deterministic 599 sequence can typically be more complex than a direct sequence and 600 include redundancy path, with one or more packet replication and 601 elimination points. 603 4.1.3. The Network Plane 605 The Network Plane represents the network devices and protocols as a 606 whole, regardless of the Layer at which the network devices operate. 607 It includes Forwarding Plane (data plane), Application, and 608 Operational Plane (control plane) aspects. 610 The network Plane comprises the Network Interface Cards (NIC) in the 611 end systems, which are typically IP hosts, and intermediate nodes, 612 which are typically IP routers and switches. Network-to-Network 613 Interfaces such as used for Traffic Engineering path reservation in 614 [RFC5921], as well as User-to-Network Interfaces (UNI) such as 615 provided by the Local Management Interface (LMI) between network and 616 end systems, are both part of the Network Plane, both in the control 617 plane and the data plane. 619 A Southbound (Network) Interface enables the entities in the 620 Controller Plane to communicate with devices in the Network Plane. 621 This interface leverages and extends TEAS to describe the physical 622 topology and resources in the Network Plane. 624 Flow Management Entity 626 End End 627 System System 629 -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 631 PCE PCE PCE PCE 633 -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 635 intermediate intermed. intermed. intermed. 636 Node Node Node Node 637 NIC NIC 638 intermediate intermed. intermed. intermed. 639 Node Node Node Node 641 Figure 2 643 The intermediate nodes (and eventually the end systems NIC) expose 644 their capabilities and physical resources to the controller (the 645 PCE), and update the PCE with their dynamic perception of the 646 topology, across the Southbound Interface. In return, the PCE(s) set 647 the per-flow paths up, providing a Flow Characterization that is more 648 tightly coupled to the intermediate node Operation than a TSpec. 650 At the Network plane, intermediate nodes may exchange information 651 regarding the state of the paths, between adjacent systems and 652 eventually with the end systems, and forward packets within 653 constraints associated to each flow, or, when unable to do so, 654 perform a last resort operation such as drop or declassify. 656 This specification focuses on the Southbound interface and the 657 operation of the Network Plane. 659 4.2. DetNet flows 661 4.2.1. Source guarantees 663 For the purposes of congestion protection, DetNet flows can be 664 synchronous or asynchronous. In synchronous DetNet flows, at least 665 the intermediate nodes (and possibly the end systems) are closely 666 time synchronized, typically to better than 1 microsecond. By 667 transmitting packets from different DetNet flows or classes of DetNet 668 flows at different times, using repeating schedules synchronized 669 among the intermediate nodes, resources such as buffers and link 670 bandwidth can be shared over the time domain among different DetNet 671 flows. There is a tradeoff among techniques for synchronous DetNet 672 flows between the burden of fine-grained scheduling and the benefit 673 of reducing the required resources, especially buffer space. 675 In contrast, asynchronous DetNet flows are not coordinated with a 676 fine-grained schedule, so relay and end systems must assume worst- 677 case interference among DetNet flows contending for buffer resources. 678 Asynchronous DetNet flows are characterized by: 680 o A maximum packet size; 682 o An observation interval; and 684 o A maximum number of transmissions during that observation 685 interval. 687 These parameters, together with knowledge of the protocol stack used 688 (and thus the size of the various headers added to a packet), limit 689 the number of bit times per observation interval that the DetNet flow 690 can occupy the physical medium. 692 The source promises that these limits will not be exceeded. If the 693 source transmits less data than this limit allows, the unused 694 resources such as link bandwidth can be made available by the system 695 to non-DetNet packets. However, making those resources available to 696 DetNet packets in other DetNet flows would serve no purpose. Those 697 other DetNet flows have their own dedicated resources, on the 698 assumption that all DetNet flows can use all of their resources over 699 a long period of time. 701 Note that there is no provision in DetNet for throttling DetNet flows 702 (reducing the transmission rate via feedback); the assumption is that 703 a DetNet flow, to be useful, must be delivered in its entirety. That 704 is, while any useful application is written to expect a certain 705 number of lost packets, the real-time applications of interest to 706 DetNet demand that the loss of data due to the network is 707 extraordinarily infrequent. 709 Although DetNet strives to minimize the changes required of an 710 application to allow it to shift from a special-purpose digital 711 network to an Internet Protocol network, one fundamental shift in the 712 behavior of network applications is impossible to avoid: the 713 reservation of resources before the application starts. In the first 714 place, a network cannot deliver finite latency and practically zero 715 packet loss to an arbitrarily high offered load. Secondly, achieving 716 practically zero packet loss for unthrottled (though bandwidth 717 limited) DetNet flows means that bridges and routers have to dedicate 718 buffer resources to specific DetNet flows or to classes of DetNet 719 flows. The requirements of each reservation have to be translated 720 into the parameters that control each system's queuing, shaping, and 721 scheduling functions and delivered to the hosts, bridges, and 722 routers. 724 4.2.2. Incomplete Networks 726 The presence in the network of transit nodes or subnets that are not 727 fully capable of offering DetNet services complicates the ability of 728 the intermediate nodes and/or controller to allocate resources, as 729 extra buffering, and thus extra latency, must be allocated at points 730 downstream from the non-DetNet intermediate node for a DetNet flow. 732 4.3. Queuing, Shaping, Scheduling, and Preemption 734 DetNet achieves congestion protection and bounded delivery latency by 735 reserving bandwidth and buffer resources at every hop along the path 736 of the DetNet flow. The reservation itself is not sufficient, 737 however. Implementors and users of a number of proprietary and 738 standard real-time networks have found that standards for specific 739 data plane techniques are required to enable these assurances to be 740 made in a multi-vendor network. The fundamental reason is that 741 latency variation in one system results in the need for extra buffer 742 space in the next-hop system(s), which in turn, increases the worst- 743 case per-hop latency. 745 Standard queuing and transmission selection algorithms allow a 746 central controller to compute the latency contribution of each 747 transit node to the end-to-end latency, to compute the amount of 748 buffer space required in each transit node for each incremental 749 DetNet flow, and most importantly, to translate from a flow 750 specification to a set of values for the managed objects that control 751 each relay or end system. The IEEE 802 has specified (and is 752 specifying) a set of queuing, shaping, and scheduling algorithms that 753 enable each transit node (bridge or router), and/or a central 754 controller, to compute these values. These algorithms include: 756 o A credit-based shaper [IEEE802.1Q-2014] Clause 34. 758 o Time-gated queues governed by a rotating time schedule, 759 synchronized among all transit nodes [IEEE802.1Qbv]. 761 o Synchronized double (or triple) buffers driven by synchronized 762 time ticks. [IEEE802.1Qch]. 764 o Pre-emption of an Ethernet packet in transmission by a packet with 765 a more stringent latency requirement, followed by the resumption 766 of the preempted packet [IEEE802.1Qbu], [IEEE802.3br]. 768 While these techniques are currently embedded in Ethernet and 769 bridging standards, we can note that they are all, except perhaps for 770 packet preemption, equally applicable to other media than Ethernet, 771 and to routers as well as bridges. 773 4.4. Coexistence with normal traffic 775 A DetNet network supports the dedication of a high proportion (e.g. 776 75%) of the network bandwidth to DetNet flows. But, no matter how 777 much is dedicated for DetNet flows, it is a goal of DetNet to coexist 778 with existing Class of Service schemes (e.g., DiffServ). It is also 779 important that non-DetNet traffic not disrupt the DetNet flow, of 780 course (see Section 4.5 and Section 7). For these reasons: 782 o Bandwidth (transmission opportunities) not utilized by a DetNet 783 flow are available to non-DetNet packets (though not to other 784 DetNet flows). 786 o DetNet flows can be shaped or scheduled, in order to ensure that 787 the highest-priority non-DetNet packet also is ensured a worst- 788 case latency (at any given hop). 790 o When transmission opportunities for DetNet flows are scheduled in 791 detail, then the algorithm constructing the schedule should leave 792 sufficient opportunities for non-DetNet packets to satisfy the 793 needs of the users of the network. Detailed scheduling can also 794 permit the time-shared use of buffer resources by different DetNet 795 flows. 797 Ideally, the net effect of the presence of DetNet flows in a network 798 on the non-DetNet packets is primarily a reduction in the available 799 bandwidth. 801 4.5. Fault Mitigation 803 One key to building robust real-time systems is to reduce the 804 infinite variety of possible failures to a number that can be 805 analyzed with reasonable confidence. DetNet aids in the process by 806 providing filters and policers to detect DetNet packets received on 807 the wrong interface, or at the wrong time, or in too great a volume, 808 and to then take actions such as discarding the offending packet, 809 shutting down the offending DetNet flow, or shutting down the 810 offending interface. 812 It is also essential that filters and service remarking be employed 813 at the network edge to prevent non-DetNet packets from being mistaken 814 for DetNet packets, and thus impinging on the resources allocated to 815 DetNet packets. 817 There exist techniques, at present and/or in various stages of 818 standardization, that can perform these fault mitigation tasks that 819 deliver a high probability that misbehaving systems will have zero 820 impact on well-behaved DetNet flows, except of course, for the 821 receiving interface(s) immediately downstream of the misbehaving 822 device. Examples of such techniques include traffic policing 823 functions (e.g. [RFC2475]) and separating flows into per-flow rate- 824 limited queues. 826 4.6. Representative Protocol Stack Model 828 Figure 3 illustrates a conceptual DetNet data plane layering model. 829 One may compare it to that in [IEEE802.1CB], Annex C, a work in 830 progress. 832 DetNet data plane protocol stack 834 | packets going | ^ packets coming ^ 835 v down the stack v | up the stack | 836 +----------------------+ +-----------------------+ 837 | Source | | Destination | 838 +----------------------+ +-----------------------+ 839 | Service layer | | Service layer | 840 | Packet sequencing | | Duplicate elimination | 841 | Flow duplication | | Flow merging | 842 | Packet encoding | | Packet decoding | 843 +----------------------+ +-----------------------+ 844 | Transport layer | | Transport layer | 845 | Congestion prot. | | Congestion prot. | 846 +----------------------+ +-----------------------+ 847 | Lower layers | | Lower layers | 848 +----------------------+ +-----------------------+ 849 v ^ 850 \_________________________/ 852 Figure 3 854 Not all layers are required for any given application, or even for 855 any given network. The layers are, from top to bottom: 857 Application 858 Shown as "source" and "destination" in the diagram. 860 OAM 861 Operations, Administration, and Maintenance leverages in-band 862 and out-of-and signaling that validates whether the service 863 is effectively obtained within QoS constraints. OAM is not 864 shown in Figure 3; it may reside in any number of the layers. 866 OAM can involve specific tagging added in the packets for 867 tracing implementation or network configuration errors; 868 traceability enables to find whether a packet is a replica, 869 which relay node performed the replication, and which segment 870 was intended for the replica. 872 Packet sequencing 873 As part of DetNet service protection, supplies the sequence 874 number for packet replication and elimination (Section 3.4). 875 Peers with Duplicate elimination. This layer is not needed 876 if a higher-layer transport protocol is expected to perform 877 any packet sequencing and duplicate elimination required by 878 the DetNet flow duplication. 880 Duplicate elimination 881 As part of the DetNet service layer, based on the sequenced 882 number supplied by its peer, packet sequencing, Duplicate 883 elimination discards any duplicate packets generated by 884 DetNet flow duplication. It can operate on member flows, 885 compound flows, or both. The duplication may also be 886 inferred from other information such as the precise time of 887 reception in a scheduled network. The duplicate elimination 888 layer may also perform resequencing of packets to restore 889 packet order in a flow that was disrupted by the loss of 890 packets on one or another of the multiple paths taken. 892 Flow duplication 893 As part of DetNet service protection, replicates packets that 894 belong to a DetNet compound flow into two or more DetNet 895 member flows. Note that this function is separate from 896 packet sequencing. Flow duplication can be an explicit 897 duplication and remarking of packets, or can be performed by, 898 for example, techniques similar to ordinary multicast 899 replication. Peers with DetNet flow merging. 901 Network flow merging 902 As part of DetNet service protection, merges DetNet member 903 flows together for packets coming up the stack belonging to a 904 specific DetNet compound flow. Peers with DetNet flow 905 duplication. DetNet flow merging, together with packet 906 sequencing, duplicate elimination, and DetNet flow 907 duplication, performs packet replication and elimination 908 (Section 3.4). 910 Packet encoding 911 As part of DetNet service protection, as an alternative to 912 packet sequencing and flow duplication, packet encoding 913 combines the information in multiple DetNet packets, perhaps 914 from different DetNet compound flows, and transmits that 915 information in packets on different DetNet member Flows. 916 Peers with Packet decoding. 918 Packet decoding 919 As part of DetNet service protection, as an alternative to 920 flow merging and duplicate elimination, packet decoding takes 921 packets from different DetNet member flows, and computes from 922 those packets the original DetNet packets from the compound 923 flows input to packet encoding. Peers with Packet encoding. 925 Congestio protection 926 The DetNet transport layer provides congestion protection. 927 See Section 4.3. The actual queuing and shaping mechaniss 928 are typically provided by underlying subnet layers, but since 929 these are can be closely associated with the means of 930 providing paths for DetNet flows (e.g. MPLS LSPs or {VLAN, 931 multicast destination MAC address} pairs), the path and the 932 congestion protection are conflated in this figure. 934 Note that the packet sequencing and duplication elimination functions 935 at the source and destination ends of a DetNet compound flow may be 936 performed either in the end system or in a DetNet edge node. The 937 reader must not confuse a DetNet edge function with other kinds of 938 edge functions, e.g. an Label Edge Router, although the two functions 939 may be performed together. The DetNet edge function is concerned 940 with sequencing packets belonging to DetNet flows. The LER with 941 encapsulating/decapsulating packets for transport, and is considered 942 part of the network underlying the DetNet transport layer. 944 4.7. Exporting flow identification 946 An interesting feature of DetNet, and one that invites 947 implementations that can be accused of "layering violations", is the 948 need for lower layers to be aware of specific flows at higher layers, 949 in order to provide specific queuing and shaping services for 950 specific flows. For example: 952 o A non-IP, strictly L2 source end system X may be sending multiple 953 flows to the same L2 destination end system Y. Those flows may 954 include DetNet flows with different QoS requirements, and may 955 include non-DetNet flows. 957 o A router may be sending any number of flows to another router. 958 Again, those flows may include DetNet flows with different QoS 959 requirements, and may include non-DetNet flows. 961 o Two routers may be separated by bridges. For these bridges to 962 perform any required per-flow queuing and shaping, they must be 963 able to identify the individual flows. 965 o A Label Edge Router (LERs) may have a Label Switched Path (LSP) 966 set up for handling traffic destined for a particular IP address 967 carrying only non-DetNet flows. If a DetNet flow to that same 968 address is requested, a separate LSP may be needed, in order that 969 all of the Label Switch Routers (LSRs) along the path to the 970 destination give that flow special queuing and shaping. 972 The need for a lower-level DetNet node to be aware of individual 973 higher-layer flows is not unique to DetNet. But, given the endless 974 complexity of layering and relayering over tunnels that is available 975 to network designers, DetNet needs to provide a model for flow 976 identification that is at least somewhat better than packet 977 inspection. That is not to say that packet inspection to layer 4 or 978 5 addresses will not be used, or the capability standardized; but, 979 there are alternatives. 981 A DetNet relay node can connect DetNet flows on different paths using 982 different flow identification methods. For example: 984 o A single unicast DetNet flow passing from router A through a 985 bridged network to router B may be assigned a {VLAN, multicast 986 destination MAC address} pair that is unique within that bridged 987 network. The bridges can then identify the flow without accessing 988 higher-layer headers. Of course, the receiving router must 989 recognize and accept that multicast MAC address. 991 o A DetNet flow passing from LSR A to LSR B may be assigned a 992 different label than that used for other flows to the same IP 993 destination. 995 In any of the above cases, it is possible that an existing DetNet 996 flow can be used as a carrier for multiple DetNet sub-flows. (Not to 997 be confused with DetNet compound vs. member flows.) Of course, this 998 requires that the aggregate DetNet flow be provisioned properly to 999 carry the sub-flows. 1001 Thus, rather than packet inspection, there is the option to export 1002 higher-layer information to the lower layer. The requirement to 1003 support one or the other method for flow identification (or both) is 1004 the essential complexity that DetNet brings to existing control plane 1005 models. 1007 4.8. Advertising resources, capabilities and adjacencies 1009 There are three classes of information that a central controller or 1010 decentralized control plane needs to know that can only be obtained 1011 from the end systems and/or transit nodes in the network. When using 1012 a peer-to-peer control plane, some of this information may be 1013 required by a system's neighbors in the network. 1015 o Details of the system's capabilities that are required in order to 1016 accurately allocate that system's resources, as well as other 1017 systems' resources. This includes, for example, which specific 1018 queuing and shaping algorithms are implemented (Section 4.3), the 1019 number of buffers dedicated for DetNet allocation, and the worst- 1020 case forwarding delay. 1022 o The dynamic state of an end or transit node's DetNet resources. 1024 o The identity of the system's neighbors, and the characteristics of 1025 the link(s) between the systems, including the length (in 1026 nanoseconds) of the link(s). 1028 4.9. Provisioning model 1030 4.9.1. Centralized Path Computation and Installation 1032 A centralized routing model, such as provided with a PCE (RFC 4655 1033 [RFC4655]), enables global and per-flow optimizations. (See 1034 Section 4.1.) The model is attractive but a number of issues are 1035 left to be solved. In particular: 1037 o Whether and how the path computation can be installed by 1) an end 1038 device or 2) a Network Management entity, 1040 o And how the path is set up, either by installing state at each hop 1041 with a direct interaction between the forwarding device and the 1042 PCE, or along a path by injecting a source-routed request at one 1043 end of the path. 1045 4.9.2. Distributed Path Setup 1047 Significant work on distributed path setup can be leveraged from MPLS 1048 Traffic Engineering, in both its GMPLS and non-GMPLS forms. The 1049 protocols within scope are Resource ReSerVation Protocol [RFC3209] 1050 [RFC3473](RSVP-TE), OSPF-TE [RFC4203] [RFC5392] and ISIS-TE [RFC5307] 1051 [RFC5316]. These should be viewed as starting points as there are 1052 feature specific extensions defined that may be applicable to DetNet. 1054 In a Layer-2 only environment, or as part of a layered approach to a 1055 mixed environment, IEEE 802.1 also has work, either completed or in 1056 progress. [IEEE802.1Q-2014] Clause 35 describes SRP, a peer-to-peer 1057 protocol for Layer-2 roughly analogous to RSVP [RFC2205]. 1058 [IEEE802.1Qca] defines how ISIS can provide multiple disjoint paths 1059 or distribution trees. Also in progress is [IEEE802.1Qcc], which 1060 expands the capabilities of SRP. 1062 The integration/interaction of the DetNet control layer with an 1063 underlying IEEE 802.1 sub-network control layer will need to be 1064 defined. 1066 4.10. Scaling to larger networks 1068 Reservations for individual DetNet flows require considerable state 1069 information in each transit node, especially when adequate fault 1070 mitigation (Section 4.5) is required. The DetNet data plane, in 1071 order to support larger numbers of DetNet flows, must support the 1072 aggregation of DetNet flows into tunnels, which themselves can be 1073 viewed by the transit nodes' data planes largely as individual DetNet 1074 flows. Without such aggregation, the per-relay system may limit the 1075 scale of DetNet networks. 1077 4.11. Connected islands vs. networks 1079 Given that users have deployed examples of the IEEE 802.1 TSN TG 1080 standards, which provide capabilities similar to DetNet, it is 1081 obvious to ask whether the IETF DetNet effort can be limited to 1082 providing Layer-2 connections (VPNs) between islands of bridged TSN 1083 networks. While this capability is certainly useful to some 1084 applications, and must not be precluded by DetNet, tunneling alone is 1085 not a sufficient goal for the DetNet WG. As shown in the 1086 Deterministic Networking Use Cases draft [I-D.ietf-detnet-use-cases], 1087 there are already deployments of Layer-2 TSN networks that are 1088 encountering the well-known problems of over-large broadcast domains. 1089 Routed solutions, and combinations routed/bridged solutions, are both 1090 required. 1092 5. Compatibility with Layer-2 1094 Standards providing similar capabilities for bridged networks (only) 1095 have been and are being generated in the IEEE 802 LAN/MAN Standards 1096 Committee. The present architecture describes an abstract model that 1097 can be applicable both at Layer-2 and Layer-3, and over links not 1098 defined by IEEE 802. It is the intention of the authors (and 1099 hopefully, as this draft progresses, of the DetNet Working Group) 1100 that IETF and IEEE 802 will coordinate their work, via the 1101 participation of common individuals, liaisons, and other means, to 1102 maximize the compatibility of their outputs. 1104 DetNet enabled end systems and intermediate nodes can be 1105 interconnected by sub-networks, i.e., Layer-2 technologies. These 1106 sub-networks will provide DetNet compatible service for support of 1107 DetNet traffic. Examples of sub-networks include 802.1TSN and a 1108 point-to-point OTN link. Of course, multi-layer DetNet systems may 1109 be possible too, where one DetNet appears as a sub-network, and 1110 provides service to, a higher layer DetNet system. 1112 6. Open Questions 1114 There are a number of architectural questions that will have to be 1115 resolved before this document can be submitted for publication. 1116 Aside from the obvious fact that this present draft is subject to 1117 change, there are specific questions to which the authors wish to 1118 direct the readers' attention. 1120 6.1. Flat vs. hierarchical control 1122 Boxes that are solely routers or solely bridges are rare in today's 1123 market. In a multi-tenant data center, multiple users' virtual 1124 Layer-2/Layer-3 topologies exist simultaneously, implemented on a 1125 network whose physical topology bears only accidental resemblance to 1126 the virtual topologies. 1128 While the forwarding topology (the bridges and routers) are an 1129 important consideration for a DetNet Flow Management Entity 1130 (Section 4.1.1), so is the purely physical topology. Ultimately, the 1131 model used by the management entities is based on boxes, queues, and 1132 links. The authors hope that the work of the TEAS WG will help to 1133 clarify exactly what model parameters need to be traded between the 1134 intermediate nodes and the controller(s). 1136 6.2. Peer-to-peer reservation protocol 1138 As described in Section 4.9.2, the DetNet WG needs to decide whether 1139 to support a peer-to-peer protocol for a source and a destination to 1140 reserve resources for a DetNet stream. Assuming that enabling the 1141 involvement of the source and/or destination is desirable (see 1142 Deterministic Networking Use Cases [I-D.ietf-detnet-use-cases]), it 1143 remains to decide whether the DetNet WG will make it possible to 1144 deploy at least some DetNet capabilities in a network using only a 1145 peer-to-peer protocol, without a central controller. 1147 (Note that a UNI (see Section 4.1.3) between an end system and a 1148 DetNet edge node, for sources and/or listeners to request DetNet 1149 services, can be either the first hop of a per-to-peer reservation 1150 protocol, or can be deflected by the DetNet edge node to a central 1151 controller for resolution. Similarly, a decision by a central 1152 controller can be effected by the controller instructing the end 1153 system or DetNet edge node to initiate a per-to-peer protocol 1154 activity.) 1156 6.3. Wireless media interactions 1158 Deterministic Networking Use Cases [I-D.ietf-detnet-use-cases] 1159 illustrates cases where wireless media are needed in a DetNet 1160 network. Some wireless media in general use, such as IEEE 802.11 1161 [IEEE802.1Q-2014], have significantly higher packet loss rates than 1162 typical wired media, such as Ethernet [IEEE802.3-2012]. IEEE 802.11 1163 includes support for such features as MAC-layer acknowledgements and 1164 retransmissions. 1166 The techniques described in Section 3 are likely to improve the 1167 ability of a mixed wired/wireless network to offer the DetNet QoS 1168 features. The interaction of these techniques with the features of 1169 specific wireless media, although they may be significant, cannot be 1170 addressed in this document. It remains to be decided to what extent 1171 the DetNet WG will address them, and to what extent other WGs, e.g. 1172 6TiSCH, will do so. 1174 7. Security Considerations 1176 Security in the context of Deterministic Networking has an added 1177 dimension; the time of delivery of a packet can be just as important 1178 as the contents of the packet, itself. A man-in-the-middle attack, 1179 for example, can impose, and then systematically adjust, additional 1180 delays into a link, and thus disrupt or subvert a real-time 1181 application without having to crack any encryption methods employed. 1182 See [RFC7384] for an exploration of this issue in a related context. 1184 Furthermore, in a control system where millions of dollars of 1185 equipment, or even human lives, can be lost if the DetNet QoS is not 1186 delivered, one must consider not only simple equipment failures, 1187 where the box or wire instantly becomes perfectly silent, but bizarre 1188 errors such as can be caused by software failures. Because there is 1189 essential no limit to the kinds of failures that can occur, 1190 protecting against realistic equipment failures is indistinguishable, 1191 in most cases, from protecting against malicious behavior, whether 1192 accidental or intentional. See also Section 4.5. 1194 Security must cover: 1196 o the protection of the signaling protocol 1197 o the authentication and authorization of the controlling systems 1199 o the identification and shaping of the DetNet flows 1201 8. Privacy Considerations 1203 DetNet is provides a Quality of Service (QoS), and as such, does not 1204 directly raise any new privacy considerations. 1206 However, the requirement for every (or almost every) node along the 1207 path of a DetNet flow to identify DetNet flows may present an 1208 additional attack surface for privacy, should the DetNet paradigm be 1209 found useful in broader environments. 1211 9. IANA Considerations 1213 This document does not require an action from IANA. 1215 10. Acknowledgements 1217 The authors wish to thank Jouni Korhonen, Erik Nordmark, George 1218 Swallow, Rudy Klecka, Anca Zamfir, David Black, Thomas Watteyne, 1219 Shitanshu Shah, Craig Gunther, Rodney Cummings, Balazs Varga, 1220 Wilfried Steiner, Marcel Kiessling, Karl Weber, Janos Farkas, Ethan 1221 Grossman, Pat Thaler, Lou Berger, and especially Michael Johas 1222 Teener, for their various contribution with this work. 1224 11. Access to IEEE 802.1 documents 1226 To access password protected IEEE 802.1 drafts, see the IETF IEEE 1227 802.1 information page at https://www.ietf.org/proceedings/52/slides/ 1228 bridge-0/tsld003.htm. 1230 12. Informative References 1232 [AVnu] http://www.avnu.org/, "The AVnu Alliance tests and 1233 certifies devices for interoperability, providing a simple 1234 and reliable networking solution for AV network 1235 implementation based on the Audio Video Bridging (AVB) 1236 standards.". 1238 [CCAMP] IETF, "Common Control and Measurement Plane", 1239 . 1241 [HART] www.hartcomm.org, "Highway Addressable Remote Transducer, 1242 a group of specifications for industrial process and 1243 control devices administered by the HART Foundation". 1245 [HSR-PRP] IEC, "High availability seamless redundancy (HSR) is a 1246 further development of the PRP approach, although HSR 1247 functions primarily as a protocol for creating media 1248 redundancy while PRP, as described in the previous 1249 section, creates network redundancy. PRP and HSR are both 1250 described in the IEC 62439 3 standard.", 1251 . 1254 [I-D.dt-detnet-dp-alt] 1255 Korhonen, J., Farkas, J., Mirsky, G., Thubert, P., 1256 Zhuangyan, Z., and L. Berger, "DetNet Data Plane Protocol 1257 and Solution Alternatives", draft-dt-detnet-dp-alt-03 1258 (work in progress), August 2016. 1260 [I-D.ietf-6tisch-architecture] 1261 Thubert, P., "An Architecture for IPv6 over the TSCH mode 1262 of IEEE 802.15.4", draft-ietf-6tisch-architecture-10 (work 1263 in progress), June 2016. 1265 [I-D.ietf-6tisch-tsch] 1266 Watteyne, T., Palattella, M., and L. Grieco, "Using 1267 IEEE802.15.4e TSCH in an IoT context: Overview, Problem 1268 Statement and Goals", draft-ietf-6tisch-tsch-06 (work in 1269 progress), March 2015. 1271 [I-D.ietf-detnet-problem-statement] 1272 Finn, N. and P. Thubert, "Deterministic Networking Problem 1273 Statement", draft-ietf-detnet-problem-statement-00 (work 1274 in progress), April 2016. 1276 [I-D.ietf-detnet-use-cases] 1277 Grossman, E., Gunther, C., Thubert, P., Wetterwald, P., 1278 Raymond, J., Korhonen, J., Kaneko, Y., Das, S., Zha, Y., 1279 Varga, B., Farkas, J., Goetz, F., and J. Schmitt, 1280 "Deterministic Networking Use Cases", draft-ietf-detnet- 1281 use-cases-10 (work in progress), July 2016. 1283 [I-D.ietf-roll-rpl-industrial-applicability] 1284 Phinney, T., Thubert, P., and R. Assimiti, "RPL 1285 applicability in industrial networks", draft-ietf-roll- 1286 rpl-industrial-applicability-02 (work in progress), 1287 October 2013. 1289 [I-D.svshah-tsvwg-deterministic-forwarding] 1290 Shah, S. and P. Thubert, "Deterministic Forwarding PHB", 1291 draft-svshah-tsvwg-deterministic-forwarding-04 (work in 1292 progress), August 2015. 1294 [IEEE802.11-2012] 1295 IEEE, "Wireless LAN Medium Access Control (MAC) and 1296 Physical Layer (PHY) Specifications", 2012, 1297 . 1300 [IEEE802.1AS-2011] 1301 IEEE, "Timing and Synchronizations (IEEE 802.1AS-2011)", 1302 2011, . 1305 [IEEE802.1BA-2011] 1306 IEEE, "AVB Systems (IEEE 802.1BA-2011)", 2011, 1307 . 1310 [IEEE802.1CB] 1311 IEEE, "Frame Replication and Elimination for Reliability 1312 (IEEE Draft P802.1CB)", 2016, 1313 . 1315 [IEEE802.1Q-2014] 1316 IEEE, "MAC Bridges and VLANs (IEEE 802.1Q-2014", 2014, 1317 . 1320 [IEEE802.1Qbu] 1321 IEEE, "Frame Preemption", 2016, 1322 . 1324 [IEEE802.1Qbv] 1325 IEEE, "Enhancements for Scheduled Traffic", 2016, 1326 . 1328 [IEEE802.1Qca] 1329 IEEE 802.1, "IEEE 802.1Qca Bridges and Bridged Networks - 1330 Amendment 24: Path Control and Reservation", IEEE 1331 P802.1Qca/D2.1 P802.1Qca, June 2015, 1332 . 1335 [IEEE802.1Qcc] 1336 IEEE, "Stream Reservation Protocol (SRP) Enhancements and 1337 Performance Improvements", 2016, 1338 . 1340 [IEEE802.1Qch] 1341 IEEE, "Cyclic Queuing and Forwarding", 2016, 1342 . 1344 [IEEE802.1TSNTG] 1345 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1346 Networks Task Group", 2013, 1347 . 1349 [IEEE802.3-2012] 1350 IEEE, "IEEE Stabdard for Ethernet", 2012, 1351 . 1354 [IEEE802.3br] 1355 IEEE, "Interspersed Express Traffic", 2016, 1356 . 1358 [IEEE802154] 1359 IEEE standard for Information Technology, "IEEE std. 1360 802.15.4, Part. 15.4: Wireless Medium Access Control (MAC) 1361 and Physical Layer (PHY) Specifications for Low-Rate 1362 Wireless Personal Area Networks", June 2011. 1364 [IEEE802154e] 1365 IEEE standard for Information Technology, "IEEE std. 1366 802.15.4e, Part. 15.4: Low-Rate Wireless Personal Area 1367 Networks (LR-WPANs) Amendment 1: MAC sublayer", April 1368 2012. 1370 [ISA100.11a] 1371 ISA/IEC, "ISA100.11a, Wireless Systems for Automation, 1372 also IEC 62734", 2011, < http://www.isa100wci.org/en- 1373 US/Documents/PDF/3405-ISA100-WirelessSystems-Future-broch- 1374 WEB-ETSI.aspx>. 1376 [ISA95] ANSI/ISA, "Enterprise-Control System Integration Part 1: 1377 Models and Terminology", 2000, . 1380 [ODVA] http://www.odva.org/, "The organization that supports 1381 network technologies built on the Common Industrial 1382 Protocol (CIP) including EtherNet/IP.". 1384 [PCE] IETF, "Path Computation Element", 1385 . 1387 [Profinet] 1388 http://us.profinet.com/technology/profinet/, "PROFINET is 1389 a standard for industrial networking in automation.", 1390 . 1392 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1393 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1394 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1395 September 1997, . 1397 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1398 and W. Weiss, "An Architecture for Differentiated 1399 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1400 . 1402 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1403 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1404 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1405 . 1407 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 1408 Switching (GMPLS) Signaling Resource ReserVation Protocol- 1409 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 1410 DOI 10.17487/RFC3473, January 2003, 1411 . 1413 [RFC4203] Kompella, K., Ed. and Y. Rekhter, Ed., "OSPF Extensions in 1414 Support of Generalized Multi-Protocol Label Switching 1415 (GMPLS)", RFC 4203, DOI 10.17487/RFC4203, October 2005, 1416 . 1418 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 1419 Element (PCE)-Based Architecture", RFC 4655, 1420 DOI 10.17487/RFC4655, August 2006, 1421 . 1423 [RFC5307] Kompella, K., Ed. and Y. Rekhter, Ed., "IS-IS Extensions 1424 in Support of Generalized Multi-Protocol Label Switching 1425 (GMPLS)", RFC 5307, DOI 10.17487/RFC5307, October 2008, 1426 . 1428 [RFC5316] Chen, M., Zhang, R., and X. Duan, "ISIS Extensions in 1429 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1430 Traffic Engineering", RFC 5316, DOI 10.17487/RFC5316, 1431 December 2008, . 1433 [RFC5392] Chen, M., Zhang, R., and X. Duan, "OSPF Extensions in 1434 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1435 Traffic Engineering", RFC 5392, DOI 10.17487/RFC5392, 1436 January 2009, . 1438 [RFC5673] Pister, K., Ed., Thubert, P., Ed., Dwars, S., and T. 1439 Phinney, "Industrial Routing Requirements in Low-Power and 1440 Lossy Networks", RFC 5673, DOI 10.17487/RFC5673, October 1441 2009, . 1443 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 1444 L., and L. Berger, "A Framework for MPLS in Transport 1445 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 1446 . 1448 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 1449 Profile (MPLS-TP) Survivability Framework", RFC 6372, 1450 DOI 10.17487/RFC6372, September 2011, 1451 . 1453 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1454 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1455 October 2014, . 1457 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1458 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1459 Defined Networking (SDN): Layers and Architecture 1460 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1461 2015, . 1463 [TEAS] IETF, "Traffic Engineering Architecture and Signaling", 1464 . 1466 [WirelessHART] 1467 www.hartcomm.org, "Industrial Communication Networks - 1468 Wireless Communication Network and Communication Profiles 1469 - WirelessHART - IEC 62591", 2010. 1471 Authors' Addresses 1473 Norman Finn 1474 Cisco Systems 1475 170 W Tasman Dr. 1476 San Jose, California 95134 1477 USA 1479 Phone: +1 408 526 4495 1480 Email: nfinn@cisco.com 1481 Pascal Thubert 1482 Cisco Systems 1483 Village d'Entreprises Green Side 1484 400, Avenue de Roumanille 1485 Batiment T3 1486 Biot - Sophia Antipolis 06410 1487 FRANCE 1489 Phone: +33 4 97 23 26 34 1490 Email: pthubert@cisco.com