idnits 2.17.1 draft-ietf-detnet-architecture-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 636 has weird spacing: '...mediate inter...' == Line 639 has weird spacing: '...mediate inter...' == Line 836 has weird spacing: '...e stack v ...' -- The document date (September 26, 2016) is 2761 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'AVnu' is defined on line 1233, but no explicit reference was found in the text == Unused Reference: 'HART' is defined on line 1242, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-architecture' is defined on line 1261, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-tsch' is defined on line 1266, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-roll-rpl-industrial-applicability' is defined on line 1284, but no explicit reference was found in the text == Unused Reference: 'I-D.svshah-tsvwg-deterministic-forwarding' is defined on line 1290, but no explicit reference was found in the text == Unused Reference: 'IEEE802.11-2012' is defined on line 1295, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1AS-2011' is defined on line 1301, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1TSNTG' is defined on line 1345, but no explicit reference was found in the text == Unused Reference: 'IEEE802154' is defined on line 1359, but no explicit reference was found in the text == Unused Reference: 'IEEE802154e' is defined on line 1365, but no explicit reference was found in the text == Unused Reference: 'ISA95' is defined on line 1377, but no explicit reference was found in the text == Unused Reference: 'ODVA' is defined on line 1381, but no explicit reference was found in the text == Unused Reference: 'Profinet' is defined on line 1388, but no explicit reference was found in the text == Unused Reference: 'RFC5673' is defined on line 1439, but no explicit reference was found in the text == Unused Reference: 'WirelessHART' is defined on line 1467, but no explicit reference was found in the text == Outdated reference: A later version (-30) exists of draft-ietf-6tisch-architecture-10 == Outdated reference: A later version (-09) exists of draft-ietf-detnet-problem-statement-00 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-10 -- Obsolete informational reference (is this intentional?): RFC 5316 (Obsoleted by RFC 9346) Summary: 0 errors (**), 0 flaws (~~), 23 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft Self-employed 4 Intended status: Standards Track P. Thubert 5 Expires: March 30, 2017 Cisco 6 September 26, 2016 8 Deterministic Networking Architecture 9 draft-ietf-detnet-architecture-00 11 Abstract 13 Deterministic Networking (DetNet) provides a capability to carry 14 specified unicast or multicast data flows for real-time applications 15 with extremely low data loss rates and bounded latency. Techniques 16 used include: 1) reserving data plane resources for individual (or 17 aggregated) DetNet flows in some or all of the intermediate nodes 18 (e.g. bridges or routers) along the path of the flow; 2) providing 19 explicit routes for DetNet flows that do not rapidly change with the 20 network topology; and 3) distributing data from DetNet flow packets 21 over time and/or space to ensure delivery of each packet's data' in 22 spite of the loss of a path. The capabilities can be managed by 23 configuration, or by manual or automatic network management. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at http://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on March 30, 2017. 42 Copyright Notice 44 Copyright (c) 2016 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 2.1. Terms used in this document . . . . . . . . . . . . . . . 4 62 2.2. IEEE 802 TSN to DetNet dictionary . . . . . . . . . . . . 5 63 3. Providing the DetNet Quality of Service . . . . . . . . . . . 6 64 3.1. Congestion protection . . . . . . . . . . . . . . . . . . 8 65 3.2. Explicit routes . . . . . . . . . . . . . . . . . . . . . 8 66 3.3. Jitter Reduction . . . . . . . . . . . . . . . . . . . . 9 67 3.4. Packet Replication and Elimination . . . . . . . . . . . 10 68 3.5. Packet encoding for service protection . . . . . . . . . 11 69 4. DetNet Architecture . . . . . . . . . . . . . . . . . . . . . 12 70 4.1. Traffic Engineering for DetNet . . . . . . . . . . . . . 12 71 4.1.1. The Application Plane . . . . . . . . . . . . . . . . 12 72 4.1.2. The Controller Plane . . . . . . . . . . . . . . . . 13 73 4.1.3. The Network Plane . . . . . . . . . . . . . . . . . . 13 74 4.2. DetNet flows . . . . . . . . . . . . . . . . . . . . . . 14 75 4.2.1. Source guarantees . . . . . . . . . . . . . . . . . . 14 76 4.2.2. Incomplete Networks . . . . . . . . . . . . . . . . . 16 77 4.3. Queuing, Shaping, Scheduling, and Preemption . . . . . . 16 78 4.4. Coexistence with normal traffic . . . . . . . . . . . . . 17 79 4.5. Fault Mitigation . . . . . . . . . . . . . . . . . . . . 17 80 4.6. Representative Protocol Stack Model . . . . . . . . . . . 18 81 4.7. Exporting flow identification . . . . . . . . . . . . . . 20 82 4.8. Advertising resources, capabilities and adjacencies . . . 22 83 4.9. Provisioning model . . . . . . . . . . . . . . . . . . . 22 84 4.9.1. Centralized Path Computation and Installation . . . . 22 85 4.9.2. Distributed Path Setup . . . . . . . . . . . . . . . 22 86 4.10. Scaling to larger networks . . . . . . . . . . . . . . . 23 87 4.11. Connected islands vs. networks . . . . . . . . . . . . . 23 88 5. Compatibility with Layer-2 . . . . . . . . . . . . . . . . . 23 89 6. Open Questions . . . . . . . . . . . . . . . . . . . . . . . 24 90 6.1. Flat vs. hierarchical control . . . . . . . . . . . . . . 24 91 6.2. Peer-to-peer reservation protocol . . . . . . . . . . . . 24 92 6.3. Wireless media interactions . . . . . . . . . . . . . . . 25 93 7. Security Considerations . . . . . . . . . . . . . . . . . . . 25 94 8. Privacy Considerations . . . . . . . . . . . . . . . . . . . 26 95 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 96 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 26 97 11. Access to IEEE 802.1 documents . . . . . . . . . . . . . . . 26 98 12. Informative References . . . . . . . . . . . . . . . . . . . 26 99 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 101 1. Introduction 103 Deterministic Networking (DetNet) is a service that can be offered by 104 a network to DetNet flows. DetNet provides these flows extremely low 105 packet loss rates and assured maximum end-to-end delivery latency. 106 This is accomplished by dedicating network resources such as link 107 bandwidth and buffer space to DetNet flows and/or classes of DetNet 108 flows, and by replicating packets along multiple paths. Unused 109 reserved resources are available to non-DetNet packets. 111 The Deterministic Networking Problem Statement 112 [I-D.ietf-detnet-problem-statement] introduces Deterministic 113 Networking, and Deterministic Networking Use Cases 114 [I-D.ietf-detnet-use-cases] summarizes the need for it. See 115 [I-D.dt-detnet-dp-alt] for a discussion of specific techniques that 116 can be used to identify DetNet Flows and assign them to specific 117 paths through a network. 119 A goal of DetNet is a converged network in all respects. That is, 120 the presence of DetNet flows does not preclude non-DetNet flows, and 121 the benefits offered DetNet flows should not, except in extreme 122 cases, prevent existing QoS mechanisms from operating in a normal 123 fashion, subject to the bandwidth required for the DetNet flows. A 124 single source-destination pair can trade both DetNet and non-DetNet 125 flows. End systems and applications need not instantiate special 126 interfaces for DetNet flows. Networks are not restricted to certain 127 topologies; connectivity is not restricted. Any application that 128 generates a data flow that can be usefully characterized as having a 129 maximum bandwidth should be able to take advantage of DetNet, as long 130 as the necessary resources can be reserved. Reservations can be made 131 by the application itself, via network management, by an applications 132 controller, or by other means. 134 Many applications of interest to Deterministic Networking require the 135 ability to synchronize the clocks in end systems to a sub-microsecond 136 accuracy. Some of the queue control techniques defined in 137 Section 4.3 also require time synchronization among relay and transit 138 nodes. The means used to achieve time synchronization are not 139 addressed in this document. DetNet should accommodate various 140 synchronization techniques and profiles that are defined elsewhere to 141 solve exchange time in different market segments. 143 The present document is an individual contribution, but it is 144 intended by the authors for adoption by the DetNet working group. 146 2. Terminology 148 2.1. Terms used in this document 150 The following special terms are used in this document in order to 151 avoid the assumption that a given element in the architecture does or 152 does not have Internet Protocol stack, functions as a router, bridge, 153 firewall, or otherwise plays a particular role at Layer-2 or higher. 155 destination 156 An end system capable of receiving a DetNet flow. 158 DetNet domain 159 The portion of a network that is DetNet aware. It includes 160 end systems and other DetNet nodes. 162 DetNet flow 163 A DetNet flow is a sequence of packets to which the DetNet 164 service is to be provided. 166 DetNet compound flow and DetNet member flow 167 A DetNet compound flow is a DetNet flow that has been 168 separated into multiple duplicate DetNet member flows, which 169 are eventually merged back into a single DetNet compound 170 flow, at the DetNet transport layer. "Compound" and "member" 171 are strictly relative to each other, not absolutes; a DetNet 172 compound flow comprising multiple DetNet member flows can, in 173 turn, be a member of a higher-order compound. 175 DetNet intermediate node 176 A DetNet relay node or transit node. 178 DetNet edge node 179 An instance of a DetNet relay node that includes either a 180 DetNet service layer proxy function for DetNet service 181 protection (e.g. the addition or removal of packet sequencing 182 information) for one or more end systems, or starts or 183 terminates congestion protection at the DetNet transport 184 layer, analogous to a Label Edge Router (LER). 186 end system 187 Commonly called a "host" or "node" in IETF documents, and an 188 "end station" is IEEE 802 documents. End systems of interest 189 to this document are either sources or destinations of DetNet 190 flows. And end system may or may not be DetNet transport 191 layer aware or DetNet service layer aware. 193 link 194 A connection between two DetNet nodes. It may be composed of 195 a physical link or a sub-network technology that can provide 196 appropriate traffic delivery for DetNet flows. 198 DetNet node 199 A DetNet aware end system, transit node, or relay node. 200 "DetNet" may be omitted in some text. 202 Detnet relay node 203 A DetNet node including a service layer function that 204 interconnects different DetNet transport layer paths to 205 provide service protection. A DetNet relay node can be a 206 bridge, a router, a firewall, or any other system that 207 participates in the DetNet service layer. It typically 208 incorporates DetNet transport layer functions as well, in 209 which case it is collocated with a transit node. 211 reservation 212 A trail of configuration between source to destination(s) 213 through transit nodes and subnets associated with a DetNet 214 flow, to provide congestion protection. 216 DetNet service layer 217 The layer at which service protection is provided, either 218 packet sequencing, replication, and elimination (Section 3.4) 219 or network coding (Section 3.5). 221 source 222 An end system capable of sourcing a DetNet flow. 224 DetNet transit node 225 A node operating at the DetNet transport layer, that utilizes 226 link layer and/or network layer switching across multiple 227 links and/or sub-networks to provide paths for DetNet service 228 layer functions. Optionally provides congestion protection 229 over those paths. An MPLS LSR is an example of a DetNet 230 transit node. 232 DetNet transport layer 233 The layer that optionally provides congestion protection for 234 DetNet flows over paths provided by the underlying network. 236 2.2. IEEE 802 TSN to DetNet dictionary 238 This section also serves as a dictionary for translating from the 239 terms used by the IEEE 802 Time-Sensitive Networking (TSN) Task Group 240 to those of the DetNet WG. 242 Listener 243 The IEEE 802 term for a destination of a DetNet flow. 245 relay system 246 The IEEE 802 term for a DetNet intermediate node. 248 Stream 249 The IEEE 802 term for a DetNet flow. 251 Talker 252 The IEEE 802 term for the source of a DetNet flow. 254 3. Providing the DetNet Quality of Service 256 The DetNet Quality of Service can be expressed in terms of: 258 o Minimum and maximum end-to-end latency from source to destination; 259 timely delivery and jitter avoidance derive from these constraints 261 o Probability of loss of a packet, under various assumptions as to 262 the operational states of the nodes and links. A derived property 263 is whether it is acceptable to deliver a duplicate packet, which 264 is an inherent risk in highly reliable and/or broadcast 265 transmissions 267 It is a distinction of DetNet that it is concerned solely with worst- 268 case values for the end-to-end latency. Average, mean, or typical 269 values are of no interest, because they do not affect the ability of 270 a real-time system to perform its tasks. In general, a trivial 271 priority-based queuing scheme will give better average latency to a 272 data flow than DetNet, but of course, the worst-case latency can be 273 essentially unbounded. 275 Three techniques are used by DetNet to provide these qualities of 276 service: 278 o Congestion protection (Section 3.1). 280 o Explicit routes (Section 3.2). 282 o Service protection. 284 Congestion protection operates by reserving resources along the path 285 of a DetNet Flow, e.g. buffer space or link bandwidth. Congestion 286 protection greatly reduces, or even eliminates entirely, packet loss 287 due to output packet congestion within the network, but it can only 288 be supplied to a DetNet flow that is limited at the source to a 289 maximum packet size and transmission rate. 291 Congestion protection addresses both of the DetNet QoS requirements 292 (latency and packet loss). Given that DetNet nodes have a finite 293 amount of buffer space, congestion protection necessarily results in 294 a maximum end-to-end latency. It also addresses the largest 295 contribution to packet loss, which is buffer congestion. 297 After congestion, the most important contributions to packet loss are 298 typically from random media errors and equipment failures. Service 299 protection is the name for the mechanisms used by DetNet to address 300 these losses. The mechanisms employed are constrained by the 301 requirement to meet the users' latency requirements. Packet 302 replication and elimination (Section 3.4) packet encoding Section 3.5 303 are described in this document to provide service protection; others 304 may be found. Both mechanisms distribute the contents of DetNet 305 flows over multiple paths in time and/or space, so that the loss of 306 some of the paths does need not cause the loss of any packets. The 307 paths are typically (but not necessarily) explicit routes, so that 308 they cannot suffer temporary interruptions caused by the 309 reconvergence of routing or bridging protocols. 311 These three techniques can be applied independently, giving eight 312 possible combinations, including none (no DetNet), although some 313 combinations are of wider utility than others. This separation keeps 314 the protocol stack coherent and maximizes interoperability with 315 existing and developing standards in this (IETF) and other Standards 316 Development Organizations. Some examples of typical expected 317 combinations: 319 o Explicit routes plus service protection are exactly the techniques 320 employed by [HSR-PRP]. Explicit routes are achieved by limiting 321 the physical topology of the network, and the sequentialization, 322 replication, and duplicate elimination are facilitated by packet 323 tags added at the front or the end of Ethernet frames. 325 o Congestion protection alone is is offered by IEEE 802.1 Audio 326 Video bridging [IEEE802.1BA-2011]. As long as the network suffers 327 no failures, zero congestion loss can be achieved through the use 328 of a reservation protocol (MSRP), shapers in every bridge, and a 329 bit of network calculus. 331 o Using all three together gives maximum protection. 333 There are, of course, simpler methods available (and employed, today) 334 to achieve levels of latency and packet loss that are satisfactory 335 for many applications. Prioritization and over-provisioning is one 336 such technique. However, these methods generally work best in the 337 absence of any significant amount of non-critical traffic in the 338 network (if, indeed, such traffic is supported at all), or work only 339 if the critical traffic constitutes only a small portion of the 340 network's theoretical capacity, or work only if all systems are 341 functioning properly, or in the absence of actions by end systems 342 that disrupt the network's operations. 344 There are any number of methods in use, defined, or in progress for 345 accomplishing each of the above techniques. It is expected that this 346 DetNet Architecture will assist various vendors, users, and/or 347 "vertical" Standards Development Organizations (dedicated to a single 348 industry) to make selections among the available means of 349 implementing DetNet networks. 351 3.1. Congestion protection 353 The primary means by which DetNet achieves its QoS assurances is to 354 reduce, or even completely eliminate, congestion at an output port as 355 a cause of packet loss. Given that a DetNet flow cannot be 356 throttled, this can be achieved only by the provision of sufficient 357 buffer storage at each hop through the network to ensure that no 358 packets are dropped due to a lack of buffer storage. 360 Ensuring adequate buffering requires, in turn, that the source, and 361 every intermediate node along the path to the destination (or nearly 362 every node -- see Section 4.2.2) be careful to regulate its output to 363 not exceed the data rate for any DetNet flow, except for brief 364 periods when making up for interfering traffic. Any packet sent 365 ahead of its time potentially adds to the number of buffers required 366 by the next hop, and may thus exceed the resources allocated for a 367 particular DetNet flow. 369 The low-level mechanisms described in Section 4.3 provide the 370 necessary regulation of transmissions by an end system or 371 intermediate node to provide congestion protection. The reservation 372 of the bandwidth and buffers for a DetNet flow requires the 373 provisioning described in Section 4.9. A DetNet node may have other 374 resources requiring allocation and/or scheduling, that might 375 otherwise be over-subscribed and trigger the rejection of a 376 reservation. 378 3.2. Explicit routes 380 In networks controlled by typical peer-to-peer protocols such as IEEE 381 802.1 ISIS bridged networks or IETF OSPF routed networks, a network 382 topology event in one part of the network can impact, at least 383 briefly, the delivery of data in parts of the network remote from the 384 failure or recovery event. Thus, even redundant paths through a 385 network, if controlled by the typical peer-to-peer protocols, do not 386 eliminate the chances of brief losses of contact. 388 Many real-time networks rely on physical rings or chains of two-port 389 devices, with a relatively simple ring control protocol. This 390 supports redundant paths for service protection with a minimum of 391 wiring. As an additional benefit, ring topologies can often utilize 392 different topology management protocols than those used for a mesh 393 network, with a consequent reduction in the response time to topology 394 changes. Of course, this comes at some cost in terms of increased 395 hop count, and thus latency, for the typical path. 397 In order to get the advantages of low hop count and still ensure 398 against even very brief losses of connectivity, DetNet employs 399 explicit routes, where the path taken by a given DetNet flow does not 400 change, at least immediately, and likely not at all, in response to 401 network topology events. Service protection (Section 3.4 or 402 Section 3.5) over explicit routes provides a high likelihood of 403 continuous connectivity. Explicit routes are commonly used in MPLS 404 TE LSPs. 406 3.3. Jitter Reduction 408 A core objective of DetNet is to enable the convergence of Non-IP 409 networks onto a common network infrastructure. This requires the 410 accurate emulation of currently deployed mission-specific networks, 411 which typically rely on point-to-point analog (e.g. 4-20mA 412 modulation) and serial-digital cables (or buses) for highly reliable, 413 synchronized and jitter-free communications. While the latency of 414 analog transmissions is basically the speed of light, legacy serial 415 links are usually slow (in the order of Kbps) compared to, say, GigE, 416 and some latency is usually acceptable. What is not acceptable is 417 the introduction of excessive jitter, which may, for instance, affect 418 the stability of control systems. 420 Applications that are designed to operate on serial links usually do 421 not provide services to recover the jitter, because jitter simply 422 does not exists there. Streams of information are expected to be 423 delivered in-order and the precise time of reception influences the 424 processes. In order to converge such existing applications, there is 425 a desire to emulate all properties of the serial cable, such as clock 426 transportation, perfect flow isolation and fixed latency. While 427 minimal jitter (in the form of specifying minimum, as well as 428 maximum, end-to-end latency) is supported by DetNet, there are 429 practical limitations on packet-based networks in this regard. In 430 general, users are encouraged to use, instead of, "do this when you 431 get the packet," a combination of: 433 o Sub-microsecond time synchronization among all source and 434 destination end systems, and 436 o Time-of-execution fields in the application packets. 438 Jitter reduction is provided by the mechanisms described in 439 Section 4.3 that also provide congestion protection. 441 3.4. Packet Replication and Elimination 443 After congestion loss has been eliminated, the most important causes 444 of packet loss are random media and/or memory faults, and equipment 445 failures. Both causes of packet loss can be greatly reduced by 446 spreading the data in a packet over multiple transmissions. One such 447 method for service protection is described in this section, which 448 sends the same packets over multiple paths. See also Section 3.5. 450 Packet replication and elimination, also known as seamless redundancy 451 [HSR-PRP], or 1+1 hitless protection, is a function of the DetNet 452 service layer. It involves three capabilities: 454 o Providing sequencing information, once, at or near the source, to 455 the packets of a DetNet compound flow. This may be done by adding 456 a sequence number or time stamp as part of DetNet, or may be 457 inherent in the packet, e.g. in a transport protocol, or 458 associated to other physical properties such as the precise time 459 (and radio channel) of reception of the packet. Section 3.2. 461 o Replicating these packets into multiple DetNet member flows and, 462 typically, sending them along at least two different paths to the 463 destination(s), e.g. over the explicit routes of 465 o Eliminating duplicated packets. This may be done at any step 466 along the path to save network resources further down, in 467 particular if multiple Replication points exist. But the most 468 common case is to perform this operation at the very edge of the 469 DetNet network, preferably in or near the receiver. 471 This function is a "hitless" version of, e.g., the 1+1 linear 472 protection in [RFC6372]. That is, instead of switching from one flow 473 to the other when a failure of a flow is detected, DetNet combines 474 both flows, and performs a packet-by-packet selection of which to 475 discard, based on sequence number. 477 In the simplest case, this amounts to replicating each packet in a 478 source that has two interfaces, and conveying them through the 479 network, along separate paths, to the similarly dual-homed 480 destinations, that discard the extras. This ensures that one path 481 (with zero congestion loss) remains, even if some intermediate node 482 fails. The sequence numbers can also be used for loss detection and 483 for re-ordering. 485 Detnet relay nodes in the network can provide replication and 486 elimination facilities at various points in the network, so that 487 multiple failures can be accommodated. 489 This is shown in the following figure, where the two relay nodes each 490 replicate (R) the DetNet flow on input, sending the DetNet member 491 flows to both the other relay node and to the end system, and 492 eliminate duplicates (E) on the output interface to the right-hand 493 end system. Any one link in the network can fail, and the Detnet 494 compound flow can still get through. Furthermore, two links can 495 fail, as long as they are in different segments of the network. 497 Packet replication and elimination 499 > > > > > > > > > relay > > > > > > > > 500 > /------------+ R node E +------------\ > 501 > / v + ^ \ > 502 end R + v | ^ + E end 503 system + v | ^ + system 504 > \ v + ^ / > 505 > \------------+ R relay E +-----------/ > 506 > > > > > > > > > node > > > > > > > > 508 Figure 1 510 Note that packet replication and elimination does not react to and 511 correct failures; it is entirely passive. Thus, intermittent 512 failures, mistakenly created packet filters, or misrouted data is 513 handled just the same as the equipment failures that are detected 514 handled by typical routing and bridging protocols. 516 If packet replication and elimination is used over paths providing 517 congestion protection (Section 3.1), and member flows that take 518 different-length paths through the network are combined, a merge 519 point may require extra buffering to equalize the delays over the 520 different paths. This equalization ensures that the resultant 521 compound flow will not exceed its contracted bandwidth even after one 522 or the other of the paths is restored after a failure. 524 3.5. Packet encoding for service protection 526 There are methods for using multiple paths to provide service 527 protection that involve encoding the information in a packet 528 belonging to a DetNet flow into multiple transmission units, 529 typically combining information from multiple packets into any given 530 transmission unit. Such techniques may be applicable for use as a 531 DetNet service protection technique, assuming that the DetNet users' 532 needs for timeliness of delivery and freedom from interference with 533 misbehaving DetNet flows can be met. 535 No specific mechanisms are defined here, at this time. This section 536 will either be enhanced or removed. Contributions are invited. 538 4. DetNet Architecture 540 4.1. Traffic Engineering for DetNet 542 Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines 543 traffic-engineering architectures for generic applicability across 544 packet and non-packet networks. From TEAS perspective, Traffic 545 Engineering (TE) refers to techniques that enable operators to 546 control how specific traffic flows are treated within their networks. 548 Because if its very nature of establishing explicit optimized paths, 549 Deterministic Networking can be seen as a new, specialized branch of 550 Traffic Engineering, and inherits its architecture with a separation 551 into planes. 553 The Deterministic Networking architecture is thus composed of three 554 planes, a (User) Application Plane, a Controller Plane, and a Network 555 Plane, which echoes that of Figure 1 of Software-Defined Networking 556 (SDN): Layers and Architecture Terminology [RFC7426].: 558 4.1.1. The Application Plane 560 Per [RFC7426], the Application Plane includes both applications and 561 services. In particular, the Application Plane incorporates the User 562 Agent, a specialized application that interacts with the end user / 563 operator and performs requests for Deterministic Networking services 564 via an abstract Flow Management Entity, (FME) which may or may not be 565 collocated with (one of) the end systems. 567 At the Application Plane, a management interface enables the 568 negotiation of flows between end systems. An abstraction of the flow 569 called a Traffic Specification (TSpec) provides the representation. 570 This abstraction is used to place a reservation over the (Northbound) 571 Service Interface and within the Application plane. It is associated 572 with an abstraction of location, such as IP addresses and DNS names, 573 to identify the end systems and eventually specify intermediate 574 nodes. 576 4.1.2. The Controller Plane 578 The Controller Plane corresponds to the aggregation of the Control 579 and Management Planes in [RFC7426], though Common Control and 580 Measurement Plane (CCAMP) [CCAMP] makes an additional distinction 581 between management and measurement. When the logical separation of 582 the Control, Measurement and other Management entities is not 583 relevant, the term Controller Plane is used for simplicity to 584 represent them all, and the term controller refers to any device 585 operating in that plane, whether is it a Path Computation entity or a 586 Network Management entity (NME). The Path Computation Element (PCE) 587 [PCE] is a core element of a controller, in charge of computing 588 Deterministic paths to be applied in the Network Plane. 590 A (Northbound) Service Interface enables applications in the 591 Application Plane to communicate with the entities in the Controller 592 Plane. 594 One or more PCE(s) collaborate to implement the requests from the FME 595 as Per-Flow Per-Hop Behaviors installed in the intermediate nodes for 596 each individual flow. The PCEs place each flow along a deterministic 597 sequence of intermediate nodes so as to respect per-flow constraints 598 such as security and latency, and optimize the overall result for 599 metrics such as an abstract aggregated cost. The deterministic 600 sequence can typically be more complex than a direct sequence and 601 include redundancy path, with one or more packet replication and 602 elimination points. 604 4.1.3. The Network Plane 606 The Network Plane represents the network devices and protocols as a 607 whole, regardless of the Layer at which the network devices operate. 608 It includes Forwarding Plane (data plane), Application, and 609 Operational Plane (control plane) aspects. 611 The network Plane comprises the Network Interface Cards (NIC) in the 612 end systems, which are typically IP hosts, and intermediate nodes, 613 which are typically IP routers and switches. Network-to-Network 614 Interfaces such as used for Traffic Engineering path reservation in 615 [RFC5921], as well as User-to-Network Interfaces (UNI) such as 616 provided by the Local Management Interface (LMI) between network and 617 end systems, are both part of the Network Plane, both in the control 618 plane and the data plane. 620 A Southbound (Network) Interface enables the entities in the 621 Controller Plane to communicate with devices in the Network Plane. 622 This interface leverages and extends TEAS to describe the physical 623 topology and resources in the Network Plane. 625 Flow Management Entity 627 End End 628 System System 630 -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 632 PCE PCE PCE PCE 634 -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 636 intermediate intermed. intermed. intermed. 637 Node Node Node Node 638 NIC NIC 639 intermediate intermed. intermed. intermed. 640 Node Node Node Node 642 Figure 2 644 The intermediate nodes (and eventually the end systems NIC) expose 645 their capabilities and physical resources to the controller (the 646 PCE), and update the PCE with their dynamic perception of the 647 topology, across the Southbound Interface. In return, the PCE(s) set 648 the per-flow paths up, providing a Flow Characterization that is more 649 tightly coupled to the intermediate node Operation than a TSpec. 651 At the Network plane, intermediate nodes may exchange information 652 regarding the state of the paths, between adjacent systems and 653 eventually with the end systems, and forward packets within 654 constraints associated to each flow, or, when unable to do so, 655 perform a last resort operation such as drop or declassify. 657 This specification focuses on the Southbound interface and the 658 operation of the Network Plane. 660 4.2. DetNet flows 662 4.2.1. Source guarantees 664 For the purposes of congestion protection, DetNet flows can be 665 synchronous or asynchronous. In synchronous DetNet flows, at least 666 the intermediate nodes (and possibly the end systems) are closely 667 time synchronized, typically to better than 1 microsecond. By 668 transmitting packets from different DetNet flows or classes of DetNet 669 flows at different times, using repeating schedules synchronized 670 among the intermediate nodes, resources such as buffers and link 671 bandwidth can be shared over the time domain among different DetNet 672 flows. There is a tradeoff among techniques for synchronous DetNet 673 flows between the burden of fine-grained scheduling and the benefit 674 of reducing the required resources, especially buffer space. 676 In contrast, asynchronous DetNet flows are not coordinated with a 677 fine-grained schedule, so relay and end systems must assume worst- 678 case interference among DetNet flows contending for buffer resources. 679 Asynchronous DetNet flows are characterized by: 681 o A maximum packet size; 683 o An observation interval; and 685 o A maximum number of transmissions during that observation 686 interval. 688 These parameters, together with knowledge of the protocol stack used 689 (and thus the size of the various headers added to a packet), limit 690 the number of bit times per observation interval that the DetNet flow 691 can occupy the physical medium. 693 The source promises that these limits will not be exceeded. If the 694 source transmits less data than this limit allows, the unused 695 resources such as link bandwidth can be made available by the system 696 to non-DetNet packets. However, making those resources available to 697 DetNet packets in other DetNet flows would serve no purpose. Those 698 other DetNet flows have their own dedicated resources, on the 699 assumption that all DetNet flows can use all of their resources over 700 a long period of time. 702 Note that there is no provision in DetNet for throttling DetNet flows 703 (reducing the transmission rate via feedback); the assumption is that 704 a DetNet flow, to be useful, must be delivered in its entirety. That 705 is, while any useful application is written to expect a certain 706 number of lost packets, the real-time applications of interest to 707 DetNet demand that the loss of data due to the network is 708 extraordinarily infrequent. 710 Although DetNet strives to minimize the changes required of an 711 application to allow it to shift from a special-purpose digital 712 network to an Internet Protocol network, one fundamental shift in the 713 behavior of network applications is impossible to avoid: the 714 reservation of resources before the application starts. In the first 715 place, a network cannot deliver finite latency and practically zero 716 packet loss to an arbitrarily high offered load. Secondly, achieving 717 practically zero packet loss for unthrottled (though bandwidth 718 limited) DetNet flows means that bridges and routers have to dedicate 719 buffer resources to specific DetNet flows or to classes of DetNet 720 flows. The requirements of each reservation have to be translated 721 into the parameters that control each system's queuing, shaping, and 722 scheduling functions and delivered to the hosts, bridges, and 723 routers. 725 4.2.2. Incomplete Networks 727 The presence in the network of transit nodes or subnets that are not 728 fully capable of offering DetNet services complicates the ability of 729 the intermediate nodes and/or controller to allocate resources, as 730 extra buffering, and thus extra latency, must be allocated at points 731 downstream from the non-DetNet intermediate node for a DetNet flow. 733 4.3. Queuing, Shaping, Scheduling, and Preemption 735 DetNet achieves congestion protection and bounded delivery latency by 736 reserving bandwidth and buffer resources at every hop along the path 737 of the DetNet flow. The reservation itself is not sufficient, 738 however. Implementors and users of a number of proprietary and 739 standard real-time networks have found that standards for specific 740 data plane techniques are required to enable these assurances to be 741 made in a multi-vendor network. The fundamental reason is that 742 latency variation in one system results in the need for extra buffer 743 space in the next-hop system(s), which in turn, increases the worst- 744 case per-hop latency. 746 Standard queuing and transmission selection algorithms allow a 747 central controller to compute the latency contribution of each 748 transit node to the end-to-end latency, to compute the amount of 749 buffer space required in each transit node for each incremental 750 DetNet flow, and most importantly, to translate from a flow 751 specification to a set of values for the managed objects that control 752 each relay or end system. The IEEE 802 has specified (and is 753 specifying) a set of queuing, shaping, and scheduling algorithms that 754 enable each transit node (bridge or router), and/or a central 755 controller, to compute these values. These algorithms include: 757 o A credit-based shaper [IEEE802.1Q-2014] Clause 34. 759 o Time-gated queues governed by a rotating time schedule, 760 synchronized among all transit nodes [IEEE802.1Qbv]. 762 o Synchronized double (or triple) buffers driven by synchronized 763 time ticks. [IEEE802.1Qch]. 765 o Pre-emption of an Ethernet packet in transmission by a packet with 766 a more stringent latency requirement, followed by the resumption 767 of the preempted packet [IEEE802.1Qbu], [IEEE802.3br]. 769 While these techniques are currently embedded in Ethernet and 770 bridging standards, we can note that they are all, except perhaps for 771 packet preemption, equally applicable to other media than Ethernet, 772 and to routers as well as bridges. 774 4.4. Coexistence with normal traffic 776 A DetNet network supports the dedication of a high proportion (e.g. 777 75%) of the network bandwidth to DetNet flows. But, no matter how 778 much is dedicated for DetNet flows, it is a goal of DetNet to coexist 779 with existing Class of Service schemes (e.g., DiffServ). It is also 780 important that non-DetNet traffic not disrupt the DetNet flow, of 781 course (see Section 4.5 and Section 7). For these reasons: 783 o Bandwidth (transmission opportunities) not utilized by a DetNet 784 flow are available to non-DetNet packets (though not to other 785 DetNet flows). 787 o DetNet flows can be shaped or scheduled, in order to ensure that 788 the highest-priority non-DetNet packet also is ensured a worst- 789 case latency (at any given hop). 791 o When transmission opportunities for DetNet flows are scheduled in 792 detail, then the algorithm constructing the schedule should leave 793 sufficient opportunities for non-DetNet packets to satisfy the 794 needs of the users of the network. Detailed scheduling can also 795 permit the time-shared use of buffer resources by different DetNet 796 flows. 798 Ideally, the net effect of the presence of DetNet flows in a network 799 on the non-DetNet packets is primarily a reduction in the available 800 bandwidth. 802 4.5. Fault Mitigation 804 One key to building robust real-time systems is to reduce the 805 infinite variety of possible failures to a number that can be 806 analyzed with reasonable confidence. DetNet aids in the process by 807 providing filters and policers to detect DetNet packets received on 808 the wrong interface, or at the wrong time, or in too great a volume, 809 and to then take actions such as discarding the offending packet, 810 shutting down the offending DetNet flow, or shutting down the 811 offending interface. 813 It is also essential that filters and service remarking be employed 814 at the network edge to prevent non-DetNet packets from being mistaken 815 for DetNet packets, and thus impinging on the resources allocated to 816 DetNet packets. 818 There exist techniques, at present and/or in various stages of 819 standardization, that can perform these fault mitigation tasks that 820 deliver a high probability that misbehaving systems will have zero 821 impact on well-behaved DetNet flows, except of course, for the 822 receiving interface(s) immediately downstream of the misbehaving 823 device. Examples of such techniques include traffic policing 824 functions (e.g. [RFC2475]) and separating flows into per-flow rate- 825 limited queues. 827 4.6. Representative Protocol Stack Model 829 Figure 3 illustrates a conceptual DetNet data plane layering model. 830 One may compare it to that in [IEEE802.1CB], Annex C, a work in 831 progress. 833 DetNet data plane protocol stack 835 | packets going | ^ packets coming ^ 836 v down the stack v | up the stack | 837 +----------------------+ +-----------------------+ 838 | Source | | Destination | 839 +----------------------+ +-----------------------+ 840 | Service layer | | Service layer | 841 | Packet sequencing | | Duplicate elimination | 842 | Flow duplication | | Flow merging | 843 | Packet encoding | | Packet decoding | 844 +----------------------+ +-----------------------+ 845 | Transport layer | | Transport layer | 846 | Congestion prot. | | Congestion prot. | 847 +----------------------+ +-----------------------+ 848 | Lower layers | | Lower layers | 849 +----------------------+ +-----------------------+ 850 v ^ 851 \_________________________/ 853 Figure 3 855 Not all layers are required for any given application, or even for 856 any given network. The layers are, from top to bottom: 858 Application 859 Shown as "source" and "destination" in the diagram. 861 OAM 862 Operations, Administration, and Maintenance leverages in-band 863 and out-of-and signaling that validates whether the service 864 is effectively obtained within QoS constraints. OAM is not 865 shown in Figure 3; it may reside in any number of the layers. 867 OAM can involve specific tagging added in the packets for 868 tracing implementation or network configuration errors; 869 traceability enables to find whether a packet is a replica, 870 which relay node performed the replication, and which segment 871 was intended for the replica. 873 Packet sequencing 874 As part of DetNet service protection, supplies the sequence 875 number for packet replication and elimination (Section 3.4). 876 Peers with Duplicate elimination. This layer is not needed 877 if a higher-layer transport protocol is expected to perform 878 any packet sequencing and duplicate elimination required by 879 the DetNet flow duplication. 881 Duplicate elimination 882 As part of the DetNet service layer, based on the sequenced 883 number supplied by its peer, packet sequencing, Duplicate 884 elimination discards any duplicate packets generated by 885 DetNet flow duplication. It can operate on member flows, 886 compound flows, or both. The duplication may also be 887 inferred from other information such as the precise time of 888 reception in a scheduled network. The duplicate elimination 889 layer may also perform resequencing of packets to restore 890 packet order in a flow that was disrupted by the loss of 891 packets on one or another of the multiple paths taken. 893 Flow duplication 894 As part of DetNet service protection, replicates packets that 895 belong to a DetNet compound flow into two or more DetNet 896 member flows. Note that this function is separate from 897 packet sequencing. Flow duplication can be an explicit 898 duplication and remarking of packets, or can be performed by, 899 for example, techniques similar to ordinary multicast 900 replication. Peers with DetNet flow merging. 902 Network flow merging 903 As part of DetNet service protection, merges DetNet member 904 flows together for packets coming up the stack belonging to a 905 specific DetNet compound flow. Peers with DetNet flow 906 duplication. DetNet flow merging, together with packet 907 sequencing, duplicate elimination, and DetNet flow 908 duplication, performs packet replication and elimination 909 (Section 3.4). 911 Packet encoding 912 As part of DetNet service protection, as an alternative to 913 packet sequencing and flow duplication, packet encoding 914 combines the information in multiple DetNet packets, perhaps 915 from different DetNet compound flows, and transmits that 916 information in packets on different DetNet member Flows. 917 Peers with Packet decoding. 919 Packet decoding 920 As part of DetNet service protection, as an alternative to 921 flow merging and duplicate elimination, packet decoding takes 922 packets from different DetNet member flows, and computes from 923 those packets the original DetNet packets from the compound 924 flows input to packet encoding. Peers with Packet encoding. 926 Congestio protection 927 The DetNet transport layer provides congestion protection. 928 See Section 4.3. The actual queuing and shaping mechaniss 929 are typically provided by underlying subnet layers, but since 930 these are can be closely associated with the means of 931 providing paths for DetNet flows (e.g. MPLS LSPs or {VLAN, 932 multicast destination MAC address} pairs), the path and the 933 congestion protection are conflated in this figure. 935 Note that the packet sequencing and duplication elimination functions 936 at the source and destination ends of a DetNet compound flow may be 937 performed either in the end system or in a DetNet edge node. The 938 reader must not confuse a DetNet edge function with other kinds of 939 edge functions, e.g. an Label Edge Router, although the two functions 940 may be performed together. The DetNet edge function is concerned 941 with sequencing packets belonging to DetNet flows. The LER with 942 encapsulating/decapsulating packets for transport, and is considered 943 part of the network underlying the DetNet transport layer. 945 4.7. Exporting flow identification 947 An interesting feature of DetNet, and one that invites 948 implementations that can be accused of "layering violations", is the 949 need for lower layers to be aware of specific flows at higher layers, 950 in order to provide specific queuing and shaping services for 951 specific flows. For example: 953 o A non-IP, strictly L2 source end system X may be sending multiple 954 flows to the same L2 destination end system Y. Those flows may 955 include DetNet flows with different QoS requirements, and may 956 include non-DetNet flows. 958 o A router may be sending any number of flows to another router. 959 Again, those flows may include DetNet flows with different QoS 960 requirements, and may include non-DetNet flows. 962 o Two routers may be separated by bridges. For these bridges to 963 perform any required per-flow queuing and shaping, they must be 964 able to identify the individual flows. 966 o A Label Edge Router (LERs) may have a Label Switched Path (LSP) 967 set up for handling traffic destined for a particular IP address 968 carrying only non-DetNet flows. If a DetNet flow to that same 969 address is requested, a separate LSP may be needed, in order that 970 all of the Label Switch Routers (LSRs) along the path to the 971 destination give that flow special queuing and shaping. 973 The need for a lower-level DetNet node to be aware of individual 974 higher-layer flows is not unique to DetNet. But, given the endless 975 complexity of layering and relayering over tunnels that is available 976 to network designers, DetNet needs to provide a model for flow 977 identification that is at least somewhat better than packet 978 inspection. That is not to say that packet inspection to layer 4 or 979 5 addresses will not be used, or the capability standardized; but, 980 there are alternatives. 982 A DetNet relay node can connect DetNet flows on different paths using 983 different flow identification methods. For example: 985 o A single unicast DetNet flow passing from router A through a 986 bridged network to router B may be assigned a {VLAN, multicast 987 destination MAC address} pair that is unique within that bridged 988 network. The bridges can then identify the flow without accessing 989 higher-layer headers. Of course, the receiving router must 990 recognize and accept that multicast MAC address. 992 o A DetNet flow passing from LSR A to LSR B may be assigned a 993 different label than that used for other flows to the same IP 994 destination. 996 In any of the above cases, it is possible that an existing DetNet 997 flow can be used as a carrier for multiple DetNet sub-flows. (Not to 998 be confused with DetNet compound vs. member flows.) Of course, this 999 requires that the aggregate DetNet flow be provisioned properly to 1000 carry the sub-flows. 1002 Thus, rather than packet inspection, there is the option to export 1003 higher-layer information to the lower layer. The requirement to 1004 support one or the other method for flow identification (or both) is 1005 the essential complexity that DetNet brings to existing control plane 1006 models. 1008 4.8. Advertising resources, capabilities and adjacencies 1010 There are three classes of information that a central controller or 1011 decentralized control plane needs to know that can only be obtained 1012 from the end systems and/or transit nodes in the network. When using 1013 a peer-to-peer control plane, some of this information may be 1014 required by a system's neighbors in the network. 1016 o Details of the system's capabilities that are required in order to 1017 accurately allocate that system's resources, as well as other 1018 systems' resources. This includes, for example, which specific 1019 queuing and shaping algorithms are implemented (Section 4.3), the 1020 number of buffers dedicated for DetNet allocation, and the worst- 1021 case forwarding delay. 1023 o The dynamic state of an end or transit node's DetNet resources. 1025 o The identity of the system's neighbors, and the characteristics of 1026 the link(s) between the systems, including the length (in 1027 nanoseconds) of the link(s). 1029 4.9. Provisioning model 1031 4.9.1. Centralized Path Computation and Installation 1033 A centralized routing model, such as provided with a PCE (RFC 4655 1034 [RFC4655]), enables global and per-flow optimizations. (See 1035 Section 4.1.) The model is attractive but a number of issues are 1036 left to be solved. In particular: 1038 o Whether and how the path computation can be installed by 1) an end 1039 device or 2) a Network Management entity, 1041 o And how the path is set up, either by installing state at each hop 1042 with a direct interaction between the forwarding device and the 1043 PCE, or along a path by injecting a source-routed request at one 1044 end of the path. 1046 4.9.2. Distributed Path Setup 1048 Significant work on distributed path setup can be leveraged from MPLS 1049 Traffic Engineering, in both its GMPLS and non-GMPLS forms. The 1050 protocols within scope are Resource ReSerVation Protocol [RFC3209] 1051 [RFC3473](RSVP-TE), OSPF-TE [RFC4203] [RFC5392] and ISIS-TE [RFC5307] 1052 [RFC5316]. These should be viewed as starting points as there are 1053 feature specific extensions defined that may be applicable to DetNet. 1055 In a Layer-2 only environment, or as part of a layered approach to a 1056 mixed environment, IEEE 802.1 also has work, either completed or in 1057 progress. [IEEE802.1Q-2014] Clause 35 describes SRP, a peer-to-peer 1058 protocol for Layer-2 roughly analogous to RSVP [RFC2205]. 1059 [IEEE802.1Qca] defines how ISIS can provide multiple disjoint paths 1060 or distribution trees. Also in progress is [IEEE802.1Qcc], which 1061 expands the capabilities of SRP. 1063 The integration/interaction of the DetNet control layer with an 1064 underlying IEEE 802.1 sub-network control layer will need to be 1065 defined. 1067 4.10. Scaling to larger networks 1069 Reservations for individual DetNet flows require considerable state 1070 information in each transit node, especially when adequate fault 1071 mitigation (Section 4.5) is required. The DetNet data plane, in 1072 order to support larger numbers of DetNet flows, must support the 1073 aggregation of DetNet flows into tunnels, which themselves can be 1074 viewed by the transit nodes' data planes largely as individual DetNet 1075 flows. Without such aggregation, the per-relay system may limit the 1076 scale of DetNet networks. 1078 4.11. Connected islands vs. networks 1080 Given that users have deployed examples of the IEEE 802.1 TSN TG 1081 standards, which provide capabilities similar to DetNet, it is 1082 obvious to ask whether the IETF DetNet effort can be limited to 1083 providing Layer-2 connections (VPNs) between islands of bridged TSN 1084 networks. While this capability is certainly useful to some 1085 applications, and must not be precluded by DetNet, tunneling alone is 1086 not a sufficient goal for the DetNet WG. As shown in the 1087 Deterministic Networking Use Cases draft [I-D.ietf-detnet-use-cases], 1088 there are already deployments of Layer-2 TSN networks that are 1089 encountering the well-known problems of over-large broadcast domains. 1090 Routed solutions, and combinations routed/bridged solutions, are both 1091 required. 1093 5. Compatibility with Layer-2 1095 Standards providing similar capabilities for bridged networks (only) 1096 have been and are being generated in the IEEE 802 LAN/MAN Standards 1097 Committee. The present architecture describes an abstract model that 1098 can be applicable both at Layer-2 and Layer-3, and over links not 1099 defined by IEEE 802. It is the intention of the authors (and 1100 hopefully, as this draft progresses, of the DetNet Working Group) 1101 that IETF and IEEE 802 will coordinate their work, via the 1102 participation of common individuals, liaisons, and other means, to 1103 maximize the compatibility of their outputs. 1105 DetNet enabled end systems and intermediate nodes can be 1106 interconnected by sub-networks, i.e., Layer-2 technologies. These 1107 sub-networks will provide DetNet compatible service for support of 1108 DetNet traffic. Examples of sub-networks include 802.1TSN and a 1109 point-to-point OTN link. Of course, multi-layer DetNet systems may 1110 be possible too, where one DetNet appears as a sub-network, and 1111 provides service to, a higher layer DetNet system. 1113 6. Open Questions 1115 There are a number of architectural questions that will have to be 1116 resolved before this document can be submitted for publication. 1117 Aside from the obvious fact that this present draft is subject to 1118 change, there are specific questions to which the authors wish to 1119 direct the readers' attention. 1121 6.1. Flat vs. hierarchical control 1123 Boxes that are solely routers or solely bridges are rare in today's 1124 market. In a multi-tenant data center, multiple users' virtual 1125 Layer-2/Layer-3 topologies exist simultaneously, implemented on a 1126 network whose physical topology bears only accidental resemblance to 1127 the virtual topologies. 1129 While the forwarding topology (the bridges and routers) are an 1130 important consideration for a DetNet Flow Management Entity 1131 (Section 4.1.1), so is the purely physical topology. Ultimately, the 1132 model used by the management entities is based on boxes, queues, and 1133 links. The authors hope that the work of the TEAS WG will help to 1134 clarify exactly what model parameters need to be traded between the 1135 intermediate nodes and the controller(s). 1137 6.2. Peer-to-peer reservation protocol 1139 As described in Section 4.9.2, the DetNet WG needs to decide whether 1140 to support a peer-to-peer protocol for a source and a destination to 1141 reserve resources for a DetNet stream. Assuming that enabling the 1142 involvement of the source and/or destination is desirable (see 1143 Deterministic Networking Use Cases [I-D.ietf-detnet-use-cases]), it 1144 remains to decide whether the DetNet WG will make it possible to 1145 deploy at least some DetNet capabilities in a network using only a 1146 peer-to-peer protocol, without a central controller. 1148 (Note that a UNI (see Section 4.1.3) between an end system and a 1149 DetNet edge node, for sources and/or listeners to request DetNet 1150 services, can be either the first hop of a per-to-peer reservation 1151 protocol, or can be deflected by the DetNet edge node to a central 1152 controller for resolution. Similarly, a decision by a central 1153 controller can be effected by the controller instructing the end 1154 system or DetNet edge node to initiate a per-to-peer protocol 1155 activity.) 1157 6.3. Wireless media interactions 1159 Deterministic Networking Use Cases [I-D.ietf-detnet-use-cases] 1160 illustrates cases where wireless media are needed in a DetNet 1161 network. Some wireless media in general use, such as IEEE 802.11 1162 [IEEE802.1Q-2014], have significantly higher packet loss rates than 1163 typical wired media, such as Ethernet [IEEE802.3-2012]. IEEE 802.11 1164 includes support for such features as MAC-layer acknowledgements and 1165 retransmissions. 1167 The techniques described in Section 3 are likely to improve the 1168 ability of a mixed wired/wireless network to offer the DetNet QoS 1169 features. The interaction of these techniques with the features of 1170 specific wireless media, although they may be significant, cannot be 1171 addressed in this document. It remains to be decided to what extent 1172 the DetNet WG will address them, and to what extent other WGs, e.g. 1173 6TiSCH, will do so. 1175 7. Security Considerations 1177 Security in the context of Deterministic Networking has an added 1178 dimension; the time of delivery of a packet can be just as important 1179 as the contents of the packet, itself. A man-in-the-middle attack, 1180 for example, can impose, and then systematically adjust, additional 1181 delays into a link, and thus disrupt or subvert a real-time 1182 application without having to crack any encryption methods employed. 1183 See [RFC7384] for an exploration of this issue in a related context. 1185 Furthermore, in a control system where millions of dollars of 1186 equipment, or even human lives, can be lost if the DetNet QoS is not 1187 delivered, one must consider not only simple equipment failures, 1188 where the box or wire instantly becomes perfectly silent, but bizarre 1189 errors such as can be caused by software failures. Because there is 1190 essential no limit to the kinds of failures that can occur, 1191 protecting against realistic equipment failures is indistinguishable, 1192 in most cases, from protecting against malicious behavior, whether 1193 accidental or intentional. See also Section 4.5. 1195 Security must cover: 1197 o the protection of the signaling protocol 1198 o the authentication and authorization of the controlling systems 1200 o the identification and shaping of the DetNet flows 1202 8. Privacy Considerations 1204 DetNet is provides a Quality of Service (QoS), and as such, does not 1205 directly raise any new privacy considerations. 1207 However, the requirement for every (or almost every) node along the 1208 path of a DetNet flow to identify DetNet flows may present an 1209 additional attack surface for privacy, should the DetNet paradigm be 1210 found useful in broader environments. 1212 9. IANA Considerations 1214 This document does not require an action from IANA. 1216 10. Acknowledgements 1218 The authors wish to thank Jouni Korhonen, Erik Nordmark, George 1219 Swallow, Rudy Klecka, Anca Zamfir, David Black, Thomas Watteyne, 1220 Shitanshu Shah, Craig Gunther, Rodney Cummings, Balazs Varga, 1221 Wilfried Steiner, Marcel Kiessling, Karl Weber, Janos Farkas, Ethan 1222 Grossman, Pat Thaler, Lou Berger, and especially Michael Johas 1223 Teener, for their various contribution with this work. 1225 11. Access to IEEE 802.1 documents 1227 To access password protected IEEE 802.1 drafts, see the IETF IEEE 1228 802.1 information page at https://www.ietf.org/proceedings/52/slides/ 1229 bridge-0/tsld003.htm. 1231 12. Informative References 1233 [AVnu] http://www.avnu.org/, "The AVnu Alliance tests and 1234 certifies devices for interoperability, providing a simple 1235 and reliable networking solution for AV network 1236 implementation based on the Audio Video Bridging (AVB) 1237 standards.". 1239 [CCAMP] IETF, "Common Control and Measurement Plane", 1240 . 1242 [HART] www.hartcomm.org, "Highway Addressable Remote Transducer, 1243 a group of specifications for industrial process and 1244 control devices administered by the HART Foundation". 1246 [HSR-PRP] IEC, "High availability seamless redundancy (HSR) is a 1247 further development of the PRP approach, although HSR 1248 functions primarily as a protocol for creating media 1249 redundancy while PRP, as described in the previous 1250 section, creates network redundancy. PRP and HSR are both 1251 described in the IEC 62439 3 standard.", 1252 . 1255 [I-D.dt-detnet-dp-alt] 1256 Korhonen, J., Farkas, J., Mirsky, G., Thubert, P., 1257 Zhuangyan, Z., and L. Berger, "DetNet Data Plane Protocol 1258 and Solution Alternatives", draft-dt-detnet-dp-alt-04 1259 (work in progress), September 2016. 1261 [I-D.ietf-6tisch-architecture] 1262 Thubert, P., "An Architecture for IPv6 over the TSCH mode 1263 of IEEE 802.15.4", draft-ietf-6tisch-architecture-10 (work 1264 in progress), June 2016. 1266 [I-D.ietf-6tisch-tsch] 1267 Watteyne, T., Palattella, M., and L. Grieco, "Using 1268 IEEE802.15.4e TSCH in an IoT context: Overview, Problem 1269 Statement and Goals", draft-ietf-6tisch-tsch-06 (work in 1270 progress), March 2015. 1272 [I-D.ietf-detnet-problem-statement] 1273 Finn, N. and P. Thubert, "Deterministic Networking Problem 1274 Statement", draft-ietf-detnet-problem-statement-00 (work 1275 in progress), April 2016. 1277 [I-D.ietf-detnet-use-cases] 1278 Grossman, E., Gunther, C., Thubert, P., Wetterwald, P., 1279 Raymond, J., Korhonen, J., Kaneko, Y., Das, S., Zha, Y., 1280 Varga, B., Farkas, J., Goetz, F., and J. Schmitt, 1281 "Deterministic Networking Use Cases", draft-ietf-detnet- 1282 use-cases-10 (work in progress), July 2016. 1284 [I-D.ietf-roll-rpl-industrial-applicability] 1285 Phinney, T., Thubert, P., and R. Assimiti, "RPL 1286 applicability in industrial networks", draft-ietf-roll- 1287 rpl-industrial-applicability-02 (work in progress), 1288 October 2013. 1290 [I-D.svshah-tsvwg-deterministic-forwarding] 1291 Shah, S. and P. Thubert, "Deterministic Forwarding PHB", 1292 draft-svshah-tsvwg-deterministic-forwarding-04 (work in 1293 progress), August 2015. 1295 [IEEE802.11-2012] 1296 IEEE, "Wireless LAN Medium Access Control (MAC) and 1297 Physical Layer (PHY) Specifications", 2012, 1298 . 1301 [IEEE802.1AS-2011] 1302 IEEE, "Timing and Synchronizations (IEEE 802.1AS-2011)", 1303 2011, . 1306 [IEEE802.1BA-2011] 1307 IEEE, "AVB Systems (IEEE 802.1BA-2011)", 2011, 1308 . 1311 [IEEE802.1CB] 1312 IEEE, "Frame Replication and Elimination for Reliability 1313 (IEEE Draft P802.1CB)", 2016, 1314 . 1316 [IEEE802.1Q-2014] 1317 IEEE, "MAC Bridges and VLANs (IEEE 802.1Q-2014", 2014, 1318 . 1321 [IEEE802.1Qbu] 1322 IEEE, "Frame Preemption", 2016, 1323 . 1325 [IEEE802.1Qbv] 1326 IEEE, "Enhancements for Scheduled Traffic", 2016, 1327 . 1329 [IEEE802.1Qca] 1330 IEEE 802.1, "IEEE 802.1Qca Bridges and Bridged Networks - 1331 Amendment 24: Path Control and Reservation", IEEE 1332 P802.1Qca/D2.1 P802.1Qca, June 2015, 1333 . 1336 [IEEE802.1Qcc] 1337 IEEE, "Stream Reservation Protocol (SRP) Enhancements and 1338 Performance Improvements", 2016, 1339 . 1341 [IEEE802.1Qch] 1342 IEEE, "Cyclic Queuing and Forwarding", 2016, 1343 . 1345 [IEEE802.1TSNTG] 1346 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1347 Networks Task Group", 2013, 1348 . 1350 [IEEE802.3-2012] 1351 IEEE, "IEEE Stabdard for Ethernet", 2012, 1352 . 1355 [IEEE802.3br] 1356 IEEE, "Interspersed Express Traffic", 2016, 1357 . 1359 [IEEE802154] 1360 IEEE standard for Information Technology, "IEEE std. 1361 802.15.4, Part. 15.4: Wireless Medium Access Control (MAC) 1362 and Physical Layer (PHY) Specifications for Low-Rate 1363 Wireless Personal Area Networks", June 2011. 1365 [IEEE802154e] 1366 IEEE standard for Information Technology, "IEEE std. 1367 802.15.4e, Part. 15.4: Low-Rate Wireless Personal Area 1368 Networks (LR-WPANs) Amendment 1: MAC sublayer", April 1369 2012. 1371 [ISA100.11a] 1372 ISA/IEC, "ISA100.11a, Wireless Systems for Automation, 1373 also IEC 62734", 2011, < http://www.isa100wci.org/en- 1374 US/Documents/PDF/3405-ISA100-WirelessSystems-Future-broch- 1375 WEB-ETSI.aspx>. 1377 [ISA95] ANSI/ISA, "Enterprise-Control System Integration Part 1: 1378 Models and Terminology", 2000, . 1381 [ODVA] http://www.odva.org/, "The organization that supports 1382 network technologies built on the Common Industrial 1383 Protocol (CIP) including EtherNet/IP.". 1385 [PCE] IETF, "Path Computation Element", 1386 . 1388 [Profinet] 1389 http://us.profinet.com/technology/profinet/, "PROFINET is 1390 a standard for industrial networking in automation.", 1391 . 1393 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1394 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1395 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1396 September 1997, . 1398 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1399 and W. Weiss, "An Architecture for Differentiated 1400 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1401 . 1403 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1404 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1405 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1406 . 1408 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 1409 Switching (GMPLS) Signaling Resource ReserVation Protocol- 1410 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 1411 DOI 10.17487/RFC3473, January 2003, 1412 . 1414 [RFC4203] Kompella, K., Ed. and Y. Rekhter, Ed., "OSPF Extensions in 1415 Support of Generalized Multi-Protocol Label Switching 1416 (GMPLS)", RFC 4203, DOI 10.17487/RFC4203, October 2005, 1417 . 1419 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 1420 Element (PCE)-Based Architecture", RFC 4655, 1421 DOI 10.17487/RFC4655, August 2006, 1422 . 1424 [RFC5307] Kompella, K., Ed. and Y. Rekhter, Ed., "IS-IS Extensions 1425 in Support of Generalized Multi-Protocol Label Switching 1426 (GMPLS)", RFC 5307, DOI 10.17487/RFC5307, October 2008, 1427 . 1429 [RFC5316] Chen, M., Zhang, R., and X. Duan, "ISIS Extensions in 1430 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1431 Traffic Engineering", RFC 5316, DOI 10.17487/RFC5316, 1432 December 2008, . 1434 [RFC5392] Chen, M., Zhang, R., and X. Duan, "OSPF Extensions in 1435 Support of Inter-Autonomous System (AS) MPLS and GMPLS 1436 Traffic Engineering", RFC 5392, DOI 10.17487/RFC5392, 1437 January 2009, . 1439 [RFC5673] Pister, K., Ed., Thubert, P., Ed., Dwars, S., and T. 1440 Phinney, "Industrial Routing Requirements in Low-Power and 1441 Lossy Networks", RFC 5673, DOI 10.17487/RFC5673, October 1442 2009, . 1444 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 1445 L., and L. Berger, "A Framework for MPLS in Transport 1446 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 1447 . 1449 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 1450 Profile (MPLS-TP) Survivability Framework", RFC 6372, 1451 DOI 10.17487/RFC6372, September 2011, 1452 . 1454 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1455 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1456 October 2014, . 1458 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1459 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1460 Defined Networking (SDN): Layers and Architecture 1461 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1462 2015, . 1464 [TEAS] IETF, "Traffic Engineering Architecture and Signaling", 1465 . 1467 [WirelessHART] 1468 www.hartcomm.org, "Industrial Communication Networks - 1469 Wireless Communication Network and Communication Profiles 1470 - WirelessHART - IEC 62591", 2010. 1472 Authors' Addresses 1473 Norman Finn 1474 Self-employed 1475 1807 Santa Rita Rd 1476 Suite D, PMB 345 1477 Pleasanton, California 94566 1478 US 1480 Phone: +1 925 980 6430 1481 Email: nfinn@alumni.caltech.edu 1483 Pascal Thubert 1484 Cisco Systems 1485 Village d'Entreprises Green Side 1486 400, Avenue de Roumanille 1487 Batiment T3 1488 Biot - Sophia Antipolis 06410 1489 FRANCE 1491 Phone: +33 4 97 23 26 34 1492 Email: pthubert@cisco.com