idnits 2.17.1 draft-finn-detnet-architecture-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 21, 2016) is 2929 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'AVnu' is defined on line 1160, but no explicit reference was found in the text == Unused Reference: 'HART' is defined on line 1169, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-architecture' is defined on line 1187, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-6tisch-tsch' is defined on line 1192, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-roll-rpl-industrial-applicability' is defined on line 1205, but no explicit reference was found in the text == Unused Reference: 'I-D.svshah-tsvwg-deterministic-forwarding' is defined on line 1211, but no explicit reference was found in the text == Unused Reference: 'IEEE802.11-2012' is defined on line 1216, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1AS-2011' is defined on line 1222, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1TSNTG' is defined on line 1263, but no explicit reference was found in the text == Unused Reference: 'IEEE802154' is defined on line 1277, but no explicit reference was found in the text == Unused Reference: 'IEEE802154e' is defined on line 1283, but no explicit reference was found in the text == Unused Reference: 'ISA95' is defined on line 1295, but no explicit reference was found in the text == Unused Reference: 'ODVA' is defined on line 1299, but no explicit reference was found in the text == Unused Reference: 'Profinet' is defined on line 1306, but no explicit reference was found in the text == Unused Reference: 'RFC2205' is defined on line 1311, but no explicit reference was found in the text == Unused Reference: 'RFC5673' is defined on line 1326, but no explicit reference was found in the text == Unused Reference: 'WirelessHART' is defined on line 1354, but no explicit reference was found in the text == Outdated reference: A later version (-30) exists of draft-ietf-6tisch-architecture-09 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-08 Summary: 0 errors (**), 0 flaws (~~), 20 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft P. Thubert 4 Intended status: Standards Track Cisco 5 Expires: September 22, 2016 M. Johas Teener 6 Broadcom 7 March 21, 2016 9 Deterministic Networking Architecture 10 draft-finn-detnet-architecture-04 12 Abstract 14 Deterministic Networking (DetNet) provides a capability to carry 15 specified unicast or multicast data flows for real-time applications 16 with extremely low data loss rates and bounded latency. Techniques 17 used include: 1) reserving data plane resources for individual (or 18 aggregated) DetNet flows in some or all of the relay systems (bridges 19 or routers) along the path of the flow; 2) providing fixed paths for 20 DetNet flows that do not rapidly change with the network topology; 21 and 3) sequentializing, replicating, and eliminating duplicate 22 packets at various points to ensure the availability of at least one 23 path. The capabilities can be managed by configuration, or by manual 24 or automatic network management. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on September 22, 2016. 43 Copyright Notice 45 Copyright (c) 2016 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 2.1. Terms used in this document . . . . . . . . . . . . . . . 4 63 2.2. IEEE 802 TSN to DetNet dictionary . . . . . . . . . . . . 5 64 3. Providing the DetNet Quality of Service . . . . . . . . . . . 5 65 3.1. Zero Congestion Loss . . . . . . . . . . . . . . . . . . 7 66 3.2. Pinned paths . . . . . . . . . . . . . . . . . . . . . . 8 67 3.3. Packet replication and deletion . . . . . . . . . . . . . 8 68 4. DetNet Architecture . . . . . . . . . . . . . . . . . . . . . 9 69 4.1. Elements of DetNet Architecture . . . . . . . . . . . . . 9 70 4.2. Traffic Engineering for DetNet . . . . . . . . . . . . . 11 71 4.2.1. The Application Plane . . . . . . . . . . . . . . . . 13 72 4.2.2. The Controller Plane . . . . . . . . . . . . . . . . 13 73 4.2.3. The Network Plane . . . . . . . . . . . . . . . . . . 14 74 4.3. DetNet flows . . . . . . . . . . . . . . . . . . . . . . 15 75 4.3.1. Source guarantees . . . . . . . . . . . . . . . . . . 15 76 4.3.2. Incomplete Networks . . . . . . . . . . . . . . . . . 16 77 4.4. Queuing, Shaping, Scheduling, and Preemption . . . . . . 16 78 4.5. Coexistence with normal traffic . . . . . . . . . . . . . 17 79 4.6. Fault Mitigation . . . . . . . . . . . . . . . . . . . . 18 80 4.7. Protocol Stack Model . . . . . . . . . . . . . . . . . . 18 81 4.8. Advertising resources, capabilities and adjacencies . . . 20 82 4.9. Provisioning model . . . . . . . . . . . . . . . . . . . 20 83 4.9.1. Centralized Path Computation and Installation . . . . 20 84 4.9.2. Distributed Path Setup . . . . . . . . . . . . . . . 21 85 4.10. Scaling to larger networks . . . . . . . . . . . . . . . 21 86 4.11. Connected islands vs. networks . . . . . . . . . . . . . 21 87 5. Compatibility with Layer-2 . . . . . . . . . . . . . . . . . 22 88 6. Open Questions . . . . . . . . . . . . . . . . . . . . . . . 22 89 6.1. Data plane shapers and schedulers . . . . . . . . . . . . 22 90 6.2. DetNet flow identification and sequencing . . . . . . . . 22 91 6.3. Flat vs. hierarchical control . . . . . . . . . . . . . . 23 92 6.4. Peer-to-peer reservation protocol . . . . . . . . . . . . 23 93 6.5. Wireless media interactions . . . . . . . . . . . . . . . 24 94 7. Security Considerations . . . . . . . . . . . . . . . . . . . 24 95 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 25 96 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 25 97 10. Access to IEEE 802.1 documents . . . . . . . . . . . . . . . 25 98 11. Informative References . . . . . . . . . . . . . . . . . . . 25 99 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 29 101 1. Introduction 103 Deterministic Networking (DetNet) is a service that can be offered by 104 a network to data flows (DetNet flows) that that are limited, at 105 their source, to a maximum data rate specified by that source. 106 DetNet provides these flows extremely low packet loss rates and 107 assured maximum end-to-end delivery latency. This is accomplished by 108 dedicating network resources such as link bandwidth and buffer space 109 to DetNet flows and/or classes of DetNet flows. Unused reserved 110 resources are available to non-DetNet packets. 112 The Deterministic Networking Problem Statement 113 [I-D.finn-detnet-problem-statement] introduces Deterministic 114 Networking, and Deterministic Networking Use Cases 115 [I-D.ietf-detnet-use-cases] summarizes the need for it. 117 A goal of DetNet is a converged network in all respects. That is, 118 the presence of DetNet flows does not preclude non-DetNet flows, and 119 the benefits offered DetNet flows should not, except in extreme 120 cases, prevent existing QoS mechanisms from operating in a normal 121 fashion, subject to the bandwidth required for the DetNet flows. A 122 single source-destination pair can trade both DetNet and non-DetNet 123 flows. End systems and applications need not instantiate special 124 interfaces for DetNet flows. Networks are not restricted to certain 125 topologies; connectivity is not restricted. Any application that 126 generates a data flow that can be usefully characterized as having a 127 maximum bandwidth should be able to take advantage of DetNet, as long 128 as the necessary resources can be reserved. Reservations can be made 129 by the application itself, via network management, by an applications 130 controller, or by other means. 132 Many applications of interest to Deterministic Networking require the 133 ability to synchronize the clocks in end systems to a sub-microsecond 134 accuracy. Some of the queue control techniques defined in 135 Section 4.4 also require time synchronization among relay systems. 136 The means used to achieve time synchronization are not addressed in 137 this document. 139 The present document is an individual contribution, intended by the 140 authors for eventual adoption by the DetNet working group. As such, 141 it expresses the only the opinions of the authors. 143 2. Terminology 145 2.1. Terms used in this document 147 The following special terms are used in this document in order to 148 avoid the assumption that a given element in the architecture does or 149 does not have Internet Protocol stack, functions as a router, bridge, 150 firewall, or otherwise plays a particular role at Layer-2 or higher. 152 destination 153 An end system capable of sinking a DetNet flow. 155 DetNet domain 156 The portion of a network that is DetNet aware. It includes 157 end systems and other DetNet nodes. 159 DetNet flow 160 A DetNet flow is a sequence of packets from a single source, 161 through some number of relay systems to one or more 162 destinations, that is limited by the source in its maximum 163 packet size and transmission rate, and can thus be ensured 164 the DetNet Quality of Service (QoS) from the network. 166 DetNet node 167 A DetNet aware end system or relay system. "DetNet" may be 168 omitted in some text. 170 end system 171 Commonly called a "host" or "node" in IETF documents, and an 172 "end station" is IEEE 802 documents. End systems of interest 173 to this document are either sources or destinations of L2 174 and/or L3 DatNet streams. Note that a system that takes non- 175 DetNet aware traffic and transmits it via a DetNet flow is 176 also an end system. (For comparison, a Label Edge Router 177 (LER) would be an MPLS "end system".) 179 link 180 A connection between two DetNet nodes. It may be composed of 181 a physical link or a sub-network technology that can provide 182 appropriate traffic delivery for DetNet flows. 184 relay system 185 A router, transit node, bridge, Label Switch Router (LSR), 186 firewall, or any other system that forwards packets from one 187 interface to another. 189 reservation 190 A trail of configuration between source to destination(s) 191 through relay systems associated with a DetNet flow, required 192 to deliver the benefits of DetNet. 194 source 195 An end system capable of sourcing a DetNet flow. 197 2.2. IEEE 802 TSN to DetNet dictionary 199 This section also serves as a dictionary for translating from the 200 terms used by the IEEE 802 Time-Sensitive Networking (TSN) Task Group 201 to those of the DetNet WG. 203 listener 204 The IEEE 802 term for a destination of a DetNet flow. 206 stream 207 The IEEE 802 term for a DetNet flow. 209 talker 210 The IEEE 802 term for the source of a DetNet flow. 212 3. Providing the DetNet Quality of Service 214 DetNet Quality of Service is expressed in terms of: 216 o Minimum and maximum end-to-end latency from source to destination; 218 o Probability of loss of a packet, under various assumptions as to 219 the operational states of the relay systems and links; 221 It is a distinction of DetNet that it is concerned solely with worst- 222 case values for the end-to-end latency. Average, mean, or typical 223 values are of no interest, because they do not affect the ability of 224 a real-time system to perform its tasks. In general, a trivial 225 priority-based queuing scheme will give better average latency to a 226 data flow than DetNet, but of course, the worst-case latency is 227 essentially unbounded. 229 Three techniques are employed by DetNet to achieve these QoS 230 parameters: 232 a. Zero congestion loss (Section 3.1). Network resources such as 233 link bandwidth, buffers, queues, shapers, and scheduled input/ 234 output slots are assigned in each relay system to the use of a 235 specific DetNet flow or class of DetNet flows. Given a finite 236 amount of buffer space, zero congestion loss necessarily ensures 237 a bounded end-to-end latency. Depending on the resources 238 employed, a minimum latency, and thus bounded jitter, can also be 239 achieved. 241 b. Pinned paths (Section 3.2). Point-to-point paths or point-to- 242 multipoint trees through the network from a source to one or more 243 destinations can be established, and DetNet flows assigned to 244 follow a particular path or tree. 246 c. Packet replication and deletion (Section 3.3). End systems and/ 247 or relay systems can number packets sequentially, replicate them, 248 and later eliminate all but one of the replicants, at multiple 249 points in the network in order to ensure that one (or more) 250 equipment failure events still leave at least one path intact for 251 a DetNet flow. This function is a "hitless" version of, e.g., 252 the 1+1 linear protection in [RFC6372]. That is, instead of 253 switching from one flow to the other when a failure of a flow is 254 detected, DetNet combines both flows, and performs a packet-by- 255 packet selection of which to discard, based on sequence number. 257 These techniques address both of the DetNet QoS requirements. Given 258 that relay nodes have a finite amount of buffer space, zero 259 congestion loss (Section 3.1) necessarily results in a maximum end- 260 to-end latency. It also addresses the largest contribution to packet 261 loss, which is buffer congestion. Packet replication and deletion 262 mitigates the other most important contributions to packet loss, 263 namely random media errors and equipment failure. 265 These three techniques can be applied independently, giving eight 266 possible combinations, including none (no DetNet), although some 267 combinations are of wider utility than others. This separation keeps 268 the protocol stack coherent and maximizes interoperability with 269 existing and developing standards in this (IETF) and other Standards 270 Development Organizations. Some examples of typical expected 271 combinations: 273 o Pinned paths (a) plus packet replication (b) are exactly the 274 techniques employed by [HSR-PRP]. Pinned paths are achieved by 275 limiting the physical topology of the network, and the 276 sequentialization, replication, and duplicate elimination are 277 facilitated by packet tags added at the front or the end of 278 Ethernet frames. 280 o Zero congestion loss (a) alone is is offered by IEEE 802.1 Audio 281 Video bridging [IEEE802.1BA-2011]. As long as the network suffers 282 no failures, zero congestion loss can be achieved through the use 283 of a reservation protocol (MSRP), shapers in every relay system 284 (bridge), and a bit of network calculus. 286 o Using all three together gives maximum protection. 288 There are, of course, simpler methods available (and employed, today) 289 to achieve levels of latency and packet loss that are satisfactory 290 for many applications. Prioritization and over-provisioning is one 291 such technique. However, these methods generally work best in the 292 absence of any significant amount of non-critical traffic in the 293 network (if, indeed, such traffic is supported at all), or work only 294 if the critical traffic constitutes only a small portion of the 295 network's theoretical capacity, or work only if all systems are 296 functioning properly, or in the absence of actions by end systems 297 that disrupt the network's operations. 299 There are any number of methods in use, defined, or in progress for 300 accomplishing each of the above techniques. It is expected that this 301 DetNet Architecture will assist various vendors, users, and/or 302 "vertical" Standards Development Organizations (dedicated to a single 303 industry) to make selections among the available means of 304 implementing DetNet networks. 306 3.1. Zero Congestion Loss 308 The primary means by which DetNet achieves its QoS assurances is to 309 completely eliminate congestion at an output port as a cause of 310 packet loss. Given that a DetNet flow cannot be throttled, this can 311 be achieved only by the provision of sufficient buffer storage at 312 each hop through the network to ensure that no packets are dropped 313 due to a lack of buffer storage. 315 Ensuring adequate buffering requires, in turn, that the source, and 316 every relay system along the path to the destination (or nearly every 317 relay system -- see Section 4.3.2) be careful to regulate its output 318 to not exceed the data rate for any DetNet flow, except for brief 319 periods when making up for interfering traffic. Any packet sent 320 ahead of its time potentially adds to the number of buffers required 321 by the next hop, and may thus exceed the resources allocated for a 322 particular DetNet flow. 324 The low-level mechanisms described in Section 4.4 provide the 325 necessary regulation of transmissions by an edge system or relay 326 system to ensure zero congestion loss. The reservation of the 327 bandwidth and buffers for a DetNet flow requires the provisioning 328 described in Section 4.9. 330 A DetNet node may have other resources requiring allocation and/or 331 scheduling, that might otherwise be over-subscribed and trigger 332 congestion loss. 334 3.2. Pinned paths 336 In networks controlled by typical peer-to-peer protocols such as IEEE 337 802.1 ISIS bridged networks or IETF OSPF routed networks, a network 338 topology event in one part of the network can impact, at least 339 briefly, the delivery of data in parts of the network remote from the 340 failure or recovery event. Thus, even redundant paths through a 341 network, if controlled by the typical peer-to-peer protocols, do not 342 eliminate the chances of brief losses of contact. 344 Many real-time networks rely on physical rings or chains of two-port 345 devices, with a relatively simple ring control protocol. This 346 supports redundant paths with a minimum of wiring. As an additional 347 benefit, ring topologies can often utilize different topology 348 management protocols than those used for a mesh network, with a 349 consequent reduction in the response time to topology changes. Of 350 course, this comes at some cost in terms of increased hop count, and 351 thus latency, for the typical path. 353 In order to get the advantages of low hop count and still ensure 354 against even very brief losses of connectivity, DetNet employs pinned 355 paths, where the path taken by a given DetNet flow does not change, 356 at least immediately, and likely not at all, in response to network 357 topology events. When combined with packet replication and deletion 358 (Section 3.3), this results in a high likelihood of continuous 359 connectivity. Pinned paths are commonly used in MPLS TE LSPs. 361 3.3. Packet replication and deletion 363 After congestion loss has been eliminated, the most important causes 364 of packet loss are random media and/or memory faults, and equipment 365 failures. Both causes of packet loss can be greatly reduced by 366 sending the same packets over multiple paths. 368 Packet replication and deletion, also known as seamless redundancy 369 [HSR-PRP], or 1+1 hitless protection, involves three capabilities: 371 o Providing sequencing information, once, to the packets of a DetNet 372 flow. This may be done by adding a sequence number or time stamp 373 as part of DetNet, or may be inherent in the packet, e.g. in a 374 transport protocol. 376 o Replicating these packets and, typically, sending them along at 377 least two different paths to the destination(s). (Often, the 378 pinned paths of Section 3.2.) 380 o Discarding duplicated packets. 382 In the simplest case, this amounts to replicating each packet in a 383 source that has two interfaces, and conveying them through the 384 network, along separate paths, to the similarly dual-homed 385 destinations, that discard the extras. This ensures that one path 386 (with zero congestion loss) remains, even if some relay system fails. 387 The sequence numbers can also be used for loss detection and for re- 388 ordering. 390 Alternatively, relay systems in the network can provide replication 391 and elimination facilities at various points in the network, so that 392 multiple failures can be accommodated. 394 This is shown in the following figure, where the two relay systems 395 each replicate (R) the DetNet flow on input, sending the DetNet flow 396 to both the other relay system and to the end system, and eliminate 397 duplicates (E) on the output interface to the right-hand end system. 398 Any one link in the network can fail, and the Detnet flow can still 399 get through. Furthermore, two links can fail, as long as they are in 400 different segments of the network. 402 > > > > > > > > relay > > > > > > > > 403 > /------------+ R system E +------------\ > 404 > / v + ^ \ > 405 end R + v | ^ + E end 406 system + v | ^ + system 407 > \ v + ^ / > 408 > \------------+ R relay E +------------/ > 409 > > > > > > > > system > > > > > > > > 411 Figure 1 413 Note that packet replication and deletion does not react to and 414 correct failures; it is entirely passive. Thus, intermittent 415 failures, mistakenly created access control lists, or misrouted data 416 is handled just the same as the equipment failures that are detected 417 handled by typical routing and bridging protocols. 419 4. DetNet Architecture 421 4.1. Elements of DetNet Architecture 423 The DetNet architecture has a number of elements, discussed in the 424 following sections. Note that not every application requires all of 425 these elements. 427 a. A model for the definition, identification, and operation of 428 DetNet flows (Section 4.3), for use by relay systems to classify 429 and process individual packets following per-flow rules. 431 b. A model for the flow of data out of an end system or through a 432 relay system that can be used to predict the bounds for that 433 system's impact on the QoS of a DetNet flow, for use by the 434 Controllers to configure policing and shaping engines in Network 435 Systems over the Southbound interface. The model includes: 437 1. A model for queuing, transmission selection, shaping, 438 preemption, and timing resources that can be used by an end 439 system or relay system to control the selection of packets 440 output on an interface. These models must have sufficiently 441 well-defined characteristics, both individually and in the 442 aggregate, to give predictable results for the QoS for DetNet 443 packets (Section 4.4). 445 2. A model for identifying misbehaving DetNet flows and 446 mitigating their impact on properly functioning DetNet flows 447 (Section 4.6). 449 c. A model for the relay system to inform the controller(s) of the 450 information it needs for adequate path computations (Section 4.2) 451 including: 453 1. Systems' individual capabilities (e.g. can do replication, 454 can do precise time). 456 2. Link capabilities and resources (e.g. bandwidth, transmission 457 delay, hardware deterministic support to the physical layer, 458 ...) 460 3. Physical resources (total and available buffers, timers, 461 queues, etc) 463 4. Network Adjacencies (neighbors) 465 d. A model for the provision of a service, by end systems or relay 466 systems, to replicate and forward a DetNet flow over redundant 467 paths. The model includes: 469 1. A model for specifying multiple stable paths across a network 470 that can perform packet forwarding at both Layer 3 and at 471 lower layers, to which specific DetNet flows can be assigned 472 (Section 4.2). 474 2. A model and data plane format(s) for sequencing and 475 replicating the packets of a DetNet flow, typically at or 476 near the source, sending the replicated DetNet flows over 477 different stable paths, merging and/or re-replicating those 478 packets at other points in the network, and finally 479 eliminating the duplicates, typically at or near the 480 destination(s), in order to provide high availability 481 (Section 3.3). 483 e. The protocol stack model for an end system and/or a relay system 484 should support the above elements in a manner that maximizes the 485 applicability of existing standards and protocols to the DetNet 486 problem, and allows for the creation of new protocols only where 487 needed, thus making DetNet an add-on feature to existing 488 networks, rather than a new way to do networking. In particular 489 this protocol stack supports networks in which the path from 490 source to destination(s) includes bridges and/or routers in any 491 order (Section 4.7). 493 f. A variety of models for the provisioning of DetNet flows can be 494 envisioned, including orchestration by a central controller or by 495 a federation of controllers, via control plane protocols running 496 on relay systems and end systems, by off-line configuration, or 497 by a combination of these methods. The provisioning models are 498 similar to existing Layer-2 and Layer-3 models, in order to 499 minimize the amount of innovation required in this area 500 (Section 4.9). 502 4.2. Traffic Engineering for DetNet 504 Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines 505 traffic-engineering architectures for generic applicability across 506 packet and non-packet networks. From TEAS perspective, Traffic 507 Engineering (TE) refers to techniques that enable operators to 508 control how specific traffic flows are treated within their networks. 510 Because if its very nature of establishing pinned optimized paths, 511 Deterministic Networking can be seen as a new, specialized branch of 512 Traffic Engineering, and inherits its architecture with a separation 513 into planes. 515 The Deterministic Networking architecture is thus composed of three 516 planes, a (User) Application Plane, a Controller Plane, and a Network 517 Plane, which echoes that of Software-Defined Networking (SDN): Layers 518 and Architecture Terminology [RFC7426] which is represented below: 520 SDN Layers and Architecture Terminology per RFC 7426 522 o--------------------------------o 523 | | 524 | +-------------+ +----------+ | 525 | | Application | | Service | | 526 | +-------------+ +----------+ | 527 | Application Plane | 528 o---------------Y----------------o 529 | 530 *-----------------------------Y---------------------------------* 531 | Network Services Abstraction Layer (NSAL) | 532 *------Y------------------------------------------------Y-------* 533 | | 534 | Service Interface | 535 | | 536 o------Y------------------o o---------------------Y------o 537 | | Control Plane | | Management Plane | | 538 | +----Y----+ +-----+ | | +-----+ +----Y----+ | 539 | | Service | | App | | | | App | | Service | | 540 | +----Y----+ +--Y--+ | | +--Y--+ +----Y----+ | 541 | | | | | | | | 542 | *----Y-----------Y----* | | *---Y---------------Y----* | 543 | | Control Abstraction | | | | Management Abstraction | | 544 | | Layer (CAL) | | | | Layer (MAL) | | 545 | *----------Y----------* | | *----------Y-------------* | 546 | | | | | | 547 o------------|------------o o------------|---------------o 548 | | 549 | CP | MP 550 | Southbound | Southbound 551 | Interface | Interface 552 | | 553 *------------Y---------------------------------Y----------------* 554 | Device and resource Abstraction Layer (DAL) | 555 *------------Y---------------------------------Y----------------* 556 | | | | 557 | o-------Y----------o +-----+ o--------Y----------o | 558 | | Forwarding Plane | | App | | Operational Plane | | 559 | o------------------o +-----+ o-------------------o | 560 | Network Device | 561 +---------------------------------------------------------------+ 563 Figure 2 565 4.2.1. The Application Plane 567 Per [RFC7426], the Application Plane includes both applications and 568 services. In particular, the Application Plane incorporates the User 569 Agent, a specialized application that interacts with the end user / 570 operator and performs requests for Deterministic Networking services 571 via an abstract Flow Management Entity, (FME) which may or may not be 572 collocated with (one of) the end systems. 574 At the Application Plane, a management interface enables the 575 negotiation of flows between end systems. An abstraction of the flow 576 called a Traffic Specification (TSpec) provides the representation. 577 This abstraction is used to place a reservation over the (Northbound) 578 Service Interface and within the Application plane. It is associated 579 with an abstraction of location, such as IP addresses and DNS names, 580 to identify the end systems and eventually specify intermediate relay 581 systems. 583 4.2.2. The Controller Plane 585 The Controller Plane corresponds to the aggregation of the Control 586 and Management Planes in [RFC7426], though Common Control and 587 Measurement Plane (CCAMP) [CCAMP] makes an additional distinction 588 between management and measurement. When the logical separation of 589 the Control, Measurement and other Management entities is not 590 relevant, the term Controller Plane is used for simplicity to 591 represent them all, and the term controller refers to any device 592 operating in that plane, whether is it a Path Computation entity or a 593 Network Management entity (NME). The Path Computation Element (PCE) 594 [PCE] is a core element of a controller, in charge of computing 595 Deterministic paths to be applied in the Network Plane. 597 A (Northbound) Service Interface enables applications in the 598 Application Plane to communicate with the entities in the Controller 599 Plane. 601 One or more PCE(s) collaborate to implement the requests from the FME 602 as Per-fFlow Per-Hop Behaviors installed in the relay systems for 603 each individual flow. The PCEs place each flow along a deterministic 604 sequence of relay systems so as to respect per-flow constraints such 605 as security and latency, and optimize the overall result for metrics 606 such as an abstract aggregated cost. The deterministic sequence can 607 typically be more complex than a direct sequence and include 608 redundancy path, with one or more packet replication and elimination 609 points. 611 4.2.3. The Network Plane 613 The Network Plane represents the network devices and protocols as a 614 whole, regardless of the Layer at which the network devices operate. 615 It includes Forwarding Plane (data plane), Application, and 616 Operational Plane (control plane) aspects. 618 The network Plane comprises the Network Interface Cards (NIC) in the 619 end systems, which are typically IP hosts, and relay systems, which 620 are typically IP routers and switches. Network-to-Network Interfaces 621 such as used for Traffic Engineering path reservation in [RFC5921], 622 as well as User-to-Network Interfaces (UNI) such as provided by the 623 Local Management Interface (LMI) between network and end systems, are 624 both part of the Network Plane, both in the control plane and the 625 data plane. 627 A Southbound (Network) Interface enables the entities in the 628 Controller Plane to communicate with devices in the Network Plane. 629 This interface leverages and extends TEAS to describe the physical 630 topology and resources in the Network Plane. 632 Flow Management Entity 634 End End 635 System System 637 -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 639 PCE PCE PCE PCE 641 -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 643 Relay Relay Relay Relay 644 System System System System 645 NIC NIC 646 Relay Relay Relay Relay 647 System System System System 649 Figure 3 651 The relay systems (and eventually the end systems NIC) expose their 652 capabilities and physical resources to the controller (the PCE), and 653 update the PCE with their dynamic perception of the topology, across 654 the Southbound Interface. In return, the PCE(s) set the per-flow 655 paths up, providing a Flow Characterization that is more tightly 656 coupled to the relay system Operation than a TSpec. 658 At the Network plane, relay systems may exchange information 659 regarding the state of the paths, between adjacent systems and 660 eventually with the end systems, and forward packets within 661 constraints associated to each flow, or, when unable to do so, 662 perform a last resort operation such as drop or declassify. 664 This specification focuses on the Southbound interface and the 665 operation of the Network Plane. 667 4.3. DetNet flows 669 4.3.1. Source guarantees 671 DetNet flows can by synchronous or asynchronous. In synchronous 672 DetNet flows, at least the relay systems (and possibly the end 673 systems) are closely time synchronized, typically to better than 1 674 microsecond. By transmitting packets from different DetNet flows or 675 classes of DetNet flows at different times, using repeating schedules 676 synchronized among the relay systems, resources such as buffers and 677 link bandwidth can be shared over the time domain among different 678 DetNet flows. There is a tradeoff among techniques for synchronous 679 DetNet flows between the burden of fine-grained scheduling and the 680 benefit of reducing the required resources, especially buffer space. 682 In contrast, asynchronous DetNet flows are not coordinated with a 683 fine-grained schedule, so relay and end systems must assume worst- 684 case interference among DetNet flows contending for buffer resources. 685 Asynchronous DetNet flows are characterized by: 687 o A maximum packet size; 689 o An observation interval; and 691 o A maximum number of transmissions during that observation 692 interval. 694 These parameters, together with knowledge of the protocol stack used 695 (and thus the size of the various headers added to a packet), limit 696 the number of bit times per observation interval that the DetNet flow 697 can occupy the physical medium. 699 The source promises that these limits will not be exceeded. If the 700 source transmits less data than this limit allows, the unused 701 resources such as link bandwidth can be made available by the system 702 to non-DetNet packets. However, making those resources available to 703 DetNet packets in other DetNet flows would serve no purpose. Those 704 other DetNet flows have their own dedicated resources, on the 705 assumption that all DetNet flows can use all of their resources over 706 a long period of time. 708 Note that there is no provision in DetNet for throttling DetNet flows 709 (reducing the transmission rate via feedback); the assumption is that 710 a DetNet flow, to be useful, must be delivered in its entirety. That 711 is, while any useful application is written to expect a certain 712 number of lost packets, the real-time applications of interest to 713 DetNet demand that the loss of data due to the network is 714 extraordinarily infrequent. 716 Although DetNet strives to minimize the changes required of an 717 application to allow it to shift from a special-purpose digital 718 network to an Internet Protocol network, one fundamental shift in the 719 behavior of network applications is impossible to avoid--the 720 reservation of resources before the application starts. In the first 721 place, a network cannot deliver finite latency and practically zero 722 packet loss to an arbitrarily high offered load. Secondly, achieving 723 practically zero packet loss for unthrottled (though bandwidth 724 limited) DetNet flows means that bridges and routers have to dedicate 725 buffer resources to specific DetNet flows or to classes of DetNet 726 flows. The requirements of each reservation have to be translated 727 into the parameters that control each system's queuing, shaping, and 728 scheduling functions and delivered to the hosts, bridges, and 729 routers. 731 4.3.2. Incomplete Networks 733 The presence in the network of relay systems that are not fully 734 capable of offering DetNet services complicates the ability of the 735 relay systems and/or controller to allocate resources, as extra 736 buffering, and thus extra latency, must be allocated at points 737 downstream from the non-DetNet relay system for a DetNet flow. 739 4.4. Queuing, Shaping, Scheduling, and Preemption 741 As described above, DetNet achieves its aims by reserving bandwidth 742 and buffer resources at every hop along the path of the DetNet flow. 743 The reservation itself is not sufficient, however. Implementors and 744 users of a number of proprietary and standard real-time networks have 745 found that standards for specific data plane techniques are required 746 to enable these assurances to be made in a multi-vendor network. The 747 fundamental reason is that latency variation in one system results in 748 the need for extra buffer space in the next-hop system(s), which in 749 turn, increases the worst-case per-hop latency. 751 Standard queuing and transmission selection algorithms allow a 752 central controller to compute the latency contribution of each relay 753 node to the end-to-end latency, to compute the amount of buffer space 754 required in each relay system for each incremental DetNet flow, and 755 most importantly, to translate from a flow specification to a set of 756 values for the managed objects that control each relay or end system. 757 The IEEE 802 has specified (and is specifying) a set of queuing, 758 shaping, and scheduling algorithms that enable each relay system 759 (bridge or router), and/or a central controller, to compute these 760 values. These algorithms include: 762 o A credit-based shaper [IEEE802.1Q-2014] Clause 34. 764 o Time-gated queues governed by a rotating time schedule, 765 synchronized among all relay nodes [IEEE802.1Qbv]. 767 o Synchronized double (or triple) buffers driven by synchronized 768 time ticks. [IEEE802.1Qch]. 770 o Pre-emption of an Ethernet packet in transmission by a packet with 771 a more stringent latency requirement, followed by the resumption 772 of the preempted packet [IEEE802.1Qbu], [IEEE802.3br]. 774 While these techniques are currently embedded in Ethernet and 775 bridging standards, we can note that they are all, except perhaps for 776 packet preemption, equally applicable to other media than Ethernet, 777 and to routers as well as bridges. 779 4.5. Coexistence with normal traffic 781 A DetNet network supports the dedication of a high proportion (e.g. 782 75%) of the network bandwidth to DetNet flows. But, no matter how 783 much is dedicated for DetNet flows, it is a goal of DetNet to coexist 784 with with existing Class of Service schemes (e.g., DiffServ). It is 785 also important that non-DetNet traffic not disrupt the DetNet flow, 786 of course (see Section 4.6 and Section 7). For these reasons: 788 o Bandwidth (transmission opportunities) not utilized by a DetNet 789 flow are available to non-DetNet packets (though not to other 790 DetNet flows). 792 o DetNet flows can be shaped or scheduled, in order to ensure that 793 the highest-priority non-DetNet packet also is ensured a worst- 794 case latency (at any given hop). 796 o When transmission opportunities for DetNet flows are scheduled in 797 detail, then the algorithm constructing the schedule should leave 798 sufficient opportunities for non-DetNet packets to satisfy the 799 needs of the users of the network. Detailed scheduling can also 800 permit the time-shared use of buffer resources by different DetNet 801 flows. 803 Ideally, the net effect of the presence of DetNet flows in a network 804 on the non-DetNet packets is primarily a reduction in the available 805 bandwidth. 807 4.6. Fault Mitigation 809 One key to building robust real-time systems is to reduce the 810 infinite variety of possible failures to a number that can be 811 analyzed with reasonable confidence. DetNet aids in the process by 812 providing filters and policers to detect DetNet packets received on 813 the wrong interface, or at the wrong time, or in too great a volume, 814 and to then take actions such as discarding the offending packet, 815 shutting down the offending DetNet flow, or shutting down the 816 offending interface. 818 It is also essential that filters and service remarking be employed 819 at the network edge to prevent non-DetNet packets from being mistaken 820 for DetNet packets, and thus impinging on the resources allocated to 821 DetNet packets. 823 There exist techniques, at present and/or in various stages of 824 standardization, that can perform these fault mitigation tasks that 825 deliver a high probability that misbehaving systems will have zero 826 impact on well-behaved DetNet flows, except of course, for the 827 receiving interface(s) immediately downstream of the misbehaving 828 device. 830 4.7. Protocol Stack Model 832 [IEEE802.1CB], Annex C, offers a description of the TSN protocol 833 stack. It may serve as the foundation for the DetNet model which 834 will be defined by the working group. While this standard is a work 835 in progress, a consensus around the basic architecture has formed. 836 This stack is summarized in Figure 4. 838 DetNet Protocol Stack 840 +--------------------------------+ 841 | Upper Layers | 842 +--------------------------------+ 843 | Sequence generation/recovery | 844 +--------------------------------+ 845 | DetNet flow splitting/merging | 846 +--------------------------------+ 847 | Sequence encode/decode | 848 +--------------------------------+ 849 | DetNet flow encode/decode | 850 +--------------------------------+ 851 | Lower layers | 852 +--------------------------------+ 854 Figure 4 856 Not all layers are required for any given application, or even for 857 any given network. The layers are, from top to bottom: 859 Sequence generation/recovery 860 Supplies the sequence number for packet replication and 861 deletion (Section 3.3) for packets going down the stack (if 862 not already present), and discards duplicate packets coming 863 up the stack. 865 DetNet flow splitting/merging 866 Replicates packets going down the stack into two DetNet 867 flows, and merges DetNet flows together for packets coming up 868 the stack, based on the packet's DetNet flow identifier. 869 Needed for packet replication and deletion (Section 3.3). 871 Sequence encode/decode 872 Encodes the sequence number into packets going down the 873 stack, and extracts the sequence number from packets coming 874 up the stack. This function may or may not be a null 875 transformation of the packet, and for some protocols, is not 876 explicitly present, being included in the DetNet flow encode/ 877 decode layer, below. 879 DetNet flow encode/decode 880 Encapsulates packets going down the stack, based on the 881 packet's locally-significant DetNet flow identifier, in order 882 to identify to which DetNet flow the packet belongs, and 883 extracts a locally-significant DetNet flow identifier from 884 packets coming up the stack. This may be a null 885 transformation (e.g., for DetNet flows identified by IP 886 5-tuple) or might be an explicit encapsulation (e.g., for 887 DetNet flows identified with an MPLS label). DetNet flow 888 identification is the basis for packet replication and 889 deletion, for assigning per-flow resources (if any) to 890 packets and for defense against misbehaving systems 891 (Section 4.6). When DetNet flows are assigned to pinned 892 paths, this layer can be indistinguishable from the data 893 forwarding layer(s). 895 The reader is likely to notice that Figure 4 does not specify the 896 relationship between the DetNet layers, the IP layers, and the link 897 layers. This is intentional, because they can usefully be placed 898 different places in the stack, and even in mulitple places, depending 899 on where their peers are placed. 901 4.8. Advertising resources, capabilities and adjacencies 903 There are three classes of information that a central controller or 904 decentralized control plane needs to know that can only be obtained 905 from the end systems and/or relay systems in the network. When using 906 a peer-to-peer control plane, some of this information may be 907 required by a system's neighbors in the network. 909 o Details of the system's capabilities that are required in order to 910 accurately allocate that system's resources, as well as other 911 systems' resources. This includes, for example, which specific 912 queuing and shaping algorithms are implemented (Section 4.4), the 913 number of buffers dedicated for DetNet allocation, and the worst- 914 case forwarding delay. 916 o The dynamic state of an end or relay system's DetNet resources. 918 o The identity of the system's neighbors, and the characteristics of 919 the link(s) between the systems, including the length (in 920 nanoseconds) of the link(s). 922 4.9. Provisioning model 924 4.9.1. Centralized Path Computation and Installation 926 A centralized routing model, such as provided with a PCE (RFC 4655 927 [RFC4655]), enables global and per-flow optimizations. (See 928 Section 4.2.) The model is attractive but a number of issues are 929 left to be solved. In particular: 931 o Whether and how the path computation can be installed by 1) an end 932 device or 2) a Network Management entity, 934 o And how the path is set up, either by installing state at each hop 935 with a direct interaction between the forwarding device and the 936 PCE, or along a path by injecting a source-routed request at one 937 end of the path. 939 4.9.2. Distributed Path Setup 941 Whether a distributed alternative without a PCE can be valuable 942 should be studied as well. Such an alternative could for instance 943 inherit from the Resource ReSerVation Protocol [RFC3209] (RSVP-TE) 944 flows. 946 In a Layer-2 only environment, or as part of a layered approach to a 947 mixed environment, IEEE 802.1 also has work, either completed or in 948 progress. [IEEE802.1Q-2014] Clause 35 describes SRP, a peer-to-peer 949 protocol for Layer-2 roughly analogous to RSVP. Almost complete is 950 [IEEE802.1Qca], which defines how ISIS can provide multiple disjoint 951 paths or distribution trees. Also in progress is [IEEE802.1Qcc], 952 which expands the capabilities of SRP. 954 The integration/interaction of the DetNet control layer an underlying 955 IEEE 802.1 sub-network control layer will need to be defined. 957 4.10. Scaling to larger networks 959 Reservations for individual DetNet flows require considerable state 960 information in each relay system, especially when adequate fault 961 mitigation (Section 4.6) is required. The DetNet data plane, in 962 order to support larger numbers of DetNet flows, must support the 963 aggregation of DetNet flows into tunnels, which themselves can be 964 viewed by the relay systems' data planes largely as individual DetNet 965 flows. Without such aggregation, the per-relay system may limit the 966 scale of DetNet networks. 968 4.11. Connected islands vs. networks 970 Given that users have deployed examples of the IEEE 802.1 TSN TG 971 standards, which provide capabilities similar to DetNet, it is 972 obvious to ask whether the IETF DetNet effort can be limited to 973 providing Layer-2 connections (VPNs) between islands of bridged TSN 974 networks. While this capability is certainly useful to some 975 applications, and must not be precluded by DetNet, tunneling alone is 976 not a sufficient goal for the DetNet WG. As shown in the 977 Deterministic Networking Use Cases draft [I-D.ietf-detnet-use-cases], 978 there are already deployments of Layer-2 TSN networks that are 979 encountering the well-known problems of over-large broadcast domains. 980 Routed solutions, and combinations routed/bridged solutions, are both 981 required. 983 5. Compatibility with Layer-2 985 Standards providing similar capabilities for bridged networks (only) 986 have been and are being generated in the IEEE 802 LAN/MAN Standards 987 Committee. The present architecture describes an abstract model that 988 can be applicable both at Layer-2 and Layer-3, and over links not 989 defined by IEEE 802. It is the intention of the authors (and 990 hopefully, as this draft progresses, of the DetNet Working Group) 991 that IETF and IEEE 802 will coordinate their work, via the 992 participation of common individuals, liaisons, and other means, to 993 maximize the compatibility of their outputs. 995 DetNet enabled systems and nodes can be interconnected by sub- 996 networks, i.e., Layer 2 technologies. These sub-networks will 997 provide DetNet compatible service for support of DetNet traffic. 998 Examples of sub-networks include 802.1TSN and a point-to-point OTN 999 link. Of course, multi-layer DetNet systems may be possible too, 1000 where one DetNet appears as a sub-network, and provides service to, a 1001 higher layer DetNet system. 1003 6. Open Questions 1005 There are a number of architectural questions that will have to be 1006 resolved before this document can be submitted for publication. 1007 Aside from the obvious fact that this present draft is subject to 1008 change, there are specific questions to which the authors wish to 1009 direct the readers' attention. 1011 6.1. Data plane shapers and schedulers 1013 A number of techniques have been defined and are being defined by 1014 IEEE 802 for queuing, shaping, and scheduling transmissions on 1015 EtherNet media, most of which are directly applicable to any other 1016 medium. Specific selections of supported techniques are required, 1017 because minimizing, and even eliminating, congestion losses depends 1018 strongly on the details of the per-hop behavior of sources and relay 1019 systems. 1021 The present authors expect that, at least, the IEEE 802 mechanisms 1022 will be supported. 1024 6.2. DetNet flow identification and sequencing 1026 The techniques to be used for DetNet flow identification must be 1027 settled. The following paragraphs provide a snapshot of the authors' 1028 opinions at the time of writing. These authors anticipate the 1029 submission of drafts in the near future on this subject. 1031 IEEE 802.1 TSN streams are identified by giving each stream (DetNet 1032 flow) a {VLAN identifier, destination MAC address} pair that is 1033 unique in the bridged network, and that the MAC address must be a 1034 multicast address. If a source is generating, for example, two 1035 unicast UDP flows to the same destination, one DetNet and one not, 1036 the DetNet flow's packets must be transformed at some point to have a 1037 multicast destination MAC address, and perhaps, a different VLAN than 1038 the non-DetNet flow's packets. 1040 A similar provision would apply to DetNet packets that are identified 1041 by MPLS labels; any bridges between the LSRs need a {VLAN identifier, 1042 destination MAC address} pair uniquely identifying the DetNet flow in 1043 the bridged network. 1045 Provision is made in current draft of [IEEE802.1CB] to make these 1046 transformations either in a Layer-2 shim in the source end system, on 1047 the output side of a router or LSR, or in a proxy function in the 1048 first-hop bridge. It remains to be seen whether this provision is 1049 adequate and/or acceptable to the IETF DetNet WG. 1051 There are also questions regarding the sequentialization of packets 1052 for use with packet replication and deletion (Section 3.3). 1053 [IEEE802.1CB] defines an EtherNet tag carrying a sequence number. If 1054 MPLS Pseudowires are used with a control word containing a sequence 1055 number, the relationship and interworking between these two formats 1056 must be defined. 1058 6.3. Flat vs. hierarchical control 1060 Boxes that are solely routers or solely bridges are rare in today's 1061 market. In a multi-tenant data center, multiple users' virtual 1062 Layer-2/Layer-3 topologies exist simultaneously, implemented on a 1063 network whose physical topology bears only accidental resemblance to 1064 the virtual topologies. 1066 While the forwarding topology (the bridges and routers) are an 1067 important consideration for a DetNet Flow Management Entity 1068 (Section 4.2.1), so is the purely physical topology. Ultimately, the 1069 model used by the management entities is based on boxes, queues, and 1070 links. The authors hope that the work of the TEAS WG will help to 1071 clarify exactly what model parameters need to be traded between the 1072 relay systems and the controller(s). 1074 6.4. Peer-to-peer reservation protocol 1076 As described in Section 4.9.2, the DetNet WG needs to decide whether 1077 to support a peer-to-peer protocol for a source and a destination to 1078 reserve resources for a DetNet stream. Assuming that enabling the 1079 involvement of the source and/or destination is desirable (see 1080 Deterministic Networking Use Cases [I-D.ietf-detnet-use-cases]), it 1081 remains to decide whether the DetNet WG will make it possible to 1082 deploy at least some DetNet capabilities in a network using only a 1083 peer-to-peer protocol, without a central controller. 1085 (Note that a UNI (see Section 4.2.3) between an end system and an 1086 edge relay system, for sources and/or listeners to request DetNet 1087 services, can be either the first hop of a per-to-peer reservation 1088 protocol, or can be deflected by the edge relay system to a central 1089 controller for resolution. Similarly, a decision by a central 1090 controller can be effected by the controller instructing the end 1091 system or edge relay system to initiate a per-to-peer protocol 1092 activity.) 1094 6.5. Wireless media interactions 1096 Deterministic Networking Use Cases [I-D.ietf-detnet-use-cases] 1097 illustrates cases where wireless media are needed in a DetNet 1098 network. Some wireless media in general use, such as IEEE 802.11 1099 [IEEE802.1Q-2014], have significantly higher packet loss rates than 1100 typical wired media, such as Ethernet [IEEE802.3-2012]. IEEE 802.11 1101 includes support for such features as MAC-layer acknowledgements and 1102 retransmissions. 1104 The techniques described in Section 3 are likely to improve the 1105 ability of a mixed wired/wireless network to offer the DetNet QoS 1106 features. The interaction of these techniques with the features of 1107 specific wireless media, although they may be significant, cannot be 1108 addressed in this document. It remains to be decided to what extent 1109 the DetNet WG will address them, and to what extent other WGs, e.g. 1110 6TiSCH, will do so. 1112 7. Security Considerations 1114 Security in the context of Deterministic Networking has an added 1115 dimension; the time of delivery of a packet can be just as important 1116 as the contents of the packet, itself. A man-in-the-middle attack, 1117 for example, can impose, and then systematically adjust, additional 1118 delays into a link, and thus disrupt or subvert a real-time 1119 application without having to crack any encryption methods employed. 1120 See [RFC7384] for an exploration of this issue in a related context. 1122 Furthermore, in a control system where millions of dollars of 1123 equipment, or even human lives, can be lost if the DetNet QoS is not 1124 delivered, one must consider not only simple equipment failures, 1125 where the box or wire instantly becomes perfectly silent, but bizarre 1126 errors such as can be caused by software failures. Because there is 1127 essential no limit to the kinds of failures that can occur, 1128 protecting against realistic equipment failures is indistinguishable, 1129 in most cases, from protecting against malicious behavior, whether 1130 accidental or intentional. See also Section 4.6. 1132 Security must cover: 1134 o the protection of the signaling protocol 1136 o the authentication and authorization of the controlling systems 1138 o the identification and shaping of the DetNet flows 1140 8. IANA Considerations 1142 This document does not require an action from IANA. 1144 9. Acknowledgements 1146 The authors wish to thank Jouni Korhonen, Erik Nordmark, George 1147 Swallow, Rudy Klecka, Anca Zamfir, David Black, Thomas Watteyne, 1148 Shitanshu Shah, Craig Gunther, Rodney Cummings, Wilfried Steiner, 1149 Marcel Kiessling, Karl Weber, Ethan Grossman, Pat Thaler, and Lou 1150 Berger for their various contribution with this work. 1152 10. Access to IEEE 802.1 documents 1154 To access password protected IEEE 802.1 drafts, see the IETF IEEE 1155 802.1 information page at https://www.ietf.org/proceedings/52/slides/ 1156 bridge-0/tsld003.htm. 1158 11. Informative References 1160 [AVnu] http://www.avnu.org/, "The AVnu Alliance tests and 1161 certifies devices for interoperability, providing a simple 1162 and reliable networking solution for AV network 1163 implementation based on the Audio Video Bridging (AVB) 1164 standards.". 1166 [CCAMP] IETF, "Common Control and Measurement Plane", 1167 . 1169 [HART] www.hartcomm.org, "Highway Addressable Remote Transducer, 1170 a group of specifications for industrial process and 1171 control devices administered by the HART Foundation". 1173 [HSR-PRP] IEC, "High availability seamless redundancy (HSR) is a 1174 further development of the PRP approach, although HSR 1175 functions primarily as a protocol for creating media 1176 redundancy while PRP, as described in the previous 1177 section, creates network redundancy. PRP and HSR are both 1178 described in the IEC 62439 3 standard.", 1179 . 1182 [I-D.finn-detnet-problem-statement] 1183 Finn, N. and P. Thubert, "Deterministic Networking Problem 1184 Statement", draft-finn-detnet-problem-statement-05 (work 1185 in progress), March 2016. 1187 [I-D.ietf-6tisch-architecture] 1188 Thubert, P., "An Architecture for IPv6 over the TSCH mode 1189 of IEEE 802.15.4", draft-ietf-6tisch-architecture-09 (work 1190 in progress), November 2015. 1192 [I-D.ietf-6tisch-tsch] 1193 Watteyne, T., Palattella, M., and L. Grieco, "Using 1194 IEEE802.15.4e TSCH in an IoT context: Overview, Problem 1195 Statement and Goals", draft-ietf-6tisch-tsch-06 (work in 1196 progress), March 2015. 1198 [I-D.ietf-detnet-use-cases] 1199 Grossman, E., Gunther, C., Thubert, P., Wetterwald, P., 1200 Raymond, J., Korhonen, J., Kaneko, Y., Das, S., Zha, Y., 1201 Varga, B., Farkas, J., Goetz, F., and J. Schmitt, 1202 "Deterministic Networking Use Cases", draft-ietf-detnet- 1203 use-cases-08 (work in progress), March 2016. 1205 [I-D.ietf-roll-rpl-industrial-applicability] 1206 Phinney, T., Thubert, P., and R. Assimiti, "RPL 1207 applicability in industrial networks", draft-ietf-roll- 1208 rpl-industrial-applicability-02 (work in progress), 1209 October 2013. 1211 [I-D.svshah-tsvwg-deterministic-forwarding] 1212 Shah, S. and P. Thubert, "Deterministic Forwarding PHB", 1213 draft-svshah-tsvwg-deterministic-forwarding-04 (work in 1214 progress), August 2015. 1216 [IEEE802.11-2012] 1217 IEEE, "Wireless LAN Medium Access Control (MAC) and 1218 Physical Layer (PHY) Specifications", 2012, 1219 . 1222 [IEEE802.1AS-2011] 1223 IEEE, "Timing and Synchronizations (IEEE 802.1AS-2011)", 1224 2011, . 1227 [IEEE802.1BA-2011] 1228 IEEE, "AVB Systems (IEEE 802.1BA-2011)", 2011, 1229 . 1232 [IEEE802.1CB] 1233 IEEE, "Frame Replication and Elimination for Reliability 1234 (IEEE Draft P802.1CB)", 2016, 1235 . 1237 [IEEE802.1Q-2014] 1238 IEEE, "MAC Bridges and VLANs (IEEE 802.1Q-2014", 2014, 1239 . 1242 [IEEE802.1Qbu] 1243 IEEE, "Frame Preemption", 2016, 1244 . 1246 [IEEE802.1Qbv] 1247 IEEE, "Enhancements for Scheduled Traffic", 2016, 1248 . 1250 [IEEE802.1Qca] 1251 IEEE, "Path Control and Reservation", 2015, 1252 . 1254 [IEEE802.1Qcc] 1255 IEEE, "Stream Reservation Protocol (SRP) Enhancements and 1256 Performance Improvements", 2016, 1257 . 1259 [IEEE802.1Qch] 1260 IEEE, "Cyclic Queuing and Forwarding", 2016, 1261 . 1263 [IEEE802.1TSNTG] 1264 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1265 Networks Task Group", 2013, 1266 . 1268 [IEEE802.3-2012] 1269 IEEE, "IEEE Stabdard for Ethernet", 2012, 1270 . 1273 [IEEE802.3br] 1274 IEEE, "Interspersed Express Traffic", 2016, 1275 . 1277 [IEEE802154] 1278 IEEE standard for Information Technology, "IEEE std. 1279 802.15.4, Part. 15.4: Wireless Medium Access Control (MAC) 1280 and Physical Layer (PHY) Specifications for Low-Rate 1281 Wireless Personal Area Networks", June 2011. 1283 [IEEE802154e] 1284 IEEE standard for Information Technology, "IEEE std. 1285 802.15.4e, Part. 15.4: Low-Rate Wireless Personal Area 1286 Networks (LR-WPANs) Amendment 1: MAC sublayer", April 1287 2012. 1289 [ISA100.11a] 1290 ISA/IEC, "ISA100.11a, Wireless Systems for Automation, 1291 also IEC 62734", 2011, < http://www.isa100wci.org/en- 1292 US/Documents/PDF/3405-ISA100-WirelessSystems-Future-broch- 1293 WEB-ETSI.aspx>. 1295 [ISA95] ANSI/ISA, "Enterprise-Control System Integration Part 1: 1296 Models and Terminology", 2000, . 1299 [ODVA] http://www.odva.org/, "The organization that supports 1300 network technologies built on the Common Industrial 1301 Protocol (CIP) including EtherNet/IP.". 1303 [PCE] IETF, "Path Computation Element", 1304 . 1306 [Profinet] 1307 http://us.profinet.com/technology/profinet/, "PROFINET is 1308 a standard for industrial networking in automation.", 1309 . 1311 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1312 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1313 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1314 September 1997, . 1316 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1317 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1318 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1319 . 1321 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 1322 Element (PCE)-Based Architecture", RFC 4655, 1323 DOI 10.17487/RFC4655, August 2006, 1324 . 1326 [RFC5673] Pister, K., Ed., Thubert, P., Ed., Dwars, S., and T. 1327 Phinney, "Industrial Routing Requirements in Low-Power and 1328 Lossy Networks", RFC 5673, DOI 10.17487/RFC5673, October 1329 2009, . 1331 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 1332 L., and L. Berger, "A Framework for MPLS in Transport 1333 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 1334 . 1336 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 1337 Profile (MPLS-TP) Survivability Framework", RFC 6372, 1338 DOI 10.17487/RFC6372, September 2011, 1339 . 1341 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1342 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1343 October 2014, . 1345 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1346 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1347 Defined Networking (SDN): Layers and Architecture 1348 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1349 2015, . 1351 [TEAS] IETF, "Traffic Engineering Architecture and Signaling", 1352 . 1354 [WirelessHART] 1355 www.hartcomm.org, "Industrial Communication Networks - 1356 Wireless Communication Network and Communication Profiles 1357 - WirelessHART - IEC 62591", 2010. 1359 Authors' Addresses 1360 Norman Finn 1361 Cisco Systems 1362 170 W Tasman Dr. 1363 San Jose, California 95134 1364 USA 1366 Phone: +1 408 526 4495 1367 Email: nfinn@cisco.com 1369 Pascal Thubert 1370 Cisco Systems 1371 Village d'Entreprises Green Side 1372 400, Avenue de Roumanille 1373 Batiment T3 1374 Biot - Sophia Antipolis 06410 1375 FRANCE 1377 Phone: +33 4 97 23 26 34 1378 Email: pthubert@cisco.com 1380 Michael Johas Teener 1381 Broadcom Corp. 1382 3151 Zanker Rd. 1383 San Jose, California 95134 1384 USA 1386 Phone: +1 831 824 4228 1387 Email: MikeJT@broadcom.com