idnits 2.17.1 draft-finn-detnet-architecture-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 9, 2015) is 3335 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'AVnu' is defined on line 858, but no explicit reference was found in the text == Unused Reference: 'HART' is defined on line 867, but no explicit reference was found in the text == Unused Reference: 'I-D.finn-detnet-problem-statement' is defined on line 880, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1AS-2011' is defined on line 908, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1Qat-2010' is defined on line 928, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1Qav' is defined on line 933, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1TSNTG' is defined on line 949, but no explicit reference was found in the text == Unused Reference: 'IEEE802154' is defined on line 954, but no explicit reference was found in the text == Unused Reference: 'ODVA' is defined on line 972, but no explicit reference was found in the text == Unused Reference: 'Profinet' is defined on line 979, but no explicit reference was found in the text == Unused Reference: 'RFC2205' is defined on line 984, but no explicit reference was found in the text == Outdated reference: A later version (-05) exists of draft-finn-detnet-problem-statement-01 == Outdated reference: A later version (-30) exists of draft-ietf-6tisch-architecture-05 == Outdated reference: A later version (-06) exists of draft-ietf-6tisch-tsch-05 == Outdated reference: A later version (-04) exists of draft-svshah-tsvwg-deterministic-forwarding-03 Summary: 0 errors (**), 0 flaws (~~), 16 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft P. Thubert 4 Intended status: Standards Track Cisco 5 Expires: September 10, 2015 March 9, 2015 7 Deterministic Networking Architecture 8 draft-finn-detnet-architecture-00 10 Abstract 12 Deterministic Networking (DetNet) provides a capability to carry 13 specified unicast or multicast data streams for real-time 14 applications with extremely low data loss rates and maximum latency. 15 Techniques used include: 1) reserving data plane resources for 16 individual (or aggregated) DetNet streams in some or all of the relay 17 systems (bridges or routers) along the path of the stream; 2) 18 providing fixed paths for DetNet streams that do not rapidly change 19 with the network topology; and 3) sequentializing, replicating, and 20 eliminating duplicate packets at various points to ensure the 21 availability of at least one path. The capabilities can be managed 22 by configuration, or by manual or automatic network management. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on September 10, 2015. 41 Copyright Notice 43 Copyright (c) 2015 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 59 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 60 3. Providing the DetNet Quality of Service . . . . . . . . . . . 5 61 3.1. Zero Congestion Loss . . . . . . . . . . . . . . . . . . 6 62 3.2. Pinned-down paths . . . . . . . . . . . . . . . . . . . . 7 63 3.3. Seamless Redundancy . . . . . . . . . . . . . . . . . . . 7 64 4. DetNet Architecture . . . . . . . . . . . . . . . . . . . . . 8 65 4.1. The Application Plane . . . . . . . . . . . . . . . . . . 11 66 4.2. The Controller Plane . . . . . . . . . . . . . . . . . . 11 67 4.3. The Network Plane . . . . . . . . . . . . . . . . . . . . 12 68 4.4. Elements of DetNet Architecture . . . . . . . . . . . . . 13 69 4.5. DetNet streams . . . . . . . . . . . . . . . . . . . . . 14 70 4.5.1. Talker guarantees . . . . . . . . . . . . . . . . . . 14 71 4.5.2. Incomplete Networks . . . . . . . . . . . . . . . . . 15 72 4.6. Data Flow Model through Systems . . . . . . . . . . . . . 16 73 4.7. Queuing, Shaping, Scheduling, and Preemption . . . . . . 16 74 4.8. Coexistence with normal traffic . . . . . . . . . . . . . 16 75 4.9. Fault Mitigation . . . . . . . . . . . . . . . . . . . . 16 76 4.10. Protocol Stack Model . . . . . . . . . . . . . . . . . . 17 77 4.11. Advertising resources, capabilities and adjacencies . . . 17 78 4.12. Provisioning model . . . . . . . . . . . . . . . . . . . 17 79 4.12.1. Centralized Path Computation and Installation . . . 17 80 4.12.2. Distributed Path Setup . . . . . . . . . . . . . . . 17 81 5. Related IETF work . . . . . . . . . . . . . . . . . . . . . . 18 82 5.1. Deterministic PHB . . . . . . . . . . . . . . . . . . . . 18 83 5.2. 6TiSCH . . . . . . . . . . . . . . . . . . . . . . . . . 18 84 6. Security Considerations . . . . . . . . . . . . . . . . . . . 19 85 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 86 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 87 9. Informative References . . . . . . . . . . . . . . . . . . . 19 88 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 90 1. Introduction 92 Operational Technology (OT) refers to industrial networks that are 93 typically used for monitoring systems and supporting control loops, 94 as well as movement detection systems for use in process control 95 (i.e., process manufacturing) and factory automation (i.e., discrete 96 manufacturing). Due to its different goals, OT has evolved in 97 parallel but in a manner that is radically different from IT/ICT, 98 focusing on highly secure, reliable and deterministic networks, with 99 limited scalability over a bounded area. 101 The convergence of IT and OT technologies, also called the Industrial 102 Internet, represents a major evolution for both sides. The work has 103 already started; in particular, the industrial automation space has 104 been developing a number of Ethernet-based replacements for existing 105 digital control systems, often not packet-based (fieldbus 106 technologies). 108 These replacements are meant to provide similar behavior as the 109 incumbent protocols, and their common focus is to transport a fully 110 characterized flow over a well-controlled environment (i.e., a 111 factory floor), with a bounded latency, extraordinarily low frame 112 loss, and a very narrow jitter. Examples of such protocols include 113 PROFINET, ODVA Ethernet/IP, and EtherCAT. 115 In parallel, the need for determinism in professional and home audio/ 116 video markets drove the formation of the Audio/Video Bridging (AVB) 117 standards effort of IEEE 802.1. With the explosion of demand for 118 connectivity and multimedia in transportation in general, the 119 Ethernet AVB technology has become one of the hottest topics, in 120 particular in the automotive connectivity. It is finding application 121 in all elements of the vehicle from head units, to rear seat 122 entertainment modules, to amplifiers and camera modules. While aimed 123 at less critical applications than some industrial networks, AVB 124 networks share the requirement for extremely low packet loss rates 125 and ensured finite latency and jitter. 127 Other instances of in-vehicle deterministic networks have arisen as 128 well for control networks in cars, trains and buses, as well as 129 avionics, with, for instance, the mission-critical "Avionics Full- 130 Duplex Switched Ethernet" (AFDX) that was designed as part of the 131 ARINC 664 standards. Existing automotive control networks such as 132 the LIN, CAN and FlexRay standards were not designed to cover these 133 increasing demands in terms of bandwidth and scalability that we see 134 with various kinds of Driver Assistance Systems (DAS) and new 135 multiplexing technologies based on Ethernet are now getting traction. 137 The generalization of the needs for more deterministic networks have 138 led to the IEEE 802.1 AVB Task Group becoming the Time-Sensitive 139 Networking (TSN) Task Group (TG), with a much-expanded constituency 140 from the industrial and vehicular markets. Along with this 141 expansion, the networks in consideration are becoming larger and 142 structured, requiring deterministic forwarding beyond the LAN 143 boundaries. For instance, Industrial Automation segregates the 144 network along the broad lines of the Purdue Enterprise Reference 145 Architecture (PERA), using different technologies at each level, and 146 public infrastructures such as Electricity Automation require 147 deterministic properties over the Wide Area. The realization is now 148 coming that the convergence of IT and OT networks requires Layer-3, 149 as well as Layer-2, capabilities. 151 The present architecture is the result of a collaboration of the IETF 152 and the IEEE and implements an abstract model that can be applicable 153 both at Layer-2 and Layer-3, and along segments of different 154 technologie. With this new work, a path may span, for instance, 155 across a (limited) number of 802.1 bridges and then a (limited) 156 number of IP routers. In that example, the IEEE 802.1 bridges may be 157 operating at Layer-2 over Ethernet whereas the IP routers may be 158 6TiSCH nodes operating at Layer-2 and/or Layer-3 over the IEEE 159 802.15.4e MAC. 161 Many applications of interest to Deterministic Networking require the 162 ability to synchronize the clocks in end systems to a sub-microsecond 163 accuracy. Some of the queue control techniques defined in 164 Section 4.7 also require time synchronization among relay systems. 165 The means used to achieve time synchronization are not addressed in 166 this document. 168 2. Terminology 170 The follwing special terms are used in this document in order to 171 avoid the assumption that a given element in the archetecture does or 172 does not have Internet Protocol stack, functions as a router or a 173 bridge, or otherwise plays a particular role at Layer-3 or higher: 175 bridge 176 A Customer Bridge as defined by IEEE 802.1Q 177 [IEEE802.1Q-2011]. 179 circuit 180 A trail of configuration from talker to listener(s) through 181 relay systems associated with a DetNet stream, required to 182 deliver the benefits of DetNet. 184 end system 185 Commonly called a "host" in IETF documents, and an "end 186 station" is IEEE 802 documents. End systems of interest to 187 this document are talkers and listeners. 189 listener 190 An end system capable of sinking a DetNet stream. 192 relay system 193 A router or a bridge. 195 stream 196 A DetNet stream is a sequence of packets from a single 197 talker, through some number of relay systems to one or more 198 listeners, that is limited by the talker in its maximum 199 packet size and transmission rate, and can thus be ensured 200 the DetNet Quality of Service (QoS) from the network. 202 talker 203 An end system capable of sourcing a DetNet stream. 205 3. Providing the DetNet Quality of Service 207 DetNet Quality of Service is expressed in terms of: 209 o Minimum and maximum end-to-end latency from talker to listener; 211 o Probability of loss of a packet, assuming the normal operation of 212 the relay systems and links; 214 o Probability of loss of a packet in the event of the failure of a 215 relay system or link. 217 It is a distinction of DetNet that it is concerned solely with worst- 218 case values for all of the above parameters. Average, mean, or 219 typical values are of no interest, because they do not affect the 220 ability of a real-time system to perform its tasks. 222 Three techniques are employed by DetNet to achieve these QoS 223 parameters: 225 a. Zero congestion loss (Section 3.1). Network resources such as 226 link bandwidth, buffers, queues, shapers, and scheduled input/ 227 output slots are assigned in each relay system to the use of a 228 specific DetNet stream or group of streams. Note that, given a 229 finite amount of buffer space), zero congestion loss necessarily 230 ensures a maximum end-to-end latency. Depending on the method 231 employed, a minimum latency can also be achieved. 233 b. Pinned-down paths (Section 3.2). Point-to-point paths or point- 234 to-multipoint trees through the network from a talker to one or 235 more listeners can be established, and DetNet streams assigned to 236 follow a particular path or tree. 238 c. Packet replication and deletion (Section 3.3). End systems and/ 239 or relay systems can sequence number, replicate, and eliminate 240 replicated packets at multiple points in the network in order to 241 ensure that one (or more) equipment failure events still leave at 242 least one path intact for a DetNet stream. 244 These three techniques can be applied independently, giving eight 245 possible combinations, including none (no DetNet), although some 246 combinations are of wider utility than others. This separation keeps 247 the protocol stack coherent and maximizes interoperability with 248 existing and developing standards in this (IETF) and other Standards 249 Development Organizations. Some examples of typical expected 250 combinations: 252 o Pinned-down paths (a) plus packet replication (b) are exactly the 253 techniques employed by [HSR-PRP]. Pinned-down paths are achieve 254 by limiting the physical topology of the network, and the 255 sequentialization, replication, and duplicate elimination 256 facilitated by packet tags added at the front or the end of 257 Ethernet frames. 259 o Zero congestion loss (a) alone is is offered by IEEE 802.1 Audio 260 Video bridging [IEEE802.1BA-2011]. As long as the network suffers 261 no failures, near-zero (at best, zero) congestion loss can be 262 achieved through the use of a reservation protocol (MSRP) and 263 shapers in every relay system (bridge). 265 o Using all three together gives maximum protection. 267 There are, of course, simpler methods available (and employed, today) 268 to achieve levels of latency and packet loss that are satisfactory 269 for many applications. However, these methods generally work best in 270 the absence of any significant amount of non-critical traffic in the 271 network (if, indeed, such traffic is supported at all), or work only 272 if the critical traffic constitutes only a small portion of the 273 network's theoretical capacity, or work only if all systems are 274 functioning properly, or in the absence of actions by end systems 275 that disrupt the network's operations. 277 There are any number of methods in use, defined, or in progress for 278 accomplishing each of the above techniques. It is expected that this 279 DetNet Architecture will assist various vendors, users, and/or 280 "vertical" Standards Development Organizations (dedicated to a single 281 industry) to make selections among the available means of 282 implementing DetNet networks. 284 3.1. Zero Congestion Loss 286 The primary means by which DetNet achieves its QoS assurances is to 287 completely eliminate congestion at an output port as a cause of 288 packet loss. Given that a DetNet stream cannot be throttled, this 289 can be achieved only by the provision of sufficient buffer storage at 290 each hop through the network to ensure that no packets are dropped 291 due to a lack of buffer storage. 293 Ensuring adequate buffering requires, in turn, that the talker, and 294 every relay system along the path to the listener (or nearly every 295 relay system -- see Section 4.5.2) be careful to regulate its output 296 to not exceed the data rate for any stream, except for brief perios 297 when making up for interfering traffic. Any packet sent ahead of its 298 time potentially adds to the number of buffers required by the next 299 hop, and may thus exceed the resources allocated for a particular 300 stream. 302 The low-level mechanisms described in Section 4.7 provide the 303 necessary regulation of transmissions by an edge system or relay 304 system to ensure zero congestion loss. Of course, the reservation of 305 the bandwidth and buffers for a stream requires the provisioning 306 described in Section 4.12. 308 3.2. Pinned-down paths 310 In networks controlled by typical peer-to-peer protocols such as IEEE 311 802.1 ISIS bridged networks or ETF OSPF routed networks, a network 312 topology event in one part of the network can impact, at least 313 briefly, the delivery of data in parts of the network remote from the 314 failure or recovery event. Thus, even redundant paths through a 315 network, if controlled by the typical peer-to-peer protocols, do not 316 eliminate the chances of brief losses of contact. For this reason, 317 many real-time networks rely on physical rings of two-port devices, 318 with a relatively simple ring control protocol. This both minimizes 319 recovery time and easily supports redundant paths. Of course, this 320 comes at the cost of increased hop count, and thus latency, for the 321 typical path. 323 In order to get the advantages of low hop count and still ensure 324 against even brief losses of connectivity, DetNet employs pinned-down 325 paths, where the path taken by a given DetNet stream does not change, 326 at least immediately, and likely not at all, in response to network 327 topology events. When combined with seamless redundancy 328 (Section 3.3), this results in a high likelihood of continuous 329 connectivity. 331 3.3. Seamless Redundancy 333 After congestion loss has been eliminated, the most important causes 334 of packet loss are random media and/or memory faults and equipment 335 failures. 337 Seamless redundancy involves three capabilities: 339 o Adding sequence numbers to the packets of a DetNet stream. 341 o Replicating these packets and, typically, sending them along at 342 least two different paths to the listener(s). 344 o Discarding duplicated packets. 346 In the simplest case, this amounts to replicating each packet in a 347 talker that has two interfaces, and conveying them through the 348 network, along separate paths, to the similarly dual-homed listeners, 349 that discard the extras. This ensures that one path (with zero 350 congestion loss) remains, even if some relay system fails. 352 Alternatively, relay systems in the network can provide replication 353 and elimination facilities at various points in the network, so that 354 multiple failures can be accommodated. 356 This is shown in the following figure, where the two relay systems 357 each replicate (R) the DetNet stream on input, sending the stream to 358 both the other relay system and to the end system, and eliminated 359 duplicates (E) on the output interface to the right-hand end system. 360 Any one links in the network can fail, and the Detnet stream can 361 still get through. Furthermore, two links can fail, as long as they 362 are in different segments of the network. 364 > > > > > > > > relay > > > > > > > > 365 > /------------+ R system E +------------\ > 366 > / v + ^ \ > 367 end R + v | ^ + E end 368 system + v | ^ + system 369 > \ v + ^ / > 370 > \------------+ R relay E +------------/ > 371 > > > > > > > > system > > > > > > > > 373 Figure 1 375 4. DetNet Architecture 377 Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines 378 traffic-engineering architectures for generic applicability across 379 packet and non-packet networks. From TEAS perspective, Traffic 380 Engineering (TE) refers to techniques that enable operators to 381 control how specific traffic flows are treated within their networks. 383 Because if its very nature of establishing pinned-down optimized 384 paths, Deterministic Networking can be seen as a new, specialized 385 branch of Traffic Engineering, and inherits its architecture with a 386 separation into planes. 388 The Deterministic Networking architecture is thus composed of three 389 planes, a (User) Application Plane, a Controller Plane, and a Network 390 Plane, which echoes that of Software-Defined Networking (SDN): Layers 391 and Architecture Terminology [RFC7426] which is represented below: 393 SDN Layers and Architecture Terminology per RFC 7426 395 o--------------------------------o 396 | | 397 | +-------------+ +----------+ | 398 | | Application | | Service | | 399 | +-------------+ +----------+ | 400 | Application Plane | 401 o---------------Y----------------o 402 | 403 *-----------------------------Y---------------------------------* 404 | Network Services Abstraction Layer (NSAL) | 405 *------Y------------------------------------------------Y-------* 406 | | 407 | Service Interface | 408 | | 409 o------Y------------------o o---------------------Y------o 410 | | Control Plane | | Management Plane | | 411 | +----Y----+ +-----+ | | +-----+ +----Y----+ | 412 | | Service | | App | | | | App | | Service | | 413 | +----Y----+ +--Y--+ | | +--Y--+ +----Y----+ | 414 | | | | | | | | 415 | *----Y-----------Y----* | | *---Y---------------Y----* | 416 | | Control Abstraction | | | | Management Abstraction | | 417 | | Layer (CAL) | | | | Layer (MAL) | | 418 | *----------Y----------* | | *----------Y-------------* | 419 | | | | | | 420 o------------|------------o o------------|---------------o 421 | | 422 | CP | MP 423 | Southbound | Southbound 424 | Interface | Interface 425 | | 426 *------------Y---------------------------------Y----------------* 427 | Device and resource Abstraction Layer (DAL) | 428 *------------Y---------------------------------Y----------------* 429 | | | | 430 | o-------Y----------o +-----+ o--------Y----------o | 431 | | Forwarding Plane | | App | | Operational Plane | | 432 | o------------------o +-----+ o-------------------o | 433 | Network Device | 434 +---------------------------------------------------------------+ 436 Figure 2 438 4.1. The Application Plane 440 Per [RFC7426], the Application Plane includes both applications and 441 services. In particular, the Application Plane incorporates the User 442 Agent, a specialized application that interacts with the end user / 443 operator and performs requests for Deterministic Networking services 444 via an abstract Stream Management Entity, (SME) which may or may not 445 be collocated with (one of) the end systems. 447 At the Application Plane, a management interface enables the 448 negotiation of streams between end systems. An abstraction of the 449 stream called a Traffic Specification (TSpec) provides the 450 representation. This abstraction is used to place a reservation over 451 the (Northbound) Service Interface and within the Application plane. 452 It is associated with an abstraction of location, such as IP 453 addresses and DNS names, to identify the end systems and eventually 454 specify intermediate relay systems. 456 4.2. The Controller Plane 458 The Controller Plane corresponds to the aggregation of the Control 459 and Management Planes in [RFC7426], though Common Control and 460 Measurement Plane (CCAMP) [CCAMP] makes an additional distinction 461 between management and measurement. When the logical separation of 462 the Control, Measurement and other Management entities is not 463 relevant, the term Controller Plane is used for simplicity to 464 represent them all, and the term controller refers to any device 465 operating in that plane, whether is it a Path Computation entity or a 466 Network Management entity (NME). The Path Computation Element (PCE) 467 [PCE] is a core element of a controller, in charge of computing 468 Deterministic paths to be applied in the Network Plane. 470 A (Northbound) Service Interface enables applications in the 471 Application Plane to communicate with the entities in the Controller 472 Plane. 474 One or more PCE(s) collaborate to implement the requests from the SME 475 as Per-Stream Per-Hop Behaviors installed in the relay systems for 476 each individual streams. The PCEs place each stream along a 477 deterministic sequence of relay systems so as to respect per-stream 478 constraints such as security and latency, and optimize the overall 479 result for metrics such as an abstract aggregated cost. The 480 deterministic sequence can typically be more complex than a direct 481 sequence and include redundancy path, with one or more packet 482 replication and elimination points. 484 4.3. The Network Plane 486 The Network Plane represents the network devices and protocols as a 487 whole, regardless of the Layer at which the network devices operate. 489 The network Plane comprises the Network Interface Cards (NIC) in the 490 end systems, which are typically IP hosts, and relay systems, which 491 are typically IP routers and switches. Network-to-Network Interfaces 492 such as used for Traffic Engineering path reservation in [RFC3209], 493 as well as User-to-Network Interfaces (UNI) such as provided by the 494 Local Management Interface (LMI) between network and end systems, are 495 all part of the Network Plane. 497 A Southbound (Network) Interface enables the entities in the 498 Controller Plane to communicate with devices in the Network Plane. 499 This interface leverages and extends TEAS to describe the physical 500 topology and resources in the Network Plane. 502 Stream Management Entity 504 End End 505 System System 507 -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- 509 PCE PCE PCE PCE 511 -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+--+-+-+-+-+-+- 513 Relay Relay Relay Relay 514 System System System System 515 NIC NIC 516 Relay Relay Relay Relay 517 System System System System 519 Figure 3 521 The relay systems (and eventually the end systems NIC) expose their 522 capabilities and physical resources to the controller (the PCE), and 523 update the PCE with their dynamic perception of the topology, across 524 the Southbound Interface. In return, the PCE(s) set the per-stream 525 paths up, providing a Stream Characterization that is more tightly 526 coupled to the relay system Operation than a TSpec. 528 At the Network plane, relay systems exchange information regarding 529 the state of the paths, between adjacent systems and eventually with 530 the end systems, and forward packets within constraints associated to 531 each stream, or, when unable to do so, perform a last resort 532 operation such as drop or declassify. 534 This specification focuses on the Southbound interface and the 535 operation of the Network Plane. 537 4.4. Elements of DetNet Architecture 539 The DetNet architecture has a number of elements, discussed in the 540 following sections: 542 a. A model for the definition, identification, and operation of 543 DetNet streams (Section 4.5), for use by relay systems to 544 classify and process individual packets following per-stream 545 rules. 547 b. A model for the flow of data from an end system or through a 548 relay system that can be used to predict the bounds for that 549 system's impact on the QoS of a DetNet stream, without 550 significantly constraining the method of implementing that 551 system, for use by the Controllers to configure policing and 552 shaping engines in Network Systems over the Southbound interface. 553 The model includes: 555 1. A model for queuing, transmission selection, shaping, 556 preemption, and timing resources that can be used by an end 557 system or relay system to control the selection of packets 558 output on an interface. These models must have sufficiently 559 well-defined characteristics, both individually and in the 560 aggregate, to give predictable results for the QoS for DetNet 561 packets (Section 4.7). 563 2. A model for identifying misbehaving DetNet streams and 564 mitigating their impact on properly functioning streams 565 (Section 4.9). 567 c. A model for the relay system to inform the controller(s) of the 568 information it needs for adequate path computations including: 570 1. Systems' individual capabilities (e.g. can do replication, 571 can do precise time). 573 2. Link capabilities and resources (e.g. bandwidth, 0 delays, 574 hardware deterministic support to the physical layer, ...) 576 3. hysical resources (total and available buffers, timers, 577 queues, etc) 579 4. Network Adjacencies (neighbors) 581 d. A model for the provision of a service, by end systems, or relay 582 systems, to forward a DetNet stream over a simple or redundant 583 path. The model includes: 585 1. A model for an abstract relaying operation of either Routing 586 or forwarding packets of a DetNet stream to a next-hop relay 587 system, across Layer boundaries. 589 2. A model of next-hop(s) information for replicating the 590 packets of a DetNet stream, typically at or near the talker, 591 merging and/or re-replicating those packets at other points 592 in the network, and finally eliminating the duplicates, 593 typically at or near the listener(s), in order to provide 594 high availability (Section 3.3). 596 e. The protocol stack model for an end system and/or a relay system 597 should support the above elements in a manner that maximizes the 598 applicability of existing standards and protocols to the DetNet 599 problem, allows for the creation of new protocols where needed, 600 thus making DetNet an add-on feature to existing networks, rather 601 than a new way to do networking. In particular this protocol 602 stack supports networks in which the path from talker to 603 listener(s) includes bridges and/or routers in any order 604 (Section 4.10). 606 f. A variety of models for the provisioning of DetNet streams can be 607 envisioned, including orchestration by a central controller or by 608 a federation of controllers, provisioning by relay systems and 609 end systems sharing peer-to-peer protocols, by off-line 610 configuration, or by a combination of these methods. The 611 provisioning models are similar to existing Layer-2 and Layer-3 612 models, in order to minimize the amount of innovation required in 613 this area (Section 4.12). 615 4.5. DetNet streams 617 4.5.1. Talker guarantees 619 DetNet streams can by synchronous or asynchronous. The transmission 620 of packets in synchronous DetNet streams uses time synchronization 621 among the end and relay systems to control the flow of packets. 622 Asynchronous DetNet streams are characterized by: 624 o A maximum packet size; 626 o An observation interval; and 627 o A maximum number of transmissions during that observation 628 interval. 630 These parameters, together with knowledge of the protocol stack used 631 (and thus the size of the various headers added to a packet), limit 632 the number of bit times per observation interval that the DetNet 633 stream can occupy the physical medium. 635 The talker promises that these limits will not be exceeded. If the 636 talker transmits less data than this limit allows, the unused 637 resources such as link bandwidth can be made available by the system 638 to non-DetNet packets. However, making those resources available to 639 DetNet packets in other streams would serve no purpose. Those other 640 streams have their own dedicated resources, on the assumption that 641 all DetNet streams can use all of their resources over a long period 642 of time. 644 Note that there is no provision in DetNet for throttling streams; the 645 assumption is that a DetNet stream, to be useful, must be delivered 646 in its entirety. That is, while any useful application is written to 647 expect a certain number of lost packets, the real-time applications 648 of interest to DetNet demand that the loss of data due to the network 649 is extraordinarily infrequent. 651 Although DetNet strives to minimize the changes required of an 652 application to allow it to shift from a special-purpose digital 653 network to an Internet Protocol network, one fundamental shift in the 654 behavior of network applications that is impossible to avoid--the 655 reservation of resources before the application starts. In the first 656 place, a network cannot deliver finite latency and practically zero 657 packet loss to an arbitrarily high offered load. Secondly, achieving 658 practically zero packet loss for unthrottled (though bandwidth 659 limited) streams means that bridges and routers have to dedicate 660 buffer resources to specific streams or to classes of streams. The 661 requirements of each reservation have to be translated into the 662 parameters that control each system's queuing, shaping, and 663 scheduling functions and delivered to the hosts, bridges, and 664 routers. 666 4.5.2. Incomplete Networks 668 The presence in the network of relay systems that are not fully 669 capable of offering DetNet services complicates the ability of the 670 relay systems and/or controller to allocate resources, as extra 671 buffering, and thus extra latency, must be allocated at each point 672 that is downstream from the non-DetNet relay system for some DetNet 673 stream. 675 4.6. Data Flow Model through Systems 677 4.7. Queuing, Shaping, Scheduling, and Preemption 679 For this reason, the IEEE 802.1 Time-Sensitive Networking Task Group 680 has defined a set of queuing, shaping, and scheduling algorithms that 681 enable each bridge or router to compute the exact number of buffers 682 to be allocated for each stream or class of streams. 684 4.8. Coexistence with normal traffic 686 A DetNet network supports the dedication of at least 75% of the 687 network bandwidth to DetNet streams. But, no matter how much is 688 dedicated for DetNet streams, It is z goal of DetNet to not interfere 689 excessively with existing QoS schemes. It is also important that 690 non-DetNet traffic not disrupt the DetNet stream, of course (see 691 Section 4.9 and Section 6). For these reasons: 693 o Bandwidth (transmission opportunities) not utilized by a DetNet 694 stream are available to non-DetNet packets (though not to other 695 DetNet streams). 697 o DetNet streams can be shaped, in order to ensure that the highest- 698 priority non-DetNet packet also is ensured a maximum latency. 700 o When transmission opportunities for DetNet streams are scheduled 701 in detail, then the algorithm constructing the schedule should 702 leave sufficient opportunities for non-DetNet packets to satisfy 703 the needs of the uses of the network. 705 Ideally, the net effect of the presence of DetNet streams in a 706 network on the non-DetNet packets is primarily a reductoin in the 707 available bandwidth. 709 4.9. Fault Mitigation 711 One key to building robust real-time systems is to reduce the 712 infinite variety of possible failures to a number that can be 713 analyzed with reasonable confidence. DetNet aids in the process by 714 providing filters and policers to detect DetNet packets received on 715 the wrong interface, or at the wrong time, or in too great a volume, 716 and to then take actions such as disabling the offending packet, 717 shutting down the offending DetNet stream, or shutting down the 718 offending interface. 720 It is also essential that filters and service remarking be employed 721 to prevent non-DetNet packets from impinging on the resources 722 allocated to DetNet packets. 724 There exist techniques, at present and/or in various stages of 725 standardization, that can perform these fault mitigation tasks that 726 deliver a high probability that misbehaving systemd will have zero 727 impact on well-behaved DetNet streams, except of course, for the 728 receiving interface(s) immediately downstream of the misbehaving 729 device. 731 4.10. Protocol Stack Model 733 This section will be further developed. See [IEEE802.1CB], Annex C, 734 for a description of the protocol stack. This is very much a work in 735 progress, not a standard. See also [IEEE802.1Qcc]. 737 4.11. Advertising resources, capabilities and adjacencies 739 4.12. Provisioning model 741 4.12.1. Centralized Path Computation and Installation 743 A centralized routing model, such as provided with a PCE (RFC 4655 744 [RFC4655]), enables global and per-stream optimizations. The model 745 is attractive but a number of issues are left to be solved. In 746 particular: 748 o whether and how the path computation can be installed by 1) an end 749 device or 2) a Network Management entity, 751 o and how the path is set up, either by installing state at each hop 752 with a direct interaction between the forwarding device and the 753 PCE, or along a path by injecting a source-routed request at one 754 end of the path. 756 4.12.2. Distributed Path Setup 758 Whether a distributed alternative without a PCE can be valuable 759 should be studied as well. Such an alternative could for instance 760 inherit from the Resource ReSerVation Protocol [RFC5127] (RSVP) 761 flows. 763 In a Layer-2 only environment, or as part of a layered approach to a 764 mixed environment, IEEE 802.1 also has work, either completed or in 765 progress. [IEEE802.1Q-2011] Clause 35 describes SRP, a peer-to-peer 766 protocol for Layer-2 roughly analogous to RSVP. Almost complete is 767 [IEEE802.1Qca], which defines how ISIS can provide multiple disjoint 768 paths or distribution trees. Also in progress is [IEEE802.1Qcc], 769 which expands the capabilities of SRP. 771 5. Related IETF work 773 5.1. Deterministic PHB 775 [I-D.svshah-tsvwg-deterministic-forwarding] defines a Differentiated 776 Services Per-Hop-Behavior (PHB) Group called Deterministic Forwarding 777 (DF). The document describes the purpose and semantics of this PHB. 778 It also describes creation and forwarding treatment of the service 779 class. The document also describes how the code-point can be mapped 780 into one of the aggregated Diffserv service classes [RFC5127]. 782 5.2. 6TiSCH 784 Industrial process control already leverages deterministic wireless 785 Low power and Lossy Networks (LLNs) to interconnect critical 786 resource-constrained devices and form wireless mesh networks, with 787 standards such as [ISA100.11a] and [WirelessHART]. 789 These standards rely on variations of the [IEEE802154e] timeSlotted 790 Channel Hopping (TSCH) [I-D.ietf-6tisch-tsch] Medium Access Control 791 (MAC), and a form of centralized Path Computation Element (PCE), to 792 deliver deterministic capabilities. 794 The TSCH MAC benefits include high reliability against interference, 795 low power consumption on characterized streams, and Traffic 796 Engineering capabilities. Typical applications are open and closed 797 control loops, as well as supervisory control streams and management. 799 The 6TiSCH Working Group focuses only on the TSCH mode of the IEEE 800 802.15.4e standard. The WG currently defines a framework for 801 managing the TSCH schedule. Future work will standardize 802 deterministic operations over so-called tracks as described in 803 [I-D.ietf-6tisch-architecture]. Tracks are an instance of a 804 deterministic path, and the DetNet work is a prerequisite to specify 805 track operations and serve process control applications. 807 [RFC5673] and [I-D.ietf-roll-rpl-industrial-applicability] section 808 2.1.3. and next discusses application-layer paradigms, such as 809 Source-sink (SS) that is a Multipeer to Multipeer (MP2MP) model that 810 is primarily used for alarms and alerts, Publish-subscribe (PS, or 811 pub/sub) that is typically used for sensor data, as well as Peer-to- 812 peer (P2P) and Peer-to-multipeer (P2MP) communications. Additional 813 considerations on Duocast and its N-cast generalization are also 814 provided for improved reliability. 816 6. Security Considerations 818 Security in the context of Deterministic Networking has an added 819 dimension; the time of delivery of a packet can be just as important 820 as the contents of the packet, itself. A man-in-the-middle attack, 821 for example, can impose, and then systematically adjust, additional 822 delays into a link, and thus disrupt or subvert a real-time 823 application without having to crack any encryption methods employed. 824 See [RFC7384] for an exploration of this issue in a related context. 826 Furthermore, in a control system where millions of dollars of 827 equipment, or even human lives, can be lost if the DetNet QoS is not 828 delivered, one must consider not only simple equipment failures, 829 where the box or wire instantly becomes perfectly silent, but bizarre 830 errors such as can be coused by software failures. Because there is 831 essentiall no limit to the kinds of failures that can occur, 832 protecting against realistic equipment failures is indistinguishable, 833 in most cases, from protecting against malicious behavior, whether 834 accidental or intentional. See also Section 4.9. 836 Security must cover: 838 o the protection of the signaling protocol 840 o the authentication and authorization of the controlling systems 842 o the identification and shaping of the streams 844 7. IANA Considerations 846 This document does not require an action from IANA. 848 8. Acknowledgements 850 The authors wish to thank Jouni Korhonen, Erik Nordmark, George 851 Swallow, Rudy Klecka, Anca Zamfir, David Black, Thomas Watteyne, 852 Shitanshu Shah, Craig Gunther, Rodney Cummings, Wilfried Steiner, 853 Marcel Kiessling, Karl Weber, Ethan Grossman and Pat Thaler, for 854 their various contribution with this work. 856 9. Informative References 858 [AVnu] http://www.avnu.org/, "The AVnu Alliance tests and 859 certifies devices for interoperability, providing a simple 860 and reliable networking solution for AV network 861 implementation based on the Audio Video Bridging (AVB) 862 standards.", . 864 [CCAMP] IETF, "Common Control and Measurement Plane", 865 . 867 [HART] www.hartcomm.org, "Highway Addressable Remote Transducer, 868 a group of specifications for industrial process and 869 control devices administered by the HART Foundation", . 871 [HSR-PRP] IEC, "High availability seamless redundancy (HSR) is a 872 further development of the PRP approach, although HSR 873 functions primarily as a protocol for creating media 874 redundancy while PRP, as described in the previous 875 section, creates network redundancy. PRP and HSR are both 876 described in the IEC 62439 3 standard.", 877 . 880 [I-D.finn-detnet-problem-statement] 881 Finn, N. and P. Thubert, "Deterministic Networking Problem 882 Statement", draft-finn-detnet-problem-statement-01 (work 883 in progress), October 2014. 885 [I-D.ietf-6tisch-architecture] 886 Thubert, P., Watteyne, T., Struik, R., and M. Richardson, 887 "An Architecture for IPv6 over the TSCH mode of IEEE 888 802.15.4e", draft-ietf-6tisch-architecture-05 (work in 889 progress), January 2015. 891 [I-D.ietf-6tisch-tsch] 892 Watteyne, T., Palattella, M., and L. Grieco, "Using 893 IEEE802.15.4e TSCH in an IoT context: Overview, Problem 894 Statement and Goals", draft-ietf-6tisch-tsch-05 (work in 895 progress), January 2015. 897 [I-D.ietf-roll-rpl-industrial-applicability] 898 Phinney, T., Thubert, P., and R. Assimiti, "RPL 899 applicability in industrial networks", draft-ietf-roll- 900 rpl-industrial-applicability-02 (work in progress), 901 October 2013. 903 [I-D.svshah-tsvwg-deterministic-forwarding] 904 Shah, S. and P. Thubert, "Deterministic Forwarding PHB", 905 draft-svshah-tsvwg-deterministic-forwarding-03 (work in 906 progress), March 2015. 908 [IEEE802.1AS-2011] 909 IEEE, "Timing and Synchronizations (IEEE 802.1AS-2011)", 910 2011, . 913 [IEEE802.1BA-2011] 914 IEEE, "AVB Systems (IEEE 802.1BA-2011)", 2011, 915 . 918 [IEEE802.1CB] 919 IEEE, "Seamless Redundancy (IEEE Draft P802.1CB)", 2015, 920 . 923 [IEEE802.1Q-2011] 924 IEEE, "MAC Bridges and VLANs (IEEE 802.1Q-2011", 2011, 925 . 928 [IEEE802.1Qat-2010] 929 IEEE, "Stream Reservation Protocol (IEEE 802.1Qat-2010)", 930 2010, . 933 [IEEE802.1Qav] 934 IEEE, "Forwarding and Queuing (IEEE 802.1Qav-2009)", 2009, 935 . 938 [IEEE802.1Qca] 939 IEEE, "Path Control and Reservation", 2015, 940 . 943 [IEEE802.1Qcc] 944 IEEE, "Stream Reservation Protocol (SRP) Enhancements and 945 Performance Improvements", 2015, 946 . 949 [IEEE802.1TSNTG] 950 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 951 Networks Task Group", 2013, 952 . 954 [IEEE802154] 955 IEEE standard for Information Technology, "IEEE std. 956 802.15.4, Part. 15.4: Wireless Medium Access Control (MAC) 957 and Physical Layer (PHY) Specifications for Low-Rate 958 Wireless Personal Area Networks", June 2011. 960 [IEEE802154e] 961 IEEE standard for Information Technology, "IEEE std. 962 802.15.4e, Part. 15.4: Low-Rate Wireless Personal Area 963 Networks (LR-WPANs) Amendment 1: MAC sublayer", April 964 2012. 966 [ISA100.11a] 967 ISA/IEC, "ISA100.11a, Wireless Systems for Automation, 968 also IEC 62734", 2011, < http://www.isa100wci.org/en- 969 US/Documents/PDF/3405-ISA100-WirelessSystems-Future-broch- 970 WEB-ETSI.aspx>. 972 [ODVA] http://www.odva.org/, "The organization that supports 973 network technologies built on the Common Industrial 974 Protocol (CIP) including EtherNet/IP.", . 976 [PCE] IETF, "Path Computation Element", 977 . 979 [Profinet] 980 http://us.profinet.com/technology/profinet/, "PROFINET is 981 a standard for industrial networking in automation.", 982 . 984 [RFC2205] Braden, B., Zhang, L., Berson, S., Herzog, S., and S. 985 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 986 Functional Specification", RFC 2205, September 1997. 988 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 989 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 990 Tunnels", RFC 3209, December 2001. 992 [RFC4655] Farrel, A., Vasseur, J., and J. Ash, "A Path Computation 993 Element (PCE)-Based Architecture", RFC 4655, August 2006. 995 [RFC5127] Chan, K., Babiarz, J., and F. Baker, "Aggregation of 996 Diffserv Service Classes", RFC 5127, February 2008. 998 [RFC5673] Pister, K., Thubert, P., Dwars, S., and T. Phinney, 999 "Industrial Routing Requirements in Low-Power and Lossy 1000 Networks", RFC 5673, October 2009. 1002 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1003 Packet Switched Networks", RFC 7384, October 2014. 1005 [RFC7426] Haleplidis, E., Pentikousis, K., Denazis, S., Hadi Salim, 1006 J., Meyer, D., and O. Koufopavlou, "Software-Defined 1007 Networking (SDN): Layers and Architecture Terminology", 1008 RFC 7426, January 2015. 1010 [TEAS] IETF, "Traffic Engineering Architecture and Signaling", 1011 . 1013 [WirelessHART] 1014 www.hartcomm.org, "Industrial Communication Networks - 1015 Wireless Communication Network and Communication Profiles 1016 - WirelessHART - IEC 62591", 2010. 1018 Authors' Addresses 1020 Norm Finn 1021 Cisco Systems 1022 170 W Tasman Dr. 1023 San Jose, California 95134 1024 USA 1026 Phone: +1 408 526 4495 1027 Email: nfinn@cisco.com 1029 Pascal Thubert 1030 Cisco Systems 1031 Village d'Entreprises Green Side 1032 400, Avenue de Roumanille 1033 Batiment T3 1034 Biot - Sophia Antipolis 06410 1035 FRANCE 1037 Phone: +33 4 97 23 26 34 1038 Email: pthubert@cisco.com