idnits 2.17.1 draft-thubert-tsvwg-detnet-transport-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 352 has weird spacing: '...e stack v ...' == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (October 30, 2017) is 2370 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-13) exists of draft-ietf-detnet-architecture-03 == Outdated reference: A later version (-09) exists of draft-ietf-detnet-problem-statement-02 == Outdated reference: A later version (-16) exists of draft-ietf-detnet-security-00 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-13 Summary: 0 errors (**), 0 flaws (~~), 7 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TSVWG P. Thubert, Ed. 3 Internet-Draft Cisco 4 Intended status: Standards Track October 30, 2017 5 Expires: May 3, 2018 7 A Transport Layer for Deterministic Networks 8 draft-thubert-tsvwg-detnet-transport-01 10 Abstract 12 This document specifies the behavior of a Transport Layer operating 13 over a Deterministic Network and implementing a DetNet Service Layer 14 and a Northbound side of the DetNet User-to-Network Interface. 16 Status of This Memo 18 This Internet-Draft is submitted in full conformance with the 19 provisions of BCP 78 and BCP 79. 21 Internet-Drafts are working documents of the Internet Engineering 22 Task Force (IETF). Note that other groups may also distribute 23 working documents as Internet-Drafts. The list of current Internet- 24 Drafts is at https://datatracker.ietf.org/drafts/current/. 26 Internet-Drafts are draft documents valid for a maximum of six months 27 and may be updated, replaced, or obsoleted by other documents at any 28 time. It is inappropriate to use Internet-Drafts as reference 29 material or to cite them other than as "work in progress." 31 This Internet-Draft will expire on May 3, 2018. 33 Copyright Notice 35 Copyright (c) 2017 IETF Trust and the persons identified as the 36 document authors. All rights reserved. 38 This document is subject to BCP 78 and the IETF Trust's Legal 39 Provisions Relating to IETF Documents 40 (https://trustee.ietf.org/license-info) in effect on the date of 41 publication of this document. Please review these documents 42 carefully, as they describe your rights and restrictions with respect 43 to this document. Code Components extracted from this document must 44 include Simplified BSD License text as described in Section 4.e of 45 the Trust Legal Provisions and are provided without warranty as 46 described in the Simplified BSD License. 48 Table of Contents 50 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 51 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 52 3. On Deterministic Networking . . . . . . . . . . . . . . . . . 5 53 3.1. Applications and Requirements . . . . . . . . . . . . . . 5 54 3.2. The DetNet User-to-Network Interface (UNI) . . . . . . . 7 55 3.3. The DetNet Stack . . . . . . . . . . . . . . . . . . . . 8 56 3.4. The DetNet Service Model . . . . . . . . . . . . . . . . 8 57 4. DetTrans Operations . . . . . . . . . . . . . . . . . . . . . 9 58 4.1. DetTrans Overview . . . . . . . . . . . . . . . . . . . . 9 59 4.2. Application Requirements . . . . . . . . . . . . . . . . 9 60 4.2.1. Packet Normalization . . . . . . . . . . . . . . . . 9 61 4.2.2. Packet Streaming . . . . . . . . . . . . . . . . . . 10 62 4.3. Deterministic Flow Services . . . . . . . . . . . . . . . 10 63 4.3.1. Deterministic Flows . . . . . . . . . . . . . . . . . 10 64 4.3.2. Deterministic Flow Encapsulation and Stitching . . . 11 65 4.3.2.1. Flow Stitching . . . . . . . . . . . . . . . . . 11 66 4.3.2.2. Load Sharing . . . . . . . . . . . . . . . . . . 11 67 4.3.2.3. Flow Aggregation . . . . . . . . . . . . . . . . 12 68 4.3.3. Deterministic Service Protection . . . . . . . . . . 13 69 4.3.3.1. PRE vs. 1+1 Redundancy . . . . . . . . . . . . . 13 70 4.3.3.2. Network Coding . . . . . . . . . . . . . . . . . 13 71 4.3.3.3. Multipath DetTrans Services . . . . . . . . . . . 13 72 5. The DetNet-UNI . . . . . . . . . . . . . . . . . . . . . . . 14 73 5.1. Local Loop Flow Control . . . . . . . . . . . . . . . . . 16 74 5.1.1. Dichotomy of a DetNet End System . . . . . . . . . . 16 75 5.1.2. Local Loop Location . . . . . . . . . . . . . . . . . 17 76 5.1.3. Network Pull vs. Rate Based Flow Control . . . . . . 18 77 5.2. DetNet-UNI Protocol Exchanges . . . . . . . . . . . . . . 18 78 5.2.1. the "More" Message . . . . . . . . . . . . . . . . . 18 79 5.2.2. the "Time-Correction" Message . . . . . . . . . . . . 19 80 5.2.3. Loss of a Control Message . . . . . . . . . . . . . . 19 81 6. Security Considerations . . . . . . . . . . . . . . . . . . . 20 82 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 83 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 20 84 9. Informative References . . . . . . . . . . . . . . . . . . . 20 85 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 22 87 1. Introduction 89 Over last twenty years, voice, data and video networks have converged 90 to digital over IP. Mail delivery has become quasi-immediate and 91 volumes have multiplied; long distance voice is now mostly free and 92 the videophone is finally a reality; TV is available on-demand and 93 games became interactive and massively multi-player. The convergence 94 of highly heterogeneous networks over IP resulted in significant 95 drops in price for the end-user while adding new distinct value to 96 the related services. Yet, and even though similar benefits can be 97 envisioned when converging new applications over the Internet, there 98 are still many disjoint branches in the networking family tree, many 99 use-cases where mission-specific applications continue to utilize 100 dedicated point-to-point analog and digital technologies for their 101 operations. 103 Forty years ago, Control Information was first encoded as an analog 104 modulation of current (typically 4 to 20 mA) that can be carried 105 virtually instantly and with no loss over a distance. Then came 106 digitization, which enabled to multiplex data with the control signal 107 and manage the devices, but at the same time introduced latency to 108 industrial processes, the necessary delay to encode a series of bits 109 on a link and transport them along, which in turn may limit the 110 amount of transported information. The need to save cable and 111 simplify wiring lead to the Time Division Multiplexing (TDM) of 112 signals from multiple devices over shared digital buses, each signal 113 being granted access to the medium at a fixed period for a fixed 114 duration; with TDM, came more latency, waiting for the next reserved 115 access time. Statistical multiplexing, with Ethernet and IP, was 116 then introduced to achieve higher speeds at lower cost, and with it 117 came jitter and congestion loss. 119 A number of Operational Technology (OT) applications are now 120 migrating to Ethernet and IP, but that comes at the expense of 121 additional latency for the flows, to compensate for the degradation 122 of the transport discussed above. This also comes at the expense of 123 additional complexity in particular, applications may need to 124 transport a sense of time, provide some Forward Error Correction 125 (FEC) and include a jitter absorption buffer. for that reason, many 126 applications were never ported and OT networks are still largely 127 operated on point-to-point serial links and TDM buses. 129 A sense of what Deterministic Networking is has emerged as the 130 capability to make the Application simple again and enable a larger 131 migration of existing applicationsby absorbing the complexity lower 132 in the stack, at the Transport, Network and Link layers. A 133 Deterministic Network should be capable to emulate point-to-point 134 wires over a packet network, sharing the network resources between 135 deterministic and non-deterministic flows in such a fashion that 136 there can no observable influence whatsoever on a deterministic flow 137 from any other flow, regardless of the load of the network. 139 The generalization of the needs for more deterministic networks have 140 led to the IEEE 802.1 AVB Task Group becoming the Time-Sensitive 141 Networking (TSN) [IEEE802.1TSNTG] Task Group (TG), with a much- 142 expanded constituency from the industrial and vehicular markets. In 143 order to address the problem at the network layer, the DetNet Working 144 Group was formed to specify the signaling elements to be used to 145 establish a path and the tagging elements to be used identify the 146 flows that are to be forwarded along that path. 148 The "Deterministic Networking Use Cases" [I-D.ietf-detnet-use-cases] 149 indicates that beyond the classical case of industrial automation and 150 control systems (IACS), there are in fact multiple industries with 151 strong and yet relatively similar needs for deterministic network 152 services such as latency guarantees and ultra-low packet loss. The 153 "Deterministic Networking Problem Statement" 154 [I-D.ietf-detnet-problem-statement] documents the specific 155 requirements for the use of routed networks to support these 156 applications and the "Deterministic Networking Architecture" 157 [I-D.ietf-detnet-architecture] introduces the model that must be 158 proposed to integrate determinism in IT technology. 160 A DetNet network will guarantee a bounded latency and a very low 161 packet loss as long as the incoming flows respect a certain Service 162 Level Agreement (SLA), as typically expressed in the form of a 163 maximum packet size, a time window of observation and a maximum 164 number of packets per time window. 166 Outside the scope of DetNet, the IETF will also need to specify the 167 necessary protocols, or protocol additions, based on relevant IETF 168 technologies, to enable end-to-end deterministic flows. One critical 169 element is the Deterministic Transport Layer (DetTrans) that adapts 170 the flows coming from the Application Layer to the SLA of the DetNet 171 Network and provide end-to-end guarantees such as loss, latency and 172 timeliness. 174 The DetTrans Layer should in particular ensure that: 176 o the Deterministic Network setup matches the needs of the 177 Application 179 o the Application flows are presented to the Deterministic Network 180 in accordance to the SLA regardless of the way the data is passed 181 from the application 183 o the use of the network is optimized so as to ensure that every 184 byte from the application can effectively be transported 186 o the application flow is delivered reliably and with a bounded 187 latency to the other Transport End Point, which may imply a FEC 188 technique such as Network Coding, Packet Replication and 189 Elimination (PRE), or basic 1+1 redundancy. 191 o the full of the application flow is served, which may require the 192 use of multiple reservations in parallel, and the reordering of 193 the flows 195 On the one hand, the Deterministic Network will typically guarantee a 196 constant rate, so the classical Transport feature of flow control 197 will not be needed in a Deterministic Transport. On the other hand, 198 the Application and Transport layers may not reside in the same 199 device as the DetNet Router and/or the IEEE Std. 802.1 TSN Bridge 200 that acts as ingress point to the Deterministic Network. It results 201 that a minimum reliability and flow control must take place over the 202 Local Loop between these devices to ensure that the Deterministic 203 Network is kept optimally fed, meaning that packets are received just 204 in time for their scheduled transmission opportunities. 206 2. Terminology 208 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 209 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 210 document are to be interpreted as described in [RFC2119]. 212 3. On Deterministic Networking 214 3.1. Applications and Requirements 216 The Internet is not the only digital network that has grown 217 dramatically over the last 30-40 years. Video and audio 218 entertainment, and control systems for machinery, manufacturing 219 processes, and vehicles are also ubiquitous, and are now based almost 220 entirely on digital technologies. Over the past 10 years, engineers 221 in these fields have come to realize that significant advantages in 222 both cost and in the ability to accelerate growth can be obtained by 223 basing all of these disparate digital technologies on packet 224 networks. 226 The goals of Deterministic Networking are to enable the migration of 227 applications that use special-purpose fieldbus technologies (HDMI, 228 CANbus, ProfiBus, etc... even RS-232!) to packet technologies in 229 general, and the Internet Protocol in particular, and to support both 230 these new applications, and existing packet network applications, 231 over the same physical network. 233 Considerable experience ([ODVA]/[EIP], [AVnu], [Profinet],[HART], 234 [IEC62439], [ISA100.11a] and [WirelessHART], etc...) has shown that 235 these applications need a some or all of a suite of deterministic 236 features. 238 That suite of deterministic features includes: 240 1. Time synchronization of all Host and network nodes (Routers and/ 241 or Bridges), accurate to something between 10 nanoseconds and 10 242 microseconds, depending on the application. 244 2. Support for critical packet flows that: 246 * Can be unicast or multicast; 248 * Need absolute guarantees of minimum and maximum latency end- 249 to-end across the network; sometimes a tight jitter is 250 required as well; 252 * Need a packet loss ratio beyond the classical range for a 253 particular medium, in the range of 10^-9 to 10^-12, or better, 254 on Ethernet, and in the order of 10^-5 in Wireless Sensor Mesh 255 Networks; 257 * Can, in total, absorb more than half of the network's 258 available bandwidth (that is, massive over-provisioning is 259 ruled out as a solution); 261 * Cannot suffer throttling, flow control, or any other network- 262 imposed latency, for flows that can be meaningfully 263 characterized either by a fixed, repeating transmission 264 schedule, or by a maximum bandwidth and packet size; 266 3. Multiple methods to schedule, shape, limit, and otherwise control 267 the transmission of critical packets at each hop through the 268 network data plane; 270 4. Robust defenses against misbehaving Hosts, Routers, or Bridges, 271 both in the data and control planes, with guarantees that a 272 critical flow within its guaranteed resources cannot be affected 273 by other flows whatever the pressures on the network; 275 5. One or more methods to reserve resources in Bridges and Routers 276 to carry these flows. 278 Robustness is a common need for networking protocols, but plays a 279 more important part in real-time control networks, where expensive 280 equipment, and even lives, can be lost due to misbehaving equipment. 281 Reserving resources before packet transmission is the one fundamental 282 shift in the behavior of network applications that is impossible to 283 avoid. In the first place, a network cannot deliver finite latency 284 and practically zero packet loss to an arbitrarily high offered load. 285 Secondly, achieving practically zero packet loss for un-throttled 286 (though bandwidth limited) flows means that Bridges and Routers have 287 to dedicate buffer resources to specific flows or to classes of 288 flows. The requirements of each reservation have to be translated 289 into the parameters that control each Host's, Bridge's, and Router's 290 queuing, shaping, and scheduling functions and delivered to the 291 Hosts, Bridges, and Routers. 293 3.2. The DetNet User-to-Network Interface (UNI) 295 The "Deterministic Networking Architecture" 296 [I-D.ietf-detnet-architecture] presents the end-to-end networking 297 model and the DetNet services; in particular, it depicts the DetNet 298 User-to-Network Interfaces (DetNet-UNIs) ("U" in Figure 1) between 299 the Edge nodes (PE) of the Deterministic Network and the End Systems. 300 These UNIs are assumed to be packet-based reference points and 301 provide connectivity over the packet network. The Architecture also 302 mentions internal reference points between the Central Processing 303 Unit (CPU) and the Network Interface Card (NIC) in the End System. 304 The DetNet-UNIs provide congestion protection services and belong to 305 the DetNet Transport Layer. 307 DetNet DetNet 308 End System End System 309 _ _ 310 / \ +----DetNet-UNI (U) / \ 311 /App\ | /App\ 312 /-----\ | /-----\ 313 | NIC | v ________ | NIC | 314 +--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+ 315 | / \__/ \ | | 316 | / +----+ +----+ \_____ | | 317 | / | | | | \_______ | | 318 +------U PE +----+ P +----+ \ _ v | 319 | | | | | | | ___/ \ | 320 | +--+-+ +----+ | +----+ | / \_ | 321 \ | | | | | / \ | 322 \ | +----+ +--+-+ +--+PE |-------- U------+ 323 \ | | | | | | | | | \_ _/ 324 \ +---+ P +----+ P +--+ +----+ | \____/ 325 \___ | | | | / 326 \ +----+__ +----+ DetNet-1 DetNet-2 327 | \_____/ \___________/ | 328 | | 329 | | End-to-End-Service | | | | 330 <----------------------------------------------------------------> 331 | | DetNet-Service | | | | 332 | <--------------------------------------------------> | 333 | | | | | | 335 Figure 1: DetNet Service Reference Model (multi-domain) 337 A specific hardware is necessary for the time-sensitive functions of 338 synchronization, shaping and scheduling. This hardware may or may 339 not be fully available on a NIC inside the Host system. This 340 specification makes a distinction between a fully DetNet-Capable NIC, 341 and a DetNet-Aware NIC that participates to the DetNet-UNI, but is 342 not synchronized and scheduled with the Deterministic Network. 344 3.3. The DetNet Stack 346 The "Deterministic Networking Architecture" 347 [I-D.ietf-detnet-architecture] presents a conceptual DetNet data 348 plane layering model. The protocol stack includes a Service Layer 349 and a Transport Layer and is illustrated in Figure 2. 351 | packets going | ^ packets coming ^ 352 v down the stack v | up the stack | 353 +------------------------+ +------------------------+ 354 | Source | | Destination | 355 +------------------------+ +------------------------+ 356 | Service layer | | Service layer | 357 | ------------------- | | -------------------- | 358 | Adaptation | | Presentation | 359 | Packet | Network | | Duplicate | Network | 360 | sequencing | Coding | | elimination | Coding | 361 | Flow | Packet | | Flow | Packet | 362 | duplication | encoding | | merging | decoding | 363 +-------------+----------+ +-------------+----------+ 364 | Transport layer | | Transport layer | 365 | ------------------- | | ------------------- | 366 | Encapsulation | | Decapsulation | 367 | Congestion protection | | Congestion protection | 368 +------------------------+ +------------------------+ 369 | Lower layers | | Lower layers | 370 +------------------------+ +------------------------+ 371 DetNet End System DetNet End System 372 v ^ 373 \________DetNet Transport______/ 375 Figure 2: DetNet-Capable End-System Protocol Stack 377 3.4. The DetNet Service Model 379 The "DetNet Service Model" [I-D.varga-detnet-service-model] provides 380 more details on the distribution of DetNet awareness and services. 382 4. DetTrans Operations 384 4.1. DetTrans Overview 386 The DetNet Service Layer mostly operates between the end-points, 387 though it is possible that some operations such as Packet Replication 388 and Elimination are also performed in selected intermediate nodes. 389 The DetNet Transport Layer represents the methods that ensure that a 390 packet is deterministically forwarded hop-by-hop from a Detnet Relay 391 to the next. The term "Transport" in the DetNet terminology must not 392 be confused with the function described in this document. This 393 document defines Detrans as a Layer-4 operation and an IETF Transport 394 Layer; DetTrans provides DetNet End-To-End Services for its 395 Applications, as well as intermediate services in selected points. 397 Following the DetNet Architecture, DetTrans mostly corresponds to the 398 DetNet Service Layer and its interface with the Detnet Transport 399 Layer for congestion protection services through the DetNet_UNI, as 400 well as for encapsulation and decapsulation services. Compared to a 401 traditional IETF Transport Layer, DetTrans performs similar operation 402 of end-to-end reliability, flow control and multipath load sharing, 403 but differs on how those functionalities are achieved. 405 Architectural variations are also introduced, for instance: 407 o Multipath operations are not necessarily end-to-end and a DetTrans 408 function may be present inside the network to relay between N 409 parallel paths and M parallel path, and or perform reliability 410 functionality such as Packet Replication and Elimination. 412 o The flow control is only needed between the DetTrans Layer and the 413 first Deterministic Transit or Relay Node, for instance a DetNet 414 Router or an IEEE Std. 802.1 TSN Bridge. From that point on, the 415 flow is strictly controlled by the DetNet operation. 416 Architecturally speaking, the flow control does not belong to the 417 DetNet Service Layer but to the DetNet Transport Layer, which 418 means that this specification also defines a sublayer from the 419 DetNet Transport Layer for DetNet-UNI operations. 421 4.2. Application Requirements 423 4.2.1. Packet Normalization 425 A typical SLA for DetNet must be simple, for instance a maximum 426 packet size, and a maximum number of packets per window of time. 427 Smaller packets will mean wasted bandwidth, and excess packets within 428 a time window will be destroyed by the ingress shaping at the first 429 DetNet Bridge or Router. 431 The way the application layer feed the DetTrans layer may not 432 necessarily match the SLA with the Deterministic Network and in order 433 to provide the expected service, the DetTrans layer must pack the 434 data in packets that are as close to the maximum packet size as 435 possible, and yet make them available for transmission before 436 scheduled time. 438 4.2.2. Packet Streaming 440 The DetTrans Layer operates on its own sense of time which may be 441 loosely connected to the shared sense of time in the Deterministic 442 Network. 444 The DetTrans layer must shape its transmissions so as to ensure that 445 packets are delivered just in time to be injected along schedule in 446 the Deterministic Network. 448 4.3. Deterministic Flow Services 450 4.3.1. Deterministic Flows 452 Deterministic forwarding can only apply on flows with well-defined 453 characteristics such as periodicity and burstiness. Before a path 454 can be established to serve them, the expression of those 455 characteristics, and how the network can serve them, for instance in 456 shaping and forwarding operations, must be specified. 458 At the time of this writing, the distinction between application 459 layer flows and lower layer flows is not clearly stated in the 460 "Deterministic Networking Architecture" 461 [I-D.ietf-detnet-architecture]. For the purpose of this document, we 462 use the term Determistic End-to-End Service Flow (DEESF), or DetTrans 463 Flow, to refer to an end-to-end application flow, and the term 464 Determistic Service Flow (DSF), or DetNet Flow, to refer to a lower 465 layer deterministic transport. This is illustrated in Figure 3. 467 +---------------+ +---------------+ 468 | Application | | Application | 469 +---------------+ +---------------+ 470 | Service >--------- DetNet End-to-End Service Flow ---------> Service | 471 +---------------+ +--------+ +--------+ +---------------+ 472 | Transport | | | | | | Transport | 473 +---------------+ |>-- DetNet Service -->| +---------------+ 474 | Lower Layers | | Flow | | Lower Layers | 475 +---------------+ +--------+ +--------+ +---------------+ 477 Figure 3: DetTrans vs. DetNet Flows 479 An application flow is established end-to-end between the DetTrans 480 layers and uses one or more lower-layer deterministic flows either in 481 parallel or in serial modes. 483 At Application and DetTrans Layers, the characteristics of a flow 484 relate to aggregate properties such as throughput, loss, and traffic 485 shape, and the Traffic Specification (TSPec) is expressed as a 486 Constant Bit Rate (CBR) or a Variable Bit Rate (VBR), burstiness 487 (e.g. video I-Frames), reliability (e.g. five nines), worst case 488 latency, amount of data to transfer, and expected duration of the 489 flow. 491 At the DetNet Transport Layer (between Relays), metrics are very 492 different, and relate to immediate actions on a packet as opposed to 493 general characteristics of a flow. DetNet Transport Layer 494 characteristics include time sync precision, time interval between 495 packets, packet size, jitter, and number of packets per window of 496 time. This is how the network SLA is defined, but this is not the 497 native terms for the application and a complex mapping must ensure 498 that the path that is setup and the DetNet Transport Layer 499 effectively matches the requirements from the DetNet Services Layer 500 and above. 502 4.3.2. Deterministic Flow Encapsulation and Stitching 504 4.3.2.1. Flow Stitching 506 The DetNet encapsulation and decapsulation of one-in-one, one-in-many 507 and many-in-one Deterministic flows belongs to the DetNet Transport 508 Layer. Direct one-in-one flow stitching also belongs to the DetNet 509 Transport Layer. This happens when a deterministic flow can be 510 directly bridged into another, resource-to-resource, without the need 511 of an upper layer adaptation such as service protection from the 512 Service Layer. A Detnet End-to-End Service flow may be stitched into 513 one Detnet Service flow, or encapsulated in one or multiple Detnet 514 Service flows. 516 4.3.2.2. Load Sharing 518 Load Sharing refers to the encapsulation of a DetNet Flow in more 519 than one DetNet flows, for instance using multiple small and more 520 manageable DetNet Service Flows in parallel to carry a large 521 Determistic End-to-End Service Flow, in order to avoid the need to 522 periodically defragment the network. Packets are sequenced at the 523 DetTrans Layer and distributed over the DetNet Transports paths in 524 accordance to their relative capacities. In case of inconsistent 525 jitter and Latency characteristics, packets may need to be reordered 526 at the receiving DetTrans Layer based on the DSF Sequence. 528 +---------------+ +--------+ +--------+ +---------------+ 529 | Application | |>-- DetNet Service -->| | Application | 530 +---------v-----+ +-------------Flow-------------+ +-----^---------+ 531 | | | | | | | | | | | | 532 | | | | +--------+ +--------+ | | | | 533 | | | | | | | | 534 | | >------- DetNet End-to-End Service Flow -------> | | 535 | | | | | | | | 536 | | +-----+ +--------+ +--------+ +-----+ | | 537 | +---+ | |>-- DetNet Service -->| | +---+ | 538 | +-------------------Flow-------------------+ | 539 | DetTrans | | | | | | | | DetTrans | 540 +---------------+ +--------+ +--------+ +---------------+ 541 DetNet Deterministic DetNet 542 End System Routers and Bridges End System 544 Figure 4: Load Sharing 546 In order to achieve this function, a Load Distribution function is 547 required at the source and a Re-Ordering Function is required at the 548 destination DetTrans End Point. 550 4.3.2.3. Flow Aggregation 552 Flow Aggregation refers to the encapsulation of more than one DetNet 553 flows in one DetNet Flow, for instance using one large and long-lived 554 DetNet Service Flow from a third party provider to carry multiple 555 more dynamic Deterministic End-to-End Service Flows across domains. 556 Packets are sequenced at the DetTrans Layer and distributed over the 557 DetNet Transports paths in accordance to their relative capacities. 558 In case of inconsistent jitter and Latency characteristics, packets 559 may need to be reordered at the receiving DetTrans Layer based on the 560 DSF Sequence. 562 +---------------+ >-- DetNet Service --> +---------------+ 563 | Application | Flow | Application | 564 +-----v---v-----+ +--------+ +--------+ +-----^---^-----+ 565 | | | | | | | | | | | | 566 | | +--------------------------------------------------+ | | 567 | | >------- DetNet End-to-End Service Flow ---------> | | 568 | +----------------------------------------------------------+ | 569 | >----------- DetNet End-to-End Service Flow -------------> | 570 | DetTrans | | | | | | | DetTrans | 571 +---------------+ +--------+ +--------+ +---------------+ 572 DetNet Deterministic DetNet 573 End System Routers and Bridges End System 575 Figure 5: Flow Aggregation 577 In order to achieve this function, a multiplexing function is 578 required at the source and a demultiplexing function is required at 579 the destination DetTrans End Point. 581 4.3.3. Deterministic Service Protection 583 4.3.3.1. PRE vs. 1+1 Redundancy 585 The DetNet Flows may also be used for Packet Replication and 586 Elimination, in which case an elimination function is required at the 587 DetTrans Termination. 589 1+1 Redundancy refers to injecting identical copies of a packet at 590 the ingress of two non-congruent paths, and eliminating the excess 591 copy when both are received at the egress of the paths. Packet 592 Replication and Elimination extends the concept by enabling more than 593 2 paths, and allowing non-end-to-end redundant paths with 594 intermediate Replication and Elimination points. 596 4.3.3.2. Network Coding 598 Redundancy and Load Sharing may be combined with the use of Network 599 Coding whereby a coded packet may carry redundancy information for 600 previous data packet and cover the loss of one, in which case the 601 recovery function is required at the other DetTrans End Point. 602 Network Coding provides a Forward Error Correction between multiple 603 packets or multiple fragments of a packet. It may be used at the DSF 604 layer to enable an efficient combination of redundancy and load 605 sharing. 607 4.3.3.3. Multipath DetTrans Services 609 A DetTrans Flow may leverage multiple DetNet Flows in parallel in 610 order to achieve its requirements in terms of reliability and 611 Aggregate throughput. The "Deterministic Networking Architecture" 612 [I-D.ietf-detnet-architecture] clearly states that the capability of 613 Replication and Elimination is not limited to the DetNet End Systems. 614 DetNet Relay Nodes that operate DetTrans but then relay the packets 615 are needed when the DetTrans operations are not end-to-end. 617 It may be that the DetTrans flow may need to traverse different 618 domains where those Services are operated differently, e.g. 619 controlled by different controllers or leveraging different 620 technologies. It may also be that the bandwidth that is required is 621 only available one segemnt at a time, and that for each segment, a 622 different number of DetNet flows must be setup to transport the full 623 amount of the DetTrans flow. 625 Figure 6 illustrates an example of the latter case, whereby The 626 DetTrans Flow is distributed over two DetNet Flows, maybe operating 627 PRE, then over three DetNet Flows, for instance operating Network 628 Coding between them but using a smaller banswidth for each flow, and 629 then two DetNet Flows again. 631 DetTrans is needed at the interconnection points to adapt the flows, 632 recover losses and reinject the appropriate rates in the next 633 segment. 635 +---+-----------+ +--------+ +--------+ 636 | C |Application| +------------------------------+ 637 | P +-----------+ | |>- DetNet Transport ->| | +---------------+ 638 | U | DetTrans | | +--------+ +--------+ | | DetTrans / | 639 +---+-----------+ | | | ------- | 640 | N | Packet +-----+ +--------+ +--------+ | | / DetTrans | 641 | I | Queues -+ | |>- DetNet Transport ->| +--------+---+---+ | 642 | C | +---------------------------------------------+---+---+ | 643 +---+-----------+ +--------+ +--------+ +----|---|---|--+ 644 | | | 645 DetNet | | | 646 Transport | | | 647 | | | 648 +---+-----------+ +--------+ +--------+ | | | 649 | C |Application| +------------------------------+ | | | 650 | P +-----------+ | |<- DetNet Transport -<| | +----|---|---|--+ 651 | U | DetTrans | | +--------+ +--------+ +--------+---+---+ | 652 +---+-----------+ | +--------+---+---+ | 653 | N | +-----+ +--------+ +--------+ | | \ DetTrans | 654 | I | <-+ | |<- DetNet Transport -<| | | ------- | 655 | C | +------------------------------------+ | DetTrans \ | 656 +---+-----------+ +--------+ +--------+ +---------------+ 657 DetNet-Aware Deterministic Deterministic 658 Host Systems Transit Nodes Relay Nodes 660 Figure 6: Intermediate Systems 662 5. The DetNet-UNI 664 Figure 7 illustrates a simple example of classical networked devices 665 implementing the DetNet architecture. In that example, applications 666 reside on Host systems and run on main CPUs; DetTrans is collocated 667 with its applications and provides them with a Deterministic Service 668 through DetTrans APIs. NICs provides the connectivity to the 669 Deterministic Routers or Bridges acting at DetNet Edge and Relay 670 Nodes - say as an example that they are IEEE Std. 802.1 TSN Bridges. 672 Deterministic 673 Host System Routers and Bridges Host System 674 +---+-----------+ | +--------+ +--------+ | +---+-----------+ 675 | C |Application| |---| |----| |---| | C |Application| 676 | P +-----------+ | |<- DetNet Transport ->|-------+ P +-----------+ 677 | U | DetTrans | | +--------+ +--------+ | | U | DetTrans | 678 +---+-----------+ <------------ DetTrans ------------> +---+-----------+ 679 | N | Lower | | +--------+ +--------+ | | N | Lower | 680 | I | Layers |---| |<- DetNet Transport ->| |---| I | Layers | 681 | C | (queues) | |---| |----| |---| | C | | 682 +---+-----------+ | +--------+ +--------+ | +---+-----------+ 684 <-UNI-> <-- DetNet Services -> <-UNI-> 686 <------------- DetNet End-To-End Services ------------------> 688 Figure 7: Example Physical Network 690 The DetTrans Layer aggregates the data coming from the application up 691 to a maximum frame size that is part of the SLA with the DetNet 692 Transport. Packets thus formed can be distributed over any of 693 multiple DetNet Transport sessions that are defined to accept that 694 packet size. Packets formed at the DetTrans Layer are queued and 695 ready to be delivered through the DetNet-UNI either with a Rate-Based 696 or a Network-Pull mechanism. 698 If the NIC is DetNet-Aware then the queue can be offboarded to the 699 NIC and it can be drained with a time gate (Rate-Base) or a message- 700 driven gate (Network-Pull). Else, the queue is handled by the CPU 701 and hopefully it can be drained within an interrupt, either for a 702 timer (Rate-Base) or for a message (Network-Pull). 704 The DetNet-UNI protocol enables the DetNet transport ingress point to 705 control when the DetTrans Layer transmits its Data packets. It may 706 happen that the DetTrans Layer has not formed a fully-sized packet 707 when time comes for sending it, in which case the packet will be sent 708 with a size below the maximum. 710 The DetNet UNI uses ICMPv6 to carry its protocol elements. Data 711 Packets across the UNI are encapsulated in order to carry DetNet-UNI 712 control information to identify the reason of a loss or a delay, and 713 determine the action to be taken in case of a packet lost or delayed 714 over the interface. 716 5.1. Local Loop Flow Control 718 5.1.1. Dichotomy of a DetNet End System 720 The logical DetNet End System depicted in Figure 2 comprises several 721 elements which may implemented in one or separate physical Systems. 722 The example dichotomy in Figure 3 segregates ingress shaping and 723 DetNet Relay functions, which are performed by IEEE Std. 802.1 TSN 724 Bridges, from a DetNet-Aware Host. 726 Hosts and Edge Bridges are connected over Ethernet and together they 727 form a DetNet End System. As it goes, this example introduces a 728 further dichotomy within the Host, between the CPU and the NIC, 729 across a local bus such as PCI, as illustrated in Figure 8. 731 +-------------+ +-------+ +-----------+ 732 | Application | | MAC | | Ingress | 733 +-------------+ <--PCI--> +-------+ <---- Ethernet ----> | Shaping | 734 | DetTrans | | PHY | | and Relay | 735 +-------------+ +-------+ +-----------+ 736 CPU NIC IEEE Std. 737 <------------- DetNet-UNI -------------> 802.1 738 <----------- Host System -------------> TSN Bridge 739 <------------------------- DetNet End System ------------------------> 741 Figure 8: Chained Functions 743 The NICs in the Host System may not participate to the network time 744 Synchronization and may not be aware of the DetNet protocols running 745 between the Deterministic Routers and Bridges, and the associated 746 scheduling rules. In that situation, the DetNet-UNI operates on a 747 Local Loop to ensure that a packet that leaves the Transport reaches 748 the Router or Bridge just in time for injection into the 749 Deterministic data plane and to provide a flow control that avoids 750 congestion loss at the interface. 752 It is also possible that the NIC participates to the Deterministic 753 Network but still has asynchronous communication with DetTrans Layer 754 running on the the CPU. Either way, a flow control over a local loop 755 must be implemented to drain the queues from the DetTrans layer and 756 feed the network just in time for the deterministic transmission. 758 Depending on the level of support by the NIC, the loop may be placed 759 on a different interface but remains functionally the same. 761 5.1.2. Local Loop Location 763 If the NIC is not aware at all of DetNet, then it is a plain pipe for 764 the Deterministic Traffic. The Local Loop operates between the Edge 765 TSN Bridge and the CPU as illustrated in Figure 9. 767 +---+-----------+ +---+-------+ +-----------+ 768 | C |Application| | N | MAC | | Ingress | 769 | P +-----------+ <--PCI--> | I +-------+ <-- Ethernet --> | Shaping | 770 | U |DetTrans | | C | PHY | (Bridged | / Transit | 771 +---+-----------+ +---+-------+ or P2P) +-----------+ 773 <------------- Local Loop -------------> Edge 802.1 774 <----------- Host System -------------> TSN Bridge 776 Figure 9: DetNet Unaware NIC 778 If the NIC is fully DetNet-Capable and participates to the 779 deterministic Network including time synchronization and scheduling, 780 then the local loop operates between the CPU and the NIC as 781 illustrated in Figure 10. 783 +---+-----------+ +---+-----------+ +----------+ 784 | C |Application| | N | Ingress | | DetNet | 785 | P +-----------+ <--PCI--> | I + Shaping | <--DetNet--> | Transit | 786 | U |DetTrans | | C | / Transit | Transport | Node | 787 +---+-----------+ +---+-----------+ +----------+ 788 <-Local- 789 -Loop -> 790 <-------------- Host System --------------> TSN Bridge 792 Figure 10: DetNet Capable NIC 794 If the NIC is DetNet-Aware and does not participates to the 795 deterministic Network including time synchronization and scheduling, 796 then there are two local loops, one that operates between the CPU and 797 the NIC and one that operates between the NIC and the Edge TSN Bridge 798 as illustrated in Figure 11. 800 +---+-----------+ +---+-------+ +-----------+ 801 | C |Application| | N | MAC | Ethernet | Ingress | 802 | P +-----------+ <--PCI--> | I +-------+ <-- (Bridged --> | Shaping | 803 | U |DetTrans | | C | PHY | or P2P) | / Transit | 804 +---+-----------+ <-Local- +---+-------+ +-----------+ 805 -Loop -> <-- Local Loop --> Edge 806 <----------- Host System -------------> DetNet Transit 808 Figure 11: DetNet Capable NIC 810 5.1.3. Network Pull vs. Rate Based Flow Control 812 The flow control at the DetNet-UNI can take any of two forms: 814 Network Pull In that Model, the DetNet Edge node drains the DetTrans 815 queue by sending a DetNet-UNI "More" command some estimated amount 816 of time ahead of the scheduled time of transmission for each 817 packet; in case of load sharing, multiple DetNet Edge nodes may 818 drain a queue at their own rates; in case of a high jitter on the 819 UNI Local Loop (e.g. there is a non-deterministic Bridge in 820 between, or the NIC is not DetNet-Aware and the flows suffer from 821 the more erratic response time of the CPU), the DetNet Edge node 822 may need to pull a window of packets to maintain its own 823 transmission queues fed at all times 825 Rate Based In that model, the NIC is aware of the rate of the 826 deterministic transmission and is drained by its internal timers. 827 Since the NIC is not synchronized with the Deterministic Network, 828 the Bridge uses a DetNet-UNI "Time-Correction" command 829 asynchronously to move forward or backward the next timeout of the 830 NIC for that flow, in order to keep the Rate-Based transmission by 831 the NIC in rough alignment with the scheduled transmission over 832 the DetNet network. 834 if the NIC is DetNet-Aware, it is expected that it maintains the 835 DetTrans queues in order to provide a deterministic response to the 836 DetNet-UNI, and in that case another control loop between the NIC and 837 the CPU is needed to ensure that the queue in the NIC is always fed 838 in time by the DetTrans Layer; this second loop may be of a different 839 nature than the DetNet-UNI one and may for instance be operated 840 within an interrupt to limit the asynchronism related to message 841 queueing. 843 5.2. DetNet-UNI Protocol Exchanges 845 5.2.1. the "More" Message 847 The "More" message enables a DetNet Transport Edge to pull one packet 848 from the DetTrans Layer in Network-Pull mode. The message is 849 associated with a future transmission opportunity for a packet. The 850 "More" messages are indexed by a wrapping More Sequence Counter 851 (MSC). The Transport Edge also maintains wrapping counters of 852 Successful Packet Transmissions (SPT) and Missed Transmit 853 Opportunities (MTO). The current value of these counters is placed 854 in the "More "message. 856 Upon reception of a "More" message, the DetTrans Layer, or the NIC on 857 behalf of the DetTrans Layer, sends the next available packet for 858 that session. The packet is encapsulated and the encapsulation 859 indicates the MSC. This enables the DetNet Transport Edge to 860 correlate the packet with the transmission opportunity and drop 861 packets that are overly delayed. 863 5.2.2. the "Time-Correction" Message 865 The "Time-Correction" message enables a DetNet Transport Edge to 866 adjust the timer associated to the DetNet-UNI session in Rate-Based 867 mode. In that mode, the DetTrans Layer sends a packet and restarts a 868 timer at a period that corresponds to the transmission opportunity of 869 the DetNet Transport Edge. If the clock in the CPU drifts, the 870 DetNet Transport Edge will start receiving packets increasingly ahead 871 of expected time or behind expected time. It is expected that the 872 DetNet Transport Edge is protected against a minimum drift by a guard 873 time, but if the drift becomes too important, then the DetNet 874 Transport Edge issues a "Time-Correction" message indicating a number 875 of time units (e.g. microseconds) by which the DetTrans Layer should 876 advance or delay is next time out. 878 5.2.3. Loss of a Control Message 880 The loss of a packet beween the DetTrans Layer and the DetNet 881 Transport Edge will correspond to a missed Transmission Opportunity 882 but this does not mean that packets are piling up at the DetTrans 883 Layer. OTOH, if a "More" message is lost, then one packet will not 884 be dequeued and the Detrans queue might grow, increasingly augmenting 885 the latency. It is thus important to differentiate these situations, 886 and in the latter case, discard an extraneous packet to restore the 887 normal level in the DetTrans queue for that session. 889 In order to do so, the DetTrans Layer maintains the record of the 890 Number of Packets Sent (NPS), that it compares with the variation of 891 the MTO and SPT counters in the "More" message. Upon a "More" 892 message, the DetTrans Layer computes the variation of NPS 893 (dNPS=NPS2-NPS1) and the variation of SPT (dSPT=SPT2-SPT1) since the 894 previous "More" Message . dNPS is typically 1 if the transport 895 always has data to send. Packets in flight when the "More" message 896 is sent are considered lost since they will be received after their 897 scheduled transmission opportunity, so the Number of Packets Losses 898 (NPL) is (NPL=dNPS-dSPT). The DetTrans Layer also computes the 899 variation of MTO since the previous "More" Message (dMTO=MTO2-MTO1). 900 Since a packet loss implies a missed transmission opportunity, there 901 cannot be more packets losses than missed opportunities, so we have 902 dMTO>=NPL. dMTO-NPL represents the number of missed opportunities 903 that are not due to a packet lost or late arrival, thus this is the 904 sub-count of MTOs due to the loss of a "More" message. 906 For each loss of a "More" message, a packet in the DetTrans queue 907 should be discarded. In order to simplify that operation and 908 outboard it to the NIC, the Transports marks some packets as "Discard 909 Eligible" (DE). A packet can be marked DE if there are enough 910 alternate transmissions of non-DE packets to recover this. For 911 instance, in case of Packet Replication and Elimination only one copy 912 can be marked DE, and the marking should alternate between the 913 sessions to cover a loss on either one rapidly. 915 6. Security Considerations 917 The generic threats against Deterministic Networking are discussed in 918 the "Deterministic Networking Security" [I-D.ietf-detnet-security] 919 document. 921 Security in the context of Deterministic Networking has an added 922 dimension; the time of delivery of a packet can be just as important 923 as the contents of the packet, itself. A man-in-the-middle attack, 924 for example, can impose, and then systematically adjust, additional 925 delays into a link, and thus disrupt or subvert a real-time 926 application without having to crack any encryption methods employed. 927 See [RFC7384] for an exploration of this issue in a related context. 929 Packet Replication and Elimination of done right can prevent a man- 930 in-the-middle attack on one leg to actually impact the flow beyond 931 the loss of an individual packet for lack of redundancy. This 932 specification expects that PRE is performed at the transport level 933 and provides specific means to protect one leg against misuse of the 934 other. 936 7. IANA Considerations 938 This document does not require an action from IANA. 940 8. Acknowledgments 942 The authors wish to thank Patrick Wetterwald, Leon Turkevitch, Balazs 943 Varga and Janos Farkas for their various contributions to this work. 944 Special thanks to Norm Finn for being a (if not the) major thought 945 leader to the whole deterministic effort, and for some text that is 946 inlined here from other IETF documents, for the convenience of the 947 reader. 949 9. Informative References 951 [AVnu] http://www.avnu.org/, "The AVnu Alliance tests and 952 certifies devices for interoperability, providing a simple 953 and reliable networking solution for AV network 954 implementation based on the IEEE Audio Video Bridging 955 (AVB) and Time-Sensitive Networking (TSN) standards.". 957 [EIP] http://www.odva.org/, "EtherNet/IP provides users with the 958 network tools to deploy standard Ethernet technology (IEEE 959 802.3 combined with the TCP/IP Suite) for industrial 960 automation applications while enabling Internet and 961 enterprise connectivity data anytime, anywhere.", 962 . 966 [HART] www.hartcomm.org, "Highway Addressable Remote Transducer, 967 a group of specifications for industrial process and 968 control devices administered by the HART Foundation". 970 [I-D.ietf-detnet-architecture] 971 Finn, N., Thubert, P., Varga, B., and J. Farkas, 972 "Deterministic Networking Architecture", draft-ietf- 973 detnet-architecture-03 (work in progress), August 2017. 975 [I-D.ietf-detnet-problem-statement] 976 Finn, N. and P. Thubert, "Deterministic Networking Problem 977 Statement", draft-ietf-detnet-problem-statement-02 (work 978 in progress), September 2017. 980 [I-D.ietf-detnet-security] 981 Mizrahi, T., Grossman, E., Hacker, A., Das, S., Dowdell, 982 J., Austad, H., Stanton, K., and N. Finn, "Deterministic 983 Networking (DetNet) Security Considerations", draft-ietf- 984 detnet-security-00 (work in progress), October 2017. 986 [I-D.ietf-detnet-use-cases] 987 Grossman, E., Gunther, C., Thubert, P., Wetterwald, P., 988 Raymond, J., Korhonen, J., Kaneko, Y., Das, S., Zha, Y., 989 Varga, B., Farkas, J., Goetz, F., Schmitt, J., Vilajosana, 990 X., Mahmoodi, T., Spirou, S., Vizarreta, P., Huang, D., 991 Geng, X., Dujovne, D., and M. Seewald, "Deterministic 992 Networking Use Cases", draft-ietf-detnet-use-cases-13 993 (work in progress), September 2017. 995 [I-D.varga-detnet-service-model] 996 Varga, B. and J. Farkas, "DetNet Service Model", draft- 997 varga-detnet-service-model-02 (work in progress), May 998 2017. 1000 [IEC62439] 1001 IEC, "Industrial communication networks - High 1002 availability automation networks - Part 3: Parallel 1003 Redundancy Protocol (PRP) and High-availability Seamless 1004 Redundancy (HSR) - IEC62439-3", 2012, 1005 . 1007 [IEEE802.1TSNTG] 1008 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1009 Networks Task Group", 2013, 1010 . 1012 [ISA100.11a] 1013 ISA/IEC, "ISA100.11a, Wireless Systems for Automation, 1014 also IEC 62734", 2011, < http://www.isa100wci.org/en- 1015 US/Documents/PDF/3405-ISA100-WirelessSystems-Future-broch- 1016 WEB-ETSI.aspx>. 1018 [ODVA] http://www.odva.org/, "The organization that supports 1019 network technologies built on the Common Industrial 1020 Protocol (CIP) including EtherNet/IP.". 1022 [Profinet] 1023 http://us.profinet.com/technology/profinet/, "PROFINET is 1024 a standard for industrial networking in automation.", 1025 . 1027 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1028 Requirement Levels", BCP 14, RFC 2119, 1029 DOI 10.17487/RFC2119, March 1997, 1030 . 1032 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1033 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1034 October 2014, . 1036 [WirelessHART] 1037 www.hartcomm.org, "Industrial Communication Networks - 1038 Wireless Communication Network and Communication Profiles 1039 - WirelessHART - IEC 62591", 2010. 1041 Author's Address 1042 Pascal Thubert (editor) 1043 Cisco Systems, Inc 1044 Building D (Regus) 45 Allee des Ormes 1045 MOUGINS - Sophia Antipolis 1046 FRANCE 1048 Phone: +33 4 97 23 26 34 1049 Email: pthubert@cisco.com