idnits 2.17.1 draft-finn-detnet-bounded-latency-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 2, 2018) is 2124 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '2' on line 236 -- Looks like a reference, but probably isn't: '4' on line 236 == Unused Reference: 'I-D.ietf-detnet-dp-alt' is defined on line 770, but no explicit reference was found in the text == Unused Reference: 'RFC4364' is defined on line 790, but no explicit reference was found in the text == Unused Reference: 'RFC6658' is defined on line 794, but no explicit reference was found in the text == Unused Reference: 'IEEE8021TSN' is defined on line 844, but no explicit reference was found in the text == Unused Reference: 'IEEE8023' is defined on line 848, but no explicit reference was found in the text == Outdated reference: A later version (-13) exists of draft-ietf-detnet-architecture-00 ** Downref: Normative reference to an Informational draft: draft-ietf-detnet-dp-alt (ref. 'I-D.ietf-detnet-dp-alt') == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-16 ** Downref: Normative reference to an Informational draft: draft-ietf-detnet-use-cases (ref. 'I-D.ietf-detnet-use-cases') ** Downref: Normative reference to an Informational RFC: RFC 7806 Summary: 5 errors (**), 0 flaws (~~), 8 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet N. Finn 3 Internet-Draft Huawei Technologies Co. Ltd 4 Intended status: Standards Track J-Y. Le Boudec 5 Expires: January 3, 2019 E. Mohammadpour 6 EPFL 7 B. Varga 8 J. Farkas 9 Ericsson 10 July 2, 2018 12 DetNet Bounded Latency 13 draft-finn-detnet-bounded-latency-01 15 Abstract 17 This document presents a parameterized timing model for Deterministic 18 Networking so that existing and future standards can achieve bounded 19 latency and zero congestion loss. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on January 3, 2019. 38 Copyright Notice 40 Copyright (c) 2018 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (https://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 56 2. Conventions Used in This Document . . . . . . . . . . . . . . 3 57 3. Terminology and Definitions . . . . . . . . . . . . . . . . . 4 58 4. DetNet bounded latency model . . . . . . . . . . . . . . . . 4 59 4.1. Flow creation . . . . . . . . . . . . . . . . . . . . . . 4 60 4.2. End-to-end model . . . . . . . . . . . . . . . . . . . . 5 61 4.3. Relay system model . . . . . . . . . . . . . . . . . . . 5 62 5. Computing End-to-end Latency Bounds . . . . . . . . . . . . . 7 63 5.1. Examples of Computations . . . . . . . . . . . . . . . . 8 64 5.1.1. Per-flow queuing . . . . . . . . . . . . . . . . . . 8 65 5.1.2. Time-Sensitive Networking with Asynchronous Traffic 66 Shaping . . . . . . . . . . . . . . . . . . . . . . . 8 67 6. Achieving zero congestion loss . . . . . . . . . . . . . . . 9 68 6.1. A General Formula . . . . . . . . . . . . . . . . . . . . 9 69 7. Queuing model . . . . . . . . . . . . . . . . . . . . . . . . 10 70 7.1. Queuing data model . . . . . . . . . . . . . . . . . . . 10 71 7.2. IEEE 802.1 Queuing Model . . . . . . . . . . . . . . . . 12 72 7.2.1. Queuing Data Model with Preemption . . . . . . . . . 12 73 7.2.2. Transmission Selection Model . . . . . . . . . . . . 13 74 7.3. Time-Sensitive Networking with Asynchronous Traffic 75 Shaping . . . . . . . . . . . . . . . . . . . . . . . . . 15 76 7.4. Other queuing models, e.g. IntServ . . . . . . . . . . . 17 77 8. Parameters for the bounded latency model . . . . . . . . . . 17 78 8.1. Sender parameters . . . . . . . . . . . . . . . . . . . . 17 79 8.2. Relay system parameters . . . . . . . . . . . . . . . . . 17 80 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 17 81 9.1. Normative References . . . . . . . . . . . . . . . . . . 17 82 9.2. Informative References . . . . . . . . . . . . . . . . . 18 83 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 85 1. Introduction 87 The ability for IETF Deterministic Networking (DetNet) or IEEE 802.1 88 Time-Sensitive Networking (TSN) to provide the DetNet services of 89 bounded latency and zero congestion loss depends upon A) configuring 90 and allocating network resources for the exclusive use of DetNet/TSN 91 flows; B) identifying, in the data plane, the resources to be 92 utilized by any given packet, and C) the detailed behavior of those 93 resources, especially transmission queue selection, so that latency 94 bounds can be reliably assured. Thus, DetNet is an example of an 95 INTSERV Guaranteed Quality of Service [RFC2212] 96 As explained in [I-D.ietf-detnet-architecture], DetNet flows are 97 characterized by 1) a maximum bandwidth, guaranteed either by the 98 transmitter or by strict input metering; and 2) a requirement for a 99 guaranteed worst-case end-to-end latency. That latency guarantee, in 100 turn, provides the opportunity for the network to supply enough 101 buffer space to guarantee zero congestion loss. To be of use to the 102 applications identified in [I-D.ietf-detnet-use-cases], it must be 103 possible to calculate, before the transmission of a DetNet flow 104 commences, both the worst-case end-to-end network latency, and the 105 amount of buffer space required at each hop to ensure against 106 congestion loss. 108 Rather than defining, in great detail, specific mechanisms to be used 109 to control packet transmission at each output port, this document 110 presents a timing model for sources, destinations, and the network 111 nodes that relay packets. The parameters specified in this model: 113 o Characterize a DetNet flow in a way that provides externally 114 measureable verification that the sender is conforming to its 115 promised maximum, can be implemented reasonably easily by a 116 sending device, and does not require excessive over-allocation of 117 resources by the network. 119 o Enable resonably accurate computation of worst-case end-to-end 120 latency, in a way that requires as little detailed knowledge as 121 possible of the behavior of the Quality of Service (QoS) 122 algorithms implemented in each devince, including queuing, 123 shaping, metering, policing, and transmission selection 124 techniques. 126 Using the model presented in this document, it should be possible for 127 an implementor, user, or standards development organization to select 128 a particular set of QoS algorithms for each device in a DetNet 129 network, and to select a resource reservation algorithm for that 130 network, so that those elements can work together to provide the 131 DetNet service. 133 This document does not specify any resource reservation protocol or 134 server. It does not describe all of the requirements for that 135 protocol or server. It does describe a set of requirements for 136 resource reservation algorithms and for QoS algorithms that, if met, 137 will enable them to work together. 139 2. Conventions Used in This Document 141 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 142 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 143 document are to be interpreted as described in [RFC2119]. 145 The lowercase forms with an initial capital "Must", "Must Not", 146 "Shall", "Shall Not", "Should", "Should Not", "May", and "Optional" 147 in this document are to be interpreted in the sense defined in 148 [RFC2119], but are used where the normative behavior is defined in 149 documents published by SDOs other than the IETF. 151 3. Terminology and Definitions 153 This document uses the terms defined in 154 [I-D.ietf-detnet-architecture]. 156 4. DetNet bounded latency model 158 4.1. Flow creation 160 The bounded latency model assusmes the use of the following paradigm 161 for provisioning a particular DetNet flow: 163 1. Perform any onfiguration required by the relay systems in the 164 network for the classes of service to be offered, including one 165 or more classes of DetNet service. This configuration is 166 general; it is not tied to any particular flow. 168 2. Characterize the DetNet flow in terms of limitations on the 169 sender Section 8.1 and flow requirements Section 8.2. 171 3. Establish the path that the DetNet flow will take through the 172 network from the source to the destination(s). This can be a 173 point-to-point or a point-to-multipoint path. 175 4. Select one of the DetNet classes of service for the DetNet flow. 177 5. Compute the worst-case end-to-end latency for the DetNet flow. 178 In the process, determine whether sufficient resources are 179 available for that flow to guarantee the required latency and 180 provide zero congestion loss. 182 6. Assuming that the resources are available, commit those resources 183 to the flow. This may or may not require adjusting the 184 parameters that control the QoS algorithms at each hop along the 185 flow's path. 187 This paradigm can be static and/or dynamic, and can be implemented 188 using peer-to-peer protocols or with a central server model. In some 189 situations, backtracking and recursing through this list may be 190 necessary. 192 Issues such as un-provisioning a DetNet flow in favor of another when 193 resources are scarce are not considered. How the path to be taken by 194 a DetNet flow is chosen is not considered in this document. 196 4.2. End-to-end model 198 [Suggestion: This is the introduction to network calculus. The 199 starting point is a model in which a relay system is a black box.] 201 4.3. Relay system model 203 [NWF I think that at least some of this will be useful. We won't 204 know until we see what J-Y has to say in Section 4.2. I'm especially 205 interested in whether J-Y thinks that the "output delay" in Figure 1 206 is useful in determining the number of buffers needed in the next 207 hop. It is possible that we can define the parameters we need 208 without this section.] 210 In Figure 1 we see a breakdown of the per-hop latency experienced by 211 a packet passing through a relay system, in terms that are suitable 212 for computing both hop-by-hop latency and per-hop buffer 213 requirements. 215 DetNet relay node A DetNet relay node B 216 +-------------------+ +-------------------+ 217 | Reg. Queue | | Reg. Queue | 218 | +-+-+ +-+-+-+ | | +-+-+ +-+-+-+ | 219 -->+ | | | | | | + +------->+ | | | | | | + +---> 220 | +-+-+ +-+-+-+ | | +-+-+ +-+-+-+ | 221 | | | | 222 +-------------------+ +-------------------+ 223 |<->|<-->|<---->|<->|<------>|<->|<-->|<---->|<->|<-- 224 2,3 4 5 6 1 2,3 4 5 6 1 2,3 225 1: Output delay 3: Preemption delay 226 2: Link delay 4: Processing delay 227 5: Regulation delay 6: Queuing delay. 229 Figure 1: Timing model for DetNet or TSN 231 In Figure 1, we see two DetNet relay nodes (typically, bridges or 232 routers), with a wired link between them. In this model, the only 233 queues we deal with explicitly are attached to the output port; other 234 queues are modeled as variations in the other delay times. (E.g., an 235 input queue could be modeled as either a variation in the link delay 236 [2] or the processing delay [4].) There are five delays that a 237 packet can experience from hop to hop. 239 1. Output delay 240 The time taken from the selection of a packet for output from a 241 queue to the transmission of the first bit of the packet on the 242 physical link. If the queue is directly attached to the physical 243 port, output delay can be a constant. But, in many 244 implementations, the queuing mechanism in a forwarding ASIC is 245 separated from a multi-port MAC/PHY, in a second ASIC, by a 246 multiplexed connection. This causes variations in the output 247 delay that are hard for the forwarding node to predict or control. 249 2. Link delay 250 The time taken from the transmission of the first bit of the 251 packet to the reception of the last bit, assuming that the 252 transmission is not suspended by a preemption event. This delay 253 has two components, the first-bit-out to first-bit-in delay and 254 the first-bit-in to last-bit-in delay that varies with packet 255 size. The former is typically measured by the Precision Time 256 Protocol and is constant (see [I-D.ietf-detnet-architecture]). 257 However, a virtual "link" could exhibit a variable link delay. 259 3. Preemption delay 260 If the packet is interrupted (e.g. [IEEE8023br] preemption) in 261 order to transmit another packet or packets, an arbitrary delay 262 can result. 264 4. Processing delay 265 This delay covers the time from the reception of the last bit of 266 the packet to that packet being eligible, if there were no other 267 packets in the queue, for selection for output. This delay can be 268 variable, and depends on the details of the operation of the 269 forwarding node. 271 5. Regulation delay 272 This is the time spent from the insertion of the packet into a 273 regulation queue until the time the packet is declared eligible 274 according to its regulation constraints. We assume that this time 275 can be calculated based on the details of regulation policy. If 276 there is no regulation, this time is zero. 278 6. Queuing delay 279 This is the time spent for a packet from being declared eligibile 280 until being selected for output on the next link. We assume that 281 this time is calculable based on the details of the queuing 282 mechanism. If there is no regulation, this time is from the 283 insertion of the packet into a queue until it is selected for 284 output on the next link. 286 Not shown in Figure 1 are the other output queues that we presume are 287 also attached to that same output port as the queue shown, and 288 against which this shown queue competes for transmission 289 opportunities. 291 The initial and final measurement point in this analysis (that is, 292 the definition of a "hop") is the point at which a packet is selected 293 for output. In general, any queue selection method that is suitable 294 for use in a DetNet network includes a detailed specification as to 295 exactly when packets are selected for transmission. Any variations 296 in any of the delay times 1-4 result in a need for additional buffers 297 in the queue. If all delays 1-4 are constant, then any variation in 298 the time at which packets are inserted into a queue depends entirely 299 on the timing of packet selection in the previous node. If the 300 delays 1-4 are not constant, then additional buffers are required in 301 the queue to absorb these variations. Thus: 303 o Variations in output delay (1) require buffers to absorb that 304 variation in the next hop, so the output delay variations of the 305 previous hop (on each input port) must be known in order to 306 calculate the buffer space required on this hop. 308 o Variations in processing delay (4) require additional output 309 buffers in the queues of that same Detnet relay node. Depending 310 on the details of the queueing delay (6) calculations, these 311 variations need not be visible outside the DetNet relay node. 313 5. Computing End-to-end Latency Bounds 315 End-to-end latency bounds can be computed using the delay model in 316 Section 4.3. Here it is important to be aware that for several 317 queuing mechanisms, the worst-case end-to-end delay is less than the 318 sum of the per-hop worst-case delays. An end-to-end latency bound 319 for one detnet flow can be computed as 321 end_to_end_latency_bound = non_queuing_latency + queuing_latency 323 The two terms in the above formula are computed as follows. First, 324 at the h-th hop along the path of this detnet flow, obtain an upper 325 bound per-hop_non_queuing_latency[h] on the sum of delays 1,2,3,4 of 326 Figure 1. These upper-bounds are expected to depend on the specific 327 technology of the node at the h-th hop but not on the T-SPEC of this 328 detnet flow. Then set non_queuing_latency = the sum of per- 329 hop_non_queuing_latency[h] over all hops h. 331 Second, compute queuing_latency as an upper bound to the sum of the 332 queuing delays along the path. The value of queuing_latency depends 333 on the T-SPEC of this flow and possibly of other flows in the 334 network, as well as the specifics of the queuing mechanisms deployed 335 along the path of this flow. 337 For several queuing mechanisms, queuing_latency is less than the sum 338 of upper bounds on the queuing delays (5,6) at every hop. 339 Section 5.1 gives such practical computation examples. 341 For other queuing mechanisms the only available value of 342 queuing_latency is the sum of the per-hop queuing delay bounds. In 343 such cases, the computation of per-hop queuing delay bounds must 344 account for the fact that the T-SPEC of a detnet flow is no longer 345 satisfied at the ingress of a hop, since burstiness increases as one 346 flow traverses one detnet node. 348 5.1. Examples of Computations 350 5.1.1. Per-flow queuing 352 [[ JYLB: THIS IS WHERE DETAILS OF END-TO-END LATENCY COMPUTATION ARE 353 GIVEN FOR PER-FLOW QUEUING]] 355 5.1.2. Time-Sensitive Networking with Asynchronous Traffic Shaping 357 Figure 2 shows an example of a network with 5 nodes, which have the 358 queuing model as Section 7.3. An end-to-end delay bound for flow f 359 of a given AVB class (A or B), traversing from node 1 to 5, is 360 calculated as following: 362 end_to_end_latency_bound_of_flow_f = C12 + C23 + C34 + S4 364 In the above formula, Cij is a bound on the aggregate response time 365 of the AVB FIFO queue with CBS (Credit Based Shaper) in node i and 366 interleaved regulator of node j, and S4 is a bound on the response 367 time of the AVB FIFO queue with CBS in node 4 for flow f. In fact, 368 using the delay definitions in Section 4.3, Cij is a bound on sum of 369 the delays 1,2,3,6 of node i and 4,5 of node j. Similarly, S4 is a 370 bound on sum of the delays 1,2,3,6 of node 4. The detail of 371 calculation for the these response time bounds can be found in 372 [TSNwithATS]. 374 f 375 -----------------------------> 376 +---+ +---+ +---+ +---+ +---+ 377 | 1 |---| 2 |---| 3 |---| 4 |---| 5 | 378 +---+ +---+ +---+ +---+ +---+ 379 \__C12_/\__C23_/\__C34_/\_S4_/ 381 Figure 2: End-to-end latency computation example 383 REMARK: The end-to-end delay bound calculation provided here gives a 384 much better upper bound in comparison with end-to-end delay bound 385 computation by adding the delay bounds of each node in the path of a 386 flow [TSNwithATS]. 388 6. Achieving zero congestion loss 390 When the input rate to an output queue exceeds the output rate for a 391 sufficient length of time, the queue must overflow. This is 392 congestion loss, and this is what deterministic networking seeks to 393 avoid. 395 6.1. A General Formula 397 To avoid congestion losses, an upper bound on the backlog present in 398 the queue of Figure 1 must be computed during path computation. This 399 bound depends on the set of flows that use this queue, the details of 400 the specific queuing mechanism and an upper bound on the processing 401 delay (4). The queue must contain the packet in transmission plus 402 all other packets that are waiting to be selected for output. 404 A conservative backlog bound, that applies to all systems, can be 405 derived as follows. 407 The backlog bound is counted in data units (bytes, or words of 408 multiple bytes) that are relevant for buffer allocation. For every 409 class we need one buffer space for the packet in transmission, plus 410 space for the packets that are waiting to be selected for output. 411 Excluding transmission and preemption times, the packets are waiting 412 in the queue since reception of the last bit, for a duration equal to 413 the processing delay (4) plus the queuing delays (5,6). 415 Let 417 o nb_classes be the number of classes of traffic that may use this 418 output port 420 o total_in_rate be the sum of the line rates of all input ports that 421 send traffic of any class to this output port. The value of 422 total_in_rate is in data units (e.g. bytes) per second. 424 o nb_input_ports be the number input ports that send traffic of any 425 class to this output port 427 o max_packet_length be the maximum packet size for packets of any 428 class that may be sent to this output port. This is counted in 429 data units. 431 o max_delay45 be an upper bound, in seconds, on the sum of the 432 processing delay (4) and the queuing delays (5,6) for a packet of 433 any class at this ouput port. 435 Then a bound on the backlog of traffic of all classes in the queue at 436 this output port is 438 backlog_bound = ( nb_classes + nb_input_ports ) * 439 max_packet_length + total_in_rate* max_delay45 441 7. Queuing model 443 [[ JYLB: THIS IS WHERE DETAILS OF END-TO-END LATENCY COMPUTATION ARE 444 GIVEN FOR PER-FLOW QUEUING AND FOR TSN WITH ATS]] 446 7.1. Queuing data model 448 Sophisticated QoS mechanisms are available in Layer 3 (L3), see, 449 e.g., [RFC7806] for an overview. In general, we assume that "Layer 450 3" queues, shapers, meters, etc., are instantiated hierarchically 451 above the "Layer 2" queuing mechanisms, among which packets compete 452 for opportunities to be transmitted on a physical (or sometimes, 453 logical) medium. These "Layer 2 queuing mechanisms" are not the 454 province solely of bridges; they are an essential part of any DetNet 455 relay node. As illustrated by numerous implementation examples, the 456 "Layer 3" some of mechanisms described in documents such as [RFC7806] 457 are often integrated, in an implementation, with the "Layer 2" 458 mechanisms also implemented in the same system. An integrated model 459 is needed in order to successfully predict the interactions among the 460 different queuing mechanisms needed in a network carrying both DetNet 461 flows and non-DetNet flows. 463 Figure 3 shows the (very simple) model for the flow of packets 464 through the queues of an IEEE 802.1Q bridge. Packets are assigned to 465 a class of service. The classes of service are mapped to some number 466 of physical FIFO queues. IEEE 802.1Q allows a maximum of 8 classes 467 of service, but it is more common to implement 2 or 4 queues on most 468 ports. 470 | 471 +--------------V---------------+ 472 | Class of Service Assignment | 473 +--+-------+---------------+---+ 474 | | | 475 +--V--+ +--V--+ +--V--+ 476 |Class| |Class| |Class| 477 | 0 | | 1 | . . . | n | 478 |queue| |queue| |queue| 479 +--+--+ +--+--+ +--+--+ 480 | | | 481 +--V-------V---------------V--+ 482 | Transmission selection | 483 +--------------+--------------+ 484 | 485 V 487 Figure 3: IEEE 802.1Q Queuing Model: Data flow 489 Some relevant mechanisms are hidden in this figure, and are performed 490 in the "Class n queue" box: 492 o Discarding packets because a queue is full. 494 o Discarding packets marked "yellow" by a metering function, in 495 preference to discarding "green" packets. 497 The Class of Service Assignment function can be quite complex, since 498 the introduction of [IEEE802.1Qci]. In addition to the Layer 2 499 priority expressed in the 802.1Q VLAN tag, a bridge can utilize any 500 of the following information to assign a packet to a particular class 501 of service (queue): 503 o Input port. 505 o Selector based on a rotating schedule that starts at regular, 506 time-synchronized intervals and has nanosecond precision. 508 o MAC addresses, VLAN ID, IP addresses, Layer 4 port numbers, DSCP. 509 (Work items expected to add MPC and other indicators.) 511 o The Class of Service Assignment function can contain metering and 512 policing functions. 514 The "Transmission selection" function decides which queue is to 515 transfer its oldest packet to the output port when a transmission 516 opportunity arises. 518 7.2. IEEE 802.1 Queuing Model 520 7.2.1. Queuing Data Model with Preemption 522 Figure 3 must be modified if the output port supports preemption 523 ([IEEE8021Qbu] and [IEEE8023br]). This modification is shown in 524 Figure 4. 526 | 527 +------------------------------V------------------------------+ 528 | Class of Service Assignment | 529 +--+-------+-------+-------+-------+-------+-------+-------+--+ 530 | | | | | | | | 531 +--V--+ +--V--+ +--V--+ +--V--+ +--V--+ +--V--+ +--V--+ +--V--+ 532 |Class| |Class| |Class| |Class| |Class| |Class| |Class| |Class| 533 | a | | b | | c | | d | | e | | f | | g | | h | 534 |queue| |queue| |queue| |queue| |queue| |queue| |queue| |queue| 535 +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ 536 | | | +-+ | | | | 537 | | | | | | | | 538 +--V-------V-------V------+ +V-----V-------V-------V-------V--+ 539 | Interrupted xmit select | | Preempting xmit select | 802.1 540 +-------------+-----------+ +----------------+----------------+ 541 | | ====== 542 +-------------V-----------+ +----------------V----------------+ 543 | Preemptible MAC | | Express MAC | 802.3 544 +--------+----------------+ +----------------+----------------+ 545 | | 546 +--------V-----------------------------------V----------------+ 547 | MAC merge sublayer | 548 +--------------------------+----------------------------------+ 549 | 550 +--------------------------V----------------------------------+ 551 | PHY (unaware of preemption) | 552 +--------------------------+----------------------------------+ 553 | 554 V 556 Figure 4: IEEE 802.1Q Queuing Model: Data flow with preemption 558 From Figure 4, we can see that, in the IEEE 802 model, the preemption 559 feature is modeled as consisting of two MAC/PHY stacks, one for 560 packets that can be interrupted, and one for packets that can 561 interrupt the interruptible packets. The Class of Service (queue) 562 determines which packets are which. In Figure 4, the classes of 563 service are marked "a, b, ..." instead of with numbers, in order to 564 avoid any implication about which numeric Layer 2 priority values 565 correspond to preemptible or preempting queues. Although it shows 566 three queues going to the preemptible MAC/PHY, any assignment is 567 possible. 569 7.2.2. Transmission Selection Model 571 In Figure 5, we expand the "Transmission selection" function of 572 Figure 4. 574 Figure 5 does NOT show the data path. It shows an example of a 575 configuration of the IEEE 802.1Q transmission selection box shown in 576 Figure 3 and Figure 4. Each queue m presents a "Class m Ready" 577 signal. These signals go through various logic, filters, and state 578 machines, until a single queue's "not empty" signal is chosen for 579 presentation to the underlying MAC/PHY. When the MAC/PHY is ready to 580 take another output packet, then a packet is selected from the one 581 queue (if any) whose signal manages to pass all the way through the 582 transmission selection function. 584 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ 585 |Class| |Class| |Class| |Class| |Class| |Class| |Class| |Class| 586 | 1 | | 0 | | 4 | | 5 | | 6 | | 7 | | 2 | | 3 | 587 |Ready| |Ready| |Ready| |Ready| |Ready| |Ready| |Ready| |Ready| 588 +--+--+ +--+--+ +--+--+ +-XXX-+ +--+--+ +--+--+ +--+--+ +--+--+ 589 | | | | | | | 590 | +--V--+ +--V--+ +--+--+ +--V--+ | +--V--+ +--V--+ 591 | |Prio.| |Prio.| |Prio.| |Prio.| | |Sha- | |Sha- | 592 | | 0 | | 4 | | 5 | | 6 | | | per| | per| 593 | | PFC | | PFC | | PFC | | PFC | | | A | | B | 594 | +--+--+ +--+--+ +-XXX-+ +-XXX-+ | +--+--+ +-XXX-+ 595 | | | | | 596 +--V--+ +--V--+ +--V--+ +--+--+ +--+--+ +--V--+ +--V--+ +--+--+ 597 |Time | |Time | |Time | |Time | |Time | |Time | |Time | |Time | 598 | Gate| | Gate| | Gate| | Gate| | Gate| | Gate| | Gate| | Gate| 599 | 1 | | 0 | | 4 | | 5 | | 6 | | 7 | | 2 | | 3 | 600 +--+--+ +-XXX-+ +--+--+ +--+--+ +-XXX-+ +--+--+ +-XXX-+ +--+--+ 601 | | | 602 +--V-------+-------V-------+--+ | 603 |802.1Q Enhanced Transmission | | 604 | Selection (ETS) = Weighted | | 605 | Fair Queuing (WFQ) | | 606 +--+-------+------XXX------+--+ | 607 | | 608 +--V-------+-------+-------+-------+-------V-------+-------+--+ 609 | Strict Priority selection (rightmost first) | 610 +-XXX------+-------+-------+-------+-------+-------+-------+--+ 611 | 612 V 614 Figure 5: 802.1Q Transmission Selection 616 The following explanatory notes apply to Figure 5 618 o The numbers in the "Class n Ready" boxes are the values of the 619 Layer 2 priority that are assigned to that Class of Service in 620 this example. The rightmost CoS is the most important, the 621 leftmost the least. Classes 2 and 3 are made the most important, 622 because they carry DetNet flows. It is all right to make them 623 more important than the priority 7 queue, which typically carries 624 critical network control protocols such as spanning tree or IS-IS, 625 because the shaper ensures that the highest priority best-effort 626 queue (7) will get reasonable access to the MAC/PHY. Note that 627 Class 5 has no Ready signal, indicating that that queue is empty. 629 o Below the Class Ready signals are shown the Priority Flow Control 630 gates (IEEE Std 802.1Qbb-2011 Priority-based Flow Control, now 631 [IEEE8021Q] clause 36) on Classes of Service 1, 0, 4, and 5, and 632 two 802.1Q shapers, A and B. Perhaps shaper A conforms to the 633 IEEE Std 802.1Qav-2009 (now [IEEE8021Q] clause 34) credit-based 634 shaper, and shaper B conforms to [IEEE8021Qcr] Asynchronous 635 Traffic Shaper. Any given Class of Service can have either a PFC 636 function or a shaper, but not both. 638 o Next are the IEEE Std 802.1Qbv time gates ([IEEE8021Qbv]). Each 639 one of the 8 Classes of Service has a time gate. The gates are 640 controlled by a repeating schedule that restarts periodically, and 641 can be programmed to turn any combination of gates on or off with 642 nanosecond precision. (Although the implementation is not 643 necessarily that accurate.) 645 o Following the time gates, any number of Classes of Service can be 646 linked to one ore more instances of the Enhanced Transmission 647 Selection function. This does weighted fair queuing among the 648 members of its group. 650 o A final selection of the one queue to be selected for output is 651 made by strict priority. Note that the priority is determined not 652 by the Layer 2 priority, but by the Class of Service. 654 o An "XXX" in the lower margin of a box (e.g. "Prio. 5 PFC" 655 indicates that the box has blocked the "Class n Ready" signal. 657 o IEEE 802.1Qch Cyclic Queuing and Forwarding [IEEE802.1Qch] is 658 accomplished using two or three queues (e.g. 2 and 3 in the 659 figure), using sophisticated time-based schedules in the Class of 660 Service Assignment function, and using the IEEE 802.1Qbv time 661 gates [IEEE8021Qbv] to swap between the output buffers. 663 7.3. Time-Sensitive Networking with Asynchronous Traffic Shaping 665 Consider a network with a set of nodes (switches and hosts) along 666 with a set of flows between hosts. Hosts are sources or destinations 667 of flows. There are four types of flows, namely, control-data 668 traffic (CDT), class A, class B, and best effort (BE) in decreasing 669 order of priority. Flows of classes A and B are together referred to 670 as AVB flows. It is assumed a subset of TSN functions as described 671 next. 673 It is also assumed that contention occurs only at the output port of 674 a TSN node. Each node output port performs per-class scheduling with 675 eight classes: one for CDT, one for class A traffic, one for class B 676 traffic, and five for BE traffic denoted as BE0-BE4 (according to TSN 677 standard). In addition, each node output port also performs per-flow 678 regulation for AVB flows using an interleaved regulator (IR), called 679 Asynchronous Traffic Shaper (ATS) in TSN. Thus, at each output port 680 of a node, there is one interleaved regulator per-input port and per- 681 class. The detailed picture of scheduling and regulation 682 architecture at a node output port is given by Figure 6. The packets 683 received at a node input port for a given class are enqueued in the 684 respective interleaved regulator at the output port. Then, the 685 packets from all the flows, including CDT and BE flows, are enqueued 686 in a class based FIFO system (CBFS) [TSNwithATS]. 688 +--+ +--+ +--+ +--+ 689 | | | | | | | | 690 |IR| |IR| |IR| |IR| 691 | | | | | | | | 692 +-++XXX++-+ +-++XXX++-+ 693 | | | | 694 | | | | 695 +-----+ +-v-XXX-v-+ +-v-XXX-v-+ +-----+ +-----+ +-----+ +-----+ +-----+ 696 | | | | | | |Class| |Class| |Class| |Class| |Class| 697 | CDT | | Class A | | Class B | | BE4 | | BE3 | | BE2 | | BE1 | | BE0 | 698 | | | | | | | | | | | | | | | | 699 +--+--+ +----+----+ +----+----+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ 700 | | | | | | | | 701 | +-v-+ +-v-+ | | | | | 702 | |CBS| |CBS| | | | | | 703 | +-+-+ +-+-+ | | | | | 704 | | | | | | | | 705 +--v---------v-----------v---------v-------V-------v-------v-------v--+ 706 | Strict Priority selection | 707 +----------------------------------+----------------------------------+ 708 | 709 V 711 Figure 6: Architecture of one TSN node output port with interleaved 712 regulators (IRs) 714 The CBFS includes two CBS subsystems, one for each class A and B. 715 The CBS serves a packet from a class according to the available 716 credit for that class. The credit for each class A or B increases 717 based on the idle slope, and decreases based on the send slope, both 718 of which are parameters of the CBS. The CDT and BE0-BE4 flows in the 719 CBFS are served by separate FIFO subsystems. Then, packets from all 720 flows are served by a transmission selection subsystem that serves 721 packets from each class based on its priority. All subsystems are 722 non-preemptive. Guarantees for AVB traffic can be provided only if 723 CDT traffic is bounded; it is assumed that the CDT traffic has an 724 affine arrival curve r t + b in each node, i.e. the amount of bits 725 entering a node within a time interval t is bounded by r t + b. 727 [[ EM: THE FOLLOWING PARAGRAPH SHOULD BE ALIGNED WITH Section 8.2. ]] 729 Additionally, it is assumed that flows are regulated at their source, 730 according to either leaky bucket (LB) or length rate quotient (LRQ). 731 The LB-type regulation forces flow f to conform to an arrival curve 732 r_f t+b_f . The LRQ-type regulation with rate r_f ensures that the 733 time separation between two consecutive packets of sizes l_n and 734 l_n+1 is at least l_n/r_f. Note that if flow f is LRQ-regulated, it 735 satisfies an arrival curve constraint r_f t + L_f where L_f is its 736 maximum packet size (but the converse may not hold). For an LRQ 737 regulated flow, b_f = L_f. At the source hosts, the traffic 738 satisfies its regulation constraint, i.e. the delay due to 739 interleaved regulator at hosts is ignored. 741 At each switch implementing an interleaved regulator, packets of 742 multiple flows are processed in one FIFO queue; the packet at the 743 head of the queue is regulated based on its regulation constraints; 744 it is released at the earliest time at which this is possible without 745 violating the constraint. The regulation type and parameters for a 746 flow are the same at its source and at all switches along its path. 748 7.4. Other queuing models, e.g. IntServ 750 [[NWF More sections that discuss specific models]] 752 8. Parameters for the bounded latency model 754 8.1. Sender parameters 756 8.2. Relay system parameters 758 [[NWF This section talks about the paramters that must be passed hop- 759 by-hop (T-SPEC? F-SPEC?) by a resoure reservation protocol.]] 761 9. References 763 9.1. Normative References 765 [I-D.ietf-detnet-architecture] 766 Finn, N. and P. Thubert, "Deterministic Networking 767 Architecture", draft-ietf-detnet-architecture-00 (work in 768 progress), September 2016. 770 [I-D.ietf-detnet-dp-alt] 771 Korhonen, J., Farkas, J., Mirsky, G., Thubert, P., 772 Zhuangyan, Z., and L. Berger, "DetNet Data Plane Protocol 773 and Solution Alternatives", draft-ietf-detnet-dp-alt-00 774 (work in progress), October 2016. 776 [I-D.ietf-detnet-use-cases] 777 Grossman, E., "Deterministic Networking Use Cases", draft- 778 ietf-detnet-use-cases-16 (work in progress), May 2018. 780 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 781 Requirement Levels", BCP 14, RFC 2119, 782 DOI 10.17487/RFC2119, March 1997, 783 . 785 [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification 786 of Guaranteed Quality of Service", RFC 2212, 787 DOI 10.17487/RFC2212, September 1997, 788 . 790 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 791 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 792 2006, . 794 [RFC6658] Bryant, S., Ed., Martini, L., Swallow, G., and A. Malis, 795 "Packet Pseudowire Encapsulation over an MPLS PSN", 796 RFC 6658, DOI 10.17487/RFC6658, July 2012, 797 . 799 [RFC7806] Baker, F. and R. Pan, "On Queuing, Marking, and Dropping", 800 RFC 7806, DOI 10.17487/RFC7806, April 2016, 801 . 803 9.2. Informative References 805 [IEEE802.1Qch] 806 IEEE, "IEEE Std 802.1Qch-2017 IEEE Standard for Local and 807 metropolitan area networks - Bridges and Bridged Networks 808 Amendment 29: Cyclic Queuing and Forwarding (amendment to 809 802.1Q-2014)", 2017, 810 . 812 [IEEE802.1Qci] 813 IEEE, "IEEE Std 802.1Qci-2017 IEEE Standard for Local and 814 metropolitan area networks - Bridges and Bridged Networks 815 - Amendment 30: Per-Stream Filtering and Policing", 2017, 816 . 818 [IEEE8021Q] 819 IEEE 802.1, "IEEE Std 802.1Q-2014: IEEE Standard for Local 820 and metropolitan area networks - Bridges and Bridged 821 Networks", 2014, . 824 [IEEE8021Qbu] 825 IEEE, "IEEE Std 802.1Qbu-2016 IEEE Standard for Local and 826 metropolitan area networks - Bridges and Bridged Networks 827 - Amendment 26: Frame Preemption", 2016, 828 . 831 [IEEE8021Qbv] 832 IEEE 802.1, "IEEE Std 802.1Qbv-2015: IEEE Standard for 833 Local and metropolitan area networks - Bridges and Bridged 834 Networks - Amendment 25: Enhancements for Scheduled 835 Traffic", 2015, . 838 [IEEE8021Qcr] 839 IEEE 802.1, "IEEE P802.1Qcr: IEEE Draft Standard for Local 840 and metropolitan area networks - Bridges and Bridged 841 Networks - Amendment: Asynchronous Traffic Shaping", 2017, 842 . 844 [IEEE8021TSN] 845 IEEE 802.1, "IEEE 802.1 Time-Sensitive Networking (TSN) 846 Task Group", . 848 [IEEE8023] 849 IEEE 802.3, "IEEE Std 802.3-2015: IEEE Standard for Local 850 and metropolitan area networks - Ethernet", 2015, 851 . 854 [IEEE8023br] 855 IEEE 802.3, "IEEE Std 802.3br-2016: IEEE Standard for 856 Local and metropolitan area networks - Ethernet - 857 Amendment 5: Specification and Management Parameters for 858 Interspersing Express Traffic", 2016, 859 . 862 [TSNwithATS] 863 E. Mohammadpour, E. Stai, M. Mohiuddin, and J.-Y. Le 864 Boudec, "End-to-end Latency and Backlog Bounds in Time- 865 Sensitive Networking with Credit Based Shapers and 866 Asynchronous Traffic Shaping", 867 . 869 Authors' Addresses 871 Norman Finn 872 Huawei Technologies Co. Ltd 873 3101 Rio Way 874 Spring Valley, California 91977 875 US 877 Phone: +1 925 980 6430 878 Email: norman.finn@mail01.huawei.com 880 Jean-Yves Le Boudec 881 EPFL 882 IC Station 14 883 Lausanne EPFL 1015 884 Switzerland 886 Email: jean-yves.leboudec@epfl.ch 888 Ehsan Mohammadpour 889 EPFL 890 IC Station 14 891 Lausanne EPFL 1015 892 Switzerland 894 Email: ehsan.mohammadpour@epfl.ch 896 Balazs Varga 897 Ericsson 898 Konyves Kalman krt. 11/B 899 Budapest 1097 900 Hungary 902 Email: balazs.a.varga@ericsson.com 904 Janos Farkas 905 Ericsson 906 Konyves Kalman krt. 11/B 907 Budapest 1097 908 Hungary 910 Email: janos.farkas@ericsson.com