idnits 2.17.1 draft-ietf-detnet-dp-sol-mpls-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The exact meaning of the all-uppercase expression 'NOT REQUIRED' is not defined in RFC 2119. If it is intended as a requirements expression, it should be rewritten using one of the combinations defined in RFC 2119; otherwise it should not be all-uppercase. -- The document date (March 10, 2019) is 1871 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Network' is mentioned on line 532, but not defined == Missing Reference: 'TBD-TSN-OVER-DETNET' is mentioned on line 548, but not defined == Outdated reference: A later version (-13) exists of draft-ietf-detnet-architecture-11 == Outdated reference: A later version (-14) exists of draft-ietf-detnet-flow-information-model-03 == Outdated reference: A later version (-14) exists of draft-ietf-pce-pcep-extension-for-pce-controller-01 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-mpls-18 -- Obsolete informational reference (is this intentional?): RFC 3272 (Obsoleted by RFC 9522) -- Obsolete informational reference (is this intentional?): RFC 6006 (Obsoleted by RFC 8306) Summary: 0 errors (**), 0 flaws (~~), 7 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet J. Korhonen, Ed. 3 Internet-Draft 4 Intended status: Standards Track B. Varga, Ed. 5 Expires: September 11, 2019 Ericsson 6 March 10, 2019 8 DetNet MPLS Data Plane Encapsulation 9 draft-ietf-detnet-dp-sol-mpls-02 11 Abstract 13 This document specifies the Deterministic Networking data plane when 14 operating over an MPLS Packet Switched Networks. 16 Status of This Memo 18 This Internet-Draft is submitted in full conformance with the 19 provisions of BCP 78 and BCP 79. 21 Internet-Drafts are working documents of the Internet Engineering 22 Task Force (IETF). Note that other groups may also distribute 23 working documents as Internet-Drafts. The list of current Internet- 24 Drafts is at https://datatracker.ietf.org/drafts/current/. 26 Internet-Drafts are draft documents valid for a maximum of six months 27 and may be updated, replaced, or obsoleted by other documents at any 28 time. It is inappropriate to use Internet-Drafts as reference 29 material or to cite them other than as "work in progress." 31 This Internet-Draft will expire on September 11, 2019. 33 Copyright Notice 35 Copyright (c) 2019 IETF Trust and the persons identified as the 36 document authors. All rights reserved. 38 This document is subject to BCP 78 and the IETF Trust's Legal 39 Provisions Relating to IETF Documents 40 (https://trustee.ietf.org/license-info) in effect on the date of 41 publication of this document. Please review these documents 42 carefully, as they describe your rights and restrictions with respect 43 to this document. Code Components extracted from this document must 44 include Simplified BSD License text as described in Section 4.e of 45 the Trust Legal Provisions and are provided without warranty as 46 described in the Simplified BSD License. 48 Table of Contents 50 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 51 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 52 2.1. Terms Used in This Document . . . . . . . . . . . . . . . 4 53 2.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4 54 3. Requirements Language . . . . . . . . . . . . . . . . . . . . 5 55 4. DetNet MPLS Data Plane Overview . . . . . . . . . . . . . . . 6 56 4.1. Layers of DetNet Data Plane . . . . . . . . . . . . . . . 6 57 4.2. DetNet MPLS Data Plane Scenarios . . . . . . . . . . . . 7 58 4.2.1. IP Over DetNet MPLS Data Plane Scenarios . . . . . . 9 59 4.2.2. IEEE 802.1 TSN Over DetNet MPLS Data Plane Scenario . 12 60 4.3. Packet Flow Example with Service Protection . . . . . . . 14 61 5. DetNet MPLS Data Plane Considerations . . . . . . . . . . . . 15 62 5.1. End-System Specific Considerations . . . . . . . . . . . 16 63 5.2. Sub-Network Considerations . . . . . . . . . . . . . . . 17 64 6. MPLS-Based DetNet Data Plane Solution . . . . . . . . . . . . 18 65 6.1. DetNet Over MPLS Encapsulation Components . . . . . . . . 18 66 6.2. MPLS Data Plane Encapsulation . . . . . . . . . . . . . . 19 67 6.2.1. DetNet Control Word and the DetNet Sequence Number . 20 68 6.2.2. S-Labels . . . . . . . . . . . . . . . . . . . . . . 21 69 6.2.3. F-Labels . . . . . . . . . . . . . . . . . . . . . . 24 70 6.3. OAM Indication . . . . . . . . . . . . . . . . . . . . . 26 71 6.4. Flow Aggregation . . . . . . . . . . . . . . . . . . . . 27 72 6.4.1. Aggregation at the LSP . . . . . . . . . . . . . . . 28 73 6.4.2. Aggregating DetNet Flows as a new DetNet flow . . . . 28 74 6.4.3. Simple Aggregation at the DetNet Layer . . . . . . . 29 75 6.5. Service Sub-Layer Considerations . . . . . . . . . . . . 29 76 6.5.1. Edge Node Processing . . . . . . . . . . . . . . . . 30 77 6.5.2. Relay Node Processing . . . . . . . . . . . . . . . . 31 78 6.6. Forwarding Sub-Layer Considerations . . . . . . . . . . . 31 79 6.6.1. Class of Service . . . . . . . . . . . . . . . . . . 31 80 6.6.2. Quality of Service . . . . . . . . . . . . . . . . . 32 81 6.6.3. Cross-DetNet Flow Resource Aggregation . . . . . . . 32 82 6.6.4. Layer 2 Addressing and QoS Considerations . . . . . . 33 83 6.6.5. Time Synchronization . . . . . . . . . . . . . . . . 34 84 7. Controller Plane (Management and Control) 85 Considerations . . . . . . . . . . . . . . . . . . . . . . . 34 86 7.1. S-Label and F-Label Assignment and Distribution . . . . . 35 87 7.2. Packet Replication, Elimination, and Ordering (PREOF) . . 36 88 7.3. Contention Loss and Jitter Reduction . . . . . . . . . . 36 89 7.4. Bidirectional Traffic . . . . . . . . . . . . . . . . . . 37 90 7.5. Flow Aggregation Control . . . . . . . . . . . . . . . . 38 91 7.6. DetNet 92 Controller (Control and Management) Plane Requirements . 38 93 8. DetNet MPLS Operation Over IEEE 802.1 TSN Sub-Networks . . . 39 94 8.1. Mapping of TSN Stream ID and Sequence Number . . . . . . 41 95 8.2. TSN Usage of FRER . . . . . . . . . . . . . . . . . . . . 42 96 8.3. Management and Control Implications . . . . . . . . . . . 42 97 9. DetNet MPLS Operation over DetNet 98 IP PSNs . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 99 10. Security Considerations . . . . . . . . . . . . . . . . . . . 45 100 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 45 101 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 45 102 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 46 103 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 47 104 14.1. Normative References . . . . . . . . . . . . . . . . . . 47 105 14.2. Informative References . . . . . . . . . . . . . . . . . 49 106 Appendix A. Example of DetNet Data Plane Operation . . . . . . . 52 107 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 52 109 1. Introduction 111 Deterministic Networking (DetNet) is a service that can be offered by 112 a network to DetNet flows. DetNet provides these flows with a low 113 packet loss rates and assured maximum end-to-end delivery latency. 114 General background and concepts of DetNet can be found in 115 [I-D.ietf-detnet-architecture]. 117 The DetNet Architecture decomposes the DetNet related data plane 118 functions into two sub-layers: a service sub-layer and a forwarding 119 sub-layer. The service sub-layer is used to provide DetNet service 120 protection and reordering. The forwarding sub-layer is used to 121 provides congestion protection (low loss, assured latency, and 122 limited reordering) leveraging MPLS Traffic Engineering mechanisms. 124 This document specifies the DetNet data plane operation and the on- 125 wire encapsulation of DetNet flows over an MPLS-based Packet Switched 126 Network (PSN). The specified encapsulation provides the building 127 blocks to enable the DetNet service and forwarding sub-layer 128 functions and supports flow identification as described in the DetNet 129 Architecture. As part of the service sub-layer functions, this 130 document describes DetNet node data plane operation. It also 131 describes the function and operation of the Packet Replication (PRF) 132 Packet Elimination (PEF) and Packet Ordering (POF) functions with an 133 MPLS data plane. It also describes an MPLS-based DetNet forwarding 134 sub-layer that eliminates (or reduces) contention loss and provides 135 bounded latency for DetNet flows. 137 MPLS encapsulated DetNet flows can be carried over network 138 technologies that can provide the DetNet required level of service. 139 This document defines examples of such, specifically carrying DetNet 140 MPLS flows over IEEE 802.1 TSN sub-networks, and over DetNet IP PSN. 142 The intent is for this document to support different traffic types 143 being mapped over DetNet MPLS, but this is out side the scope of this 144 document. An example of such can be found in 145 [I-D.ietf-detnet-dp-sol-ip]. This document also allows for, but does 146 not define, associated controller plane and Operations, 147 Administration, and Maintenance (OAM) functions. 149 2. Terminology 151 2.1. Terms Used in This Document 153 This document uses the terminology established in the DetNet 154 architecture [I-D.ietf-detnet-architecture], and the reader is 155 assumed to be familiar with that document and its terminology. 157 The following terminology is introduced in this document: 159 F-Label A Detnet "forwarding" label that identifies the LSP 160 used to forward a DetNet flow across an MPLS PSN, e.g., 161 a hop-by-hop label used between label switching routers 162 (LSR). 164 S-Label A DetNet "service" label that is used between DetNet 165 nodes that implement also the DetNet service sub-layer 166 functions. An S-Label is also used to identify a 167 DetNet flow at DetNet service sub-layer. 169 d-CW A DetNet Control Word (d-CW) is used for sequencing and 170 identifying duplicate packets of a DetNet flow at the 171 DetNet service sub-layer. 173 2.2. Abbreviations 175 The following abbreviations are used in this document: 177 AC Attachment Circuit. 179 CE Customer Edge equipment. 181 CoS Class of Service. 183 CW Control Word. 185 DetNet Deterministic Networking. 187 DF DetNet Flow. 189 DN-IWF DetNet Inter-Working Function. 191 L2 Layer 2. 193 L2VPN Layer 2 Virtual Private Network. 195 L3 Layer 3. 197 LSR Label Switching Router. 199 MPLS Multiprotocol Label Switching. 201 MPLS-TE Multiprotocol Label Switching - Traffic Engineering. 203 MPLS-TP Multiprotocol Label Switching - Transport Profile. 205 MS-PW Multi-Segment PseudoWire (MS-PW). 207 NSP Native Service Processing. 209 OAM Operations, Administration, and Maintenance. 211 PE Provider Edge. 213 PEF Packet Elimination Function. 215 PRF Packet Replication Function. 217 PREOF Packet Replication, Elimination and Ordering Functions. 219 POF Packet Ordering Function. 221 PSN Packet Switched Network. 223 PW PseudoWire. 225 QoS Quality of Service. 227 S-PE Switching Provider Edge. 229 T-PE Terminating Provider Edge. 231 TSN Time-Sensitive Network. 233 3. Requirements Language 235 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 236 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 237 "OPTIONAL" in this document are to be interpreted as described in BCP 238 14 [RFC2119] [RFC8174] when, and only when, they appear in all 239 capitals, as shown here. 241 4. DetNet MPLS Data Plane Overview 243 4.1. Layers of DetNet Data Plane 245 This document describes how DetNet flows are carried over MPLS 246 networks. The DetNet Architecture, [I-D.ietf-detnet-architecture], 247 decomposes the DetNet data plane into two sub-layers: a service sub- 248 layer and a forwarding sub-layer. The basic approach defined in this 249 document supports the DetNet service sub-layer based on existing 250 pseudowire (PW) encapsulations and mechanisms, and supports the 251 DetNet forwarding sub-layer based on existing MPLS Traffic 252 Engineering encapsulations and mechanisms. Background on PWs can be 253 found in [RFC3985] and [RFC3031]. Background on MPLS Traffic 254 Engineering can be found in [RFC3272] and [RFC3209]. 256 DetNet MPLS 257 . 258 . 259 +------------+ 260 | Service | d-CW, S-Label 261 +------------+ 262 | Forwarding | F-Label(s) 263 +------------+ 264 . 265 . 267 Figure 1: DetNet Adaptation to MPLS Data Plane 269 The DetNet MPLS data plane approach defined in this document is shown 270 in Figure 1. The service sub-layer is supported by a DetNet control 271 word (d-CW) which conforms to the Generic PW MPLS Control Word 272 (PWMCW) defined in [RFC4385]. A d-CW identifying service label 273 (S-Label) is also used. 275 A node operating on a DetNet flow in the Detnet service sub-layer, 276 i.e. a node processing a DetNet packet which has the S-Label as top 277 of stack uses the local context associated with that S-Label, for 278 example a received F-Label, to determine what local DetNet 279 operation(s) are applied to that packet. An S-Label may be unique 280 when taken from the platform label space [RFC3031], which would 281 enable correct DetNet flow identification regardless of which input 282 interface or LSP the packet arrives on. 284 The DetNet MPLS data plane builds on MPLS Traffic Engineering 285 encapsulations and mechanisms to provide a forwarding sub-layer that 286 is responsible for providing resource allocation and explicit routes. 287 The forwarding sub-layer is supported by one or more forwarding 288 labels (F-Labels). 290 4.2. DetNet MPLS Data Plane Scenarios 292 DetNet MPLS Relay Transit Relay DetNet MPLS 293 End System Node Node Node End System 294 (T-PE) (S-PE) (LSR) (S-PE) (T-PE) 295 +----------+ +----------+ 296 | Appl. |<------------ End to End Service ----------->| Appl. | 297 +----------+ +---------+ +---------+ +----------+ 298 | Service |<--| Service |-- DetNet flow --| Service |-->| Service | 299 +----------+ +---------+ +----------+ +---------+ +----------+ 300 |Forwarding| |Fwd| |Fwd| |Forwarding| |Fwd| |Fwd| |Forwarding| 301 +-------.--+ +-.-+ +-.-+ +----.---.-+ +-.-+ +-.-+ +---.------+ 302 : Link : / ,-----. \ : Link : / ,-----. \ 303 +........+ +-[ Sub ]-+ +......+ +-[ Sub ]-+ 304 [Network] [Network] 305 `-----' `-----' 306 |<- LSP -->| |<-------- LSP -----------| |<--- LSP -->| 308 |<----------------- DetNet MPLS --------------------->| 310 Figure 2: A DetNet MPLS Network 312 Figure 2 illustrates a hypothetical DetNet MPLS-only network composed 313 of DetNet aware MPLS enabled end systems, operating over a DetNet 314 aware MPLS network. In this figure, relay nodes sit at MPLS LSP 315 boundaries and transit nodes are LSRs. 317 DetNet end system and relay nodes are DetNet service sub-layer aware, 318 understand the particular needs of DetNet flows and provide both 319 DetNet service and forwarding sub-layer functions. They add, remove 320 and process d-CWs, S-Labels and F-labels as needed. MPLS enabled end 321 system and relay nodes can enhance the reliability of delivery by 322 enabling the replication of packets where multiple copies, possibly 323 over multiple paths, are forwarded through the DetNet domain. They 324 can also eliminate surplus previously replicated copies of DetNet 325 packets. DetNet MPLS nodes provide functionality similar to T-PEs 326 when they sit at the edge of an MPLS domain, and functionality 327 similar to S-PEs when they are in the middle of an MPLS domain, see 328 [RFC6073]. End system and relay nodes also include DetNet forwarding 329 sub-layer functions, support for notably explicit routes, and 330 resources allocation to eliminate (or reduce) congestion loss and 331 jitter. 333 DetNet transit nodes reside wholly within a DetNet domain, and also 334 provide DetNet forwarding sub-layer functions in accordance with the 335 performance required by a DetNet flow carried over an LSP. Unlike 336 other DetNet node types, transit nodes provide no service sub-layer 337 processing. In a DetNet MPLS network, transit nodes may be DetNet 338 service aware or may be DetNet unaware MPLS Label Switching Routers 339 (LSRs). In this latter case, such LSRs would be unaware of the 340 special requirements of the DetNet service sub-layer, but would still 341 provide traffic engineering services and the QoS need to ensure that 342 the (TE) LSPs meet the service requirements of the carried DetNet 343 flows. 345 The LSPs may be provided by any MPLS controller method. For example 346 they may be provisioned via a management plane, RSVP-TE, MPLS-TP, or 347 MPLS Segment Routing (when extended to support resource allocation). 349 Figure 3 illustrates how an end to end MPLS-based DetNet service is 350 provided in a more detail. In this figure, the end systems, CE1 and 351 CE2, are able to send and receive MPLS encapsulated DetNet flows, and 352 R1, R2 and R3 are relay nodes as they sit in the middle of a DetNet 353 network. The 'X' in the end systems, and relay nodes represents 354 potential DetNet compound flow packet replication and elimination 355 points. In this example, service protection is supported over four 356 DetNet member flows and TE LSPs. For a unidirectional flow, R1 357 supports PRF, R2 supports PREOF and R3 supports PEF and POF. Note 358 that the relay nodes may change the underlying forwarding sub-layer, 359 for example tunneling MPLS over IEEE 802.1 TSN Section 8, or simply 360 over interconnect network links. 362 DetNet DetNet 363 MPLS Service Transit Transit Service MPLS 364 DetNet | |<-Tnl->| |<-Tnl->| | DetNet 365 End | V 1 V V 2 V | End 366 System | +--------+ +--------+ +--------+ | System 367 +---+ | | R1 |=======| R2 |=======| R3 | | +---+ 368 | X...DFa...|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|.DFa..|.X | 369 |CE1|========| \ | | X | | / |======|CE2| 370 | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | 371 +---+ | |=======| |=======| | +---+ 372 ^ +--------+ +--------+ +--------+ ^ 373 | Relay Node Relay Node Relay Node | 374 | (S-PE) (S-PE) (S-PE) | 375 | | 376 |<---------------------- DetNet MPLS --------------------->| 377 | | 378 |<--------------- End to End DetNet Service -------------->| 380 -------------------------- Data Flow -------------------------> 382 X = Optional service protection (none, PRF, PREOF, PEF/POF) 383 DFx = DetNet member flow x over a TE LSP 385 Figure 3: MPLS-Based Native DetNet 387 As previously mentioned, this document specifies how MPLS is used to 388 support DetNet flows using an MPLS data plane as well as how such can 389 be mapped to IEEE 802.1 TSN and IP DetNet PSNs. An equally import 390 scenario is when IP is supported over DetNet MPLS and this is covered 391 in [I-D.ietf-detnet-dp-sol-ip]. Another important scenario is where 392 an Ethernet Layer 2 service is supported over DetNet MPLS and this is 393 covered in [TBD-TSN-OVER-DETNET]. 395 4.2.1. IP Over DetNet MPLS Data Plane Scenarios 397 [Author's note: this section to be moved to IP sol draft] 398 IP DetNet Relay Transit Relay IP DetNet 399 End System Node Node Node End System 400 (T-PE) (LSR) (T-PE) 401 +----------+ +----------+ 402 | Appl. |<------------ End to End Service ----------->| Appl. | 403 +----------+ .....-----+ +-----..... +----------+ 404 | Service |<--: Service |-- DetNet flow --| Service :-->| Service | 405 +----------+ +---------+ +----------+ +---------+ +----------+ 406 |Forwarding| |Fwd| |Fwd| |Forwarding| |Fwd| |Fwd| |Forwarding| 407 +-------.--+ +-.-+ +-.-+ +----.---.-+ +-.-+ +-.-+ +---.------+ 408 : Link : / ,-----. \ : Link : / ,-----. \ 409 +........+ +-[ Sub ]-+ +......+ +-[ Sub ]-+ 410 [Network] [Network] 411 `-----' `-----' 413 |<- DN IP->| |<---- DetNet MPLS ---->| |< -DN IP ->| 415 Figure 4: DetNet IP Over MPLS Network 417 Figure 4 illustrates DetNet enabled End Systems (hosts), connected to 418 DetNet (DN) enabled IP networks, operating over a DetNet aware MPLS 419 network. In this figure, Relay nodes sit at the boundary of the MPLS 420 domain since the non-MPLS domain is DetNet aware. This figure is 421 very similar to Figure 2. The primary difference is that the Relay 422 nodes are at the edges of the MPLS domain and therefore function as 423 T-PEs, and that service sub-layer functions are not provided over the 424 DetNet IP network. There is no difference in transit node function. 426 Figure 5 illustrates how relay nodes can provide service protection 427 over the MPLS domain. In this case, CE1 and CE2 are IP DetNet end 428 systems which are interconnected via a MPLS domain such as previously 429 shown in Figure 3. Note that R1 and R3 sit at the edges of an MPLS 430 domain and therefore are similar to T-PEs, while R2 sits in the 431 middle of the domain and is therefore similar to an S-PE. 433 DetNet DetNet 434 IP Service Transit Transit Service IP 435 DetNet |<-Tnl->| |<-Tnl->| DetNet 436 End | V 1 V V 2 V | End 437 System | +--------+ +--------+ +--------+ | System 438 +---+ | | R1 |=======| R2 |=======| R3 | | +---+ 439 | |-------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|-------| | 440 |CE1| | | \ | | X | | / | | |CE2| 441 | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | 442 +---+ | |=======| |=======| | +---+ 443 ^ +--------+ +--------+ +--------+ ^ 444 | Relay Node Relay Node Relay Node | 445 | (T-PE) (S-PE) (T-PE) | 446 | | 447 |<-DN IP-> <-------- DetNet MPLS ---------------> <-DN IP->| 448 | | 449 |<-------------- End to End DetNet Service --------------->| 451 -------------------------- Data Flow -------------------------> 453 X = Service protection (PRF, PREOF, PEF/POF) 454 DFx = DetNet member flow x over a TE LSP 456 Figure 5: DetNet IP Over DetNet MPLS Network 458 IP Edge Edge IP 459 End System Node Node End System 460 (T-PE) (LSR) (T-PE) 461 +----------+ +....-----+ +-----....+ +----------+ 462 | Appl. |<--:Svc Proxy|-- E2E Service --|Svc Proxy:-->| Appl. | 463 +----------+ +.....+---+ +---+.....+ +----------+ 464 | IP |<--:IP : |Svc|-- IP/DN Flow ---|Svc| :IP :-->| IP | 465 +----------+ +---+ +---+ +----------+ +---+ +---+ +----------+ 466 |Forwarding| |Fwd| |Fwd| |Forwarding| |Fwd| |Fwd| |Forwarding| 467 +-------.--+ +-.-+ +-.-+ +----.---.-+ +-.-+ +-.-+ +---.------+ 468 : Link : / ,-----. \ : Link : / ,-----. \ 469 +........+ +-[ Sub ]-+ +......+ +-[ Sub ]-+ 470 [Network] [Network] 471 `-----' `-----' 473 |<--- IP --->| |<----- DetNet MPLS ----->| |<--- IP --->| 475 Figure 6: Non-DetNet Aware IP Over DetNet MPLS Network 477 Figure 6 illustrates non-DetNet enabled End Systems (hosts), 478 connected to DetNet (DN) enabled MPLS network. It differs from 479 Figure 4 in that the hosts and edge IP networks are not DetNet aware. 480 In this case, edge nodes sit at the boundary of the MPLS domain since 481 it is also a DetNet domain boundary. The edge nodes provide DetNet 482 service proxies for the end applications by initiating and 483 terminating DetNet service for the application's IP flows. See 484 [I-D.ietf-detnet-dp-sol-ip] for more information. 486 Figure 7 illustrates how it is still possible to provided DetNet 487 service protection for non-DetNet aware end systems. This figures is 488 basically the same as Figure 5, with the exception that CE1 and CE2 489 are non-DetNet aware end systems and E1 and E3 are edge nodes that 490 replace the relay nodes R1 and R3. 492 IP IP 493 Non Service Transit Transit Service Non 494 DetNet |<-Tnl->| |<-Tnl->| DetNet 495 End | V 1 V V 2 V | End 496 System | +--------+ +--------+ +--------+ | System 497 +---+ | | E1 |=======| R2 |=======| E3 | | +---+ 498 | |--------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|------| | 499 |CE1| | | \ | | X | | / | | |CE2| 500 | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | 501 +---+ | |=======| |=======| | +---+ 502 +--------+ +--------+ +--------+ 503 ^ Edge Node Relay Node Edge Node^ 504 | (T-PE) (S-PE) (T-PE) | 505 | | 506 <--IP-->| <-------- IP Over DetNet MPLS ---------> |<--IP--> 507 | | 508 |<------ End to End DetNet Service ------->| 510 X = Optional service protection (none, PRF, PREOF, PEF/POF) 511 DFx = DetNet member flow x over a TE LSP 513 Figure 7: MPLS-Based DetNet (non-MPLS End System) 515 4.2.2. IEEE 802.1 TSN Over DetNet MPLS Data Plane Scenario 517 [Author's note: this section to be moved to TSN over mpls sol draft - 518 TBD-TSN-OVER-DETNET] 519 TSN Edge Transit Edge TSN 520 End System Node Node Node End System 521 (T-PE) (LSR) (T-PE) 523 +----------+ +.........+ +.........+ +----------+ 524 | Appl. |<-:Svc Proxy:--End to End Svc.--:Svc Proxy:->| Appl. | 525 +----------+ +---------+ +---------+ +----------+ 526 | TSN | |TSN| |Svc|<-- DetNet flow -->|Svc| |TSN| | TSN | 527 +----------+ +---+ +---+ +----------+ +---+ +---+ +----------+ 528 |Forwarding| |Fwd| |Fwd| |Forwarding| |Fwd| |Fwd| |Forwarding| 529 +------.---+ +--.+ +-.-+ +---.----.-+ +--.+ +-.-+ +----.-----+ 530 : Link : / ,-----. \ : Link : / ,-----. \ 531 +.........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ 532 [Network] [Network] 533 `-----' `-----' 535 |<- TSN ->| |<------ DetNet MPLS ------>| |<-- TSN --->| 537 Figure 8: A TSN over DetNet MPLS Enabled Network 539 Figure 8 shows IEEE 802.1 TSN end stations operating over a TSN aware 540 DetNet service running over an MPLS network. DetNet Edge Nodes sit 541 at the boundary of a DetNet domain. They are responsible for mapping 542 non-DetNet aware L2 traffic to DetNet services. They also support 543 the imposition and disposition of the required DetNet encapsulation. 544 These are functionally similar to pseudowire (PW) Terminating 545 Provider Edge (T-PE) nodes which use MPLS-TE LSPs. In this example 546 they understand and support IEEE 802.1 TSN and are able to map TSN 547 flows into DetNet flows. The specific of this operation are 548 discussed in [TBD-TSN-OVER-DETNET]. 550 Native TSN flow and DetNet MPLS flow differ not only by the 551 additional MPLS specific encapsulation, but DetNet MPLS flows have on 552 each DetNet node an associated DetNet specific data structure, what 553 defines flow related characteristics and required forwarding 554 functions. In this example, edge Nodes provide a service proxy 555 function that "associates" the DetNet flows and native flows at the 556 edge of the DetNet domain. This ensures that the DN Flow is properly 557 served at the Edge node (and inside the domain). 559 Figure 9 illustrates how DetNet can provide services for IEEE 560 802.1TSN end systems, CE1 and CE2, over a DetNet enabled MPLS 561 network. Similar to Figure 6, the edge nodes, E1 and E2, insert and 562 remove required DetNet data plane encapsulation. The 'X' in the edge 563 nodes and relay node, R1, represent a potential DetNet compound flow 564 packet replication and elimination point. This conceptually 565 parallels L2VPN services, and could leverage existing related 566 solutions as discussed below. 568 TSN |<------- End to End DetNet Service ------>| TSN 569 Service | Transit Transit | Service 570 TSN (AC) | |<-Tnl->| |<-Tnl->| | (AC) TSN 571 End | V V 1 V V 2 V V | End 572 System | +--------+ +--------+ +--------+ | System 573 +---+ | | E1 |=======| R1 |=======| E2 | | +---+ 574 | |--|----|._X_....|..DF1..|.._ _...|..DF3..|...._X_.|---|---| | 575 |CE1| | | \ | | X | | / | | |CE2| 576 | | | \_.|..DF2..|._/ \_..|..DF4..|._/ | | | 577 +---+ | |=======| |=======| | +---+ 578 ^ +--------+ +--------+ +--------+ ^ 579 | Edge Node Relay Node Edge Node | 580 | (T-PE) (S-PE) (T-PE) | 581 | | 582 |<- TSN -> <------- TSN Over DetNet MPLS -------> <- TSN ->| 583 | | 584 |<--- Emulated Time Sensitive Networking (TSN) Service --->| 586 X = Service protection 587 DFx = DetNet member flow x over a TE LSP 589 Figure 9: IEEE 802.1TSN Over DetNet 591 4.3. Packet Flow Example with Service Protection 593 An example DetNet MPLS network fragment and packet flow is 594 illustrated in Figure 10. 596 1 1.1 1.1 1.2.1 1.2.1 1.2.2 597 CE1----EN1--------R1-------R2-------R3--------EN2-----CE2 598 \ 1.2.1 / / 599 \1.2 /-----+ / 600 +------R4------------------------+ 601 1.2.2 603 Figure 10: Example Packet Flow in DetNet Enabled MPLS Network 605 In Figure 10 the numbers are used to identify the instance of a 606 packet. Packet 1 is the original packet, and packets 1.1, and 1.2 607 are two first generation copies of packet 1. Packet 1.2.1 is a 608 second generation copy of packet 1.2 etc. Note that these numbers 609 never appear in the packet, and are not to be confused with sequence 610 numbers, labels or any other identifier that appears in the packet. 611 They simply indicate the generation number of the original packet so 612 that its passage through the network fragment can be identified to 613 the reader. 615 Customer Equipment CE1 sends a packet into the DetNet enabled MPLS 616 network. This is packet (1). Edge Node EN1 encapsulates the packet 617 as a DetNet Packet and sends it to Relay node R1 (packet 1.1). EN1 618 makes a copy of the packet (1.2), encapsulates it and sends this copy 619 to Relay node R4. 621 Note that along the MPLS path from EN1 to R1 there may be zero or 622 more LSRs which, for clarity, are not shown. The same is true for 623 any other path between two DetNet entities shown in Figure 10. 625 Relay node R4 has been configured to send one copy of the packet to 626 Relay Node R2 (packet 1.2.1) and one copy to Edge Node EN2 (packet 627 1.2.2). 629 R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and, 630 having been configured to perform packet elimination on this DetNet 631 flow, forwards packet 1.2.1 to Relay Node R3. Packet copy 1.1 is of 632 no further use and so is discarded by R2. 634 Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives 635 packet copy 1.2.1 from R2 via relay Node R3. EN2 therefore strips 636 any DetNet encapsulation from packet copy 1.2.2 and forwards the 637 packet to CE2. When EN2 receives the later packet copy 1.2.1 this is 638 discarded. 640 The above is of course illustrative of many network scenarios that 641 can be configured. Between a pair of relay nodes there may be one or 642 more transit nodes that simply forward the DetNet traffic, but these 643 are omitted for clarity. 645 5. DetNet MPLS Data Plane Considerations 647 This section provides informative considerations related to providing 648 DetNet service to flows which are identified based on their header 649 information. At a high level, the following are provided on a per 650 flow basis: 652 Eliminating contention loss and jitter reduction: 654 Use of allocated resources (queuing, policing, shaping) to ensure 655 that the congestion-related loss and latency/jitter requirements 656 of a DetNet flow are met. 658 Explicit routes: 660 Use of a specific path for a flow. This limits misordering and 661 bounds latency. 663 Service protection: 665 Which in the case of this document primarily relates to 666 replication and elimination. Changing the explicit path after a 667 failure is detected in order to restore delivery of the required 668 DetNet service characteristics is also possible. Path changes, 669 even in the case of failure recovery, can lead to the out of order 670 delivery of data. 672 Load sharing: 674 Generally, distributing packets of the same DetNet flow over 675 multiple paths is not recommended. Such load sharing, e.g., via 676 ECMP or UCMP, impacts ordering and possibly jitter. 678 Troubleshooting: 680 For example, to support identification of misbehaving flows. 682 Recognize flow(s) for analytics: 684 For example, increase counters. 686 Correlate events with flows: 688 For example, unexpected loss. 690 The DetNet data plane also allows for the aggregation of DetNet 691 flows, e.g., via MPLS hierarchical LSPs, to improved scaling. When 692 DetNet flows are aggregated, transit nodes provide service to the 693 aggregate and not on a per-DetNet flow basis. In this case, nodes 694 performing aggregation will ensure that per-flow service requirements 695 are achieved. 697 5.1. End-System Specific Considerations 699 Data-flows requiring DetNet service are generated and terminated on 700 end-systems. Encapsulation depends on application and its 701 preferences. In a DetNet MPLS domain the DN functions use the d-CWs, 702 S-Labels and F-Labels to provide DetNet services. However, an 703 application may exchange further flow related parameters (e.g., time- 704 stamp), which are not provided by DN functions. 706 Specifics related to non-MPLS DetNet end station behavior are out 707 side the scope of this document. For example, details on support for 708 DetNet IP data flows can be found in [I-D.ietf-detnet-dp-sol-ip]. 709 This document is also useful for end stations that map IP flows to 710 DetNet flows. 712 As a general rule, DetNet MPLS domains are capable of forwarding any 713 DetNet MPLS flows and the DetNet domain does not mandate the end- 714 system or edge system encapsulation format. Unless there is a proxy 715 of some form present, end-systems peer with similar end-systems using 716 the same application encapsulation format. For example, as shown in 717 Figure 11, IP applications peer with IP applications and Ethernet 718 L2VPN applications peer with Ethernet L2VPN applications. 720 +-----+ 721 | X | +-----+ 722 +-----+ | X | 723 | Eth | ________ +-----+ 724 +-----+ _____ / \ | Eth | 725 \ / \__/ \___ +-----+ 726 \ / \ / 727 0======== tunnel-1 ========0_ 728 | \ 729 \ | 730 0========= tunnel-2 =========0 731 / \ __/ \ 732 +-----+ \__ DetNet MPLS domain / \ 733 | X | \ __ / +-----+ 734 +-----+ \_______/ \_____/ | X | 735 | IP | +-----+ 736 +-----+ | IP | 737 +-----+ 739 Figure 11: End-Systems and The DetNet MPLS Domain 741 5.2. Sub-Network Considerations 743 As shown in Figure 2, MPLS nodes are interconnected by different sub- 744 network technologies, which may include point-to-point links. Each 745 of these need to provide appropriate service to DetNet flows. In 746 some cases, e.g., on dedicated point-to-point links or TDM 747 technologies, all that is required is for a DetNet node to 748 appropriately queue its output traffic. In other cases, DetNet nodes 749 will need to map DetNet flows to the flow semantics (i.e., 750 identifiers) and mechanisms used by an underlying sub-network 751 technology. Figure 12 shows several examples of header formats that 752 can be used to carry DetNet MPLS flows over different sub-network 753 technologies. L2 represent a generic layer-2 encapsulation that 754 might be used on a point-to-point link. TSN represents the 755 encapsulation used on an IEEE 802.1 TSN network, as described in 756 Section 8. UDP/IP represents the encapsulation used on a DetNet IP 757 PSN, as described in Section 9 . 759 +------+ +------+ +------+ 760 App-Flow | X | | X | | X | 761 +-----+======+--+======+--+======+-----+ 762 DetNet-MPLS | d-CW | | d-CW | | d-CW | 763 +------+ +------+ +------+ 764 |Labels| |Labels| |Labels| 765 +-----+======+--+======+--+======+-----+ 766 Sub-Network | L2 | | TSN | | UDP | 767 +------+ +------+ +------+ 768 | IP | 769 +------+ 770 | L2 | 771 +------+ 773 Figure 12: Example DetNet MPLS Sub-Network Formats 775 6. MPLS-Based DetNet Data Plane Solution 777 6.1. DetNet Over MPLS Encapsulation Components 779 To carry DetNet over MPLS the following is required: 781 1. A method of identifying the MPLS payload type. 783 2. A method of identifying the DetNet flow group to the processing 784 element. 786 3. A method of distinguishing DetNet OAM packets from DetNet data 787 packets. 789 4. A method of carrying the DetNet sequence number. 791 5. A suitable LSP to deliver the packet to the egress PE. 793 6. A method of carrying queuing and forwarding indication. 795 In this design an MPLS service label (the S-Label), similar to a 796 pseudowire (PW) label [RFC3985], is used to identify both the DetNet 797 flow identity and the payload MPLS payload type satisfying (1) and 798 (2) in the list above. OAM traffic discrimination happens through 799 the use of the Associated Channel method described in [RFC4385]. The 800 DetNet sequence number is carried in the DetNet Control word which 801 carries the Data/OAM discriminator. To simplify implementation and 802 to maximize interoperability two sequence number sizes are supported: 803 a 16 bit sequence number and a 28 bit sequence number. The 16 bit 804 sequence number is needed to support some types of legacy clients. 805 The 28 bit sequence number is used in situations where it is 806 necessary ensure that in high speed networks the sequence number 807 space does not wrap whilst packets are in flight. 809 The LSP used to forward the DetNet packet may be of any type (MPLS- 810 LDP, MPLS-TE, MPLS-TP [RFC5921], or MPLS-SR 811 [I-D.ietf-spring-segment-routing-mpls]). The LSP (F-Label) label 812 and/or the S-Label may be used to indicate the queue processing as 813 well as the forwarding parameters. Note that the possible use of 814 Penultimate Hop Popping (PHP) means that the only label in a received 815 label stack may be the S-Label. 817 6.2. MPLS Data Plane Encapsulation 819 Figure 13 illustrates a DetNet data plane MPLS encapsulation. The 820 MPLS-based encapsulation of the DetNet flows is a good fit for the 821 scenarios described in Section 4.2.1 and Section 4.2.2. Furthermore, 822 end to end DetNet service i.e., native DetNet deployment (see 823 Section 4.2) is also possible if DetNet end systems are capable of 824 initiating and termination MPLS encapsulated packets. 826 The MPLS-based DetNet data plane encapsulation consists of: 828 o DetNet control word (d-CW) containing sequencing information for 829 packet replication and duplicate elimination purposes, and the OAM 830 indicator. 832 o DetNet service Label (S-Label) that identifies a DetNet flow at 833 the receiving DetNet service sub-layer processing node. 835 o Zero or more Detnet MPLS Forwarding label(s) (F-Label) used to 836 direct the packet along the label switched path (LSP) to the next 837 service sub-layer processing node along the path. When 838 Penultimate Hop Popping is in use there may be no label F-Label in 839 the protocol stack on the final hop. 841 o The necessary data-link encapsulation is then applied prior to 842 transmission over the physical media. 844 DetNet MPLS-based encapsulation 846 +---------------------------------+ 847 | | 848 | DetNet App-Flow | 849 | Payload Packet | 850 | | 851 +---------------------------------+ <--\ 852 | DetNet Control Word | | 853 +---------------------------------+ +--> DetNet data plane 854 | S-Label | | MPLS encapsulation 855 +---------------------------------+ | 856 | F-Label(s) | | 857 +---------------------------------+ <--/ 858 | Data-Link | 859 +---------------------------------+ 860 | Physical | 861 +---------------------------------+ 863 Figure 13: Encapsulation of a DetNet App-Flow in an MPLS(-TP) PSN 865 6.2.1. DetNet Control Word and the DetNet Sequence Number 867 A DetNet control word (d-CW) conforms to the Generic PW MPLS Control 868 Word (PWMCW) defined in [RFC4385]. The d-CW formatted as shown in 869 Figure 14 MUST be present in all DetNet packets containing app-flow 870 data. 872 0 1 2 3 873 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 874 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 875 |0 0 0 0| Sequence Number | 876 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 878 Figure 14: DetNet Control Word 880 (bits 0 to 3) 882 Per [RFC4385], MUST be set to zero (0). 884 Sequence Number (bits 4 to 31) 886 An unsigned value implementing the DetNet sequence number. 888 A separate sequence number space MUST be maintained by the the node 889 that adds the d-CW for each DetNet app-flow. The following sequence 890 number field lengths MUST be supported: 892 0 bits 894 16 bits 896 28 bits 898 The sequence number length MUST be provisioned (configured) on a per 899 app-flow basis via configuration, e.g., the controller plane 900 described in Section 7. 902 A 0 bit sequence number field length indicates that there is no 903 DetNet sequence number used for the flow. When the length is zero, 904 the sequence number field MUST be set to zero (0) on all packets sent 905 for the flow. 907 When the sequence number field length is 16 or 28 bits for a flow, 908 the sequence number MUST be incremented by one for each new app-flow 909 packet sent. When the field length is 16 bits, d-CW bits 4 to 15 910 MUST be set to zero (0). This values carried in this field can wrap 911 and it is important to note that zero (0) is a valid field value. 912 For example, were the sequence number size is 16 bits, the sequence 913 will contain: 65535, 0, 1. In this case, zero (0) is an ordinary 914 sequence number. This differs from [RFC4448] where a sequence number 915 of zero (0) does not indicate that no sequence number field value is 916 in use. 918 The sequence number is optionally used during receive processing as 919 described below in Section 6.2.2.1 and Section 6.2.2.2. 921 6.2.2. S-Labels 923 App-flow identification at a DetNet service sub-layer is realized by 924 an S-Label. Each app-flow MUST be sent by the node that adds a d-CW 925 with a single specific S-Label value. MPLS-aware DetNet end systems 926 and edge nodes, which are by definition MPLS ingress and egress 927 nodes, MUST add and remove the d-CW and S-Label. Relay nodes MAY 928 swap S-Label values when processing an app-flow. 930 The S-Label value MUST be provisioned per app-flow via configuration, 931 e.g., via the controller plane described in Section 7. Note that 932 S-Labels provide app-flow identification at the downstream DetNet 933 service sub-layer receiver, not the sender. As such, S-Labels MUST 934 be allocated by the entity that controls the service sub-layer 935 receiving node's label space, and MAY be allocated from the platform 936 label space [RFC3031]. 938 The S-Label will normally be at the bottom of the label stack, 939 immediately preceding the d-CW. To support service sub-layer level 940 OAM, an OAM Associated Channel Header (ACH) [RFC4385] together with a 941 Generic Associated Channel Label (GAL) [RFC5586] MAY be used in place 942 of a d-CW. 944 Similarly, an Entropy Label Indicator/Entropy Label (ELI/EL) 945 [RFC6790] MAY be carried below the S-Label in the label stack in 946 networks where DetNet flows would otherwise received ECMP treatment. 947 When ELs are used, the same EL value SHOULD be used for all of the 948 packets sent using a specific S-Label to force the flow to follow the 949 same path. However, as previously stated in Section 5, the use of 950 ECMP for DetNet flows is NOT RECOMMENDED. ECMP MAY be used for non- 951 DetNet flows within a DetNet domain. 953 When receiving a DetNet MPLS flow, an implementation MUST identify 954 the app-flow associated with the incoming packet based on the 955 S-Label. When a node is using platform labels for S-Labels, no 956 additional information is needed as the S-label uniquely identifies 957 the app-flow. In the case where platform labels are not used, one or 958 more F-Labels proceeding the S-Label MUST be used together with the 959 S-Label to uniquely identify the incoming app-flows. When PHP is 960 used, the incoming interface MAY also be used to together with any 961 present F-Label(s) and the S-Label to uniquely identify an incoming 962 app-flows. Note that choice to use platform label space for S-Label 963 or S-Label plus one or more F-Labels to identify app flows is a local 964 implementation choice, with one caveat. When one or more F-labels, 965 or incoming interface, is needed together with an S-Label to uniquely 966 identify, the controller plane MUST ensure that incoming DetNet MPLS 967 packets arrive with the needed information (F-label(s) and/or 968 incoming interface); the details of such are outside the scope of 969 this document. 971 While NOT REQUIRED, the use of platform labels for S-Labels matches 972 other pseudowire encapsulations. This implementation choice also 973 impacts PEF and POF processing as described in the next section. 975 6.2.2.1. Packet Elimination Function Processing 977 Implementations MAY support the Packet Elimination Function (PEF) for 978 received DetNet MPLS flows. When supported, use of the PEF for a 979 particular app-flow MUST be provisioned via configuration, e.g., via 980 the controller plane described in Section 7. 982 After an app-flow is identified for a received DetNet MPLS packet, as 983 described above, an implementation MUST check if PEF is configured 984 for that app-flow. When configured the implementation MUST track the 985 sequence number contained in received d-CWs and MUST ensure that 986 duplicate (replicated) instances of a particular sequence number are 987 discarded. The specific mechanisms used for an implementation to 988 identify which received packets are duplicates and which are new is 989 an implementation choice. Note that per Section 6.2.1 the sequence 990 number field length may be 16 or 28 bits, and the field value can 991 wrap. 993 Note that an implementation MAY wish to constrain the maximum number 994 sequence numbers that are tracked, on platform-wide or per flow 995 basis. Some implementations MAY support the provisioning of the 996 maximum number sequence numbers that are tracked number on either a 997 platform-wide or per flow basis. 999 6.2.2.2. Packet Ordering Function Processing 1001 A function that is related to PEF is the Packet Ordering Function 1002 (POF). Implementations MAY support POF. When supported, use of the 1003 POF for a particular app-flow MUST be provisioned via configuration, 1004 e.g., via the controller plane described by Section 7. 1005 Implementations MAY required that PEF and POF be used in combination. 1006 There is no requirement related to the order of execution of the 1007 Packet Elimination and Ordering Functions in an implementation. 1009 After an app-flow is identified for a received DetNet MPLS packet, as 1010 described above, an implementation MUST check if POF is configured 1011 for that app-flow. When configured the implementation MUST track the 1012 sequence number contained in received d-CWs and MUST ensure that 1013 packets are processed in the order indicated in the received d-CW 1014 sequence number field, which may not be in the order the packets are 1015 received. As defined in Section 6.2.1 the sequence number field 1016 length may be 16 or 28 bits, is incremened by one (1) for each new 1017 app-flow packet sent, and the field value can wrap. The specific 1018 mechanisms used for an implementation to identify the order of 1019 received packets is an implementation choice. 1021 Note that an implementation MAY wish to constrain the maximum number 1022 of out of order packets that can be processed, on platform-wide or 1023 per flow basis. Some implementations MAY support the provisioning of 1024 this number on either a platform-wide or per flow basis. The number 1025 of out of order packets that can be processed also impacts the 1026 latency of a flow. 1028 6.2.3. F-Labels 1030 F-Labels are support the DetNet forwarding sub-layer. F-Labels are 1031 used to provide LSP-based connectivity between DetNet service sub- 1032 layer processing nodes. 1034 6.2.3.1. Service Sub-Layer and Packet Replication Function Processing 1036 DetNet MPLS end systems, edge nodes and relay nodes may operate at 1037 the DetNet service sub-layer with understand of app-flows and their 1038 requirements. As mentioned above, when operating at this layer such 1039 nodes can push, pop or swap (pop then push) S-Labels. In all cases, 1040 the F-Labels used for the app-flow are always replaced and this 1041 section applies. 1043 When sending a DetNet flow, Zero or more F-Labels MAY be added on top 1044 of an S-Label by the node pushing an S-Label. The F-Labels to be 1045 pushed when sending a particular app-flow MUST be provisioned per 1046 app-flow via configuration, e.g., via the controller plane discussed 1047 in Section 7. To allow for the omission of F-Labels, an 1048 implementation SHOULD also allow an outgoing interface to be 1049 provisioned. 1051 The Packet Replication Function (PRF) function MAY be supported by an 1052 implementation for outgoing DetNet flows. When supported, the same 1053 app-flow data will be sent over multiple outgoing forwarding sub- 1054 layer LSPs. To support PRF an implementation MUST support the 1055 setting of different sets of F-Labels. Hereto, to allow for the 1056 omission of F-Labels, an implementation SHOULD also allow multiple 1057 outgoing interfaces to be provisioned. PRF MUST NOT be used with 1058 app-flows configured with a d-CW sequence number field length of 0 1059 bits. 1061 When a single set of F-Labels is provisioned for a particular 1062 outgoing app-flow, that set of F-labels MUST be pushed after the 1063 S-Label is pushed. The outgoing packet is then forwarded as 1064 described below in Section 6.2.3.2. When a single outgoing interface 1065 is provisioned, the outgoing packet is then forwarded as described 1066 below in Section 6.2.3.2. 1068 When multiple sets of F-Labels or interfaces are provisioned for a 1069 particular outgoing app-flow, a copy of the outgoing packet, 1070 including the pushed S-Label, MUST be made per F-label set and 1071 outgoing interface. Each set of provisioned F-Labels are then pushed 1072 onto a copy of the packet. Each copy is then forwarded as described 1073 below in Section 6.2.3.2. 1075 As described in the previous section, when receiving a DetNet MPLS 1076 flow, an implementation identifies the app-flow associated with the 1077 incoming packet based on the S-Label. When a node is using platform 1078 labels for S-Labels, any F-Labels can be popped and the S-label 1079 uniquely identifies the app-flow. In the case where platform labels 1080 are not used, F-Label(s) MUST also be provisioned for incoming app- 1081 flows. When PHP is used, incoming interface MUST be provisioned. 1082 The provisioned information MUST then be used to identify incoming 1083 app-flows based on the combination of S-Label and F-Label(s) or 1084 incoming interface. 1086 6.2.3.2. Common F-Label Processing 1088 All DetNet aware MPLS nodes process F-Labels as needed to meet the 1089 service requirements of the DetNet flow or flows carried in the LSPs 1090 represented by the F-Labels. This includes normal push, pop and swap 1091 operations. Such processing is essentially the same type of 1092 processing enabled for TE LSPs, although the specific service 1093 parameters, or traffic specification, can differ. When the DetNet 1094 service parameters of the app-flow or flows carried in an LSP 1095 represented by an F-Label can be met by an exiting TE mechanism, the 1096 forwarding sub-layer processing node MAY be a DetNet unaware, i.e., 1097 standard, MPLS LSR. Such TE LSPs may provide LSP forwarding service 1098 as defined in, but not limited to, [RFC3209], [RFC3270], [RFC3272], 1099 [RFC3473], [RFC4875], [RFC5440], and [RFC6006]. 1101 More specifically, as mentioned above, the DetNet forwarding sub- 1102 layer provides explicit routes and allocated resources, and F-Labels 1103 are used to map to each. Explicit routes are supported based on the 1104 topmost (outermost) F-Label that is pushed or swapped and the LSP 1105 that corresponds to this label. This topmost (outgoing) label MUST 1106 be associated with a provisioned outgoing interface and, for non- 1107 point-to-point outgoing interfaces, a next hop LSR. Note that this 1108 information MUST be provisioned via configuration or the controller 1109 plane. In the previously mentioned special case where there is no 1110 added F-labels and the outgoing interface is not a point-to-point 1111 interface, the outgoing interface MUST also be associated with a next 1112 hop LSR. 1114 Resources may be allocated in a hierarchical fashion per LSP that is 1115 represented by each F-Label. Each LSP MAY be provisioned with a 1116 service parameters that dictates the specific traffic treatment to be 1117 received by the traffic carried over that LSP. Implementations of 1118 this document MUST ensure that traffic carried over each LSP 1119 represented by an F-Label receives the traffic treatment provisioned 1120 for that LSP. Typical mechanisms used to provide different treatment 1121 to different flows includes the allocation of system resources (such 1122 as queues and buffers) and provisioning or related parameters (such 1123 as shaping, and policing). Support can also be provided via an 1124 underlying network technology such IEEE802.1 TSN Section 8. Other 1125 than in the TSN case, the specific mechanisms used by a DetNet node 1126 to ensure DetNet service delivery requirements are met for supported 1127 DetNet flows is outside the scope of this document. 1129 Packets that are marked in a way that does not correspond to 1130 allocated resources, e.g., lack a provisioned F-Label, can disrupt 1131 the QoS offered to properly reserved DetNet flows by using resources 1132 allocated to the reserved flows. Therefore, the network nodes of a 1133 DetNet network: 1135 o MUST defend the DetNet QoS by discarding or remarking (to an 1136 allocated DetNet flow or non-competing non-DetNet flow) packets 1137 received that are not the subject of a completed resource 1138 allocation. 1140 o MUST NOT use a DetNet allocated resource, e.g. a queue or shaper 1141 reserved for DetNet flows, for any packet that does match the 1142 corresponding DetNet flow. 1144 o MUST ensure a QoS flow does not exceed its allocated resources or 1145 provisioned service level, 1147 o MUST ensure a CoS flow or service class does not impact the 1148 service delivered to other flows. This requirement is similar to 1149 requirement for MPLS LSRs to that CoS LSPs do not impact the 1150 resources allocated to TE LSPs, e.g., via [RFC3473]. 1152 Subsequent sections provide additional considerations related to CoS 1153 (Section 6.6.1), QoS (Section 6.6.2) and aggregation (Section 6.6.3). 1155 6.3. OAM Indication 1157 OAM follows the procedures set out in [RFC5085] with the restriction 1158 that only Virtual Circuit Connectivity Verification (VCCV) type 1 is 1159 supported. 1161 As shown in Figure 3 of [RFC5085] when the first nibble of the d-CW 1162 is 0x0 the payload following the d-CW is normal user data. However, 1163 when the first nibble of the d-CW is 0X1, the payload that follows 1164 the d-DW is an OAM payload with the OAM type indicated by the value 1165 in the d-CW Channel Type field. 1167 The reader is referred to [RFC5085] for a more detailed description 1168 of the Associated Channel mechanism, and to the DetNet work on OAM 1169 for more information DetNet OAM. 1171 6.4. Flow Aggregation 1173 [Author's note: need to revisit this section and ensure that we cover 1174 (and fully specify) desired types of aggregation.] 1176 1. Aggregate at the LSP (Forwarding) 1178 2. Aggregating DetNet flows as a new DetNet flow 1180 3. Simple Aggregation at the DetNet layer 1182 The resource control and management aspects of aggregation (including 1183 the queuing/shaping/ policing implications) will be covered in other 1184 documents. 1186 The ability to aggregate individual flows, and their associated 1187 resource control, into a larger aggregate is an important technique 1188 for improving scaling of control in the data, management and control 1189 planes. The DetNet data plane allows for the aggregation of DetNet 1190 flows, to improved scaling. There are three methods of introducing 1191 flow aggregation: 1193 [Editor's note:] 1195 The following review comments were received when this section was 1196 committed to github. 1198 General comment: We should points to the major issue of aggregation, 1199 namely the Seq.Num related problem. The aggregated flows have their 1200 own Seq.Num and those are independent. We should consider to group 1201 the aggregation techniques as per their impact on what DetNet 1202 functions they allow on a DetNet flow. (E.g., aggregation without 1203 new Aggregate.Seq.Num would prohibit usage of FR, EF and in-order- 1204 delivery function on the aggregate flow). 1206 SR based aggregation can be treated as a form of H-LSP aggregation. 1207 Should we differentiate them? What are the differences? 1209 What are the issues when aggregating of different payload types? 1210 Should we add an editor note on this? 1212 Simple-aggregation-at-the-detnet-layer: is this not the same as 1213 H-LSP? The A-label can be treated just as an additional F-Label. 1215 [Editor's note: End of review comment.] 1217 6.4.1. Aggregation at the LSP 1219 DetNet flows forwarded via MPLS can leverage MPLS-TE's existing 1220 support for hierarchical LSPs (H-LSPs), see [RFC4206]. H-LSPs are 1221 typically used to aggregate control and resources, they may also be 1222 used to provide OAM or protection for the aggregated LSPs. Arbitrary 1223 levels of aggregation naturally falls out of the definition for 1224 hierarchy and the MPLS label stack [RFC3032]. DetNet nodes which 1225 support aggregation (LSP hierarchy) map one or more LSPs (labels) 1226 into and from an H-LSP. Both carried LSPs and H-LSPs may or may not 1227 use the TC field, i.e., L-LSPs or E-LSPs. Such nodes will need to 1228 ensure that traffic from aggregated LSPs are placed (shaped/policed/ 1229 enqueued) onto the H-LSPs in a fashion that ensures the required 1230 DetNet service is preserved. 1232 Additional details of the traffic control capabilities needed at a 1233 DetNet-aware node may be covered in the new service descriptions 1234 mentioned above or in separate future documents. Management and 1235 control plane mechanisms will also need to ensure that the service 1236 required on the aggregate flow (H-LSP or DSCP) are provided, which 1237 may include the discarding or remarking mentioned in the previous 1238 sections. 1240 6.4.2. Aggregating DetNet Flows as a new DetNet flow 1242 An aggregate can be built by layering DetNet flows as shown below: 1244 +---------------------------------+ 1245 | | 1246 | DetNet Flow | 1247 | Payload Packet | 1248 | | 1249 +---------------------------------+ <--\ 1250 | DetNet Control Word | | 1251 +=================================+ | 1252 | S-Label | | 1253 +---------------------------------+ +----DetNet data plane 1254 | DetNet Control Word | | MPLS encapsulation 1255 +=================================+ | 1256 | A-Label | | 1257 +---------------------------------+ | 1258 | F-Label(s) | <--/ 1259 +---------------------------------+ 1260 | Data-Link | 1261 +---------------------------------+ 1262 | Physical | 1263 +---------------------------------+ 1264 Both the Aggregation (A) label and the S-Label have their MPLS S bit 1265 set indicating bottom of stack, and the d-CW allows the PREOF to 1266 work. 1268 It is a property of the A-label that what follows is d-CW followed by 1269 an S-Label. A relay node processing the A-label would not know the 1270 underlying payload type. This would only be known to a node that was 1271 a peer of the node imposing the S-Label. However there is no real 1272 need for it to know the payload type during aggregation processing. 1274 6.4.3. Simple Aggregation at the DetNet Layer 1276 Another approach would be not to include a d-CW for the aggregated 1277 flow. This would be functionally similar to aggregation at the 1278 forwarding sub-layer using H-LSPs, but would confine knowledge of the 1279 aggregation to the DetNet layer. Such an approach shares the 1280 disadvantage that PREOF operations would not be possible. OAM 1281 operation in this mode is for further study. 1283 +---------------------------------+ 1284 | | 1285 | DetNet Flow | 1286 | Payload Packet | 1287 | | 1288 +---------------------------------+ <--\ 1289 | DetNet Control Word | | 1290 +=================================+ | 1291 | S-Label | +----DetNet data plane 1292 +---------------------------------+ | MPLS encapsulation 1293 | A-Label | | 1294 +---------------------------------+ | 1295 | F-Label(s) | <--/ 1296 +---------------------------------+ 1297 | Data-Link | 1298 +---------------------------------+ 1299 | Physical | 1300 +---------------------------------+ 1302 6.5. Service Sub-Layer Considerations 1304 The edge and relay node internal procedures related to PREOF are 1305 implementation specific. The order of a packet elimination or 1306 replication is out of scope in this specification. However, care 1307 should be taken that the replication function does not actually 1308 loopback packets as "replicas". Looped back packets include 1309 artificial delay when the node that originally initiated the packet 1310 receives it again. Also, looped back packets may make the network 1311 condition to look healthier than it actually is (in some cases link 1312 failures are not reflected properly because looped back packets make 1313 the situation appear better than it actually is). 1315 It is important that the DetNet layer is configured such that a 1316 DetNet node never receives its own replicated packets. If it were to 1317 receive such packets the replication function would make the loop 1318 more destructive of bandwidth than a conventional unicast loop. 1319 Ultimately the TTL in the S-Label will cause the packet to die during 1320 a transient, but given the sensitivity of applications to packet 1321 latency the impact on the DetNet application would be severe. 1323 6.5.1. Edge Node Processing 1325 An edge node is resposable for matching ingress packets to the 1326 service they require and encapsulating them accordingly.An edge node 1327 may participate in the packet replication and duplication 1328 elimination. 1330 The DetNet-aware forwarder selects the egress DetNet member flow 1331 segment based on the flow identification. The mapping of ingress 1332 DetNet member flow segment to egress DetNet member flow segment may 1333 be statically or dynamically configured. Additionally the DetNet- 1334 aware forwarder does duplicate frame elimination based on the flow 1335 identification and the sequence number combination. The packet 1336 replication is also done within the DetNet-aware forwarder. During 1337 elimination and the replication process the sequence number of the 1338 DetNet member flow MUST be preserved and copied to the egress DetNet 1339 member flow. 1341 The internal design of a relay node is out of scope of this document. 1342 However the reader's attention is drawn to the need to make any PREOF 1343 state available to the packet processor(s) dealing with packets to 1344 which the PREOF functions must be applied, and to maintain that state 1345 is such as way that it is available to the packet processor operation 1346 on the next packet in the DetNet flow (which may be a duplicate, a 1347 late packet, or the next packet in sequence. 1349 [Editor's note: I think the rest of this section belongs in a new 1350 "802.1 TSN (island Interconnect) over DetNet MPLS" section.] 1352 This may be done in the DetNet layer, or where the native service 1353 processing (NSP) [RFC3985] is IEEE 802.1CB [IEEE8021CB] capable, the 1354 packet replication and duplicate elimination MAY entirely be done in 1355 the NSP, bypassing the DetNet flow encapsulation and logic entirely. 1356 This enables operating over unmodified implementations and 1357 deployments. The NSP approach works only between edge nodes and 1358 cannot make use of relay nodes. 1360 The NSP approach is useful end to end tunnel and for for "island 1361 interconnect" scenarios. However, when there is a need to do PREOF 1362 in a middle of the network, such plain edge to edge operation is not 1363 sufficient. 1365 The extended forwarder MAY copy the sequencing information from the 1366 native DetNet packet into the DetNet sequence number field and vice 1367 versa. If there is no existing sequencing information available in 1368 the native packet or the forwarder chose not to copy it from the 1369 native packet, then the extended forwarder MUST maintain a sequence 1370 number counter for each DetNet flow (indexed by the DetNet flow 1371 identification). 1373 6.5.2. Relay Node Processing 1375 A DetNet Relay node operates in the DetNet forwarding sub-layer . 1376 This processing is done within an extended forwarder function. 1377 Whether an ingress DetNet member flow receives DetNet specific 1378 processing depends on how the forwarding is programmed. Some relay 1379 nodes may be DetNet service aware, while others may be unmodified 1380 LSRs that only understand how to swicth MPLS-TE LSPs. 1382 It is also possible to treat the relay node as a transit node, see 1383 Section 6.6.3. Again, this is entirely up to how the forwarding has 1384 been programmed. 1386 6.6. Forwarding Sub-Layer Considerations 1388 6.6.1. Class of Service 1390 Class and quality of service, i.e., CoS and QoS, are terms that are 1391 often used interchangeably and confused with each other. In the 1392 context of DetNet, CoS is used to refer to mechanisms that provide 1393 traffic forwarding treatment based on aggregate group basis and QoS 1394 is used to refer to mechanisms that provide traffic forwarding 1395 treatment based on a specific DetNet flow basis. Examples of 1396 existing network level CoS mechanisms include DiffServ which is 1397 enabled by IP header differentiated services code point (DSCP) field 1398 [RFC2474] and MPLS label traffic class field [RFC5462], and at Layer- 1399 2, by IEEE 802.1p priority code point (PCP). 1401 CoS for DetNet flows carried in PWs and MPLS is provided using the 1402 existing MPLS Differentiated Services (DiffServ) architecture 1403 [RFC3270]. Both E-LSP and L-LSP MPLS DiffServ modes MAY be used to 1404 support DetNet flows. The Traffic Class field (formerly the EXP 1405 field) of an MPLS label follows the definition of [RFC5462] and 1406 [RFC3270]. The Uniform, Pipe, and Short Pipe DiffServ tunneling and 1407 TTL processing models are described in [RFC3270] and [RFC3443] and 1408 MAY be used for MPLS LSPs supporting DetNet flows. MPLS ECN MAY also 1409 be used as defined in ECN [RFC5129] and updated by [RFC5462]. 1411 6.6.2. Quality of Service 1413 In addition to explicit routes, and packet replication and 1414 elimination, described in Section 6 above, DetNet provides zero 1415 congestion loss and bounded latency and jitter. As described in 1416 [I-D.ietf-detnet-architecture], there are different mechanisms that 1417 maybe used separately or in combination to deliver a zero congestion 1418 loss service. This includes Quality of Service (QoS) mechanisms at 1419 the MPLS layer, that may be combined with the mechanisms defined by 1420 the underlying network layer such as 802.1TSN. 1422 Quality of Service (QoS) mechanisms for flow specific traffic 1423 treatment typically includes a guarantee/agreement for the service, 1424 and allocation of resources to support the service. Example QoS 1425 mechanisms include discrete resource allocation, admission control, 1426 flow identification and isolation, and sometimes path control, 1427 traffic protection, shaping, policing and remarking. Example 1428 protocols that support QoS control include Resource ReSerVation 1429 Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473]. 1430 The existing MPLS mechanisms defined to support CoS [RFC3270] can 1431 also be used to reserve resources for specific traffic classes. 1433 A baseline set of QoS capabilities for DetNet flows carried in PWs 1434 and MPLS can provided by MPLS with Traffic Engineering (MPLS-TE) 1435 [RFC3209] and [RFC3473]. TE LSPs can also support explicit routes 1436 (path pinning). Current service definitions for packet TE LSPs can 1437 be found in "Specification of the Controlled Load Quality of 1438 Service", [RFC2211], "Specification of Guaranteed Quality of 1439 Service", [RFC2212], and "Ethernet Traffic Parameters", [RFC6003]. 1440 Additional service definitions are expected in future documents to 1441 support the full range of DetNet services. In all cases, the 1442 existing label-based marking mechanisms defined for TE-LSPs and even 1443 E-LSPs are use to support the identification of flows requiring 1444 DetNet QoS. 1446 6.6.3. Cross-DetNet Flow Resource Aggregation 1448 [Editor's NOTE: Isn't this section the same as "Aggregation at the 1449 LSP". -- Address as part of aggregation section cleanup.] 1451 The ability to aggregate individual flows, and their associated 1452 resource control, into a larger aggregate is an important technique 1453 for improving scaling of control in the data, management and control 1454 planes. This document identifies the traffic identification related 1455 aspects of aggregation of DetNet flows. The resource control and 1456 management aspects of aggregation (including the queuing/shaping/ 1457 policing implications) will be covered in other documents. The data 1458 plane implications of aggregation are independent for PW/MPLS and IP 1459 encapsulated DetNet flows. 1461 DetNet flows forwarded via MPLS can leverage MPLS-TE's existing 1462 support for hierarchical LSPs (H-LSPs), see [RFC4206]. H-LSPs are 1463 typically used to aggregate control and resources, they may also be 1464 used to provide OAM or protection for the aggregated LSPs. Arbitrary 1465 levels of aggregation naturally falls out of the definition for 1466 hierarchy and the MPLS label stack [RFC3032]. DetNet nodes which 1467 support aggregation (LSP hierarchy) map one or more LSPs (labels) 1468 into and from an H-LSP. Both carried LSPs and H-LSPs may or may not 1469 use the TC field, i.e., L-LSPs or E-LSPs. Such nodes will need to 1470 ensure that traffic from aggregated LSPs are placed (shaped/policed/ 1471 enqueued) onto the H-LSPs in a fashion that ensures the required 1472 DetNet service is preserved. 1474 [NOTE: This needs to be revised:] Additional details of the traffic 1475 control capabilities needed at a DetNet-aware node may be covered in 1476 the new service descriptions mentioned above or in separate future 1477 documents. Management and control plane mechanisms will also need to 1478 ensure that the service required on the aggregate flow (H-LSP or 1479 DSCP) are provided, which may include the discarding or remarking 1480 mentioned in the previous sections. 1482 6.6.4. Layer 2 Addressing and QoS Considerations 1484 [Editor's NOTE: review and simplify this section. Doesn't this 1485 belong in the TSN section? Alternatively, describe in generic/non 1486 sub-network technology specific terms.] 1488 The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1 1489 Working Group have defined (and are defining) a number of amendments 1490 to IEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and 1491 bounded latency in bridged networks. IEEE 802.1CB [IEEE8021CB] 1492 defines packet replication and elimination functions that should 1493 prove both compatible with and useful to, DetNet networks. 1495 As is the case for DetNet, a Layer 2 network node such as a bridge 1496 may need to identify the specific DetNet flow to which a packet 1497 belongs in order to provide the TSN/DetNet QoS for that packet. It 1498 also will likely need a CoS marking, such as the priority field of an 1499 IEEE Std 802.1Q VLAN tag, to give the packet proper service. 1501 Although the flow identification methods described in IEEE 802.1CB 1502 [IEEE8021CB] are flexible, and in fact, include IP 5-tuple 1503 identification methods, the baseline TSN standards assume that every 1504 Ethernet frame belonging to a TSN stream (i.e. DetNet flow) carries 1505 a multicast destination MAC address that is unique to that flow 1506 within the bridged network over which it is carried. Furthermore, 1507 IEEE 802.1CB [IEEE8021CB] describes three methods by which a packet 1508 sequence number can be encoded in an Ethernet frame. 1510 Ensuring that the proper Ethernet VLAN tag priority and destination 1511 MAC address are used on a DetNet/TSN packet may require further 1512 clarification of the customary L2/L3 transformations carried out by 1513 routers and edge label switches. Edge nodes may also have to move 1514 sequence number fields among Layer 2, PW, and IP encapsulations. 1516 6.6.5. Time Synchronization 1518 [Editor's Note: A detailed discussion of time synchronization is 1519 outside the scope of this document, and the production of a 1520 specialist text discussing this topic is encouraged. This section 1521 will be updated/removed if such a document is available before 1522 publication of this text.] 1524 Time synchronization is important both from the perspective of 1525 operating the DetNet network itself and from the perspective of 1526 transferring time across the network between client applications. 1527 Some clients may be able to use the DetNet as their provider of time 1528 and frequency, others may require the DetNet to transfer time between 1529 a client clock source and a client clock user. 1531 For example, [RFC8169] describes a method of recording the packet 1532 queuing time in an MPLS LSR on a packet by per packet basis and 1533 forwarding this information to the egress edge system. This allows 1534 compensation for any variable packet queuing delay to be applied at 1535 the packet receiver. Other mechanisms for IP/MPLS networks are 1536 defined based on IEEE Standard 1588 [IEEE1588], such as ITU-T 1537 [G.8275.1] and [G.8275.2]. 1539 A more detailed discussion of time synchronization is outside the 1540 scope of this document. 1542 7. Controller Plane (Management and Control) Considerations 1544 While management plane and control planes are traditionally 1545 considered separately, from the Data Plane perspective there is no 1546 practical difference based on the origin of flow provisioning 1547 information, and the DetNet architecture 1548 [I-D.ietf-detnet-architecture] refers to these collectively as the 1549 'Controller Plane'. This document therefore does not distinguish 1550 between information provided by distributed control plane protocols, 1551 e.g., RSVP-TE [RFC3209] and [RFC3473], or by centralized network 1552 management mechanisms, e.g., RestConf [RFC8040], YANG [RFC7950], and 1553 the Path Computation Element Communication Protocol (PCEP) 1554 [I-D.ietf-pce-pcep-extension-for-pce-controller] or any combination 1555 thereof. Specific considerations and requirements for the DetNet 1556 Controller Plane are discussed in Section 7.6. 1558 7.1. S-Label and F-Label Assignment and Distribution 1560 [Editor's note - we may need additional text on resource allocation 1561 in this section.] 1563 DetNet S-Labels (see Section 6.2.2 for their definition) are similar 1564 to other MPLS service labels that denote the contents of the MPLS 1565 packet payload such as a layer 2 pseudowire, an IP packet that is 1566 routed in a VPN context with a private address, or an Ethernet 1567 virtual private network (EVPN) service. 1569 S-Labels are expected to be allocated in the same manner as any other 1570 service labels. S-Labels uniquely identify a particular DetNet flow, 1571 and are local to the node on which the label is allocated. In the 1572 DetNet service sub-layer the explicit route consists of the set of 1573 Relay Nodes that the DetNet flow must traverse. They can be used to 1574 identify the DetNet flow that a packet belongs to as it traverses a 1575 particular node in a DetNet domain. Because labels are local to each 1576 node rather than being a global identifier within a domain, they must 1577 be advertised to their upstream DetNet service-aware peer nodes 1578 (e.g., a DetNet MPLS End System as shown in Figure 3, or a DetNet 1579 Relay or Edge Node as shown in Figure 7) and interpreted in the 1580 context of their received F-Label. 1582 As discussed in Section 4, the forwarding sub-layer uses one or more 1583 F-Labels to forward DetNet packets between DetNet service-aware nodes 1584 along explicitly defined routes at the DetNet forwarding sub-layer, 1585 which in the context of this document is the MPLS layer. F-Labels 1586 can also provide context for an S-Label. In the DetNet Forwarding 1587 (MPLS) sub-layer the explicit route consists of the set of DetNet 1588 nodes which are LSRs, links, and possibly link bundle members and 1589 queues that the DetNet packets of a flow must traverse between nodes 1590 in the DetNet service sub-layer (i.e. between a specific Edge Node 1591 and the next hop Relay Node, between specific Relay Nodes, and 1592 between a specific Relay node and the egress Edge Node. Resource 1593 allocation corresponding to the set of Services supported over the 1594 forwarding sub-layer, which may or may not include aggregation, is 1595 required at this sub-layer. Explicit routes are used to ensure that 1596 packets are routed through the resources that have been reserved for 1597 them, and hence provide the DetNet application with the required 1598 service. Multiple F-Labels may be pushed after an S-Label and there 1599 is no requirement for all F-Labels to be controlled via the same 1600 controller mechanisms. For example in EVPN, some labels are 1601 distributed using BGP while others are distributed using LDP or RSVP. 1603 Whether configuring, calculating and instantiating these routes is a 1604 single-stage or multi-stage process, or in a centralized or 1605 distributed manner, is out of scope of this document. 1607 There are a number of approaches that could be used to provide 1608 explicit routes and resource allocation in the MPLS layer: 1610 o The path could be explicitly set up by a controller which 1611 calculates the path and explicitly configures each node along that 1612 path with the appropriate forwarding and resource allocation 1613 information. 1615 o The path could be set up using RSVP-TE signaling. 1617 o The path could be implemented using MPLS-based segment routing 1618 when extended to support resource allocation. 1620 See Section 7.6 for further discussion of these alternatives. 1622 Much like other MPLS labels, there are a number of alternatives 1623 available for DetNet S-Label and F-Label advertisement to an upstream 1624 peer node. These include distributed signaling protocols such as 1625 RSVP-TE, centralized label distribution via a controller that manages 1626 both the sender and the receiver using NETCONF/YANG, BGP, PCEP, etc., 1627 and hybrid combinations of the two. The details of the controller 1628 plane solution required for the label distribution and the management 1629 of the label number space are out of scope of this document, but as 1630 mentioned above, there are particular DetNet considerations and 1631 requirements that are discussed in Section 7.6. 1633 7.2. Packet Replication, Elimination, and Ordering (PREOF) 1635 The controller plane protocol solution required for managing the 1636 PREOF processing is outside the scope of this document. That said, 1637 it should be noted that the ability to determine, for a particular 1638 flow, optimal packet replication and elimination points in the DetNet 1639 domain requires explicit support. There are be capabilities that can 1640 be used, or extended, for example GMPLS end-to-end recovery [RFC4872] 1641 and GMPLS segment recovery [RFC4873]. 1643 7.3. Contention Loss and Jitter Reduction 1645 As discussed in Section 1, this document does not specify the 1646 mechanisms needed to eliminate contention loss or reduce jitter for 1647 DetNet flows at the DetNet forwarding sub-layer. The ability to 1648 manage node and link resources to be able to provide these functions 1649 will be a necessary part of the DetNet controller plane. It will 1650 also be necessary to be able to control the required queuing 1651 mechanisms used to provide these functions along a flow's path 1652 through the network. See Section 7.6 for further discussion of these 1653 requirements. 1655 7.4. Bidirectional Traffic 1657 Some DetNet applications generate bidirectional traffic. Using MPLS 1658 definitions [RFC5654] there are associated bidirectional flows, and 1659 co-routed bidirectional flows. MPLS defines a point-to-point 1660 associated bidirectional LSP as consisting of two unidirectional 1661 point-to-point LSPs, one from A to B and the other from B to A, which 1662 are regarded as providing a single logical bidirectional forwarding 1663 path. This would be analogous of standard IP routing, or PWs running 1664 over two reciprocal unidirection LSPs. MPLS defines a point-to-point 1665 co-routed bidirectional LSP as an associated bidirectional LSP which 1666 satisfies the additional constraint that its two unidirectional 1667 component LSPs follow the same path (in terms of both nodes and 1668 links) in both directions. An important property of co-routed 1669 bidirectional LSPs is that their unidirectional component LSPs share 1670 fate. In both types of bidirectional LSPs, resource reservations may 1671 differ in each direction. The concepts of associated bidirectional 1672 flows and co-routed bidirectional flows can be applied to DetNet 1673 flows. 1675 While the MPLS data plane must support bidirectional DetNet flows, 1676 there are no special bidirectional features with respect to the data 1677 plane other than the need for the two directions of a co-routed 1678 bidirectional flow to take the same path. Fate sharing and 1679 associated vs co-routed bidirectional flows can be managed at the 1680 control level. Note that there is no stated requirement for 1681 bidirectional DetNet flows to be supported using the same MPLS Labels 1682 in each direction. 1684 DetNet's use of PREOF may increase the complexity of using co-routing 1685 bidirectional flows, since if PREOF is used, then the replication 1686 points in one direction would have to match the elimination points in 1687 the other direction, and vice versa, and the optimal points for these 1688 functions in one direction may not match the optimal points in the 1689 other. 1691 Control and management mechanisms will need to support bidirectional 1692 flows, but the specification of such mechanisms are out of scope of 1693 this document. Related control plan mechanisms have been defined in 1694 [RFC3473], [RFC6387] and [RFC7551]. 1696 This is further discussed in Section 7.6. 1698 7.5. Flow Aggregation Control 1700 Section 6.4 discusses the use of flow aggregation in DetNet. It 1701 includes flow aggregation accomplished through the use of 1702 hierarchical LSPs, aggregating multiple DetNet flows into a single 1703 new DetNet flow, and simple aggregation at the DetNet layer. It will 1704 be the responsibility of the DetNet controller plane to be able to 1705 properly provision the use of these mechanisms. These requirements 1706 are included in the next section. 1708 7.6. DetNet Controller (Control and Management) Plane Requirements 1710 While the definition of controller plane for DetNet is out of the 1711 scope of this document, there are particular considerations and 1712 requirements for such that result from the unique characteristics of 1713 the DetNet architecture [I-D.ietf-detnet-architecture] and data plane 1714 as defined herein. 1716 The primary requirements of the DetNet controller plane are that it 1717 must be able to: 1719 o Instantiate DetNet flows in a DetNet domain (which may include 1720 some or all of explicit path and PREOF replication and elimination 1721 node determination, link bandwidth reservations, node buffer and 1722 other resource reservations, specification of required queuing 1723 disciplines along the path, ability to manage bidirectional flows, 1724 etc.) as needed for a flow. 1726 o Manage DetNet S-Label and F-Label allocation and distribution, 1727 when the DetNet MPLS encapsulation is in use 1729 o The ability to support DetNet flow aggregation 1731 o Advertise static and dynamic node and link resources such as 1732 capabilities and adjacencies to other network nodes (for dynamic 1733 signaling approaches) or to network controllers (for centralized 1734 approaches) 1736 o Scale to handle the number of DetNet flows expected in a domain 1737 (which may require per-flow signaling or provisioning) 1739 o Provision flow identification information at each of the nodes 1740 along the path, and it may differ depending on the location in the 1741 network and the DetNet functionality (e.g. transit node vs. relay 1742 node). 1744 These requirements, as stated earlier, could be satisfied using 1745 distributed control protocol signaling (such as RSVP-TE), centralized 1746 network management provisioning mechanisms (such as BGP, PCEP, YANG 1747 [I-D.ietf-detnet-flow-information-model], etc.) or hybrid 1748 combinations of the two, and could also make use of MPLS-based 1749 segment routing. 1751 In the abstract, the results of either distributed signaling or 1752 centralized provisioning are equivalent from a DetNet data plane 1753 perspective - flows are instantiated, explicit routes are determined, 1754 resources are reserved, and packets are forwarded through the domain 1755 using the MPLS data plane. 1757 However, from a practical and implementation standpoint, they are not 1758 equivalent at all. Some approaches are more scalable than others in 1759 terms of signaling load on the network. Some can take advantage of 1760 global tracking of resources in the DetNet domain for better overall 1761 network resource optimization. Some are more resilient than others 1762 if link, node, or management equipment failures occur. While a 1763 detailed analysis of the control plane alternatives is out of the 1764 scope of this document, the requirements from this document can be 1765 used as the basis of a later analysis of the alternatives. 1767 8. DetNet MPLS Operation Over IEEE 802.1 TSN Sub-Networks 1769 [Editor's note: this is a place holder section. A standalone section 1770 on MPLS over IEEE 802.1 TSN. Includes RFC2119 Language.] 1772 This section covers how DetNet MPLS flows operate over an IEEE 802.1 1773 TSN sub-network. Figure 15 illustrates such a scenario, where two 1774 MPLS (DetNet) nodes are interconnected by a TSN sub-network. Node-1 1775 is single homed and Node-2 is dual-homed. MPLS nodes can be (1) 1776 DetNet MPLS End System, (2) DetNet MPLS Edge or Relay node or (3) 1777 MPLS Transit node. 1779 Note: in case of MPLS Transit node there is no DetNet Service sub- 1780 layer processing. 1782 MPLS (DetNet) MPLS (DetNet) 1783 Node-1 Node-2 1785 +----------+ +----------+ 1786 <--| Service* |-- DetNet flow ---| Service* |--> 1787 +----------+ +----------+ 1788 |Forwarding| |Forwarding| 1789 +--------.-+ <-TSN Str-> +-.-----.--+ 1790 \ ,-------. / / 1791 +----[ TSN-Sub ]---+ / 1792 [ Network ]--------+ 1793 `-------' 1794 <---------------- DetNet MPLS ---------------> 1796 Note: * no service sub-layer required for transit nodes 1798 Figure 15: DetNet Enabled MPLS Network Over a TSN Sub-Network 1800 The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1 1801 Working Group have defined (and are defining) a number of amendments 1802 to IEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and 1803 bounded latency in bridged networks. Furthermore IEEE 802.1CB 1804 [IEEE8021CB] defines frame replication and elimination functions for 1805 reliability that should prove both compatible with and useful to, 1806 DetNet networks. All these functions have to identify flows those 1807 require TSN treatment. 1809 As is the case for DetNet, a Layer 2 network node such as a bridge 1810 may need to identify the specific DetNet flow to which a packet 1811 belongs in order to provide the TSN/DetNet QoS for that packet. It 1812 also may need a CoS marking, such as the priority field of an IEEE 1813 Std 802.1Q VLAN tag, to give the packet proper service. 1815 The challange for MPLS DeNet flows is that the protocol interworking 1816 function defined in IEEE 802.1CB [IEEE8021CB] works only for IP 1817 flows. The aim of the protocol interworking function is to convert 1818 an ingress flow to use a specific multicast destination MAC address 1819 and VLAN, for example to direct the packets through a specific path 1820 inside the bridged network. A similar interworking pair at the other 1821 end of the TSN sub-network would restore the packet to its original 1822 destination MAC address and VLAN. 1824 As protocol interworking function defined in [IEEE8021CB] does not 1825 work for MPLS labeled flows, the DetNet MPLS nodes MUST ensure proper 1826 TSN sub-network specific Ethernet encapsulation of the DetNet MPLS 1827 packets. For a given TSN Stream (i.e., DetNet flow) an MPLS (DetNet) 1828 node MUST behave as a TSN-aware Talker or a Listener inside the TSN 1829 sub-network. 1831 8.1. Mapping of TSN Stream ID and Sequence Number 1833 TSN capable MPLS (DetNet) nodes are TSN-aware Talker/Listener as 1834 shown in Figure 16. MPLS (DetNet) node MUST provide the TSN sub- 1835 network specific Ethernet encapsulation over the link(s) towards the 1836 sub-network. An TSN-aware MPLS (DetNet) node MUST support the 1837 following TSN components: 1839 1. For recognizing flows: 1841 * Stream Identification (MPLS-flow-aware) 1843 2. For FRER used inside the TSN domain, additonaly: 1845 * Sequencing function (MPLS-flow-aware) 1847 * Sequence encode/decode function 1849 3. For FRER when the node is a TSN replication or elimination point, 1850 additionally: 1852 * Stream splitting function 1854 * Individual recovery function 1856 [Editor's note: Should we added here requirements regarding IEEE 1857 802.1Q C-VLAN component?] 1859 The Stream Identification and The Sequencing functions are slightly 1860 modified for frames passed down the protocol stack from the upper 1861 layers. 1863 Stream Identification MUST pair MPLS flows and TSN Streams and encode 1864 that in data plane formats as well. The packet's stream_handle 1865 subparameter (see IEEE 802.1CB [IEEE8021CB]) inside the Talker/ 1866 Listener is defined based on the Flow-ID used in the upper DetNet 1867 MPLS layer. Stream Identification function MUST encode Ethernet 1868 header fields namely (1) the destination MAC-address, (2) the VLAN-ID 1869 and (3) priority parameters with TSN sub-network specific values. 1870 Encoding is provided for the frame passed down the stack from the 1871 upper layers. 1873 The sequence generation function resides in the Sequencing function. 1874 It generates a sequence_number subparameter for each packet of a 1875 Stream passed down to the lower layers. Sequencing function MUST 1876 copy sequence information from the MPLS d-CW of the packet to the 1877 sequence_number subparameter for the frame passed down the stack from 1878 the upper layers. 1880 MPLS (DetNet) 1881 Node-1 1882 <----------> 1884 +----------+ 1885 <--| Service |-- DetNet flow ------------------ 1886 +----------+ 1887 |Forwarding| 1888 +----------+ +---------------+ 1889 | L2 with |<---| L2 Relay with |---- TSN ---- 1890 | TSN | | TSN function | Stream 1891 +-----.----+ +--.---------.--+ 1892 \__________/ \______ 1894 TSN-aware 1895 Talker / TSN-Bridge 1896 Listener Relay 1898 <--------- TSN sub-network ------------ 1900 Figure 16: MPLS (DetNet) Node with TSN Functions 1902 The Sequence encode/decode function MUST support the Redundancy tag 1903 (R-TAG) format as per Clause 7.8 of IEEE 802.1CB [IEEE8021CB]. 1905 8.2. TSN Usage of FRER 1907 TSN Streams supporting DetNet flows may use Frame Replication and 1908 Elimination for Redundancy (FRER) [802.1CB] based on the loss service 1909 requirements of the TSN Stream, which is derived from the DetNet 1910 service requirements of the DetNet mapped flow. The specific 1911 operation of FRER is not modified by the use of DetNet and follows 1912 IEEE 802.1CB [IEEE8021CB]. 1914 FRER function and the provided service recovery is available only 1915 within the TSN sub-network however as the Stream-ID and the TSN 1916 sequence number are paired with the MPLS flow parameters they can be 1917 combined with PREOF functions. 1919 8.3. Management and Control Implications 1921 [Editor's note: This section is TBD Covers Creation, mapping, removal 1922 of TSN Stream IDs, related parameters and,when needed, configuration 1923 of FRER. Supported by management/control plane.] 1925 9. DetNet MPLS Operation over DetNet IP PSNs 1927 This section specifies the DetNet encapsulation over an IP network. 1928 The approach is modeled on the operation of MPLS and PseudoWires (PW) 1929 over an IP Packet Switched Network (PSN) [RFC3985][RFC4385][RFC7510]. 1930 It maps the MPLS data plane encapsulation described in Section 6.2 to 1931 the DetNet IP data plane define in [I-D.ietf-detnet-dp-sol-ip]. 1933 To carry DetNet with full functionality at the DetNet layer over an 1934 IP network, the following components are required (these are a subset 1935 of the requirements for MPLS encapsulation listed in Section 6.1): 1937 1. A method of identifying the DetNet flow group to the processing 1938 element. 1940 2. A method of carrying the DetNet sequence number. 1942 3. A method of distinguishing DetNet OAM packets from DetNet data 1943 packets. 1945 4. A method of carrying queuing and forwarding indication. 1947 These requirements are satisfied by the DetNet over MPLS 1948 Encapsulation described in Section 6.2. 1950 This document builds on the the specification of MPLS over UDP and IP 1951 defined in [RFC7510]. It replaces the the F-Label(s) used in 1952 Section 6.2 with UDP and IP headers. The UDP and IP header 1953 information is used to identify DetNet flows, including member flows, 1954 per [I-D.ietf-detnet-dp-sol-ip]. The resulting encapsulation is 1955 shown in Figure 17. 1957 Note that this encapsulation works equally well with IPv4 and IPv6. 1959 +---------------------------------+ 1960 | | 1961 | DetNet App-Flow | 1962 | Payload Packet | 1963 | | 1964 +---------------------------------+ <--\ 1965 | DetNet Control Word | | 1966 +---------------------------------+ +--> DetNet data plane 1967 | S-Label | | MPLS encapsulation 1968 +---------------------------------+ <--X. 1969 | UDP Header | | 1970 +---------------------------------+ +--> DetNet data plane 1971 | IP Header | | IP encapsulation 1972 +---------------------------------+ <--/ 1973 | Data-Link | 1974 +---------------------------------+ 1975 | Physical | 1976 +---------------------------------+ 1978 Figure 17: IP Encapsulation of DetNet MPLS 1980 d-CW and and S-Labels are used as defined in Section 6.2 and are not 1981 modified by this section. 1983 To support outgoing DetNet MPLS over IP, an implementation MUST 1984 support the provisioning of IP/UDP header information in place of 1985 sets of F-Labels. Note that multiple sets of F-Labels can be 1986 provisioned to support PRF on transmitted DetNet flows and therefore, 1987 when PRF is supported, multiple IP/UDP headers MAY be provisioned. 1988 When multiple IP/UDP headers are provisioned for a particular 1989 outgoing app-flow, a copy of the outgoing packet, including the 1990 pushed S-Label, MUST be made for each. The headers for each outgoing 1991 packet MUST be based on the configuration information and as defined 1992 in [RFC7510], with one exception. The one exceptions is that the UDP 1993 Source Port value MUST be set to uniquely identify the DetNet 1994 (forwarding sub-layer) flow. The packet MUST then be handed as a 1995 DetNet IP packet, per [I-D.ietf-detnet-dp-sol-ip]. 1997 To support receive processing an implementation MUST also support the 1998 provisioning of received IP/UDP header information. When S-Labels 1999 are taken from platform label space, all that is required is to 2000 provision that receiving IP/UDP encapsulated DetNet MPLS packets is 2001 permitted. Once the IP/UDP header is stripped, the S-label uniquely 2002 identifies the app-flow. When S-Labels are not taken from platform 2003 label space, IP/UDP header information MUST be provisioned. The 2004 provisioned information MUST then be used to identify incoming app- 2005 flows based on the combination of S-Label and incoming IP/UDP header. 2006 Normal receive processing, including PEOF can then take place. 2008 10. Security Considerations 2010 The security considerations of DetNet in general are discussed in 2011 [I-D.ietf-detnet-architecture] and [I-D.sdt-detnet-security]. Other 2012 security considerations will be added in a future version of this 2013 draft. 2015 11. IANA Considerations 2017 This document makes no IANA requests. 2019 12. Contributors 2021 RFC7322 limits the number of authors listed on the front page of a 2022 draft to a maximum of 5, far fewer than the many individuals below 2023 who made important contributions to this draft. The editor wishes to 2024 thank and acknowledge each of the following authors for contributing 2025 text to this draft. See also Section 13. 2027 Loa Andersson 2028 Huawei 2029 Email: loa@pi.nu 2031 Yuanlong Jiang 2032 Huawei 2033 Email: jiangyuanlong@huawei.com 2035 Norman Finn 2036 Huawei 2037 3101 Rio Way 2038 Spring Valley, CA 91977 2039 USA 2040 Email: norman.finn@mail01.huawei.com 2042 Janos Farkas 2043 Ericsson 2044 Magyar Tudosok krt. 11. 2045 Budapest 1117 2046 Hungary 2047 Email: janos.farkas@ericsson.com 2049 Carlos J. Bernardos 2050 Universidad Carlos III de Madrid 2051 Av. Universidad, 30 2052 Leganes, Madrid 28911 2053 Spain 2054 Email: cjbc@it.uc3m.es 2056 Tal Mizrahi 2057 Marvell 2058 6 Hamada st. 2059 Yokneam 2060 Israel 2061 Email: talmi@marvell.com 2063 Lou Berger 2064 LabN Consulting, L.L.C. 2065 Email: lberger@labn.net 2067 Stewart Bryant 2068 Huawei Technologies 2069 Email: stewart.bryant@gmail.com 2071 Mach Chen 2072 Huawei Technologies 2073 Email: mach.chen@huawei.com 2075 Andrew G. Malis 2076 Huawei Technologies 2077 Email: agmalis@gmail.com 2079 13. Acknowledgements 2081 The author(s) ACK and NACK. 2083 The following people were part of the DetNet Data Plane Solution 2084 Design Team: 2086 Jouni Korhonen 2088 Janos Farkas 2090 Norman Finn 2092 Balazs Varga 2094 Loa Andersson 2096 Tal Mizrahi 2098 David Mozes 2100 Yuanlong Jiang 2101 Andrew Malis 2103 Carlos J. Bernardos 2105 The DetNet chairs serving during the DetNet Data Plane Solution 2106 Design Team: 2108 Lou Berger 2110 Pat Thaler 2112 Thanks for Stewart Bryant for his extensive review of the previous 2113 versions of the document. 2115 14. References 2117 14.1. Normative References 2119 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2120 Requirement Levels", BCP 14, RFC 2119, 2121 DOI 10.17487/RFC2119, March 1997, 2122 . 2124 [RFC2211] Wroclawski, J., "Specification of the Controlled-Load 2125 Network Element Service", RFC 2211, DOI 10.17487/RFC2211, 2126 September 1997, . 2128 [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification 2129 of Guaranteed Quality of Service", RFC 2212, 2130 DOI 10.17487/RFC2212, September 1997, 2131 . 2133 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2134 Label Switching Architecture", RFC 3031, 2135 DOI 10.17487/RFC3031, January 2001, 2136 . 2138 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2139 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2140 Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001, 2141 . 2143 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2144 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2145 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 2146 . 2148 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2149 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2150 Protocol Label Switching (MPLS) Support of Differentiated 2151 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 2152 . 2154 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2155 in Multi-Protocol Label Switching (MPLS) Networks", 2156 RFC 3443, DOI 10.17487/RFC3443, January 2003, 2157 . 2159 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 2160 Switching (GMPLS) Signaling Resource ReserVation Protocol- 2161 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 2162 DOI 10.17487/RFC3473, January 2003, 2163 . 2165 [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) 2166 Hierarchy with Generalized Multi-Protocol Label Switching 2167 (GMPLS) Traffic Engineering (TE)", RFC 4206, 2168 DOI 10.17487/RFC4206, October 2005, 2169 . 2171 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2172 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2173 Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385, 2174 February 2006, . 2176 [RFC5085] Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual 2177 Circuit Connectivity Verification (VCCV): A Control 2178 Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085, 2179 December 2007, . 2181 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2182 Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2183 2008, . 2185 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2186 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2187 Class" Field", RFC 5462, DOI 10.17487/RFC5462, February 2188 2009, . 2190 [RFC7510] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black, 2191 "Encapsulating MPLS in UDP", RFC 7510, 2192 DOI 10.17487/RFC7510, April 2015, 2193 . 2195 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2196 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2197 May 2017, . 2199 14.2. Informative References 2201 [G.8275.1] 2202 International Telecommunication Union, "Precision time 2203 protocol telecom profile for phase/time synchronization 2204 with full timing support from the network", ITU-T 2205 G.8275.1/Y.1369.1 G.8275.1, June 2016, 2206 . 2208 [G.8275.2] 2209 International Telecommunication Union, "Precision time 2210 protocol telecom profile for phase/time synchronization 2211 with partial timing support from the network", ITU-T 2212 G.8275.2/Y.1369.2 G.8275.2, June 2016, 2213 . 2215 [I-D.ietf-detnet-architecture] 2216 Finn, N., Thubert, P., Varga, B., and J. Farkas, 2217 "Deterministic Networking Architecture", draft-ietf- 2218 detnet-architecture-11 (work in progress), February 2019. 2220 [I-D.ietf-detnet-dp-sol-ip] 2221 Korhonen, J., Varga, B., "DetNet IP Data Plane 2222 Encapsulation", 2018. 2224 [I-D.ietf-detnet-flow-information-model] 2225 Farkas, J., Varga, B., Cummings, R., and Y. Jiang, "DetNet 2226 Flow Information Model", draft-ietf-detnet-flow- 2227 information-model-03 (work in progress), March 2019. 2229 [I-D.ietf-pce-pcep-extension-for-pce-controller] 2230 Zhao, Q., Li, Z., Negi, M., and C. Zhou, "PCEP Procedures 2231 and Protocol Extensions for Using PCE as a Central 2232 Controller (PCECC) of LSPs", draft-ietf-pce-pcep- 2233 extension-for-pce-controller-01 (work in progress), 2234 February 2019. 2236 [I-D.ietf-spring-segment-routing-mpls] 2237 Bashandy, A., Filsfils, C., Previdi, S., Decraene, B., 2238 Litkowski, S., and R. Shakir, "Segment Routing with MPLS 2239 data plane", draft-ietf-spring-segment-routing-mpls-18 2240 (work in progress), December 2018. 2242 [I-D.sdt-detnet-security] 2243 Mizrahi, T., Grossman, E., Hacker, A., Das, S., 2244 "Deterministic Networking (DetNet) Security 2245 Considerations, draft-sdt-detnet-security, work in 2246 progress", 2017. 2248 [IEEE1588] 2249 IEEE, "IEEE 1588 Standard for a Precision Clock 2250 Synchronization Protocol for Networked Measurement and 2251 Control Systems Version 2", 2008. 2253 [IEEE8021CB] 2254 Finn, N., "Draft Standard for Local and metropolitan area 2255 networks - Seamless Redundancy", IEEE P802.1CB 2256 /D2.1 P802.1CB, December 2015, 2257 . 2260 [IEEE8021Q] 2261 IEEE 802.1, "Standard for Local and metropolitan area 2262 networks--Bridges and Bridged Networks (IEEE Std 802.1Q- 2263 2014)", 2014, . 2265 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 2266 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 2267 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 2268 September 1997, . 2270 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2271 "Definition of the Differentiated Services Field (DS 2272 Field) in the IPv4 and IPv6 Headers", RFC 2474, 2273 DOI 10.17487/RFC2474, December 1998, 2274 . 2276 [RFC3272] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X. 2277 Xiao, "Overview and Principles of Internet Traffic 2278 Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002, 2279 . 2281 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 2282 Edge-to-Edge (PWE3) Architecture", RFC 3985, 2283 DOI 10.17487/RFC3985, March 2005, 2284 . 2286 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 2287 "Encapsulation Methods for Transport of Ethernet over MPLS 2288 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 2289 . 2291 [RFC4872] Lang, J., Ed., Rekhter, Y., Ed., and D. Papadimitriou, 2292 Ed., "RSVP-TE Extensions in Support of End-to-End 2293 Generalized Multi-Protocol Label Switching (GMPLS) 2294 Recovery", RFC 4872, DOI 10.17487/RFC4872, May 2007, 2295 . 2297 [RFC4873] Berger, L., Bryskin, I., Papadimitriou, D., and A. Farrel, 2298 "GMPLS Segment Recovery", RFC 4873, DOI 10.17487/RFC4873, 2299 May 2007, . 2301 [RFC4875] Aggarwal, R., Ed., Papadimitriou, D., Ed., and S. 2302 Yasukawa, Ed., "Extensions to Resource Reservation 2303 Protocol - Traffic Engineering (RSVP-TE) for Point-to- 2304 Multipoint TE Label Switched Paths (LSPs)", RFC 4875, 2305 DOI 10.17487/RFC4875, May 2007, 2306 . 2308 [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation 2309 Element (PCE) Communication Protocol (PCEP)", RFC 5440, 2310 DOI 10.17487/RFC5440, March 2009, 2311 . 2313 [RFC5586] Bocci, M., Ed., Vigoureux, M., Ed., and S. Bryant, Ed., 2314 "MPLS Generic Associated Channel", RFC 5586, 2315 DOI 10.17487/RFC5586, June 2009, 2316 . 2318 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 2319 Sprecher, N., and S. Ueno, "Requirements of an MPLS 2320 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 2321 September 2009, . 2323 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 2324 L., and L. Berger, "A Framework for MPLS in Transport 2325 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 2326 . 2328 [RFC6003] Papadimitriou, D., "Ethernet Traffic Parameters", 2329 RFC 6003, DOI 10.17487/RFC6003, October 2010, 2330 . 2332 [RFC6006] Zhao, Q., Ed., King, D., Ed., Verhaeghe, F., Takeda, T., 2333 Ali, Z., and J. Meuric, "Extensions to the Path 2334 Computation Element Communication Protocol (PCEP) for 2335 Point-to-Multipoint Traffic Engineering Label Switched 2336 Paths", RFC 6006, DOI 10.17487/RFC6006, September 2010, 2337 . 2339 [RFC6073] Martini, L., Metz, C., Nadeau, T., Bocci, M., and M. 2340 Aissaoui, "Segmented Pseudowire", RFC 6073, 2341 DOI 10.17487/RFC6073, January 2011, 2342 . 2344 [RFC6387] Takacs, A., Berger, L., Caviglia, D., Fedyk, D., and J. 2345 Meuric, "GMPLS Asymmetric Bandwidth Bidirectional Label 2346 Switched Paths (LSPs)", RFC 6387, DOI 10.17487/RFC6387, 2347 September 2011, . 2349 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2350 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2351 RFC 6790, DOI 10.17487/RFC6790, November 2012, 2352 . 2354 [RFC7551] Zhang, F., Ed., Jing, R., and R. Gandhi, Ed., "RSVP-TE 2355 Extensions for Associated Bidirectional Label Switched 2356 Paths (LSPs)", RFC 7551, DOI 10.17487/RFC7551, May 2015, 2357 . 2359 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 2360 RFC 7950, DOI 10.17487/RFC7950, August 2016, 2361 . 2363 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 2364 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 2365 . 2367 [RFC8169] Mirsky, G., Ruffini, S., Gray, E., Drake, J., Bryant, S., 2368 and A. Vainshtein, "Residence Time Measurement in MPLS 2369 Networks", RFC 8169, DOI 10.17487/RFC8169, May 2017, 2370 . 2372 Appendix A. Example of DetNet Data Plane Operation 2374 [Editor's note: Add a simplified example of DetNet data plane and how 2375 labels etc work in the case of MPLS-based PSN and utilizing PREOF. 2376 The figure is subject to change depending on the further DT decisions 2377 on the label handling..] 2379 Authors' Addresses 2381 Jouni Korhonen (editor) 2383 Email: jouni.nospam@gmail.com 2384 Balazs Varga (editor) 2385 Ericsson 2386 Magyar Tudosok krt. 11. 2387 Budapest 1117 2388 Hungary 2390 Email: balazs.a.varga@ericsson.com