idnits 2.17.1 draft-ietf-detnet-dp-sol-mpls-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 21, 2018) is 2015 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'Network' is mentioned on line 359, but not defined == Missing Reference: 'TBD' is mentioned on line 1612, but not defined == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-mpls-14 == Outdated reference: A later version (-13) exists of draft-ietf-detnet-architecture-08 Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet J. Korhonen, Ed. 3 Internet-Draft 4 Intended status: Standards Track B. Varga, Ed. 5 Expires: April 24, 2019 Ericsson 6 October 21, 2018 8 DetNet MPLS Data Plane Encapsulation 9 draft-ietf-detnet-dp-sol-mpls-01 11 Abstract 13 This document specifies Deterministic Networking data plane 14 encapsulation solutions. The described data plane solutions is 15 applied over an MPLS Packet Switched Networks. 17 Status of This Memo 19 This Internet-Draft is submitted in full conformance with the 20 provisions of BCP 78 and BCP 79. 22 Internet-Drafts are working documents of the Internet Engineering 23 Task Force (IETF). Note that other groups may also distribute 24 working documents as Internet-Drafts. The list of current Internet- 25 Drafts is at https://datatracker.ietf.org/drafts/current/. 27 Internet-Drafts are draft documents valid for a maximum of six months 28 and may be updated, replaced, or obsoleted by other documents at any 29 time. It is inappropriate to use Internet-Drafts as reference 30 material or to cite them other than as "work in progress." 32 This Internet-Draft will expire on April 24, 2019. 34 Copyright Notice 36 Copyright (c) 2018 IETF Trust and the persons identified as the 37 document authors. All rights reserved. 39 This document is subject to BCP 78 and the IETF Trust's Legal 40 Provisions Relating to IETF Documents 41 (https://trustee.ietf.org/license-info) in effect on the date of 42 publication of this document. Please review these documents 43 carefully, as they describe your rights and restrictions with respect 44 to this document. Code Components extracted from this document must 45 include Simplified BSD License text as described in Section 4.e of 46 the Trust Legal Provisions and are provided without warranty as 47 described in the Simplified BSD License. 49 Table of Contents 51 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 52 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 53 2.1. Terms used in this document . . . . . . . . . . . . . . . 4 54 2.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4 55 3. Requirements language . . . . . . . . . . . . . . . . . . . . 6 56 4. MPLS DetNet data plane overview . . . . . . . . . . . . . . . 6 57 4.1. Layers of DetNet data plane . . . . . . . . . . . . . . . 6 58 4.2. MPLS DetNet data plane scenarios . . . . . . . . . . . . 7 59 4.3. Packet flow example (service protection) . . . . . . . . 11 60 5. DetNet MPLS Data Considerations . . . . . . . . . . . . . . . 12 61 5.1. End-system specific considerations . . . . . . . . . . . 13 62 5.2. DetNet domain specific considerations . . . . . . . . . . 15 63 5.2.1. DetNet Layer Two Service . . . . . . . . . . . . . . 16 64 5.2.2. DetNet Routing Service (IP over MPLS) . . . . . . . . 17 65 5.3. DetNet Inter-Working Function (DN-IWF) . . . . . . . . . 18 66 5.3.1. Networks with multiple technology segments . . . . . 18 67 5.3.2. DN-IWF related considerations . . . . . . . . . . . . 19 68 6. MPLS-based DetNet data plane solution . . . . . . . . . . . . 20 69 6.1. DetNet over MPLS Encapsulation Components . . . . . . . . 20 70 6.2. MPLS data plane encapsulation . . . . . . . . . . . . . . 22 71 6.3. DetNet control word . . . . . . . . . . . . . . . . . . . 23 72 6.4. Flow Identification . . . . . . . . . . . . . . . . . . . 24 73 6.5. Indication of the DetNet Payload Type . . . . . . . . . . 24 74 6.6. OAM Indication . . . . . . . . . . . . . . . . . . . . . 25 75 6.7. Flow Aggregation . . . . . . . . . . . . . . . . . . . . 25 76 6.7.1. Aggregation at the LSP . . . . . . . . . . . . . . . 26 77 6.7.2. Aggregating DetNet flows as a new DetNet flow . . . . 26 78 6.7.3. Simple Aggregation at the DetNet layer . . . . . . . 27 79 6.8. Service Layer Considerations . . . . . . . . . . . . . . 28 80 6.8.1. Edge node processing . . . . . . . . . . . . . . . . 28 81 6.8.2. Relay node processing . . . . . . . . . . . . . . . . 29 82 6.9. Other DetNet data plane considerations . . . . . . . . . 30 83 6.9.1. Class of Service . . . . . . . . . . . . . . . . . . 30 84 6.9.2. Quality of Service . . . . . . . . . . . . . . . . . 31 85 6.9.3. Cross-DetNet flow resource aggregation . . . . . . . 32 86 6.9.4. Layer 2 addressing and QoS Considerations . . . . . . 33 87 6.9.5. Time Synchronization . . . . . . . . . . . . . . . . 33 88 7. Management and control considerations . . . . . . . . . . . . 34 89 7.1. MPLS-based data plane . . . . . . . . . . . . . . . . . . 34 90 7.1.1. S-Label assignment and distribution . . . . . . . . . 34 91 7.1.2. Explicit routes . . . . . . . . . . . . . . . . . . . 34 92 7.2. Packet replication and elimination . . . . . . . . . . . 35 93 7.3. Congestion protection and latency control . . . . . . . . 36 94 7.4. Bidirectional traffic . . . . . . . . . . . . . . . . . . 36 95 7.5. Flow aggregation control . . . . . . . . . . . . . . . . 36 96 8. DetNet IP Operation over DetNet MPLS Service . . . . . . . . 36 97 9. IEEE 802.1 TSN Interconnection over DetNet MPLS Service . . . 37 98 10. DetNet MPLS Transport Layer Operation over IEEE 802.1 TSN 99 Sub-Networks . . . . . . . . . . . . . . . . . . . . . . . . 37 100 10.1. Mapping of TSN Stream ID and Sequence Number . . . . . . 38 101 10.2. TSN Usage of FRER . . . . . . . . . . . . . . . . . . . 40 102 10.3. Management and Control Implications . . . . . . . . . . 40 103 11. DetNet MPLS Transport Layer Operation over IP DetNet PSNs . . 40 104 12. Security considerations . . . . . . . . . . . . . . . . . . . 42 105 13. IANA considerations . . . . . . . . . . . . . . . . . . . . . 42 106 14. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 43 107 15. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 45 108 16. References . . . . . . . . . . . . . . . . . . . . . . . . . 45 109 16.1. Normative references . . . . . . . . . . . . . . . . . . 45 110 16.2. Informative references . . . . . . . . . . . . . . . . . 48 111 Appendix A. Example of DetNet data plane operation . . . . . . . 50 112 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 50 114 1. Introduction 116 Deterministic Networking (DetNet) is a service that can be offered by 117 a network to DetNet flows. DetNet provides these flows with a low 118 packet loss rates and assured maximum end-to-end delivery latency. 119 General background and concepts of DetNet can be found in 120 [I-D.ietf-detnet-architecture]. 122 This document specifies the DetNet data plane and the on-wire 123 encapsulation of DetNet flows over an MPLS-based Packet Switched 124 Network (PSN). The specified encapsulation provides the building 125 blocks to enable the DetNet service layer functions and allow flow 126 identification as described in the DetNet Architecture. 128 The DetNet transport layer functionality that provides congestion 129 protection for DetNet flows is assumed to be in place in a DetNet 130 node. 132 Furthermore, this document also describes how DetNet flows are 133 identified, and how a DetNet Relay/Edge/Transit nodes works. It also 134 describes the function and operation of the Packet Replication (PRF) 135 Packet Elimination (PEF) and Packet Ordering (POF) functions in the 136 MPLS data plane. 138 This document does not define the associated control plane functions, 139 or Operations, Administration, and Maintenance (OAM). It also does 140 not specify traffic handling capabilities required to deliver 141 congestion protection and latency control for DetNet flows at the 142 DetNet transport layer. 144 2. Terminology 146 2.1. Terms used in this document 148 This document uses the terminology established in the DetNet 149 architecture [I-D.ietf-detnet-architecture] and the DetNet Data Plane 150 Solution Alternatives [I-D.ietf-detnet-dp-alt]. 152 T-Label A label used to identify the LSP used to transport a 153 DetNet flow across an MPLS PSN, e.g., a hop-by-hop 154 label used between label switching routers (LSR). 156 S-Label A DetNet "service" label that is used between DetNet 157 nodes that implement also the DetNet service layer 158 functions. An S-Label is also used to identify a 159 DetNet flow at DetNet service layer. 161 PEF A Packet Elimination Function (PEF) eliminates 162 duplicate copies of packets received by an edge or a 163 relay node to prevent excess packets flooding the 164 network, or to prevent duplicate packets being sent out 165 of the DetNet domain. 167 PRF A Packet Replication Function (PRF) replicates DetNet 168 flow packets and forwards them to one or more next hops 169 in the DetNet domain. The number of packet copies sent 170 to each next hop is a DetNet flow specific parameter at 171 the node doing the replication. PRF can be implemented 172 by an edge node, a relay node, or an end system. 174 POF A Packet Ordering Function (POF) re-orders packets 175 within a DetNet flow that are received out of order. 176 This function can be implemented by an edge node, a 177 relay node, or an end system. 179 PREOF Collective name for Packet Replication, Elimination, 180 and Ordering Functions. 182 d-CW A DetNet Control Word (d-CW) is used for sequencing and 183 identifying duplicate packets of a DetNet flow at the 184 DetNet service layer. 186 2.2. Abbreviations 188 The following abbreviations used in this document: 190 AC Attachment Circuit. 192 CE Customer Edge equipment. 194 CoS Class of Service. 196 CW Control Word. 198 d-CW DetNet Control Word. 200 DetNet Deterministic Networking. 202 DF DetNet Flow. 204 DN-IWF DetNet Inter-Working Function. 206 L2 Layer 2. 208 L2VPN Layer 2 Virtual Private Network. 210 L3 Layer 3. 212 LSR Label Switching Router. 214 MPLS Multiprotocol Label Switching. 216 MPLS-TE Multiprotocol Label Switching - Traffic Engineering. 218 MPLS-TP Multiprotocol Label Switching - Transport Profile. 220 MS-PW Multi-Segment PseudoWire (MS-PW). 222 NSP Native Service Processing. 224 OAM Operations, Administration, and Maintenance. 226 PE Provider Edge. 228 PEF Packet Elimination Function. 230 PRF Packet Replication Function. 232 PREOF Packet Replication, Elimination and Ordering Functions. 234 POF Packet Ordering Function. 236 PSN Packet Switched Network. 238 PW PseudoWire. 240 QoS Quality of Service. 242 S-PE Switching Provider Edge. 244 T-PE Terminating Provider Edge. 246 TSN Time-Sensitive Network. 248 3. Requirements language 250 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 251 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 252 "OPTIONAL" in this document are to be interpreted as described in BCP 253 14 [RFC2119] [RFC8174] when, and only when, they appear in all 254 capitals, as shown here. 256 4. MPLS DetNet data plane overview 258 4.1. Layers of DetNet data plane 260 This document describes how DetNet flows are carried over MPLS 261 networks. The DetNet Architecture, [I-D.ietf-detnet-architecture], 262 decomposes the DetNet data plane into two layers: a service layer and 263 a transport layer. The basic approach defined in this document 264 supports the DetNet service layer based on existing pseudowire (PW) 265 encapsulations and mechanisms, and supports the DetNet transport 266 layer based on existing MPLS Traffic Engineering encapsulations and 267 mechanisms. Background on PWs can be found in [RFC3985] and 268 [RFC3031]. 270 DetNet MPLS 271 . 272 . 273 +-----------+ 274 | Service | d-CW, S-Label 275 +-----------+ 276 | Transport | T-Label(s) 277 +-----------+ 278 . 279 . 281 Figure 1: DetNet adaptation to MPLS data plane 283 The MPLS DetNet data plane approach defined in this document is shown 284 in Figure 1. The service layer is supported by a DetNet control word 285 (d-CW) which conforms to the Generic PW MPLS Control Word (PWMCW) 286 defined in [RFC4385]. A d-CW identifying service label (S-Label) is 287 also used. The transport layer is supported by one or labels 288 (T-Labels). 290 A node operating on a DetNet flow in the Detnet layer, i.e. a node 291 processing a DetNet packet which has the S-label as top of stack uses 292 the local context associated with that S-label to determine what 293 local operation(s) are applied to that packet. The S-label has to be 294 unique on each edge and relay node, which is achieved by using a 295 label taken from the platform label space [RFC3031]. 297 4.2. MPLS DetNet data plane scenarios 299 TSN Edge Transit Edge TSN 300 End System Node Node Node End System 301 (T-PE) (LSR) (T-PE) 303 +---------+ +.........+ +.........+ +---------+ 304 | Appl. |<--:Svc Proxy:--End to End Svc.--:Svc Proxy:-->| Appl. | 305 +---------+ +---------+ +---------+ +---------+ 306 | TSN | |TSN| |Svc|<-- DetNet flow -->: Service :-->| TSN | 307 +---------+ +---+ +---+ +---------+ +---------+ +---------+ 308 |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| 309 +-------.-+ +--.+ +-.-+ +--.----.-+ +-.-+ +-.-+ +---.-----+ 310 : Link : / ,-----. \ : Link : / ,-----. \ 311 +........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ 312 [Network] [Network] 313 `-----' `-----' 315 |<- TSN ->| |<----- DetNet MPLS ---->| |<-- TSN --->| 317 Figure 2: A TSN over DetNet MPLS Enabled Network 319 Figure 2 shows several node types defined in 320 [I-D.ietf-detnet-architecture]. DetNet Edge Nodes sit at the 321 boundary of a DetNet domain. They are responsible for mapping non- 322 DetNet aware traffic to DetNet services. They also support the 323 imposition and disposition of the required DetNet encapsulation. 324 These are functionally similar to pseudowire (PW) Terminating 325 Provider Edge (T-PE) nodes which use MPLS-TE LSPs. 327 Native TSN flow and DetNet MPLS flow differ not only by the 328 additional MPLS specific encapsulation, but DetNet MPLS flows have on 329 each DetNet node an associated DetNet specific data structure, what 330 defines flow related characteristics and required forwarding 331 functions. Edge Nodes MUST provide a Service Proxy entity that 332 "associates" the DetNet flows and native flows at the edge of the 333 DetNet domain. It ensures that the DN Flow is properly served at the 334 Edge node (and inside the domain). 336 Transit nodes are normal MPLS Label Switching Routers (LSRs). They 337 are generally unaware of the special requirements of DetNet flows, 338 although they need to provide traffic engineering services and proper 339 QoS to the LSPs associated with DetNet flows to enhance the prospect 340 of the LSPs meeting the DetNet service requirements. Some 341 implementations of transit nodes may be DetNet aware, but such nodes 342 just support the DetNet transport layer. 344 The MPLS LSP may be provided by any MPLS method (provisioned, RSVP- 345 TE, MPLS- TP, or MPLS Segment Routing (SR)). 347 IP DetNet Relay Transit Relay IP DetNet 348 End System Node Node Node End System 349 (T-PE) (LSR) (T-PE) 350 +---------+ +---------+ 351 | Appl. |<--------------- End to End Service ---------->| Appl. | 352 +---------+ .....-----+ +-----..... +---------+ 353 | Service |<---: Service |-- DetNet flow ---| Service :-->| Service | 354 +---------+ +---------+ +---------+ +---------+ +---------+ 355 |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| 356 +-------.-+ +-.-+ +-.-+ +---.---.-+ +-.-+ +-.-+ +---.-----+ 357 : Link : / ,-----. \ : Link : / ,-----. \ 358 +........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ 359 [Network] [Network] 360 `-----' `-----' 362 |<-DN IP->| |<---- DetNet MPLS ---->| |<-DN IP->| 364 Figure 3: DetNet (DN) IP Over MPLS Network 366 Figure 3 and Figure 4, show different cases where relay nodes may be 367 used. Relay nodes are similar to edge nodes in that both are aware 368 of the needs of particular DetNet flows and take care to process them 369 in accordance with the required performance needs. They differ in 370 that relay nodes sit within a DetNet domain while edge nodes always 371 sit at DetNet domain boundaries. Both node types can enhance the 372 reliability of delivery by enabling the replication of packets so 373 that multiple copies, possibly over multiple paths are forwarded 374 through the DetNet domain. They also reduce the impact of 375 replication by eliminating surplus copies of DetNet packets. Relay 376 nodes may sit the boundary of an MPLS domain when the non-MPLS domain 377 is DetNet aware. Relay nodes are functionally similar to PW S-PEs 378 or, when at the edge of an MPLS network, T-PEs [RFC6073]. 380 Figure 4 illustrates how DetNet can provide services for IEEE 381 802.1TSN end systems, CE1 and CE2, over a DetNet enabled network. 382 The edge nodes, E1 and E2, insert and remove required DetNet data 383 plane encapsulation. The 'X' in the edge nodes and relay node, R1, 384 represent a potential DetNet flow packet replication and elimination 385 point. This conceptually parallels L2VPN services, and could 386 leverage existing related solutions as discussed below. 388 TSN |<------- End to End DetNet Service ------>| TSN 389 Service | Transit Transit | Service 390 TSN (AC) | |<-Tnl->| |<-Tnl->| | (AC) TSN 391 End | V V 1 V V 2 V V | End 392 System | +--------+ +--------+ +--------+ | System 393 +---+ | | E1 |=======| R1 |=======| E2 | | +---+ 394 | |--|----|._X_....|..DF1..|.._ _...|..DF3..|...._X_.|---|---| | 395 |CE1| | | \ | | X | | / | | |CE2| 396 | | | \_.|..DF2..|._/ \_..|..DF4..|._/ | | | 397 +---+ | |=======| |=======| | +---+ 398 ^ +--------+ +--------+ +--------+ ^ 399 | Edge Node Relay Node Edge Node | 400 | (T-PE) (S-PE) (T-PE) | 401 | | 402 |<-TSN-> <------- TSN Over DetNet MPLS ---------> <-TSN->| 403 | | 404 |<--- Emulated Time Sensitive Networking (TSN) Service --->| 406 DFx = DetNet Flow x 408 Figure 4: IEEE 802.1TSN over DetNet 410 [Editor's note: TSN Over DetNet MPLS arrows extended beyond the 'X' 411 (the PREF points).] 413 Figure 5 illustrates how an end to end MPLS-based DetNet service is 414 provided in a more detail. In this case, the end systems, CE1 and 415 CE2, are able to send and receive DetNet flows, and R1 and R2 are 416 relay nodes as they sit in the middle of a DetNet network. For 417 example, an end system sends data encapsulated in MPLS. The 'X' in 418 the end systems, and relay nodes represents potential DetNet flow 419 packet replication and elimination points. In this figure, the relay 420 nodes may change the underlying transport, for example tunneling MPLS 421 over IP Section 11, or simply interconnect network segments. 423 DetNet DetNet 424 MPLS Service Transit Transit Service MPLS 425 DetNet | |<-Tnl->| |<-Tnl->| | DetNet 426 End | V 1 V V 2 V | End 427 System | +--------+ +--------+ +--------+ | System 428 +---+ | | R1 |=======| R2 |=======| R3 | | +---+ 429 | X...DFa...|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|.DFa..|.X | 430 |CE1|========| \ | | X | | / |======|CE2| 431 | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | 432 +---+ | |=======| |=======| | +---+ 433 ^ +--------+ +--------+ +--------+ ^ 434 | Relay Node Relay Node Relay Node | 435 | (S-PE) (S-PE) (S-PE) | 436 | | 437 |<---------------------- DetNet MPLS --------------------->| 438 | | 439 |<--------------- End to End DetNet Service -------------->| 441 Figure 5: MPLS-Based Native DetNet 443 Figure 6 illustrates how an end to end MPLS-based DetNet service is 444 provided where the end systems are not able to send and receive 445 DetNet flows. In this example, the nodes labeled CE1 and CE2 could 446 be non-DetNet aware IP routers or hosts. Note that E1 and E2 are 447 edge nodes as they sit boundaries of the DetNet enabled domain. 449 IP IP 450 Non Service Transit Transit Service Non 451 DetNet |<-Tnl->| |<-Tnl->| DetNet 452 End | V 1 V V 2 V | End 453 System | +--------+ +--------+ +--------+ | System 454 +---+ | | E1 |=======| R2 |=======| E3 | | +---+ 455 | |--------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|------| | 456 |CE1| | | \ | | X | | / | | |CE2| 457 | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | 458 +---+ | |=======| |=======| | +---+ 459 +--------+ +--------+ +--------+ 460 ^ Edge Node Relay Node Edge Node^ 461 | (T-PE) (S-PE) (T-PE) | 462 | | 463 <--IP-->| <-------- IP Over DetNet MPLS ---------> |<--IP--> 464 | | 465 |<------ End to End DetNet Service ------->| 467 Figure 6: MPLS-Based DetNet (non-MPLS End System) 469 Figure 7 illustrates how end to end DetNet service is provided where 470 the end systems are able to send and receive IP DetNet flows, e.g., 471 per [I-D.ietf-detnet-dp-sol-ip], and the MPLS nodes optionally 472 provide service protection. In this case R1 and R3 are T-PEs and R2 473 is an S-PE and the DetNet service is end-to-end. 475 DetNet DetNet 476 IP Service Transit Transit Service IP 477 DetNet |<-Tnl->| |<-Tnl->| DetNet 478 End | V 1 V V 2 V | End 479 System | +--------+ +--------+ +--------+ | System 480 +---+ | | R1 |=======| R2 |=======| R3 | | +---+ 481 | |-------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|-------| | 482 |CE1| | | \ | | X | | / | | |CE2| 483 | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | 484 +---+ | |=======| |=======| | +---+ 485 ^ +--------+ +--------+ +--------+ ^ 486 | Relay Node Relay Node Relay Node | 487 | (T-PE) (S-PE) (T-PE) | 488 | | 489 |<-DN IP-> <-------- DetNet MPLS ---------------> <-DN IP->| 490 | | 491 |<-------------- End to End DetNet Service --------------->| 493 Figure 7: DetNet IP over DetNet (DN) MPLS 495 4.3. Packet flow example (service protection) 497 An example MPLS DetNet network fragment and packet flow is 498 illustrated in Figure 8. 500 1 1.1 1.1 1.2.1 1.2.1 1.2.2 501 CE1----EN1--------R1-------R2-------R3--------EN2-----CE2 502 \ 1.2.1 / / 503 \1.2 /-----+ / 504 +------R4------------------------+ 505 1.2.2 507 Figure 8: Example Packet flow in DetNet Enabled MPLS Network 509 In Figure 8 the numbers are used to identify the instance of a 510 packet. Packet 1 is the original packet, and packets 1.1, and 1.2 511 are two first generation copies of packet 1. Packet 1.2.1 is a 512 second generation copy of packet 1.2 etc. Note that these numbers 513 never appear in the packet, and are not to be confused with sequence 514 numbers, labels or any other identifier that appears in the packet. 515 They simply indicate the generation number of the original packet so 516 that its passage through the network fragment can be identified to 517 the reader. 519 Customer Equipment CE1 sends a packet into the DetNet enabled MPLS 520 network. This is packet (1). Edge Node EN1 encapsulates the packet 521 as a DetNet Packet and sends it to Relay node R1 (packet 1.1). EN1 522 makes a copy of the packet (1.2), encapsulates it and sends this copy 523 to Relay node R4. 525 Note that along the MPLS path from EN1 to R1 there may be zero or 526 more LSRs which, for clarity, are not shown. The same is true for 527 any other path between two DetNet entities shown in Figure 8. 529 Relay node R4 has been configured to send one copy of the packet to 530 Relay Node R2 (packet 1.2.1) and one copy to Edge Node EN2 (packet 531 1.2.2). 533 R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and, 534 having been configured to perform packet elimination on this DetNet 535 flow, forwards packet 1.2.1 to Relay Node R3. Packet copy 1.1 is of 536 no further use and so is discarded by R2. 538 Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives 539 packet copy 1.2.1 from R2 via relay Node R3. EN2 therefore strips 540 any DetNet encapsulation from packet copy 1.2.2 and forwards the 541 packet to CE2. When EN2 receives the later packet copy 1.2.1 this is 542 discarded. 544 The above is of course illustrative of many network scenarios that 545 can be configured. Between a pair of relay nodes there may be one or 546 more transport nodes that simply forward the DetNet traffic, but 547 these are omitted for clarity. 549 5. DetNet MPLS Data Considerations 551 This section provides informative considerations related to providing 552 DetNet service to flows which are identified based on their header 553 information. At a high level, the following are provided on a per 554 flow basis: 556 Congestion protection and latency control: 558 Usage of allocated resources (queuing, policing, shaping) to 559 ensure that the congestion-related loss and latency/jitter 560 requirements of a DetNet flow are met. 562 Explicit routes: 564 Use of a specific path for a flow. This limits miss-ordering and 565 latency. 567 Service protection: 569 Which in the case of this document primarily relates to 570 replication and elimination. Changing the explicit path after a 571 failure is detected in order to restore delivery of the required 572 DetNet service characteristics is also possible. Path changes, 573 even in the case of failure recovery, can lead to the out of order 574 delivery of data. 576 Load sharing: 578 Generally, distributing packets of the same DetNet flow over 579 multiple paths is not recommended. Such load sharing, e.g., via 580 ECMP or UCMP, impacts ordering and end-to-end jitter. 582 Troubleshooting: 584 For example, to support identification of misbehaving flows. 586 Recognize flow(s) for analytics: 588 For example, increase counters. 590 Correlate events with flows: 592 For example, unexpected loss. 594 The DetNet data plane also allows for the aggregation of DetNet 595 flows, e.g., via MPLS hierarchical LSPs, to improved scaling. When 596 DetNet flows are aggregated, transit nodes may have limited ability 597 to provide service on per-flow DetNet identifiers. Therefore, 598 identifying each individual DetNet flow on a transit node may not be 599 achieved in some network scenarios, but DetNet service can still be 600 assured in these scenarios through resource allocation and control. 602 5.1. End-system specific considerations 604 Data-flows requiring DetNet service are generated and terminated on 605 end-systems. Encapsulation depends on application and its 606 preferences. In a DetNet (or even a TSN) domain the DN (TSN) 607 functions use at most two flow parameters, namely Flow-ID and 608 Sequence Number. However, an application may exchange further flow 609 related parameters (e.g., time-stamp), which are not considered by DN 610 functions. 612 Two types of end-systems are distinguished: 614 o L2 (Ethernet) end-system: application directly over L2. 616 o L3 (IP) end-system: application over L3. 618 Note: An MPLS DetNet end system (as shown in Figure 5) can be treated 619 as a combination of an L3 (IP) end-system and an MPLS DetNet edge 620 node. 622 In case of Ethernet end-systems the application data is encapsulated 623 directly in L2. From the DN domain perspective no upper layer 624 protocols are visible. The Data-flow uses only Ethernet tag(s) and 625 further flow specific parameters (if needed) are hidden inside the 626 protocol data unit (PDU). 628 The IP end-system scenario is different. In this case, data-flows 629 are encapsulated directly in IP and, typically, other higher layer 630 protocols such as UDP and Real-time Transport Protocol (RTP). Many 631 valid combinations exist and it is up to applications to select 632 specific headers to be used. Details on support for DetNet IP data 633 flows can be found in [I-D.ietf-detnet-dp-sol-ip]. 635 As a general rule, DetNet domains MUST be capable of forwarding any 636 Data-flows and the DetNet domain MUST NOT mandate the end-system 637 encapsulation format. 639 Furthermore, no application-level-proxy function is envisioned inside 640 the DetNet domain, so end-systems peer with end-systems using the 641 same application encapsulation format (see figure below): 643 o L2 end-systems peer with L2 end-systems and 645 o L3 end-systems peer with L3 end-systems. 647 +-----+ 648 | X | +-----+ 649 +-----+ | X | 650 | Eth | ________ +-----+ 651 +-----+ _____ / \ | Eth | 652 \ / \__/ \___ +-----+ 653 \ / \ / 654 0======== tunnel-1 ========0_ 655 | \ 656 \ | 657 0========= tunnel-2 =========0 658 / \ __/ \ 659 +-----+ \__ DetNet domain / \ 660 | X | \ __ / +-----+ 661 +-----+ \_______/ \_____/ | X | 662 | IP | +-----+ 663 +-----+ | IP | 664 +-----+ 666 Figure 9: End-systems and the DetNet domain 668 5.2. DetNet domain specific considerations 670 From a connection type perspective, three scenarios are 671 distinguished: 673 1. Directly attached: end-system is directly connected to an edge 674 node. 676 2. Indirectly attached: end-system is behind a (L2-TSN / L3-DetNet) 677 sub-network. 679 3. DN integrated: end-system is part of the DetNet domain. 681 L3 end-systems may use any of these connection types, however L2 end- 682 systems may use only the first two (directly or indirectly attached). 683 DetNet domain MUST allow communication between any end-systems of the 684 same type (L2-L2, L3-L3), independent of their connection type and 685 DetNet capability. However directly attached and indirectly attached 686 end-systems have no knowledge about the DetNet domain and its 687 encapsulation format at all. See Figure 10 for L3 end-system 688 scenarios. 690 ____+----+ 691 +----+ _____ / | ES3| 692 | ES1|____ / \__/ +----+___ 693 +----+ \ / \ 694 + | 695 ____ \ _/ 696 +----+ __/ \ +__ DetNet domain / 697 | ES2|____/ L2/L3 |___/ \ __ __/ 698 +----+ \_______/ \_______/ \___/ 700 Figure 10: Connection types of L3 end-systems 702 5.2.1. DetNet Layer Two Service 704 The simplest DetNet service is to provide tunneling for layer two, 705 where the connected hosts are in the same broadcast (BC) domain. 706 Forwarding over the DetNet domain is based on L2 (MAC) addresses 707 (i.e. dst-MAC), or on received interface [RFC3985]. In both cases 708 the L2 headers MUST either be kept, or provision must be made for 709 their reconstruction at egress from the DetNet domain. 711 +------+ 712 | X | 713 +------+ +------+ 714 | X | | IP | 715 +------+ +------+ 716 End-system | L2 | | L2 | 717 +-----+======+-+======+--+------+ 718 DetNet tunnel | Shim | 719 +------+ 720 | MPLS | 721 +------+ 722 | L2 | 723 +------+ 725 Examples: 727 +------+ +------+ 728 | X | | X | 729 +------+ +------+ +------+ 730 | X | | IP | | IP | 731 +------+ +------+ +------+ 732 | L2 | | L2 | | L2 | 733 +-----+======+--+======+--+======+-----+ 734 | d-CW | | d-CW | | d-CW | 735 +------+ +------+ +------+ 736 | MPLS | | MPLS | | MPLS | 737 +------+ +------+ +------+ 738 | L2 | | L2 | | UDP | 739 +------+ +------+ +------+ 740 | IP | 741 +------+ 742 | L2 | 743 +------+ 745 Figure 11: Encapsulation format for DetNet Layer Two Service 747 As shown in Figure 11 both L2 and L3 end-systems can be served by 748 such a DetNet L2 encapsulation service. This encapsulation service 749 may be carried over MPLS natively Section 6.2, or over MPLS over IP 750 Section 11. 752 5.2.2. DetNet Routing Service (IP over MPLS) 754 IP traffic and IP DetNet flows, see [I-D.ietf-detnet-dp-sol-ip], can 755 be carried over a DetNet MPLS domain. In such cases, the IP headers 756 are modified per standard router behavior, e.g., TTL handling. 758 Figure 12 shows the encapsulation of an IP flow over MPLS as well as 759 when MPLS is carried over an IP PSN, see Section 11. 761 +------+ 762 | X | 763 +------+ 764 End-system | IP | 765 +-----+======+--+------+ 766 DetNet tunnel | Shim | 767 +------+ 768 | MPLS | 769 +------+ 770 | L2 | 771 +------+ 773 Examples: 775 +------+ +------+ 776 | X | | X | 777 +------+ +------+ 778 | IP | | IP | 779 +-----+======+--+======+-----+ 780 | d-CW | | d-CW | 781 +------+ +------+ 782 | MPLS | | MPLS | 783 +------+ +------+ 784 | L2 | | UDP | 785 +------+ +------+ 786 | IP | 787 +------+ 788 | L2 | 789 +------+ 791 Figure 12: Encapsulation format for DetNet Routing in MPLS PSN for L3 792 end-systems 794 5.3. DetNet Inter-Working Function (DN-IWF) 796 5.3.1. Networks with multiple technology segments 798 There are networking scenarios, where the DetNet domain contains 799 multiple technology segments (IP, MPLS, ..) and all those segments 800 are under the same administrative control (see Figure 13). 801 Furthermore, DetNet nodes may be interconnected via TSN segments. 803 An important aspect of DetNet network design is the placement of 804 DetNet functions across the domain. Designs based on segment-by- 805 segment optimization can provide only sub-optimal solutions. In 806 order to achieve global optimized Inter-Working Functions (DN-IWF) 807 can be placed at segment edge nodes, which stitch together DetNet 808 flows across connected segments. 810 DN-IWF may ensure that flow attributes are correlated across segment 811 edges. For example, there are two DetNet functions which require 812 Sequence Numbers: (1) PEF: removes duplications from flows and (2) 813 POF: ensures in-order-delivery of packet in a flow. Stitching flows 814 together and correlating attributes means for example that 815 replication of packets can happen in one segment and elimination of 816 duplicates in a different one. 818 ______ 819 ____ / \__ 820 ____ / \__/ \___ ______ 821 +----+ __/ +======+ +==+ \ +----+ 822 |src |__/ Seg1 ) | | \ Seg3 \____| dst| 823 +----+ \_______+ \ Segment-2 | \+_____/ +----+ 824 \======+_ _+===/ 825 \ __ __/ 826 \_______/ \___/ 828 +------------+ 829 +---------------E----+ | | 830 +----+ | | +----R---+ | +----+ 831 |src |-------R +---+ | E----------+ dst| 832 +----+ | | E--------+ +----+ 833 +-----------R | 834 +-----------------+ 836 Figure 13: Optimal replication and elimination placement across 837 technology segments example 839 5.3.2. DN-IWF related considerations 841 The goal of DN-IWF is to (1) match and (2) translate segment specific 842 flow attributes. The DN-IWF ensures that segment specific attributes 843 comprise per domain unique attributes for the whole DetNet domain. 844 This characteristic can ensure that DetNet functions can be based on 845 per domain attributes and not per segment attributes. 847 The two DetNet specific attributes have the following 848 characteristics: 850 o Flow-ID: it is same in all packets of a flow 852 o Sequence Number: it is different packet-by-packet 854 For the Flow-ID the DN-IWF can implement a static mapping. The 855 situation is more complicated for Sequence Number as it is different 856 packet-by-packet, so it may need more sophisticated translation 857 unless its format is exactly the same in the two technology segments. 858 In this later case the DN-IWF can simple copy the Sequence Number 859 field between the tunneling encapsulation of the two technology 860 segments. 862 In case of three technology segments (IP, MPLS and TSN) three DN-IWF 863 functions can be specified. In the rest of this section the focus is 864 on the (1) IP - MPLS network scenario. Note: the use-cases are out- 865 of-scope for (2) TSN - IP, (3) TSN - MPLS. 867 Simplest implementation of DN-IWF is provided if the flow attributes 868 have the same format. Such a common denominator of the tunnel 869 encapsulation format is the pseudowire encapsulation over both IP and 870 MPLS. 872 +--------+ 873 | | 874 | X X | 875 | ____ | 876 | / \ | 877 | | 878 |/\/\/\/\| 880 Oops! 881 404 Not Found 883 Figure 14: FIGURE Placeholder PW over X 885 [Editor's note: Where is the text describing how 802.1 TSN Streams 886 are mapping to DetNet services/flows. i.e., EVPN+] 888 6. MPLS-based DetNet data plane solution 890 6.1. DetNet over MPLS Encapsulation Components 892 To carry DetNet over MPLS the following is required: 894 1. A method of identifying the MPLS payload type. 896 2. A method of identifying the DetNet flow group to the processing 897 element. 899 3. A method of distinguishing DetNet OAM packets from DetNet data 900 packets. 902 4. A method of carrying the DetNet sequence number. 904 5. A suitable LSP to deliver the packet to the egress PE. 906 6. A method of carrying queuing and forwarding indication. 908 In this design an MPLS service label (the S-Label), similar to a 909 pseudowire (PW) label [RFC3985], is used to identify both the DetNet 910 flow identity and the payload MPLS payload type satisfying (1) and 911 (2) in the list above. OAM traffic discrimination happens through 912 the use of the Associated Channel method described in [RFC4385]. The 913 sequence number is carried in the DetNet Control word which carries 914 the Data/OAM discriminator. The LSP used to transport the DetNet 915 packet may be of any type (MPLS-LDP, MPLS-TE, MPLS-TP [RFC5921], or 916 MPLS-SR [I-D.ietf-spring-segment-routing-mpls]). The LSP (T-Label) 917 label and/or the S-Label may be used to indicate the queue processing 918 as well as the forwarding parameters. 920 To simplify implementation and to maximize interoperability two 921 sequence number sizes are supported: a 16 bit sequence number and a 922 28 bit sequence number. The 16 bit sequence number is needed to 923 support some types of legacy clients. The 28 bit sequence number is 924 used in situations where it is necessary ensure that in high speed 925 networks the sequence number space does not wrap whilst packets are 926 in flight. In addition it must be possible to send a packet with a 927 zero length sequence number, to support the case where sequence 928 numbers are not required by a particular DetNet flow. 930 Note that the concept of a zero length sequence number is not to be 931 confused with a sequence number of zero. For example, were the 932 sequence number size is 16 bits, the sequence will contain: 65535, 0, 933 1. In this case zero is an ordinary sequence number. Unlike 934 [RFC4448] a sequence number of zero does not indicate that no 935 sequence number is in use. Where sequence numbers are not in use, 936 and thus a zero length sequence number is in used, the sequence 937 number field in the packet is sent as zero. The DetNet packet 938 forwarder knows which of these cases applies through configuration 939 parameters associated with each specific DetNet flow. 941 Note that when the network consists only of DetNet enabled nodes with 942 no aggregation, Penultimate Hop Popping (PHP) means that the only 943 label in the label stack may be the S-label. 945 6.2. MPLS data plane encapsulation 947 Figure 15 illustrates a DetNet data plane MPLS encapsulation. The 948 MPLS-based encapsulation of the DetNet flows is a good fit for the 949 Layer-2 interconnect deployment cases (see Figure 4). Furthermore, 950 end to end DetNet service i.e., native DetNet deployment (see 951 Figure 5) is also possible if DetNet end systems are capable of 952 initiating and termination MPLS encapsulated packets. 954 The MPLS-based DetNet data plane encapsulation consists of: 956 o DetNet control word (d-CW) containing sequencing information for 957 packet replication and duplicate elimination purposes, and the OAM 958 indicator. There MUST be a separate sequence number space for 959 each DetNet flow. 961 o DetNet service Label (S-label) that identifies a DetNet flow to 962 the peer node that is to process it. The S-Label is allocated 963 from the platform label space [RFC3031]. 965 o Zero or more MPLS transport LSP label(s) (T-label) used to direct 966 the packet along the label switched path (LSP) to the next peer 967 node along the path. When Penultimate Hop Popping is in use there 968 may be no label T-label in the protocol stack on the final hop. 970 o The necessary data-link encapsulation is then applied prior to 971 transmission over the physical media. 973 DetNet MPLS-based encapsulation 975 +---------------------------------+ 976 | | 977 | DetNet Flow | 978 | Payload Packet | 979 | | 980 +---------------------------------+ <--\ 981 | DetNet Control Word | | 982 +---------------------------------+ +--> DetNet data plane 983 | S-Label | | MPLS encapsulation 984 +---------------------------------+ <--/ 985 | T-Label(s) | 986 +---------------------------------+ 987 | Data-Link | 988 +---------------------------------+ 989 | Physical | 990 +---------------------------------+ 992 Figure 15: Encapsulation of a DetNet flow in an MPLS(-TP) PSN 994 6.3. DetNet control word 996 A DetNet control word (d-CW) conforms to the Generic PW MPLS Control 997 Word (PWMCW) defined in [RFC4385] and is illustrated in Figure 16. 998 The upper nibble of the d-CW MUST be set to zero (0). Two sequence 999 number sizes are supported: 16 bits and 28 bits. The sequence number 1000 size in use for the d-CW associated with a DetNet flow (S-Label) is 1001 configured either by a control plane or manually for each DetNet 1002 flow. The sequence number is aligned to the right (least significant 1003 bits) and unused bits MUST be set to zero (0). Each DetNet flow MUST 1004 have its own sequence number counter. The sequence number is 1005 incremented by one for each new packet. 1007 As discussed in Section 6, zero is an ordinary sequence number with 1008 no special meaning. Also as discussed therein, where no sequence 1009 number is used by a particular DetNet flow, the sequence number field 1010 in the d-CW is set to zero. 1012 The d-CW MUST always be present in a packet. In a case where the 1013 sequence number is not used (e.g., for DetNet-t-flows) a zero length 1014 sequence number is used and the sequence number MUST be set to zero 1015 (0). 1017 0 1 2 3 1018 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1019 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1020 |0 0 0 0| Sequence Number | 1021 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1023 Figure 16: DetNet Control Word 1025 6.4. Flow Identification 1027 DetNet flow identification at a DetNet service layer is realized by 1028 an S-label. The S-label is allocated from the platform label space 1029 [RFC3031] which means that the DetNet flow is correctly identified 1030 and matched to the flow parameters, including the flow history, 1031 regardless of which input interface the packet arrives on. The 1032 S-label MUST be at the bottom label of the label stack for a DetNet- 1033 s- or DetNet-st-flow and MUST precede the d-CW. 1035 The S-label for a specific DetNet flow is unique to that DetNet flow 1036 on a specific node, but is not required to be identical with the 1037 S-label for that DetNet flow in any other node within the DetNet 1038 domain. Thus the S-label can only be used to identify the DetNet 1039 flow at the intended receiving node. 1041 6.5. Indication of the DetNet Payload Type 1043 The only nodes that needs to know the payload type of a flow are the 1044 DetNet ingress node and the DetNet egress nodes. The ingress node 1045 has to know how to process the packet it receives from the ingress AC 1046 or IP flow, and the egress edge node has to know how to prepare the 1047 packet for transmission to the next hop. 1049 On ingress a DetNet edge node has to classify the packets into those 1050 that are for transmission as Detnet packets and those that are for 1051 transmission as "normal" packets at one of more lower priorities. 1052 The packet type is indicated to the egress edge node through the 1053 value of the S-label. Thus, when the egress edge node looks up the 1054 S-label one of the parameters returned is the packet type which in 1055 turn tells the egress edge node how to prepare the packet for 1056 transmission to a next hop. 1058 The consequence of this approach is that if multiple packet 1059 encapsulations are processed on a node pair, each encapsulation will 1060 need its own S-Label. That is not generally a problems, since it is 1061 anticipated that only one encapsulation type will be present for each 1062 DetNet flow. Of course, if for some reason the multiple 1063 encapsulations are needed to support a single DetNet service, 1064 multiple S-labels will be required for that service. Note that in 1065 the unlikely case that Ipv4 and IPv6 will map to the same DetNet 1066 flow, different S-labels will be needed to differentiate between the 1067 versions of IP. 1069 6.6. OAM Indication 1071 OAM follows the procedures set out in [RFC5085] with the restriction 1072 that only Virtual Circuit Connectivity Verification (VCCV) type 1 is 1073 supported. 1075 As shown in Figure 3 of [RFC5085] when the first nibble of the d-CW 1076 is 0x0 the payload following the d-CW is normal user data. However, 1077 when the first nibble of the d-CW is 0X1, the payload that follows 1078 the d-DW is an OAM payload with the OAM type indicated by the value 1079 in the d-CW Channel Type field. 1081 The reader is referred to [RFC5085] for a more detailed description 1082 of the Associated Channel mechanism, and to the DetNet work on OAM 1083 for more information DetNet OAM. 1085 6.7. Flow Aggregation 1087 1. Aggregate at the LSP (Transport) 1089 2. Aggregating DetNet flows as a new DetNet flow 1091 3. Simple Aggregation at the DetNet layer 1093 A further method of using SR to perform aggregation is for further 1094 study. 1096 The resource control and management aspects of aggregation (including 1097 the queuing/shaping/ policing implications) will be covered in other 1098 documents. 1100 The ability to aggregate individual flows, and their associated 1101 resource control, into a larger aggregate is an important technique 1102 for improving scaling of control in the data, management and control 1103 planes. The DetNet data plane allows for the aggregation of DetNet 1104 flows, to improved scaling. There are three methods of introducing 1105 flow aggregation: 1107 The following review comments were received when this section was 1108 committed to github. 1110 General comment: We should points to the major issue of aggregation, 1111 namely the Seq.Num related problem. The aggregated flows have their 1112 own Seq.Num and those are independent. We should consider to group 1113 the aggregation techniques as per their impact on what DetNet 1114 functions they allow on a DetNet flow. (E.g., aggregation without 1115 new Aggregate.Seq.Num would prohibit usage of FR, EF and in-order- 1116 delivery function on the aggregate flow). 1118 SR based aggregation can be treated as a form of H-LSP aggregation. 1119 Should we differentiate them? What are the differences? 1121 What are the issues when aggregating of different payload types? 1122 Should we add an editor note on this? 1124 Simple-aggregation-at-the-detnet-layer: is this not the same as 1125 H-LSP? The A-label can be treated just as an additional T-label. 1127 End of review comment. 1129 6.7.1. Aggregation at the LSP 1131 DetNet flows transported via MPLS can leverage MPLS-TE?s existing 1132 support for hierarchical LSPs (H-LSPs), see [RFC4206]. H-LSPs are 1133 typically used to aggregate control and resources, they may also be 1134 used to provide OAM or protection for the aggregated LSPs. Arbitrary 1135 levels of aggregation naturally falls out of the definition for 1136 hierarchy and the MPLS label stack [RFC3032]. DetNet nodes which 1137 support aggregation (LSP hierarchy) map one or more LSPs (labels) 1138 into and from an H-LSP. Both carried LSPs and H-LSPs may or may not 1139 use the TC field, i.e., L-LSPs or E-LSPs. Such nodes will need to 1140 ensure that traffic from aggregated LSPs are placed (shaped/policed/ 1141 enqueued) onto the H-LSPs in a fashion that ensures the required 1142 DetNet service is preserved. 1144 Additional details of the traffic control capabilities needed at a 1145 DetNet-aware node may be covered in the new service descriptions 1146 mentioned above or in separate future documents. Management and 1147 control plane mechanisms will also need to ensure that the service 1148 required on the aggregate flow (H-LSP or DSCP) are provided, which 1149 may include the discarding or remarking mentioned in the previous 1150 sections. 1152 6.7.2. Aggregating DetNet flows as a new DetNet flow 1154 An aggregate can be built by layering DetNet flows as shown below: 1156 +---------------------------------+ 1157 | | 1158 | DetNet Flow | 1159 | Payload Packet | 1160 | | 1161 +---------------------------------+ <--\ 1162 | DetNet Control Word | | 1163 +=================================+ | 1164 | S-Label | | 1165 +---------------------------------+ +----DetNet data plane 1166 | DetNet Control Word | | MPLS encapsulation 1167 +=================================+ | 1168 | A-Label | | 1169 +---------------------------------+ <--/ 1170 | T-Label(s) | 1171 +---------------------------------+ 1172 | Data-Link | 1173 +---------------------------------+ 1174 | Physical | 1175 +---------------------------------+ 1177 Both the Aggregation (A) label and the S-label have their MPLS S bit 1178 set indicating bottom of stack, and the d-CW allows the PREOF to 1179 work. 1181 It is a property of the A-label that what follows is d-CW followed by 1182 an S-label. A relay node processing the A-label would not know the 1183 underlying payload type. This would only be known to a node that was 1184 a peer of the node imposing the S-label. However there is no real 1185 need for it to know the payload type during aggregation processing. 1187 6.7.3. Simple Aggregation at the DetNet layer 1189 Another approach would be not to include a d-CW for the aggregated 1190 flow. This would be functionally similar to aggregation at the 1191 transport layer using H-LSPs, but would confine knowledge of the 1192 aggregation to the DetNet layer. Such an approach shares the 1193 disadvantage that PREOF operations would not be possible. OAM 1194 operation in this mode is for further study. 1196 +---------------------------------+ 1197 | | 1198 | DetNet Flow | 1199 | Payload Packet | 1200 | | 1201 +---------------------------------+ <--\ 1202 | DetNet Control Word | | 1203 +=================================+ | 1204 | S-Label | +----DetNet data plane 1205 +---------------------------------+ | MPLS encapsulation 1206 | A-Label | | 1207 +---------------------------------+ <--/ 1208 | T-Label(s) | 1209 +---------------------------------+ 1210 | Data-Link | 1211 +---------------------------------+ 1212 | Physical | 1213 +---------------------------------+ 1215 6.8. Service Layer Considerations 1217 The edge and relay node internal procedures related to PREOF are 1218 implementation specific. The order of a packet elimination or 1219 replication is out of scope in this specification. However, care 1220 should be taken that the replication function does not actually 1221 loopback packets as "replicas". Looped back packets include 1222 artificial delay when the node that originally initiated the packet 1223 receives it again. Also, looped back packets may make the network 1224 condition to look healthier than it actually is (in some cases link 1225 failures are not reflected properly because looped back packets make 1226 the situation appear better than it actually is). 1228 It is important that the DetNet layer is configured such that a 1229 DetNet node never receives its own replicated packets. If it were to 1230 receive such packets the replication function would make the loop 1231 more destructive of bandwidth than a conventional unicast loop. 1232 Ultimately the TTL in the S-Label will cause the packet to die during 1233 a transient, but given the sensitivity of applications to packet 1234 latency the impact on the DetNet application would be severe. 1236 6.8.1. Edge node processing 1238 An edge node is resposable for matching ingress packets to the 1239 service they require and encapsulating them accordingly.An edge node 1240 may participate in the packet replication and duplication 1241 elimination. 1243 The DetNet-aware forwarder selects the egress DetNet member flow 1244 segment based on the flow identification. The mapping of ingress 1245 DetNet member flow segment to egress DetNet member flow segment may 1246 be statically or dynamically configured. Additionally the DetNet- 1247 aware forwarder does duplicate frame elimination based on the flow 1248 identification and the sequence number combination. The packet 1249 replication is also done within the DetNet-aware forwarder. During 1250 elimination and the replication process the sequence number of the 1251 DetNet member flow MUST be preserved and copied to the egress DetNet 1252 member flow. 1254 The internal design of a relay node is out of scope of this document. 1255 However the reader's attention is drawn to the need to make any PREOF 1256 state available to the packet processor(s) dealing with packets to 1257 which the PREOF functions must be applied, and to maintain that state 1258 is such as way that it is available to the packet processor operation 1259 on the next packet in the DetNet flow (which may be a duplicate, a 1260 late packet, or the next packet in sequence. 1262 [Editor's note: I think the rest of this section belongs in a new 1263 "802.1 TSN (island Interconnect) over MPLS DetNet" section.] 1265 This may be done in the DetNet layer, or where the native service 1266 processing (NSP) [RFC3985] is IEEE 802.1CB [IEEE8021CB] capable, the 1267 packet replication and duplicate elimination MAY entirely be done in 1268 the NSP, bypassing the DetNet flow encapsulation and logic entirely. 1269 This enables operating over unmodified implementations and 1270 deployments. The NSP approach works only between edge nodes and 1271 cannot make use of relay nodes. 1273 The NSP approach is useful end to end tunnel and for for "island 1274 interconnect" scenarios. However, when there is a need to do PREOF 1275 in a middle of the network, such plain edge to edge operation is not 1276 sufficient. 1278 The extended forwarder MAY copy the sequencing information from the 1279 native DetNet packet into the DetNet sequence number field and vice 1280 versa. If there is no existing sequencing information available in 1281 the native packet or the forwarder chose not to copy it from the 1282 native packet, then the extended forwarder MUST maintain a sequence 1283 number counter for each DetNet flow (indexed by the DetNet flow 1284 identification). 1286 6.8.2. Relay node processing 1288 A DetNet Relay node operates in the DetNet transport layer . This 1289 processing is done within an extended forwarder function. Whether an 1290 ingress DetNet member flow receives DetNet specific processing 1291 depends on how the forwarding is programmed. Some relay nodes may be 1292 DetNet service aware, while others may be unmodified LSRs that only 1293 understand how to swicth MPLS-TE LSPs. 1295 It is also possible to treat the relay node as a transit node, see 1296 Section 6.9.3. Again, this is entirely up to how the forwarding has 1297 been programmed. 1299 6.9. Other DetNet data plane considerations 1301 6.9.1. Class of Service 1303 [Editor's note: this section needs to updated to discuss how DetNet 1304 service is mapped to E- and L-LSPs. Perhaps this gets merged with 1305 the aggregation section or dropped?] 1307 Class and quality of service, i.e., CoS and QoS, are terms that are 1308 often used interchangeably and confused with each other. In the 1309 context of DetNet, CoS is used to refer to mechanisms that provide 1310 traffic forwarding treatment based on aggregate group basis and QoS 1311 is used to refer to mechanisms that provide traffic forwarding 1312 treatment based on a specific DetNet flow basis. Examples of 1313 existing network level CoS mechanisms include DiffServ which is 1314 enabled by IP header differentiated services code point (DSCP) field 1315 [RFC2474] and MPLS label traffic class field [RFC5462], and at Layer- 1316 2, by IEEE 802.1p priority code point (PCP). 1318 CoS for DetNet flows carried in PWs and MPLS is provided using the 1319 existing MPLS Differentiated Services (DiffServ) architecture 1320 [RFC3270]. Both E-LSP and L-LSP MPLS DiffServ modes MAY be used to 1321 support DetNet flows. The Traffic Class field (formerly the EXP 1322 field) of an MPLS label follows the definition of [RFC5462] and 1323 [RFC3270]. The Uniform, Pipe, and Short Pipe DiffServ tunneling and 1324 TTL processing models are described in [RFC3270] and [RFC3443] and 1325 MAY be used for MPLS LSPs supporting DetNet flows. MPLS ECN MAY also 1326 be used as defined in ECN [RFC5129] and updated by [RFC5462]. 1328 CoS for DetNet flows carried in IPv6 is provided using the standard 1329 differentiated services code point (DSCP) field [RFC2474] and related 1330 mechanisms. The 2-bit explicit congestion notification (ECN) 1331 [RFC3168] field MAY also be used. 1333 One additional consideration for DetNet nodes which support CoS 1334 services is that they MUST ensure that the CoS service classes do not 1335 impact the congestion protection and latency control mechanisms used 1336 to provide DetNet QoS. This requirement is similar to requirement 1337 for MPLS LSRs to that CoS LSPs do not impact the resources allocated 1338 to TE LSPs via [RFC3473]. 1340 6.9.2. Quality of Service 1342 Quality of Service (QoS) mechanisms for flow specific traffic 1343 treatment typically includes a guarantee/agreement for the service, 1344 and allocation of resources to support the service. Example QoS 1345 mechanisms include discrete resource allocation, admission control, 1346 flow identification and isolation, and sometimes path control, 1347 traffic protection, shaping, policing and remarking. Example 1348 protocols that support QoS control include Resource ReSerVation 1349 Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473]. 1350 The existing MPLS mechanisms defined to support CoS [RFC3270] can 1351 also be used to reserve resources for specific traffic classes. 1353 In addition to explicit routes, and packet replication and 1354 elimination, described in Section 6 above, DetNet provides zero 1355 congestion loss and bounded latency and jitter. As described in 1356 [I-D.ietf-detnet-architecture], there are different mechanisms that 1357 maybe used separately or in combination to deliver a zero congestion 1358 loss service. These mechanisms are provided by the either the MPLS 1359 or IP layers, and may be combined with the mechanisms defined by the 1360 underlying network layer such as 802.1TSN. 1362 A baseline set of QoS capabilities for DetNet flows carried in PWs 1363 and MPLS can provided by MPLS with Traffic Engineering (MPLS-TE) 1364 [RFC3209] and [RFC3473]. TE LSPs can also support explicit routes 1365 (path pinning). Current service definitions for packet TE LSPs can 1366 be found in "Specification of the Controlled Load Quality of 1367 Service", [RFC2211], "Specification of Guaranteed Quality of 1368 Service", [RFC2212], and "Ethernet Traffic Parameters", [RFC6003]. 1369 Additional service definitions are expected in future documents to 1370 support the full range of DetNet services. In all cases, the 1371 existing label-based marking mechanisms defined for TE-LSPs and even 1372 E-LSPs are use to support the identification of flows requiring 1373 DetNet QoS. 1375 Packets that are marked with a DetNet Class of Service value, but 1376 that have not been the subject of a completed reservation, can 1377 disrupt the QoS offered to properly reserved DetNet flows by using 1378 resources allocated to the reserved flows. Therefore, the network 1379 nodes of a DetNet network: 1381 o MUST defend the DetNet QoS by discarding or remarking (to a non- 1382 DetNet CoS) packets received that are not the subject of a 1383 completed reservation. 1385 o MUST NOT use a DetNet reserved resource, e.g. a queue or shaper 1386 reserved for DetNet flows, for any packet that does not carry a 1387 DetNet Class of Service marker. 1389 6.9.3. Cross-DetNet flow resource aggregation 1391 [Editor's NOTE: keep and extend this section.] 1393 The ability to aggregate individual flows, and their associated 1394 resource control, into a larger aggregate is an important technique 1395 for improving scaling of control in the data, management and control 1396 planes. This document identifies the traffic identification related 1397 aspects of aggregation of DetNet flows. The resource control and 1398 management aspects of aggregation (including the queuing/shaping/ 1399 policing implications) will be covered in other documents. The data 1400 plane implications of aggregation are independent for PW/MPLS and IP 1401 encapsulated DetNet flows. 1403 DetNet flows transported via MPLS can leverage MPLS-TE's existing 1404 support for hierarchical LSPs (H-LSPs), see [RFC4206]. H-LSPs are 1405 typically used to aggregate control and resources, they may also be 1406 used to provide OAM or protection for the aggregated LSPs. Arbitrary 1407 levels of aggregation naturally falls out of the definition for 1408 hierarchy and the MPLS label stack [RFC3032]. DetNet nodes which 1409 support aggregation (LSP hierarchy) map one or more LSPs (labels) 1410 into and from an H-LSP. Both carried LSPs and H-LSPs may or may not 1411 use the TC field, i.e., L-LSPs or E-LSPs. Such nodes will need to 1412 ensure that traffic from aggregated LSPs are placed (shaped/policed/ 1413 enqueued) onto the H-LSPs in a fashion that ensures the required 1414 DetNet service is preserved. 1416 DetNet flows transported via IP have more limited aggregation 1417 options, due to the available traffic flow identification fields of 1418 the IP solution. One available approach is to manage the resources 1419 associated with a DSCP identified traffic class and to map (remark) 1420 individually controlled DetNet flows onto that traffic class. This 1421 approach also requires that nodes support aggregation ensure that 1422 traffic from aggregated LSPs are placed (shaped/policed/enqueued) in 1423 a fashion that ensures the required DetNet service is preserved. 1425 In both the MPLS and IP cases, additional details of the traffic 1426 control capabilities needed at a DetNet-aware node may be covered in 1427 the new service descriptions mentioned above or in separate future 1428 documents. Management and control plane mechanisms will also need to 1429 ensure that the service required on the aggregate flow (H-LSP or 1430 DSCP) are provided, which may include the discarding or remarking 1431 mentioned in the previous sections. 1433 6.9.4. Layer 2 addressing and QoS Considerations 1435 [Editor's NOTE: review and simplify this section.] 1437 The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1 1438 Working Group have defined (and are defining) a number of amendments 1439 to IEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and 1440 bounded latency in bridged networks. IEEE 802.1CB [IEEE8021CB] 1441 defines packet replication and elimination functions that should 1442 prove both compatible with and useful to, DetNet networks. 1444 As is the case for DetNet, a Layer 2 network node such as a bridge 1445 may need to identify the specific DetNet flow to which a packet 1446 belongs in order to provide the TSN/DetNet QoS for that packet. It 1447 also will likely need a CoS marking, such as the priority field of an 1448 IEEE Std 802.1Q VLAN tag, to give the packet proper service. 1450 Although the flow identification methods described in IEEE 802.1CB 1451 [IEEE8021CB] are flexible, and in fact, include IP 5-tuple 1452 identification methods, the baseline TSN standards assume that every 1453 Ethernet frame belonging to a TSN stream (i.e. DetNet flow) carries 1454 a multicast destination MAC address that is unique to that flow 1455 within the bridged network over which it is carried. Furthermore, 1456 IEEE 802.1CB [IEEE8021CB] describes three methods by which a packet 1457 sequence number can be encoded in an Ethernet frame. 1459 Ensuring that the proper Ethernet VLAN tag priority and destination 1460 MAC address are used on a DetNet/TSN packet may require further 1461 clarification of the customary L2/L3 transformations carried out by 1462 routers and edge label switches. Edge nodes may also have to move 1463 sequence number fields among Layer 2, PW, and IPv6 encapsulations. 1465 6.9.5. Time Synchronization 1467 [Editor's Note: A detailed discussion of time synchronization is 1468 outside the scope of this document, and the production of a 1469 specialist text discussing this topic is encouraged. This section 1470 will be updated/removed if such a document is available before 1471 publication of this text.] 1473 Time synchronization is important both from the perspective of 1474 operating the DetNet network itself and from the perspective of 1475 transferring time across the network between client applications. 1476 Some clients may be able to use the DetNet as their provider of time 1477 and frequency, others may require the DetNet to transfer time between 1478 a client clock source and a client clock user. 1480 For example, [RFC8169] describes a method of recording the packet 1481 queuing time in an MPLS LSR on a packet by per packet basis and 1482 forwarding this information to the egress edge system. This allows 1483 compensation for any variable packet queuing delay to be applied at 1484 the packet receiver. Other mechanisms for IP/MPLS networks are 1485 defined based on IEEE Standard 1588 [IEEE1588], such as ITU-T 1486 [G.8275.1] and [G.8275.2]. 1488 A more detailed discussion of time synchronization is outside the 1489 scope of this document. 1491 7. Management and control considerations 1493 [Editor's note: This section needs to be different for MPLS and IP 1494 solutions. Most solutions are technology dependant. Currently most 1495 text in this section is just a draft and may have bits that are 1496 already moved to other places/documents.] 1498 While management plane and control planes are traditionally 1499 considered separately, from the Data Plane perspective there is no 1500 practical difference based on the origin of flow provisioning 1501 information. This document therefore does not distinguish between 1502 information provided by a control plane protocol, e.g., RSVP-TE 1503 [RFC3209] and [RFC3473], or by a network management mechanisms, e.g., 1504 RestConf [RFC8040] and YANG [RFC7950]. 1506 [Editor's note: This section is a work in progress. discuss here 1507 what kind of enhancements are needed for DetNet and specifically for 1508 PREOF and DetNet zero congest loss and latency control. Need to 1509 cover both traffic control (queuing) and connection control (control 1510 plane).] 1512 7.1. MPLS-based data plane 1514 7.1.1. S-Label assignment and distribution 1516 [Editor's note: Outdated and needs more work.] 1518 The DetNet S-Label distribution follows the same mechanisms specified 1519 for XYZ . The details of the control plane protocol solution required 1520 for the label distribution and the management of the label number 1521 space are out of scope of this document. 1523 7.1.2. Explicit routes 1525 It is necessary to consider explicit routes both at the DetNet layer 1526 and in the MPLS layer. In the DetNet layer the explicit route 1527 consists of the set of Relay Nodes that the DetNet flow must 1528 traverse. In the MPLS layer the explicit route consists of the set 1529 of LSRs, links, and possibly link bundle members and queues that the 1530 DetNet packets of a flow must traverse between nodes in the DetNet 1531 layer (i.e. between a specific Edge Node and the next hop Relay Node, 1532 between specific Relay Nodes, and between a specific Relay node and 1533 the egress Edge Node. This detailed steering is needed to ensure 1534 that packets are routed through the resources that have been reserved 1535 for them, and hence provide the DetNet application with the required 1536 performance. 1538 Whether configuring, calculating and instantiating this is a multi- 1539 stage process, or a single stage process is out of scope of this 1540 document. 1542 The one method of explicitly setting up the explicit path at the 1543 DetNet layer is through the use of the management controller. 1545 [Editor's note: a method of setting up a graph through the DetNet 1546 Nodes using the IGP has been proposed. A reference is needed to 1547 e.g., RFC 7813 IS-IS Path Control and Reservation.] 1549 There are a number of approaches that can be taken to provide 1550 explicit routes/paths in the MPLS layer: 1552 o The path can be explicitly set up by the management controller 1553 calculating the path and explicitly configuring each node along 1554 that path. 1556 o The LSP can be set up using RSVP-TE. Such an approach confines 1557 the packet to the explicit path. 1559 o The path can be implemented using segment routing. 1561 Where the DetNet traffic is carried over IP Section 11 explicit paths 1562 may need to be provided in the IP layer. This is for further study. 1564 7.2. Packet replication and elimination 1566 [Editor's note: Outdated and at the functional level technology 1567 independent.. but needs more work.] 1569 The control plane protocol solution required for managing the PREOF 1570 processing is outside the scope of this document. 1572 7.3. Congestion protection and latency control 1574 [Editor's note: TBD] 1576 7.4. Bidirectional traffic 1578 [Editor's NOTE: this section needs to be updated to have its scope 1579 limited to management and control.] 1581 Some DetNet applications generate bidirectional traffic. Using MPLS 1582 definitions [RFC5654] there are associated bidirectional flows, and 1583 co-routed bidirectional flows. MPLS defines a point-to-point 1584 associated bidirectional LSP as consisting of two unidirectional 1585 point-to-point LSPs, one from A to B and the other from B to A, which 1586 are regarded as providing a single logical bidirectional transport 1587 path. This would be analogous of standard IP routing, or PWs running 1588 over two reciprocal unidirection LSPs. MPLS defines a point-to-point 1589 co-routed bidirectional LSP as an associated bidirectional LSP which 1590 satisfies the additional constraint that its two unidirectional 1591 component LSPs follow the same path (in terms of both nodes and 1592 links) in both directions. An important property of co-routed 1593 bidirectional LSPs is that their unidirectional component LSPs share 1594 fate. In both types of bidirectional LSPs, resource allocations may 1595 differ in each direction. The concepts of associated bidirectional 1596 flows and co-routed bidirectional flows can be applied to DetNet 1597 flows as well whether IPv6 or MPLS is used. 1599 While the IPv6 and MPLS data planes must support bidirectional DetNet 1600 flows, there are no special bidirectional features with respect to 1601 the data plane other than need for the two directions take the same 1602 paths. Fate sharing and associated vs co-routed bidirectional flows 1603 can be managed at the control level. Note, that there is no stated 1604 requirement for bidirectional DetNet flows to be supported using the 1605 same IPv6 Flow Labels or MPLS Labels in each direction. Control 1606 mechanisms will need to support such bidirectional flows for both 1607 IPv6 and MPLS, but such mechanisms are out of scope of this document. 1608 An example control plane solution for MPLS can be found in [RFC7551]. 1610 7.5. Flow aggregation control 1612 [TBD] 1614 8. DetNet IP Operation over DetNet MPLS Service 1616 [Editor's note: this is a place holder section. A standalone section 1617 on operation of IP flows over DetNet MPLS data plane. Includes 1618 RFC2119 Language.] 1620 9. IEEE 802.1 TSN Interconnection over DetNet MPLS Service 1622 [Editor's note: this is a place holder section. A standalone section 1623 on TSN "island" interconnect over DetNet". Includes RFC2119 1624 Language.] 1626 10. DetNet MPLS Transport Layer Operation over IEEE 802.1 TSN Sub- 1627 Networks 1629 [Editor's note: this is a place holder section. A standalone section 1630 on MPLS over IEEE 802.1 TSN. Includes RFC2119 Language.] 1632 This section covers how MPLS DetNet flows operate over an IEEE 802.1 1633 TSN sub-network. Figure 17 illustrates such a scenario, where two 1634 MPLS (DetNet) nodes are interconnected by a TSN sub-network. Node-1 1635 is single homed and Node-2 is dual-homed. MPLS nodes can be (1) MPLS 1636 DetNet End System, (2) MPLS DetNet Edge or Relay node or (3) MPLS 1637 Transit node. 1639 Note: in case of MPLS Transit node there is no DetNet Service sub- 1640 layer. 1642 MPLS (DetNet) MPLS (DetNet) 1643 Node-1 Node-2 1645 +---------+ +---------+ 1646 <--| Service*|-- DetNet flow ---| Service*|--> 1647 +---------+ +---------+ 1648 |Transport| |Transport| 1649 +-------.-+ <-TSN Str-> +-.-----.-+ 1650 \ ,-------. / / 1651 +----[ TSN-Sub ]---+ / 1652 [ Network ]--------+ 1653 `-------' 1654 <---------------- DetNet MPLS ---------------> 1656 Note: * no service sub-layer for transit nodes 1658 Figure 17: DetNet Enabled MPLS Network over a TSN sub-network 1660 The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1 1661 Working Group have defined (and are defining) a number of amendments 1662 to IEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and 1663 bounded latency in bridged networks. Furthermore IEEE 802.1CB 1664 [IEEE8021CB] defines frame replication and elimination functions for 1665 reliability that should prove both compatible with and useful to, 1666 DetNet networks. All these functions have to identify flows those 1667 require TSN treatment. 1669 As is the case for DetNet, a Layer 2 network node such as a bridge 1670 may need to identify the specific DetNet flow to which a packet 1671 belongs in order to provide the TSN/DetNet QoS for that packet. It 1672 also may need a CoS marking, such as the priority field of an IEEE 1673 Std 802.1Q VLAN tag, to give the packet proper service. 1675 The challange for MPLS DeNet flows is that the protocol interworking 1676 function defined in IEEE 802.1CB [IEEE8021CB] works only for IP 1677 flows. The aim of the protocol interworking function is to convert 1678 an ingress flow to use a specific multicast destination MAC address 1679 and VLAN, for example to direct the packets through a specific path 1680 inside the bridged network. A similar interworking pair at the other 1681 end of the TSN sub-network would restore the packet to its original 1682 destination MAC address and VLAN. 1684 As protocol interworking function defined in [IEEE8021CB] does not 1685 work for MPLS labeled flows, the MPLS DetNet nodes MUST ensure proper 1686 TSN sub-network specific Ethernet encapsulation of the MPLS DetNet 1687 packets. For a given TSN Stream (i.e., DetNet flow) an MPLS (DetNet) 1688 node MUST behave as a TSN-aware Talker or a Listener inside the TSN 1689 sub-network. 1691 10.1. Mapping of TSN Stream ID and Sequence Number 1693 TSN capable MPLS (DetNet) nodes are TSN-aware Talker/Listener as 1694 shown in Figure 18. MPLS (DetNet) node MUST provide the TSN sub- 1695 network specific Ethernet encapsulation over the link(s) towards the 1696 sub-network. An TSN-aware MPLS (DetNet) node MUST support the 1697 following TSN components: 1699 1. For recognizing flows: 1701 * Stream Identification (MPLS-flow-aware) 1703 2. For FRER used inside the TSN domain, additonaly: 1705 * Sequencing function (MPLS-flow-aware) 1707 * Sequence encode/decode function 1709 3. For FRER when the node is a TSN replication or elimination point, 1710 additionally: 1712 * Stream splitting function 1714 * Individual recovery function 1716 [Editor's note: Should we added here requirements regarding IEEE 1717 802.1Q C-VLAN component?] 1719 The Stream Identification and The Sequencing functions are slightly 1720 modified for frames passed down the protocol stack from the upper 1721 layers. 1723 Stream Identification MUST pair MPLS flows and TSN Streams and encode 1724 that in data plane formats as well. The packet's stream_handle 1725 subparameter (see IEEE 802.1CB [IEEE8021CB]) inside the Talker/ 1726 Listener is defined based on the Flow-ID used in the upper MPLS 1727 DetNet layer. Stream Identification function MUST encode Ethernet 1728 header fields namely (1) the destination MAC-address, (2) the VLAN-ID 1729 and (3) priority parameters with TSN sub-network specific values. 1730 Encoding is provided for the frame passed down the stack from the 1731 upper layers. 1733 The sequence generation function resides in the Sequencing function. 1734 It generates a sequence_number subparameter for each packet of a 1735 Stream passed down to the lower layers. Sequencing function MUST 1736 copy sequence information from the MPLS d-CW of the packet to the 1737 sequence_number subparameter for the frame passed down the stack from 1738 the upper layers. 1740 MPLS (DetNet) 1741 Node-1 1742 <---------> 1744 +---------+ 1745 <--| Service |-- DetNet flow ------------------ 1746 +---------+ 1747 |Transport| 1748 +---------+ +---------------+ 1749 | L2 with |<---| L2 Relay with |---- TSN ---- 1750 | TSN | | TSN function | Stream 1751 +----.----+ +--.---------.--+ 1752 \__________/ \______ 1754 TSN-aware 1755 Talker / TSN-Bridge 1756 Listener Relay 1758 <--------- TSN sub-network ------------ 1760 Figure 18: MPLS (DetNet) node with TSN functions 1762 The Sequence encode/decode function MUST support the Redundancy tag 1763 (R-TAG) format as per Clause 7.8 of IEEE 802.1CB [IEEE8021CB]. 1765 10.2. TSN Usage of FRER 1767 TSN Streams supporting DetNet flows may use Frame Replication and 1768 Elimination for Redundancy (FRER) [802.1CB] based on the loss service 1769 requirements of the TSN Stream, which is derived from the DetNet 1770 service requirements of the DetNet mapped flow. The specific 1771 operation of FRER is not modified by the use of DetNet and follows 1772 IEEE 802.1CB [IEEE8021CB]. 1774 FRER function and the provided service recovery is available only 1775 within the TSN sub-network however as the Stream-ID and the TSN 1776 sequence number are paired with the MPLS flow parameters they can be 1777 combined with PREOF functions. 1779 10.3. Management and Control Implications 1781 [Editor's note: This section is TBD Covers Creation, mapping, removal 1782 of TSN Stream IDs, related parameters and,when needed, configuration 1783 of FRER. Supported by management/control plane.] 1785 11. DetNet MPLS Transport Layer Operation over IP DetNet PSNs 1787 This section specifies the DetNet encapsulation over an IP transport 1788 network. The approach is modeled on the operation of MPLS and 1789 PseudoWires (PW) over an IP Packet Switched Network (PSN) 1790 [RFC3985][RFC4385][RFC7510]. It is also based on the MPLS data plane 1791 encapsulation described in Section 6.2. 1793 To carry DetNet with full functionality at the DetNet layer over an 1794 IP transport network, the following components are required (these 1795 are a subset of the requirements for MPLS encapsulation listed in 1796 Section 6.1): 1798 1. A method of identifying the DetNet flow group to the processing 1799 element. 1801 2. A method of carrying the DetNet sequence number. 1803 3. A method of distinguishing DetNet OAM packets from DetNet data 1804 packets. 1806 4. A method of carrying queuing and forwarding indication. 1808 These requirements are satisfied by the DetNet over MPLS 1809 Encapsulation described in Section 6.2. 1811 To simplify operations and implementations, rather than inventing a 1812 new encapsulation, the IP encapsulation takes advantage of the MPLS 1813 encapsulation. By using the specification of MPLS over UDP and IP in 1814 [RFC7510], the T-Label(s) shown in Figure 15 in Section 6.2 can be 1815 replaced by UDP and IP, resulting in the following encapsulation: 1817 +---------------------------------+ 1818 | | 1819 | DetNet Flow | 1820 | Payload Packet | 1821 | | 1822 +---------------------------------+ <--\ 1823 | DetNet Control Word | | 1824 +---------------------------------+ +--> DetNet data plane 1825 | S-Label | | MPLS encapsulation 1826 +---------------------------------+ <--/ 1827 | UDP Header | 1828 +---------------------------------+ 1829 | IP Header | 1830 +---------------------------------+ 1831 | Data-Link | 1832 +---------------------------------+ 1833 | Physical | 1834 +---------------------------------+ 1836 Figure 19: IP Encapsulation of DetNet 1838 Where the UDP header is used as defined in Section 3 of [RFC7510]. 1840 As in Section 6.2, the S-Label is used to identify a DetNet flow to 1841 the peer node that processes it, in this case the node addressed by 1842 the IP Header in Figure 19. The S-Label is allocated from the 1843 receiving node?s platform label space [RFC3031]. 1845 In ingress Edge Nodes, the encapsulation in Figure 19 will be imposed 1846 on Detnet Flow Payload Packets as received from DetNet End Systems, 1847 and the encapsulation will be removed in egress Edge Nodes as they 1848 transmit the Payload Packets to the End Systems. 1850 Note that this encapsulation works equally well with IPv4 and IPv6. 1852 This encapsulation can also be used in conjunction with segment 1853 routing as specified in [I-D.ietf-spring-segment-routing-mpls]. In 1854 this case, the T-Label(s) in Figure 19 should be retained, and at 1855 each hop, the top T-label is popped and mapped to a corresponding 1856 UDP/IP tunnel, resulting in the following encapsulation: 1858 +---------------------------------+ 1859 | | 1860 | DetNet Flow | 1861 | Payload Packet | 1862 | | 1863 +---------------------------------+ <--\ 1864 | DetNet Control Word | | 1865 +---------------------------------+ +--> DetNet data plane 1866 | S-Label | | MPLS encapsulation 1867 +---------------------------------+ <--/ 1868 | T-Label(s) | 1869 +---------------------------------+ 1870 | UDP Header | 1871 +---------------------------------+ 1872 | IP Header | 1873 +---------------------------------+ 1874 | Data-Link | 1875 +---------------------------------+ 1876 | Physical | 1877 +---------------------------------+ 1879 Figure 20: IP Encapsulation of DetNet with MPLS-SR 1881 Again, the UDP header is used as defined in Section 3 of [RFC7510]. 1883 Note that if required in both the case of IP Encapsulation of DetNet 1884 Figure 19, and of IP Encapsulation of DetNet with MPLS-SR Figure 20, 1885 it is possible to omit the UDP header if required. Operation of MPLS 1886 directly over IP is described in [RFC4023]. In this case DetNet 1887 Service can be provided on a per IP flow basis as described in 1888 [I-D.ietf-detnet-dp-sol-ip]. 1890 12. Security considerations 1892 The security considerations of DetNet in general are discussed in 1893 [I-D.ietf-detnet-architecture] and [I-D.sdt-detnet-security]. Other 1894 security considerations will be added in a future version of this 1895 draft. 1897 13. IANA considerations 1899 This document makes no IANA requests. 1901 14. Contributors 1903 RFC7322 limits the number of authors listed on the front page of a 1904 draft to a maximum of 5, far fewer than the 20 individuals below who 1905 made important contributions to this draft. The editor wishes to 1906 thank and acknowledge each of the following authors for contributing 1907 text to this draft. See also Section 15. 1909 Loa Andersson 1910 Huawei 1911 Email: loa@pi.nu 1913 Yuanlong Jiang 1914 Huawei 1915 Email: jiangyuanlong@huawei.com 1917 Norman Finn 1918 Huawei 1919 3101 Rio Way 1920 Spring Valley, CA 91977 1921 USA 1922 Email: norman.finn@mail01.huawei.com 1924 Janos Farkas 1925 Ericsson 1926 Magyar Tudosok krt. 11. 1927 Budapest 1117 1928 Hungary 1929 Email: janos.farkas@ericsson.com 1931 Carlos J. Bernardos 1932 Universidad Carlos III de Madrid 1933 Av. Universidad, 30 1934 Leganes, Madrid 28911 1935 Spain 1936 Email: cjbc@it.uc3m.es 1938 Tal Mizrahi 1939 Marvell 1940 6 Hamada st. 1941 Yokneam 1942 Israel 1943 Email: talmi@marvell.com 1945 Lou Berger 1946 LabN Consulting, L.L.C. 1947 Email: lberger@labn.net 1949 Stewart Bryant 1950 Huawei Technologies 1951 Email: stewart.bryant@gmail.com 1953 Mach Chen 1954 Huawei Technologies 1955 Email: mach.chen@huawei.com 1957 15. Acknowledgements 1959 The author(s) ACK and NACK. 1961 The following people were part of the DetNet Data Plane Solution 1962 Design Team: 1964 Jouni Korhonen 1966 Janos Farkas 1968 Norman Finn 1970 Balazs Varga 1972 Loa Andersson 1974 Tal Mizrahi 1976 David Mozes 1978 Yuanlong Jiang 1980 Carlos J. Bernardos 1982 The DetNet chairs serving during the DetNet Data Plane Solution 1983 Design Team: 1985 Lou Berger 1987 Pat Thaler 1989 Thanks for Stewart Bryant for his extensive review of the previous 1990 versions of the document. 1992 16. References 1994 16.1. Normative references 1996 [I-D.ietf-spring-segment-routing-mpls] 1997 Bashandy, A., Filsfils, C., Previdi, S., Decraene, B., 1998 Litkowski, S., and R. Shakir, "Segment Routing with MPLS 1999 data plane", draft-ietf-spring-segment-routing-mpls-14 2000 (work in progress), June 2018. 2002 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2003 Requirement Levels", BCP 14, RFC 2119, 2004 DOI 10.17487/RFC2119, March 1997, 2005 . 2007 [RFC2211] Wroclawski, J., "Specification of the Controlled-Load 2008 Network Element Service", RFC 2211, DOI 10.17487/RFC2211, 2009 September 1997, . 2011 [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification 2012 of Guaranteed Quality of Service", RFC 2212, 2013 DOI 10.17487/RFC2212, September 1997, 2014 . 2016 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2017 "Definition of the Differentiated Services Field (DS 2018 Field) in the IPv4 and IPv6 Headers", RFC 2474, 2019 DOI 10.17487/RFC2474, December 1998, 2020 . 2022 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2023 Label Switching Architecture", RFC 3031, 2024 DOI 10.17487/RFC3031, January 2001, 2025 . 2027 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2028 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2029 Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001, 2030 . 2032 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2033 of Explicit Congestion Notification (ECN) to IP", 2034 RFC 3168, DOI 10.17487/RFC3168, September 2001, 2035 . 2037 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2038 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2039 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 2040 . 2042 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2043 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2044 Protocol Label Switching (MPLS) Support of Differentiated 2045 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 2046 . 2048 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2049 in Multi-Protocol Label Switching (MPLS) Networks", 2050 RFC 3443, DOI 10.17487/RFC3443, January 2003, 2051 . 2053 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 2054 Switching (GMPLS) Signaling Resource ReserVation Protocol- 2055 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 2056 DOI 10.17487/RFC3473, January 2003, 2057 . 2059 [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., 2060 "Encapsulating MPLS in IP or Generic Routing Encapsulation 2061 (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005, 2062 . 2064 [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) 2065 Hierarchy with Generalized Multi-Protocol Label Switching 2066 (GMPLS) Traffic Engineering (TE)", RFC 4206, 2067 DOI 10.17487/RFC4206, October 2005, 2068 . 2070 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2071 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2072 Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385, 2073 February 2006, . 2075 [RFC5085] Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual 2076 Circuit Connectivity Verification (VCCV): A Control 2077 Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085, 2078 December 2007, . 2080 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2081 Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2082 2008, . 2084 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2085 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2086 Class" Field", RFC 5462, DOI 10.17487/RFC5462, February 2087 2009, . 2089 [RFC6003] Papadimitriou, D., "Ethernet Traffic Parameters", 2090 RFC 6003, DOI 10.17487/RFC6003, October 2010, 2091 . 2093 [RFC7510] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black, 2094 "Encapsulating MPLS in UDP", RFC 7510, 2095 DOI 10.17487/RFC7510, April 2015, 2096 . 2098 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2099 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2100 May 2017, . 2102 16.2. Informative references 2104 [G.8275.1] 2105 International Telecommunication Union, "Precision time 2106 protocol telecom profile for phase/time synchronization 2107 with full timing support from the network", ITU-T 2108 G.8275.1/Y.1369.1 G.8275.1, June 2016, 2109 . 2111 [G.8275.2] 2112 International Telecommunication Union, "Precision time 2113 protocol telecom profile for phase/time synchronization 2114 with partial timing support from the network", ITU-T 2115 G.8275.2/Y.1369.2 G.8275.2, June 2016, 2116 . 2118 [I-D.ietf-detnet-architecture] 2119 Finn, N., Thubert, P., Varga, B., and J. Farkas, 2120 "Deterministic Networking Architecture", draft-ietf- 2121 detnet-architecture-08 (work in progress), September 2018. 2123 [I-D.ietf-detnet-dp-alt] 2124 Korhonen, J., Farkas, J., Mirsky, G., Thubert, P., 2125 Zhuangyan, Z., and L. Berger, "DetNet Data Plane Protocol 2126 and Solution Alternatives", draft-ietf-detnet-dp-alt-00 2127 (work in progress), October 2016. 2129 [I-D.ietf-detnet-dp-sol-ip] 2130 Korhonen, J., Varga, B., "DetNet IP Data Plane 2131 Encapsulation", 2018. 2133 [I-D.sdt-detnet-security] 2134 Mizrahi, T., Grossman, E., Hacker, A., Das, S., 2135 "Deterministic Networking (DetNet) Security 2136 Considerations, draft-sdt-detnet-security, work in 2137 progress", 2017. 2139 [IEEE1588] 2140 IEEE, "IEEE 1588 Standard for a Precision Clock 2141 Synchronization Protocol for Networked Measurement and 2142 Control Systems Version 2", 2008. 2144 [IEEE8021CB] 2145 Finn, N., "Draft Standard for Local and metropolitan area 2146 networks - Seamless Redundancy", IEEE P802.1CB 2147 /D2.1 P802.1CB, December 2015, 2148 . 2151 [IEEE8021Q] 2152 IEEE 802.1, "Standard for Local and metropolitan area 2153 networks--Bridges and Bridged Networks (IEEE Std 802.1Q- 2154 2014)", 2014, . 2156 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 2157 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 2158 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 2159 September 1997, . 2161 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 2162 Edge-to-Edge (PWE3) Architecture", RFC 3985, 2163 DOI 10.17487/RFC3985, March 2005, 2164 . 2166 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 2167 "Encapsulation Methods for Transport of Ethernet over MPLS 2168 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 2169 . 2171 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 2172 Sprecher, N., and S. Ueno, "Requirements of an MPLS 2173 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 2174 September 2009, . 2176 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 2177 L., and L. Berger, "A Framework for MPLS in Transport 2178 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 2179 . 2181 [RFC6073] Martini, L., Metz, C., Nadeau, T., Bocci, M., and M. 2182 Aissaoui, "Segmented Pseudowire", RFC 6073, 2183 DOI 10.17487/RFC6073, January 2011, 2184 . 2186 [RFC7551] Zhang, F., Ed., Jing, R., and R. Gandhi, Ed., "RSVP-TE 2187 Extensions for Associated Bidirectional Label Switched 2188 Paths (LSPs)", RFC 7551, DOI 10.17487/RFC7551, May 2015, 2189 . 2191 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 2192 RFC 7950, DOI 10.17487/RFC7950, August 2016, 2193 . 2195 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 2196 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 2197 . 2199 [RFC8169] Mirsky, G., Ruffini, S., Gray, E., Drake, J., Bryant, S., 2200 and A. Vainshtein, "Residence Time Measurement in MPLS 2201 Networks", RFC 8169, DOI 10.17487/RFC8169, May 2017, 2202 . 2204 Appendix A. Example of DetNet data plane operation 2206 [Editor's note: Add a simplified example of DetNet data plane and how 2207 labels etc work in the case of MPLS-based PSN and utilizing PREOF. 2208 The figure is subject to change depending on the further DT decisions 2209 on the label handling..] 2211 Authors' Addresses 2213 Jouni Korhonen (editor) 2215 Email: jouni.nospam@gmail.com 2217 Balazs Varga (editor) 2218 Ericsson 2219 Magyar Tudosok krt. 11. 2220 Budapest 1117 2221 Hungary 2223 Email: balazs.a.varga@ericsson.com