idnits 2.17.1 draft-dt-detnet-dp-alt-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 17, 2016) is 2809 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'Network' is mentioned on line 166, but not defined -- Looks like a reference, but probably isn't: '1' on line 2634 -- Looks like a reference, but probably isn't: '2' on line 2636 -- Looks like a reference, but probably isn't: '3' on line 2638 -- Looks like a reference, but probably isn't: '4' on line 2640 -- Looks like a reference, but probably isn't: '5' on line 2643 -- Looks like a reference, but probably isn't: '6' on line 2645 -- Looks like a reference, but probably isn't: '46' on line 2159 == Unused Reference: 'I-D.ietf-sunset4-gapanalysis' is defined on line 2245, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1Qbv' is defined on line 2281, but no explicit reference was found in the text == Unused Reference: 'IEEE802.1Qch' is defined on line 2292, but no explicit reference was found in the text == Unused Reference: 'TSNTG' is defined on line 2628, but no explicit reference was found in the text == Outdated reference: A later version (-06) exists of draft-eckert-bier-te-arch-04 == Outdated reference: A later version (-08) exists of draft-finn-detnet-architecture-07 == Outdated reference: A later version (-13) exists of draft-ietf-6man-rfc2460bis-05 == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-01 == Outdated reference: A later version (-08) exists of draft-ietf-bier-architecture-04 == Outdated reference: A later version (-09) exists of draft-ietf-detnet-problem-statement-00 == Outdated reference: A later version (-15) exists of draft-ietf-mpls-residence-time-11 == Outdated reference: A later version (-15) exists of draft-ietf-spring-segment-routing-09 == Outdated reference: A later version (-09) exists of draft-ietf-sunset4-gapanalysis-07 == Outdated reference: A later version (-02) exists of draft-ooamdt-rtgwg-ooam-requirement-01 -- Obsolete informational reference (is this intentional?): RFC 2460 (Obsoleted by RFC 8200) -- Obsolete informational reference (is this intentional?): RFC 4447 (Obsoleted by RFC 8077) -- Obsolete informational reference (is this intentional?): RFC 7752 (Obsoleted by RFC 9552) Summary: 0 errors (**), 0 flaws (~~), 16 warnings (==), 12 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet J. Korhonen, Ed. 3 Internet-Draft Broadcom 4 Intended status: Informational J. Farkas 5 Expires: February 18, 2017 G. Mirsky 6 Ericsson 7 P. Thubert 8 Cisco 9 Y. Zhuang 10 Huawei 11 L. Berger 12 LabN 13 August 17, 2016 15 DetNet Data Plane Protocol and Solution Alternatives 16 draft-dt-detnet-dp-alt-03 18 Abstract 20 This document identifies existing IP and MPLS, and other 21 encapsulations that run over IP and/or MPLS data plane technologies 22 that can be considered as the base line solution for deterministic 23 networking data plane definition. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at http://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on February 18, 2017. 42 Copyright Notice 44 Copyright (c) 2016 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 61 3. DetNet Data Plane Overview . . . . . . . . . . . . . . . . . 3 62 3.1. Example DetNet Service Scenarios . . . . . . . . . . . . 6 63 4. Criteria for data plane solution alternatives . . . . . . . . 8 64 4.1. #1 Encapsulation and overhead . . . . . . . . . . . . . . 9 65 4.2. #2 Flow identification . . . . . . . . . . . . . . . . . 9 66 4.3. #3 Packet sequencing and duplicate elimination . . . . . 9 67 4.4. #4 Explicit routes . . . . . . . . . . . . . . . . . . . 10 68 4.5. #5 Flow duplication and merging . . . . . . . . . . . . . 10 69 4.6. #6 Operations, Administration and Maintenance . . . . . . 11 70 4.7. #8 Class and quality of service capabilities . . . . . . 11 71 4.8. #9 Packet traceability . . . . . . . . . . . . . . . . . 12 72 4.9. #10 Technical maturity . . . . . . . . . . . . . . . . . 12 73 5. Data plane solution alternatives . . . . . . . . . . . . . . 12 74 5.1. DetNet Transport layer technologies . . . . . . . . . . . 13 75 5.1.1. Native IPv6 transport . . . . . . . . . . . . . . . . 13 76 5.1.2. Native IPv4 transport . . . . . . . . . . . . . . . . 16 77 5.1.3. Multiprotocol Label Switching (MPLS) . . . . . . . . 19 78 5.1.4. Bit Indexed Explicit Replication (BIER) . . . . . . . 23 79 5.1.5. BIER - Traffic Engineering (BIER-TE) . . . . . . . . 27 80 5.2. DetNet Service layer technologies . . . . . . . . . . . . 34 81 5.2.1. Generic Routing Encapsulation (GRE) . . . . . . . . . 34 82 5.2.2. MPLS-based Services for DetNet . . . . . . . . . . . 36 83 5.2.3. Pseudo Wire Emulation Edge-to-Edge (PWE3) . . . . . . 37 84 5.2.4. MPLS-Based Ethernet VPN (EVPN) . . . . . . . . . . . 41 85 5.2.5. Higher layer header fields . . . . . . . . . . . . . 43 86 6. Summary of data plane alternatives . . . . . . . . . . . . . 45 87 7. Security considerations . . . . . . . . . . . . . . . . . . . 47 88 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 47 89 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 48 90 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 48 91 10.1. Informative References . . . . . . . . . . . . . . . . . 48 92 10.2. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 57 93 Appendix A. Examples of combined DetNet Service and Transport 94 layers . . . . . . . . . . . . . . . . . . . . . . . 58 95 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 58 97 1. Introduction 99 Deterministic Networking (DetNet) [I-D.ietf-detnet-problem-statement] 100 provides a capability to carry unicast or multicast data flows for 101 real-time applications with extremely low data loss rates, timely 102 delivery and bounded packet delay variation 103 [I-D.finn-detnet-architecture]. The deterministic networking Quality 104 of Service (QoS) is expressed as 1) the minimum and the maximum end- 105 to-end latency from source (talker) to destination (listener), and 2) 106 probability of loss of a packet. Only the worst-case values for the 107 mentioned parameters are concerned. 109 There are three techniques to achieve the QoS required by 110 deterministic networks: 112 o Congestion protection, 113 o explicit routes, 114 o service protection. 116 This document identifies existing IP and Multiprotocol Label 117 Switching (MPLS) [RFC3031], layer-2 or layer-3 encapsulations and 118 transport protocols that could be considered as foundations for a 119 deterministic networking data plane. The full scope of the 120 deterministic networking data plane solution is considered including, 121 as appropriate: quality of service (QoS); Operations, Administration 122 and Maintenance (OAM); and time synchronization among other criteria 123 described in Section 4. 125 This document does not select a deterministic networking data plane 126 protocol. It does, however, elaborate what it would require to adapt 127 and use a specific protocol as the deterministic networking data 128 plane solution. This document is only concerned with data plane 129 considerations and, specifically, with topics that potentially impact 130 potential deterministic networking aware data plane hardware. 131 Control plane considerations are out of scope of this document. 133 2. Terminology 135 This document uses the terminology established in the DetNet 136 architecture [I-D.finn-detnet-architecture]. 138 3. DetNet Data Plane Overview 140 A "Deterministic Network" will be composed of DetNet enabled nodes 141 i.e., End Systems, Edge Nodes, Relay Nodes and collectively deliver 142 DetNet services. DetNet enabled nodes are interconnected via Transit 143 Nodes (i.e., routers) which support DetNet, but are not DetNet 144 service aware. Transit nodes see DetNet nodes as end points. All 145 DetNet enabled nodes are connect to sub-networks, where a point-to- 146 point link is also considered as a simple sub-network. These sub- 147 networks will provide DetNet compatible service for support of DetNet 148 traffic. Examples of sub-networks include IEEE 802.1TSN and OTN. Of 149 course, multi-layer DetNet systems may also be possible, where one 150 DetNet appears as a sub-network, and provides service to, a higher 151 layer DetNet system. A simple DetNet concept network is shown in 152 Figure 1. 154 TSN Edge Transit Relay DetNet 155 End System Node Node Node End System 157 +---------+ +.........+ +---------+ 158 | Appl. |<---:Svc Proxy:-- End to End Service ---------->| Appl. | 159 +---------+ +---------+ +---------+ +---------+ 160 | TSN | |TSN| |Svc|<-- DetNet flow ---: Service :-->| Service | 161 +---------+ +---+ +---+ +---------+ +---------+ +---------+ 162 |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| 163 +-------.-+ +-.-+ +-.-+ +--.----.-+ +-.-+ +-.-+ +---.-----+ 164 : Link : / ,-----. \ : Link : / ,-----. \ 165 +........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ 166 [Network] [Network] 167 `-----' `-----' 169 Figure 1: A Simple DetNet Enabled Network 171 The DetNet data plane is logically divided into two layers (also see 172 Figure 2): 174 DetNet Service Layer 176 The DetNet service layer provides adaptation of DetNet services. 177 It is composed of a shim layer to carry deterministic flow 178 specific attributes, which are needed during forwarding and for 179 service protection. DetNet enabled end systems originate and 180 terminate the DetNet Service layer and are peers at the DetNet 181 Service layer. DetNet relay and edge nodes also implement DetNet 182 Service layer functions. The DetNet service layer is used to 183 deliver traffic end to end across a DetNet domain. 185 DetNet Transport Layer 187 The DetNet transport layer is required on all DetNet nodes. All 188 DetNet nodes are end points and the transport layer. Non-DetNet 189 service aware transit nodes deliver traffic between DetNet nodes. 190 The DetNet transport layer operates below and supports the DetNet 191 Service layer and optionally provides congestion protection for 192 DetNet flows. 194 Distinguishing the function of these two DetNet data plane layers 195 helps to explore and evaluate various combinations of the data plane 196 solutions available. This separation of DetNet layers, while 197 helpful, should not be considered as formal requirement. For 198 example, some technologies may violate these strict layers and still 199 be able to deliver a DetNet service. 201 . 202 . 203 +-----------+ 204 | Service | PW, RTP/(UDP), GRE 205 +-----------+ 206 | Transport | (UDP)/IPv6, (UDP)/IPv4, MPLS LSPs, BIER, BIER-TE 207 +-----------+ 208 . 209 . 211 Figure 2: DetNet adaptation to data plane 213 The two logical layers defined here aim to help to identify which 214 data plane technology can be used for what purposes in the DetNet 215 context. This layering is similar to the data plane concept of MPLS, 216 where some part of the label stack is "Service" specific (e.g., PW 217 labels, VPN labels) and an other part is "Transport" specific (e.g, 218 LSP label, TE label(s)). 220 In some networking scenarios, the end system initially provides a 221 DetNet flow encapsulation, which contains all information needed by 222 DetNet nodes (e.g., Real-time Transport Protocol (RTP) [RFC3550] 223 based DetNet flow transported over a native UDP/IP network or 224 PseudoWire). In other scenarios, the encapsulation formats might 225 differ significantly. As an example, a CPRI "application's" I/Q data 226 mapped directly to Ethernet frames may have to be transported over an 227 MPLS-based packet switched network (PSN). 229 There are many valid options to create a data plane solution for 230 DetNet traffic by selecting a technology approach for the DetNet 231 Service layer and also selecting a technology approach for the DetNet 232 Transport layer. There are a high number of valid combinations. 233 Therefore, not the combinations but the different technologies are 234 evaluated along the criteria collected in Section 4. Different 235 criteria apply for the DetNet Service layer and the DetNet Transport 236 layer, however, some of the criteria are valid for both layers. 238 One of the most fundamental differences between different potential 239 data plane options is the basic addressing and headers used by DetNet 240 end systems. For example, is the basic service a Layer 2 (e.g., 241 Ethernet) or Layer 3 (i.e., IP) service. This decision impacts how 242 DetNet end systems are addressed, and the basic forwarding logic for 243 the DetNet Service layer. 245 3.1. Example DetNet Service Scenarios 247 In an attempt to illustrate a DetNet date plane, this document uses 248 the Multi-Segment Pseudowire Emulation Edge-to-Edge (PWE3) [RFC5254] 249 reference model shown in Figure 3 as the foundation for different 250 DetNet data plane deployment options and how layering could work. 251 Other reference models are possible but not covered in this document. 252 Note that other technologies can be also used to implement DetNet, 253 Multi-Segment PW is only used here to illustrate functions, features 254 and layering from the perspective of the architecture. 256 Native |<--------Multi-Segment Pseudowire----->| Native 257 Service | PSN PSN | Service 258 (AC) | |<-Tunnel->| |<-Tunnel->| | (AC) 259 | V V 1 V V 2 V V | 260 | +-----+ +-----+ +---- + | 261 +---+ | |T-PE1|==========|S-PE1|==========|T-PE2| | +---+ 262 | |---|-----|........PW1...........|...PW3..........|---|----| | 263 |CE1| | | | | | | | | |CE2| 264 | |---------|........PW2...........|...PW4..........|--------| | 265 +---+ | | |==========| |==========| | | +---+ 266 ^ +-----+ +-----+ +-----+ ^ 267 | Provider Edge 1 ^ Provider Edge 3 | 268 | | | 269 | PW switching point | 270 | | 271 |<------------------- Emulated Service ------------------->| 273 Figure 3: Pseudo Wire switching reference model 275 Figure 4 illustrates how DetNet can provide services for IEEE 276 802.1TSN end systems over a DetNet enabled network. The edge nodes 277 insert and remove required DetNet data plane encapsulation. The 'X' 278 in the edge and relay nodes represents a potential DetNet flow packet 279 replication and elimination point. This conceptually parallels L2VPN 280 services, and could leverage existing related solutions as discussed 281 below. 283 TSN |<----- End to End DetNet Service ----->| TSN 284 Service | Transit Transit | Service 285 TSN (AC) | |<-Tunnel->| |<-Tunnel->| | (AC) TSN 286 End | V V 1 V V 2 V V | End 287 System | +-----+ +-----+ +---- + | System 288 +---+ | |T-PE1|==========|S-PE1|==========|T-PE2| | +---+ 289 | |---|-----|.X_..DetNet Flow1..X..|...DF3........X.|---|----| | 290 |CE1| | | \ | | | | / | | |CE2| 291 | | |...X_...DF2........X..|...DF4......X_..| | | 292 +---+ | |==========| |==========| | +---+ 293 ^ +-----+ +-----+ +-----+ ^ 294 | Edge Node Relay Node Edge Node | 295 | | 296 |<--------------- Emulated TSN Service ------------------->| 298 Figure 4: IEEE 802.1TSN over DetNet 300 Figure 5 illustrates how end to end native DetNet service can be 301 provided. In this case, the end systems are able to send and receive 302 native DetNet flows. For example, as PseudoWire (PW) encapsulated 303 IP. Like earlier the 'X' in the end systems, edge and relay nodes 304 represents potential DetNet flow packet replication and elimination 305 points. Here the relay nodes may change the underlying transport, 306 for example replacing IP with MPLS or tunneling IP over MPLS (e.g., 307 via L3VPNs), or simply interconnect network domains. 309 DetNet DetNet 310 Service Transit Transit Service 311 DetNet | |<-Tunnel->| |<-Tunnel->| | DetNet 312 End | V 1 V V 2 V | End 313 System | +-----+ +-----+ +-----+ | System 314 +---+ | |S-PE1|==========|S-PE2|==========|S-PE3| | +---+ 315 | X....DFa.....X_.......DF1.......X_....DF3........X.....DFa...X | 316 |CE1|=========| \ | | / | | / |========|CE2| 317 | | | | \......DF2.....X_......DF4....../ | | | | 318 +---+ | |==========| |==========| | +---+ 319 ^ +-----+ +-----+ +-----+ ^ 320 | Relay Node Relay Node Relay Node | 321 | | 322 |<------------- End to End DetNet Service ---------------->| 324 Figure 5: Native DetNet 326 Figure 6 illustrates how a IEEE 802.1TSN end system could communicate 327 with a native DetNet end system through an edge node which provides a 328 TSN to DetNet inter-working capability. The edge node would add and 329 remove required DetNet data plane encapsulation as well as provide 330 any needed address mapping. As in previous figures, the 'X' in the 331 end systems, edge and relay nodes represents potential DetNet flow 332 packet duplication and elimination points. 334 TSN |<----- End to End DetNet Service -------------->| 335 Service | Transit Transit | 336 TSN (AC) | |<-Tunnel->| |<-Tunnel->| DetNet | DetNet 337 End | V V 1 V V 2 V Service | End 338 System | +-----+ +-----+ +-----+ | V System 339 +---+ | |T-PE1|==========|S-PE1|==========|S-PE2| | +---+ 340 | |---|-----|.X_.......DF1......X..|...DF3........X.|...DFa...X | 341 |CE1| | | \ | | | | / |========|CE2| 342 | | | \.....DF2.......X..|...DF4....../ | | | | 343 +---+ | |==========| |==========| | +---+ 344 ^ +-----+ +-----+ +-----+ ^ 345 | Edge Node Relay Node Relay Node | 346 | | 347 |<----------------- End to End Service ------------------->| 349 Figure 6: IEEE 802.1TSN to native DetNet 351 4. Criteria for data plane solution alternatives 353 This section provides criteria to help to evaluate potential options. 354 Each deterministic networking data plane solution alternative is 355 described and evaluated using the criteria described in this section. 356 The used criteria enumerated in this section are selected so that 357 they highlight the existence or lack of features that are expected or 358 seen important to a solution alternative for the data plane solution. 360 The criteria for the DetNet Service layer: 362 #1 Encapsulation and overhead 363 #2 Flow identification (Service ID part of the DetNet flows) 364 #3 Packet sequencing and duplicate elimination 365 #5 Flow duplication and merging 366 #6 Operations, Administration and Maintenance (capabilities) 367 #8 Class and quality of service capabilities (DetNet Service 368 specific) 369 #10 Technical maturity 371 The criteria for the DetNet Transport layer: 373 #1 Encapsulation and overhead 374 #2 Flow identification 375 #4 Explicit routes (network path) 376 #5 Flow duplication and merging (sometimes, flow duplication and 377 merging is also doable at the transport layer, not just at the 378 service layer) 380 #6 Operations, Administration and Maintenance (capabilities, 381 performance management, packet traceability) 382 #8 Class and quality of service capabilities (DetNet Transport 383 specific) 384 #9 Packet traceability (can be part of OAM) 385 #10 Technical maturity 387 [Editor's Note: numbering is off because #7 is removed.] 389 [Editor's Note: #9 should(?) be integrated into #6.] 391 Most of the criteria is relevant for both the DetNet Service and 392 DetNet Transport layers. However, different aspects of the same 393 criteria may relevant for different layers, for example, as it is the 394 case with criteria #5 Packet replication and elimination. 396 4.1. #1 Encapsulation and overhead 398 Encapsulation and overhead is related to how the DetNet data plane 399 carries DetNet flow. In several cases a DetNet flow has to be 400 encapsulated inside other protocols, for example, when transporting a 401 layer-2 Ethernet frame over an IP transport network. In some cases a 402 tunneling like encapsulation can be avoided by underlying transport 403 protocol translation, for example, translating layer-2 Ethernet frame 404 including addressing and flow identification into native IP traffic. 405 Last it is possible that sources and destinations handle 406 deterministic flows natively in layer-3. This criteria concerns what 407 is the encapsulation method the solution alternative support: 408 tunneling like encapsulation, protocol translation or native layer-3 409 transport. In addition to the encapsulation mechanism this criteria 410 is also concerned of the processing and specifically the encapsulate 411 header overhead. 413 4.2. #2 Flow identification 415 The solution alternative has to provide means to identify specific 416 deterministic flows. The flow identification can, for example, be 417 explicit field in the data plane encapsulation header or implicitly 418 encoded into the addressing scheme of the used data plane protocol or 419 their combination. This criteria concerns the availability and 420 details of deterministic flow identification the data plane protocol 421 alternative has. 423 4.3. #3 Packet sequencing and duplicate elimination 425 The solution alternative has to provide means for end systems to 426 number packets sequentially and transport that sequencing information 427 along with the sent packets. In addition to possible reordering 428 packets other important uses for sequencing are detecting duplicates 429 and lost packets. 431 In a case of intentional packet duplication a combination of flow 432 identification and packet sequencing allows for detecting and 433 eliminating duplicates at the destination (see Section 4.5 for more 434 details). 436 4.4. #4 Explicit routes 438 The solution alternative has to provide a mechanism(s) for 439 establishing explicit routes that all packets belonging to a 440 deterministic flow will follow. The explicit route can be seen as a 441 form of source routing or a pre-reserved path e.g., using some 442 network management procedure. It should be noted that the explicit 443 route does not need to be detailed to a level where every possible 444 intermediate node along the path is part of the named explicit route. 445 RSVP-TE [RFC3209] supports explicit routes, and typically provides 446 pinned data paths for established LSPs. At Layer-2, the IEEE 447 802.1Qca [IEEE802.1Qca] specification defines how to do explicit path 448 control in a bridged network and its IETF counter part is defined in 449 [RFC7813]. This criteria concerns the available mechanisms for 450 explicit routes for the data plane protocol alternative. 452 4.5. #5 Flow duplication and merging 454 Flow duplication and flow merging are methods being considered to 455 provide DetNet service protection. The objective for supporting flow 456 duplication and flow merging is to enable hitless (or lossless) 1+1 457 protection. Other methods, if so identified, are also permissible. 459 The solution alternative has to provide means for end systems, relay 460 and edge nodes to be able to duplicate packets into duplicate flows, 461 and later merge the flows into one for duplicate elimination. The 462 duplication and merging may take place at multiple points in the 463 network in order to ensure that one (or more) equipment failure 464 event(s) still leave at least one path intact for a deterministic 465 networking flow. The goal is again to enable hitless 1+1 protection 466 in a way that no packet gets lost or there is no ramp up time when 467 either one of the paths fails for one reason or another. 469 Another concern regarding packet duplication is how to enforce 470 duplicated packets to take different route or path while the final 471 destination still remains the same. With strict source routing, all 472 the intermediate hops are listed and paths can be guaranteed to be 473 non-overlapping. Loose source routing only signals some of the 474 intermediate hops and it takes additional knowledge to ensure that 475 there is no single point of failure. 477 The IEEE 802.1CB (seamless redundancy) [IEEE8021CB] is an example of 478 Ethernet-based solution that defines packet sequence numbering, flow 479 duplication, flow merging, duplicate packet identification and 480 elimination. The deterministic networking data plane solution 481 alternative at layer-3 has to provide equivalent functionality. This 482 criteria concerns the available mechanisms for packet replication and 483 duplicate deletion the data plane protocol alternative has. 485 4.6. #6 Operations, Administration and Maintenance 487 The solution alternative should demonstrate an availability of 488 appropriate standardized OAM tools that can be extended for 489 deterministic networking purposes with a reasonable effort, when 490 required. The OAM tools do not necessarily need to be specific to 491 the data plane protocol as it could be the case, for example, with 492 MPLS-based data planes. But any OAM-related implications or 493 requirements on data plane hardware must be considered. 495 The OAM includes but is not limited to tools listed in the 496 requirements for overlay networks 497 [I-D.ooamdt-rtgwg-ooam-requirement]. Specifically, the performance 498 management requirements are of interest at both service and transport 499 layers. 501 4.7. #8 Class and quality of service capabilities 503 Class and quality of service, i.e., CoS and QoS, are terms that are 504 often used interchangeably and confused. In the context of DetNet, 505 CoS is used to refer to mechanisms that provide traffic forwarding 506 treatment based on aggregate group basis and QoS is used to refer to 507 mechanisms that provide traffic forwarding treatment based on a 508 specific DetNet flow basis. Examples of CoS mechanisms include 509 DiffServ which is enabled by IP header differentiated services code 510 point (DSCP) field [RFC2474] and MPLS label traffic class field 511 [RFC5462], and at Layer-2, by IEEE 802.1p priority code point (PCP). 513 Quality of Service (QoS) mechanisms for flow specific traffic 514 treatment typically includes a guarantee/agreement for the service, 515 and allocation of resources to support the service. Example QoS 516 mechanisms include discrete resource allocation, admission control, 517 flow identification and isolation, and sometimes path control, 518 traffic protection, shaping, policing and remarking. Example 519 protocols that support QoS control include Resource ReSerVation 520 Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473]. 522 A critical DetNet service enabled by QoS (and perhaps CoS) is 523 delivering zero congestion loss. There are different mechanisms that 524 maybe used separately or in combination to deliver a zero congestion 525 loss service. The key aspect of this objective is that DetNet 526 packets are not discarded due to congestion at any point in a DetNet 527 aware network. 529 In the context of the data plane solution there should be means for 530 flow identification, which then can be used to map a flow against 531 specific resources and treatment in a node enforcing the QoS. 532 Hereto, certain aspects of CoS and QoS may be provided by the 533 underlying sub-net technology, e.g., actual queuing or IEEE 802.3x 534 priority flow control (PFC). 536 4.8. #9 Packet traceability 538 For the network management and specifically for tracing 539 implementation or network configuration errors any means to find out 540 whether a packet is a replica, which node performed replication, and 541 which path was intended for the replica, can be very useful. This 542 criteria concerns the availability of solutions for tracing packets 543 in the context of data plane protocol alternative. Packet 544 traceability can also be part of OAM. 546 4.9. #10 Technical maturity 548 The technical maturity of the data plane solution alternative is 549 crucial, since it basically defines the effort, time line and risks 550 involved for the use of the solution in deployments. For example, 551 the maturity level can be categorized as available immediately, 552 available with small extensions, available with re-purposing/ 553 redefining portions of the protocol or its header fields. Yet 554 another important measure for maturity is the deployment experience. 555 This criteria concerns the maturity of the data plane protocol 556 alternative as the solution alternative. This criteria is 557 particularly important given, as previously noted, that the DetNet 558 data plane solution is expected to impact, i.e., be supported in, 559 hardware. 561 5. Data plane solution alternatives 563 The following sections describe and rate deterministic data plane 564 solution alternatives. In "Analysis and Discussion" section each 565 alternative is evaluated against the criteria given in Section 4 and 566 rated using the following: (M)eets the criteria, (W)ork needed, and 567 (N)ot suitable or too much work envisioned. 569 5.1. DetNet Transport layer technologies 571 5.1.1. Native IPv6 transport 573 5.1.1.1. Solution description 575 This section investigates the application of native IPv6 [RFC2460] as 576 the data plane for deterministic networking along the criteria 577 collected in Section 4. 579 The application of higher OSI layer headers, i.e., headers deeper in 580 the packet, can be considered. Two aspects have to be taken into 581 account for such solutions. (i) Those header fields can be encrypted. 582 (ii) Those header fields are deeper in the packet, therefore, routers 583 have to apply deep packet inspection. See further details in 584 Section 5.2.5. 586 5.1.1.2. Analysis and Discussion 588 #1 Encapsulation and overhead (M) 590 IPv6 can encapsulate DetNet Service layer headers (and associated 591 DetNet flow payload) like any other upper-layer header indicated 592 by the Next Header. The fixed header of an IPv6 packet is 40 593 bytes [RFC2460]. This overhead is bigger if any Extension Header 594 is used, and a generic behaviour for host and forwarding nodes is 595 specified in [RFC7045]. However, the exact overhead (Section 4.1) 596 depends on what solution is actually used to provide DetNet 597 features, e.g., explicit routing or DetNet service protection if 598 any of these is applied. 600 IPv6 has two types of Extension Headers that are processed by 601 intermediate routers between the source and the final destination 602 and may be of interest for the data plane signaling, the Routing 603 Header that is used to direct the traffic via intermediate routers 604 in a strict or loose source routing way, and the Hop-by-Hop 605 Options Header that carries optional information that must be 606 examined by every node along a packet's delivery path. The Hop- 607 by-Hop Options Header, when present, must immediately follow the 608 IPv6 Header and it is not possible to limit its processing to the 609 end points of Source Routed segments. 611 IPv6 also provides a Destination Options Header that is used to 612 carry optional information to be examined only by a packet's 613 destination node(s). The encoding of the options used in the Hop- 614 by-Hop and in the Destination Options Header indicates the 615 expected behavior when a processing IPv6 node does not recognize 616 the Option Type, e.g. skip or drop; it should be noted that due to 617 performance restrictions nodes may ignore the Hop-by-Hop Option 618 Header, drop packets containing a Hop-by-Hop Option Header, or 619 assign packets containing a Hop-by-Hop Option Header to a slow 620 processing path [I-D.ietf-6man-rfc2460bis] (e.g. punt packets from 621 hardware to software forwarding which is highly detrimental to the 622 performance). 624 The creation of new Extension Headers that would need to be 625 processed by intermediate nodes is strongly discouraged. In 626 particular, new Extension Header(s) having hop-by-hop behavior 627 must not be created or specified. New options for the existing 628 Hop-by-Hop Header should not be created or specified unless no 629 alternative solution is feasible [RFC6564]. 631 #2 Flow identification (W) 633 The 20-bit flow label field of the fixed IPv6 header is suitable 634 to distinguish different deterministic flows. But guidance on the 635 use of the flow label provided by [RFC6437] places restrictions on 636 how the flow label can be used. In particular, labels should be 637 chosen from an approximation to a discrete uniform distribution. 638 Additionally, existing implementations generally do not open APIs 639 to control the flow label from the upper layers. 641 Alternatively, the Flow identification could be transported in a 642 new option in the Hop-by-Hop Options Header. 644 #4 Explicit routes (W) 646 One possibility is for a Software-Defined Networking (SDN) 647 [RFC7426] based approach to be applied to compute, establish and 648 manage the explicit routes, leveraging Traffic Engineering (TE) 649 extensions to routing protocols [RFC5305] [RFC7752] and evolving 650 to the Path Computation Element (PCE) Architecture [RFC5440], 651 though a number of issues remain to be solved [RFC7399]. 653 Segment Routing (SR) [I-D.ietf-spring-segment-routing] is a new 654 initiative to equip IPv6 with explicit routing capabilities. The 655 idea for the DetNet data plane would be to apply SR to IPv6 with 656 the addition of a new type of routing extension header 657 [I-D.ietf-6man-segment-routing-header] to explicitly signal the 658 path in the data plane between the source and the destination, 659 and/or between replication points and elimination points if this 660 functionality is used. 662 #5 Flow duplication and merging (W) 663 The functionality of replicating a packet exists in IPv6 but is 664 limited to multicast flows. In order to enforce replicated 665 packets to take different routes and eventually again merge flow 666 (bring them to a specific merging point), IP-in-IP encapsulation 667 and Segment Routing could be leveraged to signal a segment in a 668 packet. A replication point would insert a different routing 669 header in each copy it makes, the routing header providing 670 explicitly the hops to the merging point for that particular 671 replica of the packet, in a strict or in a loose source routing 672 fashion. A flow merging point would pop the routing headers from 673 the various copies it gets and do the rest of the required 674 processing for merging the two flows into one flow. 676 #6 Operations, Administration and Maintenance (M/W) 678 IPv6 enjoys the existing toolbox for generic IP network 679 management. However, IPv6 specific management features are still 680 not at the level comparable to that of IPv4. Particular areas of 681 concerns are those that are IPv6 specific, for example, related to 682 neighbor discovery protocol (ND), stateless address 683 autoconfiguration (SLAAC), subscriber identification, and 684 security. While the standards are already mostly in place the 685 implementations in deployed equipment can be lacking or inadequate 686 for commercial deployments. This is larger issue with older 687 existing equipment. 689 #8 Class and quality of service capabilities (W) 691 IPv6 provides support for CoS and QoS. CoS is provided by 692 DiffServ which is enabled by IP header differentiated services 693 code point (DSCP) and QoS is defined as part of RSVP [RFC2205]. 694 DiffServ support is widely available, while RSVP for IP packets is 695 generally not supported. 697 #9 Packet traceability (W) 699 The traceability of replicated packets involves the capability to 700 resolve which replication point issued a particular copy of a 701 packet, which segment was intended for that replica, and which 702 particular packet of which particular flow this is. Sequence also 703 depends on the sequencing mechanism. As an example, the 704 replication point may be indicated as the source of the packet if 705 IP-in-IP encapsulation is used to forward along segments. Another 706 alternate to IP-in-IP tunneling along segments would be to protect 707 the original source address in a destination option similar to the 708 Home Address option [RFC6275] and then use the address of the 709 replication point as source in the IP header. 711 The traceability also involves the capability to determine if a 712 particular segment is operational. While IPv6 as such has no 713 support for reversing a path, it appears source route extensions 714 such as the one defined for segment routing could be used for 715 tracing purposes. Though it is not a usual practice, IPv6 716 [RFC2460] expects that a Source Route path may be reversed, and 717 the standard insists that a node must not include the reverse of a 718 Routing Header in the response unless the received Routing Header 719 was authenticated. 721 #10 Technical maturity (M/W) 723 IPv6 has been around about 20 years. However, large scale global 724 and commercial IPv6 deployments are rather new dating only few 725 years back to around 2012. While IPv6 has proven itself for best 726 effort traffic, DiffServ usage is less common and QoS capabilities 727 are not currently present. Additional, there are number of small 728 issues to work on as they show up once operations experience 729 grows. 731 The Cisco 6Lab site [1] provides information on IPv6 deployment 732 per country, indicating figures for prefixes, transit AS, content 733 and users. Per this site, many countries, including Canada, 734 Brazil, the USA, Germany, France, Japan, Portugal, Sweden, 735 Finland, Norway, Greece, and Ecuador, achieve a deployment ratio 736 above 30 percent, and the overall adoption reported by Google 737 Statistics [2] is now above 10 percent. 739 5.1.1.3. Summary 741 IPv6 supports a significant portion of the identified DetNet data 742 plane criteria today. There are aspects of the DetNet data plane 743 that are not fully supported, notably QoS, but these can be 744 incrementally added or supplemented by the underlying sub-network 745 layer. IPv6 may be a choice as the DetNet Transport layer in 746 networks where other technologies such as MPLS are not deployed. 748 5.1.2. Native IPv4 transport 750 5.1.2.1. Solution description 752 IPv4 [RFC0791] is in principle the same as IPv6, except that it has a 753 smaller address space. However, IPv6 was designed around the fact 754 that extension headers are an integral part of the protocol and 755 operation from the beginning, although the practice may some times 756 prove differently [RFC7872]. IPv4 does support header options, but 757 these have historically not been supported on in hardware-based 758 forwarding so are generally blocked or handled at a much slower rate. 759 In either case, the use of IP header options is generally avoided. 760 In the context of deterministic networking data plane solutions the 761 major difference between IPv4 and IPv6 seems to be the practical 762 support for header extensibility. Anything below and above the IP 763 header independent of the version is practically the same. 765 5.1.2.2. Analysis and Discussion 767 #1 Encapsulation and overhead (M) 769 The fixed header of an IPv4 packet is 20 bytes [RFC0791]. IP 770 options add overhead, but are not generally used and are not 771 considered as part of this document. 773 #2 Flow identification (W) 775 The IPv4 header has a 16-bit identification field that was 776 originally intended for assisting fragmentation and reassembly of 777 IPv4 packets as described in [RFC0791]. The identification field 778 has also been proposed to be used for actually identifying flows 779 between two IP addresses and a given protocol for detecting and 780 removing duplicate packets [RFC1122]. However, recent update 781 [RFC6864] to both [RFC0791] and [RFC1122] restricts the use of 782 IPv4 identification field only to fragmentation purposes. 784 The IPv4 also has a stream identifier option [RFC0791], which 785 contains a 16-bit SATNET stream identifier. However, the option 786 has been deprecated [RFC6814]. The conclusion is that stream 787 identification does not work nicely with IPv4 header alone and a 788 traditional 5-tuple identification might not also be enough in a 789 case of a flow duplication or encrypted flows. For a working 790 solution, upper layer protocol headers such as RTP or PWs may be 791 required for unambiguous flow identification. There is also 792 emerging work within the IETF that may provide new flow 793 identification alternatives. 795 #4 Explicit routes (W) 797 IPv4 has two source routing option specified: the loose source and 798 record route option (LSRR), and the strict source and record route 799 option (SSRR) [RFC0791]. The support of these options in the 800 Internet is questionable but within a closed network the support 801 may be assumed. But as both these options use IP header options, 802 which are generally not supported in hardware, use of these 803 options are questionable. Of course, the same options of SDN and 804 SR approaches discussed above for IPv6 may be equally applicable 805 to IPv4. 807 #5 Flow duplication and merging (W/N) 809 The functionality of replicating a packet exists in IPv4 but is 810 limited to multicast flows. In general the issue regarding the 811 IPv6 packet replication also applies to IPv4. Duplicate packet 812 detection for IPv4 is studied in [RFC6621] to a great detail in 813 the context of simplified multicast forwarding. In general there 814 is no good way to detect duplicated packets for IPv4 without 815 additional upper layer protocol support. 817 #6 Operations, Administration and Maintenance (M) 819 IPv4 enjoys the extensive and "complete" existing toolbox for 820 generic IP network management. 822 #8 Class and quality of service capabilities (M/W) 824 IPv4 provides support for CoS and QoS. CoS is provided by 825 DiffServ which is enabled by IP header differentiated services 826 code point (DSCP) and QoS is defined as part of RSVP [RFC2205]. 827 DiffServ support is widely available, while RSVP for IP packets is 828 generally not supported. 830 #9 Packet traceability (W) 832 The IPv4 has similar needs and requirements for traceability as 833 IPv6 (see Section 5.1.1.2). The IPv4 has a traceroute option 834 [RFC6814] that could be used to record the route the packet took. 835 However, the option has been deprecated [RFC6814]. 837 #10 Technical maturity (M/W) 839 IPv4 can be considered mature technology with over 30 years of 840 implementation, deployment and operations experience. As with 841 IPv6, today's commercial implementations and deployments of IPv4 842 generally lack any support for QoS. 844 5.1.2.3. Summary 846 The IPv4 has specifications to support most of the identified DetNet 847 data plane criteria today. However, several of those have already 848 been deprecated or their wide support is not guaranteed. The DetNet 849 data plane criteria that are not fully supported could be 850 incrementally added or supplemented by the underlying sub-network 851 layer. Unfortunately, the IPv4 has had limited success getting its 852 extensions deployed at large. However, introducing new extensions 853 might have a better success in closed networks (like DetNet) than in 854 Internet. Due to the popularity of the IPv4, it should be considered 855 as a potential choice for the DetNet Transport layer. 857 5.1.3. Multiprotocol Label Switching (MPLS) 859 Multiprotocol Label Switching Architecture (MPLS) [RFC3031] and its 860 variants, MPLS with Traffic Engineering (MPLS-TE) [RFC3209] and 861 [RFC3473], and MPLS Transport Profile (MPLS-TP) [RFC5921] is a widely 862 deployed technology that switches traffic based on MPLS label stacks 863 [RFC3032] and [RFC5960]. MPLS is the foundation for Pseudowire-based 864 services Section 5.2.3 and emerging technologies such as Bit-Indexed 865 Explicit Replication (BIER) Section 5.1.4 and Source Packet Routing 866 [3]. 868 MPLS supports the equivalent of both the DetNet Service and DetNet 869 Transport layers, and provides a very rich set of mechanisms that can 870 be reused directly, and perhaps augmented in certain cases, to 871 deliver DetNet services. At the DetNet Transport layer, MPLS 872 provides forwarding, protection and OAM services. At the DetNet 873 Service Layer it provides client service adaption, directly, via 874 Pseudowires Section 5.2.3 and via other label-like mechanisms such as 875 EPVN Section 5.2.4. A representation of these options are shown in 876 Figure 7. 878 PW-Based EVPN Labeled IP 879 Services Services Transport 880 |------------| |-----------------------------| |------------| 882 Emulated EVPN over LSP EVPN w/ ESI ID IP 883 Service 884 +------------+ 885 | Payload | 886 +------------+ +------------+ +------------+ (Service) 887 | PW Payload | | Payload | |ESI Lbl(S=1)| 888 +------------+ +------------+ +------------+ +------------+ 889 |PW Lbl(S=1) | |VPN Lbl(S=1)| |VPN Lbl(S=0)| | IP | 890 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 891 |LSP Lbl(S=0)| |LSP Lbl(S=0)| |LSP Lbl(S=0)| |LSP Lbl(S=1)| 892 +------------+ +------------+ +------------+ +------------+ 893 . . . . 894 . . . . (Transport) 895 . . . . 897 ~~~~~~~~~~~ denotes DetNet Service <-> DetNet Transport layer boundary 899 Figure 7: MPLS-based Services 901 MPLS can be controlled in a number of ways including via a control 902 plane, via the management plane, or via centralized controller (SDN) 903 based approaches. MPLS also provides standard control plane 904 reference points. Additional information on MPLS architecture and 905 control can be found in [RFC5921]. A summary of MPLS control plane 906 related functions can be found in [RFC6373]. The remainder of this 907 section will focus [RFC6373]. The remainder of this section will 908 focus on the MPLS transport data plane, additional information on the 909 MPLS service data plane can be found below in Section 5.2.2. 911 5.1.3.1. Solution description 913 The following draws heavily from [RFC5960]. 915 Encapsulation and forwarding of packets traversing MPLS LSPs follows 916 standard MPLS packet encapsulation and forwarding as defined in 917 [RFC3031], [RFC3032], [RFC5331], and [RFC5332]. 919 Data plane Quality of Service capabilities are included in the MPLS 920 in the form of Traffic Engineered (TE) LSPs [RFC3209] and the MPLS 921 Differentiated Services (DiffServ) architecture [RFC3270]. Both 922 E-LSP and L-LSP MPLS DiffServ modes are defined. The Traffic Class 923 field (formerly the EXP field) of an MPLS label follows the 924 definition of [RFC5462] and [RFC3270]. 926 Except for transient packet reordering that may occur, for example, 927 during fault conditions, packets are delivered in order on L-LSPs, 928 and on E-LSPs within a specific ordered aggregate. 930 The Uniform, Pipe, and Short Pipe DiffServ tunneling and TTL 931 processing models are described in [RFC3270] and [RFC3443] and may be 932 used for MPLS LSPs. 934 Equal-Cost Multi-Path (ECMP) load-balancing is possible with MPLS 935 LSPs and can be avoided using a number of techniques. The same holds 936 for Penultimate Hop Popping (PHP). 938 MPLS includes the following LSP types: 940 o Point-to-point unidirectional 941 o Point-to-point associated bidirectional 942 o Point-to-point co-routed bidirectional 943 o Point-to-multipoint unidirectional 945 Point-to-point unidirectional LSPs are supported by the basic MPLS 946 architecture [RFC3031]. 948 A point-to-point associated bidirectional LSP between LSRs A and B 949 consists of two unidirectional point-to-point LSPs, one from A to B 950 and the other from B to A, which are regarded as a pair providing a 951 single logical bidirectional transport path. 953 A point-to-point co-routed bidirectional LSP is a point-to-point 954 associated bidirectional LSP with the additional constraint that its 955 two unidirectional component LSPs in each direction follow the same 956 path (in terms of both nodes and links). An important property of 957 co-routed bidirectional LSPs is that their unidirectional component 958 LSPs share fate. 960 A point-to-multipoint unidirectional LSP functions in the same manner 961 in the data plane, with respect to basic label processing and packet- 962 switching operations, as a point-to-point unidirectional LSP, with 963 one difference: an LSR may have more than one (egress interface, 964 outgoing label) pair associated with the LSP, and any packet it 965 transmits on the LSP is transmitted out all associated egress 966 interfaces. Point-to-multipoint LSPs are described in [RFC4875] and 967 [RFC5332]. TTL processing and exception handling for point-to- 968 multipoint LSPs is the same as for point-to-point LSPs. 970 Additional data plane capabilities include Linear Protection, 971 [RFC6378] and [RFC7271]. And the in progress work on MPLS support 972 for time synchronization [I-D.ietf-mpls-residence-time]. 974 5.1.3.2. Analysis and Discussion 976 #1 Encapsulation and overhead (M) 978 There are two perspectives to consider when looking at 979 encapsulation. The first is encapsulation to support services. 980 These considerations are part of the DetNet service layer and are 981 covered below, see Sections 5.2.3 and 5.2.4. 983 The second perspective relates to encapsulation, if any, is needed 984 to transport packets across network. In this case, the MPLS label 985 stack, [RFC3032] is used to identify flows across a network. MPLS 986 labels are compact and highly flexible. They can be stacked to 987 support client adaptation, protection, network layering, source 988 routing, etc. 990 The number of DetNet Transport layer specific labels is flexible 991 and support a wide range of applicable functions and MPLS domain 992 characteristics (e.g., TE-tunnels, Hierarchical-LSPs, etc.). 994 #2 Flow identification (M) 995 MPLS label stacks provide highly flexible ways to identify flows. 996 Basically, they enable the complete separation of traffic 997 classification from traffic treatment and thereby enable arbitrary 998 combinations of both. 1000 For the DetNet flow identification the MPLS label stack can be 1001 used to support n-layers of DetNet flow identification. For 1002 example, using dedicated LSP per DetNet flow would simplify flow 1003 identification for intermediate transport nodes, and additional 1004 hierarchical LSPs could be used to facilitate scaling. 1006 #4 Explicit routes (M) 1008 MPLS supports explicit routes based on how LSPs are established, 1009 e.g., via TE explicit routes [RFC3209]. Additional, but not 1010 required, capabilities are being defined as part of Segment 1011 Routing (SR) [I-D.ietf-spring-segment-routing]. 1013 #5 Flow duplication and merging (M/W) 1015 MPLS as DetNet Transport layer supports the replication via point- 1016 to-multipoint LSPs. At the MPLS LSP level, there are mechanisms 1017 defined to provide 1+1 protection, which could help realizing the 1018 flow merging function. The current definitions [RFC6378] and 1019 [RFC7271] use OAM mechanisms to support and coordinate protection 1020 switching and packet loss is possible during a switch. While such 1021 this level of protection may be sufficient for many DetNet 1022 applications, when truly hitless (i.e., zero loss) switching is 1023 required, additional mechanisms will be needed. It is expected 1024 that these additional mechanisms will be defined at a DetNet 1025 layer. 1027 #6 Operations, Administration and Maintenance (M) 1029 MPLS already includes a rich set of OAM functions at both the 1030 Service and Transport Layers. This includes LSP ping [ref] and 1031 those enabled via the MPLS Generic Associated Channel [RFC5586] 1032 and registered by IANA [4]. 1034 #8 Class and quality of service capabilities (M/W) 1036 As previously mentioned, Data plane Quality of Service 1037 capabilities are included in the MPLS in the form of Traffic 1038 Engineered (TE) LSPs [RFC3209] and the MPLS Differentiated 1039 Services (DiffServ) architecture [RFC3270]. Both E-LSP and L-LSP 1040 MPLS DiffServ modes are defined. The Traffic Class field 1041 (formerly the EXP field) of an MPLS label follows the definition 1042 of [RFC5462] and [RFC3270]. One potential open area of work is 1043 synchronized, time based scheduling. Another is shaping, which is 1044 generally not supported in shipping MPLS hardware. 1046 #9 Packet traceability (M) 1048 MPLS supports multiple tracing mechanisms. A control based one is 1049 defined in [RFC3209]. An OAM based mechanism is defined in MPLS 1050 On-Demand Connectivity Verification and Route Tracing [RFC6426]. 1052 #10 Technical maturity (M) 1054 MPLS as a mature technology that has been widely deployed in many 1055 networks for many years. Numerous vendor products and multiple 1056 generations of MPLS hardware have been built and deployed. 1058 5.1.3.3. Summary 1060 MPLS is a mature technology that has been widely deployed. Numerous 1061 vendor products and multiple generations of MPLS hardware have been 1062 built and deployed. MPLS LSPs support a significant portion of the 1063 identified DetNet data plane criteria today. Aspects of the DetNet 1064 data plane that are not fully supported can be incrementally added. 1065 It's worth noting that a number of limitations are in shipping 1066 hardware, versus at the protocol specification level, e.g., shaping. 1068 5.1.4. Bit Indexed Explicit Replication (BIER) 1070 Bit Indexed Explicit Replication [I-D.ietf-bier-architecture] (BIER) 1071 is a network plane replication technique that was initially intended 1072 as a new method for multicast distribution. In a nutshell, a BIER 1073 header includes a bitmap that explicitly signals the destinations 1074 that are intended for a particular packet, which means that 1) the 1075 source is aware of the individual destinations and 2) the BIER 1076 control plane is a simple extension of the unicast routing as opposed 1077 to a dedicated multicast data plane, which represents a considerable 1078 reduction in OPEX. For this reason, the technology faces a lot of 1079 traction from Service Providers. Section 5.1.4 discusses the 1080 applicability of BIER for replication in the DetNet. 1082 The simplicity of the BIER technology makes it very versatile as a 1083 network plane signaling protocol. Already, a new Traffic Engineering 1084 variation is emerging that uses bits to signal segments along a TE 1085 path. While the more classical BIER is mainly a multicast technology 1086 that typically leverages a unicast distributed control plane through 1087 IGP extensions, BIER-TE is mainly a unicast technology that leverages 1088 a central computation to setup path, compute segments and install the 1089 mapping in the intermediate nodes. Section 5.1.5 discusses the 1090 applicability of BIER-TE for replication, traceability and OAM 1091 operations in DetNet. 1093 Bit-Indexed Explicit Replication (BIER) layer may be considered to be 1094 included into Deterministic Networking data plane solution. 1095 Encapsulation of a BIER packet in MPLS network presented in Figure 8 1097 0 1 2 3 1098 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1099 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1100 | Label Stack Element | 1101 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1102 | Label Stack Element | 1103 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1104 | BIER-MPLS label | |1| | 1105 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1106 |0 1 0 1| Ver | Len | Entropy | 1107 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1108 | BitString (first 32 bits) ~ 1109 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1110 ~ ~ 1111 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1112 ~ BitString (last 32 bits) | 1113 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1114 |OAM| Reserved | Proto | BFIR-id | 1115 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1117 Figure 8: BIER packet in MPLS encapsulation 1119 5.1.4.1. Solution description 1121 The DetNet may be presented in BIER as distinctive payload type with 1122 its own Proto(col) ID. Then it is likely that DetNet will have the 1123 header that would identify: 1125 o Version; 1126 o Sequence Number; 1127 o Timestamp; 1128 o Payload type, e.g. data vs. OAM. 1130 DetNet node, collocated with BFIR, may use multiple BIER sub-domains 1131 to create replicated flows. Downstream DetNet nodes, collocated with 1132 BFER, would terminate redundant flows based on Sequence Number and/or 1133 Timestamp information. Such DetNet may be BFER in one BIER sub- 1134 domain and BFIR in another. Thus DetNet flow would traverse several 1135 BIER sub-domains. 1137 +-----+ 1138 | A | 1139 +-----+ 1140 / \ 1141 . . 1142 / . 1143 . \ 1144 / . 1145 . . 1146 / \ 1147 +-----+ +-----+ 1148 | B | | C | 1149 +-----+ +-----+ 1150 / \ / \ 1151 . . . . 1152 / \ . . 1153 . . / \ 1154 / \ . . 1155 . . . . 1156 / \ / \ 1157 +-----+ +-----+ +-----+ 1158 | D | | E | | F | 1159 +-----+ +-----+ +-----+ 1160 \ . . / 1161 . . . . 1162 \ . . . 1163 . . . / 1164 \ . . . 1165 . . . . 1166 \ . . / 1167 +-----+ +-----+ 1168 | G | | H | 1169 +-----+ +-----+ 1171 Figure 9: DetNet in BIER domain 1173 Consider DetNet flow that must traverse BIER enabled domain from A to 1174 G and H. DetNet may use three BIER subdomains: 1176 o A-B-D-E-G (dash-dot): A is BFIR, E and G are BFERs, 1177 o A-C-E-F-H (dash-double-dot): A is BFIR, E and H are BFERs, 1178 o E-G-H (dotted): E is BFIR, G and H are BFERs. 1180 DetNet node A sends DetNet into red and purple BIER sub-domains. 1181 DetNet node E receives DetNet packet and sends into green sub-domain 1182 while terminating duplicates and those that deemed too-late. 1184 DetNet nodes G and H receive DetNet flows, terminate duplicates and 1185 those that are too-late. 1187 5.1.4.2. Analysis and Discussion 1189 #1 Encapsulation and overhead (M) 1191 BIER over MPLS network encapsulation (will refer as "BIER over 1192 MPLS" further for short), Figure 8, is being defined [I-D. ietf- 1193 bier-mpls-encapsulation] within the BIER working group. 1195 #2 Flow identification (M) 1197 Flow identification and separation can be achieved through use of 1198 BIER domains and/or Entropy value in the BIER over MPLS, Figure 8. 1200 #4 Explicit routes (M) 1202 Explicit routes may be used as underlay for BIER domain. BIER 1203 underlay may be calculated using PCE and instantiated using any 1204 southbound mechanism. 1206 #5 Flow duplication and merging (M/W) 1208 Packet replication, as indicated by its name, is core function of 1209 the Bit-Indexed Explicit Replication. Elimination of the 1210 duplicates and/or too-late packets cannot be done within BIER sub- 1211 domain but may be done at DetNet overlay at the edge of the BIER 1212 sub-domain. 1214 [Editor's note: how about the flow merging?] 1216 #6 Operations, Administration and Maintenance (M/W) 1218 BIER over MPLS guarantees that OAM is fate-sharing, i.e. in-band 1219 with a data flow being monitored or measured. Additionally, BIER 1220 over MPLS enables passive performance measurement, e.g. with the 1221 marking method [I-D.mirsky-bier-pmmm-oam]. Some OAM protocols, 1222 e.g. can be applied and used in BIER over MPLS as demonstrated 1223 [I-D.ooamdt-rtgwg-oam-gap-analysis], while new protocols being 1224 worked on, e.g. ping/traceroute [I-D.kumarzheng-bier-ping] or Path 1225 MTU Discovery [I-D.mirsky-bier-path-mtu-discovery]. 1227 #8 Class and quality of service capabilities (M/W) 1228 Class of Service can be inherited from the underlay of the 1229 particular BIER sub-domain. Quality of Service, i.e. scheduling 1230 and bandwidth reservations can be used among other constrains in 1231 calculating explicit path for the BIER sub-domain's underlay. 1233 #9 Packet traceability (W) 1235 Ability to do passive performance measurement by using OAM field 1236 of the BIER over MPLS, Figure 8, is unmatched and significantly 1237 simplifies truly passive tracing of selected flows and packets 1238 within them. 1240 #10 Technical maturity (W) 1242 The BIER over MPLS is nearing finalization within the BIER WG and 1243 several experimental implementations are expected soon. 1245 5.1.4.3. Summary 1247 BIER over MPLS supports a significant portion of the identified 1248 DetNet data plane requirements, including controlled packet 1249 replication, traffic engineering, while some requirements, e.g. 1250 duplicate and too-late packet elimination may be realized as function 1251 of the DetNet overlay. BIER over MPLS is a viable candidate as the 1252 DetNet Transport layer in MPLS networks. 1254 5.1.5. BIER - Traffic Engineering (BIER-TE) 1256 An alternate use of Bit-Indexed Explicit Replication (BIER) uses bits 1257 in the BitString to represent adjacencies as opposed to destinations, 1258 as discussed in BIER Traffic Engineering (TE) 1259 [I-D.eckert-bier-te-arch]. 1261 The proposed function of BIER-TE in the DetNet data plane is to 1262 control the process of replication and elimination, as opposed to the 1263 identification of the flows or and the sequencing of packets within a 1264 flow. 1266 At the path ingress, BIER-TE identifies the adjacencies that are 1267 activated for this packet (under the rule of the controller). At the 1268 egress, BIER-TE is used to identify the adjacencies where 1269 transmission failed. This information is passed to the controller, 1270 which in turn can modify the active adjacencies for the next packets. 1272 The value is that the replication can be controlled and monitored in 1273 a loop that may involve an external controller, with the granularity 1274 of a packet and an adjacency . 1276 5.1.5.1. Solution description 1278 BIER-TE enables to activate the replication and elimination functions 1279 in a manner that is abstract to the data plane forwarding 1280 information. An adjacency, which is represented by a bit in the BIER 1281 header, can correspond in the data plane to an Ethernet hop, a Label 1282 Switched Path, or it can correspond to an IPv6 loose or strict source 1283 routed path. 1285 In a nutshell, BIER-TE is used as follows: 1287 o A controller computes a complex path, sometimes called a track, 1288 which takes the general form of a ladder. The steps and the side 1289 rails between them are the adjacencies that can be activated on 1290 demand on a per-packet basis using bits in the BIER header. 1292 ===> (A) ====> (C) ==== 1293 // ^ | ^ | \\ 1294 ingress (I) | | | | (E) egress 1295 \\ | v | v // 1296 ===> (B) ====> (D) ==== 1298 Figure 10: Ladder Shape with replication and elimination Points 1300 o The controller assigns a BIER domain, and inside that domain, 1301 assigns bits to the adjacencies. The controller assigns each bit 1302 to a replication node that sends towards the adjacency, for 1303 instance the ingress router into a segment that will insert a 1304 routing header in the packet. A single bit may be used for a step 1305 in the ladder, indicating the other end of the step in both 1306 directions. 1308 ===> (A) ====> (C) ==== 1309 // 1 ^ | 4 ^ | 7 \\ 1310 ingress (I) |2| |6| (E) egress 1311 \\ 3 | v 5 | v 8 // 1312 ===> (B) ====> (D) ==== 1314 Figure 11: Assigning Bits 1316 o The controller activates the replication by deciding the setting 1317 of the bits associated with the adjacencies. This decision can be 1318 modified at any time, but takes the latency of a controller round 1319 trip to effectively take place. Below is an example that uses 1320 replication and elimination to protect the A->C adjacency. 1322 +-------+-----------+-------+---------------------+ 1323 | Bit # | Adjacency | Owner | Example Bit Setting | 1324 +-------+-----------+-------+---------------------+ 1325 | 1 | I->A | I | 1 | 1326 | 2 | A->B | A | 1 | 1327 | | B->A | B | | 1328 | 3 | I->C | I | 0 | 1329 | 4 | A->C | A | 1 | 1330 | 5 | B->D | B | 1 | 1331 | 6 | C->D | C | 1 | 1332 | | D->C | D | | 1333 | 7 | C->E | C | 1 | 1334 | 8 | D->E | D | 0 | 1335 +-------+-----------+-------+---------------------+ 1337 replication and elimination Protecting A->C 1339 Table 1: Controlling Replication 1341 o The BIER header with the controlling BitString is injected in the 1342 packet by the ingress node of the deterministic path. That node 1343 may act as a replication point, in which case it may issue 1344 multiple copies of the packet 1346 ====> Repl ===> Elim ==== 1347 // | ^ \\ 1348 ingress | | egress 1349 v | 1350 Fwd ====> Fwd 1352 Figure 12: Enabled Adjacencies 1354 o For each of its bits that is set in the BIER header, the owner 1355 replication point resets the bit and transmits towards the 1356 associated adjacency; to achieve this, the replication point 1357 copies the packet and inserts the relevant data plane information, 1358 such as a source route header, towards the adjacency that 1359 corresponds to the bit 1360 +-----------+----------------+ 1361 | Adjacency | BIER BitString | 1362 +-----------+----------------+ 1363 | I->A | 01011110 | 1364 | A->B | 00011110 | 1365 | B->D | 00010110 | 1366 | D->C | 00010010 | 1367 | A->C | 01001110 | 1368 +-----------+----------------+ 1370 BitString in BIER Header as Packet Progresses 1372 Table 2: BIER-TE in Action 1374 o Adversely, an elimination node on the way strips the data plane 1375 information and performs a bitwise AND on the BitStrings from the 1376 various copies of the packet that it has received, before it 1377 forwards the packet with the resulting BitString. 1379 +-----------+----------------+ 1380 | Operation | BIER BitString | 1381 +-----------+----------------+ 1382 | D->C | 00010010 | 1383 | A->C | 01001110 | 1384 | | -------- | 1385 | AND in C | 00000010 | 1386 | | | 1387 | C->E | 00000000 | 1388 +-----------+----------------+ 1390 BitString Processing at Elimination Point C 1392 Table 3: BIER-TE in Action (cont.) 1394 o In this example, all the transmissions succeeded and the BitString 1395 at arrival has all the bits reset - note that the egress may be an 1396 Elimination Point in which case this is evaluated after this node 1397 has performed its AND operation on the received BitStrings). 1399 +-------------------+-----------------------+ 1400 | Failing Adjacency | Egress BIER BitString | 1401 +-------------------+-----------------------+ 1402 | I->A | Frame Lost | 1403 | I->B | Not Tried | 1404 | A->C | 00010000 | 1405 | A->B | 01001100 | 1406 | B->D | 01001100 | 1407 | D->C | 01001100 | 1408 | C->E | Frame Lost | 1409 | D->E | Not Tried | 1410 +-------------------+-----------------------+ 1412 BitString indicating failures 1414 Table 4: BIER-TE in Action (cont.) 1416 o But if a transmission failed along the way, one (or more) bit is 1417 never cleared. Table 4 provides the possible outcomes of a 1418 transmission. If the frame is lost, then it is probably due to a 1419 failure in either I->A or C->E, and the controller should enable 1420 I->B and D->E to find out. A BitString of 00010000 indicates 1421 unequivocally a transmission error on the A->C adjacency, and a 1422 BitString of 01001100 indicates a loss in either A->B, B->D or 1423 D->C; enabling D->E on the next packets may provide more 1424 information to sort things out. 1426 In more details: 1428 The BIER header is of variable size, and a DetNet network of a 1429 limited size can use a model with 64 bits if 64 adjacencies are 1430 enough, whereas a larger deployment may be able to signal up to 256 1431 adjacencies for use in very complex paths. Figure 8 illustrates a 1432 BIER header as encapsulated within MPLS. The format of this header 1433 is common to BIER and BIER-TE. 1435 For the DetNet data plane, a replication point is an ingress point 1436 for more than one adjacency, and an elimination point is an egress 1437 point for more than one adjacency. 1439 A pre-populated state in a replication node indicates which bits are 1440 served by this node and to which adjacency each of these bits 1441 corresponds. With DetNet, the state is typically installed by a 1442 controller entity such as a PCE. The way the adjacency is signaled 1443 in the packet is fully abstracted in the bit representation and must 1444 be provisioned to the replication nodes and maintained as a local 1445 state, together with the timing or shaping information for the 1446 associated flow. 1448 The DetNet data plane uses BIER-TE to control which adjacencies are 1449 used for a given packet. This is signaled from the path ingress, 1450 which sets the appropriate bits in the BIER BitString to indicate 1451 which replication must happen. 1453 The replication point clears the bit associated to the adjacency 1454 where the replica is placed, and the elimination points perform a 1455 logical AND of the BitStrings of the copies that it gets before 1456 forwarding. 1458 As is apparent in the examples above, clearing the bits enables to 1459 trace a packet to the replication points that made any particular 1460 copy. BIER-TE also enables to detect the failing adjacencies or 1461 sequences of adjacencies along a path and to activate additional 1462 replications to counter balance the failures. 1464 Finally, using the same BIER-TE bit for both directions of the steps 1465 of the ladder enables to avoid replication in both directions along 1466 the crossing adjacencies. At the time of sending along the step of 1467 the ladder, the bit may have been already reset by performing the AND 1468 operation with the copy from the other side, in which case the 1469 transmission is not needed and does not occur (since the control bit 1470 is now off). 1472 5.1.5.2. Analysis and Discussion 1474 #1 Encapsulation and overhead (W/M) 1476 The size of the BIER header depends on the number of segments in 1477 the particular path. It is very concise considering the amount of 1478 information that is carried (control of replication, traceability, 1479 and measurement of the reliability of the segments). 1481 #2 Flow identification (N) 1483 Some fields in the BIER header could be used to identify the flows 1484 but they are not the primary purpose, so it's probably not a good 1485 idea. 1487 #4 Explicit routes (N) 1489 A separate procedure must be used to set up the paths and allocate 1490 the bits for the adjacencies. The bits should be distributed as a 1491 form of tag by the route setup protocol. This procedure requires 1492 more work and is separate from the data plane method that is 1493 described here. 1495 #5 Flow duplication and merging M/W) 1497 The bitmap expresses in a very concise fashion which replication 1498 and merging (and elimination) should take place for a given 1499 packet. It also enables to control that process on a per packet 1500 basis, depending on the loss that it enables to measure. The net 1501 result is that a complex path may be installed with all the 1502 possibilities and that the decision of which possibilities are 1503 used is controlled in the data plane. 1505 #6 Operations, Administration and Maintenance (W) 1507 The setting of the bits at arrival enables to determine which 1508 adjacencies worked and which did not, enabling a dynamic control 1509 of the replication and elimination process. This is a form of OAM 1510 that is in-band with the data stream as opposed to leveraging 1511 separate packets, which is a more accurate information on the 1512 reliability of the link for the user. 1514 #8 Class and quality of service capabilities (N) 1516 BIER-TE does not signal that explicitly. 1518 #9 Packet traceability (W) 1520 This is a strong point of the solution. The solution enables to 1521 determine which is the current segment that a given packet is 1522 expected to traverse, which node performed the replication and 1523 which should perform the elimination if any 1525 #10 Technical maturity (W) 1527 Some components of the technology are more mature, e.g. segment 1528 routing and BIER. Yet, the overall solution has never been 1529 deployed as is not fully defined. It should be noted that the 1530 definition of the BIER-TE solution is outside the scope of the 1531 DetNet WG charter. 1533 5.1.5.3. Summary 1535 BIER-TE occupies a particular position in the DetNet data plane. In 1536 the one hand it is optional, and only useful if replication and 1537 elimination is taking place. In the other hand, it has unique 1538 capabilities to: 1540 o control which replication take place on a per packet basis, so 1541 that replication points can be configured but not actually 1542 utilized 1543 o trace the replication activity and determine which node replicated 1544 a particular packet 1545 o measure the quality of transmission of the actual data packet 1546 along the replication segments and use that in a control loop to 1547 adapt the setting of the bits and maintain the reliability. 1549 However, as noted earlier, BIER-TE is not yet fully specified and the 1550 required specification work is outside the scope of the current 1551 DetNet WG charter. 1553 5.2. DetNet Service layer technologies 1555 5.2.1. Generic Routing Encapsulation (GRE) 1557 5.2.1.1. Solution description 1559 Generic Routing Encapsulation (GRE) [RFC2784] provides an 1560 encapsulation of an arbitrary network layer protocol over another 1561 arbitrary network layer protocol. The encapsulation of a GRE packet 1562 can be found in Figure 13. 1564 +-------------------------------+ 1565 | Delivery Header | 1566 +-------------------------------+ 1567 | GRE Header | 1568 +-------------------------------+ 1569 | Payload packet | 1570 +-------------------------------+ 1572 Figure 13: Encapsulation of a GRE packet 1574 Based on RFC2784, [RFC2890] further includes sequencing number and 1575 Key in optional fields of the GRE header, which may help to transport 1576 DetNet traffic flows over IP networks. The format of a GRE header is 1577 presented in Figure 14. 1579 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 1580 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1581 |C| |K|S| Reserved0 | Ver | Protocol Type | 1582 +-----------------------------------------------------------------+ 1583 | Checksum (optional) | Reserved1 (Optional) | 1584 +-----------------------------------------------------------------+ 1585 | Key (optional) | 1586 +-----------------------------------------------------------------+ 1587 | Sequence Number (optional) | 1588 +-----------------------------------------------------------------+ 1590 Figure 14: Format of a GRE header 1592 5.2.1.2. Analysis and Discussion 1594 #1 Encapsulation and overhead (M) 1596 GRE can provide encapsulation at the service layer over the 1597 transport layer. A new protocol type for DetNet traffic should be 1598 allocated as an "Ether Type" in [RFC3232] and in IANA Ethernet 1599 Numbers [5]. The fixed header of a GRE packet is 4 octets while 1600 the maximum header is 16 octets with optional fields in Figure 14. 1602 #2 Flow identification (W) 1604 There is no flow identification field in GRE header. However, it 1605 can rely on the flow identification mechanism applied in the 1606 delivery protocols, such as flow identification stated in IP 1607 Sections 5.1.1 and 5.1.2 when the delivery protocols are IPv6 and 1608 IPv4 respectively. Alternatively, the Key field can also be 1609 extended to carry the flow identification. The size of Key field 1610 is 4 octets. 1612 #3 Packet sequencing and duplicate elimination (M/W) 1614 As stated in Section 5.2.1, GRE provides an optional sequencing 1615 number in its header to provide sequencing services for packets. 1616 The size of the sequencing number is 32 bits. The GRE header 1617 could be extended to indicate the duplicated packets by defining a 1618 flag in reserved fields or using the sequencing number of a flow. 1620 #5 Flow duplication and merging (W/N) 1622 GRE has no flow/packet replication and merging support in its 1623 header. It can use the transport IPv4/IPv6 protocols at the 1624 transport layer to replicate the packets and take the different 1625 routes as discussed in Section 5.1.1 and Section 5.1.2. 1627 #6 Operations, Administration and Maintenance (M) 1629 GRE uses the network management provided by the IP protocols as 1630 transport layer. 1632 #8 Class and quality of service capabilities (W) 1634 For the class of service capability, an optional code point field 1635 to indicate CoS of the traffic could be added into the GRE header. 1636 Otherwise, GRE can reuse the class and quality of service of 1637 delivery protocols at transport layer such as IPv6 and IPv4 stated 1638 in Section 5.1.1 and Section 5.1.2. 1640 #10 Technical maturity (M) 1642 GRE has been developed over 20 years. The delivery protocol 1643 mostly used is IPv4, while the IPv6 support for GRE is to be 1644 standardized now in IETF as [RFC7676]. Due to its good 1645 extensibility, GRE has also been extended to support network 1646 virtualization in Data Center, which is NVGRE [RFC7637]. 1648 5.2.1.3. Summary 1650 As a tunneling protocol, GRE can encapsulate a wide variety of 1651 network layer protocols over another network layer, which can 1652 naturally serve as the service layer protocol for DetNet. Currently, 1653 it supports a portion of the Detnet service layer criteria, and still 1654 some are not fully supported but can be incrementally added or 1655 supported by delivery protocols at as the transport layer. In 1656 general, GRE can be a choice as the DetNet service layer and can work 1657 with IPv6 and IPv4 as the DetNet Transport layer. 1659 5.2.2. MPLS-based Services for DetNet 1661 MPLS based technologies supports both the DetNet Service and DetNet 1662 Transport layers. This, as well as a general overview of MPLS, is 1663 covered above in Section 5.1.3. These sections focus on the DetNet 1664 Service Layer it provides client service adaption, via Pseudowires 1665 Section 5.2.3 and via native and other label-like mechanisms such as 1666 EPVN in Section 5.2.4. A representation of these options was 1667 previously discussed and is shown in Figure 7. 1669 The following text is adapted from [RFC5921]: 1671 The MPLS native service adaptation functions interface the client 1672 layer network service to MPLS. For Pseudowires, these adaptation 1673 functions are the payload encapsulation described in Section 4.4 1674 of [RFC3985] and Section 6 of [RFC5659]. For network layer client 1675 services, the adaptation function uses the MPLS encapsulation 1676 format as defined in [RFC3032]. 1678 The purpose of this encapsulation is to abstract the data plane of 1679 the client layer network from the MPLS data plane, thus 1680 contributing to the independent operation of the MPLS network. 1682 MPLS may itself be a client of an underlying server layer. MPLS 1683 can thus also bounded by a set of adaptation functions to this 1684 server layer network, which may itself be MPLS. These adaptation 1685 functions provide encapsulation of the MPLS frames and for the 1686 transparent transport of those frames over the server layer 1687 network. 1689 While MPLS service can provided on and true end-system to end- 1690 system basis, it's more likely that DetNet service will be 1691 provided over Pseudowires as described in Section 5.2.3 or via an 1692 EPVN-based service described in Section 5.2.4 . 1694 MPLS labels in the label stack may be used to identify transport 1695 paths, see Section 5.1.3, or as service identifiers. Typically a 1696 single label is used for service identification. 1698 Packet sequencing mechanisms are added in client-related 1699 adaptation processing, see Sections 5.2.3 and 5.2.4. 1701 The MPLS client inherits its Quality of Service (QoS) from the 1702 MPLS transport layer, which in turn inherits its QoS from the 1703 server (sub-network) layer. The server layer therefore needs to 1704 provide the necessary QoS to ensure that the MPLS client QoS 1705 commitments can be satisfied. 1707 5.2.3. Pseudo Wire Emulation Edge-to-Edge (PWE3) 1709 5.2.3.1. Solution description 1711 Pseudo Wire Emulation Edge-to-Edge (PWE3) [RFC3985] or simply 1712 PseudoWires (PW) provide means of emulating the essential attributes 1713 and behaviour of a telecommunications service over a packet switched 1714 network (PSN) using IP or MPLS transport. In addition to traditional 1715 telecommunications services such as T1 line or Frame Relay, PWs also 1716 provide transport for Ethernet service [RFC4448] and for generic 1717 packet service [RFC6658]. Figure 15 illustrate the reference PWE3 1718 stack model. 1720 +----------------+ +----------------+ 1721 |Emulated Service| |Emulated Service| 1722 |(e.g., Eth, ...)|<= Emulated Service =>|(e.g., Eth, ...)| 1723 +----------------+ +----------------+ 1724 | Payload | | Payload | CW, 1725 | Encapsulation |<=== Pseudo Wire ====>| Encapsulation | Timing, 1726 | | | | Seq., .. 1727 +----------------+ +----------------+ 1728 |PW Demultiplexer| |PW Demultiplexer| 1729 | PSN Tunnel, |<==== PSN Tunnel ====>| PSN Tunnel, | MPLS. 1730 | PSN & Physical | | PSN & Physical | L2TP, 1731 | Layers | | Layers | IP, .. 1732 +-------+--------+ ___________ +---------+------+ 1733 | / \ | 1734 +============/ PSN \==============+ 1735 \ / 1736 \___________/ 1738 Figure 15: PWE3 protocol stack reference model 1740 PWs appear as a good data plane solution alternative for a number of 1741 reasons. PWs are a proven and deployed technology with a rich OAM 1742 control plane [RFC4447], and enjoy the toolbox developed for MPLS 1743 networks in a case of MPLS-based PSN. Furthermore, PWs may have an 1744 optional Control Word (CW) as part of the payload encapsulation 1745 between the PSN and the emulated service that is, for example, 1746 capable of frame sequencing and duplicate detection. The 1747 encapsulation layer may also provide timing using RTP as described in 1748 Sections 5.2.2, 5.4.1 and 5.4.2 of [RFC3985] and utilized by 1749 [RFC4553][RFC5087]. Furthermore, advanced DetNet node functions are 1750 conceptually already supported by PW framework (with some added 1751 functional required), such as the DetNet Relay node modeled after the 1752 Multi-Segment PWE3 [RFC5254]. 1754 PWs can be also used if the PSN is IP, which enables the application 1755 of PWs in networks that do not have MPLS enabled in their core 1756 routers. One approach to provide PWs over IP is to provide MPLS over 1757 IP in some way and then leverage what is available for PWs over MPLS. 1758 The following standard solutions are available both for IPv4 and IPv6 1759 to follow this approach. The different solutions have different 1760 overhead as discussed in the following subsection. The MPLS-in-IP 1761 encapsulation is specified by [RFC4023]. The IPv4 Protocol Number 1762 field or the IPv6 Next Header field is set to 137, which indicates an 1763 MPLS unicast packet. (The use of the MPLS-in-IP encapsulation for 1764 MPLS multicast packets is not supported.) The MPLS-in-GRE 1765 encapsulation is specified in [RFC4023], where the IP header (either 1766 IPv4 or IPv6) is followed by a GRE header, which is followed by an 1767 MPLS label stack. The protocol type field in the GRE header is set 1768 to MPLS Unicast (0x8847) or Multicast (0x8848). MPLS over L2TPv3 1769 over IP encapsulation is specified by [RFC4817]. The MPLS-in-UDP 1770 encapsulation is specified by [RFC7510], where the UDP Destination 1771 Port indicates tunneled MPLS packet and the UDP Source Port is an 1772 entropy value that is generated by the encapsulator to uniquely 1773 identify a flow. MPLS-in-UDP encapsulation can be applied to enable 1774 UDP-based ECMP (Equal-Cost Multipath) or Link Aggregation. All these 1775 solutions can be secured with IPsec [RFC4303]. 1777 5.2.3.2. Analysis and Discussion 1779 #1 Encapsulation and overhead (M) 1781 PWs offer encapsulation services practically for any types of 1782 payloads over any PSN. New PW types need a code point allocation 1783 [RFC4446] and in some cases an emulated service specific document. 1785 Specifically in the case of the MPLS PSN the PW encapsulation 1786 overhead is minimal. Typically minimum two labels and a CW is 1787 needed, which totals to 12 octets. PW type specific handling 1788 might, however, allow optimizations on the emulated service in the 1789 provider edge (PE) device's native service processing (NSP) / 1790 forwarder function. These optimizations could be used, for 1791 example, to reduce header overhead. Ethernet PWs already have 1792 rather low overhead [RFC4448]. Without a CW and VLAN tags the 1793 Ethernet header gets reduced to 14 octets (minimum Ethernet header 1794 overhead is 26). 1796 The overhead is somewhat bigger in case of IP PSN if an MPLS over 1797 IP solution is applied to provide PWs. IP adds at least 20 (IPv4) 1798 or 40 (IPv6) bytes overhead to the PW over MPLS overhead; 1799 furthermore, the GRE, L2TPv3, or UDP header has to be taken into 1800 account if any of these further encapsulations is used. 1802 #2 Flow identification (M) 1804 PWs provide multiple layers of flow identification, especially in 1805 the case of the MPLS PSN. The PWs are typically prepended with an 1806 endpoint specific PW label that can be used to identify a specific 1807 PW per endpoint. Furthermore, the MPLS PSN also uses one or more 1808 labels to transport packets over a specific label switched paths 1809 (that then would carry PWs). So, a DetNet flow can be identified 1810 in this example by the service and transport layer labels. IP 1811 (and other) PSNs may need other mechanisms, such as, UDP port 1812 numbers, upper layer protocol header (like RTP) or some IP 1813 extension header to provide required flow identification. 1815 #3 Packet sequencing and duplicate elimination (M) 1817 As mentioned earlier PWs may contain an optional CW that is able 1818 to provide sequencing services. The size of the sequence number 1819 in the generic CW is 16 bits, which might be, depending on the 1820 used link and DetNet flow speed be too little. The PW duplicate 1821 detection mechanism is already conceptually specified [RFC3985] 1822 but no emulated service makes use of it currently. 1824 #5 Flow duplication and merging (W) 1826 PWs could use a (extended) version of existing transport layer 1827 provided protection mechanisms (e.g., hitless 1+1 protection) for 1828 both flow duplication and flow merging. The service layer has to 1829 provide the functionality to map DetNet flows into appropriate 1830 transport leyer connection, though. 1832 #6 Operations, Administration and Maintenance (M/W) 1834 PWs have rich control plane for OAM and in a case of the MPLS PSN 1835 enjoy the full control plane toolbox developed for MPLS network 1836 OAM likewise IP PSN have the full toolbox of IP network OAM tools. 1837 There could be, however, need for deterministic networking 1838 specific extensions for the mentioned control planes. 1840 #8 Class and quality of service capabilities (M/W) 1842 In a case of IP PSN the 6-bit differentiated services code point 1843 (DSCP) field can be used for indicating the class of service 1844 [RFC2474] and 2-bit field reserved for the explicit congestion 1845 notification (ECN) [RFC3168]. Similarly, in a case of MPLS PSN, 1846 there are 3-bit traffic class field (TC) [RFC5462] in the label 1847 reserved for for both Explicitly TC-encoded-PSC LSPs (E-LSP) 1848 [RFC3270] and ECN [RFC5129]. Due to the limited number of bits in 1849 the TC field, their use for QoS and ECN functions restricted and 1850 intended to be flexible. Although the QoS/CoS mechanism is 1851 already in place some clarifications may be required in the 1852 context of deterministic networking flows, for example, if some 1853 specific mapping between bit fields have to be done. 1855 When PWs are used over MPLS, MPLS LSPs can be used to provide both 1856 CoS (E-LSPs and L-LSPs) and QoS (dedicated TE LSPS). 1858 #10 Technical maturity (M) 1860 PWs, IP and MPLS are proven technologies with wide variety of 1861 deployments and years of operational experience. Furthermore, the 1862 estimated work for missing functionality (packet replication and 1863 elimination) does not appear to be extensive, since the existing 1864 protection mechanism already get close to what is needed from the 1865 deterministic networking data plane solution. 1867 5.2.3.3. Summary 1869 PseudoWires appear to be a strong candidate as the deterministic 1870 networking data plane solution alternative for the DetNet Service 1871 layer. The strong points are the technical maturity and the 1872 extensive control plane for OAM. This holds specifically for MPLS- 1873 based PSN. 1875 Extensions are required to realize the packet replication and 1876 duplicate detection features of the deterministic networking data 1877 plane. 1879 5.2.4. MPLS-Based Ethernet VPN (EVPN) 1881 5.2.4.1. Solution description 1883 MPLS-Based Ethernet VPN (EVPN), in the form documented in [RFC7432] 1884 and [RFC7209], is an increasingly popular approach to delivering 1885 MPLS-based Ethernet services and is designed to be the successor to 1886 Virtual Private LAN Service (VPLS), [RFC4664]. 1888 EVPN provides client adaptation and reuses the MPLS data plane 1889 discussed above in Section 5.2.2. While not required, the PW Control 1890 Word is also used. EVPN control is via BGP, [RFC7432], and may use 1891 TE-LSPs, e.g., controlled via [RFC3209] for MPLS transport. 1892 Additional EVPN related RFCs and in progress drafts are being 1893 developed by the BGP Enabled Services Working Group [6]. 1895 5.2.4.2. Analysis and Discussion 1897 #1 Encapsulation and overhead (M) 1899 EVPN generally uses a single MPLS label stack entry to support its 1900 client adaptation service. The optional addition of a second 1901 label is also supported. In certain cases PW Control Word may 1902 also be used. 1904 #2 Flow identification (W) 1906 EVPN currently uses labels to identify flows per {Ethernet Segment 1907 Identifier, VLAN} or per MAC level. Additional definition will be 1908 needed to standardize identification of finer granularity DetNet 1909 flows as well as mapping of TSN services to DetNet Services. 1911 #3 Packet sequencing and duplicate elimination (M) 1913 Like MPLS, EVPN generally orders packets similar to Ethernet. 1914 Reordering is possible primarily during path changes and 1915 protection switching. In order to avoid misordering due to ECMP, 1916 EVPN uses the "Preferred PW MPLS Control Word" [RFC4385] (in which 1917 case EVPN inherits this function from PWs) or the entropy labels 1918 [RFC6790]. 1920 If additional ordering mechanisms are required, such mechanisms 1921 will need to be defined. 1923 #5 Flow duplication and merging (M/W) 1925 EVPN relies on the MPLS layer for all protection functions. See 1926 Section 5.1.3 and Section 5.2.2. Some extensions, either at the 1927 EVPN or MPLS levels, will be need to support those DetNet 1928 applications which require true hitless (i.e., zero loss) 1+1 1929 protection switching. (Network coding may be an interesting 1930 alternative to investigate to delivering such hitless loss 1931 protection capability.) 1933 #6 Operations, Administration and Maintenance (M/W) 1935 Nodes supporting EVPN may participate in either or both Ethernet 1936 level and MPLS level OAM. It is likely that it may make sense to 1937 map or adapt the OAM functions at the different levels, but such 1938 has yet to be defined. [RFC6371] provides some useful background 1939 on this topic. 1941 #8 Class and quality of service capabilities (M/W) 1943 EVPN is largely silent on the topics of CoS and QoS, but the 802.1 1944 TSN Ethernet and existing MPLS TE mechanisms can be directly used. 1945 The inter-working of such is new work and within the scope of 1946 DetNet. The existing MPLS mechanisms include both CoS (E-LSPs and 1947 L-LSPs) and QoS (dedicated TE LSPs). 1949 #10 Technical maturity (M) 1950 EVPN is a second (or third) generation MPLS-based L2VPN service 1951 standard. From a data plane standpoint it makes uses of existing 1952 MPLS data plane mechanisms. The mechanisms have been widely 1953 implemented and deployed. 1955 5.2.4.3. Summary 1957 EVPN is the emerging successor to VPLS. EVPN is standardized, 1958 implemented and deployed. It makes use of the mature MPLS data 1959 plane. While offering a mature and very comprehensive set of 1960 features, certain DetNet required features are not fully/directly 1961 supported and additional standardization in these areas are needed. 1962 Examples include: mapping CoS and QoS; use of labels per DetNet flow, 1963 and hitless 1+1 protection. 1965 5.2.5. Higher layer header fields 1967 Fields of headers belonging to higher OSI layers can be used to 1968 implement functionality that is not provided e.g., by the IPv6 or 1969 IPv4 header fields. However, this approach cannot be always applied, 1970 e.g., due to encryption. Furthermore, even if this approach is 1971 applicable, it requires deep packet inspection from the routers and 1972 switches. There are implementation dependent limits how far into the 1973 packet the lookup can be done efficiently in the fast path. When 1974 encryption is not used, a safe bet is generally between 128 and 256 1975 octets for the maximum lookup depth. Various higher layer protocols 1976 can be applied. Some examples are provided here for the sequence 1977 numbering feature (Section 4.3). 1979 5.2.5.1. TCP 1981 The TCP header includes a sequence number parameter, which can be 1982 applied to detect and eliminate duplicate packets if DetNet service 1983 protection is used. As the TCP header is right after the IP header, 1984 it does not require very deep packet inspection; the 4-byte sequence 1985 number is conveyed by bits 32 through 63 of the TCP header. In 1986 addition to sequencing, the TCP header also contain source and 1987 destination port information that can be used for assisting the flow 1988 identification. 1990 5.2.5.2. RTP 1992 5.2.5.2.1. Solution Description 1994 Real-time Transport Protocol (RTP) [RFC3550] is often used to deliver 1995 time critical traffic in IP networks. RTP is typically carried on 1996 top of UDP/IP. However, as noted earlier in Section 5.2.3 1997 PseudoWires also have a well-defined way of embedding and 1998 transposting RTP header as part of its payload encapsulation headers/ 1999 sub-layer. RTP is also augmented by its own control protocol RTCP, 2000 which monitors of the data delivery and provides minimal control and 2001 identification functionality. RTCP packets do not carry "media 2002 payload". Although both RTP and RTCP are typically used with UDP/IP 2003 transport they are designed to be independent of the underlying 2004 transport and network layers. 2006 The RTP header includes a 2-byte sequence number, which can be used 2007 to detect and eliminate duplicate packets if DetNet service 2008 protection is used. The sequence number is conveyed by bits 16 2009 through 31 of the RTP header. In addition to the sequence number the 2010 RTP header has also timestamp field (bits 32 through 63) that can be 2011 useful for time synchronization purposes. Furthermore, the RTP 2012 header has also one or more synchronization sources (bits starting 2013 from 64) that can potentially be useful for flow identification 2014 purposes. 2016 5.2.5.2.2. Analysis and Discussion 2018 #1 Encapsulation and overhead (M) 2020 RTP adds minimum 12 octets of header overhead. Typically 8 octets 2021 overhead of UDP header has to be also added, at least in a case 2022 when RTP is transported over IP. Although RTCP packets do not 2023 contribute to the media payload transport they still consume 2024 overall network capacity, since all participants to an RTP session 2025 including sourcess and multicast session destinations are expected 2026 to send RTCP reports. 2028 #2 Flow identification (M) 2030 The RTP header contains a synchronization source (SSRC) 2031 identifier. The intent is that no two synchronization sources 2032 within the same RTP session has the same SSRC identifier. 2034 #3 Packet sequencing and duplicate elimination (M) 2036 The RTP header contains a 16 bit sequence number. The sequence 2037 number can be also used to detect duplicate packets. 2039 #5 Flow duplication and merging (M/W) 2041 RTP has precedence of being used for hitless protection switching 2042 [ST20227], which essentially is equivalent to DetNet service 2043 protection. Furthermore, recent work in IETF for RTP stream 2044 duplication [RFC7198] as a mechanism to protect media flows from 2045 packet loss is again equivalent to Detnet service protection. 2047 #6 Operations, Administration and Maintenance (M) 2049 RTP has its own control protocol RTCP for (minimal) management and 2050 stream monitoring purposes. Existing IP OAM tools can directly 2051 leveraged when RTP is deployed over IP transport. 2053 #8 Class and quality of service capabilities (M/W) 2055 TBD. [Editor's note: relies on lower layers to provide CoS/QoS] 2057 #10 Technical maturity (M) 2059 RTP has been deployed and used in large commercial systems for 2060 over ten years and can be considered a mature technology. 2062 5.2.5.2.3. Summary 2064 RTP appears to be a good candidate as the deterministic networking 2065 data plane solution alternative for the DetNet Service layer. The 2066 strong points are the technical maturity and the fact it was designed 2067 for transporting time-sensitive payload from the beginning. RTP is 2068 specifically well suited to be used with (UDP)/IP transport. 2070 Extensions may be required to realize the packet replication and 2071 duplicate detection features of the deterministic networking data 2072 plane. However, there is already precedence of similar solutions 2073 that could potentially be leveraged [ST20227][RFC7198]. 2075 6. Summary of data plane alternatives 2077 The following table summarizes the criteria (Section 4) used for the 2078 evaluation of data plane options. 2080 Applicability per Alternative 2082 +--------+---------------------------------------------+ 2083 | Item # | Meaning | 2084 +--------+---------------------------------------------+ 2085 | #1 | Encapsulation and overhead | 2086 | #2 | Flow identification | 2087 | #3 | Packet sequencing and duplicate elimination | 2088 | #4 | Explicit routes | 2089 | #5 | Flow duplication and merging | 2090 | #6 | Operations, Administration and Maintenance | 2091 | #8 | Class and quality of service capabilities | 2092 | #9 | Packet traceability | 2093 | #10 | Technical maturity | 2094 +--------+---------------------------------------------+ 2096 Table 5: Evaluation criteria (#7 obsoleted) 2098 There is no single technology that could meet all the criteria on its 2099 own. Distinguishing the DetNet Service and the DetNet Transport, as 2100 explained in (Section 3), allows a number of combinations, which can 2101 meet most of the criteria. There is no room here to evaluate all 2102 possible combinations. Therefore, only some combinations are 2103 highlighted here, which are selected based on the number of criteria 2104 that are met and the maturity of the technology (#10). 2106 The following table summarizes the evaluation of the data plane 2107 options that can be used for the DetNet Transport Layer against the 2108 evaluation criteria. Each value in the table is from the 2109 corresponding section. 2111 Applicability per Transport Alternative 2113 +----------+-----+----+----+-----+-----+-----+----+-----+ 2114 | Solution | #1 | #2 | #4 | #5 | #6 | #8 | #9 | #10 | 2115 +----------+-----+----+----+-----+-----+-----+----+-----+ 2116 | IPv6 | M | W | W | W | M | W | W | M/W | 2117 | IPv4 | M | W | W | W/N | M | M/W | W | M/W | 2118 | MPLS | M | M | M | M/W | M | M/W | M | M | 2119 | BIER | M | M | M | M/W | M/W | M/W | M | W | 2120 | BIER-TE | W/M | N | N | M/W | W | N | W | W | 2121 +----------+-----+----+----+-----+-----+-----+----+-----+ 2123 Summarizing Transport capabilities 2125 Table 6: DetNet Transport Layer 2127 The following table summarizes the evaluation of the data plane 2128 options that can be used for the DetNet Service Layer against the 2129 criteria evaluation criteria. Each value in the table is from the 2130 corresponding section. 2132 Applicability per Service Alternative 2134 +----------+----+----+-----+-----+-----+-----+-----+ 2135 | Solution | #1 | #2 | #3 | #5 | #6 | #8 | #10 | 2136 +----------+----+----+-----+-----+-----+-----+-----+ 2137 | GRE | M | W | M/W | W/N | M | W | M | 2138 | PWE3 | M | M | M | W | M/W | M/W | M | 2139 | EVPN | M | W | M | M/W | M/W | M/W | M | 2140 | RTP | M | M | M | M/W | M | M/W | M | 2141 +----------+----+----+-----+-----+-----+-----+-----+ 2143 Summarizing Service capabilities 2145 Table 7: DetNet Service Layer 2147 PseudoWire (Section 5.2.3) is a technology that is mature and meets 2148 most of the criteria for the DetNet Service layer as shown in the 2149 table above. From upper layer protocols PWs or RTP can be a 2150 candidate for non-MPLS PSNs. The identified work for PWs is to 2151 figure out how to implement duplicate detection for these protocols 2152 (e.g., based on [RFC3985]). In a case of RTP there is precedence of 2153 implementing packet duplication and duplicate elimination 2154 [ST20227][RFC7198]. 2156 PWs can be carried over MPLS or IP. MPLS is the most common 2157 technology that is used as PSN for PseudoWires; furthermore, MPLS is 2158 a mature technology and meets most DetNet Transport layer criteria. 2159 IPv[46] can be also used as PSN and both are mature technologies, 2160 although both generally only support CoS (DiffServ) in deployed 2161 networks. RTP is independent of the underlying transport technology 2162 and network. However, it is well suited for UDP/IP transport or 2163 embedded as a part of the PseudoWire timing sub-layer. 2165 7. Security considerations 2167 This document does not add any new security considerations beyond 2168 what the referenced technologies already have. 2170 8. IANA Considerations 2172 This document has no IANA considerations. 2174 9. Acknowledgements 2176 The author(s) ACK and NACK. 2178 The following people were part of the DetNet Data Plane Design Team: 2180 Jouni Korhonen 2181 Janos Farkas 2182 Norman Finn 2183 Olivier Marce 2184 Gregory Mirsky 2185 Pascal Thubert 2186 Zhuangyan Zhuang 2188 Substantial contributions were received from: 2190 Balazs Varga (service model) 2192 The DetNet chairs serving during the DetNet Data Plane Design Team: 2194 Lou Berger 2195 Pat Thaler 2197 10. References 2199 10.1. Informative References 2201 [I-D.eckert-bier-te-arch] 2202 Eckert, T., Cauchie, G., Braun, W., and M. Menth, "Traffic 2203 Engineering for Bit Index Explicit Replication BIER-TE", 2204 draft-eckert-bier-te-arch-04 (work in progress), July 2205 2016. 2207 [I-D.finn-detnet-architecture] 2208 Finn, N., Thubert, P., and M. Teener, "Deterministic 2209 Networking Architecture", draft-finn-detnet- 2210 architecture-07 (work in progress), July 2016. 2212 [I-D.ietf-6man-rfc2460bis] 2213 Deering, D. and R. Hinden, "Internet Protocol, Version 6 2214 (IPv6) Specification", draft-ietf-6man-rfc2460bis-05 (work 2215 in progress), June 2016. 2217 [I-D.ietf-6man-segment-routing-header] 2218 Previdi, S., Filsfils, C., Field, B., Leung, I., Linkova, 2219 J., Aries, E., Kosugi, T., Vyncke, E., and D. Lebrun, 2220 "IPv6 Segment Routing Header (SRH)", draft-ietf-6man- 2221 segment-routing-header-01 (work in progress), March 2016. 2223 [I-D.ietf-bier-architecture] 2224 Wijnands, I., Rosen, E., Dolganow, A., Przygienda, T., and 2225 S. Aldrin, "Multicast using Bit Index Explicit 2226 Replication", draft-ietf-bier-architecture-04 (work in 2227 progress), July 2016. 2229 [I-D.ietf-detnet-problem-statement] 2230 Finn, N. and P. Thubert, "Deterministic Networking Problem 2231 Statement", draft-ietf-detnet-problem-statement-00 (work 2232 in progress), April 2016. 2234 [I-D.ietf-mpls-residence-time] 2235 Mirsky, G., Ruffini, S., Gray, E., Drake, J., Bryant, S., 2236 and S. Vainshtein, "Residence Time Measurement in MPLS 2237 network", draft-ietf-mpls-residence-time-11 (work in 2238 progress), July 2016. 2240 [I-D.ietf-spring-segment-routing] 2241 Filsfils, C., Previdi, S., Decraene, B., Litkowski, S., 2242 and R. Shakir, "Segment Routing Architecture", draft-ietf- 2243 spring-segment-routing-09 (work in progress), July 2016. 2245 [I-D.ietf-sunset4-gapanalysis] 2246 Perreault, S., Tsou, T., Zhou, C., and P. Fan, "Gap 2247 Analysis for IPv4 Sunset", draft-ietf- 2248 sunset4-gapanalysis-07 (work in progress), April 2015. 2250 [I-D.kumarzheng-bier-ping] 2251 Kumar, N., Pignataro, C., Akiya, N., Zheng, L., Chen, M., 2252 and G. Mirsky, "BIER Ping and Trace", draft-kumarzheng- 2253 bier-ping-03 (work in progress), July 2016. 2255 [I-D.mirsky-bier-path-mtu-discovery] 2256 Mirsky, G., Przygienda, T., and A. Dolganow, "Path Maximum 2257 Transmission Unit Discovery (PMTUD) for Bit Index Explicit 2258 Replication (BIER) Layer", draft-mirsky-bier-path-mtu- 2259 discovery-01 (work in progress), April 2016. 2261 [I-D.mirsky-bier-pmmm-oam] 2262 Mirsky, G., Zheng, L., Chen, M., and G. Fioccola, 2263 "Performance Measurement (PM) with Marking Method in Bit 2264 Index Explicit Replication (BIER) Layer", draft-mirsky- 2265 bier-pmmm-oam-01 (work in progress), March 2016. 2267 [I-D.ooamdt-rtgwg-oam-gap-analysis] 2268 Mirsky, G., Nordmark, E., Pignataro, C., Kumar, N., Kumar, 2269 D., Chen, M., Yizhou, L., Mozes, D., Networks, J., and i. 2270 ibagdona@gmail.com, "Operations, Administration and 2271 Maintenance (OAM) for Overlay Networks: Gap Analysis", 2272 draft-ooamdt-rtgwg-oam-gap-analysis-02 (work in progress), 2273 July 2016. 2275 [I-D.ooamdt-rtgwg-ooam-requirement] 2276 Kumar, N., Pignataro, C., Kumar, D., Mirsky, G., Chen, M., 2277 Nordmark, E., Networks, J., and D. Mozes, "Overlay OAM 2278 Requirements", draft-ooamdt-rtgwg-ooam-requirement-01 2279 (work in progress), July 2016. 2281 [IEEE802.1Qbv] 2282 IEEE, "Enhancements for Scheduled Traffic", 2016, 2283 . 2285 [IEEE802.1Qca] 2286 IEEE 802.1, "IEEE 802.1Qca Bridges and Bridged Networks - 2287 Amendment 24: Path Control and Reservation", IEEE 2288 P802.1Qca/D2.1 P802.1Qca, June 2015, 2289 . 2292 [IEEE802.1Qch] 2293 IEEE, "Cyclic Queuing and Forwarding", 2016, 2294 . 2296 [IEEE8021CB] 2297 Finn, N., "Draft Standard for Local and metropolitan area 2298 networks - Seamless Redundancy", IEEE P802.1CB 2299 /D2.1 P802.1CB, December 2015, 2300 . 2303 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 2304 DOI 10.17487/RFC0791, September 1981, 2305 . 2307 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 2308 Communication Layers", STD 3, RFC 1122, 2309 DOI 10.17487/RFC1122, October 1989, 2310 . 2312 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 2313 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 2314 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 2315 September 1997, . 2317 [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 2318 (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460, 2319 December 1998, . 2321 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2322 "Definition of the Differentiated Services Field (DS 2323 Field) in the IPv4 and IPv6 Headers", RFC 2474, 2324 DOI 10.17487/RFC2474, December 1998, 2325 . 2327 [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. 2328 Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, 2329 DOI 10.17487/RFC2784, March 2000, 2330 . 2332 [RFC2890] Dommety, G., "Key and Sequence Number Extensions to GRE", 2333 RFC 2890, DOI 10.17487/RFC2890, September 2000, 2334 . 2336 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2337 Label Switching Architecture", RFC 3031, 2338 DOI 10.17487/RFC3031, January 2001, 2339 . 2341 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2342 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2343 Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001, 2344 . 2346 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2347 of Explicit Congestion Notification (ECN) to IP", 2348 RFC 3168, DOI 10.17487/RFC3168, September 2001, 2349 . 2351 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2352 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2353 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 2354 . 2356 [RFC3232] Reynolds, J., Ed., "Assigned Numbers: RFC 1700 is Replaced 2357 by an On-line Database", RFC 3232, DOI 10.17487/RFC3232, 2358 January 2002, . 2360 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2361 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2362 Protocol Label Switching (MPLS) Support of Differentiated 2363 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 2364 . 2366 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2367 in Multi-Protocol Label Switching (MPLS) Networks", 2368 RFC 3443, DOI 10.17487/RFC3443, January 2003, 2369 . 2371 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 2372 Switching (GMPLS) Signaling Resource ReserVation Protocol- 2373 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 2374 DOI 10.17487/RFC3473, January 2003, 2375 . 2377 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 2378 Jacobson, "RTP: A Transport Protocol for Real-Time 2379 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 2380 July 2003, . 2382 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 2383 Edge-to-Edge (PWE3) Architecture", RFC 3985, 2384 DOI 10.17487/RFC3985, March 2005, 2385 . 2387 [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., 2388 "Encapsulating MPLS in IP or Generic Routing Encapsulation 2389 (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005, 2390 . 2392 [RFC4303] Kent, S., "IP Encapsulating Security Payload (ESP)", 2393 RFC 4303, DOI 10.17487/RFC4303, December 2005, 2394 . 2396 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2397 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2398 Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385, 2399 February 2006, . 2401 [RFC4446] Martini, L., "IANA Allocations for Pseudowire Edge to Edge 2402 Emulation (PWE3)", BCP 116, RFC 4446, 2403 DOI 10.17487/RFC4446, April 2006, 2404 . 2406 [RFC4447] Martini, L., Ed., Rosen, E., El-Aawar, N., Smith, T., and 2407 G. Heron, "Pseudowire Setup and Maintenance Using the 2408 Label Distribution Protocol (LDP)", RFC 4447, 2409 DOI 10.17487/RFC4447, April 2006, 2410 . 2412 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 2413 "Encapsulation Methods for Transport of Ethernet over MPLS 2414 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 2415 . 2417 [RFC4553] Vainshtein, A., Ed. and YJ. Stein, Ed., "Structure- 2418 Agnostic Time Division Multiplexing (TDM) over Packet 2419 (SAToP)", RFC 4553, DOI 10.17487/RFC4553, June 2006, 2420 . 2422 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 2423 2 Virtual Private Networks (L2VPNs)", RFC 4664, 2424 DOI 10.17487/RFC4664, September 2006, 2425 . 2427 [RFC4817] Townsley, M., Pignataro, C., Wainner, S., Seely, T., and 2428 J. Young, "Encapsulation of MPLS over Layer 2 Tunneling 2429 Protocol Version 3", RFC 4817, DOI 10.17487/RFC4817, March 2430 2007, . 2432 [RFC4875] Aggarwal, R., Ed., Papadimitriou, D., Ed., and S. 2433 Yasukawa, Ed., "Extensions to Resource Reservation 2434 Protocol - Traffic Engineering (RSVP-TE) for Point-to- 2435 Multipoint TE Label Switched Paths (LSPs)", RFC 4875, 2436 DOI 10.17487/RFC4875, May 2007, 2437 . 2439 [RFC5087] Stein, Y(J)., Shashoua, R., Insler, R., and M. Anavi, 2440 "Time Division Multiplexing over IP (TDMoIP)", RFC 5087, 2441 DOI 10.17487/RFC5087, December 2007, 2442 . 2444 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2445 Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2446 2008, . 2448 [RFC5254] Bitar, N., Ed., Bocci, M., Ed., and L. Martini, Ed., 2449 "Requirements for Multi-Segment Pseudowire Emulation Edge- 2450 to-Edge (PWE3)", RFC 5254, DOI 10.17487/RFC5254, October 2451 2008, . 2453 [RFC5305] Li, T. and H. Smit, "IS-IS Extensions for Traffic 2454 Engineering", RFC 5305, DOI 10.17487/RFC5305, October 2455 2008, . 2457 [RFC5331] Aggarwal, R., Rekhter, Y., and E. Rosen, "MPLS Upstream 2458 Label Assignment and Context-Specific Label Space", 2459 RFC 5331, DOI 10.17487/RFC5331, August 2008, 2460 . 2462 [RFC5332] Eckert, T., Rosen, E., Ed., Aggarwal, R., and Y. Rekhter, 2463 "MPLS Multicast Encapsulations", RFC 5332, 2464 DOI 10.17487/RFC5332, August 2008, 2465 . 2467 [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation 2468 Element (PCE) Communication Protocol (PCEP)", RFC 5440, 2469 DOI 10.17487/RFC5440, March 2009, 2470 . 2472 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2473 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2474 Class" Field", RFC 5462, DOI 10.17487/RFC5462, February 2475 2009, . 2477 [RFC5586] Bocci, M., Ed., Vigoureux, M., Ed., and S. Bryant, Ed., 2478 "MPLS Generic Associated Channel", RFC 5586, 2479 DOI 10.17487/RFC5586, June 2009, 2480 . 2482 [RFC5659] Bocci, M. and S. Bryant, "An Architecture for Multi- 2483 Segment Pseudowire Emulation Edge-to-Edge", RFC 5659, 2484 DOI 10.17487/RFC5659, October 2009, 2485 . 2487 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 2488 L., and L. Berger, "A Framework for MPLS in Transport 2489 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 2490 . 2492 [RFC5960] Frost, D., Ed., Bryant, S., Ed., and M. Bocci, Ed., "MPLS 2493 Transport Profile Data Plane Architecture", RFC 5960, 2494 DOI 10.17487/RFC5960, August 2010, 2495 . 2497 [RFC6275] Perkins, C., Ed., Johnson, D., and J. Arkko, "Mobility 2498 Support in IPv6", RFC 6275, DOI 10.17487/RFC6275, July 2499 2011, . 2501 [RFC6371] Busi, I., Ed. and D. Allan, Ed., "Operations, 2502 Administration, and Maintenance Framework for MPLS-Based 2503 Transport Networks", RFC 6371, DOI 10.17487/RFC6371, 2504 September 2011, . 2506 [RFC6373] Andersson, L., Ed., Berger, L., Ed., Fang, L., Ed., Bitar, 2507 N., Ed., and E. Gray, Ed., "MPLS Transport Profile (MPLS- 2508 TP) Control Plane Framework", RFC 6373, 2509 DOI 10.17487/RFC6373, September 2011, 2510 . 2512 [RFC6378] Weingarten, Y., Ed., Bryant, S., Osborne, E., Sprecher, 2513 N., and A. Fulignoli, Ed., "MPLS Transport Profile (MPLS- 2514 TP) Linear Protection", RFC 6378, DOI 10.17487/RFC6378, 2515 October 2011, . 2517 [RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS 2518 On-Demand Connectivity Verification and Route Tracing", 2519 RFC 6426, DOI 10.17487/RFC6426, November 2011, 2520 . 2522 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 2523 "IPv6 Flow Label Specification", RFC 6437, 2524 DOI 10.17487/RFC6437, November 2011, 2525 . 2527 [RFC6564] Krishnan, S., Woodyatt, J., Kline, E., Hoagland, J., and 2528 M. Bhatia, "A Uniform Format for IPv6 Extension Headers", 2529 RFC 6564, DOI 10.17487/RFC6564, April 2012, 2530 . 2532 [RFC6621] Macker, J., Ed., "Simplified Multicast Forwarding", 2533 RFC 6621, DOI 10.17487/RFC6621, May 2012, 2534 . 2536 [RFC6658] Bryant, S., Ed., Martini, L., Swallow, G., and A. Malis, 2537 "Packet Pseudowire Encapsulation over an MPLS PSN", 2538 RFC 6658, DOI 10.17487/RFC6658, July 2012, 2539 . 2541 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2542 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2543 RFC 6790, DOI 10.17487/RFC6790, November 2012, 2544 . 2546 [RFC6814] Pignataro, C. and F. Gont, "Formally Deprecating Some IPv4 2547 Options", RFC 6814, DOI 10.17487/RFC6814, November 2012, 2548 . 2550 [RFC6864] Touch, J., "Updated Specification of the IPv4 ID Field", 2551 RFC 6864, DOI 10.17487/RFC6864, February 2013, 2552 . 2554 [RFC7045] Carpenter, B. and S. Jiang, "Transmission and Processing 2555 of IPv6 Extension Headers", RFC 7045, 2556 DOI 10.17487/RFC7045, December 2013, 2557 . 2559 [RFC7198] Begen, A. and C. Perkins, "Duplicating RTP Streams", 2560 RFC 7198, DOI 10.17487/RFC7198, April 2014, 2561 . 2563 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 2564 Henderickx, W., and A. Isaac, "Requirements for Ethernet 2565 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 2566 . 2568 [RFC7271] Ryoo, J., Ed., Gray, E., Ed., van Helvoort, H., 2569 D'Alessandro, A., Cheung, T., and E. Osborne, "MPLS 2570 Transport Profile (MPLS-TP) Linear Protection to Match the 2571 Operational Expectations of Synchronous Digital Hierarchy, 2572 Optical Transport Network, and Ethernet Transport Network 2573 Operators", RFC 7271, DOI 10.17487/RFC7271, June 2014, 2574 . 2576 [RFC7399] Farrel, A. and D. King, "Unanswered Questions in the Path 2577 Computation Element Architecture", RFC 7399, 2578 DOI 10.17487/RFC7399, October 2014, 2579 . 2581 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 2582 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 2583 Defined Networking (SDN): Layers and Architecture 2584 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 2585 2015, . 2587 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 2588 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 2589 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 2590 2015, . 2592 [RFC7510] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black, 2593 "Encapsulating MPLS in UDP", RFC 7510, 2594 DOI 10.17487/RFC7510, April 2015, 2595 . 2597 [RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network 2598 Virtualization Using Generic Routing Encapsulation", 2599 RFC 7637, DOI 10.17487/RFC7637, September 2015, 2600 . 2602 [RFC7676] Pignataro, C., Bonica, R., and S. Krishnan, "IPv6 Support 2603 for Generic Routing Encapsulation (GRE)", RFC 7676, 2604 DOI 10.17487/RFC7676, October 2015, 2605 . 2607 [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and 2608 S. Ray, "North-Bound Distribution of Link-State and 2609 Traffic Engineering (TE) Information Using BGP", RFC 7752, 2610 DOI 10.17487/RFC7752, March 2016, 2611 . 2613 [RFC7813] Farkas, J., Ed., Bragg, N., Unbehagen, P., Parsons, G., 2614 Ashwood-Smith, P., and C. Bowers, "IS-IS Path Control and 2615 Reservation", RFC 7813, DOI 10.17487/RFC7813, June 2016, 2616 . 2618 [RFC7872] Gont, F., Linkova, J., Chown, T., and W. Liu, 2619 "Observations on the Dropping of Packets with IPv6 2620 Extension Headers in the Real World", RFC 7872, 2621 DOI 10.17487/RFC7872, June 2016, 2622 . 2624 [ST20227] SMPTE 2022, "Seamless Protection Switching of SMPTE ST 2625 2022 IP Datagrams", ST 2022-7:2013, 2013, 2626 . 2628 [TSNTG] IEEE Standards Association, "IEEE 802.1 Time-Sensitive 2629 Networks Task Group", 2013, 2630 . 2632 10.2. URIs 2634 [1] http://6lab.cisco.com/stats/ 2636 [2] https://www.google.com/intl/en/ipv6/statistics.html 2638 [3] https://datatracker.ietf.org/wg/spring/charter/ 2640 [4] http://www.iana.org/assignments/g-ach-parameters/g-ach- 2641 parameters.xhtml 2643 [5] http://ftp.isi.edu/in-notes/iana/assignments/ethernet-numbers 2645 [6] https://tools.ietf.org/wg/bess/ 2647 Appendix A. Examples of combined DetNet Service and Transport layers 2649 Authors' Addresses 2651 Jouni Korhonen (editor) 2652 Broadcom 2653 3151 Zanker Road 2654 San Jose, CA 95134 2655 USA 2657 Email: jouni.nospam@gmail.com 2659 Janos Farkas 2660 Ericsson 2661 Konyves Kalman krt. 11/B 2662 Budapest 1097 2663 Hungary 2665 Email: janos.farkas@ericsson.com 2667 Gregory Mirsky 2668 Ericsson 2670 Email: gregory.mirsky@ericsson.com 2672 Pascal Thubert 2673 Cisco 2675 Email: pthubert@cisco.com 2677 Yan Zhuang 2678 Huawei 2680 Email: zhuangyan.zhuang@huawei.com 2682 Lou Berger 2683 LabN Consulting, L.L.C. 2685 Email: lberger@labn.net