idnits 2.17.1 draft-dt-detnet-dp-alt-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 21, 2016) is 2957 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'Network' is mentioned on line 161, but not defined == Missing Reference: 'DetNet-ARCH' is mentioned on line 400, but not defined -- Looks like a reference, but probably isn't: '1' on line 2333 -- Looks like a reference, but probably isn't: '2' on line 2335 -- Looks like a reference, but probably isn't: '3' on line 2337 -- Looks like a reference, but probably isn't: '4' on line 2339 -- Looks like a reference, but probably isn't: '5' on line 2342 -- Looks like a reference, but probably isn't: '6' on line 2344 == Unused Reference: 'RFC2702' is defined on line 2023, but no explicit reference was found in the text == Unused Reference: 'RFC6073' is defined on line 2191, but no explicit reference was found in the text == Unused Reference: 'RFC6718' is defined on line 2245, but no explicit reference was found in the text == Unused Reference: 'RFC6733' is defined on line 2249, but no explicit reference was found in the text == Unused Reference: 'RFC7167' is defined on line 2272, but no explicit reference was found in the text == Unused Reference: 'ST20227' is defined on line 2323, but no explicit reference was found in the text == Outdated reference: A later version (-06) exists of draft-eckert-bier-te-arch-02 == Outdated reference: A later version (-08) exists of draft-finn-detnet-architecture-02 == Outdated reference: A later version (-05) exists of draft-finn-detnet-problem-statement-04 == Outdated reference: A later version (-13) exists of draft-ietf-6man-rfc2460bis-04 == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-01 == Outdated reference: A later version (-08) exists of draft-ietf-bier-architecture-03 == Outdated reference: A later version (-15) exists of draft-ietf-mpls-residence-time-05 == Outdated reference: A later version (-15) exists of draft-ietf-spring-segment-routing-07 == Outdated reference: A later version (-09) exists of draft-ietf-sunset4-gapanalysis-07 -- Obsolete informational reference (is this intentional?): RFC 1393 (Obsoleted by RFC 6814) -- Obsolete informational reference (is this intentional?): RFC 1700 (Obsoleted by RFC 3232) -- Obsolete informational reference (is this intentional?): RFC 2460 (Obsoleted by RFC 8200) -- Obsolete informational reference (is this intentional?): RFC 4447 (Obsoleted by RFC 8077) Summary: 0 errors (**), 0 flaws (~~), 18 warnings (==), 12 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet J. Korhonen, Ed. 3 Internet-Draft Broadcom 4 Intended status: Informational J. Farkas 5 Expires: September 22, 2016 G. Mirsky 6 Ericsson 7 P. Thubert 8 Cisco 9 Y. Zhuang 10 Huawei 11 L. Berger 12 LabN 13 March 21, 2016 15 DetNet Data Plane Protocol and Solution Alternatives 16 draft-dt-detnet-dp-alt-00 18 Abstract 20 This document identifies existing IP and MPLS, and other 21 encapsulations that run over IP and/or MPLS data plane technologies 22 that can be considered as the base line solution for deterministic 23 networking data plane definition. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at http://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on September 22, 2016. 42 Copyright Notice 44 Copyright (c) 2016 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 2. DetNet Data Plane Overview . . . . . . . . . . . . . . . . . 3 61 3. Criteria for data plane solution alternatives . . . . . . . . 6 62 3.1. #? DetNet Service Interface . . . . . . . . . . . . . . . 6 63 3.2. #1 Encapsulation and overhead . . . . . . . . . . . . . . 7 64 3.3. #2 Flow identification . . . . . . . . . . . . . . . . . 7 65 3.4. #3 Packet sequencing . . . . . . . . . . . . . . . . . . 7 66 3.5. #4 Explicit routes . . . . . . . . . . . . . . . . . . . 8 67 3.6. #5 Packet replication and deletion . . . . . . . . . . . 8 68 3.7. #6 Operations, Administration and Maintenance . . . . . . 9 69 3.8. #7 Time synchronization . . . . . . . . . . . . . . . . . 9 70 3.9. #8 Class and quality of service capabilities . . . . . . 9 71 3.10. #9 Packet traceability . . . . . . . . . . . . . . . . . 10 72 3.11. #10 Technical maturity . . . . . . . . . . . . . . . . . 10 73 4. Data plane solution alternatives . . . . . . . . . . . . . . 11 74 4.1. DetNet Transport layer technologies . . . . . . . . . . . 11 75 4.1.1. Native IPv6 transport . . . . . . . . . . . . . . . . 11 76 4.1.2. Native IPv4 transport . . . . . . . . . . . . . . . . 14 77 4.1.3. Multiprotocol Label Switching (MPLS) . . . . . . . . 16 78 4.2. DetNet Service layer technologies . . . . . . . . . . . . 21 79 4.2.1. Generic Routing Encapsulation (GRE) . . . . . . . . . 21 80 4.2.2. Layer-2 Tunneling Protocol (L2TP) . . . . . . . . . . 24 81 4.2.3. Virtual Extensible LAN (VXLAN) . . . . . . . . . . . 24 82 4.2.4. MPLS-based Services . . . . . . . . . . . . . . . . . 24 83 4.2.5. Pseudo Wire Emulation Edge-to-Edge (PWE3) . . . . . . 26 84 4.2.6. MPLS-Based Ethernet VPN (EVPN) . . . . . . . . . . . 30 85 4.2.7. Bit Indexed Explicit Replication (BIER) . . . . . . . 33 86 4.2.8. Higher layer header fields . . . . . . . . . . . . . 41 87 5. Summary of data plane alternatives . . . . . . . . . . . . . 42 88 6. Security considerations . . . . . . . . . . . . . . . . . . . 42 89 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 42 90 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 42 91 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 43 92 9.1. Informative References . . . . . . . . . . . . . . . . . 43 93 9.2. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 52 94 Appendix A. Examples of combined DetNet Service and Transport 95 layers . . . . . . . . . . . . . . . . . . . . . . . 52 96 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 52 98 1. Introduction 100 Deterministic Networking (DetNet) [I-D.finn-detnet-problem-statement] 101 provides a capability to carry unicast or multicast data flows for 102 real-time applications with extremely low data loss rates and known 103 upper bound maximum latency [I-D.finn-detnet-architecture]. The 104 deterministic networking Quality of Service (QoS) is expressed as 1) 105 the maximum end-to-end latency from sender (talker) to receiver 106 (listener), and 2) probability of loss of a packet. Only the worst- 107 case values for the mentioned parameters are concerned. 109 There are three techniques to achieve the QoS required by 110 deterministic networks: 112 o zero congestion loss, 113 o explicit routes, 114 o packet replication and deletion. 116 This document identifies existing IP and Multiprotocol Label 117 Switching (MPLS) [RFC3031], layer-2 or layer-3 encapsulations and 118 transport protocols that could be considered as foundations for a 119 deterministic networking data plane. The full scope of the 120 deterministic networking data plane solution is considered including, 121 as appropriate: quality of service (QoS); Operations, Administration 122 and Maintenance (OAM); and time synchronization among other criteria 123 described in Section 3. 125 This document does not select a deterministic networking data plane 126 protocol. It does, however, elaborate what it would require to adapt 127 and use a specific protocol as the deterministic networking data 128 plane solution. This document is only concerned with data plane 129 considerations and, specifically, with topics that potentially impact 130 potential deterministic networking aware data plane hardware. 131 Control plane considerations are out of scope of this document. 133 2. DetNet Data Plane Overview 135 [Editor's note: all/portions of the following may be moved to the 136 DetNet Architecture document at some future point.] 138 A "Deterministic Network" will be composed of DetNet enabled "End 139 Systems", DetNet enabled "Edge Nodes", and DetNet enabled "Network 140 Nodes". DetNet enabled nodes will provide a DetNet service to 141 attached DetNet End Systems. All DetNet enabled systems and nodes 142 will be interconnected by sub-networks. These sub-networks will 143 provide DetNet compatible service for support of DetNet traffic. 144 Examples of sub-networks include 802.1TSN and a point-to-point OTN 145 link. Of course, multi-layer DetNet systems may be possible too, 146 where one DetNet appears as a sub-network, and provides service to, a 147 higher layer DetNet system. A simple DetNet is shown in Figure 1. 149 +------+ +------+ 150 | End |<-------------Deterministic flow--------------->| End | 151 |System| |System| 152 +--+---+ +-+----+ 153 | <--------------DetNet flow-----------------> | 154 | ,+-''''--. 155 | [ Sub ] 156 |Link [ Network ] 157 | +-....--' 158 | | 159 | +------+ +-------+ ,-'''-. +-------+ +------+ 160 | | Edge |Link |Network|--[ Sub ]--|Network|--| Edge | 161 +----| Node |-----| Node | [Network] | Node | | Node | 162 +------+ +-------+ +-...-' +-------+ +------+ 164 Figure 1: A Simple DetNet Enabled Network 166 This DetNet data plane is logically divided into two layers: 168 o The DetNet Service layer provides adaptation of DetNet services. 169 It is composed of a shim layer to carry DetNet flow specific 170 attributes, which are needed during forwarding. End systems 171 originate and terminate the DetNet Service layer and are peers at 172 the DetNet Service layer. 173 o The DetNet Transport layer is supported by all DetNet aware 174 systems and nodes. It operates below the DetNet Service layer. 175 The DetNet Transport layer is used to relay traffic end to end 176 across a DetNet domain. 178 Distinguishing the function of these two DetNet data plane layers 179 helps to explore and evaluate various combinations of the data plane 180 solutions available. This separation of DetNet layers, while 181 helpful, should not be considered as formal requirement. For 182 example, some technologies may violate these strict layers and still 183 be able to deliver a DetNet service. 185 A number of data plane technology candidates are discussed later in 186 this document. They can be mapped to the two layers as shown in 187 Figure 2. 189 . 190 . 191 . 192 +-----------+ 193 | Service | PW, RTP, UDP, GRE, L2TP, VXLAN 194 ~~~~~~~~~~~~~ 195 | Transport | IPv6, IPv4, MPLS LSPs, BIER, BIER-TE 196 +-----------+ 197 . 198 . 200 Figure 2: DetNet adaptation to data plane 202 Both the DetNet Service and the DetNet Transport layers are provided 203 by a single technology in some solutions, e.g. the DetNet Service 204 layer is PW and the DetNet Transport layer is MPLS in case of PW over 205 MPLS. Nonetheless, both the DetNet Service and the DetNet Transport 206 layers can include multiple technology layers in other solutions in 207 order to provide the capabilities needed for DetNet flows. For 208 instance, the DetNet Transport layer can comprise MPLS-in-GRE 209 (Section 4.2.5) in one solution. In another solution, the DetNet 210 Service layer can include, e.g., RTP in UDP (Section 4.2.8). 212 [Editor's note: I'm not sure if the remainder of this section says 213 anything not present in the next section. Will need to revisit as 214 part of the pre-pub review.] 216 There are many valid options to create a data plane solution for 217 DetNet traffic by selecting a technology approach for the DetNet 218 Service layer and also selecting a technology approach for the DetNet 219 Transport layer. There are a high number of valid combinations. 220 Therefore, not the combinations but the different technologies are 221 evaluated along the criteria collected in Section 3. Different 222 criteria apply for the DetNet Service layer and the DetNet Transport 223 layer, however, some of the criteria are valid for both layers. 225 The criteria for the DetNet Service layer: 227 #1 Encapsulation and overhead 228 #2 Flow identification (Flow ID part of the DetNet flows) 229 #3 Packet sequencing (sequence number) 230 #5 Packet replication and deletion (note: only the packet deletion 231 for seamless redundancy) 232 #6 Operations, Administration and Maintenance (capabilities) 233 #7 Time synchronization (e.g., time stamping) 234 #8 Class and quality of service capabilities (DetNet Service 235 specific) 236 #10 Technical maturity 238 The criteria for the DetNet Transport layer: 240 #1 Encapsulation and overhead 241 #2 Flow identification 242 #4 Explicit routes (network path) 243 #5 Packet replication and deletion (note: only the packet 244 replication for seamless redundancy) 245 #6 Operations, Administration and Maintenance (capabilities) 246 #7 Time synchronization (time/phase sync of nodes) 247 #8 Class and quality of service capabilities (DetNet Transport 248 specific) 249 #9 Packet traceability (verification purposes) 250 #10 Technical maturity 252 [Editor's note: #7 is more of OAM feature.] 254 Some of the criteria are relevant for both the DetNet Service and 255 DetNet Transport layers. The two layers provide together what is 256 needed to meet certain criteria, e.g., flow identification. 257 Different aspects are valid for the two different layers for other 258 criteria, e.g., time synchronization. Furthermore, technical 259 maturity is a criteria valid for both layers. 261 3. Criteria for data plane solution alternatives 263 This section provides criteria to help to evaluate potential options. 264 The criteria can be broken down based on layer. That is if the 265 criteria is focused on delivering DetNet service adaptation, i.e., is 266 concerned with the DetNet Service layer, or if the criteria is 267 focused on transporting flows across an end to end DetNet domain. 268 [Editor's note: which is TBD.] 270 Each deterministic networking data plane solution alternative is 271 described and evaluated using the criteria described in this section. 272 The used criteria enumerated in this section are selected so that 273 they highlight the existence or lack of features that are expected or 274 seen important to a solution alternative for the data plane solution. 276 3.1. #? DetNet Service Interface 278 [Editor's note: this criteria needs a bit more discussion.] 280 One of the most fundamental differences between different potential 281 data plane options is the basic addressing and headers used by DetNet 282 clients. For example, is the basic service a Layer 2 (e.g., 283 Ethernet) or Layer 3 (i.e., IP) service. This decision impacts how 284 DetNet end points are addressed, and the basic forwarding logic of 285 the DetNet Service layer. 287 3.2. #1 Encapsulation and overhead 289 Encapsulation and overhead is related to how the DetNet data plane 290 carries DetNet user traffic. In several cases a deterministic flow 291 has to be encapsulated inside other protocols, for example, when 292 transporting a layer-2 Ethernet frame over an IP transport network. 293 In some cases a former tunneling like encapsulation can be avoided by 294 underlying transport protocol translation, for example, translating 295 layer-2 Ethernet frame including addressing and flow identification 296 into native IP traffic. Last it is possible that talkers and 297 listeners handle deterministic flows natively in layer-3. This 298 criteria concerns what is the encapsulation method the solution 299 alternative support: tunneling like encapsulation, protocol 300 translation or native layer-3 transport. In addition to the 301 encapsulation mechanism this criteria is also concerned of the 302 processing and specifically the encapsulate header overhead. 304 3.3. #2 Flow identification 306 The solution alternative has to provide means to identify specific 307 deterministic flows. The flow identification can, for example, be 308 explicit field in the data plane encapsulation header or implicitly 309 encoded into the addressing scheme of the used data plane protocol or 310 their combination. This criteria concerns the availability and 311 details of deterministic flow identification the data plane protocol 312 alternative has. 314 3.4. #3 Packet sequencing 316 [Editor's note: is in order delivery a strict requirement? if so, it 317 should be stated as such and separately from any other requirement. 318 There are multiple ways to solve this criteria.] 320 The solution alternative has to provide means for end systems to 321 number packets sequentially and transport that sequencing information 322 along with the sent packets. In addition to possible reordering 323 packets one of the important uses for sequencing is detecting 324 duplicates. In a case of intentional packet duplication a 325 combination of flow identification and packet sequencing allows for 326 detecting and discarding duplicates at the receiver (see Section 3.6 327 for more details). This criteria concerns the availability and 328 details of the packet sequencing capabilities the data plane protocol 329 alternative has. 331 3.5. #4 Explicit routes 333 The solution alternative has to provide a mechanism(s) for 334 establishing explicit routes that all packets belonging to a 335 deterministic flow will follow. The explicit route can be seen as a 336 form of source routing or a pre-reserved path e.g., using some 337 network management procedure. It should be noted that the explicit 338 route does not need to be detailed to a level where every possible 339 intermediate node along the path is part of the named explicit route. 340 RSVP-TE [RFC3209] supports explicit routes, and typically provides 341 pinned data paths for established LSPs. At Layer-2, the IEEE 342 802.1Qca [IEEE8021Qca] specification defines how to do explicit path 343 control in a bridged network and its IETF counter part is defined in 344 [I-D.ietf-isis-pcr]. This criteria concerns the available mechanisms 345 for explicit routes for the data plane protocol alternative. 347 3.6. #5 Packet replication and deletion 349 The objective for supporting packet replication and deletion is to 350 enable hitless (or lossless) 1+1 protection, which is also called 351 Seamless redundancy in [I-D.finn-detnet-architecture]. Data plane 352 solutions need to meet this objective independent of the particular 353 solution used. In other words, a packet replication and deletion is 354 one identified method for a data plane solution to provide seamless 355 redundancy and other methods, if so identified, are permissible. 357 The solution alternative has to provide means for end systems and/or 358 relay systems to be able to replicate packets, and later eliminate 359 all but one of the replicas, at multiple points in the network in 360 order to ensure that one (or more) equipment failure event(s) still 361 leave at least one path intact for a deterministic networking flow. 362 The goal is to enable hitless 1+1 protection in a way that no packet 363 gets lost or there is no ramp up time when either one of the paths 364 fails for one reason or another. 366 Another concern regarding packet replication is how to enforce 367 replicated packets to take different route while the final 368 destination still remains the same. With strict source routing, all 369 the intermediate hops are listed and paths can be guaranteed to be 370 non-overlapping. Loose source routing only signals some of the 371 intermediate hops and it takes additional knowledge to ensure that 372 there is no single point of failure. 374 [Editor's note: at the DetNet Transport layer this criteria does not 375 concern packet deletion, only the packet replication. The packet 376 deletion belongs to the DetNet Service layer] 377 The IEEE 802.1CB [IEEE8021CB] is an example of Ethernet-based 378 solution that defines packet sequence numbering, packet replication, 379 and duplicate packet identification and deletion. The deterministic 380 networking data plane solution alternative at layer-3 has to provide 381 equivalent functionality. This criteria concerns the available 382 mechanisms for packet replication and duplicate deletion the data 383 plane protocol alternative has. 385 3.7. #6 Operations, Administration and Maintenance 387 The solution alternative should demonstrate an availability of 388 appropriate standardized OAM tools that can be extended for 389 deterministic networking purposes with a reasonable effort, when 390 required. The OAM tools do not necessarily need to be specific to 391 the data plane protocol as it could be the case, for example, with 392 MPLS-based data planes. But any OAM-related implications or 393 requirements on data plane hardware must be considered. 395 3.8. #7 Time synchronization 397 Time synchronization between DetNet systems and nodes can be used to 398 enable fine grain scheduling of traffic along an end-to-end data 399 path. Such scheduling can be used to deliver very low jitter and 400 latency. [DetNet-ARCH] refers to a synchronization target of less 401 than 1 microsecond. Meeting such time synchronization objectives is 402 likely to require specific hardware support, both at the 403 synchronization protocol level and at the (time synchronized) packet 404 scheduling level. It is worth noting that certain aspects of time 405 synchronization and packet scheduling may be provided by the 406 underlying sub-net technology, e.g., [IEEE802.1Qbv] and 407 [IEEE802.1Qch]. 409 3.9. #8 Class and quality of service capabilities 411 Class and quality of service, i.e., CoS and QoS, are terms that are 412 often used interchangeably and confused. In the context of DetNet, 413 CoS is used to refer to mechanisms that provide traffic forwarding 414 treatment based on aggregate group basis and QoS is used to refer to 415 mechanisms that provide traffic forwarding treatment based on a 416 specific DetNet flow basis. Examples of CoS mechanisms include 417 DiffServ which is enabled by IP header differentiated services code 418 point (DSCP) field [RFC2474] and MPLS label traffic class field 419 [RFC5462], and at Layer-2, by IEEE 802.1p priority code point (PCP). 421 Quality of Service (QoS) mechanisms for flow specific traffic 422 treatment typically includes a guarantee/agreement for the service, 423 and allocation of resources to support the service. Example QoS 424 mechanisms include discrete resource allocation, admission control, 425 flow identification and isolation, and sometimes path control, 426 traffic protection, shaping, policing and remarking. Example 427 protocols that support QoS control include Resource ReSerVation 428 Protocol [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473]. 430 A critical DetNet service enabled by QoS (and perhaps CoS) is 431 delivering zero congestion loss. There are different mechanisms that 432 maybe used separately or in combination to deliver a zero congestion 433 loss service. The key aspect of this objective is that DetNet 434 packets are not discarded due to congestion at any point in a DetNet 435 aware network. 437 In the context of the data plane solution there should be means for 438 flow identification, which then can be used to map a flow against 439 specific resources and treatment in a node enforcing the QoS. 440 Hereto, certain aspects of CoS and QoS may be provided by the 441 underlying sub-net technology, e.g., actual queuing or IEEE 802.3x 442 priority flow control (PFC). 444 3.10. #9 Packet traceability 446 For the network management and specifically for tracing 447 implementation or network configuration errors any means to find out 448 whether a packet is a replica, which node performed replication, and 449 which path was intended for the replica, can be very useful. This 450 criteria concerns the availability of solutions for tracing packets 451 in the context of data plane protocol alternative. Packet trace is a 452 form of OAM. 454 3.11. #10 Technical maturity 456 The technical maturity of the data plane solution alternative is 457 crucial, since it basically defines the effort, time line and risks 458 involved for the use of the solution in deployments. For example, 459 the maturity level can be categorized as available immediately, 460 available with small extensions, available with repurposing/ 461 redefining portions of the protocol or its header fields. Yet 462 another important measure for maturity is the deployment experience. 463 This criteria concerns the maturity of the data plane protocol 464 alternative as the solution alternative. This criteria is 465 particularly important given, as previously noted, that the DetNet 466 data plane solution is expected to impact, i.e., be supported in, 467 hardware. 469 4. Data plane solution alternatives 471 The following sections describe and rate deterministic data plane 472 solution alternatives. In "Analysis and Discussion" section each 473 alternative is evaluated against the criteria given in Section 3 and 474 rated using the following: (M)eets the criteria, (W)ork needed, and 475 (N)ot suitable or too much work envisioned. 477 4.1. DetNet Transport layer technologies 479 4.1.1. Native IPv6 transport 481 4.1.1.1. Solution description 483 This section investigates the application of native IPv6 [RFC2460] as 484 the data plane for deterministic networking along the criteria 485 collected in Section 3. 487 The application of higher OSI layer headers, i.e., headers deeper in 488 the packet, can be considered. Two aspects have to be taken into 489 account for such solutions. (i) Those header fields can be encrypted. 490 (ii) Those header fields are deeper in the packet, therefore, routers 491 have to apply deep packet inspection. See further details in 492 Section 4.2.8. 494 4.1.1.2. Analysis and Discussion 496 Encapsulation and overhead (M/W) 498 The DetNet Service layer encapsulated DetNet flows are assumed to 499 be handled natively at layer-3 by IPv6 at the first place. The 500 fixed header of an IPv6 packet is 40 bytes [RFC2460]. This 501 overhead is bigger if any Extension Header is used, and a generic 502 behaviour for host and forwarding nodes is specified in [RFC7045]. 503 However, the exact overhead (Section 3.2) depends on what solution 504 is actually used to provide DetNet features, e.g., explicit 505 routing or seamless redundancy if any of these is applied. 507 IPv6 has two types of Extension Headers that are processed by 508 intermediate routers between the source and the final destination 509 and may be of interest for the data plane signaling, the Routing 510 Header that is used to direct the traffic via intermediate routers 511 in a strict or loose source routing way, and the Hop-by-Hop 512 Options Header that carries optional information that must be 513 examined by every node along a packet's delivery path. The Hop- 514 by-Hop Options Header, when present, must immediately follow the 515 IPv6 Header and it is not possible to limit its processing to the 516 end points of Source Routed segments. 518 IPv6 also provides a Destination Options Header that is used to 519 carry optional information to be examined only by a packet's 520 destination node(s). The encoding of the options used in the Hop- 521 by-Hop and in the Destination Options Header indicates the 522 expected behavior when a processing IPv6 node does not recognize 523 the Option Type, e.g. skip or drop; it should be noted that due to 524 performance restrictions nodes may ignore the Hop-by-Hop Option 525 Header, drop packets containing a Hop-by-Hop Option Header, or 526 assign packets containing a Hop-by-Hop Option Header to a slow 527 processing path [I-D.ietf-6man-rfc2460bis] (e.g. punt packets from 528 hardware to software forwarding which is highly detrimental to the 529 performance). 531 The creation of new Extension Headers that would need to be 532 processed by intermediate nodes is strongly discouraged. In 533 particular, new Extension Header(s) having hop-by-hop behavior 534 must not be created or specified. New options for the existing 535 Hop-by-Hop Header should not be created or specified unless no 536 alternative solution is feasible [RFC6564]. 538 Flow identification (M/W) 540 The 20-bit flow label field of the fixed IPv6 header is suitable 541 to distinguish different deterministic flows. But guidance on the 542 use of the flow label provided by [RFC6437] places restrictions on 543 how the flow label can be used. In particular, labels should be 544 chosen from an approximation to a discrete uniform distribution. 545 Additionally, existing implementations generally do not open APIs 546 to control the flow label from the upper layers. 548 Alternatively, the Flow identification could be transported in a 549 new option in the Hop-by-Hop Options Header. 551 Explicit routes (W) 553 The general assumption is that a Software-Defined Networking (SDN) 554 [RFC7426] based approach is applied to compute, establish and 555 manage the explicit routes, leveraging Traffic Engineering (TE) 556 extensions to routing protocols [RFC5305] 557 [I-D.ietf-idr-ls-distribution] and evolving to the Path 558 Computation Element (PCE) Architecture [RFC5440], though a number 559 of issues remain to be solved [RFC7399]. 561 Segment Routing (SR) [I-D.ietf-spring-segment-routing] is a new 562 initiative to equip IPv6 with explicit routing capabilities. The 563 idea for the DetNet data plane would be to apply SR to IPv6 with 564 the addition of a new type of routing extension header 566 [I-D.ietf-6man-segment-routing-header] to explicitly signal the 567 path in the data plane between the source and the destination, 568 and/or between replication points and elimination points if this 569 functionality is used. 571 Another concern regarding packet replication is how to enforce 572 replicated packets to take different route while the final 573 destination still remains the same. With strict source routing, 574 all the intermediate hops are listed and paths can be guaranteed 575 to be non-overlapping. Loose source routing only signals some of 576 the intermediate hops and it takes additional knowledge to ensure 577 that there is no single point of failure. 579 Packet replication (W) The functionality of replicating a packet 580 exists in IPv6 but is limited to multicast flows. 582 In order to enforce replicated packets to take different routes, 583 IP-in-IP encapsulation and Segment Routing could be leveraged to 584 signal a segment in a packet. A replication point would insert a 585 different routing header in each copy it makes, the routing header 586 providing explicitly the hops to the elimination point for that 587 particular replica of the packet, in a strict or in a loose source 588 routing fashion. An elimination point would pop the routing 589 headers from the various copies it gets and forward or receive the 590 packet if it is the final destination. 592 Operations, Administration and Maintenance (M/W) 594 IPv6 enjoys the existing toolbox for generic IP network 595 management. However, IPv6 specific management features are still 596 not at the level of that IPv4 has. This specifically concerns the 597 areas that are IPv6 specific, for example, related to neighbor 598 discovery protocol (ND), stateless address autoconfiguration 599 (SLAAC), subscriber identification, and security. While the 600 standards are already mostly in place the implementations in 601 deployed equipment can be lacking or inadequate for commercial 602 deployments. This is largely only an issue with old existing 603 equipment. 605 Class and quality of service capabilities (M) 607 The traffic class field of the fixed IPv6 header is designed for 608 this purpose. 610 Packet traceability (M/W/N) 611 The traceability of replicated packets involves the capability to 612 resolve which replication point issued a particular copy of a 613 packet, which segment was intended for that replica, and which 614 particular packet of which particular flow this is. Sequence also 615 depends on the sequencing mechanism. As an example, the 616 replication point may be indicated as the source of the packet if 617 IP-in-IP encapsulation is used to forward along segments. Another 618 alternate to IP-in-IP tunneling along segments would be to protect 619 the original source address in a destination option similar to the 620 Home Address option [RFC6275] and then use the address of the 621 replication point as source in the IP header. 623 The traceability also involves the capability to determine if a 624 particular segment is operational. While IPv6 as such has no 625 support for reversing a path, it appears source route extensions 626 such as the one defined for segment routing could be used for 627 tracing purposes. Though it is not a usual practice, IPv6 628 [RFC2460] expects that a Source Route path may be reversed, and 629 the standard insists that a node must not include the reverse of a 630 Routing Header in the response unless the received Routing Header 631 was authenticated. 633 Technical maturity (M/W) 635 IPv6 has been around about 20 years. However, large scale global 636 and commercial IPv6 deployments are rather new dating only few 637 years back to around 2012. While IPv6 has proven itself there are 638 number of small issues to work on as they show up once operations 639 experience grows. 641 The Cisco 6Lab site [1] provides information on IPv6 deployment 642 per country, indicating figures for prefixes, transit AS, content 643 and users. Per this site, many countries, including Canada, 644 Brazil, the USA, Germany, France, Japan, Portugal, Sweden, 645 Finland, Norway, Greece, and Ecuador, achieve a deployment ratio 646 above 30 percent, and the overall adoption reported by Google 647 Statistics [2] is now above 10 percent. 649 4.1.1.3. Summary 651 TBD. 653 4.1.2. Native IPv4 transport 654 4.1.2.1. Solution description 656 IPv4 [RFC0791] is in principle the same as IPv6, except that it has a 657 smaller address space. However, IPv6 was designed around the fact 658 that extension headers are an integral part of the protocol and 659 operation from the beginning, although the practice may some times 660 prove differently [I-D.ietf-v6ops-ipv6-ehs-in-real-world]. IPv4 661 never really needed any extension headers to work properly, thus 662 support for IPv4 options outside closed networks cannot typically be 663 guaranteed. In the context of deterministic networking data plane 664 solutions the major difference between IPv4 and IPv6 seems to be the 665 practical support for header extensibility. Anything below and above 666 the IP header independent of the version is practically the same. 668 4.1.2.2. Analysis and Discussion 670 Encapsulation and overhead (M) 672 The fixed header of an IPv4 packet is 20 bytes [RFC0791]. IP 673 options add overhead and the maximum IPv4 header size if 60 octets 674 leaving only 40 octets for possible options. 676 Flow identification (W/N) 678 The IPv4 header has a 16-bit identification field that was 679 originally intended for assisting fragmentation and reassembly of 680 IPv4 packets as described in [RFC0791]. The identification field 681 has also been proposed to be used for actually identifying flows 682 between two IP addresses and a given protocol for detecting and 683 removing duplicate packets [RFC1122]. However, recent update 684 [RFC6864] to both [RFC0791] and [RFC1122] restricts the use of 685 IPv4 identification field only to fragmentation purposes. 687 The IPv4 also has a stream identifier option [RFC0791], which 688 contains a 16-bit SATNET stream identifier. However, the option 689 has been deprecated [RFC6814]. The conclusion is that stream 690 identification does not work nicely with IPv4 header alone and a 691 traditional 5-tuple identification might not also be enough in a 692 case of a flow duplication. For a working solution upper layer 693 protocol headers such as the RTP are required for unambiguous flow 694 identification. 696 Explicit routes (M/W) 698 IPv4 has two source routing option specified: the loose source and 699 record route option (LSRR), and the strict source and record route 700 option (SSRR) [RFC0791]. The support of these options in the 701 Internet is questionable but within a closed network the support 702 may be assumed. 704 Packet replication (W/N) 706 The functionality of replicating a packet exists in IPv4 but is 707 limited to multicast flows. In general the issue regarding the 708 IPv6 packet replication also applies to IPv4. Duplicate packet 709 detection for IPv4 is studied in [RFC6621] to a great detail in 710 the context of simplified multicast forwarding. 712 Operations, Administration and Maintenance (M) 714 IPv4 enjoys the extensive and "complete" existing toolbox for 715 generic IP network management. 717 Class and quality of service capabilities (M) 719 The type of service (TOS) field of the fixed IPv4 header is 720 designed for this purpose. 722 Packet traceability (W/N) 724 The IPv4 has a traceroute option [RFC1393] that could be used to 725 record the route the packet took. However, the option has been 726 deprecated [RFC6814]. Similarly to IPv6 new work would be needed 727 to allow better traceability of IPv4 packets. 729 Technical maturity (M/W) 731 IPv4 can be considered mature technology with over 30 years of 732 implementation, deployment and operations experience. However, no 733 new IPv4 standards development is "allowed" anymore 734 [RFC6540][I-D.ietf-sunset4-gapanalysis]. 736 4.1.2.3. Summary 738 TBD. 740 4.1.3. Multiprotocol Label Switching (MPLS) 742 4.1.3.1. Solution description 744 Multiprotocol Label Switching Architecture (MPLS) [RFC3031] and its 745 variants, MPLS with Traffic Engineering (MPLS-TE) [RFC3209] and 746 [RFC3473], and MPLS Transport Profile (MPLS-TP) [RFC5921] is a widely 747 deployed technology that switches traffic based on MPLS label stacks 749 [RFC3032] and [RFC5960]. MPLS is the foundation for Pseudowire-based 750 services Section 4.2.5 and emerging technologies such as Bit-Indexed 751 Explicit Replication (BIER) Section 4.2.7.1 and Source Packet Routing 752 [3]. 754 MPLS supports the equivalent of both the DetNet Service and DetNet 755 Transport layers, and provides a very rich set of mechanisms that can 756 be reused directly, and perhaps augmented in certain cases, to 757 deliver DetNet services. At the DetNet Transport layer, MPLS 758 provides forwarding, protection and OAM services. At the DetNet 759 Service Layer it provides client service adaption, directly, via 760 Pseudowires Section 4.2.5 and via other label-like mechanisms such as 761 EPVN Section 4.2.6. A representation of these options are shown in 762 Figure 3. 764 PW-Based EVPN Labeled IP 765 Services Services Transport 766 |------------| |-----------------------------| |------------| 768 Emulated EVPN over LSP EVPN w/ ESI ID IP 769 Service 770 +------------+ 771 | Payload | 772 +------------+ +------------+ +------------+ (Service) 773 | PW Payload | | Payload | |ESI Lbl(S=1)| 774 +------------+ +------------+ +------------+ +------------+ 775 |PW Lbl(S=1) | |VPN Lbl(S=1)| |VPN Lbl(S=0)| | IP | 776 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 777 |LSP Lbl(S=0)| |LSP Lbl(S=0)| |LSP Lbl(S=0)| |LSP Lbl(S=1)| 778 +------------+ +------------+ +------------+ +------------+ 779 . . . . 780 . . . . (Transport) 781 . . . . 783 ~~~~~~~~~~~ denotes DetNet Service <-> DetNet Transport layer boundary 785 Figure 3: MPLS-based Services 787 MPLS can be controlled in a number of ways including via a control 788 plane, via the management plane, or via centralized controller (SDN) 789 based approaches. MPLS also provides standard control plane 790 reference points. Additional information on MPLS architecture and 791 control can be found in [RFC5921]. A summary of MPLS control plane 792 related functions can be found in [RFC6373]. The remainder of this 793 section will focus [RFC6373]. The remainder of this section will 794 focus on the MPLS transport data plane, additional information on the 795 MPLS service data plane can be found below in Section 4.2.4. 797 The following is a work in progress and draws heavily from [RFC5960] 798 and may be updated, replaced or removed. 800 Encapsulation and forwarding of packets traversing MPLS LSPs follows 801 standard MPLS packet encapsulation and forwarding as defined in 802 [RFC3031], [RFC3032], [RFC5331], and [RFC5332]. 804 Data plane Quality of Service capabilities are included in the MPLS 805 in the form of Traffic Engineered (TE) LSPs [RFC3209] and the MPLS 806 Differentiated Services (DiffServ) architecture [RFC3270]. Both 807 E-LSP and L-LSP MPLS DiffServ modes are defined. The Traffic Class 808 field (formerly the EXP field) of an MPLS label follows the 809 definition of [RFC5462] and [RFC3270]. 811 Except for transient packet reordering that may occur, for example, 812 during fault conditions, packets are delivered in order on L-LSPs, 813 and on E-LSPs within a specific ordered aggregate. 815 The Uniform, Pipe, and Short Pipe DiffServ tunneling and TTL 816 processing models are described in [RFC3270] and [RFC3443] and may be 817 used for MPLS LSPs. 819 Equal-Cost Multi-Path (ECMP) load-balancing is possible with MPLS 820 LSPs and can be avoided using a number of techniques. The same holds 821 for Penultimate Hop Popping (PHP). 823 MPLS includes the following LSP types: 825 o Point-to-point unidirectional 827 o Point-to-point associated bidirectional 829 o Point-to-point co-routed bidirectional 831 o Point-to-multipoint unidirectional 833 Point-to-point unidirectional LSPs are supported by the basic MPLS 834 architecture [RFC3031]. 836 A point-to-point associated bidirectional LSP between LSRs A and B 837 consists of two unidirectional point-to-point LSPs, one from A to B 838 and the other from B to A, which are regarded as a pair providing a 839 single logical bidirectional transport path. 841 A point-to-point co-routed bidirectional LSP is a point-to-point 842 associated bidirectional LSP with the additional constraint that its 843 two unidirectional component LSPs in each direction follow the same 844 path (in terms of both nodes and links). An important property of 845 co-routed bidirectional LSPs is that their unidirectional component 846 LSPs share fate. 848 A point-to-multipoint unidirectional LSP functions in the same manner 849 in the data plane, with respect to basic label processing and packet- 850 switching operations, as a point-to-point unidirectional LSP, with 851 one difference: an LSR may have more than one (egress interface, 852 outgoing label) pair associated with the LSP, and any packet it 853 transmits on the LSP is transmitted out all associated egress 854 interfaces. Point-to-multipoint LSPs are described in [RFC4875] and 855 [RFC5332]. TTL processing and exception handling for point-to- 856 multipoint LSPs is the same as for point-to-point LSPs. 858 Additional data plane capabilities include Linear Protection, 859 [RFC6378] and [RFC7271]. And the in progress work on MPLS support 860 for time synchronization [I-D.ietf-mpls-residence-time]. 862 4.1.3.2. Analysis and Discussion 864 #? DetNet Service Interface (M) 866 The DetNet service interface is enabled through the DetNet Service 867 Layer it provides client service adaption, directly, via 868 Pseudowires Section 4.2.5 and via other label-like mechanisms such 869 as EPVN Section 4.2.6. 871 #1 Encapsulation and overhead (M) 873 There are two perspectives to consider when looking at 874 encapsulation. The first is encapsulation to support services. 875 These considerations are part of the DetNet service layer and are 876 covered below, see Sections 4.2.5 and 4.2.6. 878 The second perspective relates to encapsulation, if any, is needed 879 to transport packets across network. In this case, the MPLS label 880 stack, [RFC3032] is used to identify flows across a network. MPLS 881 labels are compact and highly flexible. They can be stacked to 882 support client adaptation, protection, network layering, source 883 routing, etc. 885 #2 Flow identification (M) 887 MPLS label stacks provide highly flexible ways to identify flows. 888 Basically, they enable the complete separation of traffic 889 classification from traffic treatment and thereby enable arbitrary 890 combinations of both. 892 #3 Packet sequencing (M) 894 Packet ordering in MPLS is generally similar to packet ordering in 895 Ethernet. MPLS implementations can also support ECMP for certain 896 types of traffic which can to lead to out of order delivery. 897 There are defined techniques to avoid ECMP and ensure in order 898 delivery during normal operation. Out of order delivery is still 899 possible in certain MPLS protection scenarios. If additional 900 ordering mechanisms are required, these are likely to be 901 implemented at the DetNet Service Layer. 903 #4 Explicit routes (M) 905 MPLS supports explicit routes based on how LSPs are established, 906 e.g., via TE explicit routes [RFC3209]. Additional, but not 907 required, additional capabilities are being defined as part of 908 Segment Routing (SR) [I-D.ietf-spring-segment-routing]. 910 #5 Packet replication and deletion (M/W) 912 At the MPLS LSP level, there are mechanisms defined to provide 1+1 913 protection. The current definitions [RFC6378] and [RFC7271] use 914 OAM mechanisms to support and coordinate protection switching and 915 packet loss is possible during a switch. While such this level of 916 protection may be sufficient for man DetNet applications, when 917 truly hitless (i.e., zero loss) switching is required additional 918 mechanisms will be needed. It is expected that these additional 919 mechanisms will be defined at the DetNet Service Layer. 921 #6 Operations, Administration and Maintenance (M) 923 MPLS already includes a rich set of OAM functions at both the 924 Service and Transport Layers. This includes LSP ping [ref] and 925 those enabled via the MPLS Generic Associated Channel [RFC5586] 926 and registered by IANA [4]. 928 #7 Time synchronization (M/W) 930 MPLS itself does not provide any time synchronization service. 931 The expectations is that the actual time-based scheduling will be 932 provided by the sub-network layer, e.g., by [TSNTG], and that the 933 DetNet transport layer will merely need to facilitate time 934 synchronization (with hardware support) across multiple sub- 935 network domains and technologies. Work is in progress 937 [I-D.ietf-mpls-residence-time] that may satisfy, or serve as a 938 building block for, DetNet time synchronization. 940 #8 Class and quality of service capabilities (M/W) 942 As previously mentioned, Data plane Quality of Service 943 capabilities are included in the MPLS in the form of Traffic 944 Engineered (TE) LSPs [RFC3209] and the MPLS Differentiated 945 Services (DiffServ) architecture [RFC3270]. Both E-LSP and L-LSP 946 MPLS DiffServ modes are defined. The Traffic Class field 947 (formerly the EXP field) of an MPLS label follows the definition 948 of [RFC5462] and [RFC3270]. One potential open area of work is 949 synchronized, time based scheduling. 951 #9 Packet traceability (M) 953 MPLS supports multiple tracing mechanisms. A control based one is 954 defined in [RFC3209]. An OAM based mechanism is defined in MPLS 955 On-Demand Connectivity Verification and Route Tracing [RFC6426]. 957 #10 Technical maturity (M) 959 MPLS as a mature technology that has been widely deployed in many 960 networks for many years. Numerous vendor products and multiple 961 generations of MPLS hardware have been built and deployed. 963 4.1.3.3. Summary 965 MPLS is a mature technology that has been widely deployed. Numerous 966 vendor products and multiple generations of MPLS hardware have been 967 built and deployed. MPLS LSPs support a significant portion of the 968 identified DetNet data plane criteria today. Aspects of the DetNet 969 data plane that are not fully supported can be incrementally added. 971 4.2. DetNet Service layer technologies 973 4.2.1. Generic Routing Encapsulation (GRE) 975 4.2.1.1. Solution description 977 Generic Routing Encapsulation (GRE) [RFC2784] provides an 978 encapsulation of an arbitrary network layer protocol over another 979 arbitrary network layer protocol. The encapsulation of a GRE packet 980 can be found in Figure 4. 982 +-------------------------------+ 983 | | 984 | Delivery Header | 985 | | 986 +-------------------------------+ 987 | | 988 | GRE Header | 989 | | 990 +-------------------------------+ 991 | | 992 | Payload packet | 993 | | 994 +-------------------------------+ 996 Figure 4: Encapsulation of a GRE packet 998 Based on RFC2784, [RFC2890] further includes sequencing number and 999 Key in optional fields of the GRE header, which may help to transport 1000 DetNet traffic flows over IP networks. The format of a GRE header is 1001 presented in Figure 5. 1003 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 1004 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1005 |C| |K|S| Reserved0 | Ver | Protocol Type | 1006 +-----------------------------------------------------------------+ 1007 | Checksum (optional) | Reserved1 (Optional) | 1008 +-----------------------------------------------------------------+ 1009 | Key (optional) | 1010 +-----------------------------------------------------------------+ 1011 | Sequence Number (optional) | 1012 +-----------------------------------------------------------------+ 1014 Figure 5: Format of a GRE header 1016 4.2.1.2. Analysis and Discussion 1018 Encapsulation and overhead (M) 1020 GRE provides encapsulation for a network layer protocol over 1021 anther network layer protocol. A new protocol type for DetNet 1022 traffics should be allocated as an "Ether Type" in [RFC1700] and 1023 in IANA Ethernet Numbers. [5] The fixed header of a GRE packet is 1024 4 octets while the maximum header is 16 octets with optional 1025 fields in Figure 5. 1027 Flow identification (W) 1029 There is no flow identification field in GRE header. However, it 1030 can rely on the flow identification mechanism applied in the 1031 delivery protocols, such as flow identification stated in IP 1032 Sections 4.1.1 and 4.1.2 when the delivery protocols are IPv6 and 1033 IPv4 respectively. Alternatively, the Key field can also be 1034 extended to carry the flow identification. The size of Key field 1035 is 4 octets. 1037 Packet sequencing (M) 1039 As stated in Section 4.2.1, GRE provides an optional sequencing 1040 number in its header to provide sequencing services for packets. 1041 The size of the sequencing number is 32 bits. 1043 Duplicate packet deletion (N) 1045 GRE has no packet replication and deletion currently in its header 1046 and should be extended or rely on delivery protocols. 1048 Operations, Administration and Maintenance (W/N) 1050 [note: rely on the delivery protocol] GRE has no packet 1051 replication and deletion currently and should be relied on 1052 delivery protocols. 1054 Time synchronization (W/N) 1056 [note: rely on the delivery protocol] GRE has no packet 1057 replication and deletion currently and should be relied on 1058 delivery protocols. 1060 Class and quality of service capabilities (W/N) 1062 [note: rely on the delivery protocol] GRE has no packet 1063 replication and deletion currently and should be relied on 1064 delivery protocols. For the class of service capability, an 1065 optional code point field to indicate CoS of a traffic can be 1066 extended in GRE header. 1068 Technical maturity (M) 1070 GRE has been developed over 20 years. The delivery protocol 1071 mostly used is IPv4, while the IPv6 support for GRE is to be 1072 standardized now in IETF as [I-D.ietf-intarea-gre-ipv6]. Due to 1073 its good extensibility, GRE is also extended to support network 1074 virtualization in Data Center, which is NVGRE [RFC7637]. 1076 4.2.1.3. Summary 1078 TBD. 1080 4.2.2. Layer-2 Tunneling Protocol (L2TP) 1082 [Editor's note: L2TPv3 [RFC3931] content to be provided later, if 1083 needed] 1085 4.2.3. Virtual Extensible LAN (VXLAN) 1087 [Editor's note: VXLAN [RFC7348] content to be provided later, if 1088 needed] 1090 4.2.4. MPLS-based Services 1092 4.2.4.1. Solution description 1094 MPLS supports the equivalent of both the DetNet Service and DetNet 1095 Transport layers. This, as well as a general overview of MPLS, is 1096 covered above in Section 4.1.3. This section will focus on the 1097 DetNet Service Layer it provides client service adaption, via 1098 Pseudowires in Section 4.2.5 and via native and other label-like 1099 mechanisms such as EPVN in Section 4.2.6. A representation of these 1100 options was previously discussed and is shown in Figure 3. 1102 4.2.4.2. Analysis and Discussion 1104 #? DetNet Service Interface (M) 1106 The following text is adapted from [RFC5921]: 1108 The MPLS native service adaptation functions interface the client 1109 layer network service to MPLS. For Pseudowires, these adaptation 1110 functions are the payload encapsulation described in Section 4.4 1111 of [RFC3985] and Section 6 of [RFC5659]. For network layer client 1112 services, the adaptation function uses the MPLS encapsulation 1113 format as defined in [RFC3032]. 1115 The purpose of this encapsulation is to abstract the data plane of 1116 the client layer network from the MPLS data plane, thus 1117 contributing to the independent operation of the MPLS network. 1119 MPLS may itself be a client of an underlying server layer. MPLS 1120 can thus also bounded by a set of adaptation functions to this 1121 server layer network, which may itself be MPLS. These adaptation 1122 functions provide encapsulation of the MPLS frames and for the 1123 transparent transport of those frames over the server layer 1124 network. 1126 While MPLS service can provided on and true end-system to end- 1127 system basis, it's more likely that DetNet service will be 1128 provided over Pseudowires as described in Section 4.2.5 or via an 1129 EPVN-based service described in Section 4.2.6 . 1131 #1 Encapsulation and overhead (M) 1133 MPLS labels in the label stack may be used to identify transport 1134 paths, see Section 4.1.3, or as service identifiers. Typically a 1135 single label is used for service identification. Additional 1136 details on how client adaptation makes use of such labels is part 1137 of actual client-related adaptation processing, see Sections 4.2.5 1138 and 4.2.6. 1140 #2 Flow identification (M) 1142 This is basically the same as MPLS at the DetNet transport layer. 1143 MPLS label stacks provide highly flexible ways to identify flows. 1144 Basically, they enable the complete separation of traffic 1145 classification from traffic treatment and thereby enable arbitrary 1146 combinations of both. Typically a separate label will be added 1147 per service being supported by a node. 1149 #3 Packet sequencing (M) 1151 This is the same as MPLS at the DetNet transport layer. If 1152 additional ordering mechanisms are required, these will be needed 1153 (and added) in client-related adaptation processing, see Sections 1154 4.2.5 and 4.2.6. 1156 #4 Explicit routes (N/A) 1158 Explicit routes are part of the DetNet transport layer, see 1159 Section 4.2.6, or as part of multi-segment PWEs, Section 4.2.5. 1161 #5 Packet replication and deletion (M/W) 1163 This is the same as MPLS at the DetNet transport layer. 1164 Additional capability may also be provided as part of client- 1165 related adaptation processing see Section 4.2.5. 1167 #6 Operations, Administration and Maintenance (M) 1169 This is the same as MPLS at the DetNet transport layer. 1170 Additional capability may also be provided as part of client- 1171 related adaptation processing. 1173 #7 Time synchronization (TBD) 1175 It's unclear at this time if any additional capability is needed 1176 at this level. 1178 #8 Class and quality of service capabilities (M/W) 1180 The MPLS client inherits its Quality of Service (QoS) from the 1181 MPLS transport layer, which in turn inherits its QoS from the 1182 server (sub-network) layer. The server layer therefore needs to 1183 provide the necessary QoS to ensure that the MPLS client QoS 1184 commitments can be satisfied. 1186 #9 Packet traceability (M) 1188 This is the same as MPLS at the DetNet transport layer. 1190 #10 Technical maturity (M) 1192 This is the same as MPLS at the DetNet transport layer. 1194 4.2.4.3. Summary 1196 This is the same as MPLS at the DetNet transport layer. MPLS is a 1197 mature technology that has been widely deployed. Numerous vendor 1198 products and multiple generations of MPLS hardware have been built 1199 and deployed. MPLS LSPs support a significant portion of the 1200 identified DetNet data plane criteria today. Aspects of the DetNet 1201 data plane that are not fully supported can be incrementally added. 1203 4.2.5. Pseudo Wire Emulation Edge-to-Edge (PWE3) 1204 4.2.5.1. Solution description 1206 PSeudo Wire Emulation Edge-to-Edge (PWE3) [RFC3985] or simply 1207 PseudoWires (PW) provide means of emulating the essential attributes 1208 and behaviour of a telecommunications service over a packet switched 1209 network (PSN) using IP or MPLS transport. In addition to traditional 1210 telecommunications services such as T1 line or Frame Relay, PWs also 1211 provide transport for Ethernet service [RFC4448] and for generic 1212 packet service [RFC6658]. Figure 6 illustrate the reference PWE3 1213 stack model. 1215 +----------------+ +----------------+ 1216 |Emulated Service| |Emulated Service| 1217 |(e.g., Eth, ...)|<= Emulated Service =>|(e.g., Eth, ...)| 1218 +----------------+ +----------------+ 1219 | Payload | | Payload | CW, 1220 | Encapsulation |<=== Pseudo Wire ====>| Encapsulation | Timing, 1221 | | | | Seq., .. 1222 +----------------+ +----------------+ 1223 |PW Demultiplexer| |PW Demultiplexer| 1224 | PSN Tunnel, |<==== PSN Tunnel ====>| PSN Tunnel, | MPLS. 1225 | PSN & Physical | | PSN & Physical | L2TP, 1226 | Layers | | Layers | IP, .. 1227 +-------+--------+ ___________ +---------+------+ 1228 | / \ | 1229 +============/ PSN \==============+ 1230 \ / 1231 \___________/ 1233 Figure 6: PWE3 protocol stack reference model 1235 PWs appear as a good data plane solution alternative for a number of 1236 reasons. PWs are a proven and deployed technology with a rich OAM 1237 control plane [RFC4447], and enjoy the toolbox developed for MPLS 1238 networks. Furthermore, PWs may have an optional Control Word (CW) as 1239 part of the payload encapsulation between the PSN and the emulated 1240 service that is, for example, capable of frame sequencing and 1241 duplicate detection. The encapsulation layer may also provide timing 1242 [RFC5087]. 1244 PWs can be also used if the PSN is IP, which enables the application 1245 of PWs in networks that do not have MPLS enabled in their core 1246 routers. One approach to provide PWs over IP is to provide MPLS over 1247 IP in some way and then leverage what is available for PWs over MPLS. 1248 The following standard solutions are available both for IPv4 and IPv6 1249 to follow this approach. The different solutions have different 1250 overhead as discussed in the following subsection. The MPLS-in-IP 1251 encapsulation is specified by [RFC4023]. The IPv4 Protocol Number 1252 field or the IPv6 Next Header field is set to 137, which indicates an 1253 MPLS unicast packet. (The use of the MPLS-in-IP encapsulation for 1254 MPLS multicast packets is not supported.) The MPLS-in-GRE 1255 encapsulation is specified in [RFC4023], where the IP header (either 1256 IPv4 or IPv6) is followed by a GRE header, which is followed by an 1257 MPLS label stack. The protocol type field in the GRE header is set 1258 to MPLS Unicast (0x8847) or Multicast (0x8848). MPLS over L2TPv3 1259 over IP encapsulation is specified by [RFC4817]. The MPLS-in-UDP 1260 encapsulation is specified by [RFC7510], where the UDP Destination 1261 Port indicates tunneled MPLS packet and the UDP Source Port is an 1262 entropy value that is generated by the encapsulator to uniquely 1263 identify a flow. MPLS-in-UDP encapsulation can be applied to enable 1264 UDP-based ECMP (Equal-Cost Multipath) or Link Aggregation. All these 1265 solutions can be secured with IPSec. 1267 4.2.5.2. Analysis and Discussion 1269 Encapsulation and overhead (M) 1271 PWs offer encapsulation services practically for any types of 1272 payloads over any PSN. New PW types need a code point allocation 1273 [RFC4446] and in some cases an emulated service specific document. 1275 Specifically in the case of the MPLS PSN the PW encapsulation 1276 overhead is minimal. Typically minimum two labels and a CW is 1277 needed, which totals to 12 octets. PW type specific handling 1278 might, however, allow optimizations on the emulated service in the 1279 provider edge (PE) device's native service processing (NSP) / 1280 forwarder function. These optimizations could be used, for 1281 example, to reduce header overhead. Ethernet PWs already have 1282 rather low overhead [RFC4448]. Without a CW and VLAN tags the 1283 Ethernet header gets reduced to 14 octets (minimum Ethernet header 1284 overhead is 26). 1286 The overhead is somewhat bigger in case of IP PSN if an MPLS over 1287 IP solution is applied to provide PWs. IP adds at least 20 (IPv4) 1288 or 40 (IPv6) bytes overhead to the PW over MPLS overhead; 1289 furthermore, the GRE, L2TPv3, or UDP header has to be taken into 1290 account if any of these further encapsulations is used. 1292 Flow identification (M) 1294 [Editor's note: this criteria has not been checked against the 1295 latest view of flow identification after the separation of 1296 transport and service layers.] 1297 PWs provide multiple layers of flow identification, especially in 1298 the case of the MPLS PSN. The PWs are typically prepended with a 1299 PW label that can be used to identify a specific PW. Furthermore, 1300 the PSN also uses one or more labels to transport packets over a 1301 specific label switched paths (that then would carry PWs). IP 1302 (and other) PSNs may need other mechanisms, such as, UDP port 1303 numbers, upper layer protocol header (like RTP) or some IP 1304 extension header to provide required flow identification. 1306 Packet sequencing (M) 1308 As mentioned earlier PWs may contain an optional CW that is able 1309 to provide sequencing services. The size of the sequence number 1310 in the generic CW is 16 bits, which might be, depending on the 1311 used link and DetNet flow speed be too little. 1313 Duplicate packet deletion (W) 1315 The PW duplicate detection mechanism also exists in theory 1316 [RFC3985] but no emulated service makes use of it currently. 1318 Operations, Administration and Maintenance (M/W) 1320 PWs have rich control plane for OAM and in a case of the MPLS PSN 1321 enjoy the full control plane toolbox developed for MPLS network 1322 OAM likewise IP PSN have the full toolbox of IP network OAM tools. 1323 There could be, however, need for deterministic networking 1324 specific extensions for the mentioned control planes. 1326 Time synchronization (M/W) 1328 It is possible to carry time synchronization information as part 1329 of the PW encapsulation layer (see for example [RFC5087]). 1330 Whether the timing precision is enough for all deterministic 1331 networking use cases vary, and it is possible existing mechanisms 1332 are not adequate for all use cases. IP PSNs have already 1333 demonstrated the use of time synchronization as a part of PWE3 1334 [RFC5086]. 1336 Class and quality of service capabilities (M) 1338 In a case of IP PSN the 6-bit differentiated services code point 1339 (DSCP) field can be used for indicating the class of service 1340 [RFC2474] and 2-bit field reserved for the explicit congestion 1341 notification (ECN) [RFC3168]. Similarly, in a case of MPLS PSN, 1342 there are 3-bit traffic class field (TC) [RFC5462] in the label 1343 reserved for for both Explicitly TC-encoded-PSC LSPs (E-LSP) 1344 [RFC3270] and ECN [RFC5129]. Due to the limited number of bits in 1345 the TC field, their use for QoS and ECN functions restricted and 1346 intended to be flexible. Although the QoS/CoS mechanism is 1347 already in place some clarifications may be required in the 1348 context of deterministic networking flows, for example, if some 1349 specific mapping between bit fields have to be done. 1351 Technical maturity (M) 1353 PWs, IP and MPLS are proven technologies with wide variety of 1354 deployments and years of operational experience. Furthermore, the 1355 estimated work for missing functionality (packet replication and 1356 deletion) does not appear to be extensive, since the existing 1357 protection mechanism already get close to what is needed from the 1358 deterministic networking data plane solution. 1360 4.2.5.3. Summary 1362 PseudoWires appear to be a strong candidate as the deterministic 1363 networking data plane solution alternative for the DetNet Service 1364 layer. The strong points are the technical maturity and the 1365 extensive control plane for OAM. This holds specifically for MPLS- 1366 based PSN. 1368 Extensions are required to realize the packet replication and 1369 duplicate detection features of the deterministic networking data 1370 plane. 1372 4.2.6. MPLS-Based Ethernet VPN (EVPN) 1374 4.2.6.1. Solution description 1376 MPLS-Based Ethernet VPN (EVPN), in the form documented in [RFC7432] 1377 and [RFC7209], is an increasingly popular approach to delivering 1378 MPLS-based Ethernet services and is designed to be the successor to 1379 Virtual Private LAN Service (VPLS), [RFC4664]. 1381 EVPN provides client adaptation and reuses the MPLS data plane 1382 discussed above in Section 4.2.4. In certain special cases, it also 1383 uses the PW MPLS Control Word. EVPN control is via BGP, [RFC7432], 1384 and may use TE-LSPs, e.g., controlled via [RFC3209] for MPLS 1385 transport. Additional EVPN related RFCs and in progress drafts are 1386 being developed by the BGP Enabled Services Working Group [6]. 1388 4.2.6.2. Analysis and Discussion 1390 #? DetNet Service Interface (M/W) 1392 The service supported by EVPN is a layer 2 Ethernet virtual 1393 private network. While EVPN is typically envisioned to be 1394 deployed on provider edge systems, it is also possible to extent 1395 the EVPN service to a DetNet end or edge system if such service is 1396 needed. 1398 #1 Encapsulation and overhead (M) 1400 EVPN generally uses a single MPLS label stack entry to support its 1401 client adaptation service. The optional addition of a second 1402 label is also supported. In certain cases PW Control Word may 1403 also be used. 1405 #2 Flow identification (W) 1407 EVPN currently uses labels to identify flows per {Ethernet Segment 1408 Identifier, VLAN} or per MAC level. Additional definition will be 1409 needed to standardize identification of finer granularity DetNet 1410 flows. 1412 #3 Packet sequencing (M/W) 1414 Like MPLS, EVPN generally orders packets similar to Ethernet. 1415 Reordering is possible primarily during path changes and 1416 protection switching. In order to avoid misordering due to ECMP, 1417 EVPN uses the "Preferred PW MPLS Control Word" [RFC4385] or the 1418 entropy labels [RFC6790]. 1420 If additional ordering mechanisms are required, such mechanisms 1421 will need to be defined. 1423 #4 Explicit routes (M) 1425 EVPN itself doesn't offer support for explicit routes as it is 1426 simply an adaptation function. Explicit routes for EVPN at the 1427 DetNet transport layer would be provided via MPLS. 1429 #5 Packet replication and deletion (M/W) 1430 EVPN relies on the MPLS layer for all protection functions. See 1431 Section 4.1.3 and Section 4.2.4. Some extensions, either at the 1432 EVPN or MPLS levels, will be need to support those DetNet 1433 applications which require true hitless (i.e., zero loss) 1+1 1434 protection switching. (Network coding may be an interesting 1435 alternative to investigate to delivering such hitless loss 1436 protection capability.) 1438 #6 Operations, Administration and Maintenance (M/W) 1440 Nodes supporting EVPN may participate in either or both Ethernet 1441 level and MPLS level OAM. It is likely that it may make sense to 1442 map or adapt the OAM functions at the different levels, but such 1443 has yet to be defined. [RFC6371] provides some useful background 1444 on this topic. 1446 #7 Time synchronization (W) 1448 The interface to the DetNet time synchronization service is still 1449 to be determined. If the service is accessed by end systems via 1450 IEEE defined mechanisms, then those mechanisms will need to be 1451 mapped to the MPLS provided mechanisms discussed in Section 4.1.3. 1453 #8 Class and quality of service capabilities (M/W) 1455 EVPN is largely silent on the topics of CoS and QoS, but the 1456 existing Ethernet and MPLS mechanisms can be directly used. While 1457 an implementation may support such mappings today, standardized 1458 mappings do not (yet) exist. 1460 #9 Packet traceability (M) 1462 EVPN nodes can utilize MPLS layers tracing mechanisms. 1464 #10 Technical maturity 1466 EVPN is a second (or third) generation MPLS-based L2VPN service 1467 standard. From a data plane standpoint it makes uses of existing 1468 MPLS data plane mechanisms. The mechanisms have been widely 1469 implemented and deployed. 1471 4.2.6.3. Summary 1473 EVPN is the emerging successor to VPLS. EVPN is standardized, 1474 implemented and deployed. It makes use of the mature MPLS data 1475 plane. While offering a mature and very comprehensive set of 1476 features, certain DetNet required features are not fully/directly 1477 supported and additional standardization in these areas are needed. 1478 Examples include: mapping CoS and QoS; use of labels per DetNet flow, 1479 and hitless 1+1 protection. 1481 4.2.7. Bit Indexed Explicit Replication (BIER) 1483 Bit Indexed Explicit Replication [I-D.ietf-bier-architecture] (BIER) 1484 is a network plane replication technique that was initially intended 1485 as a new method for multicast distribution. In a nutshell, a BIER 1486 header includes a bitmap that explicitly signals the listeners that 1487 are intended for a particular packet, which means that 1) the sender 1488 is aware of the individual listeners and 2) the BIER control plane is 1489 a simple extension of the unicast routing as opposed to a dedicated 1490 multicast data plane, which represents a considerable reduction in 1491 OPEX. For this reason, the technology faces a lot of traction from 1492 Service Providers. Section 4.2.7.1 discusses the applicability of 1493 BIER for replication in the DetNet. 1495 The simplicity of the BIER technology makes it very versatile as a 1496 network plane signaling protocol. Already, a new Traffic Engineering 1497 variation is emerging that uses bits to signal segments along a TE 1498 path. While the more classical BIER is mainly a multicast technology 1499 that typically leverages a unicast distributed control plane through 1500 IGP extensions, BIER-TE is mainly a unicast technology that leverages 1501 a central computation to setup path, compute segments and install the 1502 mapping in the intermediate nodes. Section 4.2.7.2 discusses the 1503 applicability of BIER-TE for replication, traceability and OAM 1504 operations in DetNet. 1506 4.2.7.1. Base BIER 1508 Bit-Indexed Explicit Replication (BIER) layer may be considered to be 1509 included into Deterministic Networking data plane solution. 1510 Encapsulation of a BIER packet in MPLS network presented in Figure 7 1511 0 1 2 3 1512 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1513 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1514 | Label Stack Element | 1515 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1516 | Label Stack Element | 1517 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1518 | BIER-MPLS label | |1| | 1519 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1520 |0 1 0 1| Ver | Len | Entropy | 1521 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1522 | BitString (first 32 bits) ~ 1523 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1524 ~ ~ 1525 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1526 ~ BitString (last 32 bits) | 1527 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1528 |OAM| Reserved | Proto | BFIR-id | 1529 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1531 Figure 7: BIER packet in MPLS encapsulation 1533 4.2.7.1.1. Solution description 1535 The DetNet may be presented in BIER as distinctive payload type with 1536 its own Proto(col) ID. Then it is likely that DetNet will have the 1537 header that would identify: 1539 o Version; 1540 o Sequence Number; 1541 o Timestamp; 1542 o Payload type, e.g. data vs. OAM. 1544 DetNet node, collocated with BFIR, may use multiple BIER sub-domains 1545 to create replicated flows. Downstream DetNet nodes, collocated with 1546 BFER, would terminate redundant flows based on Sequence Number and/or 1547 Timestamp information. Such DetNet may be BFER in one BIER sub- 1548 domain and BFIR in another. Thus DetNet flow would traverse several 1549 BIER sub-domains. 1551 +-----+ 1552 | A | 1553 +-----+ 1554 / \ 1555 . . 1556 / . 1557 . \ 1558 / . 1559 . . 1560 / \ 1561 +-----+ +-----+ 1562 | B | | C | 1563 +-----+ +-----+ 1564 / \ / \ 1565 . . . . 1566 / \ . . 1567 . . / \ 1568 / \ . . 1569 . . . . 1570 / \ / \ 1571 +-----+ +-----+ +-----+ 1572 | D | | E | | F | 1573 +-----+ +-----+ +-----+ 1574 \ . . / 1575 . . . . 1576 \ . . . 1577 . . . / 1578 \ . . . 1579 . . . . 1580 \ . . / 1581 +-----+ +-----+ 1582 | G | | H | 1583 +-----+ +-----+ 1585 Figure 8: DetNet in BIER domain 1587 Consider DetNet flow that must traverse BIER enabled domain from A to 1588 G and H. DetNet may use three BIER subdomains: 1590 o A-B-D-E-G (dash-dot): A is BFIR, E and G are BFERs, 1591 o A-C-E-F-H (dash-double-dot): A is BFIR, E and H are BFERs, 1592 o E-G-H (dotted): E is BFIR, G and H are BFERs. 1594 DetNet node A sends DetNet into red and purple BIER sub-domains. 1595 DetNet node E receives DetNet packet and sends into green sub-domain 1596 while terminating duplicates and those that deemed too-late. 1598 DetNet nodes G and H receive DetNet flows, terminate duplicates and 1599 those that are too-late. 1601 4.2.7.1.2. Analysis and Discussion 1603 4.2.7.1.3. Summary 1605 4.2.7.2. BIER - Traffic Engineering 1607 An alternate use of Bit-Indexed Explicit Replication (BIER) uses bits 1608 in the BitString to represent adjacencies as opposed to destinations, 1609 as discussed in BIER Traffic Engineering (TE) 1610 [I-D.eckert-bier-te-arch]. 1612 The proposed function of BIER-TE in the DetNet data plane is to 1613 control the process of replication and elimination, as opposed to the 1614 identification of the flows or and the sequencing of packets within a 1615 flow. 1617 At the path ingress, BIER-TE identifies the adjacencies that are 1618 activated for this packet (under the rule of the controller). At the 1619 egress, BIER-TE is used to identify the adjacencies where 1620 transmission failed. This information is passed to the controller, 1621 which in turn can modify the active adjacencies for the next packets. 1623 The value is that the replication can be controlled and monitored 1624 with the granularity of a packet and a adjacency in a control loops 1625 that involves an external controller. 1627 4.2.7.2.1. Solution description 1629 BIER-TE enables to activate the replication and elimination functions 1630 in a manner that is abstract to the data plane forwarding 1631 information. An adjacency, which is represented by a bit in the BIER 1632 header, can correspond in the data plane to an Ethernet hop, a Label 1633 Switched Path, or it can correspond to an IPv6 loose or strict source 1634 routed path. 1636 In a nutshell, BIER-TE is used as follows: 1638 o A controller computes a complex path, sometimes called a track, 1639 which takes the general form of a ladder. The steps and the side 1640 rails between them are the adjacencies that can be activated on 1641 demand on a per-packet basis using bits in the BIER header. 1643 ===> (A) ====> (C) ==== 1644 // ^ | ^ | \\ 1645 ingress (I) | | | | (E) egress 1646 \\ | v | v // 1647 ===> (B) ====> (D) ==== 1649 Figure 9: Ladder Shape with Replication and Elimination Points 1651 o The controller assigns a BIER domain, and inside that domain, 1652 assigns bits to the adjacencies. The controller assigns each bit 1653 to a replication node that sends towards the adjacency, for 1654 instance the ingress router into a segment that will insert a 1655 routing header in the packet. A single bit may be used for a step 1656 in the ladder, indicating the other end of the step in both 1657 directions. 1659 ===> (A) ====> (C) ==== 1660 // 1 ^ | 4 ^ | 7 \\ 1661 ingress (I) |2| |6| (E) egress 1662 \\ 3 | v 5 | v 8 // 1663 ===> (B) ====> (D) ==== 1665 Figure 10: Assigning Bits 1667 o The controller activates the replication by deciding the setting 1668 of the bits associated with the adjacencies. This decision can be 1669 modified at any time, but takes the latency of a controller round 1670 trip to effectively take place. Below is an example that uses 1671 Replication and Elimination to protect the A->C adjacency. 1673 +-------+-----------+-------+---------------------+ 1674 | Bit # | Adjacency | Owner | Example Bit Setting | 1675 +-------+-----------+-------+---------------------+ 1676 | 1 | I->A | I | 1 | 1677 | 2 | A->B | A | 1 | 1678 | | B->A | B | | 1679 | 3 | I->C | I | 0 | 1680 | 4 | A->C | A | 1 | 1681 | 5 | B->D | B | 1 | 1682 | 6 | C->D | C | 1 | 1683 | | D->C | D | | 1684 | 7 | C->E | C | 1 | 1685 | 8 | D->E | D | 0 | 1686 +-------+-----------+-------+---------------------+ 1688 Replication and Elimination Protecting A->C 1690 Table 1: Controlling Replication 1692 o The BIER header with the controlling BitString is injected in the 1693 packet by the ingress node of the deterministic path. That node 1694 may act as a replication point, in which case it may issue 1695 multiple copies of the packet 1697 ====> Repl ===> Elim ==== 1698 // | ^ \\ 1699 ingress | | egress 1700 v | 1701 Fwd ====> Fwd 1703 Figure 11: Enabled Adjacencies 1705 o For each of its bits that is set in the BIER header, the owner 1706 replication point resets the bit and transmits towards the 1707 associated adjacency; to achieve this, the replication point 1708 copies the packet and inserts the relevant data plane information, 1709 such as a source route header, towards the adjacency that 1710 corresponds to the bit 1711 +-----------+----------------+ 1712 | Adjacency | BIER BitString | 1713 +-----------+----------------+ 1714 | I->A | 01011110 | 1715 | A->B | 00011110 | 1716 | B->D | 00010110 | 1717 | D->C | 00010010 | 1718 | A->C | 01001110 | 1719 +-----------+----------------+ 1721 BitString in BIER Header as Packet Progresses 1723 Table 2: BIER-TE in Action 1725 o Adversely, an elimination node on the way strips the data plane 1726 information and performs a bitwise AND on the BitStrings from the 1727 various copies of the packet that it has received, before it 1728 forwards the packet with the resulting BitString. 1730 +-----------+----------------+ 1731 | Operation | BIER BitString | 1732 +-----------+----------------+ 1733 | D->C | 00010010 | 1734 | A->C | 01001110 | 1735 | | -------- | 1736 | AND in C | 00000010 | 1737 | | | 1738 | C->E | 00000000 | 1739 +-----------+----------------+ 1741 BitString Processing at Elimination Point C 1743 Table 3: BIER-TE in Action (cont.) 1745 o In this example, all the transmissions succeeded and the BitString 1746 at arrival has all the bits reset - note that the egress may be an 1747 Elimination Point in which case this is evaluated after this node 1748 has performed its AND operation on the received BitStrings). 1750 +-------------------+-----------------------+ 1751 | Failing Adjacency | Egress BIER BitString | 1752 +-------------------+-----------------------+ 1753 | I->A | Frame Lost | 1754 | I->B | Not Tried | 1755 | A->C | 00010000 | 1756 | A->B | 01001100 | 1757 | B->D | 01001100 | 1758 | D->C | 01001100 | 1759 | C->E | Frame Lost | 1760 | D->E | Not Tried | 1761 +-------------------+-----------------------+ 1763 BitString indicating failures 1765 Table 4: BIER-TE in Action (cont.) 1767 o But if a transmission failed along the way, one (or more) bit is 1768 never cleared. Table 4 provides the possible outcomes of a 1769 transmission. If the frame is lost, then it is probably due to a 1770 failure in either I->A or C->E, and the controller should enable 1771 I->B and D->E to find out. A BitString of 00010000 indicates 1772 unequivocally a transmission error on the A->C adjacency, and a 1773 BitString of 01001100 indicates a loss in either A->B, B->D or 1774 D->C; enabling D->E on the next packets may provide more 1775 information to sort things out. 1777 In more details: 1779 The BIER header is of variable size, and a DetNet network of a 1780 limited size can use a model with 64 bits if 64 adjacencies are 1781 enough, whereas a larger deployment may be able to signal up to 256 1782 adjacencies for use in very complex paths. Figure 7 illustrates a 1783 BIER header as encapsulated within MPLS. The format of this header 1784 is common to BIER and BIER-TE. 1786 For the DetNet data plane, a replication point is an ingress point 1787 for more than one adjacency, and an elimination point is an egress 1788 point for more than one adjacency. 1790 A pre-populated state in a replication node indicates which bits are 1791 served by this node and to which adjacency each of these bits 1792 corresponds. With DetNet, the state is typically installed by a 1793 controller entity such as a PCE. The way the adjacency is signaled 1794 in the packet is fully abstracted in the bit representation and must 1795 be provisioned to the replication nodes and maintained as a local 1796 state, together with the timing or shaping information for the 1797 associated flow. 1799 The DetNet data plane uses BIER-TE to control which adjacencies are 1800 used for a given packet. This is signaled from the path ingress, 1801 which sets the appropriate bits in the BIER BitString to indicate 1802 which replication must happen. 1804 The replication point clears the bit associated to the adjacency 1805 where the replica is placed, and the elimination points perform a 1806 logical AND of the BitStrings of the copies that it gets before 1807 forwarding. 1809 As is apparent in the examples above, clearing the bits enables to 1810 trace a packet to the replication points that made any particular 1811 copy. BIER-TE also enables to detect the failing adjacencies or 1812 sequences of adjacencies along a path and to activate additional 1813 replications to counter balance the failures. 1815 Finally, using the same BIER-TE bit for both directions of the steps 1816 of the ladder enables to avoid replication in both directions along 1817 the crossing adjacencies. At the time of sending along the step of 1818 the ladder, the bit may have been already reset by performing the AND 1819 operation with the copy from the other side, in which case the 1820 transmission is not needed and does not occur (since the control bit 1821 is now off). 1823 4.2.8. Higher layer header fields 1825 Fields of headers belonging to higher OSI layers can be used to 1826 implement functionality that is not provided e.g., by the IPv6 or 1827 IPv4 header fields. However, this approach cannot be always applied, 1828 e.g., due to encryption. Furthermore, even if this approach is 1829 applicable, it requires deep packet inspection from the routers and 1830 switches. There are implementation dependent limits how far into the 1831 packet the lookup can be done efficiently in the fast path. In 1832 general a safe bet is between 128 and 256 octets for the maximum 1833 lookup depth. Various higher layer protocols can be applied. Some 1834 examples are provided here for the sequence numbering feature 1835 (Section 3.4). 1837 4.2.8.1. TCP 1839 The TCP header includes a sequence number parameter, which can be 1840 applied to detect and eliminate duplicate packets if seamless 1841 redundancy is used. As the TCP header is right after the IP header, 1842 it does not require very deep packet inspection; the 4-byte sequence 1843 number is conveyed by bits 32 through 63 of the TCP header. In 1844 addition to sequencing, the TCP header also contain source and 1845 destination port information that can be used for assisting the flow 1846 identification. 1848 4.2.8.2. RTP 1850 RTP is often used to deliver time critical traffic in IP networks. 1851 RTP is is carried on top of IP and UDP [RFC3550]. The RTP header 1852 includes a 2-byte sequence number, which can be used to detect and 1853 eliminate duplicate packets if seamless redundancy is used. The 1854 sequence number is conveyed by bits 16 through 31 of the RTP header. 1855 In addition to the sequence number the RTP header has also timestamp 1856 field (bits 32 through 63) that can be useful for time 1857 synchronization purposes. Furthermore, the RTP header has also one 1858 or more synchronization sources (bits starting from 64) that can 1859 potentially be useful for flow identification purposes. 1861 5. Summary of data plane alternatives 1863 TBD. 1865 6. Security considerations 1867 TBD. 1869 7. IANA Considerations 1871 This document has no IANA considerations. 1873 8. Acknowledgements 1875 The author(s) ACK and NACK. 1877 The following people were part of the DetNet Data Plane Design Team: 1879 Jouni Korhonen 1880 Janos Farkas 1881 Norman Finn 1882 Olivier Marce 1883 Gregory Mirsky 1884 Pascal Thubert 1885 Zhuangyan Zhuang 1887 The DetNet chairs serving during the DetNet Data Plane Design Team: 1889 Lou Berger 1890 Pat Thaler 1892 9. References 1894 9.1. Informative References 1896 [I-D.eckert-bier-te-arch] 1897 Eckert, T. and G. Cauchie, "Traffic Enginering for Bit 1898 Index Explicit Replication BIER-TE", draft-eckert-bier-te- 1899 arch-02 (work in progress), October 2015. 1901 [I-D.finn-detnet-architecture] 1902 Finn, N., Thubert, P., and M. Teener, "Deterministic 1903 Networking Architecture", draft-finn-detnet- 1904 architecture-02 (work in progress), November 2015. 1906 [I-D.finn-detnet-problem-statement] 1907 Finn, N. and P. Thubert, "Deterministic Networking Problem 1908 Statement", draft-finn-detnet-problem-statement-04 (work 1909 in progress), October 2015. 1911 [I-D.ietf-6man-rfc2460bis] 1912 Deering, S. and B. Hinden, "Internet Protocol, Version 6 1913 (IPv6) Specification", draft-ietf-6man-rfc2460bis-04 (work 1914 in progress), March 2016. 1916 [I-D.ietf-6man-segment-routing-header] 1917 Previdi, S., Filsfils, C., Field, B., Leung, I., Linkova, 1918 J., Kosugi, T., Vyncke, E., and D. Lebrun, "IPv6 Segment 1919 Routing Header (SRH)", draft-ietf-6man-segment-routing- 1920 header-01 (work in progress), March 2016. 1922 [I-D.ietf-bier-architecture] 1923 Wijnands, I., Rosen, E., Dolganow, A., P, T., and S. 1924 Aldrin, "Multicast using Bit Index Explicit Replication", 1925 draft-ietf-bier-architecture-03 (work in progress), 1926 January 2016. 1928 [I-D.ietf-idr-ls-distribution] 1929 Gredler, H., Medved, J., Previdi, S., Farrel, A., and S. 1930 Ray, "North-Bound Distribution of Link-State and TE 1931 Information using BGP", draft-ietf-idr-ls-distribution-13 1932 (work in progress), October 2015. 1934 [I-D.ietf-intarea-gre-ipv6] 1935 Pignataro, C., Bonica, R., and S. Krishnan, "IPv6 Support 1936 for Generic Routing Encapsulation (GRE)", draft-ietf- 1937 intarea-gre-ipv6-14 (work in progress), September 2015. 1939 [I-D.ietf-isis-pcr] 1940 Farkas, J., Bragg, N., Unbehagen, P., Parsons, G., 1941 Ashwood-Smith, P., and C. Bowers, "IS-IS Path Computation 1942 and Reservation", draft-ietf-isis-pcr-05 (work in 1943 progress), February 2016. 1945 [I-D.ietf-mpls-residence-time] 1946 Mirsky, G., Ruffini, S., Gray, E., Drake, J., Bryant, S., 1947 and S. Sasha, "Residence Time Measurement in MPLS 1948 network", draft-ietf-mpls-residence-time-05 (work in 1949 progress), March 2016. 1951 [I-D.ietf-spring-segment-routing] 1952 Filsfils, C., Previdi, S., Decraene, B., Litkowski, S., 1953 and R. Shakir, "Segment Routing Architecture", draft-ietf- 1954 spring-segment-routing-07 (work in progress), December 1955 2015. 1957 [I-D.ietf-sunset4-gapanalysis] 1958 Perreault, S., Tsou, T., Zhou, C., and P. Fan, "Gap 1959 Analysis for IPv4 Sunset", draft-ietf- 1960 sunset4-gapanalysis-07 (work in progress), April 2015. 1962 [I-D.ietf-v6ops-ipv6-ehs-in-real-world] 1963 Gont, F., Linkova, J., Chown, T., and S. LIU, 1964 "Observations on the Dropping of Packets with IPv6 1965 Extension Headers in the Real World", draft-ietf-v6ops- 1966 ipv6-ehs-in-real-world-02 (work in progress), December 1967 2015. 1969 [IEEE802.1Qbv] 1970 IEEE, "Enhancements for Scheduled Traffic", 2016, 1971 . 1973 [IEEE802.1Qch] 1974 IEEE, "Cyclic Queuing and Forwarding", 2016, 1975 . 1977 [IEEE8021CB] 1978 Finn, N., "Draft Standard for Local and metropolitan area 1979 networks - Seamless Redundancy", IEEE P802.1CB 1980 /D2.1 P802.1CB, December 2015, 1981 . 1984 [IEEE8021Qca] 1985 IEEE 802.1, "IEEE 802.1Qca Bridges and Bridged Networks - 1986 Amendment 24: Path Control and Reservation", IEEE 1987 P802.1Qca/D2.1 P802.1Qca, June 2015, 1988 . 1991 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 1992 DOI 10.17487/RFC0791, September 1981, 1993 . 1995 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 1996 Communication Layers", STD 3, RFC 1122, 1997 DOI 10.17487/RFC1122, October 1989, 1998 . 2000 [RFC1393] Malkin, G., "Traceroute Using an IP Option", RFC 1393, 2001 DOI 10.17487/RFC1393, January 1993, 2002 . 2004 [RFC1700] Reynolds, J. and J. Postel, "Assigned Numbers", RFC 1700, 2005 DOI 10.17487/RFC1700, October 1994, 2006 . 2008 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 2009 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 2010 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 2011 September 1997, . 2013 [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 2014 (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460, 2015 December 1998, . 2017 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2018 "Definition of the Differentiated Services Field (DS 2019 Field) in the IPv4 and IPv6 Headers", RFC 2474, 2020 DOI 10.17487/RFC2474, December 1998, 2021 . 2023 [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and J. 2024 McManus, "Requirements for Traffic Engineering Over MPLS", 2025 RFC 2702, DOI 10.17487/RFC2702, September 1999, 2026 . 2028 [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. 2029 Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, 2030 DOI 10.17487/RFC2784, March 2000, 2031 . 2033 [RFC2890] Dommety, G., "Key and Sequence Number Extensions to GRE", 2034 RFC 2890, DOI 10.17487/RFC2890, September 2000, 2035 . 2037 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2038 Label Switching Architecture", RFC 3031, 2039 DOI 10.17487/RFC3031, January 2001, 2040 . 2042 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2043 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2044 Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001, 2045 . 2047 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2048 of Explicit Congestion Notification (ECN) to IP", 2049 RFC 3168, DOI 10.17487/RFC3168, September 2001, 2050 . 2052 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2053 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2054 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 2055 . 2057 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2058 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2059 Protocol Label Switching (MPLS) Support of Differentiated 2060 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 2061 . 2063 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2064 in Multi-Protocol Label Switching (MPLS) Networks", 2065 RFC 3443, DOI 10.17487/RFC3443, January 2003, 2066 . 2068 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 2069 Switching (GMPLS) Signaling Resource ReserVation Protocol- 2070 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 2071 DOI 10.17487/RFC3473, January 2003, 2072 . 2074 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 2075 Jacobson, "RTP: A Transport Protocol for Real-Time 2076 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 2077 July 2003, . 2079 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 2080 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 2081 RFC 3931, DOI 10.17487/RFC3931, March 2005, 2082 . 2084 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 2085 Edge-to-Edge (PWE3) Architecture", RFC 3985, 2086 DOI 10.17487/RFC3985, March 2005, 2087 . 2089 [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., 2090 "Encapsulating MPLS in IP or Generic Routing Encapsulation 2091 (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005, 2092 . 2094 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2095 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2096 Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385, 2097 February 2006, . 2099 [RFC4446] Martini, L., "IANA Allocations for Pseudowire Edge to Edge 2100 Emulation (PWE3)", BCP 116, RFC 4446, 2101 DOI 10.17487/RFC4446, April 2006, 2102 . 2104 [RFC4447] Martini, L., Ed., Rosen, E., El-Aawar, N., Smith, T., and 2105 G. Heron, "Pseudowire Setup and Maintenance Using the 2106 Label Distribution Protocol (LDP)", RFC 4447, 2107 DOI 10.17487/RFC4447, April 2006, 2108 . 2110 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 2111 "Encapsulation Methods for Transport of Ethernet over MPLS 2112 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 2113 . 2115 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 2116 2 Virtual Private Networks (L2VPNs)", RFC 4664, 2117 DOI 10.17487/RFC4664, September 2006, 2118 . 2120 [RFC4817] Townsley, M., Pignataro, C., Wainner, S., Seely, T., and 2121 J. Young, "Encapsulation of MPLS over Layer 2 Tunneling 2122 Protocol Version 3", RFC 4817, DOI 10.17487/RFC4817, March 2123 2007, . 2125 [RFC4875] Aggarwal, R., Ed., Papadimitriou, D., Ed., and S. 2126 Yasukawa, Ed., "Extensions to Resource Reservation 2127 Protocol - Traffic Engineering (RSVP-TE) for Point-to- 2128 Multipoint TE Label Switched Paths (LSPs)", RFC 4875, 2129 DOI 10.17487/RFC4875, May 2007, 2130 . 2132 [RFC5086] Vainshtein, A., Ed., Sasson, I., Metz, E., Frost, T., and 2133 P. Pate, "Structure-Aware Time Division Multiplexed (TDM) 2134 Circuit Emulation Service over Packet Switched Network 2135 (CESoPSN)", RFC 5086, DOI 10.17487/RFC5086, December 2007, 2136 . 2138 [RFC5087] Stein, Y(J)., Shashoua, R., Insler, R., and M. Anavi, 2139 "Time Division Multiplexing over IP (TDMoIP)", RFC 5087, 2140 DOI 10.17487/RFC5087, December 2007, 2141 . 2143 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2144 Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2145 2008, . 2147 [RFC5305] Li, T. and H. Smit, "IS-IS Extensions for Traffic 2148 Engineering", RFC 5305, DOI 10.17487/RFC5305, October 2149 2008, . 2151 [RFC5331] Aggarwal, R., Rekhter, Y., and E. Rosen, "MPLS Upstream 2152 Label Assignment and Context-Specific Label Space", 2153 RFC 5331, DOI 10.17487/RFC5331, August 2008, 2154 . 2156 [RFC5332] Eckert, T., Rosen, E., Ed., Aggarwal, R., and Y. Rekhter, 2157 "MPLS Multicast Encapsulations", RFC 5332, 2158 DOI 10.17487/RFC5332, August 2008, 2159 . 2161 [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation 2162 Element (PCE) Communication Protocol (PCEP)", RFC 5440, 2163 DOI 10.17487/RFC5440, March 2009, 2164 . 2166 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2167 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2168 Class" Field", RFC 5462, DOI 10.17487/RFC5462, February 2169 2009, . 2171 [RFC5586] Bocci, M., Ed., Vigoureux, M., Ed., and S. Bryant, Ed., 2172 "MPLS Generic Associated Channel", RFC 5586, 2173 DOI 10.17487/RFC5586, June 2009, 2174 . 2176 [RFC5659] Bocci, M. and S. Bryant, "An Architecture for Multi- 2177 Segment Pseudowire Emulation Edge-to-Edge", RFC 5659, 2178 DOI 10.17487/RFC5659, October 2009, 2179 . 2181 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 2182 L., and L. Berger, "A Framework for MPLS in Transport 2183 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 2184 . 2186 [RFC5960] Frost, D., Ed., Bryant, S., Ed., and M. Bocci, Ed., "MPLS 2187 Transport Profile Data Plane Architecture", RFC 5960, 2188 DOI 10.17487/RFC5960, August 2010, 2189 . 2191 [RFC6073] Martini, L., Metz, C., Nadeau, T., Bocci, M., and M. 2192 Aissaoui, "Segmented Pseudowire", RFC 6073, 2193 DOI 10.17487/RFC6073, January 2011, 2194 . 2196 [RFC6275] Perkins, C., Ed., Johnson, D., and J. Arkko, "Mobility 2197 Support in IPv6", RFC 6275, DOI 10.17487/RFC6275, July 2198 2011, . 2200 [RFC6371] Busi, I., Ed. and D. Allan, Ed., "Operations, 2201 Administration, and Maintenance Framework for MPLS-Based 2202 Transport Networks", RFC 6371, DOI 10.17487/RFC6371, 2203 September 2011, . 2205 [RFC6373] Andersson, L., Ed., Berger, L., Ed., Fang, L., Ed., Bitar, 2206 N., Ed., and E. Gray, Ed., "MPLS Transport Profile (MPLS- 2207 TP) Control Plane Framework", RFC 6373, 2208 DOI 10.17487/RFC6373, September 2011, 2209 . 2211 [RFC6378] Weingarten, Y., Ed., Bryant, S., Osborne, E., Sprecher, 2212 N., and A. Fulignoli, Ed., "MPLS Transport Profile (MPLS- 2213 TP) Linear Protection", RFC 6378, DOI 10.17487/RFC6378, 2214 October 2011, . 2216 [RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS 2217 On-Demand Connectivity Verification and Route Tracing", 2218 RFC 6426, DOI 10.17487/RFC6426, November 2011, 2219 . 2221 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 2222 "IPv6 Flow Label Specification", RFC 6437, 2223 DOI 10.17487/RFC6437, November 2011, 2224 . 2226 [RFC6540] George, W., Donley, C., Liljenstolpe, C., and L. Howard, 2227 "IPv6 Support Required for All IP-Capable Nodes", BCP 177, 2228 RFC 6540, DOI 10.17487/RFC6540, April 2012, 2229 . 2231 [RFC6564] Krishnan, S., Woodyatt, J., Kline, E., Hoagland, J., and 2232 M. Bhatia, "A Uniform Format for IPv6 Extension Headers", 2233 RFC 6564, DOI 10.17487/RFC6564, April 2012, 2234 . 2236 [RFC6621] Macker, J., Ed., "Simplified Multicast Forwarding", 2237 RFC 6621, DOI 10.17487/RFC6621, May 2012, 2238 . 2240 [RFC6658] Bryant, S., Ed., Martini, L., Swallow, G., and A. Malis, 2241 "Packet Pseudowire Encapsulation over an MPLS PSN", 2242 RFC 6658, DOI 10.17487/RFC6658, July 2012, 2243 . 2245 [RFC6718] Muley, P., Aissaoui, M., and M. Bocci, "Pseudowire 2246 Redundancy", RFC 6718, DOI 10.17487/RFC6718, August 2012, 2247 . 2249 [RFC6733] Fajardo, V., Ed., Arkko, J., Loughney, J., and G. Zorn, 2250 Ed., "Diameter Base Protocol", RFC 6733, 2251 DOI 10.17487/RFC6733, October 2012, 2252 . 2254 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2255 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2256 RFC 6790, DOI 10.17487/RFC6790, November 2012, 2257 . 2259 [RFC6814] Pignataro, C. and F. Gont, "Formally Deprecating Some IPv4 2260 Options", RFC 6814, DOI 10.17487/RFC6814, November 2012, 2261 . 2263 [RFC6864] Touch, J., "Updated Specification of the IPv4 ID Field", 2264 RFC 6864, DOI 10.17487/RFC6864, February 2013, 2265 . 2267 [RFC7045] Carpenter, B. and S. Jiang, "Transmission and Processing 2268 of IPv6 Extension Headers", RFC 7045, 2269 DOI 10.17487/RFC7045, December 2013, 2270 . 2272 [RFC7167] Frost, D., Bryant, S., Bocci, M., and L. Berger, "A 2273 Framework for Point-to-Multipoint MPLS in Transport 2274 Networks", RFC 7167, DOI 10.17487/RFC7167, April 2014, 2275 . 2277 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 2278 Henderickx, W., and A. Isaac, "Requirements for Ethernet 2279 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 2280 . 2282 [RFC7271] Ryoo, J., Ed., Gray, E., Ed., van Helvoort, H., 2283 D'Alessandro, A., Cheung, T., and E. Osborne, "MPLS 2284 Transport Profile (MPLS-TP) Linear Protection to Match the 2285 Operational Expectations of Synchronous Digital Hierarchy, 2286 Optical Transport Network, and Ethernet Transport Network 2287 Operators", RFC 7271, DOI 10.17487/RFC7271, June 2014, 2288 . 2290 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 2291 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 2292 eXtensible Local Area Network (VXLAN): A Framework for 2293 Overlaying Virtualized Layer 2 Networks over Layer 3 2294 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, 2295 . 2297 [RFC7399] Farrel, A. and D. King, "Unanswered Questions in the Path 2298 Computation Element Architecture", RFC 7399, 2299 DOI 10.17487/RFC7399, October 2014, 2300 . 2302 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 2303 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 2304 Defined Networking (SDN): Layers and Architecture 2305 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 2306 2015, . 2308 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 2309 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 2310 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 2311 2015, . 2313 [RFC7510] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black, 2314 "Encapsulating MPLS in UDP", RFC 7510, 2315 DOI 10.17487/RFC7510, April 2015, 2316 . 2318 [RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network 2319 Virtualization Using Generic Routing Encapsulation", 2320 RFC 7637, DOI 10.17487/RFC7637, September 2015, 2321 . 2323 [ST20227] SMPTE 2022, "Seamless Protection Switching of SMPTE ST 2324 2022 IP Datagrams", ST 2022-7:2013, 2013, 2325 . 2327 [TSNTG] IEEE Standards Association, "IEEE 802.1 Time-Sensitive 2328 Networks Task Group", 2013, 2329 . 2331 9.2. URIs 2333 [1] http://6lab.cisco.com/stats/ 2335 [2] https://www.google.com/intl/en/ipv6/statistics.html 2337 [3] https://datatracker.ietf.org/wg/spring/charter/ 2339 [4] http://www.iana.org/assignments/g-ach-parameters/g-ach- 2340 parameters.xhtml 2342 [5] http://ftp.isi.edu/in-notes/iana/assignments/ethernet-numbers 2344 [6] https://tools.ietf.org/wg/bess/ 2346 Appendix A. Examples of combined DetNet Service and Transport layers 2348 Authors' Addresses 2349 Jouni Korhonen (editor) 2350 Broadcom 2351 3151 Zanker Road 2352 San Jose, CA 95134 2353 USA 2355 Email: jouni.nospam@gmail.com 2357 Janos Farkas 2358 Ericsson 2359 Konyves Kalman krt. 11/B 2360 Budapest 1097 2361 Hungary 2363 Email: janos.farkas@ericsson.com 2365 Gregory Mirsky 2366 Ericsson 2368 Email: gregory.mirsky@ericsson.com 2370 Pascal Thubert 2371 Cisco 2373 Email: pthubert@cisco.com 2375 Yan Zhuang 2376 Huawei 2378 Email: zhuangyan.zhuang@huawei.com 2380 Lou Berger 2381 LabN Consulting, L.L.C. 2383 Email: lberger@labn.net