idnits 2.17.1 draft-ietf-detnet-data-plane-framework-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 194 has weird spacing: '...e stack v ...' -- The document date (May 6, 2020) is 1450 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-14) exists of draft-ietf-detnet-flow-information-model-07 == Outdated reference: A later version (-07) exists of draft-ietf-detnet-ip-05 == Outdated reference: A later version (-13) exists of draft-ietf-detnet-mpls-05 == Outdated reference: A later version (-07) exists of draft-ietf-detnet-mpls-over-tsn-02 == Outdated reference: A later version (-08) exists of draft-ietf-detnet-mpls-over-udp-ip-05 == Outdated reference: A later version (-16) exists of draft-ietf-detnet-security-09 == Outdated reference: A later version (-27) exists of draft-ietf-idr-rfc5575bis-24 == Outdated reference: A later version (-14) exists of draft-ietf-pce-pcep-extension-for-pce-controller-04 == Outdated reference: A later version (-28) exists of draft-ietf-spring-srv6-network-programming-15 -- Obsolete informational reference (is this intentional?): RFC 5575 (Obsoleted by RFC 8955) Summary: 0 errors (**), 0 flaws (~~), 11 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet B. Varga, Ed. 3 Internet-Draft J. Farkas 4 Intended status: Informational Ericsson 5 Expires: November 7, 2020 L. Berger 6 LabN Consulting, L.L.C. 7 A. Malis 8 Malis Consulting 9 S. Bryant 10 Futurewei Technologies 11 May 6, 2020 13 DetNet Data Plane Framework 14 draft-ietf-detnet-data-plane-framework-06 16 Abstract 18 This document provides an overall framework for the DetNet data 19 plane. It covers concepts and considerations that are generally 20 common to any Deterministic Networking data plane specification. It 21 describes related controller plane considerations as well. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at https://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on November 7, 2020. 40 Copyright Notice 42 Copyright (c) 2020 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (https://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 58 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 59 2.1. Terms Used in This Document . . . . . . . . . . . . . . . 4 60 2.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4 61 3. DetNet Data Plane Overview . . . . . . . . . . . . . . . . . 4 62 3.1. Data Plane Characteristics . . . . . . . . . . . . . . . 6 63 3.1.1. Data Plane Technology . . . . . . . . . . . . . . . . 6 64 3.1.2. Encapsulation . . . . . . . . . . . . . . . . . . . . 6 65 3.2. DetNet-specific Metadata . . . . . . . . . . . . . . . . 7 66 3.3. DetNet IP Data Plane . . . . . . . . . . . . . . . . . . 8 67 3.4. DetNet MPLS Data Plane . . . . . . . . . . . . . . . . . 8 68 3.5. Further DetNet Data Plane Considerations . . . . . . . . 9 69 3.5.1. Per Flow Related Functions . . . . . . . . . . . . . 9 70 3.5.2. Service Protection . . . . . . . . . . . . . . . . . 11 71 3.5.3. Aggregation Considerations . . . . . . . . . . . . . 13 72 3.5.4. End-System-Specific Considerations . . . . . . . . . 14 73 3.5.5. Sub-Network Considerations . . . . . . . . . . . . . 15 74 4. Controller Plane (Management and Control) 75 Considerations . . . . . . . . . . . . . . . . . . . . . . . 15 76 4.1. DetNet Controller Plane Requirements . . . . . . . . . . 15 77 4.2. Generic Controller Plane Considerations . . . . . . . . . 17 78 4.2.1. Flow Aggregation Control . . . . . . . . . . . . . . 17 79 4.2.2. Explicit Routes . . . . . . . . . . . . . . . . . . . 18 80 4.2.3. Contention Loss and Jitter Reduction . . . . . . . . 19 81 4.2.4. Bidirectional Traffic . . . . . . . . . . . . . . . . 19 82 4.3. Packet Replication, Elimination, and Ordering (PREOF) . . 20 83 5. Security Considerations . . . . . . . . . . . . . . . . . . . 20 84 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 85 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21 86 8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 22 87 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 88 9.1. Normative References . . . . . . . . . . . . . . . . . . 22 89 9.2. Informative References . . . . . . . . . . . . . . . . . 22 90 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 92 1. Introduction 94 DetNet (Deterministic Networking) provides a capability to carry 95 specified unicast or multicast data flows for real-time applications 96 with extremely low packet loss rates and assured maximum end-to-end 97 delivery latency. A description of the general background and 98 concepts of DetNet can be found in [RFC8655]. 100 This document describes the concepts needed by any DetNet data plane 101 specification (i.e., the DetNet-specific use of packet header fields) 102 and provides considerations for any corresponding implementation. It 103 covers the building blocks that provide the DetNet service, the 104 DetNet service sub-layer and the DetNet forwarding sub-layer 105 functions as described in the DetNet Architecture. 107 The DetNet Architecture models the DetNet related data plane 108 functions as being decomposed into two sub-layers: a service sub- 109 layer and a forwarding sub-layer. The service sub-layer is used to 110 provide DetNet service protection and reordering. The forwarding 111 sub-layer leverages Traffic Engineering mechanisms and provides 112 congestion protection (low loss, assured latency, and limited out-of- 113 order delivery). A particular forwarding sub-layer may have 114 capabilities that are not available on other forwarding-sub layers. 115 DetNet makes use of the existing forwarding sub-layers with their 116 respective capabilities and does not require 1:1 equivalence between 117 different forwarding sub-layer capabilities. 119 As part of the service sub-layer functions, this document describes 120 typical DetNet node data plane operation. It describes the 121 functionality and operation of the Packet Replication (PRF), Packet 122 Elimination (PEF), and the Packet Ordering (POF) functions within the 123 service sub-layer. Furthermore, it describes the forwarding sub- 124 layer. 126 As defined in [RFC8655], DetNet flows may be carried over network 127 technologies that can provide the DetNet required service 128 characteristics. For example, DetNet MPLS flows can be carried over 129 IEEE 802.1 Time Sensitive Network (TSN) [IEEE802.1TSNTG] sub- 130 networks. However, IEEE 802.1 TSN support is not required in DetNet. 131 TSN frame preemption is an example of a forwarding layer capability 132 that is typically not replicated in other forwarding technologies. 133 Most of DetNet benefits can be gained by running over a data link 134 layer that has not been specifically enhanced to support all TSN 135 capabilities but for such networks and traffic mixes delay and jitter 136 performance may vary due to the forwarding sub-layer intrinsic 137 properties. 139 Different application flows, such as Ethernet or IP, can be mapped on 140 top of DetNet. DetNet can optionally reuse header information 141 provided by, or shared with, applications. An example of shared 142 header fields can be found in [I-D.ietf-detnet-ip]. 144 This document also covers basic concepts related to the controller 145 plane and Operations, Administration, and Maintenance (OAM). Data 146 plane OAM specifics are out of scope for this document. 148 2. Terminology 150 2.1. Terms Used in This Document 152 This document uses the terminology established in the DetNet 153 architecture [RFC8655], and the reader is assumed to be familiar with 154 that document and its terminology. 156 2.2. Abbreviations 158 The following abbreviations are used in this document: 160 BGP Border Gateway Protocol. 161 d-CW DetNet Control Word. 162 DetNet Deterministic Networking. 163 DN DetNet. 164 GMPLS Generalized Multiprotocol Label Switching. 165 GRE Generic Routing Encapsulation. 166 IPSec IP Security. 167 L2 Layer 2. 168 LSP Label Switched Path. 169 MPLS Multiprotocol Label Switching. 170 OAM Operations, Administration, and Maintenance. 171 PCEP Path Computation Element Communication Protocol. 172 PEF Packet Elimination Function. 173 PRF Packet Replication Function. 174 PREOF Packet Replication, Elimination and Ordering Functions. 175 POF Packet Ordering Function. 176 PSN Packet Switched Network. 177 QoS Quality of Service. 178 S-Label DetNet "service" label. 179 TDM Time-Division Multiplexing. 180 TSN Time-Sensitive Network. 181 YANG Yet Another Next Generation. 183 3. DetNet Data Plane Overview 185 This document describes how application flows, or app-flows, are 186 carried over DetNet networks. The DetNet Architecture [RFC8655] 187 models the DetNet-related data plane functions as decomposed into two 188 sub-layers: a service sub-layer and a forwarding sub-layer. 190 Figure 1, reproduced from [RFC8655], shows a logical DetNet service 191 with the two sub-layers. 193 | packets going | ^ packets coming ^ 194 v down the stack v | up the stack | 195 +-----------------------+ +-----------------------+ 196 | Source | | Destination | 197 +-----------------------+ +-----------------------+ 198 | Service sub-layer: | | Service sub-layer: | 199 | Packet sequencing | | Duplicate elimination | 200 | Flow replication | | Flow merging | 201 | Packet encoding | | Packet decoding | 202 +-----------------------+ +-----------------------+ 203 | Forwarding sub-layer: | | Forwarding sub-layer: | 204 | Resource allocation | | Resource allocation | 205 | Explicit routes | | Explicit routes | 206 +-----------------------+ +-----------------------+ 207 | Lower layers | | Lower layers | 208 +-----------------------+ +-----------------------+ 209 v ^ 210 \_________________________/ 212 Figure 1: DetNet data plane protocol stack 214 The DetNet forwarding sub-layer may be directly provided by the 215 DetNet service sub-layer, for example by IP tunnels or MPLS. 216 Alternatively, an overlay approach may be used in which the packet is 217 natively carried between key nodes within the DetNet network (say 218 between PREOF nodes) and a sub-layer is used to provide the 219 information needed to reach the next hop in the overlay. 221 The forwarding sub-layer provides the QoS related functions needed by 222 the DetNet flow. It may do this directly through the use of queuing 223 techniques and traffic engineering methods, or it may do this through 224 the assistance of its underlying connectivity. For example it may 225 call upon Ethernet TSN capabilities defined in IEEE 802.1 TSN 226 [IEEE802.1TSNTG]. The forwarding sub-layer uses buffer resources for 227 packet queuing, as well as reservation and allocation of bandwidth 228 capacity resources. 230 The service sub-layer provides additional support beyond the 231 connectivity function of the forwarding sub-layer. See Section 4.3 232 as an example for Packet Replication, Elimination, and Ordering 233 functions. The ordering function (POF) uses sequence numbers added 234 to packets enabling a range of packet order protection from simple 235 ordering and dropping out-of-order packets to more complex reordering 236 of a fixed number of out-of-order, minimally delayed packets. 237 Reordering requires buffer resources and has impact on the delay and 238 jitter of packets in the DetNet flow. 240 The method of instantiating each of the layers is specific to the 241 particular DetNet data plane method, and more than one approach may 242 be applicable to a given network type. 244 3.1. Data Plane Characteristics 246 There are two major characteristics to the data plane: the technology 247 and the encapsulation, as discussed below. 249 3.1.1. Data Plane Technology 251 The DetNet data plane is provided by the DetNet service and 252 forwarding sub layers. The DetNet service sub-layer generally 253 provides its functions for the DetNet application flows by using or 254 applying existing standardized headers and/or encapsulations. The 255 Detnet forwarding sub-layer may provide capabilities leveraging that 256 same header or encapsulation technology (e.g., DN IP or DN MPLS) or 257 it may be achieved by other technologies as shown in Figure 2. 258 DetNet is currently defined for operation over packet-switched (IP) 259 networks or label-switched (MPLS) networks. 261 3.1.2. Encapsulation 263 DetNet encodes specific flow attributes (flow identity and sequence 264 number) in packets. For example, in DetNet IP, zero encapsulation is 265 used and no sequence number is available, and in DetNet MPLS, DetNet- 266 specific information may be added explicitly to the packets in the 267 form of S-label and d-CW [I-D.ietf-detnet-mpls] . 269 The encapsulation of a DetNet flow allows it to be sent over a data 270 plane technology other than its native type. DetNet uses header 271 information to perform traffic classification, i.e., identify DetNet 272 flows, and provide DetNet service and forwarding functions. As 273 mentioned above, DetNet may add headers, as is the case for DN MPLS, 274 or may use headers that are already present, as is the case in DN IP. 275 Figure 2 illustrates some relationships between the components. 277 +-----+ 278 | TSN | 279 +-------+ +-+-----+-+ 280 | DN IP | | DN MPLS | 281 +--+--+----+----+ +-+---+-----+-+ 282 | TSN | DN MPLS | | TSN | DN IP | 283 +-----+---------+ +-----+-------+ 285 Figure 2: DetNet Service Examples 287 The use of encapsulation is also required if additional information 288 (metadata) is needed by the DetNet data plane and there is either no 289 ability to include it in the client data packet, or the specification 290 of the client data plane does not permit the modification of the 291 packet to include additional data. An example of such metadata is 292 the inclusion of a sequence number required by the PREOF function. 294 Encapsulation may also be used to carry or aggregate flows for 295 equipment with limited DetNet capability. 297 3.2. DetNet-specific Metadata 299 The DetNet data plane can provide or carry the following metadata: 301 1. Flow-ID 303 2. Sequence Number 305 The DetNet data plane framework supports a Flow-ID (for 306 identification of the flow or aggregate flow) and/or a Sequence 307 Number (for PREOF) for each DetNet flow. The Flow-ID is used by both 308 the service and forwarding sub-layers, but the sequence number is 309 only used by the service layer. Metadata can also be used for OAM 310 indications and instrumentation of DetNet data plane operation. 312 Metadata inclusion can be implicit or explicit. Explicit inclusions 313 involve a dedicated header field that is used to include metadata in 314 a DetNet packet. In the implicit method, part of an already existing 315 header field is used to encode the metadata. 317 Explicit inclusion of metadata is possible through the use of IP 318 options or IP extension headers. New IP options are almost 319 impossible to get standardized or to deploy in an operational network 320 and will not be discussed further in this text. IPv6 extensions 321 headers are finding popularity in current IPv6 development work, 322 particularly in connection with Segment Routing of IPv6 (SRv6) and IP 323 OAM. The design of a new IPv6 extension header or the modification 324 of an existing one is a technique available in the tool box of the 325 DetNet IP data plane designer. 327 Explicit inclusion of metadata in an IP packet is also possible 328 through the inclusion of an MPLS label stack and the MPLS DetNet 329 Control Word using one of the methods for carrying MPLS over IP 330 [I-D.ietf-detnet-mpls-over-udp-ip]. This is described in more detail 331 in Section 3.5.5. 333 Implicit metadata in IP can be included through the use of the 334 network programming paradigm 336 [I-D.ietf-spring-srv6-network-programming] in which the suffix of an 337 IPv6 address is used to encode additional information for use by the 338 network of the receiving host. 340 An MPLS example of explicit metadata is the sequence number used by 341 the PREOF function, or even the case where all the essential 342 information being included into the DetNet over MPLS label stack (the 343 DetNet Control Word and the DetNet Service label). 345 3.3. DetNet IP Data Plane 347 An IP data plane may operate natively or through the use of an 348 encapsulation. Many types of IP encapsulation can satisfy DetNet 349 requirements and it is anticipated that more than one encapsulation 350 may be deployed, for example GRE, IPSec. 352 One method of operating an IP DetNet data plane without encapsulation 353 is to use "6-tuple" based flow identification, where "6-tuple" refers 354 to information carried in IP and higher layer protocol headers. 355 General background on the use of IP headers, and "6-tuples", to 356 identify flows and support Quality of Service (QoS) can be found in 357 [RFC3670]. [RFC7657] provides useful background on differentiated 358 services (DiffServ) and "tuple" based flow identification. DetNet 359 flow aggregation may be enabled via the use of wildcards, masks, 360 prefixes and ranges. The operation of this method is described in 361 detail in [I-D.ietf-detnet-ip]. 363 The DetNet forwarding plane may use explicit route capabilities and 364 traffic engineering capabilities to provide a forwarding sub-layer 365 that is responsible for providing resource allocation and explicit 366 routes. It is possible to include such information in a native IP 367 packet explicitly, or implicitly. 369 3.4. DetNet MPLS Data Plane 371 MPLS provides a forwarding sub-layer for traffic over implicit and 372 explicit paths to the point in the network where the next DetNet 373 service sub-layer action needs to take place. It does this through 374 the use of a stack of one or more labels with various forwarding 375 semantics. 377 MPLS also provides the ability to identify a service instance that is 378 used to process the packet through the use of a label that maps the 379 packet to a service instance. 381 In cases where metadata is needed to process an MPLS encapsulated 382 packet at the service sub-layer, the d-CW [I-D.ietf-detnet-mpls], 383 [RFC4385] can be used. Although such d-CWs are frequently 32 bits 384 long, there is no architectural constraint on the size of this 385 structure, only the requirement that it is fully understood by all 386 parties operating on it in the DetNet service sub-layer. The 387 operation of this method is described in detail in 388 [I-D.ietf-detnet-mpls]. 390 3.5. Further DetNet Data Plane Considerations 392 This section provides informative considerations related to providing 393 DetNet service to flows which are identified based on their header 394 information. 396 3.5.1. Per Flow Related Functions 398 At a high level, the following functions are provided on a per flow 399 basis. 401 3.5.1.1. Reservation and Allocation of resources 403 Resources might be reserved in order to make them available for 404 allocation to specific DetNet flows. This can eliminate packet 405 contention and packet loss for DetNet traffic. This also can reduce 406 jitter for DetNet traffic. Resources allocated to a DetNet flow 407 protect it from other traffic flows. On the other hand, DetNet flows 408 are assumed to behave with respect to the reserved traffic profile. 409 It must be possible to detect misbehaving DetNet flows and to ensure 410 that they do not compromise QoS of other flows. Queuing, policing, 411 and shaping policies can be used to ensure that the allocation of 412 resources reserved for DetNet is met. 414 3.5.1.2. Explicit routes 416 A flow can be routed over a specific, pre-computed path. This allows 417 control of the network delay by steering the packet with the ability 418 to influence the physical path. Explicit routes complement 419 reservation by ensuring that a consistent path can be associated with 420 its resources for the duration of that path. Coupled with the 421 traffic mechanism, this limits misordering and bounds latency. 422 Explicit route computation can encompass a wide set of constraints 423 and can optimize the path for a certain characteristic, e.g., highest 424 bandwidth or lowest jitter. In these cases the "best" path for any 425 set of characteristics may not be a shortest path. The selection of 426 path can take into account multiple network metrics. Some of these 427 metrics are measured and distributed by the routing system as traffic 428 engineering metrics. 430 3.5.1.3. Service protection 432 Service protection involves use of multiple packet streams using 433 multiple paths, for example 1+1 or 1:1 linear protection. For 434 DetNet, this primarily relates to packet replication and elimination 435 capabilities. MPLS offers a number of protection schemes. MPLS 436 hitless protection can be used to switch traffic to an already 437 established path in order to restore delivery rapidly after a 438 failure. Path changes, even in the case of failure recovery, can 439 lead to the out of order delivery of data requiring packet ordering 440 functions either within the DetNet service or at a high layer in the 441 application traffic. Establishment of new paths after a failure is 442 out of scope for DetNet services. 444 3.5.1.4. Network Coding 446 Network Coding [nwcrg], not to be confused with network programming, 447 comprises several techniques where multiple data flows are encoded. 448 These resulting flows can then be sent on different paths. The 449 encoding operation can combine flows and error recovery information. 450 When the encoded flows are decoded and recombined the original flows 451 can be recovered. Note that Network coding uses an alternative to 452 packet by packet PREOF. Therefore, for certain network topologies 453 and traffic loads, Network Coding can be used to improve a network's 454 throughput, efficiency, latency, and scalability, as well as 455 resilience to partition, attacks, and eavesdropping, as compared to 456 traditional methods. DetNet could use Network coding as an 457 alternative to other protection means. Network coding is often 458 applied in wireless networks and is being explored for other network 459 types. 461 3.5.1.5. Load sharing 463 Use of packet-by-packet load sharing of the same DetNet flow over 464 multiple paths is not recommended except for the cases listed above 465 where PREOF is utilized to improve protection of traffic and maintain 466 order. Packet-by-packet load sharing, e.g., via ECMP or UCMP, 467 impacts ordering and possibly jitter. 469 3.5.1.6. Troubleshooting 471 Detnet leverages many different forwarding sub-layers, each of which 472 supports various tools to troubleshoot connectivity, for example 473 identification of misbehaving flows. The DetNet Service layer can 474 leverage existing mechanisms to troubleshoot or monitor flows, such 475 as those in use by IP and MPLS networks. At the Application layer a 476 client of a DetNet service can use existing techniques to detect and 477 monitor delay and loss. 479 3.5.1.7. Flow recognition for analytics 481 Network analytics can be inherited from the technologies of the 482 Service and Forwarding sub-layers. At the DetNet service edge, 483 packet and bit counters (e.g. sent, received, dropped, and out-of- 484 sequence) can be maintained. 486 3.5.1.8. Correlation of events with flows 488 The provider of a DetNet service may provide other capabilities to 489 monitor flows, such as more detailed loss statistics and time 490 stamping of events. The details of these capabilities are out of 491 scope for this document. 493 3.5.2. Service Protection 495 Service protection allows DetNet services to increase reliability and 496 maintain a DetNet Service Assurance in the case of network congestion 497 or network failure. Detnet relies on the underlying technology 498 capabilities for various protection schemes. Protection schemes 499 enable partial or complete coverage of the network paths and active 500 protection with combinations of PRF, PEF, and POF. 502 3.5.2.1. Linear Service Protection 504 An example DetNet MPLS network fragment and packet flow is 505 illustrated in Figure 3. 507 1 1.1 1.1 1.2.1 1.2.1 1.2.2 508 CE1----EN1--------R1-------R2-------R3--------EN2-----CE2 509 \ 1.2.1 / / 510 \1.2 /-----+ / 511 +------R4------------------------+ 512 1.2.2 514 Figure 3: Example Packet Flow in DetNet Protected Network 516 In Figure 3 the numbers are used to identify the instance of a 517 packet. Packet 1 is the original packet, and packets 1.1, and 1.2 518 are two first generation copies of packet 1. Packet 1.2.1 is a 519 second generation copy of packet 1.2, etc. Note that these numbers 520 never appear in the packet, and are not to be confused with sequence 521 numbers, labels or any other identifier that appears in the packet. 522 They simply indicate the generation number of the original packet so 523 that its passage through the network fragment can be identified to 524 the reader. 526 Customer Equipment CE1 sends a packet into the DetNet enabled 527 network. This is packet (1). Edge Node EN1 encapsulates the packet 528 as a DetNet packet and sends it to Relay node R1 (packet 1.1). EN1 529 makes a copy of the packet (1.2), encapsulates it and sends this copy 530 to Relay node R4. 532 Note that along the path from EN1 to R1 there may be zero or more 533 nodes which, for clarity, are not shown. The same is true for any 534 other path between two DetNet entities shown in Figure 3. 536 Relay node R4 has been configured to send one copy of the packet to 537 Relay Node R2 (packet 1.2.1) and one copy to Edge Node EN2 (packet 538 1.2.2). 540 R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and, 541 having been configured to perform packet elimination on this DetNet 542 flow, forwards packet 1.2.1 to Relay Node R3. Packet copy 1.1 is of 543 no further use and so is discarded by R2. 545 Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives 546 packet copy 1.2.1 from R2 via relay Node R3. EN2 therefore strips 547 any DetNet encapsulation from packet copy 1.2.2 and forwards the 548 packet to CE2. When EN2 receives the later packet copy 1.2.1 this is 549 discarded. 551 The above is of course illustrative of many network scenarios that 552 can be configured. 554 This example also illustrates 1:1 protection scheme meaning there is 555 traffic over each segment of the end-to-end path. Local DetNet relay 556 nodes determine which packets are eliminated and which packets are 557 forwarded. A 1+1 scheme where only one path is used for traffic at a 558 time could use the same topology. In this case there is no PRF 559 function and traffic is switched upon detection of failure. An OAM 560 scheme that monitors the paths to detect the loss of path or traffic 561 is required to initiate the switch. A POF may still be used in this 562 case to prevent misordering of packets. In both cases the protection 563 paths are established and maintained for the duration of the DetNet 564 service. 566 3.5.2.2. Path Differential Delay 568 In the preceding example, proper working of duplicate elimination and 569 reordering of packets are dependent on the number of out-of-order 570 packets that can be buffered and the delay difference of arriving 571 packets. DetNet uses flow-specific requirements (e.g., maximum 572 number of out-of-order packets, maximum latency of the flow) for 573 configuration of POF-related buffers. If the differential delay 574 between paths is excessively large or there is excessive mis-ordering 575 of the packets, then packets may be dropped instead of being 576 reordered. Likewise, PEF uses the sequence number to identify 577 duplicate packets, and large differential delays combined with high 578 numbers of packets may exceed the ability of the PEF to work 579 properly. 581 3.5.2.3. Ring Service Protection 583 Ring protection may also be supported if the underlying technology 584 supports it. Many of the same concepts apply, however rings are 585 normally 1+1 protection for data efficiency reasons. [RFC8227] is an 586 example of MPLS-TP data plane that supports Ring protection. 588 3.5.3. Aggregation Considerations 590 The DetNet data plane also allows for the aggregation of DetNet 591 flows, which can improve scalability by reducing the per-hop state. 592 How this is accomplished is data plane or control plane dependent. 593 When DetNet flows are aggregated, transit nodes provide service to 594 the aggregate and not on a per-DetNet flow basis. When aggregating 595 DetNet flows, the flows should be compatible, i.e., the same or very 596 similar QoS and CoS characteristics. In this case, nodes performing 597 aggregation will ensure that per-flow service requirements are 598 achieved. 600 If bandwidth reservations are used, the reservation should be the sum 601 of all the individual reservations; in other words, the reservations 602 should not add up to an over-subscription of bandwidth reservation. 603 If maximum delay bounds are used, the system should ensure that the 604 aggregate does not exceed the delay bounds of the individual flows. 606 When an encapsulation is used, the choice of reserving a maximum 607 resource level and then tracking the services in the aggregated 608 service or adjusting the aggregated resources as the services are 609 added is implementation and technology specific. 611 DetNet flows at edges must be able to handle rejection to an 612 aggregation group due to lack of resources as well as conditions 613 where requirements are not satisfied. 615 3.5.3.1. IP Aggregation 617 IP aggregation has both data plane and controller plane aspects. For 618 the data plane, flows may be aggregated for treatment based on shared 619 characteristics such as 6-tuple [I-D.ietf-detnet-ip]. Alternatively, 620 an IP encapsulation may be used to tunnel an aggregate number of 621 DetNet Flows between relay nodes. 623 3.5.3.2. MPLS Aggregation 625 MPLS aggregation also has data plane and controller plane aspects. 626 MPLS flows are often tunneled in a forwarding sub-layer, under the 627 reservation associated with that MPLS tunnel. 629 3.5.4. End-System-Specific Considerations 631 Data-flows requiring DetNet service are generated and terminated on 632 end-systems. Encapsulation depends on the application and its 633 preferences. For example, in a DetNet MPLS domain the sub-layer 634 functions use the d-CWs, S-Labels and F-Labels to provide DetNet 635 services. However, an application may exchange further flow related 636 parameters (e.g., time-stamp), which are not provided by DetNet 637 functions. 639 As a general rule, DetNet domains are capable of forwarding any 640 DetNet flows and the DetNet domain does not mandate the end-system or 641 edge node encapsulation format. Unless there is a proxy of some form 642 present, end-systems peer with similar end-systems using the same 643 application encapsulation format. For example, as shown in Figure 4, 644 IP applications peer with IP applications and Ethernet applications 645 peer with Ethernet applications. 647 +-----+ 648 | X | +-----+ 649 +-----+ | X | 650 | Eth | ________ +-----+ 651 +-----+ _____ / \ | Eth | 652 \ / \__/ \___ +-----+ 653 \ / \ / 654 0======== tunnel-1 ========0_ 655 | \ 656 \ | 657 0========= tunnel-2 =========0 658 / \ __/ \ 659 +-----+ \__ DetNet MPLS domain / \ 660 | X | \ __ / +-----+ 661 +-----+ \_______/ \_____/ | X | 662 | IP | +-----+ 663 +-----+ | IP | 664 +-----+ 666 Figure 4: End-Systems and The DetNet MPLS Domain 668 3.5.5. Sub-Network Considerations 670 Any of the DetNet service types may be transported by another DetNet 671 service. MPLS nodes may be interconnected by different sub-network 672 technologies, which may include point-to-point links. Each of these 673 sub-network technologies needs to provide appropriate service to 674 DetNet flows. In some cases, e.g., on dedicated point-to-point links 675 or TDM technologies, all that is required is for a DetNet node to 676 appropriately queue its output traffic. In other cases, DetNet nodes 677 will need to map DetNet flows to the flow semantics (i.e., 678 identifiers) and mechanisms used by an underlying sub-network 679 technology. Figure 5 shows several examples of header formats that 680 can be used to carry DetNet MPLS flows over different sub-network 681 technologies. L2 represents a generic layer-2 encapsulation that 682 might be used on a point-to-point link. TSN represents the 683 encapsulation used on an IEEE 802.1 TSN network, as described in 684 [I-D.ietf-detnet-mpls-over-tsn]. UDP/IP represents the encapsulation 685 used on a DetNet IP PSN, as described in 686 [I-D.ietf-detnet-mpls-over-udp-ip]. 688 +------+ +------+ +------+ 689 App-Flow | X | | X | | X | 690 +-----+======+--+======+--+======+-----+ 691 DetNet-MPLS | d-CW | | d-CW | | d-CW | 692 +------+ +------+ +------+ 693 |Labels| |Labels| |Labels| 694 +-----+======+--+======+--+======+-----+ 695 Sub-Network | L2 | | TSN | | UDP | 696 +------+ +------+ +------+ 697 | IP | 698 +------+ 699 | L2 | 700 +------+ 702 Figure 5: Example DetNet MPLS Sub-Network Formats 704 4. Controller Plane (Management and Control) Considerations 706 4.1. DetNet Controller Plane Requirements 708 The Controller Plane corresponds to the aggregation of the Control 709 and Management Planes discussed in [RFC7426] and [RFC8655]. While 710 more details of any DetNet controller plane are out of the scope of 711 this document, there are particular considerations and requirements 712 for such that result from the unique characteristics of the DetNet 713 architecture and data plane as defined herein. 715 The primary requirements of the DetNet controller plane are that it 716 must be able to: 718 o Instantiate DetNet flows in a DetNet domain (which may include 719 some or all of explicit path determination, link bandwidth 720 reservations, restricting flows to IEEE 802.1 TSN links, node 721 buffer and other resource reservations, specification of required 722 queuing disciplines along the path, ability to manage 723 bidirectional flows, etc.) as needed for a flow. 725 o In the case of MPLS, manage DetNet S-Label and F-Label allocation 726 and distribution, where the DetNet MPLS encapsulation is in use 727 see [I-D.ietf-detnet-mpls]. 729 o Support DetNet flow aggregation. 731 o Advertise static and dynamic node and link resources such as 732 capabilities and adjacencies to other network nodes (for dynamic 733 signaling approaches) or to network controllers (for centralized 734 approaches). 736 o Scale to handle the number of DetNet flows expected in a domain 737 (which may require per-flow signaling or provisioning). 739 o Provision flow identification information at each of the nodes 740 along the path. Flow identification may differ depending on the 741 location in the network and the DetNet functionality (e.g. transit 742 node vs. relay node). 744 These requirements, as stated earlier, could be satisfied using 745 distributed control protocol signaling (such as RSVP-TE), centralized 746 network management provisioning mechanisms (such as BGP, PCEP, YANG 747 [I-D.ietf-detnet-flow-information-model], etc.) or hybrid 748 combinations of the two, and could also make use of MPLS-based 749 segment routing. 751 In the abstract, the results of either distributed signaling or 752 centralized provisioning are equivalent from a DetNet data plane 753 perspective - flows are instantiated, explicit routes are determined, 754 resources are reserved, and packets are forwarded through the domain 755 using the DetNet data plane. 757 However, from a practical and implementation standpoint, controller 758 plane alternatives are not equivalent at all. Some approaches are 759 more scalable than others in terms of signaling load on the network. 760 Some alternatives can take advantage of global tracking of resources 761 in the DetNet domain for better overall network resource 762 optimization. Some solutions are more resilient than others if link, 763 node, or management equipment failures occur. While a detailed 764 analysis of the control plane alternatives is out of the scope of 765 this document, the requirements from this document can be used as the 766 basis of a later analysis of the alternatives. 768 4.2. Generic Controller Plane Considerations 770 This section covers control plane considerations that are independent 771 of the data plane technology used for DetNet service delivery. 773 While the management plane and control planes are traditionally 774 considered separately, from the data plane perspective there is no 775 practical difference based on the origin of flow provisioning 776 information, and the DetNet architecture [RFC8655] refers to these 777 collectively as the 'Controller Plane'. This document therefore does 778 not distinguish between information provided by distributed control 779 plane protocols, e.g., RSVP-TE [RFC3209] and [RFC3473], or by 780 centralized network management mechanisms, e.g., RestConf [RFC8040], 781 YANG [RFC7950], and the Path Computation Element Communication 782 Protocol (PCEP) [I-D.ietf-pce-pcep-extension-for-pce-controller] or 783 any combination thereof. Specific considerations and requirements 784 for the DetNet Controller Plane are discussed in Section 4.1. 786 Each respective data plane document also covers the control plane 787 considerations for that technology. For example, 788 [I-D.ietf-detnet-ip] covers IP control plane normative considerations 789 and [I-D.ietf-detnet-mpls] covers MPLS control plane normative 790 considerations. 792 4.2.1. Flow Aggregation Control 794 Flow aggregation means that multiple app-flows are served by a single 795 new DetNet flow. There are many techniques to achieve aggregation. 796 For example, in the case of IP, IP flows that share 6-tuple 797 attributes or flow identifiers at the DetNet sub-layer can be 798 grouped. Another example includes aggregation accomplished through 799 the use of hierarchical LSPs in MPLS and tunnels. 801 Control of aggregation involves a set of procedures listed here. 802 Aggregation may use some or all of these capabilities and the order 803 may vary: 805 o Traffic engineering resource collection and distribution: 807 Available resources are tracked through control plane or 808 management plane databases and distributed amongst controllers 809 or nodes that can manage resources. 811 o Path computation and resource allocation: 813 When DetNet services are provisioned or requested, one or more 814 paths meeting the requirements are selected and the resources 815 verified and recorded. 817 o Resource assignment and data plane co-ordination: 819 The assignment of resources along the path depends on the 820 technology and includes assignment of specific links, 821 coordination of queueing, and other traffic management 822 capabilities such as policing and shaping. 824 o Assigned Resource recording and updating: 826 Depending on the specific technology, the assigned resources 827 are updated and distributed in the databases, preventing over- 828 subscription. 830 4.2.2. Explicit Routes 832 Explicit routes are used to ensure that packets are routed through 833 the resources that have been reserved for them, and hence provide the 834 DetNet application with the required service. A requirement for the 835 DetNet Controller Plane will be the ability to assign a particular 836 identified DetNet IP flow to a path through the DetNet domain that 837 has been assigned the required per-node resources. This provides the 838 appropriate traffic treatment for the flow and also includes 839 particular links as a part of the path that are able to support the 840 DetNet flow. For example, by using IEEE 802.1 TSN links (as 841 discussed in [I-D.ietf-detnet-mpls-over-tsn] ) DetNet parameters can 842 be maintained. Further considerations and requirements for the 843 DetNet Controller Plane are discussed in Section 4.1. 845 Whether configuring, calculating and instantiating these routes is a 846 single-stage or multi-stage process, or in a centralized or 847 distributed manner, is out of scope of this document. 849 There are several approaches that could be used to provide explicit 850 routes and resource allocation in the DetNet forwarding sub-layer. 851 For example: 853 o The path could be explicitly set up by a controller which 854 calculates the path and explicitly configures each node along that 855 path with the appropriate forwarding and resource allocation 856 information. 858 o The path could use a distributed control plane such as RSVP 859 [RFC2205] or RSVP-TE [RFC3473] extended to support DetNet IP 860 flows. 862 o The path could be implemented using IPv6-based segment routing 863 when extended to support resource allocation. 865 See Section 4.1 for further discussion of these alternatives. In 866 addition, [RFC2386] contains useful background information on QoS- 867 based routing, and [RFC5575] (being updated by 868 [I-D.ietf-idr-rfc5575bis]) discusses a specific mechanism used by BGP 869 for traffic flow specification and policy-based routing. 871 4.2.3. Contention Loss and Jitter Reduction 873 As discussed in Section 1, this document does not specify the 874 mechanisms needed to eliminate packet contention, packet loss or 875 reduce jitter for DetNet flows at the DetNet forwarding sub-layer. 876 The ability to manage node and link resources to be able to provide 877 these functions is a necessary part of the DetNet controller plane. 878 It is also necessary to be able to control the required queuing 879 mechanisms used to provide these functions along a flow's path 880 through the network. See [I-D.ietf-detnet-ip] and Section 4.1 for 881 further discussion of these requirements. Some forms of protection 882 may minimize packet loss or change jitter characteristics in the 883 cases where packets are reordered when out-of-order packets are 884 received at the service sub-layer. 886 4.2.4. Bidirectional Traffic 888 In many cases DetNet flows can be considered unidirectional and 889 independent. However, there are cases where the DetNet service 890 requires bidirectional traffic from a DetNet application service 891 perspective. IP and MPLS typically treat each direction separately 892 and do not force interdependence of each direction. MPLS has 893 considered bidirectional traffic requirements and the MPLS 894 definitions from [RFC5654] are useful to illustrate terms such as 895 associated bidirectional flows and co-routed bidirectional flows. 896 MPLS defines a point-to-point associated bidirectional LSP as 897 consisting of two unidirectional point-to-point LSPs, one from A to B 898 and the other from B to A, which are regarded as providing a single 899 logical bidirectional forwarding path. This is analogous to standard 900 IP routing. MPLS defines a point-to-point co-routed bidirectional 901 LSP as an associated bidirectional LSP which satisfies the additional 902 constraint that its two unidirectional component LSPs follow the same 903 path (in terms of both nodes and links) in both directions. An 904 important property of co-routed bidirectional LSPs is that their 905 unidirectional component LSPs share fate. In both types of 906 bidirectional LSPs, resource reservations may differ in each 907 direction. The concepts of associated bidirectional flows and co- 908 routed bidirectional flows can also be applied to DetNet IP flows. 910 While the DetNet IP data plane must support bidirectional DetNet 911 flows, there are no special bidirectional features with respect to 912 the data plane other than the need for the two directions of a co- 913 routed bidirectional flow to take the same path. That is to say that 914 bidirectional DetNet flows are solely represented at the management 915 and control plane levels, without specific support or knowledge 916 within the DetNet data plane. Fate sharing and associated or co- 917 routed bidirectional flows, can be managed at the control level. 919 DetNet's use of PREOF may increase the complexity of using co-routing 920 bidirectional flows, since if PREOF is used, then the replication 921 points in one direction would have to match the elimination points in 922 the other direction, and vice versa. In such cases the optimal 923 points for these functions in one direction may not match the optimal 924 points in the other, due to network and traffic constraints. 925 Furthermore, due to the per packet service protection nature, 926 bidirectional forwarding per packet may not be ensured. The first 927 packet of received member flows is selected by the elimination 928 function independently of which path it has taken through the 929 network. 931 Control and management mechanisms need to support bidirectional 932 flows, but the specification of such mechanisms are out of scope of 933 this document. Example control plane solutions for MPLS can be found 934 in [RFC3473] , [RFC6387] and [RFC7551]. These requirements are 935 included in Section 4.1. 937 4.3. Packet Replication, Elimination, and Ordering (PREOF) 939 The controller plane protocol solution required for managing the 940 PREOF processing is outside the scope of this document. That said, 941 it should be noted that the ability to determine, for a particular 942 flow, optimal packet replication and elimination points in the DetNet 943 domain requires explicit support. There may be existing capabilities 944 that can be used, or extended, for example GMPLS end-to-end recovery 945 [RFC4872] and GMPLS segment recovery [RFC4873]. 947 5. Security Considerations 949 Security considerations for DetNet are described in detail in 950 [I-D.ietf-detnet-security]. General security considerations for 951 DetNet architecture are described in [RFC8655]. This section 952 considers architecture-level DetNet security considerations 953 applicable to all data planes. 955 Part of what makes DetNet unique is its ability to provide specific 956 and reliable quality of service (delivering data flows with extremely 957 low packet loss rates and bounded end-to-end delivery latency), and 958 the security-related aspects of protecting that quality of service 959 are similarly unique. 961 As for all communications protocols, the primary consideration for 962 the data plane is to maintain integrity of data and delivery of the 963 associated DetNet service traversing the DetNet network. Application 964 flows can be protected through whatever means is provided by the 965 underlying technology. For example, encryption may be used, such as 966 that provided by IPSec [RFC4301] for IP flows and/or by an underlying 967 sub-net using MACSec [IEEE802.1AE-2018] for Ethernet (Layer-2) flows. 969 At the management and control level DetNet flows are identified on a 970 per-flow basis, which may provide controller plane attackers with 971 additional information about the data flows (when compared to 972 controller planes that do not include per-flow identification). This 973 is an inherent property of DetNet which has security implications 974 that should be considered when determining if DetNet is a suitable 975 technology for any given use case. 977 To provide uninterrupted availability of the DetNet service, 978 provisions can be made against DOS attacks and delay attacks. To 979 protect against DOS attacks, excess traffic due to malicious or 980 malfunctioning devices can be prevented or mitigated, for example 981 through the use of existing mechanisms such as policing and shaping 982 applied at the input of a DetNet domain. To prevent DetNet packets 983 from being delayed by an entity external to a DetNet domain, DetNet 984 technology definition can allow for the mitigation of Man-In-The- 985 Middle attacks, for example through use of authentication and 986 authorization of devices within the DetNet domain. 988 In order to prevent or mitigate DetNet attacks on other networks via 989 flow escape, edge devices can for example use existing mechanism such 990 as policing and shaping applied at the output of a DetNet domain. 992 6. IANA Considerations 994 This document makes no IANA requests. 996 7. Acknowledgements 998 The authors wish to thank Pat Thaler, Norman Finn, Loa Anderson, 999 David Black, Rodney Cummings, Ethan Grossman, Tal Mizrahi, David 1000 Mozes, Craig Gunther, George Swallow, Yuanlong Jiang and Carlos J. 1001 Bernardos for their various contributions to this work. 1003 8. Contributors 1005 The following people contributed substantially to the content of this 1006 document: 1008 Don Fedyk 1009 Jouni Korhonen 1011 9. References 1013 9.1. Normative References 1015 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1016 "Deterministic Networking Architecture", RFC 8655, 1017 DOI 10.17487/RFC8655, October 2019, 1018 . 1020 9.2. Informative References 1022 [I-D.ietf-detnet-flow-information-model] 1023 Farkas, J., Varga, B., Cummings, R., Jiang, Y., and D. 1024 Fedyk, "DetNet Flow Information Model", draft-ietf-detnet- 1025 flow-information-model-07 (work in progress), March 2020. 1027 [I-D.ietf-detnet-ip] 1028 Varga, B., Farkas, J., Berger, L., Fedyk, D., Malis, A., 1029 and S. Bryant, "DetNet Data Plane: IP", draft-ietf-detnet- 1030 ip-05 (work in progress), February 2020. 1032 [I-D.ietf-detnet-mpls] 1033 Varga, B., Farkas, J., Berger, L., Fedyk, D., Malis, A., 1034 Bryant, S., and J. Korhonen, "DetNet Data Plane: MPLS", 1035 draft-ietf-detnet-mpls-05 (work in progress), February 1036 2020. 1038 [I-D.ietf-detnet-mpls-over-tsn] 1039 Varga, B., Farkas, J., Malis, A., and S. Bryant, "DetNet 1040 Data Plane: MPLS over IEEE 802.1 Time Sensitive Networking 1041 (TSN)", draft-ietf-detnet-mpls-over-tsn-02 (work in 1042 progress), March 2020. 1044 [I-D.ietf-detnet-mpls-over-udp-ip] 1045 Varga, B., Farkas, J., Berger, L., Malis, A., and S. 1046 Bryant, "DetNet Data Plane: MPLS over UDP/IP", draft-ietf- 1047 detnet-mpls-over-udp-ip-05 (work in progress), February 1048 2020. 1050 [I-D.ietf-detnet-security] 1051 Mizrahi, T. and E. Grossman, "Deterministic Networking 1052 (DetNet) Security Considerations", draft-ietf-detnet- 1053 security-09 (work in progress), March 2020. 1055 [I-D.ietf-idr-rfc5575bis] 1056 Loibl, C., Hares, S., Raszuk, R., McPherson, D., and M. 1057 Bacher, "Dissemination of Flow Specification Rules", 1058 draft-ietf-idr-rfc5575bis-24 (work in progress), April 1059 2020. 1061 [I-D.ietf-pce-pcep-extension-for-pce-controller] 1062 Zhao, Q., Li, Z., Negi, M., Peng, S., and C. Zhou, "PCEP 1063 Procedures and Protocol Extensions for Using PCE as a 1064 Central Controller (PCECC) of LSPs", draft-ietf-pce-pcep- 1065 extension-for-pce-controller-04 (work in progress), March 1066 2020. 1068 [I-D.ietf-spring-srv6-network-programming] 1069 Filsfils, C., Camarillo, P., Leddy, J., Voyer, D., 1070 Matsushima, S., and Z. Li, "SRv6 Network Programming", 1071 draft-ietf-spring-srv6-network-programming-15 (work in 1072 progress), March 2020. 1074 [IEEE802.1AE-2018] 1075 IEEE Standards Association, "IEEE Std 802.1AE-2018 MAC 1076 Security (MACsec)", 2018, 1077 . 1079 [IEEE802.1TSNTG] 1080 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1081 Networking Task Group", . 1083 [nwcrg] IRTF, "Coding for efficient NetWork Communications 1084 Research Group (nwcrg)", 1085 . 1087 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1088 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1089 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1090 September 1997, . 1092 [RFC2386] Crawley, E., Nair, R., Rajagopalan, B., and H. Sandick, "A 1093 Framework for QoS-based Routing in the Internet", 1094 RFC 2386, DOI 10.17487/RFC2386, August 1998, 1095 . 1097 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1098 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1099 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1100 . 1102 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 1103 Switching (GMPLS) Signaling Resource ReserVation Protocol- 1104 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 1105 DOI 10.17487/RFC3473, January 2003, 1106 . 1108 [RFC3670] Moore, B., Durham, D., Strassner, J., Westerinen, A., and 1109 W. Weiss, "Information Model for Describing Network Device 1110 QoS Datapath Mechanisms", RFC 3670, DOI 10.17487/RFC3670, 1111 January 2004, . 1113 [RFC4301] Kent, S. and K. Seo, "Security Architecture for the 1114 Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, 1115 December 2005, . 1117 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 1118 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 1119 Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385, 1120 February 2006, . 1122 [RFC4872] Lang, J., Ed., Rekhter, Y., Ed., and D. Papadimitriou, 1123 Ed., "RSVP-TE Extensions in Support of End-to-End 1124 Generalized Multi-Protocol Label Switching (GMPLS) 1125 Recovery", RFC 4872, DOI 10.17487/RFC4872, May 2007, 1126 . 1128 [RFC4873] Berger, L., Bryskin, I., Papadimitriou, D., and A. Farrel, 1129 "GMPLS Segment Recovery", RFC 4873, DOI 10.17487/RFC4873, 1130 May 2007, . 1132 [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J., 1133 and D. McPherson, "Dissemination of Flow Specification 1134 Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009, 1135 . 1137 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1138 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1139 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1140 September 2009, . 1142 [RFC6387] Takacs, A., Berger, L., Caviglia, D., Fedyk, D., and J. 1143 Meuric, "GMPLS Asymmetric Bandwidth Bidirectional Label 1144 Switched Paths (LSPs)", RFC 6387, DOI 10.17487/RFC6387, 1145 September 2011, . 1147 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1148 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1149 Defined Networking (SDN): Layers and Architecture 1150 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1151 2015, . 1153 [RFC7551] Zhang, F., Ed., Jing, R., and R. Gandhi, Ed., "RSVP-TE 1154 Extensions for Associated Bidirectional Label Switched 1155 Paths (LSPs)", RFC 7551, DOI 10.17487/RFC7551, May 2015, 1156 . 1158 [RFC7657] Black, D., Ed. and P. Jones, "Differentiated Services 1159 (Diffserv) and Real-Time Communication", RFC 7657, 1160 DOI 10.17487/RFC7657, November 2015, 1161 . 1163 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 1164 RFC 7950, DOI 10.17487/RFC7950, August 2016, 1165 . 1167 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 1168 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 1169 . 1171 [RFC8227] Cheng, W., Wang, L., Li, H., van Helvoort, H., and J. 1172 Dong, "MPLS-TP Shared-Ring Protection (MSRP) Mechanism for 1173 Ring Topology", RFC 8227, DOI 10.17487/RFC8227, August 1174 2017, . 1176 Authors' Addresses 1178 Balazs Varga (editor) 1179 Ericsson 1180 Magyar Tudosok krt. 11. 1181 Budapest 1117 1182 Hungary 1184 Email: balazs.a.varga@ericsson.com 1185 Janos Farkas 1186 Ericsson 1187 Magyar Tudosok krt. 11. 1188 Budapest 1117 1189 Hungary 1191 Email: janos.farkas@ericsson.com 1193 Lou Berger 1194 LabN Consulting, L.L.C. 1196 Email: lberger@labn.net 1198 Andrew G. Malis 1199 Malis Consulting 1201 Email: agmalis@gmail.com 1203 Stewart Bryant 1204 Futurewei Technologies 1206 Email: stewart.bryant@gmail.com