idnits 2.17.1 draft-ietf-detnet-data-plane-framework-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 196 has weird spacing: '...e stack v ...' -- The document date (February 3, 2020) is 1544 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-14) exists of draft-ietf-detnet-flow-information-model-06 == Outdated reference: A later version (-07) exists of draft-ietf-detnet-ip-04 == Outdated reference: A later version (-13) exists of draft-ietf-detnet-mpls-04 == Outdated reference: A later version (-07) exists of draft-ietf-detnet-mpls-over-tsn-01 == Outdated reference: A later version (-08) exists of draft-ietf-detnet-mpls-over-udp-ip-04 == Outdated reference: A later version (-16) exists of draft-ietf-detnet-security-07 == Outdated reference: A later version (-14) exists of draft-ietf-pce-pcep-extension-for-pce-controller-03 == Outdated reference: A later version (-28) exists of draft-ietf-spring-srv6-network-programming-08 -- Obsolete informational reference (is this intentional?): RFC 5575 (Obsoleted by RFC 8955) Summary: 0 errors (**), 0 flaws (~~), 10 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet B. Varga, Ed. 3 Internet-Draft J. Farkas 4 Intended status: Informational Ericsson 5 Expires: August 6, 2020 L. Berger 6 LabN Consulting, L.L.C. 7 A. Malis 8 Independent 9 S. Bryant 10 Futurewei Technologies 11 February 3, 2020 13 DetNet Data Plane Framework 14 draft-ietf-detnet-data-plane-framework-04 16 Abstract 18 This document provides an overall framework for the DetNet data 19 plane. It covers concepts and considerations that are generally 20 common to any Deterministic Networking data plane specification. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on August 6, 2020. 39 Copyright Notice 41 Copyright (c) 2020 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (https://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 57 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 58 2.1. Terms Used in This Document . . . . . . . . . . . . . . . 4 59 2.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4 60 3. DetNet Data Plane Overview . . . . . . . . . . . . . . . . . 4 61 3.1. Data Plane Characteristics . . . . . . . . . . . . . . . 6 62 3.1.1. Data Plane Technology . . . . . . . . . . . . . . . . 6 63 3.1.2. Data Plane Format . . . . . . . . . . . . . . . . . . 6 64 3.2. Encapsulation . . . . . . . . . . . . . . . . . . . . . . 6 65 3.3. DetNet Specific Metadata . . . . . . . . . . . . . . . . 7 66 3.4. DetNet IP Data Plane . . . . . . . . . . . . . . . . . . 8 67 3.5. DetNet MPLS Data Plane . . . . . . . . . . . . . . . . . 9 68 3.6. Further DetNet Data Plane Considerations . . . . . . . . 9 69 3.6.1. Per Flow Related Functions . . . . . . . . . . . . . 9 70 3.6.2. Service Protection . . . . . . . . . . . . . . . . . 11 71 3.6.3. Aggregation Considerations . . . . . . . . . . . . . 13 72 3.6.4. End-System-Specific Considerations . . . . . . . . . 14 73 3.6.5. Sub-Network Considerations . . . . . . . . . . . . . 15 74 4. Controller Plane (Management and Control) 75 Considerations . . . . . . . . . . . . . . . . . . . . . . . 16 76 4.1. DetNet Controller Plane Requirements . . . . . . . . . . 16 77 4.2. Generic Controller Plane Considerations . . . . . . . . . 17 78 4.2.1. Flow Aggregation Control . . . . . . . . . . . . . . 18 79 4.2.2. Explicit Routes . . . . . . . . . . . . . . . . . . . 19 80 4.2.3. Contention Loss and Jitter Reduction . . . . . . . . 19 81 4.2.4. Bidirectional Traffic . . . . . . . . . . . . . . . . 20 82 4.3. Packet Replication, Elimination, and Ordering (PREOF) . . 21 83 5. Security Considerations . . . . . . . . . . . . . . . . . . . 21 84 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 85 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 22 86 8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 22 87 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 88 9.1. Normative References . . . . . . . . . . . . . . . . . . 22 89 9.2. Informative References . . . . . . . . . . . . . . . . . 23 90 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 26 92 1. Introduction 94 DetNet (Deterministic Networking) provides a capability to carry 95 specified unicast or multicast data flows for real-time applications 96 with extremely low packet loss rates and assured maximum end-to-end 97 delivery latency. A description of the general background and 98 concepts of DetNet can be found in [RFC8655]. 100 This document describes the concepts needed by any DetNet data plane 101 specification and provides considerations for any corresponding 102 implementation. It covers the building blocks that provide the 103 DetNet service, the DetNet service sub-layer and the DetNet 104 forwarding sub-layer functions as described in the DetNet 105 Architecture. 107 The DetNet Architecture models the DetNet related data plane 108 functions decomposed into two sub-layers: a service sub-layer and a 109 forwarding sub-layer. The service sub-layer is used to provide 110 DetNet service protection and reordering. The forwarding sub-layer 111 leverages Traffic Engineering mechanisms and provides congestion 112 protection (low loss, assured latency, and limited out-of-order 113 delivery). A particular forwarding sub-layer may have capabilities 114 that are not available on other forwarding-sub layers. DetNet makes 115 use of the existing forwarding sub-layers with their respective 116 capabilities and does not require 1:1 equivalence between different 117 forwarding sub-layer capabilities. 119 As part of the service sub-layer functions, this document describes 120 typical DetNet node data plane operation. It describes the function 121 and operation of the Packet Replication (PRF) Packet Elimination 122 (PEF) and the Packet Ordering (POF) functions within the service sub- 123 layer. Furthermore, it also describes the forwarding sub-layer. 125 DetNet flows may be carried over network technologies that can 126 provide the DetNet required service characteristics. For example, 127 DetNet MPLS flows can be carried over IEEE 802.1 Time Sensitive 128 Network (TSN) [IEEE802.1TSNTG] sub-networks. However, IEEE 802.1 TSN 129 support is not required in DetNet. TSN frame preemption is an 130 example of a forwarding layer capability that is typically not 131 replicated in other forwarding technologies. Most of DetNet benefits 132 can be gained by running over a data link layer that has not been 133 specifically enhanced to support all TSN capabilities but for certain 134 networks and traffic mixes delay and jitter performance may vary due 135 to the forwarding sub-layer intrinsic properties. 137 Different application flows (e.g., Ethernet, IP, etc.), can be mapped 138 on top of DetNet. DetNet can optionally reuse header information 139 provided by, or shared with, applications. An example of shared 140 header fields can be found in [I-D.ietf-detnet-ip]. 142 This document also covers basic concepts related to the controller 143 plane and Operations, Administration, and Maintenance (OAM). Data 144 plane OAM specifics are out of scope for this document. 146 2. Terminology 148 2.1. Terms Used in This Document 150 This document uses the terminology established in the DetNet 151 architecture [RFC8655], and the reader is assumed to be familiar with 152 that document and its terminology. 154 2.2. Abbreviations 156 The following abbreviations are used in this document: 158 BGP Border Gateway Protocol. 159 CW Control Word. 160 d-CW DetNet Control Word. 161 DetNet Deterministic Networking. 162 DN DetNet. 163 GMPLS Generalized Multiprotocol Label Switching. 164 GRE Generic Routing Encapsulation. 165 IPSec IP Security. 166 L2 Layer 2. 167 LSP Label Switched Path. 168 LSR Label Switching Router. 169 MPLS Multiprotocol Label Switching. 170 MPLS-TE Multiprotocol Label Switching - Traffic Engineering. 171 OAM Operations, Administration, and Maintenance. 172 PCEP Path Computation Element Communication Protocol. 173 PEF Packet Elimination Function. 174 PRF Packet Replication Function. 175 PREOF Packet Replication, Elimination and Ordering Functions. 176 POF Packet Ordering Function. 177 PSN Packet Switched Network. 178 PW PseudoWire. 179 QoS Quality of Service. 180 S-Label DetNet "service" label. 181 TDM Time-Division Multiplexing. 182 TSN Time-Sensitive Network. 183 YANG Yet Another Next Generation. 185 3. DetNet Data Plane Overview 187 This document describes how application flows, or app-flows, are 188 carried over DetNet networks. The DetNet Architecture, [RFC8655], 189 models the DetNet related data plane functions as decomposed into two 190 sub-layers: a service sub-layer and a forwarding sub-layer. 192 Figure 1 reproduced from the [RFC8655],shows a logical DetNet service 193 with the two sub-layers. 195 | packets going | ^ packets coming ^ 196 v down the stack v | up the stack | 197 +-----------------------+ +-----------------------+ 198 | Source | | Destination | 199 +-----------------------+ +-----------------------+ 200 | Service sub-layer: | | Service sub-layer: | 201 | Packet sequencing | | Duplicate elimination | 202 | Flow replication | | Flow merging | 203 | Packet encoding | | Packet decoding | 204 +-----------------------+ +-----------------------+ 205 | Forwarding sub-layer: | | Forwarding sub-layer: | 206 | Resource allocation | | Resource allocation | 207 | Explicit routes | | Explicit routes | 208 +-----------------------+ +-----------------------+ 209 | Lower layers | | Lower layers | 210 +-----------------------+ +-----------------------+ 211 v ^ 212 \_________________________/ 214 Figure 1: DetNet data plane protocol stack 216 The DetNet forwarding sub-layer may be directly provided by the 217 DetNet service sub-layer, for example by IP tunnels or MPLS. 218 Alternatively, an overlay approach may be used in which the packet is 219 natively carried between key nodes within the DetNet network (say 220 between PREOF nodes) and a sub-layer is used to provide the 221 information needed to reach the next hop in the overlay. 223 The forwarding sub-layer provides the QoS related functions needed by 224 the DetNet flow. It may do this directly through the use of queuing 225 techniques and traffic engineering methods, or it may do this through 226 the assistance of its underlying connectivity. For example it may 227 call upon Ethernet TSN capabilities defined in IEEE 802.1 TSN 228 [IEEE802.1TSNTG]. The forwarding sub-layer uses buffer resources for 229 packet queuing, as well as reservation and allocation of bandwidth 230 capacity resources. 232 The service sub-layer provides additional support beyond the 233 connectivity function of the forwarding sub-layer. An example of 234 this is Packet Replication, Elimination, and Ordering functions see 235 Section 4.3. The ordering (POF) uses sequence numbers added to 236 packets enabling a range of packet order protection from simple 237 ordering and dropping out-of-order packets to more complex reordering 238 of a fixed number of out-of-order, minimally delayed packets. 239 Reordering requires buffer resources and has impact on the delay and 240 jitter of packets in the DetNet flow. 242 The method of instantiating each of the layers is specific to the 243 particular DetNet data plane method, and more than one approach may 244 be applicable to a given bearer network type. 246 3.1. Data Plane Characteristics 248 There are two major characteristics to the data plane: the technology 249 and the encapsulation, as discussed below. 251 3.1.1. Data Plane Technology 253 The DetNet data plane is provided by the DetNet service and 254 forwarding sub layers. The DetNet service sub-layer generally 255 provides its functions for the DetNet application flows by using or 256 applying existing standardized headers and/or encapsulations. The 257 Detnet forwarding sub-layer may provide capabilities leveraging that 258 same header or encapsulation technology (e.g., DN IP or DN MPLS) or 259 it may be achieved by other technologies (e.g., Figure 2). DetNet is 260 currently defined for operation over packet switched (IP) networks or 261 label switched (MPLS) networks. 263 3.1.2. Data Plane Format 265 DetNet encodes specific flow attributes (flow identity and sequence 266 number) in packets. For example, in DetNet IP, zero encapsulation is 267 used and no sequence number is available, and in DetNet MPLS, DetNet 268 specific information may be added explicitly to the packets in the 269 format of S-label and d-CW [I-D.ietf-detnet-mpls] . 271 3.2. Encapsulation 273 The encapsulation of a DetNet flow allows it to be sent over a data 274 plane technology other than its native type. DetNet uses header 275 information to perform traffic classification, i.e., identify DetNet 276 flows, and provide DetNet service and forwarding functions. As 277 mentioned above, DetNet may add headers, as is the case for DN MPLS, 278 or may use headers that are already present, as is the case in DN IP. 279 Figure 2 illustrates some relationships between the components. 281 +-----+ 282 | TSN | 283 +-------+ +-+-----+-+ 284 | DN IP | | DN MPLS | 285 +--+--+----+----+ +-+---+-----+-+ 286 | TSN | DN MPLS | | TSN | DN IP | 287 +-----+---------+ +-----+-------+ 289 Figure 2: DetNet Service Examples 291 The use of encapsulation is also required if additional information 292 (metadata) is needed by the DetNet data plane and there is either no 293 ability to include it in the client data packet, or the specification 294 of the client data plane does not permit the modification of the 295 packet to include additional data. An example of such metadata is 296 the inclusion of a sequence number required by the PREOF function. 298 Encapsulation may also be used to carry or aggregate flows for 299 equipment with limited DetNet capability. 301 3.3. DetNet Specific Metadata 303 The DetNet data plane can provide or carry metadata: 305 1. Flow-ID 307 2. Sequence Number 309 The DetNet data plane framework supports a Flow-ID (for 310 identification of the flow or aggregate flow) and/or a Sequence 311 Number (for PREOF) for each DetNet flow. The DetNet Service sub- 312 layer requires both; the DetNet forwarding sub-layer requires only 313 Flow-ID. Metadata can also be used for OAM indications and 314 instrumentation of DetNet data plane operation. 316 Metadata can be included implicit or explicit. Explicit means that a 317 dedicated header field is used to include metadata in a DetNet 318 packet. In case of implicit method a part of an already existing 319 header field is used to encode the metadata. 321 Explicit inclusion of metadata is possible through the use of IP 322 options or IP extension headers. New IP options are almost 323 impossible to get standardized or to deploy in an operational network 324 and will not be discussed further in this text. IPv6 extensions 325 headers are finding popularity in current IPv6 development work, 326 particularly in connection with Segment Routing of IPv6 (SRv6) and IP 327 OAM. The design of a new IPv6 extension header or the modification 328 of an existing one is a technique available in the tool box of the 329 DetNet IP data plane designer. 331 Explicit inclusion of metadata in an IP packet is also possible 332 through the inclusion of an MPLS label stack and the MPLS DetNet 333 Control Word using one of the methods for carrying MPLS over IP 334 [I-D.ietf-detnet-mpls-over-udp-ip]. This is described in more detail 335 in Section 3.6.5. 337 Implicit metadata in IP can be included through the use of the 338 network programming paradigm 339 [I-D.ietf-spring-srv6-network-programming] in which the suffix of an 340 IPv6 address is used to encode additional information for use by the 341 network of the receiving host. 343 Some MPLS examples of implicit metadata include the sequence number 344 for use by the PREOF function, or even all the essential information 345 being included into the DetNet over MPLS label stack (the DetNet 346 Control Word and the DetNet Service label). 348 3.4. DetNet IP Data Plane 350 An IP data plane may operate natively or through the use of an 351 encapsulation. Many types of IP encapsulation can satisfy DetNet 352 requirements and it is anticipated that more than one encapsulation 353 may be deployed, for example GRE, IPSec etc. 355 One method of operating an IP DetNet data plane without encapsulation 356 is to use "6-tuple" based flow identification, where "6-tuple" refers 357 to information carried in IP and higher layer protocol headers. 358 General background on the use of IP headers, and "6-tuples", to 359 identify flows and support Quality of Service (QoS) can be found in 360 [RFC3670]. [RFC7657] provides useful background on differentiated 361 services (DiffServ) and "tuple" based flow identification. DetNet 362 flow aggregation may be enabled via the use of wildcards, masks, 363 prefixes and ranges. The operation of this method is described in 364 detail in [I-D.ietf-detnet-ip]. 366 The DetNet forwarding plane may use explicit route capabilities and 367 traffic engineering capabilities to provide a forwarding sub-layer 368 that is responsible for providing resource allocation and explicit 369 routes. It is possible to include such information in a native IP 370 packet explicitly, or implicitly. 372 3.5. DetNet MPLS Data Plane 374 MPLS provides a forwarding sub-layer for traffic over implicit and 375 explicit paths to the point in the network where the next DetNet 376 service sub-layer action needs to take place. It does this through 377 the use of a stack of one or more labels with various forwarding 378 semantics. 380 MPLS also provides the ability to identify a service instance that is 381 used to process the packet through the use of a label that maps the 382 packet to a service instance. 384 In cases where metadata is needed to process an MPLS encapsulated 385 packet at the service sub-layer, the d-CW [I-D.ietf-detnet-mpls], 386 [RFC4385],can be used. Although such d-CWs are frequently 32 bits 387 long, there is no architectural constraint on its size of this 388 structure, only the requirement that it is fully understood by all 389 parties operating on it in the DetNet service sub-layer. The 390 operation of this method is described in detail in 391 [I-D.ietf-detnet-mpls]. 393 3.6. Further DetNet Data Plane Considerations 395 This section provides informative considerations related to providing 396 DetNet service to flows which are identified based on their header 397 information. 399 3.6.1. Per Flow Related Functions 401 At a high level, the following functions are provided on a per flow 402 basis. 404 3.6.1.1. Reservation and Allocation of resources 406 Reservation of resources can allocate resources to specific DetNet 407 flows. This can eliminate packet contention and packet loss for 408 DetNet traffic. This also can reduce jitter for DetNet traffic. 409 Resources allocated to a DetNet flow protect it from other traffic 410 flows. On the other hand, DetNet flows are assumed to behave with 411 respect to the reserved traffic profile. Misbehaving DetNet flows 412 must be able to be detected and ensure that they do not compromise 413 QoS of other flows. Queuing, policing, and shaping policies can be 414 used to ensure that the allocation of resources reserved for DetNet 415 is met. 417 3.6.1.2. Explicit routes 419 Use of a specific path for a flow. This allows control of the 420 network delay by steering the packet with the ability to influence 421 the physical path. Explicit routes complement reservation by 422 ensuring that a consistent path can be associated with its resources 423 for the duration of that path. Coupled with the traffic mechanism, 424 this limits misordering and bounds latency. Explicit route 425 computation can encompass a wide set of constraints and optimize the 426 path for a certain characteristic e.g. highest bandwidth or lowest 427 jitter. In these cases the "best" path for any set of 428 characteristics may not be a shortest path. The selection of path 429 can take into account multiple network metrics. Some of these 430 metrics are measured and distributed by the routing system as traffic 431 engineering metrics. 433 3.6.1.3. Service protection 435 Use of multiple packet streams using multiple paths, for example 1+1 436 or 1:1 linear protection. For DetNet this primarily relates to 437 packet replication and elimination capabilities. MPLS offers a 438 number of protection schemes. MPLS hitless protection can be used to 439 switch traffic to an already established path in order to restore 440 delivery rapidly after a failure. Path changes, even in the case of 441 failure recovery, can lead to the out of order delivery of data 442 requiring packet ordering functions either within the DetNet service 443 or at a high layer in the application traffic. Establishment of new 444 paths after a failure is out of scope for DetNet services. 446 3.6.1.4. Network Coding 448 Network Coding, [nwcrg] not to be confused with network programming, 449 comprises several techniques where multiple data flows are encoded. 450 These resulting flows can then be sent on different paths. The 451 encoding operation can combine flows and error recovery information. 452 When the encoded flows are decoded and recombined the original flows 453 can be recovered. Note that Network coding uses an alternative to 454 packet by packet PREOF. Therefore, for certain network topologies 455 and traffic loads, Network Coding can be used to improve a network's 456 throughput, efficiency, latency, and scalability, as well as 457 resilience to partition, attacks, and eavesdropping, as compared to 458 traditional methods. DetNet could utilized Network coding as an 459 alternative to other protection means. Network coding is often 460 applied in wireless networks and is being explored for other network 461 types. 463 3.6.1.5. Load sharing 465 Use of packet-by-packet distribution of the same DetNet flow over 466 multiple paths is not recommended except for the cases listed above 467 where PREOF is utilized to improve protection of traffic and maintain 468 order. Packet by packet load sharing, e.g., via ECMP or UCMP, 469 impacts ordering and possibly jitter. 471 3.6.1.6. Troubleshooting 473 Detnet leverages many different forwarding sub-layers, each of which 474 supports various tools to troubleshoot connectivity, for example 475 identification of misbehaving flows. The DetNet Service layer can 476 leverage existing mechanisms to troubleshoot or monitor flows, such 477 as those in use by IP and MPLS networks. At the Application layer a 478 client of a DetNet service can use existing techniques to detect and 479 monitor delay and loss. 481 3.6.1.7. Flow recognition for analytics 483 Network analytics can be inherited from the technologies of the 484 Service and Forwarding sub-layers. At the DetNet service edge, 485 packet and bit counters (e.g. sent, received, dropped, and out-of- 486 sequence) can be maintained. 488 3.6.1.8. Correlation of events with flows 490 The provider of a DetNet service may provide other capabilities to 491 monitor flows, such as more detailed loss statistics and time 492 stamping of events. The details of these capabilities are currently 493 out of scope for this document. 495 3.6.2. Service Protection 497 Service protection allow DetNet services to increase reliability and 498 maintain a DetNet Service Assurance in the case of network congestion 499 or network failure. Detnet relies on the underlying technology 500 capabilities for various protection schemes. Protection schemes 501 enable partial or complete coverage of the network paths and active 502 protection with combinations of PRF, PEF, and POF. 504 3.6.2.1. Linear Service Protection 506 An example DetNet MPLS network fragment and packet flow is 507 illustrated in Figure 3. 509 1 1.1 1.1 1.2.1 1.2.1 1.2.2 510 CE1----EN1--------R1-------R2-------R3--------EN2-----CE2 511 \ 1.2.1 / / 512 \1.2 /-----+ / 513 +------R4------------------------+ 514 1.2.2 516 Figure 3: Example Packet Flow in DetNet protected Network 518 In Figure 3 the numbers are used to identify the instance of a 519 packet. Packet 1 is the original packet, and packets 1.1, and 1.2 520 are two first generation copies of packet 1. Packet 1.2.1 is a 521 second generation copy of packet 1.2 etc. Note that these numbers 522 never appear in the packet, and are not to be confused with sequence 523 numbers, labels or any other identifier that appears in the packet. 524 They simply indicate the generation number of the original packet so 525 that its passage through the network fragment can be identified to 526 the reader. 528 Customer Equipment CE1 sends a packet into the DetNet enabled 529 network. This is packet (1). Edge Node EN1 encapsulates the packet 530 as a DetNet Packet and sends it to Relay node R1 (packet 1.1). EN1 531 makes a copy of the packet (1.2), encapsulates it and sends this copy 532 to Relay node R4. 534 Note that along the path from EN1 to R1 there may be zero or more 535 nodes which, for clarity, are not shown. The same is true for any 536 other path between two DetNet entities shown in Figure 3 . 538 Relay node R4 has been configured to send one copy of the packet to 539 Relay Node R2 (packet 1.2.1) and one copy to Edge Node EN2 (packet 540 1.2.2). 542 R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and, 543 having been configured to perform packet elimination on this DetNet 544 flow, forwards packet 1.2.1 to Relay Node R3. Packet copy 1.1 is of 545 no further use and so is discarded by R2. 547 Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives 548 packet copy 1.2.1 from R2 via relay Node R3. EN2 therefore strips 549 any DetNet encapsulation from packet copy 1.2.2 and forwards the 550 packet to CE2. When EN2 receives the later packet copy 1.2.1 this is 551 discarded. 553 The above is of course illustrative of many network scenarios that 554 can be configured. 556 This example also illustrates 1:1 protection scheme meaning there is 557 traffic over each segment of the end to end path. Local DetNet relay 558 nodes determine which packets are eliminated and which packets are 559 forwarded. A 1+1 scheme where only one path is used for traffic at a 560 time, could use the same topology. In this case there is no PRF 561 function and traffic is switched upon detection of failure. An OAM 562 scheme that monitors the paths detects the loss of path or traffic is 563 required to initiate the switch. A POF may still be used in this 564 case to prevent misordering of packets. In both cases the protection 565 paths are established and maintained for the duration of the DetNet 566 service. 568 3.6.2.2. Path Differential Delay 570 In the preceding example, proper working of duplicate elimination and 571 reordering of packets are dependent on the number of out-of-order 572 packets that can be buffered and the delay difference of arriving 573 packets. DetNet uses flow specific requirements (e.g., maximum 574 number of out-of-order packets, maximum latency of the flow, etc.) 575 for configuration of POF related buffers. If the differential delay 576 between paths is excessively large or there is excessive mis-ordering 577 of the packets, then packets may be dropped instead of being 578 reordered. Likewise, PEF uses the sequence number to identify 579 duplicate packets, and large differential delays combined with high 580 numbers of packets may exceed the ability of the PEF to work 581 properly. 583 3.6.2.3. Ring Service Protection 585 Ring protection may also be supported if the underlying technology 586 supports it. Many of the same concepts apply however rings are 587 normally 1+1 protection for data efficiency reasons. [RFC8227] is an 588 example of MPLS-TP data plane that supports Ring protection. 590 3.6.3. Aggregation Considerations 592 The DetNet data plane also allows for the aggregation of DetNet 593 flows, which can improve scalability by reducing the per-hop state. 594 How this is accomplished is data plane or control plane dependent. 595 When DetNet flows are aggregated, transit nodes provide service to 596 the aggregate and not on a per-DetNet flow basis. When aggregating 597 DetNet flows the flows should be compatible i.e. the same or very 598 similar QoS and CoS characteristics. In this case, nodes performing 599 aggregation will ensure that per-flow service requirements are 600 achieved. 602 If bandwidth reservations are used, the sum of the reservations 603 should be the sum of all the individual reservations; in other words, 604 the reservations should not add up to an over-subscription of 605 bandwidth reservation. If maximum delay bounds are used, the system 606 should ensure that the aggregate does not exceed the delay bounds of 607 the individual flows. 609 When an encapsulation is used the choice of reserving a maximum 610 resource level and then tracking the services in the aggregated 611 service or adjusting the aggregated resources as the services are 612 added is implementation and technology specific. 614 DetNet flows at edges must be able to handle rejection to an 615 aggregation group due to lack of resources as well as conditions 616 where requirements are not satisfied. 618 3.6.3.1. IP Aggregation 620 IP aggregation has both data plane and controller plane aspects. For 621 the data plane, flows may be aggregated for treatment based on shared 622 characteristics such as 6-tuple. Alternatively, an IP encapsulation 623 may be used to tunnel an aggregate number of DetNet Flows between 624 relay nodes. 626 3.6.3.2. MPLS Aggregation 628 MPLS aggregation also has data plane and controller plane aspects. 629 MPLS flows are often tunneled in a forwarding sub-layer, under the 630 reservation associated with that MPLS tunnel. 632 3.6.4. End-System-Specific Considerations 634 Data-flows requiring DetNet service are generated and terminated on 635 end-systems. Encapsulation depends on the application and its 636 preferences. For example, in a DetNet MPLS domain the sub-layer 637 functions use the d-CWs, S-Labels and F-Labels to provide DetNet 638 services. However, an application may exchange further flow related 639 parameters (e.g., time-stamp), which are not provided by DetNet 640 functions. 642 As a general rule, DetNet domains are capable of forwarding any 643 DetNet flows and the DetNet domain does not mandate the end-system or 644 edge node encapsulation format. Unless there is a proxy of some form 645 present, end-systems peer with similar end-systems using the same 646 application encapsulation format. For example, as shown in Figure 4, 647 IP applications peer with IP applications and Ethernet applications 648 peer with Ethernet applications. 650 +-----+ 651 | X | +-----+ 652 +-----+ | X | 653 | Eth | ________ +-----+ 654 +-----+ _____ / \ | Eth | 655 \ / \__/ \___ +-----+ 656 \ / \ / 657 0======== tunnel-1 ========0_ 658 | \ 659 \ | 660 0========= tunnel-2 =========0 661 / \ __/ \ 662 +-----+ \__ DetNet MPLS domain / \ 663 | X | \ __ / +-----+ 664 +-----+ \_______/ \_____/ | X | 665 | IP | +-----+ 666 +-----+ | IP | 667 +-----+ 669 Figure 4: End-Systems and The DetNet MPLS Domain 671 3.6.5. Sub-Network Considerations 673 Any of the DetNet service types may be transported by another DetNet 674 service. MPLS nodes may interconnected by different sub-network 675 technologies, which may include point-to-point links. Each of these 676 sub-network technologies need to provide appropriate service to 677 DetNet flows. In some cases, e.g., on dedicated point-to-point links 678 or TDM technologies, all that is required is for a DetNet node to 679 appropriately queue its output traffic. In other cases, DetNet nodes 680 will need to map DetNet flows to the flow semantics (i.e., 681 identifiers) and mechanisms used by an underlying sub-network 682 technology. Figure 5 shows several examples of header formats that 683 can be used to carry DetNet MPLS flows over different sub-network 684 technologies. L2 represent a generic layer-2 encapsulation that 685 might be used on a point-to-point link. TSN represents the 686 encapsulation used on an IEEE 802.1 TSN network, as described in 687 [I-D.ietf-detnet-mpls-over-tsn]. UDP/IP represents the encapsulation 688 used on a DetNet IP PSN, as described in 689 [I-D.ietf-detnet-mpls-over-udp-ip]. 691 +------+ +------+ +------+ 692 App-Flow | X | | X | | X | 693 +-----+======+--+======+--+======+-----+ 694 DetNet-MPLS | d-CW | | d-CW | | d-CW | 695 +------+ +------+ +------+ 696 |Labels| |Labels| |Labels| 697 +-----+======+--+======+--+======+-----+ 698 Sub-Network | L2 | | TSN | | UDP | 699 +------+ +------+ +------+ 700 | IP | 701 +------+ 702 | L2 | 703 +------+ 705 Figure 5: Example DetNet MPLS Sub-Network Formats 707 4. Controller Plane (Management and Control) Considerations 709 4.1. DetNet Controller Plane Requirements 711 The Controller Plane corresponds to the aggregation of the Control 712 and Management Planes discussed in [RFC7426] and [RFC8655]. While 713 more details of any DetNet controller plane are out of the scope of 714 this document, there are particular considerations and requirements 715 for such that result from the unique characteristics of the DetNet 716 architecture [RFC8655] and data plane as defined herein. 718 The primary requirements of the DetNet controller plane are that it 719 must be able to: 721 o Instantiate DetNet flows in a DetNet domain (which may include 722 some or all of explicit path determination, link bandwidth 723 reservations, restricting flows to IEEE 802.1 TSN links, node 724 buffer and other resource reservations, specification of required 725 queuing disciplines along the path, ability to manage 726 bidirectional flows, etc.) as needed for a flow. 728 o In the case of MPLS, manage DetNet S-Label and F-Label allocation 729 and distribution, where the DetNet MPLS encapsulation is in use 730 see [I-D.ietf-detnet-mpls]. 732 o Support DetNet flow aggregation. 734 o Advertise static and dynamic node and link resources such as 735 capabilities and adjacencies to other network nodes (for dynamic 736 signaling approaches) or to network controllers (for centralized 737 approaches). 739 o Scale to handle the number of DetNet flows expected in a domain 740 (which may require per-flow signaling or provisioning). 742 o Provision flow identification information at each of the nodes 743 along the path. Flow identification may differ depending on the 744 location in the network and the DetNet functionality (e.g. transit 745 node vs. relay node). 747 These requirements, as stated earlier, could be satisfied using 748 distributed control protocol signaling (such as RSVP-TE), centralized 749 network management provisioning mechanisms (such as BGP, PCEP, YANG 750 [I-D.ietf-detnet-flow-information-model], etc.) or hybrid 751 combinations of the two, and could also make use of MPLS-based 752 segment routing. 754 In the abstract, the results of either distributed signaling or 755 centralized provisioning are equivalent from a DetNet data plane 756 perspective - flows are instantiated, explicit routes are determined, 757 resources are reserved, and packets are forwarded through the domain 758 using the DetNet data plane. 760 However, from a practical and implementation standpoint, they are not 761 equivalent at all. Some approaches are more scalable than others in 762 terms of signaling load on the network. Some can take advantage of 763 global tracking of resources in the DetNet domain for better overall 764 network resource optimization. Some are more resilient than others 765 if link, node, or management equipment failures occur. While a 766 detailed analysis of the control plane alternatives is out of the 767 scope of this document, the requirements from this document can be 768 used as the basis of a later analysis of the alternatives. 770 4.2. Generic Controller Plane Considerations 772 This section covers control plane considerations that are independent 773 of the data plane technology used for DetNet service delivery. 775 While management plane and control planes are traditionally 776 considered separately, from the Data Plane perspective there is no 777 practical difference based on the origin of flow provisioning 778 information, and the DetNet architecture [RFC8655] refers to these 779 collectively as the 'Controller Plane'. This document therefore does 780 not distinguish between information provided by distributed control 781 plane protocols, e.g., RSVP-TE [RFC3209] and [RFC3473], or by 782 centralized network management mechanisms, e.g., RestConf [RFC8040], 783 YANG [RFC7950], and the Path Computation Element Communication 784 Protocol (PCEP) [I-D.ietf-pce-pcep-extension-for-pce-controller] or 785 any combination thereof. Specific considerations and requirements 786 for the DetNet Controller Plane are discussed in Section 4.1. 788 Each respective data plane document also covers the control plane 789 considerations for that technology. For example [I-D.ietf-detnet-ip] 790 covers IP control plane normative considerations and 791 [I-D.ietf-detnet-mpls] covers MPLS control plane normative 792 considerations. 794 4.2.1. Flow Aggregation Control 796 Flow aggregation means that multiple App-flows are served by a single 797 new DetNet flow. There are many techniques to achieve aggregation, 798 for example in case of IP, it can be grouping of IP flows that share 799 6-tuple attributes or flow identifiers at the DetNet sub-layer. 800 Another example includes aggregation accomplished through the use of 801 hierarchical LSPs in MPLS and tunnels. 803 Control of aggregation involves a set of procedures listed here. 804 Aggregation may use some or all of these capabilities and the order 805 may vary: 807 o Traffic engineering resource collection and distribution: 809 Available resources are tracked through control plane or 810 management plane databases and distributed amongst controllers 811 or nodes that can manage resources. 813 o Path computation and resource allocation: 815 When DetNet services are provisioned or requested one or more 816 paths meeting the requirements are selected and the resources 817 verified and recorded. 819 o Resource assignment and data plane co-ordination: 821 The assignment of resources along the path depends on the 822 technology and it includes assignment of specific links and 823 coordination of the queuing and other traffic management 824 capabilities such as policing and shaping. 826 o Assigned Resource recording and updating: 828 Depending on the specific technology, the assigned resources 829 are updated and distributed in the databases, preventing over- 830 subscription. 832 4.2.2. Explicit Routes 834 Explicit routes are used to ensure that packets are routed through 835 the resources that have been reserved for them, and hence provide the 836 DetNet application with the required service. A requirement for the 837 DetNet Controller Plane will be the ability to assign a particular 838 identified DetNet IP flow to a path through the DetNet domain that 839 has been assigned the required nodal resources. This provides the 840 appropriate traffic treatment for the flow and also includes 841 particular links as a part of the path that are able to support the 842 DetNet flow. For example, by using IEEE 802.1 TSN links (as 843 discussed in [I-D.ietf-detnet-mpls-over-tsn] ) DetNet parameters can 844 be maintained. Further considerations and requirements for the 845 DetNet Controller Plane are discussed in Section 4.1. 847 Whether configuring, calculating and instantiating these routes is a 848 single-stage or multi-stage process, or in a centralized or 849 distributed manner, is out of scope of this document. 851 There are several approaches that could be used to provide explicit 852 routes and resource allocation in the DetNet forwarding sub-layer. 853 For example: 855 o The path could be explicitly set up by a controller which 856 calculates the path and explicitly configures each node along that 857 path with the appropriate forwarding and resource allocation 858 information. 860 o The path could use a distributed control plane such as RSVP 861 [RFC2205] or RSVP-TE [RFC3473] extended to support DetNet IP 862 flows. 864 o The path could be implemented using IPv6-based segment routing 865 when extended to support resource allocation. 867 See Section 4.1 for further discussion of these alternatives. In 868 addition, [RFC2386] contains useful background information on QoS- 869 based routing, and [RFC5575] discusses a specific mechanism used by 870 BGP for traffic flow specification and policy-based routing. 872 4.2.3. Contention Loss and Jitter Reduction 874 As discussed in Section 1, this document does not specify the 875 mechanisms needed to eliminate packet contention, packet loss or 876 reduce jitter for DetNet flows at the DetNet forwarding sub-layer. 877 The ability to manage node and link resources to be able to provide 878 these functions is a necessary part of the DetNet controller plane. 879 It is also necessary to be able to control the required queuing 880 mechanisms used to provide these functions along a flow's path 881 through the network. See [I-D.ietf-detnet-ip] and Section 4.1 for 882 further discussion of these requirements. Some forms of protection 883 may minimize packet loss or change jitter characteristics in the 884 cases where packets are reordered when out-of-order packets are 885 received at the service sub-layer. 887 4.2.4. Bidirectional Traffic 889 In many cases DetNet flows can be considered unidirectional and 890 independent. However, there are cases where the DetNet service 891 requires bidirectional traffic from a DetNet application service 892 perspective. IP and MPLS typically treat each direction separately 893 and do not force interdependence of each direction. MPLS has 894 considered bidirectional traffic requirements and the MPLS 895 definitions from [RFC5654] are useful to illustrate terms such as 896 associated bidirectional flows and co-routed bidirectional flows. 897 MPLS defines a point-to-point associated bidirectional LSP as 898 consisting of two unidirectional point-to-point LSPs, one from A to B 899 and the other from B to A, which are regarded as providing a single 900 logical bidirectional forwarding path. This is analogous to standard 901 IP routing. MPLS defines a point-to-point co-routed bidirectional 902 LSP as an associated bidirectional LSP which satisfies the additional 903 constraint that its two unidirectional component LSPs follow the same 904 path (in terms of both nodes and links) in both directions. An 905 important property of co-routed bidirectional LSPs is that their 906 unidirectional component LSPs share fate. In both types of 907 bidirectional LSPs, resource reservations may differ in each 908 direction. The concepts of associated bidirectional flows and co- 909 routed bidirectional flows can also be applied to DetNet IP flows. 911 While the DetNet IP data plane must support bidirectional DetNet 912 flows, there are no special bidirectional features with respect to 913 the data plane other than the need for the two directions of a co- 914 routed bidirectional flow to take the same path. That is to say that 915 bidirectional DetNet flows are solely represented at the management 916 and control plane levels, without specific support or knowledge 917 within the DetNet data plane. Fate sharing and associated or co- 918 routed bidirectional flows, can be managed at the control level. 920 DetNet's use of PREOF may increase the complexity of using co-routing 921 bidirectional flows, since if PREOF is used, then the replication 922 points in one direction would have to match the elimination points in 923 the other direction, and vice versa. In such cases the optimal 924 points for these functions in one direction may not match the optimal 925 points in the other, due to network and traffic constraints. 926 Furthermore, due to the per packet service protection nature, 927 bidirectional forwarding per packet may not be ensured. The first 928 packet of received member flows is selected by the elimination 929 function independently of which path it has taken through the 930 network. 932 Control and management mechanisms need to support bidirectional 933 flows, but the specification of such mechanisms are out of scope of 934 this document. An example control plane solution for MPLS can be 935 found in [RFC3473] , [RFC6387] and [RFC7551]. These requirements are 936 included in Section 4.1. 938 4.3. Packet Replication, Elimination, and Ordering (PREOF) 940 The controller plane protocol solution required for managing the 941 PREOF processing is outside the scope of this document. That said, 942 it should be noted that the ability to determine, for a particular 943 flow, optimal packet replication and elimination points in the DetNet 944 domain requires explicit support. There may be capabilities that can 945 be used, or extended, for example GMPLS end-to-end recovery [RFC4872] 946 and GMPLS segment recovery [RFC4873]. 948 5. Security Considerations 950 Security considerations for DetNet are described in detail in 951 [I-D.ietf-detnet-security]. General security considerations are 952 described in [RFC8655]. This section considers general security 953 considerations applicable to all data planes. 955 Security aspects which are unique to DetNet are those whose aim is to 956 provide the specific quality of service aspects of DetNet, which are 957 primarily to deliver data flows with extremely low packet loss rates 958 and bounded end-to-end delivery latency. 960 The primary considerations for the data plane is to maintain 961 integrity of data and delivery of the associated DetNet service 962 traversing the DetNet network. Application flows can be protected 963 through whatever means is provided by the underlying technology. For 964 example, encryption may be used, such as that provided by IPSec 965 [RFC4301] for IP flows and/or by an underlying sub-net using MACSec 966 [IEEE802.1AE-2018] for Ethernet (Layer-2) flows. 968 At the management and control level DetNet flows are identified on a 969 per-flow basis, which may provide controller plane attackers with 970 additional information about the data flows (when compared to 971 controller planes that do not include per-flow identification). This 972 is an inherent property of DetNet which has security implications 973 that should be considered when determining if DetNet is a suitable 974 technology for any given use case. 976 To provide uninterrupted availability of the DetNet service, 977 provisions can be made against DOS attacks and delay attacks. To 978 protect against DOS attacks, excess traffic due to malicious or 979 malfunctioning devices can be prevented or mitigated, for example 980 through the use of existing mechanism such as policing and shaping 981 applied at the input of a DetNet domain. To prevent DetNet packets 982 from being delayed by an entity external to a DetNet domain, DetNet 983 technology definition can allow for the mitigation of Man-In-The- 984 Middle attacks, for example through use of authentication and 985 authorization of devices within the DetNet domain. 987 In order to prevent or mitigate DetNet attacks on other networks via 988 flow escape, edge devices can for example use existing mechanism such 989 as policing and shaping applied at the output of a DetNet domain. 991 6. IANA Considerations 993 This document makes no IANA requests. 995 7. Acknowledgements 997 The authors wish to thank Pat Thaler, Norman Finn, Loa Anderson, 998 David Black, Rodney Cummings, Ethan Grossman, Tal Mizrahi, David 999 Mozes, Craig Gunther, George Swallow, Yuanlong Jiang and Carlos J. 1000 Bernardos for their various contributions to this work. 1002 8. Contributors 1004 The following people contributed substantially to the content of this 1005 document: 1007 Don Fedyk 1008 Jouni Korhonen 1010 9. References 1012 9.1. Normative References 1014 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1015 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1016 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1017 . 1019 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 1020 Switching (GMPLS) Signaling Resource ReserVation Protocol- 1021 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 1022 DOI 10.17487/RFC3473, January 2003, 1023 . 1025 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 1026 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 1027 Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385, 1028 February 2006, . 1030 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1031 "Deterministic Networking Architecture", RFC 8655, 1032 DOI 10.17487/RFC8655, October 2019, 1033 . 1035 9.2. Informative References 1037 [I-D.ietf-detnet-flow-information-model] 1038 Farkas, J., Varga, B., Cummings, R., Jiang, Y., and D. 1039 Fedyk, "DetNet Flow Information Model", draft-ietf-detnet- 1040 flow-information-model-06 (work in progress), October 1041 2019. 1043 [I-D.ietf-detnet-ip] 1044 Varga, B., Farkas, J., Berger, L., Fedyk, D., Malis, A., 1045 Bryant, S., and J. Korhonen, "DetNet Data Plane: IP", 1046 draft-ietf-detnet-ip-04 (work in progress), November 2019. 1048 [I-D.ietf-detnet-mpls] 1049 Varga, B., Farkas, J., Berger, L., Fedyk, D., Malis, A., 1050 Bryant, S., and J. Korhonen, "DetNet Data Plane: MPLS", 1051 draft-ietf-detnet-mpls-04 (work in progress), November 1052 2019. 1054 [I-D.ietf-detnet-mpls-over-tsn] 1055 Varga, B., Farkas, J., Malis, A., and S. Bryant, "DetNet 1056 Data Plane: MPLS over IEEE 802.1 Time Sensitive Networking 1057 (TSN)", draft-ietf-detnet-mpls-over-tsn-01 (work in 1058 progress), October 2019. 1060 [I-D.ietf-detnet-mpls-over-udp-ip] 1061 Varga, B., Farkas, J., Berger, L., Malis, A., Bryant, S., 1062 and J. Korhonen, "DetNet Data Plane: MPLS over UDP/IP", 1063 draft-ietf-detnet-mpls-over-udp-ip-04 (work in progress), 1064 November 2019. 1066 [I-D.ietf-detnet-security] 1067 Mizrahi, T., Grossman, E., Hacker, A., Das, S., Dowdell, 1068 J., Austad, H., and N. Finn, "Deterministic Networking 1069 (DetNet) Security Considerations", draft-ietf-detnet- 1070 security-07 (work in progress), January 2020. 1072 [I-D.ietf-pce-pcep-extension-for-pce-controller] 1073 Zhao, Q., Li, Z., Negi, M., and C. Zhou, "PCEP Procedures 1074 and Protocol Extensions for Using PCE as a Central 1075 Controller (PCECC) of LSPs", draft-ietf-pce-pcep- 1076 extension-for-pce-controller-03 (work in progress), 1077 November 2019. 1079 [I-D.ietf-spring-srv6-network-programming] 1080 Filsfils, C., Camarillo, P., Leddy, J., Voyer, D., 1081 Matsushima, S., and Z. Li, "SRv6 Network Programming", 1082 draft-ietf-spring-srv6-network-programming-08 (work in 1083 progress), January 2020. 1085 [IEEE802.1AE-2018] 1086 IEEE Standards Association, "IEEE Std 802.1AE-2018 MAC 1087 Security (MACsec)", 2018, 1088 . 1090 [IEEE802.1TSNTG] 1091 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 1092 Networking Task Group", . 1094 [nwcrg] IRTF, "Coding for efficient NetWork Communications 1095 Research Group (nwcrg)", 1096 . 1098 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1099 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1100 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1101 September 1997, . 1103 [RFC2386] Crawley, E., Nair, R., Rajagopalan, B., and H. Sandick, "A 1104 Framework for QoS-based Routing in the Internet", 1105 RFC 2386, DOI 10.17487/RFC2386, August 1998, 1106 . 1108 [RFC3670] Moore, B., Durham, D., Strassner, J., Westerinen, A., and 1109 W. Weiss, "Information Model for Describing Network Device 1110 QoS Datapath Mechanisms", RFC 3670, DOI 10.17487/RFC3670, 1111 January 2004, . 1113 [RFC4301] Kent, S. and K. Seo, "Security Architecture for the 1114 Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, 1115 December 2005, . 1117 [RFC4872] Lang, J., Ed., Rekhter, Y., Ed., and D. Papadimitriou, 1118 Ed., "RSVP-TE Extensions in Support of End-to-End 1119 Generalized Multi-Protocol Label Switching (GMPLS) 1120 Recovery", RFC 4872, DOI 10.17487/RFC4872, May 2007, 1121 . 1123 [RFC4873] Berger, L., Bryskin, I., Papadimitriou, D., and A. Farrel, 1124 "GMPLS Segment Recovery", RFC 4873, DOI 10.17487/RFC4873, 1125 May 2007, . 1127 [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J., 1128 and D. McPherson, "Dissemination of Flow Specification 1129 Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009, 1130 . 1132 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1133 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1134 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1135 September 2009, . 1137 [RFC6387] Takacs, A., Berger, L., Caviglia, D., Fedyk, D., and J. 1138 Meuric, "GMPLS Asymmetric Bandwidth Bidirectional Label 1139 Switched Paths (LSPs)", RFC 6387, DOI 10.17487/RFC6387, 1140 September 2011, . 1142 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1143 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1144 Defined Networking (SDN): Layers and Architecture 1145 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1146 2015, . 1148 [RFC7551] Zhang, F., Ed., Jing, R., and R. Gandhi, Ed., "RSVP-TE 1149 Extensions for Associated Bidirectional Label Switched 1150 Paths (LSPs)", RFC 7551, DOI 10.17487/RFC7551, May 2015, 1151 . 1153 [RFC7657] Black, D., Ed. and P. Jones, "Differentiated Services 1154 (Diffserv) and Real-Time Communication", RFC 7657, 1155 DOI 10.17487/RFC7657, November 2015, 1156 . 1158 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 1159 RFC 7950, DOI 10.17487/RFC7950, August 2016, 1160 . 1162 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 1163 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 1164 . 1166 [RFC8227] Cheng, W., Wang, L., Li, H., van Helvoort, H., and J. 1167 Dong, "MPLS-TP Shared-Ring Protection (MSRP) Mechanism for 1168 Ring Topology", RFC 8227, DOI 10.17487/RFC8227, August 1169 2017, . 1171 Authors' Addresses 1173 Balazs Varga (editor) 1174 Ericsson 1175 Magyar Tudosok krt. 11. 1176 Budapest 1117 1177 Hungary 1179 Email: balazs.a.varga@ericsson.com 1181 Janos Farkas 1182 Ericsson 1183 Magyar Tudosok krt. 11. 1184 Budapest 1117 1185 Hungary 1187 Email: janos.farkas@ericsson.com 1189 Lou Berger 1190 LabN Consulting, L.L.C. 1192 Email: lberger@labn.net 1194 Andrew G. Malis 1195 Independent 1197 Email: agmalis@gmail.com 1199 Stewart Bryant 1200 Futurewei Technologies 1202 Email: stewart.bryant@gmail.com