idnits 2.17.1 draft-ietf-teas-enhanced-vpn-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 12, 2021) is 1020 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-04) exists of draft-dong-teas-enhanced-vpn-vtn-scalability-02 == Outdated reference: A later version (-19) exists of draft-ietf-opsawg-l2nm-02 == Outdated reference: A later version (-18) exists of draft-ietf-opsawg-l3sm-l3nm-08 == Outdated reference: A later version (-13) exists of draft-ietf-opsawg-ntf-07 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-policy-11 == Outdated reference: A later version (-24) exists of draft-ietf-teas-actn-vn-yang-11 == Outdated reference: A later version (-11) exists of draft-ietf-teas-actn-yang-07 == Outdated reference: A later version (-25) exists of draft-ietf-teas-ietf-network-slices-00 == Outdated reference: A later version (-05) exists of draft-wd-teas-ietf-network-slice-nbi-yang-02 Summary: 0 errors (**), 0 flaws (~~), 10 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group J. Dong 3 Internet-Draft Huawei 4 Intended status: Informational S. Bryant 5 Expires: January 13, 2022 Futurewei 6 Z. Li 7 China Mobile 8 T. Miyasaka 9 KDDI Corporation 10 Y. Lee 11 Samsung 12 July 12, 2021 14 A Framework for Enhanced Virtual Private Network (VPN+) Services 15 draft-ietf-teas-enhanced-vpn-08 17 Abstract 19 This document describes the framework for Enhanced Virtual Private 20 Network (VPN+) services. The purpose of enhanced VPNs is to support 21 the needs of new applications, particularly applications that are 22 associated with 5G services, by utilizing an approach that is based 23 on existing VPN and Traffic Engineering (TE) technologies and adds 24 characteristics that specific services require over and above those 25 provided by traditional VPNs. 27 Typically, VPN+ will be used to underpin network slicing, but could 28 also be of use in its own right providing enhanced connectivity 29 services between customer sites. 31 It is envisaged that enhanced VPNs will be delivered using a 32 combination of existing, modified, and new networking technologies. 33 This document provides an overview of relevant technologies and 34 identifies some areas for potential new work. 36 Status of This Memo 38 This Internet-Draft is submitted in full conformance with the 39 provisions of BCP 78 and BCP 79. 41 Internet-Drafts are working documents of the Internet Engineering 42 Task Force (IETF). Note that other groups may also distribute 43 working documents as Internet-Drafts. The list of current Internet- 44 Drafts is at https://datatracker.ietf.org/drafts/current/. 46 Internet-Drafts are draft documents valid for a maximum of six months 47 and may be updated, replaced, or obsoleted by other documents at any 48 time. It is inappropriate to use Internet-Drafts as reference 49 material or to cite them other than as "work in progress." 51 This Internet-Draft will expire on January 13, 2022. 53 Copyright Notice 55 Copyright (c) 2021 IETF Trust and the persons identified as the 56 document authors. All rights reserved. 58 This document is subject to BCP 78 and the IETF Trust's Legal 59 Provisions Relating to IETF Documents 60 (https://trustee.ietf.org/license-info) in effect on the date of 61 publication of this document. Please review these documents 62 carefully, as they describe your rights and restrictions with respect 63 to this document. Code Components extracted from this document must 64 include Simplified BSD License text as described in Section 4.e of 65 the Trust Legal Provisions and are provided without warranty as 66 described in the Simplified BSD License. 68 Table of Contents 70 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 71 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 72 3. Overview of the Requirements . . . . . . . . . . . . . . . . 6 73 3.1. Performance Guarantees . . . . . . . . . . . . . . . . . 6 74 3.2. Isolation between Enhanced VPN Services . . . . . . . . . 8 75 3.2.1. A Pragmatic Approach to Isolation . . . . . . . . . . 10 76 3.3. Integration . . . . . . . . . . . . . . . . . . . . . . . 11 77 3.3.1. Abstraction . . . . . . . . . . . . . . . . . . . . . 11 78 3.4. Dynamic Changes . . . . . . . . . . . . . . . . . . . . . 11 79 3.5. Customized Control . . . . . . . . . . . . . . . . . . . 12 80 3.6. Applicability . . . . . . . . . . . . . . . . . . . . . . 12 81 3.7. Inter-Domain and Inter-Layer Network . . . . . . . . . . 13 82 4. Architecture of Enhanced VPNs . . . . . . . . . . . . . . . . 13 83 4.1. Layered Architecture . . . . . . . . . . . . . . . . . . 15 84 4.2. Multi-Point to Multi-Point (MP2MP) Connectivity . . . . . 17 85 4.3. Application Specific Data Types . . . . . . . . . . . . . 17 86 4.4. Scaling Considerations . . . . . . . . . . . . . . . . . 18 87 5. Candidate Technologies . . . . . . . . . . . . . . . . . . . 18 88 5.1. Packet Forwarding Plane Technologies . . . . . . . . . . 19 89 5.1.1. Flexible Ethernet . . . . . . . . . . . . . . . . . . 19 90 5.1.2. Dedicated Queues . . . . . . . . . . . . . . . . . . 19 91 5.1.3. Time Sensitive Networking . . . . . . . . . . . . . . 20 92 5.2. Layer Three Data Plane . . . . . . . . . . . . . . . . . 20 93 5.2.1. Deterministic Networking . . . . . . . . . . . . . . 20 94 5.2.2. MPLS Traffic Engineering (MPLS-TE) . . . . . . . . . 21 95 5.2.3. Segment Routing . . . . . . . . . . . . . . . . . . . 21 97 5.3. Non-Packet Data Plane . . . . . . . . . . . . . . . . . . 22 98 5.4. Control Plane . . . . . . . . . . . . . . . . . . . . . . 22 99 5.5. Management Plane . . . . . . . . . . . . . . . . . . . . 23 100 5.6. Applicability of Service Data Models to Enhanced VPN . . 24 101 5.6.1. An Example of Enhanced VPN Delivery . . . . . . . . . 25 102 6. Scalability Considerations . . . . . . . . . . . . . . . . . 25 103 6.1. Maximum Stack Depth of SR . . . . . . . . . . . . . . . . 26 104 6.2. RSVP-TE Scalability . . . . . . . . . . . . . . . . . . . 27 105 6.3. SDN Scaling . . . . . . . . . . . . . . . . . . . . . . . 27 106 7. OAM Considerations . . . . . . . . . . . . . . . . . . . . . 27 107 8. Telemetry Considerations . . . . . . . . . . . . . . . . . . 28 108 9. Enhanced Resiliency . . . . . . . . . . . . . . . . . . . . . 28 109 10. Operational Considerations . . . . . . . . . . . . . . . . . 29 110 11. Security Considerations . . . . . . . . . . . . . . . . . . . 29 111 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 112 13. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 30 113 14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 31 114 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 31 115 15.1. Normative References . . . . . . . . . . . . . . . . . . 31 116 15.2. Informative References . . . . . . . . . . . . . . . . . 32 117 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 37 119 1. Introduction 121 Virtual private networks (VPNs) have served the industry well as a 122 means of providing different groups of users with logically isolated 123 connectivity over a common network. The common or base network that 124 is used to provide the VPNs is often referred to as the underlay, and 125 the VPN is often called an overlay. 127 Customers of a network operator may request a connectivity services 128 with advanced characteristics such as low latency guarantees, bounded 129 jitter, or isolation from other services or customers so that changes 130 in some other service (such as changes in network load, or events 131 such as congestion or outages) have no or only acceptable effect on 132 the throughput or latency of the services provided to the customer. 133 These services are referred to as "enhanced VPNs" (known as VPN+) in 134 that they are similar to VPN services providing the customer with the 135 required connectivity, but in addition they have enhanced 136 characteristics. 138 The concept of network slicing has gained traction driven largely by 139 needs surfacing from 5G [NGMN-NS-Concept] [TS23501] [TS28530] 140 [BBF-SD406]. According to [TS28530], a 5G end-to-end network slice 141 consists of three major types of network segments: Radio Access 142 Network (RAN), Transport Network (TN), and Mobile Core Network (CN). 143 The transport network provides the connectivity between different 144 entities in RAN and CN segments of a 5G end-to-end network slice, 145 with specific performance commitment. 147 An IETF network slice [I-D.ietf-teas-ietf-network-slices] is a 148 virtual (logical) network with its own network topology and a set of 149 shared or dedicated network resources, which are used to provide the 150 network slice customer with the required connectivity, appropriate 151 isolation, and a specific set of Service Level Objectives (SLOs) and 152 Service Level Expectations (SLEs). In this document (which is solely 153 about IETF technologies) we refer to an "IETF network slice" simply 154 as a "network slice": a network slice is considered one possible use 155 case of an enhanced VPN. 157 A network slice could span multiple technologies (such as IP or 158 Optical) and multiple administrative domains. Depending on the 159 customer's requirement, a network slice could be isolated from other 160 network slices in terms of data plane, control plane, and management 161 plane resources. 163 Network slicing builds on the concepts of resource management, 164 network virtualization, and abstraction to provide performance 165 assurance, flexibility, programmability, and modularity. It may use 166 techniques such as Software Defined Networking (SDN) [RFC7149], 167 network abstraction [RFC7926] and Network Function Virtualization 168 (NFV) [RFC8172] [RFC8568] to create multiple logical (virtual) 169 networks, each tailored for use by a set of services or by a 170 particular tenant or a group of tenants that share the same or 171 similar requirements. These logical networks are created on top of a 172 common underlay network. How the network slices are engineered can 173 be deployment-specific. 175 VPN+ can be used to instantiate a network slice, but the technique 176 can also be of use in general cases to provide enhanced connectivity 177 services between customer sites. 179 The requirements of enhanced VPN services cannot be met by simple 180 overlay networks, as these services require tighter coordination and 181 integration between the underlay and the overlay network. VPN+ is 182 built from a VPN overlay and an underlying Virtual Transport Network 183 (VTN) which has a customized network topology and a set of dedicated 184 or shared resources in the underlay network. The enhanced VPN may 185 also include a set of invoked service functions located within the 186 underlay network. Thus, an enhanced VPN can achieve greater 187 isolation with strict performance guarantees. These new properties, 188 which have general applicability, are also of interest as part of a 189 network slicing solution. 191 It is not envisaged that VPN+ services will replace traditional VPN 192 services. Traditional VPN services will continue to be delivered 193 using pre-existing mechanisms and can co-exist with VPN+ services. 194 In fact, compared to traditional VPNs, it is not envisaged that large 195 numbers of VPN+ services will be deployed in a network. In other 196 words, it is not intended that all existing VPNs supported by a 197 network will use VPN+ techniques. 199 This document describes a framework for using existing, modified, and 200 potential new technologies as components to provide a VPN+ service. 201 Specifically, we are concerned with: 203 o The functional requirements and service characteristics of an 204 enhanced VPN. 206 o The design of the enhanced VPN data plane. 208 o The necessary control and management protocols in both the 209 underlay and the overlay of the enhanced VPN. 211 o The mechanisms to achieve integration between overlay and 212 underlay. 214 o The necessary Operation, Administration, and Management (OAM) 215 methods to instrument an enhanced VPN to make sure that the 216 required Service Level Agreement (SLA) between the customer and 217 the network operator is met, and to take any corrective action 218 (such as switching traffic to an alternate path) to avoid SLA 219 violation. 221 The required layered network structure to achieve this is shown in 222 Section 4.1. 224 2. Terminology 226 In this document, the relationship of the four terms "VPN", "VPN+", 227 "VTN", and "Network Slice" are as follows: 229 o A Virtual Private Network (VPN) refers to the overlay network 230 service that provides the connectivity between different customer 231 sites, and that maintains traffic separation between different 232 customers. IPVPN is defined in [RFC2764], L2VPN is defined in 233 [RFC4664], L3VPN is defined in [RFC4364], and EVPN is defined in 234 [RFC7209]. 236 o An enhanced VPN (VPN+) is an evolution of the VPN service that 237 makes additional service-specific commitments. An enhanced VPN is 238 made by integrating an overlay VPN with a set of network resources 239 allocated in the underlay network. 241 o A Virtual Transport Network (VTN) is a virtual underlay network 242 that provide the connection between the customer sites. The VTN 243 has the capability to deliver the performance characteristics 244 required by the VPN+ customers and to provide isolation between 245 separate VPN+ services. 247 o A network slice could be provided by building an enhanced VPN. 248 Other mechanisms for delivering network slices may exist but are 249 not in scope for this document. 251 The term "tenant" is used in this document to refer to the customers 252 and all of their associated enhanced VPNs. 254 The following terms are also used in this document. Some of them are 255 newly defined, some others reference existing definitions. 257 ACTN: Abstraction and Control of Traffic Engineered Networks 258 [RFC8453] 260 DetNet: Deterministic Networking. See [DETNET] and [RFC8655] 262 FlexE: Flexible Ethernet [FLEXE] 264 TSN: Time Sensitive Networking [TSN] 266 VN: Virtual Network [I-D.ietf-teas-actn-vn-yang] 268 VTP: Virtual Transport Path. A VTP is a path through the VTN which 269 provides the connection between two customer sites. 271 3. Overview of the Requirements 273 This section provides an overview of the requirements of an enhanced 274 VPN service. 276 3.1. Performance Guarantees 278 Performance guarantees are made by network operators to their 279 customers in relation to the services provided to the customers. 280 They are usually expressed in SLAs as a set of SLOs. 282 There are several kinds of performance guarantee, including 283 guaranteed maximum packet loss, guaranteed maximum delay, and 284 guaranteed delay variation. Note that these guarantees apply to 285 conformance traffic, out-of-profile traffic will be handled according 286 to a separate agreement with the customer. 288 Guaranteed maximum packet loss is usually addressed by setting packet 289 priorities, queue size, and discard policy. However this becomes 290 more difficult when the requirement is combined with latency 291 requirements. The limiting case is zero congestion loss, and that is 292 the goal of DetNet [DETNET] and TSN [TSN]. In modern optical 293 networks, loss due to transmission errors already approaches zero, 294 but there is the possibility of failure of the interface or the fiber 295 itself. This type of fault can only be addressed by some form of 296 signal duplication and transmission over diverse paths. 298 Guaranteed maximum latency is required by a number of applications 299 particularly real-time control applications and some types of virtual 300 reality applications. DetNet [DETNET] is relevant, however 301 additional methods of enhancing the underlay to better support the 302 delay guarantees may be needed, and these methods will need to be 303 integrated with the overall service provisioning mechanisms. 305 Guaranteed maximum delay variation is a performance guarantee that 306 may also be needed. [RFC8578] calls up a number of cases that need 307 this guarantee, for example in electrical utilities. Time transfer 308 is an example service that needs a performance guarantee, although it 309 is in the nature of time that the service might be delivered by the 310 underlay as a shared service and not provided through different 311 enhanced VPNs. Alternatively, a dedicated enhanced VPN might be used 312 to provide this as a shared service. 314 This suggests that a spectrum of service guarantees need to be 315 considered when deploying an enhanced VPN. As a guide to 316 understanding the design requirements we can consider four types of 317 service: 319 o Best effort 321 o Assured bandwidth 323 o Guaranteed latency 325 o Enhanced delivery 327 The best effort service is the basic service as provided by current 328 VPNs. 330 An assured bandwidth service is one in which the bandwidth over some 331 period of time is assured. This can be achieved either simply based 332 on a best effort service with over-capacity provisioning, or it can 333 be based on MPLS traffic engineered label switching paths (TE-LSPs) 334 with bandwidth reservations. Depending on the technique used, 335 however, the bandwidth is not necessarily assured at any instant. 336 Providing assured bandwidth to VPNs, for example by using per-VPN TE- 337 LSPs, is not widely deployed at least partially due to scalability 338 concerns. VPN+ aims to provide a more scalable approach for such 339 services. 341 A guaranteed latency service has an upper bound to edge-to-edge 342 latency. Assuring the upper bound is sometimes more important than 343 minimizing latency. There are several new technologies that provide 344 some assistance with this performance guarantee. Firstly, the IEEE 345 TSN project [TSN] introduces the concept of scheduling of delay- and 346 loss-sensitive packets. The DetNet work [DETNET] is also of 347 relevance in assuring an upper bound of end-to-end packet latency. 348 FlexE [FLEXE] is also useful to help provide these guarantees. The 349 use of such underlying technologies to deliver VPN+ services needs to 350 be considered. 352 An enhanced delivery service is one in which the underlay network (at 353 Layer 3) attempts to deliver the packet through multiple paths in the 354 hope of eliminating packet loss due to equipment or media failures. 355 Such a mechanism may need to be used for VPN+ service. 357 3.2. Isolation between Enhanced VPN Services 359 One element of the SLA demanded for an enhanced VPN may be a 360 guarantee that the service offered to the customer will not be 361 affected by any other traffic flows in the network. This is termed 362 "isolation" and a customer may express the requirement for isolation 363 as an SLE [I-D.ietf-teas-ietf-network-slices]. 365 One way for a network operator to meet the requirement for isolation 366 is simply by setting and conforming to all the SLOs. For example, 367 traffic congestion (interference from other services) might impact on 368 the latency experienced by a VPN+ customer. Thus, in this example, 369 conformance to a latency SLO would be the primary requirement for 370 delivery of the VPN+ service, and isolation from other services might 371 be only a means to that end. 373 Another way for a service provider to meet this SLE is to control the 374 degree to which traffic from one service is isolated from other 375 services in the network. 377 There is a fine distinction between how isolation is requested by a 378 customer and how it is delivered by the service provider. In 379 general, the customer is interested in service performance and not 380 how it is delivered. Thus, for example, the customer wants specific 381 quality guarantees and is not concerned about how the service 382 provider delivers them. However, it should be noted that some 383 aspects of isolation might be directly measurable by a customer if 384 they have information about the traffic patterns on a number services 385 supported by the same service provider. Furthermore, a customer may 386 be nervous about disruption caused by other services, contamination 387 by other traffic, or delivery of their traffic to the wrong 388 destinations. In this way, the customer may want to specify (and pay 389 for) the level of isolation provided by the service provider. 391 Isolation is achieved in the realization of a VPN+ through existing 392 technologies that may be supplemented by new mechanisms. The service 393 provider chooses which processes to use to meet this SLE just as they 394 choose how to meet all other SLOs and SLEs. Isolation may be 395 achieved in the network by various forms of resource partitioning 396 ranging from simple separation of service traffic on delivery 397 (ensuring that traffic is not delivered to the wrong customer), 398 through sharing of resources with some form of safeguards, to 399 dedicated allocation of resources for a specific enhanced VPN. For 400 example, interference avoidance may be achieved by network capacity 401 planning, allocating dedicated network resources, traffic policing or 402 shaping, prioritizing in using shared network resources, etc. 404 The terms hard and soft isolation are used to indicate different 405 levels of isolation. A service has soft isolation if the traffic of 406 one service cannot be received by the customers of another service. 407 The existing IP and MPLS VPNs are examples of services with soft 408 isolation: the network delivers the traffic only to the required 409 customer endpoints. However, with soft isolation, as the network 410 resources are shared, traffic from some services may congest the 411 network, resulting in packet loss and delay for other services. The 412 ability for a service or a group of services to be sheltered from 413 this effect is called hard isolation. Hard isolation may be needed 414 so that applications with exacting requirements can function 415 correctly, despite other demands (perhaps a burst of traffic in 416 another service) competing for the underlying resources. A customer 417 may request different degrees of isolation ranging from soft 418 isolation to hard isolation. In practice isolation may be delivered 419 on a spectrum between soft and hard, and in some cases soft and hard 420 isolation may be used in a hierarchical manner with one enhanced VPN 421 being built on another. 423 To provide the required level of isolation, resources may need to be 424 reserved in the data plane of the underlay network and dedicated to 425 traffic from a specific enhanced VPN or a specific group of enhanced 426 VPNs. This may introduce scalability concerns both in the 427 implementation (as each enhanced VPN would need to be tracked in the 428 network) and in how many resources need to be reserved and may be 429 under-used (see Section 4.4). Thus, some trade-off needs to be 430 considered to provide the isolation between enhanced VPNs while still 431 allowing reasonable resource sharing. 433 An optical underlay can offer a high degree of isolation, at the cost 434 of allocating resources on a long-term and end-to-end basis. On the 435 other hand, where adequate isolation can be achieved at the packet 436 layer, this permits the resources to be shared amongst a group of 437 services and only dedicated to a service on a temporary basis. 439 The next section explores a pragmatic approach to isolation in packet 440 networks. 442 3.2.1. A Pragmatic Approach to Isolation 444 A key question is whether it is possible to achieve hard isolation in 445 packet networks that were designed to provide statistical 446 multiplexing through sharing of data plane resources, a significant 447 economic advantage when compared to a dedicated, or a Time Division 448 Multiplexing (TDM) network. Clearly, there is no need to provide 449 more isolation than is required by the applications, and an 450 approximation to full hard isolation is sufficient in most cases. 451 For example, pseudowires [RFC3985] emulate services that would have 452 had hard isolation in their native form. 454 O=================================================O 455 | \---------------v---------------/ 456 Statistical Pragmatic Absolute 457 Multiplexing Isolation Isolation 458 (Traditional VPNs) (Enhanced VPN) (Dedicated Network) 460 Figure 1: The Spectrum of Isolation 462 Figure 1 shows a spectrum of isolation that may be delivered by a 463 network. At one end of the spectrum, we see statistical multiplexing 464 technologies that support traditional VPNs. This is a service type 465 that has served the industry well and will continue to do so. At the 466 opposite end of the spectrum, we have the absolute isolation provided 467 by dedicated transport networks. The goal of enhanced VPNs is 468 "pragmatic isolation". This is isolation that is better than what is 469 obtainable from pure statistical multiplexing, more cost effective 470 and flexible than a dedicated network, but is a practical solution 471 that is good enough for the majority of applications. Mechanisms for 472 both soft isolation and hard isolation are needed to meet different 473 levels of service requirement. 475 3.3. Integration 477 The way to achieve the characteristics demanded by an enhanced VPN 478 (such as guaranteed or predictable performance) is by integrating the 479 overlay VPN with a particular set of resources in the underlay 480 network which are allocated to meet the service requirement. This 481 needs be done in a flexible and scalable way so that it can be widely 482 deployed in operators' networks to support a reasonable number of 483 enhanced VPN customers. 485 Taking mobile networks and in particular 5G into consideration, the 486 integration of the network with service functions is likely a 487 requirement. The IETF's work on service function chaining (SFC) 488 [SFC] provides a foundation for this. Service functions can be 489 considered as part of enhanced VPN services. The detailed mechanisms 490 about the integration between service functions and enhanced VPNs are 491 out of the scope of this document. 493 3.3.1. Abstraction 495 Integration of the overlay VPN and the underlay network resources 496 does not need to be a tight mapping. As described in [RFC7926], 497 abstraction is the process of applying policy to a set of information 498 about a traffic engineered (TE) network to produce selective 499 information that represents the potential ability to connect across 500 the network. The process of abstraction presents the connectivity 501 graph in a way that is independent of the underlying network 502 technologies, capabilities, and topology so that the graph can be 503 used to plan and deliver network services in a uniform way. 505 Virtual networks can be built on top of an abstracted topology that 506 represents the connectivity capabilities of the underlay network as 507 described in the framework for Abstraction and Control of TE Networks 508 (ACTN) [RFC8453] as discussed further in Section 5.5. 509 [I-D.king-teas-applicability-actn-slicing] describes the 510 applicability of ACTN to network slicing and is, therefore, relevant 511 to the consideration of using ACTN to enable enhanced VPNs. 513 3.4. Dynamic Changes 515 Enhanced VPNs need to be created, modified, and removed from the 516 network according to service demands. An enhanced VPN that requires 517 hard isolation (Section 3.2) must not be disrupted by the 518 instantiation or modification of another enhanced VPN. Determining 519 whether modification of an enhanced VPN can be disruptive to that 520 VPN, and whether the traffic in flight will be disrupted can be a 521 difficult problem. 523 The data plane aspects of this problem are discussed further in 524 Section 5.1,Section 5.2, and Section 5.3. 526 The control plane aspects of this problem are discussed further in 527 Section 5.4. 529 The management plane aspects of this problem are discussed further in 530 Section 5.5. 532 Dynamic changes both to the enhanced VPN and to the underlay 533 transport network need to be managed to avoid disruption to services 534 that are sensitive to changes in network performance. 536 In addition to non-disruptively managing the network during changes 537 such as the inclusion of a new VPN endpoint or a change to a link, 538 VPN traffic might need to be moved because of changes to traffic 539 patterns and volumes. 541 3.5. Customized Control 543 In many cases the customers are delivered with enhanced VPN services 544 without knowing the information about the underlying VTNs. However, 545 depends on the agreement between the operator and the customer, in 546 some cases the customer may also be provided with some information 547 about the underlying VTNs. Such information can be filtered or 548 aggregated according to the operator's policy. This allows the 549 customer of the enhanced VPN to have some visibility and even control 550 over how the underlying topology and resources of the VTN are used. 551 For example, the customers may be able to specify the service paths 552 within the VTN for specific traffic flows of their enhanced VPNs. 553 Depending on the requirements, an enhanced VPN customer may have his 554 own network controller, which may be provided with an interface to 555 the control or management system run by the network operator. Note 556 that such control is within the scope of the customer's enhanced VPN, 557 any additional changes beyond this would require some intervention by 558 the network operator. 560 A description of the control plane aspects of this problem are 561 discussed further in Section 5.4. A description of the management 562 plane aspects of this feature can be found in Section 5.5. 564 3.6. Applicability 566 The concept of enhanced VPN can be applied to any existing and future 567 multi-tenancy overlay services including but not limited to : 569 o Layer-2 point-to-point services such as pseudowires [RFC3985] 570 o Layer-2 VPNs [RFC4664] 572 o Ethernet VPNs [RFC7209] 574 o Layer-3 VPNs [RFC4364], [RFC2764] 576 Where such VPN service types need enhanced isolation and delivery 577 characteristics, the technologies described in Section 5 can be used 578 to provide an underlay with the required enhanced performance. 580 3.7. Inter-Domain and Inter-Layer Network 582 In some scenarios, an enhanced VPN service may span multiple network 583 domains. A domain is considered to be any collection of network 584 elements within a common realm of address space or path computation 585 responsibility [RFC5151] for example, an Autonomous System. In some 586 domains the network operator may manage a multi-layered network, for 587 example, a packet network over an optical network. When enhanced 588 VPNs are provisioned in such network scenarios, the technologies used 589 in different network planes (data plane, control plane, and 590 management plane) need to provide mechanisms to support multi-domain 591 and multi-layer coordination and integration, so as to provide the 592 required service characteristics for different enhanced VPNs, and 593 improve network efficiency and operational simplicity. 595 4. Architecture of Enhanced VPNs 597 A number of enhanced VPN services will typically be provided by a 598 common network infrastructure. Each enhanced VPN consists of both 599 the overlay and a VTN with a specific set of network resources and 600 service functions allocated in the underlay to satisfy the needs of 601 the VPN customer. One VTN may support one of more enhanced VPNs. 602 The integration between overlay and various underlay resources 603 ensures the required isolation between different enhanced VPNs, and 604 achieves the guaranteed performance for different services. 606 An enhanced VPN needs to be designed with consideration given to: 608 o An enhanced data plane. 610 o A control plane to create enhanced VPNs, making use of the data 611 plane isolation and performance guarantee techniques. 613 o A management plane for enhanced VPN service life-cycle management. 615 These topics are expanded below. 617 o The enhanced data plane: 619 * Provides the required packet latency and jitter 620 characteristics. 622 * Provides the required packet loss characteristics. 624 * Provides the required resource isolation capability, e.g., 625 bandwidth guarantee. 627 * Provides the mechanism to associate a packet with the set of 628 resources allocated to the enhanced VPN to which the packet 629 belongs. 631 o The control plane: 633 * Collects information about the underlying network topology and 634 available resources, and exports this to nodes in the network 635 and/or a centralized controller as required. 637 * Creates the required VTNs with the resources and properties 638 needed by the enhanced VPN services that are they support. 640 * Determines the risk of SLA violation and takes appropriate 641 avoiding action. 643 * Determines the right balance of per-packet and per-node state 644 according to the needs of the enhanced VPN services to scale to 645 the required size. 647 o The management plane: 649 * Provides an interface between the enhanced VPN provider (e.g., 650 operator's network management system ) and the enhanced VPN 651 customer (e.g., an organization or a service with enhanced VPN 652 requirement) such that some of the operation requests can be 653 met without interfering with the enhanced VPN of other 654 customers. 656 * Provides an interface between the enhanced VPN provider and the 657 enhanced VPN customers to expose the network capability 658 information toward the enhanced VPN customer. 660 * Provides the service life-cycle management and operation of 661 enhanced VPNs (e.g., creation, modification, assurance/ 662 monitoring, and decommissioning). 664 o Operations, Administration, and Maintenance (OAM) 665 * Provides the tools to verify the connectivity and performance 666 of the enhanced VPN. 668 * Provides the tools to verify whether the underlay network 669 resources are correctly allocated and operating properly. 671 o Telemetry 673 * Provides the mechanisms to collect network information about 674 the operation of the data plane, control plane, and management 675 plane. More specifically, telemetry provides the mechanisms to 676 collect network data: 678 + from the underlay network for overall performance evaluation 679 and for the planning enhanced VPN services. 681 + from each enhanced VPN and for monitoring and analytics of 682 the characteristics and SLA fulfillment of the enhanced VPN 683 services. 685 4.1. Layered Architecture 687 The layered architecture of an enhanced VPN is shown in Figure 2. 689 Underpinning everything is the physical network infrastructure layer 690 which provide the underlying resources used to provision the 691 separated virtual transport networks (VTNs). This includes the 692 partitioning of link and/or node resources. Each subset of link or 693 node resource can be considered as a virtual link or virtual node 694 used to build the VTNs. 696 /\ 697 || 698 +-------------------+ Centralized 699 | Network Controller| Control & Management 700 +-------------------+ 701 || 702 \/ 703 o---------------------------o VPN Service 1 704 /-------------o 705 o____________/______________o VPN Service 2 706 _________________o 707 _____/ 708 o___/ \_________________o VPN Service 3 709 \_______________________o 710 ...... ... 711 o-----------\ /-------------o 712 o____________X______________o VPN Service n 713 __________________________ 714 / o----o-----o / 715 / / / / VTN-1 716 / o-----o-----o----o----o / 717 /_________________________/ 718 __________________________ 719 / o----o / 720 / / / \ / VTN-2 721 / o-----o----o---o------o / 722 /_________________________/ 723 ...... ... 724 ___________________________ 725 / o----o / 726 / / / / VTN-m 727 / o-----o----o----o-----o / 728 /__________________________/ 730 ++++ ++++ ++++ 731 +--+===+--+===+--+ 732 +--+===+--+===+--+ 733 ++++ +++\\ ++++ 734 || || \\ || Physical 735 || || \\ || Network 736 ++++ ++++ ++++ \\+++ ++++ Infrastructure 737 +--+===+--+===+--+===+--+===+--+ 738 +--+===+--+===+--+===+--+===+--+ 739 ++++ ++++ ++++ ++++ ++++ 741 o Virtual Node ++++ 742 +--+ Physical Node with resource partition 743 -- Virtual Link +--+ 744 ++++ 745 == Physical Link with resource partition 747 Figure 2: The Layered Architecture of VPN+ 749 Various components and techniques discussed in Section 5 can be used 750 to enable resource partitioning, such as FlexE, TSN, DetNet, 751 dedicated queues, etc. These partitions may be physical or virtual 752 so long as the SLA required by the higher layers is met. 754 Based on the network resources provided by the physical network 755 infrastructure, multiple VTNs can be provisioned, each with 756 customized topology and other attributes to meet the requirements of 757 different enhanced VPNs or different groups of enhanced VPNs. To get 758 the required characteristics, each VTN needs to be mapped to a set of 759 network nodes and links in the network infrastructure. And on each 760 node or link, the VTN is associated with a set of resources which are 761 allocated for the processing of traffic in the VTN. The VTN provides 762 the integration between the virtual network topology and the required 763 underlying network resources. The VTN is an essential scaling 764 technique, as it has the potential of eliminating per-path state from 765 the network. In addition, when a group of enhanced VPNs is supported 766 by a single VTN, there is need only to maintain network state for the 767 single VTN (see Section 4.4 for more information). 769 The centralized network controller is used to create the VTN, and to 770 instruct the network nodes to allocate the required resources to each 771 VTN and to provision the enhanced VPN services on the VTNs. A 772 distributed control plane may also be used for the distribution of 773 the VTN topology and attribute information between nodes within the 774 VTNs. 776 The process used to create VTNs and to allocate network resources for 777 use by VTNs needs to take a holistic view of the needs of all of its 778 customers and to partition the resources accordingly. However, 779 within a VTN these resources can, if required, be managed via a 780 dynamic control plane. This provides the required scalability and 781 isolation. 783 4.2. Multi-Point to Multi-Point (MP2MP) Connectivity 785 At the level of a overlay VPN service, the required connectivity for 786 an MP2MP service is usually full or partial mesh. To support such 787 VPN services, the corresponding VTN connectivity also needs to have 788 an abstracted MP2MP connectivity. 790 Other service requirements may be expressed at different 791 granularities, some of which can be applicable to the whole service, 792 while some others may only be applicable to some pairs of end points. 793 For example, when a particular level of performance guarantee is 794 required, the point-to-point path through the underlying VTN of the 795 enhanced VPN may need to be specifically engineered to meet the 796 required performance guarantee. 798 4.3. Application Specific Data Types 800 Although a lot of the traffic that will be carried over the enhanced 801 VPN will likely be IPv4 or IPv6, the design must be capable of 802 carrying other traffic types, in particular Ethernet traffic. This 803 is easily accomplished through the various pseudowire (PW) techniques 804 [RFC3985]. Where the underlay is MPLS, Ethernet can be carried over 805 the enhanced VPN encapsulated according to the method specified in 806 [RFC4448]. Where the underlay is IP, Layer Two Tunneling Protocol - 807 Version 3 (L2TPv3) [RFC3931] can be used with Ethernet traffic 808 carried according to [RFC4719]. Encapsulations have been defined for 809 most of the common Layer-2 types for both PW over MPLS and for 810 L2TPv3. 812 4.4. Scaling Considerations 814 VPNs are instantiated as overlays on top of an operator's network and 815 offered as services to the operator's customers. An important 816 feature of overlays is that they can deliver services without placing 817 per-service state in the core of the underlay network. 819 Enhanced VPNs may need to install some additional state within the 820 network to achieve the features that they require. Solutions must 821 consider minimizing and controlling the scale of such state, and 822 deployment architectures should constrain the number of enhanced VPNs 823 that would exist where such services would place additional state in 824 the network. It is expected that the number of enhanced VPNs will be 825 small at the beginning, and even in future the number of enhanced 826 VPNs will be much fewer than traditional VPNs because pre-existing 827 VPN techniques are good enough to meet the needs of most existing 828 VPN-type services. 830 In general, it is not required that the state in the network be 831 maintained in a 1:1 relationship with the VPN+ services. It will 832 usually be possible to aggregate a set or group of VPN+ services so 833 that they share the same VTN and the same set of network resources 834 (much in the same way that current VPNs are aggregated over transport 835 tunnels) so that collections of enhanced VPNs that require the same 836 behavior from the network in terms of resource reservation, latency 837 bounds, resiliency, etc. can be grouped together. This is an 838 important feature to assist with the scaling characteristics of VPN+ 839 deployments. 841 [I-D.dong-teas-enhanced-vpn-vtn-scalability] provides more details of 842 scalability considerations for enhanced VPNs, and Section 6 includes 843 a greater discussion of scalability considerations. 845 5. Candidate Technologies 847 A VPN is a network created by applying a demultiplexing technique to 848 the underlying network (the underlay) to distinguish the traffic of 849 one VPN from that of another. A VPN path that travels by other than 850 the shortest path through the underlay normally requires state in the 851 underlay to specify that path. State is normally applied to the 852 underlay through the use of the RSVP-TE signaling protocol, or 853 directly through the use of an SDN controller, although other 854 techniques may emerge as this problem is studied. This state gets 855 harder to manage as the number of VPN paths increases. Furthermore, 856 as we increase the coupling between the underlay and the overlay to 857 support the enhanced VPN service, this state will increase further. 858 Thus, an enhanced VPN solution needs tighter coupling with the 859 underlay than is the case with existing VPN techniques. We cannot, 860 for example, share the network resource between enhanced VPNs which 861 require hard isolation. 863 In an enhanced VPN, different subsets of the underlay resources can 864 be dedicated to different enhanced VPNs or different groups of 865 enhanced VPNs through the use of VTNs. 867 5.1. Packet Forwarding Plane Technologies 869 Several candidate Layer 2 packet- or frame-based data plane solutions 870 which provide the required isolation and guarantees are described in 871 the following sections. 873 5.1.1. Flexible Ethernet 875 FlexE [FLEXE] provides the ability to multiplex channels over an 876 Ethernet link to create point-to-point fixed- bandwidth connections 877 in a way that provides hard isolation. FlexE also supports bonding 878 links to create larger links out of multiple low capacity links. 880 However, FlexE is only a link level technology. When packets are 881 received by the downstream node, they need to be processed in a way 882 that preserves that isolation in the downstream node. This in turn 883 requires a queuing and forwarding implementation that preserves the 884 end-to-end isolation. 886 If different FlexE channels are used for different services, then no 887 sharing is possible between the FlexE channels. This means that it 888 may be difficult to dynamically redistribute unused bandwidth to 889 lower priority services in another FlexE channel. If one FlexE 890 channel is used by one customer, the customer can use some methods to 891 manage the relative priority of their own traffic in the FlexE 892 channel. 894 5.1.2. Dedicated Queues 896 DiffServ based queuing systems are described in [RFC2475] and 897 [RFC4594]. This approach is not sufficient to provide isolation for 898 enhanced VPNs because DiffServ does not provide enough markers to 899 differentiate between traffic of a large number of enhanced VPNs. 900 Nor does DiffServ offer the range of service classes that each VPN 901 needs to provide to its tenants. This problem is particularly acute 902 with an MPLS underlay, because MPLS only provides eight traffic 903 classes. 905 In addition, DiffServ, as currently implemented, mainly provides per- 906 hop priority-based scheduling, and it is difficult to use it to 907 achieve quantitative resource reservation. 909 To address these problems and to reduce the potential interference 910 between enhanced VPNs, it would be necessary to steer traffic to 911 dedicated input and output queues per enhanced VPN: some routers have 912 a large number of queues and sophisticated queuing systems which 913 could support this, while some routers may struggle to provide the 914 granularity and level of isolation required by the applications of 915 enhanced VPN. 917 5.1.3. Time Sensitive Networking 919 Time Sensitive Networking (TSN) [TSN] is an IEEE project to provide a 920 method of carrying time sensitive information over Ethernet. It 921 introduces the concept of packet scheduling where a packet stream may 922 be given a time slot guaranteeing that it experiences no queuing 923 delay or increase in latency beyond the very small scheduling delay. 924 The mechanisms defined in TSN can be used to meet the requirements of 925 time sensitive services of an enhanced VPN. 927 Ethernet can be emulated over a Layer 3 network using an IP or MPLS 928 pseudowire. However, a TSN Ethernet payload would be opaque to the 929 underlay and thus not treated specifically as time sensitive data. 930 The preferred method of carrying TSN over a Layer 3 network is 931 through the use of deterministic networking as explained in 932 Section 5.2.1. 934 5.2. Layer Three Data Plane 936 This section considers the problem of enhanced VPN differentiation 937 and resource representation in the network layer. 939 5.2.1. Deterministic Networking 941 Deterministic Networking (DetNet) [RFC8655] is a technique being 942 developed in the IETF to enhance the ability of Layer-3 networks to 943 deliver packets more reliably and with greater control over the 944 delay. The design cannot use re-transmission techniques such as TCP 945 since that can exceed the delay tolerated by the applications. Even 946 the delay improvements that are achieved with Stream Control 947 Transmission Protocol Partial Reliability Extension (SCTP-PR) 948 [RFC3758] may not meet the bounds set by application demands. DetNet 949 pre-emptively sends copies of the packet over various paths to 950 minimize the chance of all copies of a packet being lost. It also 951 seeks to set an upper bound on latency, but the goal is not to 952 minimize latency. 954 5.2.2. MPLS Traffic Engineering (MPLS-TE) 956 MPLS-TE [RFC2702][RFC3209] introduces the concept of reserving end- 957 to-end bandwidth for a TE-LSP, which can be used to provide a point- 958 to-point Virtual Transport Path (VTP) across the underlay network to 959 support VPNs. VPN traffic can be carried over dedicated TE-LSPs to 960 provide reserved bandwidth for each specific connection in a VPN, and 961 VPNs with similar behavior requirements may be multiplexed onto the 962 same TE-LSPs. Some network operators have concerns about the 963 scalability and management overhead of MPLS-TE system especially with 964 regard to those systems that use an active control plane, and this 965 has lead them to consider other solutions for their networks. 967 5.2.3. Segment Routing 969 Segment Routing (SR) [RFC8402] is a method that prepends instructions 970 to packets at the head-end of a path. These instructions are used to 971 specify the nodes and links to be traversed, and allow the packets to 972 be routed on paths other than the shortest path. By encoding the 973 state in the packet, per-path state is transitioned out of the 974 network. 976 An SR traffic engineered path operates with a granularity of a link. 977 Hints about priority are provided using the Traffic Class (TC) or 978 Differentiated Services Code Point (DSCP) field in the header. 979 However, to achieve the latency and isolation characteristics that 980 are sought by enhanced VPN customers, it will be necessary to steer 981 packets through specific virtual links and/or queues on the same link 982 and direct them to use specific resources. With SR, it is possible 983 to introduce such fine-grained packet steering by specifying the 984 queues and resources through an SR instruction list. 986 Note that the concept of a queue is a useful abstraction for 987 different types of underlay mechanism that may be used to provide 988 enhanced isolation and latency support. How the queue satisfies the 989 requirement is implementation specific and is transparent to the 990 layer-3 data plane and control plane mechanisms used. 992 With Segment Routing, the SR instruction list could be used to build 993 a P2P path, and a group of SR SIDs could also be used to represent an 994 MP2MP network. Thus, the SR based mechanism could be used to provide 995 both a Virtual Transport Path (VTP) and a Virtual Transport Network 996 (VTN) for enhanced VPN services. 998 5.3. Non-Packet Data Plane 1000 Non-packet underlay data plane technologies often have TE properties 1001 and behaviors, and meet many of the key requirements in particular 1002 for bandwidth guarantees, traffic isolation (with physical isolation 1003 often being an integral part of the technology), highly predictable 1004 latency and jitter characteristics, measurable loss characteristics, 1005 and ease of identification of flows. The cost is that the resources 1006 are allocated on a long-term and end-to-end basis. Such an 1007 arrangement means that the full cost of the resources has to be borne 1008 by the service that is allocated with the resources. 1010 5.4. Control Plane 1012 An enhanced VPN would likely be based on a hybrid control mechanism 1013 that takes advantage of a logically centralized controller for on- 1014 demand provisioning and global optimization, whilst still relying on 1015 a distributed control plane to provide scalability, high reliability, 1016 fast reaction, automatic failure recovery, etc. Extension to and 1017 optimization of the centralized and distributed control plane is 1018 needed to support the enhanced properties of VPN+. 1020 RSVP-TE [RFC3209] provides the signaling mechanism for establishing a 1021 TE-LSP in an MPLS network with end-to-end resource reservation. This 1022 can be seen as an approach of providing a Virtual Transport Path 1023 (VTP) which could be used to bind the VPN to specific network 1024 resources allocated within the underlay, but there remain scalability 1025 concerns as mentioned in Section 5.2.2. 1027 The control plane of SR [RFC8665] [RFC8667] 1028 [I-D.ietf-idr-bgp-ls-segment-routing-ext] does not have the 1029 capability of signaling resource reservations along the path. On the 1030 other hand, the SR approach provides a potential way of binding the 1031 underlay network resource and the enhanced VPN service without 1032 requiring per-path state to be maintained in the network. A 1033 centralized controller can perform resource planning and reservation 1034 for enhanced VPNs, while it needs to ensure that resources are 1035 correctly allocated in network nodes for the enhanced VPN service. 1036 The controller could also compute the SR paths based on the planned 1037 or collected network resource and other attributes, and provision the 1038 SR paths based on the mechanism in 1039 [I-D.ietf-spring-segment-routing-policy] to the ingress nodes of the 1040 enhanced VPN services. The distributed control plane may be used to 1041 advertise the network attributes associated with enhanced VPNs, and 1042 compute the SR paths with specific constraints of enhanced VPN 1043 services. 1045 5.5. Management Plane 1047 The management plane provides the interface between the enhanced VPN 1048 provider and the customers for life-cycle management of the service 1049 (i.e., creation, modification, assurance/monitoring, and 1050 decommissioning). It relies on a set of service data models for the 1051 description of the information and operations needed on the 1052 interface. 1054 As an example, in the context of 5G end-to-end network slicing 1055 [TS28530], the management of enhanced VPNs is considered as the 1056 management of the transport network part of the 5G end-to-end network 1057 slice. The 3GPP management system may provide the connectivity and 1058 performance related parameters as requirements to the management 1059 plane of the transport network. It may also require the transport 1060 network to expose the capabilities and status of the network slice. 1061 Thus, an interface between the enhanced VPN management plane and the 1062 5G network slice management system, and relevant service data models 1063 are needed for the coordination of 5G end-to-end network slice 1064 management. 1066 The management plane interface and data models for enhanced VPN can 1067 be based on the service models described in Section 5.6. 1069 It is important that the management life-cycle supports in-place 1070 modification of enhanced VPNs. That is, it should be possible to add 1071 and remove end points, as well as to change the requested 1072 characteristics of the service that is delivered. The management 1073 system needs to be able to assess the revised VPN+ requests and 1074 determine whether they can be provided by the existing VTN or whether 1075 changes must be made, and it will additionally need to determine 1076 whether those changes to the VTN are possible. If not, then the 1077 customer's modification request may be rejected. 1079 When the modification of an enhanced VPN is possible, the management 1080 system should make every effort to make the changes in a non- 1081 disruptive way. That is, the modification of the enhanced VPN or the 1082 underlying VTN should not perturbate traffic on the enhanced VPN in a 1083 way that causes the service level to drop below the agreed levels. 1084 Furthermore, in the spirit of isolation, changes to one enhanced VPN 1085 should not cause disruption to other enhanced VPNs. 1087 The network operator for the underlay network (i.e., the provider of 1088 the enhanced VPN) may delegate some operational aspects of the 1089 enhanced VPN to the customer. In this way, the VPN+ is presented to 1090 the customer as a virtual network, and the customer can choose how to 1091 use that network. The customer cannot exceed the capabilities of 1092 virtual links and nodes, but can decide how to load traffic onto the 1093 network, for example, by assigning different metrics to the virtual 1094 links so that the customer can control how traffic is routed through 1095 the overlay. This approach requires a management system for the 1096 overlay network, but does not necessarily require any coordination 1097 between the underlay and overlay management systems, except that the 1098 overlay management system might notice when the enhanced VPN network 1099 is close to capacity or considerably under-used and automatically 1100 request changes in the service provided by the underlay. 1102 5.6. Applicability of Service Data Models to Enhanced VPN 1104 This section describes the applicability of the existing and in- 1105 progress service data models to enhanced VPN. New service models may 1106 also be introduced for some of the required management functions. 1108 The ACTN framework[RFC8453] supports operators in viewing and 1109 controlling different domains and presenting virtualized networks to 1110 their customers. It highlights how: 1112 o Abstraction of the underlying network resources is provided to 1113 higher-layer applications and customers. 1115 o Underlying resources are virtualized and allocated for the 1116 customer, application, or service. 1118 o A virtualized environment is created allowing operators to view 1119 and control multi-domain networks as a single virtualized network. 1121 o Networks can be presented to customers as a virtual network via 1122 open and programmable interfaces. 1124 The type of network virtualization enabled by ACTN managed 1125 infrastructure provides customers with the capability to utilize and 1126 independently control allocated virtual network resources as if they 1127 were physically their own resources. Service Data models are used to 1128 represent, monitor, and manage the virtual networks and services 1129 enabled by ACTN. The VPN customer service models (e.g., the layer 3 1130 VPN service model (L3SM) [RFC8299], the layer 2 VPN service model 1131 (L2SM) [RFC8466]), or the ACTN Virtual Network (VN) model 1132 [I-D.ietf-teas-actn-vn-yang]) are a customer view of the ACTN managed 1133 infrastructure, and is presented by the ACTN provider as a set of 1134 abstracted services or resources. The layer 3 VPN network model 1135 (L3NM) [I-D.ietf-opsawg-l3sm-l3nm] and layer 2 VPN network model 1136 (L2NM) [I-D.ietf-opsawg-l2nm] provide network views of the ACTN 1137 managed infrastructure presented by the ACTN provider as a set of 1138 virtual networks and the associated resources. The VTN network model 1139 [I-D.wd-teas-vtn-network-yang] further provides the management of the 1140 virtual underlay network topology and resources for the mapping of 1141 the VPN network models. 1143 [I-D.king-teas-applicability-actn-slicing] discusses the 1144 applicability of the ACTN approach in the context of network slicing. 1145 Since there is a strong correlation between network slices and 1146 enhanced VPNs, that document can also give guidance on how ACTN can 1147 be applied to enhanced VPNs. 1149 5.6.1. An Example of Enhanced VPN Delivery 1151 One typical use case of enhanced VPN is to instantiate a network 1152 slice. In order to provide network slices to customers, a 1153 technology-agnostic network slice Northbound Interface (NBI) data 1154 model is needed for the customers to communicate the requirements and 1155 operations of network slices (end points, connectivity, SLOs, and 1156 SLEs). These requirements may then be realized using technology- 1157 specific Southbound Interfaces (SBIs) to instruct the network to 1158 instantiate an enhanced VPN service to meet the requirements of the 1159 customer. 1161 As per [RFC8453] and [I-D.ietf-teas-actn-yang], the CNC-MDSC 1162 Interface (CMI) of ACTN can be used to convey the virtual network 1163 service requirements, which is a generic interface to deliver various 1164 TE based VN services. In the context of the network slice NBI, there 1165 may be some gaps in the combination of the L3SM/L2SM and VN models. 1166 The NBI is required to communicate the connectivity of the network 1167 slice, along with the SLO parameters, SLEs, and traffic selection 1168 rules, and provides a way to monitor the state of the network slice. 1169 This can be described in a more abstract manner, so as to reduce the 1170 association with specific technologies used to realize the network 1171 slice, such as the VPN and TE technologies. The network slice NBI 1172 model as defined in [I-D.wd-teas-ietf-network-slice-nbi-yang] 1173 provides an abstract and generic approach to provide the network 1174 slice NBI functions. 1176 The MDSC-PNC Interface (MPI) models in the ACTN architecture can be 1177 used for the realization of network slices, for example, in a TE 1178 enabled network, and may also be used for cross-layer or cross-domain 1179 implementation of network slice. 1181 6. Scalability Considerations 1183 An enhanced VPN provides performance guaranteed services in packet 1184 networks, but with the potential cost of introducing additional state 1185 into the network. There are at least three ways that this additional 1186 state might be brought into the network: 1188 o Introduce the complete state into the packet, as is done in SR. 1189 This allows the controller to specify the detailed series of 1190 forwarding and processing instructions for the packet as it 1191 transits the network. The cost of this is an increase in the 1192 packet header size. The cost is also that systems will have 1193 capabilities enabled in case they are called upon by a service. 1194 This is a type of latent state, and increases as the path and 1195 resources that need to be exclusively available to a VPN are 1196 specified more precisely. 1198 o Introduce the state to the network. This is normally done by 1199 creating a path using signaling such as RSVP-TE. This could be 1200 extended to include any element that needs to be specified along 1201 the path, for example explicitly specifying queuing policy. It is 1202 also possible to use other methods to introduce path state, such 1203 as via an SDN controller, or possibly by modifying a routing 1204 protocol. With this approach there is state per path: per-path 1205 characteristic that needs to be maintained over the life of the 1206 path. This is more network state than is needed using SR, but the 1207 packets are usually shorter. 1209 o Provide a hybrid approach. One example is based on using binding 1210 SIDs [RFC8402] to create path fragments, and bind them together 1211 with SR. Dynamic creation of a VPN service path using SR requires 1212 less state maintenance in the network core at the expense of 1213 larger packet headers. The packet size can be lower if a form of 1214 loose source routing is used (using a few nodal SIDs), and it will 1215 be lower if no specific functions or resources on the routers are 1216 specified. 1218 Reducing the state in the network is important to enhanced VPN, as it 1219 requires the overlay to be more closely integrated with the underlay 1220 than with traditional VPNs. This tighter coupling would normally 1221 mean that more state needs to be created and maintained in the 1222 network, as the state about fine granularity processing would need to 1223 be loaded and maintained in the routers. However, an SR approach 1224 allows much of this state to be spread amongst the network ingress 1225 nodes, and transiently carried in the packets as SIDs. 1227 Further discussion of the scalability considerations of enhanced VPNs 1228 can be found in [I-D.dong-teas-enhanced-vpn-vtn-scalability]. 1230 6.1. Maximum Stack Depth of SR 1232 One of the challenges with SR is the stack depth that nodes are able 1233 to impose on packets [RFC8491]. This leads to a difficult balance 1234 between adding state to the network and minimizing stack depth, or 1235 minimizing state and increasing the stack depth. 1237 6.2. RSVP-TE Scalability 1239 The traditional method of creating a resource allocated path through 1240 an MPLS network is to use the RSVP-TE protocol. However, there have 1241 been concerns that this requires significant continuous state 1242 maintenance in the network. Work to improve the scalability of RSVP- 1243 TE LSPs in the control plane can be found in [RFC8370]. 1245 There is also concern at the scalability of the forwarder footprint 1246 of RSVP-TE as the number of paths through a label switching router 1247 (LSR) grows. [RFC8577] addresses this by employing SR within a 1248 tunnel established by RSVP-TE. 1250 6.3. SDN Scaling 1252 The centralized approach of SDN requires state to be stored in the 1253 network, but does not have the overhead of also requiring control 1254 plane state to be maintained. Each individual network node may need 1255 to maintain a communication channel with the SDN controller, but that 1256 compares favorably with the need for a control plane to maintain 1257 communication with all neighbors. 1259 However, SDN may transfer some of the scalability concerns from the 1260 network to the centralized controller. In particular, there may be a 1261 heavy processing burden at the controller, and a heavy load in the 1262 network surrounding the controller. A centralized controller also 1263 presents a single point of failure within the network. 1265 7. OAM Considerations 1267 The design of OAM for enhanced VPNs needs to consider the following 1268 requirements: 1270 o Instrumentation of the underlay so that the network operator can 1271 be sure that the resources committed to a tenant are operating 1272 correctly and delivering the required performance. 1274 o Instrumentation of the overlay by the tenant. This is likely to 1275 be transparent to the network operator and to use existing 1276 methods. Particular consideration needs to be given to the need 1277 to verify the isolation and the various committed performance 1278 characteristics. 1280 o Instrumentation of the overlay by the network provider to 1281 proactively demonstrate that the committed performance is being 1282 delivered. This needs to be done in a non-intrusive manner, 1283 particularly when the tenant is deploying a performance sensitive 1284 application. 1286 o Verification of the conformity of the path to the service 1287 requirement. This may need to be done as part of a commissioning 1288 test. 1290 A study of OAM in SR networks has been documented in [RFC8403]. 1292 8. Telemetry Considerations 1294 Network visibility is essential for network operation. Network 1295 telemetry has been considered as an ideal means to gain sufficient 1296 network visibility with better flexibility, scalability, accuracy, 1297 coverage, and performance than conventional OAM technologies. 1299 As defined in [I-D.ietf-opsawg-ntf], the objective of Network 1300 Telemetry is to acquire network data remotely for network monitoring 1301 and operation. It is a general term for a large set of network 1302 visibility techniques and protocols. Network telemetry addresses the 1303 current network operation issues and enables smooth evolution toward 1304 intent-driven autonomous networks. Telemetry can be applied on the 1305 forwarding plane, the control plane, and the management plane in a 1306 network. 1308 How the telemetry mechanisms could be used or extended for the 1309 enhanced VPN service is out of the scope of this document. 1311 9. Enhanced Resiliency 1313 Each enhanced VPN has a life cycle, and may need modification during 1314 deployment as the needs of its tenant change. This is discussed in 1315 Section 5.5. Additionally, as the network evolves, there may need to 1316 be garbage collection performed to consolidate resources into usable 1317 quanta. 1319 Systems in which the path is imposed, such as SR or some form of 1320 explicit routing, tend to do well in these applications, because it 1321 is possible to perform an atomic transition from one path to another. 1322 That is, a single action by the head-end that changes the path 1323 without the need for coordinated action by the routers along the 1324 path. However, implementations and the monitoring protocols need to 1325 make sure that the new path is operational and meets the required SLA 1326 before traffic is transitioned to it. It is possible for deadlocks 1327 to arise as a result of the network becoming fragmented over time, 1328 such that it is impossible to create a new path or to modify an 1329 existing path without impacting the SLA of other paths. Resolution 1330 of this situation is as much a commercial issue as it is a technical 1331 issue and is outside the scope of this document. 1333 There are, however, two manifestations of the latency problem that 1334 are for further study in any of these approaches: 1336 o The problem of packets overtaking one another if a path latency 1337 reduces during a transition. 1339 o The problem of transient variation in latency in either direction 1340 as a path migrates. 1342 There is also the matter of what happens during failure in the 1343 underlay infrastructure. Fast reroute is one approach, but that 1344 still produces a transient loss with a normal goal of rectifying this 1345 within 50ms [RFC5654]. An alternative is some form of N+1 delivery 1346 such as has been used for many years to support protection from 1347 service disruption. This may be taken to a different level using the 1348 techniques of DetNet with multiple in-network replication and the 1349 culling of later packets [RFC8655]. 1351 In addition to the approach used to protect high priority packets, 1352 consideration should be given to the impact of best effort traffic on 1353 the high priority packets during a transition. Specifically, if a 1354 conventional re-convergence process is used there will inevitably be 1355 micro-loops and whilst some form of explicit routing will protect the 1356 high priority traffic, lower priority traffic on best effort shortest 1357 paths will micro-loop without the use of a loop prevention 1358 technology. To provide the highest quality of service to high 1359 priority traffic, either this traffic must be shielded from the 1360 micro-loops, or micro-loops must be prevented completely. 1362 10. Operational Considerations 1364 It is likely that enhanced VPN services will be introduced in 1365 networks which already have traditional VPN services deployed. 1366 Depending on service requirements, the tenants or the operator may 1367 choose to use a traditional VPN or an enhanced VPN to fulfill a 1368 service requirement. The information and parameters to assist such a 1369 decision needs to be reflected on the management interface between 1370 the tenant and the operator. 1372 11. Security Considerations 1374 All types of virtual network require special consideration to be 1375 given to the isolation of traffic belonging to different tenants. 1376 That is, traffic belonging to one VPN must not be delivered to end 1377 points outside that VPN. In this regard enhanced VPNs neither 1378 introduce, nor experience a greater security risks than other VPNs. 1380 However, in an enhanced Virtual Private Network service the 1381 additional service requirements need to be considered. For example, 1382 if a service requires a specific upper bound to latency then it can 1383 be damaged by simply delaying the packets through the activities of 1384 another tenant, i.e., by introducing bursts of traffic for other 1385 services. In some respects this makes the enhanced VPN more 1386 susceptible to attacks since the SLA may be broken. But another view 1387 is that the operator must, in any case, preform monitoring of the 1388 enhanced VPN to ensure that the SLA is met, and this means that the 1389 operator may be more likely to spot the early onset of a security 1390 attack and be able to take pre-emptive protective action. 1392 The measures to address these dynamic security risks must be 1393 specified as part to the specific solution are form part of the 1394 isolation requirements of a service. 1396 While an enhanced VPN service may be sold as offering encryption and 1397 other security features as part of the service, customers would be 1398 well advised to take responsibility for their own security 1399 requirements themselves possibly by encrypting traffic before handing 1400 it off to the service provider. 1402 The privacy of enhanced VPN service customers must be preserved. It 1403 should not be possible for one customer to discover the existence of 1404 another customer, nor should the sites that are members of an 1405 enhanced VPN be externally visible. 1407 12. IANA Considerations 1409 There are no requested IANA actions. 1411 13. Contributors 1412 Daniel King 1413 Email: daniel@olddog.co.uk 1415 Adrian Farrel 1416 Email: adrian@olddog.co.uk 1418 Jeff Tansura 1419 Email: jefftant.ietf@gmail.com 1421 Zhenbin Li 1422 Email: lizhenbin@huawei.com 1424 Qin Wu 1425 Email: bill.wu@huawei.com 1427 Bo Wu 1428 Email: lana.wubo@huawei.com 1430 Daniele Ceccarelli 1431 Email: daniele.ceccarelli@ericsson.com 1433 Mohamed Boucadair 1434 Email: mohamed.boucadair@orange.com 1436 Sergio Belotti 1437 Email: sergio.belotti@nokia.com 1439 Haomian Zheng 1440 Email: zhenghaomian@huawei.com 1442 14. Acknowledgements 1444 The authors would like to thank Charlie Perkins, James N Guichard, 1445 John E Drake, Shunsuke Homma, and Luis M. Contreras for their review 1446 and valuable comments. 1448 This work was supported in part by the European Commission funded 1449 H2020-ICT-2016-2 METRO-HAUL project (G.A. 761727). 1451 15. References 1453 15.1. Normative References 1455 [RFC2764] Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. 1456 Malis, "A Framework for IP Based Virtual Private 1457 Networks", RFC 2764, DOI 10.17487/RFC2764, February 2000, 1458 . 1460 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1461 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1462 DOI 10.17487/RFC3985, March 2005, 1463 . 1465 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 1466 2 Virtual Private Networks (L2VPNs)", RFC 4664, 1467 DOI 10.17487/RFC4664, September 2006, 1468 . 1470 15.2. Informative References 1472 [BBF-SD406] 1473 "BBF SD-406: End-to-End Network Slicing", 2016, 1474 . 1477 [DETNET] "Deterministic Networking", March , 1478 . 1480 [FLEXE] "Flex Ethernet Implementation Agreement", March 2016, 1481 . 1484 [I-D.dong-teas-enhanced-vpn-vtn-scalability] 1485 Dong, J., Li, Z., Qin, F., Yang, G., and J. N. Guichard, 1486 "Scalability Considerations for Enhanced VPN (VPN+)", 1487 draft-dong-teas-enhanced-vpn-vtn-scalability-02 (work in 1488 progress), February 2021. 1490 [I-D.ietf-idr-bgp-ls-segment-routing-ext] 1491 Previdi, S., Talaulikar, K., Filsfils, C., Gredler, H., 1492 and M. Chen, "BGP Link-State extensions for Segment 1493 Routing", draft-ietf-idr-bgp-ls-segment-routing-ext-18 1494 (work in progress), April 2021. 1496 [I-D.ietf-opsawg-l2nm] 1497 Barguil, S., Dios, O. G. D., Boucadair, M., and L. A. 1498 Munoz, "A Layer 2 VPN Network YANG Model", draft-ietf- 1499 opsawg-l2nm-02 (work in progress), April 2021. 1501 [I-D.ietf-opsawg-l3sm-l3nm] 1502 Barguil, S., Dios, O. G. D., Boucadair, M., Munoz, L. A., 1503 and A. Aguado, "A Layer 3 VPN Network YANG Model", draft- 1504 ietf-opsawg-l3sm-l3nm-08 (work in progress), April 2021. 1506 [I-D.ietf-opsawg-ntf] 1507 Song, H., Qin, F., Martinez-Julia, P., Ciavaglia, L., and 1508 A. Wang, "Network Telemetry Framework", draft-ietf-opsawg- 1509 ntf-07 (work in progress), February 2021. 1511 [I-D.ietf-spring-segment-routing-policy] 1512 Filsfils, C., Talaulikar, K., Voyer, D., Bogdanov, A., and 1513 P. Mattes, "Segment Routing Policy Architecture", draft- 1514 ietf-spring-segment-routing-policy-11 (work in progress), 1515 April 2021. 1517 [I-D.ietf-teas-actn-vn-yang] 1518 Lee, Y., Dhody, D., Ceccarelli, D., Bryskin, I., and B. Y. 1519 Yoon, "A YANG Data Model for VN Operation", draft-ietf- 1520 teas-actn-vn-yang-11 (work in progress), February 2021. 1522 [I-D.ietf-teas-actn-yang] 1523 Lee, Y., Zheng, H., Ceccarelli, D., Yoon, B. Y., and S. 1524 Belotti, "Applicability of YANG models for Abstraction and 1525 Control of Traffic Engineered Networks", draft-ietf-teas- 1526 actn-yang-07 (work in progress), February 2021. 1528 [I-D.ietf-teas-ietf-network-slices] 1529 Farrel, A., Gray, E., Drake, J., Rokui, R., Homma, S., 1530 Makhijani, K., Contreras, L. M., and J. Tantsura, 1531 "Framework for IETF Network Slices", draft-ietf-teas-ietf- 1532 network-slices-00 (work in progress), April 2021. 1534 [I-D.king-teas-applicability-actn-slicing] 1535 King, D., Drake, J., Zheng, H., and A. Farrel, 1536 "Applicability of Abstraction and Control of Traffic 1537 Engineered Networks (ACTN) to Network Slicing", draft- 1538 king-teas-applicability-actn-slicing-10 (work in 1539 progress), March 2021. 1541 [I-D.wd-teas-ietf-network-slice-nbi-yang] 1542 Wu, B., Dhody, D., Han, L., and R. Rokui, "A Yang Data 1543 Model for IETF Network Slice NBI", draft-wd-teas-ietf- 1544 network-slice-nbi-yang-02 (work in progress), February 1545 2021. 1547 [I-D.wd-teas-vtn-network-yang] 1548 Wu, B. and D. Dhody, "A VTN Network YANG Module", draft- 1549 wd-teas-vtn-network-yang-00 (work in progress), June 2021. 1551 [NGMN-NS-Concept] 1552 "NGMN NS Concept", 2016, . 1556 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1557 and W. Weiss, "An Architecture for Differentiated 1558 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1559 . 1561 [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and J. 1562 McManus, "Requirements for Traffic Engineering Over MPLS", 1563 RFC 2702, DOI 10.17487/RFC2702, September 1999, 1564 . 1566 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1567 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1568 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1569 . 1571 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1572 Conrad, "Stream Control Transmission Protocol (SCTP) 1573 Partial Reliability Extension", RFC 3758, 1574 DOI 10.17487/RFC3758, May 2004, 1575 . 1577 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1578 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 1579 RFC 3931, DOI 10.17487/RFC3931, March 2005, 1580 . 1582 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1583 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1584 2006, . 1586 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 1587 "Encapsulation Methods for Transport of Ethernet over MPLS 1588 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 1589 . 1591 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 1592 Guidelines for DiffServ Service Classes", RFC 4594, 1593 DOI 10.17487/RFC4594, August 2006, 1594 . 1596 [RFC4719] Aggarwal, R., Ed., Townsley, M., Ed., and M. Dos Santos, 1597 Ed., "Transport of Ethernet Frames over Layer 2 Tunneling 1598 Protocol Version 3 (L2TPv3)", RFC 4719, 1599 DOI 10.17487/RFC4719, November 2006, 1600 . 1602 [RFC5151] Farrel, A., Ed., Ayyangar, A., and JP. Vasseur, "Inter- 1603 Domain MPLS and GMPLS Traffic Engineering -- Resource 1604 Reservation Protocol-Traffic Engineering (RSVP-TE) 1605 Extensions", RFC 5151, DOI 10.17487/RFC5151, February 1606 2008, . 1608 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1609 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1610 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1611 September 2009, . 1613 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 1614 Networking: A Perspective from within a Service Provider 1615 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 1616 . 1618 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 1619 Henderickx, W., and A. Isaac, "Requirements for Ethernet 1620 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 1621 . 1623 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 1624 Ceccarelli, D., and X. Zhang, "Problem Statement and 1625 Architecture for Information Exchange between 1626 Interconnected Traffic-Engineered Networks", BCP 206, 1627 RFC 7926, DOI 10.17487/RFC7926, July 2016, 1628 . 1630 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 1631 Network Functions and Their Infrastructure", RFC 8172, 1632 DOI 10.17487/RFC8172, July 2017, 1633 . 1635 [RFC8299] Wu, Q., Ed., Litkowski, S., Tomotaki, L., and K. Ogaki, 1636 "YANG Data Model for L3VPN Service Delivery", RFC 8299, 1637 DOI 10.17487/RFC8299, January 2018, 1638 . 1640 [RFC8370] Beeram, V., Ed., Minei, I., Shakir, R., Pacella, D., and 1641 T. Saad, "Techniques to Improve the Scalability of RSVP-TE 1642 Deployments", RFC 8370, DOI 10.17487/RFC8370, May 2018, 1643 . 1645 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1646 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1647 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1648 July 2018, . 1650 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1651 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1652 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1653 2018, . 1655 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1656 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1657 DOI 10.17487/RFC8453, August 2018, 1658 . 1660 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1661 Data Model for Layer 2 Virtual Private Network (L2VPN) 1662 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1663 2018, . 1665 [RFC8491] Tantsura, J., Chunduri, U., Aldrin, S., and L. Ginsberg, 1666 "Signaling Maximum SID Depth (MSD) Using IS-IS", RFC 8491, 1667 DOI 10.17487/RFC8491, November 2018, 1668 . 1670 [RFC8568] Bernardos, CJ., Rahman, A., Zuniga, JC., Contreras, LM., 1671 Aranda, P., and P. Lynch, "Network Virtualization Research 1672 Challenges", RFC 8568, DOI 10.17487/RFC8568, April 2019, 1673 . 1675 [RFC8577] Sitaraman, H., Beeram, V., Parikh, T., and T. Saad, 1676 "Signaling RSVP-TE Tunnels on a Shared MPLS Forwarding 1677 Plane", RFC 8577, DOI 10.17487/RFC8577, April 2019, 1678 . 1680 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 1681 RFC 8578, DOI 10.17487/RFC8578, May 2019, 1682 . 1684 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1685 "Deterministic Networking Architecture", RFC 8655, 1686 DOI 10.17487/RFC8655, October 2019, 1687 . 1689 [RFC8665] Psenak, P., Ed., Previdi, S., Ed., Filsfils, C., Gredler, 1690 H., Shakir, R., Henderickx, W., and J. Tantsura, "OSPF 1691 Extensions for Segment Routing", RFC 8665, 1692 DOI 10.17487/RFC8665, December 2019, 1693 . 1695 [RFC8667] Previdi, S., Ed., Ginsberg, L., Ed., Filsfils, C., 1696 Bashandy, A., Gredler, H., and B. Decraene, "IS-IS 1697 Extensions for Segment Routing", RFC 8667, 1698 DOI 10.17487/RFC8667, December 2019, 1699 . 1701 [SFC] "Service Function Chaining", March , 1702 . 1704 [TS23501] "3GPP TS23.501", 2016, 1705 . 1708 [TS28530] "3GPP TS28.530", 2016, 1709 . 1712 [TSN] "Time-Sensitive Networking", March , 1713 . 1715 Authors' Addresses 1717 Jie Dong 1718 Huawei 1720 Email: jie.dong@huawei.com 1722 Stewart Bryant 1723 Futurewei 1725 Email: stewart.bryant@gmail.com 1727 Zhenqiang Li 1728 China Mobile 1730 Email: lizhenqiang@chinamobile.com 1731 Takuya Miyasaka 1732 KDDI Corporation 1734 Email: ta-miyasaka@kddi.com 1736 Young Lee 1737 Samsung 1739 Email: younglee.tx@gmail.com