idnits 2.17.1 draft-ietf-teas-enhanced-vpn-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (7 March 2022) is 781 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'I-D.ietf-teas-actn-yang' is defined on line 1644, but no explicit reference was found in the text == Outdated reference: A later version (-02) exists of draft-dong-teas-nrp-scalability-01 == Outdated reference: A later version (-19) exists of draft-ietf-opsawg-l2nm-12 == Outdated reference: A later version (-08) exists of draft-ietf-spring-resource-aware-segments-03 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-policy-18 == Outdated reference: A later version (-24) exists of draft-ietf-teas-actn-vn-yang-13 == Outdated reference: A later version (-11) exists of draft-ietf-teas-actn-yang-08 == Outdated reference: A later version (-06) exists of draft-ietf-teas-applicability-actn-slicing-00 == Outdated reference: A later version (-12) exists of draft-ietf-teas-ietf-network-slice-nbi-yang-01 == Outdated reference: A later version (-25) exists of draft-ietf-teas-ietf-network-slices-08 == Outdated reference: A later version (-02) exists of draft-wd-teas-nrp-yang-00 Summary: 0 errors (**), 0 flaws (~~), 12 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group J. Dong 3 Internet-Draft Huawei 4 Intended status: Informational S. Bryant 5 Expires: 8 September 2022 University of Surrey 6 Z. Li 7 China Mobile 8 T. Miyasaka 9 KDDI Corporation 10 Y. Lee 11 Samsung 12 7 March 2022 14 A Framework for Enhanced Virtual Private Network (VPN+) Services 15 draft-ietf-teas-enhanced-vpn-10 17 Abstract 19 This document describes the framework for Enhanced Virtual Private 20 Network (VPN+) services. The purpose of enhanced VPNs is to support 21 the needs of new applications, particularly applications that are 22 associated with 5G services, by utilizing an approach that is based 23 on the VPN and Traffic Engineering (TE) technologies and adds 24 characteristics that specific services require over those provided by 25 traditional VPNs. 27 Typically, VPN+ will be used to underpin network slicing, but could 28 also be of use in its own right providing enhanced connectivity 29 services between customer sites. 31 It is envisaged that enhanced VPNs will be delivered using a 32 combination of existing, modified, and new networking technologies. 33 This document provides an overview of relevant technologies and 34 identifies some areas for potential new work. 36 Status of This Memo 38 This Internet-Draft is submitted in full conformance with the 39 provisions of BCP 78 and BCP 79. 41 Internet-Drafts are working documents of the Internet Engineering 42 Task Force (IETF). Note that other groups may also distribute 43 working documents as Internet-Drafts. The list of current Internet- 44 Drafts is at https://datatracker.ietf.org/drafts/current/. 46 Internet-Drafts are draft documents valid for a maximum of six months 47 and may be updated, replaced, or obsoleted by other documents at any 48 time. It is inappropriate to use Internet-Drafts as reference 49 material or to cite them other than as "work in progress." 51 This Internet-Draft will expire on 8 September 2022. 53 Copyright Notice 55 Copyright (c) 2022 IETF Trust and the persons identified as the 56 document authors. All rights reserved. 58 This document is subject to BCP 78 and the IETF Trust's Legal 59 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 60 license-info) in effect on the date of publication of this document. 61 Please review these documents carefully, as they describe your rights 62 and restrictions with respect to this document. Code Components 63 extracted from this document must include Revised BSD License text as 64 described in Section 4.e of the Trust Legal Provisions and are 65 provided without warranty as described in the Revised BSD License. 67 Table of Contents 69 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 70 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 71 3. Overview of the Requirements . . . . . . . . . . . . . . . . 7 72 3.1. Performance Guarantees . . . . . . . . . . . . . . . . . 7 73 3.2. Isolation between Enhanced VPN Services . . . . . . . . . 8 74 3.2.1. A Pragmatic Approach to Isolation . . . . . . . . . . 10 75 3.3. Integration . . . . . . . . . . . . . . . . . . . . . . . 11 76 3.3.1. Abstraction . . . . . . . . . . . . . . . . . . . . . 11 77 3.4. Dynamic Changes . . . . . . . . . . . . . . . . . . . . . 12 78 3.5. Customized Control . . . . . . . . . . . . . . . . . . . 12 79 3.6. Applicability to Overlay Technologies . . . . . . . . . . 13 80 3.7. Inter-Domain and Inter-Layer Network . . . . . . . . . . 13 81 4. Architecture of Enhanced VPNs . . . . . . . . . . . . . . . . 14 82 4.1. Layered Architecture . . . . . . . . . . . . . . . . . . 16 83 4.2. Multi-Point to Multi-Point (MP2MP) Connectivity . . . . . 18 84 4.3. Application Specific Data Types . . . . . . . . . . . . . 18 85 4.4. Scaling Considerations . . . . . . . . . . . . . . . . . 18 86 5. Candidate Technologies . . . . . . . . . . . . . . . . . . . 19 87 5.1. Packet Forwarding Plane Technologies . . . . . . . . . . 20 88 5.1.1. Flexible Ethernet . . . . . . . . . . . . . . . . . . 20 89 5.1.2. Dedicated Queues . . . . . . . . . . . . . . . . . . 20 90 5.1.3. Time Sensitive Networking . . . . . . . . . . . . . . 21 91 5.2. Layer Three Data Plane . . . . . . . . . . . . . . . . . 21 92 5.2.1. Deterministic Networking . . . . . . . . . . . . . . 21 93 5.2.2. MPLS Traffic Engineering (MPLS-TE) . . . . . . . . . 22 94 5.2.3. Segment Routing . . . . . . . . . . . . . . . . . . . 22 95 5.3. Non-Packet Data Plane . . . . . . . . . . . . . . . . . . 23 96 5.4. Control Plane . . . . . . . . . . . . . . . . . . . . . . 23 97 5.5. Management Plane . . . . . . . . . . . . . . . . . . . . 24 98 5.6. Applicability of Service Data Models to Enhanced VPN . . 26 99 6. Applicability to Network Slice Realization . . . . . . . . . 26 100 6.1. VTN Planning . . . . . . . . . . . . . . . . . . . . . . 27 101 6.2. VTN Instantiation . . . . . . . . . . . . . . . . . . . . 27 102 6.3. VPN+ Service Provisioning . . . . . . . . . . . . . . . . 27 103 6.4. Network Slice Traffic Steering and Forwarding . . . . . . 28 104 7. Scalability Considerations . . . . . . . . . . . . . . . . . 28 105 7.1. Maximum Stack Depth of SR . . . . . . . . . . . . . . . . 29 106 7.2. RSVP-TE Scalability . . . . . . . . . . . . . . . . . . . 29 107 7.3. SDN Scaling . . . . . . . . . . . . . . . . . . . . . . . 29 108 8. OAM Considerations . . . . . . . . . . . . . . . . . . . . . 30 109 9. Telemetry Considerations . . . . . . . . . . . . . . . . . . 30 110 10. Enhanced Resiliency . . . . . . . . . . . . . . . . . . . . . 31 111 11. Operational Considerations . . . . . . . . . . . . . . . . . 32 112 12. Security Considerations . . . . . . . . . . . . . . . . . . . 32 113 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 114 14. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 33 115 15. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 33 116 16. References . . . . . . . . . . . . . . . . . . . . . . . . . 34 117 16.1. Normative References . . . . . . . . . . . . . . . . . . 34 118 16.2. Informative References . . . . . . . . . . . . . . . . . 34 119 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 40 121 1. Introduction 123 Virtual private networks (VPNs) have served the industry well as a 124 means of providing different groups of users with logically isolated 125 connectivity over a common network. The common or base network that 126 is used to provide the VPNs is often referred to as the underlay, and 127 the VPN is often called an overlay. 129 Customers of a network operator may request a connectivity services 130 with advanced characteristics such as low latency guarantees, bounded 131 jitter, or isolation from other services or customers so that changes 132 in some other service (such as changes in network load, or events 133 such as congestion or outages) have no or only acceptable effect on 134 the throughput or latency of the services provided to the customer. 135 These services are referred to as "enhanced VPNs" (known as VPN+) in 136 that they are similar to VPN services providing the customer with the 137 required connectivity, but in addition they have enhanced 138 characteristics. 140 The concept of network slicing has gained traction driven largely by 141 needs surfacing from 5G [NGMN-NS-Concept] [TS23501] [TS28530]. 142 According to [TS28530], a 5G end-to-end network slice consists of 143 three major types of network segments: Radio Access Network (RAN), 144 Transport Network (TN), and Mobile Core Network (CN). The transport 145 network provides the connectivity between different entities in RAN 146 and CN segments of a 5G end-to-end network slice, with specific 147 performance commitment. 149 [I-D.ietf-teas-ietf-network-slices] defines the terminologies and the 150 characteristics of IETF network slices. It also discusses the 151 general framework, the components and interfaces for requesting and 152 operating IETF network slices. An IETF Network Slice Service enables 153 connectivity between a set of CEs with specific Service Level 154 Objectives (SLOs) and Service Level Expectations (SLEs) over a common 155 underlay network. An IETF Network Slice can be realized as a logical 156 network connecting a number of endpoints and is associated with a set 157 of shared or dedicated network resources that are used to satisfy the 158 Service Level Objectives (SLOs) and Service Level Expectations (SLEs) 159 requirements. In this document (which is solely about IETF 160 technologies) we refer to an "IETF network slice" simply as a 161 "network slice": a network slice is considered one possible use case 162 of an enhanced VPN. 164 A network slice could span multiple technologies (such as IP or 165 Optical) and multiple administrative domains. Depending on the 166 customer's requirement, a network slice could be isolated from other 167 network slices in terms of data plane, control plane, and management 168 plane resources. 170 Network slicing builds on the concepts of resource management, 171 network virtualization, and abstraction to provide performance 172 assurance, flexibility, programmability, and modularity. It may use 173 techniques such as Software Defined Networking (SDN) [RFC7149], 174 network abstraction [RFC7926] and Network Function Virtualization 175 (NFV) [RFC8172] [RFC8568] to create multiple logical (virtual) 176 networks, each tailored for use by a set of services or by a 177 particular tenant or a group of tenants that share the same or 178 similar requirements. These logical networks are created on top of a 179 common underlay network. How the network slices are engineered can 180 be deployment-specific. 182 The requirements of enhanced VPN services cannot be met by simple 183 overlay networks, as these services require tighter coordination and 184 integration between the underlay and the overlay network. VPN+ is 185 built from a VPN overlay and an underlying Virtual Transport Network 186 (VTN) which has a customized network topology and a set of dedicated 187 or shared resources in the underlay network. The enhanced VPN may 188 also include a set of invoked service functions located within the 189 underlay network. Thus, an enhanced VPN can achieve greater 190 isolation with strict performance guarantees. These new properties, 191 which have general applicability, are also of interest as part of a 192 network slicing solution. 194 VPN+ can be used to instantiate a network slice service, and the 195 technique can also be of use in general cases to provide enhanced 196 connectivity services between customer sites or service end points. 197 [I-D.ietf-teas-ietf-network-slices] introduces the concept Network 198 Resource Partition (NRP) as a set of network resources that are 199 available to carry traffic and meet the SLOs and SLEs. An NRP is 200 associated with a network topology to define the set of links and 201 nodes. Thus VTN and NRP are considered as similar concepts, and NRP 202 can be seen as an instantiation of VTN in the context of network 203 slicing. 205 It is not envisaged that VPN+ services will replace traditional VPN 206 services. Traditional VPN services will continue to be delivered 207 using pre-existing mechanisms and can co-exist with VPN+ services. 209 This document describes a framework for using existing, modified, and 210 potential new technologies as components to provide a VPN+ service. 211 Specifically, we are concerned with: 213 * The functional requirements and service characteristics of an 214 enhanced VPN. 216 * The design of the enhanced data plane. 218 * The necessary control and management protocols in both the 219 underlay and the overlay of the enhanced VPN. 221 * The mechanisms to achieve integration between overlay and 222 underlay. 224 * The necessary Operation, Administration, and Management (OAM) 225 methods to instrument an enhanced VPN to make sure that the 226 required Service Level Agreement (SLA) between the customer and 227 the network operator is met, and to take any corrective action 228 (such as switching traffic to an alternate path) to avoid SLA 229 violation. 231 The required layered network structure to achieve this is shown in 232 Section 4.1. 234 2. Terminology 236 In this document, the relationship of the four terms "VPN", "VPN+", 237 "VTN", and "Network Slice" are as follows: 239 * A Virtual Private Network (VPN) refers to the overlay network 240 service that provides the connectivity between different customer 241 sites, and that maintains traffic separation between different 242 customers. The typical VPN technologies are: IPVPN [RFC2764], 243 L2VPN [RFC4664], L3VPN [RFC4364], and EVPN [RFC7209]. 245 * An enhanced VPN (VPN+) is an evolution of the VPN service that 246 makes additional service-specific commitments. An enhanced VPN is 247 made by integrating an overlay VPN with a set of network resources 248 allocated in the underlay network. 250 * A Virtual Transport Network (VTN) is a virtual underlay network 251 which consists of a set of dedicated or shared network resources 252 allocated from the physical underlay network, and is associated 253 with a customized logical network topology. VTN has the 254 capability to deliver the performance characteristics required by 255 the VPN+ customers and to provide isolation between different VPN+ 256 services. 258 * A network slice could be provided by provisioning an enhanced VPN 259 in the network. Other mechanisms for delivering network slices 260 may exist but are not in scope for this document. 262 The term "tenant" is used in this document to refer to the customers 263 and all of their associated enhanced VPNs. 265 The following terms are also used in this document. Some of them are 266 newly defined, some others reference existing definitions. 268 ACTN: Abstraction and Control of Traffic Engineered Networks 269 [RFC8453] 271 DetNet: Deterministic Networking. See [DETNET] and [RFC8655] 273 FlexE: Flexible Ethernet [FLEXE] 275 TSN: Time Sensitive Networking [TSN] 277 VN: Virtual Network [I-D.ietf-teas-actn-vn-yang] 279 VTP: Virtual Transport Path. A VTP is a path through the VTN which 280 provides the required connectivity and performance between two or 281 more customer sites. 283 3. Overview of the Requirements 285 This section provides an overview of the requirements of an enhanced 286 VPN service. 288 3.1. Performance Guarantees 290 Performance guarantees are made by network operators to their 291 customers in relation to the services provided to the customers. 292 They are usually expressed in SLAs as a set of SLOs. 294 There are several kinds of performance guarantee, including 295 guaranteed maximum packet loss, guaranteed maximum delay, and 296 guaranteed delay variation. Note that these guarantees apply to 297 conformance traffic, out-of-profile traffic will be handled according 298 to a separate agreement with the customer. 300 Guaranteed maximum packet loss is usually addressed by setting packet 301 priorities, queue size, and discard policy. However this becomes 302 more difficult when the requirement is combined with latency 303 requirements. The limiting case is zero congestion loss, and that is 304 the goal of DetNet [DETNET] and TSN [TSN]. In modern optical 305 networks, loss due to transmission errors already approaches zero, 306 but there is the possibility of failure of the interface or the fiber 307 itself. This type of fault can only be addressed by some form of 308 signal duplication and transmission over diverse paths. 310 Guaranteed maximum latency is required by a number of applications 311 particularly real-time control applications and some types of virtual 312 reality applications. DetNet [DETNET] is relevant, however 313 additional methods of enhancing the underlay to better support the 314 delay guarantees may be needed, and these methods will need to be 315 integrated with the overall service provisioning mechanisms. 317 Guaranteed maximum delay variation is a performance guarantee that 318 may also be needed. [RFC8578] calls up a number of cases that need 319 this guarantee, for example in electrical utilities. Time transfer 320 is an example service that needs a performance guarantee, although it 321 is in the nature of time that the service might be delivered by the 322 underlay as a shared service and not provided through different 323 enhanced VPNs. Alternatively, a dedicated enhanced VPN might be used 324 to provide this as a shared service. 326 This suggests that a spectrum of service guarantees need to be 327 considered when deploying an enhanced VPN. As a guide to 328 understanding the design requirements we can consider four types of 329 service: 331 * Best effort 333 * Assured bandwidth 335 * Guaranteed latency 337 * Enhanced delivery 339 The best effort service is the basic service as provided by current 340 VPNs. 342 An assured bandwidth service is one in which the bandwidth over some 343 period of time is assured. This can be achieved either simply based 344 on a best effort service with over-capacity provisioning, or it can 345 be based on MPLS traffic engineered label switching paths (TE-LSPs) 346 with bandwidth reservations. Depending on the technique used, 347 however, the bandwidth is not necessarily assured at any instant. 348 Providing assured bandwidth to VPNs, for example by using per-VPN TE- 349 LSPs, is not widely deployed at least partially due to scalability 350 concerns. VPN+ aims to provide a more scalable approach for such 351 services. 353 A guaranteed latency service has an upper bound to edge-to-edge 354 latency. Assuring the upper bound is sometimes more important than 355 minimizing latency. There are several new technologies that provide 356 some assistance with this performance guarantee. Firstly, the IEEE 357 TSN project [TSN] introduces the concept of scheduling of delay- and 358 loss-sensitive packets. The DetNet work [DETNET] is also of 359 relevance in assuring an upper bound of end-to-end packet latency. 360 FlexE [FLEXE] is also useful to help provide these guarantees. The 361 use of such underlying technologies to deliver VPN+ services needs to 362 be considered. 364 An enhanced delivery service is one in which the underlay network (at 365 Layer 3) attempts to deliver the packet through multiple paths in the 366 hope of eliminating packet loss due to equipment or media failures. 367 Such a mechanism may need to be used for VPN+ service. 369 3.2. Isolation between Enhanced VPN Services 371 One element of the SLA demanded for an enhanced VPN may be a 372 guarantee that the service offered to the customer will not be 373 affected by any other traffic flows in the network. This is termed 374 "isolation" and a customer may express the requirement for isolation 375 as an SLE [I-D.ietf-teas-ietf-network-slices]. 377 One way for a network operator to meet the requirement for isolation 378 is simply by setting and conforming to all the SLOs. For example, 379 traffic congestion (interference from other services) might impact on 380 the latency experienced by a VPN+ customer. Thus, in this example, 381 conformance to a latency SLO would be the primary requirement for 382 delivery of the VPN+ service, and isolation from other services might 383 be only a means to that end. 385 Another way for a service provider to meet this SLE is to control the 386 degree to which traffic from one service is isolated from other 387 services in the network. 389 There is a fine distinction between how isolation is requested by a 390 customer and how it is delivered by the service provider. In 391 general, the customer is interested in service performance and not 392 how it is delivered. Thus, for example, the customer wants specific 393 quality guarantees and is not concerned about how the service 394 provider delivers them. However, it should be noted that some 395 aspects of isolation might be directly measurable by a customer if 396 they have information about the traffic patterns on a number services 397 supported by the same service provider. Furthermore, a customer may 398 be nervous about disruption caused by other services, contamination 399 by other traffic, or delivery of their traffic to the wrong 400 destinations. In this way, the customer may want to specify (and pay 401 for) the level of isolation provided by the service provider. 403 Isolation is achieved in the realization of a VPN+ through existing 404 technologies that may be supplemented by new mechanisms. The service 405 provider chooses which processes to use to meet this SLE just as they 406 choose how to meet all other SLOs and SLEs. Isolation may be 407 achieved in the network by various forms of resource partitioning 408 ranging from simple separation of service traffic on delivery 409 (ensuring that traffic is not delivered to the wrong customer), 410 through sharing of resources with some form of safeguards, to 411 dedicated allocation of resources for a specific enhanced VPN. For 412 example, interference avoidance may be achieved by network capacity 413 planning, allocating dedicated network resources, traffic policing or 414 shaping, prioritizing in using shared network resources, etc. 416 The terms hard and soft isolation are used to indicate different 417 levels of isolation. A service has soft isolation if the traffic of 418 one service cannot be received by the customers of another service. 419 The existing IP and MPLS VPNs are examples of services with soft 420 isolation: the network delivers the traffic only to the required 421 customer endpoints. However, with soft isolation, as the network 422 resources are shared, traffic from some services may congest the 423 network, resulting in packet loss and delay for other services. The 424 ability for a service or a group of services to be sheltered from 425 this effect is called hard isolation. Hard isolation may be needed 426 so that applications with exacting requirements can function 427 correctly, despite other demands (perhaps a burst of traffic in 428 another service) competing for the underlying resources. A customer 429 may request different degrees of isolation ranging from soft 430 isolation to hard isolation. In practice isolation may be delivered 431 on a spectrum between soft and hard, and in some cases soft and hard 432 isolation may be used in a hierarchical manner with one enhanced VPN 433 being built on another. 435 To provide the required level of isolation, resources may need to be 436 reserved in the data plane of the underlay network and dedicated to 437 traffic from a specific enhanced VPN or a specific group of enhanced 438 VPNs. This may introduce scalability concerns both in the 439 implementation (as each enhanced VPN would need to be tracked in the 440 network) and in how many resources need to be reserved and may be 441 under-used (see Section 4.4). Thus, some trade-off needs to be 442 considered to provide the isolation between enhanced VPNs while still 443 allowing reasonable resource sharing. 445 An optical underlay can offer a high degree of isolation, at the cost 446 of allocating resources on a long-term and end-to-end basis. On the 447 other hand, where adequate isolation can be achieved at the packet 448 layer, this permits the resources to be shared amongst a group of 449 services and only dedicated to a service on a temporary basis. 451 The next section explores a pragmatic approach to isolation in packet 452 networks. 454 3.2.1. A Pragmatic Approach to Isolation 456 A key question is whether it is possible to achieve hard isolation in 457 packet networks that were designed to provide statistical 458 multiplexing through sharing of data plane resources, a significant 459 economic advantage when compared to a dedicated, or a Time Division 460 Multiplexing (TDM) network. Clearly, there is no need to provide 461 more isolation than is required by the applications, and an 462 approximation to full hard isolation is sufficient in most cases. 463 For example, pseudowires [RFC3985] emulate services that would have 464 had hard isolation in their native form. 466 O=================================================O 467 | \---------------v---------------/ 468 Statistical Pragmatic Absolute 469 Multiplexing Isolation Isolation 470 (Traditional VPNs) (Enhanced VPN) (Dedicated Network) 472 Figure 1: The Spectrum of Isolation 474 Figure 1 shows a spectrum of isolation that may be delivered by a 475 network. At one end of the spectrum, we see statistical multiplexing 476 technologies that support traditional VPNs. This is a service type 477 that has served the industry well and will continue to do so. At the 478 opposite end of the spectrum, we have the absolute isolation provided 479 by dedicated transport networks. The goal of enhanced VPNs is 480 "pragmatic isolation". This is isolation that is better than what is 481 obtainable from pure statistical multiplexing, more cost effective 482 and flexible than a dedicated network, but is a practical solution 483 that is good enough for the majority of applications. Mechanisms for 484 both soft isolation and hard isolation are needed to meet different 485 levels of service requirement. 487 3.3. Integration 489 The way to achieve the characteristics demanded by an enhanced VPN 490 (such as guaranteed or predictable performance) is by integrating the 491 overlay VPN with a particular set of resources in the underlay 492 network which are allocated to meet the service requirement. This 493 needs be done in a flexible and scalable way so that it can be widely 494 deployed in operators' networks to support a reasonable number of 495 enhanced VPN customers. 497 Taking mobile networks and in particular 5G into consideration, the 498 integration of the network with service functions is likely a 499 requirement. The IETF's work on service function chaining (SFC) 500 [SFC] provides a foundation for this. Service functions can be 501 considered as part of enhanced VPN services. The detailed mechanisms 502 about the integration between service functions and enhanced VPNs are 503 out of the scope of this document. 505 3.3.1. Abstraction 507 Integration of the overlay VPN and the underlay network resources 508 does not always need to be a direct mapping. As described in 509 [RFC7926], abstraction is the process of applying policy to a set of 510 information about a traffic engineered (TE) network to produce 511 selective information that represents the potential ability to 512 connect across the network. The process of abstraction presents the 513 connectivity graph in a way that is independent of the underlying 514 network technologies, capabilities, and topology so that the graph 515 can be used to plan and deliver network services in a uniform way. 517 Virtual networks can be built on top of an abstracted topology that 518 represents the connectivity capabilities of the underlay TE based 519 network as described in the framework for Abstraction and Control of 520 TE Networks (ACTN) [RFC8453] as discussed further in Section 5.5. 521 [I-D.ietf-teas-applicability-actn-slicing] describes the 522 applicability of ACTN to network slicing and is, therefore, relevant 523 to the consideration of using ACTN to enable enhanced VPNs. 525 3.4. Dynamic Changes 527 Enhanced VPNs need to be created, modified, and removed from the 528 network according to service demands. An enhanced VPN that requires 529 hard isolation (Section 3.2) must not be disrupted by the 530 instantiation or modification of another enhanced VPN. Determining 531 whether modification of an enhanced VPN can be disruptive to that 532 VPN, and whether the traffic in flight will be disrupted can be a 533 difficult problem. 535 The data plane aspects of this problem are discussed further in 536 Section 5.1,Section 5.2, and Section 5.3. 538 The control plane aspects of this problem are discussed further in 539 Section 5.4. 541 The management plane aspects of this problem are discussed further in 542 Section 5.5. 544 Dynamic changes both to the enhanced VPN and to the underlay network 545 need to be managed to avoid disruption to services that are sensitive 546 to changes in network performance. 548 In addition to non-disruptively managing the network during changes 549 such as the inclusion of a new VPN endpoint or a change to a link, 550 VPN traffic might need to be moved because of changes to traffic 551 patterns and volumes. 553 3.5. Customized Control 555 In many cases the customers are delivered with enhanced VPN services 556 without knowing the information about the underlying VTNs. However, 557 depends on the agreement between the operator and the customer, in 558 some cases the customer may also be provided with some information 559 about the underlying VTNs. Such information can be filtered or 560 aggregated according to the operator's policy. This allows the 561 customer of the enhanced VPN to have some visibility and even control 562 over how the underlying topology and resources of the VTN are used. 563 For example, the customers may be able to specify the service paths 564 within the VTN for specific traffic flows of their enhanced VPNs. 566 Depending on the requirements, an enhanced VPN customer may have his 567 own network controller, which may be provided with an interface to 568 the control or management system run by the network operator. Note 569 that such control is within the scope of the customer's enhanced VPN, 570 any additional changes beyond this would require some intervention by 571 the network operator. 573 A description of the control plane aspects of this problem are 574 discussed further in Section 5.4. A description of the management 575 plane aspects of this feature can be found in Section 5.5. 577 3.6. Applicability to Overlay Technologies 579 The concept of enhanced VPN can be applied to any existing and future 580 multi-tenancy overlay technologies including but not limited to : 582 * Layer-2 point-to-point services such as pseudowires [RFC3985] 584 * Layer-2 VPNs [RFC4664] 586 * Ethernet VPNs [RFC7209] 588 * Layer-3 VPNs [RFC4364], [RFC2764] 590 Where such VPN service types need enhanced isolation and delivery 591 characteristics, the technologies described in Section 5 can be used 592 to provide an underlay with the required enhanced performance. 594 3.7. Inter-Domain and Inter-Layer Network 596 In some scenarios, an enhanced VPN service may span multiple network 597 domains. A domain is considered to be any collection of network 598 elements within a common realm of address space or path computation 599 responsibility [RFC5151] for example, an Autonomous System. In some 600 domains the network operator may manage a multi-layered network, for 601 example, a packet network over an optical network. When VPN+ 602 services are provisioned in such network scenarios, the technologies 603 used in different network planes (data plane, control plane, and 604 management plane) need to provide mechanisms to support multi-domain 605 and multi-layer coordination and integration, so as to provide the 606 required service characteristics for different enhanced VPNs, and 607 improve network efficiency and operational simplicity. 609 4. Architecture of Enhanced VPNs 611 A number of VPN+ services will typically be provided by a common 612 network infrastructure. Each VPN+ service is provisioned with an 613 overlay VPN and a corresponding VTN, which has a specific set of 614 network resources and functions allocated in the underlay to satisfy 615 the needs of the customer. One VTN may support one of more VPN+ 616 services. The integration between the overlay connectivity and the 617 underlay resources ensures the required isolation between different 618 VPN+ services, and achieves the guaranteed performance for different 619 customers. 621 The VPN+ architecture needs to be designed with consideration given 622 to: 624 * An enhanced data plane. 626 * A control plane to create enhanced VPNs, making use of the data 627 plane isolation and performance guarantee techniques. 629 * A management plane for enhanced VPN service life-cycle management. 631 These topics are expanded below. 633 * The enhanced data plane: 635 - Provides the required packet latency and jitter 636 characteristics. 638 - Provides the required packet loss characteristics. 640 - Provides the required resource isolation capability, e.g., 641 bandwidth guarantee. 643 - Provides the mechanism to associate a packet with the set of 644 resources allocated to a VTN which the VPN+ service packet is 645 mapped to. 647 * The control plane: 649 - Collects information about the underlying network topology and 650 network resources, and exports this to network nodes and/or a 651 centralized controller as required. 653 - Creates VTNs with the network resource and topology properties 654 needed by the VPN+ services. 656 - Distribute the attributes of VTNs to network nodes which 657 participate in the VTNs and/or a centralized controller. 659 - Compute and set up network paths in each VTN. 661 - Map VPN+ services to an appropriate VTN. 663 - Determines the risk of SLA violation and takes appropriate 664 avoiding action. 666 - Consider the right balance of per-packet and per-node state 667 according to the needs of the VPN+ services to scale to the 668 required size. 670 * The management plane: 672 - Provides an interface between the VPN+ service provider (e.g., 673 operator's network management system ) and the VPN+ customer 674 (e.g., an organization or a service with enhanced VPN 675 requirement) such that some of the operation requests can be 676 met without interfering with other VPN+ customers. 678 - Provides an interface between the VPN+ service provider and the 679 VPN+ customers to expose the network capability information 680 toward the customer. 682 - Provides the service life-cycle management and operation of 683 VPN+ services (e.g., creation, modification, assurance/ 684 monitoring, and decommissioning). 686 * Operations, Administration, and Maintenance (OAM) 688 - Provides the tools to verify the connectivity and performance 689 of the VPN+ service. 691 - Provides the tools to verify whether the underlay network 692 resources are correctly allocated and operating properly. 694 * Telemetry 696 - Provides the mechanisms to collect network information about 697 the operation of the data plane, control plane, and management 698 plane. More specifically, telemetry provides the mechanisms to 699 collect network data: 701 o from the underlay network for overall performance evaluation 702 and for the planning VPN+ services. 704 o from each VPN+ service for monitoring and analytics of the 705 characteristics and SLA fulfillment of the VPN+ services. 707 4.1. Layered Architecture 709 The layered architecture of VPN+ is shown in Figure 2. 711 Underpinning everything is the physical network infrastructure layer 712 which provide the underlying resources used to provision the 713 separated VTNs. This layer is responsbile for the partitioning of 714 link and/or node resources for different VTNs. Each subset of link 715 or node resource can be considered as a virtual link or virtual node 716 used to build the VTNs. 718 /\ 719 || 720 +-------------------+ Centralized 721 | Network Controller| Control & Management 722 +-------------------+ 723 || 724 \/ 725 o---------------------------o VPN Service 1 726 /-------------o 727 o____________/______________o VPN Service 2 728 _________________o 729 _____/ 730 o___/ \_________________o VPN Service 3 731 \_______________________o 732 ...... ... 733 o-----------\ /-------------o 734 o____________X______________o VPN Service n 736 __________________________ 737 / o----o-----o / 738 / / / / VTN-1 739 / o-----o-----o----o----o / 740 /_________________________/ 741 __________________________ 742 / o----o / 743 / / / \ / VTN-2 744 / o-----o----o---o------o / 745 /_________________________/ 746 ...... ... 747 ___________________________ 748 / o----o / 749 / / / / VTN-m 750 / o-----o----o----o-----o / 751 /__________________________/ 752 ++++ ++++ ++++ 753 +--+===+--+===+--+ 754 +--+===+--+===+--+ 755 ++++ +++\\ ++++ 756 || || \\ || Physical 757 || || \\ || Network 758 ++++ ++++ ++++ \\+++ ++++ Infrastructure 759 +--+===+--+===+--+===+--+===+--+ 760 +--+===+--+===+--+===+--+===+--+ 761 ++++ ++++ ++++ ++++ ++++ 763 o Virtual Node ++++ 764 +--+ Physical Node with resource partition 765 -- Virtual Link +--+ 766 ++++ 767 == Physical Link with resource partition 769 Figure 2: The Layered Architecture of VPN+ 771 Various components and techniques discussed in Section 5 can be used 772 to enable resource partitioning, such as FlexE, TSN, DetNet, 773 dedicated queues, etc. These partitions may be physical or virtual 774 so long as the SLA required by the higher layers is met. 776 Based on the network resource partitions provided by the physical 777 network infrastructure, multiple VTNs can be created, each with a set 778 of dedicated or shared network resources allocated from the physical 779 underlay network, and is associated with a customized logical network 780 topology, so as to meet the requirements of different VPN+ services 781 or different groups of VPN+ services. According to the associated 782 logical network topology, each VTN needs to be instantiated on a set 783 of network nodes and links which are involved in the logical 784 topology. And on each node or link, each VTN is associated with a 785 set of local resources which are allocated for the processing of 786 traffic in the VTN. The VTN provides the integration between the 787 virtual network topology and the required underlying network 788 resources. 790 According to the service requirements on connectivity, performance 791 and isolation, etc., VPN services can be mapped to the appropriate 792 VTNs in the network. Different VPN services can be mapped to 793 different VTNs, while it is also possible that multiple VPNs are 794 mapped to the same VTN. Thus VTN is an essential scaling technique, 795 as it has the potential of eliminating per-path state from the 796 network. In addition, when a group of VPN+ services are mapped to a 797 single VTN, only the network state of the single VTN needs to be 798 maintained in the network (see Section 4.4 for more information). 800 The centralized network controller is responsible for creating a VTN, 801 instructing the involved network nodes to allocate network resources 802 to the VTN, and provisioning the VPN services on the VTN. A 803 distributed control plane may be used for distributing the VTN 804 resource and topology attributes among nodes in the VTN. 806 The process used to create VTNs and to allocate network resources for 807 use by the VTNs needs to take a holistic view of the needs of all of 808 its customers and to partition the resources accordingly. However, 809 within a VTN these resources can, if required, be managed via a 810 dynamic control plane. This provides the required scalability and 811 isolation with some flexibility. 813 4.2. Multi-Point to Multi-Point (MP2MP) Connectivity 815 At the VPN service level, the required connectivity for an MP2MP VPN 816 service is usually full or partial mesh. To support such VPN 817 services, the corresponding VTN also needs to provide MP2MP 818 connectivity among the end points. 820 Other service requirements may be expressed at different 821 granularities, some of which can be applicable to the whole service, 822 while some others may only be applicable to some pairs of end points. 823 For example, when a particular level of performance guarantee is 824 required, the point-to-point path through the underlying VTN of the 825 VPN+ service may need to be specifically engineered to meet the 826 required performance guarantee. 828 4.3. Application Specific Data Types 830 Although a lot of the traffic that will be carried over VPN+ will 831 likely be IP based, the design must be capable of carrying other 832 traffic types, in particular Ethernet traffic. This is easily 833 accomplished through the various pseudowire (PW) techniques 834 [RFC3985]. Where the underlay is MPLS, Ethernet traffic can be 835 carried over VPN+ encapsulated according to the method specified in 836 [RFC4448]. Where the underlay is IP, Layer Two Tunneling Protocol - 837 Version 3 (L2TPv3) [RFC3931] can be used with Ethernet traffic 838 carried according to [RFC4719]. Encapsulations have been defined for 839 most of the common Layer-2 types for both PW over MPLS and for 840 L2TPv3. 842 4.4. Scaling Considerations 844 VPNs are instantiated as overlays on top of an operator's network and 845 offered as services to the operator's customers. An important 846 feature of overlays is that they can deliver services without placing 847 per-service state in the core of the underlay network. 849 VPN+ may need to install some additional state within the network to 850 achieve the features that they require. Solutions must consider 851 minimizing and controlling the scale of such state, and deployment 852 architectures should constrain the number of VPN+ services so that 853 the additional state introduced to the network is acceptable and 854 under control. It is expected that the number of VPN+ services will 855 be small at the beginning, and even in future the number of VPN+ 856 services will be fewer than traditional VPNs because pre-existing VPN 857 techniques are good enough to meet the needs of most existing VPN- 858 type services. 860 In general, it is not required that the state in the network be 861 maintained in a 1:1 relationship with the VPN+ services. It will 862 usually be possible to aggregate a set or group of VPN+ services so 863 that they share the same VTN and the same set of network resources 864 (much in the same way that current VPNs are aggregated over transport 865 tunnels) so that collections of VPN+ services that require the same 866 behavior from the network in terms of resource reservation, latency 867 bounds, resiliency, etc. can be grouped together. This is an 868 important feature to assist with the scaling characteristics of VPN+ 869 deployments. 871 [I-D.dong-teas-nrp-scalability] provides more details of scalability 872 considerations for the network resource partitions used to 873 instantiate VTNs, and Section 7 includes a greater discussion of 874 scalability considerations. 876 5. Candidate Technologies 878 A VPN is a network created by applying a demultiplexing technique to 879 the underlying network (the underlay) to distinguish the traffic of 880 one VPN from that of another. A VPN path that travels by other than 881 the shortest path through the underlay normally requires state in the 882 underlay to specify that path. State is normally applied to the 883 underlay through the use of the RSVP-TE signaling protocol, or 884 directly through the use of an SDN controller, although other 885 techniques may emerge as this problem is studied. This state gets 886 harder to manage as the number of VPN paths increases. Furthermore, 887 as we increase the coupling between the underlay and the overlay to 888 support the VPN+ service, this state will increase further. Thus, a 889 VPN+ solution needs tighter coupling with the underlay than is the 890 case with existing VPN techniques. We cannot, for example, share the 891 network resource between VPN+ services which require hard isolation. 893 In a VPN+ solution, different subsets of the underlay resources can 894 be dedicated to different VPN+ services or different groups of VPN+ 895 services through the use of VTNs. 897 5.1. Packet Forwarding Plane Technologies 899 Several candidate Layer 2 packet- or frame-based data plane solutions 900 which provide the required isolation and guarantees are described in 901 the following sections. 903 5.1.1. Flexible Ethernet 905 FlexE [FLEXE] provides the ability to multiplex channels over an 906 Ethernet link to create point-to-point fixed- bandwidth connections 907 in a way that provides hard isolation. FlexE also supports bonding 908 links to create larger links out of multiple low capacity links. 910 However, FlexE is only a link level technology. When packets are 911 received by the downstream node, they need to be processed in a way 912 that preserves that isolation in the downstream node. This in turn 913 requires a queuing and forwarding implementation that preserves the 914 end-to-end isolation. 916 If different FlexE channels are used for different services, then no 917 sharing is possible between the FlexE channels. This means that it 918 may be difficult to dynamically redistribute unused bandwidth to 919 lower priority services in another FlexE channel. If one FlexE 920 channel is used by one customer, the customer can use some methods to 921 manage the relative priority of their own traffic in the FlexE 922 channel. 924 5.1.2. Dedicated Queues 926 DiffServ based queuing systems are described in [RFC2475] and 927 [RFC4594]. This approach is not sufficient to provide isolation for 928 VPN+ services because DiffServ does not provide enough markers to 929 differentiate between traffic of a large number of VPN+ services. 930 Nor does DiffServ offer the range of service classes that each VPN+ 931 service needs to provide to its tenants. This problem is 932 particularly acute with an MPLS underlay, because MPLS only provides 933 eight traffic classes. 935 In addition, DiffServ, as currently implemented, mainly provides per- 936 hop priority-based scheduling, and it is difficult to use it to 937 achieve quantitative resource reservation for different VPN+ 938 services. 940 To address these problems and to reduce the potential interference 941 between VPN+ services, it would be necessary to steer traffic to 942 dedicated input and output queues per VPN+ service or per group of 943 VPN+ services: some routers have a large number of queues and 944 sophisticated queuing systems which could support this, while some 945 routers may struggle to provide the granularity and level of 946 isolation required by the applications of VPN+. 948 5.1.3. Time Sensitive Networking 950 Time Sensitive Networking (TSN) [TSN] is an IEEE project to provide a 951 method of carrying time sensitive information over Ethernet. It 952 introduces the concept of packet scheduling where a packet stream may 953 be given a time slot guaranteeing that it experiences no queuing 954 delay or increase in latency beyond the very small scheduling delay. 955 The mechanisms defined in TSN can be used to meet the requirements of 956 time sensitive traffic flows of VPN+ service. 958 Ethernet can be emulated over a Layer 3 network using an IP or MPLS 959 pseudowire. However, a TSN Ethernet payload would be opaque to the 960 underlay and thus not treated specifically as time sensitive data. 961 The preferred method of carrying TSN over a Layer 3 network is 962 through the use of deterministic networking as explained in 963 Section 5.2.1. 965 5.2. Layer Three Data Plane 967 This section considers the problem of VPN+ service differentiation 968 and the representation of underlying network resources in the network 969 layer. More specifically, it describes the possible data plane 970 mechanisms to determine the network resources and the logical network 971 topology or paths associated with a VTN. 973 5.2.1. Deterministic Networking 975 Deterministic Networking (DetNet) [RFC8655] is a technique being 976 developed in the IETF to enhance the ability of Layer-3 networks to 977 deliver packets more reliably and with greater control over the 978 delay. The design cannot use re-transmission techniques such as TCP 979 since that can exceed the delay tolerated by the applications. Even 980 the delay improvements that are achieved with Stream Control 981 Transmission Protocol Partial Reliability Extension (SCTP-PR) 982 [RFC3758] may not meet the bounds set by application demands. DetNet 983 pre-emptively sends copies of the packet over various paths to 984 minimize the chance of all copies of a packet being lost. It also 985 seeks to set an upper bound on latency, but the goal is not to 986 minimize latency. Detnet can be realized over IP data plane 987 [RFC8939] or MPLS data plane [RFC8964], and may be used to provide 988 Virtual Transport Path (VTP) for VPN+ services. 990 5.2.2. MPLS Traffic Engineering (MPLS-TE) 992 MPLS-TE [RFC2702][RFC3209] introduces the concept of reserving end- 993 to-end bandwidth for a TE-LSP, which can be used to provide a point- 994 to-point Virtual Transport Path (VTP) across the underlay network to 995 support VPN services. VPN traffic can be carried over dedicated TE- 996 LSPs to provide reserved bandwidth for each specific connection in a 997 VPN, and VPNs with similar behavior requirements may be multiplexed 998 onto the same TE-LSPs. Some network operators have concerns about 999 the scalability and management overhead of MPLS-TE system, especially 1000 with regard to those systems that use an active control plane, and 1001 this has lead them to consider other solutions for traffic 1002 engineering in their networks. 1004 5.2.3. Segment Routing 1006 Segment Routing (SR) [RFC8402] is a method that prepends instructions 1007 to packets at the head-end of a path. These instructions are used to 1008 specify the nodes and links to be traversed, and allow the packets to 1009 be routed on paths other than the shortest path. By encoding the 1010 state in the packet, per-path state is transitioned out of the 1011 network. 1013 An SR traffic engineered path operates with a granularity of a link. 1014 Hints about priority are provided using the Traffic Class (TC) or 1015 Differentiated Services Code Point (DSCP) field in the packet header. 1016 However, to achieve the performance and isolation characteristics 1017 that are sought by VPN+ customers, it will be necessary to steer 1018 packets through specific virtual links and/or queues on the same link 1019 and direct them to use specific resources. With SR, it is possible 1020 to introduce such fine-grained packet steering by specifying the 1021 queues and the associated resources through an SR instruction list. 1023 Note that the concept of a queue is a useful abstraction for 1024 different types of underlay mechanism that may be used to provide 1025 enhanced isolation and performance support. How the queue satisfies 1026 the requirement is implementation specific and is transparent to the 1027 layer-3 data plane and control plane mechanisms used. 1029 With Segment Routing, the SR instruction list could be used to build 1030 a P2P path, and a group of SR SIDs could also be used to represent an 1031 MP2MP network. Thus, the SR based mechanism could be used to provide 1032 both a Virtual Transport Path (VTP) and a Virtual Transport Network 1033 (VTN) for VPN+ services. 1035 5.3. Non-Packet Data Plane 1037 Non-packet underlay data plane technologies often have TE properties 1038 and behaviors, and meet many of the key requirements in particular 1039 for bandwidth guarantees, traffic isolation (with physical isolation 1040 often being an integral part of the technology), highly predictable 1041 latency and jitter characteristics, measurable loss characteristics, 1042 and ease of identification of flows. The cost is that the resources 1043 are allocated on a long-term and end-to-end basis. Such an 1044 arrangement means that the full cost of the resources has to be borne 1045 by the service that is allocated with the resources. 1047 5.4. Control Plane 1049 The control plane of VPN+ would likely be based on a hybrid control 1050 mechanism that takes advantage of a logically centralized controller 1051 for on-demand provisioning and global optimization, whilst still 1052 relying on a distributed control plane to provide scalability, high 1053 reliability, fast reaction, automatic failure recovery, etc. 1054 Extension to and optimization of the centralized and distributed 1055 control plane is needed to support the enhanced properties of VPN+. 1057 As described in section 4, the VPN+ control plane needs to provide 1058 the following functions: 1060 * Collects information about the underlying network topology and 1061 network resources, and exports this to network nodes and/or a 1062 centralized controller as required. 1064 * Creates VTNs with the network resource and topology properties 1065 needed by the VPN+ services. 1067 * Distribute the attributes of VTNs to network nodes which 1068 participate in the VTNs and/or the centralized controller. 1070 * Compute and set up VTPs in each VTN. 1072 * Map VPN+ services to an appropriate VTN. 1074 The collection of underlying network topology and resource 1075 information can be done using existing the IGP and BGP-LS based 1076 mechanisms. The creation of VTN and the distribution of VTN 1077 attributes may need further control protocol extensions. The 1078 computation of VTPs based on the attributes and constraints of the 1079 VTN can be performed either by the headend node of the path or a 1080 centralized Path Computation Element (PCE). 1082 There are two candidate mechanisms for the setup of VTPs in the VTN: 1083 RSVP-TE and Segment Routing (SR). 1085 * RSVP-TE [RFC3209] provides the signaling mechanism for 1086 establishing a TE-LSP in an MPLS network with end-to-end resource 1087 reservation. This can be seen as an approach of providing a 1088 Virtual Transport Path (VTP) which could be used to bind the VPN 1089 to specific network resources allocated within the underlay, but 1090 there remain scalability concerns as mentioned in Section 5.2.2. 1092 * The SR control plane [RFC8665] [RFC8667] [RFC9085] does not have 1093 the capability of signaling resource reservations along the path. 1094 On the other hand, the SR approach provides a potential way of 1095 binding the underlay network resource and the VTNs without 1096 requiring per-path state to be maintained in the network. A 1097 centralized controller can perform resource planning and 1098 reservation for VTNs, and it needs to instruct the network nodes 1099 to ensure that resources are correctly allocated for the VTN. The 1100 controller could provision the SR paths based on the mechanism in 1101 [I-D.ietf-spring-segment-routing-policy] to the headend nodes of 1102 the paths. 1104 According to the service requirements on connectivity, performance 1105 and isolation, one VPN+ service may be mapped a dedicated VTN, or a 1106 group of VPN+ services may be mapped to the same VTN. The mapping of 1107 VPN+ services to VTN can be achieved using existing control 1108 mechanisms with possible extensions, and it can be based on either 1109 the characteristics of the data packet or the attributes of the VPN 1110 service routes. 1112 5.5. Management Plane 1114 The management plane provides the interface between the VPN+ service 1115 provider and the customers for life-cycle management of the VPN+ 1116 service (i.e., creation, modification, assurance/monitoring, and 1117 decommissioning). It relies on a set of service data models for the 1118 description of the information and operations needed on the 1119 interface. 1121 As an example, in the context of 5G end-to-end network slicing 1122 [TS28530], the management of VPN+ services is considered as the 1123 management of the transport network segment of the 5G end-to-end 1124 network slice. The 3GPP management system may provide the 1125 connectivity and performance related parameters as requirements to 1126 the management plane of the transport network. It may also require 1127 the transport network to expose the capabilities and status of the 1128 network slice. Thus, an interface between the VPN+ management plane 1129 and the 5G network slice management system, and relevant service data 1130 models are needed for the coordination of 5G end-to-end network slice 1131 management. 1133 The management plane interface and data models for VPN+ services can 1134 be based on the service models described in Section 5.6. 1136 It is important that the management life-cycle supports in-place 1137 modification of VPN+ services. That is, it should be possible to add 1138 and remove end points, as well as to change the requested 1139 characteristics of the service that is delivered. The management 1140 system needs to be able to assess the revised VPN+ requests and 1141 determine whether they can be provided by the existing VTNs or 1142 whether changes must be made, and it will additionally need to 1143 determine whether those changes to the VTN are possible. If not, 1144 then the customer's modification request may be rejected. 1146 When the modification of a VPN+ service is possible, the management 1147 system should make every effort to make the changes in a non- 1148 disruptive way. That is, the modification of the VPN+ service or the 1149 underlying VTN should not perturbate traffic on the VPN+ service in a 1150 way that causes the service level to drop below the agreed levels. 1151 Furthermore, in the spirit of isolation, changes to one VPN+ service 1152 should not cause disruption to other VPN+ services. 1154 The network operator for the underlay network (i.e., the provider of 1155 the VPN+ service) may delegate some operational aspects of the 1156 overlay VPN and the underlying VTN to the customer. In this way, the 1157 VPN+ is presented to the customer as a virtual network, and the 1158 customer can choose how to use that network. The customer cannot 1159 exceed the capabilities of the virtual links and nodes, but can 1160 decide how to load traffic onto the network, for example, by 1161 assigning different metrics to the virtual links so that the customer 1162 can control how traffic is routed through the virtual network. This 1163 approach requires a management system for the virtual network, but 1164 does not necessarily require any coordination between the management 1165 systems of the virtual network and the physical network, except that 1166 the virtual network management system might notice when the VTN is 1167 close to capacity or considerably under-used and automatically 1168 request changes in the service provided by the underlay network. 1170 5.6. Applicability of Service Data Models to Enhanced VPN 1172 This section describes the applicability of the existing and in- 1173 progress service data models to VPN+. [RFC8309] describes the the 1174 scope and purpose of service models and shows where a service model 1175 might fit into a SDN based network management architecture. New 1176 service models may also be introduced for some of the required 1177 management functions. 1179 Service data models are used to represent, monitor, and manage the 1180 virtual networks and services enabled by VPN+. The VPN customer 1181 service models (e.g., the layer 3 VPN service model (L3SM) [RFC8299], 1182 the layer 2 VPN service model (L2SM) [RFC8466]), or the ACTN Virtual 1183 Network (VN) model [I-D.ietf-teas-actn-vn-yang]) are service models 1184 which can provide the customer's view of the VPN+ service. The layer 1185 3 VPN network model (L3NM) [I-D.ietf-opsawg-l3sm-l3nm], the layer 2 1186 VPN network model (L2NM) [I-D.ietf-opsawg-l2nm] provide the 1187 operator's view of the managed infrastructure as a set of virtual 1188 networks and the associated resources. The NRP model 1189 [I-D.wd-teas-nrp-yang] further provides the management of the virtual 1190 underlay network topology and resources both in the controller and in 1191 the network devices to instantiate the VTNs needed for the VPN+ 1192 services. 1194 The ACTN framework[RFC8453] supports operators in viewing and 1195 controlling different domains and presenting virtualized networks to 1196 their customers. [I-D.ietf-teas-applicability-actn-slicing] 1197 discusses the applicability of the ACTN approach in the context of 1198 network slicing. Since there is a strong correlation between network 1199 slices and enhanced VPNs, that document also give guidance on how 1200 ACTN can be applied to enhanced VPNs. 1202 6. Applicability to Network Slice Realization 1204 One of the typical use cases of enhanced VPN is to deliver IETF 1205 network slice service. This section describes the applicability of 1206 enhanced VPN to network slice realization. 1208 In order to provide IETF network slices to customers, a technology- 1209 agnostic network slice service Northbound Interface (NBI) data model 1210 [I-D.ietf-teas-ietf-network-slice-nbi-yang] is needed for the 1211 customers to communicate the requirements of IETF network slices (end 1212 points, connectivity, SLOs, and SLEs). These requirements may be 1213 realized using technology specified in this document to instruct the 1214 network to instantiate a VPN+ service to meet the requirements of the 1215 IETF network slice customers. 1217 6.1. VTN Planning 1219 According to the network operators' network resource planning policy, 1220 or based on the requirement of one or a group of customers or 1221 services, a VTN may need to be created. One of the basic 1222 requirements for a VTN is to provide a set of dedicated network 1223 resources to avoid unexpected interference from other services in the 1224 same network. Other possible requirements may include the required 1225 topology and connectivity, bandwidth, latency, reliability, etc. 1227 A centralized network controller can be responsible for calculating a 1228 subset of the underlay network topology (which is called a logical 1229 topology) to support the VTN requirement. And on the network nodes 1230 and links within the logical topology, the set of network resources 1231 to be allocated to the VTN can also be determined by the controller. 1232 Normally such calculation needs to take the underlay network 1233 connectivity information and the available network resource 1234 information of the underlay network into consideration. The network 1235 controller may also take the status of the existing VTNs into 1236 consideration in the planning and calculation of a new VTN. 1238 6.2. VTN Instantiation 1240 According to the result of the VTN planning, the network nodes and 1241 links involved in the logical topology of the VTN are instructed to 1242 allocated the required set of network resources for the VTN. One or 1243 multiple mechanisms as specified in section 5.1 can be used to 1244 partition the forwarding plane network resources and allocate 1245 different subsets of resources to different VTNs. In addition, the 1246 data plane identifiers which are used to identify the set of network 1247 resources allocated to the VTN are also provisioned on the network 1248 nodes. Depends on the data plane technologies used, the set of 1249 network resources of a VTN can be identified using either resource 1250 aware SR segments as specified in 1251 [I-D.ietf-spring-resource-aware-segments], or a dedicated VTN 1252 resource ID as specified in [I-D.dong-6man-enhanced-vpn-vtn-id] can 1253 be introduced. The network nodes involved in a VTN may distribute 1254 the logical topology information, the VTN specific network resource 1255 information and the VTN resource identifiers using the control plane. 1256 Such information could be used by the controller and the network 1257 nodes to compute the TE or shortest paths within the VTN, and install 1258 the VTN specific forwarding entries to network nodes. 1260 6.3. VPN+ Service Provisioning 1262 According to the connectivity requirements of an IETF network slice 1263 service, an overlay VPN can be created using the existing or future 1264 multi-tenancy overlay technologies as described in Section 3.6. 1266 Then according to the SLOs and SLEs requirements of the network 1267 slice, the overlay VPN is mapped to an appropriate VTN as the virtual 1268 underlay. The integration of the overlay VPN and the underlay VTN 1269 together provide an enhanced VPN service which can meet the network 1270 slice service requirements. 1272 6.4. Network Slice Traffic Steering and Forwarding 1274 At the edge of the operator's network, traffic of IETF network slices 1275 can be classified based on the rules defined by operator's policy, so 1276 that the traffic is treated as a specific VPN+ service, which is 1277 further mapped to a underlay VTN. Packets belonging to the VPN+ 1278 service will be processed and forwarded by network nodes based the TE 1279 or shortest path forwarding entries and the set of network resources 1280 of the corresponding VTN. 1282 7. Scalability Considerations 1284 VPN+ provides performance guaranteed services in packet networks, but 1285 with the potential cost of introducing additional state into the 1286 network. There are at least three ways that this additional state 1287 might be brought into the network: 1289 * Introduce the complete state into the packet, as is done in SR. 1290 This allows the controller to specify the detailed series of 1291 forwarding and processing instructions for the packet as it 1292 transits the network. The cost of this is an increase in the 1293 packet header size. The cost is also that systems will have 1294 capabilities enabled in case they are called upon by a service. 1295 This is a type of latent state, and increases as the path and 1296 resources that need to be exclusively available to a VPN are 1297 specified more precisely. 1299 * Introduce the state to the network. This is normally done by 1300 creating a path using signaling such as RSVP-TE. This could be 1301 extended to include any element that needs to be specified along 1302 the path, for example explicitly specifying queuing policy. It is 1303 also possible to use other methods to introduce path state, such 1304 as via an SDN controller, or possibly by modifying a routing 1305 protocol. With this approach there is state per path: per-path 1306 characteristic that needs to be maintained over the life of the 1307 path. This is more network state than is needed using SR, but the 1308 packets are usually shorter. 1310 * Provide a hybrid approach. One example is based on using binding 1311 SIDs [RFC8402] to create path fragments, and bind them together 1312 with SR. Dynamic creation of a VPN service path using SR requires 1313 less state maintenance in the network core at the expense of 1314 larger packet headers. The packet size can be lower if a form of 1315 loose source routing is used (using a few nodal SIDs), and it will 1316 be lower if no specific functions or resources on the routers are 1317 specified. 1319 Reducing the state in the network is important to VPN+, as it 1320 requires the overlay to be more closely integrated with the underlay 1321 than with traditional VPNs. This tighter coupling would normally 1322 mean that more state needs to be created and maintained in the 1323 network, as the state about fine granularity processing would need to 1324 be loaded and maintained in the routers. However, an SR approach 1325 allows much of this state to be spread amongst the network ingress 1326 nodes, and transiently carried in the packets as SIDs. 1328 Further discussion of the scalability considerations of the 1329 underlaying network resource partitions of VPN+ can be found in 1330 [I-D.dong-teas-nrp-scalability]. 1332 7.1. Maximum Stack Depth of SR 1334 One of the challenges with SR is the stack depth that nodes are able 1335 to impose on packets [RFC8491]. This leads to a difficult balance 1336 between adding state to the network and minimizing stack depth, or 1337 minimizing state and increasing the stack depth. 1339 7.2. RSVP-TE Scalability 1341 The traditional method of creating a resource allocated path through 1342 an MPLS network is to use the RSVP-TE protocol. However, there have 1343 been concerns that this requires significant continuous state 1344 maintenance in the network. Work to improve the scalability of RSVP- 1345 TE LSPs in the control plane can be found in [RFC8370]. 1347 There is also concern at the scalability of the forwarder footprint 1348 of RSVP-TE as the number of paths through a label switching router 1349 (LSR) grows. [RFC8577] addresses this by employing SR within a 1350 tunnel established by RSVP-TE. 1352 7.3. SDN Scaling 1354 The centralized approach of SDN requires state to be stored in the 1355 network, but does not have the overhead of also requiring control 1356 plane state to be maintained. Each individual network node may need 1357 to maintain a communication channel with the SDN controller, but that 1358 compares favorably with the need for a control plane to maintain 1359 communication with all neighbors. 1361 However, SDN may transfer some of the scalability concerns from the 1362 network to the centralized controller. In particular, there may be a 1363 heavy processing burden at the controller, and a heavy load in the 1364 network surrounding the controller. A centralized controller also 1365 presents a single point of failure within the network. 1367 8. OAM Considerations 1369 The design of OAM for VPN+ services needs to consider the following 1370 requirements: 1372 * Instrumentation of the underlay so that the network operator can 1373 be sure that the resources committed to a tenant are operating 1374 correctly and delivering the required performance. 1376 * Instrumentation of the overlay by the tenant. This is likely to 1377 be transparent to the network operator and to use existing 1378 methods. Particular consideration needs to be given to the need 1379 to verify the isolation and the various committed performance 1380 characteristics. 1382 * Instrumentation of the overlay by the network provider to 1383 proactively demonstrate that the committed performance is being 1384 delivered. This needs to be done in a non-intrusive manner, 1385 particularly when the tenant is deploying a performance sensitive 1386 application. 1388 * Verification of the conformity of the path to the service 1389 requirement. This may need to be done as part of a commissioning 1390 test. 1392 A study of OAM in SR networks has been documented in [RFC8403]. 1394 9. Telemetry Considerations 1396 Network visibility is essential for network operation. Network 1397 telemetry has been considered as an ideal means to gain sufficient 1398 network visibility with better flexibility, scalability, accuracy, 1399 coverage, and performance than conventional OAM technologies. 1401 As defined in [I-D.ietf-opsawg-ntf], the objective of Network 1402 Telemetry is to acquire network data remotely for network monitoring 1403 and operation. It is a general term for a large set of network 1404 visibility techniques and protocols. Network telemetry addresses the 1405 current network operation issues and enables smooth evolution toward 1406 intent-driven autonomous networks. Telemetry can be applied on the 1407 forwarding plane, the control plane, and the management plane in a 1408 network. 1410 How the telemetry mechanisms could be used or extended for the VPN+ 1411 service is out of the scope of this document. 1413 10. Enhanced Resiliency 1415 Each VPN+ service has a life cycle, and may need modification during 1416 deployment as the needs of its tenant change. This is discussed in 1417 Section 5.5. Additionally, as the network evolves, there may need to 1418 be garbage collection performed to consolidate resources into usable 1419 quanta. 1421 Systems in which the path is imposed, such as SR or some form of 1422 explicit routing, tend to do well in these applications, because it 1423 is possible to perform an atomic transition from one path to another. 1424 That is, a single action by the head-end that changes the path 1425 without the need for coordinated action by the routers along the 1426 path. However, implementations and the monitoring protocols need to 1427 make sure that the new path is operational and meets the required SLA 1428 before traffic is transitioned to it. It is possible for deadlocks 1429 to arise as a result of the network becoming fragmented over time, 1430 such that it is impossible to create a new path or to modify an 1431 existing path without impacting the SLA of other paths. Resolution 1432 of this situation is as much a commercial issue as it is a technical 1433 issue and is outside the scope of this document. 1435 There are, however, two manifestations of the latency problem that 1436 are for further study in any of these approaches: 1438 * The problem of packets overtaking one another if a path latency 1439 reduces during a transition. 1441 * The problem of transient variation in latency in either direction 1442 as a path migrates. 1444 There is also the matter of what happens during failure in the 1445 underlay infrastructure. Fast reroute is one approach, but that 1446 still produces a transient loss with a normal goal of rectifying this 1447 within 50ms [RFC5654]. An alternative is some form of N+1 delivery 1448 such as has been used for many years to support protection from 1449 service disruption. This may be taken to a different level using the 1450 techniques of DetNet with multiple in-network replication and the 1451 culling of later packets [RFC8655]. 1453 In addition to the approach used to protect high priority packets, 1454 consideration should be given to the impact of best effort traffic on 1455 the high priority packets during a transition. Specifically, if a 1456 conventional re-convergence process is used there will inevitably be 1457 micro-loops and whilst some form of explicit routing will protect the 1458 high priority traffic, lower priority traffic on best effort shortest 1459 paths will micro-loop without the use of a loop prevention 1460 technology. To provide the highest quality of service to high 1461 priority traffic, either this traffic must be shielded from the 1462 micro-loops, or micro-loops must be prevented completely. 1464 11. Operational Considerations 1466 It is likely that VPN+ services will be introduced in networks which 1467 already have traditional VPN services deployed. Depending on service 1468 requirements, the tenants or the operator may choose to use a 1469 traditional VPN or an enhanced VPN to fulfill a service requirement. 1470 The information and parameters to assist such a decision needs to be 1471 reflected on the management interface between the tenant and the 1472 operator. 1474 12. Security Considerations 1476 All types of virtual network require special consideration to be 1477 given to the isolation of traffic belonging to different tenants. 1478 That is, traffic belonging to one VPN must not be delivered to end 1479 points outside that VPN. In this regard VPN+ neither introduce, nor 1480 experience a greater security risks than other VPNs. 1482 However, in a VPN+ service the additional service requirements need 1483 to be considered. For example, if a service requires a specific 1484 upper bound to latency then it can be damaged by simply delaying the 1485 packets through the activities of another tenant, i.e., by 1486 introducing bursts of traffic for other services. In some respects 1487 this makes the enhanced VPN more susceptible to attacks since the SLA 1488 may be broken. But another view is that the operator must, in any 1489 case, preform monitoring of the enhanced VPN to ensure that the SLA 1490 is met, and this means that the operator may be more likely to spot 1491 the early onset of a security attack and be able to take pre-emptive 1492 protective action. 1494 The measures to address these dynamic security risks must be 1495 specified as part to the specific solution are form part of the 1496 isolation requirements of a service. 1498 While a VPN+ service may be sold as offering encryption and other 1499 security features as part of the service, customers would be well 1500 advised to take responsibility for their own security requirements 1501 themselves possibly by encrypting traffic before handing it off to 1502 the service provider. 1504 The privacy of VPN+ service customers must be preserved. It should 1505 not be possible for one customer to discover the existence of another 1506 customer, nor should the sites that are members of an VPN+ be 1507 externally visible. 1509 13. IANA Considerations 1511 There are no requested IANA actions. 1513 14. Contributors 1515 Daniel King 1516 Email: daniel@olddog.co.uk 1518 Adrian Farrel 1519 Email: adrian@olddog.co.uk 1521 Jeff Tansura 1522 Email: jefftant.ietf@gmail.com 1524 Zhenbin Li 1525 Email: lizhenbin@huawei.com 1527 Qin Wu 1528 Email: bill.wu@huawei.com 1530 Bo Wu 1531 Email: lana.wubo@huawei.com 1533 Daniele Ceccarelli 1534 Email: daniele.ceccarelli@ericsson.com 1536 Mohamed Boucadair 1537 Email: mohamed.boucadair@orange.com 1539 Sergio Belotti 1540 Email: sergio.belotti@nokia.com 1542 Haomian Zheng 1543 Email: zhenghaomian@huawei.com 1545 15. Acknowledgements 1547 The authors would like to thank Charlie Perkins, James N Guichard, 1548 John E Drake, Shunsuke Homma, and Luis M. Contreras for their review 1549 and valuable comments. 1551 This work was supported in part by the European Commission funded 1552 H2020-ICT-2016-2 METRO-HAUL project (G.A. 761727). 1554 16. References 1556 16.1. Normative References 1558 [RFC2764] Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. 1559 Malis, "A Framework for IP Based Virtual Private 1560 Networks", RFC 2764, DOI 10.17487/RFC2764, February 2000, 1561 . 1563 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1564 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1565 DOI 10.17487/RFC3985, March 2005, 1566 . 1568 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 1569 2 Virtual Private Networks (L2VPNs)", RFC 4664, 1570 DOI 10.17487/RFC4664, September 2006, 1571 . 1573 16.2. Informative References 1575 [DETNET] "Deterministic Networking", March , 1576 . 1578 [FLEXE] "Flex Ethernet Implementation Agreement", March 2016, 1579 . 1582 [I-D.dong-6man-enhanced-vpn-vtn-id] 1583 Dong, J., Li, Z., Xie, C., Ma, C., and G. Mishra, 1584 "Carrying Virtual Transport Network (VTN) Identifier in 1585 IPv6 Extension Header", Work in Progress, Internet-Draft, 1586 draft-dong-6man-enhanced-vpn-vtn-id-06, 24 October 2021, 1587 . 1590 [I-D.dong-teas-nrp-scalability] 1591 Dong, J., Li, Z., Gong, L., Yang, G., Guichard, J. N., 1592 Mishra, G., Qin, F., Saad, T., and V. P. Beeram, 1593 "Scalability Considerations for Network Resource 1594 Partition", Work in Progress, Internet-Draft, draft-dong- 1595 teas-nrp-scalability-01, 7 February 2022, 1596 . 1599 [I-D.ietf-opsawg-l2nm] 1600 Barguil, S., Dios, O. G. D., Boucadair, M., and L. A. 1601 Munoz, "A Layer 2 VPN Network YANG Model", Work in 1602 Progress, Internet-Draft, draft-ietf-opsawg-l2nm-12, 22 1603 November 2021, . 1606 [I-D.ietf-opsawg-l3sm-l3nm] 1607 Barguil, S., Dios, O. G. D., Boucadair, M., Munoz, L. A., 1608 and A. Aguado, "A YANG Network Data Model for Layer 3 1609 VPNs", Work in Progress, Internet-Draft, draft-ietf- 1610 opsawg-l3sm-l3nm-18, 8 October 2021, 1611 . 1614 [I-D.ietf-opsawg-ntf] 1615 Song, H., Qin, F., Martinez-Julia, P., Ciavaglia, L., and 1616 A. Wang, "Network Telemetry Framework", Work in Progress, 1617 Internet-Draft, draft-ietf-opsawg-ntf-13, 3 December 2021, 1618 . 1621 [I-D.ietf-spring-resource-aware-segments] 1622 Dong, J., Bryant, S., Miyasaka, T., Zhu, Y., Qin, F., Li, 1623 Z., and F. Clad, "Introducing Resource Awareness to SR 1624 Segments", Work in Progress, Internet-Draft, draft-ietf- 1625 spring-resource-aware-segments-03, 12 July 2021, 1626 . 1629 [I-D.ietf-spring-segment-routing-policy] 1630 Filsfils, C., Talaulikar, K., Voyer, D., Bogdanov, A., and 1631 P. Mattes, "Segment Routing Policy Architecture", Work in 1632 Progress, Internet-Draft, draft-ietf-spring-segment- 1633 routing-policy-18, 17 February 2022, 1634 . 1637 [I-D.ietf-teas-actn-vn-yang] 1638 Lee, Y., Dhody, D., Ceccarelli, D., Bryskin, I., and B. Y. 1639 Yoon, "A YANG Data Model for VN Operation", Work in 1640 Progress, Internet-Draft, draft-ietf-teas-actn-vn-yang-13, 1641 23 October 2021, . 1644 [I-D.ietf-teas-actn-yang] 1645 Lee, Y., Zheng, H., Ceccarelli, D., Yoon, B. Y., and S. 1646 Belotti, "Applicability of YANG models for Abstraction and 1647 Control of Traffic Engineered Networks", Work in Progress, 1648 Internet-Draft, draft-ietf-teas-actn-yang-08, 8 September 1649 2021, . 1652 [I-D.ietf-teas-applicability-actn-slicing] 1653 King, D., Drake, J., Zheng, H., and A. Farrel, 1654 "Applicability of Abstraction and Control of Traffic 1655 Engineered Networks (ACTN) to Network Slicing", Work in 1656 Progress, Internet-Draft, draft-ietf-teas-applicability- 1657 actn-slicing-00, 21 September 2021, 1658 . 1661 [I-D.ietf-teas-ietf-network-slice-nbi-yang] 1662 Wu, B., Dhody, D., Rokui, R., Saad, T., and L. Han, "IETF 1663 Network Slice Service YANG Model", Work in Progress, 1664 Internet-Draft, draft-ietf-teas-ietf-network-slice-nbi- 1665 yang-01, 4 March 2022, . 1668 [I-D.ietf-teas-ietf-network-slices] 1669 Farrel, A., Drake, J., Rokui, R., Homma, S., Makhijani, 1670 K., Contreras, L. M., and J. Tantsura, "Framework for IETF 1671 Network Slices", Work in Progress, Internet-Draft, draft- 1672 ietf-teas-ietf-network-slices-08, 6 March 2022, 1673 . 1676 [I-D.wd-teas-nrp-yang] 1677 Wu, B., Dhody, D., and Y. Cheng, "A YANG Data Model for 1678 Network Resource Partition (NRP)", Work in Progress, 1679 Internet-Draft, draft-wd-teas-nrp-yang-00, 30 January 1680 2022, . 1683 [NGMN-NS-Concept] 1684 hao ,, "NGMN NS Concept", 2016, 1685 . 1688 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1689 and W. Weiss, "An Architecture for Differentiated 1690 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1691 . 1693 [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and J. 1694 McManus, "Requirements for Traffic Engineering Over MPLS", 1695 RFC 2702, DOI 10.17487/RFC2702, September 1999, 1696 . 1698 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1699 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1700 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1701 . 1703 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1704 Conrad, "Stream Control Transmission Protocol (SCTP) 1705 Partial Reliability Extension", RFC 3758, 1706 DOI 10.17487/RFC3758, May 2004, 1707 . 1709 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1710 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 1711 RFC 3931, DOI 10.17487/RFC3931, March 2005, 1712 . 1714 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1715 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1716 2006, . 1718 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 1719 "Encapsulation Methods for Transport of Ethernet over MPLS 1720 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 1721 . 1723 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 1724 Guidelines for DiffServ Service Classes", RFC 4594, 1725 DOI 10.17487/RFC4594, August 2006, 1726 . 1728 [RFC4719] Aggarwal, R., Ed., Townsley, M., Ed., and M. Dos Santos, 1729 Ed., "Transport of Ethernet Frames over Layer 2 Tunneling 1730 Protocol Version 3 (L2TPv3)", RFC 4719, 1731 DOI 10.17487/RFC4719, November 2006, 1732 . 1734 [RFC5151] Farrel, A., Ed., Ayyangar, A., and JP. Vasseur, "Inter- 1735 Domain MPLS and GMPLS Traffic Engineering -- Resource 1736 Reservation Protocol-Traffic Engineering (RSVP-TE) 1737 Extensions", RFC 5151, DOI 10.17487/RFC5151, February 1738 2008, . 1740 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1741 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1742 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1743 September 2009, . 1745 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 1746 Networking: A Perspective from within a Service Provider 1747 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 1748 . 1750 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 1751 Henderickx, W., and A. Isaac, "Requirements for Ethernet 1752 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 1753 . 1755 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 1756 Ceccarelli, D., and X. Zhang, "Problem Statement and 1757 Architecture for Information Exchange between 1758 Interconnected Traffic-Engineered Networks", BCP 206, 1759 RFC 7926, DOI 10.17487/RFC7926, July 2016, 1760 . 1762 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 1763 Network Functions and Their Infrastructure", RFC 8172, 1764 DOI 10.17487/RFC8172, July 2017, 1765 . 1767 [RFC8299] Wu, Q., Ed., Litkowski, S., Tomotaki, L., and K. Ogaki, 1768 "YANG Data Model for L3VPN Service Delivery", RFC 8299, 1769 DOI 10.17487/RFC8299, January 2018, 1770 . 1772 [RFC8309] Wu, Q., Liu, W., and A. Farrel, "Service Models 1773 Explained", RFC 8309, DOI 10.17487/RFC8309, January 2018, 1774 . 1776 [RFC8370] Beeram, V., Ed., Minei, I., Shakir, R., Pacella, D., and 1777 T. Saad, "Techniques to Improve the Scalability of RSVP-TE 1778 Deployments", RFC 8370, DOI 10.17487/RFC8370, May 2018, 1779 . 1781 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1782 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1783 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1784 July 2018, . 1786 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1787 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1788 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1789 2018, . 1791 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1792 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1793 DOI 10.17487/RFC8453, August 2018, 1794 . 1796 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1797 Data Model for Layer 2 Virtual Private Network (L2VPN) 1798 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1799 2018, . 1801 [RFC8491] Tantsura, J., Chunduri, U., Aldrin, S., and L. Ginsberg, 1802 "Signaling Maximum SID Depth (MSD) Using IS-IS", RFC 8491, 1803 DOI 10.17487/RFC8491, November 2018, 1804 . 1806 [RFC8568] Bernardos, CJ., Rahman, A., Zuniga, JC., Contreras, LM., 1807 Aranda, P., and P. Lynch, "Network Virtualization Research 1808 Challenges", RFC 8568, DOI 10.17487/RFC8568, April 2019, 1809 . 1811 [RFC8577] Sitaraman, H., Beeram, V., Parikh, T., and T. Saad, 1812 "Signaling RSVP-TE Tunnels on a Shared MPLS Forwarding 1813 Plane", RFC 8577, DOI 10.17487/RFC8577, April 2019, 1814 . 1816 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 1817 RFC 8578, DOI 10.17487/RFC8578, May 2019, 1818 . 1820 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1821 "Deterministic Networking Architecture", RFC 8655, 1822 DOI 10.17487/RFC8655, October 2019, 1823 . 1825 [RFC8665] Psenak, P., Ed., Previdi, S., Ed., Filsfils, C., Gredler, 1826 H., Shakir, R., Henderickx, W., and J. Tantsura, "OSPF 1827 Extensions for Segment Routing", RFC 8665, 1828 DOI 10.17487/RFC8665, December 2019, 1829 . 1831 [RFC8667] Previdi, S., Ed., Ginsberg, L., Ed., Filsfils, C., 1832 Bashandy, A., Gredler, H., and B. Decraene, "IS-IS 1833 Extensions for Segment Routing", RFC 8667, 1834 DOI 10.17487/RFC8667, December 2019, 1835 . 1837 [RFC8939] Varga, B., Ed., Farkas, J., Berger, L., Fedyk, D., and S. 1838 Bryant, "Deterministic Networking (DetNet) Data Plane: 1839 IP", RFC 8939, DOI 10.17487/RFC8939, November 2020, 1840 . 1842 [RFC8964] Varga, B., Ed., Farkas, J., Berger, L., Malis, A., Bryant, 1843 S., and J. Korhonen, "Deterministic Networking (DetNet) 1844 Data Plane: MPLS", RFC 8964, DOI 10.17487/RFC8964, January 1845 2021, . 1847 [RFC9085] Previdi, S., Talaulikar, K., Ed., Filsfils, C., Gredler, 1848 H., and M. Chen, "Border Gateway Protocol - Link State 1849 (BGP-LS) Extensions for Segment Routing", RFC 9085, 1850 DOI 10.17487/RFC9085, August 2021, 1851 . 1853 [SFC] "Service Function Chaining", March , 1854 . 1856 [TS23501] "3GPP TS23.501", 2016, 1857 . 1860 [TS28530] "3GPP TS28.530", 2016, 1861 . 1864 [TSN] "Time-Sensitive Networking", March , 1865 . 1867 Authors' Addresses 1869 Jie Dong 1870 Huawei 1871 Email: jie.dong@huawei.com 1873 Stewart Bryant 1874 University of Surrey 1875 Email: stewart.bryant@gmail.com 1876 Zhenqiang Li 1877 China Mobile 1878 Email: lizhenqiang@chinamobile.com 1880 Takuya Miyasaka 1881 KDDI Corporation 1882 Email: ta-miyasaka@kddi.com 1884 Young Lee 1885 Samsung 1886 Email: younglee.tx@gmail.com