idnits 2.17.1 draft-ietf-teas-enhanced-vpn-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 13, 2020) is 1377 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-18) exists of draft-ietf-idr-bgp-ls-segment-routing-ext-16 == Outdated reference: A later version (-19) exists of draft-ietf-opsawg-l2nm-00 == Outdated reference: A later version (-18) exists of draft-ietf-opsawg-l3sm-l3nm-03 == Outdated reference: A later version (-13) exists of draft-ietf-opsawg-ntf-03 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-policy-08 == Outdated reference: A later version (-24) exists of draft-ietf-teas-actn-vn-yang-08 == Outdated reference: A later version (-11) exists of draft-ietf-teas-actn-yang-05 Summary: 0 errors (**), 0 flaws (~~), 8 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group J. Dong 3 Internet-Draft Huawei 4 Intended status: Informational S. Bryant 5 Expires: January 14, 2021 Futurewei 6 Z. Li 7 China Mobile 8 T. Miyasaka 9 KDDI Corporation 10 Y. Lee 11 Samsung 12 July 13, 2020 14 A Framework for Enhanced Virtual Private Networks (VPN+) Service 15 draft-ietf-teas-enhanced-vpn-06 17 Abstract 19 This document describes the framework for Enhanced Virtual Private 20 Network (VPN+) service. The purpose is to support the needs of new 21 applications, particularly applications that are associated with 5G 22 services, by utilizing an approach that is based on existing VPN and 23 Traffic Engineering (TE) technologies and adds features that specific 24 services require over and above traditional VPNs. 26 Typically, VPN+ will be used to form the underpinning of network 27 slicing, but could also be of use in its own right providing enhanced 28 connectivity services between customer sites. 30 It is envisaged that enhanced VPNs will be delivered using a 31 combination of existing, modified, and new networking technologies. 32 This document provides an overview of relevant technologies and 33 identifies some areas for potential new work. 35 Comparing to traditional VPNs, It is not envisaged that quite large 36 numbers of VPN+ services will be deployed in a network. In other 37 word, it is not intended that all existing VPNs supported by a 38 network will use VPN+ related techniques. 40 Status of This Memo 42 This Internet-Draft is submitted in full conformance with the 43 provisions of BCP 78 and BCP 79. 45 Internet-Drafts are working documents of the Internet Engineering 46 Task Force (IETF). Note that other groups may also distribute 47 working documents as Internet-Drafts. The list of current Internet- 48 Drafts is at https://datatracker.ietf.org/drafts/current/. 50 Internet-Drafts are draft documents valid for a maximum of six months 51 and may be updated, replaced, or obsoleted by other documents at any 52 time. It is inappropriate to use Internet-Drafts as reference 53 material or to cite them other than as "work in progress." 55 This Internet-Draft will expire on January 14, 2021. 57 Copyright Notice 59 Copyright (c) 2020 IETF Trust and the persons identified as the 60 document authors. All rights reserved. 62 This document is subject to BCP 78 and the IETF Trust's Legal 63 Provisions Relating to IETF Documents 64 (https://trustee.ietf.org/license-info) in effect on the date of 65 publication of this document. Please review these documents 66 carefully, as they describe your rights and restrictions with respect 67 to this document. Code Components extracted from this document must 68 include Simplified BSD License text as described in Section 4.e of 69 the Trust Legal Provisions and are provided without warranty as 70 described in the Simplified BSD License. 72 Table of Contents 74 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 75 2. Terminologies . . . . . . . . . . . . . . . . . . . . . . . . 5 76 3. Overview of the Requirements . . . . . . . . . . . . . . . . 6 77 3.1. Isolation between Enhanced VPN Services . . . . . . . . . 6 78 3.1.1. A Pragmatic Approach to Isolation . . . . . . . . . . 8 79 3.2. Performance Guarantee . . . . . . . . . . . . . . . . . . 9 80 3.3. Integration . . . . . . . . . . . . . . . . . . . . . . . 10 81 3.3.1. Abstraction . . . . . . . . . . . . . . . . . . . . . 11 82 3.4. Dynamic Management . . . . . . . . . . . . . . . . . . . 11 83 3.5. Customized Control . . . . . . . . . . . . . . . . . . . 12 84 3.6. Applicability . . . . . . . . . . . . . . . . . . . . . . 12 85 3.7. Inter-Domain and Inter-Layer Network . . . . . . . . . . 12 86 4. Architecture of Enhanced VPN . . . . . . . . . . . . . . . . 13 87 4.1. Layered Architecture . . . . . . . . . . . . . . . . . . 14 88 4.2. Multi-Point to Multi-Point (MP2MP) Connectivity . . . . . 17 89 4.3. Application Specific Network Types . . . . . . . . . . . 17 90 4.4. Scaling Considerations . . . . . . . . . . . . . . . . . 17 91 5. Candidate Technologies . . . . . . . . . . . . . . . . . . . 18 92 5.1. Layer-Two Data Plane . . . . . . . . . . . . . . . . . . 18 93 5.1.1. Flexible Ethernet . . . . . . . . . . . . . . . . . . 18 94 5.1.2. Dedicated Queues . . . . . . . . . . . . . . . . . . 19 95 5.1.3. Time Sensitive Networking . . . . . . . . . . . . . . 19 96 5.2. Layer-Three Data Plane . . . . . . . . . . . . . . . . . 20 97 5.2.1. Deterministic Networking . . . . . . . . . . . . . . 20 98 5.2.2. MPLS Traffic Engineering (MPLS-TE) . . . . . . . . . 20 99 5.2.3. Segment Routing . . . . . . . . . . . . . . . . . . . 20 100 5.3. Non-Packet Data Plane . . . . . . . . . . . . . . . . . . 21 101 5.4. Control Plane . . . . . . . . . . . . . . . . . . . . . . 21 102 5.5. Management Plane . . . . . . . . . . . . . . . . . . . . 22 103 5.6. Applicability of Service Data Models to Enhanced VPN . . 22 104 5.6.1. Network Slice Delivery via Coordinated Service Data 105 Models . . . . . . . . . . . . . . . . . . . . . . . 23 106 6. Scalability Considerations . . . . . . . . . . . . . . . . . 24 107 6.1. Maximum Stack Depth of SR . . . . . . . . . . . . . . . . 25 108 6.2. RSVP Scalability . . . . . . . . . . . . . . . . . . . . 25 109 6.3. SDN Scaling . . . . . . . . . . . . . . . . . . . . . . . 25 110 7. OAM Considerations . . . . . . . . . . . . . . . . . . . . . 25 111 8. Telemetry Considerations . . . . . . . . . . . . . . . . . . 26 112 9. Enhanced Resiliency . . . . . . . . . . . . . . . . . . . . . 26 113 10. Operational Considerations . . . . . . . . . . . . . . . . . 27 114 11. Security Considerations . . . . . . . . . . . . . . . . . . . 27 115 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28 116 13. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 28 117 14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 29 118 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 29 119 15.1. Normative References . . . . . . . . . . . . . . . . . . 29 120 15.2. Informative References . . . . . . . . . . . . . . . . . 30 121 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35 123 1. Introduction 125 Virtual private networks (VPNs) have served the industry well as a 126 means of providing different groups of users with logically isolated 127 connectivity over a common network. The common or base network that 128 is used to provide the VPNs is often referred to as the underlay, and 129 the VPN is often called an overlay. 131 Customers of a network operator may request a connectivity services 132 with advanced characteristics such as enhanced isolation from other 133 services so that changes in some other service (such as changes in 134 network load, or events such as congestion or outages) have no or 135 acceptable effect on the throughput or latency of the services 136 provided to the customer. These services are "enhanced VPNs" (known 137 as VPN+) in that they are similar to VPN services as they provide the 138 customer with required connectivity, but have enhanced 139 characteristics. 141 Driven largely by needs surfacing from 5G, the concept of network 142 slicing has gained traction [NGMN-NS-Concept] [TS23501] [TS28530] 143 [BBF-SD406]. According to [TS28530], a 5G end-to-end network slice 144 consists of three major types network segments: Radio Access Network 145 (RAN), Transport Network (TN) and Mobile Core Network (CN). The 146 transport network provides the required connectivity between 147 different entities in RAN and CN segments of an end-to-end network 148 slice, with specific performance commitment. 150 A transport network slice is a virtual (logical) network with a 151 particular network topology and a set of shared or dedicated network 152 resources, which are used to provide the network slice consumer with 153 the required connectivity, appropriate isolation and specific Service 154 Level Objective (SLO). 156 A transport network slice could span multiple technologies (such as 157 IP or Optical) and multiple administrative domains. Depending on the 158 consumer's requirement, a transport network slice could be isolated 159 from other, often concurrent transport network slices in terms of 160 data plane, control plane, and management plane resources. 162 In this document the term "network slice" refers to a transport 163 network slice, and is considered as one typical use case of enhanced 164 VPN. 166 Network slicing builds on the concept of resource management, network 167 virtualization, and abstraction to provide performance assurance, 168 flexibility, programmability and modularity. It may use techniques 169 such as Software Defined Networking (SDN) [RFC7149], network 170 abstraction [RFC7926] and Network Function Virtualization (NFV) 171 [RFC8172] [RFC8568] to create multiple logical (virtual) networks, 172 each tailored for a set of services or a particular tenant or a group 173 of tenants that share the same or similar set of requirements, on top 174 of a common network. How the network slices are engineered can be 175 deployment-specific. 177 VPN+ could be used to form the underpinning of transport network 178 slice, but could also be of use in general cases providing enhanced 179 connectivity services between customer sites. 181 The requirement of enhanced VPN services cannot be met by simple 182 overlay networks, as they require tighter coordination and 183 integration between the underlay and the overlay network. VPN+ is 184 built from a VPN overlay and a underlying Virtual Transport Network 185 (VTN) which has a customized network topology and a set of dedicated 186 or shared network resources. It may optionally include a set of 187 invoked service functions allocated from the underlay network. Thus 188 an enhanced VPN can achieve greater isolation with strict performance 189 guarantees. These new properties, which have general applicability, 190 may also be of interest as part of a network slicing solution. It is 191 not envisaged that VPN+ services will replace traditional VPN 192 services that can continue to be deployed using pre- existing 193 mechanisms. 195 This document specifies a framework for using existing, modified, and 196 potential new technologies as components to provide a VPN+ service. 197 Specifically we are concerned with: 199 o The design of the enhanced data plane. 201 o The necessary protocols in both the underlay and the overlay of 202 the enhanced VPN. 204 o The mechanisms to achieve integration between overlay and 205 underlay. 207 o The necessary Operation, Administration, and Management (OAM) 208 methods to instrument an enhanced VPN to make sure that the 209 required Service Level Agreement (SLA) is met, and to take any 210 corrective action to avoid SLA violation, such as switching to an 211 alternate path. 213 The required layered network structure to achieve this is shown in 214 Section 4.1. 216 Note that, in this document, the relationship of the four terms 217 "VPN", "VPN+", "VTN", and "Transport Network Slice" are described as 218 below: 220 o A VPN refers to the overlay virtual private network which provides 221 the required service connectivity and traffic separation between 222 different VPN customers. 224 o A Virtual Transport Network (VTN) is a virtual underlay network 225 that connects customer edge points with the additional capability 226 of providing the isolation and performance characteristics 227 required by an enhanced VPN customer. 229 o An enhanced VPN (VPN+) can be considered as an evolution of VPN 230 service, but with additional service-specific commitments. An 231 enhanced VPN (VPN+) is made by integrating an overlay VPN and a 232 VTN with a set of network resources allocated in the underlay 233 network. 235 o A transport network slice could be provided with an enhanced VPN 236 (VPN+). 238 2. Terminologies 240 The following terms are used in this document. Some of them are 241 newly defined, some others reference existing definitions: 243 ACTN: Abstraction and Control of TE Networks [RFC8453] 245 Detnet: Deterministic Networking [DETNET] 247 FlexE: Flexible Ethernet [FLEXE] 249 TSN: Time Sensitive Networking [TSN] 251 VN: Virtual Network [I-D.ietf-teas-actn-vn-yang] 253 VPN: Virtual Private Network. IPVPN is defined in [RFC2764], L2VPN 254 is defined in [RFC4664]. 256 VPN+: Enhanced VPN service. An enhanced VPN service (VPN+) can be 257 considered as an evolution of VPN service, but with additional 258 service-specific commitments such as enhanced isolation and 259 performance guarantee. 261 VTP: Virtual Transport Path. A VTP is a virtual underlay path which 262 connects two customer edge points with the capability of providing 263 the isolation and performance characteristics required by an enhanced 264 VPN customer. A VTP usually has a customized path with a set of 265 reserved network resources along the path. 267 VTN: Virtual Transport Network. A VTN is a virtual underlay network 268 that connects customer edge points with the capability of providing 269 the isolation and performance characteristics required by an enhanced 270 VPN customer. A VTN usually has a customized topology and a set of 271 dedicated or shared network resources. 273 3. Overview of the Requirements 275 In this section we provide an overview of the requirements of an 276 enhanced VPN service. 278 3.1. Isolation between Enhanced VPN Services 280 One element of the SLA demanded for an enhanced VPN is a guarantee 281 that the service offered to the customer will not be perturbed by any 282 other traffic flows in the network. One way for a service provider 283 to guarantee the customer's SLA is by controlling the degree of 284 isolation from other services in the network. Isolation is a feature 285 that can be requested by customers. There are different grades of 286 how isolation may be enabled by a network operator and that may 287 result in different levels of service perceived by the customer. 288 These range from simple separation of service traffic on delivery 289 (ensuring that traffic is not delivered to the wrong customer), all 290 the way to complete separation within the underlay so that the 291 traffic from different services use distinct network resources. 293 The terms hard and soft isolation are used to illustrate different 294 levels of isolation. A VPN has soft isolation if the traffic of one 295 VPN cannot be received by the customers of another VPN. Both IP and 296 MPLS VPNs are examples of VPNs with soft isolation: the network 297 delivers the traffic only to the required VPN endpoints. However, 298 with soft isolation, traffic from VPNs and regular non-VPN traffic 299 may congest the network resulting in packet loss and delay for other 300 VPNs operating normally. The ability for a VPN service or a group of 301 VPN services to be sheltered from this effect is called hard 302 isolation, and this property is required by some applications. Hard 303 isolation is needed so that applications with exacting requirements 304 can function correctly, despite other demands (perhaps a burst of 305 traffic in another VPN) competing for the underlying resources. In 306 practice isolation may be offered as a spectrum between soft and 307 hard, and in some cases soft and hard isolation may be used in a 308 hierarchical manner. An operator may offer its customers a choice of 309 different degrees of isolation ranging from soft isolation up to hard 310 isolation. 312 An example of the requirement for hard isolation is a network 313 supporting both emergency services and public broadband multi-media 314 services. During a major incident the VPNs supporting these services 315 would both be expected to experience high data volumes, and it is 316 important that both make progress in the transmission of their data. 317 In these circumstances the VPN services would require an appropriate 318 degree of isolation to be able to continue to operate acceptably. On 319 the other hand, VPNs servicing ordinary bulk data may expect to 320 contest for network resources and queue packets so that traffic is 321 delivered within SLAs, but with some potential delays and 322 interference. 324 In order to provide the required level of isolation, resources may 325 have to be reserved in the data plane of the underlay network and 326 dedicated to traffic from a specific VPN or a specific group of VPNs 327 to form different enhanced VPNs in the network. This may introduce 328 scalability concerns, thus some trade-off needs to be considered to 329 provide the required isolation between some enhanced VPNs while still 330 allowing reasonable sharing. 332 An optical layer can offer a high degree of isolation, at the cost of 333 allocating resources on a long term and end-to-end basis. On the 334 other hand, where adequate isolation can be achieved at the packet 335 layer, this permits the resources to be shared amongst a group of 336 services and only dedicated to a service on a temporary basis. 338 There are several new technologies that provide some assistance with 339 these data plane issues. Firstly there is the IEEE project on Time 340 Sensitive Networking [TSN] which introduces the concept of packet 341 scheduling of delay and loss sensitive packets. Then there is 342 [FLEXE] which provides the ability to multiplex multiple channels 343 over one or more Ethernet links in a way that provides hard 344 isolation. Finally there are advanced queueing approaches which 345 allow the construction of virtual sub-interfaces, each of which is 346 provided with dedicated resource in a shared physical interface. 347 These approaches are described in more detail later in this document. 349 Section 3.1.1 explores pragmatic approaches to isolation in packet 350 networks. 352 3.1.1. A Pragmatic Approach to Isolation 354 A key question is whether it is possible to achieve hard isolation in 355 packet networks that were never designed to support hard isolation. 356 On the contrary, they were designed to provide statistical 357 multiplexing, a significant economic advantage when compared to a 358 dedicated, or a Time Division Multiplexing (TDM) network. However, 359 there is no need to provide any harder isolation than is required by 360 the applications. An approximation to this requirement is sufficient 361 in most cases. Pseudowires[RFC3985] emulate services that would have 362 had hard isolation in their native form. 364 This spectrum of isolation is shown in Figure 1: 366 O=================================================O 367 | \---------------v---------------/ 368 Statistical Pragmatic Absolute 369 Multiplexing Isolation Isolation 370 (Traditional VPNs) (Enhanced VPN) (Dedicated Network) 372 Figure 1: The Spectrum of Isolation 374 Figure 1 shows the spectrum of isolation that may be delivered by a 375 network. At one end of the figure, we have traditional statistical 376 multiplexing technologies that support VPNs. This is a service type 377 that has served the industry well and will continue to do so. At the 378 opposite end of the spectrum, we have the absolute isolation provided 379 by dedicated transport networks. The goal of enhanced VPNs is 380 "pragmatic isolation". This is isolation that is better than is 381 obtainable from pure statistical multiplexing, more cost effective 382 and flexible than a dedicated network, but is a practical solution 383 that is good enough for the majority of applications. Mechanisms for 384 both soft isolation and hard isolation would be needed to meet 385 different levels of service requirement. 387 3.2. Performance Guarantee 389 There are several kinds of performance guarantee, including 390 guaranteed maximum packet loss, guaranteed maximum delay, and 391 guaranteed delay variation. Note that these guarantees apply to 392 conformance traffic, out-of-profile traffic will be handled according 393 to other requirements. 395 Guaranteed maximum packet loss is a common parameter, and is usually 396 addressed by setting packet priorities, queue size, and discard 397 policy. However this becomes more difficult when the requirement is 398 combined with latency requirements. The limiting case is zero 399 congestion loss, and that is the goal of the Deterministic Networking 400 work that the IETF [DETNET] and IEEE [TSN] are pursuing. In modern 401 optical networks, loss due to transmission errors already approaches 402 zero, but there are the possibilities of failure of the interface or 403 the fiber itself. This can only be addressed by some form of signal 404 duplication and transmission over diverse paths. 406 Guaranteed maximum latency is required in a number of applications 407 particularly real-time control applications and some types of virtual 408 reality applications. The work of the IETF Deterministic Networking 409 (DetNet) Working Group [DETNET] is relevant, however additional 410 methods of enhancing the underlay to better support the delay 411 guarantees may be needed, and these methods will need to be 412 integrated with the overall service provisioning mechanisms. 414 Guaranteed maximum delay variation is a service that may also be 415 needed. [RFC8578] calls up a number of cases where this is needed, 416 for example in electrical utilities. Time transfer is one example of 417 a service that needs this, although it is in the nature of time that 418 the service might be delivered by the underlay as a shared service 419 and not provided through different enhanced VPNs. Alternatively a 420 dedicated enhanced VPN may be used to provide this as a shared 421 service. 423 This suggests that a spectrum of service guarantee be considered when 424 deploying an enhanced VPN. As a guide to understanding the design 425 requirements we can consider four types: 427 o Best effort 429 o Assured bandwidth 431 o Guaranteed latency 432 o Enhanced delivery 434 Best effort service is the basic service that current VPNs can 435 provide. 437 An assured bandwidth service is one in which the bandwidth over some 438 period of time is assured. This can be achieved either simply based 439 on best effort with over-capacity provisioning, or it can be based on 440 TE-LSPs with bandwidth reservation. The instantaneous bandwidth is 441 however, not necessarily assured, depending on the technique used. 442 Providing assured bandwidth to VPNs, for example by using per-VPN TE- 443 LSPs, is not widely deployed at least partially due to scalability 444 concerns. VPN+ aims to provide a more scalable approach for such 445 kind of service. 447 A guaranteed latency service has a latency upper bound provided by 448 the network. Assuring the upper bound is sometimes more important 449 than minimizing latency. There are several new technologies that 450 provide some assistance with performance guarantee. Firstly there is 451 the IEEE project on Time Sensitive Networking [TSN] which introduces 452 the concept of packet scheduling of delay and loss sensitive packets. 453 Then the DetNet work is also of greater relevance in assuring upper 454 bound of end-to-end packet latency. Flex Ethernet [FLEXE] is also 455 useful to provide these guarantees. The usage of such underlying 456 technologies for VPN+ service needs to be considered. 458 An enhanced delivery service is one in which the underlay network (at 459 Layer 3) attempts to deliver the packet through multiple paths in the 460 hope of eliminating packet loss due to equipment or media failures. 461 Such mechanism may need to be used for VPN+ service. 463 3.3. Integration 465 The only way to achieve the enhanced characteristics provided by an 466 enhanced VPN (such as guaranteed or predicted performance) is by 467 integrating the overlay VPN with a particular set of network 468 resources in the underlay network which are allocated to meet the 469 service requirement. This needs be done in a flexible and scalable 470 way so that it can be widely deployed in operator networks to support 471 a reasonable number of enhanced VPN customers. 473 Taking mobile networks and in particular 5G into consideration, the 474 integration of network and the service functions is a likely 475 requirement. The work in IETF SFC working group [SFC] provides a 476 foundation for this integration. 478 3.3.1. Abstraction 480 Integration of the overlay VPN and the underlay network resources 481 does not need to be a tight mapping. As described in [RFC7926], 482 abstraction is the process of applying policy to a set of information 483 about a TE network to produce selective information that represents 484 the potential ability to connect across the network. The process of 485 abstraction presents the connectivity graph in a way that is 486 independent of the underlying network technologies, capabilities, and 487 topology so that the graph can be used to plan and deliver network 488 services in a uniform way. 490 Virtual networks can be built on top of an abstracted topology that 491 represents the connectivity capabilities of the underlay network as 492 described in the framework for Abstraction and Control of TE Networks 493 (ACTN) [RFC8453] as discussed further in Section 5.5. 495 3.4. Dynamic Management 497 Enhanced VPNs need to be created, modified, and removed from the 498 network according to service demand. An enhanced VPN that requires 499 hard isolation (Section 3.1) must not be disrupted by the 500 instantiation or modification of another enhanced VPN. Determining 501 whether modification of an enhanced VPN can be disruptive to that 502 VPN, and in particular whether the traffic in flight will be 503 disrupted can be a difficult problem. 505 The data plane aspects of this problem are discussed further in 506 Sections Section 5.1,Section 5.2 and Section 5.3. 508 The control plane aspects of this problem are discussed further in 509 Section 5.4. 511 The management plane aspects of this problem are discussed further in 512 Section 5.5. 514 Dynamic changes both to the VPN and to the underlay transport network 515 need to be managed to avoid disruption to services that are sensitive 516 to the change of network performance. 518 In addition to non-disruptively managing the network as a result of 519 gross change such as the inclusion of a new VPN endpoint or a change 520 to a link, VPN traffic might need to be moved as a result of traffic 521 volume changes. 523 3.5. Customized Control 525 In some cases it is desirable that an enhanced VPN has a customized 526 control plane, so that the tenant of the enhanced VPN can have some 527 control of how the resources and functions allocated to this enhanced 528 VPN are used. For example, the tenant may be able to specify the 529 service paths in his own enhanced VPN. Depending on the requirement, 530 an enhanced VPN may have its own dedicated controller, which may be 531 provided with an interface to the control system provided by the 532 network operator. Note that such control is within the scope of the 533 tenant's enhanced VPN, any change beyond that would require some 534 intervention of the operator. 536 A description of the control plane aspects of this problem are 537 discussed further in Section 5.4. A description of the management 538 plane aspects of this feature can be found in Section 5.5. 540 3.6. Applicability 542 The technologies described in this document should be applicable to a 543 number types of VPN overlay services such as: 545 o Layer 2 point-to-point services such as pseudowires [RFC3985] 547 o Layer 2 VPNs [RFC4664] 549 o Ethernet VPNs [RFC7209] 551 o Layer 3 VPNs [RFC4364], [RFC2764] 553 Where such VPN types need enhanced isolation and delivery 554 characteristics, the technologies described in Section 5 can be used 555 to provide an underlay with the required enhanced performance. 557 3.7. Inter-Domain and Inter-Layer Network 559 In some scenarios, an enhanced VPN services may span multiple network 560 domains. A domain is considered to be any collection of network 561 elements within a common realm of address space or path computation 562 responsibility [RFC5151]. In some domains the operator may manage a 563 multi-layered network, for example, a packet network over an optical 564 network. When enhanced VPNs are provisioned in such network 565 scenarios, the technologies used in different network planes (data 566 plane, control plane, and management plane) need to provide 567 mechanisms to support multi-domain and multi-layer coordination and 568 integration, so as to provide the required service characteristics 569 for different enhanced VPNs, and improve network efficiency and 570 operational simplicity. 572 4. Architecture of Enhanced VPN 574 A number of enhanced VPN services will typically be provided by a 575 common network infrastructure. Each enhanced VPN consists of both 576 the overlay and a corresponding VTN with a specific set of network 577 resources and functions allocated in the underlay to satisfy the 578 needs of the VPN tenant. The integration between overlay and various 579 underlay resources ensures the required isolation between different 580 enhanced VPNs, and achieves the guaranteed performance for different 581 services. 583 An enhanced VPN needs to be designed with consideration given to: 585 o A enhanced data plane 587 o A control plane to create enhanced VPNs, making use of the data 588 plane isolation and performance guarantee techniques. 590 o A management plane for enhanced VPN service life-cycle management. 592 These required characteristics are expanded below: 594 o Enhanced data plane 596 * Provides the required resource isolation capability, e.g. 597 bandwidth guarantee. 599 * Provides the required packet latency and jitter 600 characteristics. 602 * Provides the required packet loss characteristics. 604 * Provides the mechanism to associate a packet with the set of 605 resources allocated to the enhanced VPN which the packet 606 belongs. 608 o Control plane 610 * Collect information about the underlying network topology and 611 resources available and export this to nodes in the network 612 and/or the centralized controller as required. 614 * Create the required virtual transport networks (VTNs) with the 615 resource and properties needed by the enhanced VPN services 616 that are assigned to them. 618 * Determine the risk of SLA violation and take appropriate 619 avoiding action. 621 * Determine the right balance of per-packet and per-node state 622 according to the needs of enhanced VPN service to scale to the 623 required size. 625 o Management plane 627 * Provides an interface between the enhanced VPN provider (e.g. 628 the Transport Network (TN) Manager) and the enhanced VPN 629 clients (e.g. the 3GPP Management System) such that some of the 630 operation requests can be met without interfering with the 631 enhanced VPN of other clients. 633 * Provides an interface between the enhanced VPN provider and the 634 enhanced VPN clients to expose transport network capability 635 information toward the enhanced VPN client. 637 * Provides the service life-cycle management and operation of 638 enhanced VPN (e.g. creation, modification, assurance/monitoring 639 and decommissioning). 641 o Operations, Administration, and Maintenance (OAM) 643 * Provides the OAM tools to verify the connectivity and 644 performance of the enhanced VPN. 646 * Provide the OAM tools to verify whether the underlay network 647 resources are correctly allocated and operated properly. 649 o Telemetry 651 * Provides the mechanism to collect the data plane, control plane 652 and management plane data of the network, more specifically: 654 * 656 + Provides the mechanism to collect network data from the 657 underlay network for overall performance evaluation and the 658 enhanced VPN service planning. 660 + Provides the mechanism to collect network data of each 661 enhanced VPN for the monitoring and analytics of the 662 characteristics and SLA fulfilment of enhanced VPN services. 664 4.1. Layered Architecture 666 The layered architecture of an enhanced VPN is shown in Figure 2. 668 Underpinning everything is the physical network infrastructure layer 669 which provide the underlying resources used to provision the 670 separated virtual transport networks (VTNs). This includes the 671 partitioning of link and/or node resources. Each subset of link or 672 node resource can be considered as a virtual link or virtual node 673 used to build the VTNs. 675 A 676 | | 677 +-------------------+ Centralized 678 | Network Controller| Control & Management 679 +-------------------+ 680 || 681 \/ 682 o---------------------------o 683 /-------------o 684 o____________/______________o VPN Services 685 ...... (P2P,P2MP,MP2MP...) 686 o-----------\ /-------------o 687 o____________X______________o 689 __________________________ 690 / o----o----o / 691 / / / / VTN-1 692 / o-----o-----o----o----o / 693 /_________________________/ 694 __________________________ 695 / o----o / 696 / / / \ / VTN-2 697 / o-----o----o----o-----o / 698 /_________________________/ 699 ...... ... 700 ___________________________ 701 / o----o / 702 / / / / VTN-3 703 / o-----o----o----o-----o / 704 /__________________________/ 706 ++++ ++++ ++++ 707 +--+===+--+===+--+ 708 +--+===+--+===+--+ 709 ++++ +++\\ ++++ Physical 710 || || \\ || 711 || || \\ || Network 712 ++++ ++++ ++++ \\+++ ++++ 713 +--+===+--+===+--+===+--+===+--+ Infrastructure 714 +--+===+--+===+--+===+--+===+--+ 715 ++++ ++++ ++++ ++++ ++++ 717 O Virtual Node 719 -- Virtual Link 721 ++++ 722 +--+ Physical Node with resource partition 723 +--+ 724 ++++ 726 == Physical Link with resource partition 728 Figure 2: The Layered Architecture of VPN+ 730 Various components and techniques discussed in Section 5 can be used 731 to enable resource partition, such as FlexE, Time Sensitive 732 Networking, Deterministic Networking, Dedicated queues, etc. These 733 partitions may be physical, or virtual so long as the SLA required by 734 the higher layers is met. 736 Based on the network resources provided by the physical network 737 infrastructure, multiple VTNs can be provisioned, each with 738 customized topology and other attributes to meet the requirement of 739 different enhanced VPNs or different groups of enhanced VPNs. To get 740 the required characteristic, each VTN needs to be mapped to a set of 741 network nodes and links in the network infrastructure. And on each 742 node or link, the VTN is associated with a set of resources which are 743 allocated for the processing of traffic in the VTN. VTN provides the 744 integration between the virtual network topology and the required 745 underlying network resources. 747 The centralized controller is used to create the VTN, and to instruct 748 the network nodes to allocate the required resources to each VTN and 749 to provision the enhanced VPN services on the VTNs. A distributed 750 control plane may also be used for the distribution of the VTN 751 topology and attribute information between nodes within the VTNs. 753 The process used to create VTNs and to allocate network resources for 754 use by VTNs needs to take a holistic view of the needs of all of its 755 tenants (i.e., of all customers and their associated VTNs), and to 756 partition the resources accordingly. However, within a VTN these 757 resources can, if required, be managed via a dynamic control plane. 758 This provides the required scalability and isolation. 760 4.2. Multi-Point to Multi-Point (MP2MP) Connectivity 762 At the VPN service level, the required connectivity is usually mesh 763 or partial-mesh. To support such kinds of VPN service, the 764 corresponding VTN in underlay is also an abstract MP2MP medium. 765 Other service requirements may be expressed at different granularity, 766 some of which can be applicable to the whole service, while some 767 others may be only applicable to some pairs of end points. For 768 example, when particular level of performance guarantee is required, 769 the point-to-point path through the underlay of the enhanced VPN may 770 need to be specifically engineered to meet the required performance 771 guarantee. 773 4.3. Application Specific Network Types 775 Although a lot of the traffic that will be carried over the enhanced 776 VPN will likely be IPv4 or IPv6, the design has to be capable of 777 carrying other traffic types, in particular Ethernet traffic. This 778 is easily accomplished through the various pseudowire (PW) techniques 779 [RFC3985]. Where the underlay is MPLS, Ethernet can be carried over 780 the enhanced VPN encapsulated according to the method specified in 781 [RFC4448]. Where the underlay is IP, Layer Two Tunneling Protocol - 782 Version 3 (L2TPv3) [RFC3931] can be used with Ethernet traffic 783 carried according to [RFC4719]. Encapsulations have been defined for 784 most of the common Layer 2 types for both PW over MPLS and for 785 L2TPv3. 787 4.4. Scaling Considerations 789 VPNs are instantiated as overlays on top of an operator's network and 790 offered as services to the operator's customers. An important 791 feature of overlays is that they are able to deliver services without 792 placing per-service state in the core of the underlay network. 794 Enhanced VPNs may need to install some additional state within the 795 network to achieve the additional features that they require. 796 Solutions must consider minimizing and controlling the scale of such 797 state, and deployment architectures should constrain the number of 798 enhanced VPNs that would exist where such services would place 799 additional state in the network. It is expected that the number of 800 enhanced VPN would be small in the beginning, and even in future the 801 number of enhanced VPN will be much fewer than traditional VPNs, 802 because pre-existing VPN techniques are be good enough to meet the 803 needs of most existing VPN-type services. 805 In general, it is not required that the state in the network be 806 maintained in a 1:1 relationship with the VPN+ services. It will 807 usually be possible to aggregate a set of VPN+ services so that they 808 share the same VTN and the same set of network resources (much in the 809 way that current VPNs are aggregated over transport tunnels) so that 810 collections of enhanced VPNs that require the same behaviour from the 811 network in terms of resource reservation, latency bounds, resiliency, 812 etc. are able to be grouped together. This is an important feature 813 to assist with the scaling characteristics of VPN+ deployments. 815 See Section 6 for a greater discussion of scalability considerations. 817 5. Candidate Technologies 819 A VPN is a network created by applying a demultiplexing technique to 820 the underlying network (the underlay) in order to distinguish the 821 traffic of one VPN from that of another. A VPN path that travels by 822 other than the shortest path through the underlay normally requires 823 state in the underlay to specify that path. State is normally 824 applied to the underlay through the use of the RSVP signaling 825 protocol, or directly through the use of an SDN controller, although 826 other techniques may emerge as this problem is studied. This state 827 gets harder to manage as the number of VPN paths increases. 828 Furthermore, as we increase the coupling between the underlay and the 829 overlay to support the enhanced VPN service, this state will increase 830 further. 832 In an enhanced VPN different subsets of the underlay resources can be 833 dedicated to different enhanced VPNs or different groups of enhanced 834 VPNs. An enhanced VPN solution thus needs tighter coupling with 835 underlay than is the case with existing VPNs. We cannot, for 836 example, share the network resource between enhanced VPNs which 837 require hard isolation. 839 5.1. Layer-Two Data Plane 841 A number of candidate Layer 2 packet or frame-based data plane 842 solutions which can be used provide the required isolation and 843 guarantees are described in following sections. 845 5.1.1. Flexible Ethernet 847 FlexE [FLEXE] provides the ability to multiplex channels over an 848 Ethernet link to create point-to-point fixed-bandwidth connections in 849 a way that provides hard isolation. FlexE also supports bonding 850 links to create larger links out of multiple low capacity links. 852 However, FlexE is only a link level technology. When packets are 853 received by the downstream node, they need to be processed in a way 854 that preserves that isolation in the downstream node. This in turn 855 requires a queuing and forwarding implementation that preserves the 856 end-to-end isolation. 858 If different FlexE channels are used for different services, then no 859 sharing is possible between the FlexE channels. This means that it 860 may be difficult to dynamically redistribute unused bandwidth to 861 lower priority services in another FlexE channel. If one FlexE 862 channel is used by one tenant, the tenant can use some methods to 863 manage the relative priority of his own traffic in the FlexE channel. 865 5.1.2. Dedicated Queues 867 DiffServ based queuing systems are described in [RFC2475] and 868 [RFC4594]. This is considered insufficient to provide isolation for 869 enhanced VPNs because DiffServ does not always provide enough markers 870 to differentiate between traffic of many enhanced VPNs, or offer the 871 range of service classes that each VPN needs to provide to its 872 tenants. This problem is particularly acute with an MPLS underlay, 873 because MPLS only provides eight Traffic Classes. 875 In addition, DiffServ, as currently implemented, mainly provides per- 876 hop priority-based scheduling, and it is difficult to use it to 877 achieve quantitive resource reservation. 879 In order to address these problems and to reduce the potential 880 interference between enhanced VPNs, it would be necessary to steer 881 traffic to dedicated input and output queues per enhanced VPN: some 882 routers have a large number of queues and sophisticated queuing 883 systems, which could support this, while some routers may struggle to 884 provide the granularity and level of isolation required by the 885 applications of enhanced VPN. 887 5.1.3. Time Sensitive Networking 889 Time Sensitive Networking (TSN) [TSN] is an IEEE project that is 890 designing a method of carrying time sensitive information over 891 Ethernet. It introduces the concept of packet scheduling where a 892 packet stream may be given a time slot guaranteeing that it 893 experiences no queuing delay or increase in latency. The mechanisms 894 defined in TSN can be used to meet the requirements of time sensitive 895 services of an enhanced VPN. 897 Ethernet can be emulated over a Layer 3 network using an IP or MPLS 898 pseudowire. However, a TSN Ethernet payload would be opaque to the 899 underlay and thus not treated specifically as time sensitive data. 900 The preferred method of carrying TSN over a Layer 3 network is 901 through the use of deterministic networking as explained in 902 Section 5.2.1. 904 5.2. Layer-Three Data Plane 906 We now consider the problem of slice differentiation and resource 907 representation in the network layer. 909 5.2.1. Deterministic Networking 911 Deterministic Networking (DetNet) [RFC8655] is a technique being 912 developed in the IETF to enhance the ability of Layer 3 networks to 913 deliver packets more reliably and with greater control over the 914 delay. The design cannot use re-transmission techniques such as TCP 915 since that can exceed the delay tolerated by the applications. Even 916 the delay improvements that are achieved with Stream Control 917 Transmission Protocol Partial Reliability Extension (SCTP-PR) 918 [RFC3758] may not meet the bounds set by application demands. DetNet 919 pre-emptively sends copies of the packet over various paths to 920 minimize the chance of all copies of a packet being lost. It also 921 seeks to set an upper bound on latency, but the goal is not to 922 minimize latency. 924 5.2.2. MPLS Traffic Engineering (MPLS-TE) 926 MPLS-TE [RFC2702][RFC3209] introduces the concept of reserving end- 927 to-end bandwidth for a TE-LSP, which can be used to provide point- 928 to-point Virtual Transport Path (VTP) across the underlay network to 929 support VPNs. VPN traffic can be carried over dedicated TE-LSPs to 930 provide reserved bandwidth for each specific connection in a VPN, and 931 VPNs with similar behaviour requirements may be multiplexed onto the 932 same TE-LSPs. Some network operators have concerns about the 933 scalability and management overhead of MPLS-TE system, and this has 934 lead them to consider other solutions for their networks. 936 5.2.3. Segment Routing 938 Segment Routing (SR) [RFC8402] is a method that prepends instructions 939 to packets at the head-end of a path. These instructions are used to 940 specify the nodes and links to be traversed and allow the packets to 941 be routed on paths other than the shortest path. By encoding the 942 state in the packet, per-path state is transitioned out of the 943 network. 945 An SR traffic engineered path operates with a granularity of a link 946 with hints about priority provided through the use of the traffic 947 class (TC) or Differentiated Services Code Point (DSCP) field in the 948 header. However to achieve the latency and isolation characteristics 949 that are sought by the enhanced VPN users, steering packets through 950 specific queues and resources will likely be required. With SR, it 951 is possible to introduce such fine-grained packet steering by 952 specifying the queues and resources through an SR instruction list. 954 Note that the concept of queue is a useful abstraction for different 955 types of underlay mechanism that may be used to provide enhanced 956 isolation and latency support. How the queue satisfies the 957 requirement is implementation specific and is transparent to the 958 layer-3 data plane and control plane mechanisms used. 960 With Segment Routing, the SR instruction list could be used to build 961 a P2P path, a group of SR SIDs could also be used to represent a 962 MP2MP network. Thus the SR based mechanism could be used to provide 963 both Virtual Transport Path (VTP) and Virtual Transport Network (VTN) 964 for enhanced VPN services. 966 5.3. Non-Packet Data Plane 968 Non-packet underlay data plane technologies often have TE properties 969 and behaviours, and meet many of the key requirements in particular 970 for bandwidth guarantees, traffic isolation (with physical isolation 971 often being an integral part of the technology), highly predictable 972 latency and jitter characteristics, measurable loss characteristics, 973 and ease of identification of flows. The cost is the resources are 974 allocated on a long term and end-to-end basis. Such an arrangement 975 means that the full cost of the resources has be borne by the service 976 that is allocated with the resources. 978 5.4. Control Plane 980 Enhanced VPN would likely be based on a hybrid control mechanism, 981 which takes advantage of the logically centralized controller for on- 982 demand provisioning and global optimization, whilst still relying on 983 a distributed control plane to provide scalability, high reliability, 984 fast reaction, automatic failure recovery, etc. Extension to and 985 optimization of the distributed control plane is needed to support 986 the enhanced properties of VPN+. 988 RSVP-TE [RFC3209] provides the signaling mechanism for establishing a 989 TE-LSP in an MPLS network with end-to-end resource reservation. This 990 can be seen as an approach of providing Virtual Transport Path (VTP), 991 which could be used to bind the VPN to specific network resources 992 allocated within the underlay, but there remain scalability concerns 993 mentioned in Section 5.2.2. 995 The control plane of SR [RFC8665] [RFC8667] 996 [I-D.ietf-idr-bgp-ls-segment-routing-ext] does not have the 997 capability of signaling resource reservations along the path. On the 998 other hand, the SR approach provides a potential way of binding the 999 underlay network resource and the enhanced VPN service without 1000 requiring per-path state to be maintained in the network. A 1001 centralized controller can perform resource planning and reservation 1002 for enhanced VPNs, while it needs to ensure that resources are 1003 correctly allocated in network nodes for the enhanced VPN service. 1004 The controller could also compute the SR paths based on the planned 1005 or collected network resource and other attributes, and provision the 1006 SR paths based on the mechanism in 1007 [I-D.ietf-spring-segment-routing-policy] to the ingress nodes of the 1008 enhanced VPN services. The distributed control plane may be used to 1009 advertise the network attributes associated with enhanced VPNs, and 1010 compute the SR paths with specific constraints of enhanced VPN 1011 services. 1013 5.5. Management Plane 1015 The management plane provides the interface between the enhanced VPN 1016 provider and the clients for the service life-cycle management (e.g. 1017 creation, modification, assurance/monitoring and decommissioning). 1018 It relies on a set of service data models for the description of the 1019 information and operations needed on the interface. 1021 As an example, in the context of 5G end-to-end network slicing 1022 [TS28530], the management of enhanced VPNs is considered as the 1023 management of the transport network part of the end-to-end network 1024 slice. 3GPP management system may provide the connectivity and 1025 performance related parameters as requirements to the management 1026 plane of the transport network. It may also require the transport 1027 network to expose the capability and status of the transport network 1028 slice. Thus, an interface between the enhanced VPN management plane 1029 and the 3GPP network slice management system, and relevant service 1030 data models are needed for the coordination of end-to-end network 1031 slice management. 1033 The management plane interface and data models for enhanced VPN can 1034 be based on the service models described in Section 5.6 1036 5.6. Applicability of Service Data Models to Enhanced VPN 1038 ACTN supports operators in viewing and controlling different domains 1039 and presenting virtualized networks to their customers. The ACTN 1040 framework [RFC8453] highlights how: 1042 o Abstraction of the underlying network resources is provided to 1043 higher-layer applications and customers. 1045 o Underlying resources are virtualized and allocated for the 1046 customer, application, or service. 1048 o A virtualized environment is created allowing operators to view 1049 and control multi-domain networks as a single virtualized network. 1051 o Networks can be presented to customers as a virtual network via 1052 open and programmable interfaces. 1054 The type of network virtualization enabled by ACTN managed 1055 infrastructure provides customers and applications (tenants) with the 1056 capability to utilize and independently control allocated virtual 1057 network resources as if they were physically their own resources. 1058 Service Data models are used to represent, monitor, and manage the 1059 virtual networks and services enabled by ACTN. The Customer VPN 1060 model (e.g. L3SM [RFC8299], L2SM [RFC8466]) or an ACTN Virtual 1061 Network (VN) [I-D.ietf-teas-actn-vn-yang] model is a customer view of 1062 the ACTN managed infrastructure, and is presented by the ACTN 1063 provider as a set of abstracted services or resources. The L3VPN 1064 network model [I-D.ietf-opsawg-l3sm-l3nm] and [I-D.ietf-opsawg-l2nm] 1065 provide a network view of the ACTN managed infrastructure presented 1066 by the ACTN provider as a set of transport resources. 1068 5.6.1. Network Slice Delivery via Coordinated Service Data Models 1070 In order to support network slice service in transport network, a 1071 Transport Slice (TS) Northbound Interface (NBI) data model may be 1072 needed for a consumer to express the requirements for transport 1073 slices, which can be technology-agnostic. Then these requirements 1074 may be realized using technology-specific Southbound Interface (SBI). 1076 As per [RFC8453] and [I-D.ietf-teas-actn-yang], the CNC-MDSC 1077 Interface (CMI) of ACTN is used to convey the virtual network service 1078 requirements, which is a generic interface to deliver various TE 1079 based VN services. In the context of network slice northbound 1080 interface, there may be some gaps in L3SM/L2SM or VN model, or the 1081 combination of them. The TS NBI is required to communicate the 1082 connectivity of the transport slice, along with the service level 1083 objective (SLO) parameters and traffic selection rules, and provides 1084 a way to monitor the state of the transport slice. This can be 1085 described in a more abstracted manner, so as to reduce the 1086 association with specific realization technologies of transport 1087 network slice, such as the VPN and TE technologies. The transport 1088 slice model as defined in [I-D.wd-teas-transport-slice-yang] provides 1089 an abstracted and generic approach to meet the transport slice NBI 1090 requirement. 1092 The MDSC-PNC Interface (MPI) models in the ACTN architecture can be 1093 used for the realization of transport slices, for example, in a TE 1094 enabled transport network, and may also be used for cross-layer or 1095 cross-domain implementation of transport slice. 1097 6. Scalability Considerations 1099 Enhanced VPN provides performance guaranteed services in packet 1100 networks, but with the potential cost of introducing additional 1101 states into the network. There are at least three ways that this 1102 additional state might be presented in the network: 1104 o Introduce the complete state into the packet, as is done in SR. 1105 This allows the controller to specify the detailed series of 1106 forwarding and processing instructions for the packet as it 1107 transits the network. The cost of this is an increase in the 1108 packet header size. The cost is also that systems will have 1109 capabilities enabled in case they are called upon by a service. 1110 This is a type of latent state, and increases as we more precisely 1111 specify the path and resources that need to be exclusively 1112 available to a VPN. 1114 o Introduce the state to the network. This is normally done by 1115 creating a path using RSVP-TE, which can be extended to introduce 1116 any element that needs to be specified along the path, for example 1117 explicitly specifying queuing policy. It is possible to use other 1118 methods to introduce path state, such as via a Software Defined 1119 Network (SDN) controller, or possibly by modifying a routing 1120 protocol. With this approach there is state per path, per path 1121 characteristic that needs to be maintained over its life-cycle. 1122 This is more state than is needed using SR, but the packets are 1123 shorter. 1125 o Provide a hybrid approach. One example is based on using binding 1126 SIDs [RFC8402] to create path fragments, and bind them together 1127 with SR. Dynamic creation of a VPN service path using SR requires 1128 less state maintenance in the network core at the expense of 1129 larger packet headers. The packet size can be lower if a form of 1130 loose source routing is used (using a few nodal SIDs), and it will 1131 be lower if no specific functions or resources on the routers are 1132 specified. 1134 Reducing the state in the network is important to enhanced VPN, as it 1135 requires the overlay to be more closely integrated with the underlay 1136 than with traditional VPNs. This tighter coupling would normally 1137 mean that more state needed to be created and maintained in the 1138 network, as the state about fine granularity processing would need to 1139 be loaded and maintained in the routers. However, a segment routed 1140 approach allows much of this state to be spread amongst the network 1141 ingress nodes, and transiently carried in the packets as SIDs. 1143 6.1. Maximum Stack Depth of SR 1145 One of the challenges with SR is the stack depth that nodes are able 1146 to impose on packets [RFC8491]. This leads to a difficult balance 1147 between adding state to the network and minimizing stack depth, or 1148 minimizing state and increasing the stack depth. 1150 6.2. RSVP Scalability 1152 The traditional method of creating a resource allocated path through 1153 an MPLS network is to use the RSVP protocol. However there have been 1154 concerns that this requires significant continuous state maintenance 1155 in the network. Work to improve the scalability of RSVP-TE LSPs in 1156 the control plane can be found in [RFC8370]. 1158 There is also concern at the scalability of the forwarder footprint 1159 of RSVP as the number of paths through an LSR grows. [RFC8577] 1160 proposes to address this by employing SR within a tunnel established 1161 by RSVP-TE. 1163 6.3. SDN Scaling 1165 The centralized approach of SDN requires state to be stored in the 1166 network, but does not have the overhead of also requiring control 1167 plane state to be maintained. Each individual network node may need 1168 to maintain a communication channel with the SDN controller, but that 1169 compares favourably with the need for a control plane to maintain 1170 communication with all neighbors. 1172 However, SDN may transfer some of the scalability concerns from the 1173 network to the centralized controller. In particular, there may be a 1174 heavy processing burden at the controller, and a heavy load in the 1175 network surrounding the controller. 1177 7. OAM Considerations 1179 The enhanced VPN OAM design needs to consider the following 1180 requirements: 1182 o Instrumentation of the underlay so that the network operator can 1183 be sure that the resources committed to a tenant are operating 1184 correctly and delivering the required performance. 1186 o Instrumentation of the overlay by the tenant. This is likely to 1187 be transparent to the network operator and to use existing 1188 methods. Particular consideration needs to be given to the need 1189 to verify the isolation and the various committed performance 1190 characteristics. 1192 o Instrumentation of the overlay by the network provider to 1193 proactively demonstrate that the committed performance is being 1194 delivered. This needs to be done in a non-intrusive manner, 1195 particularly when the tenant is deploying a performance sensitive 1196 application. 1198 o Verification of the conformity of the path to the service 1199 requirement. This may need to be done as part of a commissioning 1200 test. 1202 A study of OAM in SR networks has been documented in [RFC8403]. 1204 8. Telemetry Considerations 1206 Network visibility is essential for network operation. Network 1207 telemetry has been considered as an ideal means to gain sufficient 1208 network visibility with better flexibility, scalability, accuracy, 1209 coverage, and performance than conventional OAM technologies. 1211 As defined in [I-D.ietf-opsawg-ntf], Network Telemetry is to acquire 1212 network data remotely for network monitoring and operation. It is a 1213 general term for a large set of network visibility techniques and 1214 protocols. Network telemetry addresses the current network operation 1215 issues and enables smooth evolution toward intent-driven autonomous 1216 networks. Telemetry can be applied on the forwarding plane, the 1217 control plane, and the management plane in a network. 1219 How the telemetry mechanisms could be used or extended for the 1220 enhanced VPN service is out of the scope of this document. 1222 9. Enhanced Resiliency 1224 Each enhanced VPN has a life-cycle, and may need modification during 1225 deployment as the needs of its tenant change. Additionally, as the 1226 network as a whole evolves, there may need to be garbage collection 1227 performed to consolidate resources into usable quanta. 1229 Systems in which the path is imposed such as SR, or some form of 1230 explicit routing tend to do well in these applications, because it is 1231 possible to perform an atomic transition from one path to another. 1232 This is a single action by the head-end changes the path without the 1233 need for coordinated action by the routers along the path. However, 1234 implementations and the monitoring protocols need to make sure that 1235 the new path is up and meets the required SLA before traffic is 1236 transitioned to it. It is possible for deadlocks to arise as a 1237 result of the network becoming fragmented over time, such that it is 1238 impossible to create a new path or to modify an existing path without 1239 impacting the SLA of other paths. Resolution of this situation is as 1240 much a commercial issue as it is a technical issue and is outside the 1241 scope of this document. 1243 There are, however, two manifestations of the latency problem that 1244 are for further study in any of these approaches: 1246 o The problem of packets overtaking one and other if a path latency 1247 reduces during a transition. 1249 o The problem of transient variation in latency in either direction 1250 as a path migrates. 1252 There is also the matter of what happens during failure in the 1253 underlay infrastructure. Fast reroute is one approach, but that 1254 still produces a transient loss with a normal goal of rectifying this 1255 within 50ms [RFC5654]. An alternative is some form of N+1 delivery 1256 such as has been used for many years to support protection from 1257 service disruption. This may be taken to a different level using the 1258 techniques proposed by the IETF deterministic network work with 1259 multiple in-network replication and the culling of later packets 1260 [RFC8655]. 1262 In addition to the approach used to protect high priority packets, 1263 consideration has to be given to the impact of best effort traffic on 1264 the high priority packets during a transient. Specifically if a 1265 conventional re-convergence process is used there will inevitably be 1266 micro-loops and whilst some form of explicit routing will protect the 1267 high priority traffic, lower priority traffic on best effort shortest 1268 paths will micro-loop without the use of a loop prevention 1269 technology. To provide the highest quality of service to high 1270 priority traffic, either this traffic must be shielded from the 1271 micro-loops, or micro-loops must be prevented. 1273 10. Operational Considerations 1275 It is likely that enhanced VPN service will be introduced in networks 1276 which already have traditional VPN services deployed. Depends on 1277 service requirement, the tenants or the operator may choose to use 1278 traditional VPN or enhanced VPN to fulfil the service requirement. 1279 The information and parameters to assist such decision needs to be 1280 reflected on the management interface between the tenants and the 1281 operator. 1283 11. Security Considerations 1285 All types of virtual network require special consideration to be 1286 given to the isolation of traffic belonging to different tenants. 1287 That is, traffic belonging to one VPN must not be delivered to end 1288 points outside that VPN. In this regard enhanced VPNs neither 1289 introduce, no experience a greater security risks than other VPNs. 1291 However, in an enhanced Virtual Private Network service the 1292 additional service requirements need to be considered. For example, 1293 if a service requires a specific upper bound to latency then it can 1294 be damaged by simply delaying the packets through the activities of 1295 another tenant, i.e., by introducing bursts of traffic for other 1296 services. 1298 The measures to address these dynamic security risks must be 1299 specified as part to the specific solution are form part of the 1300 isolation requirements of a service. 1302 While an enhanced VPN service may be sold as offering encryption and 1303 other security features as part of the service, customers would be 1304 well advised to take responsibility for their own security 1305 requirements themselves possibly by encrypting traffic before handing 1306 it off to the service provider. 1308 The privacy of enhanced VPN service customers must be preserved. It 1309 should not be possible for one customer to discover the existence of 1310 another customer, nor should the sites that are members of an 1311 enhanced VPN be externally visible. 1313 12. IANA Considerations 1315 There are no requested IANA actions. 1317 13. Contributors 1318 Daniel King 1319 Email: daniel@olddog.co.uk 1321 Adrian Farrel 1322 Email: adrian@olddog.co.uk 1324 Jeff Tansura 1325 Email: jefftant.ietf@gmail.com 1327 Zhenbin Li 1328 Email: lizhenbin@huawei.com 1330 Qin Wu 1331 Email: bill.wu@huawei.com 1333 Bo Wu 1334 Email: lana.wubo@huawei.com 1336 Daniele Ceccarelli 1337 Email: daniele.ceccarelli@ericsson.com 1339 Mohamed Boucadair 1340 Email: mohamed.boucadair@orange.com 1342 Sergio Belotti 1343 Email: sergio.belotti@nokia.com 1345 Haomian Zheng 1346 Email: zhenghaomian@huawei.com 1348 14. Acknowledgements 1350 The authors would like to thank Charlie Perkins, James N Guichard, 1351 John E Drake and Shunsuke Homma for their review and valuable 1352 comments. 1354 This work was supported in part by the European Commission funded 1355 H2020-ICT-2016-2 METRO-HAUL project (G.A. 761727). 1357 15. References 1359 15.1. Normative References 1361 [RFC2764] Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. 1362 Malis, "A Framework for IP Based Virtual Private 1363 Networks", RFC 2764, DOI 10.17487/RFC2764, February 2000, 1364 . 1366 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1367 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1368 DOI 10.17487/RFC3985, March 2005, 1369 . 1371 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 1372 2 Virtual Private Networks (L2VPNs)", RFC 4664, 1373 DOI 10.17487/RFC4664, September 2006, 1374 . 1376 15.2. Informative References 1378 [BBF-SD406] 1379 "BBF SD-406: End-to-End Network Slicing", 2016, 1380 . 1383 [DETNET] "Deterministic Networking", March , 1384 . 1386 [FLEXE] "Flex Ethernet Implementation Agreement", March 2016, 1387 . 1390 [I-D.ietf-idr-bgp-ls-segment-routing-ext] 1391 Previdi, S., Talaulikar, K., Filsfils, C., Gredler, H., 1392 and M. Chen, "BGP Link-State extensions for Segment 1393 Routing", draft-ietf-idr-bgp-ls-segment-routing-ext-16 1394 (work in progress), June 2019. 1396 [I-D.ietf-opsawg-l2nm] 1397 Barguil, S., Dios, O., Boucadair, M., Munoz, L., Jalil, 1398 L., and J. Ma, "A Layer 2 VPN Network YANG Model", draft- 1399 ietf-opsawg-l2nm-00 (work in progress), July 2020. 1401 [I-D.ietf-opsawg-l3sm-l3nm] 1402 Barguil, S., Dios, O., Boucadair, M., Munoz, L., and A. 1403 Aguado, "A Layer 3 VPN Network YANG Model", draft-ietf- 1404 opsawg-l3sm-l3nm-03 (work in progress), April 2020. 1406 [I-D.ietf-opsawg-ntf] 1407 Song, H., Qin, F., Martinez-Julia, P., Ciavaglia, L., and 1408 A. Wang, "Network Telemetry Framework", draft-ietf-opsawg- 1409 ntf-03 (work in progress), April 2020. 1411 [I-D.ietf-spring-segment-routing-policy] 1412 Filsfils, C., Talaulikar, K., Voyer, D., Bogdanov, A., and 1413 P. Mattes, "Segment Routing Policy Architecture", draft- 1414 ietf-spring-segment-routing-policy-08 (work in progress), 1415 July 2020. 1417 [I-D.ietf-teas-actn-vn-yang] 1418 Lee, Y., Dhody, D., Ceccarelli, D., Bryskin, I., and B. 1419 Yoon, "A Yang Data Model for VN Operation", draft-ietf- 1420 teas-actn-vn-yang-08 (work in progress), March 2020. 1422 [I-D.ietf-teas-actn-yang] 1423 Lee, Y., Zheng, H., Ceccarelli, D., Yoon, B., Dios, O., 1424 Shin, J., and S. Belotti, "Applicability of YANG models 1425 for Abstraction and Control of Traffic Engineered 1426 Networks", draft-ietf-teas-actn-yang-05 (work in 1427 progress), February 2020. 1429 [I-D.wd-teas-transport-slice-yang] 1430 Bo, W., Dhody, D., Han, L., and R. Rokui, "A Yang Data 1431 Model for Transport Slice NBI", draft-wd-teas-transport- 1432 slice-yang-02 (work in progress), July 2020. 1434 [NGMN-NS-Concept] 1435 "NGMN NS Concept", 2016, . 1439 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1440 and W. Weiss, "An Architecture for Differentiated 1441 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1442 . 1444 [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and J. 1445 McManus, "Requirements for Traffic Engineering Over MPLS", 1446 RFC 2702, DOI 10.17487/RFC2702, September 1999, 1447 . 1449 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1450 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1451 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1452 . 1454 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1455 Conrad, "Stream Control Transmission Protocol (SCTP) 1456 Partial Reliability Extension", RFC 3758, 1457 DOI 10.17487/RFC3758, May 2004, 1458 . 1460 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1461 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 1462 RFC 3931, DOI 10.17487/RFC3931, March 2005, 1463 . 1465 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1466 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1467 2006, . 1469 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 1470 "Encapsulation Methods for Transport of Ethernet over MPLS 1471 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 1472 . 1474 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 1475 Guidelines for DiffServ Service Classes", RFC 4594, 1476 DOI 10.17487/RFC4594, August 2006, 1477 . 1479 [RFC4719] Aggarwal, R., Ed., Townsley, M., Ed., and M. Dos Santos, 1480 Ed., "Transport of Ethernet Frames over Layer 2 Tunneling 1481 Protocol Version 3 (L2TPv3)", RFC 4719, 1482 DOI 10.17487/RFC4719, November 2006, 1483 . 1485 [RFC5151] Farrel, A., Ed., Ayyangar, A., and JP. Vasseur, "Inter- 1486 Domain MPLS and GMPLS Traffic Engineering -- Resource 1487 Reservation Protocol-Traffic Engineering (RSVP-TE) 1488 Extensions", RFC 5151, DOI 10.17487/RFC5151, February 1489 2008, . 1491 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1492 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1493 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1494 September 2009, . 1496 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 1497 Networking: A Perspective from within a Service Provider 1498 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 1499 . 1501 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 1502 Henderickx, W., and A. Isaac, "Requirements for Ethernet 1503 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 1504 . 1506 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 1507 Ceccarelli, D., and X. Zhang, "Problem Statement and 1508 Architecture for Information Exchange between 1509 Interconnected Traffic-Engineered Networks", BCP 206, 1510 RFC 7926, DOI 10.17487/RFC7926, July 2016, 1511 . 1513 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 1514 Network Functions and Their Infrastructure", RFC 8172, 1515 DOI 10.17487/RFC8172, July 2017, 1516 . 1518 [RFC8299] Wu, Q., Ed., Litkowski, S., Tomotaki, L., and K. Ogaki, 1519 "YANG Data Model for L3VPN Service Delivery", RFC 8299, 1520 DOI 10.17487/RFC8299, January 2018, 1521 . 1523 [RFC8370] Beeram, V., Ed., Minei, I., Shakir, R., Pacella, D., and 1524 T. Saad, "Techniques to Improve the Scalability of RSVP-TE 1525 Deployments", RFC 8370, DOI 10.17487/RFC8370, May 2018, 1526 . 1528 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1529 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1530 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1531 July 2018, . 1533 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1534 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1535 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1536 2018, . 1538 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1539 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1540 DOI 10.17487/RFC8453, August 2018, 1541 . 1543 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1544 Data Model for Layer 2 Virtual Private Network (L2VPN) 1545 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1546 2018, . 1548 [RFC8491] Tantsura, J., Chunduri, U., Aldrin, S., and L. Ginsberg, 1549 "Signaling Maximum SID Depth (MSD) Using IS-IS", RFC 8491, 1550 DOI 10.17487/RFC8491, November 2018, 1551 . 1553 [RFC8568] Bernardos, CJ., Rahman, A., Zuniga, JC., Contreras, LM., 1554 Aranda, P., and P. Lynch, "Network Virtualization Research 1555 Challenges", RFC 8568, DOI 10.17487/RFC8568, April 2019, 1556 . 1558 [RFC8577] Sitaraman, H., Beeram, V., Parikh, T., and T. Saad, 1559 "Signaling RSVP-TE Tunnels on a Shared MPLS Forwarding 1560 Plane", RFC 8577, DOI 10.17487/RFC8577, April 2019, 1561 . 1563 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 1564 RFC 8578, DOI 10.17487/RFC8578, May 2019, 1565 . 1567 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1568 "Deterministic Networking Architecture", RFC 8655, 1569 DOI 10.17487/RFC8655, October 2019, 1570 . 1572 [RFC8665] Psenak, P., Ed., Previdi, S., Ed., Filsfils, C., Gredler, 1573 H., Shakir, R., Henderickx, W., and J. Tantsura, "OSPF 1574 Extensions for Segment Routing", RFC 8665, 1575 DOI 10.17487/RFC8665, December 2019, 1576 . 1578 [RFC8667] Previdi, S., Ed., Ginsberg, L., Ed., Filsfils, C., 1579 Bashandy, A., Gredler, H., and B. Decraene, "IS-IS 1580 Extensions for Segment Routing", RFC 8667, 1581 DOI 10.17487/RFC8667, December 2019, 1582 . 1584 [SFC] "Service Function Chaining", March , 1585 . 1587 [TS23501] "3GPP TS23.501", 2016, 1588 . 1591 [TS28530] "3GPP TS28.530", 2016, 1592 . 1595 [TSN] "Time-Sensitive Networking", March , 1596 . 1598 Authors' Addresses 1600 Jie Dong 1601 Huawei 1603 Email: jie.dong@huawei.com 1605 Stewart Bryant 1606 Futurewei 1608 Email: stewart.bryant@gmail.com 1610 Zhenqiang Li 1611 China Mobile 1613 Email: lizhenqiang@chinamobile.com 1615 Takuya Miyasaka 1616 KDDI Corporation 1618 Email: ta-miyasaka@kddi.com 1620 Young Lee 1621 Samsung 1623 Email: younglee.tx@gmail.com