idnits 2.17.1 draft-ietf-teas-enhanced-vpn-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 08, 2019) is 1753 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2119' is defined on line 1476, but no explicit reference was found in the text == Outdated reference: A later version (-24) exists of draft-ietf-teas-actn-vn-yang-06 == Outdated reference: A later version (-12) exists of draft-ietf-teas-sf-aware-topo-model-03 == Outdated reference: A later version (-15) exists of draft-ietf-teas-te-service-mapping-yang-01 Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group J. Dong 3 Internet-Draft S. Bryant 4 Intended status: Informational Huawei 5 Expires: January 9, 2020 Z. Li 6 China Mobile 7 T. Miyasaka 8 KDDI Corporation 9 Y. Lee 10 Futurewei 11 July 08, 2019 13 A Framework for Enhanced Virtual Private Networks (VPN+) Service 14 draft-ietf-teas-enhanced-vpn-02 16 Abstract 18 This document specifies a framework for using existing, modified and 19 potential new networking technologies as components to provide an 20 Enhanced Virtual Private Networks (VPN+) service. The purpose is to 21 support the needs of new applications, particularly applications that 22 are associated with 5G services by utilizing an approach that is 23 based on existing VPN technologies and adds features that specific 24 services require over and above traditional VPNs. 26 Typically, VPN+ will be used to form the underpinning of network 27 slicing, but could also be of use in its own right. It is not 28 envisaged that large numbers of VPN+ instances will be deployed in a 29 network and, in particular, it is not intended that all VPNs 30 supported by a network will use VPN+ techniques. 32 Status of This Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at https://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on January 9, 2020. 49 Copyright Notice 51 Copyright (c) 2019 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents 56 (https://trustee.ietf.org/license-info) in effect on the date of 57 publication of this document. Please review these documents 58 carefully, as they describe your rights and restrictions with respect 59 to this document. Code Components extracted from this document must 60 include Simplified BSD License text as described in Section 4.e of 61 the Trust Legal Provisions and are provided without warranty as 62 described in the Simplified BSD License. 64 Table of Contents 66 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 67 2. Overview of the Requirements . . . . . . . . . . . . . . . . 5 68 2.1. Isolation between Virtual Networks . . . . . . . . . . . 5 69 2.1.1. A Pragmatic Approach to Isolation . . . . . . . . . . 7 70 2.2. Performance Guarantee . . . . . . . . . . . . . . . . . . 8 71 2.3. Integration . . . . . . . . . . . . . . . . . . . . . . . 10 72 2.3.1. Abstraction . . . . . . . . . . . . . . . . . . . . . 10 73 2.4. Dynamic Configuration . . . . . . . . . . . . . . . . . . 10 74 2.5. Customized Control . . . . . . . . . . . . . . . . . . . 11 75 2.6. Applicability . . . . . . . . . . . . . . . . . . . . . . 11 76 2.7. Inter-Domain and Inter-Layer Network . . . . . . . . . . 11 77 3. Architecture of Enhanced VPN . . . . . . . . . . . . . . . . 12 78 3.1. Layered Architecture . . . . . . . . . . . . . . . . . . 13 79 3.2. Multi-Point to Multi-Point . . . . . . . . . . . . . . . 15 80 3.3. Application Specific Network Types . . . . . . . . . . . 15 81 3.4. Scaling Considerations . . . . . . . . . . . . . . . . . 15 82 4. Candidate Technologies . . . . . . . . . . . . . . . . . . . 16 83 4.1. Underlay Packet and Frame-Based Data Planes . . . . . . . 16 84 4.1.1. FlexE . . . . . . . . . . . . . . . . . . . . . . . . 17 85 4.1.2. Dedicated Queues . . . . . . . . . . . . . . . . . . 17 86 4.1.3. Time Sensitive Networking . . . . . . . . . . . . . . 18 87 4.2. Packet and Frame-Based Network Layer . . . . . . . . . . 18 88 4.2.1. Deterministic Networking . . . . . . . . . . . . . . 18 89 4.2.2. MPLS Traffic Engineering (MPLS-TE) . . . . . . . . . 19 90 4.2.3. Segment Routing . . . . . . . . . . . . . . . . . . . 19 91 4.3. Non-Packet Technologies . . . . . . . . . . . . . . . . . 21 92 4.4. Control Plane . . . . . . . . . . . . . . . . . . . . . . 21 93 4.5. Management Plane . . . . . . . . . . . . . . . . . . . . 22 94 4.6. Applicability of ACTN to Enhanced VPN . . . . . . . . . . 22 95 4.6.1. ACTN Used for VPN+ Delivery . . . . . . . . . . . . . 24 96 4.6.2. Enhanced VPN Features with ACTN . . . . . . . . . . . 26 98 5. Scalability Considerations . . . . . . . . . . . . . . . . . 28 99 5.1. Maximum Stack Depth of SR . . . . . . . . . . . . . . . . 29 100 5.2. RSVP Scalability . . . . . . . . . . . . . . . . . . . . 29 101 6. OAM Considerations . . . . . . . . . . . . . . . . . . . . . 30 102 7. Enhanced Resiliency . . . . . . . . . . . . . . . . . . . . . 30 103 8. Security Considerations . . . . . . . . . . . . . . . . . . . 31 104 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 105 10. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 32 106 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 32 107 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 32 108 12.1. Normative References . . . . . . . . . . . . . . . . . . 33 109 12.2. Informative References . . . . . . . . . . . . . . . . . 33 110 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 37 112 1. Introduction 114 Virtual private networks (VPNs) have served the industry well as a 115 means of providing different groups of users with logically isolated 116 access to a common network. The common or base network that is used 117 to provide the VPNs is often referred to as the underlay, and the VPN 118 is often called an overlay. 120 Customers of a network operator may request enhanced overlay services 121 with advanced characteristics such as complete isolation from other 122 services so that changes in network load or event of other services 123 have no effect on the throughput or latency of the service provided 124 to the customer. 126 Driven largely by needs surfacing from 5G, the concept of network 127 slicing has gained traction [NGMN-NS-Concept] [TS23501] [TS28530] 128 [BBF-SD406]. Network slicing requires the underlying network to 129 support partitioning the network resources to provide the client with 130 dedicated (private) networking, computing, and storage resources 131 drawn from a shared pool. The slices may be seen as (and operated 132 as) virtual networks. 134 Network abstraction is a technique that can be applied to a network 135 domain to select network resources by policy to obtain a view of 136 potential connectivity and a set of service functions. 138 Network slicing is an approach to network operations that builds on 139 the concept of network abstraction to provide programmability, 140 flexibility, and modularity. It may use techniques such as Software 141 Defined Networking (SDN) [RFC7149] and Network Function 142 Virtualization (NFV) to create multiple logical (virtual) networks, 143 each tailored for a set of services or a particular tenant or a group 144 of tenants that share the same set of requirements, on top of a 145 common network. How the network slices are engineered can be 146 deployment-specific. 148 Thus, there is a need to create virtual networks with enhanced 149 characteristics. The tenant of such a virtual network can require a 150 degree of isolation and performance that previously could not be 151 satisfied by traditional overlay VPNs. Additionally, the tenant may 152 ask for some level of control to their virtual networks, e.g., to 153 customize the service paths in a network slice. 155 These enhanced properties cannot be met with pure overlay networks, 156 as they require tighter coordination and integration between the 157 underlay and the overlay network. This document introduces a new 158 network service called Enhanced VPN: VPN+. VPN+ is built on a virtual 159 network which has a customized network topology and a set of 160 dedicated or shared network resources, including invoked service 161 functions, allocated from the underlay network. Unlike a traditional 162 VPN, an enhanced VPN can achieve greater isolation with strict 163 guaranteed performance. These new properties, which have general 164 applicability, may also be of interest as part of a network slicing 165 solution, but it is not envisaged that VPN+ techniques will be 166 applied to normal VPN services that can continue to be deployed using 167 pre-existing mechanisms. Furthermore, it is not intended that large 168 numbers of VPN+ instances will be deployed within a single network. 169 See Section 5 for a discussion of scalability considerations. 171 This document specifies a framework for using existing, modified and 172 potential new networking technologies as components to provide a VPN+ 173 service. Specifically we are concerned with: 175 o The design of the enhanced data plane. 177 o The necessary protocols in both the underlay and the overlay of 178 enhanced VPN. 180 o The mechanisms to achieve integration between overlay and 181 underlay. 183 o The necessary Operation, Administration and Management (OAM) 184 methods to instrument an enhanced VPN to make sure that the 185 required Service Level Agreement (SLA) are met, and to take any 186 corrective action to avoid SLA violation, such as switching to an 187 alternate path. 189 The required network layered structure to achieve this is shown in 190 Section 3.1. 192 Note that, in this document, the four terms "VPN", "Enhanced VPN" (or 193 "VPN+"), "Virtual Network (VN)", and "Network Slice" may be 194 considered as describing similar concepts dependent on the viewpoint 195 from which they are used. 197 o An enhanced VPN can be considered as a form of VPN, but with 198 additional service-specific commitments. Thus, care must be taken 199 with the term "VPN" to distinguish normal or legacy VPNs from VPN+ 200 instances. 202 o A Virtual Network is a type of service that connects customer edge 203 points with the additional possibility of requesting further 204 service characteristics in the manner of an enhanced VPN. 206 o An enhanced VPN or VN is made by creating a slice through the 207 resources of the underlay network. 209 o The general concept of network slicing in a TE network is a larger 210 problem space than is addressed by VPN+ or VN, but those concepts 211 are tools to address some aspects or realizations of network 212 slicing. 214 2. Overview of the Requirements 216 In this section we provide an overview of the requirements of an 217 enhanced VPN. 219 2.1. Isolation between Virtual Networks 221 One element of the SLA demanded for an enhanced VPN is the degree of 222 isolation from other services in the network. Isolation is a feature 223 requested by some particular customers in the network. Such feature 224 is offered by a network operator where the traffic from one service 225 instance is isolated from the traffic of other services. There are 226 different grades of isolation that range from simple separation of 227 traffic on delivery (ensuring that traffic is not delivered to the 228 wrong customer) all the way to complete separation within the 229 underlay so that the traffic from different services use distinct 230 network resources. 232 The terms hard and soft isolation are introduced to give example of 233 different isolation cases. A VPN has soft isolation if the traffic 234 of one VPN cannot be received by the customers of another VPN. Both 235 IP and MPLS VPNs are examples of soft isolated VPNs because the 236 network delivers the traffic only to the required VPN endpoints. 237 However, with soft isolation, traffic from one or more VPNs and 238 regular non-VPN traffic may congest the network resulting in packet 239 loss and delay for other VPNs operating normally. The ability for a 240 VPN or a group of VPNs to be sheltered from this effect is called 241 hard isolation, and this property is required by some critical 242 applications. 244 The requirement is for an operator to provide both hard and soft 245 isolation between the tenants/applications using one enhanced VPN and 246 the tenants/applications using another enhanced VPN. Hard isolation 247 is needed so that applications with exacting requirements can 248 function correctly, despite other demands (perhaps a burst on another 249 VPN) competing for the underlying resources. In practice isolation 250 may be offered as a spectrum between soft and hard, and in some cases 251 soft and hard isolation may be used in a hierarchical manner. 253 An example of hard isolation is a network supporting both emergency 254 services and public broadband multi-media services. During a major 255 incident the VPNs supporting these services would both be expected to 256 experience high data volumes, and it is important that both make 257 progress in the transmission of their data. In these circumstances 258 the VPNs would require an appropriate degree of isolation to be able 259 to continue to operate acceptably. 261 In order to provide the required isolation, resources may have to be 262 reserved in the data plane of the underlay network and dedicated to 263 traffic from a specific VPN or a specific group of VPNs. This may 264 introduce scalability concerns, thus some trade-off needs to be 265 considered to provide the required isolation between network slices 266 while still allowing reasonable sharing inside each network slice. 268 An optical layer can offer a high degree of isolation, at the cost of 269 allocating resources on a long term and end-to-end basis. Such an 270 arrangement means that the full cost of the resources must be borne 271 by the service that is allocated with the resources. On the other 272 hand, where adequate isolation can be achieved at the packet layer, 273 this permits the resources to be shared amongst many services and 274 only dedicated to a service on a temporary basis. This in turn, 275 allows greater statistical multiplexing of network resources and thus 276 amortizes the cost over many services, leading to better economy. 277 However, the different degrees of isolation required by network 278 slicing cannot be entirely met with existing mechanisms such as 279 Traffic Engineered Label Switched Paths (TE-LSPs). This is because 280 most implementations enforce the bandwidth in the data-plane only at 281 the PEs, but at the P routers the bandwidth is only reserved in the 282 control plane, thus bursts of data can accidentally occur at a P 283 router with higher than committed data rate. 285 There are several new technologies that provide some assistance with 286 these data plane issues. Firstly there is the IEEE project on Time 287 Sensitive Networking [TSN] which introduces the concept of packet 288 scheduling of delay and loss sensitive packets. Then there is 289 [FLEXE] which provides the ability to multiplex multiple channels 290 over one or more Ethernet links in a way that provides hard 291 isolation. Finally there are advanced queueing approaches which 292 allow the construction of virtual sub-interfaces, each of which is 293 provided with dedicated resource in a shared physical interface. 294 These approaches are described in more detail later in this document. 296 In the remainder of this section we explore how isolation may be 297 achieved in packet networks. 299 2.1.1. A Pragmatic Approach to Isolation 301 A key question is whether it is possible to achieve hard isolation in 302 packet networks, which were never designed to support hard isolation. 303 On the contrary, they were designed to provide statistical 304 multiplexing, a significant economic advantage when compared to a 305 dedicated, or a Time Division Multiplexing (TDM) network. However 306 there is no need to provide any harder isolation than is required by 307 the application. Pseudowires [RFC3985] emulate services that would 308 have had hard isolation in their native form. An approximation to 309 this requirement is sufficient in most cases. 311 Thus, for example, using FlexE or a virtual sub-interface together 312 with packet scheduling as isolation mechanism of interface resources, 313 optionally along with the partitioning of node resources, a type of 314 hard isolation can be provided that is adequate for many VPN+ 315 applications. Other applications may be either satisfied with a 316 classical VPN with or without reserved bandwidth, or may need 317 dedicated point to point underlay connection. The needs of each 318 application must be quantified in order to provide an economic 319 solution that satisfies those needs without over-engineering. 321 This spectrum of isolation is shown in Figure 1: 323 O=================================================O 324 | \---------------v---------------/ 325 Statistical Pragmatic Absolute 326 Multiplexing Isolation Isolation 327 (Traditional VPNs) (Enhanced VPN) (Dedicated Network) 329 Figure 1: The Spectrum of Isolation 331 At one end of the above figure, we have traditional statistical 332 multiplexing technologies that support VPNs. This is a service type 333 that has served the industry well and will continue to do so. At the 334 opposite end of the spectrum, we have the absolute isolation provided 335 by dedicated transport networks. The goal of enhanced VPN is 336 pragmatic isolation. This is isolation that is better than is 337 obtainable from pure statistical multiplexing, more cost effective 338 and flexible than a dedicated network, but which is a practical 339 solution that is good enough for the majority of applications. 340 Mechanisms for both soft isolation and hard isolation would be needed 341 to meet different levels of service requirement. 343 2.2. Performance Guarantee 345 There are several kinds of performance guarantees, including 346 guaranteed maximum packet loss, guaranteed maximum delay and 347 guaranteed delay variation. Note that these guarantees apply to the 348 conformance traffic, the out-of-profile traffic will be handled 349 following other requirements. 351 Guaranteed maximum packet loss is a common parameter, and is usually 352 addressed by setting the packet priorities, queue size and discard 353 policy. However this becomes more difficult when the requirement is 354 combined with the latency requirement. The limiting case is zero 355 congestion loss, and that is the goal of the Deterministic Networking 356 work that the IETF [DETNET] and IEEE [TSN] are pursuing. In modern 357 optical networks, loss due to transmission errors is already 358 approaches zero, but there are the possibilities of failure of the 359 interface or the fiber itself. This can only be addressed by some 360 form of signal duplication and transmission over diverse paths. 362 Guaranteed maximum latency is required in a number of applications 363 particularly real-time control applications and some types of virtual 364 reality applications. The work of the IETF Deterministic Networking 365 (DetNet) Working Group [DETNET] is relevant; however the scope needs 366 to be extended to methods of enhancing the underlay to better support 367 the delay guarantee, and to integrate these enhancements with the 368 overall service provision. 370 Guaranteed maximum delay variation is a service that may also be 371 needed. [I-D.ietf-detnet-use-cases] calls up a number of cases where 372 this is needed, for example electrical utilities have an operational 373 need for this. Time transfer is one example of a service that needs 374 this, although it is in the nature of time that the service might be 375 delivered by the underlay as a shared service and not provided 376 through different virtual networks. Alternatively a dedicated 377 virtual network may be used to provide this as a shared service. 379 This suggests that a spectrum of service guarantee be considered when 380 deploying an enhanced VPN. As a guide to understanding the design 381 requirements we can consider four types: 383 o Best effort 385 o Assured bandwidth 387 o Guaranteed latency 389 o Enhanced delivery 391 Best effort service is the basic service that current VPNs can 392 provide. 394 An assured bandwidth service is one in which the bandwidth over some 395 period of time is assured, this can be achieved either simply based 396 on best effort with over-capacity provisioning, or it can be based on 397 TE-LSPs with bandwidth reservation. The instantaneous bandwidth is 398 however, not necessarily assured, depending on the technique used. 399 Providing assured bandwidth to VPNs, for example by using TE-LSPs, is 400 not widely deployed at least partially due to scalability concerns. 401 Guaranteed latency and enhanced delivery are not yet integrated with 402 VPNs. 404 A guaranteed latency service has a latency upper bound provided by 405 the network. Assuring the upper bound is more important than 406 achieving the minimum latency. 408 In Section 2.1 we considered the work of the IEEE Time Sensitive 409 Networking (TSN) project [TSN] and the work of the IETF DetNet 410 Working group [DETNET] in the context of isolation. The TSN and 411 DetNet work is of greater relevance in assuring end-to-end packet 412 latency. It is also of importance in considering enhanced delivery. 414 An enhanced delivery service is one in which the underlay network (at 415 layer 3) attempts to deliver the packet through multiple paths in the 416 hope of eliminating packet loss due to equipment or media failures. 418 It is these last two characteristics that an enhanced VPN adds to a 419 VPN service. 421 Flex Ethernet [FLEXE] is a useful underlay to provide these 422 guarantees. This is a method of providing time-slot based 423 channelization over an Ethernet bearer. Such channels are fully 424 isolated from other channels running over the same Ethernet bearer. 425 As noted elsewhere this produces hard isolation but makes the 426 reclamation of unused bandwidth more difficult. 428 These approaches can be used in tandem. It is possible to use FlexE 429 to provide tenant isolation, and then to use the TSN/Detnet approach 430 to provide a performance guarantee inside the a slice or tenant VPN. 432 2.3. Integration 434 A solution to the enhanced VPN problem has to provide close 435 integration of both overlay VPN and the underlay network resource. 436 This needs be done in a flexible and scalable way so that it can be 437 widely deployed in operator networks to support a reasonable number 438 of enhanced VPN customers. 440 Taking mobile networks and in particular 5G into consideration, the 441 integration of network and the service functions is a likely 442 requirement. The work in IETF SFC working group [SFC] provides a 443 foundation for this integration. 445 2.3.1. Abstraction 447 Integration of the overlay VPN and the underlay network resources 448 does not need to be a tight mapping. As described in [RFC7926], 449 abstraction is the process of applying policy to a set of information 450 about a TE network to produce selective information that represents 451 the potential ability to connect across the network. The process of 452 abstraction presents the connectivity graph in a way that is 453 independent of the underlying network technologies, capabilities, and 454 topology so that the graph can be used to plan and deliver network 455 services in a uniform way. 457 Virtual networks can be built on top of an abstracted topology that 458 represents the connectivity capabilities of the underlay network as 459 described in the framework for Abstraction and Control of TE Networks 460 (ACTN) described in [RFC8453] as discussed further in Section 4.5. 462 2.4. Dynamic Configuration 464 Enhanced VPNs need to be created, modified, and removed from the 465 network according to service demand. An enhanced VPN that requires 466 hard isolation must not be disrupted by the instantiation or 467 modification of another enhanced VPN. Determining whether 468 modification of an enhanced VPN can be disruptive to that VPN, and in 469 particular the traffic in flight will be disrupted can be a difficult 470 problem. 472 The data plane aspects of this problem are discussed further in 473 Section 4. 475 The control plane aspects of this problem are discussed further in 476 Section 4.4. 478 The management plane aspects of this problem are discussed further in 479 Section 4.5 480 Dynamic changes both to the VPN and to the underlay transport network 481 need to be managed to avoid disruption to sensitive services. 483 In addition to non-disruptively managing the network as a result of 484 gross change such as the inclusion of a new VPN endpoint or a change 485 to a link, VPN traffic might need to be moved as a result of traffic 486 volume changes. 488 2.5. Customized Control 490 In some cases it is desirable that an enhanced VPN has a customized 491 control plane, so that the tenant of the enhanced VPN can have some 492 control to the resources and functions allocated to this enhanced 493 VPN. For example, the tenant may be able to specify the service 494 paths in his own enhanced VPN. Depending on the requirement, an 495 enhanced VPN may have its own dedicated controller, or it may be 496 provided with an interface to a control system which is shared with a 497 set of other tenants, or it may be provided with an interface to the 498 control system provided by the network operator. 500 Further detail on this requirement will be provided in a future 501 version of the draft. A description of the management plane aspects 502 of this feature can be found in Section 4.5. 504 2.6. Applicability 506 The technologies described in this document should be applicable to a 507 number types of VPN services such as: 509 o Layer 2 point to point services such as pseudowires [RFC3985] 511 o Layer 2 VPNs [RFC4664] 513 o Ethernet VPNs [RFC7209] 515 o Layer 3 VPNs [RFC4364], [RFC2764] 517 o Virtual Networks (VNs) [RFC8453] 519 Where such VPN or VN types need enhanced isolation and delivery 520 characteristics, the technology described here can be used to provide 521 an underlay with the required enhanced performance. 523 2.7. Inter-Domain and Inter-Layer Network 525 In some scenarios, an enhanced VPN services may span multiple network 526 domains. And in some domains the operator may own a multi-layered 527 network. When enhanced VPNs are provisioned in such network 528 scenarios, the technologies used in different network plane (data 529 plane, control plane and management plane) need to provide necessary 530 mechanisms to support multi-domain and multi-layer coordination and 531 integration, so as to provide the required service characterstics for 532 different enhanced VPNs, and improve network efficiency and 533 operational simplicity. 535 3. Architecture of Enhanced VPN 537 A number of enhanced VPN services will typically be provided by a 538 common network infrastructure. Each enhanced VPN consists of both 539 the overlay and a specific set of dedicated network resources and 540 functions allocated in the underlay to satisfy the needs of the VPN 541 tenant. The integration between overlay and various underlay 542 resources ensures the isolation between different enhanced VPNs, and 543 achieves the guaranteed performance for different services. 545 An enhanced VPN needs to be designed with consideration given to: 547 o A enhanced data plane 549 o A control plane to create enhanced VPN, making use of the data 550 plane isolation and guarantee techniques 552 o A management plane for enhanced VPN service life-cycle management 554 These required characteristics are expanded below: 556 o Enhanced data plane 558 * Provides the required resource isolation capability, e.g. 559 bandwidth guarantee. 561 * Provides the required packet latency and jitter 562 characteristics. 564 * Provides the required packet loss characteristics. 566 * Provides the mechanism to identify network slice and the 567 associated resources. 569 o Control plane 571 * Collect the underlying network topology and resources available 572 and export this to other nodes and/or the centralized 573 controller as required. 575 * Create the required virtual networks with the resource and 576 properties needed by the enhanced VPN services that are 577 assigned to it. 579 * Determine the risk of SLA violation and take appropriate 580 avoiding action. 582 * Determine the right balance of per-packet and per-node state 583 according to the needs of enhanced VPN service to scale to the 584 required size. 586 o Management plane 588 * Provides the life-cycle management (creation, modification, 589 decommissioning) of enhanced VPN. 591 * Provides an interface between the enhanced VPN provider and the 592 enhanced VPN clients such that some of the operation requests 593 can be met without interfering with the enhanced VPN of other 594 clients. 596 3.1. Layered Architecture 598 The layered architecture of enhanced VPN is shown in Figure 2. 600 +-------------------+ } 601 | Network Controller| } Centralized 602 +-------------------+ } Control 603 . . . . . 604 . . . . . 605 . N----N----N . } 606 . / / . } 607 N-----N-----N----N-----N } 608 N----N } 609 / / \ } Virtual 610 N-----N----N----N-----N } Networks 611 N----N } 612 / / } 613 N-----N-----N----N-----N } 615 +----+ ===== +----+ ===== +----+ ===== +----+ } 616 +----+ ===== +----+ ===== +----+ ===== +----+ } Physical 617 +----+ ===== +----+ ===== +----+ ===== +----+ } Network 618 +----+ +----+ +----+ +----+ } 619 N L N L N L N 621 N = Partitioned node 622 L = Partitioned link 624 +----+ = Partition within a node 625 +----+ 627 ====== = Partition within a link 629 Figure 2: The Layered Architecture 631 Underpinning everything is the physical infrastructure layer 632 consisting of partitioned links and nodes which provide the 633 underlying resources used to provision the separated virtual 634 networks. Various components and techniques as discussed in 635 Section 4 can be used to provide the resource partition, such as 636 FlexE, Time Sensitive Networking, Deterministic Networking, etc. 637 These partitions may be physical, or virtual so long as the SLA 638 required by the higher layers is met. 640 These techniques can be used to provision the virtual networks with 641 dedicated resources that they need. To get the required 642 functionality there needs to be integration between these overlays 643 and the underlay providing the physical resources. 645 The centralized controller is used to create the virtual networks, to 646 allocate the resources to each virtual network and to provision the 647 enhanced VPN services within the virtual networks. A distributed 648 control plane may also be used for the distribution of the topology 649 and attribute information of the virtual networks. 651 The creation and allocation process needs to take a holistic view of 652 the needs of all of its tenants, and to partition the resources 653 accordingly. However within a virtual network these resources can if 654 required be managed via a dynamic control plane. This provides the 655 required scalability and isolation. 657 3.2. Multi-Point to Multi-Point 659 At the VPN service level, the connectivity are usually mesh or 660 partial-mesh. To support such kind of VPN service, the corresponding 661 underlay is also an abstract MP2MP medium. However when service 662 guarantees are provided, the point-to-point path through the underlay 663 of the enhanced VPN needs to be specifically engineered to meet the 664 required performance guarantees. 666 3.3. Application Specific Network Types 668 Although a lot of the traffic that will be carried over the enhanced 669 VPN will likely be IPv4 or IPv6, the design has to be capable of 670 carrying other traffic types, in particular Ethernet traffic. This 671 is easily accomplished through the various pseudowire (PW) techniques 672 [RFC3985]. Where the underlay is MPLS, Ethernet can be carried over 673 the enhanced VPN encapsulated according to the method specified in 674 [RFC4448]. Where the underlay is IP, Layer Two Tunneling Protocol - 675 Version 3 (L2TPv3) [RFC3931] can be used with Ethernet traffic 676 carried according to [RFC4719]. Encapsulations have been defined for 677 most of the common layer-2 types for both PW over MPLS and for 678 L2TPv3. 680 3.4. Scaling Considerations 682 VPNs are instantiated as overlays on top of an operators network and 683 offered as services to the operators customers. An important feature 684 of overlays is that they are able to deliver services without placing 685 per-service state in the core of the underlay network. 687 Enhanced VPNs may need to install some additional state within the 688 network to achieve the additional features that they require. 689 Solutions must consider minimising and controlling the scale of such 690 state, and deployment architectures should constrain the number of 691 enhanced VPNs that would exist where such services would place 692 additional state in the network. It is expected that the number of 693 enhanced VPN would be a small number in the beginning, and even in 694 future the number of enhanced VPN will be much less than traditional 695 VPNs, because traditional VPN would be enough for most existing 696 services. 698 In general, it is not required that the state in the network to be 699 maintained in a 1:1 relationship with the VPN+ instances. It will 700 usually be possible to aggregate a set of VPN+ services so that they 701 share a same virtual network and the same set of network resources 702 (much in the way that current VPNs are aggregated over transport 703 tunnels) so that collections of enhanced VPNs that require the same 704 behaviour from the network in terms of resource reservation, latency 705 bounds, resiliency, etc. are able to be grouped together. This is an 706 important feature to assist with the scaling characteristics of VPN+ 707 deployments. 709 See Section 5 for a greater discussion of scalability considerations. 711 4. Candidate Technologies 713 A VPN is a network created by applying a multiplexing technique to 714 the underlying network (the underlay) in order to distinguish the 715 traffic of one VPN from that of another. A VPN path that travels by 716 other than the shortest path through the underlay normally requires 717 state in the underlay to specify that path. State is normally 718 applied to the underlay through the use of the RSVP Signaling 719 protocol, or directly through the use of an SDN controller, although 720 other techniques may emerge as this problem is studied. This state 721 gets harder to manage as the number of VPN paths increases. 722 Furthermore, as we increase the coupling between the underlay and the 723 overlay to support the enhanced VPN service, this state will increase 724 further. 726 In an enhanced VPN different subsets of the underlay resources can be 727 dedicated to different enhanced VPNs or different groups of enhanced 728 VPNs. An enhanced VPN solution thus needs tighter coupling with 729 underlay than is the case with existing VPNs. We cannot for example 730 share the tunnel between enhanced VPNs which require hard isolation. 732 4.1. Underlay Packet and Frame-Based Data Planes 734 A number of candidate underlay packet or frame-based data plane 735 solutions which can be used provide the required isolation and 736 guarantee are described in following sections. 738 o FlexE 740 o Time Sensitive Networking 742 o Dedicated Queues 744 4.1.1. FlexE 746 FlexE [FLEXE] is a method of creating a point-to-point Ethernet with 747 a specific fixed bandwidth. FlexE provides the ability to multiplex 748 multiple channels over an Ethernet link in a way that provides hard 749 isolation. FlexE also supports the bonding of multiple links, which 750 can be used to create larger links out of multiple slower links in a 751 more efficient way that traditional link aggregation. FlexE also 752 supports the sub-rating of links, which allows an operator to only 753 use a portion of a link. However it is a only a link level 754 technology. When packets are received by the downstream node, they 755 need to be processed in a way that preserves that isolation in the 756 downstream node. This in turn requires a queuing and forwarding 757 implementation that preserves the end-to-end isolation. 759 If different FlexE channels are used for different services, then no 760 sharing is possible between the FlexE channels. This in turn means 761 that it may be difficult to dynamically redistribute unused bandwidth 762 to lower priority services. This may increase the cost of providing 763 services on the network. On the other hand, FlexE can be used to 764 provide hard isolation between different tenants on a shared 765 interface. The tenant can then use other methods to manage the 766 relative priority of their own traffic in each FlexE channel. 768 Methods of dynamically re-sizing FlexE channels and the implication 769 for enhanced VPN is for further study. 771 4.1.2. Dedicated Queues 773 In order to provide multiple isolated virtual networks for enhanced 774 VPN, the conventional Diff-Serv based queuing system [RFC2475] 775 [RFC4594] is insufficient, due to the limited number of queues which 776 cannot differentiate between traffic of different enhanced VPNs, and 777 the range of service classes that each need to provide to their 778 tenants. This problem is particularly acute with an MPLS underlay 779 due to the small number of traffic class services available. In 780 order to address this problem and reduce the interference between 781 enhanced VPNs, it is necessary to steer traffic of VPNs to dedicated 782 input and output queues. Routers usually have large amount of queues 783 and sophisticated queuing systems, which could be used or enhanced to 784 provide the levels of isolation required by the applications of 785 enhanced VPN. For example, on one physical interface, the queuing 786 system can provide a set of virtual sub-interfaces, each allocated 787 with dedicated queueing and buffer resources. Sophisticated queuing 788 systems of this type may be used to provide end-to-end virtual 789 isolation between traffic of different enhanced VPNs. 791 4.1.3. Time Sensitive Networking 793 Time Sensitive Networking (TSN) [TSN] is an IEEE project that is 794 designing a method of carrying time sensitive information over 795 Ethernet. It introduces the concept of packet scheduling where a 796 high priority packet stream may be given a scheduled time slot 797 thereby guaranteeing that it experiences no queuing delay and hence a 798 reduced latency. However, when no scheduled packet arrives, its 799 reserved time-slot is handed over to best effort traffic, thereby 800 improving the economics of the network. The mechanisms defined in 801 TSN can be used to meet the requirements of time sensitive services 802 of an enhanced VPN. 804 Ethernet can be emulated over a Layer 3 network using a pseudowire. 805 However the TSN payload would be opaque to the underlay and thus not 806 treated specifically as time sensitive data. The preferred method of 807 carrying TSN over a layer 3 network is through the use of 808 deterministic networking as explained in the following section of 809 this document. 811 4.2. Packet and Frame-Based Network Layer 813 We now consider the problem of slice differentiation and resource 814 representation in the network layer. The candidate technologies are: 816 o Deterministic Networking 818 o MPLS-TE 820 o Segment Routing 822 4.2.1. Deterministic Networking 824 Deterministic Networking (DetNet) [I-D.ietf-detnet-architecture] is a 825 technique being developed in the IETF to enhance the ability of layer 826 3 networks to deliver packets more reliably and with greater control 827 over the delay. The design cannot use re-transmission techniques 828 such as TCP since that can exceed the delay tolerated by the 829 applications. Even the delay improvements that are achieved with 830 Stream Control Transmission Protocol Partial Reliability Extenstion 831 (SCTP-PR) [RFC3758] do not meet the bounds set by application 832 demands. DetNet pre-emptively sends copies of the packet over 833 various paths to minimize the chance of all packets being lost, and 834 trims duplicate packets to prevent excessive flooding of the network 835 and to prevent multiple packets being delivered to the destination. 836 It also seeks to set an upper bound on latency. The goal is not to 837 minimize latency; the optimum upper bound paths may not be the 838 minimum latency paths. 840 DetNet is based on flows. It currently does not specify the use of 841 underlay topology other than the base topology. To be of use for 842 enhanced VPN, DetNet needs to be integrated with different virtual 843 topologies of enhanced VPNs. 845 The detailed design that allows the use DetNet in a multi-tenant 846 network, and how to improve the scalability of DetNet in a multi- 847 tenant network are topics for further study. 849 4.2.2. MPLS Traffic Engineering (MPLS-TE) 851 MPLS-TE introduces the concept of reserving end-to-end bandwidth for 852 a TE-LSP, which can be used as the underlay of VPNs. It also 853 introduces the concept of non-shortest path routing through the use 854 of the Explicit Route Object [RFC3209]. VPN traffic can be run over 855 dedicated TE-LSPs to provide reserved bandwidth for each specific 856 connection in a VPN. Some network operators have concerns about the 857 scalability and management overhead of RSVP-TE system, and this has 858 lead them to consider other solutions for their networks. 860 4.2.3. Segment Routing 862 Segment Routing [RFC8402] is a method that prepends instructions to 863 packets at the head-end node and optionally at various points as it 864 passes though the network. These instructions allow the packets to 865 be routed on paths other than the shortest path for various traffic 866 engineering reasons. These paths can be strict or loose paths, 867 depending on the compactness required of the instruction list and the 868 degree of autonomy granted to the network, for example to support 869 Equal Cost Multipath load-balancing (ECMP) [RFC2992]. 871 With SR, a path needs to be dynamically created through a set of 872 segments by simply specifying the Segment Identifiers (SIDs), i.e. 873 instructions rooted at a particular point in the network. Thus if a 874 path is to be provisioned from some ingress point A to some egress 875 point B in the underlay, A is provided with a SID list from A to B 876 and instructions on how to identify the packets to which the SID list 877 is to be prepended. 879 By encoding the state in the packet, as is done in Segment Routing, 880 per-path state is transitioned out of the network. 882 However, there are a number of limitations in current SR, which limit 883 its applicability to enhanced VPNs: 885 o Segments are shared between different VPNs paths 887 o There is no reservation of bandwidth or other network resources 888 o There is limited differentiation in the data plane. 890 Thus some extensions to SR are needed to provide isolation between 891 different enhanced VPNs. This can be achieved by including a finer 892 granularity of state in the network in anticipation of its future use 893 by authorized services. We therefore need to evaluate the balance 894 between this additional state and the performance delivered by the 895 network. 897 With current segment routing, the instructions are used to specify 898 the nodes and links to be traversed. However, in order to achieve 899 the required isolation between different services, new instructions 900 can be created which can be prepended to a packet to steer it through 901 specific network resources and functions. 903 Traditionally, an SR traffic engineered path operates with a 904 granularity of a link with hints about priority provided through the 905 use of the traffic class (TC) field in the header. However to 906 achieve the latency and isolation characteristics that are sought by 907 the enhanced VPN users, steering packets through specific queues and 908 resources will likely be required. The extent to which these needs 909 can be satisfied through existing QoS mechanisms is to be determined. 910 What is clear is that a fine control of which services wait for 911 which, with a fine granularity of queue management policy is needed. 912 Note that the concept of a queue is a useful abstraction for many 913 types of underlay mechanism that may be used to provide enhanced 914 isolation and latency support. 916 From the perspective of the segment routing, the method of steering a 917 packet to a queue that provides the required properties is an 918 abstraction that hides the details of the underlying implementation. 919 How the queue satisfies the requirement is implementation specific 920 and is transparent to the control plane and data plane mechanisms 921 used. Thus, for example, a FlexE channel, or a time sensitive 922 networking packet scheduling slot are abstracted to the same concept 923 and bound to the data plane in a common manner. 925 We can also introduce such fine grained packet steering by specifying 926 the queues through an SR instruction list. Thus new SR instructions 927 may be created to specify not only which resources are traversed, but 928 in some cases how they are traversed. For example, it may be 929 possible to specify not only the queue to be used but the policy to 930 be applied when enqueuing and dequeuing. 932 This concept could be further generalized, since as well as queuing 933 to the output port of a router, it is possible to consider queuing 934 data to any resource, for example: 936 o A network processor unit (NPU) 938 o A central processing unit (CPU) Core 940 o A Look-up engine 942 Both SR-MPLS and SRv6 are candidate network layer technologies for 943 enhanced VPN. In some cases they can be supported by DetNet to meet 944 the packet loss, delay and jitter requirement of particular service. 945 However, currently the "pure" IP variant of DetNet 946 [I-D.ietf-detnet-dp-sol-ip] does not support the Packet Replication, 947 Elimination, and Re-ordering (PREOF) [I-D.ietf-detnet-architecture] 948 functions. How to provide the DetNet enhanced delivery in an SRv6 949 environment needs further study. 951 4.3. Non-Packet Technologies 953 Non-packet underlay data plane technologies often have TE properties 954 and behaviors, and meet many of the key requirements in particular 955 for bandwidth guarantees, traffic isolation (with physical isolation 956 often being an integral part of the technology), highly predictable 957 latency and jitter characteristics, measurable loss characteristics, 958 and ease of identification of flows (and hence slices). 960 The control and management planes for non-packet data plane 961 technologies have most in common with MPLS-TE (Section 4.2.2) and 962 offer the same set of advanced features [RFC3945]. Furthermore, 963 management techniques such as ACTN ([RFC8453] and Section 4.4) can be 964 used to aid in the reporting of underlying network topologies, and 965 the creation of virtual networks with the resource and properties 966 needed by the enhanced VPN services. 968 4.4. Control Plane 970 Enhanced VPN would likely be based on a hybrid control mechanism, 971 which takes advantage of the logically centralized controller for on- 972 demand provisioning and global optimization, whilst still relies on 973 distributed control plane to provide scalability, high reliability, 974 fast reaction, automatic failure recovery etc. Extension and 975 optimization to the distributed control plane is needed to support 976 the enhanced properties of VPN+. 978 RSVP-TE provides the signaling mechanism of establishing a TE-LSP 979 with end-to-end resource reservation. It can be used to bind the VPN 980 to specific network resource allocated within the underlay, but there 981 are the above mentioned scalability concerns. 983 SR does not have the capability of signaling the resource reservation 984 along the path, nor do its currently specified distributed link state 985 routing protocols. On the other hand, the SR approach provides a way 986 of efficiently binding the network underlay and the enhanced VPN 987 overlay, as it reduces the amount of state to be maintained in the 988 network. An SR-based approach with per-slice resource reservation 989 can easily create dedicated SR network slices, and the VPN services 990 can be bound to a particular SR network slice. A centralized 991 controller can perform resource planning and reservation from the 992 controller's point of view, but this does not ensure resource 993 reservation is actually done in the network nodes. Thus, if a 994 distributed control plane is needed, either in place of an SDN 995 controller or as an assistant to it, the design of the control system 996 needs to ensure that resources are uniquely allocated in the network 997 nodes for the correct services, and not allocated to other services 998 causing unintended resource conflict. 1000 In addition, in multi-domain and multi-layer networks, the 1001 centralized and distributed control mechanisms will be used for 1002 inter-domain coordination and inter-layer optimization, so that the 1003 diverse and customized enhanced VPN service requirement could be met. 1004 The detailed mechanisms will be described in a future version. 1006 4.5. Management Plane 1008 The management plane mechanisms for enhanced VPN can be based on the 1009 VPN service models as defined in [RFC8299] and [RFC8466], possible 1010 augmentations and extensions to these models may be needed, which is 1011 out of the scope of this document. 1013 Abstraction and Control of Traffic Engineered Networks (ACTN) 1014 [RFC8453] specifies the SDN based architecture for the control of TE 1015 networks. The ACTN related data models such as 1016 [I-D.ietf-teas-actn-vn-yang] and 1017 [I-D.ietf-teas-te-service-mapping-yang] can be applicable in the 1018 provisioning of enhanced VPN service. The details are described in 1019 Section 4.6. 1021 4.6. Applicability of ACTN to Enhanced VPN 1023 ACTN facilitates end-to-end connections and provides them to the 1024 user. The ACTN framework [RFC8453] highlights how: 1026 o Abstraction of the underlying network resources are provided to 1027 higher-layer applications and customers. 1029 o Virtualization of underlying resources, whose selection criterion 1030 is the allocation of those resources for the customer, 1031 application, or service. 1033 o Creation of a virtualized environment allowing operators to view 1034 and control multi-domain networks as a single virtualized network. 1036 o The presentation to customers of networks as a virtual network via 1037 open and programmable interfaces. 1039 The infrastructure managed through ACTN comprises traffic engineered 1040 network resources, which may include: 1042 o Statistical packet bandwidth. 1044 o Physical forwarding plane sources, such as: wavelengths and time 1045 slots. 1047 o Forwarding and cross-connect capabilities. 1049 The type of network virtualization enabled by ACTN provides customers 1050 and applications (tenants) with the capability to utilize and 1051 independently control allocated virtual network resources as if they 1052 were physically their own resources. 1054 An ACTN Virtual Network (VN) is a client view of the ACTN managed 1055 infrastructure, and is presented by the ACTN provider as a set of 1056 abstracted resources. 1058 Depending on the agreement between client and provider various VN 1059 operations and VN views are possible. 1061 o Virtual Network Creation: A VN could be pre-configured and created 1062 via static or dynamic request and negotiation between customer and 1063 provider. It must meet the specified SLA attributes which satisfy 1064 the customer's objectives. 1066 o Virtual Network Operations: The virtual network may be further 1067 modified and deleted based on customer request to request changes 1068 in the network resources reserved for the customer, and used to 1069 construct the network slice. The customer can further act upon 1070 the virtual network to manage traffic flow across the virtual 1071 network. 1073 o Virtual Network View: The VN topology from a customer point of 1074 view. These may be a variety of tunnels, or an entire VN 1075 topology. Such connections may comprise of customer end points, 1076 access links, intra-domain paths, and inter-domain links. 1078 Dynamic VN Operations allow a customer to modify or delete the VN. 1079 The customer can further act upon the virtual network to 1080 create/modify/delete virtual links and nodes. These changes will 1081 result in subsequent tunnel management in the operator's networks. 1083 4.6.1. ACTN Used for VPN+ Delivery 1085 ACTN provides VPN connections between multiple sites as requested via 1086 a VPN requestor enabled by the Customer Network Controller (CNC). 1087 The CNC is managed by the customer themselves, and interacts with the 1088 network provider's Multi-Domain Service Controller (MDSC). The 1089 Provisioning Network Controllers (PNC) remain entirely under the 1090 management of the network provider and are not visible to the 1091 customer. 1093 The benefits of this model include: 1095 o Provision of edge-to-edge VPN multi-access connectivity. 1097 o Management is mostly performed by the network provider, with some 1098 flexibility delegated to the customer-managed CNC. 1100 ---------------- ---------------- 1101 | Site-A Users |----------- ------------| Site-B Users | 1102 ---------------- | | ---------------- 1103 ------- 1104 | CNC | 1105 ------- 1106 Boundary | 1107 Between ==========================|========================== 1108 Customer & | 1109 Network Operator | 1110 --------------- 1111 | MDSC | 1112 --------------- 1113 _________/ | \__________ 1114 / | \ 1115 / | \ 1116 --------- --------- --------- 1117 | PNC | | PNC | | PNC | 1118 --------- --------- --------- 1119 | | / 1120 | | / 1121 ----- ----- ----- 1122 ( ) ( ) ( ) 1123 ---( Phys. )------------( Phys. )-------( Phys. )--- 1124 ( Net ) ( Net ) ( Net ) 1125 ----- ----- ----- 1127 Figure 3: VPN Delivery in the ACTN Architecture 1129 Figure 4 presents a more general representation of how multiple 1130 enhanced VPNs may be created from the resources of multiple physical 1131 networks using the CNC, MDSC, and PNC components of the ACTN 1132 architecture. Each enhanced VPN is controlled by its own CNC. The 1133 CNCs send requests to the provider's MDSC. The provider manages two 1134 physical networks each under the control of PNC. The MDSC asks the 1135 PNCs to allocate and provision resources to achieve the enhanced 1136 VPNs. In this figure, one enhanced VPN is constructed solely from 1137 the resources of one of the physical networks, while the the VPN uses 1138 resources from both physical networks. 1140 --------------- ( ) 1141 | CNC |---------->( VPN+ ) 1142 --------^------ ( ) 1143 | _(_________ _) 1144 --------------- ( ) ^ 1145 | CNC |----------->( VPN+ ) : 1146 ------^-------- ( ) : 1147 | | (___________) : 1148 | | ^ ^ : 1149 Boundary | | : : : 1150 Between ==========|====|===================:====:====:======== 1151 Customer & | | : : : 1152 Network Provider | | : : : 1153 v v : : : 1154 --------------- : :....: 1155 | MDSC | : : 1156 --------------- : : 1157 ^ ---^------ ... 1158 | ( ) . 1159 v ( Physical ) . 1160 ---------------- ( Network ) . 1161 | PNC |<-------->( ) ---^------ 1162 ---------------- | -------- ( ) 1163 | |-- ( Physical ) 1164 | PNC |<------------------------->( Network ) 1165 --------------- ( ) 1166 -------- 1168 Figure 4: Generic VPN+ Delivery in the ACTN Architecture 1170 4.6.2. Enhanced VPN Features with ACTN 1172 This section discusses how the features of ACTN can fulfill the 1173 enhanced VPN requirements described earlier in this document. As 1174 previously noted, key requirements of the enhanced VPN include: 1176 1. Isolation between VPNs 1178 2. Guaranteed Performance 1180 3. Integration 1182 4. Dynamic Configuration 1184 5. Customized Control Plane 1186 The subsections that follow outline how each requirement is met using 1187 ACTN. 1189 4.6.2.1. Isolation Between VPNs 1191 The ACTN VN YANG model [I-D.ietf-teas-actn-vn-yang] and the TE- 1192 service mapping model [I-D.ietf-teas-te-service-mapping-yang] fulfill 1193 the VPN isolation requirement by providing the following features for 1194 the VNs: 1196 o Each VN is identified with a unique identifier (vn-id and vn-name) 1197 and so is each VN member that belongs to the VN (vn-member-id). 1199 o Each instantiated VN is managed and controlled independent of 1200 other VNs in the network with proper protection level 1201 (protection). 1203 o Each VN is instantiated with an isolation requirement described by 1204 the TE-service mapping model 1205 [I-D.ietf-teas-te-service-mapping-yang]. This mapping supports: 1207 * Hard isolation with deterministic characteristics (e.g., this 1208 case may need an optical bypass tunnel or a DetNet/TSN tunnel 1209 to guarantee latency with no jitter) 1211 * Hard isolation (i.e., dedicated TE resources in all underlays) 1213 * Soft isolation (i.e., resource in some layer may be shared 1214 while in some other layers is dedicated). 1216 * No isolation (i.e., sharing with other VN). 1218 4.6.2.2. Guaranteed Performance 1220 Performance objectives of a VN need first to be expressed in order to 1221 assure the performance guarantee. [I-D.ietf-teas-actn-vn-yang] and 1222 [I-D.ietf-teas-yang-te-topo] allow configuration of several 1223 parameters that may affect the VN performance objectives as follows: 1225 o Bandwidth 1227 o Objective function (e.g., min cost path, min load path, etc.) 1229 o Metric Types and their threshold: 1231 * TE cost, IGP cost, Hop count, or Unidirectional Delay (e.g., 1232 can set all path delay <= threshold) 1234 Once these requests are instantiated, the resources are committed and 1235 guaranteed through the life cycle of the VN. 1237 4.6.2.3. Integration 1239 ACTN provides mechanisms to correlate customer's VN and the actual TE 1240 tunnels instantiated in the provider's network. Specifically: 1242 o Link each VN member to actual TE tunnel. 1244 o Each VN can be monitored on a various level such as VN level, VN 1245 member level, TE-tunnel level, and link/node level. 1247 Service function integration with network topology (L3 and TE 1248 topology) is in progress in [I-D.ietf-teas-sf-aware-topo-model]. 1249 Specifically, [I-D.ietf-teas-sf-aware-topo-model] addresses a number 1250 of use-cases that show how TE topology supports various service 1251 functions. 1253 4.6.2.4. Dynamic Configuration 1255 ACTN provides an architecture that allows the CNC to interact with 1256 the MDSC which is network provider's SDN controller. This gives the 1257 customer control of their VNs. 1259 Specifically, the ACTN VN model [I-D.ietf-teas-actn-vn-yang] allows 1260 the VN to create, modify, and delete VNs. 1262 4.6.2.5. Customized Control 1264 ACTN provides a YANG model that allows the CNC to control a VN as a 1265 "Type 2 VN" that allows the customer to provision tunnels that 1266 connect their endpoints over the customized VN topology. 1268 For some VN members, the customers are allowed to configure the path 1269 (i.e., the sequence of virtual nodes and virtual links) over the VN/ 1270 abstract topology. 1272 5. Scalability Considerations 1274 Enhanced VPN provides the performance guaranteed services in packet 1275 networks, but with the potential cost of introducing additional 1276 states into the network. There are at least three ways that this 1277 adding state might be presented in the network: 1279 o Introduce the complete state into the packet, as is done in SR. 1280 This allows the controller to specify the detailed series of 1281 forwarding and processing instructions for the packet as it 1282 transits the network. The cost of this is an increase in the 1283 packet header size. The cost is also that systems will have 1284 capabilities enabled in case they are called upon by a service. 1286 This is a type of latent state, and increases as we more precisely 1287 specify the path and resources that need to be exclusively 1288 available to a VPN. 1290 o Introduce the state to the network. This is normally done by 1291 creating a path using RSVP-TE, which can be extended to introduce 1292 any element that needs to be specified along the path, for example 1293 explicitly specifying queuing policy. It is of course possible to 1294 use other methods to introduce path state, such as via a Software 1295 Defined Network (SDN) controller, or possibly by modifying a 1296 routing protocol. With this approach there is state per path per 1297 path characteristic that needs to be maintained over its life- 1298 cycle. This is more state than is needed using SR, but the packet 1299 are shorter. 1301 o Provide a hybrid approach based on using binding SIDs to create 1302 path fragments, and bind them together with SR. 1304 Dynamic creation of a VPN path using SR requires less state 1305 maintenance in the network core at the expense of larger VPN headers 1306 on the packet. The packet size can be lower if a form of loose 1307 source routing is used (using a few nodal SIDs), and it will be lower 1308 if no specific functions or resource on the routers are specified. 1309 Reducing the state in the network is important to enhanced VPN, as it 1310 requires the overlay to be more closely integrated with the underlay 1311 than with traditional VPNs. This tighter coupling would normally 1312 mean that more state needed to be created and maintained in the 1313 network, as the state about fine granularity processing would need to 1314 be loaded and maintained in the routers. However, a segment routed 1315 approach allows much of this state to be spread amongst the network 1316 ingress nodes, and transiently carried in the packets as SIDs. 1318 These approaches are for further study. 1320 5.1. Maximum Stack Depth of SR 1322 One of the challenges with SR is the stack depth that nodes are able 1323 to impose on packets [RFC8491]. This leads to a difficult balance 1324 between adding state to the network and minimizing stack depth, or 1325 minimizing state and increasing the stack depth. 1327 5.2. RSVP Scalability 1329 The traditional method of creating a resource allocated path through 1330 an MPLS network is to use the RSVP protocol. However there have been 1331 concerns that this requires significant continuous state maintenance 1332 in the network. There are ongoing works to improve the scalability 1333 of RSVP-TE LSPs in the control plane [RFC8370]. 1335 There is also concern at the scalability of the forwarder footprint 1336 of RSVP as the number of paths through an LSR grows [RFC8577] 1337 proposes to address this by employing SR within a tunnel established 1338 by RSVP-TE. 1340 6. OAM Considerations 1342 A study of OAM in SR networks has been documented in [RFC8403]. 1344 The enhanced VPN OAM design needs to consider the following 1345 requirements: 1347 o Instrumentation of the underlay so that the network operator can 1348 be sure that the resources committed to a tenant are operating 1349 correctly and delivering the required performance. 1351 o Instrumentation of the overlay by the tenant. This is likely to 1352 be transparent to the network operator and to use existing 1353 methods. Particular consideration needs to be given to the need 1354 to verify the isolation and the various committed performance 1355 characteristics. 1357 o Instrumentation of the overlay by the network provider to 1358 proactively demonstrate that the committed performance is being 1359 delivered. This needs to be done in a non-intrusive manner, 1360 particularly when the tenant is deploying a performance sensitive 1361 application 1363 o Verification of the conformity of the path to the service 1364 requirement. This may need to be done as part of a commissioning 1365 test. 1367 These issues will be discussed in a future version of this document. 1369 7. Enhanced Resiliency 1371 Each enhanced VPN has a life-cycle, and needs modification during 1372 deployment as the needs of its tenant change. Additionally, as the 1373 network as a whole evolves, there will need to be garbage collection 1374 performed to consolidate resources into usable quanta. 1376 Systems in which the path is imposed such as SR, or some form of 1377 explicit routing tend to do well in these applications, because it is 1378 possible to perform an atomic transition from one path to another. 1379 This is a single action by the head-end changes the path without the 1380 need for coordinated action by the routers along the path. However, 1381 implementations and the monitoring protocols need to make sure that 1382 the new path is up and meet the required SLA before traffic is 1383 transitioned to it. It is possible for deadlocks arise as a result 1384 of the network becoming fragmented over time, such that it is 1385 impossible to create a new path or modify a existing path without 1386 impacting the SLA of other paths. Resolution of this situation is as 1387 much a commercial issue as it is a technical issue and is outside the 1388 scope of this document. 1390 There are however two manifestations of the latency problem that are 1391 for further study in any of these approaches: 1393 o The problem of packets overtaking one and other if a path latency 1394 reduces during a transition. 1396 o The problem of the latency transient in either direction as a path 1397 migrates. 1399 There is also the matter of what happens during failure in the 1400 underlay infrastructure. Fast reroute is one approach, but that 1401 still produces a transient loss with a normal goal of rectifying this 1402 within 50ms [RFC5654] . An alternative is some form of N+1 delivery 1403 such as has been used for many years to support protection from 1404 service disruption. This may be taken to a different level using the 1405 techniques proposed by the IETF deterministic network work with 1406 multiple in-network replication and the culling of later packets 1407 [I-D.ietf-detnet-architecture]. 1409 In addition to the approach used to protect high priority packets, 1410 consideration has to be given to the impact of best effort traffic on 1411 the high priority packets during a transient. Specifically if a 1412 conventional re-convergence process is used there will inevitably be 1413 micro-loops and whilst some form of explicit routing will protect the 1414 high priority traffic, lower priority traffic on best effort shortest 1415 paths will micro-loop without the use of a loop prevention 1416 technology. To provide the highest quality of service to high 1417 priority traffic, either this traffic must be shielded from the 1418 micro-loops, or micro-loops must be prevented. 1420 8. Security Considerations 1422 All types of virtual network require special consideration to be 1423 given to the isolation between the tenants. In this regard enhanced 1424 VPNs neither introduce, no experience a greater security risk than 1425 another VPN of the same base type. However, in an enhanced virtual 1426 network service the isolation requirement needs to be considered. If 1427 a service requires a specific latency then it can be damaged by 1428 simply delaying the packet through the activities of another tenant. 1429 In a network with virtual functions, depriving a function used by 1430 another tenant of compute resources can be just as damaging as 1431 delaying transmission of a packet in the network. The measures to 1432 address these dynamic security risks must be specified as part to the 1433 specific solution. 1435 9. IANA Considerations 1437 There are no requested IANA actions. 1439 10. Contributors 1441 Daniel King 1442 Email: daniel@olddog.co.uk 1444 Adrian Farrel 1445 Email: adrian@olddog.co.uk 1447 Jeff Tansura 1448 Email: jefftant.ietf@gmail.com 1450 Qin Wu 1451 Email: bill.wu@huawei.com 1453 Daniele Ceccarelli 1454 Email: daniele.ceccarelli@ericsson.com 1456 Mohamed Boucadair 1457 Email: mohamed.boucadair@orange.com 1459 Sergio Belotti 1460 Email: sergio.belotti@nokia.com 1462 Haomian Zheng 1463 Email: zhenghaomian@huawei.com 1465 11. Acknowledgements 1467 The authors would like to thank Charlie Perkins, James N Guichard and 1468 John E Drake for their review and valuable comments. 1470 This work was supported in part by the European Commission funded 1471 H2020-ICT-2016-2 METRO-HAUL project (G.A. 761727). 1473 12. References 1474 12.1. Normative References 1476 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1477 Requirement Levels", BCP 14, RFC 2119, 1478 DOI 10.17487/RFC2119, March 1997, 1479 . 1481 12.2. Informative References 1483 [BBF-SD406] 1484 "BBF SD-406: End-to-End Network Slicing", 2016, 1485 . 1488 [DETNET] "Deterministic Networking", March , 1489 . 1491 [FLEXE] "Flex Ethernet Implementation Agreement", March 2016, 1492 . 1495 [I-D.ietf-detnet-architecture] 1496 Finn, N., Thubert, P., Varga, B., and J. Farkas, 1497 "Deterministic Networking Architecture", draft-ietf- 1498 detnet-architecture-13 (work in progress), May 2019. 1500 [I-D.ietf-detnet-dp-sol-ip] 1501 Korhonen, J. and B. Varga, "DetNet IP Data Plane 1502 Encapsulation", draft-ietf-detnet-dp-sol-ip-02 (work in 1503 progress), March 2019. 1505 [I-D.ietf-detnet-use-cases] 1506 Grossman, E., "Deterministic Networking Use Cases", draft- 1507 ietf-detnet-use-cases-20 (work in progress), December 1508 2018. 1510 [I-D.ietf-teas-actn-vn-yang] 1511 Lee, Y., Dhody, D., Ceccarelli, D., Bryskin, I., and B. 1512 Yoon, "A Yang Data Model for VN Operation", draft-ietf- 1513 teas-actn-vn-yang-06 (work in progress), July 2019. 1515 [I-D.ietf-teas-sf-aware-topo-model] 1516 Bryskin, I., Liu, X., Lee, Y., Guichard, J., Contreras, 1517 L., Ceccarelli, D., and J. Tantsura, "SF Aware TE Topology 1518 YANG Model", draft-ietf-teas-sf-aware-topo-model-03 (work 1519 in progress), March 2019. 1521 [I-D.ietf-teas-te-service-mapping-yang] 1522 Lee, Y., Dhody, D., Ceccarelli, D., Tantsura, J., 1523 Fioccola, G., and Q. Wu, "Traffic Engineering and Service 1524 Mapping Yang Model", draft-ietf-teas-te-service-mapping- 1525 yang-01 (work in progress), March 2019. 1527 [I-D.ietf-teas-yang-te-topo] 1528 Liu, X., Bryskin, I., Beeram, V., Saad, T., Shah, H., and 1529 O. Dios, "YANG Data Model for Traffic Engineering (TE) 1530 Topologies", draft-ietf-teas-yang-te-topo-22 (work in 1531 progress), June 2019. 1533 [NGMN-NS-Concept] 1534 "NGMN NS Concept", 2016, . 1538 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1539 and W. Weiss, "An Architecture for Differentiated 1540 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1541 . 1543 [RFC2764] Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. 1544 Malis, "A Framework for IP Based Virtual Private 1545 Networks", RFC 2764, DOI 10.17487/RFC2764, February 2000, 1546 . 1548 [RFC2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path 1549 Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000, 1550 . 1552 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1553 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1554 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1555 . 1557 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1558 Conrad, "Stream Control Transmission Protocol (SCTP) 1559 Partial Reliability Extension", RFC 3758, 1560 DOI 10.17487/RFC3758, May 2004, 1561 . 1563 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1564 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 1565 RFC 3931, DOI 10.17487/RFC3931, March 2005, 1566 . 1568 [RFC3945] Mannie, E., Ed., "Generalized Multi-Protocol Label 1569 Switching (GMPLS) Architecture", RFC 3945, 1570 DOI 10.17487/RFC3945, October 2004, 1571 . 1573 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1574 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1575 DOI 10.17487/RFC3985, March 2005, 1576 . 1578 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1579 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1580 2006, . 1582 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 1583 "Encapsulation Methods for Transport of Ethernet over MPLS 1584 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 1585 . 1587 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 1588 Guidelines for DiffServ Service Classes", RFC 4594, 1589 DOI 10.17487/RFC4594, August 2006, 1590 . 1592 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 1593 2 Virtual Private Networks (L2VPNs)", RFC 4664, 1594 DOI 10.17487/RFC4664, September 2006, 1595 . 1597 [RFC4719] Aggarwal, R., Ed., Townsley, M., Ed., and M. Dos Santos, 1598 Ed., "Transport of Ethernet Frames over Layer 2 Tunneling 1599 Protocol Version 3 (L2TPv3)", RFC 4719, 1600 DOI 10.17487/RFC4719, November 2006, 1601 . 1603 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1604 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1605 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1606 September 2009, . 1608 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 1609 Networking: A Perspective from within a Service Provider 1610 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 1611 . 1613 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 1614 Henderickx, W., and A. Isaac, "Requirements for Ethernet 1615 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 1616 . 1618 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 1619 Ceccarelli, D., and X. Zhang, "Problem Statement and 1620 Architecture for Information Exchange between 1621 Interconnected Traffic-Engineered Networks", BCP 206, 1622 RFC 7926, DOI 10.17487/RFC7926, July 2016, 1623 . 1625 [RFC8299] Wu, Q., Ed., Litkowski, S., Tomotaki, L., and K. Ogaki, 1626 "YANG Data Model for L3VPN Service Delivery", RFC 8299, 1627 DOI 10.17487/RFC8299, January 2018, 1628 . 1630 [RFC8370] Beeram, V., Ed., Minei, I., Shakir, R., Pacella, D., and 1631 T. Saad, "Techniques to Improve the Scalability of RSVP-TE 1632 Deployments", RFC 8370, DOI 10.17487/RFC8370, May 2018, 1633 . 1635 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1636 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1637 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1638 July 2018, . 1640 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1641 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1642 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1643 2018, . 1645 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1646 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1647 DOI 10.17487/RFC8453, August 2018, 1648 . 1650 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1651 Data Model for Layer 2 Virtual Private Network (L2VPN) 1652 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1653 2018, . 1655 [RFC8491] Tantsura, J., Chunduri, U., Aldrin, S., and L. Ginsberg, 1656 "Signaling Maximum SID Depth (MSD) Using IS-IS", RFC 8491, 1657 DOI 10.17487/RFC8491, November 2018, 1658 . 1660 [RFC8577] Sitaraman, H., Beeram, V., Parikh, T., and T. Saad, 1661 "Signaling RSVP-TE Tunnels on a Shared MPLS Forwarding 1662 Plane", RFC 8577, DOI 10.17487/RFC8577, April 2019, 1663 . 1665 [SFC] "Service Function Chaining", March , 1666 . 1668 [TS23501] "3GPP TS23.501", 2016, 1669 . 1672 [TS28530] "3GPP TS28.530", 2016, 1673 . 1676 [TSN] "Time-Sensitive Networking", March , 1677 . 1679 Authors' Addresses 1681 Jie Dong 1682 Huawei 1684 Email: jie.dong@huawei.com 1686 Stewart Bryant 1687 Huawei 1689 Email: stewart.bryant@gmail.com 1691 Zhenqiang Li 1692 China Mobile 1694 Email: lizhenqiang@chinamobile.com 1696 Takuya Miyasaka 1697 KDDI Corporation 1699 Email: ta-miyasaka@kddi.com 1700 Young Lee 1701 Futurewei 1703 Email: younglee.tx@gmail.com