idnits 2.17.1 draft-dong-teas-enhanced-vpn-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 15, 2018) is 1979 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2119' is defined on line 1407, but no explicit reference was found in the text == Outdated reference: A later version (-13) exists of draft-ietf-detnet-architecture-09 == Outdated reference: A later version (-02) exists of draft-ietf-detnet-dp-sol-ip-01 == Outdated reference: A later version (-20) exists of draft-ietf-detnet-use-cases-19 == Outdated reference: A later version (-24) exists of draft-ietf-teas-actn-vn-yang-02 == Outdated reference: A later version (-12) exists of draft-ietf-teas-sf-aware-topo-model-02 == Outdated reference: A later version (-22) exists of draft-ietf-teas-yang-te-topo-18 == Outdated reference: A later version (-13) exists of draft-lee-teas-te-service-mapping-yang-12 Summary: 0 errors (**), 0 flaws (~~), 9 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group J. Dong 3 Internet-Draft S. Bryant 4 Intended status: Informational Huawei 5 Expires: May 19, 2019 Z. Li 6 China Mobile 7 T. Miyasaka 8 KDDI Corporation 9 Y. Lee 10 Huawei 11 November 15, 2018 13 A Framework for Enhanced Virtual Private Networks (VPN+) Service 14 draft-dong-teas-enhanced-vpn-03 16 Abstract 18 This document specifies a framework for using existing, modified and 19 potential new networking technologies as components to provide an 20 Enhanced Virtual Private Networks (VPN+) service. The purpose is to 21 enable VPNs to support the needs of new applications, particularly 22 applications that are associated with 5G services. Typically, VPN+ 23 can be used to form the underpinning of network slicing, but will 24 also be of use in its own right. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at https://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on May 19, 2019. 43 Copyright Notice 45 Copyright (c) 2018 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (https://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 2. Overview of the Requirements . . . . . . . . . . . . . . . . 5 62 2.1. Isolation between Virtual Networks . . . . . . . . . . . 5 63 2.1.1. A Pragmatic Approach to Isolation . . . . . . . . . . 6 64 2.2. Performance Guarantee . . . . . . . . . . . . . . . . . . 7 65 2.3. Integration . . . . . . . . . . . . . . . . . . . . . . . 9 66 2.3.1. Abstraction . . . . . . . . . . . . . . . . . . . . . 9 67 2.4. Dynamic Configuration . . . . . . . . . . . . . . . . . . 10 68 2.5. Customized Control . . . . . . . . . . . . . . . . . . . 10 69 2.6. Applicability . . . . . . . . . . . . . . . . . . . . . . 11 70 3. Architecture of Enhanced VPN . . . . . . . . . . . . . . . . 11 71 3.1. Layered Architecture . . . . . . . . . . . . . . . . . . 12 72 3.2. Multi-Point to Multi-Point . . . . . . . . . . . . . . . 14 73 3.3. Application Specific Network Types . . . . . . . . . . . 14 74 4. Candidate Technologies . . . . . . . . . . . . . . . . . . . 14 75 4.1. Underlay Packet and Frame-Based Data Planes . . . . . . . 15 76 4.1.1. FlexE . . . . . . . . . . . . . . . . . . . . . . . . 15 77 4.1.2. Dedicated Queues . . . . . . . . . . . . . . . . . . 16 78 4.1.3. Time Sensitive Networking . . . . . . . . . . . . . . 16 79 4.2. Packet and Frame-Based Network Layer . . . . . . . . . . 16 80 4.2.1. Deterministic Networking . . . . . . . . . . . . . . 17 81 4.2.2. MPLS Traffic Engineering (MPLS-TE) . . . . . . . . . 17 82 4.2.3. Segment Routing . . . . . . . . . . . . . . . . . . . 17 83 4.3. Non-Packet Technologies . . . . . . . . . . . . . . . . . 19 84 4.4. Control Plane . . . . . . . . . . . . . . . . . . . . . . 20 85 4.5. Management Plane . . . . . . . . . . . . . . . . . . . . 20 86 4.6. Applicability of ACTN to Enhanced VPN . . . . . . . . . . 21 87 4.6.1. ACTN Used for VPN+ Delivery . . . . . . . . . . . . . 22 88 4.6.2. Enhanced VPN Features with ACTN . . . . . . . . . . . 24 89 5. Scalability Considerations . . . . . . . . . . . . . . . . . 26 90 5.1. Maximum Stack Depth of SR . . . . . . . . . . . . . . . . 27 91 5.2. RSVP Scalability . . . . . . . . . . . . . . . . . . . . 27 92 6. OAM Considerations . . . . . . . . . . . . . . . . . . . . . 28 93 7. Enhanced Resiliency . . . . . . . . . . . . . . . . . . . . . 28 94 8. Security Considerations . . . . . . . . . . . . . . . . . . . 29 95 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 96 10. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 30 97 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 98 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 31 99 12.1. Normative References . . . . . . . . . . . . . . . . . . 31 100 12.2. Informative References . . . . . . . . . . . . . . . . . 31 101 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35 103 1. Introduction 105 Virtual private networks (VPNs) have served the industry well as a 106 means of providing different groups of users with logically isolated 107 access to a common network. The common or base network that is used 108 to provide the VPNs is often referred to as the underlay, and the VPN 109 is often called an overlay. 111 Customers of a network operator may request enhanced VPN services 112 with additional characteristics such as complete isolation from other 113 VPNs so that changes in network load have no effect on the throughput 114 or latency of the service provided to the customer. 116 Driven largely by needs surfacing from 5G, the concept of network 117 slicing has gained traction [NGMN-NS-Concept] [TS23501] [TS28530] 118 [BBF-SD406]. Network slicing requires the underlying network to 119 support partitioning the network resources to provide the client with 120 dedicated (private) networking, computing, and storage resources 121 drawn from a shared pool. The slices may be seen as (and operated 122 as) virtual networks. 124 Network abstraction is a technique that can be applied to a network 125 domain to select network resources by policy to obtain a view of 126 potential connectivity and a set of service functions. 128 Network slicing is an approach to network operations that builds on 129 the concept of network abstraction to provide programmability, 130 flexibility, and modularity. It may use techniques such as Software 131 Defined Networking (SDN) [RFC7149] and Network Function 132 Virtualization (NFV) to create multiple logical (virtual) networks, 133 each tailored for a set of services or a particular tenant that share 134 the same set of requirements, on top of a common network. How the 135 network slices are engineered can be deployment-specific. 137 Thus, there is a need to create virtual networks with enhanced 138 characteristics. The tenant of such a virtual network can require a 139 degree of isolation and performance that previously could only be 140 satisfied by dedicated networks. Additionally, the tenant may ask 141 for some level of control to their virtual networks, e.g., to 142 customize the service forwarding paths in a network slice. 144 These enhanced properties cannot be met with pure overlay networks, 145 as they require tighter coordination and integration between the 146 underlay and the overlay network. This document introduces a new 147 network service called Enhanced VPN: VPN+. VPN+ refers to a virtual 148 network which has dedicated network resources, including invoked 149 service functions, allocated from the underlay network. Unlike a 150 traditional VPN, an enhanced VPN can achieve greater isolation with 151 strict guaranteed performance. These new properties, which have 152 general applicability, may also be of interest as part of a network 153 slicing solution. 155 This document specifies a framework for using existing, modified and 156 potential new networking technologies as components to provide a VPN+ 157 service. Specifically we are concerned with: 159 o The design of the enhanced data plane. 161 o The necessary protocols in both underlay and the overlay of 162 enhanced VPN. 164 o The mechanisms to achieve integration between overlay and 165 underlay. 167 o The necessary Operation, Administration and Management (OAM) 168 methods to instrument an enhanced VPN to make sure that the 169 required Service Level Agreement (SLA) are met, and to take any 170 corrective action to avoid SLA violation, such as switching to an 171 alternate path. 173 The required network layered structure to achieve this is shown in 174 Section 3.1. 176 Note that, in this document, the four terms "VPN", "Enhanced VPN" (or 177 "VPN+"), "Virtual Network (VN)", and "Network Slice" may be 178 considered as describing similar concepts dependent on the viewpoint 179 from which they are used. 181 o An enhanced VPN is clearly a form of VPN, but with additional 182 service-specific commitments. 184 o A VN is a type of service that connects customer edge points with 185 the additional possibility of requesting further service 186 characteristics in the manner of an enhanced VPN. 188 o An enhanced VPN or VN is made by creating a slice through the 189 resources of the underlay network. 191 o The general concept of network slicing in a TE network is a larger 192 problem space than is addressed by VPN+ or VN, but those concepts 193 are tools to address some aspects or realizations of network 194 slicing. 196 2. Overview of the Requirements 198 In this section we provide an overview of the requirements of an 199 enhanced VPN. 201 2.1. Isolation between Virtual Networks 203 Isolation is a feature requested by some particular customers in the 204 network. Such feature is offered by a network operator where the 205 traffic from one service instance is isolated from the traffic of 206 other services. There are different grades of isolation that range 207 from simple separation of traffic on delivery (ensuring that traffic 208 is not delivered to the wrong customer) all the way to complete 209 separation within the underlay so that the traffic from different 210 services use distinct network resources. 212 The terms hard and soft isolation are introduced to give example of 213 different isolation cases. A VPN has soft isolation if the traffic 214 of one VPN cannot be received by the customers of another VPN. Both 215 IP and MPLS VPNs are examples of soft isolated VPNs because the 216 network delivers the traffic only to the required VPN endpoints. 217 However, the traffic from one or more VPNs and regular network 218 traffic may congest the network resulting in packet loss and delay 219 for other VPNs operating normally. The ability for a VPN to be 220 sheltered from this effect is called hard isolation, and this 221 property is required by some critical applications. 223 The requirement is for an operator to provide both hard and soft 224 isolation between the tenants/applications using one enhanced VPN and 225 the tenants/applications using another enhanced VPN. Hard isolation 226 is needed so that applications with exacting requirements can 227 function correctly, despite other demands (perhaps a burst on another 228 VPN) competing for the underlying resources. In practice isolation 229 may be offered as a spectrum between soft and hard. 231 An example of hard isolation is a network supporting both emergency 232 services and public broadband multi-media services. During a major 233 incident the VPNs supporting these services would both be expected to 234 experience high data volumes, and it is important that both make 235 progress in the transmission of their data. In these circumstances 236 the VPNs would require an appropriate degree of isolation to be able 237 to continue to operate acceptably. 239 In order to provide the required isolation, resources may have to be 240 reserved in the data plane of the underlay network and dedicated to 241 traffic from a specific VPN. This may introduce scalability 242 concerns, thus some trade-off needs to be considered to provide the 243 required isolation between network slices while still allowing 244 reasonable sharing inside each network slice. 246 An optical layer can offer a high degree of isolation, at the cost of 247 allocating resources on a long term and end-to-end basis. Such an 248 arrangement means that the full cost of the resources must be borne 249 by the service that is allocated with the resources. On the other 250 hand, where adequate isolation can be achieved at the packet layer, 251 this permits the resources to be shared amongst many services and 252 only dedicated to a service on a temporary basis. This in turn, 253 allows greater statistical multiplexing of network resources and thus 254 amortizes the cost over many services, leading to better economy. 255 However, the degree of isolation required by network slicing cannot 256 be entirely met with existing mechanisms such as Traffic Engineered 257 Label Switched Paths (TE-LSPs). This is because most implementations 258 enforce the bandwidth in the data-plane only at the PEs, but at the P 259 routers the bandwidth is only reserved in the control plane, thus 260 bursts of data can accidentally occur at a P router with higher than 261 committed data rate. 263 There are several new technologies that provide some assistance with 264 these data plane issues. Firstly there is the IEEE project on Time 265 Sensitive Networking [TSN] which introduces the concept of packet 266 scheduling of delay and loss sensitive packets. Then there is 267 [FLEXE] which provides the ability to multiplex multiple channels 268 over one or more Ethernet links in a way that provides hard 269 isolation. Finally there are advanced queueing approaches which 270 allow the construction of virtual sub-interfaces, each of which is 271 provided with dedicated resource in a shared physical interface. 272 These approaches are described in more detail later in this document. 274 In the remainder of this section we explore how isolation may be 275 achieved in packet networks. 277 2.1.1. A Pragmatic Approach to Isolation 279 A key question is whether it is possible to achieve hard isolation in 280 packet networks, which were never designed to support hard isolation. 281 On the contrary, they were designed to provide statistical 282 multiplexing, a significant economic advantage when compared to a 283 dedicated, or a Time Division Multiplexing (TDM) network. However 284 there is no need to provide any harder isolation than is required by 285 the application. Pseudowires [RFC3985] emulate services that would 286 have had hard isolation in their native form. An approximation to 287 this requirement is sufficient in most cases. 289 Thus, for example, using FlexE or a channelized sub-interface 290 together with packet scheduling as interface slicing, optionally 291 along with the slicing of node resources, a type of hard isolation 292 can be provided that is adequate for many VPN+ applications. Other 293 applications may be either satisfied with a classical VPN with or 294 without reserved bandwidth, or may need dedicated point to point 295 fiber. The needs of each application must be quantified in order to 296 provide an economic solution that satisfies those needs without over- 297 engineering. 299 This spectrum of isolation is shown in Figure 1: 301 O=================================================O 302 | \---------------v---------------/ 303 Statistical Pragmatic Absolute 304 Multiplexing Isolation Isolation 305 (Traditional VPNs) (Enhanced VPN) (Dedicated Network) 307 Figure 1: The Spectrum of Isolation 309 At one end of the above figure, we have traditional statistical 310 multiplexing technologies that support VPNs. This is a service type 311 that has served the industry well and will continue to do so. At the 312 opposite end of the spectrum we have the absolute isolation provided 313 by traditional transport networks. The goal of enhanced VPN is 314 pragmatic isolation. This is isolation that is better than is 315 obtainable from pure statistical multiplexing, more cost effective 316 and flexible than a dedicated network, but which is a practical 317 solution that is good enough for the majority of applications. 319 2.2. Performance Guarantee 321 There are several kinds of performance guarantees, including 322 guaranteed maximum packet loss, guaranteed maximum delay and 323 guaranteed delay variation. Note that these guarantees apply to the 324 conformance traffic, the out-of-profile traffic will be handled 325 following other requirements. 327 Guaranteed maximum packet loss is a common parameter, and is usually 328 addressed by setting the packet priorities, queue size and discard 329 policy. However this becomes more difficult when the requirement is 330 combined with the latency requirement. The limiting case is zero 331 congestion loss, and that is the goal of the Deterministic Networking 332 work that the IETF [DETNET] and IEEE [TSN] are pursuing. In modern 333 optical networks, loss due to transmission errors is already 334 approaches zero, but there are the possibilities of failure of the 335 interface or the fiber itself. This can only be addressed by some 336 form of signal duplication and transmission over diverse paths. 338 Guaranteed maximum latency is required in a number of applications 339 particularly real-time control applications and some types of virtual 340 reality applications. The work of the IETF Deterministic Networking 341 (DetNet) Working Group [DETNET] is relevant; however the scope needs 342 to be extended to methods of enhancing the underlay to better support 343 the delay guarantee, and to integrate these enhancements with the 344 overall service provision. 346 Guaranteed maximum delay variation is a service that may also be 347 needed. [I-D.ietf-detnet-use-cases] calls up a number of cases where 348 this is needed, for example electrical utilities have an operational 349 need for this. Time transfer is one example of a service that needs 350 this, although it is in the nature of time that the service might be 351 delivered by the underlay as a shared service and not provided 352 through different virtual networks. Alternatively a dedicated 353 virtual network may be used to provide this as a shared service. 355 This suggests that a spectrum of service guarantee be considered when 356 deploying an enhanced VPN. As a guide to understanding the design 357 requirements we can consider four types: 359 o Best effort 361 o Assured bandwidth 363 o Guaranteed latency 365 o Enhanced delivery 367 Best effort service is the basic service that current VPNs can 368 provide. 370 An assured bandwidth service is one in which the bandwidth over some 371 period of time is assured, this can be achieved either simply based 372 on best effort with over-capacity provisioning, or it can be based on 373 TE-LSPs with bandwidth reservation. The instantaneous bandwidth is 374 however, not necessarily assured, depending on the technique used. 375 Providing assured bandwidth to VPNs, for example by using TE-LSPs, is 376 not widely deployed at least partially due to scalability concerns. 377 Guaranteed latency and enhanced delivery are not yet integrated with 378 VPNs. 380 A guaranteed latency service has a latency upper bound provided by 381 the network. Assuring the upper bound is more important than 382 achieving the minimum latency. 384 In Section 2.1 we considered the work of the IEEE Time Sensitive 385 Networking (TSN) project [TSN] and the work of the IETF DetNet 386 Working group [DETNET] in the context of isolation. The TSN and 387 DetNet work is of greater relevance in assuring end-to-end packet 388 latency. It is also of importance in considering enhanced delivery. 390 An enhanced delivery service is one in which the underlay network (at 391 layer 3) attempts to deliver the packet through multiple paths in the 392 hope of eliminating packet loss due to equipment or media failures. 394 It is these last two characteristics that an enhanced VPN adds to a 395 VPN service. 397 Flex Ethernet [FLEXE] is a useful underlay to provide these 398 guarantees. This is a method of providing time-slot based 399 channelization over an Ethernet bearer. Such channels are fully 400 isolated from other channels running over the same Ethernet bearer. 401 As noted elsewhere this produces hard isolation but makes the 402 reclamation of unused bandwidth more difficult. 404 These approaches can be used in tandem. It is possible to use FlexE 405 to provide tenant isolation, and then to use the TSN/Detnet approach 406 to provide a performance guarantee inside the a slice or tenant VPN. 408 2.3. Integration 410 A solution to the enhanced VPN problem has to provide close 411 integration of both overlay VPN and the underlay network resource. 412 This needs be done in a flexible and scalable way so that it can be 413 widely deployed in operator networks to support a reasonable number 414 of enhanced VPN customers. 416 Taking mobile networks and in particular 5G into consideration, the 417 integration of network and the service functions is a likely 418 requirement. The work in IETF SFC working group [SFC] provides a 419 foundation for this integration. 421 2.3.1. Abstraction 423 Integration of the overlay VPN and the underlay network resources 424 does not need to be a tight mapping. As described in [RFC7926], 425 abstraction is the process of applying policy to a set of information 426 about a TE network to produce selective information that represents 427 the potential ability to connect across the network. The process of 428 abstraction presents the connectivity graph in a way that is 429 independent of the underlying network technologies, capabilities, and 430 topology so that the graph can be used to plan and deliver network 431 services in a uniform way. 433 Virtual networks can be built on top of an abstracted topology that 434 represents the connectivity capabilities of the underlay network as 435 described in the framework for Abstraction and Control of TE Networks 436 (ACTN) described in [RFC8453] as discussed further in Section 4.5. 438 2.4. Dynamic Configuration 440 Enhanced VPNs need to be created, modified, and removed from the 441 network according to service demand. An enhanced VPN that requires 442 hard isolation must not be disrupted by the instantiation or 443 modification of another enhanced VPN. Determining whether 444 modification of an enhanced VPN can be disruptive to that VPN, and in 445 particular the traffic in flight will be disrupted can be a difficult 446 problem. 448 The data plane aspects of this problem are discussed further in 449 Section 4. 451 The control plane aspects of this problem are discussed further in 452 Section 4.4. 454 The management plane aspects of this problem are discussed further in 455 Section 4.5 457 Dynamic changes both to the VPN and to the underlay transport network 458 need to be managed to avoid disruption to sensitive services. 460 In addition to non-disruptively managing the network as a result of 461 gross change such as the inclusion of a new VPN endpoint or a change 462 to a link, VPN traffic might need to be moved as a result of traffic 463 volume changes. 465 2.5. Customized Control 467 In some cases it is desirable that an enhanced VPN has a customized 468 control plane, so that the tenant of the enhanced VPN can have some 469 control to the resources and functions allocated to this enhanced 470 VPN. For example, the tenant may be able to specify the service 471 paths in his own enhanced VPN. Depending on the requirement, an 472 enhanced VPN may have its own dedicated controller, or it may be 473 provided with an interface to a control system which is shared with a 474 set of other tenants, or it may be provided with an interface to the 475 control system provided by the network operator. 477 Further detail on this requirement will be provided in a future 478 version of the draft. A description of the management plane aspects 479 of this feature can be found in Section 4.5. 481 2.6. Applicability 483 The technologies described in this document should be applicable to a 484 number types of VPN services such as: 486 o Layer 2 point to point services such as pseudowires [RFC3985] 488 o Layer 2 VPNs [RFC4664] 490 o Ethernet VPNs [RFC7209] 492 o Layer 3 VPNs [RFC4364], [RFC2764] 494 o Virtual Networks (VNs) [RFC8453] 496 Where such VPN or VN types need enhanced isolation and delivery 497 characteristics, the technology described here can be used to provide 498 an underlay with the required enhanced performance. 500 3. Architecture of Enhanced VPN 502 A number of enhanced VPN services will typically be provided by a 503 common network infrastructure. Each enhanced VPN consists of both 504 the overlay and a specific set of dedicated network resources and 505 functions allocated in the underlay to satisfy the needs of the VPN 506 tenant. The integration between overlay and various underlay 507 resources ensures the isolation between different enhanced VPNs, and 508 achieves the guaranteed performance for different services. 510 An enhanced VPN needs to be designed with consideration given to: 512 o A enhanced data plane 514 o A control plane to create enhanced VPN, making use of the data 515 plane isolation and guarantee techniques 517 o A management plane for enhanced VPN service life-cycle management 519 These required characteristics are expanded below: 521 o Enhanced data plane 523 * Provides the required resource isolation capability, e.g. 524 bandwidth guarantee. 526 * Provides the required packet latency and jitter characteristics 528 * Provides the required packet loss characteristics 530 * Provides the mechanism to identify network slice and the 531 associated resources 533 o Control plane 535 * Collect the underlying network topology and resources available 536 and export this to other nodes and/or the centralized 537 controller as required. 539 * Create the required virtual networks with the resource and 540 properties needed by the enhanced VPN services that are 541 assigned to it. 543 * Determine the risk of SLA violation and take appropriate 544 avoiding action 546 * Determine the right balance of per-packet and per-node state 547 according to the needs of enhanced VPN service to scale to the 548 required size 550 o Management plane 552 * Provides the life-cycle management (creation, modification, 553 decommissioning) of enhanced VPN 555 * Provides an interface between the enhanced VPN provider and the 556 enhanced VPN clients such that some of the operation requests 557 can be met without interfering with the enhanced VPN of other 558 clients. 560 3.1. Layered Architecture 562 The layered architecture of enhanced VPN is shown in Figure 2. 564 +-------------------+ } 565 | Network Controller| } Centralized 566 +-------------------+ } Control 567 . . . . . 568 . . . . . 569 . N----N----N . } 570 . / / . } 571 N-----N-----N----N-----N } 572 N----N } 573 / / \ } Virtual 574 N-----N----N----N-----N } Networks 575 N----N } 576 / / } 577 N-----N-----N----N-----N } 579 +----+ ===== +----+ ===== +----+ ===== +----+ } 580 +----+ ===== +----+ ===== +----+ ===== +----+ } Physical 581 +----+ ===== +----+ ===== +----+ ===== +----+ } Network 582 +----+ +----+ +----+ +----+ } 583 N L N L N L N 585 N = Partitioned node 586 L = Partitioned link 588 +----+ = Partition within a node 589 +----+ 591 ====== = Partition within a link 593 Figure 2: The Layered Architecture 595 Underpinning everything is the physical infrastructure layer 596 consisting of partitioned links and nodes which provide the 597 underlying resources used to provision the separated virtual 598 networks. Various components and techniques as discussed in 599 Section 4 can be used to provide the resource partition, such as 600 FlexE, Time Sensitive Networking, Deterministic Networking, etc. 601 These partitions may be physical, or virtual so long as the SLA 602 required by the higher layers is met. 604 These techniques can be used to provision the virtual networks with 605 dedicated resources that they need. To get the required 606 functionality there needs to be integration between these overlays 607 and the underlay providing the physical resources. 609 The centralized controller is used to create the virtual networks, to 610 allocate the resources to each virtual network and to provision the 611 enhanced VPN services within the virtual networks. A distributed 612 control plane may also be used for the distribution of the topology 613 and attribute information of the virtual networks. 615 The creation and allocation process needs to take a holistic view of 616 the needs of all of its tenants, and to partition the resources 617 accordingly. However within a virtual network these resources can if 618 required be managed via a dynamic control plane. This provides the 619 required scalability and isolation. 621 3.2. Multi-Point to Multi-Point 623 At the VPN service level, the connectivity are usually mesh or 624 partial-mesh. To support such kind of VPN service, the corresponding 625 underlay is also an abstract MP2MP medium. However when service 626 guarantees are provided, the point-to-point path through the underlay 627 of the enhanced VPN needs to be specifically engineered to meet the 628 required performance guarantees. 630 3.3. Application Specific Network Types 632 Although a lot of the traffic that will be carried over the enhanced 633 VPN will likely be IPv4 or IPv6, the design has to be capable of 634 carrying other traffic types, in particular Ethernet traffic. This 635 is easily accomplished through the various pseudowire (PW) techniques 636 [RFC3985]. Where the underlay is MPLS, Ethernet can be carried over 637 the enhanced VPN encapsulated according to the method specified in 638 [RFC4448]. Where the underlay is IP, Layer Two Tunneling Protocol - 639 Version 3 (L2TPv3) [RFC3931] can be used with Ethernet traffic 640 carried according to [RFC4719]. Encapsulations have been defined for 641 most of the common layer two type for both PW over MPLS and for 642 L2TPv3. 644 4. Candidate Technologies 646 A VPN is a network created by applying a multiplexing technique to 647 the underlying network (the underlay) in order to distinguish the 648 traffic of one VPN from that of another. A VPN path that travels by 649 other than the shortest path through the underlay normally requires 650 state in the underlay to specify that path. State is normally 651 applied to the underlay through the use of the RSVP Signaling 652 protocol, or directly through the use of an SDN controller, although 653 other techniques may emerge as this problem is studied. This state 654 gets harder to manage as the number of VPN paths increases. 655 Furthermore, as we increase the coupling between the underlay and the 656 overlay to support the enhanced VPN service, this state will increase 657 further. 659 In an enhanced VPN different subsets of the underlay resources are 660 dedicated to different enhanced VPNs. Any enhanced VPN solution thus 661 needs tighter coupling with underlay than is the case with existing 662 VPNs. We cannot for example share the tunnel between enhanced VPNs 663 which require hard isolation. 665 4.1. Underlay Packet and Frame-Based Data Planes 667 A number of candidate underlay packet or frame-based data plane 668 solutions which can be used provide the required isolation and 669 guarantee are described in following sections. 671 o FlexE 673 o Time Sensitive Networking 675 o Dedicated Queues 677 4.1.1. FlexE 679 FlexE [FLEXE] is a method of creating a point-to-point Ethernet with 680 a specific fixed bandwidth. FlexE provides the ability to multiplex 681 multiple channels over an Ethernet link in a way that provides hard 682 isolation. FlexE also supports the bonding of multiple links, which 683 can be used to create larger links out of multiple slower links in a 684 more efficient way that traditional link aggregation. FlexE also 685 supports the sub-rating of links, which allows an operator to only 686 use a portion of a link. However it is a only a link level 687 technology. When packets are received by the downstream node, they 688 need to be processed in a way that preserves that isolation in the 689 downstream node. This in turn requires a queuing and forwarding 690 implementation that preserves the end-to-end isolation. 692 If different FlexE channels are used for different services, then no 693 sharing is possible between the FlexE channels. This in turn means 694 that it may be difficult to dynamically redistribute unused bandwidth 695 to lower priority services. This may increase the cost of providing 696 services on the network. On the other hand, FlexE can be used to 697 provide hard isolation between different tenants on a shared 698 interface. The tenant can then use other methods to manage the 699 relative priority of their own traffic in each FlexE channel. 701 Methods of dynamically re-sizing FlexE channels and the implication 702 for enhanced VPN is for further study. 704 4.1.2. Dedicated Queues 706 In order to provide multiple isolated virtual networks for enhanced 707 VPN, the conventional Diff-Serv based queuing system [RFC2475] 708 [RFC4594] is insufficient, due to the limited number of queues which 709 cannot differentiate between traffic of different enhanced VPNs, and 710 the range of service classes that each need to provide to their 711 tenants. This problem is particularly acute with an MPLS underlay 712 due to the small number of traffic class services available. In 713 order to address this problem and reduce the interference between 714 enhanced VPNs, it is necessary to steer traffic of VPNs to dedicated 715 input and output queues. Routers usually have large amount of queues 716 and sophisticated queuing systems, which could be used or enhanced to 717 provide the levels of isolation required by the applications of 718 enhanced VPN. For example, on one physical interface, the queuing 719 system can provide a set of virtual sub-interfaces, each allocated 720 with dedicated queueing and buffer resources. Sophisticated queuing 721 systems of this type may be used to provide end-to-end virtual 722 isolation between traffic of different enhanced VPNs. 724 4.1.3. Time Sensitive Networking 726 Time Sensitive Networking (TSN) [TSN] is an IEEE project that is 727 designing a method of carrying time sensitive information over 728 Ethernet. It introduces the concept of packet scheduling where a 729 high priority packet stream may be given a scheduled time slot 730 thereby guaranteeing that it experiences no queuing delay and hence a 731 reduced latency. However, when no scheduled packet arrives, its 732 reserved time-slot is handed over to best effort traffic, thereby 733 improving the economics of the network. The mechanisms defined in 734 TSN can be used to meet the requirements of time sensitive services 735 of an enhanced VPN. 737 Ethernet can be emulated over a Layer 3 network using a pseudowire. 738 However the TSN payload would be opaque to the underlay and thus not 739 treated specifically as time sensitive data. The preferred method of 740 carrying TSN over a layer 3 network is through the use of 741 deterministic networking as explained in the following section of 742 this document. 744 4.2. Packet and Frame-Based Network Layer 746 We now consider the problem of slice differentiation and resource 747 representation in the overlay network. The candidate technologies 748 are: 750 o Deterministic Networking 751 o MPLS-TE 753 o Segment Routing 755 4.2.1. Deterministic Networking 757 Deterministic Networking (DetNet) [I-D.ietf-detnet-architecture] is a 758 technique being developed in the IETF to enhance the ability of layer 759 3 networks to deliver packets more reliably and with greater control 760 over the delay. The design cannot use re-transmission techniques 761 such as TCP since that can exceed the delay tolerated by the 762 applications. Even the delay improvements that are achieved with 763 Stream Control Transmission Protocol Partial Reliability Extenstion 764 (SCTP-PR) [RFC3758] do not meet the bounds set by application 765 demands. DetNet pre-emptively sends copies of the packet over 766 various paths to minimize the chance of all packets being lost, and 767 trims duplicate packets to prevent excessive flooding of the network 768 and to prevent multiple packets being delivered to the destination. 769 It also seeks to set an upper bound on latency. The goal is not to 770 minimize latency; the optimum upper bound paths may not be the 771 minimum latency paths. 773 DetNet is based on flows. It currently does not specify the use of 774 underlay topology other than the base topology. To be of use for 775 enhanced VPN, DetNet needs to be integrated with different virtual 776 topologies of enhanced VPNs. 778 The detailed design that allows the use DetNet in a multi-tenant 779 network, and how to improve the scalability of DetNet in a multi- 780 tenant network are topics for further study. 782 4.2.2. MPLS Traffic Engineering (MPLS-TE) 784 MPLS-TE introduces the concept of reserving end-to-end bandwidth for 785 a TE-LSP, which can be used as the underlay of VPNs. It also 786 introduces the concept of non-shortest path routing through the use 787 of the Explicit Route Object [RFC3209]. VPN traffic can be run over 788 dedicated TE-LSPs to provide reserved bandwidth for each specific 789 connection in a VPN. Some network operators have concerns about the 790 scalability and management overhead of RSVP-TE system, and this has 791 lead them to consider other solutions for their networks. 793 4.2.3. Segment Routing 795 Segment Routing [RFC8402] is a method that prepends instructions to 796 packets at the head-end node and optionally at various points as it 797 passes though the network. These instructions allow the packets to 798 be routed on paths other than the shortest path for various traffic 799 engineering reasons. These paths can be strict or loose paths, 800 depending on the compactness required of the instruction list and the 801 degree of autonomy granted to the network, for example to support 802 Equal Cost Multipath load-balancing (ECMP) [RFC2992]. 804 With SR, a path needs to be dynamically created through a set of 805 segments by simply specifying the Segment Identifiers (SIDs), i.e. 806 instructions rooted at a particular point in the network. Thus if a 807 path is to be provisioned from some ingress point A to some egress 808 point B in the underlay, A is provided with a SID list from A to B 809 and instructions on how to identify the packets to which the SID list 810 is to be prepended. 812 By encoding the state in the packet, as is done in Segment Routing, 813 per-path state is transitioned out of the network. 815 However, there are a number of limitations in current SR, which limit 816 its applicability to enhanced VPNs: 818 o Segments are shared between different VPNs paths 820 o There is no reservation of bandwidth 822 o There is limited differentiation in the data plane. 824 Thus some extensions to SR are needed to provide isolation between 825 different enhanced VPNs. This can be achieved by including a finer 826 granularity of state in the network in anticipation of its future use 827 by authorized services. We therefore need to evaluate the balance 828 between this additional state and the performance delivered by the 829 network. 831 With current segment routing, the instructions are used to specify 832 the nodes and links to be traversed. However, in order to achieve 833 the required isolation between different services, new instructions 834 can be created which can be prepended to a packet to steer it through 835 specific network resources and functions. 837 Traditionally an SR traffic engineered path operates with a 838 granularity of a link with hints about priority provided through the 839 use of the traffic class (TC) field in the header. However to 840 achieve the latency and isolation characteristics that are sought by 841 the enhanced VPN users, steering packets through specific queues and 842 resources will likely be required. The extent to which these needs 843 can be satisfied through existing QoS mechanisms is to be determined. 844 What is clear is that a fine control of which services wait for 845 which, with a fine granularity of queue management policy is needed. 846 Note that the concept of a queue is a useful abstraction for many 847 types of underlay mechanism that may be used to provide enhanced 848 isolation and latency support. 850 From the perspective of the control plane, and from the perspective 851 of the segment routing, the method of steering a packet to a queue 852 that provides the required properties is an abstraction that hides 853 the details of the underlying implementation. How the queue 854 satisfies the requirement is implementation specific and is 855 transparent to the control plane and data plane mechanisms used. 856 Thus, for example, a FlexE channel, or a time sensitive networking 857 packet scheduling slot are abstracted to the same concept and bound 858 to the data plane in a common manner. 860 We can also introduce such fine grained packet steering by specifying 861 the queues through an SR instruction list. Thus new SR instructions 862 may be created to specify not only which resources are traversed, but 863 in some cases how they are traversed. For example, it may be 864 possible to specify not only the queue to be used but the policy to 865 be applied when enqueuing and dequeuing. 867 This concept could be further generalized, since as well as queuing 868 to the output port of a router, it is possible to consider queuing 869 data to any resource, for example: 871 o A network processor unit (NPU) 873 o A central processing unit (CPU) Core 875 o A Look-up engine 877 Both SR-MPLS and SRv6 are candidate network layer technologies for 878 enhanced VPN. In some cases they can be supported by DetNet to meet 879 the packet loss, delay and jitter requirement of particular service. 880 However, currently the "pure" IP variant of DetNet 881 [I-D.ietf-detnet-dp-sol-ip] does not support the Packet Replication, 882 Elimination, and Re-ordering (PREOF) [I-D.ietf-detnet-architecture] 883 functions. How to provide the DetNet enhanced delivery in an SRv6 884 environment needs further study. 886 4.3. Non-Packet Technologies 888 Non-packet underlay data plane technologies often have TE properties 889 and behaviors, and meet many of the key requirements in particular 890 for bandwidth guarantees, traffic isolation (with physical isolation 891 often being an integral part of the technology), highly predictable 892 latency and jitter characteristics, measurable loss characteristics, 893 and ease of identification of flows (and hence slices). 895 The control and management planes for non-packet data plane 896 technologies have most in common with MPLS-TE (Section 4.2.2) and 897 offer the same set of advanced features [RFC3945]. Furthermore, 898 management techniques such as ACTN ([RFC8453] and Section 4.4) can be 899 used to aid in the reporting of underlying network topologies, and 900 the creation of virtual networks with the resource and properties 901 needed by the enhanced VPN services. 903 4.4. Control Plane 905 Enhanced VPN would likely be based on a hybrid control mechanism, 906 which takes advantage of the logically centralized controller for on- 907 demand provisioning and global optimization, whilst still relies on 908 distributed control plane to provide scalability, high reliability, 909 fast reaction, automatic failure recovery etc. Extension and 910 optimization to the distributed control plane is needed to support 911 the enhanced properties of VPN+. 913 RSVP-TE provides the signaling mechanism of establishing a TE-LSP 914 with end-to-end resource reservation. It can be used to bind the VPN 915 to specific network resource allocated within the underlay, but there 916 are the above mentioned scalability concerns. 918 SR does not have the capability of signaling the resource reservation 919 along the path, nor do its currently specified distributed link state 920 routing protocols. On the other hand, the SR approach provides a way 921 of efficiently binding the network underlay and the enhanced VPN 922 overlay, as it reduces the amount of state to be maintained in the 923 network. An SR-based approach with per-slice resource reservation 924 can easily create dedicated SR network slices, and the VPN services 925 can be bound to a particular SR network slice. A centralized 926 controller can perform resource planning and reservation from the 927 controller's point of view, but this does not ensure resource 928 reservation is actually done in the network nodes. Thus, if a 929 distributed control plane is needed, either in place of an SDN 930 controller or as an assistant to it, the design of the control system 931 needs to ensure that resources are uniquely allocated in the network 932 nodes for the correct service, and not allocated to multiple services 933 causing unintended resource conflict. 935 4.5. Management Plane 937 The management plane mechanisms for enhanced VPN can be based on the 938 VPN service models as defined in [RFC8299] and [RFC8466], possible 939 augmentations and extensions to these models may be needed, which is 940 out of the scope of this document. 942 Abstraction and Control of Traffic Engineered Networks (ACTN) 943 [RFC8453] specifies the SDN based architecture for the control of TE 944 networks. The ACTN related data models such as 945 [I-D.ietf-teas-actn-vn-yang] and 946 [I-D.lee-teas-te-service-mapping-yang] can be applicable in the 947 provisioning of enhanced VPN service. The details are described in 948 Section 4.6. 950 4.6. Applicability of ACTN to Enhanced VPN 952 ACTN facilitates end-to-end connections and provides them to the 953 user. The ACTN framework [RFC8453] highlights how: 955 o Abstraction of the underlying network resources are provided to 956 higher-layer applications and customers. 958 o Virtualization of underlying resources, whose selection criterion 959 is the allocation of those resources for the customer, 960 application, or service. 962 o Creation of a virtualized environment allowing operators to view 963 and control multi-domain networks as a single virtualized network. 965 o The presentation to customers of networks as a virtual network via 966 open and programmable interfaces. 968 The infrastructure managed through ACTN comprises traffic engineered 969 network resources, which may include: 971 o Statistical packet bandwidth. 973 o Physical forwarding plane sources, such as: wavelengths and time 974 slots. 976 o Forwarding and cross-connect capabilities. 978 The type of network virtualization enabled by ACTN provides customers 979 and applications (tenants) with the capability to utilize and 980 independently control allocated virtual network resources as if they 981 were physically their own resources. 983 An ACTN Virtual Network (VN) is a client view of the ACTN managed 984 infrastructure, and is presented by the ACTN provider as a set of 985 abstracted resources. 987 Depending on the agreement between client and provider various VN 988 operations and VN views are possible. 990 o Virtual Network Creation: A VN could be pre-configured and created 991 via static or dynamic request and negotiation between customer and 992 provider. It must meet the specified SLA attributes which satisfy 993 the customer's objectives. 995 o Virtual Network Operations: The virtual network may be further 996 modified and deleted based on customer request to request changes 997 in the network resources reserved for the customer, and used to 998 construct the network slice. The customer can further act upon 999 the virtual network to manage traffic flow across the virtual 1000 network. 1002 o Virtual Network View: The VN topology from a customer point of 1003 view. These may be a variety of tunnels, or an entire VN 1004 topology. Such connections may comprise of customer end points, 1005 access links, intra-domain paths, and inter-domain links. 1007 Dynamic VN Operations allow a customer to modify or delete the VN. 1008 The customer can further act upon the virtual network to 1009 create/modify/delete virtual links and nodes. These changes will 1010 result in subsequent tunnel management in the operator's networks. 1012 4.6.1. ACTN Used for VPN+ Delivery 1014 ACTN provides VPN connections between multiple sites as requested via 1015 a VPN requestor enabled by the Customer Network Controller (CNC). 1016 The CNC is managed by the customer themselves, and interacts with the 1017 network provider's Multi-Domain Service Controller (MDSC). The 1018 Provisioning Network Controllers (PNC) remain entirely under the 1019 management of the network provider and are not visible to the 1020 customer. 1022 The benefits of this model include: 1024 o Provision of edge-to-edge VPN multi-access connectivity. 1026 o Management is mostly performed by the network provider, with some 1027 flexibility delegated to the customer-managed CNC. 1029 ---------------- ---------------- 1030 | Site-A Users |----------- ------------| Site-B Users | 1031 ---------------- | | ---------------- 1032 ------- 1033 | CNC | 1034 ------- 1035 Boundary | 1036 Between ==========================|========================== 1037 Customer & | 1038 Network Operator | 1039 --------------- 1040 | MDSC | 1041 --------------- 1042 _________/ | \__________ 1043 / | \ 1044 / | \ 1045 --------- --------- --------- 1046 | PNC | | PNC | | PNC | 1047 --------- --------- --------- 1048 | | / 1049 | | / 1050 ----- ----- ----- 1051 ( ) ( ) ( ) 1052 ---( Phys. )------------( Phys. )-------( Phys. )--- 1053 ( Net ) ( Net ) ( Net ) 1054 ----- ----- ----- 1056 Figure 3: VPN Delivery in the ACTN Architecture 1058 Figure 4 presents a more general representation of how multiple 1059 enhanced VPNs may be created from the resources of multiple physical 1060 networks using the CNC, MDSC, and PNC components of the ACTN 1061 architecture. Each enhanced VPN is controlled by its own CNC. The 1062 CNCs send requests to the provider's MDSC. The provider manages two 1063 physical networks each under the control of PNC. The MDSC asks the 1064 PNCs to allocate and provision resources to achieve the enhanced 1065 VPNs. In this figure, one enhanced VPN is constructed solely from 1066 the resources of one of the physical networks, while the the VPN uses 1067 resources from both physical networks. 1069 --------------- ( ) 1070 | CNC |---------->( VPN+ ) 1071 --------^------ ( ) 1072 | _(_________ _) 1073 --------------- ( ) ^ 1074 | CNC |----------->( VPN+ ) : 1075 ------^-------- ( ) : 1076 | | (___________) : 1077 | | ^ ^ : 1078 Boundary | | : : : 1079 Between ==========|====|===================:====:====:======== 1080 Customer & | | : : : 1081 Network Provider | | : : : 1082 v v : : : 1083 --------------- : :....: 1084 | MDSC | : : 1085 --------------- : : 1086 ^ ---^------ ... 1087 | ( ) . 1088 v ( Physical ) . 1089 ---------------- ( Network ) . 1090 | PNC |<-------->( ) ---^------ 1091 ---------------- | -------- ( ) 1092 | |-- ( Physical ) 1093 | PNC |<------------------------->( Network ) 1094 --------------- ( ) 1095 -------- 1097 Figure 4: Generic VPN+ Delivery in the ACTN Architecture 1099 4.6.2. Enhanced VPN Features with ACTN 1101 This section discusses how the features of ACTN can fulfill the 1102 enhanced VPN requirements described earlier in this document. As 1103 previously noted, key requirements of the enhanced VPN include: 1105 1. Isolation between VPNs 1107 2. Guaranteed Performance 1109 3. Integration 1111 4. Dynamic Configuration 1113 5. Customized Control Plane 1115 The subsections that follow outline how each requirement is met using 1116 ACTN. 1118 4.6.2.1. Isolation Between VPNs 1120 The ACTN VN YANG model [I-D.ietf-teas-actn-vn-yang] and the TE- 1121 service mapping model [I-D.lee-teas-te-service-mapping-yang] fulfill 1122 the VPN isolation requirement by providing the following features for 1123 the VNs: 1125 o Each VN is identified with a unique identifier (vn-id and vn-name) 1126 and so is each VN member that belongs to the VN (vn-member-id). 1128 o Each instantiated VN is managed and controlled independent of 1129 other VNs in the network with proper protection level 1130 (protection). 1132 o Each VN is instantiated with an isolation requirement described by 1133 the TE-service mapping model 1134 [I-D.lee-teas-te-service-mapping-yang]. This mapping supports: 1136 * Hard isolation with deterministic characteristics (e.g., this 1137 case may need an optical bypass tunnel or a DetNet/TSN tunnel 1138 to guarantee latency with no jitter) 1140 * Hard isolation (i.e., dedicated TE resources in all underlays) 1142 * Soft isolation (i.e., resource in some layer may be shared 1143 while in some other layers is dedicated). 1145 * No isolation (i.e., sharing with other VN). 1147 4.6.2.2. Guaranteed Performance 1149 Performance objectives of a VN need first to be expressed in order to 1150 assure the performance guarantee. [I-D.ietf-teas-actn-vn-yang] and 1151 [I-D.ietf-teas-yang-te-topo] allow configuration of several 1152 parameters that may affect the VN performance objectives as follows: 1154 o Bandwidth 1156 o Objective function (e.g., min cost path, min load path, etc.) 1158 o Metric Types and their threshold: 1160 * TE cost, IGP cost, Hop count, or Unidirectional Delay (e.g., 1161 can set all path delay <= threshold) 1163 Once these requests are instantiated, the resources are committed and 1164 guaranteed through the life cycle of the VN. 1166 4.6.2.3. Integration 1168 ACTN provides mechanisms to correlate customer's VN and the actual TE 1169 tunnels instantiated in the provider's network. Specifically: 1171 o Link each VN member to actual TE tunnel. 1173 o Each VN can be monitored on a various level such as VN level, VN 1174 member level, TE-tunnel level, and link/node level. 1176 Service function integration with network topology (L3 and TE 1177 topology) is in progress in [I-D.ietf-teas-sf-aware-topo-model]. 1178 Specifically, [I-D.ietf-teas-sf-aware-topo-model] addresses a number 1179 of use-cases that show how TE topology supports various service 1180 functions. 1182 4.6.2.4. Dynamic Configuration 1184 ACTN provides an architecture that allows the CNC to interact with 1185 the MDSC which is network provider's SDN controller. This gives the 1186 customer control of their VNs. 1188 Specifically, the ACTN VN model [I-D.ietf-teas-actn-vn-yang] allows 1189 the VN to create, modify, and delete VNs. 1191 4.6.2.5. Customized Control 1193 ACTN provides a YANG model that allows the CNC to control a VN as a 1194 "Type 2 VN" that allows the customer to provision tunnels that 1195 connect their endpoints over the customized VN topology. 1197 For some VN members, the customers are allowed to configure the path 1198 (i.e., the sequence of virtual nodes and virtual links) over the VN/ 1199 abstract topology. 1201 5. Scalability Considerations 1203 Enhanced VPN provides the performance guaranteed services in packet 1204 networks, with the cost of introducing necessary additional states 1205 into the network. There are at least three ways of adding the state 1206 needed for VPN+: 1208 o Introduce the complete state into the packet, as is done in SR. 1209 This allows the controller to specify the detailed series of 1210 forwarding and processing instructions for the packet as it 1211 transits the network. The cost of this is an increase in the 1212 packet header size. The cost is also that systems will have 1213 capabilities enabled in case they are called upon by a service. 1215 This is a type of latent state, and increases as we more precisely 1216 specify the path and resources that need to be exclusively 1217 available to a VPN. 1219 o Introduce the state to the network. This is normally done by 1220 creating a path using RSVP-TE, which can be extended to introduce 1221 any element that needs to be specified along the path, for example 1222 explicitly specifying queuing policy. It is of course possible to 1223 use other methods to introduce path state, such as via a Software 1224 Defined Network (SDN) controller, or possibly by modifying a 1225 routing protocol. With this approach there is state per path per 1226 path characteristic that needs to be maintained over its life- 1227 cycle. This is more state than is needed using SR, but the packet 1228 are shorter. 1230 o Provide a hybrid approach based on using binding SIDs to create 1231 path fragments, and bind them together with SR. 1233 Dynamic creation of a VPN path using SR requires less state 1234 maintenance in the network core at the expense of larger VPN headers 1235 on the packet. The packet size can be lower if a form of loose 1236 source routing is used (using a few nodal SIDs), and it will be lower 1237 if no specific functions or resource on the routers are specified. 1238 Reducing the state in the network is important to enhanced VPN, as it 1239 requires the overlay to be more closely integrated with the underlay 1240 than with traditional VPNs. This tighter coupling would normally 1241 mean that more state needed to be created and maintained in the 1242 network, as the state about fine granularity processing would need to 1243 be loaded and maintained in the routers. However, a segment routed 1244 approach allows much of this state to be spread amongst the network 1245 ingress nodes, and transiently carried in the packets as SIDs. 1247 These approaches are for further study. 1249 5.1. Maximum Stack Depth of SR 1251 One of the challenges with SR is the stack depth that nodes are able 1252 to impose on packets [I-D.ietf-isis-segment-routing-msd]. This leads 1253 to a difficult balance between adding state to the network and 1254 minimizing stack depth, or minimizing state and increasing the stack 1255 depth. 1257 5.2. RSVP Scalability 1259 The traditional method of creating a resource allocated path through 1260 an MPLS network is to use the RSVP protocol. However there have been 1261 concerns that this requires significant continuous state maintenance 1262 in the network. There are ongoing works to improve the scalability 1263 of RSVP-TE LSPs in the control plane [RFC8370]. 1265 There is also concern at the scalability of the forwarder footprint 1266 of RSVP as the number of paths through an LSR grows 1267 [I-D.sitaraman-mpls-rsvp-shared-labels] proposes to address this by 1268 employing SR within a tunnel established by RSVP-TE. 1270 6. OAM Considerations 1272 A study of OAM in SR networks has been documented in [RFC8403]. 1274 The enhanced VPN OAM design needs to consider the following 1275 requirements: 1277 o Instrumentation of the underlay so that the network operator can 1278 be sure that the resources committed to a tenant are operating 1279 correctly and delivering the required performance. 1281 o Instrumentation of the overlay by the tenant. This is likely to 1282 be transparent to the network operator and to use existing 1283 methods. Particular consideration needs to be given to the need 1284 to verify the isolation and the various committed performance 1285 characteristics. 1287 o Instrumentation of the overlay by the network provider to 1288 proactively demonstrate that the committed performance is being 1289 delivered. This needs to be done in a non-intrusive manner, 1290 particularly when the tenant is deploying a performance sensitive 1291 application 1293 o Verification of the conformity of the path to the service 1294 requirement. This may need to be done as part of a commissioning 1295 test. 1297 These issues will be discussed in a future version of this document. 1299 7. Enhanced Resiliency 1301 Each enhanced VPN has a life-cycle, and needs modification during 1302 deployment as the needs of its tenant change. Additionally, as the 1303 network as a whole evolves, there will need to be garbage collection 1304 performed to consolidate resources into usable quanta. 1306 Systems in which the path is imposed such as SR, or some form of 1307 explicit routing tend to do well in these applications, because it is 1308 possible to perform an atomic transition from one path to another. 1309 This is a single action by the head-end changes the path without the 1310 need for coordinated action by the routers along the path. However, 1311 implementations and the monitoring protocols need to make sure that 1312 the new path is up and meet the required SLA before traffic is 1313 transitioned to it. It is possible for deadlocks arise as a result 1314 of the network becoming fragmented over time, such that it is 1315 impossible to create a new path or modify a existing path without 1316 impacting the SLA of other paths. Resolution of this situation is as 1317 much a commercial issue as it is a technical issue and is outside the 1318 scope of this document. 1320 There are however two manifestations of the latency problem that are 1321 for further study in any of these approaches: 1323 o The problem of packets overtaking one and other if a path latency 1324 reduces during a transition. 1326 o The problem of the latency transient in either direction as a path 1327 migrates. 1329 There is also the matter of what happens during failure in the 1330 underlay infrastructure. Fast reroute is one approach, but that 1331 still produces a transient loss with a normal goal of rectifying this 1332 within 50ms [RFC5654] . An alternative is some form of N+1 delivery 1333 such as has been used for many years to support protection from 1334 service disruption. This may be taken to a different level using the 1335 techniques proposed by the IETF deterministic network work with 1336 multiple in-network replication and the culling of later packets 1337 [I-D.ietf-detnet-architecture]. 1339 In addition to the approach used to protect high priority packets, 1340 consideration has to be given to the impact of best effort traffic on 1341 the high priority packets during a transient. Specifically if a 1342 conventional re-convergence process is used there will inevitably be 1343 micro-loops and whilst some form of explicit routing will protect the 1344 high priority traffic, lower priority traffic on best effort shortest 1345 paths will micro-loop without the use of a loop prevention 1346 technology. To provide the highest quality of service to high 1347 priority traffic, either this traffic must be shielded from the 1348 micro-loops, or micro-loops must be prevented. 1350 8. Security Considerations 1352 All types of virtual network require special consideration to be 1353 given to the isolation between the tenants. In this regard enhanced 1354 VPNs neither introduce, no experience a greater security risk than 1355 another VPN of the same base type. However, in an enhanced virtual 1356 network service the isolation requirement needs to be considered. If 1357 a service requires a specific latency then it can be damaged by 1358 simply delaying the packet through the activities of another tenant. 1359 In a network with virtual functions, depriving a function used by 1360 another tenant of compute resources can be just as damaging as 1361 delaying transmission of a packet in the network. The measures to 1362 address these dynamic security risks must be specified as part to the 1363 specific solution. 1365 9. IANA Considerations 1367 There are no requested IANA actions. 1369 10. Contributors 1371 Daniel King 1372 Email: daniel@olddog.co.uk 1374 Adrian Farrel 1375 Email: adrian@olddog.co.uk 1377 Jeff Tansura 1378 Email: jefftant.ietf@gmail.com 1380 Qin Wu 1381 Email: bill.wu@huawei.com 1383 Daniele Ceccarelli 1384 Email: daniele.ceccarelli@ericsson.com 1386 Mohamed Boucadair 1387 Email: mohamed.boucadair@orange.com 1389 Sergio Belotti 1390 Email: sergio.belotti@nokia.com 1392 Haomian Zheng 1393 Email: zhenghaomian@huawei.com 1395 11. Acknowledgements 1397 The authors would like to thank Charlie Perkins and James N Guichard 1398 for their review and valuable comments. 1400 This work was supported in part by the European Commission funded 1401 H2020-ICT-2016-2 METRO-HAUL project (G.A. 761727). 1403 12. References 1405 12.1. Normative References 1407 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1408 Requirement Levels", BCP 14, RFC 2119, 1409 DOI 10.17487/RFC2119, March 1997, 1410 . 1412 12.2. Informative References 1414 [BBF-SD406] 1415 "BBF SD-406: End-to-End Network Slicing", 2016, 1416 . 1419 [DETNET] "Deterministic Networking", March , 1420 . 1422 [FLEXE] "Flex Ethernet Implementation Agreement", March 2016, 1423 . 1426 [I-D.ietf-detnet-architecture] 1427 Finn, N., Thubert, P., Varga, B., and J. Farkas, 1428 "Deterministic Networking Architecture", draft-ietf- 1429 detnet-architecture-09 (work in progress), October 2018. 1431 [I-D.ietf-detnet-dp-sol-ip] 1432 Korhonen, J. and B. Varga, "DetNet IP Data Plane 1433 Encapsulation", draft-ietf-detnet-dp-sol-ip-01 (work in 1434 progress), October 2018. 1436 [I-D.ietf-detnet-use-cases] 1437 Grossman, E., "Deterministic Networking Use Cases", draft- 1438 ietf-detnet-use-cases-19 (work in progress), October 2018. 1440 [I-D.ietf-isis-segment-routing-msd] 1441 Tantsura, J., Chunduri, U., Aldrin, S., and L. Ginsberg, 1442 "Signaling MSD (Maximum SID Depth) using IS-IS", draft- 1443 ietf-isis-segment-routing-msd-19 (work in progress), 1444 October 2018. 1446 [I-D.ietf-teas-actn-vn-yang] 1447 Lee, Y., Dhody, D., Ceccarelli, D., Bryskin, I., Yoon, B., 1448 Wu, Q., and P. Park, "A Yang Data Model for ACTN VN 1449 Operation", draft-ietf-teas-actn-vn-yang-02 (work in 1450 progress), September 2018. 1452 [I-D.ietf-teas-sf-aware-topo-model] 1453 Bryskin, I., Liu, X., Lee, Y., Guichard, J., Contreras, 1454 L., Ceccarelli, D., and J. Tantsura, "SF Aware TE Topology 1455 YANG Model", draft-ietf-teas-sf-aware-topo-model-02 (work 1456 in progress), September 2018. 1458 [I-D.ietf-teas-yang-te-topo] 1459 Liu, X., Bryskin, I., Beeram, V., Saad, T., Shah, H., and 1460 O. Dios, "YANG Data Model for Traffic Engineering (TE) 1461 Topologies", draft-ietf-teas-yang-te-topo-18 (work in 1462 progress), June 2018. 1464 [I-D.lee-teas-te-service-mapping-yang] 1465 Lee, Y., Dhody, D., Ceccarelli, D., Tantsura, J., 1466 Fioccola, G., and Q. Wu, "Traffic Engineering and Service 1467 Mapping Yang Model", draft-lee-teas-te-service-mapping- 1468 yang-12 (work in progress), October 2018. 1470 [I-D.sitaraman-mpls-rsvp-shared-labels] 1471 Sitaraman, H., Beeram, V., Parikh, T., and T. Saad, 1472 "Signaling RSVP-TE tunnels on a shared MPLS forwarding 1473 plane", draft-sitaraman-mpls-rsvp-shared-labels-03 (work 1474 in progress), December 2017. 1476 [NGMN-NS-Concept] 1477 "NGMN NS Concept", 2016, . 1481 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1482 and W. Weiss, "An Architecture for Differentiated 1483 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1484 . 1486 [RFC2764] Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. 1487 Malis, "A Framework for IP Based Virtual Private 1488 Networks", RFC 2764, DOI 10.17487/RFC2764, February 2000, 1489 . 1491 [RFC2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path 1492 Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000, 1493 . 1495 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1496 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1497 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1498 . 1500 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1501 Conrad, "Stream Control Transmission Protocol (SCTP) 1502 Partial Reliability Extension", RFC 3758, 1503 DOI 10.17487/RFC3758, May 2004, 1504 . 1506 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1507 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 1508 RFC 3931, DOI 10.17487/RFC3931, March 2005, 1509 . 1511 [RFC3945] Mannie, E., Ed., "Generalized Multi-Protocol Label 1512 Switching (GMPLS) Architecture", RFC 3945, 1513 DOI 10.17487/RFC3945, October 2004, 1514 . 1516 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1517 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1518 DOI 10.17487/RFC3985, March 2005, 1519 . 1521 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1522 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1523 2006, . 1525 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 1526 "Encapsulation Methods for Transport of Ethernet over MPLS 1527 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 1528 . 1530 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 1531 Guidelines for DiffServ Service Classes", RFC 4594, 1532 DOI 10.17487/RFC4594, August 2006, 1533 . 1535 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 1536 2 Virtual Private Networks (L2VPNs)", RFC 4664, 1537 DOI 10.17487/RFC4664, September 2006, 1538 . 1540 [RFC4719] Aggarwal, R., Ed., Townsley, M., Ed., and M. Dos Santos, 1541 Ed., "Transport of Ethernet Frames over Layer 2 Tunneling 1542 Protocol Version 3 (L2TPv3)", RFC 4719, 1543 DOI 10.17487/RFC4719, November 2006, 1544 . 1546 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1547 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1548 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1549 September 2009, . 1551 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 1552 Networking: A Perspective from within a Service Provider 1553 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 1554 . 1556 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 1557 Henderickx, W., and A. Isaac, "Requirements for Ethernet 1558 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 1559 . 1561 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 1562 Ceccarelli, D., and X. Zhang, "Problem Statement and 1563 Architecture for Information Exchange between 1564 Interconnected Traffic-Engineered Networks", BCP 206, 1565 RFC 7926, DOI 10.17487/RFC7926, July 2016, 1566 . 1568 [RFC8299] Wu, Q., Ed., Litkowski, S., Tomotaki, L., and K. Ogaki, 1569 "YANG Data Model for L3VPN Service Delivery", RFC 8299, 1570 DOI 10.17487/RFC8299, January 2018, 1571 . 1573 [RFC8370] Beeram, V., Ed., Minei, I., Shakir, R., Pacella, D., and 1574 T. Saad, "Techniques to Improve the Scalability of RSVP-TE 1575 Deployments", RFC 8370, DOI 10.17487/RFC8370, May 2018, 1576 . 1578 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1579 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1580 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1581 July 2018, . 1583 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1584 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1585 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1586 2018, . 1588 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1589 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1590 DOI 10.17487/RFC8453, August 2018, 1591 . 1593 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1594 Data Model for Layer 2 Virtual Private Network (L2VPN) 1595 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1596 2018, . 1598 [SFC] "Service Function Chaining", March , 1599 . 1601 [TS23501] "3GPP TS23.501", 2016, 1602 . 1605 [TS28530] "3GPP TS28.530", 2016, 1606 . 1609 [TSN] "Time-Sensitive Networking", March , 1610 . 1612 Authors' Addresses 1614 Jie Dong 1615 Huawei 1617 Email: jie.dong@huawei.com 1619 Stewart Bryant 1620 Huawei 1622 Email: stewart.bryant@gmail.com 1624 Zhenqiang Li 1625 China Mobile 1627 Email: lizhenqiang@chinamobile.com 1629 Takuya Miyasaka 1630 KDDI Corporation 1632 Email: ta-miyasaka@kddi.com 1633 Young Lee 1634 Huawei 1636 Email: leeyoung@huawei.com