idnits 2.17.1 draft-ietf-teas-enhanced-vpn-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (January 23, 2020) is 1555 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC2702' is mentioned on line 902, but not defined == Missing Reference: 'APL' is mentioned on line 1184, but not defined == Unused Reference: 'I-D.ietf-idr-bgp-ls-segment-routing-ext' is defined on line 1552, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-teas-yang-te' is defined on line 1572, but no explicit reference was found in the text == Unused Reference: 'I-D.www-bess-yang-vpn-service-pm' is defined on line 1582, but no explicit reference was found in the text == Unused Reference: 'RFC2992' is defined on line 1596, but no explicit reference was found in the text == Outdated reference: A later version (-24) exists of draft-ietf-teas-actn-vn-yang-07 == Outdated reference: A later version (-15) exists of draft-ietf-teas-te-service-mapping-yang-02 == Outdated reference: A later version (-18) exists of draft-ietf-idr-bgp-ls-segment-routing-ext-16 == Outdated reference: A later version (-18) exists of draft-ietf-opsawg-l3sm-l3nm-01 == Outdated reference: A later version (-13) exists of draft-ietf-opsawg-ntf-02 == Outdated reference: A later version (-12) exists of draft-ietf-teas-sf-aware-topo-model-04 == Outdated reference: A later version (-36) exists of draft-ietf-teas-yang-te-22 == Outdated reference: A later version (-06) exists of draft-www-bess-yang-vpn-service-pm-04 Summary: 0 errors (**), 0 flaws (~~), 15 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 TEAS working group J. Dong 2 Internet-Draft Huawei 3 Intended status: Informational S. Bryant 4 Expires: July 2020 Futurewei 5 Z. Li 6 China Mobile 7 T. Miyasaka 8 KDDI Corporation 9 Y.Lee 10 Sung Kyun Kwan University 11 January 23, 2020 13 A Framework for Enhanced Virtual Private Networks (VPN+) Services 15 draft-ietf-teas-enhanced-vpn-04 17 Abstract 19 This document describes the framework for Enhanced Virtual Private 20 Network (VPN+) service. The purpose is to support the needs of new 21 applications, particularly applications that are associated with 5G 22 services, by utilizing an approach that is based on existing VPN and 23 TE technologies and adds features that specific services require 24 over and above traditional VPNs. 26 Typically, VPN+ will be used to form the underpinning of network 27 slicing, but could also be of use in its own right providing 28 enhanced connectivity services between customer sites. 30 It is envisaged that enhanced VPNs will be delivered using a 31 combination of existing, modified, and new networking technologies. 32 This document provides an overview of relevant technologies and 33 identifies some areas for potential new work. 35 It is not envisaged that large numbers of VPN+ instances will be 36 deployed in a network and, in particular, it is not intended that 37 all VPNs supported by a network will use VPN+ related techniques. 39 Status of This Memo 41 This Internet-Draft is submitted in full conformance with the 42 provisions of BCP 78 and BCP 79. 44 Internet-Drafts are working documents of the Internet Engineering 45 Task Force (IETF). Note that other groups may also distribute 46 working documents as Internet-Drafts. The list of current Internet- 47 Drafts is at https://datatracker.ietf.org/drafts/current/. 49 Internet-Drafts are draft documents valid for a maximum of six 50 months and may be updated, replaced, or obsoleted by other documents 51 at any time. It is inappropriate to use Internet-Drafts as reference 52 material or to cite them other than as "work in progress." 54 This Internet-Draft will expire on July 22, 2020. 56 Copyright Notice 58 Copyright (c) 2020 IETF Trust and the persons identified as the 59 document authors. All rights reserved. 61 This document is subject to BCP 78 and the IETF Trust's Legal 62 Provisions Relating to IETF Documents 63 (https://trustee.ietf.org/license-info) in effect on the date of 64 publication of this document. Please review these documents 65 carefully, as they describe your rights and restrictions with 66 respect to this document. Code Components extracted from this 67 document must include Simplified BSD License text as described in 68 Section 4.e of the Trust Legal Provisions and are provided without 69 warranty as described in the Simplified BSD License. 71 Table of Contents 73 1. Introduction ................................................ 3 74 2. Overview of the Requirements ................................ 6 75 2.1. Isolation between Virtual Networks ..................... 6 76 2.1.1. A Pragmatic Approach to Isolation ................. 8 77 2.2. Performance Guarantee .................................. 8 78 2.3. Integration ........................................... 10 79 2.3.1. Abstraction ...................................... 11 80 2.4. Dynamic Management .................................... 11 81 2.5. Customized Control .................................... 12 82 2.6. Applicability ......................................... 12 83 2.7. Inter-Domain and Inter-Layer Network .................. 12 84 3. Architecture of Enhanced VPN ............................... 13 85 3.1. Layered Architecture .................................. 15 86 3.2. Multi-Point to Multi-Point (MP2MP) Connectivity ....... 17 87 3.3. Application Specific Network Types .................... 18 88 3.4. Scaling Considerations ................................ 18 89 4. Candidate Technologies ..................................... 19 90 4.1. Layer-Two Data Plane .................................. 19 91 4.1.1. Flexible Ethernet ................................ 19 92 4.1.2. Dedicated Queues ................................. 20 93 4.1.3. Time Sensitive Networking ........................ 20 95 4.2. Layer-Three Data Plane ................................ 21 96 4.2.1. Deterministic Networking ......................... 21 97 4.2.2. MPLS Traffic Engineering (MPLS-TE) ............... 21 98 4.2.3. Segment Routing .................................. 21 99 4.3. Non-Packet Data Plane ................................. 22 100 4.4. Control Plane ......................................... 22 101 4.5. Management Plane ...................................... 23 102 4.6. Applicability of Service Data Models to Enhanced VPN .. 23 103 4.6.1. Enhanced VPN Delivery in the ACTN Architecture ... 24 104 4.6.2. Enhanced VPN Features with Service Data Models ... 25 105 4.6.3. 5G Transport Service Delivery via Coordinated Data 106 Modules ................................................. 27 107 5. Scalability Considerations ................................. 29 108 5.1. Maximum Stack Depth of SR ............................. 30 109 5.2. RSVP Scalability ...................................... 30 110 5.3. SDN Scaling ........................................... 30 111 6. OAM Considerations ......................................... 30 112 7. Telemetry Considerations ................................... 31 113 8. Enhanced Resiliency ........................................ 31 114 9. Operational Considerations ................................. 33 115 10. Security Considerations ................................... 33 116 11. IANA Considerations........................................ 33 117 12. Contributors .............................................. 34 118 13. Acknowledgments ........................................... 34 119 14. References ................................................ 34 120 14.1. Normative References ................................. 34 121 14.2. Informative References ............................... 36 122 Authors' Addresses ............................................ 40 124 1. Introduction 126 Virtual private networks (VPNs) have served the industry well as a 127 means of providing different groups of users with logically isolated 128 connectivity over a common network. The common or base network that 129 is used to provide the VPNs is often referred to as the underlay, 130 and the VPN is often called an overlay. 132 Customers of a network operator may request a connectivity services 133 with advanced characteristics such as complete isolation from other 134 services so that changes in some other service (such as changes in 135 network load, or events such as congestion or outages) have no 136 effect on the throughput or latency of the services provided to the 137 customer. These services are "enhanced VPNs" (known as VPN+) in that 138 they are similar to VPN services as they provide the customer with 139 required connectivity, but have enhanced characteristics. 141 Driven largely by needs surfacing from 5G, the concept of network 142 slicing has gained traction [NGMN-NS-Concept] [TS23501] [TS28530] 143 [BBF-SD406]. According to [TS28530], a 5G end-to-end network slice 144 consists of three major types network segments: Radio Access Network 145 (RAN), Transport Network (TN) and Mobile Core Network (CN). The 146 transport network provides the required connectivity between 147 different entities in RAN and CN segments of an end-to-end network 148 slice, with specific performance commitment. VPN+ could be used to 149 form the underpinning of network slicing, but could also be of use 150 in general cases providing enhanced connectivity services between 151 customer sites. 153 A transport network slice is a virtual (logical) network with a 154 particular network topology and a set of shared or dedicated network 155 resources, which are used to provide the network slice consumer with 156 the required connectivity, appropriate isolation and specific 157 Service Level Agreement (SLA) or Service Level Objective (SLO). 159 A transport network slice could span multiple technologies (such as 160 IP or Optical) and multiple administrative domains. Depending on the 161 consumer's requirement, a transport network slice could be isolated 162 from other, often concurrent transport network slices in terms of 163 data plane, control plane, and management plane resources. 165 In this document the term "network slice" refers to a transport 166 network slice, and is considered as one typical use case of enhanced 167 VPN. 169 Network slicing builds on the concept of resource management, 170 network virtualization, and abstraction to provide performance 171 assurance, flexibility, programmability and modularity. It may use 172 techniques such as Software Defined Networking (SDN) [RFC7149], 173 network abstraction [RFC7926] and Network Function Virtualization 174 (NFV) [RFC8172] [RFC8568] to create multiple logical (virtual) 175 networks, each tailored for a set of services or a particular tenant 176 or a group of tenants that share the same or similar set of 177 requirements, on top of a common network. How the network slices 178 are engineered can be deployment-specific. 180 Thus, there is a need to create virtual networks with enhanced 181 characteristics to support enhanced VPN services. The tenant of 182 such a virtual network can require a degree of isolation and 183 performance that previously could not be satisfied by traditional 184 overlay VPNs. Additionally, the tenant may ask for some level of 185 control to their virtual networks, e.g., to customize the service 186 paths in a network slice. 188 These enhanced properties cannot be met by simple overlay networks, 189 as they require tighter coordination and integration between the 190 underlay and the overlay network. This document introduces the 191 Enhanced VPN (otherwise known as VPN+). VPN+ is built from a virtual 192 network which has a customized network topology and a set of 193 dedicated or shared network resources, optionally including invoked 194 service functions, allocated from the underlay network. Unlike a 195 traditional VPN, an enhanced VPN can achieve greater isolation with 196 strict performance guarantees. These new properties, which have 197 general applicability, may also be of interest as part of a network 198 slicing solution, but it is not envisaged that VPN+ services will 199 replace traditional VPN services that can continue to be deployed 200 using pre-existing mechanisms. Furthermore, it is not intended that 201 large numbers of VPN+ instances will be deployed within a single 202 network. See Section 5 for a discussion of scalability 203 considerations. 205 This document specifies a framework for using existing, modified, 206 and potential new technologies as components to provide a VPN+ 207 service. Specifically we are concerned with: 209 o The design of the enhanced data plane. 211 o The necessary protocols in both the underlay and the overlay 212 of the enhanced VPN. 214 o The mechanisms to achieve integration between overlay and 215 underlay. 217 o The necessary Operation, Administration, and Management (OAM) 218 methods to instrument an enhanced VPN to make sure that the required 219 Service Level Agreement (SLA) is met, and to take any corrective 220 action to avoid SLA violation, such as switching to an alternate 221 path. 223 The required layered network structure to achieve this is shown in 224 Section 3.1. 226 Note that, in this document, the four terms "VPN", "Enhanced VPN" 227 (or "VPN+"), "Virtual Network (VN)", and "Network Slice" may be 228 considered as describing similar concepts dependent on the viewpoint 229 from which they are used. 231 o An enhanced VPN can be considered as an evolution of VPN, but 232 with additional service-specific commitments. Thus, care must be 233 taken with the term "VPN" to distinguish normal or legacy VPNs from 234 VPN+ instances. 236 o A Virtual Network (VN) is a type of service that connects 237 customer edge points with the additional possibility of requesting 238 further service characteristics in the manner of an enhanced VPN. 240 o An enhanced VPN or VN is made by creating a slice through the 241 resources of the underlay network. 243 o The general concept of network slicing in a TE network 244 provides tools to address some aspects or realizations of enhanced 245 VPN. 247 2. Overview of the Requirements 249 In this section we provide an overview of the requirements of an 250 enhanced VPN service. 252 2.1. Isolation between Virtual Networks 254 One element of the SLA demanded for an enhanced VPN is a guarantee 255 that the service offered to the customer will not be perturbed by 256 any other traffic flows in the network. One way for a service 257 provider to guarantee the customer's SLA is by controlling the 258 degree of isolation from other services in the network. Isolation 259 is a feature that can be requested by customers. There are 260 different grades of how isolation may be enabled by a network 261 operator and that may result in different levels of service 262 perceived by the customer. These range from simple separation of 263 service traffic on delivery (ensuring that traffic is not delivered 264 to the wrong customer), all the way to complete separation within 265 the underlay so that the traffic from different services use 266 distinct network resources. 268 The terms hard and soft isolation are used to identify different 269 levels of isolation. A VPN has soft isolation if the traffic of one 270 VPN cannot be received by the customers of another VPN. Both IP and 271 MPLS VPNs are examples of VPNs with soft isolation: the network 272 delivers the traffic only to the required VPN endpoints. However, 273 with soft isolation, traffic from VPNs and regular non-VPN traffic 274 may congest the network resulting in packet loss and delay for other 275 VPNs operating normally. The ability for a VPN service or a group 276 of VPN services to be sheltered from this effect is called hard 277 isolation, and this property is required by some applications. Hard 278 isolation is needed so that applications with exacting requirements 279 can function correctly, despite other demands (perhaps a burst of 280 traffic in another VPN) competing for the underlying resources. In 281 practice isolation may be offered as a spectrum between soft and 282 hard, and in some cases soft and hard isolation may be used in a 283 hierarchical manner. An operator may offer its customers a choice 284 of different degrees of isolation ranging from soft isolation up to 285 hard isolation. 287 An example of the requirement for hard isolation is a network 288 supporting both emergency services and public broadband multi-media 289 services. During a major incident the VPNs supporting these 290 services would both be expected to experience high data volumes, and 291 it is important that both make progress in the transmission of their 292 data. In these circumstances the VPN services would require an 293 appropriate degree of isolation to be able to continue to operate 294 acceptably. On the other hand, VPNs servicing ordinary bulk data 295 may expect to contest for network resources and queue packets so 296 that traffic is delivered within SLAs, but with some potential 297 delays and interference. 299 In order to provide the required level of isolation, resources may 300 have to be reserved in the data plane of the underlay network and 301 dedicated to traffic from a specific VPN or a specific group of VPNs 302 to form different enhanced VPNs in the network. This may introduce 303 scalability concerns, thus some trade-off needs to be considered to 304 provide the required isolation between some enhanced VPNs while 305 still allowing reasonable sharing. 307 An optical layer can offer a high degree of isolation, at the cost 308 of allocating resources on a long term and end-to-end basis. On the 309 other hand, where adequate isolation can be achieved at the packet 310 layer, this permits the resources to be shared amongst a group of 311 services and only dedicated to a service on a temporary basis. 313 There are several new technologies that provide some assistance with 314 these data plane issues. Firstly there is the IEEE project on Time 315 Sensitive Networking [TSN] which introduces the concept of packet 316 scheduling of delay and loss sensitive packets. Then there is 317 [FLEXE] which provides the ability to multiplex multiple channels 318 over one or more Ethernet links in a way that provides hard 319 isolation. Finally there are advanced queueing approaches which 320 allow the construction of virtual sub-interfaces, each of which is 321 provided with dedicated resource in a shared physical interface. 322 These approaches are described in more detail later in this document. 324 Section 2.1.1 explores pragmatic approaches to isolation in packet 325 networks. 327 2.1.1. A Pragmatic Approach to Isolation 329 A key question is whether it is possible to achieve hard isolation 330 in packet networks that were never designed to support hard 331 isolation. On the contrary, they were designed to provide 332 statistical multiplexing, a significant economic advantage when 333 compared to a dedicated, or a Time Division Multiplexing (TDM) 334 network. However, there is no need to provide any harder isolation 335 than is required by the applications. An approximation to this 336 requirement is sufficient in most cases. Pseudowires [RFC3985] 337 emulate services that would have had hard isolation in their native 338 form. 340 This spectrum of isolation is shown in Figure 1: 342 O=================================================O 343 | \---------------v---------------/ 344 Statistical Pragmatic Absolute 345 Multiplexing Isolation Isolation 346 (Traditional VPNs) (Enhanced VPN) (Dedicated Network) 348 Figure 1 The Spectrum of Isolation 350 Figure 1 shows the spectrum of isolation that may be delivered by a 351 network. At one end of the figure, we have traditional statistical 352 multiplexing technologies that support VPNs. This is a service type 353 that has served the industry well and will continue to do so. At 354 the opposite end of the spectrum, we have the absolute isolation 355 provided by dedicated transport networks. The goal of enhanced VPNs 356 is "pragmatic isolation". This is isolation that is better than is 357 obtainable from pure statistical multiplexing, more cost effective 358 and flexible than a dedicated network, but which is a practical 359 solution that is good enough for the majority of applications. 360 Mechanisms for both soft isolation and hard isolation would be 361 needed to meet different levels of service requirement. 363 2.2. Performance Guarantee 365 There are several kinds of performance guarantee, including 366 guaranteed maximum packet loss, guaranteed maximum delay, and 367 guaranteed delay variation. Note that these guarantees apply to 368 conformance traffic, out-of-profile traffic will be handled 369 according to other requirements. 371 Guaranteed maximum packet loss is a common parameter, and is usually 372 addressed by setting packet priorities, queue size, and discard 373 policy. However this becomes more difficult when the requirement is 374 combined with latency requirements. The limiting case is zero 375 congestion loss, and that is the goal of the Deterministic 376 Networking work that the IETF [DETNET] and IEEE [TSN] are pursuing. 377 In modern optical networks, loss due to transmission errors already 378 approaches zero, but there are the possibilities of failure of the 379 interface or the fiber itself. This can only be addressed by some 380 form of signal duplication and transmission over diverse paths. 382 Guaranteed maximum latency is required in a number of applications 383 particularly real-time control applications and some types of 384 virtual reality applications. The work of the IETF Deterministic 385 Networking (DetNet) Working Group [DETNET] is relevant; however 386 additional methods of enhancing the underlay to better support the 387 delay guarantees may be needed, and these methods will need to be 388 integrated with the overall service provisioning mechanisms. 390 Guaranteed maximum delay variation is a service that may also be 391 needed. [RFC8578] calls up a number of cases where this is needed, 392 for example in electrical utilities. Time transfer is one example of 393 a service that needs this, although it is in the nature of time that 394 the service might be delivered by the underlay as a shared service 395 and not provided through different virtual networks. Alternatively 396 a dedicated virtual network may be used to provide this as a shared 397 service. 399 This suggests that a spectrum of service guarantee be considered 400 when deploying an enhanced VPN. As a guide to understanding the 401 design requirements we can consider four types: 403 o Best effort 405 o Assured bandwidth 407 o Guaranteed latency 409 o Enhanced delivery 411 Best effort service is the basic service that current VPNs provide. 413 An assured bandwidth service is one in which the bandwidth over some 414 period of time is assured. This can be achieved either simply based 415 on best effort with over-capacity provisioning, or it can be based 416 on TE-LSPs with bandwidth reservation. The instantaneous bandwidth 417 is however, not necessarily assured, depending on the technique used. 418 Providing assured bandwidth to VPNs, for example by using per-VPN 419 TE-LSPs, is not widely deployed at least partially due to 420 scalability concerns. 422 Guaranteed latency and enhanced delivery are not yet integrated with 423 VPNs. A guaranteed latency service has a latency upper bound 424 provided by the network. Assuring the upper bound is sometimes more 425 important than minimizing latency. 427 There are several new technologies that provide some assistance with 428 performance guarantee. Firstly there is the IEEE project on Time 429 Sensitive Networking [TSN] which introduces the concept of packet 430 scheduling of delay and loss sensitive packets. Then the DetNet work 431 is also of greater relevance in assuring upper bound of end-to-end 432 packet latency. Flex Ethernet [FLEXE] is also useful to provide 433 these guarantees. 435 An enhanced delivery service is one in which the underlay network 436 (at Layer 3) attempts to deliver the packet through multiple paths 437 in the hope of eliminating packet loss due to equipment or media 438 failures. 440 It is these last two characteristics (guaranteed upper bound to 441 latency and elimination of packet loss) that an enhanced VPN adds to 442 a VPN service. 444 2.3. Integration 446 The only way to achieve the enhanced characteristics provided by an 447 enhanced VPN (such as guaranteed or predicted performance) is by 448 integrating the overlay VPN with a particular set of network 449 resources in the underlay network which are allocated to meet the 450 service requirement. This needs be done in a flexible and scalable 451 way so that it can be widely deployed in operator networks to 452 support a reasonable number of enhanced VPN customers. 454 Taking mobile networks and in particular 5G into consideration, the 455 integration of network and the service functions is a likely 456 requirement. The work in IETF SFC working group [SFC] provides a 457 foundation for this integration. 459 2.3.1. Abstraction 461 Integration of the overlay VPN and the underlay network resources 462 does not need to be a tight mapping. As described in [RFC7926], 463 abstraction is the process of applying policy to a set of 464 information about a TE network to produce selective information that 465 represents the potential ability to connect across the network. The 466 process of abstraction presents the connectivity graph in a way that 467 is independent of the underlying network technologies, capabilities, 468 and topology so that the graph can be used to plan and deliver 469 network services in a uniform way. 471 Virtual networks can be built on top of an abstracted topology that 472 represents the connectivity capabilities of the underlay network as 473 described in the framework for Abstraction and Control of TE 474 Networks (ACTN) described in [RFC8453] as discussed further in 475 Section 4.5. 477 2.4. Dynamic Management 479 Enhanced VPNs need to be created, modified, and removed from the 480 network according to service demand. An enhanced VPN that requires 481 hard isolation (section 2.1) must not be disrupted by the 482 instantiation or modification of another enhanced VPN. Determining 483 whether modification of an enhanced VPN can be disruptive to that 484 VPN, and in particular whether the traffic in flight will be 485 disrupted can be a difficult problem. 487 The data plane aspects of this problem are discussed further in 488 Sections 4.1, 4.2, and 4.3. 490 The control plane aspects of this problem are discussed further in 491 Section 4.4. 493 The management plane aspects of this problem are discussed further 494 in Section 4.5 496 Dynamic changes both to the VPN and to the underlay transport 497 network need to be managed to avoid disruption to services that are 498 sensitive to the change of network performance. 500 In addition to non-disruptively managing the network as a result of 501 gross change such as the inclusion of a new VPN endpoint or a change 502 to a link, VPN traffic might need to be moved as a result of traffic 503 volume changes. 505 2.5. Customized Control 507 In some cases it is desirable that an enhanced VPN has a customized 508 control plane, so that the tenant of the enhanced VPN can have some 509 control of how the resources and functions allocated to this 510 enhanced VPN are used. For example, the tenant may be able to 511 specify the service paths in his own enhanced VPN. Depending on the 512 requirement, an enhanced VPN may have its own dedicated controller, 513 which may be provided with an interface to the control system 514 provided by the network operator. Note that such control is within 515 the scope of the tenant's enhanced VPN, any change beyond that would 516 require some intervention of the operator. 518 A description of the control plane aspects of this problem are 519 discussed further in Section 4.4. A description of the management 520 plane aspects of this feature can be found in Section 4.5. 522 2.6. Applicability 524 The technologies described in this document should be applicable to 525 a number of types of VPN services such as: 527 o Layer 2 point-to-point services such as pseudowires [RFC3985] 529 o Layer 2 VPNs [RFC4664] 531 o Ethernet VPNs [RFC7209] 533 o Layer 3 VPNs [RFC4364], [RFC2764] 535 Where such VPN types need enhanced isolation and delivery 536 characteristics, the technologies described in section 4 can be used 537 to provide an underlay with the required enhanced performance. 539 2.7. Inter-Domain and Inter-Layer Network 541 In some scenarios, an enhanced VPN services may span multiple 542 network domains. A domain is considered to be any collection of 543 network elements within a common realm of address space or path 544 computation responsibility [RFC5151]. In some domains the operator 545 may manage a multi-layered network, for example, a packet network 546 over an optical network. When enhanced VPNs are provisioned in such 547 network scenarios, the technologies used in different network planes 548 (data plane, control plane, and management plane) need to provide 549 mechanisms to support multi-domain and multi-layer coordination and 550 integration, so as to provide the required service characteristics 551 for different enhanced VPNs, and improve network efficiency and 552 operational simplicity. 554 3. Architecture of Enhanced VPN 556 A number of enhanced VPN services will typically be provided by a 557 common network infrastructure. Each enhanced VPN consists of both 558 the overlay and a specific set of network resources and functions 559 allocated in the underlay to satisfy the needs of the VPN tenant. 560 The integration between overlay and various underlay resources 561 ensures the required isolation between different enhanced VPNs, and 562 achieves the guaranteed performance for different services. 564 An enhanced VPN needs to be designed with consideration given to: 566 o An enhanced data plane 568 o A control plane to create enhanced VPNs, making use of the 569 data plane isolation and performance guarantee techniques 571 o A management plane for enhanced VPN service life-cycle 572 management. 574 These required characteristics are expanded below: 576 o Enhanced data plane 578 * Provides the required resource isolation capability, e.g. 579 bandwidth guarantee. 581 * Provides the required packet latency and jitter 582 characteristics. 584 * Provides the required packet loss characteristics. 586 * Provides the mechanism to associate a packet with the set 587 of resources allocated to the enhanced VPN which the packet belongs. 589 o Control plane 591 * Collect information about the underlying network topology 592 and resources available and export this to nodes in the network 593 and/or the centralized controller as required. 595 * Create the required virtual networks with the resource and 596 properties needed by the enhanced VPN services that are assigned to 597 them. 599 * Determine the risk of SLA violation and take appropriate 600 avoiding action. 602 * Determine the right balance of per-packet and per-node 603 state according to the needs of enhanced VPN service to scale to the 604 required size. 606 o Management plane 608 * Provides an interface between the enhanced VPN provider 609 (e.g., the Transport Network Manager) and the enhanced VPN clients 610 (e.g., the 3GPP Management System) such that some of the operation 611 requests can be met without interfering with the enhanced VPN of 612 other clients. 614 * Provides an interface between the enhanced VPN provider and 615 the enhanced VPN clients to expose transport network capability 616 information toward the enhanced VPN client. 618 * Provides the service life-cycle management and operation of 619 enhanced VPN (e.g. creation, modification, assurance/monitoring and 620 decommissioning). 622 o Operations, Administration, and Maintenance (OAM) 624 * Provides the OAM tools to verify the connectivity and 625 performance of the enhanced VPN. 627 * Provide the OAM tools to verify whether the underlay 628 network resources are correctly allocated and operated properly. 630 o Telemetry 632 * Provides the mechanism to collect data plane, control plane, 633 and management plane information about the network. More 634 specifically: 636 + Provides the mechanism to collect network data from the 637 underlay network for overall performance evaluation and the enhanced 638 VPN service planning. 640 + Provides the mechanism to collect network data about 641 each enhanced VPN for monitoring and analytics of the 642 characteristics and SLA fulfilment of enhanced VPN services. 644 3.1. Layered Architecture 646 The layered architecture of an enhanced VPN is shown in Figure 2. 648 Underpinning everything is the physical network infrastructure layer 649 which provide the underlying resources used to provision the 650 separated virtual networks. This includes the partitioning of link 651 and/or node resources. Each subset of link or node resource can be 652 considered as a virtual link or virtual node used to build the 653 virtual networks. 655 A 656 | | 657 +-------------------+ Centralized 658 | Network Controller| Control& Management 659 +-------------------+ 660 || 661 \/ 662 __________________________ 663 / o----o----o / 664 / / / / Virtual 665 / o-----o-----o----o----o / Network-1 666 /_________________________/ 667 __________________________ 668 / o----o / 669 / / / \ / Virtual 670 / o-----o----o----o-----o / Network-2 671 /_________________________/ 672 ...... ... 673 ___________________________ 674 / o----o / 675 / / / / Virtual 676 / o-----o----o----o-----o / Network-N 677 /__________________________/ 679 ++++ ++++ ++++ 680 +--+===+--+===+--+ 681 +--+===+--+===+--+ 682 ++++ +++-\ ++++ Physical 683 || || \\ || 684 || || \\ || Network 685 ++++ ++++ ++++ \\+++ ++++ 686 +--+===+--+===+--+===+--+===+--+ Infrastructure 687 +--+===+--+===+--+===+--+===+--+ 688 ++++ ++++ ++++ ++++ ++++ 690 O Virtual Node 692 -- Virtual Link 694 ++++ 695 +--+ Physical Node with resource partition 696 +--+ 697 ++++ 698 == Physical Link with resource partition 700 Figure 2 The Layered Architecture 702 Various components and techniques discussed in Section 4 can be used 703 to enable resource partition, such as FlexE, Time Sensitive 704 Networking, Deterministic Networking, Dedicated queues, etc. These 705 partitions may be physical, or virtual so long as the SLA required 706 by the higher layers is met. 708 Based on the network resources provided by the physical network 709 infrastructure, multiple virtual networks can be provisioned, each 710 with customized topology and other attributes to meet the 711 requirement of different enhanced VPNs or different groups of 712 enhanced VPNs. To get the required characteristic, each virtual 713 network needs to be mapped to a set of network nodes and links in 714 the network infrastructure. And on each node or link, the virtual 715 network is mapped to a set of resources which are allocated for the 716 service processing of the virtual network. Such mapping provides the 717 integration between the virtual networks and the required underlying 718 network resources. 720 The centralized controller is used to create the virtual networks, 721 to instruct the network nodes to allocate the required resources to 722 each virtual network and to provision the enhanced VPN services 723 within the virtual networks. A distributed control plane may also 724 be used for the distribution of the topology and attribute 725 information between nodes within the virtual networks. 727 The process used to create virtual networks and to allocate physical 728 resources for use by virtual networks needs to take a holistic view 729 of the needs of all of its tenants (i.e., of all customers and their 730 virtual networks), and to partition the resources accordingly. 731 However, within a virtual network these resources can, if required, 732 be managed via a dynamic control plane. This provides the required 733 scalability and isolation. 735 3.2. Multi-Point to Multi-Point (MP2MP) Connectivity 737 At the VPN service level, the required connectivity is usually mesh 738 or partial-mesh. To support such kinds of VPN service, the 739 corresponding underlay is also an abstract MP2MP medium. Other 740 service requirements may be expressed at different granularity, some 741 of which can be applicable to the whole service, while some others 742 may be only applicable to some pairs of end points. For example, 743 when performance guarantee is required, the point-to-point path 744 through the underlay of the enhanced VPN may need to be specifically 745 engineered to meet the required performance guarantee. 747 3.3. Application Specific Network Types 749 Although a lot of the traffic that will be carried over the enhanced 750 VPN will likely be IPv4 or IPv6, the design has to be capable of 751 carrying other traffic types, in particular Ethernet traffic. This 752 is easily accomplished through the various pseudowire (PW) 753 techniques [RFC3985]. Where the underlay is MPLS, Ethernet can be 754 carried over the enhanced VPN encapsulated according to the method 755 specified in [RFC4448]. Where the underlay is IP, Layer Two 756 Tunneling Protocol - Version 3 (L2TPv3) [RFC3931] can be used with 757 Ethernet traffic carried according to [RFC4719]. Encapsulations 758 have been defined for most of the common Layer 2 types for both PW 759 over MPLS and for L2TPv3. 761 3.4. Scaling Considerations 763 VPNs are instantiated as overlays on top of an operator's network 764 and offered as services to the operator's customers. An important 765 feature of overlays is that they are able to deliver services 766 without placing per-service state in the core of the underlay 767 network. 769 Enhanced VPNs may need to install some additional state within the 770 network to achieve the additional features that they require. 771 Solutions must consider minimizing and controlling the scale of such 772 state, and deployment architectures should constrain the number of 773 enhanced VPNs that would exist where such services would place 774 additional state in the network. It is expected that the number of 775 enhanced VPN would be small in the beginning, and even in future the 776 number of enhanced VPN will be much fewer than traditional VPNs, 777 because pre-existing VPN techniques are be good enough to meet the 778 needs of most existing VPN-type services. 780 In general, it is not required that the state in the network be 781 maintained in a 1:1 relationship with the VPN+ instances. It will 782 usually be possible to aggregate a set of VPN+ services so that they 783 share the same virtual network and the same set of network resources 784 (much in the way that current VPNs are aggregated over transport 785 tunnels) so that collections of enhanced VPNs that require the same 786 behaviour from the network in terms of resource reservation, latency 787 bounds, resiliency, etc. are able to be grouped together. This is 788 an important feature to assist with the scaling characteristics of 789 VPN+ deployments. 791 See Section 5 for a further discussion of scalability considerations. 793 4. Candidate Technologies 795 A VPN is a network created by applying a demultiplexing technique to 796 the underlying network (the underlay) in order to distinguish the 797 traffic of one VPN from that of another. A VPN path that travels by 798 other than the shortest path through the underlay normally requires 799 state in the underlay to specify that path. State is normally 800 applied to the underlay through the use of the RSVP signaling 801 protocol, or directly through the use of an SDN controller, although 802 other techniques may emerge as this problem is studied. This state 803 gets harder to manage as the number of VPN paths increases. 804 Furthermore, as we increase the coupling between the underlay and 805 the overlay to support the enhanced VPN service, this state will 806 increase further. 808 In an enhanced VPN different subsets of the underlay resources can 809 be dedicated to different enhanced VPNs or different groups of 810 enhanced VPNs. An enhanced VPN solution thus needs tighter coupling 811 with underlay than is the case with existing VPNs. We cannot, for 812 example, share the network resource between enhanced VPNs which 813 require hard isolation. 815 4.1. Layer-Two Data Plane 817 A number of candidate Layer 2 packet or frame-based data plane 818 solutions which can be used provide the required isolation and 819 guarantees are described in following sections. 821 4.1.1. Flexible Ethernet 823 FlexE [FLEXE] provides the ability to multiplex channels over an 824 Ethernet link to create point-to-point fixed-bandwidth connections 825 in a way that provides hard isolation. FlexE also supports bonding 826 links to create larger links out of multiple low capacity links. 828 However, FlexE is only a link level technology. When packets are 829 received by the downstream node, they need to be processed in a way 830 that preserves that isolation in the downstream node. This in turn 831 requires a queuing and forwarding implementation that preserves the 832 end-to-end isolation. 834 If different FlexE channels are used for different services, then no 835 sharing is possible between the FlexE channels. This means that it 836 may be difficult to dynamically redistribute unused bandwidth to 837 lower priority services in another FlexE channel. If one FlexE 838 channel is used by one tenant, the tenant can use some methods to 839 manage the relative priority of his own traffic in the FlexE channel. 841 4.1.2. Dedicated Queues 843 DiffServ based queuing systems are described in [RFC2475] and 844 [RFC4594]. This is considered insufficient to provide isolation for 845 enhanced VPNs because DiffServ does not always provide enough 846 markers to differentiate between traffic of many enhanced VPNs, or 847 offer the range of service classes that each VPN needs to provide to 848 its tenants. This problem is particularly acute with an MPLS 849 underlay, because MPLS only provides eight Traffic Classes. 851 In addition, DiffServ, as currently implemented, mainly provides 852 per-hop priority-based scheduling, and it is difficult to use it to 853 achieve quantitive resource reservation. 855 In order to address these problems and to reduce the potential 856 interference between enhanced VPNs, it would be necessary to steer 857 traffic to dedicated input and output queues per enhanced VPN: some 858 routers have a large number of queues and sophisticated queuing 859 systems, which could support this, while some routers may struggle 860 to provide the granularity and level of isolation required by the 861 applications of enhanced VPN. 863 4.1.3. Time Sensitive Networking 865 Time Sensitive Networking (TSN) [TSN] is an IEEE project that is 866 designing a method of carrying time sensitive information over 867 Ethernet. It introduces the concept of packet scheduling where a 868 packet stream may be given a time slot guaranteeing that it 869 experiences no queuing delay or increase in latency. The mechanisms 870 defined in TSN can be used to meet the requirements of time 871 sensitive services of an enhanced VPN. 873 Ethernet can be emulated over a Layer 3 network using an IP or MPLS 874 pseudowire. However, a TSN Ethernet payload would be opaque to the 875 underlay and thus not treated specifically as time sensitive data. 876 The preferred method of carrying TSN over a Layer 3 network is 877 through the use of deterministic networking as explained in Section 878 4.2.1. 880 4.2. Layer-Three Data Plane 882 We now consider the problem of slice differentiation and resource 883 representation in the network layer. 885 4.2.1. Deterministic Networking 887 Deterministic Networking (DetNet) [RFC8655] is a technique being 888 developed in the IETF to enhance the ability of Layer 3 networks to 889 deliver packets more reliably and with greater control over the 890 delay. The design cannot use re-transmission techniques such as TCP 891 since that can exceed the delay tolerated by the applications. Even 892 the delay improvements that are achieved with Stream Control 893 Transmission Protocol Partial Reliability Extension (SCTP-PR) 894 [RFC3758] do not meet the bounds set by application demands. DetNet 895 pre-emptively sends copies of the packet over various paths to 896 minimize the chance of all copies of a packet being lost. It also 897 seeks to set an upper bound on latency, but the goal is not to 898 minimize latency. 900 4.2.2. MPLS Traffic Engineering (MPLS-TE) 902 MPLS-TE [RFC2702] [RFC3209] introduces the concept of reserving end- 903 to-end bandwidth for a TE-LSP, which can be used as connectivity 904 across the underlay network to support VPNs. VPN traffic can be 905 carried over dedicated TE-LSPs to provide reserved bandwidth for 906 each specific connection in a VPN, and VPNs with similar behavior 907 requirements may be multiplexed onto the same TE-LSPs. Some network 908 operators have concerns about the scalability and management 909 overhead of MPLS-TE system, and this has lead them to consider other 910 solutions for their networks. 912 4.2.3. Segment Routing 914 Segment Routing (SR) [RFC8402] is a method that prepends 915 instructions to packets at the head-end of a path. These 916 instructions are used to specify the nodes and links to be traversed 917 and allow the packets to be routed on paths other than the shortest 918 path. By encoding the state in the packet, per-path state is 919 transitioned out of the network. 921 An SR traffic engineered path operates with a granularity of a link 922 with hints about priority provided through the use of the traffic 923 class (TC) or Differentiated Services Code Point (DSCP) field in the 924 header. However to achieve the latency and isolation 925 characteristics that are sought by the enhanced VPN users, steering 926 packets through specific queues and resources will likely be 927 required. With SR, it is possible to introduce such fine-grained 928 packet steering by specifying the queues and resources through an SR 929 instruction list. 931 Note that the concept of queue is a useful abstraction for different 932 types of underlay mechanism that may be used to provide enhanced 933 isolation and latency support. How the queue satisfies the 934 requirement is implementation specific and is transparent to the 935 layer-3 data plane and control plane mechanisms used. 937 4.3. Non-Packet Data Plane 939 Non-packet underlay data plane technologies often have TE properties 940 and behaviors, and meet many of the key requirements in particular 941 for bandwidth guarantees, traffic isolation (with physical isolation 942 often being an integral part of the technology), highly predictable 943 latency and jitter characteristics, measurable loss characteristics, 944 and ease of identification of flows. The cost is the resources are 945 allocated on a long term and end-to-end basis. Such an arrangement 946 means that the full cost of the resources has be borne by the 947 service that is allocated with the resources. 949 4.4. Control Plane 951 Enhanced VPN would likely be based on a hybrid control mechanism, 952 which takes advantage of the logically centralized controller for 953 on-demand provisioning and global optimization, whilst still relying 954 on a distributed control plane to provide scalability, high 955 reliability, fast reaction, automatic failure recovery, etc. 956 Extension to and optimization of the distributed control plane is 957 needed to support the enhanced properties of VPN+. 959 RSVP-TE [RFC3209] provides the signaling mechanism for establishing 960 a TE-LSP in an MPLS network with end-to-end resource reservation. 961 It could be used to bind the VPN to specific network resources 962 allocated within the underlay, but there remain scalability concerns 963 mentioned in Section 4.2.2. 965 The control plane of SR [RFC8665] [RFC8667] [I-D.ietf-idr-bgp-ls- 966 segment-routing-ext] does not have the capability of signaling 967 resource reservations along the path. On the other hand, the SR 968 approach provides a potential way of binding the underlay network 969 resource and the enhanced VPN service without requiring per-path 970 state to be maintained in the network. A centralized controller can 971 perform resource planning and reservation for enhanced VPNs, while 972 it needs to ensure that resources are correctly allocated in network 973 nodes for the enhanced VPN service. 975 4.5. Management Plane 977 The management plane provides the interface between the enhanced VPN 978 provider and the clients for the service life-cycle management (e.g. 979 creation, modification, assurance/monitoring and decommissioning). 980 It relies on a set of service data models for the description of the 981 information and operations needed on the interface. 983 In the context of 5G end-to-end network slicing [TS28530], the 984 management of enhanced VPNs is considered as the management of the 985 transport network part of the end-to-end network slice. 3GPP 986 management system may provide the connectivity and performance 987 related parameters as requirements to the management plane of the 988 transport network. It may also require the transport network to 989 expose the capability and status of the transport network slice. 990 Thus, an interface between the enhanced VPN management plane and the 991 3GPP network slice management system, and relevant service data 992 models are needed for the coordination of end-to-end network slice 993 management. 995 The management plane interface and data models for enhanced VPN can 996 be based on the service models described in Section 4.6. 998 4.6. Applicability of Service Data Models to Enhanced VPN 1000 ACTN supports operators in viewing and controlling different domains 1001 and presenting virtualized networks to their customers. The ACTN 1002 framework [RFC8453] highlights how: 1004 o Abstraction of the underlying network resources is provided to 1005 higher-layer applications and customers. 1007 o Underlying resources are virtualized allocating those resources 1008 for the customer, application, or service. 1010 o A virtualized environment is created allowing operators to view 1011 and control multi-domain networks as a single virtualized network. 1013 o Networks can be presented to customers as a virtual network via 1014 open and programmable interfaces. 1016 The type of network virtualization enabled by ACTN managed 1017 infrastructure provides customers and applications (tenants) with 1018 the capability to utilize and independently control allocated 1019 virtual network resources as if they were physically their own 1020 resources. Service Data models are used to represent, monitor, and 1021 manage the virtual networks and services enabled by ACTN. The 1022 Customer VPN model (e.g. L3SM [RFC8299]) or an ACTN Virtual Network 1023 (VN) [I-D.ietf-teas-actn-vn-yang] model is a customer view of the 1024 ACTN managed infrastructure, and is presented by the ACTN provider 1025 as a set of abstracted services or resources. The L3VPN network 1026 model [I-D.ietf-opsawg-l3sm-l3nm] and the TE tunnel model [I-D.ietf- 1027 teas-yang-te] provide a network view of the ACTN managed 1028 infrastructure presented by the ACTN provider as a set of transport 1029 resources. 1031 4.6.1. Enhanced VPN Delivery in the ACTN Architecture 1033 ACTN provides VPN connections between multiple sites as requested 1034 via the Customer Network Controller (CNC). The CNC is managed by 1035 the customer themselves, and interacts with the network provider's 1036 Multi-Domain Service Controller (MDSC). The Provisioning Network 1037 Controllers (PNC) are responsible for network resource management, 1038 thus the PNCs are remain entirely under the management of the 1039 network provider and are not visible to the customer so that 1040 management is mostly performed by the network provider, with some 1041 flexibility delegated to the customer-managed CNC. 1043 Figure 3 presents a more general representation of how multiple 1044 enhanced VPNs may be created from the resources of multiple physical 1045 networks using the CNC, MDSC, and PNC components of the ACTN 1046 architecture. Each enhanced VPN is controlled by its own CNC. The 1047 CNCs send requests to the provider's MDSC. The provider manages two 1048 different physical networks each under the control of PNC. The MDSC 1049 asks the PNCs to allocate and provision resources to achieve the 1050 enhanced VPNs. In this figure, one enhanced VPN is constructed 1051 solely from the resources of one of the physical networks, while the 1052 the VPN uses resources from both physical networks. 1054 --------------- ( ) 1055 | CNC |---------->( VPN+ ) 1056 --------^------ ( ) 1057 | _(_________ _) 1058 --------------- ( ) ^ 1059 | CNC |----------->( VPN+ ) : 1060 ------^-------- ( ) : 1061 | | (___________) : 1062 | | ^ ^ : 1063 Boundary | | : : : 1064 Between ==========|====|===================:====:====:======== 1065 Customer & | | : : : 1066 Network Provider | | : : : 1067 v v : : : 1068 --------------- : :....: 1069 | MDSC | : : 1070 --------------- : : 1071 ^ ---^------ ... 1072 | ( ) . 1073 v ( Physical ) . 1074 ---------------- ( Network ) . 1075 | PNC |<-------->( ) ---^------ 1076 ---------------- | -------- ( ) 1077 | |-- ( Physical ) 1078 | PNC |<------------------------->( Network ) 1079 --------------- ( ) 1080 -------- 1082 Figure 3 Generic VPN+ Delivery in the ACTN Architecture 1084 4.6.2. Enhanced VPN Features with Service Data Models 1086 This section discusses how the service data models can fulfil the 1087 enhanced VPN requirements described earlier in this document within 1088 the scope of the ACTN architecture. 1090 4.6.2.1. Isolation Between VPNs 1092 The VN YANG model [I-D.ietf-teas-actn-vn-yang] and the TE-service 1093 mapping model [I-D.ietf-teas-te-service-mapping-yang] fulfil the VPN 1094 isolation requirement by providing the following features for the 1095 VPNs: 1097 o Each VPN is identified with a unique identifier (vpn-id) and 1098 can be mapped to a specific VN. Multiple VPNs may mapped to the 1099 same VN according to service requirements and operator's policy. 1101 o Each VPN is managed and controlled independent of other VPNs. 1103 o Each VPN is instantiated with an isolation requirement 1104 described by the TE-service mapping model [I-D.ietf-teas-te-service- 1105 mapping-yang]. This mapping supports all levels of isolation (hard 1106 isolation with deterministic characteristics, hard isolation, soft 1107 isolation, or no isolation). 1109 4.6.2.2. Guaranteed Performance 1111 Performance objectives of a VPN [RFC8299][RFC8466] are expressed 1112 through a QoS profile including the following properties: 1114 o Rate-limit 1116 o Bandwidth 1118 o Latency 1120 o Jitter 1122 [I-D.ietf-teas-actn-vn-yang] and [I-D.ietf-teas-yang-te-topo] allow 1123 configuration of several TE parameters that may help to meet the VPN 1124 performance objectives as follows: 1126 o Bandwidth 1128 o Objective function (e.g., min cost path, min load path, etc.) 1130 o Metric Types and their threshold: 1132 * TE cost, IGP cost, Hop count, or Unidirectional Delay (e.g., 1133 can set all path delay <= threshold) 1135 Once these requests are instantiated, the resources are committed 1136 and guaranteed through the life cycle of the VPN. 1138 4.6.2.3. Integration 1140 The L3VPN network model provides mechanism to correlate customer's 1141 VPN and the VPN service related resources (e.g., RT and RD) 1142 allocated in the provider's network. 1144 The VPN/Network performance monitoring model [I-D.www-bess-yang-vpn- 1145 service-pm] provides mechanisms to monitor and manage network 1146 Performance on the topology at different layer or the overlay 1147 topology between VPN sites. 1149 These two models provide mechanisms to correlate the customer's VPN 1150 and the actual TE tunnels instantiated in the provider's network. 1152 Service function integration with network topology (L3 and TE 1153 topology) is in progress in [I-D.ietf-teas-sf-aware-topo-model] 1154 which addresses a number of use-cases that show how TE topology 1155 supports various service functions. 1157 4.6.2.4. Dynamic and Customized Management 1159 The ACTN architecture allows the CNC to interact with the provider's 1160 MDSC. This gives the customer dynamic control of their VPNs. 1162 For example, the ACTN VN model [I-D.ietf-teas-actn-vn-yang] allows 1163 life-cycle management to create, modify, and delete VNs on demand. 1164 Customers may also be allowed more customized control of the VN 1165 topology by provisioning tunnels to connect their endpoints, and 1166 even configuring the paths of those tunnels. 1168 Another example is the L3VPN service model [RFC8299] which allows 1169 VPN lifecycle management such as VPN creation, modification, and 1170 deletion on demand. 1172 4.6.3. 5G Transport Service Delivery via Coordinated Data Modules 1174 The overview of network slice structure as defined in the 3GPP 5GS 1175 is shown in Figure 4. The terms are described in specific 3GPP 1176 documents [TS23501] [TS28530]. 1178 <================== E2E-NSI =======================> 1179 : : : : : 1180 : : : : : 1181 <====== RAN-NSSI ======><=TN-NSSI=><====== CN-NSSI ======>VL[APL] 1182 : : : : : : : : : 1183 : : : : : : : : : 1184 RW[NFs ]<=TRN-NSSI=>[NFs ]<=TN-NSSI=>[NFs ]<=TRN-NSSI=>[NFs ]VL[APL] 1185 . . . . . . . . . . . . .. . . . . . . . . . . . . .. 1186 .,----. ,----. ,----.. ,----. .,----. ,----. ,----.. 1187 UE--|RAN |---| TN |---|RAN |---| TN |--|CN |--| TN |--|CN |--[APL] 1188 .|NFs | `----' |NFs |. `----' .|NFs | `----' |NFs |. 1189 .`----' `----'. .`----' `----'. 1190 . . . . . . . . . . . . .. . . . . . . . . . . . .. 1192 RW RAN MBH CN DN 1194 *Legends 1195 UE: User Equipment 1196 RAN: Radio Access Network 1197 CN: Core Network 1198 DN: Data Network 1199 TN: Transport Network 1200 MBH: Mobile Backhaul 1201 RW: Radio Wave 1202 NF: Network Function 1203 APL: Application Server 1204 NSI: Network Slice Instance 1205 NSSI: Network Slice Subnet Instance 1207 Figure 4 Overview of Structure of Network Slice in 3GPP 5GS 1209 The L3VPN service model [RFC8299] and TEAS VN model [I-D.ietf-teas- 1210 actn-vn-yang] can both be used to describe the 5G MBB Transport 1211 Service or connectivity service. The L3VPN service model is used to 1212 describe end-to-end IP connectivity service, while the TEAS VN model 1213 is used to describe TE connectivity service between VPN sites or 1214 between RAN NFs and Core network NFs. 1216 A VN in the TEAS VN model with its support of point-to-point or 1217 multipoint-to-multipoint connectivity services can be seen as one 1218 example of a network slice. 1220 The TE Service mapping model can be used to map L3VPN service 1221 requests onto underlying network resource and TE models to get the 1222 TE network provisioned. 1224 For IP VPN service provisioning, the service parameters in the L3VPN 1225 service model [RFC8299] can be decomposed into a set of 1226 configuration parameters described in the L3VPN network model [I- 1227 D.ietf-opsawg-l3sm-l3nm] which will get the VPN network provisioned. 1229 5. Scalability Considerations 1231 Enhanced VPN provides performance guaranteed services in packet 1232 networks, but with the potential cost of introducing additional 1233 states into the network. There are at least three ways that this 1234 additional state might be presented in the network: 1236 o Introduce the complete state into the packet, as is done in SR. 1237 This allows the controller to specify a detailed series of 1238 forwarding and processing instructions for the packet as it transits 1239 the network. The cost of this is an increase in the packet header 1240 size. The cost is also that systems will have capabilities enabled 1241 in case they are called upon by a service. This is a type of latent 1242 state, and increases as we more precisely specify the path and 1243 resources that need to be exclusively available to a VPN. 1245 o Introduce the state to the network. This is normally done by 1246 creating a path using RSVP-TE, which can be extended to introduce 1247 any element that needs to be specified along the path, for example 1248 explicitly specifying queuing policy. It is possible to use other 1249 methods to introduce path state, such as via a Software Defined 1250 Network (SDN) controller, or possibly by modifying a routing 1251 protocol. With this approach there is state per path, per path 1252 characteristic that needs to be maintained over its life-cycle. 1253 This is more state than is needed using SR, but the packets are 1254 shorter. 1256 o Provide a hybrid approach. One example is based on using 1257 binding SIDs [RFC8402] to create path fragments, and bind them 1258 together with SR. Dynamic creation of a VPN service path using SR 1259 requires less state maintenance in the network core at the expense 1260 of larger packet headers. The packet size can be lower if a form of 1261 loose source routing is used (using a few nodal SIDs), and it will 1262 be lower if no specific functions or resources on the routers are 1263 specified. 1265 Reducing the state in the network is important to enhanced VPN, as 1266 it requires the overlay to be more closely integrated with the 1267 underlay than with traditional VPNs. This tighter coupling would 1268 normally mean that more state needed to be created and maintained in 1269 the network, as the state about fine granularity processing would 1270 need to be loaded and maintained in the routers. However, a segment 1271 routed approach allows much of this state to be spread amongst the 1272 network ingress nodes, and transiently carried in the packets as 1273 SIDs. 1275 5.1. Maximum Stack Depth of SR 1277 One of the challenges with SR is the stack depth that nodes are able 1278 to impose on packets [RFC8491]. This leads to a difficult balance 1279 between adding state to the network and minimizing stack depth, or 1280 minimizing state and increasing the stack depth. 1282 5.2. RSVP Scalability 1284 The traditional method of creating a resource allocated path through 1285 an MPLS network is to use the RSVP protocol. However there have 1286 been concerns that this requires significant continuous state 1287 maintenance in the network. Work to improve the scalability of 1288 RSVP-TE LSPs in the control plane can be found in [RFC8370]. 1290 There is also concern at the scalability of the forwarder footprint 1291 of RSVP as the number of paths through an LSR grows. [RFC8577] 1292 proposes to address this by employing SR within a tunnel established 1293 by RSVP-TE. 1295 5.3. SDN Scaling 1297 The centralized approach of SDN requires state to be stored in the 1298 network, but does not have the overhead of also requiring control 1299 plane state to be maintained. Each individual network node may need 1300 to maintain a communication channel with the SDN controller, but 1301 that compares favourably with the need for a control plane to 1302 maintain communication with all neighbors. 1304 However, SDN may transfer some of the scalability concerns from the 1305 network to the centralized controller. In particular, there may be 1306 a heavy processing burden at the controller, and a heavy load in the 1307 network surrounding the controller. 1309 6. OAM Considerations 1311 The enhanced VPN OAM design needs to consider the following 1312 requirements: 1314 o Instrumentation of the underlay so that the network operator can 1315 be sure that the resources committed to a tenant are operating 1316 correctly and delivering the required performance. 1318 o Instrumentation of the overlay by the tenant. This is likely to 1319 be transparent to the network operator and to use existing methods. 1320 Particular consideration needs to be given to the need to verify the 1321 isolation and the various committed performance characteristics. 1323 o Instrumentation of the overlay by the network provider to 1324 proactively demonstrate that the committed performance is being 1325 delivered. This needs to be done in a non-intrusive manner, 1326 particularly when the tenant is deploying a performance sensitive 1327 application. 1329 o Verification of the conformity of the path to the service 1330 requirement. This may need to be done as part of a commissioning 1331 test. 1333 A study of OAM in SR networks has been documented in [RFC8403]. 1335 7. Telemetry Considerations 1337 Network visibility is essential for network operation. Network 1338 telemetry has been considered as an ideal means to gain sufficient 1339 network visibility with better flexibility, scalability, accuracy, 1340 coverage, and performance than conventional OAM technologies. 1342 As defined in [I-D.ietf-opsawg-ntf], the purpose of Network 1343 Telemetry is to acquire network data remotely for network monitoring 1344 and operation. It is a general term for a large set of network 1345 visibility techniques and protocols. Network telemetry addresses 1346 the current network operation issues and enables smooth evolution 1347 toward intent-driven autonomous networks. Telemetry can be applied 1348 on the forwarding plane, the control plane, and the management plane 1349 in a network. 1351 How the telemetry mechanisms could be used or extended for the 1352 enhanced VPN service will be described in a separate document. 1354 8. Enhanced Resiliency 1356 Each enhanced VPN has a life-cycle, and may need modification during 1357 deployment as the needs of its tenant change. Additionally, as the 1358 network as a whole evolves, there may need to be garbage collection 1359 performed to consolidate resources into usable quanta. 1361 Systems in which the path is imposed such as SR, or some form of 1362 explicit routing tend to do well in these applications, because it 1363 is possible to perform an atomic transition from one path to another. 1364 This is a single action by the head-end changes the path without the 1365 need for coordinated action by the routers along the path. However, 1366 implementations and the monitoring protocols need to make sure that 1367 the new path is up and meets the required SLA before traffic is 1368 transitioned to it. It is possible for deadlocks to arise as a 1369 result of the network becoming fragmented over time, such that it is 1370 impossible to create a new path or to modify an existing path 1371 without impacting the SLA of other paths. Resolution of this 1372 situation is as much a commercial issue as it is a technical issue 1373 and is outside the scope of this document. 1375 There are, however, two manifestations of the latency problem that 1376 are for further study in any of these approaches: 1378 o The problem of packets overtaking one and other if a path latency 1379 reduces during a transition. 1381 o The problem of transient variation in latency in either direction 1382 as a path migrates. 1384 There is also the matter of what happens during failure in the 1385 underlay infrastructure. Fast reroute is one approach, but that 1386 still produces a transient loss with a normal goal of rectifying 1387 this within 50ms [RFC5654]. An alternative is some form of N+1 1388 delivery such as has been used for many years to support protection 1389 from service disruption. This may be taken to a different level 1390 using the techniques proposed by the IETF deterministic network work 1391 with multiple in-network replication and the culling of later 1392 packets [RFC8655]. 1394 In addition to the approach used to protect high priority packets, 1395 consideration has to be given to the impact of best effort traffic 1396 on the high priority packets during a transient. Specifically if a 1397 conventional re-convergence process is used there will inevitably be 1398 micro-loops and whilst some form of explicit routing will protect 1399 the high priority traffic, lower priority traffic on best effort 1400 shortest paths will micro-loop without the use of a loop prevention 1401 technology. To provide the highest quality of service to high 1402 priority traffic, either this traffic must be shielded from the 1403 micro-loops, or micro-loops must be prevented. 1405 9. Operational Considerations 1407 It is likely that enhanced VPN service will be introduced in 1408 networks which already have traditional VPN services deployed. 1409 Depends on service requirement, the tenants or the operator may 1410 choose to use traditional VPN or enhanced VPN to fulfil the service 1411 requirement. The information and parameters to assist such decision 1412 needs to be reflected on the management interface between the 1413 tenants and the operator. 1415 10. Security Considerations 1417 All types of virtual network require special consideration to be 1418 given to the isolation of traffic belonging to different tenants. 1419 That is, traffic belonging to one VPN must not be delivered to end 1420 points outside that VPN. In this regard enhanced VPNs neither 1421 introduce, no experience a greater security risks than other VPNs. 1423 However, in an enhanced Virtual Private Network service the 1424 additional service requirements need to be considered. For example, 1425 if a service requires a specific upper bound to latency then it can 1426 be damaged by simply delaying the packets through the activities of 1427 another tenant, i.e., by introducing bursts of traffic for other 1428 services. 1430 The measures to address these dynamic security risks must be 1431 specified as part to the specific solution are form part of the 1432 isolation requirements of a service. 1434 While an enhanced VPN service may be sold as offering encryption and 1435 other security features as part of the service, customers would be 1436 well advised to take responsibility for their own security 1437 requirements themselves possibly by encrypting traffic before 1438 handing it off to the service provider. 1440 The privacy of enhanced VPN service customers must be preserved. It 1441 should not be possible for one customer to discover the existence of 1442 another customer, nor should the sites that are members of an 1443 enhanced VPN be externally visible. 1445 11. IANA Considerations 1447 There are no requested IANA actions. 1449 12. Contributors 1451 Daniel King 1452 Email: daniel@olddog.co.uk 1454 Adrian Farrel 1455 Email: adrian@olddog.co.uk 1457 Jeff Tansura 1458 Email: jefftant.ietf@gmail.com 1460 Qin Wu 1461 Email: bill.wu@huawei.com 1463 Daniele Ceccarelli 1464 Email: daniele.ceccarelli@ericsson.com 1466 Mohamed Boucadair 1467 Email: mohamed.boucadair@orange.com 1469 Sergio Belotti 1470 Email: sergio.belotti@nokia.com 1472 Haomian Zheng 1473 Email: zhenghaomian@huawei.com 1475 13. Acknowledgments 1477 The authors would like to thank Charlie Perkins, James N Guichard, 1478 John E Drake and Shunsuke Homma for their review and valuable 1479 comments. 1481 This work was supported in part by the European Commission funded 1482 H2020-ICT-2016-2 METRO-HAUL project (G.A. 761727). 1484 14. References 1486 14.1. Normative References 1488 [I-D.ietf-teas-actn-vn-yang] Lee, Y., Dhody, D., Ceccarelli, D., 1489 Bryskin, I., and B. Yoon, "A Yang Data Model for VN 1490 Operation", draft-ietf-teas-actn-vn-yang-07 (work in 1491 progress), October 2019. 1493 [I-D.ietf-teas-te-service-mapping-yang] Lee, Y., Dhody, D., Fioccola, 1494 G., Wu, Q., Ceccarelli, D., and J. Tantsura, "Traffic 1495 Engineering (TE) and Service Mapping Yang Model", draft- 1496 ietf-teas-te-service-mapping-yang-02 (work in progress), 1497 September 2019. 1499 [RFC2764] Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. 1500 Malis, "A Framework for IP Based Virtual Private Networks", 1501 RFC 2764, DOI 10.17487/RFC2764, February 2000, 1502 . 1504 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1505 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1506 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1507 . 1509 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1510 Edge-to-Edge (PWE3) Architecture", RFC 3985, DOI 1511 10.17487/RFC3985, March 2005, . 1514 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 1515 2 Virtual Private Networks (L2VPNs)", RFC 4664, DOI 1516 10.17487/RFC4664, September 2006, 1519 [RFC8299] Wu, Q., Ed., Litkowski, S., Tomotaki, L., and K. Ogaki, 1520 "YANG Data Model for L3VPN Service Delivery", RFC 8299, 1521 DOI 10.17487/RFC8299, January 2018, . 1524 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1525 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1526 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1527 July 2018, . 1529 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1530 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1531 DOI 10.17487/RFC8453, August 2018, . 1534 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1535 Data Model for Layer 2 Virtual Private Network (L2VPN) 1536 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1537 2018, . 1539 14.2. Informative References 1541 [BBF-SD406] "BBF SD-406: End-to-End Network Slicing", 2016, 1542 . 1545 [DETNET] "Deterministic Networking", March , 1546 . 1548 [FLEXE] "Flex Ethernet Implementation Agreement", March 2016, 1549 . 1552 [I-D.ietf-idr-bgp-ls-segment-routing-ext] Previdi, S., Talaulikar, 1553 K., Filsfils, C., Gredler, H., and M. Chen, "BGP Link- 1554 State extensions for Segment Routing", draft-ietf-idr-bgp- 1555 ls-segment-routing-ext-16 (work in progress), June 2019. 1557 [I-D.ietf-opsawg-l3sm-l3nm] Aguado, A., Dios, O., Lopezalvarez, V., 1558 daniel.voyer@bell.ca, d., and L. Munoz, "Layer 3 VPN 1559 Network Model", draft-ietf-opsawg-l3sm-l3nm-01, (work in 1560 progress), November 2019. 1562 [I-D.ietf-opsawg-ntf] Song, H., Qin, F., Martinez-Julia, P., 1563 Ciavaglia, L., and A. Wang, "Network Telemetry Framework", 1564 draft-ietf-opsawg-ntf-02 (work in progress), October 2019. 1566 [I-D.ietf-teas-sf-aware-topo-model] Bryskin, I., Liu, X., Lee, Y., 1567 Guichard, J., Contreras, L., Ceccarelli, D., and J. 1568 Tantsura, "SF Aware TE Topology YANG Model", draft-ietf- 1569 teas-sf-aware-topo-model-04 (work in progress), November 1570 2019. 1572 [I-D.ietf-teas-yang-te] Saad, T., Gandhi, R., Liu, X., Beeram, V., 1573 and I. Bryskin, "A YANG Data Model for Traffic Engineering 1574 Tunnels and Interfaces", draft-ietf-teas-yang-te-22 (work 1575 in progress), November 2019. 1577 [I-D.ietf-teas-yang-te-topo] Liu, X., Bryskin, I., Beeram, V., Saad, 1578 T., Shah, H., and O. Dios, "YANG Data Model for Traffic 1579 Engineering (TE) Topologies", draft-ietf-teas-yang-te- 1580 topo-22 (work in progress), June 2019. 1582 [I-D.www-bess-yang-vpn-service-pm] Wang, Z., Wu, Q., Even, R., Wen, 1583 B., and C. Liu, "A YANG Model for Network and VPN Service 1584 Performance Monitoring", draft-www-bess-yang-vpn-service- 1585 pm-04 (work in progress), November 2019. 1587 [NGMN-NS-Concept] "NGMN NS Concept", 2016, 1588 . 1591 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1592 and W. Weiss, "An Architecture for Differentiated 1593 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1594 . 1596 [RFC2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path 1597 Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000, 1598 . 1600 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1601 Conrad, "Stream Control Transmission Protocol (SCTP) 1602 Partial Reliability Extension", RFC 3758, DOI 1603 10.17487/RFC3758, May 2004, . 1606 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1607 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 1608 3931, DOI 10.17487/RFC3931, March 2005, . 1611 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1612 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1613 2006, . 1615 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 1616 "Encapsulation Methods for Transport of Ethernet over MPLS 1617 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 1618 . 1620 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 1621 Guidelines for DiffServ Service Classes", RFC 4594, DOI 1622 10.17487/RFC4594, August 2006, . 1625 [RFC4719] Aggarwal, R., Ed., Townsley, M., Ed., and M. Dos Santos, 1626 Ed., "Transport of Ethernet Frames over Layer 2 Tunneling 1627 Protocol Version 3 (L2TPv3)", RFC 4719, DOI 1628 10.17487/RFC4719, November 2006, . 1631 [RFC5151] Farrel, A., Ed., Ayyangar, A., and JP. Vasseur, "Inter- 1632 Domain MPLS and GMPLS Traffic Engineering - Resource 1633 Reservation Protocol-Traffic Engineering (RSVP-TE) 1634 Extensions", RFC 5151, DOI 10.17487/RFC5151, February 2008, 1635 . 1637 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1638 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1639 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1640 September 2009, 1642 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 1643 Networking: A Perspective from within a Service Provider 1644 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 1645 . 1647 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 1648 Henderickx, W., and A. Isaac, "Requirements for Ethernet 1649 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 1650 . 1652 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 1653 Ceccarelli, D., and X. Zhang, "Problem Statement and 1654 Architecture for Information Exchange between 1655 Interconnected Traffic-Engineered Networks", BCP 206, RFC 1656 7926, DOI 10.17487/RFC7926, July 2016, . 1659 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 1660 Network Functions and Their Infrastructure", RFC 8172, DOI 1661 10.17487/RFC8172, July 2017, . 1664 [RFC8370] Beeram, V., Ed., Minei, I., Shakir, R., Pacella, D., and T. 1665 Saad, "Techniques to Improve the Scalability of RSVP-TE 1666 Deployments", RFC 8370, DOI 10.17487/RFC8370, May 2018, 1667 . 1669 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1670 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1671 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1672 2018, . 1674 [RFC8491] Tantsura, J., Chunduri, U., Aldrin, S., and L. Ginsberg, 1675 "Signaling Maximum SID Depth (MSD) Using IS-IS", RFC 8491, 1676 DOI 10.17487/RFC8491, November 2018, . 1679 [RFC8568] Bernardos, CJ., Rahman, A., Zuniga, JC., Contreras, LM., 1680 Aranda, P., and P. Lynch, "Network Virtualization Research 1681 Challenges", RFC 8568, DOI 10.17487/RFC8568, April 2019, 1682 . 1684 [RFC8577] Sitaraman, H., Beeram, V., Parikh, T., and T. Saad, 1685 "Signaling RSVP-TE Tunnels on a Shared MPLS Forwarding 1686 Plane", RFC 8577, DOI 10.17487/RFC8577, April 2019, 1687 . 1689 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 1690 RFC 8578, DOI 10.17487/RFC8578, May 2019, 1691 . 1693 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1694 "Deterministic Networking Architecture", RFC 8655, DOI 1695 10.17487/RFC8655, October 2019, . 1698 [RFC8665] Psenak, P., Previdi, S., Filsfils, C., Gredler, H., Shakir, 1699 R., Henderickx, W., and J. Tantsura, "OSPF Extensions for 1700 Segment Routing", RFC 8665, DOI 10.17487/RFC8665, December 1701 2019, . 1703 [RFC8667] Previdi, S., Ginsberg, L., Filsfils, C., Bashandy, A., 1704 Gredler, H., and B. Decraene, "IS-IS Extensions for 1705 Segment Routing", RFC 8667, DOI 10.17487/RFC8667, December 1706 2019, . 1708 [SFC] "Service Function Chaining", 1709 . 1711 [TS23501] "3GPP TS23.501", 2019, 1712 . 1715 [TS28530] "3GPP TS28.530", 2019, 1716 . 1719 [TSN] "Time-Sensitive Networking", . 1721 Authors' Addresses 1723 Jie Dong 1724 Huawei 1726 Email: jie.dong@huawei.com 1728 Stewart Bryant 1729 Futurewei 1731 Email: stewart.bryant@gmail.com 1733 Zhenqiang Li 1734 China Mobile 1736 Email: lizhenqiang@chinamobile.com 1738 Takuya Miyasaka 1739 KDDI Corporation 1741 Email: ta-miyasaka@kddi.com 1743 Young Lee 1744 Sung Kyun Kwan University 1746 Email: younglee.tx@gmail.com