idnits 2.17.1 draft-ietf-teas-enhanced-vpn-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 3 instances of too long lines in the document, the longest one being 1 character in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (September 12, 2019) is 1687 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'APL' is mentioned on line 1328, but not defined == Unused Reference: 'I-D.ietf-detnet-dp-sol-ip' is defined on line 1704, but no explicit reference was found in the text == Unused Reference: 'RFC2992' is defined on line 1755, but no explicit reference was found in the text == Outdated reference: A later version (-24) exists of draft-ietf-teas-actn-vn-yang-06 == Outdated reference: A later version (-15) exists of draft-ietf-teas-te-service-mapping-yang-02 == Outdated reference: A later version (-02) exists of draft-aguado-opsawg-l3sm-l3nm-01 == Outdated reference: A later version (-01) exists of draft-geng-spring-srv6-for-detnet-00 == Outdated reference: A later version (-13) exists of draft-ietf-opsawg-ntf-01 == Outdated reference: A later version (-12) exists of draft-ietf-teas-actn-pm-telemetry-autonomics-00 == Outdated reference: A later version (-12) exists of draft-ietf-teas-sf-aware-topo-model-03 == Outdated reference: A later version (-36) exists of draft-ietf-teas-yang-te-21 == Outdated reference: A later version (-06) exists of draft-www-bess-yang-vpn-service-pm-03 Summary: 1 error (**), 0 flaws (~~), 13 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group J. Dong 3 Internet-Draft Huawei 4 Intended status: Informational S. Bryant 5 Expires: March 15, 2020 Futurewei 6 Z. Li 7 China Mobile 8 T. Miyasaka 9 KDDI Corporation 10 Y. Lee 11 Sung Kyun Kwan University 12 September 12, 2019 14 A Framework for Enhanced Virtual Private Networks (VPN+) Service 15 draft-ietf-teas-enhanced-vpn-03 17 Abstract 19 This document specifies a framework for using existing, modified and 20 potential new networking technologies as components to provide an 21 Enhanced Virtual Private Network (VPN+) service. The purpose is to 22 support the needs of new applications, particularly applications that 23 are associated with 5G services, by utilizing an approach that is 24 based on existing VPN and TE technologies and adds features that 25 specific services require over and above traditional VPNs. 27 Typically, VPN+ will be used to form the underpinning of network 28 slicing, but could also be of use in its own right. It is not 29 envisaged that large numbers of VPN+ instances will be deployed in a 30 network and, in particular, it is not intended that all VPNs 31 supported by a network will use VPN+ techniques. 33 Status of This Memo 35 This Internet-Draft is submitted in full conformance with the 36 provisions of BCP 78 and BCP 79. 38 Internet-Drafts are working documents of the Internet Engineering 39 Task Force (IETF). Note that other groups may also distribute 40 working documents as Internet-Drafts. The list of current Internet- 41 Drafts is at https://datatracker.ietf.org/drafts/current/. 43 Internet-Drafts are draft documents valid for a maximum of six months 44 and may be updated, replaced, or obsoleted by other documents at any 45 time. It is inappropriate to use Internet-Drafts as reference 46 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on March 15, 2020. 50 Copyright Notice 52 Copyright (c) 2019 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (https://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 68 2. Overview of the Requirements . . . . . . . . . . . . . . . . 6 69 2.1. Isolation between Virtual Networks . . . . . . . . . . . 6 70 2.1.1. A Pragmatic Approach to Isolation . . . . . . . . . . 7 71 2.2. Performance Guarantee . . . . . . . . . . . . . . . . . . 8 72 2.3. Integration . . . . . . . . . . . . . . . . . . . . . . . 10 73 2.3.1. Abstraction . . . . . . . . . . . . . . . . . . . . . 11 74 2.4. Dynamic Management . . . . . . . . . . . . . . . . . . . 11 75 2.5. Customized Control . . . . . . . . . . . . . . . . . . . 12 76 2.6. Applicability . . . . . . . . . . . . . . . . . . . . . . 12 77 2.7. Inter-Domain and Inter-Layer Network . . . . . . . . . . 12 78 3. Architecture of Enhanced VPN . . . . . . . . . . . . . . . . 13 79 3.1. Layered Architecture . . . . . . . . . . . . . . . . . . 15 80 3.2. Multi-Point to Multi-Point (MP2MP) . . . . . . . . . . . 16 81 3.3. Application Specific Network Types . . . . . . . . . . . 16 82 3.4. Scaling Considerations . . . . . . . . . . . . . . . . . 16 83 4. Candidate Technologies . . . . . . . . . . . . . . . . . . . 17 84 4.1. Layer-Two Data Plane . . . . . . . . . . . . . . . . . . 17 85 4.1.1. FlexE . . . . . . . . . . . . . . . . . . . . . . . . 18 86 4.1.2. Dedicated Queues . . . . . . . . . . . . . . . . . . 18 87 4.1.3. Time Sensitive Networking . . . . . . . . . . . . . . 19 88 4.2. Layer-Three Data Plane . . . . . . . . . . . . . . . . . 19 89 4.2.1. Deterministic Networking . . . . . . . . . . . . . . 19 90 4.2.2. MPLS Traffic Engineering (MPLS-TE) . . . . . . . . . 20 91 4.2.3. Segment Routing . . . . . . . . . . . . . . . . . . . 20 92 4.3. Non-Packet Data Plane . . . . . . . . . . . . . . . . . . 21 93 4.4. Control Plane . . . . . . . . . . . . . . . . . . . . . . 21 94 4.5. Management Plane . . . . . . . . . . . . . . . . . . . . 22 95 4.6. Applicability of Service Data Models to Enhanced VPN . . 23 96 4.6.1. Enhanced VPN Delivery in ACTN Architecture . . . . . 24 97 4.6.2. Enhanced VPN Features with Service Data Models . . . 25 98 4.6.3. 5G Transport Service Delivery via Coordinated Data 99 Modules . . . . . . . . . . . . . . . . . . . . . . . 28 100 5. Scalability Considerations . . . . . . . . . . . . . . . . . 30 101 5.1. Maximum Stack Depth of SR . . . . . . . . . . . . . . . . 31 102 5.2. RSVP Scalability . . . . . . . . . . . . . . . . . . . . 31 103 5.3. SDN Scaling . . . . . . . . . . . . . . . . . . . . . . . 31 104 6. OAM Considerations . . . . . . . . . . . . . . . . . . . . . 31 105 7. Telemetry Considerations . . . . . . . . . . . . . . . . . . 32 106 8. Enhanced Resiliency . . . . . . . . . . . . . . . . . . . . . 32 107 9. Operational Considerations . . . . . . . . . . . . . . . . . 33 108 10. Security Considerations . . . . . . . . . . . . . . . . . . . 33 109 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 34 110 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 34 111 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 35 112 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 35 113 14.1. Normative References . . . . . . . . . . . . . . . . . . 35 114 14.2. Informative References . . . . . . . . . . . . . . . . . 36 115 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 41 117 1. Introduction 119 Virtual private networks (VPNs) have served the industry well as a 120 means of providing different groups of users with logically isolated 121 access to a common network. The common or base network that is used 122 to provide the VPNs is often referred to as the underlay, and the VPN 123 is often called an overlay. 125 Customers of a network operator may request enhanced overlay services 126 with advanced characteristics such as complete isolation from other 127 services so that changes in one service (such as changes in network 128 load, or events such as congestion or outages) have no effect on the 129 throughput or latency of other services provided to the customer. 131 Driven largely by needs surfacing from 5G, the concept of network 132 slicing has gained traction [NGMN-NS-Concept] [TS23501] [TS28530] 133 [BBF-SD406]. In [TS23501], Network Slice is defined as "a logical 134 network that provides specific network capabilities and network 135 characteristics", and Network Slice Instance is defined as "A set of 136 Network Function instances and the required resources (e.g. compute, 137 storage and networking resources) which form a deployed Network 138 Slice". According to [TS28530], an end-to-end network slice consists 139 of three major network segments: Radio Access Network (RAN), 140 Transport Network (TN) and Core Network (CN). Transport network 141 provides the required connectivity within and between RAN and CN 142 parts, with specific performance commitment. For each end-to-end 143 network slice, the topology and performance requirement on transport 144 network can be very different, which requires transport network to 145 have the capability of supporting multiple different transport 146 network slices. 148 A transport network slice is a virtual (logical) network with a 149 particular network topology and a set of shared or dedicated network 150 resources, which are used to provide the network slice consumer with 151 the required connectivity, appropriate isolation and specific Service 152 Level Agreement (SLA). A transport network slice could span multiple 153 technology (IP, Optical) and multiple administrative domains. 154 Depends on the consumer's requirement, a transport network slice 155 could be isolated from other, often concurrent transport network 156 slices in terms of data plane, control plane and management plane. 158 In the following sections of this document, network slice refers to 159 transport network slice, and is interchangable with enhanced VPN. 160 End-to-end network slice is used to refer to the 5G network slice. 162 Network abstraction is a technique that can be applied to a network 163 domain to select network resources by policy to obtain a view of 164 potential connectivity and a set of service functions. 166 Network slicing builds on the concept of resource management, network 167 virtualization and abstraction to provide performance assurance, 168 flexibility, programmability and modularity. It may use techniques 169 such as Software Defined Networking (SDN) [RFC7149] and Network 170 Function Virtualization (NFV) [RFC8172][RFC8568] to create multiple 171 logical (virtual) networks, each tailored for a set of services or a 172 particular tenant or a group of tenants that share the same set of 173 requirements, on top of a common network. How the network slices are 174 engineered can be deployment-specific. 176 Thus, there is a need to create virtual networks with enhanced 177 characteristics. The tenant of such a virtual network can require a 178 degree of isolation and performance that previously could not be 179 satisfied by traditional overlay VPNs. Additionally, the tenant may 180 ask for some level of control to their virtual networks, e.g., to 181 customize the service paths in a network slice. 183 These enhanced properties cannot be met with pure overlay networks, 184 as they require tighter coordination and integration between the 185 underlay and the overlay network. This document introduces a new 186 network service called Enhanced VPN: VPN+. VPN+ is built from a 187 virtual network which has a customized network topology and a set of 188 dedicated or shared network resources, including invoked service 189 functions, allocated from the underlay network. Unlike a traditional 190 VPN, an enhanced VPN can achieve greater isolation with strict 191 performance guarantees. These new properties, which have general 192 applicability, may also be of interest as part of a network slicing 193 solution, but it is not envisaged that VPN+ techniques will be 194 applied to normal VPN services that can continue to be deployed using 195 pre-existing mechanisms. Furthermore, it is not intended that large 196 numbers of VPN+ instances will be deployed within a single network. 197 See Section 5 for a discussion of scalability considerations. 199 This document specifies a framework for using existing, modified and 200 potential new technologies as components to provide a VPN+ service. 201 Specifically we are concerned with: 203 o The design of the enhanced data plane. 205 o The necessary protocols in both the underlay and the overlay of 206 the enhanced VPN. 208 o The mechanisms to achieve integration between overlay and 209 underlay. 211 o The necessary Operation, Administration, and Management (OAM) 212 methods to instrument an enhanced VPN to make sure that the 213 required Service Level Agreement (SLA) is met, and to take any 214 corrective action to avoid SLA violation, such as switching to an 215 alternate path. 217 The required layered network structure to achieve this is shown in 218 Section 3.1. 220 Note that, in this document, the four terms "VPN", "Enhanced VPN" (or 221 "VPN+"), "Virtual Network (VN)", and "Network Slice" may be 222 considered as describing similar concepts dependent on the viewpoint 223 from which they are used. 225 o An enhanced VPN can be considered as a form of VPN, but with 226 additional service-specific commitments. Thus, care must be taken 227 with the term "VPN" to distinguish normal or legacy VPNs from VPN+ 228 instances. 230 o A Virtual Network is a type of service that connects customer edge 231 points with the additional possibility of requesting further 232 service characteristics in the manner of an enhanced VPN. 234 o An enhanced VPN or VN is made by creating a slice through the 235 resources of the underlay network. 237 o The general concept of network slicing in a TE network is a larger 238 problem space than is addressed by VPN+ or VN, but those concepts 239 are tools to address some aspects or realizations of network 240 slicing. 242 2. Overview of the Requirements 244 In this section we provide an overview of the requirements of an 245 enhanced VPN. 247 2.1. Isolation between Virtual Networks 249 One element of the SLA demanded for an enhanced VPN is the degree of 250 isolation from other services in the network. Isolation is a feature 251 requested by some particular customers in the network. Such a 252 feature is offered by a network operator where the traffic from one 253 service instance is isolated from the traffic of other services. 254 There are different grades of isolation that range from simple 255 separation of traffic on delivery (ensuring that traffic is not 256 delivered to the wrong customer) all the way to complete separation 257 within the underlay so that the traffic from different services use 258 distinct network resources. 260 The terms hard and soft isolation are introduced to identify 261 different isolation cases. A VPN has soft isolation if the traffic 262 of one VPN cannot be received by the customers of another VPN. Both 263 IP and MPLS VPNs are examples of soft isolated VPNs because the 264 network delivers the traffic only to the required VPN endpoints. 265 However, with soft isolation, traffic from one or more VPNs and 266 regular non-VPN traffic may congest the network resulting in packet 267 loss and delay for other VPNs operating normally. The ability for a 268 VPN or a group of VPNs to be sheltered from this effect is called 269 hard isolation, and this property is required by some critical 270 applications. 272 The requirement is for an operator to offer its customers a choice of 273 different degrees of isolation ranging from soft isolation up to hard 274 isolation so that the traffic of tenants/applications using one 275 enhanced VPN can be separated from the traffic of tenants/ 276 applications using another enhanced VPN appropriately. Hard 277 isolation is needed so that applications with exacting requirements 278 can function correctly, despite other demands (perhaps a burst of 279 traffic in another VPN) competing for the underlying resources. In 280 practice isolation may be offered as a spectrum between soft and 281 hard, and in some cases soft and hard isolation may be used in a 282 hierarchical manner. 284 An example of the requirement for hard isolation is a network 285 supporting both emergency services and public broadband multi-media 286 services. During a major incident the VPNs supporting these services 287 would both be expected to experience high data volumes, and it is 288 important that both make progress in the transmission of their data. 290 In these circumstances the VPNs would require an appropriate degree 291 of isolation to be able to continue to operate acceptably. 293 In order to provide the required isolation, resources may have to be 294 reserved in the data plane of the underlay network and dedicated to 295 traffic from a specific VPN or a specific group of VPNs to form 296 different network slices in the underlay network. This may introduce 297 scalability concerns, thus some trade-off needs to be considered to 298 provide the required isolation between network slices while still 299 allowing reasonable sharing inside each network slice. 301 An optical layer can offer a high degree of isolation, at the cost of 302 allocating resources on a long term and end-to-end basis. Such an 303 arrangement means that the full cost of the resources must be borne 304 by the service that is allocated with the resources. On the other 305 hand, where adequate isolation can be achieved at the packet layer, 306 this permits the resources to be shared amongst many services and 307 only dedicated to a service on a temporary basis. This in turn, 308 allows greater statistical multiplexing of network resources and thus 309 amortizes the cost over many services, leading to better economy. 310 However, the different degrees of isolation required by network 311 slicing cannot be entirely met with existing mechanisms such as 312 Traffic Engineered Label Switched Paths (TE-LSPs). This is because 313 most implementations enforce the bandwidth in the data-plane only at 314 the PEs, but at the P routers the bandwidth is only reserved in the 315 control plane, thus bursts of data can accidentally occur at a P 316 router with higher than committed data rate. 318 There are several new technologies that provide some assistance with 319 these data plane issues. Firstly there is the IEEE project on Time 320 Sensitive Networking [TSN] which introduces the concept of packet 321 scheduling of delay and loss sensitive packets. Then there is 322 [FLEXE] which provides the ability to multiplex multiple channels 323 over one or more Ethernet links in a way that provides hard 324 isolation. Finally there are advanced queueing approaches which 325 allow the construction of virtual sub-interfaces, each of which is 326 provided with dedicated resource in a shared physical interface. 327 These approaches are described in more detail later in this document. 329 In the remainder of this section we explore how isolation may be 330 achieved in packet networks. 332 2.1.1. A Pragmatic Approach to Isolation 334 A key question is whether it is possible to achieve hard isolation in 335 packet networks, which were never designed to support hard isolation. 336 On the contrary, they were designed to provide statistical 337 multiplexing, a significant economic advantage when compared to a 338 dedicated, or a Time Division Multiplexing (TDM) network. However 339 there is no need to provide any harder isolation than is required by 340 the application. Pseudowires [RFC3985] emulate services that would 341 have had hard isolation in their native form. An approximation to 342 this requirement is sufficient in most cases. 344 Thus, for example, using FlexE or a virtual sub-interface together 345 with packet scheduling as the isolation mechanism of interface 346 resources, optionally along with the partitioning of node resources, 347 a type of hard isolation can be provided that is adequate for many 348 enhanced VPN applications. Other applications may be either 349 satisfied with a classical VPN with or without reserved bandwidth, or 350 may need a dedicated point to point underlay connection. The needs 351 of each application must be quantified in order to provide an 352 economic solution that satisfies those needs without over- 353 engineering. 355 This spectrum of isolation is shown in Figure 1: 357 O=================================================O 358 | \---------------v---------------/ 359 Statistical Pragmatic Absolute 360 Multiplexing Isolation Isolation 361 (Traditional VPNs) (Enhanced VPN) (Dedicated Network) 363 Figure 1: The Spectrum of Isolation 365 At one end of the above figure, we have traditional statistical 366 multiplexing technologies that support VPNs. This is a service type 367 that has served the industry well and will continue to do so. At the 368 opposite end of the spectrum, we have the absolute isolation provided 369 by dedicated transport networks. The goal of enhanced VPN is 370 pragmatic isolation. This is isolation that is better than is 371 obtainable from pure statistical multiplexing, more cost effective 372 and flexible than a dedicated network, but which is a practical 373 solution that is good enough for the majority of applications. 374 Mechanisms for both soft isolation and hard isolation would be needed 375 to meet different levels of service requirement. 377 2.2. Performance Guarantee 379 There are several kinds of performance guarantees, including 380 guaranteed maximum packet loss, guaranteed maximum delay and 381 guaranteed delay variation. Note that these guarantees apply to the 382 conformance traffic, the out-of-profile traffic will be handled 383 following other requirements. 385 Guaranteed maximum packet loss is a common parameter, and is usually 386 addressed by setting the packet priorities, queue size and discard 387 policy. However this becomes more difficult when the requirement is 388 combined with the latency requirement. The limiting case is zero 389 congestion loss, and that is the goal of the Deterministic Networking 390 work that the IETF [DETNET] and IEEE [TSN] are pursuing. In modern 391 optical networks, loss due to transmission errors already approaches 392 zero, but there are the possibilities of failure of the interface or 393 the fiber itself. This can only be addressed by some form of signal 394 duplication and transmission over diverse paths. 396 Guaranteed maximum latency is required in a number of applications 397 particularly real-time control applications and some types of virtual 398 reality applications. The work of the IETF Deterministic Networking 399 (DetNet) Working Group [DETNET] is relevant; however the scope needs 400 to be extended to methods of enhancing the underlay to better support 401 the delay guarantee, and to integrate these enhancements with the 402 overall service provision. 404 Guaranteed maximum delay variation is a service that may also be 405 needed. [RFC8578] calls up a number of cases where this is needed, 406 for example electrical utilities have an operational need for this. 407 Time transfer is one example of a service that needs this, although 408 it is in the nature of time that the service might be delivered by 409 the underlay as a shared service and not provided through different 410 virtual networks. Alternatively a dedicated virtual network may be 411 used to provide this as a shared service. 413 This suggests that a spectrum of service guarantee be considered when 414 deploying an enhanced VPN. As a guide to understanding the design 415 requirements we can consider four types: 417 o Best effort 419 o Assured bandwidth 421 o Guaranteed latency 423 o Enhanced delivery 425 Best effort service is the basic service that current VPNs can 426 provide. 428 An assured bandwidth service is one in which the bandwidth over some 429 period of time is assured, this can be achieved either simply based 430 on best effort with over-capacity provisioning, or it can be based on 431 TE-LSPs with bandwidth reservation. The instantaneous bandwidth is 432 however, not necessarily assured, depending on the technique used. 434 Providing assured bandwidth to VPNs, for example by using TE-LSPs, is 435 not widely deployed at least partially due to scalability concerns. 436 Guaranteed latency and enhanced delivery are not yet integrated with 437 VPNs. 439 A guaranteed latency service has a latency upper bound provided by 440 the network. Assuring the upper bound is more important than 441 achieving the minimum latency. 443 In Section 2.1 we considered the work of the IEEE Time Sensitive 444 Networking (TSN) project [TSN] and the work of the IETF DetNet 445 Working group [DETNET] in the context of isolation. The TSN and 446 DetNet work is of greater relevance in assuring end-to-end packet 447 latency. It is also of importance in considering enhanced delivery. 449 An enhanced delivery service is one in which the underlay network (at 450 layer 3) attempts to deliver the packet through multiple paths in the 451 hope of eliminating packet loss due to equipment or media failures. 453 It is these last two characteristics that an enhanced VPN adds to a 454 VPN service. 456 Flex Ethernet [FLEXE] is a useful underlay to provide these 457 guarantees. This is a method of providing time-slot based 458 channelization over an Ethernet bearer. Such channels are fully 459 isolated from other channels running over the same Ethernet bearer. 460 As noted elsewhere this produces hard isolation but makes the 461 reclamation of unused bandwidth more difficult. 463 These approaches can be used in tandem. It is possible to use FlexE 464 to provide tenant isolation, and then to use the TSN/Detnet approach 465 to provide a performance guarantee inside the a slice or tenant VPN. 467 2.3. Integration 469 The only way to achieve the enhanced characteristics provided by an 470 enhanced VPN (such as guaranteed or predicted performance) is by 471 integrating the overlay VPN with a particular set of network 472 resources in the underlay network. This needs be done in a flexible 473 and scalable way so that it can be widely deployed in operator 474 networks to support a reasonable number of enhanced VPN customers. 476 Taking mobile networks and in particular 5G into consideration, the 477 integration of network and the service functions is a likely 478 requirement. The work in IETF SFC working group [SFC] provides a 479 foundation for this integration. 481 2.3.1. Abstraction 483 Integration of the overlay VPN and the underlay network resources 484 does not need to be a tight mapping. As described in [RFC7926], 485 abstraction is the process of applying policy to a set of information 486 about a TE network to produce selective information that represents 487 the potential ability to connect across the network. The process of 488 abstraction presents the connectivity graph in a way that is 489 independent of the underlying network technologies, capabilities, and 490 topology so that the graph can be used to plan and deliver network 491 services in a uniform way. 493 Virtual networks can be built on top of an abstracted topology that 494 represents the connectivity capabilities of the underlay network as 495 described in the framework for Abstraction and Control of TE Networks 496 (ACTN) described in [RFC8453] as discussed further in Section 4.5. 498 2.4. Dynamic Management 500 Enhanced VPNs need to be created, modified, and removed from the 501 network according to service demand. An enhanced VPN that requires 502 hard isolation must not be disrupted by the instantiation or 503 modification of another enhanced VPN. Determining whether 504 modification of an enhanced VPN can be disruptive to that VPN, and in 505 particular whether the traffic in flight will be disrupted can be a 506 difficult problem. 508 The data plane aspects of this problem are discussed further in 509 Section 4. 511 The control plane aspects of this problem are discussed further in 512 Section 4.4. 514 The management plane aspects of this problem are discussed further in 515 Section 4.5 517 Dynamic changes both to the VPN and to the underlay transport network 518 need to be managed to avoid disruption to services that are sensitive 519 to the change of network performance? 521 In addition to non-disruptively managing the network as a result of 522 gross change such as the inclusion of a new VPN endpoint or a change 523 to a link, VPN traffic might need to be moved as a result of traffic 524 volume changes. 526 2.5. Customized Control 528 In some cases it is desirable that an enhanced VPN has a customized 529 control plane, so that the tenant of the enhanced VPN can have some 530 control to the resources and functions allocated to this enhanced 531 VPN. For example, the tenant may be able to specify the service 532 paths in his own enhanced VPN. Depending on the requirement, an 533 enhanced VPN may have its own dedicated controller, or it may be 534 provided with an interface to a control system which is shared with a 535 set of other tenants, or it may be provided with an interface to the 536 control system provided by the network operator. 538 Further detail on this requirement will be provided in a future 539 version of the draft. 541 A description of the control plane aspects of this problem are 542 discussed further in Section 4.4. A description of the management 543 plane aspects of this feature can be found in Section 4.5. 545 2.6. Applicability 547 The technologies described in this document should be applicable to a 548 number types of VPN services such as: 550 o Layer 2 point to point services such as pseudowires [RFC3985] 552 o Layer 2 VPNs [RFC4664] 554 o Ethernet VPNs [RFC7209] 556 o Layer 3 VPNs [RFC4364], [RFC2764] 558 Where such VPN types need enhanced isolation and delivery 559 characteristics, the technology described here can be used to provide 560 an underlay with the required enhanced performance. 562 2.7. Inter-Domain and Inter-Layer Network 564 In some scenarios, an enhanced VPN services may span multiple network 565 domains. A domain is considered to be any collection of network 566 elements within a common realm of address space or path computation 567 responsibility[RFC5151]. And in some domains the operator may own a 568 multi-layered network, for example, a packet network over an optical 569 network. When enhanced VPNs are provisioned in such network 570 scenarios, the technologies used in different network plane (data 571 plane, control plane and management plane) need to provide necessary 572 mechanisms to support multi-domain and multi-layer coordination and 573 integration, so as to provide the required service characteristics 574 for different enhanced VPNs, and improve network efficiency and 575 operational simplicity. 577 3. Architecture of Enhanced VPN 579 A number of enhanced VPN services will typically be provided by a 580 common network infrastructure. Each enhanced VPN consists of both 581 the overlay and a specific set of dedicated network resources and 582 functions allocated in the underlay to satisfy the needs of the VPN 583 tenant. The integration between overlay and various underlay 584 resources ensures the isolation between different enhanced VPNs, and 585 achieves the guaranteed performance for different services. 587 An enhanced VPN needs to be designed with consideration given to: 589 o A enhanced data plane 591 o A control plane to create enhanced VPN, making use of the data 592 plane isolation and guarantee techniques 594 o A management plane for enhanced VPN service life-cycle management 596 These required characteristics are expanded below: 598 o Enhanced data plane 600 * Provides the required resource isolation capability, e.g. 601 bandwidth guarantee. 603 * Provides the required packet latency and jitter 604 characteristics. 606 * Provides the required packet loss characteristics. 608 * Provides the mechanism to identify network slice and the 609 associated resources. 611 o Control plane 613 * Collect the underlying network topology and resources available 614 and export this to other nodes and/or the centralized 615 controller as required. 617 * Create the required virtual networks with the resource and 618 properties needed by the enhanced VPN services that are 619 assigned to it. 621 * Determine the risk of SLA violation and take appropriate 622 avoiding action. 624 * Determine the right balance of per-packet and per-node state 625 according to the needs of enhanced VPN service to scale to the 626 required size. 628 o Management plane 630 * Provides an interface between the enhanced VPN provider (e.g. 631 the Transport Network (TN) Manager) and the enhanced VPN 632 clients (e.g. the 3GPP Management System) such that some of the 633 operation requests can be met without interfering with the 634 enhanced VPN of other clients. 636 * Provides an interface between the enhanced VPN provider and the 637 enhanced VPN clients to expose transport network capability 638 information toward the enhanced VPN client. 640 * Provides the service life-cycle management and operation of 641 enhanced VPN (e.g. creation, modification, assurance/monitoring 642 and decommissioning). 644 OAM 646 * Provides the OAM tools to verify the connectivity and 647 performance of the enhanced VPN. 649 * Provide the OAM tools to verify whether the underlay network 650 resources are correctly allocated and operated properly. 652 o Telemetry 654 * Provides the mechanism to collect the data plane, control plane 655 and management plane data of the network, more specifically: 657 * 659 + Provides the mechanism to collect network data of the 660 underlay network for overall performance evaluation and the 661 enhanced VPN service planning. 663 + Provides the mechanism to collect network data of each 664 enhanced VPN for the monitoring and analytics of the 665 characteristics and SLA fulfilment of enhanced VPN services. 667 3.1. Layered Architecture 669 The layered architecture of enhanced VPN is shown in Figure 2. 671 +-------------------+ } 672 | Network Controller| } Centralized 673 +-------------------+ } Control 674 . . . . . 675 . . . . . 676 . N----N----N . } 677 . / / . } 678 N-----N-----N----N-----N } 679 N----N } 680 / / \ } Virtual 681 N-----N----N----N-----N } Networks 682 N----N } 683 / / } 684 N-----N-----N----N-----N } 686 +----+ ===== +----+ ===== +----+ ===== +----+ } 687 +----+ ===== +----+ ===== +----+ ===== +----+ } Physical 688 +----+ ===== +----+ ===== +----+ ===== +----+ } Network 689 +----+ +----+ +----+ +----+ } 690 N L N L N L N 692 N = Partitioned node 693 L = Partitioned link 695 +----+ = Partition within a node 696 +----+ 698 ====== = Partition within a link 700 Figure 2: The Layered Architecture 702 Underpinning everything is the physical network infrastructure layer 703 consisting of partitioned links and nodes which provide the 704 underlying resources used to provision the separated virtual 705 networks. Various components and techniques as discussed in 706 Section 4 can be used to provide the resource partition, such as 707 FlexE, Time Sensitive Networking, Deterministic Networking, etc. 708 These partitions may be physical, or virtual so long as the SLA 709 required by the higher layers is met. 711 These techniques can be used to provision the virtual networks with 712 the dedicated resources that they need. To get the required 713 functionality there needs to be integration between these overlays 714 and the underlay providing the physical resources. 716 The centralized controller is used to create the virtual networks, to 717 allocate the resources to each virtual network and to provision the 718 enhanced VPN services within the virtual networks. A distributed 719 control plane may also be used for the distribution of the topology 720 and attribute information of the virtual networks. 722 The creation and allocation process needs to take a holistic view of 723 the needs of all of its tenants, and to partition the resources 724 accordingly. However within a virtual network these resources can, 725 if required, be managed via a dynamic control plane. This provides 726 the required scalability and isolation. 728 3.2. Multi-Point to Multi-Point (MP2MP) 730 At the VPN service level, the connectivity is usually mesh or 731 partial-mesh. To support such kinds of VPN service, the 732 corresponding underlay is also an abstract MP2MP medium. However 733 when service guarantees are provided, the point-to-point path through 734 the underlay of the enhanced VPN needs to be specifically engineered 735 to meet the required performance guarantees. 737 3.3. Application Specific Network Types 739 Although a lot of the traffic that will be carried over the enhanced 740 VPN will likely be IPv4 or IPv6, the design has to be capable of 741 carrying other traffic types, in particular Ethernet traffic. This 742 is easily accomplished through the various pseudowire (PW) techniques 743 [RFC3985]. Where the underlay is MPLS, Ethernet can be carried over 744 the enhanced VPN encapsulated according to the method specified in 745 [RFC4448]. Where the underlay is IP, Layer Two Tunneling Protocol - 746 Version 3 (L2TPv3) [RFC3931] can be used with Ethernet traffic 747 carried according to [RFC4719]. Encapsulations have been defined for 748 most of the common layer-2 types for both PW over MPLS and for 749 L2TPv3. 751 3.4. Scaling Considerations 753 VPNs are instantiated as overlays on top of an operator's network and 754 offered as services to the operator's customers. An important 755 feature of overlays is that they are able to deliver services without 756 placing per-service state in the core of the underlay network. 758 Enhanced VPNs may need to install some additional state within the 759 network to achieve the additional features that they require. 760 Solutions must consider minimising and controlling the scale of such 761 state, and deployment architectures should constrain the number of 762 enhanced VPNs that would exist where such services would place 763 additional state in the network. It is expected that the number of 764 enhanced VPN would be a small number in the beginning, and even in 765 future the number of enhanced VPN will be much less than traditional 766 VPNs, because pre-existing VPN techniques would be good enough to 767 meet the needs of most existing VPN-type services. 769 In general, it is not required that the state in the network be 770 maintained in a 1:1 relationship with the VPN+ instances. It will 771 usually be possible to aggregate a set of VPN+ services so that they 772 share the same virtual network and the same set of network resources 773 (much in the way that current VPNs are aggregated over transport 774 tunnels) so that collections of enhanced VPNs that require the same 775 behaviour from the network in terms of resource reservation, latency 776 bounds, resiliency, etc. are able to be grouped together. This is an 777 important feature to assist with the scaling characteristics of VPN+ 778 deployments. 780 See Section 5 for a greater discussion of scalability considerations. 782 4. Candidate Technologies 784 A VPN is a network created by applying a multiplexing technique to 785 the underlying network (the underlay) in order to distinguish the 786 traffic of one VPN from that of another. A VPN path that travels by 787 other than the shortest path through the underlay normally requires 788 state in the underlay to specify that path. State is normally 789 applied to the underlay through the use of the RSVP Signaling 790 protocol, or directly through the use of an SDN controller, although 791 other techniques may emerge as this problem is studied. This state 792 gets harder to manage as the number of VPN paths increases. 793 Furthermore, as we increase the coupling between the underlay and the 794 overlay to support the enhanced VPN service, this state will increase 795 further. 797 In an enhanced VPN different subsets of the underlay resources can be 798 dedicated to different enhanced VPNs or different groups of enhanced 799 VPNs. An enhanced VPN solution thus needs tighter coupling with 800 underlay than is the case with existing VPNs. We cannot, for 801 example, share the network resource between enhanced VPNs which 802 require hard isolation. 804 4.1. Layer-Two Data Plane 806 A number of candidate Layer-2 packet or frame-based data plane 807 solutions which can be used provide the required isolation and 808 guarantee are described in following sections. 810 o FlexE 812 o Time Sensitive Networking 814 o Dedicated Queues 816 4.1.1. FlexE 818 FlexE [FLEXE] is a method of creating a point-to-point Ethernet with 819 a specific fixed bandwidth. FlexE provides the ability to multiplex 820 multiple channels over an Ethernet link in a way that provides hard 821 isolation. FlexE also supports the bonding of multiple links, which 822 can be used to create larger links out of multiple low capacity links 823 in a more efficient way that traditional link aggregation. FlexE 824 also supports the sub-rating of links, which allows an operator to 825 only use a portion of a link. However it is a only a link level 826 technology. When packets are received by the downstream node, they 827 need to be processed in a way that preserves that isolation in the 828 downstream node. This in turn requires a queuing and forwarding 829 implementation that preserves the end-to-end isolation. 831 If different FlexE channels are used for different services, then no 832 sharing is possible between the FlexE channels. This in turn means 833 that it may be difficult to dynamically redistribute unused bandwidth 834 to lower priority services. This may increase the cost of providing 835 services on the network. On the other hand, FlexE can be used to 836 provide hard isolation between different tenants on a shared 837 interface. The tenant can then use other methods to manage the 838 relative priority of their own traffic in each FlexE channel. 840 Methods of dynamically re-sizing FlexE channels and the implication 841 for enhanced VPN are for further study. 843 4.1.2. Dedicated Queues 845 In order to provide multiple isolated virtual networks for enhanced 846 VPN, the conventional DiffServ based queuing system [RFC2475] 847 [RFC4594] is considered insufficient, as DiffServ does not always 848 provide enough queues to differentiate between traffic of different 849 enhanced VPNs, or the range of service classes that each need to 850 provide to their tenants. This problem is particularly acute with an 851 MPLS underlay, because MPLS only provides 8 Traffic Classes (TC), and 852 it's highly likely that there will be more than eight enhanced VPN 853 instances supported by a network. In addition, DiffServ, as 854 currently implemented, mainly provides relative priority-based 855 scheduling, and is difficult to achieve quantitive resource 856 reservation. In order to address this problem and reduce the 857 interference between enhanced VPNs, it is necessary to steer traffic 858 of enhanced VPNs to dedicated input and output queues. Some routers 859 have large amount of queues and sophisticated queuing systems, which 860 could be used or enhanced to provide the granularity and level of 861 isolation required by the applications of enhanced VPN. For example, 862 on one physical interface, the queuing system can provide a set of 863 virtual sub-interfaces, each allocated with dedicated queueing and 864 buffer resources. Sophisticated queuing systems of this type may be 865 used to provide end-to-end virtual isolation between traffic of 866 different enhanced VPNs. 868 4.1.3. Time Sensitive Networking 870 Time Sensitive Networking (TSN) [TSN] is an IEEE project that is 871 designing a method of carrying time sensitive information over 872 Ethernet. It introduces the concept of packet scheduling where a 873 high priority packet stream may be given a scheduled time slot 874 thereby guaranteeing that it experiences no queuing delay and hence a 875 reduced latency. However, when no scheduled packet arrives, its 876 reserved time-slot is handed over to best effort traffic, thereby 877 improving the economics of the network. The mechanisms defined in 878 TSN can be used to meet the requirements of time sensitive services 879 of an enhanced VPN. 881 Ethernet can be emulated over a Layer 3 network using a pseudowire. 882 However the TSN payload would be opaque to the underlay and thus not 883 treated specifically as time sensitive data. The preferred method of 884 carrying TSN over a layer 3 network is through the use of 885 deterministic networking as explained in the following section of 886 this document. 888 4.2. Layer-Three Data Plane 890 We now consider the problem of slice differentiation and resource 891 representation in the network layer. The candidate technologies are: 893 o Deterministic Networking 895 o MPLS-TE 897 o Segment Routing 899 4.2.1. Deterministic Networking 901 Deterministic Networking (DetNet) [I-D.ietf-detnet-architecture] is a 902 technique being developed in the IETF to enhance the ability of 903 layer-3 networks to deliver packets more reliably and with greater 904 control over the delay. The design cannot use re-transmission 905 techniques such as TCP since that can exceed the delay tolerated by 906 the applications. Even the delay improvements that are achieved with 907 Stream Control Transmission Protocol Partial Reliability Extenstion 908 (SCTP-PR) [RFC3758] do not meet the bounds set by application 909 demands. DetNet pre-emptively sends copies of the packet over 910 various paths to minimize the chance of all copies of a packet being 911 lost, and trims duplicate packets to prevent excessive flooding of 912 the network and to prevent multiple packets being delivered to the 913 destination. It also seeks to set an upper bound on latency. The 914 goal is not to minimize latency; the optimum upper bound paths may 915 not be the minimum latency paths. 917 DetNet is based on flows. It currently does not specify the use of 918 underlay topology other than the base topology. To be of use for 919 enhanced VPN, DetNet needs to be integrated with different virtual 920 topologies of enhanced VPNs. 922 The detailed design that allows the use DetNet in a multi-tenant 923 network, and how to improve the scalability of DetNet in a multi- 924 tenant network are topics for further study. 926 4.2.2. MPLS Traffic Engineering (MPLS-TE) 928 MPLS-TE introduces the concept of reserving end-to-end bandwidth for 929 a TE-LSP, which can be used as the underlay of VPNs. It also 930 introduces the concept of non-shortest path routing through the use 931 of the Explicit Route Object [RFC3209]. VPN traffic can be run over 932 dedicated TE-LSPs to provide reserved bandwidth for each specific 933 connection in a VPN. Some network operators have concerns about the 934 scalability and management overhead of RSVP-TE system, and this has 935 lead them to consider other solutions for their networks. 937 4.2.3. Segment Routing 939 Segment Routing [RFC8402] is a method that prepends instructions to 940 packets at the head-end node and optionally at various points as it 941 passes though the network. These instructions allow the packets to 942 be routed on paths other than the shortest path for various traffic 943 engineering reasons. With SR, a path needs to be dynamically created 944 through a set of segments by simply specifying the Segment 945 Identifiers (SIDs), i.e. instructions rooted at a particular point in 946 the network. By encoding the state in the packet, per-path state is 947 transitioned out of the network. 949 With current segment routing, the instructions are used to specify 950 the nodes and links to be traversed. An SR traffic engineered path 951 operates with a granularity of a link with hints about priority 952 provided through the use of the traffic class (TC) or Differentiated 953 Services Code Point (DSCP) field in the header. However to achieve 954 the latency and isolation characteristics that are sought by the 955 enhanced VPN users, steering packets through specific queues and 956 resources will likely be required. With SR, it is possible to 957 introduce such fine-grained packet steering by specifying the queues 958 and resources through an SR instruction list. 960 Note that the concept of a queue is a useful abstraction for many 961 types of underlay mechanism that may be used to provide enhanced 962 isolation and latency support. How the queue satisfies the 963 requirement is implementation specific and is transparent to the 964 layer-3 data plane and control plane mechanisms used. 966 Both SR-MPLS and SRv6 are candidate data plane technologies for 967 enhanced VPN. In some cases they can further be used for DetNet to 968 meet the packet loss, delay and jitter requirement of particular 969 service. How to provide the DetNet enhanced delivery in an SRv6 970 environment is specified in [I-D.geng-spring-srv6-for-detnet]. 972 4.3. Non-Packet Data Plane 974 Non-packet underlay data plane technologies often have TE properties 975 and behaviors, and meet many of the key requirements in particular 976 for bandwidth guarantees, traffic isolation (with physical isolation 977 often being an integral part of the technology), highly predictable 978 latency and jitter characteristics, measurable loss characteristics, 979 and ease of identification of flows (and hence slices). 981 The control and management planes for non-packet data plane 982 technologies have most in common with MPLS-TE (Section 4.2.2) and 983 offer the same set of advanced features [RFC3945]. Furthermore, 984 management techniques such as ACTN ([RFC8453] and Section 4.6 can be 985 used to aid in the reporting of underlying network topologies, and 986 the creation of virtual networks with the resource and properties 987 needed by the enhanced VPN services. 989 4.4. Control Plane 991 Enhanced VPN would likely be based on a hybrid control mechanism, 992 which takes advantage of the logically centralized controller for on- 993 demand provisioning and global optimization, whilst still relies on 994 distributed control plane to provide scalability, high reliability, 995 fast reaction, automatic failure recovery etc. Extension and 996 optimization to the distributed control plane is needed to support 997 the enhanced properties of VPN+. 999 RSVP-TE provides the signaling mechanism of establishing a TE-LSP 1000 with end-to-end resource reservation. It can be used to bind the VPN 1001 to specific network resource allocated within the underlay, but there 1002 are the above mentioned scalability concerns. 1004 SR does not have the capability of signaling the resource reservation 1005 along the path, nor do its currently specified distributed link state 1006 routing protocols. On the other hand, the SR approach provides a way 1007 of efficiently binding the network underlay and the enhanced VPN 1008 overlay, as it reduces the amount of state to be maintained in the 1009 network. An SR-based approach with per-slice resource reservation 1010 can easily create dedicated SR network slices, and the VPN services 1011 can be bound to a particular SR network slice. A centralized 1012 controller can perform resource planning and reservation from the 1013 controller's point of view, but this does not ensure resource 1014 reservation is actually done in the network nodes. Thus, if a 1015 distributed control plane is needed, either in place of an SDN 1016 controller or as an assistant to it, the design of the control system 1017 needs to ensure that resources are uniquely allocated in the network 1018 nodes for the correct services, and not allocated to other services 1019 causing unintended resource conflict. 1021 In addition, in multi-domain and multi-layer networks, the 1022 centralized and distributed control mechanisms will be used for 1023 inter-domain coordination and inter-layer optimization, so that the 1024 diverse and customized enhanced VPN service requirement could be met. 1025 The detailed mechanisms will be described in a future version. 1027 4.5. Management Plane 1029 In the context of 5G end-to-end network slicing, the management of 1030 enhanced VPN is considered as the management of transport network 1031 part of the end-to-end network slice. 3GPP management system may 1032 provide the topology and QoS parameters as requirement to the 1033 management plane of transport network. It may also require the 1034 transport network to expose the capability and status of the 1035 transport network slice. Thus an interface between enhanced VPN 1036 management plane and 3GPP network slice management system and 1037 relevant service data models are needed for the coordination of end- 1038 to-end network slice management. 1040 The management plane interface and data models for enhanced VPN can 1041 be based on the service models such as: 1043 o VPN service models defined in [RFC8299] and [RFC8466] 1045 o Possible augmentations and extensions 1046 (e.g.,[I-D.ietf-teas-te-service-mapping-yang]) to VPN service 1047 models 1049 o ACTN related service models such as [I-D.ietf-teas-actn-vn-yang] 1050 and [I-D.ietf-teas-actn-pm-telemetry-autonomics]. 1052 o VPN network model as defined in [I-D.aguado-opsawg-l3sm-l3nm]. 1054 o TE Tunnel model as defined in [I-D.ietf-teas-yang-te]. 1056 These data models can be applicable in the provisioning of enhanced 1057 VPN service. The details are described in Section 4.6. 1059 4.6. Applicability of Service Data Models to Enhanced VPN 1061 ACTN supports operators in viewing and controlling different domains 1062 and presenting virtualized networks to their customers. The ACTN 1063 framework [RFC8453] highlights how: 1065 o Abstraction of the underlying network resources are provided to 1066 higher-layer applications and customers. 1068 o Virtualization of underlying resources, whose selection criterion 1069 is the allocation of those resources for the customer, 1070 application, or service. 1072 o Creation of a virtualized environment allowing operators to view 1073 and control multi-domain networks as a single virtualized network. 1075 o The presentation to customers of networks as a virtual network via 1076 open and programmable interfaces. 1078 The infrastructure managed through the Service Data models comprises 1079 traffic engineered network resources (e.g. bandwidth, time slot, 1080 wavelength) and VPN service related resources (e.g. Route Target 1081 (RT) and Route Distinguisher (RD)). 1083 The type of network virtualization enabled by ACTN managed 1084 infrastructure provides customers and applications (tenants) with the 1085 capability to utilize and independently control allocated virtual 1086 network resources as if they were physically their own resources. 1088 The Customer VPN model (e.g. L3SM) or an ACTN Virtual Network (VN) 1089 model is a customer view of the ACTN managed infrastructure, and is 1090 presented by the ACTN provider as a set of abstracted services or 1091 resources. 1093 L3VPN network model or TE tunnel model is a network view of the ACTN 1094 managed infrastructure, and is presented by the ACTN provider as a 1095 set of transport resources. 1097 Depending on the agreement between customer and provider, various 1098 VPN/VN operations and VPN/VN views are possible. 1100 o Virtual Network Creation: A VPN/VN could be pre-configured and 1101 created via static or dynamic request and negotiation between 1102 customer and provider. It must meet the specified SLA attributes 1103 which satisfy the customer's objectives. 1105 o Virtual Network Operations: The virtual network may be further 1106 modified and deleted based on customer request to request changes 1107 in the network resources reserved for the customer, and used to 1108 construct the network slice. The customer can further act upon 1109 the virtual network to manage traffic flow across the virtual 1110 network. 1112 o Virtual Network View: The VPN/VN topology from a customer point of 1113 view. These may be a variety of tunnels, or an entire VN 1114 topology, or an VPN service topology. Such connections may 1115 comprise of customer end points, access links, intra-domain paths, 1116 and inter-domain links. 1118 Dynamic VPN/VN Operations allow a customer to modify or delete the 1119 VPN/VN. The customer can further act upon the virtual network to 1120 create/modify/delete virtual links and nodes or VPN sites. These 1121 changes will result in subsequent tunnel management or VPN service 1122 management in the operator's networks. 1124 4.6.1. Enhanced VPN Delivery in ACTN Architecture 1126 ACTN provides VPN connections or VN connections between multiple 1127 sites as requested via a VPN requestor enabled by the Customer 1128 Network Controller (CNC). The CNC is managed by the customer 1129 themselves, and interacts with the network provider's Multi-Domain 1130 Service Controller (MDSC). The Provisioning Network Controllers 1131 (PNC) are responible for network resource management, thus the PNCs 1132 are remain entirely under the management of the network provider and 1133 are not visible to the customer. 1135 The benefits of this model include: 1137 o Provision of edge-to-edge VPN multi-access connectivity. 1139 o Management is mostly performed by the network provider, with some 1140 flexibility delegated to the customer-managed CNC. 1142 Figure 3 presents a more general representation of how multiple 1143 enhanced VPNs may be created from the resources of multiple physical 1144 networks using the CNC, MDSC, and PNC components of the ACTN 1145 architecture. Each enhanced VPN is controlled by its own CNC. The 1146 CNCs send requests to the provider's MDSC. The provider manages two 1147 different physical networks each under the control of PNC. The MDSC 1148 asks the PNCs to allocate and provision resources to achieve the 1149 enhanced VPNs. In this figure, one enhanced VPN is constructed 1150 solely from the resources of one of the physical networks, while the 1151 the VPN uses resources from both physical networks. 1153 ___________ 1154 --------------- ( ) 1155 | CNC |---------->( VPN+ ) 1156 --------^------ ( ) 1157 | _(_________ _) 1158 --------------- ( ) ^ 1159 | CNC |----------->( VPN+ ) : 1160 ------^-------- ( ) : 1161 | | (___________) : 1162 | | ^ ^ : 1163 Boundary | | : : : 1164 Between ==========|====|===================:====:====:======== 1165 Customer & | | : : : 1166 Network Provider | | : : : 1167 v v : : : 1168 --------------- : :....: 1169 | MDSC | : : 1170 --------------- : : 1171 ^ ---^------ ... 1172 | ( ) . 1173 v ( Physical ) . 1174 ---------------- ( Network ) . 1175 | PNC |<-------->( ) ---^------ 1176 ---------------- | -------- ( ) 1177 | |-- ( Physical ) 1178 | PNC |<------------------------->( Network ) 1179 --------------- ( ) 1180 -------- 1182 Figure 3: Generic VPN+ Delivery in the ACTN Architecture 1184 4.6.2. Enhanced VPN Features with Service Data Models 1186 This section discusses how the service data models can fulfill the 1187 enhanced VPN requirements described earlier in this document. As 1188 previously noted, key requirements of the enhanced VPN include: 1190 1. Isolation between VPNs/VNs 1192 2. Guaranteed Performance 1193 3. Integration 1195 4. Dynamic Management 1197 5. Customized Control 1199 The subsections that follow outline how each requirement is met using 1200 ACTN. 1202 4.6.2.1. Isolation Between VPN/VNs 1204 The VN YANG model [I-D.ietf-teas-actn-vn-yang] and the TE-service 1205 mapping model [I-D.ietf-teas-te-service-mapping-yang] fulfill the 1206 VPN/VN isolation requirement by providing the following features for 1207 the VPN/VNs: 1209 o Each VN is identified with a unique identifier (vn-id and vn-name) 1210 and so is each VN member that belongs to the VN (vn-member-id). 1212 o Each VPN is identified with a unique identifier (vpn-id) and can 1213 be mapped to one specific VN. While multiple VPNs may mapped to 1214 the same VN according to service requirement and operator's 1215 policy. 1217 o Each VPN and the corresponding VN is managed and controlled 1218 independent of other VPNs/VNs in the network with proper 1219 availability level. 1221 o Each VPN/VN is instantiated with an isolation requirement 1222 described by the TE-service mapping model 1223 [I-D.ietf-teas-te-service-mapping-yang]. This mapping supports: 1225 * Hard isolation with deterministic characteristics (e.g., this 1226 case may need an optical bypass tunnel or a DetNet/TSN tunnel 1227 to guarantee latency with no jitter) 1229 * Hard isolation (i.e., dedicated TE resources in all underlays) 1231 * Soft isolation (i.e., resource in some layer may be shared 1232 while in some other layers is dedicated). 1234 * No isolation (i.e., sharing with other VPN/VN). 1236 4.6.2.2. Guaranteed Performance 1238 Performance objectives of a VN need first to be expressed in order to 1239 assure the performance guarantee. 1241 Performance objectives of a VPN [RFC8299][RFC8466] are expressed with 1242 QoS profile, either standard profile or customer profile. The 1243 customer QoS profile include the following properties: 1245 o Rate-limit 1247 o Bandwidth 1249 o Latency 1251 o Jitter 1253 [I-D.ietf-teas-actn-vn-yang] and [I-D.ietf-teas-yang-te-topo] allow 1254 configuration of several TE parameters that may affect the VN 1255 performance objectives as follows: 1257 o Bandwidth 1259 o Objective function (e.g., min cost path, min load path, etc.) 1261 o Metric Types and their threshold: 1263 * TE cost, IGP cost, Hop count, or Unidirectional Delay (e.g., 1264 can set all path delay <= threshold) 1266 Once these requests are instantiated, the resources are committed and 1267 guaranteed through the life cycle of the VPN/VN. 1269 4.6.2.3. Integration 1271 L3VPN network model provides mechanism to correlate customer's VPN 1272 and the VPN service related resources (e.g.RT and RD) allocated in 1273 the provider's network. 1275 VPN/Network performance monitoring model 1276 [I-D.www-bess-yang-vpn-service-pm] provides mechanisms to monitor and 1277 manage network Performance on the topology at different layer or the 1278 overlay topology between VPN sites. 1280 VN model and Performance Monitoring Telemetry model provides 1281 mechanisms to correlate customer's VN and the actual TE tunnels 1282 instantiated in the provider's network. Specifically: 1284 o Link each VN member to actual TE tunnel. 1286 o Each VN can be monitored on a various level such as VN level, VN 1287 member level, TE-tunnel level, and link/node level. 1289 Service function integration with network topology (L3 and TE 1290 topology) is in progress in [I-D.ietf-teas-sf-aware-topo-model]. 1291 Specifically, [I-D.ietf-teas-sf-aware-topo-model] addresses a number 1292 of use-cases that show how TE topology supports various service 1293 functions. 1295 4.6.2.4. Dynamic Management 1297 ACTN provides an architecture that allows the CNC to interact with 1298 the MDSC which is network provider's SDN controller. This gives the 1299 customer control of their VPN or VNs. 1301 e.g., the ACTN VN model [I-D.ietf-teas-actn-vn-yang] allows the VN to 1302 life-cycle management such as create, modify, and delete VNs on 1303 demand. Another example is L3VPN servicel model [RFC8299] which 1304 allows the VPN lifecycle management such as VPN creation, 1305 modification and deletion on demand. 1307 4.6.2.5. Customized Control 1309 ACTN provides a YANG model that allows the CNC to control a VN as a 1310 "Type 2 VN" that allows the customer to provision tunnels that 1311 connect their endpoints over the customized VN topology. 1313 For some VN members, the customers are allowed to configure the path 1314 (i.e., the sequence of virtual nodes and virtual links) over the VN/ 1315 abstract topology. 1317 4.6.3. 5G Transport Service Delivery via Coordinated Data Modules 1319 The overview of network slice structure as defined in the 3GPP 5GS is 1320 shown in Figure 5. The terms are described in specific 3GPP 1321 documents (e.g. [TS23501] and [TS28530].) 1322 <================== E2E-NSI =======================> 1323 : : : : : 1324 : : : : : 1325 <====== RAN-NSSI ======><=TRN-NSSI=><====== CN-NSSI ======>VL[APL] 1326 : : : : : : : : : 1327 : : : : : : : : : 1328 RW[NFs ]<=TRN-NSSI=>[NFs ]<=TRN-NSSI=>[NFs ]<=TRN-NSSI=>[NFs ]VL[APL] 1330 . . . . . . . . . . . . .. . . . . . . . . . . . . .. 1331 .,----. ,----. ,----.. ,----. .,----. ,----. ,----.. 1332 UE--|RAN |---| TN |---|RAN |---| TN |---|CN |---| TN |---|CN |--[APL] 1333 .|NFs | `----' |NFs |. `----' .|NFs | `----' |NFs |. 1334 .`----' `----'. .`----' `----'. 1335 . . . . . . . . . . . . .. . . . . . . . . . . . . .. 1337 RW RAN MBH CN DN 1339 *Legends 1340 UE: User Equipment 1341 RAN: Radio Access Network 1342 CN: Core Network 1343 DN: Data Network 1344 TN: Transport Network 1345 MBH: Mobile Backhaul 1346 RW: Radio Wave 1347 NF: Network Function 1348 APL: Application Server 1349 NSI: Network Slice Instance 1350 NSSI: Network Slice Subnet Instance 1352 Figure 4: Overview of Structure of Network Slice in 3GPP 5GS 1354 To support 5G service (e.g., 5G MBB service), L3VPN service model 1355 [RFC8299] and TEAS VN model [I-D.ietf-teas-actn-vn-yang] can be both 1356 provided to describe 5G MBB Transport Service or connectivity 1357 service. L3VPN service model is used to describe end-to-end IP 1358 connectivity service while TEAS VN model is used to describe TE 1359 connectivity service between VPN sites or between RAN NFs and Core 1360 network NFs. 1362 VN in TEAS VN model and support point-to-point or multipoint-to- 1363 multipoint connectivity service and can be seen as one example of 1364 network slice. 1366 TE Service mapping model can be used to map L3VPN service requests 1367 onto underlying network resource and TE models to get TE network 1368 setup. 1370 For IP VPN service provision, the service parameters in the L3VPN 1371 service model [RFC8299] can be decomposed into a set of configuration 1372 parameters described in the L3VPN network model 1373 [I-D.aguado-opsawg-l3sm-l3nm] which will get VPN network setup. 1375 5. Scalability Considerations 1377 Enhanced VPN provides the performance guaranteed services in packet 1378 networks, but with the potential cost of introducing additional 1379 states into the network. There are at least three ways that this 1380 adding state might be presented in the network: 1382 o Introduce the complete state into the packet, as is done in SR. 1383 This allows the controller to specify the detailed series of 1384 forwarding and processing instructions for the packet as it 1385 transits the network. The cost of this is an increase in the 1386 packet header size. The cost is also that systems will have 1387 capabilities enabled in case they are called upon by a service. 1388 This is a type of latent state, and increases as we more precisely 1389 specify the path and resources that need to be exclusively 1390 available to a VPN. 1392 o Introduce the state to the network. This is normally done by 1393 creating a path using RSVP-TE, which can be extended to introduce 1394 any element that needs to be specified along the path, for example 1395 explicitly specifying queuing policy. It is of course possible to 1396 use other methods to introduce path state, such as via a Software 1397 Defined Network (SDN) controller, or possibly by modifying a 1398 routing protocol. With this approach there is state per path per 1399 path characteristic that needs to be maintained over its life- 1400 cycle. This is more state than is needed using SR, but the packet 1401 are shorter. 1403 o Provide a hybrid approach based on using binding SIDs to create 1404 path fragments, and bind them together with SR. 1406 Dynamic creation of a VPN path using SR requires less state 1407 maintenance in the network core at the expense of larger VPN headers 1408 on the packet. The packet size can be lower if a form of loose 1409 source routing is used (using a few nodal SIDs), and it will be lower 1410 if no specific functions or resource on the routers are specified. 1411 Reducing the state in the network is important to enhanced VPN, as it 1412 requires the overlay to be more closely integrated with the underlay 1413 than with traditional VPNs. This tighter coupling would normally 1414 mean that more state needed to be created and maintained in the 1415 network, as the state about fine granularity processing would need to 1416 be loaded and maintained in the routers. However, a segment routed 1417 approach allows much of this state to be spread amongst the network 1418 ingress nodes, and transiently carried in the packets as SIDs. 1420 These approaches are for further study. 1422 5.1. Maximum Stack Depth of SR 1424 One of the challenges with SR is the stack depth that nodes are able 1425 to impose on packets [RFC8491]. This leads to a difficult balance 1426 between adding state to the network and minimizing stack depth, or 1427 minimizing state and increasing the stack depth. 1429 5.2. RSVP Scalability 1431 The traditional method of creating a resource allocated path through 1432 an MPLS network is to use the RSVP protocol. However there have been 1433 concerns that this requires significant continuous state maintenance 1434 in the network. There are ongoing works to improve the scalability 1435 of RSVP-TE LSPs in the control plane [RFC8370]. 1437 There is also concern at the scalability of the forwarder footprint 1438 of RSVP as the number of paths through an LSR grows [RFC8577] 1439 proposes to address this by employing SR within a tunnel established 1440 by RSVP-TE. 1442 5.3. SDN Scaling 1444 The centralized approach of SDN requires state to be stored in the 1445 network, but does not have the overhead of also requiring control 1446 plane state to be maintained. Each individual network node may need 1447 to maintain a communication channel with the SDN controller, but that 1448 compares favourably with the need for a control plane to maintain 1449 communication with all neighbors. 1451 However, SDN may transfer some of the scalability concerns from the 1452 network to the centralized controller. In particular, there may be a 1453 heavy processing burden at the controller, and a heavy load in the 1454 network surrounding the controller. 1456 6. OAM Considerations 1458 The enhanced VPN OAM design needs to consider the following 1459 requirements: 1461 o Instrumentation of the underlay so that the network operator can 1462 be sure that the resources committed to a tenant are operating 1463 correctly and delivering the required performance. 1465 o Instrumentation of the overlay by the tenant. This is likely to 1466 be transparent to the network operator and to use existing 1467 methods. Particular consideration needs to be given to the need 1468 to verify the isolation and the various committed performance 1469 characteristics. 1471 o Instrumentation of the overlay by the network provider to 1472 proactively demonstrate that the committed performance is being 1473 delivered. This needs to be done in a non-intrusive manner, 1474 particularly when the tenant is deploying a performance sensitive 1475 application 1477 o Verification of the conformity of the path to the service 1478 requirement. This may need to be done as part of a commissioning 1479 test. 1481 A study of OAM in SR networks has been documented in [RFC8403]. 1483 7. Telemetry Considerations 1485 Network visibility is essential for network operation. Network 1486 telemetry has been considered as an ideal means to gain sufficient 1487 network visibility with better flexibility, scalability, accuracy, 1488 coverage, and performance than conventional OAM technologies. 1490 As defined in [I-D.ietf-opsawg-ntf], Network Telemetry is to acquire 1491 network data remotely for network monitoring and operation. It is a 1492 general term for a large set of network visibility techniques and 1493 protocols. Network telemetry addresses the current network operation 1494 issues and enables smooth evolution toward intent-driven autonomous 1495 networks. Telemetry can be applied on the forwarding plane, the 1496 control plane, and the management plane in a network. 1498 How the telemetry mechanisms could be used or extended for the 1499 enhanced VPN service will be described in a future version. 1501 8. Enhanced Resiliency 1503 Each enhanced VPN has a life-cycle, and needs modification during 1504 deployment as the needs of its tenant change. Additionally, as the 1505 network as a whole evolves, there will need to be garbage collection 1506 performed to consolidate resources into usable quanta. 1508 Systems in which the path is imposed such as SR, or some form of 1509 explicit routing tend to do well in these applications, because it is 1510 possible to perform an atomic transition from one path to another. 1511 This is a single action by the head-end changes the path without the 1512 need for coordinated action by the routers along the path. However, 1513 implementations and the monitoring protocols need to make sure that 1514 the new path is up and meet the required SLA before traffic is 1515 transitioned to it. It is possible for deadlocks arise as a result 1516 of the network becoming fragmented over time, such that it is 1517 impossible to create a new path or modify a existing path without 1518 impacting the SLA of other paths. Resolution of this situation is as 1519 much a commercial issue as it is a technical issue and is outside the 1520 scope of this document. 1522 There are however two manifestations of the latency problem that are 1523 for further study in any of these approaches: 1525 o The problem of packets overtaking one and other if a path latency 1526 reduces during a transition. 1528 o The problem of the latency transient in either direction as a path 1529 migrates. 1531 There is also the matter of what happens during failure in the 1532 underlay infrastructure. Fast reroute is one approach, but that 1533 still produces a transient loss with a normal goal of rectifying this 1534 within 50ms [RFC5654] . An alternative is some form of N+1 delivery 1535 such as has been used for many years to support protection from 1536 service disruption. This may be taken to a different level using the 1537 techniques proposed by the IETF deterministic network work with 1538 multiple in-network replication and the culling of later packets 1539 [I-D.ietf-detnet-architecture]. 1541 In addition to the approach used to protect high priority packets, 1542 consideration has to be given to the impact of best effort traffic on 1543 the high priority packets during a transient. Specifically if a 1544 conventional re-convergence process is used there will inevitably be 1545 micro-loops and whilst some form of explicit routing will protect the 1546 high priority traffic, lower priority traffic on best effort shortest 1547 paths will micro-loop without the use of a loop prevention 1548 technology. To provide the highest quality of service to high 1549 priority traffic, either this traffic must be shielded from the 1550 micro-loops, or micro-loops must be prevented. 1552 9. Operational Considerations 1554 TBD in a future version. 1556 10. Security Considerations 1558 All types of virtual network require special consideration to be 1559 given to the isolation between the tenants. In this regard enhanced 1560 VPNs neither introduce, no experience a greater security risk than 1561 another VPN of the same base type. However, in an enhanced virtual 1562 network service the isolation requirement needs to be considered. If 1563 a service requires a specific latency then it can be damaged by 1564 simply delaying the packet through the activities of another tenant. 1565 In a network with virtual functions, depriving a function used by 1566 another tenant of compute resources can be just as damaging as 1567 delaying transmission of a packet in the network. The measures to 1568 address these dynamic security risks must be specified as part to the 1569 specific solution. 1571 While an enhanced VPN service may be sold as offering encryption and 1572 other security features as part of the service, customers would be 1573 well advised to take responsibility for their own security 1574 requirements themselves possibly by encrypting traffic before handing 1575 it off to the service provider. 1577 The privacy of enhanced VPN service customers must be preserved. It 1578 should not be possible for one customer to discover the existence of 1579 another customer, nor should the sites that are members of an 1580 enhanced VPN be externally visible. 1582 11. IANA Considerations 1584 There are no requested IANA actions. 1586 12. Contributors 1587 Daniel King 1588 Email: daniel@olddog.co.uk 1590 Adrian Farrel 1591 Email: adrian@olddog.co.uk 1593 Jeff Tansura 1594 Email: jefftant.ietf@gmail.com 1596 Qin Wu 1597 Email: bill.wu@huawei.com 1599 Daniele Ceccarelli 1600 Email: daniele.ceccarelli@ericsson.com 1602 Mohamed Boucadair 1603 Email: mohamed.boucadair@orange.com 1605 Sergio Belotti 1606 Email: sergio.belotti@nokia.com 1608 Haomian Zheng 1609 Email: zhenghaomian@huawei.com 1611 13. Acknowledgements 1613 The authors would like to thank Charlie Perkins, James N Guichard and 1614 John E Drake for their review and valuable comments. 1616 This work was supported in part by the European Commission funded 1617 H2020-ICT-2016-2 METRO-HAUL project (G.A. 761727). 1619 14. References 1621 14.1. Normative References 1623 [I-D.ietf-teas-actn-vn-yang] 1624 Lee, Y., Dhody, D., Ceccarelli, D., Bryskin, I., and B. 1625 Yoon, "A Yang Data Model for VN Operation", draft-ietf- 1626 teas-actn-vn-yang-06 (work in progress), July 2019. 1628 [I-D.ietf-teas-te-service-mapping-yang] 1629 Lee, Y., Dhody, D., Fioccola, G., Wu, Q., Ceccarelli, D., 1630 and J. Tantsura, "Traffic Engineering (TE) and Service 1631 Mapping Yang Model", draft-ietf-teas-te-service-mapping- 1632 yang-02 (work in progress), September 2019. 1634 [RFC2764] Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. 1635 Malis, "A Framework for IP Based Virtual Private 1636 Networks", RFC 2764, DOI 10.17487/RFC2764, February 2000, 1637 . 1639 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1640 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1641 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1642 . 1644 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1645 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1646 DOI 10.17487/RFC3985, March 2005, 1647 . 1649 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 1650 2 Virtual Private Networks (L2VPNs)", RFC 4664, 1651 DOI 10.17487/RFC4664, September 2006, 1652 . 1654 [RFC8299] Wu, Q., Ed., Litkowski, S., Tomotaki, L., and K. Ogaki, 1655 "YANG Data Model for L3VPN Service Delivery", RFC 8299, 1656 DOI 10.17487/RFC8299, January 2018, 1657 . 1659 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1660 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1661 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1662 July 2018, . 1664 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1665 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1666 DOI 10.17487/RFC8453, August 2018, 1667 . 1669 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1670 Data Model for Layer 2 Virtual Private Network (L2VPN) 1671 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1672 2018, . 1674 14.2. Informative References 1676 [BBF-SD406] 1677 "BBF SD-406: End-to-End Network Slicing", 2016, 1678 . 1681 [DETNET] "Deterministic Networking", March , 1682 . 1684 [FLEXE] "Flex Ethernet Implementation Agreement", March 2016, 1685 . 1688 [I-D.aguado-opsawg-l3sm-l3nm] 1689 Aguado, A., Dios, O., Lopezalvarez, V., 1690 daniel.voyer@bell.ca, d., and L. Munoz, "Layer 3 VPN 1691 Network Model", draft-aguado-opsawg-l3sm-l3nm-01 (work in 1692 progress), July 2019. 1694 [I-D.geng-spring-srv6-for-detnet] 1695 Geng, X., Li, Z., and M. Chen, "SRv6 for Deterministic 1696 Networking (DetNet)", draft-geng-spring-srv6-for-detnet-00 1697 (work in progress), July 2019. 1699 [I-D.ietf-detnet-architecture] 1700 Finn, N., Thubert, P., Varga, B., and J. Farkas, 1701 "Deterministic Networking Architecture", draft-ietf- 1702 detnet-architecture-13 (work in progress), May 2019. 1704 [I-D.ietf-detnet-dp-sol-ip] 1705 Korhonen, J. and B. Varga, "DetNet IP Data Plane 1706 Encapsulation", draft-ietf-detnet-dp-sol-ip-02 (work in 1707 progress), March 2019. 1709 [I-D.ietf-opsawg-ntf] 1710 Song, H., Qin, F., Martinez-Julia, P., Ciavaglia, L., and 1711 A. Wang, "Network Telemetry Framework", draft-ietf-opsawg- 1712 ntf-01 (work in progress), June 2019. 1714 [I-D.ietf-teas-actn-pm-telemetry-autonomics] 1715 Lee, Y., Dhody, D., Karunanithi, S., Vilata, R., King, D., 1716 and D. Ceccarelli, "YANG models for VN & TE Performance 1717 Monitoring Telemetry and Scaling Intent Autonomics", 1718 draft-ietf-teas-actn-pm-telemetry-autonomics-00 (work in 1719 progress), July 2019. 1721 [I-D.ietf-teas-sf-aware-topo-model] 1722 Bryskin, I., Liu, X., Lee, Y., Guichard, J., Contreras, 1723 L., Ceccarelli, D., and J. Tantsura, "SF Aware TE Topology 1724 YANG Model", draft-ietf-teas-sf-aware-topo-model-03 (work 1725 in progress), March 2019. 1727 [I-D.ietf-teas-yang-te] 1728 Saad, T., Gandhi, R., Liu, X., Beeram, V., and I. Bryskin, 1729 "A YANG Data Model for Traffic Engineering Tunnels and 1730 Interfaces", draft-ietf-teas-yang-te-21 (work in 1731 progress), April 2019. 1733 [I-D.ietf-teas-yang-te-topo] 1734 Liu, X., Bryskin, I., Beeram, V., Saad, T., Shah, H., and 1735 O. Dios, "YANG Data Model for Traffic Engineering (TE) 1736 Topologies", draft-ietf-teas-yang-te-topo-22 (work in 1737 progress), June 2019. 1739 [I-D.www-bess-yang-vpn-service-pm] 1740 Wang, Z., Wu, Q., Even, R., Wen, B., and C. Liu, "A YANG 1741 Model for Network and VPN Service Performance Monitoring", 1742 draft-www-bess-yang-vpn-service-pm-03 (work in progress), 1743 July 2019. 1745 [NGMN-NS-Concept] 1746 "NGMN NS Concept", 2016, . 1750 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1751 and W. Weiss, "An Architecture for Differentiated 1752 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1753 . 1755 [RFC2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path 1756 Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000, 1757 . 1759 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1760 Conrad, "Stream Control Transmission Protocol (SCTP) 1761 Partial Reliability Extension", RFC 3758, 1762 DOI 10.17487/RFC3758, May 2004, 1763 . 1765 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1766 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 1767 RFC 3931, DOI 10.17487/RFC3931, March 2005, 1768 . 1770 [RFC3945] Mannie, E., Ed., "Generalized Multi-Protocol Label 1771 Switching (GMPLS) Architecture", RFC 3945, 1772 DOI 10.17487/RFC3945, October 2004, 1773 . 1775 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1776 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1777 2006, . 1779 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 1780 "Encapsulation Methods for Transport of Ethernet over MPLS 1781 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 1782 . 1784 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 1785 Guidelines for DiffServ Service Classes", RFC 4594, 1786 DOI 10.17487/RFC4594, August 2006, 1787 . 1789 [RFC4719] Aggarwal, R., Ed., Townsley, M., Ed., and M. Dos Santos, 1790 Ed., "Transport of Ethernet Frames over Layer 2 Tunneling 1791 Protocol Version 3 (L2TPv3)", RFC 4719, 1792 DOI 10.17487/RFC4719, November 2006, 1793 . 1795 [RFC5151] Farrel, A., Ed., Ayyangar, A., and JP. Vasseur, "Inter- 1796 Domain MPLS and GMPLS Traffic Engineering -- Resource 1797 Reservation Protocol-Traffic Engineering (RSVP-TE) 1798 Extensions", RFC 5151, DOI 10.17487/RFC5151, February 1799 2008, . 1801 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1802 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1803 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1804 September 2009, . 1806 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 1807 Networking: A Perspective from within a Service Provider 1808 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 1809 . 1811 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 1812 Henderickx, W., and A. Isaac, "Requirements for Ethernet 1813 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 1814 . 1816 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 1817 Ceccarelli, D., and X. Zhang, "Problem Statement and 1818 Architecture for Information Exchange between 1819 Interconnected Traffic-Engineered Networks", BCP 206, 1820 RFC 7926, DOI 10.17487/RFC7926, July 2016, 1821 . 1823 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 1824 Network Functions and Their Infrastructure", RFC 8172, 1825 DOI 10.17487/RFC8172, July 2017, 1826 . 1828 [RFC8370] Beeram, V., Ed., Minei, I., Shakir, R., Pacella, D., and 1829 T. Saad, "Techniques to Improve the Scalability of RSVP-TE 1830 Deployments", RFC 8370, DOI 10.17487/RFC8370, May 2018, 1831 . 1833 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1834 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1835 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1836 2018, . 1838 [RFC8491] Tantsura, J., Chunduri, U., Aldrin, S., and L. Ginsberg, 1839 "Signaling Maximum SID Depth (MSD) Using IS-IS", RFC 8491, 1840 DOI 10.17487/RFC8491, November 2018, 1841 . 1843 [RFC8568] Bernardos, CJ., Rahman, A., Zuniga, JC., Contreras, LM., 1844 Aranda, P., and P. Lynch, "Network Virtualization Research 1845 Challenges", RFC 8568, DOI 10.17487/RFC8568, April 2019, 1846 . 1848 [RFC8577] Sitaraman, H., Beeram, V., Parikh, T., and T. Saad, 1849 "Signaling RSVP-TE Tunnels on a Shared MPLS Forwarding 1850 Plane", RFC 8577, DOI 10.17487/RFC8577, April 2019, 1851 . 1853 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 1854 RFC 8578, DOI 10.17487/RFC8578, May 2019, 1855 . 1857 [SFC] "Service Function Chaining", March , 1858 . 1860 [TS23501] "3GPP TS23.501", 2016, 1861 . 1864 [TS28530] "3GPP TS28.530", 2016, 1865 . 1868 [TSN] "Time-Sensitive Networking", March , 1869 . 1871 Authors' Addresses 1873 Jie Dong 1874 Huawei 1876 Email: jie.dong@huawei.com 1878 Stewart Bryant 1879 Futurewei 1881 Email: stewart.bryant@gmail.com 1883 Zhenqiang Li 1884 China Mobile 1886 Email: lizhenqiang@chinamobile.com 1888 Takuya Miyasaka 1889 KDDI Corporation 1891 Email: ta-miyasaka@kddi.com 1893 Young Lee 1894 Sung Kyun Kwan University 1896 Email: younglee.tx@gmail.com