idnits 2.17.1 draft-irtf-nfvrg-gaps-network-virtualization-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 11, 2017) is 2604 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-05) exists of draft-bernardos-nfvrg-multidomain-01 == Outdated reference: A later version (-02) exists of draft-defoy-netslices-3gpp-network-slicing-00 == Outdated reference: A later version (-05) exists of draft-ietf-bmwg-virtual-net-04 Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFVRG CJ. Bernardos 3 Internet-Draft UC3M 4 Intended status: Informational A. Rahman 5 Expires: September 12, 2017 InterDigital 6 JC. Zuniga 7 SIGFOX 8 LM. Contreras 9 TID 10 P. Aranda 11 UC3M 12 March 11, 2017 14 Network Virtualization Research Challenges 15 draft-irtf-nfvrg-gaps-network-virtualization-04 17 Abstract 19 This document describes open research challenges for network 20 virtualization. Network virtualization is following a similar path 21 as previously taken by cloud computing. Specifically, Cloud 22 computing popularized migration of computing functions (e.g., 23 applications) and storage from local, dedicated, physical resources 24 to remote virtual functions accessible through the Internet. In a 25 similar manner, network virtualization is encouraging migration of 26 networking functions from dedicated physical hardware nodes to a 27 virtualized pool of resources. However, network virtualization can 28 be considered to be a more complex problem than cloud computing as it 29 not only involves virtualization of computing and storage functions 30 but also involves abstraction of the network itself. This document 31 describes current research challenges in network virtualization 32 including guaranteeing quality-of-service, performance improvement, 33 supporting multiple domains, network slicing, service composition, 34 device virtualization, privacy and security. In addition, some 35 proposals are made for new activities in IETF/IRTF that could address 36 some of these challenges. 38 Status of This Memo 40 This Internet-Draft is submitted in full conformance with the 41 provisions of BCP 78 and BCP 79. 43 Internet-Drafts are working documents of the Internet Engineering 44 Task Force (IETF). Note that other groups may also distribute 45 working documents as Internet-Drafts. The list of current Internet- 46 Drafts is at http://datatracker.ietf.org/drafts/current/. 48 Internet-Drafts are draft documents valid for a maximum of six months 49 and may be updated, replaced, or obsoleted by other documents at any 50 time. It is inappropriate to use Internet-Drafts as reference 51 material or to cite them other than as "work in progress." 53 This Internet-Draft will expire on September 12, 2017. 55 Copyright Notice 57 Copyright (c) 2017 IETF Trust and the persons identified as the 58 document authors. All rights reserved. 60 This document is subject to BCP 78 and the IETF Trust's Legal 61 Provisions Relating to IETF Documents 62 (http://trustee.ietf.org/license-info) in effect on the date of 63 publication of this document. Please review these documents 64 carefully, as they describe your rights and restrictions with respect 65 to this document. Code Components extracted from this document must 66 include Simplified BSD License text as described in Section 4.e of 67 the Trust Legal Provisions and are provided without warranty as 68 described in the Simplified BSD License. 70 Table of Contents 72 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 73 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 74 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 75 3.1. Network Function Virtualization . . . . . . . . . . . . . 5 76 3.2. Software Defined Networking . . . . . . . . . . . . . . . 7 77 3.3. Mobile Edge Computing . . . . . . . . . . . . . . . . . . 11 78 3.4. IEEE 802.1CF (OmniRAN) . . . . . . . . . . . . . . . . . 12 79 3.5. Distributed Management Task Force . . . . . . . . . . . . 12 80 3.6. Open Source initiatives . . . . . . . . . . . . . . . . . 12 81 3.7. Internet of Things (IoT) . . . . . . . . . . . . . . . . 14 82 4. Network Virtualization Challenges . . . . . . . . . . . . . . 14 83 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 14 84 4.2. Guaranteeing quality-of-service . . . . . . . . . . . . . 15 85 4.2.1. Virtualization Technologies . . . . . . . . . . . . . 15 86 4.2.2. Metrics for NFV characterization . . . . . . . . . . 15 87 4.2.3. Predictive analysis . . . . . . . . . . . . . . . . . 16 88 4.2.4. Portability . . . . . . . . . . . . . . . . . . . . . 17 89 4.3. Performance improvement . . . . . . . . . . . . . . . . . 17 90 4.3.1. Energy Efficiency . . . . . . . . . . . . . . . . . . 17 91 4.3.2. Improved link usage . . . . . . . . . . . . . . . . . 18 92 4.4. Multiple Domains . . . . . . . . . . . . . . . . . . . . 18 93 4.5. 5G and Network Slicing . . . . . . . . . . . . . . . . . 18 94 4.5.1. Virtual Network Operators . . . . . . . . . . . . . . 19 95 4.5.2. Extending Virtual Networks and Systems to the 96 Internet of Things . . . . . . . . . . . . . . . . . 20 97 4.6. Service Composition . . . . . . . . . . . . . . . . . . . 21 98 4.7. End-user device virtualization . . . . . . . . . . . . . 22 99 4.8. Security and Privacy . . . . . . . . . . . . . . . . . . 22 100 4.9. Separation of control concerns . . . . . . . . . . . . . 24 101 4.10. Testing . . . . . . . . . . . . . . . . . . . . . . . . . 24 102 4.10.1. Changes in methodology . . . . . . . . . . . . . . . 24 103 4.10.2. New functionality . . . . . . . . . . . . . . . . . 26 104 4.10.3. Opportunities . . . . . . . . . . . . . . . . . . . 26 105 5. Technology Gaps and Potential IETF Efforts . . . . . . . . . 27 106 6. Mapping to NFVRG Near-Term work items . . . . . . . . . . . . 27 107 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28 108 8. Security Considerations . . . . . . . . . . . . . . . . . . . 28 109 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 28 110 10. Informative References . . . . . . . . . . . . . . . . . . . 28 111 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32 113 1. Introduction 115 The telecommunications sector is experiencing a major revolution that 116 will shape the way networks and services are designed and deployed 117 for the next few decades. In order to cope with continuously 118 increasing demand and cost, network operators are taking lessons from 119 the IT paradigm of cloud computing. This new approach of 120 virtualizing network functions will enable multi-fold advantages by 121 outsourcing communication services from bespoke hardware in the 122 operator's core network to Commercial off-the-shelf (COTS) equipment 123 distributed across datacenters. 125 Some of the network virtualization mechanisms that are being 126 considered include: sharing of network infrastructure to reduce 127 costs, virtualization of core servers running in data centers as a 128 way of supporting their load- aware elastic dimensioning, and dynamic 129 energy policies to reduce the electricity consumption. 131 This document presents research challenges in Network Function 132 Virtualization (NFV) that need to be addressed in order to achieve 133 these goals. The objective of this memo is to document the technical 134 challenges and corresponding current approaches and to expose 135 requirements that should be addressed by future research and 136 standards work. 138 2. Terminology 140 The following terms used in this document are defined by the ETSI NVF 141 ISG [etsi_gs_nfv_003], the ONF [onf_tr_521] and the IETF [RFC7665]: 143 Application Plane - The collection of applications and services 144 that program network behavior. 146 Control Plane (CP) - The collection of functions responsible for 147 controlling one or more network devices. CP instructs network 148 devices with respect to how to process and forward packets. The 149 control plane interacts primarily with the forwarding plane and, 150 to a lesser extent, with the operational plane. 152 Forwarding Plane (FP) - The collection of resources across all 153 network devices responsible for forwarding traffic. 155 Management Plane (MP) - The collection of functions responsible 156 for monitoring, configuring, and maintaining one or more network 157 devices or parts of network devices. The management plane is 158 mostly related to the operational plane (it is related less to the 159 forwarding plane). 161 NFV Infrastructure (NFVI): totality of all hardware and software 162 components which build up the environment in which VNFs are 163 deployed 165 NFV Management and Orchestration (NFV-MANO): functions 166 collectively provided by NFVO, VNFM, and VIM. 168 NFV Orchestrator (NFVO): functional block that manages the Network 169 Service (NS) lifecycle and coordinates the management of NS 170 lifecycle, VNF lifecycle (supported by the VNFM) and NFVI 171 resources (supported by the VIM) to ensure an optimized allocation 172 of the necessary resources and connectivity. 174 OpenFlow protocol (OFP): allowing vendor independent programming 175 of control functions in network nodes. 177 Operational Plane (OP) - The collection of resources responsible 178 for managing the overall operation of individual network devices. 180 Physical Network Function (PNF): Physical implementation of a 181 Network Function in a monolithic realization. 183 Service Function Chain (SFC): for a given service, the abstracted 184 view of the required service functions and the order in which they 185 are to be applied. This is somehow equivalent to the Network 186 Function Forwarding Graph (NF-FG) at ETSI. 188 Service Function Path (SFP): the selection of specific service 189 function instances on specific network nodes to form a service 190 graph through which an SFC is instantiated. 192 Virtualized Infrastructure Manager (VIM): functional block that is 193 responsible for controlling and managing the NFVI compute, storage 194 and network resources, usually within one operator's 195 Infrastructure Domain. 197 Virtualized Network Function (VNF): implementation of a Network 198 Function that can be deployed on a Network Function Virtualization 199 Infrastructure (NFVI). 201 Virtualized Network Function Manager (VNFM): functional block that 202 is responsible for the lifecycle management of VNF. 204 3. Background 206 3.1. Network Function Virtualization 208 The ETSI ISG NFV is a working group which, since 2012, aims to evolve 209 quasi-standard IT virtualization technology to consolidate many 210 network equipment types into industry standard high volume servers, 211 switches, and storage. It enables implementing network functions in 212 software that can run on a range of industry standard server hardware 213 and can be moved to, or loaded in, various locations in the network 214 as required, without the need to install new equipment. To date, 215 ETSI NFV is by far the most accepted NFV reference framework and 216 architectural footprint [etsi_nvf_whitepaper_2]. The ETSI NFV 217 framework architecture framework is composed of three domains 218 (Figure 1): 220 o Virtualized Network Function, running over the NFVI. 222 o NFV Infrastructure (NFVI), including the diversity of physical 223 resources and how these can be virtualized. NFVI supports the 224 execution of the VNFs. 226 o NFV Management and Orchestration, which covers the orchestration 227 and life-cycle management of physical and/or software resources 228 that support the infrastructure virtualization, and the life-cycle 229 management of VNFs. NFV Management and Orchestration focuses on 230 all virtualization specific management tasks necessary in the NFV 231 framework. 233 +-------------------------------------------+ +---------------+ 234 | Virtualized Network Functions (VNFs) | | | 235 | ------- ------- ------- ------- | | | 236 | | | | | | | | | | | | 237 | | VNF | | VNF | | VNF | | VNF | | | | 238 | | | | | | | | | | | | 239 | ------- ------- ------- ------- | | | 240 +-------------------------------------------+ | | 241 | | 242 +-------------------------------------------+ | | 243 | NFV Infrastructure (NFVI) | | NFV | 244 | ----------- ----------- ----------- | | Management | 245 | | Virtual | | Virtual | | Virtual | | | and | 246 | | Compute | | Storage | | Network | | | Orchestration | 247 | ----------- ----------- ----------- | | | 248 | +---------------------------------------+ | | | 249 | | Virtualization Layer | | | | 250 | +---------------------------------------+ | | | 251 | +---------------------------------------+ | | | 252 | | ----------- ----------- ----------- | | | | 253 | | | Compute | | Storage | | Network | | | | | 254 | | ----------- ----------- ----------- | | | | 255 | | Hardware resources | | | | 256 | +---------------------------------------+ | | | 257 +-------------------------------------------+ +---------------+ 259 Figure 1: ETSI NFV framework 261 The NFV architectural framework identifies functional blocks and the 262 main reference points between such blocks. Some of these are already 263 present in current deployments, whilst others might be necessary 264 additions in order to support the virtualization process and 265 consequent operation. The functional blocks are (Figure 2): 267 o Virtualized Network Function (VNF). 269 o Element Management (EM). 271 o NFV Infrastructure, including: Hardware and virtualized resources, 272 and Virtualization Layer. 274 o Virtualized Infrastructure Manager(s) (VIM). 276 o NFV Orchestrator. 278 o VNF Manager(s). 280 o Service, VNF and Infrastructure Description. 282 o Operations and Business Support Systems (OSS/BSS). 284 +--------------------+ 285 +-------------------------------------------+ | ---------------- | 286 | OSS/BSS | | | NFV | | 287 +-------------------------------------------+ | | Orchestrator +-- | 288 | ---+------------ | | 289 +-------------------------------------------+ | | | | 290 | --------- --------- --------- | | | | | 291 | | EM 1 | | EM 2 | | EM 3 | | | | | | 292 | ----+---- ----+---- ----+---- | | ---+---------- | | 293 | | | | |--|-| VNF | | | 294 | ----+---- ----+---- ----+---- | | | manager(s) | | | 295 | | VNF 1 | | VNF 2 | | VNF 3 | | | ---+---------- | | 296 | ----+---- ----+---- ----+---- | | | | | 297 +------|-------------|-------------|--------+ | | | | 298 | | | | | | | 299 +------+-------------+-------------+--------+ | | | | 300 | NFV Infrastructure (NFVI) | | | | | 301 | ----------- ----------- ----------- | | | | | 302 | | Virtual | | Virtual | | Virtual | | | | | | 303 | | Compute | | Storage | | Network | | | | | | 304 | ----------- ----------- ----------- | | ---+------ | | 305 | +---------------------------------------+ | | | | | | 306 | | Virtualization Layer | |--|-| VIM(s) +-------- | 307 | +---------------------------------------+ | | | | | 308 | +---------------------------------------+ | | ---------- | 309 | | ----------- ----------- ----------- | | | | 310 | | | Compute | | Storage | | Network | | | | | 311 | | | hardware| | hardware| | hardware| | | | | 312 | | ----------- ----------- ----------- | | | | 313 | | Hardware resources | | | NFV Management | 314 | +---------------------------------------+ | | and Orchestration | 315 +-------------------------------------------+ +--------------------+ 317 Figure 2: ETSI NFV reference architecture 319 3.2. Software Defined Networking 321 The Software Defined Networking (SDN) paradigm pushes the 322 intelligence currently residing in the network elements to a central 323 controller implementing the network functionality through software. 324 In contrast to traditional approaches, in which the network's control 325 plane is distributed throughout all network devices, with SDN the 326 control plane is logically centralized. In this way, the deployment 327 of new characteristics in the network no longer requires of complex 328 and costly changes in equipment or firmware updates, but only a 329 change in the software running in the controller. The main advantage 330 of this approach is the flexibility it provides operators with to 331 manage their network, i.e., an operator can easily change its 332 policies on how traffic is distributed throughout the network. 334 The most visible of the SDN protocol stacks is the OpenFlow protocol 335 (OFP), which is maintained and extended by the Open Network 336 Foundation (ONF: https://www.opennetworking.org/). Originally this 337 protocol was developed specifically for IEEE 802.1 switches 338 conforming to the ONF OpenFlow Switch specification. As the benefits 339 of the SDN paradigm have reached a wider audience, its application 340 has been extended to more complex scenarios such as Wireless and 341 Mobile networks. Within this area of work, the ONF is actively 342 developing new OFP extensions addressing three key scenarios: (i) 343 Wireless backhaul, (ii) Cellular Evolved Packet Core (EPC), and (iii) 344 Unified access and management across enterprise wireless and fixed 345 networks. 347 +----------+ 348 | ------- | 349 | |Oper.| | O 350 | |Mgmt.| |<........> -+- Network Operator 351 | |Iface| | ^ 352 | ------- | +----------------------------------------+ 353 | | | +------------------------------------+ | 354 | | | | --------- --------- --------- | | 355 |--------- | | | | App 1 | | App 2 | ... | App n | | | 356 ||Plugins| |<....>| | --------- --------- --------- | | 357 |--------- | | | Plugins | | 358 | | | +------------------------------------+ | 359 | | | Application Plane | 360 | | +----------------------------------------+ 361 | | A 362 | | | 363 | | V 364 | | +----------------------------------------+ 365 | | | +------------------------------------+ | 366 |--------- | | | ------------ ------------ | | 367 || Netw. | | | | | Module 1 | | Module 2 | | | 368 ||Engine | |<....>| | ------------ ------------ | | 369 |--------- | | | Network Engine | | 370 | | | +------------------------------------+ | 371 | | | Controller Plane | 372 | | +----------------------------------------+ 373 | | A 374 | | | 375 | | V 376 | | +----------------------------------------+ 377 | | | +--------------+ +--------------+ | 378 | | | | ------------ | | ------------ | | 379 |----------| | | | OpenFlow | | | | OpenFlow | | | 380 ||OpenFlow||<....>| | ------------ | | ------------ | | 381 |----------| | | NE | | NE | | 382 | | | +--------------+ +--------------+ | 383 | | | Data Plane | 384 |Management| +----------------------------------------+ 385 +----------+ 387 Figure 3: High level SDN ONF architecture 389 Figure 3 shows the blocks and the functional interfaces of the ONF 390 architecture, which comprises three planes: Data, Controller, and 391 Application. The Data plane comprehends several Network Entities 392 (NE), which expose their capabilities toward the Controller plane via 393 a Southbound API. The Controller plane includes several cooperating 394 modules devoted to the creation and maintenance of an abstracted 395 resource model of the underneath network. Such model is exposed to 396 the applications via a Northbound API where the Application plane 397 comprises several applications/services, each of which has exclusive 398 control of a set of exposed resources. 400 The Management plane spans its functionality across all planes 401 performing the initial configuration of the network elements in the 402 Data plane, the assignment of the SDN controller and the resources 403 under its responsibility. In the Controller plane, the Management 404 needs to configure the policies defining the scope of the control 405 given to the SDN applications, to monitor the performance of the 406 system, and to configure the parameters required by the SDN 407 controller modules. In the Application plane, Management configures 408 the parameters of the applications and the service level agreements. 409 In addition to the these interactions, the Management plane exposes 410 several functions to network operators which can easily and quickly 411 configure and tune the network at each layer. 413 The IRTF Software-Defined Networking Research Group (SDNRG) 414 documented in RFC7426 [RFC7426], a layer model of an SDN 415 architecture, since this has been a controvertial discussion topic: 416 what is exactly SDN? what is the layer structure of the SDN 417 architecture? how do layers interface with each oter? etc. 419 Figure 4 reproduces the figure included in RFC7426 [RFC7426] to 420 summarize the SDN architecture abstractions in the form of a 421 detailed, high-level schematic. In a particular implementation, 422 planes can be collocated with other planes or can be physically 423 separated. 425 In SDN, a controller manipulates controlled entities via an 426 interface.Interfaces, when local, are mostly API invocations through 427 some library or system call. However, such interfaces may be 428 extended via some protocol definition, which may use local inter- 429 process communication (IPC) or a protocol that could also act 430 remotely; the protocol may be defined as an open standard or in a 431 proprietary manner. 433 SDN expands multiple planes: Forwarding, Operational, Control, 434 Management and Applications. All planes mentioned above are 435 connected via interfaces. Additionally, RFC7426 [RFC7426] considers 436 four abstraction layers: the Device and resource Abstraction Layer 437 (DAL), the Control Abstraction Layer (CAL), the Management 438 Abstraction Layer (MAL) and the Network Services Abstraction Layer 439 (NSAL). 441 o--------------------------------o 442 | | 443 | +-------------+ +----------+ | 444 | | Application | | Service | | 445 | +-------------+ +----------+ | 446 | Application Plane | 447 o---------------Y----------------o 448 | 449 *-----------------------------Y---------------------------------* 450 | Network Services Abstraction Layer (NSAL) | 451 *------Y------------------------------------------------Y-------* 452 | | 453 | Service Interface | 454 | | 455 o------Y------------------o o---------------------Y------o 456 | | Control Plane | | Management Plane | | 457 | +----Y----+ +-----+ | | +-----+ +----Y----+ | 458 | | Service | | App | | | | App | | Service | | 459 | +----Y----+ +--Y--+ | | +--Y--+ +----Y----+ | 460 | | | | | | | | 461 | *----Y-----------Y----* | | *---Y---------------Y----* | 462 | | Control Abstraction | | | | Management Abstraction | | 463 | | Layer (CAL) | | | | Layer (MAL) | | 464 | *----------Y----------* | | *----------Y-------------* | 465 | | | | | | 466 o------------|------------o o------------|---------------o 467 | | 468 | CP | MP 469 | Southbound | Southbound 470 | Interface | Interface 471 | | 472 *------------Y---------------------------------Y----------------* 473 | Device and resource Abstraction Layer (DAL) | 474 *------------Y---------------------------------Y----------------* 475 | | | | 476 | o-------Y----------o +-----+ o--------Y----------o | 477 | | Forwarding Plane | | App | | Operational Plane | | 478 | o------------------o +-----+ o-------------------o | 479 | Network Device | 480 +---------------------------------------------------------------+ 482 Figure 4: SDN Layer Architecture 484 3.3. Mobile Edge Computing 486 Mobile Edge Computing capabilities deployed in the edge of the mobile 487 network can facilitate the efficient and dynamic provision of 488 services to mobile users. The ETSI ISG MEC working group, operative 489 from end of 2014, intends to specify an open environment for 490 integrating MEC capabilities with service providers networks, 491 including also applications from 3rd parties. These distributed 492 computing capabilities will make available IT infrastructure as in a 493 cloud environment for the deployment of functions in mobile access 494 networks. It can be seen then as a complement to both NFV and SDN. 496 3.4. IEEE 802.1CF (OmniRAN) 498 The IEEE 802.1CF Recommended Practice [omniran] specifies an access 499 network, which connects terminals to their access routers, utilizing 500 technologies based on the family of IEEE 802 Standards (e.g., 802.3 501 Ethernet, 802.11 Wi-Fi, etc.). The specification defines an access 502 network reference model, including entities and reference points 503 along with behavioral and functional descriptions of communications 504 among those entities. 506 The goal of this project is to help unifying the support of different 507 interfaces, enabling shared network control and use of software 508 defined network (SDN) principles, thereby lowering the barriers to 509 new network technologies, to new network operators, and to new 510 service providers. 512 3.5. Distributed Management Task Force 514 The DMTF is an industry standards organization working to simplify 515 the manageability of network-accessible technologies through open and 516 collaborative efforts by some technology companies. The DMTF is 517 involved in the creation and adoption of interoperable management 518 standards, supporting implementations that enable the management of 519 diverse traditional and emerging technologies including cloud, 520 virtualization, network and infrastructure. 522 There are several DMTF initiatives that are relevant to the network 523 virtualization area, such as the Open Virtualization Format (OVF), 524 for VNF packaging; the Cloud Infrastructure Management Interface 525 (CIM), for cloud infrastructure management; the Network Management 526 (NETMAN), for VNF management; and, the Virtualization Management 527 (VMAN), for virtualization infrastructure management. 529 3.6. Open Source initiatives 531 The Open Source community is especially active in the area of network 532 virtualization. We next summarize some of the active efforts: 534 o OpenStack. OpenStack is a free and open-source cloud-computing 535 software platform. OpenStack software controls large pools of 536 compute, storage, and networking resources throughout a 537 datacenter, managed through a dashboard or via the OpenStack API. 539 o OpenDayLight. OpenDaylight (ODL) is a highly available, modular, 540 extensible, scalable and multi-protocol controller infrastructure 541 built for SDN deployments on modern heterogeneous multi-vendor 542 networks. It provides a model-driven service abstraction platform 543 that allows users to write apps that easily work across a wide 544 variety of hardware and southbound protocols. 546 o ONOS. The ONOS (Open Network Operating System) project is an open 547 source community hosted by The Linux Foundation. The goal of the 548 project is to create a software-defined networking (SDN) operating 549 system for communications service providers that is designed for 550 scalability, high performance and high availability. 552 o OpenContrail. OpenContrail is an Apache 2.0-licensed project that 553 is built using standards-based protocols and provides all the 554 necessary components for network virtualization-SDN controller, 555 virtual router, analytics engine, and published northbound APIs. 556 It has an extensive REST API to configure and gather operational 557 and analytics data from the system. 559 o OPNFV. OPNFV is a carrier-grade, integrated, open source platform 560 to accelerate the introduction of new NFV products and services. 561 By integrating components from upstream projects, the OPNFV 562 community aims at conducting performance and use case-based 563 testing to ensure the platform's suitability for NFV use cases. 564 The scope of OPNFV's initial release is focused on building NFV 565 Infrastructure (NFVI) and Virtualized Infrastructure Management 566 (VIM) by integrating components from upstream projects such as 567 OpenDaylight, OpenStack, Ceph Storage, KVM, Open vSwitch, and 568 Linux. These components, along with application programmable 569 interfaces (APIs) to other NFV elements form the basic 570 infrastructure required for Virtualized Network Functions (VNF) 571 and Management and Network Orchestration (MANO) components. 572 OPNFV's goal is to increase performance and power efficiency; 573 improve reliability, availability, and serviceability; and deliver 574 comprehensive platform instrumentation. 576 o OSM. Open Source Mano (OSM) is an ETSI-hosted project to develop 577 an Open Source NFV Management and Orchestration (MANO) software 578 stack aligned with ETSI NFV. OSM is based on components from 579 previous projects, such Telefonica's OpenMANO or Canonical's Juju, 580 among others. 582 o OpenBaton. OpenBaton is a ETSI NFV compliant Network Function 583 Virtualization Orchestrator (NFVO). OpenBaton was part of the 584 OpenSDNCore project started with the objective of providing a 585 compliant implementation of the ETSI NFV specification. 587 Among the main areas that are being developed by the former open 588 source activities that related to network virtualization research, we 589 can highlight: policy-based resource management, analytics for 590 visibility and orchestration, service verification with regards to 591 security and resiliency. 593 3.7. Internet of Things (IoT) 595 The Internet of Things (IoT) refers to the vision of connecting a 596 multitude of automated devices (e.g. lights, environmental sensors, 597 traffic lights, parking meters, health and security systems, etc.) to 598 the Internet for purposes of reporting, and remote command and 599 control of the device. This vision is being realized by a multi- 600 pronged approach of standardization in various forums and 601 complementary open source activities. For example, in IETF, support 602 of IoT web services has been defined by an HTTP-like protocol adapted 603 for IoT called CoAP [RFC7252], and lately a group has been studying 604 the need to develop a new network layer to support IP applications 605 over Low Power Wide Area Networks (LPWAN). 607 Elsewhere, for 5G cellular evolution there is much discussion on the 608 need for supporting virtual "network slices" for the expected massive 609 numbers of IoT devices. A separate virtual network slice is 610 considered necessary for different 5G IoT use cases because devices 611 will have very different characteristics than typical cellular 612 devices like smart phones [ngmn_5G_whitepaper], and the number of IoT 613 devices is expected to be at least one or two orders of magnitude 614 higher than other 5G devices. 616 4. Network Virtualization Challenges 618 4.1. Introduction 620 Network Virtualization is changing the way the telecommunications 621 sector will deploy, extend and operate their networks. These new 622 technologies aim at reducing the overall costs by outsourcing 623 communication services from specific hardware in the operators' core 624 to server farms scattered in datacenters (i.e. compute and storage 625 virtualization). In addition, the connecting networks are 626 fundamentally affected in the way they route, process and control 627 traffic (i.e. network virtualization). 629 4.2. Guaranteeing quality-of-service 631 Guaranteeing a given quality-of-service in an NFV environment is not 632 an easy task. For example, ensuring a guaranteed and stable 633 forwarding data rate has proven not to be straightforward when the 634 forwarding function is virtualized and runs on top of COTS server 635 hardware [openmano_dataplane] 636 [I-D.mlk-nfvrg-nfv-reliability-using-cots] [etsi_nvf_whitepaper_3]. 637 We next identify some of the challenges that this poses. 639 4.2.1. Virtualization Technologies 641 The issue of guaranteeing a network quality-of-service is less of an 642 issue for "traditional cloud computing" because the workloads that 643 are treated there are servers or clients in the networking sense and 644 hardly ever process packets. Cloud computing provides hosting for 645 applications on shared servers in a highly separated way. Its main 646 advantage is that the infrastructure costs are shared among tenants 647 and that the Cloud infrastructure provides levels of reliability that 648 can not be achieved on individual premises in a cost-efficient way 649 [intel_10_differences_nfv_cloud]. NFV poses very strict requirements 650 posed in terms of performance, stability and consistency. Although 651 there are some tools and mechanisms to improve this, such as Enhanced 652 Performance Awareness (EPA), SR-IOV, NUMA, DPDK, etc, these are still 653 unsolved challenges. One open research issue is finding out 654 technologies that are different from VM and more suitable for dealing 655 with network functionalities. 657 Lately, a number of light-weight virtualization technologies 658 including containers, unikernels (specialized VMs) and minimalistic 659 distributions of general-purpose OSes have appeared as virtualization 660 approaches that can be used when constructing an NFV platform. 661 [I-D.natarajan-nfvrg-containers-for-nfv] describes the challenges in 662 building such a platform and discusses to what extent these 663 technologies, as well as traditional VMs, are able to address them. 665 4.2.2. Metrics for NFV characterization 667 Another relevant aspect is the need for tools for diagnostics and 668 measurement suited for NFV. There is a pressing need to define 669 metrics and associated protocols to measure the performance of NFV. 670 Specifically, since NFV is based on the concept of taking centralized 671 functions and evolving it to highly distributed SW functions, there 672 is a commensurate need to fully understand and measure the baseline 673 performance of such systems. 675 The IP Performance Metrics (IPPM) WG defines metrics that can be used 676 to measure the quality and performance of Internet services and 677 applications running over transport layer protocols (e.g., TCP, UPD) 678 over IP. It also develops and maintains protocols for the 679 measurement of these metrics. While the IPPM WG is a long running WG 680 that started in 1997 it does not have a charter item or active drafts 681 related to the topic of network virtualization. In addition to using 682 IPPM metrics to evaluate the QoS, there is a need for specific 683 metrics for assessing the performance of network virtualization 684 techniques. 686 The Benchmarking Methodology Working Group (BMWG) is also performing 687 work related to NFV metrics. For example, 688 [I-D.ietf-bmwg-virtual-net] investigates additional methodological 689 considerations necessary when benchmarking VNFs instantiated and 690 hosted in general- purpose hardware, using bare-metal hypervisors or 691 other isolation environments such as Linux containers. An essential 692 consideration is benchmarking physical and virtual network functions 693 in the same way when possible, thereby allowing direct comparison. 695 As stated in the document [I-D.ietf-bmwg-virtual-net], there is a 696 clear motivation for the work on performance metrics for NFV 697 [etsi_gs_nfv_per_001], that is worth replicating here: "I'm designing 698 and building my NFV Infrastructure platform. The first steps were 699 easy because I had a small number of categories of VNFs to support 700 and the VNF vendor gave HW recommendations that I followed. Now I 701 need to deploy more VNFs from new vendors, and there are different 702 hardware recommendations. How well will the new VNFs perform on my 703 existing hardware? Which among several new VNFs in a given category 704 are most efficient in terms of capacity they deliver? And, when I 705 operate multiple categories of VNFs (and PNFs) *concurrently* on a 706 hardware platform such that they share resources, what are the new 707 performance limits, and what are the software design choices I can 708 make to optimize my chosen hardware platform? Conversely, what 709 hardware platform upgrades should I pursue to increase the capacity 710 of these concurrently operating VNFs?" 712 Lately, there are also some efforts lately lookinh into VNF 713 benchmarking. The selection of an NFV Infrastructure Point of 714 Presence to host a VNF or allocation of resources (e.g., virtual 715 CPUs, memory) needs to be done over virtualized (abstracted and 716 simplified) resource views [vnf_benchmarking] 717 [I-D.rorosz-nfvrg-vbaas]. 719 4.2.3. Predictive analysis 721 On top of diagnostic tools that enable an assessment of the QoS, 722 predictive analyses are required to react before anomalies occur. 723 Due to the SW characteristics of VNFs, a reliable diagnosis framework 724 could potentially enable the prevention of issues by a proper 725 diagnosis and then a reaction in terms of acting on the potentially 726 impacted service (e.g., migration to a different compute node, 727 scaling in/out, up/down, etc). 729 4.2.4. Portability 731 Portability in NFV refers to the ability to run a given VNF on 732 multiple NFVIs, that is, that it is possible to guarantee that the 733 VNF would be able to perform its functions with a high and 734 predictable performance given that a set of requirements on the NFVI 735 resources is met. Therefore, portability is a key feature that, if 736 fully enabled, would contribute to making the NFV environment achieve 737 a better reliability than a traditional system. The fact of running 738 functionality in SW over "commodity" infrastructure should make much 739 easier to port/move functions from one place to another. However 740 this is not yet as ideal as it sounds and there are aspects not fully 741 tackled. The existence of different hypervisors, specific hardware 742 dependencies (e.g., EPA related) or state synchronization aspects are 743 just some examples of trouble-makers for portability purposes. 745 The ETSI NFV ISG is doing work in relation to portability. 746 [etsi_gs_nfv_per_001] provides a list of minimal features which the 747 VM Descriptor and Compute Host Descriptor should contain for the 748 appropriate deployment of VM Images over an NFVI (i.e. a "telco 749 datacentre"), in order to guarantee high and predictable performance 750 of data plane workloads while assuring their portability. In 751 addition, the document provides a set of recommendations on the 752 minimum requirements which HW and hypervisor should have for a "telco 753 datacentre" suitable for different workloads (data-plane, control- 754 plane, etc.) present in VNFs. The purpose of this document is to 755 provide the list of VM requirements that should be included in the VM 756 Descriptor template, and the list of HW capabilities that should be 757 included in the Compute Host Descriptor (CHD) to assure predictable 758 high performance. ETSI NFV assumes that the MANO Functions will make 759 the mix & match. There are therefore still quite several research 760 challenges to be addressed here. 762 4.3. Performance improvement 764 4.3.1. Energy Efficiency 766 Virtualization is typically seen as a direct enabler of energy 767 savings. Some of the enablers for this that are often mentioned 768 [nfv_sota_research_challenges] are: (i) the multiplexing gains 769 achieved by centralizing functions in data centers reduce overall the 770 energy consumed, (ii) the flexibility brought by network 771 programmability enables to switch off infrastructure as needed in a 772 much easier way. However there is still a lot of room for 773 improvement in terms of virtualization techniques to reduce the power 774 consumption, such as enhanced hypervisor technologies. 776 4.3.2. Improved link usage 778 The use of NFV and SDN technologies can help improving link usage. 779 SDN has shown already that it can greatly increase average link usage 780 (e.g., Google example [google_sdn_wan]). NFV adds more complexity 781 (e.g., due to service function chaining / VNF forwarding drafts) 782 which need to be considered. Aspects like the ones described in 783 [I-D.bagnulo-nfvrg-topology] on NFV data center topology design have 784 to be carefully looked as well. 786 4.4. Multiple Domains 788 Market fragmentation has resulted in a multitude of network operators 789 each focused on different countries and regions. This makes it 790 difficult to create infrastructure services spanning multiple 791 countries, such as virtual connectivity or compute resources, as no 792 single operator has a footprint everywhere. Cross-domain 793 orchestration of services over multiple administrations or over 794 multi-domain single administrations will allow end-to-end network and 795 service elements to mix in multi-vendor, heterogeneous technology and 796 resource environments. 798 For the specific use case of 'Network as a Service', it becomes even 799 more important to ensure, that Cross Domain Orchestration also takes 800 care of hierarchy of networks and their association, with respect to 801 provisioning tunnels and overlays. 803 Multi-domain orchestration is currently an active research topic, 804 which is being tackled, among others, by ETSI NFV ISG and the 5GEx 805 project (https://www.5gex.eu/) [I-D.bernardos-nfvrg-multidomain]. 807 4.5. 5G and Network Slicing 809 From the beginning of all 5G discussions in the research and industry 810 fora, it has been agreed that 5G will have to address much more use 811 cases than the preceding wireless generations, which first focused on 812 voice services, and then on voice and high speed packet data 813 services. In this case, 5G should be able to handle not only the 814 same (or enhanced) voice and packet data services, but also new 815 emerging services like tactile Internet and IoT. These use cases 816 take the requirements to opposite extremes, as some of them require 817 ultra-low latency and higher-speed, whereas some others require 818 ultra-low power consumption and high delay tolerance. 820 Because of these very extreme 5G use cases, it is envisioned that 821 different radio access networks are needed to better address the 822 specific requirements of each one of the use cases. However, on the 823 core network side, virtualization techniques can allow tailoring the 824 network resources on separate slices, specifically for each radio 825 access network and use case, in an efficient manner. 827 Network slicing techniques can also allow dedicating resources for 828 even more specific use cases within the major 5G categories. For 829 example, within the major IoT category, which is perhaps the most 830 disrupting one, some autonomous IoT devices will have very low 831 throughput, will have much longer sleep cycles (and therefore high 832 latency), and a battery life thousands of times longer compared to 833 smart phones or some other connected IoT devices that will have 834 almost continuous control and data communications. Hence, it is 835 envisioned that a single virtual core network could be used by 836 slicing separate resources to dedicated radio access networks (RANs) 837 that are better suited for specific use cases. 839 The actual definition of network slicing is still a sensitive 840 subject, currently under heavy discussion 841 [I-D.gdmb-netslices-intro-and-ps] 842 [I-D.defoy-netslices-3gpp-network-slicing] [ngmn_5G_whitepaper]. 843 Network slicing is a key for introducing new actors in existing 844 market at low cost -- by letting new players rent "blocks" of 845 capacity, if this new market provides performance that are adequate 846 with the application needs (e.g., broadcasting updates to many 847 sensors with satellite broadcasting capabilities). However, more 848 work needs to be done to define how network slicing will impact 849 existing architectures like ETSI NFV, and to define the impacts of 850 network slicing to guaranteeing quality-of-service as described in 851 Section 4.2. 853 4.5.1. Virtual Network Operators 855 The widespread of system and network virtualization technologies has 856 conducted to new business opportunities, enlarging the offer of IT 857 resources with virtual network and computing resources, among others. 858 As a consequence, the network ecosystem now differentiates between 859 the owner of physical resources, the Infrastructure Provider (InP), 860 and the intermediary that conforms and delivers network services to 861 the final customers, the Virtual Network Operator (VNO). 863 VNOs aim to exploit the virtualized infrastructures to deliver new 864 and improved services to their customers. However, current network 865 virtualization techniques offer poor support for VNOs to control 866 their resources. It has been considered that the InP is responsible 867 of the reliability of the virtual resources but there are several 868 situations in which an VNO requires to gain a finer control on its 869 resources. For instance, dynamic events, such as the identification 870 of new requirements or the detection of incidents within the virtual 871 system, might urge a VNO to quickly reform its virtual infrastructure 872 and resource allocation. However, the interfaces offered by current 873 virtualization platforms do not offer the necessary functions for 874 VNOs to perform the elastic adaptations they require to tackle with 875 their dynamic operation environments. 877 Beyond their heterogeneity, which can be resolved by software 878 adapters, current virtualization platforms do not have common methods 879 and functions, so it is difficult for the virtual network controllers 880 used by the VNOs to actually manage and control virtual resources 881 instantiated on different platforms, not even considering different 882 InPs. Therefore it is necessary to reach a common definition of the 883 functions that should be offered by underlying platforms to enable 884 such overlay controllers with the possibility of allocate and 885 deallocate resources dynamically and get monitoring data about them. 887 Such common methods should be offered by all underlying controllers, 888 regardless of being network-oriented (e.g. ODL, ONOS, Ryu) or 889 computing-oriented (e.g. OpenStack, OpenNebula, Eucalyptus). 890 Furthermore, it is also important for those platforms to offer some 891 "PUSH" function to report resource state, avoiding the need for the 892 VNO's controller to "POLL" for such data. A starting point to get 893 proper notifications within current REST APIs could be to consider 894 the protocol proposed by the WEBPUSH WG. 896 Finally, in order to establish a proper order and allow the 897 coexistence and collaboration of different systems, a common ontology 898 regarding network and system virtualization should be defined and 899 agreed, so different and heterogeneous systems can understand each 900 other without requiring to rely on specific adatpation mechanisms 901 that might break with any update on any side of the relation. 903 4.5.2. Extending Virtual Networks and Systems to the Internet of Things 905 The specific nature of the Internet of Things (IoT) ecosystem, 906 particularly reflected in the Machine-to-Machine (M2M) 907 communications, conducts to the creation of new and highly 908 distributed systems which demand location-based network and computing 909 services. An specific example can be represented by a set of 910 "things" that suddenly require to set-up a firewall to allow external 911 entities to access their data while outsourcing some computation 912 requirements to more powerful systems relying on Cloud-based 913 services. This representative use case exposes important 914 requirements for both NFV and the underlying Cloud infrastructures. 916 In order to provide the aforementioned location-based functions 917 integrated with highly distributed systems, the so called FOG 918 infrastructures should be able to instantiate VNFs, placing them in 919 the required place, e.g. close to their consumers. This requirement 920 implies that the interfaces offered by virtualization platforms must 921 support the specification of location-based resources, which is a key 922 function in those scenarios. Moreover, those platforms must also be 923 able to interpret and understand the references used by IoT systems 924 to their location (e.g., "My-AP", "5BLDG+2F") and also the 925 specification of identifiers linked to other resources, such as the 926 case of requiring the infrastructure to establish a link between a 927 specific AP and a specific virtual computing node. 929 4.6. Service Composition 931 Current network services deployed by operators often involve the 932 composition of several individual functions (such as packet 933 filtering, deep packet inspection, load balancing). These services 934 are typically implemented by the ordered combination of a number of 935 service functions that are deployed at different points within a 936 network, not necessary on the direct data path. This requires 937 traffic to be steered through the required service functions, 938 wherever they are deployed [RFC7498]. 940 For a given service, the abstracted view of the required service 941 functions and the order in which they are to be applied is called a 942 Service Function Chain (SFC), which is called Network Function 943 Forwarding Graph (NF-FG) in ETSI. An SFC is instantiated through 944 selection of specific service function instances on specific network 945 nodes to form a service graph: this is called a Service Function Path 946 (SFP). The service functions may be applied at any layer within the 947 network protocol stack (network layer, transport layer, application 948 layer, etc.). 950 Service composition is a powerful tool which can provide significant 951 benefits when applied in a softwarized network environment. There 952 are however many research challenges in this area, as for example the 953 ones related to composition mechanisms and algorithms to enable load 954 balancing and improve reliability. The service composition should 955 also act as an enabler to gather information across all hierarchies 956 (underlays and overlays) of network deployments which may span across 957 multiple operators, for faster serviceability thus facilitating in 958 accomplishing aforementioned goals of "load balancing and improve 959 reliability". 961 The SFC working group is working on an architecture for service 962 function chaining [RFC7665] that includes the necessary protocols or 963 protocol extensions to convey the Service Function Chain and Service 964 Function Path information to nodes that are involved in the 965 implementation of service functions and Service Function Chains, as 966 well as mechanisms for steering traffic through service functions. 968 In terms of actual work items, the SFC WG is has not yet considered 969 working on the management and configuration of SFC components related 970 to the support of Service Function Chaining. This part is of special 971 interest for operators and would be required in order to actually put 972 SFC mechanisms into operation. Similarly, redundancy and reliability 973 mechanisms are currently not dealt with by any WG in the IETF. While 974 this was the main goal of the VNFpool BoF efforts, it still remains 975 unaddressed. 977 4.7. End-user device virtualization 979 So far, most of the network softwarization efforts have focused on 980 virtualizing functions of network elements. While virtualization of 981 network elements started with the core, mobile networks architectures 982 are now heavily switching to also virtualize radio access network 983 (RAN) functions. The next natural step is to get virtualization down 984 at the level of the end-user device (i.e., virtualizing a smartphone) 985 [virtualization_mobile_device]. The cloning of a device in the cloud 986 (central or local) bears attractive benefits to both the device and 987 network operations alike (e.g., power saving at the device by 988 offloading computational-heaving functions to the cloud, optimized 989 networking -- both device-to-device and device-to-infrastructure) for 990 service delivery through tighter integration of the device (via its 991 clone in the networking infrastructure). This is being explored for 992 example by the European H2020 ICIRRUS project (www.icirrus-5gnet.eu). 994 4.8. Security and Privacy 996 Similar to any other situation where resources are shared, security 997 and privacy are two important aspects that need to be taken into 998 account. 1000 In the case of security, there are situations where multiple vendors 1001 will need to coexist in a virtual or hybrid physical/virtual 1002 environment. This requires attestation procedures amongst different 1003 virtual/physical functions and resources, as well as ongoing external 1004 monitoring. Similarly, different network slices operating on the 1005 same infrastructure can present security problems, for instance if 1006 one slice running critical applications (e.g. support for a safety 1007 system) is affected by another slice running a less critical 1008 application. In general, the minimum common denominator for security 1009 measures on a shared system should be equal or higher than the one 1010 required by the most critical application. Multiple and continuous 1011 threat model analysis, as well as DevOps model are required to 1012 maintain certain level of security in an NFV system. 1014 On the other hand, privacy in its strictest interpretation, refers to 1015 concerns about exposing users of the system to individual threats 1016 such as surveillance, identification, stored data compromise, 1017 secondary use, intrusion, etc. In this case, the storage, 1018 transmission, collection, and potential correlation of information in 1019 the NFV system, for purposes not originally intended or not known by 1020 the user, should be avoided. This is particularly challenging, as 1021 future intentions and threats cannot be easily predicted, and still 1022 can be applied for instance on data collected in the past. 1023 Therefore, well-known techniques such as data minimization, using 1024 privacy features as default, and allowing users to opt in/out should 1025 be used to prevent potential privacy issues. 1027 Compared to traditional networks, NFV will result in networks that 1028 are much more dynamic (in function distribution and topology) and 1029 elastic (in size and boundaries). NFV will thus require network 1030 operators to evolve their operational and administrative security 1031 solutions to work in this new environment. For example, in NFV the 1032 network orchestrator will become a key node to provide security 1033 policy orchestration across the different physical and virtual 1034 components of the virtualized network. For highly confidential data, 1035 for example, the network orchestrator should take into account if 1036 certain physical HW of the network is considered more secure (e.g., 1037 because it is located in secure premises) than other HW. 1039 Traditional telecom networks typically run under a single 1040 administrative domain controlled by an operator. With NFV, it is 1041 expected that in many cases, the telecom operator will now become a 1042 tenant (running the VNFs), and the infrastructure (NFVI) may be run 1043 by a different operator and/or cloud service provider (see also 1044 Section 4.4). Thus, there will be multiple administrative domains 1045 which will make coordination of security policy more complex. For 1046 example, who will be in charge of provisioning and maintaining 1047 security credentials such as public and private keys? Also, should 1048 private keys be allowed to be replicated across the NFV for 1049 redundancy reasons? 1051 On a positive note, NFV will allow better defense against Denial of 1052 Service (DoS) attacks because of the distributed nature of the 1053 network (i.e. no single point of failure) and the ability to steer 1054 (undesirable) traffic quickly [etsi_gs_nfv_sec_001]. Also, NFVs 1055 which have physical HW which is distributed across multiple data 1056 centers will also provide better fault isolation environments. 1057 Especially, if each data center is protected separately via fire 1058 walls, DMZs and other network protection techniques. 1060 4.9. Separation of control concerns 1062 NFV environments offer two possible levels of SDN control. One level 1063 is the need for controlling the NFVI to provide connectivity end-to- 1064 end among VNFs or among VNFs and PNFs (Physical Network Functions). 1065 A second level is the control and configuration of the VNFs 1066 themselves (in other words, the configuration of the network service 1067 implemented by those VNFs), taking profit of the programmability 1068 brought by SDN. Both control concerns are separated in nature. 1069 However, interaction between both could be expected in order to 1070 optimize, scale or influence each other. 1072 Clear mechanisms for such interaction are needed in order to avoid 1073 mal-functioning or interference among concerns. These ideas are 1074 considered in [etsi_gs_nfv_eve005] and [I-D.irtf-sdnrg-layered-sdn] 1076 4.10. Testing 1078 The impacts of network virtualization on testing can be divided into 1079 3 groups: 1081 1. Changes in methodology. 1083 2. New functionality. 1085 3. Opportunities. 1087 4.10.1. Changes in methodology 1089 The largest impact of NFV is the ability to isolate the System Under 1090 Test (SUT). When testing Physical Network Functions (PNF), isolating 1091 the SUT means that all the other devices that the SUT communicates 1092 with are replaced with simulations (or controlled executions) in 1093 order to place the SUT under test by itself. The SUT may be 1094 comprised of one or more devices. The simulations use the 1095 appropriate traffic type and protocols in order to execute test 1096 cases. See Figure 5. 1098 +--------+ +-----------+ +--------+ 1099 | | | | | | 1100 | Sim A | | SUT | | Sim B | 1101 | +------+ +-----+ | 1102 | | | | | | 1103 +--------+ +-----------+ +--------+ 1105 Figure 5: Testing methodology 1107 As shown in Figure 2, NFV provides a common architecture for all 1108 functions to use. A VNF is executed using resources offered by the 1109 NFVI, which have been allocated using the MANO function. It is not 1110 possible to test a VNF by itself, without the entire supporting 1111 environment present. This fundamentally changes how to consider the 1112 SUT. In the case of a VNF (or multiple VNFs), the SUT is part of a 1113 larger architecture which is necessary in order to run the SUTs. 1115 Isolation of the SUT therefore becomes controlling the environment in 1116 a disciplined manner. The components of the environment necessary to 1117 run the SUTs that are not part of the SUT become the test 1118 environment. In the case of VNFs which are the SUT, then the NFVI 1119 and MANO become the test environment. The configurations and 1120 policies that guide the test environment should remain constant 1121 during the execution of the tests, and also from test to test. 1122 Configurations such as CPU pinning, NUMA configuration, the SW 1123 versions and configurations of the hypervisor, vSwitch and NICs 1124 should remain constant. The only variables in the testing should be 1125 those controlling the SUT itself. If any configuration in the test 1126 environment is changed from test to test, then the results become 1127 very difficult, if not impossible, to compare since the test 1128 environment behavior may change the results as a consequence of the 1129 configuration change. 1131 Testing the NFVI itself also presents new considerations. With a 1132 PNF, the dedicated hardware supporting it is optimized for the 1133 particular workload of the function. Routing hardware is specially 1134 built to support packet forwarding functions, while the hardware to 1135 support a purely control plane application (say, a DNS server, or a 1136 Diameter function) will not have this specialized capability. In 1137 NFV, the NFVI is required to support all types of potentially 1138 different workload types. 1140 Testing the NFVI therefore requires careful consideration to what 1141 types of metrics are sought. This, in turn, depends on the workload 1142 type the expected VNF will be. Examples of different workload types 1143 are data forwarding, control plane, encryption, and authentication. 1144 All these types of expected workloads will determine the types of 1145 metrics that should be sought. For example, if the workload is 1146 control plane, then a metric such as jitter is not useful, but 1147 dropped packets is critical. In a multi-tenant environment, then the 1148 NFVI could support various types of workloads. In this case, testing 1149 with a variety of traffic types while measuring the corresponding 1150 metrics simultaneously becomes necessary. 1152 4.10.2. New functionality 1154 NFV presents a collection of new functionality in order to support 1155 the goal of software networking. Each component on the architecture 1156 shown in Figure 2 has an associated set of functionality that allows 1157 VNFs to run: onboarding, lifecycle management for VNFs and Networks 1158 Services (NS), resource allocation, hypervisor functions, etc. 1160 One of the new capabilities enabled by NFV is VNFFG (VNF Forwarding 1161 Graphs). This refers to the graph that represents a Network Service 1162 by chaining together VNFs into a forwarding path. In practice, the 1163 forwarding path can be implemented in a variety of ways using 1164 different networking capabilities: vSwitch, SDN, SDN with a 1165 northbound application, and the VNFFG might use tunneling protocols 1166 like VXLAN. The dynamic allocation and implementation of these 1167 networking paths will have different performance characteristics 1168 depending on the methods used. The path implementation mechanism 1169 becomes a variable in the network testing of the NSs. The 1170 methodology used to test the various mechanisms should largely remain 1171 the same, and as usual, the test environment should remain constant 1172 for each of the tests, focusing on varying the path establishment 1173 method. 1175 Scaling refers to the change in allocation of resources to a VNF or 1176 NS. It happens dynamically at run-time, based on defined policies 1177 and triggers. The triggers can be network, compute or storage based. 1178 Scaling can allocate more resources in times of need, or reduce the 1179 amount of resources allocated when the demand is reduced. The SUT in 1180 this case becomes much larger than the VNF itself: MANO controls how 1181 scaling is done based on policies, and then allocates the resources 1182 accordingly in the NFVI. Essentially, the testing of scaling 1183 includes the entire NFV architecture components into the SUT. 1185 4.10.3. Opportunities 1187 Softwarization of networking functionality leads to softwarization of 1188 test as well. As Physical Network Functions (PNF) are being 1189 transformed into VNFs, so have the test tools. This leads to the 1190 fact that test tools are also being controlled and executed in the 1191 same environment as the VNFs are. This presents an opportunity to 1192 include VNF-based test tools along with the deployment of the VNFs 1193 supporting the services of the service provider into the host data 1194 centers. Tests can therefore be automatically executed upon 1195 deployment in the target environment, for each deployment, and each 1196 service. With PNFs, this was very difficult to achieve. 1198 This new concept helps to enable modern concepts like DevOps and CI/ 1199 CD in the NFV environment. Simplistically, DevOps is a process that 1200 combines multiple functions into single cohesive teams in order to 1201 quickly produce quality software. It typically relies on also 1202 applying the Agile development process, which focusses on (among many 1203 things) dividing large features into multiple, smaller deliveries. 1204 One part of this is to immediately test the new smaller features in 1205 order to get immediate feedback on errors so that if present, they 1206 can be immediately fixed and redeployed. The CI/CD (Continuous 1207 Integration and Continuous Deployment) pipeline supports this 1208 concept. It consists of a series of tools, among which immediate 1209 testing is an integral part, to deliver software from source to 1210 deployment. The ability to deploy the test tools themselves into the 1211 production environment stretches the CI/CD pipeline all the way to 1212 production deployment, allowing a range of tests to be executed. The 1213 tests can be simple, with a goal of verifying the correct deployment 1214 and networking establishment, to the more complex like testing VNF 1215 functionality. 1217 5. Technology Gaps and Potential IETF Efforts 1219 Table 1 correlates the open network virtualization research areas 1220 identified in this document to potential IETF groups that could 1221 address some aspects of them. An example of a specific gap that the 1222 group could potentially address is identified in parenthetical beside 1223 the group name. 1225 +----------------------------------+--------------------------------+ 1226 | Open Research Area | Potential IETF/IRTF Group | 1227 +----------------------------------+--------------------------------+ 1228 | 1-Guaranteeing QoS | IPPM WG (Measurements of NFVI) | 1229 | 2-Performance improvement | VNFPOOL BoF (NFV resilience) | 1230 | 3-Multiple Domains | NFVRG | 1231 | 4-Network Slicing | NVO3 WG, NETSLICES bar BoF | 1232 | 5-Service Composition | SFC WG (SFC Mgmt and Config) | 1233 | 6-End-user device virtualization | N/A | 1234 | 7-Security | N/A | 1235 | 8-Separation of control concerns | NFVRG | 1236 +----------------------------------+--------------------------------+ 1238 Table 1: Mapping of Open Research Areas to Potential IETF Groups 1240 6. Mapping to NFVRG Near-Term work items 1242 Table 2 correlates the currently identified NFVRG near-work items to 1243 the open network virtualization research areas enumerated in this 1244 document. This can help the NFVRG in identifying and prioritizing 1245 research topics. 1247 +--------------------------------------+-------------------------+ 1248 | NFVRG Near-Term work item | Open Research Area | 1249 +--------------------------------------+-------------------------+ 1250 | 1-Policy-based resource management | - Performance improvem. | 1251 | | - Network Slicing | 1252 | 2-Analytics for visibility & orches. | - Guaranteeing QoS | 1253 | 3-Security and service verification | - Security | 1254 | 4-Reliability and fault detection | - Guaranteeing QoS | 1255 | 5-Service orchestration & lifecycle | - Multiple Domains | 1256 | | - Network Slicing | 1257 | | - Service Composition | 1258 | 6-Real-time properties | - Guaranteeing QoS | 1259 | | | 1260 | (other) | - End-user device virt. | 1261 | | - Separation of control | 1262 +--------------------------------------+-------------------------+ 1264 Table 2: Mapping of NFVRG Near-Term work items to Open Research Areas 1266 7. IANA Considerations 1268 N/A. 1270 8. Security Considerations 1272 This is an informational document, which therefore does not introduce 1273 any security threat. Research challenges and gaps related to 1274 security and privacy have been included in Section 4.8. 1276 9. Acknowledgments 1278 The authors want to thank Dirk von Hugo, Rafa Marin, Diego Lopez, 1279 Ramki Krishnan, Kostas Pentikousis, Rana Pratap Sircar, Alfred 1280 Morton, Nicolas Kuhn and Saumya Dikshit for their very useful reviews 1281 and comments to the document. Special thanks to Pierre Lynch, who 1282 contributed text for the testing section, and to Pedro Martinez- 1283 Julia, who provided text for the network slicing section. 1285 The work of Carlos J. Bernardos and Luis M. Contreras is partially 1286 supported by the H2020-ICT-2014 project 5GEx (Grant Agreement no. 1287 671636). 1289 10. Informative References 1291 [etsi_gs_nfv_003] 1292 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1293 Terminology for Main Concepts in NFV", ETSI GS NFV 003 1294 V1.2.1 NFV 003, December 2014, 1295 . 1298 [etsi_gs_nfv_eve005] 1299 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1300 Ecosystem; Report on SDN Usage in NFV Architectural 1301 Framework", ETSI GS NFV-EVE 005 V1.1.1 NFV-EVE 005, 1302 December 2015, . 1305 [etsi_gs_nfv_per_001] 1306 ETSI NFV ISG, "Network Functions Virtualisation (NFV); NFV 1307 Performance & Portability Best Practises", ETSI GS NFV-PER 1308 001 V1.1.2 NFV-PER 001, December 2014, 1309 . 1312 [etsi_gs_nfv_sec_001] 1313 ETSI NFV ISG, "Network Functions Virtualisation (NFV); NFV 1314 Security; Problem Statement", ETSI GS NFV-SEC 001 V1.1.1 1315 NFV-SEC 001, October 2014, 1316 . 1319 [etsi_nvf_whitepaper_2] 1320 "Network Functions Virtualisation (NFV). White Paper 2", 1321 October 2013. 1323 [etsi_nvf_whitepaper_3] 1324 "Network Functions Virtualisation (NFV). White Paper 3", 1325 October 2014. 1327 [google_sdn_wan] 1328 "B4: experience with a globally-deployed Software Defined 1329 WAN", Proceedings of the ACM SIGCOMM 2013 , August 2013. 1331 [I-D.bagnulo-nfvrg-topology] 1332 Bagnulo, M. and D. Dolson, "NFVI PoP Network Topology: 1333 Problem Statement", draft-bagnulo-nfvrg-topology-01 (work 1334 in progress), March 2016. 1336 [I-D.bernardos-nfvrg-multidomain] 1337 Bernardos, C., Contreras, L., and I. Vaishnavi, "Multi- 1338 domain Network Virtualization", draft-bernardos-nfvrg- 1339 multidomain-01 (work in progress), October 2016. 1341 [I-D.defoy-netslices-3gpp-network-slicing] 1342 Foy, X. and A. Rahman, "Network Slicing - 3GPP Use Case", 1343 draft-defoy-netslices-3gpp-network-slicing-00 (work in 1344 progress), March 2017. 1346 [I-D.gdmb-netslices-intro-and-ps] 1347 Galis, A., Dong, J., kiran.makhijani@huawei.com, k., 1348 Bryant, S., Boucadair, M., and P. Martinez-Julia, "Network 1349 Slicing - Introductory Document and Revised Problem 1350 Statement", draft-gdmb-netslices-intro-and-ps-02 (work in 1351 progress), February 2017. 1353 [I-D.ietf-bmwg-virtual-net] 1354 Morton, A., "Considerations for Benchmarking Virtual 1355 Network Functions and Their Infrastructure", draft-ietf- 1356 bmwg-virtual-net-04 (work in progress), August 2016. 1358 [I-D.irtf-sdnrg-layered-sdn] 1359 Contreras, L., Bernardos, C., Lopez, D., Boucadair, M., 1360 and P. Iovanna, "Cooperating Layered Architecture for 1361 SDN", draft-irtf-sdnrg-layered-sdn-01 (work in progress), 1362 October 2016. 1364 [I-D.mlk-nfvrg-nfv-reliability-using-cots] 1365 Mo, L. and B. Khasnabish, "NFV Reliability using COTS 1366 Hardware", draft-mlk-nfvrg-nfv-reliability-using-cots-01 1367 (work in progress), October 2015. 1369 [I-D.natarajan-nfvrg-containers-for-nfv] 1370 natarajan.sriram@gmail.com, n., Krishnan, R., Ghanwani, 1371 A., Krishnaswamy, D., Willis, P., Chaudhary, A., and F. 1372 Huici, "An Analysis of Lightweight Virtualization 1373 Technologies for NFV", draft-natarajan-nfvrg-containers- 1374 for-nfv-03 (work in progress), July 2016. 1376 [I-D.rorosz-nfvrg-vbaas] 1377 Rosa, R., Rothenberg, C., and R. Szabo, "VNF Benchmark-as- 1378 a-Service", draft-rorosz-nfvrg-vbaas-00 (work in 1379 progress), October 2015. 1381 [intel_10_differences_nfv_cloud] 1382 Intel, "Discover the Top 10 Differences Between NFV and 1383 Cloud Environments", November 2015, 1384 . 1387 [nfv_sota_research_challenges] 1388 , , , , , and , "Network Function Virtualization: State- 1389 of-the-art and Research Challenges", IEEE Communications 1390 Surveys & Tutorials Volume: 18, Issue: 1, September 2015. 1392 [ngmn_5G_whitepaper] 1393 "NGMN 5G. White Paper", February 2015. 1395 [omniran] IEEE, "802.1CF Network Reference Model and Functional 1396 Description of IEEE 802 Access Network", 802.1cf, Draft 1397 0.4 802.1cf, February 2017. 1399 [onf_tr_521] 1400 ONF, "SDN Architecture, Issue 1.1", ONF TR-521 TR-521, 1401 February 2016, 1402 . 1406 [openmano_dataplane] 1407 Telefonica I+D, "OpenMANO: The Dataplane Ready Open Source 1408 NFV MANO Stack", March 2015, 1409 . 1412 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 1413 Application Protocol (CoAP)", RFC 7252, 1414 DOI 10.17487/RFC7252, June 2014, 1415 . 1417 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1418 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1419 Defined Networking (SDN): Layers and Architecture 1420 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1421 2015, . 1423 [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for 1424 Service Function Chaining", RFC 7498, 1425 DOI 10.17487/RFC7498, April 2015, 1426 . 1428 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1429 Chaining (SFC) Architecture", RFC 7665, 1430 DOI 10.17487/RFC7665, October 2015, 1431 . 1433 [virtualization_mobile_device] 1434 "Virtualization of Mobile Device User Experience", Patent 1435 US 9.542.062 B2 , January 2017. 1437 [vnf_benchmarking] 1438 FEEC/UNICAMP, FEEC/UNICAMP, and Ericsson, "A VNF Testing 1439 Framework Design, Implementation and Partial Results", 1440 November 2016, 1441 . 1444 Authors' Addresses 1446 Carlos J. Bernardos 1447 Universidad Carlos III de Madrid 1448 Av. Universidad, 30 1449 Leganes, Madrid 28911 1450 Spain 1452 Phone: +34 91624 6236 1453 Email: cjbc@it.uc3m.es 1454 URI: http://www.it.uc3m.es/cjbc/ 1456 Akbar Rahman 1457 InterDigital Communications, LLC 1458 1000 Sherbrooke Street West, 10th floor 1459 Montreal, Quebec H3A 3G4 1460 Canada 1462 Email: Akbar.Rahman@InterDigital.com 1463 URI: http://www.InterDigital.com/ 1465 Juan Carlos Zuniga 1466 SIGFOX 1467 425 rue Jean Rostand 1468 Labege 31670 1469 France 1471 Email: j.c.zuniga@ieee.org 1472 URI: http://www.sigfox.com/ 1473 Luis M. Contreras 1474 Telefonica I+D 1475 Ronda de la Comunicacion, S/N 1476 Madrid 28050 1477 Spain 1479 Email: luismiguel.contrerasmurillo@telefonica.com 1481 Pedro Aranda 1482 Universidad Carlos III de Madrid 1483 Av. Universidad, 30 1484 Leganes, Madrid 28911 1485 Spain 1487 Email: pedroandres.aranda@uc3m.es