idnits 2.17.1 draft-irtf-nfvrg-gaps-network-virtualization-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 13, 2017) is 2599 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-05) exists of draft-bernardos-nfvrg-multidomain-01 == Outdated reference: A later version (-02) exists of draft-defoy-netslices-3gpp-network-slicing-00 == Outdated reference: A later version (-05) exists of draft-ietf-bmwg-virtual-net-04 Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFVRG CJ. Bernardos 3 Internet-Draft UC3M 4 Intended status: Informational A. Rahman 5 Expires: September 14, 2017 InterDigital 6 JC. Zuniga 7 SIGFOX 8 LM. Contreras 9 TID 10 P. Aranda 11 UC3M 12 P. Lynch 13 Ixia 14 March 13, 2017 16 Network Virtualization Research Challenges 17 draft-irtf-nfvrg-gaps-network-virtualization-05 19 Abstract 21 This document describes open research challenges for network 22 virtualization. Network virtualization is following a similar path 23 as previously taken by cloud computing. Specifically, Cloud 24 computing popularized migration of computing functions (e.g., 25 applications) and storage from local, dedicated, physical resources 26 to remote virtual functions accessible through the Internet. In a 27 similar manner, network virtualization is encouraging migration of 28 networking functions from dedicated physical hardware nodes to a 29 virtualized pool of resources. However, network virtualization can 30 be considered to be a more complex problem than cloud computing as it 31 not only involves virtualization of computing and storage functions 32 but also involves abstraction of the network itself. This document 33 describes current research challenges in network virtualization 34 including guaranteeing quality-of-service, performance improvement, 35 supporting multiple domains, network slicing, service composition, 36 device virtualization, privacy and security. In addition, some 37 proposals are made for new activities in IETF/IRTF that could address 38 some of these challenges. 40 Status of This Memo 42 This Internet-Draft is submitted in full conformance with the 43 provisions of BCP 78 and BCP 79. 45 Internet-Drafts are working documents of the Internet Engineering 46 Task Force (IETF). Note that other groups may also distribute 47 working documents as Internet-Drafts. The list of current Internet- 48 Drafts is at http://datatracker.ietf.org/drafts/current/. 50 Internet-Drafts are draft documents valid for a maximum of six months 51 and may be updated, replaced, or obsoleted by other documents at any 52 time. It is inappropriate to use Internet-Drafts as reference 53 material or to cite them other than as "work in progress." 55 This Internet-Draft will expire on September 14, 2017. 57 Copyright Notice 59 Copyright (c) 2017 IETF Trust and the persons identified as the 60 document authors. All rights reserved. 62 This document is subject to BCP 78 and the IETF Trust's Legal 63 Provisions Relating to IETF Documents 64 (http://trustee.ietf.org/license-info) in effect on the date of 65 publication of this document. Please review these documents 66 carefully, as they describe your rights and restrictions with respect 67 to this document. Code Components extracted from this document must 68 include Simplified BSD License text as described in Section 4.e of 69 the Trust Legal Provisions and are provided without warranty as 70 described in the Simplified BSD License. 72 Table of Contents 74 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 75 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 76 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 77 3.1. Network Function Virtualization . . . . . . . . . . . . . 5 78 3.2. Software Defined Networking . . . . . . . . . . . . . . . 7 79 3.3. Mobile Edge Computing . . . . . . . . . . . . . . . . . . 11 80 3.4. IEEE 802.1CF (OmniRAN) . . . . . . . . . . . . . . . . . 12 81 3.5. Distributed Management Task Force . . . . . . . . . . . . 12 82 3.6. Open Source initiatives . . . . . . . . . . . . . . . . . 12 83 3.7. Internet of Things (IoT) . . . . . . . . . . . . . . . . 14 84 4. Network Virtualization Challenges . . . . . . . . . . . . . . 14 85 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 14 86 4.2. Guaranteeing quality-of-service . . . . . . . . . . . . . 15 87 4.2.1. Virtualization Technologies . . . . . . . . . . . . . 15 88 4.2.2. Metrics for NFV characterization . . . . . . . . . . 15 89 4.2.3. Predictive analysis . . . . . . . . . . . . . . . . . 16 90 4.2.4. Portability . . . . . . . . . . . . . . . . . . . . . 17 91 4.3. Performance improvement . . . . . . . . . . . . . . . . . 17 92 4.3.1. Energy Efficiency . . . . . . . . . . . . . . . . . . 17 93 4.3.2. Improved link usage . . . . . . . . . . . . . . . . . 18 94 4.4. Multiple Domains . . . . . . . . . . . . . . . . . . . . 18 95 4.5. 5G and Network Slicing . . . . . . . . . . . . . . . . . 18 96 4.5.1. Virtual Network Operators . . . . . . . . . . . . . . 19 97 4.5.2. Extending Virtual Networks and Systems to the 98 Internet of Things . . . . . . . . . . . . . . . . . 20 99 4.6. Service Composition . . . . . . . . . . . . . . . . . . . 21 100 4.7. End-user device virtualization . . . . . . . . . . . . . 22 101 4.8. Security and Privacy . . . . . . . . . . . . . . . . . . 22 102 4.9. Separation of control concerns . . . . . . . . . . . . . 24 103 4.10. Testing . . . . . . . . . . . . . . . . . . . . . . . . . 24 104 4.10.1. Changes in methodology . . . . . . . . . . . . . . . 24 105 4.10.2. New functionality . . . . . . . . . . . . . . . . . 26 106 4.10.3. Opportunities . . . . . . . . . . . . . . . . . . . 26 107 5. Technology Gaps and Potential IETF Efforts . . . . . . . . . 27 108 6. Mapping to NFVRG Near-Term work items . . . . . . . . . . . . 27 109 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28 110 8. Security Considerations . . . . . . . . . . . . . . . . . . . 28 111 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 28 112 10. Informative References . . . . . . . . . . . . . . . . . . . 28 113 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32 115 1. Introduction 117 The telecommunications sector is experiencing a major revolution that 118 will shape the way networks and services are designed and deployed 119 for the next few decades. In order to cope with continuously 120 increasing demand and cost, network operators are taking lessons from 121 the IT paradigm of cloud computing. This new approach of 122 virtualizing network functions will enable multi-fold advantages by 123 outsourcing communication services from bespoke hardware in the 124 operator's core network to Commercial off-the-shelf (COTS) equipment 125 distributed across datacenters. 127 Some of the network virtualization mechanisms that are being 128 considered include: sharing of network infrastructure to reduce 129 costs, virtualization of core servers running in data centers as a 130 way of supporting their load- aware elastic dimensioning, and dynamic 131 energy policies to reduce the electricity consumption. 133 This document presents research challenges in Network Function 134 Virtualization (NFV) that need to be addressed in order to achieve 135 these goals. The objective of this memo is to document the technical 136 challenges and corresponding current approaches and to expose 137 requirements that should be addressed by future research and 138 standards work. 140 2. Terminology 142 The following terms used in this document are defined by the ETSI NVF 143 ISG [etsi_gs_nfv_003], the ONF [onf_tr_521] and the IETF [RFC7665]: 145 Application Plane - The collection of applications and services 146 that program network behavior. 148 Control Plane (CP) - The collection of functions responsible for 149 controlling one or more network devices. CP instructs network 150 devices with respect to how to process and forward packets. The 151 control plane interacts primarily with the forwarding plane and, 152 to a lesser extent, with the operational plane. 154 Forwarding Plane (FP) - The collection of resources across all 155 network devices responsible for forwarding traffic. 157 Management Plane (MP) - The collection of functions responsible 158 for monitoring, configuring, and maintaining one or more network 159 devices or parts of network devices. The management plane is 160 mostly related to the operational plane (it is related less to the 161 forwarding plane). 163 NFV Infrastructure (NFVI): totality of all hardware and software 164 components which build up the environment in which VNFs are 165 deployed 167 NFV Management and Orchestration (NFV-MANO): functions 168 collectively provided by NFVO, VNFM, and VIM. 170 NFV Orchestrator (NFVO): functional block that manages the Network 171 Service (NS) lifecycle and coordinates the management of NS 172 lifecycle, VNF lifecycle (supported by the VNFM) and NFVI 173 resources (supported by the VIM) to ensure an optimized allocation 174 of the necessary resources and connectivity. 176 OpenFlow protocol (OFP): allowing vendor independent programming 177 of control functions in network nodes. 179 Operational Plane (OP) - The collection of resources responsible 180 for managing the overall operation of individual network devices. 182 Physical Network Function (PNF): Physical implementation of a 183 Network Function in a monolithic realization. 185 Service Function Chain (SFC): for a given service, the abstracted 186 view of the required service functions and the order in which they 187 are to be applied. This is somehow equivalent to the Network 188 Function Forwarding Graph (NF-FG) at ETSI. 190 Service Function Path (SFP): the selection of specific service 191 function instances on specific network nodes to form a service 192 graph through which an SFC is instantiated. 194 Virtualized Infrastructure Manager (VIM): functional block that is 195 responsible for controlling and managing the NFVI compute, storage 196 and network resources, usually within one operator's 197 Infrastructure Domain. 199 Virtualized Network Function (VNF): implementation of a Network 200 Function that can be deployed on a Network Function Virtualization 201 Infrastructure (NFVI). 203 Virtualized Network Function Manager (VNFM): functional block that 204 is responsible for the lifecycle management of VNF. 206 3. Background 208 3.1. Network Function Virtualization 210 The ETSI ISG NFV is a working group which, since 2012, aims to evolve 211 quasi-standard IT virtualization technology to consolidate many 212 network equipment types into industry standard high volume servers, 213 switches, and storage. It enables implementing network functions in 214 software that can run on a range of industry standard server hardware 215 and can be moved to, or loaded in, various locations in the network 216 as required, without the need to install new equipment. To date, 217 ETSI NFV is by far the most accepted NFV reference framework and 218 architectural footprint [etsi_nvf_whitepaper_2]. The ETSI NFV 219 framework architecture framework is composed of three domains 220 (Figure 1): 222 o Virtualized Network Function, running over the NFVI. 224 o NFV Infrastructure (NFVI), including the diversity of physical 225 resources and how these can be virtualized. NFVI supports the 226 execution of the VNFs. 228 o NFV Management and Orchestration, which covers the orchestration 229 and life-cycle management of physical and/or software resources 230 that support the infrastructure virtualization, and the life-cycle 231 management of VNFs. NFV Management and Orchestration focuses on 232 all virtualization specific management tasks necessary in the NFV 233 framework. 235 +-------------------------------------------+ +---------------+ 236 | Virtualized Network Functions (VNFs) | | | 237 | ------- ------- ------- ------- | | | 238 | | | | | | | | | | | | 239 | | VNF | | VNF | | VNF | | VNF | | | | 240 | | | | | | | | | | | | 241 | ------- ------- ------- ------- | | | 242 +-------------------------------------------+ | | 243 | | 244 +-------------------------------------------+ | | 245 | NFV Infrastructure (NFVI) | | NFV | 246 | ----------- ----------- ----------- | | Management | 247 | | Virtual | | Virtual | | Virtual | | | and | 248 | | Compute | | Storage | | Network | | | Orchestration | 249 | ----------- ----------- ----------- | | | 250 | +---------------------------------------+ | | | 251 | | Virtualization Layer | | | | 252 | +---------------------------------------+ | | | 253 | +---------------------------------------+ | | | 254 | | ----------- ----------- ----------- | | | | 255 | | | Compute | | Storage | | Network | | | | | 256 | | ----------- ----------- ----------- | | | | 257 | | Hardware resources | | | | 258 | +---------------------------------------+ | | | 259 +-------------------------------------------+ +---------------+ 261 Figure 1: ETSI NFV framework 263 The NFV architectural framework identifies functional blocks and the 264 main reference points between such blocks. Some of these are already 265 present in current deployments, whilst others might be necessary 266 additions in order to support the virtualization process and 267 consequent operation. The functional blocks are (Figure 2): 269 o Virtualized Network Function (VNF). 271 o Element Management (EM). 273 o NFV Infrastructure, including: Hardware and virtualized resources, 274 and Virtualization Layer. 276 o Virtualized Infrastructure Manager(s) (VIM). 278 o NFV Orchestrator. 280 o VNF Manager(s). 282 o Service, VNF and Infrastructure Description. 284 o Operations and Business Support Systems (OSS/BSS). 286 +--------------------+ 287 +-------------------------------------------+ | ---------------- | 288 | OSS/BSS | | | NFV | | 289 +-------------------------------------------+ | | Orchestrator +-- | 290 | ---+------------ | | 291 +-------------------------------------------+ | | | | 292 | --------- --------- --------- | | | | | 293 | | EM 1 | | EM 2 | | EM 3 | | | | | | 294 | ----+---- ----+---- ----+---- | | ---+---------- | | 295 | | | | |--|-| VNF | | | 296 | ----+---- ----+---- ----+---- | | | manager(s) | | | 297 | | VNF 1 | | VNF 2 | | VNF 3 | | | ---+---------- | | 298 | ----+---- ----+---- ----+---- | | | | | 299 +------|-------------|-------------|--------+ | | | | 300 | | | | | | | 301 +------+-------------+-------------+--------+ | | | | 302 | NFV Infrastructure (NFVI) | | | | | 303 | ----------- ----------- ----------- | | | | | 304 | | Virtual | | Virtual | | Virtual | | | | | | 305 | | Compute | | Storage | | Network | | | | | | 306 | ----------- ----------- ----------- | | ---+------ | | 307 | +---------------------------------------+ | | | | | | 308 | | Virtualization Layer | |--|-| VIM(s) +-------- | 309 | +---------------------------------------+ | | | | | 310 | +---------------------------------------+ | | ---------- | 311 | | ----------- ----------- ----------- | | | | 312 | | | Compute | | Storage | | Network | | | | | 313 | | | hardware| | hardware| | hardware| | | | | 314 | | ----------- ----------- ----------- | | | | 315 | | Hardware resources | | | NFV Management | 316 | +---------------------------------------+ | | and Orchestration | 317 +-------------------------------------------+ +--------------------+ 319 Figure 2: ETSI NFV reference architecture 321 3.2. Software Defined Networking 323 The Software Defined Networking (SDN) paradigm pushes the 324 intelligence currently residing in the network elements to a central 325 controller implementing the network functionality through software. 326 In contrast to traditional approaches, in which the network's control 327 plane is distributed throughout all network devices, with SDN the 328 control plane is logically centralized. In this way, the deployment 329 of new characteristics in the network no longer requires of complex 330 and costly changes in equipment or firmware updates, but only a 331 change in the software running in the controller. The main advantage 332 of this approach is the flexibility it provides operators with to 333 manage their network, i.e., an operator can easily change its 334 policies on how traffic is distributed throughout the network. 336 The most visible of the SDN protocol stacks is the OpenFlow protocol 337 (OFP), which is maintained and extended by the Open Network 338 Foundation (ONF: https://www.opennetworking.org/). Originally this 339 protocol was developed specifically for IEEE 802.1 switches 340 conforming to the ONF OpenFlow Switch specification. As the benefits 341 of the SDN paradigm have reached a wider audience, its application 342 has been extended to more complex scenarios such as Wireless and 343 Mobile networks. Within this area of work, the ONF is actively 344 developing new OFP extensions addressing three key scenarios: (i) 345 Wireless backhaul, (ii) Cellular Evolved Packet Core (EPC), and (iii) 346 Unified access and management across enterprise wireless and fixed 347 networks. 349 +----------+ 350 | ------- | 351 | |Oper.| | O 352 | |Mgmt.| |<........> -+- Network Operator 353 | |Iface| | ^ 354 | ------- | +----------------------------------------+ 355 | | | +------------------------------------+ | 356 | | | | --------- --------- --------- | | 357 |--------- | | | | App 1 | | App 2 | ... | App n | | | 358 ||Plugins| |<....>| | --------- --------- --------- | | 359 |--------- | | | Plugins | | 360 | | | +------------------------------------+ | 361 | | | Application Plane | 362 | | +----------------------------------------+ 363 | | A 364 | | | 365 | | V 366 | | +----------------------------------------+ 367 | | | +------------------------------------+ | 368 |--------- | | | ------------ ------------ | | 369 || Netw. | | | | | Module 1 | | Module 2 | | | 370 ||Engine | |<....>| | ------------ ------------ | | 371 |--------- | | | Network Engine | | 372 | | | +------------------------------------+ | 373 | | | Controller Plane | 374 | | +----------------------------------------+ 375 | | A 376 | | | 377 | | V 378 | | +----------------------------------------+ 379 | | | +--------------+ +--------------+ | 380 | | | | ------------ | | ------------ | | 381 |----------| | | | OpenFlow | | | | OpenFlow | | | 382 ||OpenFlow||<....>| | ------------ | | ------------ | | 383 |----------| | | NE | | NE | | 384 | | | +--------------+ +--------------+ | 385 | | | Data Plane | 386 |Management| +----------------------------------------+ 387 +----------+ 389 Figure 3: High level SDN ONF architecture 391 Figure 3 shows the blocks and the functional interfaces of the ONF 392 architecture, which comprises three planes: Data, Controller, and 393 Application. The Data plane comprehends several Network Entities 394 (NE), which expose their capabilities toward the Controller plane via 395 a Southbound API. The Controller plane includes several cooperating 396 modules devoted to the creation and maintenance of an abstracted 397 resource model of the underneath network. Such model is exposed to 398 the applications via a Northbound API where the Application plane 399 comprises several applications/services, each of which has exclusive 400 control of a set of exposed resources. 402 The Management plane spans its functionality across all planes 403 performing the initial configuration of the network elements in the 404 Data plane, the assignment of the SDN controller and the resources 405 under its responsibility. In the Controller plane, the Management 406 needs to configure the policies defining the scope of the control 407 given to the SDN applications, to monitor the performance of the 408 system, and to configure the parameters required by the SDN 409 controller modules. In the Application plane, Management configures 410 the parameters of the applications and the service level agreements. 411 In addition to the these interactions, the Management plane exposes 412 several functions to network operators which can easily and quickly 413 configure and tune the network at each layer. 415 The IRTF Software-Defined Networking Research Group (SDNRG) 416 documented in RFC7426 [RFC7426], a layer model of an SDN 417 architecture, since this has been a controvertial discussion topic: 418 what is exactly SDN? what is the layer structure of the SDN 419 architecture? how do layers interface with each oter? etc. 421 Figure 4 reproduces the figure included in RFC7426 [RFC7426] to 422 summarize the SDN architecture abstractions in the form of a 423 detailed, high-level schematic. In a particular implementation, 424 planes can be collocated with other planes or can be physically 425 separated. 427 In SDN, a controller manipulates controlled entities via an 428 interface.Interfaces, when local, are mostly API invocations through 429 some library or system call. However, such interfaces may be 430 extended via some protocol definition, which may use local inter- 431 process communication (IPC) or a protocol that could also act 432 remotely; the protocol may be defined as an open standard or in a 433 proprietary manner. 435 SDN expands multiple planes: Forwarding, Operational, Control, 436 Management and Applications. All planes mentioned above are 437 connected via interfaces. Additionally, RFC7426 [RFC7426] considers 438 four abstraction layers: the Device and resource Abstraction Layer 439 (DAL), the Control Abstraction Layer (CAL), the Management 440 Abstraction Layer (MAL) and the Network Services Abstraction Layer 441 (NSAL). 443 o--------------------------------o 444 | | 445 | +-------------+ +----------+ | 446 | | Application | | Service | | 447 | +-------------+ +----------+ | 448 | Application Plane | 449 o---------------Y----------------o 450 | 451 *-----------------------------Y---------------------------------* 452 | Network Services Abstraction Layer (NSAL) | 453 *------Y------------------------------------------------Y-------* 454 | | 455 | Service Interface | 456 | | 457 o------Y------------------o o---------------------Y------o 458 | | Control Plane | | Management Plane | | 459 | +----Y----+ +-----+ | | +-----+ +----Y----+ | 460 | | Service | | App | | | | App | | Service | | 461 | +----Y----+ +--Y--+ | | +--Y--+ +----Y----+ | 462 | | | | | | | | 463 | *----Y-----------Y----* | | *---Y---------------Y----* | 464 | | Control Abstraction | | | | Management Abstraction | | 465 | | Layer (CAL) | | | | Layer (MAL) | | 466 | *----------Y----------* | | *----------Y-------------* | 467 | | | | | | 468 o------------|------------o o------------|---------------o 469 | | 470 | CP | MP 471 | Southbound | Southbound 472 | Interface | Interface 473 | | 474 *------------Y---------------------------------Y----------------* 475 | Device and resource Abstraction Layer (DAL) | 476 *------------Y---------------------------------Y----------------* 477 | | | | 478 | o-------Y----------o +-----+ o--------Y----------o | 479 | | Forwarding Plane | | App | | Operational Plane | | 480 | o------------------o +-----+ o-------------------o | 481 | Network Device | 482 +---------------------------------------------------------------+ 484 Figure 4: SDN Layer Architecture 486 3.3. Mobile Edge Computing 488 Mobile Edge Computing capabilities deployed in the edge of the mobile 489 network can facilitate the efficient and dynamic provision of 490 services to mobile users. The ETSI ISG MEC working group, operative 491 from end of 2014, intends to specify an open environment for 492 integrating MEC capabilities with service providers networks, 493 including also applications from 3rd parties. These distributed 494 computing capabilities will make available IT infrastructure as in a 495 cloud environment for the deployment of functions in mobile access 496 networks. It can be seen then as a complement to both NFV and SDN. 498 3.4. IEEE 802.1CF (OmniRAN) 500 The IEEE 802.1CF Recommended Practice [omniran] specifies an access 501 network, which connects terminals to their access routers, utilizing 502 technologies based on the family of IEEE 802 Standards (e.g., 802.3 503 Ethernet, 802.11 Wi-Fi, etc.). The specification defines an access 504 network reference model, including entities and reference points 505 along with behavioral and functional descriptions of communications 506 among those entities. 508 The goal of this project is to help unifying the support of different 509 interfaces, enabling shared network control and use of software 510 defined network (SDN) principles, thereby lowering the barriers to 511 new network technologies, to new network operators, and to new 512 service providers. 514 3.5. Distributed Management Task Force 516 The DMTF is an industry standards organization working to simplify 517 the manageability of network-accessible technologies through open and 518 collaborative efforts by some technology companies. The DMTF is 519 involved in the creation and adoption of interoperable management 520 standards, supporting implementations that enable the management of 521 diverse traditional and emerging technologies including cloud, 522 virtualization, network and infrastructure. 524 There are several DMTF initiatives that are relevant to the network 525 virtualization area, such as the Open Virtualization Format (OVF), 526 for VNF packaging; the Cloud Infrastructure Management Interface 527 (CIM), for cloud infrastructure management; the Network Management 528 (NETMAN), for VNF management; and, the Virtualization Management 529 (VMAN), for virtualization infrastructure management. 531 3.6. Open Source initiatives 533 The Open Source community is especially active in the area of network 534 virtualization. We next summarize some of the active efforts: 536 o OpenStack. OpenStack is a free and open-source cloud-computing 537 software platform. OpenStack software controls large pools of 538 compute, storage, and networking resources throughout a 539 datacenter, managed through a dashboard or via the OpenStack API. 541 o OpenDayLight. OpenDaylight (ODL) is a highly available, modular, 542 extensible, scalable and multi-protocol controller infrastructure 543 built for SDN deployments on modern heterogeneous multi-vendor 544 networks. It provides a model-driven service abstraction platform 545 that allows users to write apps that easily work across a wide 546 variety of hardware and southbound protocols. 548 o ONOS. The ONOS (Open Network Operating System) project is an open 549 source community hosted by The Linux Foundation. The goal of the 550 project is to create a software-defined networking (SDN) operating 551 system for communications service providers that is designed for 552 scalability, high performance and high availability. 554 o OpenContrail. OpenContrail is an Apache 2.0-licensed project that 555 is built using standards-based protocols and provides all the 556 necessary components for network virtualization-SDN controller, 557 virtual router, analytics engine, and published northbound APIs. 558 It has an extensive REST API to configure and gather operational 559 and analytics data from the system. 561 o OPNFV. OPNFV is a carrier-grade, integrated, open source platform 562 to accelerate the introduction of new NFV products and services. 563 By integrating components from upstream projects, the OPNFV 564 community aims at conducting performance and use case-based 565 testing to ensure the platform's suitability for NFV use cases. 566 The scope of OPNFV's initial release is focused on building NFV 567 Infrastructure (NFVI) and Virtualized Infrastructure Management 568 (VIM) by integrating components from upstream projects such as 569 OpenDaylight, OpenStack, Ceph Storage, KVM, Open vSwitch, and 570 Linux. These components, along with application programmable 571 interfaces (APIs) to other NFV elements form the basic 572 infrastructure required for Virtualized Network Functions (VNF) 573 and Management and Network Orchestration (MANO) components. 574 OPNFV's goal is to increase performance and power efficiency; 575 improve reliability, availability, and serviceability; and deliver 576 comprehensive platform instrumentation. 578 o OSM. Open Source Mano (OSM) is an ETSI-hosted project to develop 579 an Open Source NFV Management and Orchestration (MANO) software 580 stack aligned with ETSI NFV. OSM is based on components from 581 previous projects, such Telefonica's OpenMANO or Canonical's Juju, 582 among others. 584 o OpenBaton. OpenBaton is a ETSI NFV compliant Network Function 585 Virtualization Orchestrator (NFVO). OpenBaton was part of the 586 OpenSDNCore project started with the objective of providing a 587 compliant implementation of the ETSI NFV specification. 589 Among the main areas that are being developed by the former open 590 source activities that related to network virtualization research, we 591 can highlight: policy-based resource management, analytics for 592 visibility and orchestration, service verification with regards to 593 security and resiliency. 595 3.7. Internet of Things (IoT) 597 The Internet of Things (IoT) refers to the vision of connecting a 598 multitude of automated devices (e.g. lights, environmental sensors, 599 traffic lights, parking meters, health and security systems, etc.) to 600 the Internet for purposes of reporting, and remote command and 601 control of the device. This vision is being realized by a multi- 602 pronged approach of standardization in various forums and 603 complementary open source activities. For example, in IETF, support 604 of IoT web services has been defined by an HTTP-like protocol adapted 605 for IoT called CoAP [RFC7252], and lately a group has been studying 606 the need to develop a new network layer to support IP applications 607 over Low Power Wide Area Networks (LPWAN). 609 Elsewhere, for 5G cellular evolution there is much discussion on the 610 need for supporting virtual "network slices" for the expected massive 611 numbers of IoT devices. A separate virtual network slice is 612 considered necessary for different 5G IoT use cases because devices 613 will have very different characteristics than typical cellular 614 devices like smart phones [ngmn_5G_whitepaper], and the number of IoT 615 devices is expected to be at least one or two orders of magnitude 616 higher than other 5G devices. 618 4. Network Virtualization Challenges 620 4.1. Introduction 622 Network Virtualization is changing the way the telecommunications 623 sector will deploy, extend and operate their networks. These new 624 technologies aim at reducing the overall costs by outsourcing 625 communication services from specific hardware in the operators' core 626 to server farms scattered in datacenters (i.e. compute and storage 627 virtualization). In addition, the connecting networks are 628 fundamentally affected in the way they route, process and control 629 traffic (i.e. network virtualization). 631 4.2. Guaranteeing quality-of-service 633 Guaranteeing a given quality-of-service in an NFV environment is not 634 an easy task. For example, ensuring a guaranteed and stable 635 forwarding data rate has proven not to be straightforward when the 636 forwarding function is virtualized and runs on top of COTS server 637 hardware [openmano_dataplane] 638 [I-D.mlk-nfvrg-nfv-reliability-using-cots] [etsi_nvf_whitepaper_3]. 639 We next identify some of the challenges that this poses. 641 4.2.1. Virtualization Technologies 643 The issue of guaranteeing a network quality-of-service is less of an 644 issue for "traditional cloud computing" because the workloads that 645 are treated there are servers or clients in the networking sense and 646 hardly ever process packets. Cloud computing provides hosting for 647 applications on shared servers in a highly separated way. Its main 648 advantage is that the infrastructure costs are shared among tenants 649 and that the Cloud infrastructure provides levels of reliability that 650 can not be achieved on individual premises in a cost-efficient way 651 [intel_10_differences_nfv_cloud]. NFV poses very strict requirements 652 posed in terms of performance, stability and consistency. Although 653 there are some tools and mechanisms to improve this, such as Enhanced 654 Performance Awareness (EPA), SR-IOV, NUMA, DPDK, etc, these are still 655 unsolved challenges. One open research issue is finding out 656 technologies that are different from VM and more suitable for dealing 657 with network functionalities. 659 Lately, a number of light-weight virtualization technologies 660 including containers, unikernels (specialized VMs) and minimalistic 661 distributions of general-purpose OSes have appeared as virtualization 662 approaches that can be used when constructing an NFV platform. 663 [I-D.natarajan-nfvrg-containers-for-nfv] describes the challenges in 664 building such a platform and discusses to what extent these 665 technologies, as well as traditional VMs, are able to address them. 667 4.2.2. Metrics for NFV characterization 669 Another relevant aspect is the need for tools for diagnostics and 670 measurement suited for NFV. There is a pressing need to define 671 metrics and associated protocols to measure the performance of NFV. 672 Specifically, since NFV is based on the concept of taking centralized 673 functions and evolving it to highly distributed SW functions, there 674 is a commensurate need to fully understand and measure the baseline 675 performance of such systems. 677 The IP Performance Metrics (IPPM) WG defines metrics that can be used 678 to measure the quality and performance of Internet services and 679 applications running over transport layer protocols (e.g., TCP, UPD) 680 over IP. It also develops and maintains protocols for the 681 measurement of these metrics. While the IPPM WG is a long running WG 682 that started in 1997 it does not have a charter item or active drafts 683 related to the topic of network virtualization. In addition to using 684 IPPM metrics to evaluate the QoS, there is a need for specific 685 metrics for assessing the performance of network virtualization 686 techniques. 688 The Benchmarking Methodology Working Group (BMWG) is also performing 689 work related to NFV metrics. For example, 690 [I-D.ietf-bmwg-virtual-net] investigates additional methodological 691 considerations necessary when benchmarking VNFs instantiated and 692 hosted in general- purpose hardware, using bare-metal hypervisors or 693 other isolation environments such as Linux containers. An essential 694 consideration is benchmarking physical and virtual network functions 695 in the same way when possible, thereby allowing direct comparison. 697 As stated in the document [I-D.ietf-bmwg-virtual-net], there is a 698 clear motivation for the work on performance metrics for NFV 699 [etsi_gs_nfv_per_001], that is worth replicating here: "I'm designing 700 and building my NFV Infrastructure platform. The first steps were 701 easy because I had a small number of categories of VNFs to support 702 and the VNF vendor gave HW recommendations that I followed. Now I 703 need to deploy more VNFs from new vendors, and there are different 704 hardware recommendations. How well will the new VNFs perform on my 705 existing hardware? Which among several new VNFs in a given category 706 are most efficient in terms of capacity they deliver? And, when I 707 operate multiple categories of VNFs (and PNFs) *concurrently* on a 708 hardware platform such that they share resources, what are the new 709 performance limits, and what are the software design choices I can 710 make to optimize my chosen hardware platform? Conversely, what 711 hardware platform upgrades should I pursue to increase the capacity 712 of these concurrently operating VNFs?" 714 Lately, there are also some efforts lately lookinh into VNF 715 benchmarking. The selection of an NFV Infrastructure Point of 716 Presence to host a VNF or allocation of resources (e.g., virtual 717 CPUs, memory) needs to be done over virtualized (abstracted and 718 simplified) resource views [vnf_benchmarking] 719 [I-D.rorosz-nfvrg-vbaas]. 721 4.2.3. Predictive analysis 723 On top of diagnostic tools that enable an assessment of the QoS, 724 predictive analyses are required to react before anomalies occur. 725 Due to the SW characteristics of VNFs, a reliable diagnosis framework 726 could potentially enable the prevention of issues by a proper 727 diagnosis and then a reaction in terms of acting on the potentially 728 impacted service (e.g., migration to a different compute node, 729 scaling in/out, up/down, etc). 731 4.2.4. Portability 733 Portability in NFV refers to the ability to run a given VNF on 734 multiple NFVIs, that is, that it is possible to guarantee that the 735 VNF would be able to perform its functions with a high and 736 predictable performance given that a set of requirements on the NFVI 737 resources is met. Therefore, portability is a key feature that, if 738 fully enabled, would contribute to making the NFV environment achieve 739 a better reliability than a traditional system. The fact of running 740 functionality in SW over "commodity" infrastructure should make much 741 easier to port/move functions from one place to another. However 742 this is not yet as ideal as it sounds and there are aspects not fully 743 tackled. The existence of different hypervisors, specific hardware 744 dependencies (e.g., EPA related) or state synchronization aspects are 745 just some examples of trouble-makers for portability purposes. 747 The ETSI NFV ISG is doing work in relation to portability. 748 [etsi_gs_nfv_per_001] provides a list of minimal features which the 749 VM Descriptor and Compute Host Descriptor should contain for the 750 appropriate deployment of VM Images over an NFVI (i.e. a "telco 751 datacentre"), in order to guarantee high and predictable performance 752 of data plane workloads while assuring their portability. In 753 addition, the document provides a set of recommendations on the 754 minimum requirements which HW and hypervisor should have for a "telco 755 datacentre" suitable for different workloads (data-plane, control- 756 plane, etc.) present in VNFs. The purpose of this document is to 757 provide the list of VM requirements that should be included in the VM 758 Descriptor template, and the list of HW capabilities that should be 759 included in the Compute Host Descriptor (CHD) to assure predictable 760 high performance. ETSI NFV assumes that the MANO Functions will make 761 the mix & match. There are therefore still quite several research 762 challenges to be addressed here. 764 4.3. Performance improvement 766 4.3.1. Energy Efficiency 768 Virtualization is typically seen as a direct enabler of energy 769 savings. Some of the enablers for this that are often mentioned 770 [nfv_sota_research_challenges] are: (i) the multiplexing gains 771 achieved by centralizing functions in data centers reduce overall the 772 energy consumed, (ii) the flexibility brought by network 773 programmability enables to switch off infrastructure as needed in a 774 much easier way. However there is still a lot of room for 775 improvement in terms of virtualization techniques to reduce the power 776 consumption, such as enhanced hypervisor technologies. 778 4.3.2. Improved link usage 780 The use of NFV and SDN technologies can help improving link usage. 781 SDN has shown already that it can greatly increase average link usage 782 (e.g., Google example [google_sdn_wan]). NFV adds more complexity 783 (e.g., due to service function chaining / VNF forwarding drafts) 784 which need to be considered. Aspects like the ones described in 785 [I-D.bagnulo-nfvrg-topology] on NFV data center topology design have 786 to be carefully looked as well. 788 4.4. Multiple Domains 790 Market fragmentation has resulted in a multitude of network operators 791 each focused on different countries and regions. This makes it 792 difficult to create infrastructure services spanning multiple 793 countries, such as virtual connectivity or compute resources, as no 794 single operator has a footprint everywhere. Cross-domain 795 orchestration of services over multiple administrations or over 796 multi-domain single administrations will allow end-to-end network and 797 service elements to mix in multi-vendor, heterogeneous technology and 798 resource environments. 800 For the specific use case of 'Network as a Service', it becomes even 801 more important to ensure, that Cross Domain Orchestration also takes 802 care of hierarchy of networks and their association, with respect to 803 provisioning tunnels and overlays. 805 Multi-domain orchestration is currently an active research topic, 806 which is being tackled, among others, by ETSI NFV ISG and the 5GEx 807 project (https://www.5gex.eu/) [I-D.bernardos-nfvrg-multidomain]. 809 4.5. 5G and Network Slicing 811 From the beginning of all 5G discussions in the research and industry 812 fora, it has been agreed that 5G will have to address much more use 813 cases than the preceding wireless generations, which first focused on 814 voice services, and then on voice and high speed packet data 815 services. In this case, 5G should be able to handle not only the 816 same (or enhanced) voice and packet data services, but also new 817 emerging services like tactile Internet and IoT. These use cases 818 take the requirements to opposite extremes, as some of them require 819 ultra-low latency and higher-speed, whereas some others require 820 ultra-low power consumption and high delay tolerance. 822 Because of these very extreme 5G use cases, it is envisioned that 823 different radio access networks are needed to better address the 824 specific requirements of each one of the use cases. However, on the 825 core network side, virtualization techniques can allow tailoring the 826 network resources on separate slices, specifically for each radio 827 access network and use case, in an efficient manner. 829 Network slicing techniques can also allow dedicating resources for 830 even more specific use cases within the major 5G categories. For 831 example, within the major IoT category, which is perhaps the most 832 disrupting one, some autonomous IoT devices will have very low 833 throughput, will have much longer sleep cycles (and therefore high 834 latency), and a battery life thousands of times longer compared to 835 smart phones or some other connected IoT devices that will have 836 almost continuous control and data communications. Hence, it is 837 envisioned that a single virtual core network could be used by 838 slicing separate resources to dedicated radio access networks (RANs) 839 that are better suited for specific use cases. 841 The actual definition of network slicing is still a sensitive 842 subject, currently under heavy discussion 843 [I-D.gdmb-netslices-intro-and-ps] 844 [I-D.defoy-netslices-3gpp-network-slicing] [ngmn_5G_whitepaper]. 845 Network slicing is a key for introducing new actors in existing 846 market at low cost -- by letting new players rent "blocks" of 847 capacity, if this new market provides performance that are adequate 848 with the application needs (e.g., broadcasting updates to many 849 sensors with satellite broadcasting capabilities). However, more 850 work needs to be done to define how network slicing will impact 851 existing architectures like ETSI NFV, and to define the impacts of 852 network slicing to guaranteeing quality-of-service as described in 853 Section 4.2. 855 4.5.1. Virtual Network Operators 857 The widespread of system and network virtualization technologies has 858 conducted to new business opportunities, enlarging the offer of IT 859 resources with virtual network and computing resources, among others. 860 As a consequence, the network ecosystem now differentiates between 861 the owner of physical resources, the Infrastructure Provider (InP), 862 and the intermediary that conforms and delivers network services to 863 the final customers, the Virtual Network Operator (VNO). 865 VNOs aim to exploit the virtualized infrastructures to deliver new 866 and improved services to their customers. However, current network 867 virtualization techniques offer poor support for VNOs to control 868 their resources. It has been considered that the InP is responsible 869 of the reliability of the virtual resources but there are several 870 situations in which an VNO requires to gain a finer control on its 871 resources. For instance, dynamic events, such as the identification 872 of new requirements or the detection of incidents within the virtual 873 system, might urge a VNO to quickly reform its virtual infrastructure 874 and resource allocation. However, the interfaces offered by current 875 virtualization platforms do not offer the necessary functions for 876 VNOs to perform the elastic adaptations they require to tackle with 877 their dynamic operation environments. 879 Beyond their heterogeneity, which can be resolved by software 880 adapters, current virtualization platforms do not have common methods 881 and functions, so it is difficult for the virtual network controllers 882 used by the VNOs to actually manage and control virtual resources 883 instantiated on different platforms, not even considering different 884 InPs. Therefore it is necessary to reach a common definition of the 885 functions that should be offered by underlying platforms to enable 886 such overlay controllers with the possibility of allocate and 887 deallocate resources dynamically and get monitoring data about them. 889 Such common methods should be offered by all underlying controllers, 890 regardless of being network-oriented (e.g. ODL, ONOS, Ryu) or 891 computing-oriented (e.g. OpenStack, OpenNebula, Eucalyptus). 892 Furthermore, it is also important for those platforms to offer some 893 "PUSH" function to report resource state, avoiding the need for the 894 VNO's controller to "POLL" for such data. A starting point to get 895 proper notifications within current REST APIs could be to consider 896 the protocol proposed by the WEBPUSH WG. 898 Finally, in order to establish a proper order and allow the 899 coexistence and collaboration of different systems, a common ontology 900 regarding network and system virtualization should be defined and 901 agreed, so different and heterogeneous systems can understand each 902 other without requiring to rely on specific adatpation mechanisms 903 that might break with any update on any side of the relation. 905 4.5.2. Extending Virtual Networks and Systems to the Internet of Things 907 The specific nature of the Internet of Things (IoT) ecosystem, 908 particularly reflected in the Machine-to-Machine (M2M) 909 communications, conducts to the creation of new and highly 910 distributed systems which demand location-based network and computing 911 services. An specific example can be represented by a set of 912 "things" that suddenly require to set-up a firewall to allow external 913 entities to access their data while outsourcing some computation 914 requirements to more powerful systems relying on Cloud-based 915 services. This representative use case exposes important 916 requirements for both NFV and the underlying Cloud infrastructures. 918 In order to provide the aforementioned location-based functions 919 integrated with highly distributed systems, the so called FOG 920 infrastructures should be able to instantiate VNFs, placing them in 921 the required place, e.g. close to their consumers. This requirement 922 implies that the interfaces offered by virtualization platforms must 923 support the specification of location-based resources, which is a key 924 function in those scenarios. Moreover, those platforms must also be 925 able to interpret and understand the references used by IoT systems 926 to their location (e.g., "My-AP", "5BLDG+2F") and also the 927 specification of identifiers linked to other resources, such as the 928 case of requiring the infrastructure to establish a link between a 929 specific AP and a specific virtual computing node. 931 4.6. Service Composition 933 Current network services deployed by operators often involve the 934 composition of several individual functions (such as packet 935 filtering, deep packet inspection, load balancing). These services 936 are typically implemented by the ordered combination of a number of 937 service functions that are deployed at different points within a 938 network, not necessary on the direct data path. This requires 939 traffic to be steered through the required service functions, 940 wherever they are deployed [RFC7498]. 942 For a given service, the abstracted view of the required service 943 functions and the order in which they are to be applied is called a 944 Service Function Chain (SFC), which is called Network Function 945 Forwarding Graph (NF-FG) in ETSI. An SFC is instantiated through 946 selection of specific service function instances on specific network 947 nodes to form a service graph: this is called a Service Function Path 948 (SFP). The service functions may be applied at any layer within the 949 network protocol stack (network layer, transport layer, application 950 layer, etc.). 952 Service composition is a powerful tool which can provide significant 953 benefits when applied in a softwarized network environment. There 954 are however many research challenges in this area, as for example the 955 ones related to composition mechanisms and algorithms to enable load 956 balancing and improve reliability. The service composition should 957 also act as an enabler to gather information across all hierarchies 958 (underlays and overlays) of network deployments which may span across 959 multiple operators, for faster serviceability thus facilitating in 960 accomplishing aforementioned goals of "load balancing and improve 961 reliability". 963 The SFC working group is working on an architecture for service 964 function chaining [RFC7665] that includes the necessary protocols or 965 protocol extensions to convey the Service Function Chain and Service 966 Function Path information to nodes that are involved in the 967 implementation of service functions and Service Function Chains, as 968 well as mechanisms for steering traffic through service functions. 970 In terms of actual work items, the SFC WG is has not yet considered 971 working on the management and configuration of SFC components related 972 to the support of Service Function Chaining. This part is of special 973 interest for operators and would be required in order to actually put 974 SFC mechanisms into operation. Similarly, redundancy and reliability 975 mechanisms are currently not dealt with by any WG in the IETF. While 976 this was the main goal of the VNFpool BoF efforts, it still remains 977 unaddressed. 979 4.7. End-user device virtualization 981 So far, most of the network softwarization efforts have focused on 982 virtualizing functions of network elements. While virtualization of 983 network elements started with the core, mobile networks architectures 984 are now heavily switching to also virtualize radio access network 985 (RAN) functions. The next natural step is to get virtualization down 986 at the level of the end-user device (i.e., virtualizing a smartphone) 987 [virtualization_mobile_device]. The cloning of a device in the cloud 988 (central or local) bears attractive benefits to both the device and 989 network operations alike (e.g., power saving at the device by 990 offloading computational-heaving functions to the cloud, optimized 991 networking -- both device-to-device and device-to-infrastructure) for 992 service delivery through tighter integration of the device (via its 993 clone in the networking infrastructure). This is being explored for 994 example by the European H2020 ICIRRUS project (www.icirrus-5gnet.eu). 996 4.8. Security and Privacy 998 Similar to any other situation where resources are shared, security 999 and privacy are two important aspects that need to be taken into 1000 account. 1002 In the case of security, there are situations where multiple vendors 1003 will need to coexist in a virtual or hybrid physical/virtual 1004 environment. This requires attestation procedures amongst different 1005 virtual/physical functions and resources, as well as ongoing external 1006 monitoring. Similarly, different network slices operating on the 1007 same infrastructure can present security problems, for instance if 1008 one slice running critical applications (e.g. support for a safety 1009 system) is affected by another slice running a less critical 1010 application. In general, the minimum common denominator for security 1011 measures on a shared system should be equal or higher than the one 1012 required by the most critical application. Multiple and continuous 1013 threat model analysis, as well as DevOps model are required to 1014 maintain certain level of security in an NFV system. 1016 On the other hand, privacy in its strictest interpretation, refers to 1017 concerns about exposing users of the system to individual threats 1018 such as surveillance, identification, stored data compromise, 1019 secondary use, intrusion, etc. In this case, the storage, 1020 transmission, collection, and potential correlation of information in 1021 the NFV system, for purposes not originally intended or not known by 1022 the user, should be avoided. This is particularly challenging, as 1023 future intentions and threats cannot be easily predicted, and still 1024 can be applied for instance on data collected in the past. 1025 Therefore, well-known techniques such as data minimization, using 1026 privacy features as default, and allowing users to opt in/out should 1027 be used to prevent potential privacy issues. 1029 Compared to traditional networks, NFV will result in networks that 1030 are much more dynamic (in function distribution and topology) and 1031 elastic (in size and boundaries). NFV will thus require network 1032 operators to evolve their operational and administrative security 1033 solutions to work in this new environment. For example, in NFV the 1034 network orchestrator will become a key node to provide security 1035 policy orchestration across the different physical and virtual 1036 components of the virtualized network. For highly confidential data, 1037 for example, the network orchestrator should take into account if 1038 certain physical HW of the network is considered more secure (e.g., 1039 because it is located in secure premises) than other HW. 1041 Traditional telecom networks typically run under a single 1042 administrative domain controlled by an operator. With NFV, it is 1043 expected that in many cases, the telecom operator will now become a 1044 tenant (running the VNFs), and the infrastructure (NFVI) may be run 1045 by a different operator and/or cloud service provider (see also 1046 Section 4.4). Thus, there will be multiple administrative domains 1047 which will make coordination of security policy more complex. For 1048 example, who will be in charge of provisioning and maintaining 1049 security credentials such as public and private keys? Also, should 1050 private keys be allowed to be replicated across the NFV for 1051 redundancy reasons? 1053 On a positive note, NFV will allow better defense against Denial of 1054 Service (DoS) attacks because of the distributed nature of the 1055 network (i.e. no single point of failure) and the ability to steer 1056 (undesirable) traffic quickly [etsi_gs_nfv_sec_001]. Also, NFVs 1057 which have physical HW which is distributed across multiple data 1058 centers will also provide better fault isolation environments. 1059 Especially, if each data center is protected separately via fire 1060 walls, DMZs and other network protection techniques. 1062 4.9. Separation of control concerns 1064 NFV environments offer two possible levels of SDN control. One level 1065 is the need for controlling the NFVI to provide connectivity end-to- 1066 end among VNFs or among VNFs and PNFs (Physical Network Functions). 1067 A second level is the control and configuration of the VNFs 1068 themselves (in other words, the configuration of the network service 1069 implemented by those VNFs), taking profit of the programmability 1070 brought by SDN. Both control concerns are separated in nature. 1071 However, interaction between both could be expected in order to 1072 optimize, scale or influence each other. 1074 Clear mechanisms for such interaction are needed in order to avoid 1075 mal-functioning or interference among concerns. These ideas are 1076 considered in [etsi_gs_nfv_eve005] and [I-D.irtf-sdnrg-layered-sdn] 1078 4.10. Testing 1080 The impacts of network virtualization on testing can be divided into 1081 3 groups: 1083 1. Changes in methodology. 1085 2. New functionality. 1087 3. Opportunities. 1089 4.10.1. Changes in methodology 1091 The largest impact of NFV is the ability to isolate the System Under 1092 Test (SUT). When testing Physical Network Functions (PNF), isolating 1093 the SUT means that all the other devices that the SUT communicates 1094 with are replaced with simulations (or controlled executions) in 1095 order to place the SUT under test by itself. The SUT may be 1096 comprised of one or more devices. The simulations use the 1097 appropriate traffic type and protocols in order to execute test 1098 cases. See Figure 5. 1100 +--------+ +-----------+ +--------+ 1101 | | | | | | 1102 | Sim A | | SUT | | Sim B | 1103 | +------+ +-----+ | 1104 | | | | | | 1105 +--------+ +-----------+ +--------+ 1107 Figure 5: Testing methodology 1109 As shown in Figure 2, NFV provides a common architecture for all 1110 functions to use. A VNF is executed using resources offered by the 1111 NFVI, which have been allocated using the MANO function. It is not 1112 possible to test a VNF by itself, without the entire supporting 1113 environment present. This fundamentally changes how to consider the 1114 SUT. In the case of a VNF (or multiple VNFs), the SUT is part of a 1115 larger architecture which is necessary in order to run the SUTs. 1117 Isolation of the SUT therefore becomes controlling the environment in 1118 a disciplined manner. The components of the environment necessary to 1119 run the SUTs that are not part of the SUT become the test 1120 environment. In the case of VNFs which are the SUT, then the NFVI 1121 and MANO become the test environment. The configurations and 1122 policies that guide the test environment should remain constant 1123 during the execution of the tests, and also from test to test. 1124 Configurations such as CPU pinning, NUMA configuration, the SW 1125 versions and configurations of the hypervisor, vSwitch and NICs 1126 should remain constant. The only variables in the testing should be 1127 those controlling the SUT itself. If any configuration in the test 1128 environment is changed from test to test, then the results become 1129 very difficult, if not impossible, to compare since the test 1130 environment behavior may change the results as a consequence of the 1131 configuration change. 1133 Testing the NFVI itself also presents new considerations. With a 1134 PNF, the dedicated hardware supporting it is optimized for the 1135 particular workload of the function. Routing hardware is specially 1136 built to support packet forwarding functions, while the hardware to 1137 support a purely control plane application (say, a DNS server, or a 1138 Diameter function) will not have this specialized capability. In 1139 NFV, the NFVI is required to support all types of potentially 1140 different workload types. 1142 Testing the NFVI therefore requires careful consideration to what 1143 types of metrics are sought. This, in turn, depends on the workload 1144 type the expected VNF will be. Examples of different workload types 1145 are data forwarding, control plane, encryption, and authentication. 1146 All these types of expected workloads will determine the types of 1147 metrics that should be sought. For example, if the workload is 1148 control plane, then a metric such as jitter is not useful, but 1149 dropped packets is critical. In a multi-tenant environment, then the 1150 NFVI could support various types of workloads. In this case, testing 1151 with a variety of traffic types while measuring the corresponding 1152 metrics simultaneously becomes necessary. 1154 4.10.2. New functionality 1156 NFV presents a collection of new functionality in order to support 1157 the goal of software networking. Each component on the architecture 1158 shown in Figure 2 has an associated set of functionality that allows 1159 VNFs to run: onboarding, lifecycle management for VNFs and Networks 1160 Services (NS), resource allocation, hypervisor functions, etc. 1162 One of the new capabilities enabled by NFV is VNFFG (VNF Forwarding 1163 Graphs). This refers to the graph that represents a Network Service 1164 by chaining together VNFs into a forwarding path. In practice, the 1165 forwarding path can be implemented in a variety of ways using 1166 different networking capabilities: vSwitch, SDN, SDN with a 1167 northbound application, and the VNFFG might use tunneling protocols 1168 like VXLAN. The dynamic allocation and implementation of these 1169 networking paths will have different performance characteristics 1170 depending on the methods used. The path implementation mechanism 1171 becomes a variable in the network testing of the NSs. The 1172 methodology used to test the various mechanisms should largely remain 1173 the same, and as usual, the test environment should remain constant 1174 for each of the tests, focusing on varying the path establishment 1175 method. 1177 Scaling refers to the change in allocation of resources to a VNF or 1178 NS. It happens dynamically at run-time, based on defined policies 1179 and triggers. The triggers can be network, compute or storage based. 1180 Scaling can allocate more resources in times of need, or reduce the 1181 amount of resources allocated when the demand is reduced. The SUT in 1182 this case becomes much larger than the VNF itself: MANO controls how 1183 scaling is done based on policies, and then allocates the resources 1184 accordingly in the NFVI. Essentially, the testing of scaling 1185 includes the entire NFV architecture components into the SUT. 1187 4.10.3. Opportunities 1189 Softwarization of networking functionality leads to softwarization of 1190 test as well. As Physical Network Functions (PNF) are being 1191 transformed into VNFs, so have the test tools. This leads to the 1192 fact that test tools are also being controlled and executed in the 1193 same environment as the VNFs are. This presents an opportunity to 1194 include VNF-based test tools along with the deployment of the VNFs 1195 supporting the services of the service provider into the host data 1196 centers. Tests can therefore be automatically executed upon 1197 deployment in the target environment, for each deployment, and each 1198 service. With PNFs, this was very difficult to achieve. 1200 This new concept helps to enable modern concepts like DevOps and CI/ 1201 CD in the NFV environment. Simplistically, DevOps is a process that 1202 combines multiple functions into single cohesive teams in order to 1203 quickly produce quality software. It typically relies on also 1204 applying the Agile development process, which focusses on (among many 1205 things) dividing large features into multiple, smaller deliveries. 1206 One part of this is to immediately test the new smaller features in 1207 order to get immediate feedback on errors so that if present, they 1208 can be immediately fixed and redeployed. The CI/CD (Continuous 1209 Integration and Continuous Deployment) pipeline supports this 1210 concept. It consists of a series of tools, among which immediate 1211 testing is an integral part, to deliver software from source to 1212 deployment. The ability to deploy the test tools themselves into the 1213 production environment stretches the CI/CD pipeline all the way to 1214 production deployment, allowing a range of tests to be executed. The 1215 tests can be simple, with a goal of verifying the correct deployment 1216 and networking establishment, to the more complex like testing VNF 1217 functionality. 1219 5. Technology Gaps and Potential IETF Efforts 1221 Table 1 correlates the open network virtualization research areas 1222 identified in this document to potential IETF groups that could 1223 address some aspects of them. An example of a specific gap that the 1224 group could potentially address is identified in parenthetical beside 1225 the group name. 1227 +----------------------------------+--------------------------------+ 1228 | Open Research Area | Potential IETF/IRTF Group | 1229 +----------------------------------+--------------------------------+ 1230 | 1-Guaranteeing QoS | IPPM WG (Measurements of NFVI) | 1231 | 2-Performance improvement | VNFPOOL BoF (NFV resilience) | 1232 | 3-Multiple Domains | NFVRG | 1233 | 4-Network Slicing | NVO3 WG, NETSLICES bar BoF | 1234 | 5-Service Composition | SFC WG (SFC Mgmt and Config) | 1235 | 6-End-user device virtualization | N/A | 1236 | 7-Security | N/A | 1237 | 8-Separation of control concerns | NFVRG | 1238 +----------------------------------+--------------------------------+ 1240 Table 1: Mapping of Open Research Areas to Potential IETF Groups 1242 6. Mapping to NFVRG Near-Term work items 1244 Table 2 correlates the currently identified NFVRG near-work items to 1245 the open network virtualization research areas enumerated in this 1246 document. This can help the NFVRG in identifying and prioritizing 1247 research topics. 1249 +--------------------------------------+-------------------------+ 1250 | NFVRG Near-Term work item | Open Research Area | 1251 +--------------------------------------+-------------------------+ 1252 | 1-Policy-based resource management | - Performance improvem. | 1253 | | - Network Slicing | 1254 | 2-Analytics for visibility & orches. | - Guaranteeing QoS | 1255 | 3-Security and service verification | - Security | 1256 | 4-Reliability and fault detection | - Guaranteeing QoS | 1257 | 5-Service orchestration & lifecycle | - Multiple Domains | 1258 | | - Network Slicing | 1259 | | - Service Composition | 1260 | 6-Real-time properties | - Guaranteeing QoS | 1261 | | | 1262 | (other) | - End-user device virt. | 1263 | | - Separation of control | 1264 +--------------------------------------+-------------------------+ 1266 Table 2: Mapping of NFVRG Near-Term work items to Open Research Areas 1268 7. IANA Considerations 1270 N/A. 1272 8. Security Considerations 1274 This is an informational document, which therefore does not introduce 1275 any security threat. Research challenges and gaps related to 1276 security and privacy have been included in Section 4.8. 1278 9. Acknowledgments 1280 The authors want to thank Dirk von Hugo, Rafa Marin, Diego Lopez, 1281 Ramki Krishnan, Kostas Pentikousis, Rana Pratap Sircar, Alfred 1282 Morton, Nicolas Kuhn and Saumya Dikshit for their very useful reviews 1283 and comments to the document. Special thanks to Pedro Martinez- 1284 Julia, who provided text for the network slicing section. 1286 The work of Carlos J. Bernardos and Luis M. Contreras is partially 1287 supported by the H2020-ICT-2014 project 5GEx (Grant Agreement no. 1288 671636). 1290 10. Informative References 1292 [etsi_gs_nfv_003] 1293 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1294 Terminology for Main Concepts in NFV", ETSI GS NFV 003 1295 V1.2.1 NFV 003, December 2014, 1296 . 1299 [etsi_gs_nfv_eve005] 1300 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1301 Ecosystem; Report on SDN Usage in NFV Architectural 1302 Framework", ETSI GS NFV-EVE 005 V1.1.1 NFV-EVE 005, 1303 December 2015, . 1306 [etsi_gs_nfv_per_001] 1307 ETSI NFV ISG, "Network Functions Virtualisation (NFV); NFV 1308 Performance & Portability Best Practises", ETSI GS NFV-PER 1309 001 V1.1.2 NFV-PER 001, December 2014, 1310 . 1313 [etsi_gs_nfv_sec_001] 1314 ETSI NFV ISG, "Network Functions Virtualisation (NFV); NFV 1315 Security; Problem Statement", ETSI GS NFV-SEC 001 V1.1.1 1316 NFV-SEC 001, October 2014, 1317 . 1320 [etsi_nvf_whitepaper_2] 1321 "Network Functions Virtualisation (NFV). White Paper 2", 1322 October 2013. 1324 [etsi_nvf_whitepaper_3] 1325 "Network Functions Virtualisation (NFV). White Paper 3", 1326 October 2014. 1328 [google_sdn_wan] 1329 "B4: experience with a globally-deployed Software Defined 1330 WAN", Proceedings of the ACM SIGCOMM 2013 , August 2013. 1332 [I-D.bagnulo-nfvrg-topology] 1333 Bagnulo, M. and D. Dolson, "NFVI PoP Network Topology: 1334 Problem Statement", draft-bagnulo-nfvrg-topology-01 (work 1335 in progress), March 2016. 1337 [I-D.bernardos-nfvrg-multidomain] 1338 Bernardos, C., Contreras, L., and I. Vaishnavi, "Multi- 1339 domain Network Virtualization", draft-bernardos-nfvrg- 1340 multidomain-01 (work in progress), October 2016. 1342 [I-D.defoy-netslices-3gpp-network-slicing] 1343 Foy, X. and A. Rahman, "Network Slicing - 3GPP Use Case", 1344 draft-defoy-netslices-3gpp-network-slicing-00 (work in 1345 progress), March 2017. 1347 [I-D.gdmb-netslices-intro-and-ps] 1348 Galis, A., Dong, J., kiran.makhijani@huawei.com, k., 1349 Bryant, S., Boucadair, M., and P. Martinez-Julia, "Network 1350 Slicing - Introductory Document and Revised Problem 1351 Statement", draft-gdmb-netslices-intro-and-ps-02 (work in 1352 progress), February 2017. 1354 [I-D.ietf-bmwg-virtual-net] 1355 Morton, A., "Considerations for Benchmarking Virtual 1356 Network Functions and Their Infrastructure", draft-ietf- 1357 bmwg-virtual-net-04 (work in progress), August 2016. 1359 [I-D.irtf-sdnrg-layered-sdn] 1360 Contreras, L., Bernardos, C., Lopez, D., Boucadair, M., 1361 and P. Iovanna, "Cooperating Layered Architecture for 1362 SDN", draft-irtf-sdnrg-layered-sdn-01 (work in progress), 1363 October 2016. 1365 [I-D.mlk-nfvrg-nfv-reliability-using-cots] 1366 Mo, L. and B. Khasnabish, "NFV Reliability using COTS 1367 Hardware", draft-mlk-nfvrg-nfv-reliability-using-cots-01 1368 (work in progress), October 2015. 1370 [I-D.natarajan-nfvrg-containers-for-nfv] 1371 natarajan.sriram@gmail.com, n., Krishnan, R., Ghanwani, 1372 A., Krishnaswamy, D., Willis, P., Chaudhary, A., and F. 1373 Huici, "An Analysis of Lightweight Virtualization 1374 Technologies for NFV", draft-natarajan-nfvrg-containers- 1375 for-nfv-03 (work in progress), July 2016. 1377 [I-D.rorosz-nfvrg-vbaas] 1378 Rosa, R., Rothenberg, C., and R. Szabo, "VNF Benchmark-as- 1379 a-Service", draft-rorosz-nfvrg-vbaas-00 (work in 1380 progress), October 2015. 1382 [intel_10_differences_nfv_cloud] 1383 Intel, "Discover the Top 10 Differences Between NFV and 1384 Cloud Environments", November 2015, 1385 . 1388 [nfv_sota_research_challenges] 1389 , , , , , and , "Network Function Virtualization: State- 1390 of-the-art and Research Challenges", IEEE Communications 1391 Surveys & Tutorials Volume: 18, Issue: 1, September 2015. 1393 [ngmn_5G_whitepaper] 1394 "NGMN 5G. White Paper", February 2015. 1396 [omniran] IEEE, "802.1CF Network Reference Model and Functional 1397 Description of IEEE 802 Access Network", 802.1cf, Draft 1398 0.4 802.1cf, February 2017. 1400 [onf_tr_521] 1401 ONF, "SDN Architecture, Issue 1.1", ONF TR-521 TR-521, 1402 February 2016, 1403 . 1407 [openmano_dataplane] 1408 Telefonica I+D, "OpenMANO: The Dataplane Ready Open Source 1409 NFV MANO Stack", March 2015, 1410 . 1413 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 1414 Application Protocol (CoAP)", RFC 7252, 1415 DOI 10.17487/RFC7252, June 2014, 1416 . 1418 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1419 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1420 Defined Networking (SDN): Layers and Architecture 1421 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1422 2015, . 1424 [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for 1425 Service Function Chaining", RFC 7498, 1426 DOI 10.17487/RFC7498, April 2015, 1427 . 1429 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1430 Chaining (SFC) Architecture", RFC 7665, 1431 DOI 10.17487/RFC7665, October 2015, 1432 . 1434 [virtualization_mobile_device] 1435 "Virtualization of Mobile Device User Experience", Patent 1436 US 9.542.062 B2 , January 2017. 1438 [vnf_benchmarking] 1439 FEEC/UNICAMP, FEEC/UNICAMP, and Ericsson, "A VNF Testing 1440 Framework Design, Implementation and Partial Results", 1441 November 2016, 1442 . 1445 Authors' Addresses 1447 Carlos J. Bernardos 1448 Universidad Carlos III de Madrid 1449 Av. Universidad, 30 1450 Leganes, Madrid 28911 1451 Spain 1453 Phone: +34 91624 6236 1454 Email: cjbc@it.uc3m.es 1455 URI: http://www.it.uc3m.es/cjbc/ 1457 Akbar Rahman 1458 InterDigital Communications, LLC 1459 1000 Sherbrooke Street West, 10th floor 1460 Montreal, Quebec H3A 3G4 1461 Canada 1463 Email: Akbar.Rahman@InterDigital.com 1464 URI: http://www.InterDigital.com/ 1466 Juan Carlos Zuniga 1467 SIGFOX 1468 425 rue Jean Rostand 1469 Labege 31670 1470 France 1472 Email: j.c.zuniga@ieee.org 1473 URI: http://www.sigfox.com/ 1474 Luis M. Contreras 1475 Telefonica I+D 1476 Ronda de la Comunicacion, S/N 1477 Madrid 28050 1478 Spain 1480 Email: luismiguel.contrerasmurillo@telefonica.com 1482 Pedro Aranda 1483 Universidad Carlos III de Madrid 1484 Av. Universidad, 30 1485 Leganes, Madrid 28911 1486 Spain 1488 Email: pedroandres.aranda@uc3m.es 1490 Pierre Lynch 1491 Ixia 1493 Email: plynch@ixiacom.com