idnits 2.17.1 draft-irtf-nfvrg-gaps-network-virtualization-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 30, 2017) is 2364 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-05) exists of draft-bernardos-nfvrg-multidomain-03 Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFVRG CJ. Bernardos 3 Internet-Draft UC3M 4 Intended status: Informational A. Rahman 5 Expires: May 3, 2018 InterDigital 6 JC. Zuniga 7 SIGFOX 8 LM. Contreras 9 TID 10 P. Aranda 11 UC3M 12 P. Lynch 13 Ixia 14 October 30, 2017 16 Network Virtualization Research Challenges 17 draft-irtf-nfvrg-gaps-network-virtualization-07 19 Abstract 21 This document describes open research challenges for network 22 virtualization. Network virtualization is following a similar path 23 as previously taken by cloud computing. Specifically, cloud 24 computing popularized migration of computing functions (e.g., 25 applications) and storage from local, dedicated, physical resources 26 to remote virtual functions accessible through the Internet. In a 27 similar manner, network virtualization is encouraging migration of 28 networking functions from dedicated physical hardware nodes to a 29 virtualized pool of resources. However, network virtualization can 30 be considered to be a more complex problem than cloud computing as it 31 not only involves virtualization of computing and storage functions 32 but also involves abstraction of the network itself. This document 33 describes current research and engineering challenges in network 34 virtualization including guaranteeing quality-of-service, performance 35 improvement, supporting multiple domains, network slicing, service 36 composition, device virtualization, privacy and security, separation 37 of control concerns, network function placement and testing. In 38 addition, some proposals are made for new activities in IETF/IRTF 39 that could address some of these challenges. 41 Status of This Memo 43 This Internet-Draft is submitted in full conformance with the 44 provisions of BCP 78 and BCP 79. 46 Internet-Drafts are working documents of the Internet Engineering 47 Task Force (IETF). Note that other groups may also distribute 48 working documents as Internet-Drafts. The list of current Internet- 49 Drafts is at http://datatracker.ietf.org/drafts/current/. 51 Internet-Drafts are draft documents valid for a maximum of six months 52 and may be updated, replaced, or obsoleted by other documents at any 53 time. It is inappropriate to use Internet-Drafts as reference 54 material or to cite them other than as "work in progress." 56 This Internet-Draft will expire on May 3, 2018. 58 Copyright Notice 60 Copyright (c) 2017 IETF Trust and the persons identified as the 61 document authors. All rights reserved. 63 This document is subject to BCP 78 and the IETF Trust's Legal 64 Provisions Relating to IETF Documents 65 (http://trustee.ietf.org/license-info) in effect on the date of 66 publication of this document. Please review these documents 67 carefully, as they describe your rights and restrictions with respect 68 to this document. Code Components extracted from this document must 69 include Simplified BSD License text as described in Section 4.e of 70 the Trust Legal Provisions and are provided without warranty as 71 described in the Simplified BSD License. 73 Table of Contents 75 1. Introduction and scope . . . . . . . . . . . . . . . . . . . 3 76 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 77 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 78 3.1. Network Function Virtualization . . . . . . . . . . . . . 5 79 3.2. Software Defined Networking . . . . . . . . . . . . . . . 7 80 3.3. ITU-T functional architecture of SDN . . . . . . . . . . 12 81 3.4. Multi-access Edge Computing . . . . . . . . . . . . . . . 13 82 3.5. IEEE 802.1CF (OmniRAN) . . . . . . . . . . . . . . . . . 14 83 3.6. Distributed Management Task Force . . . . . . . . . . . . 14 84 3.7. Open Source initiatives . . . . . . . . . . . . . . . . . 14 85 4. Network Virtualization Challenges . . . . . . . . . . . . . . 16 86 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 16 87 4.2. Guaranteeing quality-of-service . . . . . . . . . . . . . 16 88 4.2.1. Virtualization Technologies . . . . . . . . . . . . . 17 89 4.2.2. Metrics for NFV characterization . . . . . . . . . . 17 90 4.2.3. Predictive analysis . . . . . . . . . . . . . . . . . 18 91 4.2.4. Portability . . . . . . . . . . . . . . . . . . . . . 19 92 4.3. Performance improvement . . . . . . . . . . . . . . . . . 19 93 4.3.1. Energy Efficiency . . . . . . . . . . . . . . . . . . 19 94 4.3.2. Improved link usage . . . . . . . . . . . . . . . . . 20 95 4.4. Multiple Domains . . . . . . . . . . . . . . . . . . . . 20 96 4.5. 5G and Network Slicing . . . . . . . . . . . . . . . . . 21 97 4.5.1. Virtual Network Operators . . . . . . . . . . . . . . 22 98 4.5.2. Extending Virtual Networks and Systems to the 99 Internet of Things . . . . . . . . . . . . . . . . . 23 100 4.6. Service Composition . . . . . . . . . . . . . . . . . . . 24 101 4.7. End-user device virtualization . . . . . . . . . . . . . 25 102 4.8. Security and Privacy . . . . . . . . . . . . . . . . . . 26 103 4.9. Separation of control concerns . . . . . . . . . . . . . 27 104 4.10. Network Function placement . . . . . . . . . . . . . . . 28 105 4.11. Testing . . . . . . . . . . . . . . . . . . . . . . . . . 28 106 4.11.1. Changes in methodology . . . . . . . . . . . . . . . 28 107 4.11.2. New functionality . . . . . . . . . . . . . . . . . 30 108 4.11.3. Opportunities . . . . . . . . . . . . . . . . . . . 30 109 5. Technology Gaps and Potential IETF Efforts . . . . . . . . . 31 110 6. NFVRG focus areas . . . . . . . . . . . . . . . . . . . . . . 31 111 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 112 8. Security Considerations . . . . . . . . . . . . . . . . . . . 32 113 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 33 114 10. Informative References . . . . . . . . . . . . . . . . . . . 33 115 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 38 117 1. Introduction and scope 119 The telecommunications sector is experiencing a major revolution that 120 will shape the way networks and services are designed and deployed 121 for the next few decades. In order to cope with continuously 122 increasing demand and cost, network operators are taking lessons from 123 the IT paradigm of cloud computing. This new approach of 124 virtualizing network functions will enable multi-fold advantages by 125 moving communication services from bespoke hardware in the operator's 126 core network to Commercial off-the-shelf (COTS) equipment distributed 127 across datacenters. 129 Some of the network virtualization mechanisms that are being 130 considered include: sharing of network infrastructure to reduce 131 costs, virtualization of core and edge servers/services running in 132 data centers as a way of supporting their load-aware elastic 133 dimensioning, and dynamic energy policies to reduce the electricity 134 consumption. 136 This document presents research and engineering challenges in network 137 virtualization that need to be addressed in order to achieve these 138 goals, spanning from pure research and engineering/standards space. 139 The objective of this memo is to document the technical challenges 140 and corresponding current approaches and to expose requirements that 141 should be addressed by future research and standards work. 143 2. Terminology 145 The following terms used in this document are defined by the ETSI 146 Network Function Virtualization (NVF) Industrial Study Group (ISG) 147 [etsi_gs_nfv_003], the ONF [onf_tr_521] and the IETF [RFC7426] 148 [RFC7665]: 150 Application Plane - The collection of applications and services 151 that program network behavior. 153 Control Plane (CP) - The collection of functions responsible for 154 controlling one or more network devices. CP instructs network 155 devices with respect to how to process and forward packets. The 156 control plane interacts primarily with the forwarding plane and, 157 to a lesser extent, with the operational plane. 159 Forwarding Plane (FP) - The collection of resources across all 160 network devices responsible for forwarding traffic. 162 Management Plane (MP) - The collection of functions responsible 163 for monitoring, configuring, and maintaining one or more network 164 devices or parts of network devices. The management plane is 165 mostly related to the operational plane (it is related less to the 166 forwarding plane). 168 NFV Infrastructure (NFVI): totality of all hardware and software 169 components which build up the environment in which VNFs are 170 deployed. 172 NFV Management and Orchestration (NFV-MANO): functions 173 collectively provided by NFVO, VNFM, and VIM. 175 NFV Orchestrator (NFVO): functional block that manages the Network 176 Service (NS) lifecycle and coordinates the management of NS 177 lifecycle, VNF lifecycle (supported by the VNFM) and NFVI 178 resources (supported by the VIM) to ensure an optimized allocation 179 of the necessary resources and connectivity. 181 Operational Plane (OP) - The collection of resources responsible 182 for managing the overall operation of individual network devices. 184 Physical Network Function (PNF): Physical implementation of a 185 Network Function in a monolithic realization. 187 Service Function Chain (SFC): for a given service, the abstracted 188 view of the required service functions and the order in which they 189 are to be applied. This is somehow equivalent to the Network 190 Function Forwarding Graph (NF-FG) at ETSI. 192 Service Function Path (SFP): the selection of specific service 193 function instances on specific network nodes to form a service 194 graph through which an SFC is instantiated. 196 Virtualized Infrastructure Manager (VIM): functional block that is 197 responsible for controlling and managing the NFVI compute, storage 198 and network resources, usually within one infrastructure 199 operator's Domain. 201 Virtualized Network Function (VNF): implementation of a Network 202 Function that can be deployed on a Network Function Virtualization 203 Infrastructure (NFVI). 205 Virtualized Network Function Manager (VNFM): functional block that 206 is responsible for the lifecycle management of VNF. 208 3. Background 210 This section briefly describes some basic background technologies, as 211 well as other standards developing organizations and open source 212 initiatives working on network virtualization or related topics. 214 3.1. Network Function Virtualization 216 The ETSI ISG NFV is a working group which, since 2012, aims to evolve 217 quasi-standard IT virtualization technology to consolidate many 218 network equipment types into industry standard high volume servers, 219 switches, and storage. It enables implementing network functions in 220 software that can run on a range of industry standard server hardware 221 and can be moved to, or loaded in, various locations in the network 222 as required, without the need to install new equipment. The ETSI NFV 223 is one of the predominant NFV reference framework and architectural 224 footprints [nfv_sota_research_challenges]. The ETSI NFV framework 225 architecture framework is composed of three domains (Figure 1): 227 o Virtualized Network Function, running over the NFVI. 229 o NFV Infrastructure (NFVI), including the diversity of physical 230 resources and how these can be virtualized. NFVI supports the 231 execution of the VNFs. 233 o NFV Management and Orchestration, which covers the orchestration 234 and life-cycle management of physical and/or software resources 235 that support the infrastructure virtualization, and the life-cycle 236 management of VNFs. NFV Management and Orchestration focuses on 237 all virtualization specific management tasks necessary in the NFV 238 framework. 240 +-------------------------------------------+ +---------------+ 241 | Virtualized Network Functions (VNFs) | | | 242 | ------- ------- ------- ------- | | | 243 | | | | | | | | | | | | 244 | | VNF | | VNF | | VNF | | VNF | | | | 245 | | | | | | | | | | | | 246 | ------- ------- ------- ------- | | | 247 +-------------------------------------------+ | | 248 | | 249 +-------------------------------------------+ | | 250 | NFV Infrastructure (NFVI) | | NFV | 251 | ----------- ----------- ----------- | | Management | 252 | | Virtual | | Virtual | | Virtual | | | and | 253 | | Compute | | Storage | | Network | | | Orchestration | 254 | ----------- ----------- ----------- | | | 255 | +---------------------------------------+ | | | 256 | | Virtualization Layer | | | | 257 | +---------------------------------------+ | | | 258 | +---------------------------------------+ | | | 259 | | ----------- ----------- ----------- | | | | 260 | | | Compute | | Storage | | Network | | | | | 261 | | ----------- ----------- ----------- | | | | 262 | | Hardware resources | | | | 263 | +---------------------------------------+ | | | 264 +-------------------------------------------+ +---------------+ 266 Figure 1: ETSI NFV framework 268 The NFV architectural framework identifies functional blocks and the 269 main reference points between such blocks. Some of these are already 270 present in current deployments, whilst others might be necessary 271 additions in order to support the virtualization process and 272 consequent operation. The functional blocks are (Figure 2): 274 o Virtualized Network Function (VNF). 276 o Element Management (EM). 278 o NFV Infrastructure, including: Hardware and virtualized resources, 279 and Virtualization Layer. 281 o Virtualized Infrastructure Manager(s) (VIM). 283 o NFV Orchestrator. 285 o VNF Manager(s). 287 o Service, VNF and Infrastructure Description. 289 o Operations and Business Support Systems (OSS/BSS). 291 +--------------------+ 292 +-------------------------------------------+ | ---------------- | 293 | OSS/BSS | | | NFV | | 294 +-------------------------------------------+ | | Orchestrator +-- | 295 | ---+------------ | | 296 +-------------------------------------------+ | | | | 297 | --------- --------- --------- | | | | | 298 | | EM 1 | | EM 2 | | EM 3 | | | | | | 299 | ----+---- ----+---- ----+---- | | ---+---------- | | 300 | | | | |--|-| VNF | | | 301 | ----+---- ----+---- ----+---- | | | manager(s) | | | 302 | | VNF 1 | | VNF 2 | | VNF 3 | | | ---+---------- | | 303 | ----+---- ----+---- ----+---- | | | | | 304 +------|-------------|-------------|--------+ | | | | 305 | | | | | | | 306 +------+-------------+-------------+--------+ | | | | 307 | NFV Infrastructure (NFVI) | | | | | 308 | ----------- ----------- ----------- | | | | | 309 | | Virtual | | Virtual | | Virtual | | | | | | 310 | | Compute | | Storage | | Network | | | | | | 311 | ----------- ----------- ----------- | | ---+------ | | 312 | +---------------------------------------+ | | | | | | 313 | | Virtualization Layer | |--|-| VIM(s) +-------- | 314 | +---------------------------------------+ | | | | | 315 | +---------------------------------------+ | | ---------- | 316 | | ----------- ----------- ----------- | | | | 317 | | | Compute | | Storage | | Network | | | | | 318 | | | hardware| | hardware| | hardware| | | | | 319 | | ----------- ----------- ----------- | | | | 320 | | Hardware resources | | | NFV Management | 321 | +---------------------------------------+ | | and Orchestration | 322 +-------------------------------------------+ +--------------------+ 324 Figure 2: ETSI NFV reference architecture 326 3.2. Software Defined Networking 328 The Software Defined Networking (SDN) paradigm pushes the 329 intelligence currently residing in the network elements to a central 330 controller implementing the network functionality through software. 331 In contrast to traditional approaches, in which the network's control 332 plane is distributed throughout all network devices, with SDN the 333 control plane is logically centralized. In this way, the deployment 334 of new characteristics in the network no longer requires complex and 335 costly changes in equipment or firmware updates, but only a change in 336 the software running in the controller. The main advantage of this 337 approach is the flexibility it provides operators to manage their 338 network, i.e., an operator can easily change its policies on how 339 traffic is distributed throughout the network. 341 One of the most well known protocols for the SDN control plane 342 between the central controller and the networking elements is the 343 OpenFlow protocol (OFP), which is maintained and extended by the Open 344 Network Foundation (ONF: https://www.opennetworking.org/). 345 Originally this protocol was developed specifically for IEEE 802.1 346 switches conforming to the ONF OpenFlow Switch specification. As the 347 benefits of the SDN paradigm have reached a wider audience, its 348 application has been extended to more complex scenarios such as 349 Wireless and Mobile networks. Within this area of work, the ONF is 350 actively developing new OFP extensions addressing three key 351 scenarios: (i) Wireless backhaul, (ii) Cellular Evolved Packet Core 352 (EPC), and (iii) Unified access and management across enterprise 353 wireless and fixed networks. 355 +----------+ 356 | ------- | 357 | |Oper.| | O 358 | |Mgmt.| |<........> -+- Network Operator 359 | |Iface| | ^ 360 | ------- | +----------------------------------------+ 361 | | | +------------------------------------+ | 362 | | | | --------- --------- --------- | | 363 |--------- | | | | App 1 | | App 2 | ... | App n | | | 364 ||Plugins| |<....>| | --------- --------- --------- | | 365 |--------- | | | Plugins | | 366 | | | +------------------------------------+ | 367 | | | Application Plane | 368 | | +----------------------------------------+ 369 | | A 370 | | | 371 | | V 372 | | +----------------------------------------+ 373 | | | +------------------------------------+ | 374 |--------- | | | ------------ ------------ | | 375 || Netw. | | | | | Module 1 | | Module 2 | | | 376 ||Engine | |<....>| | ------------ ------------ | | 377 |--------- | | | Network Engine | | 378 | | | +------------------------------------+ | 379 | | | Controller Plane | 380 | | +----------------------------------------+ 381 | | A 382 | | | 383 | | V 384 | | +----------------------------------------+ 385 | | | +--------------+ +--------------+ | 386 | | | | ------------ | | ------------ | | 387 |----------| | | | OpenFlow | | | | OpenFlow | | | 388 ||OpenFlow||<....>| | ------------ | | ------------ | | 389 |----------| | | NE | | NE | | 390 | | | +--------------+ +--------------+ | 391 | | | Data Plane | 392 |Management| +----------------------------------------+ 393 +----------+ 395 Figure 3: High level SDN ONF architecture 397 Figure 3 shows the blocks and the functional interfaces of the ONF 398 architecture, which comprises three planes: Data, Controller, and 399 Application. The Data plane comprehends several Network Entities 400 (NE), which expose their capabilities toward the Controller plane via 401 a Southbound API. The Controller plane includes several cooperating 402 modules devoted to the creation and maintenance of an abstracted 403 resource model of the underlying network. Such model is exposed to 404 the applications via a Northbound API where the Application plane 405 comprises several applications/services, each of which has exclusive 406 control of a set of exposed resources. 408 The Management plane spans its functionality across all planes 409 performing the initial configuration of the network elements in the 410 Data plane, the assignment of the SDN controller and the resources 411 under its responsibility. In the Controller plane, the Management 412 needs to configure the policies defining the scope of the control 413 given to the SDN applications, to monitor the performance of the 414 system, and to configure the parameters required by the SDN 415 controller modules. In the Application plane, Management configures 416 the parameters of the applications and the service level agreements. 417 In addition to these interactions, the Management plane exposes 418 several functions to network operators which can easily and quickly 419 configure and tune the network at each layer. 421 In RFC7426 [RFC7426], the IRTF Software-Defined Networking Research 422 Group (SDNRG) documented a layer model of an SDN architecture, since 423 this has been a controversial discussion topic: what exactly is SDN? 424 what is the layer structure of the SDN architecture? how do layers 425 interface with each other? etc. 427 Figure 4 reproduces the figure included in RFC7426 [RFC7426] to 428 summarize the SDN architecture abstractions in the form of a 429 detailed, high-level schematic. In a particular implementation, 430 planes can be collocated with other planes or can be physically 431 separated. 433 In SDN, a controller manipulates controlled entities via an 434 interface. Interfaces, when local, are mostly API invocations 435 through some library or system call. However, such interfaces may be 436 extended via some protocol definition, which may use local inter- 437 process communication (IPC) or a protocol that could also act 438 remotely; the protocol may be defined as an open standard or in a 439 proprietary manner. 441 SDN expands multiple planes: Forwarding, Operational, Control, 442 Management and Applications. All planes mentioned above are 443 connected via interfaces. Additionally, RFC7426 [RFC7426] considers 444 four abstraction layers: the Device and resource Abstraction Layer 445 (DAL), the Control Abstraction Layer (CAL), the Management 446 Abstraction Layer (MAL) and the Network Services Abstraction Layer 447 (NSAL). 449 o--------------------------------o 450 | | 451 | +-------------+ +----------+ | 452 | | Application | | Service | | 453 | +-------------+ +----------+ | 454 | Application Plane | 455 o---------------Y----------------o 456 | 457 *-----------------------------Y---------------------------------* 458 | Network Services Abstraction Layer (NSAL) | 459 *------Y------------------------------------------------Y-------* 460 | | 461 | Service Interface | 462 | | 463 o------Y------------------o o---------------------Y------o 464 | | Control Plane | | Management Plane | | 465 | +----Y----+ +-----+ | | +-----+ +----Y----+ | 466 | | Service | | App | | | | App | | Service | | 467 | +----Y----+ +--Y--+ | | +--Y--+ +----Y----+ | 468 | | | | | | | | 469 | *----Y-----------Y----* | | *---Y---------------Y----* | 470 | | Control Abstraction | | | | Management Abstraction | | 471 | | Layer (CAL) | | | | Layer (MAL) | | 472 | *----------Y----------* | | *----------Y-------------* | 473 | | | | | | 474 o------------|------------o o------------|---------------o 475 | | 476 | CP | MP 477 | Southbound | Southbound 478 | Interface | Interface 479 | | 480 *------------Y---------------------------------Y----------------* 481 | Device and resource Abstraction Layer (DAL) | 482 *------------Y---------------------------------Y----------------* 483 | | | | 484 | o-------Y----------o +-----+ o--------Y----------o | 485 | | Forwarding Plane | | App | | Operational Plane | | 486 | o------------------o +-----+ o-------------------o | 487 | Network Device | 488 +---------------------------------------------------------------+ 490 Figure 4: SDN Layer Architecture 492 While SDN is often directly associated to OpenFlow, this is just one 493 (relevant) example of a southbound protocol between the central 494 controller and the network entities. Other relevant examples of 495 protocols in the SDN family are NETCONF [RFC6241], RESTCONF [RFC8040] 496 and ForCES [RFC5810]. 498 3.3. ITU-T functional architecture of SDN 500 The Telecommunication standardization sector of the International 501 Telecommunication Union (ITU) -- the ITU-T -- has also looked into 502 SDN architectures, defining a slightly modified one from what other 503 SDOs have done. ITU-T provides in the recommendation ITU-T Y.3302 504 [itu-t-y.3302] a functional architecture of SDN with descriptions of 505 functional components and reference points. The described functional 506 architecture is intended to be used as an enabler for further studies 507 on other aspects such as protocols and security as well as being used 508 to customize SDN in support of appropriate use cases (e.g., cloud 509 computing, mobile networks). This recommendation is based on ITU-T 510 Y.3300 [itu-t-y.3300] and ITU-T Y.3301 [itu-t-y.3301]. While the 511 first describes the framework of SDN (including definitions, 512 objectives, high-level capabilities, requirements and the high-level 513 architecture of SDN), the second describes more detailed 514 requirements. 516 Figure 5 shows the SDN functional architecture defined by the ITU-T. 517 It is a layered architecture composed of the SDN application layer 518 (SDN-AL), the SDN control layer (SDN-CL) and the SDN resource layer 519 (SDN-RL). It also has multi-layer management functions (MMF), which 520 provides functionalities for managing the functionalities of SDN 521 layers, i.e., SDN-AL, SDN-CL and SDN-RL. MMF interacts with these 522 layers using MMFA, MMFC, and MMFR reference points. 524 The SDN-AL enables a service-aware behavior of the underlying network 525 in a programmatic manner. The SDN-CL provides programmable means to 526 control the behavior of SDN-RL resources (such as data transport and 527 processing), following requests received from the SDN-AL according to 528 MMF policies. The SDN-RL is where the physical or virtual network 529 elements perform transport and/or processing of data packets 530 according to SDN-CL decisions. 532 MMFO MMFA 533 +-----+ . +---------------------+ . +--------------------+ 534 | | . |+---+ +---+ +-------+| . |+---------+ +-----+ | 535 | | . || | | | | || . || AL. | | | | 536 | | . || E | | | | App. || . || mngmt. | | SDN | | SDN-AL 537 | | . || x | | M | | layer || . || support | | app | | 538 | | . || t.| | u | | Mngmt.|| . || & orch. | | | | 539 | | . || | | l | +-------+| . |+---------+ +-----+ | 540 | | . || r | | t | | . +--------------------+ 541 | | . || e | | i | |MMFC ..................... ACI 542 | | . || l | | | | . +--------------------+ 543 | | . || a | | l | +-------+| . |+------+ +---------+| 544 | OSS/| . || t | | a | | || . || | | App. || 545 | BSS | . || i | | y | | || . || | | support || 546 | | . || o | | e | | || . || | +---------+| 547 | | . || n | | r | | || . || CL | +---------+| 548 | | . || s | | | |Control|| . ||mngmt.| | Control || 549 | | . || h | | m | | layer || . || supp.| | layer || SDN-CL 550 | | . || i | | a | | mngmt.|| . || and | | serv. || 551 | | . || p | | n | | || . || orch.| +---------+| 552 | | . || | | a | | || . || | +---------+| 553 | | . || m | | g | | || . || | | Resource|| 554 | | . || n | | e | | || . || | | abstrac.|| 555 | | . || g | | m | +-------+| . |+------+ +---------+| 556 | | . || m | | e | | . +--------------------+ 557 | | . || t.| | n | |MMFR ..................... RCI 558 | | . || | | t | | . +--------------------+ 559 +-----+ . |+---+ | | +-------+| . |+------++----------+| 560 | | o | | || . || ||RL control|| 561 | | r | |Resour.|| . || RL |+----------+| 562 MMF | | c | | layer || . ||mngmt.|+----++----+| SDN-RL 563 | | h.| | mngmt.|| . || supp.||Data||Data|| 564 | | | | || . || ||tran||proc|| 565 | +---+ +-------+| . |+------++----++----+| 566 +---------------------+ . +--------------------+ 568 Figure 5: ITU-T SDN functional architecture 570 3.4. Multi-access Edge Computing 572 Multi-access Edge Computing (MEC) -- formerly known as Mobile Edge 573 Computing -- capabilities deployed in the edge of the mobile network 574 can facilitate the efficient and dynamic provision of services to 575 mobile users. The ETSI ISG MEC working group, operative from end of 576 2014, intends to specify an open environment for integrating MEC 577 capabilities with service providers' networks, including also 578 applications from 3rd parties. These distributed computing 579 capabilities will make available IT infrastructure as in a cloud 580 environment for the deployment of functions in mobile access 581 networks. It can be seen then as a complement to both NFV and SDN. 583 3.5. IEEE 802.1CF (OmniRAN) 585 The IEEE 802.1CF Recommended Practice [omniran] specifies an access 586 network, which connects terminals to their access routers, utilizing 587 technologies based on the family of IEEE 802 Standards (e.g., 802.3 588 Ethernet, 802.11 Wi-Fi, etc.). The specification defines an access 589 network reference model, including entities and reference points 590 along with behavioral and functional descriptions of communications 591 among those entities. 593 The goal of this project is to help unifying the support of different 594 interfaces, enabling shared network control and use of SDN 595 principles, thereby lowering the barriers to new network 596 technologies, to new network operators, and to new service providers. 598 3.6. Distributed Management Task Force 600 The DMTF (https://www.dmtf.org/) is an industry standards 601 organization working to simplify the manageability of network- 602 accessible technologies through open and collaborative efforts by 603 some technology companies. The DMTF is involved in the creation and 604 adoption of interoperable management standards, supporting 605 implementations that enable the management of diverse traditional and 606 emerging technologies including cloud, virtualization, network and 607 infrastructure. 609 There are several DMTF initiatives that are relevant to the network 610 virtualization area, such as the Open Virtualization Format (OVF), 611 for VNF packaging; the Cloud Infrastructure Management Interface 612 (CIM), for cloud infrastructure management; the Network Management 613 (NETMAN), for VNF management; and, the Virtualization Management 614 (VMAN), for virtualization infrastructure management. 616 3.7. Open Source initiatives 618 The Open Source community is especially active in the area of network 619 virtualization and orchestration. We next summarize some of the 620 active efforts: 622 o OpenStack. OpenStack is a free and open-source cloud-computing 623 software platform. OpenStack software controls large pools of 624 compute, storage, and networking resources throughout a 625 datacenter, managed through a dashboard or via the OpenStack API. 627 o Kubernetes. Kubernetes is an open-source system for automating 628 deployment, scaling and management of containerized applications. 629 Kubernetes can schedule and run application containers on clusters 630 of physical or virtual machines. Kubernetes allows: (i) Scale on 631 the fly, (ii) Limit hardware usage to required resources only, 632 (iii) Load balancing Monitoring, and (iv) Efficient lifecycle 633 management. 635 o OpenDayLight. OpenDaylight (ODL) is a highly available, modular, 636 extensible and scalable multi-protocol controller infrastructure 637 built for SDN deployments on modern heterogeneous multi-vendor 638 networks. It provides a model-driven service abstraction platform 639 that allows users to write apps that easily work across a wide 640 variety of hardware and southbound protocols. 642 o ONOS. The ONOS (Open Network Operating System) project is an open 643 source community hosted by The Linux Foundation. The goal of the 644 project is to create a SDN operating system for communications 645 service providers that is designed for scalability, high 646 performance and high availability. 648 o OpenContrail. OpenContrail is an Apache 2.0-licensed project that 649 is built using standards-based protocols and provides all the 650 necessary components for network virtualization-SDN controller, 651 virtual router, analytics engine, and published northbound APIs. 652 It has an extensive REST API to configure and gather operational 653 and analytics data from the system. 655 o OPNFV. OPNFV is a carrier-grade, integrated, open source platform 656 to accelerate the introduction of new NFV products and services. 657 By integrating components from upstream projects, the OPNFV 658 community aims at conducting performance and use case-based 659 testing to ensure the platform's suitability for NFV use cases. 660 The scope of OPNFV's initial release is focused on building NFV 661 Infrastructure (NFVI) and Virtualized Infrastructure Management 662 (VIM) by integrating components from upstream projects such as 663 OpenDaylight, OpenStack, Ceph Storage, KVM, Open vSwitch, and 664 Linux. These components, along with application programmable 665 interfaces (APIs) to other NFV elements form the basic 666 infrastructure required for Virtualized Network Functions (VNF) 667 and Management and Network Orchestration (MANO) components. 668 OPNFV's goal is to (i) increase performance and power efficiency, 669 (ii) improve reliability, availability, and serviceability, and 670 (iii) deliver comprehensive platform instrumentation. 672 o OSM. Open Source Mano (OSM) is an ETSI-hosted project to develop 673 an Open Source NFV Management and Orchestration (MANO) software 674 stack aligned with ETSI NFV. OSM is based on components from 675 previous projects, such Telefonica's OpenMANO or Canonical's Juju, 676 among others. 678 o OpenBaton. OpenBaton is a ETSI NFV compliant Network Function 679 Virtualization Orchestrator (NFVO). OpenBaton was part of the 680 OpenSDNCore project started with the objective of providing a 681 compliant implementation of the ETSI NFV specification. 683 o ONAP. ONAP (Open Network Automation Platform) is an open source 684 software platform that delivers capabilities for the design, 685 creation, orchestration, monitoring, and life cycle management of: 686 (i) Virtual Network Functions (VNFs), (ii) The carrier-scale 687 Software Defined Networks (SDNs) that contain them, and (iii) 688 Higher-level services that combine the above. ONAP (derived from 689 the AT&T's ECOMP) provides for automatic, policy-driven 690 interaction of these functions and services in a dynamic, real- 691 time cloud environment. 693 o SONA. SONA (Simplified Overlay Network Architecture) is an 694 extension to ONOS to have a almost full SDN network control in 695 OpenStack for virtual tenant network provisioning. Basically, 696 SONA is an SDN-based network virtualization solution for cloud DC. 698 Among the main areas that are being developed by the former open 699 source activities that relate to network virtualization research, we 700 can highlight: policy-based resource management, analytics for 701 visibility and orchestration, service verification with regards to 702 security and resiliency. 704 4. Network Virtualization Challenges 706 4.1. Introduction 708 Network Virtualization is changing the way the telecommunications 709 sector will deploy, extend and operate their networks. These new 710 technologies aim at reducing the overall costs by moving 711 communication services from specific hardware in the operators' core 712 to server farms scattered in datacenters (i.e. compute and storage 713 virtualization). In addition, the networks interconnecting the 714 functions that compose a network service are fundamentally affected 715 in the way they route, process and control traffic (i.e. network 716 virtualization). 718 4.2. Guaranteeing quality-of-service 720 Achieving a given quality-of-service in an NFV environment with 721 virtualized and distributed computing, storage and networking 722 functions is more challenging than providing the equivalent in 723 discrete non-virtualized components. For example, ensuring a 724 guaranteed and stable forwarding data rate has proven not to be 725 straightforward when the forwarding function is virtualized and runs 726 on top of COTS server hardware [openmano_dataplane] 727 [I-D.mlk-nfvrg-nfv-reliability-using-cots] [etsi_nvf_whitepaper_3]. 728 Again, the comparison point is against a router or forwarder built on 729 optimized hardware. We next identify some of the challenges that 730 this poses. 732 4.2.1. Virtualization Technologies 734 The issue of guaranteeing a network quality-of-service is less of an 735 issue for "traditional cloud computing" because the workloads that 736 are treated there are servers or clients in the networking sense and 737 hardly ever process packets. Cloud computing provides hosting for 738 applications on shared servers in a highly separated way. Its main 739 advantage is that the infrastructure costs are shared among tenants 740 and that the cloud infrastructure provides levels of reliability that 741 can not be achieved on individual premises in a cost-efficient way 742 [intel_10_differences_nfv_cloud]. NFV has very strict requirements 743 posed in terms of performance, stability and consistency. Although 744 there are some tools and mechanisms to improve this, such as Enhanced 745 Performance Awareness (EPA), Single Root I/O Virtualization (SR-IOV), 746 Non-Uniform Memory Access (NUMA), Data Plane Development Kit (DPDK), 747 etc, these are still unsolved challenges. One open research issue is 748 finding out technologies that are different from VM and more suitable 749 for dealing with network functionalities. 751 Lately, a number of light-weight virtualization technologies 752 including containers, unikernels (specialized VMs) and minimalistic 753 distributions of general-purpose OSes have appeared as virtualization 754 approaches that can be used when constructing an NFV platform. 755 [I-D.natarajan-nfvrg-containers-for-nfv] describes the challenges in 756 building such a platform and discusses to what extent these 757 technologies, as well as traditional VMs, are able to address them. 759 4.2.2. Metrics for NFV characterization 761 Another relevant aspect is the need for tools for diagnostics and 762 measurement suited for NFV. There is a pressing need to define 763 metrics and associated protocols to measure the performance of NFV. 764 Specifically, since NFV is based on the concept of taking centralized 765 functions and evolving it to highly distributed SW functions, there 766 is a commensurate need to fully understand and measure the baseline 767 performance of such systems. 769 The IP Performance Metrics (IPPM) WG defines metrics that can be used 770 to measure the quality and performance of Internet services and 771 applications running over transport layer protocols (e.g., TCP, UDP) 772 over IP. It also develops and maintains protocols for the 773 measurement of these metrics. While the IPPM WG is a long running WG 774 that started in 1997, at the time of writing it does not have a 775 charter item or active drafts related to the topic of network 776 virtualization. In addition to using IPPM metrics to evaluate the 777 QoS, there is a need for specific metrics for assessing the 778 performance of network virtualization techniques. 780 The Benchmarking Methodology Working Group (BMWG) is also performing 781 work related to NFV metrics. For example, [RFC8172] investigates 782 additional methodological considerations necessary when benchmarking 783 VNFs instantiated and hosted in general- purpose hardware, using 784 bare-metal hypervisors or other isolation environments such as Linux 785 containers. An essential consideration is benchmarking physical and 786 virtual network functions in the same way when possible, thereby 787 allowing direct comparison. 789 As stated in the document [RFC8172], there is a clear motivation for 790 the work on performance metrics for NFV [etsi_gs_nfv_per_001], that 791 is worth replicating here: "I'm designing and building my NFV 792 Infrastructure platform. The first steps were easy because I had a 793 small number of categories of VNFs to support and the VNF vendor gave 794 HW recommendations that I followed. Now I need to deploy more VNFs 795 from new vendors, and there are different hardware recommendations. 796 How well will the new VNFs perform on my existing hardware? Which 797 among several new VNFs in a given category are most efficient in 798 terms of capacity they deliver? And, when I operate multiple 799 categories of VNFs (and PNFs) *concurrently* on a hardware platform 800 such that they share resources, what are the new performance limits, 801 and what are the software design choices I can make to optimize my 802 chosen hardware platform? Conversely, what hardware platform 803 upgrades should I pursue to increase the capacity of these 804 concurrently operating VNFs?" 806 Lately, there are also some efforts looking into VNF benchmarking. 807 The selection of an NFV Infrastructure Point of Presence to host a 808 VNF or allocation of resources (e.g., virtual CPUs, memory) needs to 809 be done over virtualized (abstracted and simplified) resource views 810 [vnf_benchmarking] [I-D.rorosz-nfvrg-vbaas]. 812 4.2.3. Predictive analysis 814 On top of diagnostic tools that enable an assessment of the QoS, 815 predictive analyses are required to react before anomalies occur. 816 Due to the SW characteristics of VNFs, a reliable diagnosis framework 817 could potentially enable the prevention of issues by a proper 818 diagnosis and then a reaction in terms of acting on the potentially 819 impacted service (e.g., migration to a different compute node, 820 scaling in/out, up/down, etc). 822 4.2.4. Portability 824 Portability in NFV refers to the ability to run a given VNF on 825 multiple NFVIs, that is, guaranteeing that the VNF would be able to 826 perform its functions with a high and predictable performance given 827 that a set of requirements on the NFVI resources is met. Therefore, 828 portability is a key feature that, if fully enabled, would contribute 829 to making the NFV environment achieve a better reliability than a 830 traditional system. Implementing functionality in SW over 831 "commodity" infrastructure should make it much easier to port/move 832 functions from one place to another. However this is not yet as 833 ideal as it sounds, and there are aspects that are not fully tackled. 834 The existence of different hypervisors, specific hardware 835 dependencies (e.g., EPA related) or state synchronization aspects are 836 just some examples of trouble-makers for portability purposes. 838 The ETSI NFV ISG is doing work in relation to portability. 839 [etsi_gs_nfv_per_001] provides a list of minimal features which the 840 VM Descriptor and Compute Host Descriptor should contain for the 841 appropriate deployment of VM images over an NFVI (i.e. a "telco 842 datacenter"), in order to guarantee high and predictable performance 843 of data plane workloads while assuring their portability. In 844 addition, the document provides a set of recommendations on the 845 minimum requirements which HW and hypervisor should have for a "telco 846 datacenter" suitable for different workloads (data-plane, control- 847 plane, etc.) present in VNFs. The purpose of this document is to 848 provide the list of VM requirements that should be included in the VM 849 Descriptor template, and the list of HW capabilities that should be 850 included in the Compute Host Descriptor (CHD) to assure predictable 851 high performance. ETSI NFV assumes that the MANO Functions will make 852 the mix & match. There are therefore still several research 853 challenges to be addressed here. 855 4.3. Performance improvement 857 4.3.1. Energy Efficiency 859 Virtualization is typically seen as a direct enabler of energy 860 savings. Some of the enablers for this that are often mentioned 861 [nfv_sota_research_challenges] are: (i) the multiplexing gains 862 achieved by centralizing functions in data centers reduce the overall 863 energy consumed, (ii) the flexibility brought by network 864 programmability enables to switch off infrastructure as needed in a 865 much easier way. However there is still a lot of room for 866 improvement in terms of virtualization techniques to reduce the power 867 consumption, such as enhanced hypervisor technologies. 869 Some additional examples of research topics that could enable energy 870 savings are [nfv_sota_research_challenges]: 872 o Energy aware scaling (e.g., reductions in CPU speeds and partially 873 turning off some hardware com- ponents to meet a given energy 874 consumption target. 876 o Energy-aware function placement. 878 o Scheduling and chaining algorithms, for example adapting the 879 network topology and operating parameters to minimize the 880 operation cost (e.g., tracking energy costs to identify the 881 cheapest prices). 883 Note that it is also important to analyze the trade-off between 884 energy efficiency and network performance. 886 4.3.2. Improved link usage 888 The use of NFV and SDN technologies can help improve link usage. SDN 889 has already shown that it can greatly increase average link 890 utilization (e.g., Google example [google_sdn_wan]). NFV adds more 891 complexity (e.g., due to service function chaining / VNF forwarding 892 graphs) which need to be considered. Aspects like the ones described 893 in [I-D.bagnulo-nfvrg-topology] on NFV data center topology design 894 have to be carefully looked at as well. 896 4.4. Multiple Domains 898 Market fragmentation has resulted in a multitude of network operators 899 each focused on different countries and regions. This makes it 900 difficult to create infrastructure services spanning multiple 901 countries, such as virtual connectivity or compute resources, as no 902 single operator has a footprint everywhere. Cross-domain 903 orchestration of services over multiple administrations or over 904 multi-domain single administrations will allow end-to-end network and 905 service elements to mix in multi-vendor, heterogeneous technology and 906 resource environments [multi-domain_5GEx]. 908 For the specific use case of 'Network as a Service', it becomes even 909 more important to ensure that Cross Domain Orchestration also takes 910 care of hierarchy of networks and their association, with respect to 911 provisioning tunnels and overlays. 913 Multi-domain orchestration is currently an active research topic, 914 which is being tackled, among others, by ETSI NFV ISG and the 5GEx 915 project (https://www.5gex.eu/) [I-D.bernardos-nfvrg-multidomain] 916 [multi-domain_5GEx]. 918 Another side of the multi-domain problem is the integration/ 919 harmonization of different management domains. A key example comes 920 from Multi-access Edge Computing, which, according to ETSI, comes 921 with its own MANO system, and would require to be integrated if 922 interconnected to a generic NFV system. 924 4.5. 5G and Network Slicing 926 From the beginning of all 5G discussions in the research and industry 927 fora, it has been agreed that 5G will have to address much more use 928 cases than the preceding wireless generations, which first focused on 929 voice services, and then on voice and high speed packet data 930 services. In this case, 5G should be able to handle not only the 931 same (or enhanced) voice and packet data services, but also new 932 emerging services like tactile Internet and IoT. These use cases 933 take the requirements to opposite extremes, as some of them require 934 ultra-low latency and higher-speed, whereas some others require 935 ultra-low power consumption and high delay tolerance. 937 Because of these very extreme 5G use cases, it is envisioned that 938 selective combinations of radio access networks and core network 939 components will have to be combined into a given network slice to 940 address the specific requirements of each use case. 942 For example, within the major IoT category, which is perhaps the most 943 disrupting one, some autonomous IoT devices will have very low 944 throughput, will have much longer sleep cycles (and therefore high 945 latency), and a battery life time exceeding by a factor of thousands 946 that of smart phones or some other devices that will have almost 947 continuous control and data communications. Hence, it is envisioned 948 that a customized network slice will have to be stitched together 949 from virtual resources or sub-slices to meet these requirements. 951 The actual definition of network slice from an IP infrastructure 952 viewpoint is currently undergoing intense debate 953 [I-D.galis-netslices-revised-problem-statement] 954 [I-D.gdmb-netslices-intro-and-ps] 955 [I-D.defoy-netslices-3gpp-network-slicing] [ngmn_5G_whitepaper]. 956 Network slicing is a key for introducing new actors in existing 957 market at low cost -- by letting new players rent "blocks" of 958 capacity, if the new business model enables performance that meets 959 the application needs (e.g., broadcasting updates to many sensors 960 with satellite broadcasting capabilities). However, more work needs 961 to be done to define the basic architectural approach of how network 962 slices will be defined and formed. For example, is it mostly a 963 matter of defining the appropriate network models (e.g. YANG) to 964 stitch the network slice from existing components. Or do end-to-end 965 timing, synchronization and other low level requirements mean that 966 more fundamental research has to be done. 968 4.5.1. Virtual Network Operators 970 The widespread use/discussion/practice of system and network 971 virtualization technologies has led to new business opportunities, 972 enlarging the offer of IT resources with virtual network and 973 computing resources, among others. As a consequence, the network 974 ecosystem now differentiates between the owner of physical resources, 975 the Infrastructure Provider (InP), and the intermediary that conforms 976 and delivers network services to the final customers, the Virtual 977 Network Operator (VNO). 979 VNOs aim to exploit the virtualized infrastructures to deliver new 980 and improved services to their customers. However, current network 981 virtualization techniques offer poor support for VNOs to control 982 their resources. It has been considered that the InP is responsible 983 for the reliability of the virtual resources but there are several 984 situations in which a VNO requires to gain a finer control on its 985 resources. For instance, dynamic events, such as the identification 986 of new requirements or the detection of incidents within the virtual 987 system, might urge a VNO to quickly reform its virtual infrastructure 988 and resource allocation. However, the interfaces offered by current 989 virtualization platforms do not offer the necessary functions for 990 VNOs to perform the elastic adaptations they require to tackle with 991 their dynamic operation environments. 993 Beyond their heterogeneity, which can be resolved by software 994 adapters, current virtualization platforms do not have common methods 995 and functions, so it is difficult for the virtual network controllers 996 used by the VNOs to actually manage and control virtual resources 997 instantiated on different platforms, not even considering different 998 InPs. Therefore it is necessary to reach a common definition of the 999 functions that should be offered by underlying platforms to give such 1000 overlay controllers the possibility to allocate and deallocate 1001 resources dynamically and get monitoring data about them. 1003 Such common methods should be offered by all underlying controllers, 1004 regardless of being network-oriented (e.g. ODL, ONOS, Ryu) or 1005 computing-oriented (e.g. OpenStack, OpenNebula, Eucalyptus). 1006 Furthermore, it is also important for those platforms to offer some 1007 "PUSH" function to report resource state, avoiding the need for the 1008 VNO's controller to "POLL" for such data. A starting point to get 1009 proper notifications within current REST APIs could be to consider 1010 the protocol proposed by the WEBPUSH WG [RFC8030]. 1012 Finally, in order to establish a proper order and allow the 1013 coexistence and collaboration of different systems, a common ontology 1014 regarding network and system virtualization should be defined and 1015 agreed, so different and heterogeneous systems can understand each 1016 other without requiring to rely on specific adaptation mechanisms 1017 that might break with any update on any side of the relation. 1019 4.5.2. Extending Virtual Networks and Systems to the Internet of Things 1021 The Internet of Things (IoT) refers to the vision of connecting a 1022 multitude of automated devices (e.g. lights, environmental sensors, 1023 traffic lights, parking meters, health and security systems, etc.) to 1024 the Internet for purposes of reporting, and remote command and 1025 control of the device. This vision is being realized by a multi- 1026 pronged approach of standardization in various forums and 1027 complementary open source activities. For example, in the IETF, 1028 support of IoT web services has been defined by an HTTP-like protocol 1029 adapted for IoT called CoAP [RFC7252], and lately a group has been 1030 studying the need to develop a new network layer to support IP 1031 applications over Low Power Wide Area Networks (LPWAN). 1033 Elsewhere, for 5G cellular evolution there is much discussion on the 1034 need for supporting virtual "network slices" for the expected massive 1035 numbers of IoT devices. A separate virtual network slice is 1036 considered necessary for different 5G IoT use cases because devices 1037 will have very different characteristics than typical cellular 1038 devices like smart phones [ngmn_5G_whitepaper], and the number of IoT 1039 devices is expected to be at least one or two orders of magnitude 1040 higher than other 5G devices (see Section 4.5). 1042 The specific nature of the IoT ecosystem, particularly reflected in 1043 the Machine-to-Machine (M2M) communications, leads to the creation of 1044 new and highly distributed systems which demand location-based 1045 network and computing services. A specific example can be 1046 represented by a set of "things" that suddenly require to set-up a 1047 firewall to allow external entities to access their data while 1048 outsourcing some computation requirements to more powerful systems 1049 relying on cloud-based services. This representative use case 1050 exposes important requirements for both NFV and the underlying cloud 1051 infrastructures. 1053 In order to provide the aforementioned location-based functions 1054 integrated with highly distributed systems, the so called fog 1055 infrastructures should be able to instantiate VNFs, placing them in 1056 the required place, e.g. close to their consumers. This requirement 1057 implies that the interfaces offered by virtualization platforms must 1058 support the specification of location-based resources, which is a key 1059 function in those scenarios. Moreover, those platforms must also be 1060 able to interpret and understand the references used by IoT systems 1061 to their location (e.g., "My-AP", "5BLDG+2F") and also the 1062 specification of identifiers linked to other resources, such as the 1063 case of requiring the infrastructure to establish a link between a 1064 specific AP and a specific virtual computing node. In summary, the 1065 research gap is exact localization of VNFs at far network edge 1066 infrastructure which is highly distributed and dynamic. 1068 4.6. Service Composition 1070 Current network services deployed by operators often involve the 1071 composition of several individual functions (such as packet 1072 filtering, deep packet inspection, load balancing). These services 1073 are typically implemented by the ordered combination of a number of 1074 service functions that are deployed at different points within a 1075 network, not necessarily on the direct data path. This requires 1076 traffic to be steered through the required service functions, 1077 wherever they are deployed [RFC7498]. 1079 For a given service, the abstracted view of the required service 1080 functions and the order in which they are to be applied is called a 1081 Service Function Chain (SFC) [sfc_challenges], which is called 1082 Network Function Forwarding Graph (NF-FG) in ETSI. An SFC is 1083 instantiated through selection of specific service function instances 1084 on specific network nodes to form a service graph: this is called a 1085 Service Function Path (SFP). The service functions may be applied at 1086 any layer within the network protocol stack (network layer, transport 1087 layer, application layer, etc.). 1089 Service composition is a powerful means which can provide significant 1090 benefits when applied in a softwarized network environment. There 1091 are however many research challenges in this area, as for example the 1092 ones related to composition mechanisms and algorithms to enable load 1093 balancing and improve reliability. The service composition should 1094 also act as an enabler to gather information across all hierarchies 1095 (underlays and overlays) of network deployments which may span across 1096 multiple operators, for faster serviceability thus facilitating 1097 accomplishing aforementioned goals of "load balancing and improve 1098 reliability". 1100 As described in [dynamic_chaining], different algorithms can be used 1101 to enable dynamic service composition that optimizes a QoS-based 1102 utility function (e.g., minimizing the latency per-application 1103 traffic flows) for a given composition plan. Such algorithms can 1104 consider the computation capabilities and load status of resources 1105 executing the VNF instances, either deduced through estimations from 1106 historical usage data or collected through real-time monitoring 1107 (i.e., context-aware selection). For this reason, selections should 1108 include references to dynamic information on the status of the 1109 service instance and its constituent elements, i.e., monitoring 1110 information related to individual VNF instances and links connecting 1111 them as well as derived monitoring information at the chain level 1112 (e.g., end-to-end delay). At runtime, if one or more VNF instances 1113 are no more available or QoS degrades below a given threshold, the 1114 service selection task can be rerun to perform service substitution. 1116 There are different research directions that relate to the previous 1117 point. For example, the use of Integer Linear Programming (ILP) 1118 techniques can be explored to optimize the management of diverse 1119 traffic flows. Deep machine learning can also be applied to optimize 1120 service chains using information parameters such as some of the ones 1121 mentioned above. Newer scheduling paradigms, like co-flows, can also 1122 be used. 1124 The SFC working group is working on an architecture for service 1125 function chaining [RFC7665] that includes the necessary protocols or 1126 protocol extensions to convey the Service Function Chain and Service 1127 Function Path information to nodes that are involved in the 1128 implementation of service functions and Service Function Chains, as 1129 well as mechanisms for steering traffic through service functions. 1131 In terms of actual work items, the SFC WG is has not yet considered 1132 working on the management and configuration of SFC components related 1133 to the support of Service Function Chaining. This part is of special 1134 interest for operators and would be required in order to actually put 1135 SFC mechanisms into operation. Similarly, redundancy and reliability 1136 mechanisms for service function chaining are currently not dealt with 1137 by any WG in the IETF. While this was the main goal of the VNFpool 1138 BoF efforts, it still remains unaddressed. 1140 4.7. End-user device virtualization 1142 So far, most of the network softwarization efforts have focused on 1143 virtualizing functions of network elements. While virtualization of 1144 network elements started with the core, mobile networks architectures 1145 are now heavily switching to also virtualize radio access network 1146 (RAN) functions. The next natural step is to get virtualization down 1147 at the level of the end-user device (e.g., virtualizing a smartphone) 1148 [virtualization_mobile_device]. The cloning of a device in the cloud 1149 (central or local) bears attractive benefits to both the device and 1150 network operations alike (e.g., power saving at the device by 1151 offloading computational-heaving functions to the cloud, optimized 1152 networking -- both device-to-device and device-to-infrastructure) for 1153 service delivery through tighter integration of the device (via its 1154 clone in the networking infrastructure). This is, for example, being 1155 explored by the European H2020 ICIRRUS project (www.icirrus- 1156 5gnet.eu). 1158 4.8. Security and Privacy 1160 Similar to any other situation where resources are shared, security 1161 and privacy are two important aspects that need to be taken into 1162 account. 1164 In the case of security, there are situations where multiple vendors 1165 will need to coexist in a virtual or hybrid physical/virtual 1166 environment. This requires attestation procedures amongst different 1167 virtual/physical functions and resources, as well as ongoing external 1168 monitoring. Similarly, different network slices operating on the 1169 same infrastructure can present security problems, for instance if 1170 one slice running critical applications (e.g. support for a safety 1171 system) is affected by another slice running a less critical 1172 application. In general, the minimum common denominator for security 1173 measures on a shared system should be equal or higher than the one 1174 required by the most critical application. Multiple and continuous 1175 threat model analysis, as well as DevOps model are required to 1176 maintain a certain level of security in an NFV system. 1177 Simplistically, DevOps is a process that combines multiple functions 1178 into single cohesive teams in order to quickly produce quality 1179 software. It typically relies on also applying the Agile development 1180 process, which focuses on (among many things) dividing large features 1181 into multiple, smaller deliveries. One part of this is to 1182 immediately test the new smaller features in order to get immediate 1183 feedback on errors so that if present, they can be immediately fixed 1184 and redeployed. 1186 On the other hand, privacy in its strictest interpretation refers to 1187 concerns about exposing users of the system to individual threats 1188 such as surveillance, identification, stored data compromise, 1189 secondary use, intrusion, etc. In this case, the storage, 1190 transmission, collection, and potential correlation of information in 1191 the NFV system, for purposes not originally intended or not known by 1192 the user, should be avoided. This is particularly challenging, as 1193 future intentions and threats cannot be easily predicted, and still 1194 can be applied on data collected in the past. Therefore, well-known 1195 techniques such as data minimization, using privacy features as 1196 default, and allowing users to opt in/out should be used to prevent 1197 potential privacy issues. 1199 Compared to traditional networks, NFV will result in networks that 1200 are much more dynamic (in function distribution and topology) and 1201 elastic (in size and boundaries). NFV will thus require network 1202 operators to evolve their operational and administrative security 1203 solutions to work in this new environment. For example, in NFV the 1204 network orchestrator will become a key node to provide security 1205 policy orchestration across the different physical and virtual 1206 components of the virtualized network. For highly confidential data, 1207 for example, the network orchestrator should take into account if 1208 certain physical hardware (HW) of the network is considered more 1209 secure (e.g., because it is located in secure premises) than other 1210 HW. 1212 Traditional telecom networks typically run under a single 1213 administrative domain controlled by (exactly) one operator. With 1214 NFV, it is expected that in many cases, the telecom operator will now 1215 become a tenant (running the VNFs), and the infrastructure (NFVI) may 1216 be run by a different operator and/or cloud service provider (see 1217 also Section 4.4). Thus, there will be multiple administrative 1218 domains involved, making security policy coordination more complex. 1219 For example, who will be in charge of provisioning and maintaining 1220 security credentials such as public and private keys? Also, should 1221 private keys be allowed to be replicated across the NFV for 1222 redundancy reasons? Alternatively, it can be investigated how to 1223 develop a mechanism that avoid such a security policy coordination, 1224 this making the system more robust. 1226 On a positive note, NFV will may better defense against Denial of 1227 Service (DoS) attacks because of the distributed nature of the 1228 network (i.e. no single point of failure) and the ability to steer 1229 (undesirable) traffic quickly [etsi_gs_nfv_sec_001]. Also, NFVs 1230 which have physical HW which is distributed across multiple data 1231 centers will also provide better fault isolation environments. This 1232 holds true in particular if each data center is protected separately 1233 via firewalls, DMZs and other network protection techniques. 1235 SDN can also be used to help improve security by facilitating the 1236 operation of existing protocols, such as Authentication, 1237 Authorization and Accounting (AAA). The management of AAA 1238 infrastructures, namely the management of AAA routing and the 1239 establishment of security associations between AAA entities, can be 1240 performed using SDN, as analyzed in [I-D.marin-sdnrg-sdn-aaa-mng]. 1242 4.9. Separation of control concerns 1244 NFV environments offer two possible levels of SDN control. One level 1245 is the need for controlling the NFVI to provide connectivity end-to- 1246 end among VNFs or among VNFs and PNFs (Physical Network Functions). 1247 A second level is the control and configuration of the VNFs 1248 themselves (in other words, the configuration of the network service 1249 implemented by those VNFs), taking advantage of the programmability 1250 brought by SDN. Both control concerns are separated in nature. 1251 However, interaction between both could be expected in order to 1252 optimize, scale or influence each other. 1254 Clear mechanisms for such interaction are needed in order to avoid 1255 malfunctioning or interference concerns. These ideas are considered 1256 in [etsi_gs_nfv_eve005] and [I-D.irtf-sdnrg-layered-sdn] 1258 4.10. Network Function placement 1260 Network function placement is a problem in any kind of network 1261 telecommunications infrastructure. Moreover, the increased degree of 1262 freedom added by network virtualization makes this problem even more 1263 important, and also harder to tackle. Deciding where to place 1264 virtual network functions is a resource allocation problem which 1265 needs to (or may) take into consideration quite a few aspects: 1266 resiliency, (anti-)affinity, security, privacy, energy efficiency, 1267 etc. 1269 When several functions are chained (typical scenario), placement 1270 algorithms become more complex and important (as described in 1271 Section 4.6). While there has been research on the topic 1272 [nfv_piecing] [dynamic_placement][vnf-p], this still remains an open 1273 challenges that requires more attention. Multi-domain also adds 1274 another component of complexity to this problem that has to be 1275 considered. 1277 4.11. Testing 1279 The impacts of network virtualization on testing can be divided into 1280 3 groups: 1282 1. Changes in methodology. 1284 2. New functionality. 1286 3. Opportunities. 1288 4.11.1. Changes in methodology 1290 The largest impact of NFV is the ability to isolate the System Under 1291 Test (SUT). When testing Physical Network Functions (PNF), isolating 1292 the SUT means that all the other devices that the SUT communicates 1293 with are replaced with simulations (or controlled executions) in 1294 order to place the SUT under test by itself. The SUT may be 1295 comprised of one or more devices. The simulations use the 1296 appropriate traffic type and protocols in order to execute test 1297 cases. 1299 As shown in Figure 2, NFV provides a common architecture for all 1300 functions to use. A VNF is executed using resources offered by the 1301 NFVI, which have been allocated using the MANO function. It is not 1302 possible to test a VNF by itself, without the entire supporting 1303 environment present. This fundamentally changes how to consider the 1304 SUT. In the case of a VNF (or multiple VNFs), the SUT is part of a 1305 larger architecture which is necessary in order to run the SUTs. 1307 Isolation of the SUT therefore becomes controlling the environment in 1308 a disciplined manner. The components of the environment necessary to 1309 run the SUTs that are not part of the SUT become the test 1310 environment. In the case of VNFs which are the SUT, the NFVI and 1311 MANO become the test environment. The configurations and policies 1312 that guide the test environment should remain constant during the 1313 execution of the tests, and also from test to test. Configurations 1314 such as CPU pinning, NUMA configuration, the SW versions and 1315 configurations of the hypervisor, vSwitch and NICs should remain 1316 constant. The only variables in the testing should be those 1317 controlling the SUT itself. If any configuration in the test 1318 environment is changed from test to test, the results become very 1319 difficult, if not impossible, to compare since the test environment 1320 behavior may change the results as a consequence of the configuration 1321 change. 1323 Testing the NFVI itself also presents new considerations. With a 1324 PNF, the dedicated hardware supporting it is optimized for the 1325 particular workload of the function. Routing hardware is specially 1326 built to support packet forwarding functions, while the hardware to 1327 support a purely control plane application (say, a DNS server, or a 1328 Diameter function) will not have this specialized capability. In 1329 NFV, the NFVI is required to support all types of potentially 1330 different workload types. 1332 Testing the NFVI therefore requires careful consideration about what 1333 types of metrics are sought. This, in turn, depends on the workload 1334 type the expected VNF will be. Examples of different workload types 1335 are data forwarding, control plane, encryption, and authentication. 1336 All these types of expected workloads will determine the types of 1337 metrics that should be sought. For example, if the workload is 1338 control plane, then a metric such as jitter is not useful, but 1339 dropped packets are critical. In a multi-tenant environment, the 1340 NFVI could support various types of workloads. In this case, testing 1341 with a variety of traffic types while measuring the corresponding 1342 metrics simultaneously becomes necessary. 1344 4.11.2. New functionality 1346 NFV presents a collection of new functionality in order to support 1347 the goal of software networking. Each component on the architecture 1348 shown in Figure 2 has an associated set of functionality that allows 1349 VNFs to run: onboarding, lifecycle management for VNFs and Networks 1350 Services (NS), resource allocation, hypervisor functions, etc. 1352 One of the new capabilities enabled by NFV is VNFFG (VNF Forwarding 1353 Graphs). This refers to the graph that represents a Network Service 1354 by chaining together VNFs into a forwarding path. In practice, the 1355 forwarding path can be implemented in a variety of ways using 1356 different networking capabilities: vSwitch, SDN, SDN with a 1357 northbound application, and the VNFFG might use tunneling protocols 1358 like VXLAN. The dynamic allocation and implementation of these 1359 networking paths will have different performance characteristics 1360 depending on the methods used. The path implementation mechanism 1361 becomes a variable in the network testing of the NSs. The 1362 methodology used to test the various mechanisms should largely remain 1363 the same, and as usual, the test environment should remain constant 1364 for each of the tests, focusing on varying the path establishment 1365 method. 1367 Scaling refers to the change in allocation of resources to a VNF or 1368 NS. It happens dynamically at run-time, based on defined policies 1369 and triggers. The triggers can be network, compute or storage based. 1370 Scaling can allocate more resources in times of need, or reduce the 1371 amount of resources allocated when the demand is reduced. The SUT in 1372 this case becomes much larger than the VNF itself: MANO controls how 1373 scaling is done based on policies, and then allocates the resources 1374 accordingly in the NFVI. Essentially, the testing of scaling 1375 includes the entire NFV architecture components into the SUT. 1377 4.11.3. Opportunities 1379 Softwarization of networking functionality leads to softwarization of 1380 test as well. As Physical Network Functions (PNF) are being 1381 transformed into VNFs, so have the test tools. This leads to the 1382 fact that test tools are also being controlled and executed in the 1383 same environment as the VNFs are. This presents an opportunity to 1384 include VNF-based test tools along with the deployment of the VNFs 1385 supporting the services of the service provider into the host data 1386 centers. Tests can therefore be automatically executed upon 1387 deployment in the target environment, for each deployment, and each 1388 service. With PNFs, this was very difficult to achieve. 1390 This new concept helps to enable modern concepts like DevOps and 1391 Continuous Integration and Continuous Deployment in the NFV 1392 environment. The CI/CD pipeline supports this concept. It consists 1393 of a series of tools, among which immediate testing is an integral 1394 part, to deliver software from source to deployment. The ability to 1395 deploy the test tools themselves into the production environment 1396 stretches the CI/CD pipeline all the way to production deployment, 1397 allowing a range of tests to be executed. The tests can be simple, 1398 with a goal of verifying the correct deployment and networking 1399 establishment, but can also be more complex, like testing VNF 1400 functionality. 1402 5. Technology Gaps and Potential IETF Efforts 1404 Table 1 correlates the open network virtualization research areas 1405 identified in this document to potential IETF and IRTF groups that 1406 could address some aspects of them. An example of a specific gap 1407 that the group could potentially address is identified in 1408 parenthetical beside the group name. 1410 +-------------------------+-----------------------------------------+ 1411 | Open Research Area | Potential IETF/IRTF Group | 1412 +-------------------------+-----------------------------------------+ 1413 | 1-Guaranteeing QoS | IPPM WG (Measurements of NFVI) | 1414 | 2-Performance | SFC WG, NFVRG (energy driven | 1415 | improvement | orchestration) | 1416 | 3-Multiple Domains | NFVRG (multi-domain orchestration) | 1417 | 4-Network Slicing | NVO3 WG, NETSLICES bar BoF (multi- | 1418 | | tenancy support) | 1419 | 5-Service Composition | SFC WG (SFC Mgmt and Config) | 1420 | 6-End-user device | N/A | 1421 | virtualization | | 1422 | 7-Security | N/A | 1423 | 8-Separation of control | NFVRG (separation between transport | 1424 | concerns | control and services) | 1425 | 9-Testing | NFVRG (testing of scaling) | 1426 | 10-Function placement | NFVRG, SFC WG (VNF placement algorithms | 1427 | | and protocols) | 1428 +-------------------------+-----------------------------------------+ 1430 Table 1: Mapping of Open Research Areas to Potential IETF Groups 1432 6. NFVRG focus areas 1434 Table 2 correlates the currently identified NFVRG topics of 1435 interests/focus areas to the open network virtualization research 1436 areas enumerated in this document. This can help the NFVRG in 1437 identifying and prioritizing research topics. The current list of 1438 NFVRG focus points is the following: 1440 o Re-architecting functions, including aspects such as new 1441 architectural and design patterns (e.g., containerization, 1442 statelessness, serverless, control/data plane separation), SDN 1443 integration, and proposals on programmability. 1445 o New management frameworks, considering aspects related to new OAM 1446 mechanisms (e.g., configuration control, hybrid descriptors) and 1447 lightweight MANO proposals. 1449 o Techniques to guarantee low latency, resource isolation, and other 1450 dataplane features, including hardware acceleration, functional 1451 offloading to dataplane elements (including NICs), and related 1452 approaches. 1454 o Measurement and benchmarking, addressing both internal 1455 measurements and external applications. 1457 +----------------------------------------+-------------------------+ 1458 | NFVRG Focus Point | Open Research Area | 1459 +----------------------------------------+-------------------------+ 1460 | 1-Re-architecting functions | - Performance improvem. | 1461 | | - Network Slicing | 1462 | | - Guaranteeing QoS | 1463 | | - Security | 1464 | | - End-user device virt. | 1465 | | - Separation of control | 1466 | 2-New management frameworks | - Multiple Domains | 1467 | | - Service Composition | 1468 | | - End-user device virt. | 1469 | 3-Low latency, resource isolation, etc | - Performance improvem. | 1470 | | - Separation of control | 1471 | 4-Measurement and benchmarking | - Guaranteeing QoS | 1472 | | - Testing | 1473 +----------------------------------------+-------------------------+ 1475 Table 2: Mapping of NFVRG Focus Points to Open Research Areas 1477 7. IANA Considerations 1479 N/A. 1481 8. Security Considerations 1483 This is an informational document, which therefore does not introduce 1484 any security threat. Research challenges and gaps related to 1485 security and privacy have been included in Section 4.8. 1487 9. Acknowledgments 1489 The authors want to thank Dirk von Hugo, Rafa Marin, Diego Lopez, 1490 Ramki Krishnan, Kostas Pentikousis, Rana Pratap Sircar, Alfred 1491 Morton, Nicolas Kuhn, Saumya Dikshit, Fabio Giust, Evangelos 1492 Haleplidis, Angeles Vazquez-Castro, Barbara Martini, Jose Saldana and 1493 Gino Carrozzo for their very useful reviews and comments to the 1494 document. Special thanks to Pedro Martinez-Julia, who provided text 1495 for the network slicing section. 1497 The authors want to also thank Dave Oran and Michael Welzl for their 1498 very detailed IRSG reviews. 1500 The work of Carlos J. Bernardos and Luis M. Contreras is partially 1501 supported by the H2020 5GEx (Grant Agreement no. 671636) and 5G- 1502 TRANSFORMER (Grant Agreement no. 761536) projects. 1504 10. Informative References 1506 [dynamic_chaining] 1507 Martini, B. and F. Paganelli, "A Service-Oriented Approach 1508 for Dynamic Chaining of Virtual Network Functions over 1509 Multi-Provider Software-Defined Networks", Future 1510 Internet vol. 8, no. 2, June 2016. 1512 [dynamic_placement] 1513 Clayman, S., Maini, E., and A. Galis, "The dynamic 1514 placement of virtual network functions", 2014 IEEE Network 1515 Operations and Management Symposium (NOMS) pp. 1-9, May 1516 2014. 1518 [etsi_gs_nfv_003] 1519 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1520 Terminology for Main Concepts in NFV", ETSI GS NFV 003 1521 V1.2.1 NFV 003, December 2014, 1522 . 1525 [etsi_gs_nfv_eve005] 1526 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1527 Ecosystem; Report on SDN Usage in NFV Architectural 1528 Framework", ETSI GS NFV-EVE 005 V1.1.1 NFV-EVE 005, 1529 December 2015, . 1532 [etsi_gs_nfv_per_001] 1533 ETSI GS NFV-PER 001 V1.1.2, , "Network Functions 1534 Virtualisation (NFV); NFV Performance & Portability Best 1535 Practises", ETSI GS NFV-PER 001 V1.1.2 NFV-PER 001, 1536 December 2014, . 1539 [etsi_gs_nfv_sec_001] 1540 ETSI GS NFV-SEC 001 V1.1.1, , "Network Functions 1541 Virtualisation (NFV); NFV Security; Problem Statement", 1542 ETSI GS NFV-SEC 001 V1.1.1 NFV-SEC 001, October 2014, 1543 . 1546 [etsi_nvf_whitepaper_3] 1547 "Network Functions Virtualisation (NFV). White Paper 3", 1548 October 2014. 1550 [google_sdn_wan] 1551 Sushant Jain et al., , "B4: experience with a globally- 1552 deployed Software Defined WAN", Proceedings of the ACM 1553 SIGCOMM 2013 , August 2013. 1555 [I-D.bagnulo-nfvrg-topology] 1556 Bagnulo, M. and D. Dolson, "NFVI PoP Network Topology: 1557 Problem Statement", draft-bagnulo-nfvrg-topology-01 (work 1558 in progress), March 2016. 1560 [I-D.bernardos-nfvrg-multidomain] 1561 Bernardos, C., Contreras, L., Vaishnavi, I., and R. Szabo, 1562 "Multi-domain Network Virtualization", draft-bernardos- 1563 nfvrg-multidomain-03 (work in progress), September 2017. 1565 [I-D.defoy-netslices-3gpp-network-slicing] 1566 Foy, X. and A. Rahman, "Network Slicing - 3GPP Use Case", 1567 draft-defoy-netslices-3gpp-network-slicing-02 (work in 1568 progress), October 2017. 1570 [I-D.galis-netslices-revised-problem-statement] 1571 Galis, A., "Network Slicing - Revised Problem Statement", 1572 draft-galis-netslices-revised-problem-statement-01 (work 1573 in progress), July 2017. 1575 [I-D.gdmb-netslices-intro-and-ps] 1576 Galis, A., Dong, J., kiran.makhijani@huawei.com, k., 1577 Bryant, S., Boucadair, M., and P. Martinez-Julia, "Network 1578 Slicing - Introductory Document and Revised Problem 1579 Statement", draft-gdmb-netslices-intro-and-ps-02 (work in 1580 progress), February 2017. 1582 [I-D.irtf-sdnrg-layered-sdn] 1583 Contreras, L., Bernardos, C., Lopez, D., Boucadair, M., 1584 and P. Iovanna, "Cooperating Layered Architecture for 1585 SDN", draft-irtf-sdnrg-layered-sdn-01 (work in progress), 1586 October 2016. 1588 [I-D.marin-sdnrg-sdn-aaa-mng] 1589 Lopez, R. and G. Lopez-Millan, "Software-Defined 1590 Networking (SDN)-based AAA Infrastructures Management", 1591 draft-marin-sdnrg-sdn-aaa-mng-00 (work in progress), 1592 November 2015. 1594 [I-D.mlk-nfvrg-nfv-reliability-using-cots] 1595 Mo, L. and B. Khasnabish, "NFV Reliability using COTS 1596 Hardware", draft-mlk-nfvrg-nfv-reliability-using-cots-01 1597 (work in progress), October 2015. 1599 [I-D.natarajan-nfvrg-containers-for-nfv] 1600 natarajan.sriram@gmail.com, n., Krishnan, R., Ghanwani, 1601 A., Krishnaswamy, D., Willis, P., Chaudhary, A., and F. 1602 Huici, "An Analysis of Lightweight Virtualization 1603 Technologies for NFV", draft-natarajan-nfvrg-containers- 1604 for-nfv-03 (work in progress), July 2016. 1606 [I-D.rorosz-nfvrg-vbaas] 1607 Rosa, R., Rothenberg, C., and R. Szabo, "VNF Benchmark-as- 1608 a-Service", draft-rorosz-nfvrg-vbaas-00 (work in 1609 progress), October 2015. 1611 [intel_10_differences_nfv_cloud] 1612 Torre, P., "Discover the Top 10 Differences Between NFV 1613 and Cloud Environments", November 2015, 1614 . 1617 [itu-t-y.3300] 1618 ITU-T, "Y.3300: Framework of software-defined networking", 1619 ITU-T Recommendation Y.3300 (06/14), June 2014, 1620 . 1622 [itu-t-y.3301] 1623 ITU-T, "Y.3301: Functional requirements of software- 1624 defined networking", ITU-T Recommendation Y.3301 (09/16), 1625 September 2016, 1626 . 1628 [itu-t-y.3302] 1629 ITU-T, "Y.3302: Functional architecture of software- 1630 defined networking", ITU-T Recommendation Y.3302 (01/17), 1631 January 2017, 1632 . 1634 [multi-domain_5GEx] 1635 Bernardos, C., Geroe, B., Di Girolamo, M., Kern, A., 1636 Martini, B., and I. Vaishnavi, "5GEx: Realizing a Europe 1637 wide Multi-domain Framework for Software Defined 1638 Infrastructures", Transactions on Emerging 1639 Telecommunications Technologies vol. 27, no. 9, pp. 1640 1271-1280, September 2016. 1642 [nfv_piecing] 1643 Luizelli, M., Bays, L., and L. Buriol, "Piecing together 1644 the NFV provisioning puzzle: Efficient placement and 1645 chaining of virtual network functions", 2015 IFIP/IEEE 1646 International Symposium on Integrated Network Management 1647 (IM) pp. 98-106, May 2015. 1649 [nfv_sota_research_challenges] 1650 Mijumbi, R., Serrat, J., Gorricho, J-L., Bouten, N., De 1651 Turck, F., and R. Boutaba, "Network Function 1652 Virtualization: State-of-the-art and Research Challenges", 1653 IEEE Communications Surveys & Tutorials Volume: 18, Issue: 1654 1, September 2015. 1656 [ngmn_5G_whitepaper] 1657 "NGMN 5G. White Paper", February 2015. 1659 [omniran] 802.1cf, Draft 0.4, , "802.1CF Network Reference Model and 1660 Functional Description of IEEE 802 Access Network", 1661 802.1cf, Draft 0.4 802.1cf, February 2017. 1663 [onf_tr_521] 1664 ONF, "SDN Architecture, Issue 1.1", ONF TR-521 TR-521, 1665 February 2016, 1666 . 1670 [openmano_dataplane] 1671 Lopez, D., "OpenMANO: The Dataplane Ready Open Source NFV 1672 MANO Stack", March 2015, 1673 . 1676 [RFC5810] Doria, A., Ed., Hadi Salim, J., Ed., Haas, R., Ed., 1677 Khosravi, H., Ed., Wang, W., Ed., Dong, L., Gopal, R., and 1678 J. Halpern, "Forwarding and Control Element Separation 1679 (ForCES) Protocol Specification", RFC 5810, 1680 DOI 10.17487/RFC5810, March 2010, . 1683 [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., 1684 and A. Bierman, Ed., "Network Configuration Protocol 1685 (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, 1686 . 1688 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 1689 Application Protocol (CoAP)", RFC 7252, 1690 DOI 10.17487/RFC7252, June 2014, . 1693 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1694 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1695 Defined Networking (SDN): Layers and Architecture 1696 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1697 2015, . 1699 [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for 1700 Service Function Chaining", RFC 7498, 1701 DOI 10.17487/RFC7498, April 2015, . 1704 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1705 Chaining (SFC) Architecture", RFC 7665, 1706 DOI 10.17487/RFC7665, October 2015, . 1709 [RFC8030] Thomson, M., Damaggio, E., and B. Raymor, Ed., "Generic 1710 Event Delivery Using HTTP Push", RFC 8030, 1711 DOI 10.17487/RFC8030, December 2016, . 1714 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 1715 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 1716 . 1718 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 1719 Network Functions and Their Infrastructure", RFC 8172, 1720 DOI 10.17487/RFC8172, July 2017, . 1723 [sfc_challenges] 1724 Medhat, A., Taleb, T., Elmangoush, A., Carella, G., 1725 Covaci, S., and T. Magedanz, "Service Function Chaining in 1726 Next Generation Networks: State of the Art and Research 1727 Challenges", IEEE Communications Magazine vol. 55, no. 2, 1728 pp. 216-223, February 2017. 1730 [virtualization_mobile_device] 1731 William D. Sproule, , "Virtualization of Mobile Device 1732 User Experience", Patent US 9.542.062 B2 , January 2017. 1734 [vnf-p] Moens, H. and Filip De Turck, "VNF-P: A model for 1735 efficient placement of virtualized network functions", 1736 10th International Conference on Network and Service 1737 Management (CNSM) and Workshop pp. 418-423, 2014. 1739 [vnf_benchmarking] 1740 Rosa, R., Rothenberg, C., and R. Szabo, "A VNF Testing 1741 Framework Design, Implementation and Partial Results", 1742 November 2016, 1743 . 1746 Authors' Addresses 1748 Carlos J. Bernardos 1749 Universidad Carlos III de Madrid 1750 Av. Universidad, 30 1751 Leganes, Madrid 28911 1752 Spain 1754 Phone: +34 91624 6236 1755 Email: cjbc@it.uc3m.es 1756 URI: http://www.it.uc3m.es/cjbc/ 1757 Akbar Rahman 1758 InterDigital Communications, LLC 1759 1000 Sherbrooke Street West, 10th floor 1760 Montreal, Quebec H3A 3G4 1761 Canada 1763 Email: Akbar.Rahman@InterDigital.com 1764 URI: http://www.InterDigital.com/ 1766 Juan Carlos Zuniga 1767 SIGFOX 1768 425 rue Jean Rostand 1769 Labege 31670 1770 France 1772 Email: j.c.zuniga@ieee.org 1773 URI: http://www.sigfox.com/ 1775 Luis M. Contreras 1776 Telefonica I+D 1777 Ronda de la Comunicacion, S/N 1778 Madrid 28050 1779 Spain 1781 Email: luismiguel.contrerasmurillo@telefonica.com 1783 Pedro Aranda 1784 Universidad Carlos III de Madrid 1785 Av. Universidad, 30 1786 Leganes, Madrid 28911 1787 Spain 1789 Email: pedroandres.aranda@uc3m.es 1791 Pierre Lynch 1792 Ixia 1794 Email: plynch@ixiacom.com