idnits 2.17.1 draft-irtf-nfvrg-gaps-network-virtualization-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 3, 2017) is 2482 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-05) exists of draft-bernardos-nfvrg-multidomain-02 == Outdated reference: A later version (-02) exists of draft-defoy-netslices-3gpp-network-slicing-01 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFVRG CJ. Bernardos 3 Internet-Draft UC3M 4 Intended status: Informational A. Rahman 5 Expires: January 4, 2018 InterDigital 6 JC. Zuniga 7 SIGFOX 8 LM. Contreras 9 TID 10 P. Aranda 11 UC3M 12 P. Lynch 13 Ixia 14 July 3, 2017 16 Network Virtualization Research Challenges 17 draft-irtf-nfvrg-gaps-network-virtualization-06 19 Abstract 21 This document describes open research challenges for network 22 virtualization. Network virtualization is following a similar path 23 as previously taken by cloud computing. Specifically, cloud 24 computing popularized migration of computing functions (e.g., 25 applications) and storage from local, dedicated, physical resources 26 to remote virtual functions accessible through the Internet. In a 27 similar manner, network virtualization is encouraging migration of 28 networking functions from dedicated physical hardware nodes to a 29 virtualized pool of resources. However, network virtualization can 30 be considered to be a more complex problem than cloud computing as it 31 not only involves virtualization of computing and storage functions 32 but also involves abstraction of the network itself. This document 33 describes current research challenges in network virtualization 34 including guaranteeing quality-of-service, performance improvement, 35 supporting multiple domains, network slicing, service composition, 36 device virtualization, privacy and security, separation of control 37 concerns, network function placement and testing. In addition, some 38 proposals are made for new activities in IETF/IRTF that could address 39 some of these challenges. 41 Status of This Memo 43 This Internet-Draft is submitted in full conformance with the 44 provisions of BCP 78 and BCP 79. 46 Internet-Drafts are working documents of the Internet Engineering 47 Task Force (IETF). Note that other groups may also distribute 48 working documents as Internet-Drafts. The list of current Internet- 49 Drafts is at http://datatracker.ietf.org/drafts/current/. 51 Internet-Drafts are draft documents valid for a maximum of six months 52 and may be updated, replaced, or obsoleted by other documents at any 53 time. It is inappropriate to use Internet-Drafts as reference 54 material or to cite them other than as "work in progress." 56 This Internet-Draft will expire on January 4, 2018. 58 Copyright Notice 60 Copyright (c) 2017 IETF Trust and the persons identified as the 61 document authors. All rights reserved. 63 This document is subject to BCP 78 and the IETF Trust's Legal 64 Provisions Relating to IETF Documents 65 (http://trustee.ietf.org/license-info) in effect on the date of 66 publication of this document. Please review these documents 67 carefully, as they describe your rights and restrictions with respect 68 to this document. Code Components extracted from this document must 69 include Simplified BSD License text as described in Section 4.e of 70 the Trust Legal Provisions and are provided without warranty as 71 described in the Simplified BSD License. 73 Table of Contents 75 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 76 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 77 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 78 3.1. Network Function Virtualization . . . . . . . . . . . . . 5 79 3.2. Software Defined Networking . . . . . . . . . . . . . . . 7 80 3.3. ITU-T functional architecture of SDN . . . . . . . . . . 12 81 3.4. Multi-access Edge Computing . . . . . . . . . . . . . . . 13 82 3.5. IEEE 802.1CF (OmniRAN) . . . . . . . . . . . . . . . . . 14 83 3.6. Distributed Management Task Force . . . . . . . . . . . . 14 84 3.7. Open Source initiatives . . . . . . . . . . . . . . . . . 14 85 4. Network Virtualization Challenges . . . . . . . . . . . . . . 16 86 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 16 87 4.2. Guaranteeing quality-of-service . . . . . . . . . . . . . 16 88 4.2.1. Virtualization Technologies . . . . . . . . . . . . . 17 89 4.2.2. Metrics for NFV characterization . . . . . . . . . . 17 90 4.2.3. Predictive analysis . . . . . . . . . . . . . . . . . 18 91 4.2.4. Portability . . . . . . . . . . . . . . . . . . . . . 19 92 4.3. Performance improvement . . . . . . . . . . . . . . . . . 19 93 4.3.1. Energy Efficiency . . . . . . . . . . . . . . . . . . 19 94 4.3.2. Improved link usage . . . . . . . . . . . . . . . . . 20 95 4.4. Multiple Domains . . . . . . . . . . . . . . . . . . . . 20 96 4.5. 5G and Network Slicing . . . . . . . . . . . . . . . . . 20 97 4.5.1. Virtual Network Operators . . . . . . . . . . . . . . 21 98 4.5.2. Extending Virtual Networks and Systems to the 99 Internet of Things . . . . . . . . . . . . . . . . . 22 100 4.6. Service Composition . . . . . . . . . . . . . . . . . . . 23 101 4.7. End-user device virtualization . . . . . . . . . . . . . 25 102 4.8. Security and Privacy . . . . . . . . . . . . . . . . . . 25 103 4.9. Separation of control concerns . . . . . . . . . . . . . 27 104 4.10. Network Function placement . . . . . . . . . . . . . . . 27 105 4.11. Testing . . . . . . . . . . . . . . . . . . . . . . . . . 27 106 4.11.1. Changes in methodology . . . . . . . . . . . . . . . 28 107 4.11.2. New functionality . . . . . . . . . . . . . . . . . 29 108 4.11.3. Opportunities . . . . . . . . . . . . . . . . . . . 30 109 5. Technology Gaps and Potential IETF Efforts . . . . . . . . . 30 110 6. NFVRG focus areas . . . . . . . . . . . . . . . . . . . . . . 31 111 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 112 8. Security Considerations . . . . . . . . . . . . . . . . . . . 32 113 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32 114 10. Informative References . . . . . . . . . . . . . . . . . . . 33 115 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 37 117 1. Introduction 119 The telecommunications sector is experiencing a major revolution that 120 will shape the way networks and services are designed and deployed 121 for the next few decades. In order to cope with continuously 122 increasing demand and cost, network operators are taking lessons from 123 the IT paradigm of cloud computing. This new approach of 124 virtualizing network functions will enable multi-fold advantages by 125 outsourcing communication services from bespoke hardware in the 126 operator's core network to Commercial off-the-shelf (COTS) equipment 127 distributed across datacenters. 129 Some of the network virtualization mechanisms that are being 130 considered include: sharing of network infrastructure to reduce 131 costs, virtualization of core servers running in data centers as a 132 way of supporting their load-aware elastic dimensioning, and dynamic 133 energy policies to reduce the electricity consumption. 135 This document presents research challenges in Network Function 136 Virtualization (NFV) that need to be addressed in order to achieve 137 these goals. The objective of this memo is to document the technical 138 challenges and corresponding current approaches and to expose 139 requirements that should be addressed by future research and 140 standards work. 142 2. Terminology 144 The following terms used in this document are defined by the ETSI NVF 145 ISG [etsi_gs_nfv_003], the ONF [onf_tr_521] and the IETF [RFC7426] 146 [RFC7665]: 148 Application Plane - The collection of applications and services 149 that program network behavior. 151 Control Plane (CP) - The collection of functions responsible for 152 controlling one or more network devices. CP instructs network 153 devices with respect to how to process and forward packets. The 154 control plane interacts primarily with the forwarding plane and, 155 to a lesser extent, with the operational plane. 157 Forwarding Plane (FP) - The collection of resources across all 158 network devices responsible for forwarding traffic. 160 Management Plane (MP) - The collection of functions responsible 161 for monitoring, configuring, and maintaining one or more network 162 devices or parts of network devices. The management plane is 163 mostly related to the operational plane (it is related less to the 164 forwarding plane). 166 NFV Infrastructure (NFVI): totality of all hardware and software 167 components which build up the environment in which VNFs are 168 deployed. 170 NFV Management and Orchestration (NFV-MANO): functions 171 collectively provided by NFVO, VNFM, and VIM. 173 NFV Orchestrator (NFVO): functional block that manages the Network 174 Service (NS) lifecycle and coordinates the management of NS 175 lifecycle, VNF lifecycle (supported by the VNFM) and NFVI 176 resources (supported by the VIM) to ensure an optimized allocation 177 of the necessary resources and connectivity. 179 Operational Plane (OP) - The collection of resources responsible 180 for managing the overall operation of individual network devices. 182 Physical Network Function (PNF): Physical implementation of a 183 Network Function in a monolithic realization. 185 Service Function Chain (SFC): for a given service, the abstracted 186 view of the required service functions and the order in which they 187 are to be applied. This is somehow equivalent to the Network 188 Function Forwarding Graph (NF-FG) at ETSI. 190 Service Function Path (SFP): the selection of specific service 191 function instances on specific network nodes to form a service 192 graph through which an SFC is instantiated. 194 Virtualized Infrastructure Manager (VIM): functional block that is 195 responsible for controlling and managing the NFVI compute, storage 196 and network resources, usually within one operator's 197 Infrastructure Domain. 199 Virtualized Network Function (VNF): implementation of a Network 200 Function that can be deployed on a Network Function Virtualization 201 Infrastructure (NFVI). 203 Virtualized Network Function Manager (VNFM): functional block that 204 is responsible for the lifecycle management of VNF. 206 3. Background 208 This section briefly describes some basic background technologies, as 209 well as other standards developing organizations and open source 210 initiatives working on network virtualization or related topics. 212 3.1. Network Function Virtualization 214 The ETSI ISG NFV is a working group which, since 2012, aims to evolve 215 quasi-standard IT virtualization technology to consolidate many 216 network equipment types into industry standard high volume servers, 217 switches, and storage. It enables implementing network functions in 218 software that can run on a range of industry standard server hardware 219 and can be moved to, or loaded in, various locations in the network 220 as required, without the need to install new equipment. The ETSI NFV 221 is one of the predominant NFV reference framework and architectural 222 footprints [nfv_sota_research_challenges]. The ETSI NFV framework 223 architecture framework is composed of three domains (Figure 1): 225 o Virtualized Network Function, running over the NFVI. 227 o NFV Infrastructure (NFVI), including the diversity of physical 228 resources and how these can be virtualized. NFVI supports the 229 execution of the VNFs. 231 o NFV Management and Orchestration, which covers the orchestration 232 and life-cycle management of physical and/or software resources 233 that support the infrastructure virtualization, and the life-cycle 234 management of VNFs. NFV Management and Orchestration focuses on 235 all virtualization specific management tasks necessary in the NFV 236 framework. 238 +-------------------------------------------+ +---------------+ 239 | Virtualized Network Functions (VNFs) | | | 240 | ------- ------- ------- ------- | | | 241 | | | | | | | | | | | | 242 | | VNF | | VNF | | VNF | | VNF | | | | 243 | | | | | | | | | | | | 244 | ------- ------- ------- ------- | | | 245 +-------------------------------------------+ | | 246 | | 247 +-------------------------------------------+ | | 248 | NFV Infrastructure (NFVI) | | NFV | 249 | ----------- ----------- ----------- | | Management | 250 | | Virtual | | Virtual | | Virtual | | | and | 251 | | Compute | | Storage | | Network | | | Orchestration | 252 | ----------- ----------- ----------- | | | 253 | +---------------------------------------+ | | | 254 | | Virtualization Layer | | | | 255 | +---------------------------------------+ | | | 256 | +---------------------------------------+ | | | 257 | | ----------- ----------- ----------- | | | | 258 | | | Compute | | Storage | | Network | | | | | 259 | | ----------- ----------- ----------- | | | | 260 | | Hardware resources | | | | 261 | +---------------------------------------+ | | | 262 +-------------------------------------------+ +---------------+ 264 Figure 1: ETSI NFV framework 266 The NFV architectural framework identifies functional blocks and the 267 main reference points between such blocks. Some of these are already 268 present in current deployments, whilst others might be necessary 269 additions in order to support the virtualization process and 270 consequent operation. The functional blocks are (Figure 2): 272 o Virtualized Network Function (VNF). 274 o Element Management (EM). 276 o NFV Infrastructure, including: Hardware and virtualized resources, 277 and Virtualization Layer. 279 o Virtualized Infrastructure Manager(s) (VIM). 281 o NFV Orchestrator. 283 o VNF Manager(s). 285 o Service, VNF and Infrastructure Description. 287 o Operations and Business Support Systems (OSS/BSS). 289 +--------------------+ 290 +-------------------------------------------+ | ---------------- | 291 | OSS/BSS | | | NFV | | 292 +-------------------------------------------+ | | Orchestrator +-- | 293 | ---+------------ | | 294 +-------------------------------------------+ | | | | 295 | --------- --------- --------- | | | | | 296 | | EM 1 | | EM 2 | | EM 3 | | | | | | 297 | ----+---- ----+---- ----+---- | | ---+---------- | | 298 | | | | |--|-| VNF | | | 299 | ----+---- ----+---- ----+---- | | | manager(s) | | | 300 | | VNF 1 | | VNF 2 | | VNF 3 | | | ---+---------- | | 301 | ----+---- ----+---- ----+---- | | | | | 302 +------|-------------|-------------|--------+ | | | | 303 | | | | | | | 304 +------+-------------+-------------+--------+ | | | | 305 | NFV Infrastructure (NFVI) | | | | | 306 | ----------- ----------- ----------- | | | | | 307 | | Virtual | | Virtual | | Virtual | | | | | | 308 | | Compute | | Storage | | Network | | | | | | 309 | ----------- ----------- ----------- | | ---+------ | | 310 | +---------------------------------------+ | | | | | | 311 | | Virtualization Layer | |--|-| VIM(s) +-------- | 312 | +---------------------------------------+ | | | | | 313 | +---------------------------------------+ | | ---------- | 314 | | ----------- ----------- ----------- | | | | 315 | | | Compute | | Storage | | Network | | | | | 316 | | | hardware| | hardware| | hardware| | | | | 317 | | ----------- ----------- ----------- | | | | 318 | | Hardware resources | | | NFV Management | 319 | +---------------------------------------+ | | and Orchestration | 320 +-------------------------------------------+ +--------------------+ 322 Figure 2: ETSI NFV reference architecture 324 3.2. Software Defined Networking 326 The Software Defined Networking (SDN) paradigm pushes the 327 intelligence currently residing in the network elements to a central 328 controller implementing the network functionality through software. 329 In contrast to traditional approaches, in which the network's control 330 plane is distributed throughout all network devices, with SDN the 331 control plane is logically centralized. In this way, the deployment 332 of new characteristics in the network no longer requires complex and 333 costly changes in equipment or firmware updates, but only a change in 334 the software running in the controller. The main advantage of this 335 approach is the flexibility it provides operators with to manage 336 their network, i.e., an operator can easily change its policies on 337 how traffic is distributed throughout the network. 339 One of the most well known protocols for the SDN control plane 340 between the central controller and the networking elements is the 341 OpenFlow protocol (OFP), which is maintained and extended by the Open 342 Network Foundation (ONF: https://www.opennetworking.org/). 343 Originally this protocol was developed specifically for IEEE 802.1 344 switches conforming to the ONF OpenFlow Switch specification. As the 345 benefits of the SDN paradigm have reached a wider audience, its 346 application has been extended to more complex scenarios such as 347 Wireless and Mobile networks. Within this area of work, the ONF is 348 actively developing new OFP extensions addressing three key 349 scenarios: (i) Wireless backhaul, (ii) Cellular Evolved Packet Core 350 (EPC), and (iii) Unified access and management across enterprise 351 wireless and fixed networks. 353 +----------+ 354 | ------- | 355 | |Oper.| | O 356 | |Mgmt.| |<........> -+- Network Operator 357 | |Iface| | ^ 358 | ------- | +----------------------------------------+ 359 | | | +------------------------------------+ | 360 | | | | --------- --------- --------- | | 361 |--------- | | | | App 1 | | App 2 | ... | App n | | | 362 ||Plugins| |<....>| | --------- --------- --------- | | 363 |--------- | | | Plugins | | 364 | | | +------------------------------------+ | 365 | | | Application Plane | 366 | | +----------------------------------------+ 367 | | A 368 | | | 369 | | V 370 | | +----------------------------------------+ 371 | | | +------------------------------------+ | 372 |--------- | | | ------------ ------------ | | 373 || Netw. | | | | | Module 1 | | Module 2 | | | 374 ||Engine | |<....>| | ------------ ------------ | | 375 |--------- | | | Network Engine | | 376 | | | +------------------------------------+ | 377 | | | Controller Plane | 378 | | +----------------------------------------+ 379 | | A 380 | | | 381 | | V 382 | | +----------------------------------------+ 383 | | | +--------------+ +--------------+ | 384 | | | | ------------ | | ------------ | | 385 |----------| | | | OpenFlow | | | | OpenFlow | | | 386 ||OpenFlow||<....>| | ------------ | | ------------ | | 387 |----------| | | NE | | NE | | 388 | | | +--------------+ +--------------+ | 389 | | | Data Plane | 390 |Management| +----------------------------------------+ 391 +----------+ 393 Figure 3: High level SDN ONF architecture 395 Figure 3 shows the blocks and the functional interfaces of the ONF 396 architecture, which comprises three planes: Data, Controller, and 397 Application. The Data plane comprehends several Network Entities 398 (NE), which expose their capabilities toward the Controller plane via 399 a Southbound API. The Controller plane includes several cooperating 400 modules devoted to the creation and maintenance of an abstracted 401 resource model of the underlying network. Such model is exposed to 402 the applications via a Northbound API where the Application plane 403 comprises several applications/services, each of which has exclusive 404 control of a set of exposed resources. 406 The Management plane spans its functionality across all planes 407 performing the initial configuration of the network elements in the 408 Data plane, the assignment of the SDN controller and the resources 409 under its responsibility. In the Controller plane, the Management 410 needs to configure the policies defining the scope of the control 411 given to the SDN applications, to monitor the performance of the 412 system, and to configure the parameters required by the SDN 413 controller modules. In the Application plane, Management configures 414 the parameters of the applications and the service level agreements. 415 In addition to these interactions, the Management plane exposes 416 several functions to network operators which can easily and quickly 417 configure and tune the network at each layer. 419 The IRTF Software-Defined Networking Research Group (SDNRG) 420 documented in RFC7426 [RFC7426], a layer model of an SDN 421 architecture, since this has been a controversial discussion topic: 422 what is exactly SDN? what is the layer structure of the SDN 423 architecture? how do layers interface with each other? etc. 425 Figure 4 reproduces the figure included in RFC7426 [RFC7426] to 426 summarize the SDN architecture abstractions in the form of a 427 detailed, high-level schematic. In a particular implementation, 428 planes can be collocated with other planes or can be physically 429 separated. 431 In SDN, a controller manipulates controlled entities via an 432 interface. Interfaces, when local, are mostly API invocations 433 through some library or system call. However, such interfaces may be 434 extended via some protocol definition, which may use local inter- 435 process communication (IPC) or a protocol that could also act 436 remotely; the protocol may be defined as an open standard or in a 437 proprietary manner. 439 SDN expands multiple planes: Forwarding, Operational, Control, 440 Management and Applications. All planes mentioned above are 441 connected via interfaces. Additionally, RFC7426 [RFC7426] considers 442 four abstraction layers: the Device and resource Abstraction Layer 443 (DAL), the Control Abstraction Layer (CAL), the Management 444 Abstraction Layer (MAL) and the Network Services Abstraction Layer 445 (NSAL). 447 o--------------------------------o 448 | | 449 | +-------------+ +----------+ | 450 | | Application | | Service | | 451 | +-------------+ +----------+ | 452 | Application Plane | 453 o---------------Y----------------o 454 | 455 *-----------------------------Y---------------------------------* 456 | Network Services Abstraction Layer (NSAL) | 457 *------Y------------------------------------------------Y-------* 458 | | 459 | Service Interface | 460 | | 461 o------Y------------------o o---------------------Y------o 462 | | Control Plane | | Management Plane | | 463 | +----Y----+ +-----+ | | +-----+ +----Y----+ | 464 | | Service | | App | | | | App | | Service | | 465 | +----Y----+ +--Y--+ | | +--Y--+ +----Y----+ | 466 | | | | | | | | 467 | *----Y-----------Y----* | | *---Y---------------Y----* | 468 | | Control Abstraction | | | | Management Abstraction | | 469 | | Layer (CAL) | | | | Layer (MAL) | | 470 | *----------Y----------* | | *----------Y-------------* | 471 | | | | | | 472 o------------|------------o o------------|---------------o 473 | | 474 | CP | MP 475 | Southbound | Southbound 476 | Interface | Interface 477 | | 478 *------------Y---------------------------------Y----------------* 479 | Device and resource Abstraction Layer (DAL) | 480 *------------Y---------------------------------Y----------------* 481 | | | | 482 | o-------Y----------o +-----+ o--------Y----------o | 483 | | Forwarding Plane | | App | | Operational Plane | | 484 | o------------------o +-----+ o-------------------o | 485 | Network Device | 486 +---------------------------------------------------------------+ 488 Figure 4: SDN Layer Architecture 490 While SDN is often directly associated to OpenFlow, this is just one 491 (relevant) example of southbound protocol between the central 492 controller and the network entities. Other relevant examples of 493 protocols in the SDN family are NETCONF [RFC6241], RESTCONF [RFC8040] 494 and ForCES [RFC5810]. 496 3.3. ITU-T functional architecture of SDN 498 The Telecommunication standardization sector of the International 499 Telecommunication Union (ITU) -- the ITU-T -- has also looked into 500 SDN architectures, defining a slightly modified one from what other 501 SDOs have done. ITU-T provides in the recommendation ITU-T Y.3302 502 [itu-t-y.3302] provides a functional architecture of SDN with 503 descriptions of functional components and reference points. The 504 described functional architecture is intended to be used as an 505 enabler for further studies on other aspects such as protocols and 506 security as well as being used to customize SDN in support of 507 appropriate use cases (e.g., cloud computing, mobile networks). This 508 recommendation is based on ITU-T Y.3300 [itu-t-y.3300] and ITU-T 509 Y.3301 [itu-t-y.3301]. While the first describes the framework of 510 SDN (including definitions, objectives, high-level capabilities, 511 requirements and the high-level architecture of SDN), the second 512 describes more detailed requirements. 514 Figure 5 shows the SDN functional architecture defined by the ITU-T. 515 It is a layered architecture composed of the SDN application layer 516 (SDN-AL), the SDN control layer (SDN-CL) and the SDN resource layer 517 (SDN-RL). It also has multi-layer management functions (MMF), which 518 provides functionalities for managing the functionalities of SDN 519 layers, i.e., SDN-AL, SDN-CL and SDN-RL. MMF interacts with these 520 layers using MMFA, MMFC, and MMFR reference points. 522 The SDN-AL enables a service-aware behaviour of the underlying 523 network in a programmatic manner. The SDN-CL provides programmable 524 means to control the behaviour of SDN-RL resources (such as data 525 transport and processing), following requests received from the SDN- 526 AL and according to MMF policies. The SDN-RL is where the physical 527 or virtual network elements perform transport and/or processing of 528 data packets according to SDN-CL decisions. 530 MMFO MMFA 531 +-----+ . +---------------------+ . +--------------------+ 532 | | . |+---+ +---+ +-------+| . |+---------+ +-----+ | 533 | | . || | | | | || . || AL. | | | | 534 | | . || E | | | | App. || . || mngmt. | | SDN | | SDN-AL 535 | | . || x | | M | | layer || . || support | | app | | 536 | | . || t.| | u | | Mngmt.|| . || & orch. | | | | 537 | | . || | | l | +-------+| . |+---------+ +-----+ | 538 | | . || r | | t | | . +--------------------+ 539 | | . || e | | i | |MMFC ..................... ACI 540 | | . || l | | | | . +--------------------+ 541 | | . || a | | l | +-------+| . |+------+ +---------+| 542 | OSS/| . || t | | a | | || . || | | App. || 543 | BSS | . || i | | y | | || . || | | support || 544 | | . || o | | e | | || . || | +---------+| 545 | | . || n | | r | | || . || CL | +---------+| 546 | | . || s | | | |Control|| . ||mngmt.| | Control || 547 | | . || h | | m | | layer || . || supp.| | layer || SDN-CL 548 | | . || i | | a | | mngmt.|| . || and | | serv. || 549 | | . || p | | n | | || . || orch.| +---------+| 550 | | . || | | a | | || . || | +---------+| 551 | | . || m | | g | | || . || | | Resource|| 552 | | . || n | | e | | || . || | | abstrac.|| 553 | | . || g | | m | +-------+| . |+------+ +---------+| 554 | | . || m | | e | | . +--------------------+ 555 | | . || t.| | n | |MMFR ..................... RCI 556 | | . || | | t | | . +--------------------+ 557 +-----+ . |+---+ | | +-------+| . |+------++----------+| 558 | | o | | || . || ||RL control|| 559 | | r | |Resour.|| . || RL |+----------+| 560 MMF | | c | | layer || . ||mngmt.|+----++----+| SDN-RL 561 | | h.| | mngmt.|| . || supp.||Data||Data|| 562 | | | | || . || ||tran||proc|| 563 | +---+ +-------+| . |+------++----++----+| 564 +---------------------+ . +--------------------+ 566 Figure 5: ITU-T SDN functional architecture 568 3.4. Multi-access Edge Computing 570 Multi-access Edge Computing (MEC) -- formerly known as Mobile Edge 571 Computing -- capabilities deployed in the edge of the mobile network 572 can facilitate the efficient and dynamic provision of services to 573 mobile users. The ETSI ISG MEC working group, operative from end of 574 2014, intends to specify an open environment for integrating MEC 575 capabilities with service providers' networks, including also 576 applications from 3rd parties. These distributed computing 577 capabilities will make available IT infrastructure as in a cloud 578 environment for the deployment of functions in mobile access 579 networks. It can be seen then as a complement to both NFV and SDN. 581 3.5. IEEE 802.1CF (OmniRAN) 583 The IEEE 802.1CF Recommended Practice [omniran] specifies an access 584 network, which connects terminals to their access routers, utilizing 585 technologies based on the family of IEEE 802 Standards (e.g., 802.3 586 Ethernet, 802.11 Wi-Fi, etc.). The specification defines an access 587 network reference model, including entities and reference points 588 along with behavioral and functional descriptions of communications 589 among those entities. 591 The goal of this project is to help unifying the support of different 592 interfaces, enabling shared network control and use of SDN 593 principles, thereby lowering the barriers to new network 594 technologies, to new network operators, and to new service providers. 596 3.6. Distributed Management Task Force 598 The DMTF (https://www.dmtf.org/) is an industry standards 599 organization working to simplify the manageability of network- 600 accessible technologies through open and collaborative efforts by 601 some technology companies. The DMTF is involved in the creation and 602 adoption of interoperable management standards, supporting 603 implementations that enable the management of diverse traditional and 604 emerging technologies including cloud, virtualization, network and 605 infrastructure. 607 There are several DMTF initiatives that are relevant to the network 608 virtualization area, such as the Open Virtualization Format (OVF), 609 for VNF packaging; the Cloud Infrastructure Management Interface 610 (CIM), for cloud infrastructure management; the Network Management 611 (NETMAN), for VNF management; and, the Virtualization Management 612 (VMAN), for virtualization infrastructure management. 614 3.7. Open Source initiatives 616 The Open Source community is especially active in the area of network 617 virtualization and orchestration. We next summarize some of the 618 active efforts: 620 o OpenStack. OpenStack is a free and open-source cloud-computing 621 software platform. OpenStack software controls large pools of 622 compute, storage, and networking resources throughout a 623 datacenter, managed through a dashboard or via the OpenStack API. 625 o Kubernetes. Kubernetes is an open-source system for automating 626 deployment, scaling and management of containerized applications. 627 Kubernetes can schedule and run application containers on clusters 628 of physical or virtual machines. Kubernetes allows: (i) Scale on 629 the fly, (ii) Limit hardware usage to required resources only, 630 (iii) Load balancing Monitoring, and (iv) Efficient lifecycle 631 management. 633 o OpenDayLight. OpenDaylight (ODL) is a highly available, modular, 634 extensible, scalable and multi-protocol controller infrastructure 635 built for SDN deployments on modern heterogeneous multi-vendor 636 networks. It provides a model-driven service abstraction platform 637 that allows users to write apps that easily work across a wide 638 variety of hardware and southbound protocols. 640 o ONOS. The ONOS (Open Network Operating System) project is an open 641 source community hosted by The Linux Foundation. The goal of the 642 project is to create a SDN operating system for communications 643 service providers that is designed for scalability, high 644 performance and high availability. 646 o OpenContrail. OpenContrail is an Apache 2.0-licensed project that 647 is built using standards-based protocols and provides all the 648 necessary components for network virtualization-SDN controller, 649 virtual router, analytics engine, and published northbound APIs. 650 It has an extensive REST API to configure and gather operational 651 and analytics data from the system. 653 o OPNFV. OPNFV is a carrier-grade, integrated, open source platform 654 to accelerate the introduction of new NFV products and services. 655 By integrating components from upstream projects, the OPNFV 656 community aims at conducting performance and use case-based 657 testing to ensure the platform's suitability for NFV use cases. 658 The scope of OPNFV's initial release is focused on building NFV 659 Infrastructure (NFVI) and Virtualized Infrastructure Management 660 (VIM) by integrating components from upstream projects such as 661 OpenDaylight, OpenStack, Ceph Storage, KVM, Open vSwitch, and 662 Linux. These components, along with application programmable 663 interfaces (APIs) to other NFV elements form the basic 664 infrastructure required for Virtualized Network Functions (VNF) 665 and Management and Network Orchestration (MANO) components. 666 OPNFV's goal is to increase performance and power efficiency; 667 improve reliability, availability, and serviceability; and deliver 668 comprehensive platform instrumentation. 670 o OSM. Open Source Mano (OSM) is an ETSI-hosted project to develop 671 an Open Source NFV Management and Orchestration (MANO) software 672 stack aligned with ETSI NFV. OSM is based on components from 673 previous projects, such Telefonica's OpenMANO or Canonical's Juju, 674 among others. 676 o OpenBaton. OpenBaton is a ETSI NFV compliant Network Function 677 Virtualization Orchestrator (NFVO). OpenBaton was part of the 678 OpenSDNCore project started with the objective of providing a 679 compliant implementation of the ETSI NFV specification. 681 o ONAP. ONAP (Open Network Automation Platform) is an open source 682 software platform that delivers capabilities for the design, 683 creation, orchestration, monitoring, and life cycle management of: 684 (i) Virtual Network Functions (VNFs), (ii) The carrier-scale 685 Software Defined Networks (SDNs) that contain them, and (iii) 686 Higher-level services that combine the above. ONAP (derived from 687 the AT&T's ECOMP) provides for automatic, policy-driven 688 interaction of these functions and services in a dynamic, real- 689 time cloud environment. 691 o SONA. SONA (Simplified Overlay Network Architecture) is an 692 extension to ONOS to have a almost full SDN network control in 693 OpenStack for virtual tenant network provisioning. Basically, 694 SONA is a SDN-based network virtualization solution for cloud DC. 696 Among the main areas that are being developed by the former open 697 source activities that related to network virtualization research, we 698 can highlight: policy-based resource management, analytics for 699 visibility and orchestration, service verification with regards to 700 security and resiliency. 702 4. Network Virtualization Challenges 704 4.1. Introduction 706 Network Virtualization is changing the way the telecommunications 707 sector will deploy, extend and operate their networks. These new 708 technologies aim at reducing the overall costs by outsourcing 709 communication services from specific hardware in the operators' core 710 to server farms scattered in datacenters (i.e. compute and storage 711 virtualization). In addition, the connecting networks are 712 fundamentally affected in the way they route, process and control 713 traffic (i.e. network virtualization). 715 4.2. Guaranteeing quality-of-service 717 Guaranteeing a given quality-of-service in an NFV environment is not 718 an easy task. For example, ensuring a guaranteed and stable 719 forwarding data rate has proven not to be straightforward when the 720 forwarding function is virtualized and runs on top of COTS server 721 hardware [openmano_dataplane] 722 [I-D.mlk-nfvrg-nfv-reliability-using-cots] [etsi_nvf_whitepaper_3]. 723 We next identify some of the challenges that this poses. 725 4.2.1. Virtualization Technologies 727 The issue of guaranteeing a network quality-of-service is less of an 728 issue for "traditional cloud computing" because the workloads that 729 are treated there are servers or clients in the networking sense and 730 hardly ever process packets. Cloud computing provides hosting for 731 applications on shared servers in a highly separated way. Its main 732 advantage is that the infrastructure costs are shared among tenants 733 and that the cloud infrastructure provides levels of reliability that 734 can not be achieved on individual premises in a cost-efficient way 735 [intel_10_differences_nfv_cloud]. NFV has very strict requirements 736 posed in terms of performance, stability and consistency. Although 737 there are some tools and mechanisms to improve this, such as Enhanced 738 Performance Awareness (EPA), Single Root I/O Virtualization (SR-IOV), 739 Non-Uniform Memory Access (NUMA), Data Plane Development Kit (DPDK), 740 etc, these are still unsolved challenges. One open research issue is 741 finding out technologies that are different from VM and more suitable 742 for dealing with network functionalities. 744 Lately, a number of light-weight virtualization technologies 745 including containers, unikernels (specialized VMs) and minimalistic 746 distributions of general-purpose OSes have appeared as virtualization 747 approaches that can be used when constructing an NFV platform. 748 [I-D.natarajan-nfvrg-containers-for-nfv] describes the challenges in 749 building such a platform and discusses to what extent these 750 technologies, as well as traditional VMs, are able to address them. 752 4.2.2. Metrics for NFV characterization 754 Another relevant aspect is the need for tools for diagnostics and 755 measurement suited for NFV. There is a pressing need to define 756 metrics and associated protocols to measure the performance of NFV. 757 Specifically, since NFV is based on the concept of taking centralized 758 functions and evolving it to highly distributed SW functions, there 759 is a commensurate need to fully understand and measure the baseline 760 performance of such systems. 762 The IP Performance Metrics (IPPM) WG defines metrics that can be used 763 to measure the quality and performance of Internet services and 764 applications running over transport layer protocols (e.g., TCP, UDP) 765 over IP. It also develops and maintains protocols for the 766 measurement of these metrics. While the IPPM WG is a long running WG 767 that started in 1997, it does not have a charter item or active 768 drafts related to the topic of network virtualization. In addition 769 to using IPPM metrics to evaluate the QoS, there is a need for 770 specific metrics for assessing the performance of network 771 virtualization techniques. 773 The Benchmarking Methodology Working Group (BMWG) is also performing 774 work related to NFV metrics. For example, 775 [I-D.ietf-bmwg-virtual-net] investigates additional methodological 776 considerations necessary when benchmarking VNFs instantiated and 777 hosted in general- purpose hardware, using bare-metal hypervisors or 778 other isolation environments such as Linux containers. An essential 779 consideration is benchmarking physical and virtual network functions 780 in the same way when possible, thereby allowing direct comparison. 782 As stated in the document [I-D.ietf-bmwg-virtual-net], there is a 783 clear motivation for the work on performance metrics for NFV 784 [etsi_gs_nfv_per_001], that is worth replicating here: "I'm designing 785 and building my NFV Infrastructure platform. The first steps were 786 easy because I had a small number of categories of VNFs to support 787 and the VNF vendor gave HW recommendations that I followed. Now I 788 need to deploy more VNFs from new vendors, and there are different 789 hardware recommendations. How well will the new VNFs perform on my 790 existing hardware? Which among several new VNFs in a given category 791 are most efficient in terms of capacity they deliver? And, when I 792 operate multiple categories of VNFs (and PNFs) *concurrently* on a 793 hardware platform such that they share resources, what are the new 794 performance limits, and what are the software design choices I can 795 make to optimize my chosen hardware platform? Conversely, what 796 hardware platform upgrades should I pursue to increase the capacity 797 of these concurrently operating VNFs?" 799 Lately, there are also some efforts looking into VNF benchmarking. 800 The selection of an NFV Infrastructure Point of Presence to host a 801 VNF or allocation of resources (e.g., virtual CPUs, memory) needs to 802 be done over virtualized (abstracted and simplified) resource views 803 [vnf_benchmarking] [I-D.rorosz-nfvrg-vbaas]. 805 4.2.3. Predictive analysis 807 On top of diagnostic tools that enable an assessment of the QoS, 808 predictive analyses are required to react before anomalies occur. 809 Due to the SW characteristics of VNFs, a reliable diagnosis framework 810 could potentially enable the prevention of issues by a proper 811 diagnosis and then a reaction in terms of acting on the potentially 812 impacted service (e.g., migration to a different compute node, 813 scaling in/out, up/down, etc). 815 4.2.4. Portability 817 Portability in NFV refers to the ability to run a given VNF on 818 multiple NFVIs, that is, that it is possible to guarantee that the 819 VNF would be able to perform its functions with a high and 820 predictable performance given that a set of requirements on the NFVI 821 resources is met. Therefore, portability is a key feature that, if 822 fully enabled, would contribute to making the NFV environment achieve 823 a better reliability than a traditional system. The fact of running 824 functionality in SW over "commodity" infrastructure should make much 825 easier to port/move functions from one place to another. However 826 this is not yet as ideal as it sounds and there are aspects not fully 827 tackled. The existence of different hypervisors, specific hardware 828 dependencies (e.g., EPA related) or state synchronization aspects are 829 just some examples of trouble-makers for portability purposes. 831 The ETSI NFV ISG is doing work in relation to portability. 832 [etsi_gs_nfv_per_001] provides a list of minimal features which the 833 VM Descriptor and Compute Host Descriptor should contain for the 834 appropriate deployment of VM images over an NFVI (i.e. a "telco 835 datacenter"), in order to guarantee high and predictable performance 836 of data plane workloads while assuring their portability. In 837 addition, the document provides a set of recommendations on the 838 minimum requirements which HW and hypervisor should have for a "telco 839 datacenter" suitable for different workloads (data-plane, control- 840 plane, etc.) present in VNFs. The purpose of this document is to 841 provide the list of VM requirements that should be included in the VM 842 Descriptor template, and the list of HW capabilities that should be 843 included in the Compute Host Descriptor (CHD) to assure predictable 844 high performance. ETSI NFV assumes that the MANO Functions will make 845 the mix & match. There are therefore still quite several research 846 challenges to be addressed here. 848 4.3. Performance improvement 850 4.3.1. Energy Efficiency 852 Virtualization is typically seen as a direct enabler of energy 853 savings. Some of the enablers for this that are often mentioned 854 [nfv_sota_research_challenges] are: (i) the multiplexing gains 855 achieved by centralizing functions in data centers reduce overall the 856 energy consumed, (ii) the flexibility brought by network 857 programmability enables to switch off infrastructure as needed in a 858 much easier way. However there is still a lot of room for 859 improvement in terms of virtualization techniques to reduce the power 860 consumption, such as enhanced hypervisor technologies. 862 4.3.2. Improved link usage 864 The use of NFV and SDN technologies can help improving link usage. 865 SDN has shown already that it can greatly increase average link usage 866 (e.g., Google example [google_sdn_wan]). NFV adds more complexity 867 (e.g., due to service function chaining / VNF forwarding graphs) 868 which need to be considered. Aspects like the ones described in 869 [I-D.bagnulo-nfvrg-topology] on NFV data center topology design have 870 to be carefully looked at as well. 872 4.4. Multiple Domains 874 Market fragmentation has resulted in a multitude of network operators 875 each focused on different countries and regions. This makes it 876 difficult to create infrastructure services spanning multiple 877 countries, such as virtual connectivity or compute resources, as no 878 single operator has a footprint everywhere. Cross-domain 879 orchestration of services over multiple administrations or over 880 multi-domain single administrations will allow end-to-end network and 881 service elements to mix in multi-vendor, heterogeneous technology and 882 resource environments. 884 For the specific use case of 'Network as a Service', it becomes even 885 more important to ensure, that Cross Domain Orchestration also takes 886 care of hierarchy of networks and their association, with respect to 887 provisioning tunnels and overlays. 889 Multi-domain orchestration is currently an active research topic, 890 which is being tackled, among others, by ETSI NFV ISG and the 5GEx 891 project (https://www.5gex.eu/) [I-D.bernardos-nfvrg-multidomain]. 893 Another side of the multi-domain problem is the integration/ 894 harmonization of different management domains. A key example comes 895 from Multi-access Edge Computing, which, according to ETSI, comes 896 with its own MANO system, and would require ti be integrated if 897 interconnected to a generic NFV system. 899 4.5. 5G and Network Slicing 901 From the beginning of all 5G discussions in the research and industry 902 fora, it has been agreed that 5G will have to address much more use 903 cases than the preceding wireless generations, which first focused on 904 voice services, and then on voice and high speed packet data 905 services. In this case, 5G should be able to handle not only the 906 same (or enhanced) voice and packet data services, but also new 907 emerging services like tactile Internet and IoT. These use cases 908 take the requirements to opposite extremes, as some of them require 909 ultra-low latency and higher-speed, whereas some others require 910 ultra-low power consumption and high delay tolerance. 912 Because of these very extreme 5G use cases, it is envisioned that 913 different radio access networks are needed to better address the 914 specific requirements of each one of the use cases. However, on the 915 core network side, virtualization techniques can allow tailoring the 916 network resources on separate slices, specifically for each radio 917 access network and use case, in an efficient manner. 919 Network slicing techniques can also allow dedicating resources for 920 even more specific use cases within the major 5G categories. For 921 example, within the major IoT category, which is perhaps the most 922 disrupting one, some autonomous IoT devices will have very low 923 throughput, will have much longer sleep cycles (and therefore high 924 latency), and a battery life time exceeding by a factor of thousands 925 that of smart phones or some other connected IoT devices that will 926 have almost continuous control and data communications. Hence, it is 927 envisioned that a single virtual core network could be used by 928 slicing separate resources to dedicated radio access networks (RANs) 929 that are better suited for specific use cases. 931 The actual definition of network slicing is still a sensitive 932 subject, currently under heavy discussion 933 [I-D.gdmb-netslices-intro-and-ps] 934 [I-D.defoy-netslices-3gpp-network-slicing] [ngmn_5G_whitepaper]. 935 Network slicing is a key for introducing new actors in existing 936 market at low cost -- by letting new players rent "blocks" of 937 capacity, if the new business model enables performance that meets 938 the application needs (e.g., broadcasting updates to many sensors 939 with satellite broadcasting capabilities). However, more work needs 940 to be done to define how network slicing will impact existing 941 architectures like ETSI NFV, and to define the impacts of network 942 slicing to guaranteeing quality-of-service as described in 943 Section 4.2. 945 4.5.1. Virtual Network Operators 947 The widespread use/discussion/practice of system and network 948 virtualization technologies has conducted to new business 949 opportunities, enlarging the offer of IT resources with virtual 950 network and computing resources, among others. As a consequence, the 951 network ecosystem now differentiates between the owner of physical 952 resources, the Infrastructure Provider (InP), and the intermediary 953 that conforms and delivers network services to the final customers, 954 the Virtual Network Operator (VNO). 956 VNOs aim to exploit the virtualized infrastructures to deliver new 957 and improved services to their customers. However, current network 958 virtualization techniques offer poor support for VNOs to control 959 their resources. It has been considered that the InP is responsible 960 for the reliability of the virtual resources but there are several 961 situations in which a VNO requires to gain a finer control on its 962 resources. For instance, dynamic events, such as the identification 963 of new requirements or the detection of incidents within the virtual 964 system, might urge a VNO to quickly reform its virtual infrastructure 965 and resource allocation. However, the interfaces offered by current 966 virtualization platforms do not offer the necessary functions for 967 VNOs to perform the elastic adaptations they require to tackle with 968 their dynamic operation environments. 970 Beyond their heterogeneity, which can be resolved by software 971 adapters, current virtualization platforms do not have common methods 972 and functions, so it is difficult for the virtual network controllers 973 used by the VNOs to actually manage and control virtual resources 974 instantiated on different platforms, not even considering different 975 InPs. Therefore it is necessary to reach a common definition of the 976 functions that should be offered by underlying platforms to enable 977 such overlay controllers with the possibility of allocate and 978 deallocate resources dynamically and get monitoring data about them. 980 Such common methods should be offered by all underlying controllers, 981 regardless of being network-oriented (e.g. ODL, ONOS, Ryu) or 982 computing-oriented (e.g. OpenStack, OpenNebula, Eucalyptus). 983 Furthermore, it is also important for those platforms to offer some 984 "PUSH" function to report resource state, avoiding the need for the 985 VNO's controller to "POLL" for such data. A starting point to get 986 proper notifications within current REST APIs could be to consider 987 the protocol proposed by the WEBPUSH WG [RFC8030]. 989 Finally, in order to establish a proper order and allow the 990 coexistence and collaboration of different systems, a common ontology 991 regarding network and system virtualization should be defined and 992 agreed, so different and heterogeneous systems can understand each 993 other without requiring to rely on specific adaptation mechanisms 994 that might break with any update on any side of the relation. 996 4.5.2. Extending Virtual Networks and Systems to the Internet of Things 998 The Internet of Things (IoT) refers to the vision of connecting a 999 multitude of automated devices (e.g. lights, environmental sensors, 1000 traffic lights, parking meters, health and security systems, etc.) to 1001 the Internet for purposes of reporting, and remote command and 1002 control of the device. This vision is being realized by a multi- 1003 pronged approach of standardization in various forums and 1004 complementary open source activities. For example, in IETF, support 1005 of IoT web services has been defined by an HTTP-like protocol adapted 1006 for IoT called CoAP [RFC7252], and lately a group has been studying 1007 the need to develop a new network layer to support IP applications 1008 over Low Power Wide Area Networks (LPWAN). 1010 Elsewhere, for 5G cellular evolution there is much discussion on the 1011 need for supporting virtual "network slices" for the expected massive 1012 numbers of IoT devices. A separate virtual network slice is 1013 considered necessary for different 5G IoT use cases because devices 1014 will have very different characteristics than typical cellular 1015 devices like smart phones [ngmn_5G_whitepaper], and the number of IoT 1016 devices is expected to be at least one or two orders of magnitude 1017 higher than other 5G devices. 1019 The specific nature of the IoT ecosystem, particularly reflected in 1020 the Machine-to-Machine (M2M) communications, conducts to the creation 1021 of new and highly distributed systems which demand location-based 1022 network and computing services. A specific example can be 1023 represented by a set of "things" that suddenly require to set-up a 1024 firewall to allow external entities to access their data while 1025 outsourcing some computation requirements to more powerful systems 1026 relying on cloud-based services. This representative use case 1027 exposes important requirements for both NFV and the underlying cloud 1028 infrastructures. 1030 In order to provide the aforementioned location-based functions 1031 integrated with highly distributed systems, the so called fog 1032 infrastructures should be able to instantiate VNFs, placing them in 1033 the required place, e.g. close to their consumers. This requirement 1034 implies that the interfaces offered by virtualization platforms must 1035 support the specification of location-based resources, which is a key 1036 function in those scenarios. Moreover, those platforms must also be 1037 able to interpret and understand the references used by IoT systems 1038 to their location (e.g., "My-AP", "5BLDG+2F") and also the 1039 specification of identifiers linked to other resources, such as the 1040 case of requiring the infrastructure to establish a link between a 1041 specific AP and a specific virtual computing node. 1043 4.6. Service Composition 1045 Current network services deployed by operators often involve the 1046 composition of several individual functions (such as packet 1047 filtering, deep packet inspection, load balancing). These services 1048 are typically implemented by the ordered combination of a number of 1049 service functions that are deployed at different points within a 1050 network, not necessarily on the direct data path. This requires 1051 traffic to be steered through the required service functions, 1052 wherever they are deployed [RFC7498]. 1054 For a given service, the abstracted view of the required service 1055 functions and the order in which they are to be applied is called a 1056 Service Function Chain (SFC) [sfc_challenges], which is called 1057 Network Function Forwarding Graph (NF-FG) in ETSI. An SFC is 1058 instantiated through selection of specific service function instances 1059 on specific network nodes to form a service graph: this is called a 1060 Service Function Path (SFP). The service functions may be applied at 1061 any layer within the network protocol stack (network layer, transport 1062 layer, application layer, etc.). 1064 Service composition is a powerful mean which can provide significant 1065 benefits when applied in a softwarized network environment. There 1066 are however many research challenges in this area, as for example the 1067 ones related to composition mechanisms and algorithms to enable load 1068 balancing and improve reliability. The service composition should 1069 also act as an enabler to gather information across all hierarchies 1070 (underlays and overlays) of network deployments which may span across 1071 multiple operators, for faster serviceability thus facilitating in 1072 accomplishing aforementioned goals of "load balancing and improve 1073 reliability". 1075 As described in [dynamic_chaining], different algorithms can be used 1076 to enable dynamic service composition that optimize a QoS-based 1077 utility function (e.g., minimizing the latency per-application 1078 traffic flows) for a given composition plan. Such algorithms can 1079 consider the computation capabilities and load status of resources 1080 executing the VNF instances, either deduced through estimations from 1081 usage historical data or collected through real-time monitoring data 1082 (i.e., context-aware selection). For this reason selections should 1083 include references to dynamic information on the status of the 1084 service instance and its constituent elements, i.e., monitoring 1085 information related to individual VNF instances and links connecting 1086 them as well as derived monitoring information at the chain level 1087 (e.g., end-to-end delay). At runtime, if one or more VNF instances 1088 are no more available or QoS degrades below a given threshold, the 1089 service selection task can be rerun to perform service substitution. 1091 The SFC working group is working on an architecture for service 1092 function chaining [RFC7665] that includes the necessary protocols or 1093 protocol extensions to convey the Service Function Chain and Service 1094 Function Path information to nodes that are involved in the 1095 implementation of service functions and Service Function Chains, as 1096 well as mechanisms for steering traffic through service functions. 1098 In terms of actual work items, the SFC WG is has not yet considered 1099 working on the management and configuration of SFC components related 1100 to the support of Service Function Chaining. This part is of special 1101 interest for operators and would be required in order to actually put 1102 SFC mechanisms into operation. Similarly, redundancy and reliability 1103 mechanisms are currently not dealt with by any WG in the IETF. While 1104 this was the main goal of the VNFpool BoF efforts, it still remains 1105 unaddressed. 1107 4.7. End-user device virtualization 1109 So far, most of the network softwarization efforts have focused on 1110 virtualizing functions of network elements. While virtualization of 1111 network elements started with the core, mobile networks architectures 1112 are now heavily switching to also virtualize radio access network 1113 (RAN) functions. The next natural step is to get virtualization down 1114 at the level of the end-user device (i.e., virtualizing a smartphone) 1115 [virtualization_mobile_device]. The cloning of a device in the cloud 1116 (central or local) bears attractive benefits to both the device and 1117 network operations alike (e.g., power saving at the device by 1118 offloading computational-heaving functions to the cloud, optimized 1119 networking -- both device-to-device and device-to-infrastructure) for 1120 service delivery through tighter integration of the device (via its 1121 clone in the networking infrastructure). This is being explored for 1122 example by the European H2020 ICIRRUS project (www.icirrus-5gnet.eu). 1124 4.8. Security and Privacy 1126 Similar to any other situation where resources are shared, security 1127 and privacy are two important aspects that need to be taken into 1128 account. 1130 In the case of security, there are situations where multiple vendors 1131 will need to coexist in a virtual or hybrid physical/virtual 1132 environment. This requires attestation procedures amongst different 1133 virtual/physical functions and resources, as well as ongoing external 1134 monitoring. Similarly, different network slices operating on the 1135 same infrastructure can present security problems, for instance if 1136 one slice running critical applications (e.g. support for a safety 1137 system) is affected by another slice running a less critical 1138 application. In general, the minimum common denominator for security 1139 measures on a shared system should be equal or higher than the one 1140 required by the most critical application. Multiple and continuous 1141 threat model analysis, as well as DevOps model are required to 1142 maintain certain level of security in an NFV system. 1144 On the other hand, privacy in its strictest interpretation, refers to 1145 concerns about exposing users of the system to individual threats 1146 such as surveillance, identification, stored data compromise, 1147 secondary use, intrusion, etc. In this case, the storage, 1148 transmission, collection, and potential correlation of information in 1149 the NFV system, for purposes not originally intended or not known by 1150 the user, should be avoided. This is particularly challenging, as 1151 future intentions and threats cannot be easily predicted, and still 1152 can be applied for instance on data collected in the past. 1153 Therefore, well-known techniques such as data minimization, using 1154 privacy features as default, and allowing users to opt in/out should 1155 be used to prevent potential privacy issues. 1157 Compared to traditional networks, NFV will result in networks that 1158 are much more dynamic (in function distribution and topology) and 1159 elastic (in size and boundaries). NFV will thus require network 1160 operators to evolve their operational and administrative security 1161 solutions to work in this new environment. For example, in NFV the 1162 network orchestrator will become a key node to provide security 1163 policy orchestration across the different physical and virtual 1164 components of the virtualized network. For highly confidential data, 1165 for example, the network orchestrator should take into account if 1166 certain physical HW of the network is considered more secure (e.g., 1167 because it is located in secure premises) than other HW. 1169 Traditional telecom networks typically run under a single 1170 administrative domain controlled by (exactly) one operator. With 1171 NFV, it is expected that in many cases, the telecom operator will now 1172 become a tenant (running the VNFs), and the infrastructure (NFVI) may 1173 be run by a different operator and/or cloud service provider (see 1174 also Section 4.4). Thus, there will be multiple administrative 1175 domains which will make coordination of security policy more complex. 1176 For example, who will be in charge of provisioning and maintaining 1177 security credentials such as public and private keys? Also, should 1178 private keys be allowed to be replicated across the NFV for 1179 redundancy reasons? 1181 On a positive note, NFV will allow better defense against Denial of 1182 Service (DoS) attacks because of the distributed nature of the 1183 network (i.e. no single point of failure) and the ability to steer 1184 (undesirable) traffic quickly [etsi_gs_nfv_sec_001]. Also, NFVs 1185 which have physical HW which is distributed across multiple data 1186 centers will also provide better fault isolation environments. This 1187 holds true in particular, if each data center is protected separately 1188 via fire walls, DMZs and other network protection techniques. 1190 SDN can also be used to help improving security by facilitating the 1191 operation of existing protocols, such as Authentication, 1192 Authorization and Accounting (AAA). The management of AAA 1193 infrastructures, namely the management of AAA routing and the 1194 establishment of security associations between AAA entities, can be 1195 performed using SDN, as analyzed in [I-D.marin-sdnrg-sdn-aaa-mng]. 1197 4.9. Separation of control concerns 1199 NFV environments offer two possible levels of SDN control. One level 1200 is the need for controlling the NFVI to provide connectivity end-to- 1201 end among VNFs or among VNFs and PNFs (Physical Network Functions). 1202 A second level is the control and configuration of the VNFs 1203 themselves (in other words, the configuration of the network service 1204 implemented by those VNFs), taking advantage of the programmability 1205 brought by SDN. Both control concerns are separated in nature. 1206 However, interaction between both could be expected in order to 1207 optimize, scale or influence each other. 1209 Clear mechanisms for such interaction are needed in order to avoid 1210 malfunctioning or interference concerns. These ideas are considered 1211 in [etsi_gs_nfv_eve005] and [I-D.irtf-sdnrg-layered-sdn] 1213 4.10. Network Function placement 1215 Network function placement is a problem in any kind of network 1216 telecommunications infrastructure. But the increased degree of 1217 freedom added by network virtualization makes this problem even more 1218 important, and also harder to tackle. Deciding where to place 1219 virtual network functions is a resource allocation problem which 1220 needs to (or may) take into consideration quite a few aspects: 1221 resiliency, (anti-)affinity, security, privacy, energy efficiency, 1222 etc. 1224 When several functions are chained (typical scenario), placement 1225 algorithms become more complex and important (as described in 1226 Section 4.6). While there have been research on the topic 1227 [nfv_piecing] [dynamic_placement][vnf-p], this still remains an open 1228 challenges that requires more attention. Multi-domain also adds 1229 another component of complexity to this problem that have to be 1230 considered. 1232 4.11. Testing 1234 The impacts of network virtualization on testing can be divided into 1235 3 groups: 1237 1. Changes in methodology. 1239 2. New functionality. 1241 3. Opportunities. 1243 4.11.1. Changes in methodology 1245 The largest impact of NFV is the ability to isolate the System Under 1246 Test (SUT). When testing Physical Network Functions (PNF), isolating 1247 the SUT means that all the other devices that the SUT communicates 1248 with are replaced with simulations (or controlled executions) in 1249 order to place the SUT under test by itself. The SUT may be 1250 comprised of one or more devices. The simulations use the 1251 appropriate traffic type and protocols in order to execute test 1252 cases. See Figure 6. 1254 +--------+ +-----------+ +--------+ 1255 | | | | | | 1256 | Sim A | | SUT | | Sim B | 1257 | +------+ +-----+ | 1258 | | | | | | 1259 +--------+ +-----------+ +--------+ 1261 Figure 6: Testing methodology 1263 As shown in Figure 2, NFV provides a common architecture for all 1264 functions to use. A VNF is executed using resources offered by the 1265 NFVI, which have been allocated using the MANO function. It is not 1266 possible to test a VNF by itself, without the entire supporting 1267 environment present. This fundamentally changes how to consider the 1268 SUT. In the case of a VNF (or multiple VNFs), the SUT is part of a 1269 larger architecture which is necessary in order to run the SUTs. 1271 Isolation of the SUT therefore becomes controlling the environment in 1272 a disciplined manner. The components of the environment necessary to 1273 run the SUTs that are not part of the SUT become the test 1274 environment. In the case of VNFs which are the SUT, then the NFVI 1275 and MANO become the test environment. The configurations and 1276 policies that guide the test environment should remain constant 1277 during the execution of the tests, and also from test to test. 1278 Configurations such as CPU pinning, NUMA configuration, the SW 1279 versions and configurations of the hypervisor, vSwitch and NICs 1280 should remain constant. The only variables in the testing should be 1281 those controlling the SUT itself. If any configuration in the test 1282 environment is changed from test to test, then the results become 1283 very difficult, if not impossible, to compare since the test 1284 environment behavior may change the results as a consequence of the 1285 configuration change. 1287 Testing the NFVI itself also presents new considerations. With a 1288 PNF, the dedicated hardware supporting it is optimized for the 1289 particular workload of the function. Routing hardware is specially 1290 built to support packet forwarding functions, while the hardware to 1291 support a purely control plane application (say, a DNS server, or a 1292 Diameter function) will not have this specialized capability. In 1293 NFV, the NFVI is required to support all types of potentially 1294 different workload types. 1296 Testing the NFVI therefore requires careful consideration to what 1297 types of metrics are sought. This, in turn, depends on the workload 1298 type the expected VNF will be. Examples of different workload types 1299 are data forwarding, control plane, encryption, and authentication. 1300 All these types of expected workloads will determine the types of 1301 metrics that should be sought. For example, if the workload is 1302 control plane, then a metric such as jitter is not useful, but 1303 dropped packets is critical. In a multi-tenant environment, then the 1304 NFVI could support various types of workloads. In this case, testing 1305 with a variety of traffic types while measuring the corresponding 1306 metrics simultaneously becomes necessary. 1308 4.11.2. New functionality 1310 NFV presents a collection of new functionality in order to support 1311 the goal of software networking. Each component on the architecture 1312 shown in Figure 2 has an associated set of functionality that allows 1313 VNFs to run: onboarding, lifecycle management for VNFs and Networks 1314 Services (NS), resource allocation, hypervisor functions, etc. 1316 One of the new capabilities enabled by NFV is VNFFG (VNF Forwarding 1317 Graphs). This refers to the graph that represents a Network Service 1318 by chaining together VNFs into a forwarding path. In practice, the 1319 forwarding path can be implemented in a variety of ways using 1320 different networking capabilities: vSwitch, SDN, SDN with a 1321 northbound application, and the VNFFG might use tunneling protocols 1322 like VXLAN. The dynamic allocation and implementation of these 1323 networking paths will have different performance characteristics 1324 depending on the methods used. The path implementation mechanism 1325 becomes a variable in the network testing of the NSs. The 1326 methodology used to test the various mechanisms should largely remain 1327 the same, and as usual, the test environment should remain constant 1328 for each of the tests, focusing on varying the path establishment 1329 method. 1331 Scaling refers to the change in allocation of resources to a VNF or 1332 NS. It happens dynamically at run-time, based on defined policies 1333 and triggers. The triggers can be network, compute or storage based. 1334 Scaling can allocate more resources in times of need, or reduce the 1335 amount of resources allocated when the demand is reduced. The SUT in 1336 this case becomes much larger than the VNF itself: MANO controls how 1337 scaling is done based on policies, and then allocates the resources 1338 accordingly in the NFVI. Essentially, the testing of scaling 1339 includes the entire NFV architecture components into the SUT. 1341 4.11.3. Opportunities 1343 Softwarization of networking functionality leads to softwarization of 1344 test as well. As Physical Network Functions (PNF) are being 1345 transformed into VNFs, so have the test tools. This leads to the 1346 fact that test tools are also being controlled and executed in the 1347 same environment as the VNFs are. This presents an opportunity to 1348 include VNF-based test tools along with the deployment of the VNFs 1349 supporting the services of the service provider into the host data 1350 centers. Tests can therefore be automatically executed upon 1351 deployment in the target environment, for each deployment, and each 1352 service. With PNFs, this was very difficult to achieve. 1354 This new concept helps to enable modern concepts like DevOps and CI/ 1355 CD in the NFV environment. Simplistically, DevOps is a process that 1356 combines multiple functions into single cohesive teams in order to 1357 quickly produce quality software. It typically relies on also 1358 applying the Agile development process, which focuses on (among many 1359 things) dividing large features into multiple, smaller deliveries. 1360 One part of this is to immediately test the new smaller features in 1361 order to get immediate feedback on errors so that if present, they 1362 can be immediately fixed and redeployed. The CI/CD (Continuous 1363 Integration and Continuous Deployment) pipeline supports this 1364 concept. It consists of a series of tools, among which immediate 1365 testing is an integral part, to deliver software from source to 1366 deployment. The ability to deploy the test tools themselves into the 1367 production environment stretches the CI/CD pipeline all the way to 1368 production deployment, allowing a range of tests to be executed. The 1369 tests can be simple, with a goal of verifying the correct deployment 1370 and networking establishment, to the more complex like testing VNF 1371 functionality. 1373 5. Technology Gaps and Potential IETF Efforts 1375 Table 1 correlates the open network virtualization research areas 1376 identified in this document to potential IETF groups that could 1377 address some aspects of them. An example of a specific gap that the 1378 group could potentially address is identified in parenthetical beside 1379 the group name. 1381 +-------------------------+-----------------------------------------+ 1382 | Open Research Area | Potential IETF/IRTF Group | 1383 +-------------------------+-----------------------------------------+ 1384 | 1-Guaranteeing QoS | IPPM WG (Measurements of NFVI) | 1385 | 2-Performance | SFC WG, NFVRG (energy driven | 1386 | improvement | orchestration) | 1387 | 3-Multiple Domains | NFVRG (multi-domain orchestration) | 1388 | 4-Network Slicing | NVO3 WG, NETSLICES bar BoF (multi- | 1389 | | tenancy support) | 1390 | 5-Service Composition | SFC WG (SFC Mgmt and Config) | 1391 | 6-End-user device | N/A | 1392 | virtualization | | 1393 | 7-Security | N/A | 1394 | 8-Separation of control | NFVRG (separation between transport | 1395 | concerns | control and services) | 1396 | 9-Testing | NFVRG (testing of scaling) | 1397 | 10-Function placement | NFVRG, SFC WG (VNF placement algorithms | 1398 | | and protocols) | 1399 +-------------------------+-----------------------------------------+ 1401 Table 1: Mapping of Open Research Areas to Potential IETF Groups 1403 6. NFVRG focus areas 1405 Table 2 correlates the currently identified NFVRG topics of 1406 interests/focus areas to the open network virtualization research 1407 areas enumerated in this document. This can help the NFVRG in 1408 identifying and prioritizing research topics. The current list of 1409 NFVRG focus points is the following: 1411 o Re-architecting functions, including aspects such as new 1412 architectural and design patterns (e.g., containerization, 1413 statelessness, serverless, control/data plane separation), SDN 1414 integration, and proposals on programmability. 1416 o New management frameworks, considering aspects related to new OAM 1417 mechanisms (e.g., configuration control, hybrid descriptors) and 1418 lightweight MANO proposals. 1420 o Techniques to guarantee low latency, resource isolation, and other 1421 dataplane features, including hardware acceleration, functional 1422 offloading to dataplane elements (including NICs), and related 1423 approaches. 1425 o Measurement and benchmarking, addressing both internal 1426 measurements and external applications. 1428 +----------------------------------------+-------------------------+ 1429 | NFVRG Focus Point | Open Research Area | 1430 +----------------------------------------+-------------------------+ 1431 | 1-Re-architecting functions | - Performance improvem. | 1432 | | - Network Slicing | 1433 | | - Guaranteeing QoS | 1434 | | - Security | 1435 | | - End-user device virt. | 1436 | | - Separation of control | 1437 | 2-New management frameworks | - Multiple Domains | 1438 | | - Service Composition | 1439 | | - End-user device virt. | 1440 | 3-Low latency, resource isolation, etc | - Performance improvem. | 1441 | | - Separation of control | 1442 | 4-Measurement and benchmarking | - Guaranteeing QoS | 1443 | | - Testing | 1444 +----------------------------------------+-------------------------+ 1446 Table 2: Mapping of NFVRG Focus Points to Open Research Areas 1448 7. IANA Considerations 1450 N/A. 1452 8. Security Considerations 1454 This is an informational document, which therefore does not introduce 1455 any security threat. Research challenges and gaps related to 1456 security and privacy have been included in Section 4.8. 1458 9. Acknowledgments 1460 The authors want to thank Dirk von Hugo, Rafa Marin, Diego Lopez, 1461 Ramki Krishnan, Kostas Pentikousis, Rana Pratap Sircar, Alfred 1462 Morton, Nicolas Kuhn, Saumya Dikshit, Fabio Giust, Evangelos 1463 Haleplidis, Angeles Vazquez-Castro, Barbara Martini, Jose Saldana and 1464 Gino Carrozzo for their very useful reviews and comments to the 1465 document. Special thanks to Pedro Martinez-Julia, who provided text 1466 for the network slicing section. 1468 The work of Carlos J. Bernardos and Luis M. Contreras is partially 1469 supported by the H2020 5GEx (Grant Agreement no. 671636) and 5G- 1470 TRANSFORMER (Grant Agreement no. 761536) projects. 1472 10. Informative References 1474 [dynamic_chaining] 1475 and , "A Service-Oriented Approach for Dynamic Chaining of 1476 Virtual Network Functions over Multi-Provider Software- 1477 Defined Networks", Future Internet vol. 8, no. 2, June 1478 2016. 1480 [dynamic_placement] 1481 , , and , "The dynamic placement of virtual network 1482 functions", 2014 IEEE Network Operations and Management 1483 Symposium (NOMS) pp. 1-9, May 2014. 1485 [etsi_gs_nfv_003] 1486 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1487 Terminology for Main Concepts in NFV", ETSI GS NFV 003 1488 V1.2.1 NFV 003, December 2014, 1489 . 1492 [etsi_gs_nfv_eve005] 1493 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1494 Ecosystem; Report on SDN Usage in NFV Architectural 1495 Framework", ETSI GS NFV-EVE 005 V1.1.1 NFV-EVE 005, 1496 December 2015, . 1499 [etsi_gs_nfv_per_001] 1500 ETSI NFV ISG, "Network Functions Virtualisation (NFV); NFV 1501 Performance & Portability Best Practises", ETSI GS NFV-PER 1502 001 V1.1.2 NFV-PER 001, December 2014, 1503 . 1506 [etsi_gs_nfv_sec_001] 1507 ETSI NFV ISG, "Network Functions Virtualisation (NFV); NFV 1508 Security; Problem Statement", ETSI GS NFV-SEC 001 V1.1.1 1509 NFV-SEC 001, October 2014, 1510 . 1513 [etsi_nvf_whitepaper_3] 1514 "Network Functions Virtualisation (NFV). White Paper 3", 1515 October 2014. 1517 [google_sdn_wan] 1518 "B4: experience with a globally-deployed Software Defined 1519 WAN", Proceedings of the ACM SIGCOMM 2013 , August 2013. 1521 [I-D.bagnulo-nfvrg-topology] 1522 Bagnulo, M. and D. Dolson, "NFVI PoP Network Topology: 1523 Problem Statement", draft-bagnulo-nfvrg-topology-01 (work 1524 in progress), March 2016. 1526 [I-D.bernardos-nfvrg-multidomain] 1527 Bernardos, C., Contreras, L., Vaishnavi, I., and R. Szabo, 1528 "Multi-domain Network Virtualization", draft-bernardos- 1529 nfvrg-multidomain-02 (work in progress), March 2017. 1531 [I-D.defoy-netslices-3gpp-network-slicing] 1532 Foy, X. and A. Rahman, "Network Slicing - 3GPP Use Case", 1533 draft-defoy-netslices-3gpp-network-slicing-01 (work in 1534 progress), April 2017. 1536 [I-D.gdmb-netslices-intro-and-ps] 1537 Galis, A., Dong, J., kiran.makhijani@huawei.com, k., 1538 Bryant, S., Boucadair, M., and P. Martinez-Julia, "Network 1539 Slicing - Introductory Document and Revised Problem 1540 Statement", draft-gdmb-netslices-intro-and-ps-02 (work in 1541 progress), February 2017. 1543 [I-D.ietf-bmwg-virtual-net] 1544 Morton, A., "Considerations for Benchmarking Virtual 1545 Network Functions and Their Infrastructure", draft-ietf- 1546 bmwg-virtual-net-05 (work in progress), March 2017. 1548 [I-D.irtf-sdnrg-layered-sdn] 1549 Contreras, L., Bernardos, C., Lopez, D., Boucadair, M., 1550 and P. Iovanna, "Cooperating Layered Architecture for 1551 SDN", draft-irtf-sdnrg-layered-sdn-01 (work in progress), 1552 October 2016. 1554 [I-D.marin-sdnrg-sdn-aaa-mng] 1555 Lopez, R. and G. Lopez-Millan, "Software-Defined 1556 Networking (SDN)-based AAA Infrastructures Management", 1557 draft-marin-sdnrg-sdn-aaa-mng-00 (work in progress), 1558 November 2015. 1560 [I-D.mlk-nfvrg-nfv-reliability-using-cots] 1561 Mo, L. and B. Khasnabish, "NFV Reliability using COTS 1562 Hardware", draft-mlk-nfvrg-nfv-reliability-using-cots-01 1563 (work in progress), October 2015. 1565 [I-D.natarajan-nfvrg-containers-for-nfv] 1566 natarajan.sriram@gmail.com, n., Krishnan, R., Ghanwani, 1567 A., Krishnaswamy, D., Willis, P., Chaudhary, A., and F. 1568 Huici, "An Analysis of Lightweight Virtualization 1569 Technologies for NFV", draft-natarajan-nfvrg-containers- 1570 for-nfv-03 (work in progress), July 2016. 1572 [I-D.rorosz-nfvrg-vbaas] 1573 Rosa, R., Rothenberg, C., and R. Szabo, "VNF Benchmark-as- 1574 a-Service", draft-rorosz-nfvrg-vbaas-00 (work in 1575 progress), October 2015. 1577 [intel_10_differences_nfv_cloud] 1578 Intel, "Discover the Top 10 Differences Between NFV and 1579 Cloud Environments", November 2015, 1580 . 1583 [itu-t-y.3300] 1584 ITU-T, "Y.3300: Framework of software-defined networking", 1585 ITU-T Recommendation Y.3300 (06/14), June 2014, 1586 . 1588 [itu-t-y.3301] 1589 ITU-T, "Y.3301: Functional requirements of software- 1590 defined networking", ITU-T Recommendation Y.3301 (09/16), 1591 September 2016, 1592 . 1594 [itu-t-y.3302] 1595 ITU-T, "Y.3302: Functional architecture of software- 1596 defined networking", ITU-T Recommendation Y.3302 (01/17), 1597 January 2017, 1598 . 1600 [nfv_piecing] 1601 , , and , "Piecing together the NFV provisioning puzzle: 1602 Efficient placement and chaining of virtual network 1603 functions", 2015 IFIP/IEEE International Symposium on 1604 Integrated Network Management (IM) pp. 98-106, May 2015. 1606 [nfv_sota_research_challenges] 1607 , , , , , and , "Network Function Virtualization: State- 1608 of-the-art and Research Challenges", IEEE Communications 1609 Surveys & Tutorials Volume: 18, Issue: 1, September 2015. 1611 [ngmn_5G_whitepaper] 1612 "NGMN 5G. White Paper", February 2015. 1614 [omniran] IEEE, "802.1CF Network Reference Model and Functional 1615 Description of IEEE 802 Access Network", 802.1cf, Draft 1616 0.4 802.1cf, February 2017. 1618 [onf_tr_521] 1619 ONF, "SDN Architecture, Issue 1.1", ONF TR-521 TR-521, 1620 February 2016, 1621 . 1625 [openmano_dataplane] 1626 Telefonica I+D, "OpenMANO: The Dataplane Ready Open Source 1627 NFV MANO Stack", March 2015, 1628 . 1631 [RFC5810] Doria, A., Ed., Hadi Salim, J., Ed., Haas, R., Ed., 1632 Khosravi, H., Ed., Wang, W., Ed., Dong, L., Gopal, R., and 1633 J. Halpern, "Forwarding and Control Element Separation 1634 (ForCES) Protocol Specification", RFC 5810, 1635 DOI 10.17487/RFC5810, March 2010, 1636 . 1638 [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., 1639 and A. Bierman, Ed., "Network Configuration Protocol 1640 (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, 1641 . 1643 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 1644 Application Protocol (CoAP)", RFC 7252, 1645 DOI 10.17487/RFC7252, June 2014, 1646 . 1648 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1649 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1650 Defined Networking (SDN): Layers and Architecture 1651 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1652 2015, . 1654 [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for 1655 Service Function Chaining", RFC 7498, 1656 DOI 10.17487/RFC7498, April 2015, 1657 . 1659 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1660 Chaining (SFC) Architecture", RFC 7665, 1661 DOI 10.17487/RFC7665, October 2015, 1662 . 1664 [RFC8030] Thomson, M., Damaggio, E., and B. Raymor, Ed., "Generic 1665 Event Delivery Using HTTP Push", RFC 8030, 1666 DOI 10.17487/RFC8030, December 2016, 1667 . 1669 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 1670 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 1671 . 1673 [sfc_challenges] 1674 , , , , , and , "Service Function Chaining in Next 1675 Generation Networks: State of the Art and Research 1676 Challenges", IEEE Communications Magazine vol. 55, no. 2, 1677 pp. 216-223, February 2017. 1679 [virtualization_mobile_device] 1680 "Virtualization of Mobile Device User Experience", Patent 1681 US 9.542.062 B2 , January 2017. 1683 [vnf-p] and , "VNF-P: A model for efficient placement of 1684 virtualized network functions", 10th International 1685 Conference on Network and Service Management (CNSM) and 1686 Workshop pp. 418-423, 2014. 1688 [vnf_benchmarking] 1689 FEEC/UNICAMP, FEEC/UNICAMP, and Ericsson, "A VNF Testing 1690 Framework Design, Implementation and Partial Results", 1691 November 2016, 1692 . 1695 Authors' Addresses 1697 Carlos J. Bernardos 1698 Universidad Carlos III de Madrid 1699 Av. Universidad, 30 1700 Leganes, Madrid 28911 1701 Spain 1703 Phone: +34 91624 6236 1704 Email: cjbc@it.uc3m.es 1705 URI: http://www.it.uc3m.es/cjbc/ 1706 Akbar Rahman 1707 InterDigital Communications, LLC 1708 1000 Sherbrooke Street West, 10th floor 1709 Montreal, Quebec H3A 3G4 1710 Canada 1712 Email: Akbar.Rahman@InterDigital.com 1713 URI: http://www.InterDigital.com/ 1715 Juan Carlos Zuniga 1716 SIGFOX 1717 425 rue Jean Rostand 1718 Labege 31670 1719 France 1721 Email: j.c.zuniga@ieee.org 1722 URI: http://www.sigfox.com/ 1724 Luis M. Contreras 1725 Telefonica I+D 1726 Ronda de la Comunicacion, S/N 1727 Madrid 28050 1728 Spain 1730 Email: luismiguel.contrerasmurillo@telefonica.com 1732 Pedro Aranda 1733 Universidad Carlos III de Madrid 1734 Av. Universidad, 30 1735 Leganes, Madrid 28911 1736 Spain 1738 Email: pedroandres.aranda@uc3m.es 1740 Pierre Lynch 1741 Ixia 1743 Email: plynch@ixiacom.com