idnits 2.17.1 draft-irtf-nfvrg-gaps-network-virtualization-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 28, 2018) is 2221 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-05) exists of draft-bernardos-nfvrg-multidomain-03 == Outdated reference: A later version (-04) exists of draft-geng-coms-problem-statement-01 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFVRG CJ. Bernardos 3 Internet-Draft UC3M 4 Intended status: Informational A. Rahman 5 Expires: September 1, 2018 InterDigital 6 JC. Zuniga 7 SIGFOX 8 LM. Contreras 9 TID 10 P. Aranda 11 UC3M 12 P. Lynch 13 Ixia 14 February 28, 2018 16 Network Virtualization Research Challenges 17 draft-irtf-nfvrg-gaps-network-virtualization-09 19 Abstract 21 This document describes open research challenges for network 22 virtualization. Network virtualization is following a similar path 23 as previously taken by cloud computing. Specifically, cloud 24 computing popularized migration of computing functions (e.g., 25 applications) and storage from local, dedicated, physical resources 26 to remote virtual functions accessible through the Internet. In a 27 similar manner, network virtualization is encouraging migration of 28 networking functions from dedicated physical hardware nodes to a 29 virtualized pool of resources. However, network virtualization can 30 be considered to be a more complex problem than cloud computing as it 31 not only involves virtualization of computing and storage functions 32 but also involves abstraction of the network itself. This document 33 describes current research and engineering challenges in network 34 virtualization including guaranteeing quality-of-service, performance 35 improvement, supporting multiple domains, network slicing, service 36 composition, device virtualization, privacy and security, separation 37 of control concerns, network function placement and testing. In 38 addition, some proposals are made for new activities in IETF/IRTF 39 that could address some of these challenges. This document is a 40 product of the Network Function Virtualization Research Group 41 (NFVRG). 43 Status of This Memo 45 This Internet-Draft is submitted in full conformance with the 46 provisions of BCP 78 and BCP 79. 48 Internet-Drafts are working documents of the Internet Engineering 49 Task Force (IETF). Note that other groups may also distribute 50 working documents as Internet-Drafts. The list of current Internet- 51 Drafts is at https://datatracker.ietf.org/drafts/current/. 53 Internet-Drafts are draft documents valid for a maximum of six months 54 and may be updated, replaced, or obsoleted by other documents at any 55 time. It is inappropriate to use Internet-Drafts as reference 56 material or to cite them other than as "work in progress." 58 This Internet-Draft will expire on September 1, 2018. 60 Copyright Notice 62 Copyright (c) 2018 IETF Trust and the persons identified as the 63 document authors. All rights reserved. 65 This document is subject to BCP 78 and the IETF Trust's Legal 66 Provisions Relating to IETF Documents 67 (https://trustee.ietf.org/license-info) in effect on the date of 68 publication of this document. Please review these documents 69 carefully, as they describe your rights and restrictions with respect 70 to this document. Code Components extracted from this document must 71 include Simplified BSD License text as described in Section 4.e of 72 the Trust Legal Provisions and are provided without warranty as 73 described in the Simplified BSD License. 75 Table of Contents 77 1. Introduction and scope . . . . . . . . . . . . . . . . . . . 3 78 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 79 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 80 3.1. Network Function Virtualization . . . . . . . . . . . . . 5 81 3.2. Software Defined Networking . . . . . . . . . . . . . . . 7 82 3.3. ITU-T functional architecture of SDN . . . . . . . . . . 12 83 3.4. Multi-access Edge Computing . . . . . . . . . . . . . . . 13 84 3.5. IEEE 802.1CF (OmniRAN) . . . . . . . . . . . . . . . . . 14 85 3.6. Distributed Management Task Force . . . . . . . . . . . . 14 86 3.7. Open Source initiatives . . . . . . . . . . . . . . . . . 14 87 4. Network Virtualization Challenges . . . . . . . . . . . . . . 16 88 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 16 89 4.2. Guaranteeing quality-of-service . . . . . . . . . . . . . 16 90 4.2.1. Virtualization Technologies . . . . . . . . . . . . . 17 91 4.2.2. Metrics for NFV characterization . . . . . . . . . . 17 92 4.2.3. Predictive analysis . . . . . . . . . . . . . . . . . 18 93 4.2.4. Portability . . . . . . . . . . . . . . . . . . . . . 19 94 4.3. Performance improvement . . . . . . . . . . . . . . . . . 19 95 4.3.1. Energy Efficiency . . . . . . . . . . . . . . . . . . 19 96 4.3.2. Improved link usage . . . . . . . . . . . . . . . . . 20 97 4.4. Multiple Domains . . . . . . . . . . . . . . . . . . . . 20 98 4.5. 5G and Network Slicing . . . . . . . . . . . . . . . . . 21 99 4.5.1. Virtual Network Operators . . . . . . . . . . . . . . 22 100 4.5.2. Extending Virtual Networks and Systems to the 101 Internet of Things . . . . . . . . . . . . . . . . . 23 102 4.6. Service Composition . . . . . . . . . . . . . . . . . . . 24 103 4.7. End-user device virtualization . . . . . . . . . . . . . 25 104 4.8. Security and Privacy . . . . . . . . . . . . . . . . . . 26 105 4.9. Separation of control concerns . . . . . . . . . . . . . 27 106 4.10. Network Function placement . . . . . . . . . . . . . . . 28 107 4.11. Testing . . . . . . . . . . . . . . . . . . . . . . . . . 28 108 4.11.1. Changes in methodology . . . . . . . . . . . . . . . 28 109 4.11.2. New functionality . . . . . . . . . . . . . . . . . 30 110 4.11.3. Opportunities . . . . . . . . . . . . . . . . . . . 31 111 5. Technology Gaps and Potential IETF Efforts . . . . . . . . . 31 112 6. NFVRG focus areas . . . . . . . . . . . . . . . . . . . . . . 32 113 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 114 8. Security Considerations . . . . . . . . . . . . . . . . . . . 33 115 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 33 116 10. Informative References . . . . . . . . . . . . . . . . . . . 34 117 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 119 1. Introduction and scope 121 The telecommunications sector is experiencing a major revolution that 122 will shape the way networks and services are designed and deployed 123 for the next few decades. In order to cope with continuously 124 increasing demand and cost, network operators are taking lessons from 125 the IT paradigm of cloud computing. This new approach of 126 virtualizing network functions will enable multi-fold advantages by 127 moving communication services from bespoke hardware in the operator's 128 core network to Commercial off-the-shelf (COTS) equipment distributed 129 across datacenters. 131 Some of the network virtualization mechanisms that are being 132 considered include: sharing of network infrastructure to reduce 133 costs, virtualization of core and edge servers/services running in 134 data centers as a way of supporting their load-aware elastic 135 dimensioning, and dynamic energy policies to reduce the electricity 136 consumption. 138 This document presents research and engineering challenges in network 139 virtualization that need to be addressed in order to achieve these 140 goals, spanning from pure research and engineering/standards space. 141 The objective of this memo is to document the technical challenges 142 and corresponding current approaches and to expose requirements that 143 should be addressed by future research and standards work. 145 This document represents the consensus of the NFV Research Group. It 146 has been reviewed by the Research Group members active in the 147 specific areas of work covered by the document. 149 2. Terminology 151 The following terms used in this document are defined by the ETSI 152 Network Function Virtualization (NVF) Industrial Study Group (ISG) 153 [etsi_gs_nfv_003], the ONF [onf_tr_521] and the IETF [RFC7426] 154 [RFC7665]: 156 Application Plane - The collection of applications and services 157 that program network behavior. 159 Control Plane (CP) - The collection of functions responsible for 160 controlling one or more network devices. CP instructs network 161 devices with respect to how to process and forward packets. The 162 control plane interacts primarily with the forwarding plane and, 163 to a lesser extent, with the operational plane. 165 Forwarding Plane (FP) - The collection of resources across all 166 network devices responsible for forwarding traffic. 168 Management Plane (MP) - The collection of functions responsible 169 for monitoring, configuring, and maintaining one or more network 170 devices or parts of network devices. The management plane is 171 mostly related to the operational plane (it is related less to the 172 forwarding plane). 174 NFV Infrastructure (NFVI): totality of all hardware and software 175 components which build up the environment in which VNFs are 176 deployed. 178 NFV Management and Orchestration (NFV-MANO): functions 179 collectively provided by NFVO, VNFM, and VIM. 181 NFV Orchestrator (NFVO): functional block that manages the Network 182 Service (NS) lifecycle and coordinates the management of NS 183 lifecycle, VNF lifecycle (supported by the VNFM) and NFVI 184 resources (supported by the VIM) to ensure an optimized allocation 185 of the necessary resources and connectivity. 187 Operational Plane (OP) - The collection of resources responsible 188 for managing the overall operation of individual network devices. 190 Physical Network Function (PNF): Physical implementation of a 191 Network Function in a monolithic realization. 193 Service Function Chain (SFC): for a given service, the abstracted 194 view of the required service functions and the order in which they 195 are to be applied. This is somehow equivalent to the Network 196 Function Forwarding Graph (NF-FG) at ETSI. 198 Service Function Path (SFP): the selection of specific service 199 function instances on specific network nodes to form a service 200 graph through which an SFC is instantiated. 202 Virtualized Infrastructure Manager (VIM): functional block that is 203 responsible for controlling and managing the NFVI compute, storage 204 and network resources, usually within one infrastructure 205 operator's Domain. 207 Virtualized Network Function (VNF): implementation of a Network 208 Function that can be deployed on a Network Function Virtualization 209 Infrastructure (NFVI). 211 Virtualized Network Function Manager (VNFM): functional block that 212 is responsible for the lifecycle management of VNF. 214 3. Background 216 This section briefly describes some basic background technologies, as 217 well as other standards developing organizations and open source 218 initiatives working on network virtualization or related topics. 220 3.1. Network Function Virtualization 222 The ETSI ISG NFV is a working group which, since 2012, aims to evolve 223 quasi-standard IT virtualization technology to consolidate many 224 network equipment types into industry standard high volume servers, 225 switches, and storage. It enables implementing network functions in 226 software that can run on a range of industry standard server hardware 227 and can be moved to, or loaded in, various locations in the network 228 as required, without the need to install new equipment. The ETSI NFV 229 is one of the predominant NFV reference framework and architectural 230 footprints [nfv_sota_research_challenges]. The ETSI NFV framework 231 architecture framework is composed of three domains (Figure 1): 233 o Virtualized Network Function, running over the NFVI. 235 o NFV Infrastructure (NFVI), including the diversity of physical 236 resources and how these can be virtualized. NFVI supports the 237 execution of the VNFs. 239 o NFV Management and Orchestration, which covers the orchestration 240 and life-cycle management of physical and/or software resources 241 that support the infrastructure virtualization, and the life-cycle 242 management of VNFs. NFV Management and Orchestration focuses on 243 all virtualization specific management tasks necessary in the NFV 244 framework. 246 +-------------------------------------------+ +---------------+ 247 | Virtualized Network Functions (VNFs) | | | 248 | ------- ------- ------- ------- | | | 249 | | | | | | | | | | | | 250 | | VNF | | VNF | | VNF | | VNF | | | | 251 | | | | | | | | | | | | 252 | ------- ------- ------- ------- | | | 253 +-------------------------------------------+ | | 254 | | 255 +-------------------------------------------+ | | 256 | NFV Infrastructure (NFVI) | | NFV | 257 | ----------- ----------- ----------- | | Management | 258 | | Virtual | | Virtual | | Virtual | | | and | 259 | | Compute | | Storage | | Network | | | Orchestration | 260 | ----------- ----------- ----------- | | | 261 | +---------------------------------------+ | | | 262 | | Virtualization Layer | | | | 263 | +---------------------------------------+ | | | 264 | +---------------------------------------+ | | | 265 | | ----------- ----------- ----------- | | | | 266 | | | Compute | | Storage | | Network | | | | | 267 | | ----------- ----------- ----------- | | | | 268 | | Hardware resources | | | | 269 | +---------------------------------------+ | | | 270 +-------------------------------------------+ +---------------+ 272 Figure 1: ETSI NFV framework 274 The NFV architectural framework identifies functional blocks and the 275 main reference points between such blocks. Some of these are already 276 present in current deployments, whilst others might be necessary 277 additions in order to support the virtualization process and 278 consequent operation. The functional blocks are (Figure 2): 280 o Virtualized Network Function (VNF). 282 o Element Management (EM). 284 o NFV Infrastructure, including: Hardware and virtualized resources, 285 and Virtualization Layer. 287 o Virtualized Infrastructure Manager(s) (VIM). 289 o NFV Orchestrator. 291 o VNF Manager(s). 293 o Service, VNF and Infrastructure Description. 295 o Operations and Business Support Systems (OSS/BSS). 297 +--------------------+ 298 +-------------------------------------------+ | ---------------- | 299 | OSS/BSS | | | NFV | | 300 +-------------------------------------------+ | | Orchestrator +-- | 301 | ---+------------ | | 302 +-------------------------------------------+ | | | | 303 | --------- --------- --------- | | | | | 304 | | EM 1 | | EM 2 | | EM 3 | | | | | | 305 | ----+---- ----+---- ----+---- | | ---+---------- | | 306 | | | | |--|-| VNF | | | 307 | ----+---- ----+---- ----+---- | | | manager(s) | | | 308 | | VNF 1 | | VNF 2 | | VNF 3 | | | ---+---------- | | 309 | ----+---- ----+---- ----+---- | | | | | 310 +------|-------------|-------------|--------+ | | | | 311 | | | | | | | 312 +------+-------------+-------------+--------+ | | | | 313 | NFV Infrastructure (NFVI) | | | | | 314 | ----------- ----------- ----------- | | | | | 315 | | Virtual | | Virtual | | Virtual | | | | | | 316 | | Compute | | Storage | | Network | | | | | | 317 | ----------- ----------- ----------- | | ---+------ | | 318 | +---------------------------------------+ | | | | | | 319 | | Virtualization Layer | |--|-| VIM(s) +-------- | 320 | +---------------------------------------+ | | | | | 321 | +---------------------------------------+ | | ---------- | 322 | | ----------- ----------- ----------- | | | | 323 | | | Compute | | Storage | | Network | | | | | 324 | | | hardware| | hardware| | hardware| | | | | 325 | | ----------- ----------- ----------- | | | | 326 | | Hardware resources | | | NFV Management | 327 | +---------------------------------------+ | | and Orchestration | 328 +-------------------------------------------+ +--------------------+ 330 Figure 2: ETSI NFV reference architecture 332 3.2. Software Defined Networking 334 The Software Defined Networking (SDN) paradigm pushes the 335 intelligence currently residing in the network elements to a central 336 controller implementing the network functionality through software. 338 In contrast to traditional approaches, in which the network's control 339 plane is distributed throughout all network devices, with SDN the 340 control plane is logically centralized. In this way, the deployment 341 of new characteristics in the network no longer requires complex and 342 costly changes in equipment or firmware updates, but only a change in 343 the software running in the controller. The main advantage of this 344 approach is the flexibility it provides operators to manage their 345 network, i.e., an operator can easily change its policies on how 346 traffic is distributed throughout the network. 348 One of the most well known protocols for the SDN control plane 349 between the central controller and the networking elements is the 350 OpenFlow protocol (OFP), which is maintained and extended by the Open 351 Network Foundation (ONF: https://www.opennetworking.org/). 352 Originally this protocol was developed specifically for IEEE 802.1 353 switches conforming to the ONF OpenFlow Switch specification. As the 354 benefits of the SDN paradigm have reached a wider audience, its 355 application has been extended to more complex scenarios such as 356 Wireless and Mobile networks. Within this area of work, the ONF is 357 actively developing new OFP extensions addressing three key 358 scenarios: (i) Wireless backhaul, (ii) Cellular Evolved Packet Core 359 (EPC), and (iii) Unified access and management across enterprise 360 wireless and fixed networks. 362 +----------+ 363 | ------- | 364 | |Oper.| | O 365 | |Mgmt.| |<........> -+- Network Operator 366 | |Iface| | ^ 367 | ------- | +----------------------------------------+ 368 | | | +------------------------------------+ | 369 | | | | --------- --------- --------- | | 370 |--------- | | | | App 1 | | App 2 | ... | App n | | | 371 ||Plugins| |<....>| | --------- --------- --------- | | 372 |--------- | | | Plugins | | 373 | | | +------------------------------------+ | 374 | | | Application Plane | 375 | | +----------------------------------------+ 376 | | A 377 | | | 378 | | V 379 | | +----------------------------------------+ 380 | | | +------------------------------------+ | 381 |--------- | | | ------------ ------------ | | 382 || Netw. | | | | | Module 1 | | Module 2 | | | 383 ||Engine | |<....>| | ------------ ------------ | | 384 |--------- | | | Network Engine | | 385 | | | +------------------------------------+ | 386 | | | Controller Plane | 387 | | +----------------------------------------+ 388 | | A 389 | | | 390 | | V 391 | | +----------------------------------------+ 392 | | | +--------------+ +--------------+ | 393 | | | | ------------ | | ------------ | | 394 |----------| | | | OpenFlow | | | | OpenFlow | | | 395 ||OpenFlow||<....>| | ------------ | | ------------ | | 396 |----------| | | NE | | NE | | 397 | | | +--------------+ +--------------+ | 398 | | | Data Plane | 399 |Management| +----------------------------------------+ 400 +----------+ 402 Figure 3: High level SDN ONF architecture 404 Figure 3 shows the blocks and the functional interfaces of the ONF 405 architecture, which comprises three planes: Data, Controller, and 406 Application. The Data plane comprehends several Network Entities 407 (NE), which expose their capabilities toward the Controller plane via 408 a Southbound API. The Controller plane includes several cooperating 409 modules devoted to the creation and maintenance of an abstracted 410 resource model of the underlying network. Such model is exposed to 411 the applications via a Northbound API where the Application plane 412 comprises several applications/services, each of which has exclusive 413 control of a set of exposed resources. 415 The Management plane spans its functionality across all planes 416 performing the initial configuration of the network elements in the 417 Data plane, the assignment of the SDN controller and the resources 418 under its responsibility. In the Controller plane, the Management 419 needs to configure the policies defining the scope of the control 420 given to the SDN applications, to monitor the performance of the 421 system, and to configure the parameters required by the SDN 422 controller modules. In the Application plane, Management configures 423 the parameters of the applications and the service level agreements. 424 In addition to these interactions, the Management plane exposes 425 several functions to network operators which can easily and quickly 426 configure and tune the network at each layer. 428 In RFC7426 [RFC7426], the IRTF Software-Defined Networking Research 429 Group (SDNRG) documented a layer model of an SDN architecture, since 430 this has been a controversial discussion topic: what exactly is SDN? 431 what is the layer structure of the SDN architecture? how do layers 432 interface with each other? etc. 434 Figure 4 reproduces the figure included in RFC7426 [RFC7426] to 435 summarize the SDN architecture abstractions in the form of a 436 detailed, high-level schematic. In a particular implementation, 437 planes can be collocated with other planes or can be physically 438 separated. 440 In SDN, a controller manipulates controlled entities via an 441 interface. Interfaces, when local, are mostly API invocations 442 through some library or system call. However, such interfaces may be 443 extended via some protocol definition, which may use local inter- 444 process communication (IPC) or a protocol that could also act 445 remotely; the protocol may be defined as an open standard or in a 446 proprietary manner. 448 SDN expands multiple planes: Forwarding, Operational, Control, 449 Management and Applications. All planes mentioned above are 450 connected via interfaces. Additionally, RFC7426 [RFC7426] considers 451 four abstraction layers: the Device and resource Abstraction Layer 452 (DAL), the Control Abstraction Layer (CAL), the Management 453 Abstraction Layer (MAL) and the Network Services Abstraction Layer 454 (NSAL). 456 o--------------------------------o 457 | | 458 | +-------------+ +----------+ | 459 | | Application | | Service | | 460 | +-------------+ +----------+ | 461 | Application Plane | 462 o---------------Y----------------o 463 | 464 *-----------------------------Y---------------------------------* 465 | Network Services Abstraction Layer (NSAL) | 466 *------Y------------------------------------------------Y-------* 467 | | 468 | Service Interface | 469 | | 470 o------Y------------------o o---------------------Y------o 471 | | Control Plane | | Management Plane | | 472 | +----Y----+ +-----+ | | +-----+ +----Y----+ | 473 | | Service | | App | | | | App | | Service | | 474 | +----Y----+ +--Y--+ | | +--Y--+ +----Y----+ | 475 | | | | | | | | 476 | *----Y-----------Y----* | | *---Y---------------Y----* | 477 | | Control Abstraction | | | | Management Abstraction | | 478 | | Layer (CAL) | | | | Layer (MAL) | | 479 | *----------Y----------* | | *----------Y-------------* | 480 | | | | | | 481 o------------|------------o o------------|---------------o 482 | | 483 | CP | MP 484 | Southbound | Southbound 485 | Interface | Interface 486 | | 487 *------------Y---------------------------------Y----------------* 488 | Device and resource Abstraction Layer (DAL) | 489 *------------Y---------------------------------Y----------------* 490 | | | | 491 | o-------Y----------o +-----+ o--------Y----------o | 492 | | Forwarding Plane | | App | | Operational Plane | | 493 | o------------------o +-----+ o-------------------o | 494 | Network Device | 495 +---------------------------------------------------------------+ 497 Figure 4: SDN Layer Architecture 499 While SDN is often directly associated to OpenFlow, this is just one 500 (relevant) example of a southbound protocol between the central 501 controller and the network entities. Other relevant examples of 502 protocols in the SDN family are NETCONF [RFC6241], RESTCONF [RFC8040] 503 and ForCES [RFC5810]. 505 3.3. ITU-T functional architecture of SDN 507 The Telecommunication standardization sector of the International 508 Telecommunication Union (ITU) -- the ITU-T -- has also looked into 509 SDN architectures, defining a slightly modified one from what other 510 SDOs have done. ITU-T provides in the recommendation ITU-T Y.3302 511 [itu-t-y.3302] a functional architecture of SDN with descriptions of 512 functional components and reference points. The described functional 513 architecture is intended to be used as an enabler for further studies 514 on other aspects such as protocols and security as well as being used 515 to customize SDN in support of appropriate use cases (e.g., cloud 516 computing, mobile networks). This recommendation is based on ITU-T 517 Y.3300 [itu-t-y.3300] and ITU-T Y.3301 [itu-t-y.3301]. While the 518 first describes the framework of SDN (including definitions, 519 objectives, high-level capabilities, requirements and the high-level 520 architecture of SDN), the second describes more detailed 521 requirements. 523 Figure 5 shows the SDN functional architecture defined by the ITU-T. 524 It is a layered architecture composed of the SDN application layer 525 (SDN-AL), the SDN control layer (SDN-CL) and the SDN resource layer 526 (SDN-RL). It also has multi-layer management functions (MMF), which 527 provides functionalities for managing the functionalities of SDN 528 layers, i.e., SDN-AL, SDN-CL and SDN-RL. MMF interacts with these 529 layers using MMFA, MMFC, and MMFR reference points. 531 The SDN-AL enables a service-aware behavior of the underlying network 532 in a programmatic manner. The SDN-CL provides programmable means to 533 control the behavior of SDN-RL resources (such as data transport and 534 processing), following requests received from the SDN-AL according to 535 MMF policies. The SDN-RL is where the physical or virtual network 536 elements perform transport and/or processing of data packets 537 according to SDN-CL decisions. 539 MMFO MMFA 540 +-----+ . +---------------------+ . +--------------------+ 541 | | . |+---+ +---+ +-------+| . |+---------+ +-----+ | 542 | | . || | | | | || . || AL. | | | | 543 | | . || E | | | | App. || . || mngmt. | | SDN | | SDN-AL 544 | | . || x | | M | | layer || . || support | | app | | 545 | | . || t.| | u | | Mngmt.|| . || & orch. | | | | 546 | | . || | | l | +-------+| . |+---------+ +-----+ | 547 | | . || r | | t | | . +--------------------+ 548 | | . || e | | i | |MMFC ..................... ACI 549 | | . || l | | | | . +--------------------+ 550 | | . || a | | l | +-------+| . |+------+ +---------+| 551 | OSS/| . || t | | a | | || . || | | App. || 552 | BSS | . || i | | y | | || . || | | support || 553 | | . || o | | e | | || . || | +---------+| 554 | | . || n | | r | | || . || CL | +---------+| 555 | | . || s | | | |Control|| . ||mngmt.| | Control || 556 | | . || h | | m | | layer || . || supp.| | layer || SDN-CL 557 | | . || i | | a | | mngmt.|| . || and | | serv. || 558 | | . || p | | n | | || . || orch.| +---------+| 559 | | . || | | a | | || . || | +---------+| 560 | | . || m | | g | | || . || | | Resource|| 561 | | . || n | | e | | || . || | | abstrac.|| 562 | | . || g | | m | +-------+| . |+------+ +---------+| 563 | | . || m | | e | | . +--------------------+ 564 | | . || t.| | n | |MMFR ..................... RCI 565 | | . || | | t | | . +--------------------+ 566 +-----+ . |+---+ | | +-------+| . |+------++----------+| 567 | | o | | || . || ||RL control|| 568 | | r | |Resour.|| . || RL |+----------+| 569 MMF | | c | | layer || . ||mngmt.|+----++----+| SDN-RL 570 | | h.| | mngmt.|| . || supp.||Data||Data|| 571 | | | | || . || ||tran||proc|| 572 | +---+ +-------+| . |+------++----++----+| 573 +---------------------+ . +--------------------+ 575 Figure 5: ITU-T SDN functional architecture 577 3.4. Multi-access Edge Computing 579 Multi-access Edge Computing (MEC) -- formerly known as Mobile Edge 580 Computing -- capabilities deployed in the edge of the mobile network 581 can facilitate the efficient and dynamic provision of services to 582 mobile users. The ETSI ISG MEC working group, operative from end of 583 2014, intends to specify an open environment for integrating MEC 584 capabilities with service providers' networks, including also 585 applications from 3rd parties. These distributed computing 586 capabilities will make available IT infrastructure as in a cloud 587 environment for the deployment of functions in mobile access 588 networks. It can be seen then as a complement to both NFV and SDN. 590 3.5. IEEE 802.1CF (OmniRAN) 592 The IEEE 802.1CF Recommended Practice [omniran] specifies an access 593 network, which connects terminals to their access routers, utilizing 594 technologies based on the family of IEEE 802 Standards (e.g., 802.3 595 Ethernet, 802.11 Wi-Fi, etc.). The specification defines an access 596 network reference model, including entities and reference points 597 along with behavioral and functional descriptions of communications 598 among those entities. 600 The goal of this project is to help unifying the support of different 601 interfaces, enabling shared network control and use of SDN 602 principles, thereby lowering the barriers to new network 603 technologies, to new network operators, and to new service providers. 605 3.6. Distributed Management Task Force 607 The DMTF (https://www.dmtf.org/) is an industry standards 608 organization working to simplify the manageability of network- 609 accessible technologies through open and collaborative efforts by 610 some technology companies. The DMTF is involved in the creation and 611 adoption of interoperable management standards, supporting 612 implementations that enable the management of diverse traditional and 613 emerging technologies including cloud, virtualization, network and 614 infrastructure. 616 There are several DMTF initiatives that are relevant to the network 617 virtualization area, such as the Open Virtualization Format (OVF), 618 for VNF packaging; the Cloud Infrastructure Management Interface 619 (CIM), for cloud infrastructure management; the Network Management 620 (NETMAN), for VNF management; and, the Virtualization Management 621 (VMAN), for virtualization infrastructure management. 623 3.7. Open Source initiatives 625 The Open Source community is especially active in the area of network 626 virtualization and orchestration. We next summarize some of the 627 active efforts: 629 o OpenStack. OpenStack is a free and open-source cloud-computing 630 software platform. OpenStack software controls large pools of 631 compute, storage, and networking resources throughout a 632 datacenter, managed through a dashboard or via the OpenStack API. 634 o Kubernetes. Kubernetes is an open-source system for automating 635 deployment, scaling and management of containerized applications. 636 Kubernetes can schedule and run application containers on clusters 637 of physical or virtual machines. Kubernetes allows: (i) Scale on 638 the fly, (ii) Limit hardware usage to required resources only, 639 (iii) Load balancing Monitoring, and (iv) Efficient lifecycle 640 management. 642 o OpenDayLight. OpenDaylight (ODL) is a highly available, modular, 643 extensible and scalable multi-protocol controller infrastructure 644 built for SDN deployments on modern heterogeneous multi-vendor 645 networks. It provides a model-driven service abstraction platform 646 that allows users to write apps that easily work across a wide 647 variety of hardware and southbound protocols. 649 o ONOS. The ONOS (Open Network Operating System) project is an open 650 source community hosted by The Linux Foundation. The goal of the 651 project is to create a SDN operating system for communications 652 service providers that is designed for scalability, high 653 performance and high availability. 655 o OpenContrail. OpenContrail is an Apache 2.0-licensed project that 656 is built using standards-based protocols and provides all the 657 necessary components for network virtualization-SDN controller, 658 virtual router, analytics engine, and published northbound APIs. 659 It has an extensive REST API to configure and gather operational 660 and analytics data from the system. 662 o OPNFV. OPNFV is a carrier-grade, integrated, open source platform 663 to accelerate the introduction of new NFV products and services. 664 By integrating components from upstream projects, the OPNFV 665 community aims at conducting performance and use case-based 666 testing to ensure the platform's suitability for NFV use cases. 667 The scope of OPNFV's initial release is focused on building NFV 668 Infrastructure (NFVI) and Virtualized Infrastructure Management 669 (VIM) by integrating components from upstream projects such as 670 OpenDaylight, OpenStack, Ceph Storage, KVM, Open vSwitch, and 671 Linux. These components, along with application programmable 672 interfaces (APIs) to other NFV elements form the basic 673 infrastructure required for Virtualized Network Functions (VNF) 674 and Management and Network Orchestration (MANO) components. 675 OPNFV's goal is to (i) increase performance and power efficiency, 676 (ii) improve reliability, availability, and serviceability, and 677 (iii) deliver comprehensive platform instrumentation. 679 o OSM. Open Source Mano (OSM) is an ETSI-hosted project to develop 680 an Open Source NFV Management and Orchestration (MANO) software 681 stack aligned with ETSI NFV. OSM is based on components from 682 previous projects, such Telefonica's OpenMANO or Canonical's Juju, 683 among others. 685 o OpenBaton. OpenBaton is a ETSI NFV compliant Network Function 686 Virtualization Orchestrator (NFVO). OpenBaton was part of the 687 OpenSDNCore project started with the objective of providing a 688 compliant implementation of the ETSI NFV specification. 690 o ONAP. ONAP (Open Network Automation Platform) is an open source 691 software platform that delivers capabilities for the design, 692 creation, orchestration, monitoring, and life cycle management of: 693 (i) Virtual Network Functions (VNFs), (ii) The carrier-scale 694 Software Defined Networks (SDNs) that contain them, and (iii) 695 Higher-level services that combine the above. ONAP (derived from 696 the AT&T's ECOMP) provides for automatic, policy-driven 697 interaction of these functions and services in a dynamic, real- 698 time cloud environment. 700 o SONA. SONA (Simplified Overlay Network Architecture) is an 701 extension to ONOS to have a almost full SDN network control in 702 OpenStack for virtual tenant network provisioning. Basically, 703 SONA is an SDN-based network virtualization solution for cloud DC. 705 Among the main areas that are being developed by the former open 706 source activities that relate to network virtualization research, we 707 can highlight: policy-based resource management, analytics for 708 visibility and orchestration, service verification with regards to 709 security and resiliency. 711 4. Network Virtualization Challenges 713 4.1. Introduction 715 Network Virtualization is changing the way the telecommunications 716 sector will deploy, extend and operate their networks. These new 717 technologies aim at reducing the overall costs by moving 718 communication services from specific hardware in the operators' core 719 to server farms scattered in datacenters (i.e. compute and storage 720 virtualization). In addition, the networks interconnecting the 721 functions that compose a network service are fundamentally affected 722 in the way they route, process and control traffic (i.e. network 723 virtualization). 725 4.2. Guaranteeing quality-of-service 727 Achieving a given quality-of-service in an NFV environment with 728 virtualized and distributed computing, storage and networking 729 functions is more challenging than providing the equivalent in 730 discrete non-virtualized components. For example, ensuring a 731 guaranteed and stable forwarding data rate has proven not to be 732 straightforward when the forwarding function is virtualized and runs 733 on top of COTS server hardware [openmano_dataplane] 734 [I-D.mlk-nfvrg-nfv-reliability-using-cots] [etsi_nvf_whitepaper_3]. 735 Again, the comparison point is against a router or forwarder built on 736 optimized hardware. We next identify some of the challenges that 737 this poses. 739 4.2.1. Virtualization Technologies 741 The issue of guaranteeing a network quality-of-service is less of an 742 issue for "traditional cloud computing" because the workloads that 743 are treated there are servers or clients in the networking sense and 744 hardly ever process packets. Cloud computing provides hosting for 745 applications on shared servers in a highly separated way. Its main 746 advantage is that the infrastructure costs are shared among tenants 747 and that the cloud infrastructure provides levels of reliability that 748 can not be achieved on individual premises in a cost-efficient way 749 [intel_10_differences_nfv_cloud]. NFV has very strict requirements 750 posed in terms of performance, stability and consistency. Although 751 there are some tools and mechanisms to improve this, such as Enhanced 752 Performance Awareness (EPA), Single Root I/O Virtualization (SR-IOV), 753 Non-Uniform Memory Access (NUMA), Data Plane Development Kit (DPDK), 754 etc, these are still unsolved challenges. One open research issue is 755 finding out technologies that are different from VM and more suitable 756 for dealing with network functionalities. 758 Lately, a number of light-weight virtualization technologies 759 including containers, unikernels (specialized VMs) and minimalistic 760 distributions of general-purpose OSes have appeared as virtualization 761 approaches that can be used when constructing an NFV platform. 762 [I-D.natarajan-nfvrg-containers-for-nfv] describes the challenges in 763 building such a platform and discusses to what extent these 764 technologies, as well as traditional VMs, are able to address them. 766 4.2.2. Metrics for NFV characterization 768 Another relevant aspect is the need for tools for diagnostics and 769 measurement suited for NFV. There is a pressing need to define 770 metrics and associated protocols to measure the performance of NFV. 771 Specifically, since NFV is based on the concept of taking centralized 772 functions and evolving it to highly distributed SW functions, there 773 is a commensurate need to fully understand and measure the baseline 774 performance of such systems. 776 The IP Performance Metrics (IPPM) WG defines metrics that can be used 777 to measure the quality and performance of Internet services and 778 applications running over transport layer protocols (e.g., TCP, UDP) 779 over IP. It also develops and maintains protocols for the 780 measurement of these metrics. While the IPPM WG is a long running WG 781 that started in 1997, at the time of writing it does not have a 782 charter item or active drafts related to the topic of network 783 virtualization. In addition to using IPPM metrics to evaluate the 784 QoS, there is a need for specific metrics for assessing the 785 performance of network virtualization techniques. 787 The Benchmarking Methodology Working Group (BMWG) is also performing 788 work related to NFV metrics. For example, [RFC8172] investigates 789 additional methodological considerations necessary when benchmarking 790 VNFs instantiated and hosted in general-purpose hardware, using bare- 791 metal hypervisors or other isolation environments such as Linux 792 containers. An essential consideration is benchmarking physical and 793 virtual network functions in the same way when possible, thereby 794 allowing direct comparison. 796 As stated in the document [RFC8172], there is a clear motivation for 797 the work on performance metrics for NFV [etsi_gs_nfv_per_001], that 798 is worth replicating here: "I'm designing and building my NFV 799 Infrastructure platform. The first steps were easy because I had a 800 small number of categories of VNFs to support and the VNF vendor gave 801 HW recommendations that I followed. Now I need to deploy more VNFs 802 from new vendors, and there are different hardware recommendations. 803 How well will the new VNFs perform on my existing hardware? Which 804 among several new VNFs in a given category are most efficient in 805 terms of capacity they deliver? And, when I operate multiple 806 categories of VNFs (and PNFs) *concurrently* on a hardware platform 807 such that they share resources, what are the new performance limits, 808 and what are the software design choices I can make to optimize my 809 chosen hardware platform? Conversely, what hardware platform 810 upgrades should I pursue to increase the capacity of these 811 concurrently operating VNFs?" 813 Lately, there are also some efforts looking into VNF benchmarking. 814 The selection of an NFV Infrastructure Point of Presence to host a 815 VNF or allocation of resources (e.g., virtual CPUs, memory) needs to 816 be done over virtualized (abstracted and simplified) resource views 817 [vnf_benchmarking] [I-D.rorosz-nfvrg-vbaas]. 819 4.2.3. Predictive analysis 821 On top of diagnostic tools that enable an assessment of the QoS, 822 predictive analyses are required to react before anomalies occur. 823 Due to the SW characteristics of VNFs, a reliable diagnosis framework 824 could potentially enable the prevention of issues by a proper 825 diagnosis and then a reaction in terms of acting on the potentially 826 impacted service (e.g., migration to a different compute node, 827 scaling in/out, up/down, etc). 829 4.2.4. Portability 831 Portability in NFV refers to the ability to run a given VNF on 832 multiple NFVIs, that is, guaranteeing that the VNF would be able to 833 perform its functions with a high and predictable performance given 834 that a set of requirements on the NFVI resources is met. Therefore, 835 portability is a key feature that, if fully enabled, would contribute 836 to making the NFV environment achieve a better reliability than a 837 traditional system. Implementing functionality in SW over 838 "commodity" infrastructure should make it much easier to port/move 839 functions from one place to another. However this is not yet as 840 ideal as it sounds, and there are aspects that are not fully tackled. 841 The existence of different hypervisors, specific hardware 842 dependencies (e.g., EPA related) or state synchronization aspects are 843 just some examples of trouble-makers for portability purposes. 845 The ETSI NFV ISG is doing work in relation to portability. 846 [etsi_gs_nfv_per_001] provides a list of minimal features which the 847 VM Descriptor and Compute Host Descriptor should contain for the 848 appropriate deployment of VM images over an NFVI (i.e. a "telco 849 datacenter"), in order to guarantee high and predictable performance 850 of data plane workloads while assuring their portability. In 851 addition, the document provides a set of recommendations on the 852 minimum requirements which HW and hypervisor should have for a "telco 853 datacenter" suitable for different workloads (data-plane, control- 854 plane, etc.) present in VNFs. The purpose of this document is to 855 provide the list of VM requirements that should be included in the VM 856 Descriptor template, and the list of HW capabilities that should be 857 included in the Compute Host Descriptor (CHD) to assure predictable 858 high performance. ETSI NFV assumes that the MANO Functions will make 859 the mix & match. There are therefore still several research 860 challenges to be addressed here. 862 4.3. Performance improvement 864 4.3.1. Energy Efficiency 866 Virtualization is typically seen as a direct enabler of energy 867 savings. Some of the enablers for this that are often mentioned 868 [nfv_sota_research_challenges] are: (i) the multiplexing gains 869 achieved by centralizing functions in data centers reduce the overall 870 energy consumed, (ii) the flexibility brought by network 871 programmability enables to switch off infrastructure as needed in a 872 much easier way. However there is still a lot of room for 873 improvement in terms of virtualization techniques to reduce the power 874 consumption, such as enhanced hypervisor technologies. 876 Some additional examples of research topics that could enable energy 877 savings are [nfv_sota_research_challenges]: 879 o Energy aware scaling (e.g., reductions in CPU speeds and partially 880 turning off some hardware com- ponents to meet a given energy 881 consumption target. 883 o Energy-aware function placement. 885 o Scheduling and chaining algorithms, for example adapting the 886 network topology and operating parameters to minimize the 887 operation cost (e.g., tracking energy costs to identify the 888 cheapest prices). 890 Note that it is also important to analyze the trade-off between 891 energy efficiency and network performance. 893 4.3.2. Improved link usage 895 The use of NFV and SDN technologies can help improve link usage. SDN 896 has already shown that it can greatly increase average link 897 utilization (e.g., Google example [google_sdn_wan]). NFV adds more 898 complexity (e.g., due to service function chaining / VNF forwarding 899 graphs) which need to be considered. Aspects like the ones described 900 in [I-D.bagnulo-nfvrg-topology] on NFV data center topology design 901 have to be carefully looked at as well. 903 4.4. Multiple Domains 905 Market fragmentation has resulted in a multitude of network operators 906 each focused on different countries and regions. This makes it 907 difficult to create infrastructure services spanning multiple 908 countries, such as virtual connectivity or compute resources, as no 909 single operator has a footprint everywhere. Cross-domain 910 orchestration of services over multiple administrations or over 911 multi-domain single administrations will allow end-to-end network and 912 service elements to mix in multi-vendor, heterogeneous technology and 913 resource environments [multi-domain_5GEx]. 915 For the specific use case of 'Network as a Service', it becomes even 916 more important to ensure that Cross Domain Orchestration also takes 917 care of hierarchy of networks and their association, with respect to 918 provisioning tunnels and overlays. 920 Multi-domain orchestration is currently an active research topic, 921 which is being tackled, among others, by ETSI NFV ISG and the 5GEx 922 project (https://www.5gex.eu/) [I-D.bernardos-nfvrg-multidomain] 923 [multi-domain_5GEx]. 925 Another side of the multi-domain problem is the integration/ 926 harmonization of different management domains. A key example comes 927 from Multi-access Edge Computing, which, according to ETSI, comes 928 with its own MANO system, and would require to be integrated if 929 interconnected to a generic NFV system. 931 4.5. 5G and Network Slicing 933 From the beginning of all 5G discussions in the research and industry 934 fora, it has been agreed that 5G will have to address much more use 935 cases than the preceding wireless generations, which first focused on 936 voice services, and then on voice and high speed packet data 937 services. In this case, 5G should be able to handle not only the 938 same (or enhanced) voice and packet data services, but also new 939 emerging services like tactile Internet and IoT. These use cases 940 take the requirements to opposite extremes, as some of them require 941 ultra-low latency and higher-speed, whereas some others require 942 ultra-low power consumption and high delay tolerance. 944 Because of these very extreme 5G use cases, it is envisioned that 945 selective combinations of radio access networks and core network 946 components will have to be combined into a given network slice to 947 address the specific requirements of each use case. 949 For example, within the major IoT category, which is perhaps the most 950 disrupting one, some autonomous IoT devices will have very low 951 throughput, will have much longer sleep cycles (and therefore high 952 latency), and a battery life time exceeding by a factor of thousands 953 that of smart phones or some other devices that will have almost 954 continuous control and data communications. Hence, it is envisioned 955 that a customized network slice will have to be stitched together 956 from virtual resources or sub-slices to meet these requirements. 958 The actual definition of network slice from an IP infrastructure 959 viewpoint is currently undergoing intense debate 960 [I-D.geng-coms-problem-statement] [I-D.gdmb-netslices-intro-and-ps] 961 [I-D.defoy-netslices-3gpp-network-slicing] [ngmn_5G_whitepaper]. 962 Network slicing is a key for introducing new actors in existing 963 market at low cost -- by letting new players rent "blocks" of 964 capacity, if the new business model enables performance that meets 965 the application needs (e.g., broadcasting updates to many sensors 966 with satellite broadcasting capabilities). However, more work needs 967 to be done to define the basic architectural approach of how network 968 slices will be defined and formed. For example, is it mostly a 969 matter of defining the appropriate network models (e.g. YANG) to 970 stitch the network slice from existing components. Or do end-to-end 971 timing, synchronization and other low level requirements mean that 972 more fundamental research has to be done. 974 4.5.1. Virtual Network Operators 976 The widespread use/discussion/practice of system and network 977 virtualization technologies has led to new business opportunities, 978 enlarging the offer of IT resources with virtual network and 979 computing resources, among others. As a consequence, the network 980 ecosystem now differentiates between the owner of physical resources, 981 the Infrastructure Provider (InP), and the intermediary that conforms 982 and delivers network services to the final customers, the Virtual 983 Network Operator (VNO). 985 VNOs aim to exploit the virtualized infrastructures to deliver new 986 and improved services to their customers. However, current network 987 virtualization techniques offer poor support for VNOs to control 988 their resources. It has been considered that the InP is responsible 989 for the reliability of the virtual resources but there are several 990 situations in which a VNO requires to gain a finer control on its 991 resources. For instance, dynamic events, such as the identification 992 of new requirements or the detection of incidents within the virtual 993 system, might urge a VNO to quickly reform its virtual infrastructure 994 and resource allocation. However, the interfaces offered by current 995 virtualization platforms do not offer the necessary functions for 996 VNOs to perform the elastic adaptations they require to tackle with 997 their dynamic operation environments. 999 Beyond their heterogeneity, which can be resolved by software 1000 adapters, current virtualization platforms do not have common methods 1001 and functions, so it is difficult for the virtual network controllers 1002 used by the VNOs to actually manage and control virtual resources 1003 instantiated on different platforms, not even considering different 1004 InPs. Therefore it is necessary to reach a common definition of the 1005 functions that should be offered by underlying platforms to give such 1006 overlay controllers the possibility to allocate and deallocate 1007 resources dynamically and get monitoring data about them. 1009 Such common methods should be offered by all underlying controllers, 1010 regardless of being network-oriented (e.g. ODL, ONOS, Ryu) or 1011 computing-oriented (e.g. OpenStack, OpenNebula, Eucalyptus). 1012 Furthermore, it is also important for those platforms to offer some 1013 "PUSH" function to report resource state, avoiding the need for the 1014 VNO's controller to "POLL" for such data. A starting point to get 1015 proper notifications within current REST APIs could be to consider 1016 the protocol proposed by the WEBPUSH WG [RFC8030]. 1018 Finally, in order to establish a proper order and allow the 1019 coexistence and collaboration of different systems, a common ontology 1020 regarding network and system virtualization should be defined and 1021 agreed, so different and heterogeneous systems can understand each 1022 other without requiring to rely on specific adaptation mechanisms 1023 that might break with any update on any side of the relation. 1025 4.5.2. Extending Virtual Networks and Systems to the Internet of Things 1027 The Internet of Things (IoT) refers to the vision of connecting a 1028 multitude of automated devices (e.g. lights, environmental sensors, 1029 traffic lights, parking meters, health and security systems, etc.) to 1030 the Internet for purposes of reporting, and remote command and 1031 control of the device. This vision is being realized by a multi- 1032 pronged approach of standardization in various forums and 1033 complementary open source activities. For example, in the IETF, 1034 support of IoT web services has been defined by an HTTP-like protocol 1035 adapted for IoT called CoAP [RFC7252], and lately a group has been 1036 studying the need to develop a new network layer to support IP 1037 applications over Low Power Wide Area Networks (LPWAN). 1039 Elsewhere, for 5G cellular evolution there is much discussion on the 1040 need for supporting virtual "network slices" for the expected massive 1041 numbers of IoT devices. A separate virtual network slice is 1042 considered necessary for different 5G IoT use cases because devices 1043 will have very different characteristics than typical cellular 1044 devices like smart phones [ngmn_5G_whitepaper], and the number of IoT 1045 devices is expected to be at least one or two orders of magnitude 1046 higher than other 5G devices (see Section 4.5). 1048 The specific nature of the IoT ecosystem, particularly reflected in 1049 the Machine-to-Machine (M2M) communications, leads to the creation of 1050 new and highly distributed systems which demand location-based 1051 network and computing services. A specific example can be 1052 represented by a set of "things" that suddenly require to set-up a 1053 firewall to allow external entities to access their data while 1054 outsourcing some computation requirements to more powerful systems 1055 relying on cloud-based services. This representative use case 1056 exposes important requirements for both NFV and the underlying cloud 1057 infrastructures. 1059 In order to provide the aforementioned location-based functions 1060 integrated with highly distributed systems, the so called fog 1061 infrastructures should be able to instantiate VNFs, placing them in 1062 the required place, e.g. close to their consumers. This requirement 1063 implies that the interfaces offered by virtualization platforms must 1064 support the specification of location-based resources, which is a key 1065 function in those scenarios. Moreover, those platforms must also be 1066 able to interpret and understand the references used by IoT systems 1067 to their location (e.g., "My-AP", "5BLDG+2F") and also the 1068 specification of identifiers linked to other resources, such as the 1069 case of requiring the infrastructure to establish a link between a 1070 specific AP and a specific virtual computing node. In summary, the 1071 research gap is exact localization of VNFs at far network edge 1072 infrastructure which is highly distributed and dynamic. 1074 4.6. Service Composition 1076 Current network services deployed by operators often involve the 1077 composition of several individual functions (such as packet 1078 filtering, deep packet inspection, load balancing). These services 1079 are typically implemented by the ordered combination of a number of 1080 service functions that are deployed at different points within a 1081 network, not necessarily on the direct data path. This requires 1082 traffic to be steered through the required service functions, 1083 wherever they are deployed [RFC7498]. 1085 For a given service, the abstracted view of the required service 1086 functions and the order in which they are to be applied is called a 1087 Service Function Chain (SFC) [sfc_challenges], which is called 1088 Network Function Forwarding Graph (NF-FG) in ETSI. An SFC is 1089 instantiated through selection of specific service function instances 1090 on specific network nodes to form a service graph: this is called a 1091 Service Function Path (SFP). The service functions may be applied at 1092 any layer within the network protocol stack (network layer, transport 1093 layer, application layer, etc.). 1095 Service composition is a powerful means which can provide significant 1096 benefits when applied in a softwarized network environment. There 1097 are however many research challenges in this area, as for example the 1098 ones related to composition mechanisms and algorithms to enable load 1099 balancing and improve reliability. The service composition should 1100 also act as an enabler to gather information across all hierarchies 1101 (underlays and overlays) of network deployments which may span across 1102 multiple operators, for faster serviceability thus facilitating 1103 accomplishing aforementioned goals of "load balancing and improve 1104 reliability". 1106 As described in [dynamic_chaining], different algorithms can be used 1107 to enable dynamic service composition that optimizes a QoS-based 1108 utility function (e.g., minimizing the latency per-application 1109 traffic flows) for a given composition plan. Such algorithms can 1110 consider the computation capabilities and load status of resources 1111 executing the VNF instances, either deduced through estimations from 1112 historical usage data or collected through real-time monitoring 1113 (i.e., context-aware selection). For this reason, selections should 1114 include references to dynamic information on the status of the 1115 service instance and its constituent elements, i.e., monitoring 1116 information related to individual VNF instances and links connecting 1117 them as well as derived monitoring information at the chain level 1118 (e.g., end-to-end delay). At runtime, if one or more VNF instances 1119 are no more available or QoS degrades below a given threshold, the 1120 service selection task can be rerun to perform service substitution. 1122 There are different research directions that relate to the previous 1123 point. For example, the use of Integer Linear Programming (ILP) 1124 techniques can be explored to optimize the management of diverse 1125 traffic flows. Deep machine learning can also be applied to optimize 1126 service chains using information parameters such as some of the ones 1127 mentioned above. Newer scheduling paradigms, like co-flows, can also 1128 be used. 1130 The SFC working group is working on an architecture for service 1131 function chaining [RFC7665] that includes the necessary protocols or 1132 protocol extensions to convey the Service Function Chain and Service 1133 Function Path information to nodes that are involved in the 1134 implementation of service functions and Service Function Chains, as 1135 well as mechanisms for steering traffic through service functions. 1137 In terms of actual work items, the SFC WG is has not yet considered 1138 working on the management and configuration of SFC components related 1139 to the support of Service Function Chaining. This part is of special 1140 interest for operators and would be required in order to actually put 1141 SFC mechanisms into operation. Similarly, redundancy and reliability 1142 mechanisms for service function chaining are currently not dealt with 1143 by any WG in the IETF. While this was the main goal of the VNFpool 1144 BoF efforts, it still remains unaddressed. 1146 4.7. End-user device virtualization 1148 So far, most of the network softwarization efforts have focused on 1149 virtualizing functions of network elements. While virtualization of 1150 network elements started with the core, mobile networks architectures 1151 are now heavily switching to also virtualize radio access network 1152 (RAN) functions. The next natural step is to get virtualization down 1153 at the level of the end-user device (e.g., virtualizing a smartphone) 1154 [virtualization_mobile_device]. The cloning of a device in the cloud 1155 (central or local) bears attractive benefits to both the device and 1156 network operations alike (e.g., power saving at the device by 1157 offloading computational-heaving functions to the cloud, optimized 1158 networking -- both device-to-device and device-to-infrastructure) for 1159 service delivery through tighter integration of the device (via its 1160 clone in the networking infrastructure). This is, for example, being 1161 explored by the European H2020 ICIRRUS project (www.icirrus- 1162 5gnet.eu). 1164 4.8. Security and Privacy 1166 Similar to any other situation where resources are shared, security 1167 and privacy are two important aspects that need to be taken into 1168 account. 1170 In the case of security, there are situations where multiple service 1171 providers will need to coexist in a virtual or hybrid physical/ 1172 virtual environment. This requires attestation procedures amongst 1173 different virtual/physical functions and resources, as well as 1174 ongoing external monitoring. Similarly, different network slices 1175 operating on the same infrastructure can present security problems, 1176 for instance if one slice running critical applications (e.g. support 1177 for a safety system) is affected by another slice running a less 1178 critical application. In general, the minimum common denominator for 1179 security measures on a shared system should be equal or higher than 1180 the one required by the most critical application. Multiple and 1181 continuous threat model analysis, as well as DevOps model are 1182 required to maintain a certain level of security in an NFV system. 1183 Simplistically, DevOps is a process that combines multiple functions 1184 into single cohesive teams in order to quickly produce quality 1185 software. It typically relies on also applying the Agile development 1186 process, which focuses on (among many things) dividing large features 1187 into multiple, smaller deliveries. One part of this is to 1188 immediately test the new smaller features in order to get immediate 1189 feedback on errors so that if present, they can be immediately fixed 1190 and redeployed. 1192 On the other hand, privacy refers to concerns about the control of 1193 personal data and the decision of what to reveal to whom. In this 1194 case, the storage, transmission, collection, and potential 1195 correlation of information in the NFV system, for purposes not 1196 originally intended or not known by the user, should be avoided. 1197 This is particularly challenging, as future intentions and threats 1198 cannot be easily predicted, and still can be applied on data 1199 collected in the past. Therefore, well-known techniques such as data 1200 minimization, using privacy features as default, and allowing users 1201 to opt in/out should be used to prevent potential privacy issues. 1203 Compared to traditional networks, NFV will result in networks that 1204 are much more dynamic (in function distribution and topology) and 1205 elastic (in size and boundaries). NFV will thus require network 1206 operators to evolve their operational and administrative security 1207 solutions to work in this new environment. For example, in NFV the 1208 network orchestrator will become a key node to provide security 1209 policy orchestration across the different physical and virtual 1210 components of the virtualized network. For highly confidential data, 1211 for example, the network orchestrator should take into account if 1212 certain physical hardware (HW) of the network is considered more 1213 secure (e.g., because it is located in secure premises) than other 1214 HW. 1216 Traditional telecom networks typically run under a single 1217 administrative domain controlled by (exactly) one operator. With 1218 NFV, it is expected that in many cases, the telecom operator will now 1219 become a tenant (running the VNFs), and the infrastructure (NFVI) may 1220 be run by a different operator and/or cloud service provider (see 1221 also Section 4.4). Thus, there will be multiple administrative 1222 domains involved, making security policy coordination more complex. 1223 For example, who will be in charge of provisioning and maintaining 1224 security credentials such as public and private keys? Also, should 1225 private keys be allowed to be replicated across the NFV for 1226 redundancy reasons? Alternatively, it can be investigated how to 1227 develop a mechanism that avoid such a security policy coordination, 1228 this making the system more robust. 1230 On a positive note, NFV may better defense against Denial of Service 1231 (DoS) attacks because of the distributed nature of the network (i.e. 1232 no single point of failure) and the ability to steer (undesirable) 1233 traffic quickly [etsi_gs_nfv_sec_001]. Also, NFVs which have 1234 physical HW which is distributed across multiple data centers will 1235 also provide better fault isolation environments. This holds true in 1236 particular if each data center is protected separately via firewalls, 1237 DMZs and other network protection techniques. 1239 SDN can also be used to help improve security by facilitating the 1240 operation of existing protocols, such as Authentication, 1241 Authorization and Accounting (AAA). The management of AAA 1242 infrastructures, namely the management of AAA routing and the 1243 establishment of security associations between AAA entities, can be 1244 performed using SDN, as analyzed in [I-D.marin-sdnrg-sdn-aaa-mng]. 1246 4.9. Separation of control concerns 1248 NFV environments offer two possible levels of SDN control. One level 1249 is the need for controlling the NFVI to provide connectivity end-to- 1250 end among VNFs or among VNFs and PNFs (Physical Network Functions). 1251 A second level is the control and configuration of the VNFs 1252 themselves (in other words, the configuration of the network service 1253 implemented by those VNFs), taking advantage of the programmability 1254 brought by SDN. Both control concerns are separated in nature. 1256 However, interaction between both could be expected in order to 1257 optimize, scale or influence each other. 1259 Clear mechanisms for such interaction are needed in order to avoid 1260 malfunctioning or interference concerns. These ideas are considered 1261 in [etsi_gs_nfv_eve005] and [I-D.irtf-sdnrg-layered-sdn] 1263 4.10. Network Function placement 1265 Network function placement is a problem in any kind of network 1266 telecommunications infrastructure. Moreover, the increased degree of 1267 freedom added by network virtualization makes this problem even more 1268 important, and also harder to tackle. Deciding where to place 1269 virtual network functions is a resource allocation problem which 1270 needs to (or may) take into consideration quite a few aspects: 1271 resiliency, (anti-)affinity, security, privacy, energy efficiency, 1272 etc. 1274 When several functions are chained (typical scenario), placement 1275 algorithms become more complex and important (as described in 1276 Section 4.6). While there has been research on the topic 1277 [nfv_piecing] [dynamic_placement][vnf-p], this still remains an open 1278 challenges that requires more attention. Multi-domain also adds 1279 another component of complexity to this problem that has to be 1280 considered. 1282 4.11. Testing 1284 The impacts of network virtualization on testing can be divided into 1285 3 groups: 1287 1. Changes in methodology. 1289 2. New functionality. 1291 3. Opportunities. 1293 4.11.1. Changes in methodology 1295 The largest impact of NFV is the ability to isolate the System Under 1296 Test (SUT). When testing Physical Network Functions (PNF), isolating 1297 the SUT means that all the other devices that the SUT communicates 1298 with are replaced with simulations (or controlled executions) in 1299 order to place the SUT under test by itself. The SUT may be 1300 comprised of one or more devices. The simulations use the 1301 appropriate traffic type and protocols in order to execute test 1302 cases. 1304 As shown in Figure 2, NFV provides a common architecture for all 1305 functions to use. A VNF is executed using resources offered by the 1306 NFVI, which have been allocated using the MANO function. It is not 1307 possible to test a VNF by itself, without the entire supporting 1308 environment present. This fundamentally changes how to consider the 1309 SUT. In the case of a VNF (or multiple VNFs), the SUT is part of a 1310 larger architecture which is necessary in order to run the SUTs. 1312 Isolation of the SUT therefore becomes controlling the environment in 1313 a disciplined manner. The components of the environment necessary to 1314 run the SUTs that are not part of the SUT become the test 1315 environment. In the case of VNFs which are the SUT, the NFVI and 1316 MANO become the test environment. The configurations and policies 1317 that guide the test environment should remain constant during the 1318 execution of the tests, and also from test to test. Configurations 1319 such as CPU pinning, NUMA configuration, the SW versions and 1320 configurations of the hypervisor, vSwitch and NICs should remain 1321 constant. The only variables in the testing should be those 1322 controlling the SUT itself. If any configuration in the test 1323 environment is changed from test to test, the results become very 1324 difficult, if not impossible, to compare since the test environment 1325 behavior may change the results as a consequence of the configuration 1326 change. 1328 Testing the NFVI itself also presents new considerations. With a 1329 PNF, the dedicated hardware supporting it is optimized for the 1330 particular workload of the function. Routing hardware is specially 1331 built to support packet forwarding functions, while the hardware to 1332 support a purely control plane application (say, a DNS server, or a 1333 Diameter function) will not have this specialized capability. In 1334 NFV, the NFVI is required to support all types of potentially 1335 different workload types. 1337 Testing the NFVI therefore requires careful consideration about what 1338 types of metrics are sought. This, in turn, depends on the workload 1339 type the expected VNF will be. Examples of different workload types 1340 are data forwarding, control plane, encryption, and authentication. 1341 All these types of expected workloads will determine the types of 1342 metrics that should be sought. For example, if the workload is 1343 control plane, then a metric such as jitter is not useful, but 1344 dropped packets are critical. In a multi-tenant environment, the 1345 NFVI could support various types of workloads. In this case, testing 1346 with a variety of traffic types while measuring the corresponding 1347 metrics simultaneously becomes necessary. 1349 Test beds for any type of testing for an NFV-based system will be 1350 largely similar to previously used test architectures. The methods 1351 are impacted by virtualization, as described above, but the design of 1352 test beds are similar as in the past. There are two main new 1353 considerations: 1355 o Since networking is based on software, which has lead to greater 1356 automation in deployment, the test system should also be 1357 deployable with the rest of the system in order to fully automate 1358 the system. This is especially relevant in a DevOps environment 1359 supported by a CI/CD tool chain (see Section 4.11.3 below). 1361 o In any performance test bed, the test system should not share the 1362 same resources as the System Under Test (SUT). While multi- 1363 tenenacy is a reality in virtualization, having the test system 1364 share resources with the SUT will impact the measured results in a 1365 performance test bed. The test system should be deployed on a 1366 separate platform in order to not to impact the resources 1367 available to the SUT. 1369 4.11.2. New functionality 1371 NFV presents a collection of new functionality in order to support 1372 the goal of software networking. Each component on the architecture 1373 shown in Figure 2 has an associated set of functionality that allows 1374 VNFs to run: onboarding, lifecycle management for VNFs and Networks 1375 Services (NS), resource allocation, hypervisor functions, etc. 1377 One of the new capabilities enabled by NFV is VNFFG (VNF Forwarding 1378 Graphs). This refers to the graph that represents a Network Service 1379 by chaining together VNFs into a forwarding path. In practice, the 1380 forwarding path can be implemented in a variety of ways using 1381 different networking capabilities: vSwitch, SDN, SDN with a 1382 northbound application, and the VNFFG might use tunneling protocols 1383 like VXLAN. The dynamic allocation and implementation of these 1384 networking paths will have different performance characteristics 1385 depending on the methods used. The path implementation mechanism 1386 becomes a variable in the network testing of the NSs. The 1387 methodology used to test the various mechanisms should largely remain 1388 the same, and as usual, the test environment should remain constant 1389 for each of the tests, focusing on varying the path establishment 1390 method. 1392 Scaling refers to the change in allocation of resources to a VNF or 1393 NS. It happens dynamically at run-time, based on defined policies 1394 and triggers. The triggers can be network, compute or storage based. 1395 Scaling can allocate more resources in times of need, or reduce the 1396 amount of resources allocated when the demand is reduced. The SUT in 1397 this case becomes much larger than the VNF itself: MANO controls how 1398 scaling is done based on policies, and then allocates the resources 1399 accordingly in the NFVI. Essentially, the testing of scaling 1400 includes the entire NFV architecture components into the SUT. 1402 4.11.3. Opportunities 1404 Softwarization of networking functionality leads to softwarization of 1405 test as well. As Physical Network Functions (PNF) are being 1406 transformed into VNFs, so have the test tools. This leads to the 1407 fact that test tools are also being controlled and executed in the 1408 same environment as the VNFs are. This presents an opportunity to 1409 include VNF-based test tools along with the deployment of the VNFs 1410 supporting the services of the service provider into the host data 1411 centers. Tests can therefore be automatically executed upon 1412 deployment in the target environment, for each deployment, and each 1413 service. With PNFs, this was very difficult to achieve. 1415 This new concept helps to enable modern concepts like DevOps and 1416 Continuous Integration and Continuous Deployment in the NFV 1417 environment. The CI/CD pipeline supports this concept. It consists 1418 of a series of tools, among which immediate testing is an integral 1419 part, to deliver software from source to deployment. The ability to 1420 deploy the test tools themselves into the production environment 1421 stretches the CI/CD pipeline all the way to production deployment, 1422 allowing a range of tests to be executed. The tests can be simple, 1423 with a goal of verifying the correct deployment and networking 1424 establishment, but can also be more complex, like testing VNF 1425 functionality. 1427 5. Technology Gaps and Potential IETF Efforts 1429 Table 1 correlates the open network virtualization research areas 1430 identified in this document to potential IETF and IRTF groups that 1431 could address some aspects of them. An example of a specific gap 1432 that the group could potentially address is identified in 1433 parenthetical beside the group name. 1435 +-------------------------+-----------------------------------------+ 1436 | Open Research Area | Potential IETF/IRTF Group | 1437 +-------------------------+-----------------------------------------+ 1438 | 1-Guaranteeing QoS | IPPM WG (Measurements of NFVI) | 1439 | 2-Performance | SFC WG, NFVRG (energy driven | 1440 | improvement | orchestration) | 1441 | 3-Multiple Domains | NFVRG (multi-domain orchestration) | 1442 | 4-Network Slicing | NVO3 WG, NETSLICES bar BoF (multi- | 1443 | | tenancy support) | 1444 | 5-Service Composition | SFC WG (SFC Mgmt and Config) | 1445 | 6-End-user device | N/A | 1446 | virtualization | | 1447 | 7-Security | N/A | 1448 | 8-Separation of control | NFVRG (separation between transport | 1449 | concerns | control and services) | 1450 | 9-Testing | NFVRG (testing of scaling) | 1451 | 10-Function placement | NFVRG, SFC WG (VNF placement algorithms | 1452 | | and protocols) | 1453 +-------------------------+-----------------------------------------+ 1455 Table 1: Mapping of Open Research Areas to Potential IETF Groups 1457 6. NFVRG focus areas 1459 Table 2 correlates the currently identified NFVRG topics of 1460 interests/focus areas to the open network virtualization research 1461 areas enumerated in this document. This can help the NFVRG in 1462 identifying and prioritizing research topics. The current list of 1463 NFVRG focus points is the following: 1465 o Re-architecting functions, including aspects such as new 1466 architectural and design patterns (e.g., containerization, 1467 statelessness, serverless, control/data plane separation), SDN 1468 integration, and proposals on programmability. 1470 o New management frameworks, considering aspects related to new OAM 1471 mechanisms (e.g., configuration control, hybrid descriptors) and 1472 lightweight MANO proposals. 1474 o Techniques to guarantee low latency, resource isolation, and other 1475 dataplane features, including hardware acceleration, functional 1476 offloading to dataplane elements (including NICs), and related 1477 approaches. 1479 o Measurement and benchmarking, addressing both internal 1480 measurements and external applications. 1482 +----------------------------------------+-------------------------+ 1483 | NFVRG Focus Point | Open Research Area | 1484 +----------------------------------------+-------------------------+ 1485 | 1-Re-architecting functions | - Performance improvem. | 1486 | | - Network Slicing | 1487 | | - Guaranteeing QoS | 1488 | | - Security | 1489 | | - End-user device virt. | 1490 | | - Separation of control | 1491 | 2-New management frameworks | - Multiple Domains | 1492 | | - Service Composition | 1493 | | - End-user device virt. | 1494 | 3-Low latency, resource isolation, etc | - Performance improvem. | 1495 | | - Separation of control | 1496 | 4-Measurement and benchmarking | - Guaranteeing QoS | 1497 | | - Testing | 1498 +----------------------------------------+-------------------------+ 1500 Table 2: Mapping of NFVRG Focus Points to Open Research Areas 1502 7. IANA Considerations 1504 N/A. 1506 8. Security Considerations 1508 This is an informational document, which therefore does not introduce 1509 any security threat. Research challenges and gaps related to 1510 security and privacy have been included in Section 4.8. 1512 9. Acknowledgments 1514 The authors want to thank Dirk von Hugo, Rafa Marin, Diego Lopez, 1515 Ramki Krishnan, Kostas Pentikousis, Rana Pratap Sircar, Alfred 1516 Morton, Nicolas Kuhn, Saumya Dikshit, Fabio Giust, Evangelos 1517 Haleplidis, Angeles Vazquez-Castro, Barbara Martini, Jose Saldana and 1518 Gino Carrozzo for their very useful reviews and comments to the 1519 document. Special thanks to Pedro Martinez-Julia, who provided text 1520 for the network slicing section. 1522 The authors want to also thank Dave Oran and Michael Welzl for their 1523 very detailed IRSG reviews. 1525 The work of Carlos J. Bernardos and Luis M. Contreras is partially 1526 supported by the H2020 5GEx (Grant Agreement no. 671636) and 5G- 1527 TRANSFORMER (Grant Agreement no. 761536) projects. 1529 10. Informative References 1531 [dynamic_chaining] 1532 Martini, B. and F. Paganelli, "A Service-Oriented Approach 1533 for Dynamic Chaining of Virtual Network Functions over 1534 Multi-Provider Software-Defined Networks", Future 1535 Internet vol. 8, no. 2, June 2016. 1537 [dynamic_placement] 1538 Clayman, S., Maini, E., and A. Galis, "The dynamic 1539 placement of virtual network functions", 2014 IEEE Network 1540 Operations and Management Symposium (NOMS) pp. 1-9, May 1541 2014. 1543 [etsi_gs_nfv_003] 1544 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1545 Terminology for Main Concepts in NFV", ETSI GS NFV 003 1546 V1.2.1 NFV 003, December 2014, 1547 . 1550 [etsi_gs_nfv_eve005] 1551 ETSI NFV ISG, "Network Functions Virtualisation (NFV); 1552 Ecosystem; Report on SDN Usage in NFV Architectural 1553 Framework", ETSI GS NFV-EVE 005 V1.1.1 NFV-EVE 005, 1554 December 2015, . 1557 [etsi_gs_nfv_per_001] 1558 ETSI GS NFV-PER 001 V1.1.2, "Network Functions 1559 Virtualisation (NFV); NFV Performance & Portability Best 1560 Practises", ETSI GS NFV-PER 001 V1.1.2 NFV-PER 001, 1561 December 2014, . 1564 [etsi_gs_nfv_sec_001] 1565 ETSI GS NFV-SEC 001 V1.1.1, "Network Functions 1566 Virtualisation (NFV); NFV Security; Problem Statement", 1567 ETSI GS NFV-SEC 001 V1.1.1 NFV-SEC 001, October 2014, 1568 . 1571 [etsi_nvf_whitepaper_3] 1572 "Network Functions Virtualisation (NFV). White Paper 3", 1573 October 2014. 1575 [google_sdn_wan] 1576 Sushant Jain et al., "B4: experience with a globally- 1577 deployed Software Defined WAN", Proceedings of the ACM 1578 SIGCOMM 2013 , August 2013. 1580 [I-D.bagnulo-nfvrg-topology] 1581 Bagnulo, M. and D. Dolson, "NFVI PoP Network Topology: 1582 Problem Statement", draft-bagnulo-nfvrg-topology-01 (work 1583 in progress), March 2016. 1585 [I-D.bernardos-nfvrg-multidomain] 1586 Bernardos, C., Contreras, L., Vaishnavi, I., and R. Szabo, 1587 "Multi-domain Network Virtualization", draft-bernardos- 1588 nfvrg-multidomain-03 (work in progress), September 2017. 1590 [I-D.defoy-netslices-3gpp-network-slicing] 1591 Foy, X. and A. Rahman, "Network Slicing - 3GPP Use Case", 1592 draft-defoy-netslices-3gpp-network-slicing-02 (work in 1593 progress), October 2017. 1595 [I-D.gdmb-netslices-intro-and-ps] 1596 Galis, A., Dong, J., kiran.makhijani@huawei.com, k., 1597 Bryant, S., Boucadair, M., and P. Martinez-Julia, "Network 1598 Slicing - Introductory Document and Revised Problem 1599 Statement", draft-gdmb-netslices-intro-and-ps-02 (work in 1600 progress), February 2017. 1602 [I-D.geng-coms-problem-statement] 1603 67, 4., Wang, L., Slawomir, S., Qiang, L., Matsushima, S., 1604 Galis, A., and L. Contreras, "Problem Statement of 1605 Supervised Heterogeneous Network Slicing", draft-geng- 1606 coms-problem-statement-01 (work in progress), October 1607 2017. 1609 [I-D.irtf-sdnrg-layered-sdn] 1610 Contreras, L., Bernardos, C., Lopez, D., Boucadair, M., 1611 and P. Iovanna, "Cooperating Layered Architecture for 1612 SDN", draft-irtf-sdnrg-layered-sdn-01 (work in progress), 1613 October 2016. 1615 [I-D.marin-sdnrg-sdn-aaa-mng] 1616 Lopez, R. and G. Lopez-Millan, "Software-Defined 1617 Networking (SDN)-based AAA Infrastructures Management", 1618 draft-marin-sdnrg-sdn-aaa-mng-00 (work in progress), 1619 November 2015. 1621 [I-D.mlk-nfvrg-nfv-reliability-using-cots] 1622 Mo, L. and B. Khasnabish, "NFV Reliability using COTS 1623 Hardware", draft-mlk-nfvrg-nfv-reliability-using-cots-01 1624 (work in progress), October 2015. 1626 [I-D.natarajan-nfvrg-containers-for-nfv] 1627 natarajan.sriram@gmail.com, n., Krishnan, R., Ghanwani, 1628 A., Krishnaswamy, D., Willis, P., Chaudhary, A., and F. 1629 Huici, "An Analysis of Lightweight Virtualization 1630 Technologies for NFV", draft-natarajan-nfvrg-containers- 1631 for-nfv-03 (work in progress), July 2016. 1633 [I-D.rorosz-nfvrg-vbaas] 1634 Rosa, R., Rothenberg, C., and R. Szabo, "VNF Benchmark-as- 1635 a-Service", draft-rorosz-nfvrg-vbaas-00 (work in 1636 progress), October 2015. 1638 [intel_10_differences_nfv_cloud] 1639 Torre, P., "Discover the Top 10 Differences Between NFV 1640 and Cloud Environments", November 2015, 1641 . 1644 [itu-t-y.3300] 1645 ITU-T, "Y.3300: Framework of software-defined networking", 1646 ITU-T Recommendation Y.3300 (06/14), June 2014, 1647 . 1649 [itu-t-y.3301] 1650 ITU-T, "Y.3301: Functional requirements of software- 1651 defined networking", ITU-T Recommendation Y.3301 (09/16), 1652 September 2016, 1653 . 1655 [itu-t-y.3302] 1656 ITU-T, "Y.3302: Functional architecture of software- 1657 defined networking", ITU-T Recommendation Y.3302 (01/17), 1658 January 2017, 1659 . 1661 [multi-domain_5GEx] 1662 Bernardos, C., Geroe, B., Di Girolamo, M., Kern, A., 1663 Martini, B., and I. Vaishnavi, "5GEx: Realizing a Europe 1664 wide Multi-domain Framework for Software Defined 1665 Infrastructures", Transactions on Emerging 1666 Telecommunications Technologies vol. 27, no. 9, pp. 1667 1271-1280, September 2016. 1669 [nfv_piecing] 1670 Luizelli, M., Bays, L., and L. Buriol, "Piecing together 1671 the NFV provisioning puzzle: Efficient placement and 1672 chaining of virtual network functions", 2015 IFIP/IEEE 1673 International Symposium on Integrated Network Management 1674 (IM) pp. 98-106, May 2015. 1676 [nfv_sota_research_challenges] 1677 Mijumbi, R., Serrat, J., Gorricho, J-L., Bouten, N., De 1678 Turck, F., and R. Boutaba, "Network Function 1679 Virtualization: State-of-the-art and Research Challenges", 1680 IEEE Communications Surveys & Tutorials Volume: 18, Issue: 1681 1, September 2015. 1683 [ngmn_5G_whitepaper] 1684 "NGMN 5G. White Paper", February 2015. 1686 [omniran] IEEE 802.1CF, "Recommended Practice for Network Reference 1687 Model and Functional Description of IEEE 802 Access 1688 Network", Draft 1.0 , December 2017. 1690 [onf_tr_521] 1691 ONF, "SDN Architecture, Issue 1.1", ONF TR-521 TR-521, 1692 February 2016, 1693 . 1697 [openmano_dataplane] 1698 Lopez, D., "OpenMANO: The Dataplane Ready Open Source NFV 1699 MANO Stack", March 2015, 1700 . 1703 [RFC5810] Doria, A., Ed., Hadi Salim, J., Ed., Haas, R., Ed., 1704 Khosravi, H., Ed., Wang, W., Ed., Dong, L., Gopal, R., and 1705 J. Halpern, "Forwarding and Control Element Separation 1706 (ForCES) Protocol Specification", RFC 5810, 1707 DOI 10.17487/RFC5810, March 2010, 1708 . 1710 [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., 1711 and A. Bierman, Ed., "Network Configuration Protocol 1712 (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, 1713 . 1715 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 1716 Application Protocol (CoAP)", RFC 7252, 1717 DOI 10.17487/RFC7252, June 2014, 1718 . 1720 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 1721 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 1722 Defined Networking (SDN): Layers and Architecture 1723 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 1724 2015, . 1726 [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for 1727 Service Function Chaining", RFC 7498, 1728 DOI 10.17487/RFC7498, April 2015, 1729 . 1731 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1732 Chaining (SFC) Architecture", RFC 7665, 1733 DOI 10.17487/RFC7665, October 2015, 1734 . 1736 [RFC8030] Thomson, M., Damaggio, E., and B. Raymor, Ed., "Generic 1737 Event Delivery Using HTTP Push", RFC 8030, 1738 DOI 10.17487/RFC8030, December 2016, 1739 . 1741 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 1742 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 1743 . 1745 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 1746 Network Functions and Their Infrastructure", RFC 8172, 1747 DOI 10.17487/RFC8172, July 2017, 1748 . 1750 [sfc_challenges] 1751 Medhat, A., Taleb, T., Elmangoush, A., Carella, G., 1752 Covaci, S., and T. Magedanz, "Service Function Chaining in 1753 Next Generation Networks: State of the Art and Research 1754 Challenges", IEEE Communications Magazine vol. 55, no. 2, 1755 pp. 216-223, February 2017. 1757 [virtualization_mobile_device] 1758 William D. Sproule, "Virtualization of Mobile Device User 1759 Experience", Patent US 9.542.062 B2 , January 2017. 1761 [vnf-p] Moens, H. and Filip De Turck, "VNF-P: A model for 1762 efficient placement of virtualized network functions", 1763 10th International Conference on Network and Service 1764 Management (CNSM) and Workshop pp. 418-423, 2014. 1766 [vnf_benchmarking] 1767 Rosa, R., Rothenberg, C., and R. Szabo, "A VNF Testing 1768 Framework Design, Implementation and Partial Results", 1769 November 2016, 1770 . 1773 Authors' Addresses 1775 Carlos J. Bernardos 1776 Universidad Carlos III de Madrid 1777 Av. Universidad, 30 1778 Leganes, Madrid 28911 1779 Spain 1781 Phone: +34 91624 6236 1782 Email: cjbc@it.uc3m.es 1783 URI: http://www.it.uc3m.es/cjbc/ 1785 Akbar Rahman 1786 InterDigital Communications, LLC 1787 1000 Sherbrooke Street West, 10th floor 1788 Montreal, Quebec H3A 3G4 1789 Canada 1791 Email: Akbar.Rahman@InterDigital.com 1792 URI: http://www.InterDigital.com/ 1794 Juan Carlos Zuniga 1795 SIGFOX 1796 425 rue Jean Rostand 1797 Labege 31670 1798 France 1800 Email: j.c.zuniga@ieee.org 1801 URI: http://www.sigfox.com/ 1802 Luis M. Contreras 1803 Telefonica I+D 1804 Ronda de la Comunicacion, S/N 1805 Madrid 28050 1806 Spain 1808 Email: luismiguel.contrerasmurillo@telefonica.com 1810 Pedro Aranda 1811 Universidad Carlos III de Madrid 1812 Av. Universidad, 30 1813 Leganes, Madrid 28911 1814 Spain 1816 Email: pedroandres.aranda@uc3m.es 1818 Pierre Lynch 1819 Ixia 1821 Email: plynch@ixiacom.com