idnits 2.17.1 draft-rosa-bmwg-vnfbench-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 813: '... and MUST NOT be connected to device...' RFC 2119 keyword, line 817: '...ial capabilities SHOULD NOT exist in t...' RFC 2119 keyword, line 820: '...loyment scenario SHOULD be identical i...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 2, 2018) is 2115 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BMWG R. Rosa, Ed. 3 Internet-Draft C. Rothenberg 4 Intended status: Informational UNICAMP 5 Expires: January 3, 2019 M. Peuster 6 H. Karl 7 UPB 8 July 2, 2018 10 Methodology for VNF Benchmarking Automation 11 draft-rosa-bmwg-vnfbench-02 13 Abstract 15 This document describes a common methodology for automated 16 benchmarking of Virtualized Network Functions (VNFs) executed on 17 general-purpose hardware. Specific cases of benchmarking 18 methodologies for particular VNFs can be derived from this document. 19 Two open source reference implementations are reported as running 20 code embodiments of the proposed, automated benchmarking methodology. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on January 3, 2019. 39 Copyright Notice 41 Copyright (c) 2018 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (https://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 57 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 58 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 59 4. Considerations . . . . . . . . . . . . . . . . . . . . . . . 4 60 4.1. VNF Testing Methods . . . . . . . . . . . . . . . . . . . . 4 61 4.2. Generic VNF Benchmarking Setup . . . . . . . . . . . . . . 5 62 4.3. Deployment Scenarios . . . . . . . . . . . . . . . . . . . 7 63 4.4. Influencing Aspects . . . . . . . . . . . . . . . . . . . . 8 64 5. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 9 65 5.1. VNF Benchmarking Descriptor (VNF-BD) . . . . . . . . . . . 10 66 5.1.1. Procedures Configuration . . . . . . . . . . . . . . . . 10 67 5.1.2. Target Information . . . . . . . . . . . . . . . . . . . 10 68 5.1.3. Deployment Scenario . . . . . . . . . . . . . . . . . . . 10 69 5.2. VNF Performance Profile (VNF-PP) . . . . . . . . . . . . . 11 70 5.2.1. Execution Environment . . . . . . . . . . . . . . . . . . 12 71 5.2.2. Measurement Results . . . . . . . . . . . . . . . . . . . 12 72 5.3. Automated Benchmarking Procedures . . . . . . . . . . . . . 13 73 5.4. Particular Cases . . . . . . . . . . . . . . . . . . . . . 15 74 6. Open Source Reference Implementations . . . . . . . . . . . . 16 75 6.1. Gym . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 76 6.2. tng-bench . . . . . . . . . . . . . . . . . . . . . . . . . 17 77 7. Security Considerations . . . . . . . . . . . . . . . . . . . 18 78 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 79 9. Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 18 80 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 81 10.1. Normative References . . . . . . . . . . . . . . . . . . . 18 82 10.2. Informative References . . . . . . . . . . . . . . . . . . 19 83 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 85 1. Introduction 87 Benchmarking Methodology Working Group (BMWG) initiated efforts, 88 approaching considerations in [RFC8172], to develop methodologies for 89 benchmarking VNFs. Similarly described in [RFC8172], VNF benchmark 90 motivating aspects define: (i) pre-deployment infrastructure 91 dimensioning to realize associated VNF performance profiles; (ii) 92 comparison factor with physical network functions; (iii) and output 93 results for analytical VNF development. 95 Having no strict and clear execution boundaries, different from 96 earlier self-contained black-box benchmarking methodologies described 97 in BMWG, a VNF depends on underlying virtualized environment 98 parameters [ETS14a], intrinsic factors to be analyzed when one 99 investivages the performance of a VNF. This document stands as a 100 ground methodology guide for VNF benchmarking automation. It 101 addresses the state-of-the-art publications and the current 102 developments in similar standardization efforts (e.g., [ETS14c] and 103 [RFC8204]) towards bechmarking VNFs. 105 Automating the extraction of VNF performance metrics propitiates: (i) 106 the development of agile performance-focused DevOps methodologies for 107 Continuous Integration and Delivery (CI/CD) of VNFs; (ii) the 108 creation of on-demand VNF test descriptors for upcoming execution 109 environments; (iii) the path for precise-analytics of extensively 110 automated catalogues of VNF performance profiles; (iv) and run-time 111 profiling mechanisms to assist VNF lifecycle orchestration/management 112 workflows. 114 2. Terminology 116 Common benchmarking terminology contained in this document is derived 117 from [RFC1242]. Also, the reader is assumed to be familiar with the 118 terminology as defined in the European Telecommunications Standards 119 Institute (ETSI) NFV document [ETS14b]. Some of these terms, and 120 others commonly used in this document, are defined below. 122 NFV: Network Function Virtualization - The principle of separating 123 network functions from the hardware they run on by using virtual 124 hardware abstraction. 126 NFVI PoP: NFV Infrastructure Point of Presence - Any combination of 127 virtualized compute, storage and network resources. 129 NFVI: NFV Infrastructure - Collection of NFVI PoPs under one 130 orchestrator. 132 VIM: Virtualized Infrastructure Manager - functional block that is 133 responsible for controlling and managing the NFVI compute, storage 134 and network resources, usually within one operator's 135 Infrastructure Domain (e.g. NFVI-PoP). 137 VNFM: Virtualized Network Function Manager - functional block that 138 is responsible for controlling and managing the VNF life-cycle. 140 NFVO: NFV Orchestrator - functional block that manages the Network 141 Service (NS) life-cycle and coordinates the management of NS life- 142 cycle, VNF life-cycle (supported by the VNFM) and NFVI resources 143 (supported by the VIM) to ensure an optimized allocation of the 144 necessary resources and connectivity. 146 VNF: Virtualized Network Function - a software-based network 147 function. A VNF can be either represented by a single entity or 148 be composed by a set of smaller, interconnected software 149 components, called VNF components (VNFCs) [ETS14d]. Those VNFs 150 are also called composed VNFs. 152 VNFD: Virtualised Network Function Descriptor - configuration 153 template that describes a VNF in terms of its deployment and 154 operational behaviour, and is used in the process of VNF on- 155 boarding and managing the life cycle of a VNF instance. 157 VNFC: Virtualized Network Function Component - a software component 158 that implements (parts of) the VNF functionality. A VNF can 159 consist of a single VNFC or multiple, interconnected VNFCs 160 [ETS14d] 162 VNF-FG: Virtualized Network Function Forwarding Graph - an ordered 163 list of VNFs or VNFCs creating a service chain. 165 3. Scope 167 This document assumes VNFs as black boxes when defining their 168 benchmarking methodologies. White box approaches are assumed and 169 analysed as a particular case under the proper considerations of 170 internal VNF instrumentation, later discussed in this document. 172 In what follows, this document outlines a basis methodology for VNF 173 benchmarking, specifically addressing its automation. 175 4. Considerations 177 VNF benchmarking considerations are defined in [RFC8172]. 178 Additionally, VNF pre-deployment testing considerations are well 179 explored in [ETS14c]. 181 4.1. VNF Testing Methods 183 Following the ETSI's model in [ETS14c], we distinguish three methods 184 for VNF evaluation: 186 Benchmarking: Where parameters (e.g., cpu, memory, storage) are 187 provided and the corresponding performance metrics (e.g., latency, 188 throughput) are obtained. Note, such request might create 189 multiple reports, for example, with minimal latency or maximum 190 throughput results. 192 Verification: Both parameters and performance metrics are provided 193 and a stimulus verify if the given association is correct or not. 195 Dimensioning: Where performance metrics are provided and the 196 corresponding parameters obtained. Note, multiple deployment 197 interactions may be required, or if possible, underlying allocated 198 resources need to be dynamically altered. 200 Note: Verification and Dimensioning can be reduced to Benchmarking. 201 Therefore, we detail Benchmarking in what follows. 203 4.2. Generic VNF Benchmarking Setup 205 A generic VNF benchmarking setup is shown in Figure 1, and its 206 components are explained below. Note here, not all components are 207 mandatory, and VNF benchmarking scenarios, further explained, can 208 dispose its components in varied settings. 210 +---------------+ 211 | Manager | 212 Control | (Coordinator) | 213 Interface +---+-------+---+ 214 +--------+-----------+ +-------------------+ 215 | | | 216 | | +-------------------------+ | 217 | | | System Under Test | | 218 | | | | | 219 | | | +-----------------+ | | 220 | +--+------- + VNF | | | 221 | | | | | | 222 | | | +----+ +----+ | | | 223 | | | |VNFC|...|VNFC| | | | 224 | | | +----+ +----+ | | | 225 | | +----.---------.--+ | | 226 +-----+---+ | Monitor | : : | +-----+----+ 227 | Agent | |{listeners}|----^---------V--+ | | Agent | 228 |(Sender) | | | Execution | | |(Receiver)| 229 | | | | Environment | | | | 230 |{Probers}| +-----------| | | |{Probers} | 231 +-----.---+ | +----.---------.--+ | +-----.----+ 232 : +---------^---------V-----+ : 233 V : : : 234 :................>.....: :............>..: 235 Stimulus Traffic Flow 237 Figure 1: Generic VNF Benchmarking Setup 239 Agent -- executes active stimulus using probers, i.e., benchmarking 240 tools, to benchmark and collect network and system performance 241 metrics. While a single Agent is capable of performing localized 242 benchmarks in execution environments (e.g., stress tests on CPU, 243 memory, disk I/O), the interaction among distributed Agents enable 244 the generation and collection of VNF end-to-end metrics (e.g., 245 frame loss rate, latency). In a benchmarking setup, one Agent can 246 create the stimuli and the other end be the VNF itself where, for 247 example, one-way latency is evaluated. An Agent can be defined by 248 a physical or virtual network function. 250 Prober -- defines a software/hardware-based tool able to generate 251 stimulut traffic specific to a VNF (e.g., sipp) or generic to 252 multiple VNFs (e.g., pktgen). A Prober must provide 253 programmable interfaces for its life cycle management 254 workflows, e.g., configuration of operational parameters, 255 execution of stilumi, parsing of extracted metrics, and 256 debugging options. Specific Probers might be developed to 257 abstract and to realize the description of particular VNF 258 benchmarking methodologies. 260 Monitor -- when possible, it is instantiated inside the target VNF 261 or NFVI PoP (e.g., as a plug-in process in a virtualized 262 environment) to perform passive monitoring, using listeners, for 263 metrics collection based on benchmark tests evaluated according to 264 Agents` stimuli. Different from the active approach of Agents 265 that can be seen as generic benchmarking VNFs, Monitors observe 266 particular properties according to NFVI PoPs and VNFs 267 capabilities. A Monitor can be defined as a virtual network 268 function. 270 Listener -- defines one or more software interfaces for the 271 extraction of particular metrics monitored in a target VNF and/ 272 or execution environment. A Listener must provide programmable 273 interfaces for its life cycle management workflows, e.g., 274 configuration of operational parameters, execution of 275 monitoring captures, parsing of extracted metrics, and 276 debugging options. White-box benchmarking approaches must be 277 carefully analyzed, as varied methods of performance monitoring 278 might be coded as a Listener, possibly impacting the VNF and/or 279 execution environment performance results. 281 Manager -- in a VNF benchmarking setup, a Manager is responsible for 282 (i) the coordination and synchronization of activities of Agents 283 and Monitors, (ii) collecting and parsing all VNF benchmarking 284 results, and (iii) aggregating the inputs and parsed benchmark 285 outputs to construct a VNF performance profile, which defines a 286 report that correlates the VNF stimuli and the monitored metrics. 287 A Manager executes the main configuration, operation, and 288 management actions to deliver the VNF benchmarking results. A 289 Manager can be defined as a physical or virtual network function, 290 and be split into multiple sub-components, each responsible for 291 separated functional aspects of the overall Manager component. 293 Virtualized Network Function (VNF) -- consists of one or more 294 software components, so called VNF components (VNFC), adequate for 295 performing a network function according to allocated virtual 296 resources and satisfied requirements in an execution environment. 297 A VNF can demand particular configurations for benchmarking 298 specifications, demonstrating variable performance profiles based 299 on available virtual resources/parameters and configured 300 enhancements targeting specific technologies (e.g., NUMA, SR-IOV, 301 CPU-Pinning). 303 Execution Environment -- defines a virtualized and controlled 304 composition of capabilities necessary for the execution of a VNF. 305 An execution environment stands as a general purpose level of 306 virtualization with abstracted resources available for one or more 307 VNFs. It can also define specific technology habilitation, 308 incurring in viable settings for enhancing the performance of 309 VNFs. 311 4.3. Deployment Scenarios 313 A VNF benchmark deployment scenario establishes the physical and/or 314 virtual instantiation of components defined in a VNF benchmarking 315 setup. 317 The following considerations hold for deployment scenarios: 319 o Components can be composed in a single entity and be defined as 320 black or white boxes. For instance, Manager and Agents could 321 jointly define one hardware/software entity to perform a VNF 322 benchmark and present results. 324 o Monitor is not a mandatory component and must be considered only 325 when performed white box benchmarking approaches for a VNF and/or 326 its execution environment. 328 o Monitor can be defined by multiple instances of software 329 components, each addressing a VNF or execution environment and 330 their respective open interfaces for the extraction of metrics. 332 o Agents can be disposed in varied topology setups, included the 333 possibility of multiple input and output ports of a VNF being 334 directly connected each in one Agent. 336 o All benchmarking components defined in a deployment scenario must 337 perform the synchronization of clocks. 339 4.4. Influencing Aspects 341 In general, VNF benchmarks must capture relevant causes of 342 performance variability. Concerning a deployment scenario, 343 influencing aspects on the performance of a VNF can be observed in: 345 Deployment Scenario Topology: The disposition of components can 346 define particular interconnections among them composing a specific 347 case/method of VNF benchmarking. 349 Execution Environment: The availability of generic and specific 350 capabilities satisfying VNF requirements define a skeleton of 351 opportunities for the allocation of VNF resources. In addition, 352 particular cases can define multiple VNFs interacting in the same 353 execution environment of a benchmarking setup. 355 VNF: A detailed description of functionalities performed by a VNF 356 sets possible traffic forwarding and processing operations it can 357 perform on packets, added to its running requirements and specific 358 configurations, which might affect and compose a benchmarking 359 setup. 361 Agent: The toolset available for the benchmarking stimulus of a VNF 362 and its characteristics of packets format and workload can 363 interfere in a benchmarking setup. VNFs can support specific 364 traffic format as stimulus. 366 Monitor: In a particular benchmarking setup where measurements of 367 VNF and/or execution environment metrics are available for 368 extraction, an important analysis consist in verifying if the 369 Monitor components can impact performance metrics of the VNF and 370 the underlying execution environment. 372 Manager: The overall composition of VNF benchmarking procedures can 373 determine arrangements of internal states inside a VNF, which can 374 interfere in observed benchmarking metrics. 376 The listed influencing aspects must be carefully analyzed while 377 automating a VNF benchmarking methodology. 379 5. Methodology 381 Portability is an intrinsic characteristic of VNFs and allows them to 382 be deployed in multiple environments. This enables various 383 benchmarking procedures in varied deployment scenarios. A VNF 384 benchmarking methodology must be described in a clear and objective 385 manner in order to allow effective repeatability and comparability of 386 the test results. Those results, the outcome of a VNF benchmarking 387 process, are captured in a VNF Benchmarking Report (VNF-BR) as shown 388 in Figure 2. 390 X 391 / \ 392 / \ 393 / \ 394 +--------+ / \ 395 | | / \ 396 | VNF-BD |--(defines)-->| Benchmark | 397 | | \ Process / 398 +--------+ \ / 399 \ / 400 \ / 401 \ / 402 V 403 | 404 (generates) 405 | 406 v 407 +-------------------------+ 408 | VNF-BR | 409 | +--------+ +--------+ | 410 | | | | | | 411 | | VNF-BD | | VNF-PP | | 412 | | {copy} | | | | 413 | +--------+ +--------+ | 414 +-------------------------+ 416 Figure 2: VNF benchmarking process inputs and outputs 418 VNF Benchmarking reports comprise two parts: 420 VNF Benchmarking Descriptor (VNF-BD) -- contains all required 421 definitions and requirements to configure, execute and reproduce 422 VNF benchmarking experiments. VNF-BDs are defined by the 423 developer of a benchmarking experiment and serve as input to the 424 benchmarking process, before being included in the generated VNF- 425 BR. 427 VNF Performance Profile (VNF-PP) -- contains all measured metrics 428 resulting from the execution of a benchmark. Additionally, it 429 might also contain additional recordings of configuration 430 parameters used during the execution of the benchmarking scenario 431 to facilitate comparability of VNF-BRs. 433 A VNF-BR correlates structural and functional parameters of VNF-BD 434 with extracted VNF benchmarking metrics of the obtained VNF-PP. The 435 content of each part of a VNF-BR is described in the following 436 sections. 438 5.1. VNF Benchmarking Descriptor (VNF-BD) 440 VNF Benchmarking Descriptor (VNF-BD) -- an artifact that specifies a 441 method of how to measure a VNF Performance Profile. The 442 specification includes structural and functional instructions and 443 variable parameters at different abstraction levels (e.g., topology 444 of the deployment scenario, benchmarking target metrics, parameters 445 of benchmarking components). VNF-BD may be specific to a VNF or 446 applicable to several VNF types. A VNF-BD can be used to elaborate a 447 VNF benchmark deployment scenario aiming at the extraction of 448 particular VNF performance metrics. 450 The following items define the VNF-BD contents. 452 5.1.1. Procedures Configuration 454 The definition of parameters concerning the execution of the 455 benchmarking procedures (see Section 5.3), for instance, containing 456 the number of repetitions and duration of each test. 458 5.1.2. Target Information 460 General information addressing the target VNF, with references to any 461 of its specific characteristics (e.g., type, model, version/release, 462 architectural components, etc). In addition, it defines the metrics 463 to be extracted when running the benchmarking tests. 465 5.1.3. Deployment Scenario 467 This section of a VNF-BD contains all information needed to describe 468 the deployment of all involved components used during the 469 benchmarking test. 471 5.1.3.1. Topology 473 Information about the experiment topology, concerning the disposition 474 of the components in a benchmarking setup (see Section 4.2). It must 475 define the role of each component and how they are interconnected 476 (i.e., interface, link and network characteristics). 478 5.1.3.2. Requirements 480 Involves the definition of execution environment requirements to 481 execute the tests. Therefore, they concern all required capabilities 482 needed for the execution of the target VNF and the other components 483 composing the benchmarking setup. Examples of specifications 484 involve: min/max allocation of resources, specific enabling 485 technologies (e.g., DPDK, SR-IOV, PCIE). 487 5.1.3.3. Parameters 489 Involves any specific configuration of benchmarking components in a 490 setup described the the deployment scenario topology. 492 VNF Configurations: Defines any specific configuration that must be 493 loaded into the VNF to execute the benchmarking experiments (e.g., 494 routing table, firewall rules, vIMS subscribers profile). 496 VNF Resources: Contains particular VNF resource configurations that 497 should be tested during the benchmarking process, e.g., test the 498 VNF for configurations with 2, 4, and 8 vCPUs associated. 500 Agents: Defines the configured toolset of available probers and 501 related benchmarking/active metrics, available workloads, traffic 502 formats/traces, and configurations to enable hardware capabilities 503 (if existent). 505 Monitors: defines the configured toolset of available listeners and 506 related monitoring/passive metrics, configuration of the 507 interfaces with the monitoring target (VNF and/or execution 508 environment), and configurations to enable specific hardware 509 capabilities (if existent). 511 5.2. VNF Performance Profile (VNF-PP) 513 VNF Performance Profile (VNF-PP) -- defines a mapping between 514 resources allocated to a VNF (e.g., CPU, memory) as well as assigned 515 configurations (e.g., routing table used by the VNF) and the VNF 516 performance metrics (e.g., throughput, latency between in/out ports) 517 obtained in a benchmarking test conducted using a VNF-BD. Logically, 518 packet processing metrics are presented in a specific format 519 addressing statistical significance (e.g., median, standard 520 deviation, percentiles) where a correspondence among VNF parameters 521 and the delivery of a measured VNF performance exists. 523 The following items define the VNF-PP contents. 525 5.2.1. Execution Environment 527 Execution environment information is has to be included in every VNF- 528 PP and is required to describe the environment on which a benchmark 529 was actually executed. 531 Ideally, any person who has a VNF-BD and its complementing VNF-PP 532 with its execution environment information available, must be able to 533 reproduce the same deployment scenario and VNF benchmarking tests to 534 obtain identical VNF-PP measurement results. 536 If not already defined by the VNF-BD deployment scenario requirements 537 (Section 5.1.3), for each component in the VNF benchmarking setup, 538 the following topics must be detailed: 540 Hardware Specs: Contains any information associated with the 541 underlying hardware capabilities offered and used by the component 542 during the benchmarking tests. Examples of such specification 543 include allocated CPU architecture, connected NIC specs, allocated 544 memory DIMM, etc. In addition, any information concerning details 545 of resource isolation must also be described in this part of the 546 VNF-PP. 548 Software Specs: Contains any information associated with the 549 software apparatus offered and used during the benchmarking tests. 550 Examples include versions of operating systems, kernels, 551 hypervisors, container image versions, etc. 553 Optionally, a VNF-PP execution environment might contain references 554 to an orchestration description document (e.g., HEAT template) to 555 clarify technological aspects of the execution environment and any 556 specific parameters that it might contain for the VNF-PP. 558 5.2.2. Measurement Results 560 Measurement results concern the extracted metrics, output of 561 benchmarking procedures, classified into: 563 VNF Processing/Active Metrics: Concerns metrics explicitly defined 564 by or extracted from direct interactions of Agents with a VNF. 565 Those can be defined as generic metric related to network packet 566 processing (e.g., throughput, latency) or metrics specific to a 567 particular VNF (e.g., vIMS confirmed transactions, DNS replies). 569 VNF Monitored/Passive Metrics: Concerns the Monitors' metrics 570 captured from a VNF execution, classified according to the 571 virtualization level (e.g., baremetal, VM, container) and 572 technology domain (e.g., related to CPU, memory, disk) from where 573 they were obtained. 575 Depending on the configuration of the benchmarking setup and the 576 planned use cases for the resulting VNF-PPs, measurement results can 577 be stored as raw data, e.g., time series data about CPU utilization 578 of the VNF during a throughput benchmark. In the case of VNFs 579 composed of multiple VNFCs, those resulting data should be 580 represented as vectors, capturing the behavior of each VNFC, if 581 available from the used monitoring systems. Alternatively, more 582 compact representation formats can be used, e.g., statistical 583 information about a series of latency measurements, including 584 averages and standard deviations. The exact output format to be used 585 is defined in the complementing VNF-BD (Section 5.1). 587 A VNF performance profile must address the combined set of classified 588 items in the 3x3 Matrix Coverage defined in [RFC8172]. 590 5.3. Automated Benchmarking Procedures 592 VNF benchmarking offers the possibility of defining distinct aspects/ 593 steps that may or may not be automated: 595 Orchestration: placement (assignment/allocation of resources) and 596 interconnection (physical/virtual) of network function(s) and 597 benchmark components (e.g., OpenStack/Kubernetes templates, NFV 598 description solutions, like OSM, SONATA, ONAP) -> Defines 599 deployment scenario. 601 Management/Configuration: benchmark components and VNF are 602 configured to execute the experiment/test (e.g., populate routing 603 table, load pcap source files in agent) -> Realizes VNF-BD 604 settings. 606 Execution: Tests/experiments are executed according to 607 configuration and orchestrated components -> Runs the VNF 608 benchmarking cases. 610 Output: There might be generic VNF footprint metrics (e.g., CPU, 611 memory) and specific VNF traffic processing metrics (e.g., 612 transactions or throughput). Output processing must be taken into 613 account (e.g., if sampling is applied or not) in a generic 614 (statistics) or specific (clustering) ways -> Generates VNF-PP. 616 For the purposes of dissecting the execution procedures, consider the 617 following definitions: 619 Trial: is a single process or iteration to obtain VNF benchmarking 620 metrics as a singular measurement. 622 Test: Defines strict parameters for benchmarking components to 623 perform one or more trials. Multiple Trials must be performed for 624 statistical significance of the obtained benchmarking results of a 625 Test. Each Trial must be executed following a particular 626 deployment scenario composed by a VNF-BD. Proper measures must be 627 taken to ensure statistic validity (e.g., independence across 628 trials of generated load patterns). 630 Method: Consists of a VNF-BD, including one or more Tests to 631 benchmark a VNF. A Method can explicitly list ranges of parameter 632 values for the configuration of benchmarking components. Each 633 value of such a range is to be realized in a Test. I.e., Methods 634 can define parameter studies. 636 The following sequence of events composes basic, general procedures 637 to execute a Test (as defined above). 639 1. A VNF-BD must be defined to be later instantiated into and 640 executed as a deployment scenario. Such a description must 641 contain all the structural and functional settings defined in 642 Section 5.1. At the end of this step, the complete method of 643 benchmarking the target VNF is defined. 645 2. Via an automated orchestrator or in a manual process, all the 646 components of the VNF benchmark setup must be allocated and 647 interconnected. VNF and the execution environment must be 648 configured to properly address the VNF benchmark stimuli. 650 3. Manager, Agent(s) and Monitor(s) (if existing), must be started 651 and configured to execute the benchmark stimuli and retrieve 652 expected metrics captured during or at the end of each Trial. One 653 or more trials realize the required measurements to characterize 654 the performance behavior of a VNF according to the benchmark setup 655 defined in the VNF-BD. 657 4. Output results from each obtained benchmarking test must be 658 collected by the Manager. In an automated or manual process, 659 intended metrics, as described in the VNF-BD, are extracted and 660 combined to the final VNF-PP. The combination of used VNF-BD and 661 generated VNF-PP make up the resulting VNF benchmark report (VNF- 662 BR). 664 5.4. Particular Cases 666 Configurations and procedures concerning particular cases of VNF 667 benchmarks address testing methodologies proposed in [RFC8172]. In 668 addition to the general description previously defined, some details 669 must be taken into consideration in the following VNF benchmarking 670 cases. 672 Noisy Neighbor: An Agent can assume the role of a noisy neighbor, 673 generating a particular workload in synchrony with a benchmarking 674 procedure over a VNF. Adjustments of the noisy workload stimulus 675 type, frequency, virtualization level, among others, must be 676 detailed in the VNF-BD. 678 Representative Capacity: An average value of workload must be 679 specified as an Agent stimulus. Considering a long-term analysis, 680 the VNF must be configured to properly address a desired average 681 behavior of performance in comparison with the value of the 682 workload stimulus. 684 Flexibility and Elasticity: Having the possibility of a VNF be 685 composed by multiple components (VNFCs), internal events of the 686 VNF might trigger changes in VNF behavior, e.g., activating 687 functionalities associated with elasticity such as load balancing. 688 In this sense, a detailed characterization of a VNF must be 689 specified (e.g. the VNFs scaling state) and be contained in the 690 VNF-PP and thus the VNF benchmarking report. 692 On Failures: Similar to the case before, VNF benchmarking setups 693 must also capture the dynamics involved in VNF behavior. In case 694 of failures, a VNF might restart itself and possibly result in an 695 offline period (e.g., self healing). A VNF-PP and benchmarking 696 report must capture such variation of VNF states. 698 White Box VNF: A benchmarking setup must define deployment 699 scenarios to be compared with and without monitor components into 700 the VNF and/or the execution environment, in order to analyze if 701 the VNF performance is affected. The VNF-PP and benchmarking 702 report must contain such analysis of performance variability, 703 together with all the extracted VNF performance metrics. 705 6. Open Source Reference Implementations 707 There are two open source reference implementations that are build to 708 automate benchmarking of Virtualized Network Functions (VNFs). 710 6.1. Gym 712 The software, named Gym, is a framework for automated benchmarking of 713 Virtualized Network Functions (VNFs). It was coded following the 714 initial ideas presented in a 2015 scientific paper entitled "VBaaS: 715 VNF Benchmark-as-a-Service" [Rosa-a]. Later, the evolved design and 716 prototyping ideas were presented at IETF/IRTF meetings seeking impact 717 into NFVRG and BMWG. 719 Gym was built to receive high-level test descriptors and execute them 720 to extract VNFs profiles, containing measurements of performance 721 metrics - especially to associate resources allocation (e.g., vCPU) 722 with packet processing metrics (e.g., throughput) of VNFs. From the 723 original research ideas [Rosa-a], such output profiles might be used 724 by orchestrator functions to perform VNF lifecycle tasks (e.g., 725 deployment, maintenance, tear-down). 727 The proposed guiding principles, elaborated in [Rosa-b], to design 728 and build Gym can be composed in multiple practical ways for 729 different VNF testing purposes: 731 o Comparability: Output of tests shall be simple to understand and 732 process, in a human-read able format, coherent, and easily 733 reusable (e.g., inputs for analytic applications). 735 o Repeatability: Test setup shall be comprehensively defined through 736 a flexible design model that can be interpreted and executed by 737 the testing platform repeatedly but supporting customization. 739 o Configurability: Open interfaces and extensible messaging models 740 shall be available between components for flexible composition of 741 test descriptors and platform configurations. 743 o Interoperability: Tests shall be ported to different environments 744 using lightweight components. 746 In [Rosa-b] Gym was utilized to benchmark a decomposed IP Multimedia 747 Subsystem VNF. And in [Rosa-c], a virtual switch (Open vSwitch - 748 OVS) was the target VNF of Gym for the analysis of VNF benchmarking 749 automation. Such articles validated Gym as a prominent open source 750 reference implementation for VNF benchmarking tests. Such articles 751 set important contributions as discussion of the lessons learned and 752 the overall NFV performance testing landscape, included automation. 754 Gym stands as one open source reference implementation that realizes 755 the VNF benchmarking methodologies presented in this document. Gym 756 is being released open source at [Gym]. The code repository includes 757 also VNF Benchmarking Descriptor (VNF-BD) examples on the vIMS and 758 OVS targets as described in [Rosa-b] and [Rosa-c]. 760 6.2. tng-bench 762 Another software that focuses on implementing a framework to 763 benchmark VNFs is the "5GTANGO VNF/NS Benchmarking Framework" also 764 called "tng-bench" (previously "son-profile") and was is as part of 765 the two European Union H2020 projects SONATA NFV and 5GTANGO [tango]. 766 Its initial ideas were presented in [Peu-a] and the system design of 767 the end-to-end prototype was presented in [Peu-b]. 769 Tng-bench's aims to act as a framework for the end-to-end automation 770 of VNF benchmarking processes. Its goal is to automate the 771 benchmarking process in such a way that VNF-PPs can be generated 772 without further human interaction. This enables the integration of 773 VNF benchmarking into continuous integration and continuous delivery 774 (CI/CD) pipelines so that new VNF-PPs are generated on-the-fly for 775 every new software version of a VNF. Those automatically generated 776 VNF-PPs can then be bundled with the VNFs and serve as inputs for 777 orchestration systems, fitting to the original research ideas 778 presented in [Rosa-a] and [Peu-a]. 780 Following the same high-level VNF testing purposes as Gym, namely: 781 Comparability, repeatability, configurability, and interoperability, 782 tng-bench specifically aims to explore description approaches for VNF 783 benchmarking experiments. In [Peu-b] a prototype specification VNF- 784 BDs is presented which not only allows to specify generic, abstract 785 VNF benchmarking experiments, it also allows to describe sets of 786 parameter configurations to be tested during the benchmarking 787 process, allowing the system to automatically execute complex 788 parameter studies on the SUT, e.g., testing a VNF's performance under 789 different CPU, memory, or software configurations. 791 Tng-bench was used to perform a set of initial benchmarking 792 experiments using different VNFs, like a Squid proxy, an Nginx load 793 balancer, and a Socat TCP relay in [Peu-b]. Those VNFs have not only 794 been benchmarked in isolation, but also in combined setups in which 795 up to three VNFs were chained one after each other. These 796 experiments were used to test tng-bench for scenarios in which 797 composed VNFs, consisting of multiple VNF components (VNFCs), have to 798 be benchmarked. The presented results highlight the need to 799 benchmark composed VNFs in end-to-end scenarios rather than only 800 benchmark each individual component in isolation, to produce 801 meaningful VNF-PPs for the complete VNF. 803 Tng-bench is actively developed and released as open source tool 804 under Apache 2.0 license [tng-bench]. 806 7. Security Considerations 808 Benchmarking tests described in this document are limited to the 809 performance characterization of VNFs in a lab environment with 810 isolated network. 812 The benchmarking network topology will be an independent test setup 813 and MUST NOT be connected to devices that may forward the test 814 traffic into a production network, or misroute traffic to the test 815 management network. 817 Special capabilities SHOULD NOT exist in the VNF benchmarking 818 deployment scenario specifically for benchmarking purposes. Any 819 implications for network security arising from the VNF benchmarking 820 deployment scenario SHOULD be identical in the lab and in production 821 networks. 823 8. IANA Considerations 825 This document does not require any IANA actions. 827 9. Acknowledgement 829 The authors would like to thank the support of Ericsson Research, 830 Brazil. Parts of this work have received funding from the European 831 Union's Horizon 2020 research and innovation programme under grant 832 agreement No. H2020-ICT-2016-2 761493 (5GTANGO: https://5gtango.eu). 834 10. References 836 10.1. Normative References 838 [ETS14a] ETSI, "Architectural Framework - ETSI GS NFV 002 V1.2.1", 839 Dec 2014, . 842 [ETS14b] ETSI, "Terminology for Main Concepts in NFV - ETSI GS NFV 843 003 V1.2.1", Dec 2014, 844 . 847 [ETS14c] ETSI, "NFV Pre-deployment Testing - ETSI GS NFV TST001 848 V1.1.1", April 2016, 849 . 852 [ETS14d] ETSI, "Network Functions Virtualisation (NFV); Virtual 853 Network Functions Architecture - ETSI GS NFV SWA001 854 V1.1.1", December 2014, 855 . 859 [RFC1242] S. Bradner, "Benchmarking Terminology for Network 860 Interconnection Devices", July 1991, 861 . 863 [RFC8172] A. Morton, "Considerations for Benchmarking Virtual 864 Network Functions and Their Infrastructure", July 2017, 865 . 867 [RFC8204] M. Tahhan, B. O'Mahony, A. Morton, "Benchmarking Virtual 868 Switches in the Open Platform for NFV (OPNFV)", September 869 2017, . 871 10.2. Informative References 873 [Gym] "Gym Home Page", . 875 [Peu-a] M. Peuster, H. Karl, "Understand Your Chains: Towards 876 Performance Profile-based Network Service Management", 877 Fifth European Workshop on Software Defined Networks 878 (EWSDN) , 2016, 879 . 881 [Peu-b] M. Peuster, H. Karl, "Profile Your Chains, Not Functions: 882 Automated Network Service Profiling in DevOps 883 Environments", IEEE Conference on Network Function 884 Virtualization and Software Defined Networks (NFV-SDN) , 885 2017, . 887 [Rosa-a] R. V. Rosa, C. E. Rothenberg, R. Szabo, "VBaaS: VNF 888 Benchmark-as-a-Service", Fourth European Workshop on 889 Software Defined Networks , Sept 2015, 890 . 892 [Rosa-b] R. Rosa, C. Bertoldo, C. Rothenberg, "Take your VNF to the 893 Gym: A Testing Framework for Automated NFV Performance 894 Benchmarking", IEEE Communications Magazine Testing 895 Series , Sept 2017, 896 . 898 [Rosa-c] R. V. Rosa, C. E. Rothenberg, "Taking Open vSwitch to the 899 Gym: An Automated Benchmarking Approach", IV Workshop pre- 900 IETF/IRTF, CSBC Brazil, July 2017, 901 . 904 [tango] "5GTANGO: Development and validation platform for global 905 industry-specific network services and apps", 906 . 908 [tng-bench] 909 "5GTANGO VNF/NS Benchmarking Framework", 910 . 912 Authors' Addresses 914 Raphael Vicente Rosa (editor) 915 University of Campinas 916 Av. Albert Einstein, 400 917 Campinas, Sao Paulo 13083-852 918 Brazil 920 Email: rvrosa@dca.fee.unicamp.br 921 URI: https://intrig.dca.fee.unicamp.br/raphaelvrosa/ 923 Christian Esteve Rothenberg 924 University of Campinas 925 Av. Albert Einstein, 400 926 Campinas, Sao Paulo 13083-852 927 Brazil 929 Email: chesteve@dca.fee.unicamp.br 930 URI: http://www.dca.fee.unicamp.br/~chesteve/ 932 Manuel Peuster 933 Paderborn University 934 Warburgerstr. 100 935 Paderborn 33098 936 Germany 938 Email: manuel.peuster@upb.de 939 URI: http://go.upb.de/peuster 940 Holger Karl 941 Paderborn University 942 Warburgerstr. 100 943 Paderborn 33098 944 Germany 946 Email: holger.karl@upb.de 947 URI: https://cs.uni-paderborn.de/cn/