idnits 2.17.1 draft-rosa-bmwg-vnfbench-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 985: '... and MUST NOT be connected to device...' RFC 2119 keyword, line 989: '...ial capabilities SHOULD NOT exist in t...' RFC 2119 keyword, line 992: '...loyment scenario SHOULD be identical i...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (December 29, 2018) is 1944 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BMWG R. Rosa, Ed. 3 Internet-Draft C. Rothenberg 4 Intended status: Informational UNICAMP 5 Expires: July 2, 2019 M. Peuster 6 H. Karl 7 UPB 8 December 29, 2018 10 Methodology for VNF Benchmarking Automation 11 draft-rosa-bmwg-vnfbench-03 13 Abstract 15 This document describes a common methodology for the automated 16 benchmarking of Virtualized Network Functions (VNFs) executed on 17 general-purpose hardware. Specific cases of benchmarking 18 methodologies for particular VNFs can be derived from this document. 19 Two open source reference implementations are reported as running 20 code embodiments of the proposed, automated benchmarking methodology. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on July 2, 2019. 39 Copyright Notice 41 Copyright (c) 2018 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (https://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 57 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 58 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 59 4. Considerations . . . . . . . . . . . . . . . . . . . . . . . 5 60 4.1. VNF Testing Methods . . . . . . . . . . . . . . . . . . . . 5 61 4.2. Benchmarking Procedures . . . . . . . . . . . . . . . . . . 6 62 5. A Generic VNF Benchmarking Architectural Framework . . . . . 7 63 5.1. Deployment Scenarios . . . . . . . . . . . . . . . . . . . 9 64 6. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 10 65 6.1. VNF Benchmarking Descriptor (VNF-BD) . . . . . . . . . . . 11 66 6.1.1. Descriptor Headers . . . . . . . . . . . . . . . . . . . 11 67 6.1.2. Target Information . . . . . . . . . . . . . . . . . . . 11 68 6.1.3. Deployment Scenario . . . . . . . . . . . . . . . . . . . 11 69 6.1.4. Settings . . . . . . . . . . . . . . . . . . . . . . . . 12 70 6.2. VNF Performance Profile (VNF-PP) . . . . . . . . . . . . . 13 71 6.2.1. Execution Environment . . . . . . . . . . . . . . . . . . 13 72 6.2.2. Measurement Results . . . . . . . . . . . . . . . . . . . 14 73 6.3. Procedures . . . . . . . . . . . . . . . . . . . . . . . . 15 74 6.3.1. Pre-Execution . . . . . . . . . . . . . . . . . . . . . . 15 75 6.3.2. Automated Execution . . . . . . . . . . . . . . . . . . . 15 76 6.3.3. Post-Execution . . . . . . . . . . . . . . . . . . . . . 17 77 6.4. Particular Cases . . . . . . . . . . . . . . . . . . . . . 17 78 6.4.1. Capacity . . . . . . . . . . . . . . . . . . . . . . . . 17 79 6.4.2. Isolation . . . . . . . . . . . . . . . . . . . . . . . . 17 80 6.4.3. Failure Handling . . . . . . . . . . . . . . . . . . . . 18 81 6.4.4. Elasticity and Flexibility . . . . . . . . . . . . . . . 18 82 6.4.5. Handling Configurations . . . . . . . . . . . . . . . . . 18 83 6.4.6. White Box VNF . . . . . . . . . . . . . . . . . . . . . . 18 84 7. Relevant Influencing Aspects . . . . . . . . . . . . . . . . 18 85 8. Open Source Reference Implementations . . . . . . . . . . . . 19 86 8.1. Gym . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 87 8.2. tng-bench . . . . . . . . . . . . . . . . . . . . . . . . . 21 88 9. Security Considerations . . . . . . . . . . . . . . . . . . . 22 89 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 90 11. Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 22 91 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 92 12.1. Normative References . . . . . . . . . . . . . . . . . . . 22 93 12.2. Informative References . . . . . . . . . . . . . . . . . . 23 94 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 96 1. Introduction 98 The Benchmarking Methodology Working Group (BMWG) already presented 99 considerations for benchmarking of VNFs and their infrastructure in 100 [RFC8172]. Similar to the motivation given in [RFC8172], the 101 following aspects motivate the need for VNF benchmarking: (i) pre- 102 deployment infrastructure dimensioning to realize associated VNF 103 performance profiles; (ii) comparison factor with physical network 104 functions; (iii) and output results for analytical VNF development. 106 Even if many methodologies already described by the BMWG, e.g., self- 107 contained black-box benchmarking, can be applied to VNF benchmarking 108 scenarios, further considerations have to be made. This is, on the 109 one hand, because VNFs, which are software components, do not have 110 strict and clear execution boundaries and depend on underlying 111 virtualization environment parameters as well as management and 112 orchestration decisions [ETS14a]. On the other hand, can and should 113 the flexible, software-based nature of VNFs be exploited to fully 114 automate the entire benchmarking procedure end-to-end. This is an 115 inherent need to align VNF benchmarking with the agile methods 116 enabled by the concept of network function virtualization (NFV) 117 [ETS14e] More specifically it allows: (i) the development of agile 118 performance-focused DevOps methodologies for Continuous Integration 119 and Delivery (CI/CD) of VNFs; (ii) the creation of on-demand VNF test 120 descriptors for upcoming execution environments; (iii) the path for 121 precise-analytics of extensively automated catalogues of VNF 122 performance profiles; (iv) and run-time mechanisms to assist VNF 123 lifecycle orchestration/management workflows, e.g., automated 124 resource dimensioning based on benchmarking insights. 126 This document describes basic methodologies and guidelines to fully 127 automate VNF benchmarking procedures, without limiting the automated 128 process to a specific benchmark or infrastructure. After presenting 129 initial considerations, the document first describes a generic 130 architectural framework to setup automated benchmarking experiments. 131 Second, the automation methodology is discussed, with a particular 132 focus on experiment and procedure description approaches to support 133 reproducibility of the automated benchmarks, a key challenge in VNF 134 benchmarking. Finally, two independent, open-source reference 135 implementations are presented. The document addresses state-of-the- 136 art work on VNF benchmarking from scientific publications and current 137 developments in other standardization bodies (e.g., [ETS14c] and 138 [RFC8204]) wherever possible. 140 2. Terminology 142 Common benchmarking terminology contained in this document is derived 143 from [RFC1242]. Also, the reader is assumed to be familiar with the 144 terminology as defined in the European Telecommunications Standards 145 Institute (ETSI) NFV document [ETS14b]. Some of these terms, and 146 others commonly used in this document, are defined below. 148 NFV: Network Function Virtualization - the principle of separating 149 network functions from the hardware they run on by using virtual 150 hardware abstraction. 152 VNF: Virtualized Network Function - a software-based network 153 function. A VNF can be either represented by a single entity or 154 be composed by a set of smaller, interconnected software 155 components, called VNF components (VNFCs) [ETS14d]. Those VNFs 156 are also called composed VNFs. 158 NS: Network Service - a collection of interconnected VNFs forming a 159 end-to-end service. The interconnection is often done using 160 chaining of functions based on a VNF-FG. 162 VNF-FG: Virtualized Network Function Forwarding Graph - an ordered 163 list of VNFs or VNFCs creating a service chain. 165 NFVI: NFV Infrastructure - collection of NFVI PoPs under one 166 orchestrator. 168 NFVI PoP: NFV Infrastructure Point of Presence - any combination of 169 virtualized compute, storage, and network resources. 171 VIM: Virtualized Infrastructure Manager - functional block that is 172 responsible for controlling and managing the NFVI compute, 173 storage, and network resources, usually within one operator's 174 Infrastructure Domain (e.g. NFVI-PoP). 176 VNFM: Virtualized Network Function Manager - functional block that 177 is responsible for controlling and managing the VNF life-cycle. 179 NFVO: NFV Orchestrator - functional block coordinates the management 180 of network service (NS) life-cycles, VNF life-cycles (supported by 181 the VNFM) and NFVI resources (supported by the VIM) to ensure an 182 optimized allocation of the necessary resources and connectivity. 184 VNFD: Virtualised Network Function Descriptor - configuration 185 template that describes a VNF in terms of its deployment and 186 operational behaviour, and is used in the process of VNF on- 187 boarding and managing the life cycle of a VNF instance. 189 VNFC: Virtualized Network Function Component - a software component 190 that implements (parts of) the VNF functionality. A VNF can 191 consist of a single VNFC or multiple, interconnected VNFCs 192 [ETS14d] 194 3. Scope 196 This document assumes VNFs as black boxes when defining their 197 benchmarking methodologies. White box approaches are assumed and 198 analysed as a particular case under the proper considerations of 199 internal VNF instrumentation, later discussed in this document. 201 This document outlines a methodology for VNF benchmarking, 202 specifically addressing its automation. 204 4. Considerations 206 VNF benchmarking considerations are defined in [RFC8172]. 207 Additionally, VNF pre-deployment testing considerations are well 208 explored in [ETS14c]. 210 4.1. VNF Testing Methods 212 Following ETSI's model in [ETS14c], we distinguish three methods for 213 VNF evaluation: 215 Benchmarking: Where parameters (e.g., CPU, memory, storage) are 216 provided and the corresponding performance metrics (e.g., latency, 217 throughput) are obtained. Note, such evaluations might create 218 multiple reports, for example, with minimal latency or maximum 219 throughput results. 221 Verification: Both parameters and performance metrics are provided 222 and a stimulus verifies if the given association is correct or 223 not. 225 Dimensioning: Performance metrics are provided and the corresponding 226 parameters obtained. Note, multiple deployments may be required, 227 or if possible, underlying allocated resources need to be 228 dynamically altered. 230 Note: Verification and Dimensioning can be reduced to Benchmarking. 231 Therefore, we focus on Benchmarking in the rest of the document. 233 4.2. Benchmarking Procedures 235 VNF benchmarking procedures contain multiple aspects that may or may 236 not be automated: 238 Orchestration: Placement (assignment/allocation of resources) and 239 interconnection (physical/virtual) of network function(s) and 240 benchmark components (e.g., OpenStack/Kubernetes templates, NFV 241 description solutions, like OSM/ONAP). 243 Configuration: Benchmark components and VNFs are configured to 244 execute the test settings (e.g., populate routing table, load PCAP 245 source files in source of stimulus). 247 Execution: Experiments run repeatedly according to configuration 248 and orchestrated components for each VNF benchmarking test case. 250 Output: There might be generic VNF footprint metrics (e.g., CPU and 251 memory consumption) and specific VNF traffic processing metrics 252 (e.g., transactions or throughput), which can be parsed and 253 processed in generic or specific ways (e.g., by statistics or 254 machine learning algorithms). 256 For the purposes of dissecting the automated execution procedures, 257 consider the following definitions: 259 Trial: is a single process or iteration to obtain VNF benchmarking 260 metrics from measurement. A Benchmarking Test should always run 261 multiple Trails to get statistical confidence about the obtained 262 measurements. 264 Test: Defines parameters, e.g., configurations, resource 265 assignment, for benchmarked components to perform one or multiple 266 trials. Each Trial must be executed following a particular 267 deployment scenario composed by a Method. Proper measures must be 268 taken to ensure statistic validity (e.g., independence across 269 trials of generated load patterns). 271 Method: Consists of one or more Tests to benchmark a VNF. A Method 272 can explicitly list ranges of parameter values for the 273 configuration of benchmarking components. Each value of such a 274 range is to be realized in a Test. I.e., Methods can define 275 parameter studies. 277 5. A Generic VNF Benchmarking Architectural Framework 279 A generic VNF benchmarking architectural framework, shown in 280 Figure 1, establishes the disposal of essential components and 281 control interfaces, explained below, that enable the automation of 282 VNF benchmarking methodologies. 284 +---------------+ 285 | Manager | 286 Control | (Coordinator) | 287 Interface +---+-------+---+ 288 +--------+-----------+ +-------------------+ 289 | | | 290 | | +-------------------------+ | 291 | | | System Under Test | | 292 | | | | | 293 | | | +-----------------+ | | 294 | +--+------- + VNF | | | 295 | | | | | | 296 | | | +----+ +----+ | | | 297 | | | |VNFC|...|VNFC| | | | 298 | | | +----+ +----+ | | | 299 | | +----.---------.--+ | | 300 +-----+---+ | Monitor | : : | +-----+----+ 301 | Agent | |{listeners}|----^---------V--+ | | Agent | 302 |(Sender) | | | Execution | | |(Receiver)| 303 | | | | Environment | | | | 304 |{Probers}| +-----------| | | |{Probers} | 305 +-----.---+ | +----.---------.--+ | +-----.----+ 306 : +---------^---------V-----+ : 307 V : : : 308 :................>.....: :............>..: 309 Stimulus Traffic Flow 311 Figure 1: Generic VNF Benchmarking Setup 313 Agent -- executes active stimulus using probers, i.e., benchmarking 314 tools, to benchmark and collect network and system performance 315 metrics. A single Agent can perform localized benchmarks in 316 execution environments (e.g., stress tests on CPU, memory, disk I/ 317 O) or can generate stimulus traffic and the other end be the VNF 318 itself where, for example, one-way latency is evaluated. The 319 interaction among distributed Agents enable the generation and 320 collection of end-to-end metrics (e.g., frame loss rate, latency) 321 measured from stimulus traffic flowing through a VNF. An Agent 322 can be defined by a physical or virtual network function. In 323 addition, Agent must expose to Manager its available Probers and 324 execution environment capabilities. 326 Prober -- defines an abstraction layer for a software or hardware 327 tool able to generate stimulus traffic to a VNF or perform 328 stress tests on execution environments. Probers might be 329 specific or generic to an execution environment or a VNF. For 330 an Agent, a Prober must provide programmable interfaces for its 331 life cycle management workflows, e.g., configuration of 332 operational parameters, execution of stilumus, parsing of 333 extracted metrics, and debugging options. Specific Probers 334 might be developed to abstract and to realize the description 335 of particular benchmarking methodologies. 337 Monitor -- when possible is instantiated inside the System Under 338 Test, VNF and/or NFVI PoP (e.g., as a plug-in process in a 339 virtualized environment), to perform the passive monitoring, using 340 Listeners, for the extraction of metrics while Agents` stimuli 341 takes place. Monitors observe particular properties according to 342 NFVI PoPs and VNFs capabilities, i.e., exposed passive monitoring 343 interfaces. Multiple Listeners can be executed at once in 344 synchrony with a Prober' stimulus on a SUT. A Monitor can be 345 defined as a virtual network function. In addition, Monitor must 346 expose to Manager its available Listeners and execution 347 environment capabilities. 349 Listener -- defines one or more software interfaces for the 350 extraction of particular metrics monitored in a target VNF and/ 351 or execution environment. A Listener must provide programmable 352 interfaces for its life cycle management workflows, e.g., 353 configuration of operational parameters, execution of passive 354 monitoring captures, parsing of extracted metrics, and 355 debugging options. White-box benchmarking approaches must be 356 carefully analyzed, as varied methods of passive performance 357 monitoring might be implemented as a Listener, possibly 358 impacting the VNF and/or execution environment performance 359 results. 361 Manager -- performs (i) the discovery of available Agents/Monitors 362 and their respective features (i.e., available Probers/Listeners 363 and execution environment capabilities), (ii) the coordination and 364 synchronization of activities of Agents and Monitors to perform a 365 benchmark test, (iii) the collection, processing and aggregation 366 of all VNF benchmarking results that correlates the VNF stimuli 367 and the, possible, SUT monitored metrics. A Manager executes the 368 main configuration, operation, and management actions to deliver 369 the VNF benchmarking results. A Manager can be defined as a 370 physical or virtual network function. 372 Virtualized Network Function (VNF) -- consists of one or more 373 software components, so called VNF components (VNFC), adequate for 374 performing a network function according to allocated virtual 375 resources and satisfied requirements in an execution environment. 376 A VNF can demand particular configurations for benchmarking 377 specifications, demonstrating variable performance based on 378 available virtual resources/parameters and configured enhancements 379 targeting specific technologies (e.g., NUMA, SR-IOV, CPU-Pinning). 381 Execution Environment -- defines a virtualized and controlled 382 composition of capabilities necessary for the execution of a VNF. 383 An execution environment stands as a general purpose level of 384 virtualization with abstracted resources available for one or more 385 VNFs. It can also define specific technology habilitation, 386 incurring in viable settings for enhancing the performance of 387 VNFs. 389 5.1. Deployment Scenarios 391 A deployment scenario realizes the instantiation of physical and/or 392 virtual of components of a Generic VNF Benchmarking Architectural 393 Framework needed to habilitate the execution of an automated VNF 394 benchmarking methodology. 396 The following considerations hold for deployment scenarios: 398 o Not all components are mandatory, possible to be disposed in 399 varied settings. 401 o Components can be composed in a single entity and be defined as 402 black or white boxes. For instance, Manager and Agents could 403 jointly define one hardware/software entity to perform a VNF 404 benchmark and present results. 406 o Monitor is not a mandatory component and must be considered only 407 when performed white box benchmarking approaches for a VNF and/or 408 its execution environment. 410 o Monitor can be defined by multiple instances of software 411 components, each addressing a VNF or execution environment and 412 their respective open interfaces for the extraction of metrics. 414 o Agents can be disposed in varied topology setups, included the 415 possibility of multiple input and output ports of a VNF being 416 directly connected each in one Agent. 418 o All benchmarking components defined in a deployment scenario must 419 perform the synchronization of clocks. 421 6. Methodology 423 Portability is an intrinsic characteristic of VNFs and allows them to 424 be deployed in multiple environments. This enables various 425 benchmarking setups in varied deployment scenarios. A VNF 426 benchmarking methodology must be described in a clear and objective 427 manner in order to allow effective repeatability and comparability of 428 the test results. Those results, the outcome of a VNF benchmarking 429 process, must be captured in a VNF Benchmarking Report (VNF-BR), as 430 shown in Figure 2. 432 ______________ 433 +--------+ | | 434 | | | Automated | 435 | VNF-BD |--(defines)-->| Benchmarking | 436 | | | Process | 437 +--------+ |______________| 438 V 439 | 440 (generates) 441 | 442 v 443 +-------------------------+ 444 | VNF-BR | 445 | +--------+ +--------+ | 446 | | | | | | 447 | | VNF-BD | | VNF-PP | | 448 | | {copy} | | | | 449 | +--------+ +--------+ | 450 +-------------------------+ 452 Figure 2: VNF benchmarking process inputs and outputs 454 A VNF Benchmarking Report consist of two parts: 456 VNF Benchmarking Descriptor (VNF-BD) -- contains all required 457 definitions and requirements to deploy, configure, execute, and 458 reproduce VNF benchmarking tests. VNF-BDs are defined by the 459 developer of a benchmarking methodology and serve as input to the 460 benchmarking process, before being included in the generated VNF- 461 BR. 463 VNF Performance Profile (VNF-PP) -- contains all measured metrics 464 resulting from the execution of a benchmarking. Additionally, it 465 might also contain additional recordings of configuration 466 parameters used during the execution of the benchmarking scenario 467 to facilitate comparability of VNF-BRs. 469 A VNF-BR correlates structural and functional parameters of VNF-BD 470 with extracted VNF benchmarking metrics of the obtained VNF-PP. The 471 content of each part of a VNF-BR is described in the following 472 sections. 474 6.1. VNF Benchmarking Descriptor (VNF-BD) 476 VNF Benchmarking Descriptor (VNF-BD) -- an artifact that specifies a 477 method of how to measure a VNF Performance Profile. The 478 specification includes structural and functional instructions and 479 variable parameters at different abstraction levels (e.g., topology 480 of the deployment scenario, benchmarking target metrics, parameters 481 of benchmarking components). VNF-BD may be specific to a VNF or 482 applicable to several VNF types. A VNF-BD can be used to elaborate a 483 VNF benchmark deployment scenario aiming at the extraction of 484 particular VNF performance metrics. 486 The following items define the VNF-BD contents. 488 6.1.1. Descriptor Headers 490 The definition of parameters concerning the descriptor file, e.g., 491 its version, identidier, name, author and description. 493 6.1.2. Target Information 495 General information addressing the target VNF(s) the VNF-BD is 496 applicable, with references to any specific characteristics, i.e., 497 the VNF type, model, version/release, author, vendor, architectural 498 components, among any other particular features. 500 6.1.3. Deployment Scenario 502 This section contains all information needed to describe the 503 deployment of all involved functional components mandatory for the 504 execution of the benchmarking tests addressed by the VNF-BD. 506 6.1.3.1. Topology 508 Information about the experiment topology, concerning the disposition 509 of the components in a benchmarking setup (see Section 5). It must 510 define the type of each component and how they are interconnected 511 (i.e., interface, link and network characteristics). Acceptable 512 topology descriptors might include references to external 513 configuration files particular of orchestration technologies (e.g., 514 TOSCA, YANG). 516 6.1.3.2. Requirements 518 Involves the definition of execution environment requirements for the 519 tests. Therefore, they concern all required capabilities needed for 520 the execution of the target VNF and the other components composing 521 the benchmarking setup. Examples of requirements include the 522 allocation of CPUs, memory and disk for each component in the 523 deployment scenario. 525 6.1.3.3. Policies 527 Involves the definition of execution environment policies to run the 528 tests. Policies might specify the (anti)-affinity placement rules 529 for each component in the topology, min/max allocation of resources, 530 specific enabling technologies (e.g., DPDK, SR-IOV, PCIE) needed for 531 each component. 533 6.1.4. Settings 535 Involves any specific configuration of benchmarking components in a 536 setup described the the deployment scenario topology. 538 6.1.4.1. Components 540 Specifies the details of each component in the described topology in 541 the deployment scenario. For instante, it contains the role of each 542 component and its particular parameters, as the cases detailed below: 544 VNF Configurations: Defines any specific configuration that must be 545 loaded into the VNF to execute the benchmarking experiments (e.g., 546 routing table, firewall rules, subscribers profile). 548 Agents: Defines the configured toolset of probers and related 549 benchmarking/active metrics, available workloads, traffic formats/ 550 traces, and configurations to enable hardware capabilities (if 551 existent). In addition, it defines metrics from each prober to be 552 extracted when running the benchmarking tests. 554 Monitors: defines the configured toolset of listeners and related 555 monitoring/passive metrics, configuration of the interfaces with 556 the monitoring target (VNF and/or execution environment), and 557 configurations to enable specific hardware capabilities (if 558 existent). In addition, it defines metrics from each listener to 559 be extracted when running the benchmarking tests. 561 6.1.4.2. Environment 563 The definition of parameters concerning the execution environment of 564 the VNF-BD, for instance, containing name, description, plugin/ 565 driver, and parameters to realize the interface with an orchestration 566 component responsible to instantiate each VNF-BD deployment scenario. 568 6.1.4.3. Procedures Configuration 570 The definition of parameters concerning the execution of the 571 benchmarking procedures, for instance, containing the number of 572 repetitions for each trial, test, and the whole VNF-BD (method). 574 6.2. VNF Performance Profile (VNF-PP) 576 VNF Performance Profile (VNF-PP) -- defines a mapping between 577 resources allocated to a VNF (e.g., CPU, memory) as well as assigned 578 configurations (e.g., routing table used by the VNF) and the VNF 579 performance metrics (e.g., throughput, latency, CPU, memory) obtained 580 in a benchmarking test conducted using a VNF-BD. Logically, packet 581 processing metrics are presented in a specific format addressing 582 statistical significance (e.g., median, standard deviation, 583 percentiles) where a correspondence among VNF parameters and the 584 delivery of a measured VNF performance exists. 586 The following items define the VNF-PP contents. 588 6.2.1. Execution Environment 590 Execution environment information is has to be included in every VNF- 591 PP and is required to describe the environment on which a benchmark 592 test was actually executed. 594 Ideally, any person who has a VNF-BD and its complementing VNF-PP 595 with its execution environment information available, must be able to 596 reproduce the same deployment scenario and VNF benchmarking tests to 597 obtain identical VNF-PP measurement results. 599 If not already defined by the VNF-BD deployment scenario requirements 600 (Section 6.1.3), for each component in the deployment scenario of the 601 VNF benchmarking setup, the following topics must be detailed: 603 Hardware Specs: Contains any information associated with the 604 underlying hardware capabilities offered and used by the component 605 during the benchmarking tests. Examples of such specification 606 include allocated CPU architecture, connected NIC specs, allocated 607 memory DIMM, etc. In addition, any information concerning details 608 of resource isolation must also be described in this part of the 609 VNF-PP. 611 Software Specs: Contains any information associated with the 612 software apparatus offered and used during the benchmarking tests. 613 Examples include versions of operating systems, kernels, 614 hypervisors, container image versions, etc. 616 Optionally, a VNF-PP execution environment might contain references 617 to an orchestration description document (e.g., HEAT template) to 618 clarify technological aspects of the execution environment and any 619 specific parameters that it might contain for the VNF-PP. 621 6.2.2. Measurement Results 623 Measurement results concern the extracted metrics, output of 624 benchmarking procedures, classified into: 626 VNF Processing/Active Metrics: Concerns metrics explicitly defined 627 by or extracted from direct interactions of Agents with a VNF. 628 Those can be defined as generic metric related to network packet 629 processing (e.g., throughput, latency) or metrics specific to a 630 particular VNF (e.g., vIMS confirmed transactions, DNS replies). 632 VNF Monitored/Passive Metrics: Concerns the Monitors' metrics 633 captured from a VNF execution, classified according to the 634 virtualization level (e.g., baremetal, VM, container) and 635 technology domain (e.g., related to CPU, memory, disk) from where 636 they were obtained. 638 Depending on the configuration of the benchmarking setup and the 639 planned use cases for the resulting VNF-PPs, measurement results can 640 be stored as raw data, e.g., time series data about CPU utilization 641 of the VNF during a throughput benchmark. In the case of VNFs 642 composed of multiple VNFCs, those resulting data should be 643 represented as vectors, capturing the behavior of each VNFC, if 644 available from the used monitoring systems. Alternatively, more 645 compact representation formats can be used, e.g., statistical 646 information about a series of latency measurements, including 647 averages and standard deviations. The exact output format to be used 648 is defined in the complementing VNF-BD (Section 6.1). 650 The representation format of a VNF-PP must be easily translated to 651 address the combined set of classified items in the 3x3 Matrix 652 Coverage defined in [RFC8172]. 654 6.3. Procedures 656 The methodology for VNF Benchmarking Automation encompasses the 657 process defined in Figure 2, i.e., the procedures that translate a 658 VNF-BD into a VNF-PP composing a VNF-BR by the means of the 659 components specified in Figure 1. This section details the sequence 660 of events that realize such process. 662 6.3.1. Pre-Execution 664 Before the execution of benchmark tests, some procedures must be 665 performed: 667 1. A VNF-BD must be defined to be later instantiated into a 668 deployment scenario and executed its tests. Such a description 669 must contain all the structural and functional settings defined in 670 Section 6.1. At the end of this step, the complete method of 671 benchmarking the target VNF is defined. 673 2. The environment needed for a VNF-BD must be defined to realize 674 its deployment scenario, in an automated or manual method. This 675 step might count on the instantiation of orchestration platforms 676 and the composition of specific topology descriptors needed by 677 those platforms to realize the VNF-BD deployment scenario. At the 678 end of this step, the whole environment needed to instantiate the 679 components of a VNF-BD deployment scenario is defined. 681 3. The VNF target image must be prepared to be benchmarked, having 682 all its capabilities fully described. In addition all the probers 683 and listeners defined in the VNF-BD must be implemented to realize 684 the benchmark tests. At the end of this step, the complete set of 685 components of the benchmarking VNF-BD deployment scenario is 686 defined. 688 6.3.2. Automated Execution 690 Satisfied all the pre-execution procedures, the automated execution 691 of the tests specified by the VNF-BD follow: 693 1. Upon the parsing of a VNF-BD, the Manager must detect the VNF-BD 694 variable input field (e.g., list of resources values) and compose 695 the all the permutations of parameters. For each permutation, the 696 Manager must elaborate a VNF-BD instance. Each VNF-BD instance 697 defines a test, and it will have its deployment scenario 698 instantiated accordingly. I.e., the Manager must interface an 699 orchestration platform to realize the automated instantiation of 700 each deployment scenario defined by a VNF-BD instance (i.e., a 701 test). The Manager must iterate through all the VNF-BD instances 702 to finish the whole set of tests defined by all the permutations 703 of the VNF-BD input fields. 705 2. Given a VNF-BD instance, the Manager, using the VNF-BD 706 environment settings, must interface an orchestrator platform 707 requesting the deployment of a scenario to realize a test. To 708 perform such step, The Manager might interface a plugin/driver 709 responsible to properly parse the deployment scenario 710 specifications into the orchestration platform interface format. 712 3. An orchestration platform must deploy the scenario requested by 713 the Manager, assuring the requirements and policies specified on 714 it. In addition, the orchestration platform must acknowledge the 715 deployed scenario to the Manager specifying the management 716 interfaces of the VNF and the other components in the running 717 instances for the benchmarking test. 719 4. Agent(s) and Monitor(s) (if existing) and the target VNF must be 720 configured by the Manager according to the components settings 721 defined in the VNF-BD instance. After this step, the whole VNF-BD 722 test will be ready to be performed. 724 5. Manager must interface Agent(s) and Monitor(s) (if existing) via 725 control interfaces to required the execution of the benchmark 726 stimuli (and monitoring, if existing) and retrieve expected 727 metrics captured during or at the end of each Trial. I.e., for a 728 single test, according to the VNF-BD execution settings, the 729 Manager must guarantee that one or more trials realize the 730 required measurements to characterize the performance behavior of 731 a VNF. 733 6. Output measurements from each obtained benchmarking test, and 734 its possible trials, must be collected by the Manager, until all 735 tests be finished. In the execution settings of the parsed VNF- 736 BD, the Manager must check the method repetition, and perform the 737 whole VNF-BD tests (i.e., since step 1), until all methods are 738 finished. 740 7. Collected all measurements from the VNF-BD (trials, tests and 741 methods) execution, the intended metrics, as described in the VNF- 742 BD, must be parsed, extracted and combined to create the 743 corresponding VNF-PP. The combination of used VNF-BD and 744 generated VNF-PP make up the resulting VNF benchmark report (VNF- 745 BR). 747 6.3.3. Post-Execution 749 After the process of a VNF-BD, generated the associated VNF-BR, some 750 procedures must be guaranteed: 752 1. Perform a statistical analysis of the output VNF-BR. 754 2. Perform a machine learning based analysis of the output VNF-BR. 756 3. Research the analysis outputs to the detect any possible cause- 757 effect factors and/or intrinsic correlations in the VNF-BR (e.g., 758 outliers). 760 4. Review the input VNF-BD and modify it to realize the proper 761 extraction of the target VNF metrics based on the performed 762 research Iterate in the previous steps until composing a stable 763 and representative VNF-BR. 765 6.4. Particular Cases 767 As described in [RFC8172], VNF benchmarking might require to change 768 and adapt existing benchmarking methodologies. More specifically, 769 the following cases need to be considered. 771 6.4.1. Capacity 773 VNFs are usually deployed inside containers or VMs to build an 774 abstraction layer between physical resources and the resources 775 available to the VNF. According to [RFC8172], it may be more 776 representative to design experiments in a way that the VMs hosting 777 the VNFs are operating at maximum of 50% utilization and split the 778 workload among several VMs, to mitigateside effects of overloaded 779 VMs. Those cases are supported by the presented automation 780 methodologies through VNF-BDs that enable direct control over the 781 resource assignments and topology layouts used for a benchmarking 782 experiment. 784 6.4.2. Isolation 786 One of the main challenges of NFV is to create isolation between 787 VNFs. Benchmarking the quality of this isolation behavior can be 788 achieved by Agents that take the role of a noisy neighbor, generating 789 a particular workload in synchrony with a benchmarking procedure over 790 a VNF. Adjustments of the Agent's noisy workload, frequency, 791 virtualization level, among others, must be detailed in the VNF- BD. 793 6.4.3. Failure Handling 795 Hardware and software components will fail or have errors and thus 796 trigger healing actions of the benchmarked VNFs (self-healing). 797 Benchmarking procedures must also capture the dynamics of this VNF 798 behavior, e.g., if a container or VM restarts because the VNF 799 software crashed. This results in offline periods that must be 800 captured in the benchmarking reports, introducing additional metrics, 801 e.g., max. time-to-heal. The presented concept, with a flexible VNF- 802 PP structure to record arbitrary metrics, enables automation of this 803 case. 805 6.4.4. Elasticity and Flexibility 807 Having software based network functions and the possibility of a VNF 808 to be composed by multiple components (VNFCs), internal events of the 809 VNF might trigger changes in VNF behavior, e.g.,activating 810 functionalities associated with elasticity such as automated scaling. 811 These state changes and triggers (e.g. the VNF's scaling state) must 812 be captured in the benchmarking results (VNF-PP) to provide a 813 detailed characterization of the VNF's performance behavior in 814 different states. 816 6.4.5. Handling Configurations 818 As described in [RFC8172], does the sheer number of test conditions 819 and configuration combinations create a challenge for VNF 820 benchmarking. As suggested, machine readable output formats, as they 821 are presented in this document, will allow automated benchmarking 822 procedures to optimize the tested configurations. Approaches for 823 this are, e.g., machine learning-based configuration space sub- 824 sampling methods, such as [Peu-c]. 826 6.4.6. White Box VNF 828 A benchmarking setup must be able to define scenarios with and 829 without monitoring components inside the VNFs and/or the hosting 830 container or VM. If no monitoring solution is available from within 831 the VNFs, the benchmark is following the black-box concept. If, in 832 contrast, those additional sources of information from within the VNF 833 are available, VNF-PPs must be able to handle these additional VNF 834 performance metrics. 836 7. Relevant Influencing Aspects 838 In general, automated VNF benchmarking tests as herein described must 839 capture relevant causes of performance variability. Concerning a 840 deployment scenario, influencing aspects on the performance of a VNF 841 can be observed in: 843 Deployment Scenario Topology: The disposition of components can 844 define particular interconnections among them composing a specific 845 case/method of VNF benchmarking. 847 Execution Environment: The availability of generic and specific 848 capabilities satisfying VNF requirements define a skeleton of 849 opportunities for the allocation of VNF resources. In addition, 850 particular cases can define multiple VNFs interacting in the same 851 execution environment of a benchmarking setup. 853 VNF: A detailed description of functionalities performed by a VNF 854 sets possible traffic forwarding and processing operations it can 855 perform on packets, added to its running requirements and specific 856 configurations, which might affect and compose a benchmarking 857 setup. 859 Agent: The toolset available for the benchmarking stimulus of a VNF 860 and its characteristics of packets format and workload can 861 interfere in a benchmarking setup. VNFs can support specific 862 traffic format as stimulus. 864 Monitor: In a particular benchmarking setup where measurements of 865 VNF and/or execution environment metrics are available for 866 extraction, an important analysis consist in verifying if the 867 Monitor components can impact performance metrics of the VNF and 868 the underlying execution environment. 870 Manager: The overall composition of VNF benchmarking procedures can 871 determine arrangements of internal states inside a VNF, which can 872 interfere in observed benchmarking metrics. 874 The listed influencing aspects must be carefully analyzed while 875 automating a VNF benchmarking methodology. 877 8. Open Source Reference Implementations 879 There are two open source reference implementations that are build to 880 automate benchmarking of Virtualized Network Functions (VNFs). 882 8.1. Gym 884 The software, named Gym, is a framework for automated benchmarking of 885 Virtualized Network Functions (VNFs). It was coded following the 886 initial ideas presented in a 2015 scientific paper entitled "VBaaS: 887 VNF Benchmark-as-a-Service" [Rosa-a]. Later, the evolved design and 888 prototyping ideas were presented at IETF/IRTF meetings seeking impact 889 into NFVRG and BMWG. 891 Gym was built to receive high-level test descriptors and execute them 892 to extract VNFs profiles, containing measurements of performance 893 metrics - especially to associate resources allocation (e.g., vCPU) 894 with packet processing metrics (e.g., throughput) of VNFs. From the 895 original research ideas [Rosa-a], such output profiles might be used 896 by orchestrator functions to perform VNF lifecycle tasks (e.g., 897 deployment, maintenance, tear-down). 899 The proposed guiding principles, elaborated in [Rosa-b], to design 900 and build Gym can be composed in multiple practical ways for 901 different VNF testing purposes: 903 o Comparability: Output of tests shall be simple to understand and 904 process, in a human-read able format, coherent, and easily 905 reusable (e.g., inputs for analytic applications). 907 o Repeatability: Test setup shall be comprehensively defined through 908 a flexible design model that can be interpreted and executed by 909 the testing platform repeatedly but supporting customization. 911 o Configurability: Open interfaces and extensible messaging models 912 shall be available between components for flexible composition of 913 test descriptors and platform configurations. 915 o Interoperability: Tests shall be ported to different environments 916 using lightweight components. 918 In [Rosa-b] Gym was utilized to benchmark a decomposed IP Multimedia 919 Subsystem VNF. And in [Rosa-c], a virtual switch (Open vSwitch - 920 OVS) was the target VNF of Gym for the analysis of VNF benchmarking 921 automation. Such articles validated Gym as a prominent open source 922 reference implementation for VNF benchmarking tests. Such articles 923 set important contributions as discussion of the lessons learned and 924 the overall NFV performance testing landscape, included automation. 926 Gym stands as one open source reference implementation that realizes 927 the VNF benchmarking methodologies presented in this document. Gym 928 is being released open source at [Gym]. The code repository includes 929 also VNF Benchmarking Descriptor (VNF-BD) examples on the vIMS and 930 OVS targets as described in [Rosa-b] and [Rosa-c]. 932 8.2. tng-bench 934 Another software that focuses on implementing a framework to 935 benchmark VNFs is the "5GTANGO VNF/NS Benchmarking Framework" also 936 called "tng-bench" (previously "son-profile") and was developed as 937 part of the two European Union H2020 projects SONATA NFV and 5GTANGO 938 [tango]. Its initial ideas were presented in [Peu-a] and the system 939 design of the end-to-end prototype was presented in [Peu-b]. 941 Tng-bench aims to be a framework for the end-to-end automation of VNF 942 benchmarking processes. Its goal is to automate the benchmarking 943 process in such a way that VNF-PPs can be generated without further 944 human interaction. This enables the integration of VNF benchmarking 945 into continuous integration and continuous delivery (CI/CD) pipelines 946 so that new VNF-PPs are generated on-the-fly for every new software 947 version of a VNF. Those automatically generated VNF-PPs can then be 948 bundled with the VNFs and serve as inputs for orchestration systems, 949 fitting to the original research ideas presented in [Rosa-a] and 950 [Peu-a]. 952 Following the same high-level VNF testing purposes as Gym, namely: 953 Comparability, repeatability, configurability, and interoperability, 954 tng- bench specifically aims to explore description approaches for 955 VNF benchmarking experiments. In [Peu-b] a prototype specification 956 for VNF-BDs is presented which not only allows to specify generic, 957 abstract VNF benchmarking experiments, it also allows to describe 958 sets of parameter configurations to be tested during the benchmarking 959 process, allowing the system to automatically execute complex 960 parameter studies on the SUT, e.g., testing a VNF's performance under 961 different CPU, memory, or software configurations. 963 Tng-bench was used to perform a set of initial benchmarking 964 experiments using different VNFs, like a Squid proxy, an Nginx load 965 balancer, and a Socat TCP relay in [Peu-b]. Those VNFs have not only 966 been benchmarked in isolation, but also in combined setups in which 967 up to three VNFs were chained one after each other. These 968 experiments were used to test tng-bench for scenarios in which 969 composed VNFs, consisting of multiple VNF components (VNFCs), have to 970 be benchmarked. The presented results highlight the need to 971 benchmark composed VNFs in end-to-end scenarios rather than only 972 benchmark each individual component in isolation, to produce 973 meaningful VNF- PPs for the complete VNF. 975 Tng-bench is actively developed and released as open source tool 976 under Apache 2.0 license [tng-bench]. 978 9. Security Considerations 980 Benchmarking tests described in this document are limited to the 981 performance characterization of VNFs in a lab environment with 982 isolated network. 984 The benchmarking network topology will be an independent test setup 985 and MUST NOT be connected to devices that may forward the test 986 traffic into a production network, or misroute traffic to the test 987 management network. 989 Special capabilities SHOULD NOT exist in the VNF benchmarking 990 deployment scenario specifically for benchmarking purposes. Any 991 implications for network security arising from the VNF benchmarking 992 deployment scenario SHOULD be identical in the lab and in production 993 networks. 995 10. IANA Considerations 997 This document does not require any IANA actions. 999 11. Acknowledgement 1001 The authors would like to thank the support of Ericsson Research, 1002 Brazil. Parts of this work have received funding from the European 1003 Union's Horizon 2020 research and innovation programme under grant 1004 agreement No. H2020-ICT-2016-2 761493 (5GTANGO: https://5gtango.eu). 1006 12. References 1008 12.1. Normative References 1010 [ETS14a] ETSI, "Architectural Framework - ETSI GS NFV 002 V1.2.1", 1011 Dec 2014, . 1014 [ETS14b] ETSI, "Terminology for Main Concepts in NFV - ETSI GS NFV 1015 003 V1.2.1", Dec 2014, 1016 . 1019 [ETS14c] ETSI, "NFV Pre-deployment Testing - ETSI GS NFV TST001 1020 V1.1.1", April 2016, 1021 . 1024 [ETS14d] ETSI, "Network Functions Virtualisation (NFV); Virtual 1025 Network Functions Architecture - ETSI GS NFV SWA001 1026 V1.1.1", December 2014, 1027 . 1031 [ETS14e] ETSI, "Report on CI/CD and Devops - ETSI GS NFV TST006 1032 V0.0.9", April 2018, 1033 . 1036 [RFC1242] S. Bradner, "Benchmarking Terminology for Network 1037 Interconnection Devices", July 1991, 1038 . 1040 [RFC8172] A. Morton, "Considerations for Benchmarking Virtual 1041 Network Functions and Their Infrastructure", July 2017, 1042 . 1044 [RFC8204] M. Tahhan, B. O'Mahony, A. Morton, "Benchmarking Virtual 1045 Switches in the Open Platform for NFV (OPNFV)", September 1046 2017, . 1048 12.2. Informative References 1050 [Gym] "Gym Home Page", . 1052 [Peu-a] M. Peuster, H. Karl, "Understand Your Chains: Towards 1053 Performance Profile-based Network Service Management", 1054 Fifth European Workshop on Software Defined Networks 1055 (EWSDN) , 2016, 1056 . 1058 [Peu-b] M. Peuster, H. Karl, "Profile Your Chains, Not Functions: 1059 Automated Network Service Profiling in DevOps 1060 Environments", IEEE Conference on Network Function 1061 Virtualization and Software Defined Networks (NFV-SDN) , 1062 2017, . 1064 [Peu-c] M. Peuster, H. Karl, "Understand your chains and keep your 1065 deadlines: Introducing time-constrained profiling for 1066 NFV", IEEE/IFIP 14th International Conference on Network 1067 and Service Management (CNSM) , 2018, 1068 . 1070 [Rosa-a] R. V. Rosa, C. E. Rothenberg, R. Szabo, "VBaaS: VNF 1071 Benchmark-as-a-Service", Fourth European Workshop on 1072 Software Defined Networks , Sept 2015, 1073 . 1075 [Rosa-b] R. Rosa, C. Bertoldo, C. Rothenberg, "Take your VNF to the 1076 Gym: A Testing Framework for Automated NFV Performance 1077 Benchmarking", IEEE Communications Magazine Testing 1078 Series , Sept 2017, 1079 . 1081 [Rosa-c] R. V. Rosa, C. E. Rothenberg, "Taking Open vSwitch to the 1082 Gym: An Automated Benchmarking Approach", IV Workshop pre- 1083 IETF/IRTF, CSBC Brazil, July 2017, 1084 . 1087 [tango] "5GTANGO: Development and validation platform for global 1088 industry-specific network services and apps", 1089 . 1091 [tng-bench] 1092 "5GTANGO VNF/NS Benchmarking Framework", 1093 . 1095 Authors' Addresses 1097 Raphael Vicente Rosa (editor) 1098 University of Campinas 1099 Av. Albert Einstein, 400 1100 Campinas, Sao Paulo 13083-852 1101 Brazil 1103 Email: rvrosa@dca.fee.unicamp.br 1104 URI: https://intrig.dca.fee.unicamp.br/raphaelvrosa/ 1106 Christian Esteve Rothenberg 1107 University of Campinas 1108 Av. Albert Einstein, 400 1109 Campinas, Sao Paulo 13083-852 1110 Brazil 1112 Email: chesteve@dca.fee.unicamp.br 1113 URI: http://www.dca.fee.unicamp.br/~chesteve/ 1114 Manuel Peuster 1115 Paderborn University 1116 Warburgerstr. 100 1117 Paderborn 33098 1118 Germany 1120 Email: manuel.peuster@upb.de 1121 URI: http://go.upb.de/peuster 1123 Holger Karl 1124 Paderborn University 1125 Warburgerstr. 100 1126 Paderborn 33098 1127 Germany 1129 Email: holger.karl@upb.de 1130 URI: https://cs.uni-paderborn.de/cn/