idnits 2.17.1 draft-rosa-bmwg-vnfbench-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 981: '... and MUST NOT be connected to device...' RFC 2119 keyword, line 985: '...ial capabilities SHOULD NOT exist in t...' RFC 2119 keyword, line 988: '...loyment scenario SHOULD be identical i...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 24, 2019) is 1761 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BMWG R. Rosa, Ed. 3 Internet-Draft C. Rothenberg 4 Intended status: Informational UNICAMP 5 Expires: December 26, 2019 M. Peuster 6 H. Karl 7 UPB 8 June 24, 2019 10 Methodology for VNF Benchmarking Automation 11 draft-rosa-bmwg-vnfbench-04 13 Abstract 15 This document describes a common methodology for the automated 16 benchmarking of Virtualized Network Functions (VNFs) executed on 17 general-purpose hardware. Specific cases of automated benchmarking 18 methodologies for particular VNFs can be derived from this document. 19 Two open source reference implementations are reported as running 20 code embodiments of the proposed, automated benchmarking methodology. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on December 26, 2019. 39 Copyright Notice 41 Copyright (c) 2019 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (https://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 57 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 58 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 59 4. Considerations . . . . . . . . . . . . . . . . . . . . . . . 4 60 4.1. VNF Testing Methods . . . . . . . . . . . . . . . . . . . . 5 61 4.2. Benchmarking Procedures . . . . . . . . . . . . . . . . . . 5 62 4.2.1. Phase I: Deployment . . . . . . . . . . . . . . . . . . . 6 63 4.2.2. Phase II: Configuration . . . . . . . . . . . . . . . . . 6 64 4.2.3. Phase III: Execution . . . . . . . . . . . . . . . . . . 6 65 4.2.4. Phase IV: Report . . . . . . . . . . . . . . . . . . . . 7 66 5. Generic VNF Benchmarking Architectural Framework . . . . . . 7 67 5.1. Deployment Scenarios . . . . . . . . . . . . . . . . . . . 10 68 6. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 10 69 6.1. VNF Benchmarking Descriptor (VNF-BD) . . . . . . . . . . . 12 70 6.1.1. Descriptor Headers . . . . . . . . . . . . . . . . . . . 12 71 6.1.2. Target Information . . . . . . . . . . . . . . . . . . . 12 72 6.1.3. Experiments . . . . . . . . . . . . . . . . . . . . . . . 12 73 6.1.4. Environment . . . . . . . . . . . . . . . . . . . . . . . 12 74 6.1.5. Scenario . . . . . . . . . . . . . . . . . . . . . . . . 13 75 6.1.6. Proceedings . . . . . . . . . . . . . . . . . . . . . . . 14 76 6.2. VNF Performance Profile (VNF-PP) . . . . . . . . . . . . . 14 77 6.2.1. Execution Environment . . . . . . . . . . . . . . . . . . 14 78 6.2.2. Measurement Results . . . . . . . . . . . . . . . . . . . 15 79 6.3. Procedures . . . . . . . . . . . . . . . . . . . . . . . . 16 80 6.3.1. Pre-Execution . . . . . . . . . . . . . . . . . . . . . . 16 81 6.3.2. Automated Execution . . . . . . . . . . . . . . . . . . . 17 82 6.3.3. Post-Execution . . . . . . . . . . . . . . . . . . . . . 18 83 6.4. Particular Cases . . . . . . . . . . . . . . . . . . . . . 18 84 6.4.1. Capacity . . . . . . . . . . . . . . . . . . . . . . . . 18 85 6.4.2. Isolation . . . . . . . . . . . . . . . . . . . . . . . . 19 86 6.4.3. Failure Handling . . . . . . . . . . . . . . . . . . . . 19 87 6.4.4. Elasticity and Flexibility . . . . . . . . . . . . . . . 19 88 6.4.5. Handling Configurations . . . . . . . . . . . . . . . . . 19 89 6.4.6. White Box VNF . . . . . . . . . . . . . . . . . . . . . . 19 90 7. Open Source Reference Implementations . . . . . . . . . . . . 20 91 7.1. Gym . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 92 7.2. tng-bench . . . . . . . . . . . . . . . . . . . . . . . . . 21 93 8. Security Considerations . . . . . . . . . . . . . . . . . . . 22 94 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 95 10. Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 22 96 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 97 11.1. Normative References . . . . . . . . . . . . . . . . . . . 22 98 11.2. Informative References . . . . . . . . . . . . . . . . . . 23 99 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 101 1. Introduction 103 The Benchmarking Methodology Working Group (BMWG) already presented 104 considerations for benchmarking of VNFs and their infrastructure in 105 [RFC8172]. Similar to the motivation given in [RFC8172], the 106 following aspects justify the need for VNF benchmarking: (i) pre- 107 deployment infrastructure dimensioning to realize associated VNF 108 performance profiles; (ii) comparison factor with physical network 109 functions; (iii) and output results for analytical VNF development. 111 Even if many methodologies already described by the BMWG, e.g., self- 112 contained black-box benchmarking, can be applied to VNF benchmarking 113 scenarios, further considerations have to be made. This is, on the 114 one hand, because VNFs, which are software components, do not have 115 strict and clear execution boundaries and depend on underlying 116 virtualization environment parameters as well as management and 117 orchestration decisions [ETS14a]. On the other hand, can and should 118 the flexible, software-based nature of VNFs be exploited to fully 119 automate the entire benchmarking procedure end-to-end. This is an 120 inherent need to align VNF benchmarking with the agile methods 121 enabled by the concept of Network Functions Virtualization (NFV) 122 [ETS14e]. More specifically it allows: (i) the development of agile 123 performance-focused DevOps methodologies for Continuous Integration 124 and Delivery (CI/CD) of VNFs; (ii) the creation of on-demand VNF test 125 descriptors for upcoming execution environments; (iii) the path for 126 precise-analytics of automated catalogues of VNF performance 127 profiles; (iv) and run-time mechanisms to assist VNF lifecycle 128 orchestration/management workflows, e.g., automated resource 129 dimensioning based on benchmarking insights. 131 This document describes basic methodologies and guidelines to fully 132 automate VNF benchmarking procedures, without limiting the automated 133 process to a specific benchmark or infrastructure. After presenting 134 initial considerations, the document first describes a generic 135 architectural framework to setup automated benchmarking experiments. 136 Second, the automation methodology is discussed, with a particular 137 focus on experiment and procedure description approaches to support 138 reproducibility of the automated benchmarks, a key challenge in VNF 139 benchmarking. Finally, two independent, open-source reference 140 implementations are presented. The document addresses state-of-the- 141 art work on VNF benchmarking from scientific publications and current 142 developments in other standardization bodies (e.g., [ETS14c] and 143 [RFC8204]) wherever possible. 145 2. Terminology 147 Common benchmarking terminology contained in this document is derived 148 from [RFC1242]. The reader is assumed to be familiar with the 149 terminology as defined in the European Telecommunications Standards 150 Institute (ETSI) NFV document [ETS14b]. Some of these terms, and 151 others commonly used in this document, are defined below. 153 NFV: Network Function Virtualization - the principle of separating 154 network functions from the hardware they run on by using virtual 155 hardware abstraction. 157 VNF: Virtualized Network Function - a software-based network 158 function. A VNF can be either represented by a single entity or 159 be composed by a set of smaller, interconnected software 160 components, called VNF components (VNFCs) [ETS14d]. Those VNFs 161 are also called composed VNFs. 163 VNFC: Virtualized Network Function Component - a software component 164 that implements (parts of) the VNF functionality. A VNF can 165 consist of a single VNFC or multiple, interconnected VNFCs 166 [ETS14d] 168 VNFD: Virtualised Network Function Descriptor - configuration 169 template that describes a VNF in terms of its deployment and 170 operational behaviour, and is used in the process of VNF on- 171 boarding and managing the life cycle of a VNF instance. 173 NS: Network Service - a collection of interconnected VNFs forming a 174 end-to-end service. The interconnection is often done using 175 chaining of functions. 177 3. Scope 179 This document assumes VNFs as black boxes when defining their 180 benchmarking methodologies. White box approaches are assumed and 181 analysed as a particular case under the proper considerations of 182 internal VNF instrumentation, later discussed in this document. 184 This document outlines a methodology for VNF benchmarking, 185 specifically addressing its automation. 187 4. Considerations 189 VNF benchmarking considerations are defined in [RFC8172]. 190 Additionally, VNF pre-deployment testing considerations are well 191 explored in [ETS14c]. 193 4.1. VNF Testing Methods 195 Following ETSI's model in [ETS14c], we distinguish three methods for 196 VNF evaluation: 198 Benchmarking: Where parameters (e.g., CPU, memory, storage) are 199 provided and the corresponding performance metrics (e.g., latency, 200 throughput) are obtained. Note, such evaluations might create 201 multiple reports, for example, with minimal latency or maximum 202 throughput results. 204 Verification: Both parameters and performance metrics are provided 205 and a stimulus verifies if the given association is correct or 206 not. 208 Dimensioning: Performance metrics are provided and the corresponding 209 parameters obtained. Note, multiple deployments may be required, 210 or if possible, underlying allocated resources need to be 211 dynamically altered. 213 Note: Verification and Dimensioning can be reduced to Benchmarking. 214 Therefore, we focus on Benchmarking in the rest of the document. 216 4.2. Benchmarking Procedures 218 A (automated) benchmarking procedure can be divided into three sub- 219 procedures: 221 Trial: Is a single process or iteration to obtain VNF performance 222 metrics from benchmarking measurements. A Test should always run 223 multiple Trials to get statistical confidence about the obtained 224 measurements. 226 Test: Defines unique structural and functional parameters (e.g., 227 configurations, resource assignment) for benchmarked components to 228 perform one or multiple Trials. Each Test must be executed 229 following a particular benchmarking scenario composed by a Method. 230 Proper measures must be taken to ensure statistical validity 231 (e.g., independence across Trials of generated load patterns). 233 Method: Consists of one or more Tests to benchmark a VNF. A Method 234 can explicitly list ranges of parameter values for the 235 configuration of a benchmarking scenario and its components. Each 236 value of such a range is to be realized in a Test. I.e., Methods 237 can define parameter studies. 239 In general, automated VNF benchmarking Tests must capture relevant 240 causes of performance variability. To dissect a VNF benchmarking 241 Test, in the sections that follow different benchmarking phases are 242 categorized defining generic operations that may be automated. When 243 automating a VNF benchmarking methodology, all the influencing 244 aspects on the performance of a VNF must be carefully analyzed and 245 comprehensively reported, in each phase of the overall benchmarking 246 process. 248 4.2.1. Phase I: Deployment 250 The placement (i.e., assignment and allocation of resources) and the 251 interconnection, physical and/or virtual, of network function(s) and 252 benchmarking components can be realized by orchestration platforms 253 (e.g., OpenStack, Kubernetes, Open Source MANO). In automated 254 manners, the realization of a benchmarking testbed/scenario through 255 those means usually rely on network service templates (e.g., TOSCA, 256 Heat, YANG). Such descriptors have to capture all relevant details 257 of the execution environment to allow the benchmarking framework to 258 correctly instantiate the SUT as well as helper functions required 259 for a Test. 261 4.2.2. Phase II: Configuration 263 The configuration of benchmarking components and VNFs (e.g., populate 264 routing table, load PCAP source files in source of traffic stimulus) 265 to execute the Test settings can be realized by programming 266 interfaces in an automated way. In the scope of NFV, there might 267 exist management interfaces to control a VNF during a benchmarking 268 Test. Likewise, infrastructure or orchestration components can 269 establish the proper configuration of an execution environment to 270 realize all the capabilities enabling the description of the 271 benchmarking Test. Each configuration registry, its deployment 272 timestamp and target, must all be contained in the VNF benchmarking 273 report. 275 4.2.3. Phase III: Execution 277 In the execution of a benchmarking Test, the VNF configuration can be 278 programmed to be changed by itself or by a VNF management platform. 279 It means that during a Trial execution, particular behaviors of a VNF 280 can be automatically triggered, e.g., auto-scaling of its internal 281 components. Those must be captured in the detailed procedures of the 282 VNF execution and its performance report. I.e., the execution of a 283 Trial can determine arrangements of internal states inside a VNF, 284 which can interfere in observed benchmarking metrics. For instance, 285 in a particular benchmarking case where the monitoring measurements 286 of the VNF and/or execution environment are available for extraction, 287 Tests should be run to verify if the monitoring of the VNF and/or 288 execution environment can impact the VNF performance metrics. 290 4.2.4. Phase IV: Report 292 The report of a VNF benchmarking Method might contain generic metrics 293 (e.g., CPU and memory consumption) and VNF-specific traffic 294 processing metrics (e.g., transactions or throughput), which can be 295 stored and processed in generic or specific ways (e.g., by statistics 296 or machine learning algorithms). If automated procedures are applied 297 over the generation of a benchmarking report, those must be detailed 298 in the report itself, jointly with their input raw measurements and 299 output processed data. I.e., any algorithm used in the generation of 300 processed metrics must be disclosed in the report. 302 5. Generic VNF Benchmarking Architectural Framework 304 A generic VNF benchmarking architectural framework, shown in 305 Figure 1, establishes the disposal of essential components and 306 control interfaces, explained below, that enable the automation of 307 VNF benchmarking methodologies. 309 +---------------+ 310 | Manager | 311 Control | (Coordinator) | 312 Interface +---+-------+---+ 313 +--------+-----------+ +-------------------+ 314 | | | 315 | | +-------------------------+ | 316 | | | System Under Test | | 317 | | | | | 318 | | | +-----------------+ | | 319 | +--+------- + VNF | | | 320 | | | | | | 321 | | | +----+ +----+ | | | 322 | | | |VNFC|...|VNFC| | | | 323 | | | +----+ +----+ | | | 324 | | +----.---------.--+ | | 325 +-----+---+ | Monitor | : : | +-----+----+ 326 | Agent | |{listeners}|----^---------V--+ | | Agent | 327 |(Sender) | | | Execution | | |(Receiver)| 328 | | | | Environment | | | | 329 |{Probers}| +-----------| | | |{Probers} | 330 +-----.---+ | +----.---------.--+ | +-----.----+ 331 : +---------^---------V-----+ : 332 V : : : 333 :................>.....: :............>..: 334 Stimulus Traffic Flow 336 Figure 1: Generic VNF Benchmarking Setup 338 Virtualized Network Function (VNF) -- consists of one or more 339 software components, so called VNF components (VNFC), adequate for 340 performing a network function according to allocated virtual 341 resources and satisfied requirements in an execution environment. 342 A VNF can demand particular configurations for benchmarking 343 specifications, demonstrating variable performance based on 344 available virtual resources/parameters and configured enhancements 345 targeting specific technologies (e.g., NUMA, SR-IOV, CPU-Pinning). 347 Execution Environment -- defines a virtualized and controlled 348 composition of capabilities necessary for the execution of a VNF. 349 An execution environment stands as a general purpose level of 350 virtualization with abstracted resources available for one or more 351 VNFs. It can also define specific technology habilitation, 352 incurring in viable settings for enhancing the performance of 353 VNFs. 355 Agent -- executes active stimulus using probers, i.e., benchmarking 356 tools, to benchmark and collect network and system performance 357 metrics. A single Agent can perform localized benchmarks in 358 execution environments (e.g., stress tests on CPU, memory, disk I/ 359 O) or can generate stimulus traffic and the other end be the VNF 360 itself where, for example, one-way latency is evaluated. The 361 interaction among distributed Agents enable the generation and 362 collection of end-to-end metrics (e.g., frame loss rate, latency) 363 measured from stimulus traffic flowing through a VNF. An Agent 364 can be defined by a physical or virtual network function. 366 Prober -- defines an abstraction layer for a software or hardware 367 tool able to generate stimulus traffic to a VNF or perform 368 stress tests on execution environments. Probers might be 369 specific or generic to an execution environment or a VNF. For 370 an Agent, a Prober must provide programmable interfaces for its 371 life cycle management, e.g., configuration of operational 372 parameters, execution of stilumus, parsing of extracted 373 metrics, and debugging options. Specific Probers might be 374 developed to abstract and to realize the description of 375 particular VNF benchmarking methodologies. 377 Monitor -- when possible is instantiated inside the System Under 378 Test, VNF and/or infrastructure (e.g., as a plug-in process in a 379 virtualized execution environment), to perform the passive 380 monitoring, using Listeners, for the extraction of metrics while 381 Agents` stimuli takes place. Monitors observe particular 382 properties according to the execution environment and VNFs 383 capabilities, i.e., exposed passive monitoring interfaces. 384 Multiple Listeners can be executed at once in synchrony with a 385 Prober' stimulus on a SUT. A Monitor can be defined as a 386 virtualized network function. 388 Listener -- defines one or more software interfaces for the 389 extraction of metrics monitored in a target VNF and/or 390 execution environment. A Listener must provide programmable 391 interfaces for its life cycle management workflows, e.g., 392 configuration of operational parameters, execution of passive 393 monitoring captures, parsing of extracted metrics, and 394 debugging options. Varied methods of passive performance 395 monitoring might be implemented as a Listener, depending on the 396 interfaces exposed by the VNF and/or execution environment. 398 Manager -- performs (i) the discovery of available Agents/Monitors 399 and their respective features (i.e., available Probers/Listeners 400 and execution environment capabilities), (ii) the coordination and 401 synchronization of activities of Agents and Monitors to perform a 402 benchmarking Test, (iii) the collection, processing and 403 aggregation of all VNF benchmarking measurements that correlates 404 the VNF stimuli and the, possible, SUT monitored metrics. A 405 Manager executes the main configuration, operation, and management 406 actions to deliver the VNF benchmarking report. A Manager can be 407 defined as a physical or virtualized network function. 409 5.1. Deployment Scenarios 411 A deployment scenario realizes the actual instantiation of physical 412 and/or virtual components of a Generic VNF Benchmarking Architectural 413 Framework needed to habilitate the execution of an automated VNF 414 benchmarking methodology. The following considerations hold for a 415 deployment scenario: 417 o Not all components are mandatory for a Test, possible to be 418 disposed in varied settings. 420 o Components can be composed in a single entity and be defined as 421 black or white boxes. For instance, Manager and Agents could 422 jointly define one hardware/software entity to perform a VNF 423 benchmarking Test and present measurement results. 425 o Monitor can be defined by multiple instances of software 426 components, each addressing a VNF or execution environment. 428 o Agents can be disposed in varied topology setups, included the 429 possibility of multiple input and output ports of a VNF being 430 directly connected each in one Agent. 432 o All benchmarking components defined in a deployment scenario must 433 perform the synchronization of clocks. 435 6. Methodology 437 Portability is an intrinsic characteristic of VNFs and allows them to 438 be deployed in multiple environments. This enables various 439 benchmarking setups in varied deployment scenarios. A VNF 440 benchmarking methodology must be described in a clear and objective 441 manner following four basic principles: 443 o Comparability: Output of Tests shall be simple to understand and 444 process, in a human-readable format, coherent, and easily reusable 445 (e.g., inputs for analytic applications). 447 o Repeatability: A Test setup shall be comprehensively defined 448 through a flexible design model that can be interpreted and 449 executed by the testing platform repeatedly but supporting 450 customization. 452 o Configurability: Open interfaces and extensible messaging models 453 shall be available between benchmarking components for flexible 454 composition of Test descriptors and platform configurations. 456 o Interoperability: Tests shall be ported to different environments 457 using lightweight components. 459 ______________ 460 +--------+ | | 461 | | | Automated | 462 | VNF-BD |--(defines)-->| Benchmarking | 463 | | | Methodology | 464 +--------+ |______________| 465 V 466 | 467 (generates) 468 | 469 v 470 +-------------------------+ 471 | VNF-BR | 472 | +--------+ +--------+ | 473 | | | | | | 474 | | VNF-BD | | VNF-PP | | 475 | | {copy} | | | | 476 | +--------+ +--------+ | 477 +-------------------------+ 479 Figure 2: VNF benchmarking process inputs and outputs 481 As shown in Figure 2, the outcome of an automated VNF benchmarking 482 methodology, must be captured in a VNF Benchmarking Report (VNF-BR), 483 consisting of two parts: 485 VNF Benchmarking Descriptor (VNF-BD) -- contains all required 486 definitions and requirements to deploy, configure, execute, and 487 reproduce VNF benchmarking tests. VNF-BDs are defined by the 488 developer of a benchmarking methodology and serve as input to the 489 benchmarking process, before being included in the generated VNF- 490 BR. 492 VNF Performance Profile (VNF-PP) -- contains all measured metrics 493 resulting from the execution of a benchmarking. Additionally, it 494 might also contain additional recordings of configuration 495 parameters used during the execution of the benchmarking scenario 496 to facilitate comparability of VNF-BRs. 498 A VNF-BR correlates structural and functional parameters of VNF-BD 499 with extracted VNF benchmarking metrics of the obtained VNF-PP. The 500 content of each part of a VNF-BR is described in the following 501 sections. 503 6.1. VNF Benchmarking Descriptor (VNF-BD) 505 VNF Benchmarking Descriptor (VNF-BD) -- an artifact that specifies a 506 Method of how to measure a VNF Performance Profile. The 507 specification includes structural and functional instructions and 508 variable parameters at different abstraction levels (e.g., topology 509 of the deployment scenario, benchmarking target metrics, parameters 510 of benchmarking components). A VNF-BD may be specific to a VNF or 511 applicable to several VNF types. It can be used to elaborate a VNF 512 benchmark deployment scenario aiming at the extraction of particular 513 VNF performance metrics. 515 The following items define the VNF-BD contents. 517 6.1.1. Descriptor Headers 519 The definition of parameters concerning the descriptor file, e.g., 520 its version, identidier, name, author and description. 522 6.1.2. Target Information 524 General information addressing the target VNF(s) the VNF-BD is 525 applicable, with references to any specific characteristics, i.e., 526 the VNF type, model, version/release, author, vendor, architectural 527 components, among any other particular features. 529 6.1.3. Experiments 531 The specification of the number of executions for Trials, Tests and 532 Method. The execution of a VNF-BD corresponds to the execution of 533 the specified Method. 535 6.1.4. Environment 537 The details referring to the name, description, and information 538 associated with the interfaces needed for the orchestration, if 539 necessary, of the specified VNF-BD scenario. I.e., it refers to a 540 specific interface that receives the VNF-BD scenario information and 541 converts it to the template needed for an orchestration platform. In 542 this case, the means to the Manager component interface such 543 orchestration platform must be provided, as well as its outcome 544 orchestration status information (e.g., management interfaces of 545 deployed components). 547 6.1.5. Scenario 549 This section contains all information needed to describe the 550 deployment of all involved functional components mandatory for the 551 execution of the benchmarking Tests addressed by the VNF-BD. 553 6.1.5.1. Nodes 555 Information about each component in a benchmarking setup (see 556 Section 5). It contains the identification, name, image, role (i.e., 557 agent, monitor, sut), connection-points and resource requirements 558 (i.e., allocation of cpu, memory, disk). 560 The lifecycle specification of a node lists all the workflows that 561 must be realized on it during a Test. For instance, main workflows 562 include: create, start, stop, delete. Particular workflows can be 563 specified containing the required parameters and implementation. 564 Those details must reflect the actions taken on or by a node that 565 might affect the VNF performance profile. 567 6.1.5.2. Links 569 Links contain information about the data plane links interconnecting 570 the components of a benchmarking setup. Links refer to two or more 571 connection-points of a node. A link might refer to be part of a 572 network. Depending on the link type, the network might be 573 implemented as a layer 2 mesh, or as directional-oriented traffic 574 forwarding flow entries. Links also detain resource requirements, 575 specifying the minimum bandwidth, latency, and frame loss rate for 576 the execution of benchmarking Tests. 578 6.1.5.3. Policies 580 Involves the definition of execution environment policies to run the 581 Tests. Policies might specify the (anti-)affinity placement rules 582 for each component in the topology, min/max allocation of resources, 583 and specific enabling technologies (e.g., DPDK, SR-IOV, PCIE) needed 584 for each component. 586 6.1.6. Proceedings 588 This information is utilized by the Manager component to execute the 589 benchmarking Tests. It consists of agent(s) and monitor(s) settings, 590 detailing their prober(s)/listener(s) specification and running 591 parameters. 593 Agents: Defines a list containing the Agent(s) needed for the VNF- 594 BD tests. The information of each Agent contains its host 595 environment, making reference to a node specified in the VNF-BD 596 scenario (Section 6.1.5). In addition, each Agent also is defined 597 with the the configured toolset of the Prober(s) and their running 598 parameters fulfilled (e.g., stimulus workload, traffic format/ 599 trace, configurations to enable hardware capabilities, if 600 existent). In each Prober, it is also detailed the output metrics 601 to be extracted from it when running the benchmarking Tests. 603 Monitors: Defines a list containing the Monitor(s) needed for the 604 VNF-BD tests. The information of each Monitor contains its host 605 environment, making reference to a node specified in the VNF-BD 606 scenario (Section 6.1.5) and detailing the placement settings of 607 it (e.g., internal or external with the target VNF and/or 608 execution environment). In addition, each Monitor also is defined 609 with the the configured toolset of the Listener(s) and their 610 running parameters fulfilled (e.g., tap interfaces, period of 611 monitoring, interval among the measurements). In each Listener, 612 it is also detailed the output metrics to be extracted from it 613 when running the benchmarking Tests. 615 6.2. VNF Performance Profile (VNF-PP) 617 VNF Performance Profile (VNF-PP) -- defines a mapping between 618 resources allocated to a VNF (e.g., CPU, memory) as well as assigned 619 configurations (e.g., routing table used by the VNF) and the VNF 620 performance metrics (e.g., throughput, latency, CPU, memory) obtained 621 in a benchmarking Test conducted using a VNF-BD. Logically, packet 622 processing metrics are presented in a specific format addressing 623 statistical significance (e.g., median, standard deviation, 624 percentiles) where a correspondence among VNF parameters and the 625 delivery of a measured VNF performance exists. 627 The following items define the VNF-PP contents. 629 6.2.1. Execution Environment 631 Execution environment information has to be included in every VNF-PP 632 and is required to describe the environment on which a benchmark Test 633 was actually executed. 635 Ideally, any person who has a VNF-BD and its complementing VNF-PP 636 with its execution environment information available, must be able to 637 reproduce the same deployment scenario and VNF benchmarking Tests to 638 obtain identical VNF-PP measurement results. 640 If not already defined by the VNF-BD deployment scenario requirements 641 (Section 6.1.5), for each component in the deployment scenario of the 642 VNF benchmarking setup, the following topics must be detailed: 644 Hardware Specs: Contains any information associated with the 645 underlying hardware capabilities offered and used by the component 646 during the benchmarking Tests. Examples of such specification 647 include allocated CPU architecture, connected NIC specs, allocated 648 memory DIMM, etc. In addition, any information concerning details 649 of resource isolation must also be described in this part of the 650 VNF-PP. 652 Software Specs: Contains any information associated with the 653 software apparatus offered and used during the benchmarking Tests. 654 Examples include versions of operating systems, kernels, 655 hypervisors, container image versions, etc. 657 Optionally, a VNF-PP execution environment might contain references 658 to an orchestration description document (e.g., HEAT template) to 659 clarify technological aspects of the execution environment and any 660 specific parameters that it might contain for the VNF-PP. 662 6.2.2. Measurement Results 664 Measurement results concern the extracted metrics, output of 665 benchmarking procedures, classified into: 667 VNF Processing/Active Metrics: Concerns metrics explicitly defined 668 by or extracted from direct interactions of Agents with a VNF. 669 Those can be defined as generic metric related to network packet 670 processing (e.g., throughput, latency) or metrics specific to a 671 particular VNF (e.g., HTTP confirmed transactions, DNS replies). 673 VNF Monitored/Passive Metrics: Concerns the Monitors' metrics 674 captured from a VNF execution, classified according to the 675 virtualization level (e.g., baremetal, VM, container) and 676 technology domain (e.g., related to CPU, memory, disk) from where 677 they were obtained. 679 Depending on the configuration of the benchmarking setup and the 680 planned use cases for the resulting VNF-PPs, measurement results can 681 be stored as raw data, e.g., time series data about CPU utilization 682 of the VNF during a throughput benchmark. In the case of VNFs 683 composed of multiple VNFCs, those resulting data should be 684 represented as vectors, capturing the behavior of each VNFC, if 685 available from the used monitoring systems. Alternatively, more 686 compact representation formats can be used, e.g., statistical 687 information about a series of latency measurements, including 688 averages and standard deviations. The exact output format to be used 689 is defined in the complementing VNF-BD (Section 6.1). 691 The representation format of a VNF-PP must be easily translated to 692 address the combined set of classified items in the 3x3 Matrix 693 Coverage defined in [RFC8172]. 695 6.3. Procedures 697 The methodology for VNF Benchmarking Automation encompasses the 698 process defined in Figure 2, i.e., the procedures that translate a 699 VNF-BD into a VNF-PP composing a VNF-BR by the means of the 700 components specified in Figure 1. This section details the sequence 701 of events that realize such process. 703 6.3.1. Pre-Execution 705 Before the execution of benchmarking Tests, some procedures must be 706 performed: 708 1. A VNF-BD must be defined to be later instantiated into a 709 deployment scenario and have executed its Tests. Such a 710 description must contain all the structural and functional 711 settings defined in Section 6.1. At the end of this step, the 712 complete Method of benchmarking the target VNF is defined. 714 2. The VNF target image must be prepared to be benchmarked, having 715 all its capabilities fully described. In addition all the probers 716 and listeners defined in the VNF-BD must be implemented to realize 717 the benchmark Tests. At the end of this step, the complete set of 718 components of the benchmarking VNF-BD deployment scenario is 719 defined. 721 3. The environment needed for a VNF-BD must be defined to realize 722 its deployment scenario, in an automated or manual method. This 723 step might count on the instantiation of orchestration platforms 724 and the composition of specific topology descriptors needed by 725 those platforms to realize the VNF-BD deployment scenario. At the 726 end of this step, the whole environment needed to instantiate the 727 components of a VNF-BD deployment scenario is defined. 729 6.3.2. Automated Execution 731 Satisfied all the pre-execution procedures, the automated execution 732 of the Tests specified by the VNF-BD follow: 734 1. Upon the parsing of a VNF-BD, the Manager must detect the VNF-BD 735 variable input field (e.g., list of resources values) and compose 736 the all the permutations of parameters. For each permutation, the 737 Manager must elaborate a VNF-BD instance. Each VNF-BD instance 738 defines a Test, and it will have its deployment scenario 739 instantiated accordingly. I.e., the Manager must interface an 740 orchestration platform to realize the automated instantiation of 741 each deployment scenario defined by a VNF-BD instance (i.e., a 742 Test). The Manager must iterate through all the VNF-BD instances 743 to finish the whole set of Tests defined by all the permutations 744 of the VNF-BD input fields. 746 2. Given a VNF-BD instance, the Manager, using the VNF-BD 747 environment settings, must interface an orchestrator platform 748 requesting the deployment of a scenario to realize a Test. To 749 perform such step, The Manager might interface a management 750 function responsible to properly parse the deployment scenario 751 specifications into the orchestration platform interface format. 753 3. An orchestration platform must deploy the scenario requested by 754 the Manager, assuring the requirements and policies specified on 755 it. In addition, the orchestration platform must acknowledge the 756 deployed scenario to the Manager specifying the management 757 interfaces of the VNF and the other components in the running 758 instances for the benchmarking Test. 760 4. Agent(s) and Monitor(s) (if existing) and the target VNF must be 761 configured by the Manager according to the components settings 762 defined in the VNF-BD instance. After this step, the whole VNF-BD 763 Test will be ready to be executed. 765 5. Manager must interface Agent(s) and Monitor(s) (if existing) via 766 management interfaces to require the execution of the benchmarking 767 probers (and listeners, if existing), and retrieve expected 768 metrics captured during or at the end of each Trial. I.e., for a 769 single Test, according to the VNF-BD execution settings, the 770 Manager must guarantee that one or more Trials realize the 771 required measurements to characterize the performance behavior of 772 a VNF. 774 6. Output measurements from each obtained benchmarking Test, and 775 its possible Trials, must be collected by the Manager, until all 776 the Tests are finished. In the execution settings of the parsed 777 VNF-BD, the Manager must check the Method repetition, and perform 778 the whole set of VNF-BD Tests (i.e., since step 1), until all 779 methods specified are finished. 781 7. Collected all measurements from the VNF-BD (Trials, Tests and 782 Methods) execution, the intended metrics, as described in the VNF- 783 BD, must be parsed, extracted and combined to create the 784 corresponding VNF-PP. The combination of used VNF-BD and 785 generated VNF-PP compose the resulting VNF benchmark report (VNF- 786 BR). 788 6.3.3. Post-Execution 790 After the process of a VNF-BD execution, some automated procedures, 791 not necessarily mandatory, can be performed to improve the quality 792 and utility of a VNF-BR: 794 1. Archive the raw output contained in the VNF-PP, perform 795 statistical analysis on it, or train machine learning models with 796 the collected data. 798 2. Evaluate the analysis output to the detection of any possible 799 cause-effect factors and/or intrinsic correlations in the VNF-BR 800 (e.g., outliers). 802 3. Review the input VNF-BD and modify it to realize the proper 803 extraction of the target VNF metrics based on the performed 804 research. Iterate in the previous steps until composing a stable 805 and representative VNF-BR. 807 6.4. Particular Cases 809 As described in [RFC8172], VNF benchmarking might require to change 810 and adapt existing benchmarking methodologies. More specifically, 811 the following cases need to be considered. 813 6.4.1. Capacity 815 VNFs are usually deployed inside containers or VMs to build an 816 abstraction layer between physical resources and the resources 817 available to the VNF. According to [RFC8172], it may be more 818 representative to design experiments in a way that the VMs hosting 819 the VNFs are operating at maximum of 50% utilization and split the 820 workload among several VMs, to mitigateside effects of overloaded 821 VMs. Those cases are supported by the presented automation 822 methodologies through VNF-BDs that enable direct control over the 823 resource assignments and topology layouts used for a benchmarking 824 experiment. 826 6.4.2. Isolation 828 One of the main challenges of NFV is to create isolation between 829 VNFs. Benchmarking the quality of this isolation behavior can be 830 achieved by Agents that take the role of a noisy neighbor, generating 831 a particular workload in synchrony with a benchmarking procedure over 832 a VNF. Adjustments of the Agent's noisy workload, frequency, 833 virtualization level, among others, must be detailed in the VNF- BD. 835 6.4.3. Failure Handling 837 Hardware and software components will fail or have errors and thus 838 trigger healing actions of the benchmarked VNFs (self-healing). 839 Benchmarking procedures must also capture the dynamics of this VNF 840 behavior, e.g., if a container or VM restarts because the VNF 841 software crashed. This results in offline periods that must be 842 captured in the benchmarking reports, introducing additional metrics, 843 e.g., max. time-to-heal. The presented concept, with a flexible VNF- 844 PP structure to record arbitrary metrics, enables automation of this 845 case. 847 6.4.4. Elasticity and Flexibility 849 Having software based network functions and the possibility of a VNF 850 to be composed by multiple components (VNFCs), internal events of the 851 VNF might trigger changes in VNF behavior, e.g.,activating 852 functionalities associated with elasticity such as automated scaling. 853 These state changes and triggers (e.g. the VNF's scaling state) must 854 be captured in the benchmarking results (VNF-PP) to provide a 855 detailed characterization of the VNF's performance behavior in 856 different states. 858 6.4.5. Handling Configurations 860 As described in [RFC8172], does the sheer number of test conditions 861 and configuration combinations create a challenge for VNF 862 benchmarking. As suggested, machine readable output formats, as they 863 are presented in this document, will allow automated benchmarking 864 procedures to optimize the tested configurations. Approaches for 865 this are, e.g., machine learning-based configuration space sub- 866 sampling methods, such as [Peu-c]. 868 6.4.6. White Box VNF 870 A benchmarking setup must be able to define scenarios with and 871 without monitoring components inside the VNFs and/or the hosting 872 container or VM. If no monitoring solution is available from within 873 the VNFs, the benchmark is following the black-box concept. If, in 874 contrast, those additional sources of information from within the VNF 875 are available, VNF-PPs must be able to handle these additional VNF 876 performance metrics. 878 7. Open Source Reference Implementations 880 Currently, technical motivating factors in favor of the automation of 881 VNF benchmarking methodologies comprise: (i) the facility to run 882 high-fidelity and commodity traffic generators by software; (ii) the 883 existent means to construct synthetic traffic workloads purely by 884 software (e.g., handcrafted pcap files); (iii) the increasing 885 availability of datasets containing actual sources of production 886 traffic able to be reproduced in benchmarking tests; (iv) the 887 existence of a myriad of automating tools and open interfaces to 888 programmatically manage VNFs; (v) the varied set of orchestration 889 platforms enabling the allocation of resources and instantition of 890 VNFs through automated machineries based on well-defined templates; 891 (vi) the ability to utilize a large tool set of software components 892 to compose pipelines that mathematically analyze benchmarking metrics 893 in automated ways. 895 In simple terms, network softwarization enables automation. There 896 are two open source reference implementations that are build to 897 automate benchmarking of Virtualized Network Functions (VNFs). 899 7.1. Gym 901 The software, named Gym, is a framework for automated benchmarking of 902 Virtualized Network Functions (VNFs). It was coded following the 903 initial ideas presented in a 2015 scientific paper entitled "VBaaS: 904 VNF Benchmark-as-a-Service" [Rosa-a]. Later, the evolved design and 905 prototyping ideas were presented at IETF/IRTF meetings seeking impact 906 into NFVRG and BMWG. 908 Gym was built to receive high-level test descriptors and execute them 909 to extract VNFs profiles, containing measurements of performance 910 metrics - especially to associate resources allocation (e.g., vCPU) 911 with packet processing metrics (e.g., throughput) of VNFs. From the 912 original research ideas [Rosa-a], such output profiles might be used 913 by orchestrator functions to perform VNF lifecycle tasks (e.g., 914 deployment, maintenance, tear-down). 916 In [Rosa-b] Gym was utilized to benchmark a decomposed IP Multimedia 917 Subsystem VNF. And in [Rosa-c], a virtual switch (Open vSwitch - 918 OVS) was the target VNF of Gym for the analysis of VNF benchmarking 919 automation. Such articles validated Gym as a prominent open source 920 reference implementation for VNF benchmarking tests. Such articles 921 set important contributions as discussion of the lessons learned and 922 the overall NFV performance testing landscape, included automation. 924 Gym stands as one open source reference implementation that realizes 925 the VNF benchmarking methodologies presented in this document. Gym 926 is released as open source tool under Apache 2.0 license [gym]. 928 7.2. tng-bench 930 Another software that focuses on implementing a framework to 931 benchmark VNFs is the "5GTANGO VNF/NS Benchmarking Framework" also 932 called "tng-bench" (previously "son-profile") and was developed as 933 part of the two European Union H2020 projects SONATA NFV and 5GTANGO 934 [tango]. Its initial ideas were presented in [Peu-a] and the system 935 design of the end-to-end prototype was presented in [Peu-b]. 937 Tng-bench aims to be a framework for the end-to-end automation of VNF 938 benchmarking processes. Its goal is to automate the benchmarking 939 process in such a way that VNF-PPs can be generated without further 940 human interaction. This enables the integration of VNF benchmarking 941 into continuous integration and continuous delivery (CI/CD) pipelines 942 so that new VNF-PPs are generated on-the-fly for every new software 943 version of a VNF. Those automatically generated VNF-PPs can then be 944 bundled with the VNFs and serve as inputs for orchestration systems, 945 fitting to the original research ideas presented in [Rosa-a] and 946 [Peu-a]. 948 Following the same high-level VNF testing purposes as Gym, namely: 949 Comparability, repeatability, configurability, and interoperability, 950 tng- bench specifically aims to explore description approaches for 951 VNF benchmarking experiments. In [Peu-b] a prototype specification 952 for VNF-BDs is presented which not only allows to specify generic, 953 abstract VNF benchmarking experiments, it also allows to describe 954 sets of parameter configurations to be tested during the benchmarking 955 process, allowing the system to automatically execute complex 956 parameter studies on the SUT, e.g., testing a VNF's performance under 957 different CPU, memory, or software configurations. 959 Tng-bench was used to perform a set of initial benchmarking 960 experiments using different VNFs, like a Squid proxy, an Nginx load 961 balancer, and a Socat TCP relay in [Peu-b]. Those VNFs have not only 962 been benchmarked in isolation, but also in combined setups in which 963 up to three VNFs were chained one after each other. These 964 experiments were used to test tng-bench for scenarios in which 965 composed VNFs, consisting of multiple VNF components (VNFCs), have to 966 be benchmarked. The presented results highlight the need to 967 benchmark composed VNFs in end-to-end scenarios rather than only 968 benchmark each individual component in isolation, to produce 969 meaningful VNF- PPs for the complete VNF. 971 Tng-bench is actively developed and released as open source tool 972 under Apache 2.0 license [tng-bench]. 974 8. Security Considerations 976 Benchmarking tests described in this document are limited to the 977 performance characterization of VNFs in a lab environment with 978 isolated network. 980 The benchmarking network topology will be an independent test setup 981 and MUST NOT be connected to devices that may forward the test 982 traffic into a production network, or misroute traffic to the test 983 management network. 985 Special capabilities SHOULD NOT exist in the VNF benchmarking 986 deployment scenario specifically for benchmarking purposes. Any 987 implications for network security arising from the VNF benchmarking 988 deployment scenario SHOULD be identical in the lab and in production 989 networks. 991 9. IANA Considerations 993 This document does not require any IANA actions. 995 10. Acknowledgement 997 The authors would like to thank the support of Ericsson Research, 998 Brazil. Parts of this work have received funding from the European 999 Union's Horizon 2020 research and innovation programme under grant 1000 agreement No. H2020-ICT-2016-2 761493 (5GTANGO: https://5gtango.eu). 1002 11. References 1004 11.1. Normative References 1006 [ETS14a] ETSI, "Architectural Framework - ETSI GS NFV 002 V1.2.1", 1007 Dec 2014, . 1010 [ETS14b] ETSI, "Terminology for Main Concepts in NFV - ETSI GS NFV 1011 003 V1.2.1", Dec 2014, 1012 . 1015 [ETS14c] ETSI, "NFV Pre-deployment Testing - ETSI GS NFV TST001 1016 V1.1.1", April 2016, 1017 . 1020 [ETS14d] ETSI, "Network Functions Virtualisation (NFV); Virtual 1021 Network Functions Architecture - ETSI GS NFV SWA001 1022 V1.1.1", December 2014, 1023 . 1027 [ETS14e] ETSI, "Report on CI/CD and Devops - ETSI GS NFV TST006 1028 V0.0.9", April 2018, 1029 . 1032 [RFC1242] S. Bradner, "Benchmarking Terminology for Network 1033 Interconnection Devices", July 1991, 1034 . 1036 [RFC8172] A. Morton, "Considerations for Benchmarking Virtual 1037 Network Functions and Their Infrastructure", July 2017, 1038 . 1040 [RFC8204] M. Tahhan, B. O'Mahony, A. Morton, "Benchmarking Virtual 1041 Switches in the Open Platform for NFV (OPNFV)", September 1042 2017, . 1044 11.2. Informative References 1046 [gym] "Gym Framework Source Code", 1047 . 1049 [Peu-a] M. Peuster, H. Karl, "Understand Your Chains: Towards 1050 Performance Profile-based Network Service Management", 1051 Fifth European Workshop on Software Defined Networks 1052 (EWSDN) , 2016, 1053 . 1055 [Peu-b] M. Peuster, H. Karl, "Profile Your Chains, Not Functions: 1056 Automated Network Service Profiling in DevOps 1057 Environments", IEEE Conference on Network Function 1058 Virtualization and Software Defined Networks (NFV-SDN) , 1059 2017, . 1061 [Peu-c] M. Peuster, H. Karl, "Understand your chains and keep your 1062 deadlines: Introducing time-constrained profiling for 1063 NFV", IEEE/IFIP 14th International Conference on Network 1064 and Service Management (CNSM) , 2018, 1065 . 1067 [Rosa-a] R. V. Rosa, C. E. Rothenberg, R. Szabo, "VBaaS: VNF 1068 Benchmark-as-a-Service", Fourth European Workshop on 1069 Software Defined Networks , Sept 2015, 1070 . 1072 [Rosa-b] R. Rosa, C. Bertoldo, C. Rothenberg, "Take your VNF to the 1073 Gym: A Testing Framework for Automated NFV Performance 1074 Benchmarking", IEEE Communications Magazine Testing 1075 Series , Sept 2017, 1076 . 1078 [Rosa-c] R. V. Rosa, C. E. Rothenberg, "Taking Open vSwitch to the 1079 Gym: An Automated Benchmarking Approach", IV Workshop pre- 1080 IETF/IRTF, CSBC Brazil, July 2017, 1081 . 1084 [tango] "5GTANGO: Development and validation platform for global 1085 industry-specific network services and apps", 1086 . 1088 [tng-bench] 1089 "5GTANGO VNF/NS Benchmarking Framework", 1090 . 1092 Authors' Addresses 1094 Raphael Vicente Rosa (editor) 1095 University of Campinas 1096 Av. Albert Einstein, 400 1097 Campinas, Sao Paulo 13083-852 1098 Brazil 1100 Email: rvrosa@dca.fee.unicamp.br 1101 URI: https://intrig.dca.fee.unicamp.br/raphaelvrosa/ 1102 Christian Esteve Rothenberg 1103 University of Campinas 1104 Av. Albert Einstein, 400 1105 Campinas, Sao Paulo 13083-852 1106 Brazil 1108 Email: chesteve@dca.fee.unicamp.br 1109 URI: http://www.dca.fee.unicamp.br/~chesteve/ 1111 Manuel Peuster 1112 Paderborn University 1113 Warburgerstr. 100 1114 Paderborn 33098 1115 Germany 1117 Email: manuel.peuster@upb.de 1118 URI: http://go.upb.de/peuster 1120 Holger Karl 1121 Paderborn University 1122 Warburgerstr. 100 1123 Paderborn 33098 1124 Germany 1126 Email: holger.karl@upb.de 1127 URI: https://cs.uni-paderborn.de/cn/