idnits 2.17.1 draft-ietf-bmwg-virtual-net-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 21, 2016) is 2957 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2330' is defined on line 615, but no explicit reference was found in the text == Unused Reference: 'RFC2679' is defined on line 625, but no explicit reference was found in the text == Unused Reference: 'RFC2680' is defined on line 629, but no explicit reference was found in the text == Unused Reference: 'RFC2681' is defined on line 634, but no explicit reference was found in the text == Unused Reference: 'RFC3393' is defined on line 638, but no explicit reference was found in the text == Unused Reference: 'RFC3432' is defined on line 643, but no explicit reference was found in the text == Unused Reference: 'RFC5357' is defined on line 658, but no explicit reference was found in the text == Unused Reference: 'RFC5905' is defined on line 663, but no explicit reference was found in the text == Unused Reference: 'RFC6248' is defined on line 711, but no explicit reference was found in the text == Unused Reference: 'RFC6390' is defined on line 716, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) == Outdated reference: A later version (-09) exists of draft-ietf-bmwg-sdn-controller-benchmark-meth-01 == Outdated reference: A later version (-02) exists of draft-vsperf-bmwg-vswitch-opnfv-01 Summary: 2 errors (**), 0 flaws (~~), 13 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Morton 3 Internet-Draft AT&T Labs 4 Intended status: Informational March 21, 2016 5 Expires: September 22, 2016 7 Considerations for Benchmarking Virtual Network Functions and Their 8 Infrastructure 9 draft-ietf-bmwg-virtual-net-02 11 Abstract 13 Benchmarking Methodology Working Group has traditionally conducted 14 laboratory characterization of dedicated physical implementations of 15 internetworking functions. This memo investigates additional 16 considerations when network functions are virtualized and performed 17 in general purpose hardware. 19 See the new version history section for updates. 21 Requirements Language 23 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 24 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 25 document are to be interpreted as described in RFC 2119 [RFC2119]. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at http://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on September 22, 2016. 44 Copyright Notice 46 Copyright (c) 2016 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (http://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 62 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 63 3. Considerations for Hardware and Testing . . . . . . . . . . . 4 64 3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 4 65 3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5 66 3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6 67 3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7 68 4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 7 69 4.1. Comparison with Physical Network Functions . . . . . . . 7 70 4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8 71 4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 8 72 4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 9 73 4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12 74 5. Security Considerations . . . . . . . . . . . . . . . . . . . 12 75 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 76 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 77 8. Version history . . . . . . . . . . . . . . . . . . . . . . . 13 78 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 79 9.1. Normative References . . . . . . . . . . . . . . . . . . 13 80 9.2. Informative References . . . . . . . . . . . . . . . . . 15 81 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 16 83 1. Introduction 85 Benchmarking Methodology Working Group (BMWG) has traditionally 86 conducted laboratory characterization of dedicated physical 87 implementations of internetworking functions (or physical network 88 functions, PNFs). The Black-box Benchmarks of Throughput, Latency, 89 Forwarding Rates and others have served our industry for many years. 90 [RFC1242] and [RFC2544] are the cornerstones of the work. 92 An emerging set of service provider and vendor development goals is 93 to reduce costs while increasing flexibility of network devices, and 94 drastically accelerate their deployment. Network Function 95 Virtualization (NFV) has the promise to achieve these goals, and 96 therefore has garnered much attention. It now seems certain that 97 some network functions will be virtualized following the success of 98 cloud computing and virtual desktops supported by sufficient network 99 path capacity, performance, and widespread deployment; many of the 100 same techniques will help achieve NFV. 102 In the context of Virtualized Network Functions (VNF), the supporting 103 Infrastructure requires general-purpose computing systems, storage 104 systems, networking systems, virtualization support systems (such as 105 hypervisors), and management systems for the virtual and physical 106 resources. There will be many potential suppliers of Infrastructure 107 systems and significant flexibility in configuring the systems for 108 best performance. There are also many potential suppliers of VNFs, 109 adding to the combinations possible in this environment. The 110 separation of hardware and software suppliers has a profound 111 implication on benchmarking activities: much more of the internal 112 configuration of the black-box device under test (DUT) must now be 113 specified and reported with the results, to foster both repeatability 114 and comparison testing at a later time. 116 Consider the following User Story as further background and 117 motivation: 119 "I'm designing and building my NFV Infrastructure platform. The 120 first steps were easy because I had a small number of categories of 121 VNFs to support and the VNF vendor gave HW recommendations that I 122 followed. Now I need to deploy more VNFs from new vendors, and there 123 are different hardware recommendations. How well will the new VNFs 124 perform on my existing hardware? Which among several new VNFs in a 125 given category are most efficient in terms of capacity they deliver? 126 And, when I operate multiple categories of VNFs (and PNFs) 127 *concurrently* on a hardware platform such that they share resources, 128 what are the new performance limits, and what are the software design 129 choices I can make to optimize my chosen hardware platform? 130 Conversely, what hardware platform upgrades should I pursue to 131 increase the capacity of these concurrently operating VNFs?" 133 See http://www.etsi.org/technologies-clusters/technologies/nfv for 134 more background, for example, the white papers there may be a useful 135 starting place. The Performance and Portability Best Practices 136 [NFV.PER001] are particularly relevant to BMWG. There are documents 137 available in the Open Area http://docbox.etsi.org/ISG/NFV/Open/ 138 Latest_Drafts/ including drafts describing Infrastructure aspects and 139 service quality. 141 2. Scope 143 BMWG will consider the new topic of Virtual Network Functions and 144 related Infrastructure to ensure that common issues are recognized 145 from the start, using background materials from industry and SDOs 146 (e.g., IETF, ETSI NFV). 148 This memo investigates additional methodological considerations 149 necessary when benchmarking VNFs instantiated and hosted in general- 150 purpose hardware, using bare-metal hypervisors or other isolation 151 environments such as Linux containers. An essential consideration is 152 benchmarking physical and virtual network functions in the same way 153 when possible, thereby allowing direct comparison. Also, 154 benchmarking combinations of physical and virtual devices and 155 functions in a System Under Test. 157 A clearly related goal: the benchmarks for the capacity of a general- 158 purpose platform to host a plurality of VNF instances should be 159 investigated. Existing networking technology benchmarks will also be 160 considered for adaptation to NFV and closely associated technologies. 162 A non-goal is any overlap with traditional computer benchmark 163 development and their specific metrics (SPECmark suites such as 164 SPECCPU). 166 A colossal non-goal is any form of architecture development related 167 to NFV and associated technologies in BMWG, consistent with all 168 chartered work since BMWG began in 1989. 170 3. Considerations for Hardware and Testing 172 This section lists the new considerations which must be addressed to 173 benchmark VNF(s) and their supporting infrastructure. The System 174 Under Test (SUT) is composed of the hardware platform components, the 175 VNFs installed, and many other supporting systems. It is critical to 176 document all aspects of the SUT to foster repeatability. 178 3.1. Hardware Components 180 New Hardware devices will become part of the test set-up. 182 1. High volume server platforms (general-purpose, possibly with 183 virtual technology enhancements). 185 2. Storage systems with large capacity, high speed, and high 186 reliability. 188 3. Network Interface ports specially designed for efficient service 189 of many virtual NICs. 191 4. High capacity Ethernet Switches. 193 Labs conducting comparisons of different VNFs may be able to use the 194 same hardware platform over many studies, until the steady march of 195 innovations overtakes their capabilities (as happens with the lab's 196 traffic generation and testing devices today). 198 3.2. Configuration Parameters 200 It will be necessary to configure and document the settings for the 201 entire general-purpose platform to ensure repeatability and foster 202 future comparisons, including but clearly not limited-to the 203 following: 205 o number of server blades (shelf occupation) 207 o CPUs 209 o caches 211 o memory 213 o storage system 215 o I/O 217 as well as configurations that support the devices which host the VNF 218 itself: 220 o Hypervisor (or other forms of virtual function hosting) 222 o Virtual Machine (VM) 224 o Infrastructure Virtual Network (which interconnects Virtual 225 Machines with physical network interfaces, or with each other 226 through virtual switches, for example) 228 and finally, the VNF itself, with items such as: 230 o specific function being implemented in VNF 232 o reserved resources for each function (e.g., CPU pinning) 233 o number of VNFs (or sub-VNF components, each with its own VM) in 234 the service function chain (see section 1.1 of [RFC7498] for a 235 definition of service function chain) 237 o number of physical interfaces and links transited in the service 238 function chain 240 In the physical device benchmarking context, most of the 241 corresponding infrastructure configuration choices were determined by 242 the vendor. Although the platform itself is now one of the 243 configuration variables, it is important to maintain emphasis on the 244 networking benchmarks and capture the platform variables as input 245 factors. 247 3.3. Testing Strategies 249 The concept of characterizing performance at capacity limits may 250 change. For example: 252 1. It may be more representative of system capacity to characterize 253 the case where Virtual Machines (VM, hosting the VNF) are 254 operating at 50% Utilization, and therefore sharing the "real" 255 processing power across many VMs. 257 2. Another important case stems from the need for partitioning 258 functions. A noisy neighbor (VM hosting a VNF in an infinite 259 loop) would ideally be isolated and the performance of other VMs 260 would continue according to their specifications. 262 3. System errors will likely occur as transients, implying a 263 distribution of performance characteristics with a long tail 264 (like latency), leading to the need for longer-term tests of each 265 set of configuration and test parameters. 267 4. The desire for elasticity and flexibility among network functions 268 will include tests where there is constant flux in the number of 269 VM instances, the resources the VMs require, and the set-up/tear- 270 down of network paths that support VM connectivity. Requests for 271 and instantiation of new VMs, along with Releases for VMs hosting 272 VNFs that are no longer needed would be an normal operational 273 condition. In other words, benchmarking should include scenarios 274 with production life cycle management of VMs and their VNFs and 275 network connectivity in-progress, as well as static 276 configurations. 278 5. All physical things can fail, and benchmarking efforts can also 279 examine recovery aided by the virtual architecture with different 280 approaches to resiliency. 282 3.4. Attention to Shared Resources 284 Since many components of the new NFV Infrastructure are virtual, test 285 set-up design must have prior knowledge of inter-actions/dependencies 286 within the various resource domains in the System Under Test (SUT). 287 For example, a virtual machine performing the role of a traditional 288 tester function such as generating and/or receiving traffic should 289 avoid sharing any SUT resources with the Device Under Test DUT. 290 Otherwise, the results will have unexpected dependencies not 291 encountered in physical device benchmarking. 293 Note: The term "tester" has traditionally referred to devices 294 dedicated to testing in BMWG literature. In this new context, 295 "tester" additionally refers to functions dedicated to testing, which 296 may be either virtual or physical. "Tester" has never referred to 297 the individuals performing the tests. 299 The shared-resource aspect of test design remains one of the critical 300 challenges to overcome in a way to produce useful results. 301 Benchmarking set-ups may designate isolated resources for the DUT and 302 other critical support components (such as the host/kernel) as the 303 first baseline step, and add other loading processes. The added 304 complexity of each set-up leads to shared-resource testing scenarios, 305 where the characteristics of the competing load (in terms of memory, 306 storage, and CPU utilization) will directly affect the benchmarking 307 results (and variability of the results), but the results should 308 reconcile with the baseline. 310 The physical test device remains a solid foundation to compare with 311 results using combinations of physical and virtual test functions, or 312 results using only virtual testers when necessary to assess virtual 313 interfaces and other virtual functions. 315 4. Benchmarking Considerations 317 This section discusses considerations related to Benchmarks 318 applicable to VNFs and their associated technologies. 320 4.1. Comparison with Physical Network Functions 322 In order to compare the performance of VNFs and system 323 implementations with their physical counterparts, identical 324 benchmarks must be used. Since BMWG has already developed 325 specifications for many network functions, there will be re-use of 326 existing benchmarks through references, while allowing for the 327 possibility of benchmark curation during development of new 328 methodologies. Consideration should be given to quantifying the 329 number of parallel VNFs required to achieve comparable scale/capacity 330 with a given physical device, or whether some limit of scale was 331 reached before the VNFs could achieve the comparable level. Again, 332 implementation based-on different hypervisors or other virtual 333 function hosting remain as critical factors in performance 334 assessment. 336 4.2. Continued Emphasis on Black-Box Benchmarks 338 When the network functions under test are based on Open Source code, 339 there may be a tendency to rely on internal measurements to some 340 extent, especially when the externally-observable phenomena only 341 support an inference of internal events (such as routing protocol 342 convergence observed in the dataplane). Examples include CPU/Core 343 utilization, Network utilization, Storage utilization, and Memory 344 Comitted/used. These "white-box" metrics provide one view of the 345 resource footprint of a VNF. Note: The resource utilization metrics 346 do not easily match the 3x4 Matrix. 348 However, external observations remain essential as the basis for 349 Benchmarks. Internal observations with fixed specification and 350 interpretation may be provided in parallel (as auxilliary metrics), 351 to assist the development of operations procedures when the 352 technology is deployed, for example. Internal metrics and 353 measurements from Open Source implementations may be the only direct 354 source of performance results in a desired dimension, but 355 corroborating external observations are still required to assure the 356 integrity of measurement discipline was maintained for all reported 357 results. 359 A related aspect of benchmark development is where the scope includes 360 multiple approaches to a common function under the same benchmark. 361 For example, there are many ways to arrange for activation of a 362 network path between interface points and the activation times can be 363 compared if the start-to-stop activation interval has a generic and 364 unambiguous definition. Thus, generic benchmark definitions are 365 preferred over technology/protocol specific definitions where 366 possible. 368 4.3. New Benchmarks and Related Metrics 370 There will be new classes of benchmarks needed for network design and 371 assistance when developing operational practices (possibly automated 372 management and orchestration of deployment scale). Examples follow 373 in the paragraphs below, many of which are prompted by the goals of 374 increased elasticity and flexibility of the network functions, along 375 with accelerated deployment times. 377 o Time to deploy VNFs: In cases where the general-purpose hardware 378 is already deployed and ready for service, it is valuable to know 379 the response time when a management system is tasked with 380 "standing-up" 100's of virtual machines and the VNFs they will 381 host. 383 o Time to migrate VNFs: In cases where a rack or shelf of hardware 384 must be removed from active service, it is valuable to know the 385 response time when a management system is tasked with "migrating" 386 some number of virtual machines and the VNFs they currently host 387 to alternate hardware that will remain in-service. 389 o Time to create a virtual network in the general-purpose 390 infrastructure: This is a somewhat simplified version of existing 391 benchmarks for convergence time, in that the process is initiated 392 by a request from (centralized or distributed) control, rather 393 than inferred from network events (link failure). The successful 394 response time would remain dependent on dataplane observations to 395 confirm that the network is ready to perform. 397 o Effect of verification measurements on performance: A complete 398 VNF, or something as simple as a new poicy to implement in a VNF, 399 is implemented. The action to verify instantiation of the VNF or 400 policy could affect performance during normal operation. 402 Also, it appears to be valuable to measure traditional packet 403 transfer performance metrics during the assessment of traditional and 404 new benchmarks, including metrics that may be used to support service 405 engineering such as the Spatial Composition metrics found in 406 [RFC6049]. Examples include Mean one-way delay in section 4.1 of 407 [RFC6049], Packet Delay Variation (PDV) in [RFC5481], and Packet 408 Reordering [RFC4737] [RFC4689]. 410 4.4. Assessment of Benchmark Coverage 412 It can be useful to organize benchmarks according to their applicable 413 life cycle stage and the performance criteria they were designed to 414 assess. The table below provides a way to organize benchmarks such 415 that there is a clear indication of coverage for the intersection of 416 life cycle stages and performance criteria. 418 |----------------------------------------------------------| 419 | | | | | 420 | | SPEED | ACCURACY | RELIABILITY | 421 | | | | | 422 |----------------------------------------------------------| 423 | | | | | 424 | Activation | | | | 425 | | | | | 426 |----------------------------------------------------------| 427 | | | | | 428 | Operation | | | | 429 | | | | | 430 |----------------------------------------------------------| 431 | | | | | 432 | De-activation | | | | 433 | | | | | 434 |----------------------------------------------------------| 436 For example, the "Time to deploy VNFs" benchmark described above 437 would be placed in the intersection of Activation and Speed, making 438 it clear that there are other potential performance criteria to 439 benchmark, such as the "percentage of unsuccessful VM/VNF stand-ups" 440 in a set of 100 attempts. This example emphasizes that the 441 Activation and De-activation life cycle stages are key areas for NFV 442 and related infrastructure, and encourage expansion beyond 443 traditional benchmarks for normal operation. Thus, reviewing the 444 benchmark coverage using this table (sometimes called the 3x3 matrix) 445 can be a worthwhile exercise in BMWG. 447 In one of the first applications of the 3x3 matrix in BMWG 448 [I-D.ietf-bmwg-sdn-controller-benchmark-meth], we discovered that 449 metrics on measured size, capacity, or scale do not easily match one 450 of the three columns above. Following discussion, this was resolved 451 in two ways: 453 o Add a column, Scale, for use when categorizing and assessing the 454 coverage of benchmarks (without measured results). Examples of 455 this use are found in 456 [I-D.ietf-bmwg-sdn-controller-benchmark-meth] and 457 [I-D.vsperf-bmwg-vswitch-opnfv]. This is the 3x4 Matrix. 459 o If using the matrix to report results in an organized way, keep 460 size, capacity, and scale metrics separate from the 3x3 matrix and 461 incorporate them in the report with other qualifications of the 462 results. 464 Note: The resource utilization (e.g., CPU) metrics do not fit in the 465 Matrix. They are not benchmarks, and omitting them confirms their 466 status as auxilliary metrics. Resource assignments are configuration 467 parameters, and these are reported seperately. 469 This approach encourages use of the 3x3 matrix to organize reports of 470 results, where the capacity at which the various metrics were 471 measured could be included in the title of the matrix (and results 472 for multiple capacities would result in separate 3x3 matrices, if 473 there were sufficient measurements/results to organize in that way). 475 For example, results for each VM and VNF could appear in the 3x3 476 matrix, organized to illustrate resource occupation (CPU Cores) in a 477 particular physical computing system, as shown below. 479 VNF#1 480 .-----------. 481 |__|__|__|__| 482 Core 1 |__|__|__|__| 483 |__|__|__|__| 484 | | | | | 485 '-----------' 486 VNF#2 487 .-----------. 488 |__|__|__|__| 489 Cores 2-5 |__|__|__|__| 490 |__|__|__|__| 491 | | | | | 492 '-----------' 493 VNF#3 VNF#4 VNF#5 494 .-----------. .-----------. .-----------. 495 |__|__|__|__| |__|__|__|__| |__|__|__|__| 496 Core 6 |__|__|__|__| |__|__|__|__| |__|__|__|__| 497 |__|__|__|__| |__|__|__|__| |__|__|__|__| 498 | | | | | | | | | | | | | | | 499 '-----------' '-----------' '-----------' 500 VNF#6 501 .-----------. 502 |__|__|__|__| 503 Core 7 |__|__|__|__| 504 |__|__|__|__| 505 | | | | | 506 '-----------' 508 The combination of tables above could be built incrementally, 509 beginning with VNF#1 and one Core, then adding VNFs according to 510 their supporting core assignments. X-Y plots of critical benchmarks 511 would also provide insight to the effect of increased HW utilization. 512 All VNFs might be of the same type, or to match a production 513 environment there could be VNFs of multiple types and categories. In 514 this figure, VNFs #3-#5 are assumed to require small CPU resources, 515 while VNF#2 requires 4 cores to perform its function. 517 4.5. Power Consumption 519 Although there is incomplete work to benchmark physical network 520 function power consumption in a meaningful way, the desire to measure 521 the physical infrastructure supporting the virtual functions only 522 adds to the need. Both maximum power consumption and dynamic power 523 consumption (with varying load) would be useful. The IPMI standard 524 [IPMI2.0] has been implemented by many manufacturers, and supports 525 measurement of instantaneous energy consumption. 527 To assess the instantaneous energy consumption of virtual resources, 528 it may be possible to estimate the value using an overall metric 529 based on utilization readings, according to 530 [I-D.krishnan-nfvrg-policy-based-rm-nfviaas]. 532 5. Security Considerations 534 Benchmarking activities as described in this memo are limited to 535 technology characterization of a Device Under Test/System Under Test 536 (DUT/SUT) using controlled stimuli in a laboratory environment, with 537 dedicated address space and the constraints specified in the sections 538 above. 540 The benchmarking network topology will be an independent test setup 541 and MUST NOT be connected to devices that may forward the test 542 traffic into a production network, or misroute traffic to the test 543 management network. 545 Further, benchmarking is performed on a "black-box" basis, relying 546 solely on measurements observable external to the DUT/SUT. 548 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 549 benchmarking purposes. Any implications for network security arising 550 from the DUT/SUT SHOULD be identical in the lab and in production 551 networks. 553 6. IANA Considerations 555 No IANA Action is requested at this time. 557 7. Acknowledgements 559 The author acknowledges an encouraging conversation on this topic 560 with Mukhtiar Shaikh and Ramki Krishnan in November 2013. Bhavani 561 Parise and Ilya Varlashkin have provided useful suggestions to expand 562 these considerations. Bhuvaneswaran Vengainathan has already tried 563 the 3x3 matrix with SDN controller draft, and contributed to many 564 discussions. Scott Bradner quickly pointed out shared resource 565 dependencies in an early vSwitch measurement proposal, and the topic 566 was included here as a key consideration. Further development was 567 encouraged by Barry Constantine's comments following the IETF-92 BMWG 568 session: the session itself was an affirmation for this memo. There 569 have been many interesting contributions from Maryam Tahhan, Marius 570 Georgescu, Jacob Rapp, Saurabh Chattopadhyay, and others. 572 8. Version history 574 (This section should be removed by the RFC Editor.) 576 Version 02: 578 New version history section. 580 Added Memory in section 3.2, configuration. 582 Updated ACKs and References. 584 Version 01: 586 Addressed Ramki Krishnan's comments on section 4.5, power, see that 587 section (7/27 message to the list). Addressed Saurabh 588 Chattopadhyay's 7/24 comments on VNF resources and other resource 589 conditions and their effect on benchmarking, see section 3.4. 590 Addressed Marius Georgescu's 7/17 comments on the list (sections 4.3 591 and 4.4). 593 AND, comments from the extended discussion during IETF-93 BMWG 594 session: 596 Section 4.2: VNF footprint and auxilliary metrics (Maryam Tahhan), 597 Section 4.3: Verification affect metrics (Ramki Krishnan); 598 Section 4.4: Auxilliary metrics in the Matrix (Maryam Tahhan, Scott 599 Bradner, others) 601 9. References 603 9.1. Normative References 605 [NFV.PER001] 606 "Network Function Virtualization: Performance and 607 Portability Best Practices", Group Specification ETSI GS 608 NFV-PER 001 V1.1.1 (2014-06), June 2014. 610 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 611 Requirement Levels", BCP 14, RFC 2119, 612 DOI 10.17487/RFC2119, March 1997, 613 . 615 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 616 "Framework for IP Performance Metrics", RFC 2330, 617 DOI 10.17487/RFC2330, May 1998, 618 . 620 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 621 Network Interconnect Devices", RFC 2544, 622 DOI 10.17487/RFC2544, March 1999, 623 . 625 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 626 Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679, 627 September 1999, . 629 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 630 Packet Loss Metric for IPPM", RFC 2680, 631 DOI 10.17487/RFC2680, September 1999, 632 . 634 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 635 Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, 636 September 1999, . 638 [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation 639 Metric for IP Performance Metrics (IPPM)", RFC 3393, 640 DOI 10.17487/RFC3393, November 2002, 641 . 643 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 644 performance measurement with periodic streams", RFC 3432, 645 DOI 10.17487/RFC3432, November 2002, 646 . 648 [RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, 649 "Terminology for Benchmarking Network-layer Traffic 650 Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689, 651 October 2006, . 653 [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, 654 S., and J. Perser, "Packet Reordering Metrics", RFC 4737, 655 DOI 10.17487/RFC4737, November 2006, 656 . 658 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 659 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 660 RFC 5357, DOI 10.17487/RFC5357, October 2008, 661 . 663 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 664 "Network Time Protocol Version 4: Protocol and Algorithms 665 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 666 . 668 [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for 669 Service Function Chaining", RFC 7498, 670 DOI 10.17487/RFC7498, April 2015, 671 . 673 9.2. Informative References 675 [I-D.ietf-bmwg-sdn-controller-benchmark-meth] 676 Vengainathan, B., Basil, A., Tassinari, M., Manral, V., 677 and S. Banks, "Benchmarking Methodology for SDN Controller 678 Performance", draft-ietf-bmwg-sdn-controller-benchmark- 679 meth-01 (work in progress), March 2016. 681 [I-D.krishnan-nfvrg-policy-based-rm-nfviaas] 682 Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D., 683 Wright, S., Hinrichs, T., Krishnaswamy, R., and A. Yerra, 684 "NFVIaaS Architectural Framework for Policy Based Resource 685 Placement and Scheduling", draft-krishnan-nfvrg-policy- 686 based-rm-nfviaas-06 (work in progress), March 2016. 688 [I-D.vsperf-bmwg-vswitch-opnfv] 689 Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking 690 Virtual Switches in OPNFV", draft-vsperf-bmwg-vswitch- 691 opnfv-01 (work in progress), October 2015. 693 [IPMI2.0] "Intelligent Platform Management Interface, v2.0 with 694 latest Errata", 695 http://www.intel.com/content/www/us/en/servers/ipmi/ipmi- 696 intelligent-platform-mgt-interface-spec-2nd-gen-v2-0-spec- 697 update.html, April 2015. 699 [RFC1242] Bradner, S., "Benchmarking Terminology for Network 700 Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, 701 July 1991, . 703 [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation 704 Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, 705 March 2009, . 707 [RFC6049] Morton, A. and E. Stephan, "Spatial Composition of 708 Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011, 709 . 711 [RFC6248] Morton, A., "RFC 4148 and the IP Performance Metrics 712 (IPPM) Registry of Metrics Are Obsolete", RFC 6248, 713 DOI 10.17487/RFC6248, April 2011, 714 . 716 [RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New 717 Performance Metric Development", BCP 170, RFC 6390, 718 DOI 10.17487/RFC6390, October 2011, 719 . 721 Author's Address 723 Al Morton 724 AT&T Labs 725 200 Laurel Avenue South 726 Middletown,, NJ 07748 727 USA 729 Phone: +1 732 420 1571 730 Fax: +1 732 368 1192 731 Email: acmorton@att.com 732 URI: http://home.comcast.net/~acmacm/