idnits 2.17.1 draft-ietf-bmwg-vswitch-opnfv-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 17 instances of too long lines in the document, the longest one being 2 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 8, 2017) is 2511 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) == Outdated reference: A later version (-03) exists of draft-huang-bmwg-virtual-network-performance-02 Summary: 2 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group M. Tahhan 3 Internet-Draft B. O'Mahony 4 Intended status: Informational Intel 5 Expires: December 10, 2017 A. Morton 6 AT&T Labs 7 June 8, 2017 9 Benchmarking Virtual Switches in OPNFV 10 draft-ietf-bmwg-vswitch-opnfv-04 12 Abstract 14 This memo describes the contributions of the Open Platform for NFV 15 (OPNFV) project on virtual switch performance "VSPERF", particularly 16 in the areas of test set-ups and configuration parameters for the 17 system under test. This project has extended the current and 18 completed work of the Benchmarking Methodology Working Group in IETF, 19 and references existing literature. The Benchmarking Methodology 20 Working Group has traditionally conducted laboratory characterization 21 of dedicated physical implementations of internetworking functions. 22 Therefore, this memo describes the additional considerations when 23 virtual switches are implemented in general-purpose hardware. The 24 expanded tests and benchmarks are also influenced by the OPNFV 25 mission to support virtualization of the "telco" infrastructure. 27 Requirements Language 29 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 30 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 31 document are to be interpreted as described in RFC 2119 [RFC2119]. 33 Status of This Memo 35 This Internet-Draft is submitted in full conformance with the 36 provisions of BCP 78 and BCP 79. 38 Internet-Drafts are working documents of the Internet Engineering 39 Task Force (IETF). Note that other groups may also distribute 40 working documents as Internet-Drafts. The list of current Internet- 41 Drafts is at http://datatracker.ietf.org/drafts/current/. 43 Internet-Drafts are draft documents valid for a maximum of six months 44 and may be updated, replaced, or obsoleted by other documents at any 45 time. It is inappropriate to use Internet-Drafts as reference 46 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on December 10, 2017. 50 Copyright Notice 52 Copyright (c) 2017 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (http://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 68 1.1. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 3 69 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 70 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5 71 3.1. Comparison with Physical Network Functions . . . . . . . 5 72 3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 5 73 3.3. New Configuration Parameters . . . . . . . . . . . . . . 6 74 3.4. Flow classification . . . . . . . . . . . . . . . . . . . 8 75 3.5. Benchmarks using Baselines with Resource Isolation . . . 8 76 4. VSPERF Specification Summary . . . . . . . . . . . . . . . . 10 77 5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 18 78 5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 19 79 5.2. Accuracy of Activation section . . . . . . . . . . . . . 19 80 5.3. Reliability of Activation . . . . . . . . . . . . . . . . 19 81 5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 19 82 5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 19 83 5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 19 84 5.7. Reliability of Operation . . . . . . . . . . . . . . . . 20 85 5.8. Scalability of Operation . . . . . . . . . . . . . . . . 20 86 5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 20 87 6. Security Considerations . . . . . . . . . . . . . . . . . . . 20 88 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 89 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21 90 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 91 9.1. Normative References . . . . . . . . . . . . . . . . . . 21 92 9.2. Informative References . . . . . . . . . . . . . . . . . 22 93 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 95 1. Introduction 97 Benchmarking Methodology Working Group (BMWG) has traditionally 98 conducted laboratory characterization of dedicated physical 99 implementations of internetworking functions. The Black-box 100 Benchmarks of Throughput, Latency, Forwarding Rates and others have 101 served our industry for many years. Now, Network Function 102 Virtualization (NFV) has the goal to transform how internetwork 103 functions are implemented, and therefore has garnered much attention. 105 A virtual switch (vswitch) is an important aspect of the NFV 106 infrastructure; it provides connectivity between and among physical 107 network functions and virtual network functions. As a result, there 108 are many vswitch benchmarking efforts, but few specifications to 109 guide the many new test design choices. This is a complex problem 110 and an industry-wide work-in-progress. In future, several of BMWG's 111 fundamental specifications will likely be updated as more testing 112 experience helps to form consensus around new methodologies, and BMWG 113 should continue to collaborate with all organizations who share the 114 same goal. 116 This memo describes the contributions of the Open Platform for NFV 117 (OPNFV) project on virtual switch performance characterization, 118 "VSPERF", through the Danube 3.0 (fourth) release [DanubeRel] to the 119 chartered work of the BMWG (with stable references to their test 120 descriptions). This project has extended the current and completed 121 work of the BMWG in IETF, and references existing literature. For 122 example, the most often referenced RFC is [RFC2544] (which depends on 123 [RFC1242]), so the foundation of the benchmarking work in OPNFV is 124 common and strong. The recommended extensions are specifically in 125 the areas of test set-ups and configuration parameters for the system 126 under test. 128 See [VSPERFhome] for more background, and the OPNFV website for 129 general information [OPNFV]. 131 The authors note that OPNFV distinguishes itself from other open 132 source compute and networking projects through its emphasis on 133 existing "telco" services as opposed to cloud-computing. There are 134 many ways in which telco requirements have different emphasis on 135 performance dimensions when compared to cloud computing: support for 136 and transfer of isochronous media streams is one example. 138 1.1. Abbreviations 140 For the purposes of this document, the following abbreviations apply: 142 ACK Acknowledge 143 ACPI Advanced Configuration and Power Interface 144 BIOS Basic Input Output System 145 BMWG Benchmarking Methodology Working Group 146 CPDP Control Plane Data Plane 147 CPU Central Processing Unit 148 DIMM Dual In-line Memory Module 149 DPDK Data Plane Development Kit 150 DUT Device Under Test 151 GRUB Grand Unified Bootloader 152 ID Identification 153 IMIX Internet Mix 154 IP Internet Protocol 155 IPPM IP Performance Metrics 156 LAN Local Area Network 157 LTD Level Test Design 158 NFV Network Functions Virtualisation 159 NIC Network Interface Card 160 NUMA Non Uniform Memory Access 161 OPNFV Open Platform for NFV 162 OS Operating System 163 PCI Peripheral Component Interconnect 164 PDV Packet Delay Variation 165 SR/IOV Single Root/Input Output Virtualization 166 SUT System Under Test 167 SW Software 168 TCP Transmission control Protocol 169 TSO TCP Segment Offload 170 UDP User Datagram Protocol 171 VM Virtual Machine 172 VNF Virtualised Network Function 173 VSPERF OPNFV vSwitch Performance Project 175 2. Scope 177 The primary purpose and scope of the memo is to describe key aspects 178 of vswitch benchmarking, particularly in the areas of test set-ups 179 and configuration parameters for the system under test, and extend 180 the body of extensive BMWG literature and experience. Initial 181 feedback indicates that many of these extensions may be applicable 182 beyond this memo's current scope (to hardware switches in the NFV 183 Infrastructure and to virtual routers, for example). Additionally, 184 this memo serves as a vehicle to include more detail and relevant 185 commentary from BMWG and other Open Source communities, under BMWG's 186 chartered work to characterize the NFV Infrastructure. 188 The benchmarking covered in this memo should be applicable to many 189 types of vswitches, and remain vswitch-agnostic to great degree. 191 There has been no attempt to track and test all features of any 192 specific vswitch implementation. 194 3. Benchmarking Considerations 196 This section highlights some specific considerations (from 197 [I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual 198 switches. The OPNFV project is sharing its present view on these 199 areas, as they develop their specifications in the Level Test Design 200 (LTD) document. 202 3.1. Comparison with Physical Network Functions 204 To compare the performance of virtual designs and implementations 205 with their physical counterparts, identical benchmarks are needed. 206 BMWG has developed specifications for many physical network 207 functions. The BMWG has recommended to re-use existing benchmarks 208 and methods in [I-D.ietf-bmwg-virtual-net], and the OPNFV LTD expands 209 on them as described here. A key configuration aspect for vswitches 210 is the number of parallel CPU cores required to achieve comparable 211 performance with a given physical device, or whether some limit of 212 scale will be reached before the vswitch can achieve the comparable 213 performance level. 215 It's unlikely that the virtual switch will be the only application 216 running on the System Under Test (SUT), so CPU utilization, Cache 217 utilization, and Memory footprint should also be recorded for the 218 virtual implementations of internetworking functions. However, 219 internally-measured metrics such as these are not benchmarks; they 220 may be useful for the audience (operations) to know, and may also be 221 useful if there is a problem encountered during testing. 223 Benchmark Comparability between virtual and physical/hardware 224 implementations of equivalent functions will likely place more 225 detailed and exact requirements on the *testing systems* (in terms of 226 stream generation, algorithms to search for max values, and their 227 configurations of course). This is another area for standards 228 development to appreciate. However, the is a topic for a future 229 draft. 231 3.2. Continued Emphasis on Black-Box Benchmarks 233 External observations remain essential as the basis for Benchmarks. 234 Internal observations with fixed specification and interpretation 235 will be provided in parallel to assist the development of operations 236 procedures when the technology is deployed. 238 3.3. New Configuration Parameters 240 A key consideration when conducting any sort of benchmark is trying 241 to ensure the consistency and repeatability of test results. When 242 benchmarking the performance of a vswitch there are many factors that 243 can affect the consistency of results, one key factor is matching the 244 various hardware and software details of the SUT. This section lists 245 some of the many new parameters which this project believes are 246 critical to report in order to achieve repeatability. 248 It has been the goal of the project to produce repeatable results, 249 and a large set of the parameters believed to be critical is provided 250 so that the benchmarking community can better appreciate the increase 251 in configuration complexity inherent in this work. The parameter set 252 below is assumed sufficient for the infrastructure in use by the 253 VSPERF project to obtain repeatable results from test-to-test. 255 Hardware details (platform, processor, memory, and network) 256 including: 258 o BIOS version, release date and any configurations that were 259 modified 261 o Power management at all levels (ACPI sleep states, processor 262 package, OS...) 264 o CPU microcode level 266 o Number of enabled cores 268 o Number of cores used for the test 270 o Memory information (type and size) 272 o Memory DIMM configurations (quad rank performance may not be the 273 same as dual rank) in size, freq and slot locations 275 o Number of physical NICs, as well as their details (manufacturer, 276 versions, type and the PCI slot they are plugged into) 278 o NIC interrupt configuration (and any special features in use) 280 o PCI configuration parameters (payload size, early ACK option, 281 etc.) 283 Software details including: 285 o OS parameters and behavior (text vs graphical no one typing at the 286 console on one system) 288 o OS version (for host and VNF) 290 o Kernel version (for host and VNF) 292 o GRUB boot parameters (for host and VNF) 294 o Hypervisor details (Type and version) 296 o Selected vswitch, version number or commit id used 298 o vswitch launch command line if it has been parameterised 300 o Memory allocation to the vswitch 302 o which NUMA node it is using, and how many memory channels 304 o DPDK or any other SW dependency version number or commit id used 306 o Memory allocation to a VM - if it's from Hugepages/elsewhere 308 o VM storage type: snapshot/independent persistent/independent non- 309 persistent 311 o Number of VMs 313 o Number of Virtual NICs (vNICs), versions, type and driver 315 o Number of virtual CPUs and their core affinity on the host 317 o Number vNIC interrupt configuration 319 o Thread affinitization for the applications (including the vswitch 320 itself) on the host 322 o Details of Resource isolation, such as CPUs designated for Host/ 323 Kernel (isolcpu) and CPUs designated for specific processes 324 (taskset). - Test duration. - Number of flows. 326 Test Traffic Information: 328 o Traffic type - UDP, TCP, others. 330 o Frame Sizes - fixed or IMIX [RFC6985](with [IEEE802.1ac], frames 331 may be longer than 1500 bytes, and up to 2000 bytes) 333 o Deployment Scenario - defines the communications path in the SUT 335 3.4. Flow classification 337 Virtual switches group packets into flows by processing and matching 338 particular packet or frame header information, or by matching packets 339 based on the input ports. Thus a flow can be thought of a sequence 340 of packets that have the same set of header field values, or have 341 arrived on the same physical or logical port. Performance results 342 can vary based on the parameters the vswitch uses to match for a 343 flow. The recommended flow classification parameters for any vswitch 344 performance tests are: the input port (physical or logical), the 345 source MAC address, the destination MAC address, the source IP 346 address, the destination IP address and the Ethernet protocol type 347 field (although classification may take place on other fields, such 348 as source and destination transport port numbers). It is essential 349 to increase the flow timeout time on a vswitch before conducting any 350 performance tests that do not intend to measure the flow setup time, 351 see Section 3 of [RFC2889]. Normally the first packet of a 352 particular stream will install the flow in the virtual switch which 353 adds an additional latency, subsequent packets of the same flow are 354 not subject to this latency if the flow is already installed on the 355 vswitch. 357 3.5. Benchmarks using Baselines with Resource Isolation 359 This outline describes measurement of baseline with isolated 360 resources at a high level, which is the intended approach at this 361 time. 363 1. Baselines: 365 * Optional: Benchmark platform forwarding capability without a 366 vswitch or VNF for at least 72 hours (serves as a means of 367 platform validation and a means to obtain the base performance 368 for the platform in terms of its maximum forwarding rate and 369 latency). 371 Figure 1 Benchmark platform forwarding capability 373 __ 374 +--------------------------------------------------+ | 375 | +------------------------------------------+ | | 376 | | | | | 377 | | Simple Forwarding App | | Host 378 | | | | | 379 | +------------------------------------------+ | | 380 | | NIC | | | 381 +---+------------------------------------------+---+ __| 382 ^ : 383 | | 384 : v 385 +--------------------------------------------------+ 386 | | 387 | traffic generator | 388 | | 389 +--------------------------------------------------+ 391 * Benchmark VNF forwarding capability with direct connectivity 392 (vswitch bypass, e.g., SR/IOV) for at least 72 hours (serves 393 as a means of VNF validation and a means to obtain the base 394 performance for the VNF in terms of its maximum forwarding 395 rate and latency). The metrics gathered from this test will 396 serve as a key comparison point for vswitch bypass 397 technologies performance and vswitch performance. 399 Figure 2 Benchmark VNF forwarding capability 401 __ 402 +--------------------------------------------------+ | 403 | +------------------------------------------+ | | 404 | | | | | 405 | | VNF | | | 406 | | | | | 407 | +------------------------------------------+ | | 408 | | Passthrough/SR-IOV | | Host 409 | +------------------------------------------+ | | 410 | | NIC | | | 411 +---+------------------------------------------+---+ __| 412 ^ : 413 | | 414 : v 415 +--------------------------------------------------+ 416 | | 417 | traffic generator | 418 | | 419 +--------------------------------------------------+ 421 * Benchmarking with isolated resources alone, with other 422 resources (both HW&SW) disabled Example, vswitch and VM are 423 SUT 425 * Benchmarking with isolated resources alone, leaving some 426 resources unused 428 * Benchmark with isolated resources and all resources occupied 430 2. Next Steps 432 * Limited sharing 434 * Production scenarios 436 * Stressful scenarios 438 4. VSPERF Specification Summary 440 The overall specification in preparation is referred to as a Level 441 Test Design (LTD) document, which will contain a suite of performance 442 tests. The base performance tests in the LTD are based on the pre- 443 existing specifications developed by BMWG to test the performance of 444 physical switches. These specifications include: 446 o [RFC2544] Benchmarking Methodology for Network Interconnect 447 Devices 449 o [RFC2889] Benchmarking Methodology for LAN Switching 451 o [RFC6201] Device Reset Characterization 453 o [RFC5481] Packet Delay Variation Applicability Statement 455 Some of the above/newer RFCs are being applied in benchmarking for 456 the first time, and represent a development challenge for test 457 equipment developers. Fortunately, many members of the testing 458 system community have engaged on the VSPERF project, including an 459 open source test system. 461 In addition to this, the LTD also re-uses the terminology defined by: 463 o [RFC2285] Benchmarking Terminology for LAN Switching Devices 465 It is recommended that these references are included in future 466 benchmarking specifications: 468 o [RFC3918] Methodology for IP Multicast Benchmarking 470 o [RFC4737] Packet Reordering Metrics 472 As one might expect, the most fundamental internetworking 473 characteristics of Throughput and Latency remain important when the 474 switch is virtualized, and these benchmarks figure prominently in the 475 specification. 477 When considering characteristics important to "telco" network 478 functions, additional performance metrics are needed. In this case, 479 the project specifications have referenced metrics from the IETF IP 480 Performance Metrics (IPPM) literature. This means that the [RFC2544] 481 test of Latency is replaced by measurement of a metric derived from 482 IPPM's [RFC2679], where a set of statistical summaries will be 483 provided (mean, max, min, and percentiles). Further metrics planned 484 to be benchmarked include packet delay variation as defined by 485 [RFC5481] , reordering, burst behaviour, DUT availability, DUT 486 capacity and packet loss in long term testing at Throughput level, 487 where some low-level of background loss may be present and 488 characterized. 490 Tests have been designed to collect the metrics below: 492 o Throughput Tests to measure the maximum forwarding rate (in frames 493 per second or fps) and bit rate (in Mbps) for a constant load (as 494 defined by [RFC1242]) without traffic loss. 496 o Packet and Frame Delay Distribution Tests to measure average, min 497 and max packet and frame delay for constant loads. 499 o Packet Delay Tests to understand latency distribution for 500 different packet sizes and over an extended test run to uncover 501 outliers. 503 o Scalability Tests to understand how the virtual switch performs 504 with increasing number of flows, number of active ports, 505 configuration complexity of the forwarding logic, etc. 507 o Stream Performance Tests (TCP, UDP) to measure bulk data transfer 508 performance, i.e. how fast systems can send and receive data 509 through the switch. 511 o Control Path and Datapath Coupling Tests, to understand how 512 closely the datapath and the control path are coupled, as well as 513 the effect of this coupling on the performance of the DUT 514 (example: delay of the initial packet of a flow). 516 o CPU and Memory Consumption Tests to understand the virtual 517 switch's footprint on the system, conducted as auxiliary 518 measurements with benchmarks above. They include: CPU 519 utilization, Cache utilization and Memory footprint. 521 o The so-called "Soak" tests, where the selected test is conducted 522 over a long period of time (with an ideal duration of 24 hours, 523 but only long enough to determine that stability issues exist when 524 found; there is no requirement to continue a test when a DUT 525 exhibits instability over time). The key performance 526 characteristics and benchmarks for a DUT are determined (using 527 short duration tests) prior to conducting soak tests. The purpose 528 of soak tests is to capture transient changes in performance which 529 may occur due to infrequent processes, memory leaks, or the low 530 probability coincidence of two or more processes. The stability 531 of the DUT is the paramount consideration, so performance must be 532 evaluated periodically during continuous testing, and this results 533 in use of [RFC2889] Frame Rate metrics instead of [RFC2544] 534 Throughput (which requires stopping traffic to allow time for all 535 traffic to exit internal queues), for example. 537 Additional test specification development should include: 539 o Request/Response Performance Tests (TCP, UDP) which measure the 540 transaction rate through the switch. 542 o Noisy Neighbour Tests, to understand the effects of resource 543 sharing on the performance of a virtual switch. 545 o Tests derived from examination of ETSI NFV Draft GS IFA003 546 requirements [IFA003] on characterization of acceleration 547 technologies applied to vswitches. 549 The flexibility of deployment of a virtual switch within a network 550 means that it is necessary to characterize the performance of a 551 vswitch in various deployment scenarios. The deployment scenarios 552 under consideration include: 554 Figure 3 Physical port to virtual switch to physical port 556 __ 557 +--------------------------------------------------+ | 558 | +--------------------+ | | 559 | | | | | 560 | | v | | Host 561 | +--------------+ +--------------+ | | 562 | | phy port | vswitch | phy port | | | 563 +---+--------------+------------+--------------+---+ __| 564 ^ : 565 | | 566 : v 567 +--------------------------------------------------+ 568 | | 569 | traffic generator | 570 | | 571 +--------------------------------------------------+ 573 Figure 4 Physical port to virtual switch to VNF to virtual switch to 574 physical port 576 __ 577 +---------------------------------------------------+ | 578 | | | 579 | +-------------------------------------------+ | | 580 | | Application | | | 581 | +-------------------------------------------+ | | 582 | ^ : | | 583 | | | | | Guest 584 | : v | | 585 | +---------------+ +---------------+ | | 586 | | logical port 0| | logical port 1| | | 587 +---+---------------+-----------+---------------+---+ __| 588 ^ : 589 | | 590 : v __ 591 +---+---------------+----------+---------------+---+ | 592 | | logical port 0| | logical port 1| | | 593 | +---------------+ +---------------+ | | 594 | ^ : | | 595 | | | | | Host 596 | : v | | 597 | +--------------+ +--------------+ | | 598 | | phy port | vswitch | phy port | | | 599 +---+--------------+------------+--------------+---+ __| 600 ^ : 601 | | 602 : v 603 +--------------------------------------------------+ 604 | | 605 | traffic generator | 606 | | 607 +--------------------------------------------------+ 609 Figure 5 Physical port to virtual switch to VNF to virtual switch to 610 VNF to virtual switch to physical port 612 __ 613 +----------------------+ +----------------------+ | 614 | Guest 1 | | Guest 2 | | 615 | +---------------+ | | +---------------+ | | 616 | | Application | | | | Application | | | 617 | +---------------+ | | +---------------+ | | 618 | ^ | | | ^ | | | 619 | | v | | | v | | Guests 620 | +---------------+ | | +---------------+ | | 621 | | logical ports | | | | logical ports | | | 622 | | 0 1 | | | | 0 1 | | | 623 +---+---------------+--+ +---+---------------+--+__| 624 ^ : ^ : 625 | | | | 626 : v : v _ 627 +---+---------------+---------+---------------+--+ | 628 | | 0 1 | | 3 4 | | | 629 | | logical ports | | logical ports | | | 630 | +---------------+ +---------------+ | | 631 | ^ | ^ | | | Host 632 | | |-----------------| v | | 633 | +--------------+ +--------------+ | | 634 | | phy ports | vswitch | phy ports | | | 635 +---+--------------+----------+--------------+---+_| 636 ^ : 637 | | 638 : v 639 +--------------------------------------------------+ 640 | | 641 | traffic generator | 642 | | 643 +--------------------------------------------------+ 645 Figure 6 Physical port to virtual switch to VNF 647 __ 648 +---------------------------------------------------+ | 649 | | | 650 | +-------------------------------------------+ | | 651 | | Application | | | 652 | +-------------------------------------------+ | | 653 | ^ | | 654 | | | | Guest 655 | : | | 656 | +---------------+ | | 657 | | logical port 0| | | 658 +---+---------------+-------------------------------+ __| 659 ^ 660 | 661 : __ 662 +---+---------------+------------------------------+ | 663 | | logical port 0| | | 664 | +---------------+ | | 665 | ^ | | 666 | | | | Host 667 | : | | 668 | +--------------+ | | 669 | | phy port | vswitch | | 670 +---+--------------+------------ -------------- ---+ __| 671 ^ 672 | 673 : 674 +--------------------------------------------------+ 675 | | 676 | traffic generator | 677 | | 678 +--------------------------------------------------+ 680 Figure 7 VNF to virtual switch to physical port 682 __ 683 +---------------------------------------------------+ | 684 | | | 685 | +-------------------------------------------+ | | 686 | | Application | | | 687 | +-------------------------------------------+ | | 688 | : | | 689 | | | | Guest 690 | v | | 691 | +---------------+ | | 692 | | logical port | | | 693 +-------------------------------+---------------+---+ __| 694 : 695 | 696 v __ 697 +------------------------------+---------------+---+ | 698 | | logical port | | | 699 | +---------------+ | | 700 | : | | 701 | | | | Host 702 | v | | 703 | +--------------+ | | 704 | vswitch | phy port | | | 705 +-------------------------------+--------------+---+ __| 706 : 707 | 708 v 709 +--------------------------------------------------+ 710 | | 711 | traffic generator | 712 | | 713 +--------------------------------------------------+ 715 Figure 8 VNF to virtual switch to VNF 717 __ 718 +----------------------+ +----------------------+ | 719 | Guest 1 | | Guest 2 | | 720 | +---------------+ | | +---------------+ | | 721 | | Application | | | | Application | | | 722 | +---------------+ | | +---------------+ | | 723 | | | | ^ | | 724 | v | | | | | Guests 725 | +---------------+ | | +---------------+ | | 726 | | logical ports | | | | logical ports | | | 727 | | 0 | | | | 0 | | | 728 +---+---------------+--+ +---+---------------+--+__| 729 : ^ 730 | | 731 v : _ 732 +---+---------------+---------+---------------+--+ | 733 | | 1 | | 1 | | | 734 | | logical ports | | logical ports | | | 735 | +---------------+ +---------------+ | | 736 | | ^ | | Host 737 | L-----------------+ | | 738 | | | 739 | vswitch | | 740 +------------------------------------------------+_| 742 A set of Deployment Scenario figures is available on the VSPERF Test 743 Methodology Wiki page [TestTopo]. 745 5. 3x3 Matrix Coverage 747 This section organizes the many existing test specifications into the 748 "3x3" matrix (introduced in [I-D.ietf-bmwg-virtual-net]). Because 749 the LTD specification ID names are quite long, this section is 750 organized into lists for each occupied cell of the matrix (not all 751 are occupied, also the matrix has grown to 3x4 to accommodate scale 752 metrics when displaying the coverage of many metrics/benchmarks). 753 The current version of the LTD specification is available [LTD]. 755 The tests listed below assess the activation of paths in the data 756 plane, rather than the control plane. 758 A complete list of tests with short summaries is available on the 759 VSPERF "LTD Test Spec Overview" Wiki page [LTDoverV]. 761 5.1. Speed of Activation 763 o Activation.RFC2889.AddressLearningRate 765 o PacketLatency.InitialPacketProcessingLatency 767 5.2. Accuracy of Activation section 769 o CPDP.Coupling.Flow.Addition 771 5.3. Reliability of Activation 773 o Throughput.RFC2544.SystemRecoveryTime 775 o Throughput.RFC2544.ResetTime 777 5.4. Scale of Activation 779 o Activation.RFC2889.AddressCachingCapacity 781 5.5. Speed of Operation 783 o Throughput.RFC2544.PacketLossRate 785 o Stress.RFC2544.0PacketLoss 787 o Throughput.RFC2544.PacketLossRateFrameModification 789 o Throughput.RFC2544.BackToBackFrames 791 o Throughput.RFC2889.MaxForwardingRate 793 o Throughput.RFC2889.ForwardPressure 795 o Throughput.RFC2889.BroadcastFrameForwarding 797 o Throughput.RFC2544.WorstN-BestN 799 o Throughput.Overlay.Network..RFC2544.PacketLossRatio 801 5.6. Accuracy of Operation 803 o Throughput.RFC2889.ErrorFramesFiltering 805 o Throughput.RFC2544.Profile 807 5.7. Reliability of Operation 809 o Throughput.RFC2889.Soak 811 o Throughput.RFC2889.SoakFrameModification 813 o PacketDelayVariation.RFC3393.Soak 815 5.8. Scalability of Operation 817 o Scalability.RFC2544.0PacketLoss 819 o MemoryBandwidth.RFC2544.0PacketLoss.Scalability 821 o Scalability.VNF.RFC2544.PacketLossProfile 823 o Scalability.VNF.RFC2544.PacketLossRatio 825 5.9. Summary 827 |------------------------------------------------------------------------| 828 | | | | | | 829 | | SPEED | ACCURACY | RELIABILITY | SCALE | 830 | | | | | | 831 |------------------------------------------------------------------------| 832 | | | | | | 833 | Activation | X | X | X | X | 834 | | | | | | 835 |------------------------------------------------------------------------| 836 | | | | | | 837 | Operation | X | X | X | X | 838 | | | | | | 839 |------------------------------------------------------------------------| 840 | | | | | | 841 | De-activation | | | | | 842 | | | | | | 843 |------------------------------------------------------------------------| 845 6. Security Considerations 847 Benchmarking activities as described in this memo are limited to 848 technology characterization of a Device Under Test/System Under Test 849 (DUT/SUT) using controlled stimuli in a laboratory environment, with 850 dedicated address space and the constraints specified in the sections 851 above. 853 The benchmarking network topology will be an independent test setup 854 and MUST NOT be connected to devices that may forward the test 855 traffic into a production network, or misroute traffic to the test 856 management network. 858 Further, benchmarking is performed on a "black-box" basis, relying 859 solely on measurements observable external to the DUT/SUT. 861 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 862 benchmarking purposes. Any implications for network security arising 863 from the DUT/SUT SHOULD be identical in the lab and in production 864 networks. 866 7. IANA Considerations 868 No IANA Action is requested at this time. 870 8. Acknowledgements 872 The authors appreciate and acknowledge comments from Scott Bradner, 873 Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, 874 Christian Trautman, and others for their reviews. 876 We also acknowledge the early work in 877 [I-D.huang-bmwg-virtual-network-performance], and useful discussion 878 with the authors. 880 9. References 882 9.1. Normative References 884 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 885 Requirement Levels", BCP 14, RFC 2119, 886 DOI 10.17487/RFC2119, March 1997, 887 . 889 [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN 890 Switching Devices", RFC 2285, DOI 10.17487/RFC2285, 891 February 1998, . 893 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 894 Network Interconnect Devices", RFC 2544, 895 DOI 10.17487/RFC2544, March 1999, 896 . 898 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 899 Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679, 900 September 1999, . 902 [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology 903 for LAN Switching Devices", RFC 2889, 904 DOI 10.17487/RFC2889, August 2000, 905 . 907 [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast 908 Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October 909 2004, . 911 [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, 912 S., and J. Perser, "Packet Reordering Metrics", RFC 4737, 913 DOI 10.17487/RFC4737, November 2006, 914 . 916 [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera, 917 "Device Reset Characterization", RFC 6201, 918 DOI 10.17487/RFC6201, March 2011, 919 . 921 [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet 922 Sizes for Additional Testing", RFC 6985, 923 DOI 10.17487/RFC6985, July 2013, 924 . 926 9.2. Informative References 928 [DanubeRel] 929 "Danube, Fourth OPNFV Release 930 https://wiki.opnfv.org/display/SWREL/Danube". 932 [I-D.huang-bmwg-virtual-network-performance] 933 Huang, L., Rong, G., Mandeville, B., and B. Hickman, 934 "Benchmarking Methodology for Virtualization Network 935 Performance", draft-huang-bmwg-virtual-network- 936 performance-02 (work in progress), March 2017. 938 [I-D.ietf-bmwg-virtual-net] 939 Morton, A., "Considerations for Benchmarking Virtual 940 Network Functions and Their Infrastructure", draft-ietf- 941 bmwg-virtual-net-05 (work in progress), March 2017. 943 [IEEE802.1ac] 944 https://standards.ieee.org/findstds/standard/802.1AC- 945 2016.html, "802.1AC-2016 - IEEE Standard for Local and 946 metropolitan area networks -- Media Access Control (MAC) 947 Service Definition", 2016. 949 [IFA003] "https://docbox.etsi.org/ISG/NFV/Open/Drafts/ 950 IFA003_Acceleration_-_vSwitch_Spec/". 952 [LTD] Note: if the Danube Release LTD is available in artifacts 953 at publication, we will use that URL instead., "LTD Test S 954 pecificationhttp://artifacts.opnfv.org/vswitchperf/colorad 955 o/docs/requirements/vswitchperf_ltd.html". 957 [LTDoverV] 958 "LTD Test Spec Overview 959 https://wiki.opnfv.org/display/vsperf/ 960 LTD+Test+Spec+Overview". 962 [OPNFV] "OPNFV Home https://www.opnfv.org/". 964 [RFC1242] Bradner, S., "Benchmarking Terminology for Network 965 Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, 966 July 1991, . 968 [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation 969 Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, 970 March 2009, . 972 [TestTopo] 973 "Test Topologies https://wiki.opnfv.org/display/vsperf/ 974 Test+Methodology". 976 [VSPERFhome] 977 "VSPERF Home https://wiki.opnfv.org/display/vsperf/ 978 VSperf+Home". 980 Authors' Addresses 982 Maryam Tahhan 983 Intel 985 Email: maryam.tahhan@intel.com 987 Billy O'Mahony 988 Intel 990 Email: billy.o.mahony@intel.com 991 Al Morton 992 AT&T Labs 993 200 Laurel Avenue South 994 Middletown,, NJ 07748 995 USA 997 Phone: +1 732 420 1571 998 Fax: +1 732 368 1192 999 Email: acmorton@att.com 1000 URI: http://home.comcast.net/~acmacm/