idnits 2.17.1 draft-ietf-bmwg-vswitch-opnfv-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 17 instances of too long lines in the document, the longest one being 2 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 14, 2016) is 2751 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2330' is defined on line 817, but no explicit reference was found in the text == Unused Reference: 'RFC2680' is defined on line 831, but no explicit reference was found in the text == Unused Reference: 'RFC2681' is defined on line 836, but no explicit reference was found in the text == Unused Reference: 'RFC3393' is defined on line 845, but no explicit reference was found in the text == Unused Reference: 'RFC3432' is defined on line 850, but no explicit reference was found in the text == Unused Reference: 'RFC4689' is defined on line 859, but no explicit reference was found in the text == Unused Reference: 'RFC5357' is defined on line 869, but no explicit reference was found in the text == Unused Reference: 'RFC5905' is defined on line 874, but no explicit reference was found in the text == Unused Reference: 'RFC6049' is defined on line 919, but no explicit reference was found in the text == Unused Reference: 'RFC6248' is defined on line 923, but no explicit reference was found in the text == Unused Reference: 'RFC6390' is defined on line 928, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) == Outdated reference: A later version (-03) exists of draft-huang-bmwg-virtual-network-performance-01 == Outdated reference: A later version (-05) exists of draft-ietf-bmwg-virtual-net-04 Summary: 3 errors (**), 0 flaws (~~), 14 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group M. Tahhan 3 Internet-Draft B. O'Mahony 4 Intended status: Informational Intel 5 Expires: April 17, 2017 A. Morton 6 AT&T Labs 7 October 14, 2016 9 Benchmarking Virtual Switches in OPNFV 10 draft-ietf-bmwg-vswitch-opnfv-01 12 Abstract 14 This memo describes the progress of the Open Platform for NFV (OPNFV) 15 project on virtual switch performance "VSWITCHPERF". This project 16 intends to build on the current and completed work of the 17 Benchmarking Methodology Working Group in IETF, by referencing 18 existing literature. The Benchmarking Methodology Working Group has 19 traditionally conducted laboratory characterization of dedicated 20 physical implementations of internetworking functions. Therefore, 21 this memo begins to describe the additional considerations when 22 virtual switches are implemented in general-purpose hardware. The 23 expanded tests and benchmarks are also influenced by the OPNFV 24 mission to support virtualization of the "telco" infrastructure. 26 Requirements Language 28 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 29 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 30 document are to be interpreted as described in RFC 2119 [RFC2119]. 32 Status of This Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at http://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on April 17, 2017. 49 Copyright Notice 51 Copyright (c) 2016 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents 56 (http://trustee.ietf.org/license-info) in effect on the date of 57 publication of this document. Please review these documents 58 carefully, as they describe your rights and restrictions with respect 59 to this document. Code Components extracted from this document must 60 include Simplified BSD License text as described in Section 4.e of 61 the Trust Legal Provisions and are provided without warranty as 62 described in the Simplified BSD License. 64 Table of Contents 66 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 67 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 68 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 4 69 3.1. Comparison with Physical Network Functions . . . . . . . 4 70 3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 4 71 3.3. New Configuration Parameters . . . . . . . . . . . . . . 4 72 3.4. Flow classification . . . . . . . . . . . . . . . . . . . 6 73 3.5. Benchmarks using Baselines with Resource Isolation . . . 7 74 4. VSWITCHPERF Specification Summary . . . . . . . . . . . . . . 8 75 5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 16 76 5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 17 77 5.2. Accuracy of Activation section . . . . . . . . . . . . . 17 78 5.3. Reliability of Activation . . . . . . . . . . . . . . . . 17 79 5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 17 80 5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 17 81 5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 17 82 5.7. Reliability of Operation . . . . . . . . . . . . . . . . 17 83 5.8. Scalability of Operation . . . . . . . . . . . . . . . . 18 84 5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 18 85 6. Security Considerations . . . . . . . . . . . . . . . . . . . 18 86 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 87 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 88 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 89 9.1. Normative References . . . . . . . . . . . . . . . . . . 19 90 9.2. Informative References . . . . . . . . . . . . . . . . . 21 91 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 93 1. Introduction 95 Benchmarking Methodology Working Group (BMWG) has traditionally 96 conducted laboratory characterization of dedicated physical 97 implementations of internetworking functions. The Black-box 98 Benchmarks of Throughput, Latency, Forwarding Rates and others have 99 served our industry for many years. Now, Network Function 100 Virtualization (NFV) has the goal to transform how internetwork 101 functions are implemented, and therefore has garnered much attention. 103 This memo summarizes the progress of the Open Platform for NFV 104 (OPNFV) project on virtual switch performance characterization, 105 "VSWITCHPERF", through the Brahmaputra (second) release [BrahRel]. 106 This project intends to build on the current and completed work of 107 the Benchmarking Methodology Working Group in IETF, by referencing 108 existing literature. For example, currently the most often 109 referenced RFC is [RFC2544] (which depends on [RFC1242]) and 110 foundation of the benchmarking work in OPNFV is common and strong. 112 See https://wiki.opnfv.org/ 113 characterize_vswitch_performance_for_telco_nfv_use_cases for more 114 background, and the OPNFV website for general information: 115 https://www.opnfv.org/ 117 The authors note that OPNFV distinguishes itself from other open 118 source compute and networking projects through its emphasis on 119 existing "telco" services as opposed to cloud-computing. There are 120 many ways in which telco requirements have different emphasis on 121 performance dimensions when compared to cloud computing: support for 122 and transfer of isochronous media streams is one example. 124 Note also that the move to NFV Infrastructure has resulted in many 125 new benchmarking initiatives across the industry. The authors are 126 currently doing their best to maintain alignment with many other 127 projects, and this Internet Draft is one part of the efforts. We 128 acknowledge the early work in 129 [I-D.huang-bmwg-virtual-network-performance], and useful discussion 130 with the authors. 132 2. Scope 134 The primary purpose and scope of the memo is to inform the industry 135 of work-in-progress that builds on the body of extensive BMWG 136 literature and experience, and describe the extensions needed for 137 benchmarking virtual switches. Inital feedback indicates that many 138 of these extensions may be applicable beyond the current scope (to 139 hardware switches in the NFV Infrastructure and to virtual routers, 140 for example). Additionally, this memo serves as a vehicle to include 141 more detail and commentary from BMWG and other Open Source 142 communities, under BMWG's chartered work to characterize the NFV 143 Infrastructure (a virtual switch is an important aspect of that 144 infrastructure). 146 The benchmarking covered in this memo should be applicable to many 147 types of vswitches, and remain vswitch-agnostic to great degree. 148 There has been no attempt to track and test all features of any 149 specific vswitch implementation. 151 3. Benchmarking Considerations 153 This section highlights some specific considerations (from 154 [I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual 155 switches. The OPNFV project is sharing its present view on these 156 areas, as they develop their specifications in the Level Test Design 157 (LTD) document. 159 3.1. Comparison with Physical Network Functions 161 To compare the performance of virtual designs and implementations 162 with their physical counterparts, identical benchmarks are needed. 163 BMWG has developed specifications for many network functions this 164 memo re-uses existing benchmarks through references, and expands them 165 during development of new methods. A key configuration aspect is the 166 number of parallel cores required to achieve comparable performance 167 with a given physical device, or whether some limit of scale was 168 reached before the cores could achieve the comparable level. 170 It's unlikely that the virtual switch will be the only application 171 running on the SUT, so CPU utilization, Cache utilization, and Memory 172 footprint should also be recorded for the virtual implementations of 173 internetworking functions. 175 3.2. Continued Emphasis on Black-Box Benchmarks 177 External observations remain essential as the basis for Benchmarks. 178 Internal observations with fixed specification and interpretation 179 will be provided in parallel to assist the development of operations 180 procedures when the technology is deployed. 182 3.3. New Configuration Parameters 184 A key consideration when conducting any sort of benchmark is trying 185 to ensure the consistency and repeatability of test results. When 186 benchmarking the performance of a vSwitch there are many factors that 187 can affect the consistency of results, one key factor is matching the 188 various hardware and software details of the SUT. This section lists 189 some of the many new parameters which this project believes are 190 critical to report in order to achieve repeatability. 192 Hardware details including: 194 o Platform details 196 o Processor details 198 o Memory information (type and size) 200 o Number of enabled cores 202 o Number of cores used for the test 204 o Number of physical NICs, as well as their details (manufacturer, 205 versions, type and the PCI slot they are plugged into) 207 o NIC interrupt configuration 209 o BIOS version, release date and any configurations that were 210 modified 212 o CPU microcode level 214 o Memory DIMM configurations (quad rank performance may not be the 215 same as dual rank) in size, freq and slot locations 217 o PCI configuration parameters (payload size, early ack option...) 219 o Power management at all levels (ACPI sleep states, processor 220 package, OS...) 222 Software details including: 224 o OS parameters and behavior (text vs graphical no one typing at the 225 console on one system) 227 o OS version (for host and VNF) 229 o Kernel version (for host and VNF) 231 o GRUB boot parameters (for host and VNF) 233 o Hypervisor details (Type and version) 235 o Selected vSwitch, version number or commit id used 236 o vSwitch launch command line if it has been parameterised 238 o Memory allocation to the vSwitch 240 o which NUMA node it is using, and how many memory channels 242 o DPDK or any other SW dependency version number or commit id used 244 o Memory allocation to a VM - if it's from Hugpages/elsewhere 246 o VM storage type: snapshot/independent persistent/independent non- 247 persistent 249 o Number of VMs 251 o Number of Virtual NICs (vNICs), versions, type and driver 253 o Number of virtual CPUs and their core affinity on the host 255 o Number vNIC interrupt configuration 257 o Thread affinitization for the applications (including the vSwitch 258 itself) on the host 260 o Details of Resource isolation, such as CPUs designated for Host/ 261 Kernel (isolcpu) and CPUs designated for specific processes 262 (taskset). - Test duration. - Number of flows. 264 Test Traffic Information: 266 o Traffic type - UDP, TCP, IMIX / Other 268 o Packet Sizes 270 o Deployment Scenario 272 3.4. Flow classification 274 Virtual switches group packets into flows by processing and matching 275 particular packet or frame header information, or by matching packets 276 based on the input ports. Thus a flow can be thought of a sequence 277 of packets that have the same set of header field values (5-tuple) or 278 have arrived on the same port. Performance results can vary based on 279 the parameters the vSwitch uses to match for a flow. The recommended 280 flow classification parameters for any vSwitch performance tests are: 281 the input port, the source IP address, the destination IP address and 282 the Ethernet protocol type field. It is essential to increase the 283 flow timeout time on a vSwitch before conducting any performance 284 tests that do not measure the flow setup time. Normally the first 285 packet of a particular stream will install the flow in the virtual 286 switch which adds an additional latency, subsequent packets of the 287 same flow are not subject to this latency if the flow is already 288 installed on the vSwitch. 290 3.5. Benchmarks using Baselines with Resource Isolation 292 This outline describes measurement of baseline with isolated 293 resources at a high level, which is the intended approach at this 294 time. 296 1. Baselines: 298 * Optional: Benchmark platform forwarding capability without a 299 vswitch or VNF for at least 72 hours (serves as a means of 300 platform validation and a means to obtain the base performance 301 for the platform in terms of its maximum forwarding rate and 302 latency). 304 Benchmark platform forwarding capability 306 __ 307 +--------------------------------------------------+ | 308 | +------------------------------------------+ | | 309 | | | | | 310 | | Simple Forwarding App | | Host 311 | | | | | 312 | +------------------------------------------+ | | 313 | | NIC | | | 314 +---+------------------------------------------+---+ __| 315 ^ : 316 | | 317 : v 318 +--------------------------------------------------+ 319 | | 320 | traffic generator | 321 | | 322 +--------------------------------------------------+ 324 * Benchmark VNF forwarding capability with direct connectivity 325 (vSwitch bypass, e.g., SR/IOV) for at least 72 hours (serves 326 as a means of VNF validation and a means to obtain the base 327 performance for the VNF in terms of its maximum forwarding 328 rate and latency). The metrics gathered from this test will 329 serve as a key comparison point for vSwitch bypass 330 technologies performance and vSwitch performance. 332 Benchmark VNF forwarding capability 334 __ 335 +--------------------------------------------------+ | 336 | +------------------------------------------+ | | 337 | | | | | 338 | | VNF | | | 339 | | | | | 340 | +------------------------------------------+ | | 341 | | Passthrough/SR-IOV | | Host 342 | +------------------------------------------+ | | 343 | | NIC | | | 344 +---+------------------------------------------+---+ __| 345 ^ : 346 | | 347 : v 348 +--------------------------------------------------+ 349 | | 350 | traffic generator | 351 | | 352 +--------------------------------------------------+ 354 * Benchmarking with isolated resources alone, with other 355 resources (both HW&SW) disabled Example, vSw and VM are SUT 357 * Benchmarking with isolated resources alone, leaving some 358 resources unused 360 * Benchmark with isolated resources and all resources occupied 362 2. Next Steps 364 * Limited sharing 366 * Production scenarios 368 * Stressful scenarios 370 4. VSWITCHPERF Specification Summary 372 The overall specification in preparation is referred to as a Level 373 Test Design (LTD) document, which will contain a suite of performance 374 tests. The base performance tests in the LTD are based on the pre- 375 existing specifications developed by BMWG to test the performance of 376 physical switches. These specifications include: 378 o [RFC2544] Benchmarking Methodology for Network Interconnect 379 Devices 381 o [RFC2889] Benchmarking Methodology for LAN Switching 383 o [RFC6201] Device Reset Characterization 385 o [RFC5481] Packet Delay Variation Applicability Statement 387 Some of the above/newer RFCs are being applied in benchmarking for 388 the first time, and represent a development challenge for test 389 equipment developers. Fortunately, many members of the testing 390 system community have engaged on the VSPERF project, including an 391 open source test system. 393 In addition to this, the LTD also re-uses the terminology defined by: 395 o [RFC2285] Benchmarking Terminology for LAN Switching Devices 397 o [RFC5481] Packet Delay Variation Applicability Statement 399 Specifications to be included in future updates of the LTD include: 401 o [RFC3918] Methodology for IP Multicast Benchmarking 403 o [RFC4737] Packet Reordering Metrics 405 As one might expect, the most fundamental internetworking 406 characteristics of Throughput and Latency remain important when the 407 switch is virtualized, and these benchmarks figure prominently in the 408 specification. 410 When considering characteristics important to "telco" network 411 functions, we must begin to consider additional performance metrics. 412 In this case, the project specifications have referenced metrics from 413 the IETF IP Performance Metrics (IPPM) literature. This means that 414 the [RFC2544] test of Latency is replaced by measurement of a metric 415 derived from IPPM's [RFC2679], where a set of statistical summaries 416 will be provided (mean, max, min, etc.). Further metrics planned to 417 be benchmarked include packet delay variation as defined by [RFC5481] 418 , reordering, burst behaviour, DUT availability, DUT capacity and 419 packet loss in long term testing at Throughput level, where some low- 420 level of background loss may be present and characterized. 422 Tests have been (or will be) designed to collect the metrics below: 424 o Throughput Tests to measure the maximum forwarding rate (in frames 425 per second or fps) and bit rate (in Mbps) for a constant load (as 426 defined by [RFC1242]) without traffic loss. 428 o Packet and Frame Delay Distribution Tests to measure average, min 429 and max packet and frame delay for constant loads. 431 o Packet Delay Tests to understand latency distribution for 432 different packet sizes and over an extended test run to uncover 433 outliers. 435 o Scalability Tests to understand how the virtual switch performs as 436 the number of flows, active ports, complexity of the forwarding 437 logic's configuration... it has to deal with increases. 439 o Stream Performance Tests (TCP, UDP) to measure bulk data transfer 440 performance, i.e. how fast systems can send and receive data 441 through the switch. 443 o Control Path and Datapath Coupling Tests, to understand how 444 closely coupled the datapath and the control path are as well as 445 the effect of this coupling on the performance of the DUT 446 (example: delay of the initial packet of a flow). 448 o CPU and Memory Consumption Tests to understand the virtual 449 switch's footprint on the system, usually conducted as auxiliary 450 measurements with benchmarks above. They include: CPU 451 utilization, Cache utilization and Memory footprint. 453 o The so-called "Soak" tests, where the selected test is conducted 454 over a long period of time (with an ideal duration of 24 hours, 455 but only long enough to determine that stability issues exist when 456 found; there is no requirement to continue a test when a DUT 457 exhibits instability over time). The key performance 458 characteristics and benchmarks for a DUT are determined (using 459 short duration tests) prior to conducting soak tests. The purpose 460 of soak tests is to capture transient changes in performance which 461 may occur due to infrequent processes, memory leaks, or the low 462 probability coincidence of two or more processes. The stability 463 of the DUT is the paramount consideration, so performance must be 464 evaluated periodically during continuous testing, and this results 465 in use of [RFC2889] Frame Rate metrics instead of [RFC2544] 466 Throughput (which requires stopping traffic to allow time for all 467 traffic to exit internal queues), for example. 469 Future/planned test specs include: 471 o Request/Response Performance Tests (TCP, UDP) which measure the 472 transaction rate through the switch. 474 o Noisy Neighbour Tests, to understand the effects of resource 475 sharing on the performance of a virtual switch. 477 o Tests derived from examination of ETSI NFV Draft GS IFA003 478 requirements [IFA003] on characterization of acceleration 479 technologies applied to vswitches. 481 The flexibility of deployment of a virtual switch within a network 482 means that the BMWG IETF existing literature needs to be used to 483 characterize the performance of a switch in various deployment 484 scenarios. The deployment scenarios under consideration include: 486 Physical port to virtual switch to physical port 488 __ 489 +--------------------------------------------------+ | 490 | +--------------------+ | | 491 | | | | | 492 | | v | | Host 493 | +--------------+ +--------------+ | | 494 | | phy port | vSwitch | phy port | | | 495 +---+--------------+------------+--------------+---+ __| 496 ^ : 497 | | 498 : v 499 +--------------------------------------------------+ 500 | | 501 | traffic generator | 502 | | 503 +--------------------------------------------------+ 505 Physical port to virtual switch to VNF to virtual switch to physical 506 port 508 __ 509 +---------------------------------------------------+ | 510 | | | 511 | +-------------------------------------------+ | | 512 | | Application | | | 513 | +-------------------------------------------+ | | 514 | ^ : | | 515 | | | | | Guest 516 | : v | | 517 | +---------------+ +---------------+ | | 518 | | logical port 0| | logical port 1| | | 519 +---+---------------+-----------+---------------+---+ __| 520 ^ : 521 | | 522 : v __ 523 +---+---------------+----------+---------------+---+ | 524 | | logical port 0| | logical port 1| | | 525 | +---------------+ +---------------+ | | 526 | ^ : | | 527 | | | | | Host 528 | : v | | 529 | +--------------+ +--------------+ | | 530 | | phy port | vSwitch | phy port | | | 531 +---+--------------+------------+--------------+---+ __| 532 ^ : 533 | | 534 : v 535 +--------------------------------------------------+ 536 | | 537 | traffic generator | 538 | | 539 +--------------------------------------------------+ 541 Physical port to virtual switch to VNF to virtual switch to VNF to 542 virtual switch to physical port 544 __ 545 +----------------------+ +----------------------+ | 546 | Guest 1 | | Guest 2 | | 547 | +---------------+ | | +---------------+ | | 548 | | Application | | | | Application | | | 549 | +---------------+ | | +---------------+ | | 550 | ^ | | | ^ | | | 551 | | v | | | v | | Guests 552 | +---------------+ | | +---------------+ | | 553 | | logical ports | | | | logical ports | | | 554 | | 0 1 | | | | 0 1 | | | 555 +---+---------------+--+ +---+---------------+--+__| 556 ^ : ^ : 557 | | | | 558 : v : v _ 559 +---+---------------+---------+---------------+--+ | 560 | | 0 1 | | 3 4 | | | 561 | | logical ports | | logical ports | | | 562 | +---------------+ +---------------+ | | 563 | ^ | ^ | | | Host 564 | | |-----------------| v | | 565 | +--------------+ +--------------+ | | 566 | | phy ports | vSwitch | phy ports | | | 567 +---+--------------+----------+--------------+---+_| 568 ^ : 569 | | 570 : v 571 +--------------------------------------------------+ 572 | | 573 | traffic generator | 574 | | 575 +--------------------------------------------------+ 577 Physical port to virtual switch to VNF 579 __ 580 +---------------------------------------------------+ | 581 | | | 582 | +-------------------------------------------+ | | 583 | | Application | | | 584 | +-------------------------------------------+ | | 585 | ^ | | 586 | | | | Guest 587 | : | | 588 | +---------------+ | | 589 | | logical port 0| | | 590 +---+---------------+-------------------------------+ __| 591 ^ 592 | 593 : __ 594 +---+---------------+------------------------------+ | 595 | | logical port 0| | | 596 | +---------------+ | | 597 | ^ | | 598 | | | | Host 599 | : | | 600 | +--------------+ | | 601 | | phy port | vSwitch | | 602 +---+--------------+------------ -------------- ---+ __| 603 ^ 604 | 605 : 606 +--------------------------------------------------+ 607 | | 608 | traffic generator | 609 | | 610 +--------------------------------------------------+ 612 VNF to virtual switch to physical port 614 __ 615 +---------------------------------------------------+ | 616 | | | 617 | +-------------------------------------------+ | | 618 | | Application | | | 619 | +-------------------------------------------+ | | 620 | : | | 621 | | | | Guest 622 | v | | 623 | +---------------+ | | 624 | | logical port | | | 625 +-------------------------------+---------------+---+ __| 626 : 627 | 628 v __ 629 +------------------------------+---------------+---+ | 630 | | logical port | | | 631 | +---------------+ | | 632 | : | | 633 | | | | Host 634 | v | | 635 | +--------------+ | | 636 | vSwitch | phy port | | | 637 +-------------------------------+--------------+---+ __| 638 : 639 | 640 v 641 +--------------------------------------------------+ 642 | | 643 | traffic generator | 644 | | 645 +--------------------------------------------------+ 647 VNF to virtual switch to VNF 649 __ 650 +----------------------+ +----------------------+ | 651 | Guest 1 | | Guest 2 | | 652 | +---------------+ | | +---------------+ | | 653 | | Application | | | | Application | | | 654 | +---------------+ | | +---------------+ | | 655 | | | | ^ | | 656 | v | | | | | Guests 657 | +---------------+ | | +---------------+ | | 658 | | logical ports | | | | logical ports | | | 659 | | 0 | | | | 0 | | | 660 +---+---------------+--+ +---+---------------+--+__| 661 : ^ 662 | | 663 v : _ 664 +---+---------------+---------+---------------+--+ | 665 | | 1 | | 1 | | | 666 | | logical ports | | logical ports | | | 667 | +---------------+ +---------------+ | | 668 | | ^ | | Host 669 | L-----------------+ | | 670 | | | 671 | vSwitch | | 672 +------------------------------------------------+_| 674 A set of Deployment Scenario figures is available on the VSPERF Test 675 Methodology Wiki page [TestTopo]. 677 5. 3x3 Matrix Coverage 679 This section organizes the many existing test specifications into the 680 "3x3" matrix (introduced in [I-D.ietf-bmwg-virtual-net]). Because 681 the LTD specification ID names are quite long, this section is 682 organized into lists for each occupied cell of the matrix (not all 683 are occupied, also the matrix has grown to 3x4 to accommodate scale 684 metrics when displaying the coverage of many metrics/benchmarks). 685 The current version of the LTD specification is available [LTD]. 687 The tests listed below assess the activation of paths in the data 688 plane, rather than the control plane. 690 A complete list of tests with short summaries is available on the 691 VSPERF "LTD Test Spec Overview" Wiki page [LTDoverV]. 693 5.1. Speed of Activation 695 o Activation.RFC2889.AddressLearningRate 697 o PacketLatency.InitialPacketProcessingLatency 699 5.2. Accuracy of Activation section 701 o CPDP.Coupling.Flow.Addition 703 5.3. Reliability of Activation 705 o Throughput.RFC2544.SystemRecoveryTime 707 o Throughput.RFC2544.ResetTime 709 5.4. Scale of Activation 711 o Activation.RFC2889.AddressCachingCapacity 713 5.5. Speed of Operation 715 o Throughput.RFC2544.PacketLossRate 717 o CPU.RFC2544.0PacketLoss 719 o Throughput.RFC2544.PacketLossRateFrameModification 721 o Throughput.RFC2544.BackToBackFrames 723 o Throughput.RFC2889.MaxForwardingRate 725 o Throughput.RFC2889.ForwardPressure 727 o Throughput.RFC2889.BroadcastFrameForwarding 729 5.6. Accuracy of Operation 731 o Throughput.RFC2889.ErrorFramesFiltering 733 o Throughput.RFC2544.Profile 735 5.7. Reliability of Operation 737 o Throughput.RFC2889.Soak 739 o Throughput.RFC2889.SoakFrameModification 740 o PacketDelayVariation.RFC3393.Soak 742 5.8. Scalability of Operation 744 o Scalability.RFC2544.0PacketLoss 746 o MemoryBandwidth.RFC2544.0PacketLoss.Scalability 748 5.9. Summary 750 |------------------------------------------------------------------------| 751 | | | | | | 752 | | SPEED | ACCURACY | RELIABILITY | SCALE | 753 | | | | | | 754 |------------------------------------------------------------------------| 755 | | | | | | 756 | Activation | X | X | X | X | 757 | | | | | | 758 |------------------------------------------------------------------------| 759 | | | | | | 760 | Operation | X | X | X | X | 761 | | | | | | 762 |------------------------------------------------------------------------| 763 | | | | | | 764 | De-activation | | | | | 765 | | | | | | 766 |------------------------------------------------------------------------| 768 6. Security Considerations 770 Benchmarking activities as described in this memo are limited to 771 technology characterization of a Device Under Test/System Under Test 772 (DUT/SUT) using controlled stimuli in a laboratory environment, with 773 dedicated address space and the constraints specified in the sections 774 above. 776 The benchmarking network topology will be an independent test setup 777 and MUST NOT be connected to devices that may forward the test 778 traffic into a production network, or misroute traffic to the test 779 management network. 781 Further, benchmarking is performed on a "black-box" basis, relying 782 solely on measurements observable external to the DUT/SUT. 784 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 785 benchmarking purposes. Any implications for network security arising 786 from the DUT/SUT SHOULD be identical in the lab and in production 787 networks. 789 7. IANA Considerations 791 No IANA Action is requested at this time. 793 8. Acknowledgements 795 The authors appreciate and acknowledge comments from Scott Bradner, 796 Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, 797 Christian Trautman, and others for their reviews. 799 9. References 801 9.1. Normative References 803 [NFV.PER001] 804 "Network Function Virtualization: Performance and 805 Portability Best Practices", Group Specification ETSI GS 806 NFV-PER 001 V1.1.1 (2014-06), June 2014. 808 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 809 Requirement Levels", BCP 14, RFC 2119, 810 DOI 10.17487/RFC2119, March 1997, 811 . 813 [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN 814 Switching Devices", RFC 2285, DOI 10.17487/RFC2285, 815 February 1998, . 817 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 818 "Framework for IP Performance Metrics", RFC 2330, 819 DOI 10.17487/RFC2330, May 1998, 820 . 822 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 823 Network Interconnect Devices", RFC 2544, 824 DOI 10.17487/RFC2544, March 1999, 825 . 827 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 828 Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679, 829 September 1999, . 831 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 832 Packet Loss Metric for IPPM", RFC 2680, 833 DOI 10.17487/RFC2680, September 1999, 834 . 836 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 837 Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, 838 September 1999, . 840 [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology 841 for LAN Switching Devices", RFC 2889, 842 DOI 10.17487/RFC2889, August 2000, 843 . 845 [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation 846 Metric for IP Performance Metrics (IPPM)", RFC 3393, 847 DOI 10.17487/RFC3393, November 2002, 848 . 850 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 851 performance measurement with periodic streams", RFC 3432, 852 DOI 10.17487/RFC3432, November 2002, 853 . 855 [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast 856 Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October 857 2004, . 859 [RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, 860 "Terminology for Benchmarking Network-layer Traffic 861 Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689, 862 October 2006, . 864 [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, 865 S., and J. Perser, "Packet Reordering Metrics", RFC 4737, 866 DOI 10.17487/RFC4737, November 2006, 867 . 869 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 870 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 871 RFC 5357, DOI 10.17487/RFC5357, October 2008, 872 . 874 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 875 "Network Time Protocol Version 4: Protocol and Algorithms 876 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 877 . 879 [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera, 880 "Device Reset Characterization", RFC 6201, 881 DOI 10.17487/RFC6201, March 2011, 882 . 884 9.2. Informative References 886 [BrahRel] "Brahmaputra, Second OPNFV Release https://www.opnfv.org/ 887 brahmaputra". 889 [I-D.huang-bmwg-virtual-network-performance] 890 Huang, L., Rong, G., Mandeville, B., and B. Hickman, 891 "Benchmarking Methodology for Virtualization Network 892 Performance", draft-huang-bmwg-virtual-network- 893 performance-01 (work in progress), April 2015. 895 [I-D.ietf-bmwg-virtual-net] 896 Morton, A., "Considerations for Benchmarking Virtual 897 Network Functions and Their Infrastructure", draft-ietf- 898 bmwg-virtual-net-04 (work in progress), August 2016. 900 [IFA003] "https://docbox.etsi.org/ISG/NFV/Open/Drafts/ 901 IFA003_Acceleration_-_vSwitch_Spec/". 903 [LTD] "LTD Test Specification 904 http://artifacts.opnfv.org/vswitchperf/brahmaputra/docs/ 905 requirements/index.html". 907 [LTDoverV] 908 "LTD Test Spec Overview https://wiki.opnfv.org/wiki/ 909 vswitchperf_test_spec_review". 911 [RFC1242] Bradner, S., "Benchmarking Terminology for Network 912 Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, 913 July 1991, . 915 [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation 916 Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, 917 March 2009, . 919 [RFC6049] Morton, A. and E. Stephan, "Spatial Composition of 920 Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011, 921 . 923 [RFC6248] Morton, A., "RFC 4148 and the IP Performance Metrics 924 (IPPM) Registry of Metrics Are Obsolete", RFC 6248, 925 DOI 10.17487/RFC6248, April 2011, 926 . 928 [RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New 929 Performance Metric Development", BCP 170, RFC 6390, 930 DOI 10.17487/RFC6390, October 2011, 931 . 933 [TestTopo] 934 "Test Topologies https://wiki.opnfv.org/vsperf/ 935 test_methodology". 937 Authors' Addresses 939 Maryam Tahhan 940 Intel 942 Email: maryam.tahhan@intel.com 944 Billy O'Mahony 945 Intel 947 Email: billy.o.mahony@intel.com 949 Al Morton 950 AT&T Labs 951 200 Laurel Avenue South 952 Middletown,, NJ 07748 953 USA 955 Phone: +1 732 420 1571 956 Fax: +1 732 368 1192 957 Email: acmorton@att.com 958 URI: http://home.comcast.net/~acmacm/