Network Working Group Vic Liu Internet Draft Dapeng Liu Intended status: Informational China Mobile Bob Mandeville Iometrix Brooks Hickman Spirent Communications Guang Zhang IXIA Expires: January 2015 July 4, 2014 Benchmarking Methodology for Virtualization Network Performance draft-liu-bmwg-virtual-network-benchmark-00.txt Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. This document may not be modified, and derivative works of it may not be created, and it may not be published except as an Internet-Draft. This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. This document may not be modified, and derivative works of it may not be created, except to publish it as an RFC and to translate it into languages other than English. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that Expires January4,2015 [Page 1] Internet-Draft virtual network performance Benchmark July 2014 other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on January4,2009. Copyright Notice Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info)in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Abstract As the virtual network have been wide establish in IDC. The performance of virtual network has become a consideration to the IDC managers. This draft introduce a benchmarking methodology for virtualization network performance. Table of Contents 1. Introduction ................................................ 3 Vic Liu Expires January4,2015 [Page 2] Internet-Draft virtual network performance Benchmark July 2014 2. Peculiar issues ............................................. 3 3. Key Performance Index........................................ 6 4. Test Setup .................................................. 6 5. Proposed Benchmark Tests..................................... 9 5.1. Throughput ............................................. 9 5.2. CPU consumption........................................ 10 5.3. Memory consumption..................................... 12 5.4. Latency ............................................... 13 6. Formal Syntax .............................................. 14 7. Security Considerations..................................... 15 8. IANA Considerations ........................................ 15 9. Conclusions ................................................ 15 10. References ................................................ 15 10.1. Normative References.................................. 15 10.2. Informative References................................ 15 11. Acknowledgments ........................................... 15 1. Introduction As the virtual network have been wide establish in IDC. The performance of virtual network has become a consideration to the IDC managers. This draft introduce a benchmarking methodology for virtualization network performance. 2. Peculiar issues In a conventional test setup with real test ports, it is quite legitimate to assume test ports provide the golden standard and in measuring the performance metrics. If and when test results are sub optimal, it is automatically assumed it's the Device-Under-Test Vic Liu Expires January4,2015 [Page 3] Internet-Draft virtual network performance Benchmark July 2014 (DUT) that is at fault. For example, when testing the max no-drop throughput at a given frame size, if the test result shows less than 100% throughput, we can safely conclude that it's the DUT that can't deliver line rate forwarding at that frame size(s). We never doubt that tester will be an issue. In a virtual test environment where both the DUT as well as the test tool itself are VM based, it's quite a different story. Just like the DUT, tester in VM shape will have its own performance peak under various conditions. Even worse, conditions are multitude and in many forms. Tester's calibration without DUT is essential in benchmarking testing in a virtual environment. Furthermore, to reduce the enormous combination of various conditions, tester must be calibrated with the exact same combination and parameter set the user wants to measure against a real DUT. A slight variation of conditions and parameter value will cause inaccurate measurements of the DUT. While the exact combination and parameter set are hard to list, below table attempts to give a most common example how to calibrate a tester before testing a real DUT under the same condition. Sample calibration permutation ---------------------------------------------------------------- | Hypervisor | VM VNIC | VM Memory | Packet | No Drop | | Type | Speed |CPU Allocation | Size | Throughput| ---------------------------------------------------------------- | ESXi 1G/10G 512M/1Core | 64 | | | | 128 | | | | 256 | | | | 512 | | | | 1024 | | Vic Liu Expires January4,2015 [Page 4] Internet-Draft virtual network performance Benchmark July 2014 | | 1518 | | ---------------------------------------------------------------- Key points are as following: a)The hypervisor type is of ultimate importance to the test results. Tester VM must be installed on the same hypervisor type as the real DUT. When feasible, tester VM software should be installed on a separate, but identical type of, hypervisor. b)The VNIC speed will have impact on test results. Tester must calibrate against all VNIC speeds to be tested against a real DUT. c)VM allocation of CPU resources and memory will affect test results d)Packet sizes will affect test results dramatically due to the nature of virtual machine e)Other possible extensions of above table: The number of VMs to be created, latency and/or jitter readings, one VNIC per VM vs. multiple VM sharing one VNIC, and uni-directional traffic vs. bi- directional traffic. It's important to have a test environment for tester's calibration as close as possible to when a real DUT will be involved for the benchmark test. Above test setup illustrate below key points: 1.One or more tester's VM need to be created for both traffic generation and analysis 2.vSwitch is need to accommodate performance penalty due to extra VM addition 3.VNIC and its type is needed in the test setup to once again accommodate any performance penalty when real DUT VM is created 4.ToR switch is needed to accommodate delays introduced by the external device In summary, calibration should be done in such an environment that other than the DUT VM, all possible factors that may negatively impact test results should be accommodated. Vic Liu Expires January4,2015 [Page 5] Internet-Draft virtual network performance Benchmark July 2014 3. Key Performance Index We listed numbers of key performance index for virtual network as follows: a) No drop throughput under various frame sizes: Forwarding performance under various frame sizes is a key performance index of interest. Once this performance number is obtained, vendors can always allocate more CPU and memory for mission critical applications where line rate performance is expected. b) DUT consumption of CPU and memory: when adding one or more VM. With addition of each VM, DUT will consume more CPU and memory. c) Latency readings: Some applications are highly sensitive on latency.It's important to get the latency reading with respective to various conditions. VxLAN performance can be looked at from two perspectives. First, addition of VxLAN on an existing VM will consume extra CPU resources and memory. This can be easily included in the benchmark table. Tester VM are strictly traffic generator and analyzer. No calibration is needed when adding VxLAN to DUT VM. Once basic performance metric is obtained with respective to single VxLAN, we need to look at performance metrics with many VM and VxLAN. The idea is verify how many VM/VxLAN can be created and what their forwarding performance (no drop throughput), latency, and CPU/memory consumptions are. 4. Test Setup The test bed is constituted by two physical server with 10GE NIC, a test center, a 10GE TOR switch for test traffic and a 1GE TOR switch for management. Vic Liu Expires January4,2015 [Page 6] Internet-Draft virtual network performance Benchmark July 2014 ---------------------- |Test Center PHY 10GE*2| ---------------------- || || ---------- =====| 10GE TOR |======= || ---------- || || || || || ------------------- ------------------- | -------------- | | -------------- | | |V-switch(VTEP)| | | |V-switch(VTEP)| | | -------------- | | -------------- | | | | | | | | | | ----- ----- | | ----- ----- | | |TCVM1| |TCVM2|| | |TCVM1| |TCVM2|| | ----- ----- | | ----- ----- | ------------------- ------------------- Server1 Server2 Vic Liu Expires January4,2015 [Page 7] Internet-Draft virtual network performance Benchmark July 2014 Two Dell server are R710XD (CPU: E5-2460) and R710 (CPU: E5-2430) with a pair of 10GE NIC. And in the server we allocate 2 vCPU and 8G memory to each Test Center Virtual Machine (TCVM). In traffic model A: We use a physical test center connect to each server to verify the benchmark of each server. ---------------------- |Test Center PHY 10GE*2| ---------------------- || || ------------------- | -------------- | | |V-switch(VTEP)| | | -------------- | | | | | | ----- ----- | | |TCVM1| |TCVM2|| | ----- ----- | ------------------- Server1 In traffic model B: We use the benchmark to test the performance of VxLAN. ---------- =====| 10GE TOR |======= || ---------- || || || || || ------------------- ------------------- | -------------- | | -------------- | | |V-switch(VTEP)| | | |V-switch(VTEP)| | | -------------- | | -------------- | | | | | | | | | | ----- ----- | | ----- ----- | | |TCVM1| |TCVM2|| | |TCVM1| |TCVM2|| | ----- ----- | | ----- ----- | ------------------- ------------------- Server1 Server2 Vic Liu Expires January4,2015 [Page 8] Internet-Draft virtual network performance Benchmark July 2014 5. Proposed Benchmark Tests 5.1. Throughput Unlike the traditional test cases which the DUT and the tester are separated, it has brought unparalleled challenges to virtual network test. In that case, the tester and the DUT (visual switches) are in one server (physically converged), so the CPU and MEM share the same resources. Theoretically, the tester's operation may has some influences on the DUT's performances. However, for the specialty of virtualization, this method is the only way to assess the truth of assessment method. Under the background of existing technology, when we mean to test the virtual switch's throughput, the concept of traditional physical switch will not be applicable. The traditional throughput indicates the switches' largest transmit capability, for certain selected bytes and selected cycle under zero-packet-lose conditions. But in virtual environment, the fluctuant of performance on virtual network will be much greater than dedicated physical devices. In the same time, because the DUT and the tester cannot be separated, which only proved the DUT realize same network performances under certain circumstances, it also means the DUT may achieve higher capability. Therefore, we change the throughout in virtual environment to actual throughput, hoping in future, as the improvement of technique, the actual throughput will approach the theoretical throughput gradually. Of course, under actual condition, this throughout have certain referential meanings. In most cases, common throughput application cannot compare with professional tester, so for virtual application and data center's deployment, the actual throughout already have great refinance value. 5.1.1. Objectives Under the condition of certain hardware configuration, test the DUT(virtual switch) can support maximum throughput. 5.1.2. Configuration parameters Network parameters should be define as follows: Vic Liu Expires January4,2015 [Page 9] Internet-Draft virtual network performance Benchmark July 2014 a) the number of virtual tester (VMs) b) the number of vNIC of virtual tester c) the CPU type of the server d) vCPU allocated for virtual tester (VMs) e) memory allocated for virtual tester (VMs) f) the number and rate of server NIC 5.1.3. The test parameters a) test repeated times b) test packet length 5.1.4. Testing process 1. Configure the virtual tester to output traffic through V-Switch. 2. Increase the number of vCPU in the tester until the traffic has no packet loss. 3. Record the max throughput on VSwitch 4. Change the packet length and repeat step1 and record test results. 5.1.5. Test results formats ---------------------- | Byte| Throughput (GE)| ---------------------- | 0 | 0 | ---------------------- | 128 | 0.46 | ---------------------- | 256 | 0.84 | ---------------------- | 512 | 1.56 | ---------------------- | 1024| 2.88 | ---------------------- | 1518| 4.00 | ---------------------- 5.2. CPU consumption The operation of DUT (VSwitch) can increase the CPU load of host server. Different V-Switches have different CPU occupation. This can Vic Liu Expires January4,2015 [Page 10] Internet-Draft virtual network performance Benchmark July 2014 be an important indicator in benchmark the Virtual network performance. 5.2.1. Objectives The objectives of this test is verified the CPU consumption caused by the DUT (VSwitch). 5.2.2. Configuration parameters Network parameters should be define as follows: a)The number of virtual tester (VMs) b)The number of vNIC of virtual tester c)The CPU type of the server d)vCPU allocated for virtual tester (VMs) e)Memory allocated for virtual tester (VMs) f)The number and rate of server NIC 5.2.3. The test parameters: a)test repeated times b)test packet length 5.2.4. Testing process 1.Record CPU load value of server according to the steps of 5.1.3. 2.Under the same throughput, Shut down or bypass the DUT (VSwitch) record the CPU load value of server. 3.Calculate the increase of the CPU load value due to establish the DUT (VSwitch). 5.2.5. Test results formats --------------------------------------------------- | Byte| Throughput(GE)| Server CPU MHZ | VM CPU | --------------------------------------------------- | 0 | 0 | 515 | 3042 | --------------------------------------------------- | 128 | 0.46 | 6395 | 3040 | --------------------------------------------------- | 256 | 0.84 | 6517 | 3042 | --------------------------------------------------- Vic Liu Expires January4,2015 [Page 11] Internet-Draft virtual network performance Benchmark July 2014 | 512 | 1.56 | 6668 | 3041 | --------------------------------------------------- | 1024| 2.88 | 6280 | 3043 | --------------------------------------------------- | 1450| 4.00 | 6233 | 3045 | --------------------------------------------------- 5.3. Memory consumption The operation of DUT (VSwitch) can increase the CPU load of host server. Different V-Switches have different memory occupation. This can be an important indicator in benchmark the Virtual network performance. 5.3.1. Objectives The objective of this test is to verify the memory consumption by the DUT (VSwitch) on the Host server. 5.3.2. Configuration parameters Network parameters should be define as follows: a) The number of virtual tester (VMs) b) The number of vNIC of virtual tester c) The CPU type of the server d) vCPU allocated for virtual tester (VMs) e) Memory allocated for virtual tester (VMs) f) The number and rate of server NIC 5.3.3. The test parameters: a) test repeated times b) test packet length 5.3.4. Testing process 1. Record memory consumption value of server according to the steps of 5.1.3. 2. Under the same throughput, Shut down or bypass the DUT (VSwitch) record the memory consumption value of server. Vic Liu Expires January4,2015 [Page 12] Internet-Draft virtual network performance Benchmark July 2014 3. Calculate the increase of the memory consumption value due to establish the DUT (VSwitch). 5.3.5. Test results formats ------------------------------------------------- | Byte| Throughput(GE)| Host Memory | VM Memory | | ------------------------------------------------- | 0 | 0 | 3042 | 696 | ------------------------------------------------- | 128 | 0.46 | 3040 | 696 | ------------------------------------------------- | 256 | 0.84 | 3042 | 696 | ------------------------------------------------- | 512 | 1.56 | 3041 | 696 | ------------------------------------------------- | 1024| 2.88 | 3043 | 696 | ------------------------------------------------- | 1450| 4.00 | 3045 | 696 | ------------------------------------------------- 5.4. Latency Physical tester's time reference from its own clock or other time source, such as GPS, which can achieve the accuracy of 10ns. In virtual network circumstances, the virtual tester gets its reference time from Linux systems. But the clock on Linux of different server or VM can't synchronized accuracy due to current method. Although VM of some higher versions of CentOS or Fedora can achieve the accuracy of 1ms, if the network can provide better NTP connections, the result will be better. In the future, we may consider some other ways to have a better synchronization of the time to improve the accuracy of the test. 5.4.1. Objectives The objective of this test is to verify the DUT (VSwitch) for latency of the flow. This can be an important indicator in benchmark the Virtual network performance. 5.4.2. Configuration parameters Network parameters should be define as follows: Vic Liu Expires January4,2015 [Page 13] Internet-Draft virtual network performance Benchmark July 2014 a) The number of virtual tester (VMs) b) The number of vNIC of virtual tester c) The CPU type of the server d) vCPU allocated for virtual tester (VMs) e) Memory allocated for virtual tester (VMs) f) The number and rate of server NIC 5.4.3. The test parameters: a) test repeated times b) test packet length 5.4.4. Testing process 1. Record latency value of server according to the steps of 5.1.3. 2. Under the same throughput, Shut down or bypass the DUT (VSwitch) record the latency value of server. 3. Calculate the increase of the latency value due to establish the DUT (VSwitch). 5.4.5. Test results formats TBD. 6. Formal Syntax The following syntax specification uses the augmented Backus-Naur Form (BNF) as described in RFC-2234[RFC2234]. Vic Liu Expires January4,2015 [Page 14] Internet-Draft virtual network performance Benchmark July 2014 7. Security Considerations 8. IANA Considerations 9. Conclusions 10. References 10.1. Normative References [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [2] Crocker, D. and Overell, P.(Editors), "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, Internet Mail Consortium and Demon Internet Ltd., November 1997. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2234] Crocker, D. and Overell, P.(Editors), "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, Internet Mail Consortium and Demon Internet Ltd., November 1997. 10.2. Informative References [3] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in TCP and Its Effect on Busy Servers", Proc. Infocom 1999 pp. 1573- 1583. [Fab1999] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in TCP and Its Effect on Busy Servers", Proc. Infocom 1999 pp. 1573-1583. 11. Acknowledgments Thanks for Paul To, alick luo of sprient communication and Jianjun Lu of VMware Beijing Office supporting this draft. This document was prepared using 2-Word-v2.0.template.dot. Vic Liu Expires January4,2015 [Page 15] Internet-Draft virtual network performance Benchmark July 2014 Authors' Addresses Vic Liu China Mobile 32 Xuanwumen West Ave, Beijing, China Email: liuzhiheng@chinamobile.com Dapeng Liu China Mobile 32 Xuanwumen West Ave, Beijing, China Email: liudapeng@chinamobile.com Bob Mandeville Iometrix 3600 Fillmore Street Suite 409 San Francisco, CA 94123 USA bob@iometrix.com Brooks Hickman Spirent Communications 1325 Borregas Ave Sunnyvale, CA 94089 USA Brooks.Hickman@spirent.com Guang Zhang IXIA Email: GZhang@ixiacom.com Vic Liu Expires January4,2015 [Page 16]