idnits 2.17.1 draft-liu-bmwg-virtual-network-benchmark-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- -- The document has an IETF Trust Provisions (28 Dec 2009) Section 6.c(ii) Publication Limitation clause. If this document is intended for submission to the IESG for publication, this constitutes an error. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of lines with control characters in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to contain a disclaimer for pre-RFC5378 work, but was first submitted on or after 10 November 2008. The disclaimer is usually necessary only for documents that revise or obsolete older RFCs, and that take significant amounts of text from those RFCs. If you can contact all authors of the source material and they are willing to grant the BCP78 rights to the IETF Trust, you can and should remove the disclaimer. Otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (July 4, 2014) is 3584 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: '1' is defined on line 554, but no explicit reference was found in the text == Unused Reference: '2' is defined on line 557, but no explicit reference was found in the text == Unused Reference: 'RFC2119' is defined on line 561, but no explicit reference was found in the text == Unused Reference: '3' is defined on line 570, but no explicit reference was found in the text == Unused Reference: 'Fab1999' is defined on line 574, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2234 (ref. '2') (Obsoleted by RFC 4234) -- Duplicate reference: RFC2119, mentioned in 'RFC2119', was also mentioned in '1'. -- Duplicate reference: RFC2234, mentioned in 'RFC2234', was also mentioned in '2'. ** Obsolete normative reference: RFC 2234 (Obsoleted by RFC 4234) Summary: 3 errors (**), 0 flaws (~~), 7 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group Vic Liu 2 Internet Draft Dapeng Liu 3 Intended status: Informational China Mobile 4 Bob Mandeville 5 Iometrix 6 Brooks Hickman 7 Spirent Communications 8 Guang Zhang 9 IXIA 10 Expires: January 2015 July 4, 2014 12 Benchmarking Methodology for Virtualization Network Performance 13 draft-liu-bmwg-virtual-network-benchmark-00.txt 15 Status of this Memo 17 This Internet-Draft is submitted in full conformance with the 18 provisions of BCP 78 and BCP 79. 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. This document may not be modified, 22 and derivative works of it may not be created, and it may not be 23 published except as an Internet-Draft. 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. This document may not be modified, 27 and derivative works of it may not be created, except to publish it 28 as an RFC and to translate it into languages other than English. 30 This document may contain material from IETF Documents or IETF 31 Contributions published or made publicly available before November 32 10, 2008. The person(s) controlling the copyright in some of this 33 material may not have granted the IETF Trust the right to allow 34 modifications of such material outside the IETF Standards Process. 35 Without obtaining an adequate license from the person(s) controlling 36 the copyright in such materials, this document may not be modified 37 outside the IETF Standards Process, and derivative works of it may 38 not be created outside the IETF Standards Process, except to format 39 it for publication as an RFC or to translate it into languages other 40 than English. 42 Internet-Drafts are working documents of the Internet Engineering 43 Task Force (IETF), its areas, and its working groups. Note that 44 other groups may also distribute working documents as Internet- 45 Drafts. 47 Internet-Drafts are draft documents valid for a maximum of six 48 months and may be updated, replaced, or obsoleted by other documents 49 at any time. It is inappropriate to use Internet-Drafts as 50 reference material or to cite them other than as "work in progress." 52 The list of current Internet-Drafts can be accessed 53 at http://www.ietf.org/ietf/1id-abstracts.txt 55 The list of Internet-Draft Shadow Directories can be accessed 56 at http://www.ietf.org/shadow.html 58 This Internet-Draft will expire on January4,2009. 60 Copyright Notice 62 Copyright (c) 2014 IETF Trust and the persons identified as the 63 document authors. All rights reserved. 65 This document is subject to BCP 78 and the IETF Trust's Legal 66 Provisions Relating to IETF Documents 67 (http://trustee.ietf.org/license-info)in effect on the date of 68 publication of this document. Please review these documents 69 carefully, as they describe your rights and restrictions with 70 respect to this document. Code Components extracted from this 71 document must include Simplified BSD License text as described in 72 Section 4.e of the Trust Legal Provisions and are provided without 73 warranty as described in the Simplified BSD License. 75 This document is subject to BCP 78 and the IETF Trust's Legal 76 Provisions Relating to IETF Documents 77 (http://trustee.ietf.org/license-info) in effect on the date of 78 publication of this document. Please review these documents 79 carefully, as they describe your rights and restrictions with 80 respect to this document. 82 Abstract 84 As the virtual network have been wide establish in IDC. The 85 performance of virtual network has become a consideration to the IDC 86 managers. This draft introduce a benchmarking methodology for 87 virtualization network performance. 89 Table of Contents 91 1. Introduction ................................................ 3 92 2. Peculiar issues ............................................. 3 93 3. Key Performance Index........................................ 6 94 4. Test Setup .................................................. 6 95 5. Proposed Benchmark Tests..................................... 9 96 5.1. Throughput ............................................. 9 97 5.2. CPU consumption........................................ 10 98 5.3. Memory consumption..................................... 12 99 5.4. Latency ............................................... 13 100 6. Formal Syntax .............................................. 14 101 7. Security Considerations..................................... 15 102 8. IANA Considerations ........................................ 15 103 9. Conclusions ................................................ 15 104 10. References ................................................ 15 105 10.1. Normative References.................................. 15 106 10.2. Informative References................................ 15 107 11. Acknowledgments ........................................... 15 109 1. Introduction 111 As the virtual network have been wide establish in IDC. The 112 performance of virtual network has become a consideration to the IDC 113 managers. This draft introduce a benchmarking methodology for 114 virtualization network performance. 116 2. Peculiar issues 118 In a conventional test setup with real test ports, it is quite 119 legitimate to assume test ports provide the golden standard and in 120 measuring the performance metrics. If and when test results are sub 121 optimal, it is automatically assumed it's the Device-Under-Test 122 (DUT) that is at fault. For example, when testing the max no-drop 123 throughput at a given frame size, if the test result shows less than 124 100% throughput, we can safely conclude that it's the DUT that can't 125 deliver line rate forwarding at that frame size(s). We never doubt 126 that tester will be an issue. 128 In a virtual test environment where both the DUT as well as the test 129 tool itself are VM based, it's quite a different story. Just like 130 the DUT, tester in VM shape will have its own performance peak under 131 various conditions. Even worse, conditions are multitude and in many 132 forms. 134 Tester's calibration without DUT is essential in benchmarking 135 testing in a virtual environment. Furthermore, to reduce the 136 enormous combination of various conditions, tester must be 137 calibrated with the exact same combination and parameter set the 138 user wants to measure against a real DUT. A slight variation of 139 conditions and parameter value will cause inaccurate measurements of 140 the DUT. 142 While the exact combination and parameter set are hard to list, 143 below table attempts to give a most common example how to calibrate 144 a tester before testing a real DUT under the same condition. 146 Sample calibration permutation 148 ---------------------------------------------------------------- 150 | Hypervisor | VM VNIC | VM Memory | Packet | No Drop | 152 | Type | Speed |CPU Allocation | Size | Throughput| 154 ---------------------------------------------------------------- 156 | ESXi 1G/10G 512M/1Core | 64 | | 158 | | 128 | | 160 | | 256 | | 162 | | 512 | | 164 | | 1024 | | 165 | | 1518 | | 167 ---------------------------------------------------------------- 169 Key points are as following: 171 a)The hypervisor type is of ultimate importance to the test results. 172 Tester VM must be installed on the same hypervisor type as the 173 real DUT. When feasible, tester VM software should be installed on 174 a separate, but identical type of, hypervisor. 175 b)The VNIC speed will have impact on test results. Tester must 176 calibrate against all VNIC speeds to be tested against a real DUT. 177 c)VM allocation of CPU resources and memory will affect test results 178 d)Packet sizes will affect test results dramatically due to the 179 nature of virtual machine 180 e)Other possible extensions of above table: The number of VMs to be 181 created, latency and/or jitter readings, one VNIC per VM vs. 182 multiple VM sharing one VNIC, and uni-directional traffic vs. bi- 183 directional traffic. 185 It's important to have a test environment for tester's calibration 186 as close as possible to when a real DUT will be involved for the 187 benchmark test. Above test setup illustrate below key points: 189 1.One or more tester's VM need to be created for both traffic 190 generation and analysis 191 2.vSwitch is need to accommodate performance penalty due to extra VM 192 addition 193 3.VNIC and its type is needed in the test setup to once again 194 accommodate any performance penalty when real DUT VM is created 195 4.ToR switch is needed to accommodate delays introduced by the 196 external device 198 In summary, calibration should be done in such an environment that 199 other than the DUT VM, all possible factors that may negatively 200 impact test results should be accommodated. 202 3. Key Performance Index 204 We listed numbers of key performance index for virtual network as 205 follows: 207 a) No drop throughput under various frame sizes: Forwarding 208 performance under various frame sizes is a key performance index 209 of interest. Once this performance number is obtained, vendors can 210 always allocate more CPU and memory for mission critical 211 applications where line rate performance is expected. 212 b) DUT consumption of CPU and memory: when adding one or more VM. 213 With addition of each VM, DUT will consume more CPU and memory. 214 c) Latency readings: Some applications are highly sensitive on 215 latency.It's important to get the latency reading with 216 respective to various conditions. 218 VxLAN performance can be looked at from two perspectives. First, 219 addition of VxLAN on an existing VM will consume extra CPU resources 220 and memory. This can be easily included in the benchmark table. 221 Tester VM are strictly traffic generator and analyzer. No 222 calibration is needed when adding VxLAN to DUT VM. 224 Once basic performance metric is obtained with respective to single 225 VxLAN, we need to look at performance metrics with many VM and 226 VxLAN. The idea is verify how many VM/VxLAN can be created and what 227 their forwarding performance (no drop throughput), latency, and 228 CPU/memory consumptions are. 230 4. Test Setup 232 The test bed is constituted by two physical server with 10GE NIC, a 233 test center, a 10GE TOR switch for test traffic and a 1GE TOR switch 234 for management. 236 ---------------------- 237 |Test Center PHY 10GE*2| 238 ---------------------- 239 || 240 || 241 ---------- 242 =====| 10GE TOR |======= 243 || ---------- || 244 || || 245 || || 246 ------------------- ------------------- 247 | -------------- | | -------------- | 248 | |V-switch(VTEP)| | | |V-switch(VTEP)| | 249 | -------------- | | -------------- | 250 | | | | | | | | 251 | ----- ----- | | ----- ----- | 252 | |TCVM1| |TCVM2|| | |TCVM1| |TCVM2|| 253 | ----- ----- | | ----- ----- | 254 ------------------- ------------------- 255 Server1 Server2 257 Two Dell server are R710XD (CPU: E5-2460) and R710 (CPU: E5-2430) 258 with a pair of 10GE NIC. And in the server we allocate 2 vCPU and 8G 259 memory to each Test Center Virtual Machine (TCVM). 261 In traffic model A: We use a physical test center connect to each 262 server to verify the benchmark of each server. 264 ---------------------- 265 |Test Center PHY 10GE*2| 266 ---------------------- 267 || 268 || 269 ------------------- 270 | -------------- | 271 | |V-switch(VTEP)| | 272 | -------------- | 273 | | | | 274 | ----- ----- | 275 | |TCVM1| |TCVM2|| 276 | ----- ----- | 277 ------------------- 278 Server1 280 In traffic model B: We use the benchmark to test the performance of 281 VxLAN. 283 ---------- 284 =====| 10GE TOR |======= 285 || ---------- || 286 || || 287 || || 288 ------------------- ------------------- 289 | -------------- | | -------------- | 290 | |V-switch(VTEP)| | | |V-switch(VTEP)| | 291 | -------------- | | -------------- | 292 | | | | | | | | 293 | ----- ----- | | ----- ----- | 294 | |TCVM1| |TCVM2|| | |TCVM1| |TCVM2|| 295 | ----- ----- | | ----- ----- | 296 ------------------- ------------------- 297 Server1 Server2 299 5. Proposed Benchmark Tests 301 5.1. Throughput 303 Unlike the traditional test cases which the DUT and the tester are 304 separated, it has brought unparalleled challenges to virtual network 305 test. In that case, the tester and the DUT (visual switches) are in 306 one server (physically converged), so the CPU and MEM share the same 307 resources. Theoretically, the tester's operation may has some 308 influences on the DUT's performances. However, for the specialty of 309 virtualization, this method is the only way to assess the truth of 310 assessment method. 312 Under the background of existing technology, when we mean to test 313 the virtual switch's throughput, the concept of traditional physical 314 switch will not be applicable. The traditional throughput indicates 315 the switches' largest transmit capability, for certain selected 316 bytes and selected cycle under zero-packet-lose conditions. But in 317 virtual environment, the fluctuant of performance on virtual network 318 will be much greater than dedicated physical devices. In the same 319 time, because the DUT and the tester cannot be separated, which only 320 proved the DUT realize same network performances under certain 321 circumstances, it also means the DUT may achieve higher capability. 323 Therefore, we change the throughout in virtual environment to actual 324 throughput, hoping in future, as the improvement of technique, the 325 actual throughput will approach the theoretical throughput 326 gradually. 328 Of course, under actual condition, this throughout have certain 329 referential meanings. In most cases, common throughput application 330 cannot compare with professional tester, so for virtual application 331 and data center's deployment, the actual throughout already have 332 great refinance value. 334 5.1.1. Objectives 336 Under the condition of certain hardware configuration, test the 337 DUT(virtual switch) can support maximum throughput. 339 5.1.2. Configuration parameters 341 Network parameters should be define as follows: 343 a) the number of virtual tester (VMs) 344 b) the number of vNIC of virtual tester 345 c) the CPU type of the server 346 d) vCPU allocated for virtual tester (VMs) 347 e) memory allocated for virtual tester (VMs) 348 f) the number and rate of server NIC 350 5.1.3. The test parameters 352 a) test repeated times 353 b) test packet length 355 5.1.4. Testing process 357 1. Configure the virtual tester to output traffic through V-Switch. 358 2. Increase the number of vCPU in the tester until the traffic has no 359 packet loss. 360 3. Record the max throughput on VSwitch 361 4. Change the packet length and repeat step1 and record test results. 363 5.1.5. Test results formats 365 ---------------------- 366 | Byte| Throughput (GE)| 367 ---------------------- 368 | 0 | 0 | 369 ---------------------- 370 | 128 | 0.46 | 371 ---------------------- 372 | 256 | 0.84 | 373 ---------------------- 374 | 512 | 1.56 | 375 ---------------------- 376 | 1024| 2.88 | 377 ---------------------- 378 | 1518| 4.00 | 379 ---------------------- 381 5.2. CPU consumption 383 The operation of DUT (VSwitch) can increase the CPU load of host 384 server. Different V-Switches have different CPU occupation. This can 385 be an important indicator in benchmark the Virtual network 386 performance. 388 5.2.1. Objectives 390 The objectives of this test is verified the CPU consumption caused 391 by the DUT (VSwitch). 393 5.2.2. Configuration parameters 395 Network parameters should be define as follows: 397 a)The number of virtual tester (VMs) 398 b)The number of vNIC of virtual tester 399 c)The CPU type of the server 400 d)vCPU allocated for virtual tester (VMs) 401 e)Memory allocated for virtual tester (VMs) 402 f)The number and rate of server NIC 404 5.2.3. The test parameters: 406 a)test repeated times 407 b)test packet length 409 5.2.4. Testing process 411 1.Record CPU load value of server according to the steps of 5.1.3. 412 2.Under the same throughput, Shut down or bypass the DUT (VSwitch) 413 record the CPU load value of server. 414 3.Calculate the increase of the CPU load value due to establish the 415 DUT (VSwitch). 417 5.2.5. Test results formats 419 --------------------------------------------------- 420 | Byte| Throughput(GE)| Server CPU MHZ | VM CPU | 421 --------------------------------------------------- 422 | 0 | 0 | 515 | 3042 | 423 --------------------------------------------------- 424 | 128 | 0.46 | 6395 | 3040 | 425 --------------------------------------------------- 426 | 256 | 0.84 | 6517 | 3042 | 427 --------------------------------------------------- 428 | 512 | 1.56 | 6668 | 3041 | 429 --------------------------------------------------- 430 | 1024| 2.88 | 6280 | 3043 | 431 --------------------------------------------------- 432 | 1450| 4.00 | 6233 | 3045 | 433 --------------------------------------------------- 435 5.3. Memory consumption 437 The operation of DUT (VSwitch) can increase the CPU load of host 438 server. Different V-Switches have different memory occupation. This 439 can be an important indicator in benchmark the Virtual network 440 performance. 442 5.3.1. Objectives 444 The objective of this test is to verify the memory consumption by the 445 DUT (VSwitch) on the Host server. 447 5.3.2. Configuration parameters 449 Network parameters should be define as follows: 451 a) The number of virtual tester (VMs) 452 b) The number of vNIC of virtual tester 453 c) The CPU type of the server 454 d) vCPU allocated for virtual tester (VMs) 455 e) Memory allocated for virtual tester (VMs) 456 f) The number and rate of server NIC 458 5.3.3. The test parameters: 460 a) test repeated times 461 b) test packet length 463 5.3.4. Testing process 465 1. Record memory consumption value of server according to the steps 466 of 5.1.3. 467 2. Under the same throughput, Shut down or bypass the DUT (VSwitch) 468 record the memory consumption value of server. 470 3. Calculate the increase of the memory consumption value due to 471 establish the DUT (VSwitch). 473 5.3.5. Test results formats 475 ------------------------------------------------- 476 | Byte| Throughput(GE)| Host Memory | VM Memory | | 477 ------------------------------------------------- 478 | 0 | 0 | 3042 | 696 | 479 ------------------------------------------------- 480 | 128 | 0.46 | 3040 | 696 | 481 ------------------------------------------------- 482 | 256 | 0.84 | 3042 | 696 | 483 ------------------------------------------------- 484 | 512 | 1.56 | 3041 | 696 | 485 ------------------------------------------------- 486 | 1024| 2.88 | 3043 | 696 | 487 ------------------------------------------------- 488 | 1450| 4.00 | 3045 | 696 | 489 ------------------------------------------------- 491 5.4. Latency 493 Physical tester's time reference from its own clock or other time 494 source, such as GPS, which can achieve the accuracy of 10ns. In 495 virtual network circumstances, the virtual tester gets its reference 496 time from Linux systems. But the clock on Linux of different server 497 or VM can't synchronized accuracy due to current method. Although VM 498 of some higher versions of CentOS or Fedora can achieve the accuracy 499 of 1ms, if the network can provide better NTP connections, the 500 result will be better. 502 In the future, we may consider some other ways to have a better 503 synchronization of the time to improve the accuracy of the test. 505 5.4.1. Objectives 507 The objective of this test is to verify the DUT (VSwitch) for latency 508 of the flow. This can be an important indicator in benchmark the 509 Virtual network performance. 511 5.4.2. Configuration parameters 513 Network parameters should be define as follows: 515 a) The number of virtual tester (VMs) 516 b) The number of vNIC of virtual tester 517 c) The CPU type of the server 518 d) vCPU allocated for virtual tester (VMs) 519 e) Memory allocated for virtual tester (VMs) 520 f) The number and rate of server NIC 522 5.4.3. The test parameters: 524 a) test repeated times 525 b) test packet length 527 5.4.4. Testing process 529 1. Record latency value of server according to the steps of 5.1.3. 530 2. Under the same throughput, Shut down or bypass the DUT (VSwitch) 531 record the latency value of server. 532 3. Calculate the increase of the latency value due to establish the 533 DUT (VSwitch). 535 5.4.5. Test results formats 537 TBD. 539 6. Formal Syntax 541 The following syntax specification uses the augmented Backus-Naur 542 Form (BNF) as described in RFC-2234[RFC2234]. 544 7. Security Considerations 546 8. IANA Considerations 548 9. Conclusions 550 10. References 552 10.1. Normative References 554 [1] Bradner, S., "Key words for use in RFCs to Indicate 555 Requirement Levels", BCP 14, RFC 2119, March 1997. 557 [2] Crocker, D. and Overell, P.(Editors), "Augmented BNF for 558 Syntax Specifications: ABNF", RFC 2234, Internet Mail 559 Consortium and Demon Internet Ltd., November 1997. 561 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 562 Requirement Levels", BCP 14, RFC 2119, March 1997. 564 [RFC2234] Crocker, D. and Overell, P.(Editors), "Augmented BNF for 565 Syntax Specifications: ABNF", RFC 2234, Internet Mail 566 Consortium and Demon Internet Ltd., November 1997. 568 10.2. Informative References 570 [3] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in TCP 571 and Its Effect on Busy Servers", Proc. Infocom 1999 pp. 1573- 572 1583. 574 [Fab1999] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in 575 TCP and Its Effect on Busy Servers", Proc. Infocom 1999 576 pp. 1573-1583. 578 11. Acknowledgments 580 Thanks for Paul To, alick luo of sprient communication and 581 Jianjun Lu of VMware Beijing Office supporting this draft. 583 This document was prepared using 2-Word-v2.0.template.dot. 585 Authors' Addresses 587 Vic Liu 588 China Mobile 589 32 Xuanwumen West Ave, Beijing, China 591 Email: liuzhiheng@chinamobile.com 593 Dapeng Liu 594 China Mobile 595 32 Xuanwumen West Ave, Beijing, China 597 Email: liudapeng@chinamobile.com 599 Bob Mandeville 600 Iometrix 601 3600 Fillmore Street 602 Suite 409 603 San Francisco, CA 94123 604 USA 605 bob@iometrix.com 607 Brooks Hickman 608 Spirent Communications 609 1325 Borregas Ave 610 Sunnyvale, CA 94089 611 USA 612 Brooks.Hickman@spirent.com 614 Guang Zhang 615 IXIA 616 Email: GZhang@ixiacom.com