idnits 2.17.1 draft-ietf-bmwg-traffic-management-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 29, 2015) is 3309 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group B. Constantine 2 Internet Draft JDSU 3 Intended status: Informational R. Krishnan 4 Expires: September 2015 Brocade Communications 5 March 29, 2015 7 Traffic Management Benchmarking 8 draft-ietf-bmwg-traffic-management-04.txt 10 Status of this Memo 12 This Internet-Draft is submitted in full conformance with the 13 provisions of BCP 78 and BCP 79. 15 Internet-Drafts are working documents of the Internet Engineering 16 Task Force (IETF). Note that other groups may also distribute 17 working documents as Internet-Drafts. The list of current Internet- 18 Drafts is at http://datatracker.ietf.org/drafts/current/. 20 Internet-Drafts are draft documents valid for a maximum of six months 21 and may be updated, replaced, or obsoleted by other documents at any 22 time. It is inappropriate to use Internet-Drafts as reference 23 material or to cite them other than as "work in progress." 25 This Internet-Draft will expire on September 29, 2015. 27 Copyright Notice 29 Copyright (c) 2015 IETF Trust and the persons identified as the 30 document authors. All rights reserved. 32 This document is subject to BCP 78 and the IETF Trust's Legal 33 Provisions Relating to IETF Documents 34 (http://trustee.ietf.org/license-info) in effect on the date of 35 publication of this document. Please review these documents 36 carefully, as they describe your rights and restrictions with respect 37 to this document. Code Components extracted from this document must 38 include Simplified BSD License text as described in Section 4.e of 39 the Trust Legal Provisions and are provided without warranty as 40 described in the Simplified BSD License. 42 Abstract 44 This framework describes a practical methodology for benchmarking the 45 traffic management capabilities of networking devices (i.e. policing, 46 shaping, etc.). The goal is to provide a repeatable test method that 47 objectively compares performance of the device's traffic management 48 capabilities and to specify the means to benchmark traffic management 49 with representative application traffic. 51 Table of Contents 53 1. Introduction...................................................4 54 1.1. Traffic Management Overview...............................4 55 1.2. DUT Lab Configuration and Testing Overview................5 56 2. Conventions used in this document..............................7 57 3. Scope and Goals................................................8 58 4. Traffic Benchmarking Metrics...................................9 59 4.1. Metrics for Stateless Traffic Tests.......................9 60 4.2. Metrics for Stateful Traffic Tests.......................11 61 5. Tester Capabilities...........................................11 62 5.1. Stateless Test Traffic Generation........................11 63 5.1.1. Burst Hunt with Stateless Traffic...................11 64 5.2. Stateful Test Pattern Generation.........................12 65 5.2.1. TCP Test Pattern Definitions........................13 66 6. Traffic Benchmarking Methodology..............................14 67 6.1. Policing Tests...........................................15 68 6.1.1 Policer Individual Tests................................15 69 6.1.2 Policer Capacity Tests..............................16 70 6.1.2.1 Maximum Policers on Single Physical Port..........17 71 6.1.2.2 Single Policer on All Physical Ports..............18 72 6.1.2.3 Maximum Policers on All Physical Ports............19 73 6.2. Queue/Scheduler Tests....................................20 74 6.2.1 Queue/Scheduler Individual Tests........................20 75 6.2.1.1 Testing Queue/Scheduler with Stateless Traffic....21 76 6.2.1.2 Testing Queue/Scheduler with Stateful Traffic.....21 77 6.2.2 Queue / Scheduler Capacity Tests......................23 78 6.2.2.1 Multiple Queues / Single Port Active..............23 79 6.2.2.1.1 Strict Priority on Egress Port..................24 80 6.2.2.1.2 Strict Priority + Weighted Fair Queue (WFQ).....24 81 6.2.2.2 Single Queue per Port / All Ports Active..........25 82 6.2.2.3 Multiple Queues per Port, All Ports Active........25 83 6.3. Shaper tests.............................................26 84 6.3.1 Shaper Individual Tests...............................26 85 6.3.1.1 Testing Shaper with Stateless Traffic.............27 86 6.3.1.2 Testing Shaper with Stateful Traffic..............28 87 6.3.2 Shaper Capacity Tests.................................30 88 6.3.2.1 Single Queue Shaped, All Physical Ports Active....30 89 6.3.2.2 All Queues Shaped, Single Port Active.............30 90 6.3.2.3 All Queues Shaped, All Ports Active...............31 91 6.4. Concurrent Capacity Load Tests...........................32 92 Appendix A: Open Source Tools for Traffic Management Testing..32 93 Appendix B: Stateful TCP Test Patterns........................33 94 7. Security Considerations.......................................37 95 8. IANA Considerations...........................................37 96 9. Acknowledgments...............................................37 97 10. References...................................................37 98 10.1. Normative References....................................37 99 10.2. Informative References..................................38 101 1. Introduction 103 Traffic management (i.e. policing, shaping, etc.) is an increasingly 104 important component when implementing network Quality of Service 105 (QoS). 107 There is currently no framework to benchmark these features 108 although some standards address specific areas which are described 109 in Section 1.1. 111 This draft provides a framework to conduct repeatable traffic 112 management benchmarks for devices and systems in a lab environment. 114 Specifically, this framework defines the methods to characterize 115 the capacity of the following traffic management features in network 116 devices; classification, policing, queuing / scheduling, and 117 traffic shaping. 119 This benchmarking framework can also be used as a test procedure to 120 assist in the tuning of traffic management parameters before service 121 activation. In addition to Layer 2/3 (Ethernet / IP) benchmarking, 122 Layer 4 (TCP) test patterns are proposed by this draft in order to 123 more realistically benchmark end-user traffic. 125 1.1. Traffic Management Overview 127 In general, a device with traffic management capabilities performs 128 the following functions: 130 - Traffic classification: identifies traffic according to various 131 configuration rules (for example IEEE 802.1Q Virtual LAN (VLAN), 132 Differential Services Code Point (DSCP) etc.) and marks this traffic 133 internally to the network device. Multiple external priorities 134 (DSCP, 802.1p, etc.) can map to the same priority in the device. 135 - Traffic policing: limits the rate of traffic that enters a network 136 device according to the traffic classification. If the traffic 137 exceeds the provisioned limits, the traffic is either dropped or 138 remarked and forwarded onto to the next network device 139 - Traffic Scheduling: provides traffic classification within the 140 network device by directing packets to various types of queues and 141 applies a dispatching algorithm to assign the forwarding sequence 142 of packets 143 - Traffic shaping: a traffic control technique that actively buffers 144 and smooths the output rate in an attempt to adapt bursty traffic 145 to the configured limits 146 - Active Queue Management (AQM): 147 AQM involves monitoring the status of internal queues and 148 proactively dropping (or remarking) packets, which causes hosts 149 using congestion-aware protocols to back-off and in turn alleviate 150 queue congestion [AQM-RECO]. On the other hand, classic traffic 151 management techniques reactively drop (or remark) packets based on 152 queue full condition. The benchmarking scenarios for AQM are 153 different and is outside of the scope of this testing framework. 155 The following diagram is a generic model of the traffic management 156 capabilities within a network device. It is not intended to 157 represent all variations of manufacturer traffic management 158 capabilities, but provide context to this test framework. 160 |----------| |----------------| |--------------| |----------| 161 | | | | | | | | 162 |Interface | |Ingress Actions | |Egress Actions| |Interface | 163 |Input | |(classification,| |(scheduling, | |Output | 164 |Queues | | marking, | | shaping, | |Queues | 165 | |-->| policing or |-->| active queue |-->| | 166 | | | shaping) | | management | | | 167 | | | | | remarking) | | | 168 |----------| |----------------| |--------------| |----------| 170 Figure 1: Generic Traffic Management capabilities of a Network Device 172 Ingress actions such as classification are defined in [RFC4689] 173 and include IP addresses, port numbers, DSCP, etc. In terms of 174 marking, [RFC2697] and [RFC2698] define a single rate and dual rate, 175 three color marker, respectively. 177 The Metro Ethernet Forum (MEF) specifies policing and shaping in 178 terms of Ingress and Egress Subscriber/Provider Conditioning 179 Functions in MEF12.1 [MEF-12.1]; Ingress and Bandwidth Profile 180 attributes in MEF10.2 [MEF-10.2] and MEF 26 [MEF-26]. 182 1.2 Lab Configuration and Testing Overview 184 The following is the description of the lab set-up for the traffic 185 management tests: 187 +--------------+ +-------+ +----------+ +-----------+ 188 | Transmitting | | | | | | Receiving | 189 | Test Host | | | | | | Test Host | 190 | |-----| Device|---->| Network |--->| | 191 | | | Under | | Delay | | | 192 | | | Test | | Emulator | | | 193 | |<----| |<----| |<---| | 194 | | | | | | | | 195 +--------------+ +-------+ +----------+ +-----------+ 197 As shown in the test diagram, the framework supports uni-directional 198 and bi-directional traffic management tests (where the transmitting 199 and receiving roles would be reversed on the return path). 201 This testing framework describes the tests and metrics for each of 202 the following traffic management functions: 203 - Policing 204 - Queuing / Scheduling 205 - Shaping 207 The tests are divided into individual and rated capacity tests. 208 The individual tests are intended to benchmark the traffic management 209 functions according to the metrics defined in Section 4. The 210 capacity tests verify traffic management functions under the load of 211 many simultaneous individual tests and their flows. 213 This involves concurrent testing of multiple interfaces with the 214 specific traffic management function enabled, and increasing load to 215 the capacity limit of each interface. 217 As an example: a device is specified to be capable of shaping on all 218 of its egress ports. The individual test would first be conducted to 219 benchmark the specified shaping function against the metrics defined 220 in section 4. Then the capacity test would be executed to test the 221 shaping function concurrently on all interfaces and with maximum 222 traffic load. 224 The Network Delay Emulator (NDE) is required for TCP stateful tests 225 in order to allow TCP to utilize a significant size TCP window in its 226 control loop. 228 Also note that the Network Delay Emulator (NDE) SHOULD be passive in 229 nature such as a fiber spool. This is recommended to eliminate the 230 potential effects that an active delay element (i.e. test impairment 231 generator) may have on the test flows. In the case where a fiber 232 spool is not practical due to the desired latency, an active NDE MUST 233 be independently verified to be capable of adding the configured 234 delay without loss. In other words, the DUT would be removed and the 235 NDE performance benchmarked independently. 237 Note the NDE SHOULD be used in "full pipe" delay mode. Most NDEs 238 allow for per flow delay actions, emulating QoS prioritization. For 239 this framework, the NDE's sole purpose is simply to add delay to all 240 packets (emulate network latency). So to benchmark the performance of 241 the NDE, maximum offered load should be tested against the following 242 frame sizes: 128, 256, 512, 768, 1024, 1500,and 9600 bytes. The delay 243 accuracy at each of these packet sizes can then be used to calibrate 244 the range of expected Bandwidth Delay Product (BDP) for the TCP 245 stateful tests. 247 2. Conventions used in this document 249 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 250 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 251 document are to be interpreted as described in [RFC2119]. 253 The following acronyms are used: 255 AQM: Active Queue Management 257 BB: Bottleneck Bandwidth 259 BDP: Bandwidth Delay Product 261 BSA: Burst Size Achieved 263 CBS: Committed Burst Size 265 CIR: Committed Information Rate 267 DUT: Device Under Test 269 EBS: Excess Burst Size 271 EIR: Excess Information Rate 273 NDE: Network Delay Emulator 275 SP: Strict Priority Queuing 277 QL: Queue Length 279 QoS: Quality of Service 281 RTH: Receiving Test Host 283 RTT: Round Trip Time 285 SBB: Shaper Burst Bytes 287 SBI: Shaper Burst Interval 289 SR: Shaper Rate 291 SSB: Send Socket Buffer 293 Tc: CBS Time Interval 295 Te: EBS Time Interval 297 Ti Transmission Interval 299 TTH: Transmitting Test Host 300 TTP: TCP Test Pattern 302 TTPET: TCP Test Pattern Execution Time 304 3. Scope and Goals 306 The scope of this work is to develop a framework for benchmarking and 307 testing the traffic management capabilities of network devices in the 308 lab environment. These network devices may include but are not 309 limited to: 310 - Switches (including Layer 2/3 devices) 311 - Routers 312 - Firewalls 313 - General Layer 4-7 appliances (Proxies, WAN Accelerators, etc.) 315 Essentially, any network device that performs traffic management as 316 defined in section 1.1 can be benchmarked or tested with this 317 framework. 319 The primary goal is to assess the maximum forwarding performance 320 deemed to be within the provisioned traffic limits that a network 321 device can sustain without dropping or impairing packets, or 322 compromising the accuracy of multiple instances of traffic 323 management functions. This is the benchmark for comparison between 324 devices. 326 Within this framework, the metrics are defined for each traffic 327 management test but do not include pass / fail criterion, which is 328 not within the charter of BMWG. This framework provides the test 329 methods and metrics to conduct repeatable testing, which will 330 provide the means to compare measured performance between DUTs. 332 As mentioned in section 1.2, these methods describe the individual 333 tests and metrics for several management functions. It is also within 334 scope that this framework will benchmark each function in terms of 335 overall rated capacity. This involves concurrent testing of multiple 336 interfaces with the specific traffic management function enabled, up 337 to the capacity limit of each interface. 339 It is not within scope of this of this framework to specify the 340 procedure for testing multiple configurations of traffic management 341 functions concurrently. The multitudes of possible combinations is 342 almost unbounded and the ability to identify functional "break 343 points" would be almost impossible. 345 However, section 6.4 provides suggestions for some profiles of 346 concurrent functions that would be useful to benchmark. The key 347 requirement for any concurrent test function is that tests MUST 348 produce reliable and repeatable results. 350 Also, it is not within scope to perform conformance testing. Tests 351 defined in this framework benchmark the traffic management functions 352 according to the metrics defined in section 4 and do not address any 353 conformance to standards related to traffic management. The current 354 specifications don't specify exact behavior or implementation and the 355 specifications that do exist (cited in section 1.1) allow 356 implementations to vary w.r.t. short term rate accuracy and other 357 factors. This is a primary driver for this framework: to provide 358 an objective means to compare vendor traffic management functions. 360 Another goal is to devise methods that utilize flows with 361 congestion-aware transport (TCP) as part of the traffic load and 362 still produce repeatable results in the isolated test environment. 363 This framework will derive stateful test patterns (TCP or 364 application layer) that can also be used to further benchmark the 365 performance of applicable traffic management techniques such as 366 queuing / scheduling and traffic shaping. In cases where the 367 network device is stateful in nature (i.e. firewall, etc.), 368 stateful test pattern traffic is important to test along with 369 stateless, UDP traffic in specific test scenarios (i.e. 370 applications using TCP transport and UDP VoIP, etc.) 372 As mentioned earlier in the document, repeatability of test results 373 is critical, especially considering the nature of stateful TCP 374 traffic. To this end, the stateful tests will use TCP test patterns 375 to emulate applications. This framework also provides guidelines 376 for application modeling and open source tools to achieve the 377 repeatable stimulus. And finally, TCP metrics from [RFC6349] MUST 378 be measured for each stateful test and provide the means to compare 379 each repeated test. 381 4. Traffic Benchmarking Metrics 383 The metrics to be measured during the benchmarks are divided into two 384 (2) sections: packet layer metrics used for the stateless traffic 385 testing and TCP layer metrics used for the stateful traffic 386 testing. 388 4.1. Metrics for Stateless Traffic Tests 390 Stateless traffic measurements require that sequence number and 391 time-stamp be inserted into the payload for lost packet analysis. 392 Delay analysis may be achieved by insertion of timestamps directly 393 into the packets or timestamps stored elsewhere (packet captures). 394 This framework does not specify the packet format to carry sequence 395 number or timing information. 397 However,[RFC4737] and [RFC4689] provide recommendations 398 for sequence tracking along with definitions of in-sequence and 399 out-of-order packets. 401 The following are the metrics that MUST be measured during the 402 stateless traffic benchmarking components of the tests: 404 - Burst Size Achieved (BSA): for the traffic policing and network 405 queue tests, the tester will be configured to send bursts to test 406 either the Committed Burst Size (CBS) or Excess Burst Size (EBS) of 407 a policer or the queue / buffer size configured in the DUT. The 408 Burst Size Achieved metric is a measure of the actual burst size 409 received at the egress port of the DUT with no lost packets. As an 410 example, the configured CBS of a DUT is 64KB and after the burst 411 test, only a 63 KB can be achieved without packet loss. Then 63KB is 412 the BSA. Also, the average Packet Delay Variation (PDV see below) as 413 experienced by the packets sent at the BSA burst size should be 414 recorded. This metric shall be reported in units of bytes, KBytes, 415 or MBytes. 417 - Lost Packets(LP): For all traffic management tests, the tester will 418 transmit the test packets into the DUT ingress port and the number of 419 packets received at the egress port will be measured. The difference 420 between packets transmitted into the ingress port and received at the 421 egress port is the number of lost packets as measured at the egress 422 port. These packets must have unique identifiers such that only the 423 test packets are measured. For cases where multiple flows are 424 transmitted from ingress to egress port (e.g. IP conversations), each 425 flow must have sequence numbers within the test packets stream. 427 [RFC6703] and [RFC2680] describe the need to establish the 428 time threshold to wait before a packet is declared as lost, and this 429 threshold MUST be reported with the results. This metric shall be 430 reported as an integer number which cannot be negative. (see: 431 http://tools.ietf.org/html/rfc6703#section-4.1) 433 - Out of Order (OOO): in additions to the LP metric, the test 434 packets must be monitored for sequence. [RFC4689] defines the 435 general function of sequence tracking, as well as definitions 436 for in-sequence and out-of-order packets. Out-of-order packets 437 will be counted per [RFC4737]. This metric shall be reported as 438 an integer number which cannot be negative. 440 - Packet Delay (PD): the Packet Delay metric is the difference 441 between the timestamp of the received egress port packets and the 442 packets transmitted into the ingress port and specified in [RFC1242]. 443 The transmitting host and receiving host time must be in 444 time sync using NTP , GPS, etc. This metric SHALL be reported as a 445 real number of seconds, where a negative measurement usually 446 indicates a time synchronization problem between test devices. 448 - Packet Delay Variation (PDV): the Packet Delay Variation metric is 449 the variation between the timestamp of the received egress port 450 packets and specified in [RFC5481]. Note that per [RFC5481], 451 this PDV is the variation of one-way delay across many packets in 452 the traffic flow. Per the measurement formula in [RFC5481], select 453 the high percentile of 99% and units of measure will be a real 454 number of seconds (negative is not possible for PDV and would 455 indicate a measurement error). 457 - Shaper Rate (SR): The SR represents the average DUT output 458 rate (bps) over the test interval. The Shaper Rate is only 459 applicable to the traffic shaping tests. 461 - Shaper Burst Bytes (SBB): A traffic shaper will emit packets in 462 different size "trains"; these are frames "back-to-back", respect 463 the mandatory inter-frame gap. This metric characterizes the method 464 by which the shaper emits traffic. Some shapers transmit larger 465 bursts per interval, and a burst of 1 packet would apply to the 466 extreme case of a shaper sending a CBR stream of single packets. 467 This metric SHALL be reported in units of bytes, KBytes, or MBytes. 468 Shaper Burst Bytes is only applicable to thetraffic shaping tests. 470 - Shaper Burst Interval(SBI): the SBI is the time between shaper 471 emitted bursts and is measured at the DUT egress port. This metric 472 shall be reported as an real number of seconds. Shaper Burst 473 Interval is only applicable to the traffic shaping tests, 475 4.2. Metrics for Stateful Traffic Tests 477 The stateful metrics will be based on [RFC6349] TCP metrics and 478 MUST include: 480 - TCP Test Pattern Execution Time (TTPET): [RFC6349] defined the TCP 481 Transfer Time for bulk transfers, which is simply the measured time 482 to transfer bytes across single or concurrent TCP connections. The 483 TCP test patterns used in traffic management tests will include bulk 484 transfer and interactive applications. The interactive patterns 485 include instances such as HTTP business applications, database 486 applications, etc. The TTPET will be the measure of the time for a 487 single execution of a TCP Test Pattern (TTP). Average, minimum, and 488 maximum times will be measured or calculated and expressed as a real 489 number of seconds. 491 An example would be an interactive HTTP TTP session which should take 492 5 seconds on a GigE network with 0.5 millisecond latency. During ten 493 (10) executions of this TTP, the TTPET results might be: average of 494 6.5 seconds, minimum of 5.0 seconds, and maximum of 7.9 seconds. 496 - TCP Efficiency: after the execution of the TCP Test Pattern, TCP 497 Efficiency represents the percentage of Bytes that were not 498 retransmitted. 500 Transmitted Bytes - Retransmitted Bytes 502 TCP Efficiency % = --------------------------------------- X 100 504 Transmitted Bytes 506 Transmitted Bytes are the total number of TCP Bytes to be transmitted 507 including the original and the retransmitted Bytes. These 508 retransmitted bytes should be recorded from the sender's TCP/IP stack 509 perspective, to avoid any misinterpretation that a reordered packet 510 is a retransmitted packet (as may be the case with packet decode 511 interpretation). 513 - Buffer Delay: represents the increase in RTT during a TCP test 514 versus the baseline DUT RTT (non congested, inherent latency). RTT 515 and the technique to measure RTT (average versus baseline) are 516 defined in [RFC6349]. Referencing [RFC6349], the average RTT is 517 derived from the total of all measured RTTs during the actual test 518 sampled at every second divided by the test duration in seconds. 520 Total RTTs during transfer 521 Average RTT during transfer = ----------------------------- 522 Transfer duration in seconds 524 Average RTT during Transfer - Baseline RTT 525 Buffer Delay % = ------------------------------------------ X 100 526 Baseline RTT 528 Note that even though this was not explicitly stated in [RFC6349], 529 retransmitted packets should not be used in RTT measurements. 531 Also, the test results should record the average RTT in millisecond 532 across the entire test duration and number of samples. 534 5. Tester Capabilities 536 The testing capabilities of the traffic management test environment 537 are divided into two (2) sections: stateless traffic testing and 538 stateful traffic testing 540 5.1. Stateless Test Traffic Generation 542 The test device MUST be capable of generating traffic at up to the 543 link speed of the DUT. The test device must be calibrated to verify 544 that it will not drop any packets. The test device's inherent PD and 545 PDV must also be calibrated and subtracted from the PD and PDV 546 metrics. The test device must support the encapsulation to be 547 tested such as IEEE 802.1Q VLAN, IEEE 802.1ad Q-in-Q, Multiprotocol 548 Label Switching (MPLS), etc. Also, the test device must allow 549 control of the classification techniques defined in [RFC4689] 550 (i.e. IP address, DSCP, TOS, etc classification). 552 The open source tool "iperf" can be used to generate stateless UDP 553 traffic and is discussed in Appendix A. Since iperf is a software 554 based tool, there will be performance limitations at higher link 555 speeds (e.g. GigE, 10 GigE, etc.). Careful calibration of any test 556 environment using iperf is important. At higher link speeds, it is 557 recommended to use hardware based packet test equipment. 559 5.1.1 Burst Hunt with Stateless Traffic 561 A central theme for the traffic management tests is to benchmark the 562 specified burst parameter of traffic management function, since burst 563 parameters of SLAs are specified in bytes. For testing efficiency, 564 it is recommended to include a burst hunt feature, which automates 565 the manual process of determining the maximum burst size which can 566 be supported by a traffic management function. 568 The burst hunt algorithm should start at the target burst size 569 (maximum burst size supported by the traffic management function) 570 and will send single bursts until it can determine the largest burst 571 that can pass without loss. If the target burst size passes, then 572 the test is complete. The hunt aspect occurs when the target burst 573 size is not achieved; the algorithm will drop down to a configured 574 minimum burst size and incrementally increase the burst until the 575 maximum burst supported by the DUT is discovered. The recommended 576 granularity of the incremental burst size increase is 1 KB. 578 Optionally for a policer function and if the burst size passes, the 579 burst should be increased by increments of 1 KB to verify that the 580 policer is truly configured properly (or enabled at all). 582 5.2. Stateful Test Pattern Generation 584 The TCP test host will have many of the same attributes as the TCP 585 test host defined in [RFC6349]. The TCP test device may be a 586 standard computer or a dedicated communications test instrument. In 587 both cases, it must be capable of emulating both a client and a 588 server. 590 For any test using stateful TCP test traffic, the Network Delay 591 Emulator (NDE function from the lab set-up diagram) must be used in 592 order to provide a meaningful BDP. As referenced in section 2, the 593 target traffic rate and configured RTT MUST be verified independently 594 using just the NDE for all stateful tests (to ensure the NDE can 595 delay without loss). 597 The TCP test host MUST be capable to generate and receive stateful 598 TCP test traffic at the full link speed of the DUT. As a general 599 rule of thumb, testing TCP Throughput at rates greater than 500 Mbps 600 may require high performance server hardware or dedicated hardware 601 based test tools. 603 The TCP test host MUST allow adjusting both Send and Receive Socket 604 Buffer sizes. The Socket Buffers must be large enough to fill the 605 BDP for bulk transfer TCP test application traffic. 607 Measuring RTT and retransmissions per connection will generally 608 require a dedicated communications test instrument. In the absence of 609 dedicated hardware based test tools, these measurements may need to 610 be conducted with packet capture tools, i.e. conduct TCP Throughput 611 tests and analyze RTT and retransmissions in packet captures. 613 The TCP implementation used by the test host MUST be specified in 614 the test results (e.g. TCP New Reno, TCP options supported, etc.). 615 Additionally, the test results SHALL provide specific congestion 616 control algorithm details, as per [RFC3148]. 618 While [RFC6349] defined the means to conduct throughput tests of TCP 619 bulk transfers, the traffic management framework will extend TCP test 620 execution into interactive TCP application traffic. Examples include 621 email, HTTP, business applications, etc. This interactive traffic is 622 bi-directional and can be chatty, meaning many turns in traffic 623 communication during the course of a transaction (versus the 624 relatively uni-directional flow of bulk transfer applications). 626 The test device must not only support bulk TCP transfer application 627 traffic but MUST also support chatty traffic. A valid stress test 628 SHOULD include both traffic types. This is due to the non-uniform, 629 bursty nature of chatty applications versus the relatively uniform 630 nature of bulk transfers (the bulk transfer smoothly stabilizes to 631 equilibrium state under lossless conditions). 633 While iperf is an excellent choice for TCP bulk transfer testing, 634 the netperf open source tool provides the ability to control the 635 client and server request / response behavior. The netperf-wrapper 636 tool is a Python wrapper to run multiple simultaneous netperf 637 instances and aggregate the results. Appendix A provides an overview 638 of netperf / netperf-wrapper and another open source application 639 emulation tools, iperf. As with any software based tool, the 640 performance must be qualified to the link speed to be tested. 641 Hardware-based test equipment should be considered for reliable 642 results at higher links speeds (e.g. 1 GigE, 10 GigE). 644 5.2.1. TCP Test Pattern Definitions 646 As mentioned in the goals of this framework, techniques are defined 647 to specify TCP traffic test patterns to benchmark traffic 648 management technique(s) and produce repeatable results. Some 649 network devices such as firewalls, will not process stateless test 650 traffic which is another reason why stateful TCP test traffic must 651 be used. 653 An application could be fully emulated up to Layer 7, however this 654 framework proposes that stateful TCP test patterns be used in order 655 to provide granular and repeatable control for the benchmarks. The 656 following diagram illustrates a simple Web Browsing application 657 (HTTP). 659 GET url 661 Client ------------------------> Web 663 Web 200 OK 100ms | 665 Browser <------------------------ Server 667 In this example, the Client Web Browser (Client) requests a URL and 668 then the Web Server delivers the web page content to the Client 669 (after a Server delay of 100 millisecond). This asynchronous, 670 "request/response" behavior is intrinsic to most TCP based 671 applications such as Email (SMTP), File Transfers (FTP and SMB), 672 Database (SQL), Web Applications (SOAP), REST, etc. The impact to 673 the network elements is due to the multitudes of Clients and the 674 variety of bursty traffic, which stresses traffic management 675 functions. The actual emulation of the specific application 676 protocols is not required and TCP test patterns can be defined to 677 mimic the application network traffic flows and produce repeatable 678 results. 680 Application modeling techniques have been proposed in 681 "3GPP2 C.R1002-0 v1.0" and provides examples to model the behavior of 682 HTTP, FTP, and WAP applications at the TCP layer. The models have 683 been defined with various mathematical distributions for the 684 Request/Response bytes and inter-request gap times. The model 685 definition format described in this work are the basis for the 686 guidelines provides in Appendix B and are also similar to formats 687 used by network modeling tools. Packet captures can also be used to 688 characterize application traffic and specify some of the test 689 patterns listed in Appendix B. 691 This framework does not specify a fixed set of TCP test patterns, but 692 does provide test cases that SHOULD be performed in Appendix B. Some 693 of these examples reflect those specified in "draft-ietf-bmwg-ca- 694 bench-meth-04" which suggests traffic mixes for a variety of 695 representative application profiles. Other examples are simply 696 well-known application traffic types such as HTTP. 698 6. Traffic Benchmarking Methodology 700 The traffic benchmarking methodology uses the test set-up from 701 section 2 and metrics defined in section 4. 703 Each test SHOULD compare the network device's internal statistics 704 (available via command line management interface, SNMP, etc.) to the 705 measured metrics defined in section 4. This evaluates the accuracy 706 of the internal traffic management counters under individual test 707 conditions and capacity test conditions that are defined in each 708 subsection. 710 From a device configuration standpoint, scheduling and shaping 711 functionality can be applied to logical ports such Link Aggregation 712 (LAG). This would result in the same scheduling and shaping 713 configuration applied to all the member physical ports. The focus of 714 this draft is only on tests at a physical port level. 716 The following sections provide the objective, procedure, metrics, and 717 reporting format for each test. For all test steps, the following 718 global parameters must be specified: 720 Test Runs (Tr). Defines the number of times the test needs to be run 721 to ensure accurate and repeatable results. The recommended value is 722 a minimum of 10. 724 Test Duration (Td). Defines the duration of a test iteration, 725 expressed in seconds. The recommended minimum value is 60 seconds. 727 The variability in the test results MUST be measured between Test 728 Runs and if the variation is characterized as a significant portion 729 of the measured values, the next step may be to revise the methods to 730 achieve better consistency. 732 6.1. Policing Tests 734 A policer is defined as the entity performing the policy function. 735 The intent of the policing tests is to verify the policer performance 736 (i.e. CIR-CBS and EIR-EBS parameters). The tests will verify that the 737 network device can handle the CIR with CBS and the EIR with EBS and 738 will use back-back packet testing concepts from [RFC2544] (but 739 adapted to burst size algorithms and terminology). Also [MEF-14], 740 [MEF-19], and [MEF-37] provide some basis for specific components of 741 this test. The burst hunt algorithm defined in section 5.1.1 can 742 also be used to automate the measurement of the CBS value. 744 The tests are divided into two (2) sections; individual policer 745 tests and then full capacity policing tests. It is important to 746 benchmark the basic functionality of the individual policer then 747 proceed into the fully rated capacity of the device. This capacity 748 may include the number of policing policies per device and the 749 number of policers simultaneously active across all ports. 751 6.1.1 Policer Individual Tests 753 Objective: 754 Test a policer as defined by [RFC4115] or MEF 10.2, depending upon 755 the equipment's specification. In addition to verifying that the 756 policer allows the specified CBS and EBS bursts to pass, the policer 757 test MUST verify that the policer will remark or drop excess, and 758 pass traffic at the specified CBS/EBS values. 760 Test Summary: 761 Policing tests should use stateless traffic. Stateful TCP test 762 traffic will generally be adversely affected by a policer in the 763 absence of traffic shaping. So while TCP traffic could be used, 764 it is more accurate to benchmark a policer with stateless traffic. 766 As an example for [RFC4115], consider a CBS and EBS of 64KB and CIR 767 and EIR of 100 Mbps on a 1GigE physical link (in color-blind mode). 768 A stateless traffic burst of 64KB would be sent into the policer at 769 the GigE rate. This equates to approximately a 0.512 millisecond 770 burst time (64 KB at 1 GigE). The traffic generator must space these 771 bursts to ensure that the aggregate throughput does not exceed the 772 CIR. The Ti between the bursts would equal CBS * 8 / CIR = 5.12 773 millisecond in this example. 775 Test Metrics: 776 The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL 777 be measured at the egress port and recorded. 779 Procedure: 780 1. Configure the DUT policing parameters for the desired CIR/EIR and 781 CBS/EBS values to be tested 783 2. Configure the tester to generate a stateless traffic burst equal 784 to CBS and an interval equal to Ti (CBS in bits / CIR) 786 3. Compliant Traffic Step: Generate bursts of CBS + EBS traffic into 787 the policer ingress port and measure the metrics defined in 788 section 4.1 (BSA, LP. OOS, PD, and PDV) at the egress port and 789 across the entire Td (default 60 seconds duration) 791 4. Excess Traffic Test: Generate bursts of greater than CBS + EBS 792 limit traffic into the policer ingress port and verify that the 793 policer only allowed the BSA bytes to exit the egress. The excess 794 burst MUST be recorded and the recommended value is 1000 bytes. 795 Additional tests beyond the simple color-blind example might 796 include: color-aware mode, configurations where EIR is greater 797 than CIR, etc. 799 Reporting Format: 800 The policer individual report MUST contain all results for each 801 CIR/EIR/CBS/EBS test run and a recommended format is as follows: 803 ******************************************************** 804 Test Configuration Summary: Tr, Td 806 DUT Configuration Summary: CIR, EIR, CBS, EBS 808 The results table should contain entries for each test run, (Test #1 809 to Test #Tr). 811 Compliant Traffic Test: BSA, LP, OOS, PD, and PDV 813 Excess Traffic Test: BSA 814 ******************************************************** 816 6.1.2 Policer Capacity Tests 818 Objective: 819 The intent of the capacity tests is to verify the policer performance 820 in a scaled environment with multiple ingress customer policers on 821 multiple physical ports. This test will benchmark the maximum number 822 of active policers as specified by the device manufacturer. 824 Test Summary: 825 The specified policing function capacity is generally expressed in 826 terms of the number of policers active on each individual physical 827 port as well as the number of unique policer rates that are utilized. 828 For all of the capacity tests, the benchmarking test procedure and 829 report format described in Section 6.1.1 for a single policer MUST 830 be applied to each of the physical port policers. 832 As an example, a Layer 2 switching device may specify that each of 833 the 32 physical ports can be policed using a pool of policing service 834 policies. The device may carry a single customer's traffic on each 835 physical port and a single policer is instantiated per physical port. 836 Another possibility is that a single physical port may carry multiple 837 customers, in which case many customer flows would be policed 838 concurrently on an individual physical port (separate policers per 839 customer on an individual port). 841 Test Metrics: 842 The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL 843 be measured at the egress port and recorded. 845 The following sections provide the specific test scenarios, 846 procedures, and reporting formats for each policer capacity test. 848 6.1.2.1 Maximum Policers on Single Physical Port Test 850 Test Summary: 851 The first policer capacity test will benchmark a single physical 852 port, maximum policers on that physical port. 854 Assume multiple categories of ingress policers at rates r1, r2,...rn. 855 There are multiple customers on a single physical port. Each customer 856 could be represented by a single tagged vlan, double tagged vlan, 857 VPLS instance etc. Each customer is mapped to a different policer. 858 Each of the policers can be of rates r1, r2,..., rn. 860 An example configuration would be 861 - Y1 customers, policer rate r1 862 - Y2 customers, policer rate r2 863 - Y3 customers, policer rate r3 864 ... 865 - Yn customers, policer rate rn 867 Some bandwidth on the physical port is dedicated for other traffic 868 non customer traffic); this includes network control protocol 869 traffic. There is a separate policer for the other traffic. Typical 870 deployments have 3 categories of policers; there may be some 871 deployments with more or less than 3 categories of ingress 872 policers. 874 Test Procedure: 875 1. Configure the DUT policing parameters for the desired CIR/EIR and 876 CBS/EBS values for each policer rate (r1-rn) to be tested 878 2. Configure the tester to generate a stateless traffic burst equal 879 to CBS and an interval equal to TI (CBS in bits/CIR) for each 880 customer stream (Y1 - Yn). The encapsulation for each customer 881 must also be configured according to the service tested (VLAN, 882 VPLS, IP mapping, etc.). 884 3. Compliant Traffic Step: Generate bursts of CBS + EBS traffic into 885 the policer ingress port for each customer traffic stream and 886 measure the metrics defined in section 4.1 (BSA, LP, OOS, PD, and 887 PDV) at the egress port for each stream and across the entire Td 888 (default 30 seconds duration) 890 4. Excess Traffic Test: Generate bursts of greater than CBS + EBS 891 limit traffic into the policer ingress port for each customer 892 traffic stream and verify that the policer only allowed the BSA 893 bytes to exit the egress for each stream. The excess burst MUST 894 recorded and the recommended value is 1000 bytes. 896 Reporting Format: 897 The policer individual report MUST contain all results for each 898 CIR/EIR/CBS/EBS test run, per customer traffic stream. 900 A recommended format is as follows: 902 ******************************************************** 903 Test Configuration Summary: Tr, Td 905 Customer traffic stream Encapsulation: Map each stream to VLAN, 906 VPLS, IP address 908 DUT Configuration Summary per Customer Traffic Stream: CIR, EIR, 909 CBS, EBS 911 The results table should contain entries for each test run, (Test #1 912 to Test #Tr). 914 Customer Stream Y1-Yn (see note), Compliant Traffic Test: BSA, LP, 915 OOS, PD, and PDV 917 Customer Stream Y1-Yn (see note), Excess Traffic Test: BSA 918 ******************************************************** 920 Note: For each test run, there will be a two (2) rows for each 921 customer stream, the compliant traffic result and the excess traffic 922 result. 924 6.1.2.2 Single Policer on All Physical Ports 926 Test Summary: 927 The second policer capacity test involves a single Policer function 928 per physical port with all physical ports active. In this test, 929 there is a single policer per physical port. The policer can have 930 one of the rates r1, r2,.., rn. All the physical ports in the 931 networking device are active. 933 Procedure: 934 The procedure is identical to 6.1.1, the configured parameters must 935 be reported per port and the test report must include results per 936 measured egress port 938 6.1.2.3 Maximum Policers on All Physical Ports 940 Finally the third policer capacity test involves a combination of the 941 first and second capacity test, namely maximum policers active per 942 physical port and all physical ports are active. 944 Procedure: 945 Uses the procedural method from 6.1.2.1 and the configured parameters 946 must be reported per port and the test report must include per stream 947 results per measured egress port. 949 6.2. Queue and Scheduler Tests 951 Queues and traffic Scheduling are closely related in that a queue's 952 priority dictates the manner in which the traffic scheduler 953 transmits packets out of the egress port. 955 Since device queues / buffers are generally an egress function, this 956 test framework will discuss testing at the egress (although the 957 technique can be applied to ingress side queues). 959 Similar to the policing tests, the tests are divided into two 960 sections; individual queue/scheduler function tests and then full 961 capacity tests. 963 6.2.1 Queue/Scheduler Individual Tests Overview 965 The various types of scheduling techniques include FIFO, Strict 966 Priority (SP), Weighted Fair Queueing (WFQ) along with other 967 variations. This test framework recommends to test at a minimum 968 of three techniques although it is the discretion of the tester 969 to benchmark other device scheduling algorithms. 971 6.2.1.1 Queue/Scheduler with Stateless Traffic Test 973 Objective: 974 Verify that the configured queue and scheduling technique can 975 handle stateless traffic bursts up to the queue depth. 977 Test Summary: 978 A network device queue is memory based unlike a policing function, 979 which is token or credit based. However, the same concepts from 980 section 6.1 can be applied to testing network device queues. 982 The device's network queue should be configured to the desired size 983 in KB (queue length, QL) and then stateless traffic should be 984 transmitted to test this QL. 986 A queue should be able to handle repetitive bursts with the 987 transmission gaps proportional to the bottleneck bandwidth. This 988 gap is referred to as the transmission interval (Ti). Ti can 989 be defined for the traffic bursts and is based off of the QL and 990 Bottleneck Bandwidth (BB) of the egress interface. 992 Ti = QL * 8 / BB 994 Note that this equation is similar to the Ti required for 995 transmission into a policer (QL = CBS, BB = CIR). Also note that the 996 burst hunt algorithm defined in section 5.1.1 can also be used to 997 automate the measurement of the queue value. 999 The stateless traffic burst shall be transmitted at the link speed 1000 and spaced within the Ti time interval. The metrics defined in 1001 section 4.1 shall be measured at the egress port and recorded; the 1002 primary result is to verify the BSA and that no packets are dropped. 1004 The scheduling function must also be characterized to benchmark the 1005 device's ability to schedule the queues according to the priority. 1006 An example would be 2 levels of priority including SP and FIFO 1007 queueing. Under a flow load greater the egress port speed, the 1008 higher priority packets should be transmitted without drops (and 1009 also maintain low latency), while the lower priority (or best 1010 effort) queue may be dropped. 1012 Test Metrics: 1013 The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL 1014 be measured at the egress port and recorded. 1016 Procedure: 1017 1. Configure the DUT queue length (QL) and scheduling technique 1018 (FIFO, SP, etc) parameters 1020 2. Configure the tester to generate a stateless traffic burst equal 1021 to QL and an interval equal to Ti (QL in bits/BB) 1023 3. Generate bursts of QL traffic into the DUT and measure the 1024 metrics defined in section 4.1 (LP, OOS, PD, and PDV) at the 1025 egress port and across the entire Td (default 30 seconds 1026 duration) 1028 Report Format: 1029 The Queue/Scheduler Stateless Traffic individual report MUST contain 1030 all results for each QL/BB test run and a recommended format is as 1031 follows: 1033 ******************************************************** 1034 Test Configuration Summary: Tr, Td 1036 DUT Configuration Summary: Scheduling technique, BB and QL 1038 The results table should contain entries for each test run 1039 as follows, 1041 (Test #1 to Test #Tr). 1043 - LP, OOS, PD, and PDV 1044 ******************************************************** 1046 6.2.1.2 Testing Queue/Scheduler with Stateful Traffic 1048 Objective: 1049 Verify that the configured queue and scheduling technique can handle 1050 stateful traffic bursts up to the queue depth. 1052 Test Background and Summary: 1053 To provide a more realistic benchmark and to test queues in layer 4 1054 devices such as firewalls, stateful traffic testing is recommended 1055 for the queue tests. Stateful traffic tests will also utilize the 1056 Network Delay Emulator (NDE) from the network set-up configuration in 1057 section 2. 1059 The BDP of the TCP test traffic must be calibrated to the QL of the 1060 device queue. Referencing [RFC6349], the BDP is equal to: 1062 BB * RTT / 8 (in bytes) 1064 The NDE must be configured to an RTT value which is large enough to 1065 allow the BDP to be greater than QL. An example test scenario is 1066 defined below: 1068 - Ingress link = GigE 1069 - Egress link = 100 Mbps (BB) 1070 - QL = 32KB 1072 RTT(min) = QL * 8 / BB and would equal 2.56 millisecond (and the 1073 BDP = 32KB) 1075 In this example, one (1) TCP connection with window size / SSB of 1076 32KB would be required to test the QL of 32KB. This Bulk Transfer 1077 Test can be accomplished using iperf as described in Appendix A. 1079 Two types of TCP tests MUST be performed: Bulk Transfer test and 1080 Micro Burst Test Pattern as documented in Appendix B. The Bulk 1081 Transfer Test only bursts during the TCP Slow Start (or Congestion 1082 Avoidance) state, while the Micro Burst test emulates application 1083 layer bursting which may occur any time during the TCP connection. 1085 Other tests types SHOULD include: Simple Web Site, Complex Web Site, 1086 Business Applications, Email, SMB/CIFS File Copy (which are also 1087 documented in Appendix B). 1089 Test Metrics: 1090 The test results will be recorded per the stateful metrics defined in 1091 section 4.2, primarily the TCP Test Pattern Execution Time (TTPET), 1092 TCP Efficiency, and Buffer Delay. 1094 Procedure: 1096 1. Configure the DUT queue length (QL) and scheduling technique 1097 (FIFO, SP, etc) parameters 1099 2. Configure the tester* to generate a profile of emulated of an 1100 application traffic mixture 1102 - The application mixture MUST be defined in terms of percentage 1103 of the total bandwidth to be tested 1105 - The rate of transmission for each application within the mixture 1106 MUST be also be configurable 1108 * The tester MUST be capable of generating a precise TCP test 1109 patterns for each application specified, to ensure repeatable 1110 results. 1112 3. Generate application traffic between the ingress (client side) and 1113 egress (server side) ports of the DUT and measure application 1114 throughput the metrics (TTPET, TCP Efficiency, and Buffer Delay), 1116 per application stream and at the ingress and egress port (across 1117 the entire Td, default 60 seconds duration). 1119 Concerning application measurements, a couple of items require 1120 clarification. An application session may be comprised of a single 1121 TCP connection or multiple TCP connections. 1123 For the single TCP connection application sessions, the application 1124 thoughput / metrics have a 1-1 relationship to the TCP connection 1125 measurements. 1127 If an application session (i.e. HTTP-based application) utilizes 1128 multiple TCP connections, then all of the TCP connections are 1129 aggregated in the application throughput measurement / metrics for 1130 that application. 1132 Then there is the case of mulitlple instances of an application 1133 session (i.e. multiple FTPs emulating multiple clients). In this 1134 situation, the test should measure / record each FTP application 1135 session independently, tabulating the minimum, maximum, and average 1136 for all FTP sessions. 1138 Finally, application throughput measurements are based on Layer 4 1139 TCP throughput and do not include bytes retransmitted. The TCP 1140 Efficiency metric MUST be measured during the test and provides a 1141 measure of "goodput" during each test. 1143 Reporting Format: 1144 The Queue/Scheduler Stateful Traffic individual report MUST contain 1145 all results for each traffic scheduler and QL/BB test run and a 1146 recommended format is as follows: 1148 ******************************************************** 1149 Test Configuration Summary: Tr, Td 1151 DUT Configuration Summary: Scheduling technique, BB and QL 1153 Application Mixture and Intensities: this is the percent configured 1154 of each application type 1156 The results table should contain entries for each test run with 1157 minimum, maximum, and average per application session as follows, 1158 (Test #1 to Test #Tr) 1160 - Per Application Throughout (bps) and TTPET 1161 - Per Application Bytes In and Bytes Out 1162 - Per Application TCP Efficiency, and Buffer Delay 1163 ******************************************************** 1165 6.2.2 Queue / Scheduler Capacity Tests 1167 Objective: 1168 The intent of these capacity tests is to benchmark queue/scheduler 1169 performance in a scaled environment with multiple queues/schedulers 1170 active on multiple egress physical ports. This test will benchmark 1171 the maximum number of queues and schedulers as specified by the 1172 device manufacturer. Each priority in the system will map to a 1173 separate queue. 1175 Test Metrics: 1176 The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL 1177 be measured at the egress port and recorded. 1179 The following sections provide the specific test scenarios, 1180 procedures, and reporting formats for each queue / scheduler capacity 1181 test. 1183 6.2.2.1 Multiple Queues / Single Port Active 1185 For the first scheduler / queue capacity test, multiple queues per 1186 port will be tested on a single physical port. In this case, 1187 all the queues (typically 8) are active on a single physical port. 1188 Traffic from multiple ingress physical ports are directed to the 1189 same egress physical port which will cause oversubscription on the 1190 egress physical port. 1192 There are many types of priority schemes and combinations of 1193 priorities that are managed by the scheduler. The following 1194 sections specify the priority schemes that should be tested. 1196 6.2.2.1.1 Strict Priority on Egress Port 1198 Test Summary: 1199 For this test, Strict Priority (SP) scheduling on the egress 1200 physical port should be tested and the benchmarking methodology 1201 specified in section 6.2.1.1 and 6.2.1.2 (procedure, metrics, 1202 and reporting format) should be applied here. For a given 1203 priority, each ingress physical port should get a fair share of 1204 the egress physical port bandwidth. 1206 Since this is a capacity test, the configuration and report 1207 results format from 6.2.1.1 and 6.2.1.2 MUST also include: 1209 Configuration: 1210 - The number of physical ingress ports active during the test 1211 - The classication marking (DSCP, VLAN, etc.) for each physical 1212 ingress port 1213 - The traffic rate for stateful traffic and the traffic rate 1214 / mixture for stateful traffic for each physical ingress port 1216 Report results: 1217 - For each ingress port traffic stream, the achieved throughput 1218 rate and metrics at the egress port 1220 6.2.2.1.2 Strict Priority + Weighted Fair Queue (WFQ) on Egress Port 1222 Test Summary: 1223 For this test, Strict Priority (SP) and Weighted Fair Queue (WFQ) 1224 should be enabled simultaneously in the scheduler but on a single 1225 egress port. The benchmarking methodology specified in Section 1227 6.2.1.1 and 6.2.1.2 (procedure, metrics, and reporting format) 1228 should be applied here. Additionally, the egress port bandwidth 1229 sharing among weighted queues should be proportional to the assigned 1230 weights. For a given priority, each ingress physical port should get 1231 a fair share of the egress physical port bandwidth. 1233 Since this is a capacity test, the configuration and report results 1234 format from 6.2.1.1 and 6.2.1.2 MUST also include: 1236 Configuration: 1237 - The number of physical ingress ports active during the test 1238 - The classication marking (DSCP, VLAN, etc.) for each physical 1239 ingress port 1240 - The traffic rate for stateful traffic and the traffic rate / 1241 mixture for stateful traffic for each physical ingress port 1243 Report results: 1244 - For each ingress port traffic stream, the achieved throughput rate 1245 and metrics at each queue of the egress port queue (both the SP 1246 and WFQ queue). 1248 Example: 1249 - Egress Port SP Queue: throughput and metrics for ingress streams 1250 1-n 1251 - Egress Port WFQ Queue: throughput and metrics for ingress streams 1252 1-n 1254 6.2.2.2 Single Queue per Port / All Ports Active 1256 Test Summary: 1257 Traffic from multiple ingress physical ports are directed to the 1258 same egress physical port, which will cause oversubscription on the 1259 egress physical port. Also, the same amount of traffic is directed 1260 to each egress physical port. 1262 The benchmarking methodology specified in Section 6.2.1.1 1263 and 6.2.1.2 (procedure, metrics, and reporting format) should be 1264 applied here. Each ingress physical port should get a fair share of 1265 the egress physical port bandwidth. Additionally, each egress 1266 physical port should receive the same amount of traffic. 1268 Since this is a capacity test, the configuration and report results 1269 format from 6.2.1.1 and 6.2.1.2 MUST also include: 1271 Configuration: 1272 - The number of ingress ports active during the test 1273 - The number of egress ports active during the test 1274 - The classication marking (DSCP, VLAN, etc.) for each physical 1275 ingress port 1276 - The traffic rate for stateful traffic and the traffic rate / 1277 mixture for stateful traffic for each physical ingress port 1279 Report results: 1280 - For each egress port, the achieved throughput rate and metrics at 1281 the egress port queue for each ingress port stream. 1283 Example: 1284 - Egress Port 1: throughput and metrics for ingress streams 1-n 1285 - Egress Port n: throughput and metrics for ingress streams 1-n 1287 6.2.2.3 Multiple Queues per Port, All Ports Active 1289 Traffic from multiple ingress physical ports are directed to all 1290 queues of each egress physical port, which will cause 1291 oversubscription on the egress physical ports. Also, the same 1292 amount of traffic is directed to each egress physical port. 1294 The benchmarking methodology specified in Section 6.2.1.1 1295 and 6.2.1.2 (procedure, metrics, and reporting format) should be 1296 applied here. For a given priority, each ingress physical port 1297 should get a fair share of the egress physical port bandwidth. 1298 Additionally, each egress physical port should receive the same 1299 amount of traffic. 1301 Since this is a capacity test, the configuration and report results 1302 format from 6.2.1.1 and 6.2.1.2 MUST also include: 1304 Configuration: 1305 - The number of physical ingress ports active during the test 1306 - The classication marking (DSCP, VLAN, etc.) for each physical 1307 ingress port 1308 - The traffic rate for stateful traffic and the traffic rate / 1309 mixture for stateful traffic for each physical ingress port 1311 Report results: 1312 - For each egress port, the achieved throughput rate and metrics at 1313 each egress port queue for each ingress port stream. 1315 Example: 1316 - Egress Port 1, SP Queue: throughput and metrics for ingress 1317 streams 1-n 1318 - Egress Port 2, WFQ Queue: throughput and metrics for ingress 1319 streams 1-n 1320 . 1321 . 1322 - Egress Port n, SP Queue: throughput and metrics for ingress 1323 streams 1-n 1324 - Egress Port n, WFQ Queue: throughput and metrics for ingress 1325 streams 1-n 1327 6.3. Shaper tests 1329 A traffic shaper is memory based like a queue, but with the added 1330 intelligence of an active traffic scheduler. The same concepts from 1331 section 6.2 (Queue testing) can be applied to testing network device 1332 shaper. 1334 Again, the tests are divided into two sections; individual shaper 1335 benchmark tests and then full capacity shaper benchmark tests. 1337 6.3.1 Shaper Individual Tests Overview 1339 A traffic shaper generally has three (3) components that can be 1340 configured: 1342 - Ingress Queue bytes 1343 - Shaper Rate, bps 1344 - Burst Committed (Bc) and Burst Excess (Be), bytes 1346 The Ingress Queue holds burst traffic and the shaper then meters 1347 traffic out of the egress port according to the Shaper Rate and 1348 Bc/Be parameters. Shapers generally transmit into policers, so 1349 the idea is for the emitted traffic to conform to the policer's 1350 limits. 1352 6.3.1.1 Testing Shaper with Stateless Traffic 1354 Objective: 1355 Test a shaper by transmitting stateless traffic bursts into the 1356 shaper ingress port and verifying that the egress traffic is shaped 1357 according to the shaper traffic profile. 1359 Test Summary: 1360 The stateless traffic must be burst into the DUT ingress port and 1361 not exceed the Ingress Queue. The burst can be a single burst or 1362 multiple bursts. If multiple bursts are transmitted, then the 1363 Ti (Time interval) must be large enough so that the Shaper Rate is 1364 not exceeded. An example will clarify single and multiple burst 1365 test cases. 1367 In the example, the shaper's ingress and egress ports are both full 1368 duplex Gigabit Ethernet. The Ingress Queue is configured to be 1369 512,000 bytes, the Shaper Rate (SR) = 50 Mbps, and both Bc/Be 1370 configured to be 32,000 bytes. For a single burst test, the 1371 transmitting test device would burst 512,000 bytes maximum into the 1372 ingress port and then stop transmitting. 1374 If a multiple burst test is to be conducted, then the burst bytes 1375 divided by the time interval between the 512,000 byte bursts must 1376 not exceed the Shaper Rate. The time interval (Ti) must adhere to 1377 a similar formula as described in section 6.2.1.1 for queues, namely: 1379 Ti = Ingress Queue x 8 / Shaper Rate 1381 For the example from the previous paragraph, Ti between bursts must 1382 be greater than 82 millisecond (512,000 bytes x 8 / 50,000,000 bps). 1383 This yields an average rate of 50 Mbps so that an Input Queue 1384 would not overflow. 1386 Test Metrics: 1387 - The metrics defined in section 4.1 (LP, OOS, PDV, SR, SBB, SBI) 1388 SHALL be measured at the egress port and recorded. 1390 Procedure: 1391 1. Configure the DUT shaper ingress queue length (QL) and shaper 1392 egress rate parameters (SR, Bc, Be) parameters 1394 2. Configure the tester to generate a stateless traffic burst equal 1395 to QL and an interval equal to Ti (QL in bits/BB) 1397 3. Generate bursts of QL traffic into the DUT and measure the metrics 1398 defined in section 4.1 (LP, OOS, PDV, SR, SBB, SBI) at the egress 1399 port and across the entire Td (default 30 seconds duration) 1401 Report Format: 1402 The Shaper Stateless Traffic individual report MUST contain all 1403 results for each QL/SR test run and a recommended format is as 1404 follows: 1405 ******************************************************** 1406 Test Configuration Summary: Tr, Td 1408 DUT Configuration Summary: Ingress Burst Rate, QL, SR 1410 The results table should contain entries for each test run as 1411 follows,(Test #1 to Test #Tr). 1413 - LP, OOS, PDV, SR, SBB, SBI 1414 ******************************************************** 1416 6.3.1.2 Testing Shaper with Stateful Traffic 1418 Objective: 1419 Test a shaper by transmitting stateful traffic bursts into the shaper 1420 ingress port and verifying that the egress traffic is shaped 1421 according to the shaper traffic profile. 1423 Test Summary: 1424 To provide a more realistic benchmark and to test queues in layer 4 1425 devices such as firewalls, stateful traffic testing is also 1426 recommended for the shaper tests. Stateful traffic tests will also 1427 utilize the Network Delay Emulator (NDE) from the network set-up 1428 configuration in section 2. 1430 The BDP of the TCP test traffic must be calculated as described in 1431 section 6.2.2. To properly stress network buffers and the traffic 1432 shaping function, the cumulative TCP window should exceed the BDP 1433 which will stress the shaper. BDP factors of 1.1 to 1.5 are 1434 recommended, but the values are the discretion of the tester and 1435 should be documented. 1437 The cumulative TCP Window Sizes* (RWND at the receiving end & CWND 1438 at the transmitting end) equates to: 1440 TCP window size* for each connection x number of connections 1442 * as described in section 3 of [RFC6349], the SSB MUST be large 1443 enough to fill the BDP 1445 Example, if the BDP is equal to 256 Kbytes and a connection size of 1446 64Kbytes is used for each connection, then it would require four (4) 1447 connections to fill the BDP and 5-6 connections (over subscribe the 1448 BDP) to stress test the traffic shaping function. 1450 Two types of TCP tests MUST be performed: Bulk Transfer test and 1451 Micro Burst Test Pattern as documented in Appendix B. The Bulk 1452 Transfer Test only bursts during the TCP Slow Start (or Congestion 1453 Avoidance) state, while the Micro Burst test emulates application 1454 layer bursting which may any time during the TCP connection. 1456 Other tests types SHOULD include: Simple Web Site, Complex Web Site, 1457 Business Applications, Email, SMB/CIFS File Copy (which are also 1458 documented in Appendix B). 1460 Test Metrics: 1461 The test results will be recorded per the stateful metrics defined in 1462 section 4.2, primarily the TCP Test Pattern Execution Time (TTPET), 1463 TCP Efficiency, and Buffer Delay. 1465 Procedure: 1466 1. Configure the DUT shaper ingress queue length (QL) and shaper 1467 egress rate parameters (SR, Bc, Be) parameters 1469 2. Configure the tester* to generate a profile of emulated of an 1470 application traffic mixture 1472 - The application mixture MUST be defined in terms of percentage 1473 of the total bandwidth to be tested 1475 - The rate of transmission for each application within the mixture 1476 MUST be also be configurable 1478 *The tester MUST be capable of generating precise TCP test patterns 1479 for each application specified, to ensure repeatable results. 1481 3. Generate application traffic between the ingress (client side) and 1482 egress (server side) ports of the DUT and measure the metrics 1483 (TTPET, TCP Efficiency, and Buffer Delay) per application stream 1484 and at the ingress and egress port (across the entire Td, default 1485 30 seconds duration). 1487 Reporting Format: 1488 The Shaper Stateful Traffic individual report MUST contain all 1489 results for each traffic scheduler and QL/SR test run and a 1490 recommended format is as follows: 1492 ******************************************************** 1493 Test Configuration Summary: Tr, Td 1495 DUT Configuration Summary: Ingress Burst Rate, QL, SR 1497 Application Mixture and Intensities: this is the percent configured 1498 of each application type 1499 The results table should contain entries for each test run with 1500 minimum, maximum, and average per application session as follows, 1501 (Test #1 to Test #Tr) 1503 - Per Application Throughout (bps) and TTPET 1504 - Per Application Bytes In and Bytes Out 1505 - Per Application TCP Efficiency, and Buffer Delay 1506 ******************************************************** 1508 6.3.2 Shaper Capacity Tests 1510 Objective: 1511 The intent of these scalability tests is to verify shaper performance 1512 in a scaled environment with shapers active on multiple queues on 1513 multiple egress physical ports. This test will benchmark the maximum 1514 number of shapers as specified by the device manufacturer. 1516 The following sections provide the specific test scenarios, 1517 procedures, and reporting formats for each shaper capacity test. 1519 6.3.2.1 Single Queue Shaped, All Physical Ports Active 1521 Test Summary: 1522 The first shaper capacity test involves per port shaping, all 1523 physical ports active. Traffic from multiple ingress physical ports 1524 are directed to the same egress physical port and this will cause 1525 oversubscription on the egress physical port. Also, the same amount 1526 of traffic is directed to each egress physical port. 1528 The benchmarking methodology specified in Section 6.3.1 (procedure, 1529 metrics, and reporting format) should be applied here. Since this is 1530 a capacity test, the configuration and report results format from 1531 6.3.1 MUST also include: 1533 Configuration: 1534 - The number of physical ingress ports active during the test 1535 - The classication marking (DSCP, VLAN, etc.) for each physical 1536 ingress port 1537 - The traffic rate for stateful traffic and the traffic rate / 1538 mixture for stateful traffic for each physical ingress port 1539 - The shaped egress ports shaper parameters (QL, SR, Bc, Be) 1541 Report results: 1542 - For each active egress port, the achieved throughput rate and 1543 shaper metrics for each ingress port traffic stream 1545 Example: 1546 - Egress Port 1: throughput and metrics for ingress streams 1-n 1547 - Egress Port n: throughput and metrics for ingress streams 1-n 1549 6.3.2.2 All Queues Shaped, Single Port Active 1551 Test Summary: 1552 The second shaper capacity test is conducted with all queues actively 1553 shaping on a single physical port. The benchmarking methodology 1554 described in per port shaping test (previous section) serves as the 1555 foundation for this. Additionally, each of the SP queues on the 1556 egress physical port is configured with a shaper. For the highest 1557 priority queue, the maximum amount of bandwidth available is limited 1558 by the bandwidth of the shaper. For the lower priority queues, the 1559 maximum amount of bandwidth available is limited by the bandwidth of 1560 the shaper and traffic in higher priority queues. 1562 The benchmarking methodology specified in Section 6.3.1 (procedure, 1563 metrics, and reporting format) should be applied here. Since this is 1564 a capacity test, the configuration and report results format from 1565 6.3.1 MUST also include: 1567 Configuration: 1568 - The number of physical ingress ports active during the test 1569 - The classication marking (DSCP, VLAN, etc.) for each physical 1570 ingress port 1571 - The traffic rate for stateful traffic and the traffic rate/mixture 1572 for stateful traffic for each physical ingress port 1573 - For the active egress port, each shaper queue parameters (QL, SR, 1574 Bc, Be) 1576 Report results: 1577 - For each queue of the active egress port, the achieved throughput 1578 rate and shaper metrics for each ingress port traffic stream 1580 Example: 1581 - Egress Port High Priority Queue: throughput and metrics for 1582 ingress streams 1-n 1583 - Egress Port Lower Priority Queue: throughput and metrics for 1584 ingress streams 1-n 1586 6.3.2.3 All Queues Shaped, All Ports Active 1588 Test Summary: 1589 And for the third shaper capacity test (which is a combination of the 1590 tests in the previous two sections),all queues will be actively 1591 shaping and all physical ports active. 1593 The benchmarking methodology specified in Section 6.3.1 (procedure, 1594 metrics, and reporting format) should be applied here. Since this is 1595 a capacity test, the configuration and report results format from 1596 6.3.1 MUST also include: 1598 Configuration: 1599 - The number of physical ingress ports active during the test 1600 - The classication marking (DSCP, VLAN, etc.) for each physical 1601 ingress port 1602 - The traffic rate for stateful traffic and the traffic rate / 1603 mixture for stateful traffic for each physical ingress port 1604 - For each of the active egress ports, shaper port and per queue 1605 parameters(QL, SR, Bc, Be) 1607 Report results: 1608 - For each queue of each active egress port, the achieved throughput 1609 rate and shaper metrics for each ingress port traffic stream 1611 Example: 1612 - Egress Port 1 High Priority Queue: throughput and metrics for 1613 ingress streams 1-n 1614 - Egress Port 1 Lower Priority Queue: throughput and metrics for 1615 ingress streams 1-n 1616 . 1617 - Egress Port n High Priority Queue: throughput and metrics for 1618 ingress streams 1-n 1619 - Egress Port n Lower Priority Queue: throughput and metrics for 1620 ingress streams 1-n 1622 6.4 Concurrent Capacity Load Tests 1624 As mentioned in the scope of this document, it is impossible to 1625 specify the various permutations of concurrent traffic management 1626 functions that should be tested in a device for capacity testing. 1627 However, some profiles are listed below which may be useful 1628 to test under capacity as well: 1630 - Policers on ingress and queuing on egress 1631 - Policers on ingress and shapers on egress (not intended for a 1632 flow to be policed then shaped, these would be two different 1633 flows tested at the same time) 1634 - etc. 1636 The test procedures and reporting formatting from the previous 1637 sections may be modified to accommodate the capacity test profile. 1639 Appendix A: Open Source Tools for Traffic Management Testing 1641 This framework specifies that stateless and stateful behaviors SHOULD 1642 both be tested. Some open source tools that can be used to 1643 accomplish many of the tests proposed in this framework are: 1644 iperf, netperf (with netperf-wrapper),uperf, TMIX, 1645 TCP-incast-generator, and D-ITG (Distributed Internet Traffic 1646 Generator). 1648 Iperf can generate UDP or TCP based traffic; a client and server must 1649 both run the iperf software in the same traffic mode. The server is 1650 set up to listen and then the test traffic is controlled from the 1651 client. Both uni-directional and bi-directional concurrent testing 1652 are supported. 1654 The UDP mode can be used for the stateless traffic testing. The 1655 target bandwidth, packet size, UDP port, and test duration can be 1656 controlled. A report of bytes transmitted, packets lost, and delay 1657 variation are provided by the iperf receiver. 1659 Iperf (TCP mode), TCP-incast-generator, and D-ITG can be used for 1660 stateful traffic testing to test bulk transfer traffic. The TCP 1661 Window size (which is actually the SSB), the number of connections, 1662 the packet size, TCP port and the test duration can be controlled. 1663 A report of bytes transmitted and throughput achieved are provided 1664 by the iperf sender, while TCP-incast-generator and D-ITG provide 1665 even more statistics. 1667 Netperf is a software application that provides network bandwidth 1668 testing between two hosts on a network. It supports Unix domain 1669 sockets, TCP, SCTP, DLPI and UDP via BSD Sockets. Netperf provides 1670 a number of predefined tests e.g. to measure bulk (unidirectional) 1671 data transfer or request response performance 1672 http://en.wikipedia.org/wiki/Netperf). Netperf-wrapper is a Python 1673 script that runs multiple simultaneous netperf instances and 1674 aggregate the results. 1676 uperf uses a description (or model) of an application mixture and 1677 the tool generates the load according to the model desciptor. uperf 1678 is more flexible than Netperf in it's ability to generate request 1679 / response application behavior within a single TCP connection. The 1680 application model descriptor can be based off of empirical data, but 1681 currently the import of packet captures is not directly supported. 1683 Tmix is another application traffic emulation tool and uses packet 1684 captures directly to create the traffic profile. The packet trace is 1685 'reverse compiled' into a source-level characterization, called a 1686 connection vector, of each TCP connection present in the trace. While 1687 most widely used in ns2 simulation environment, TMix also runs on 1688 Linux hosts. 1690 These open source tool's traffic generation capabilities facilitate 1691 the emulation of the TCP test patterns which are discussed in 1692 Appendix B. 1694 Appendix B: Stateful TCP Test Patterns 1696 This framework recommends at a minimum the following TCP test 1697 patterns since they are representative of real world application 1698 traffic (section 5.2.1 describes some methods to derive other 1699 application-based TCP test patterns). 1701 - Bulk Transfer: generate concurrent TCP connections whose aggregate 1702 number of in-flight data bytes would fill the BDP. Guidelines 1703 from [RFC6349] are used to create this TCP traffic pattern. 1705 - Micro Burst: generate precise burst patterns within a single or 1706 multiple TCP connections(s). The idea is for TCP to establish 1707 equilibrium and then burst application bytes at defined sizes. The 1708 test tool must allow the burst size and burst time interval to be 1709 configurable. 1711 - Web Site Patterns: The HTTP traffic model from 1712 "3GPP2 C.R1002-0 v1.0" is referenced (Table 4.1.3.2.1) to develop 1713 these TCP test patterns. In summary, the HTTP traffic model consists 1714 of the following parameters: 1715 - Main object size (Sm) 1716 - Embedded object size (Se) 1717 - Number of embedded objects per page (Nd) 1718 - Client processing time (Tcp) 1719 - Server processing time (Tsp) 1721 Web site test patterns are illustrated with the following examples: 1723 - Simple Web Site: mimic the request / response and object 1724 download behavior of a basic web site (small company). 1725 - Complex Web Site: mimic the request / response and object 1726 download behavior of a complex web site (ecommerce site). 1728 Referencing the HTTP traffic model parameters , the following table 1729 was derived (by analysis and experimentation) for Simple and Complex 1730 Web site TCP test patterns: 1732 Simple Complex 1733 Parameter Web Site Web Site 1734 ----------------------------------------------------- 1735 Main object Ave. = 10KB Ave. = 300KB 1736 size (Sm) Min. = 100B Min. = 50KB 1737 Max. = 500KB Max. = 2MB 1739 Embedded object Ave. = 7KB Ave. = 10KB 1740 size (Se) Min. = 50B Min. = 100B 1741 Max. = 350KB Max. = 1MB 1743 Number of embedded Ave. = 5 Ave. = 25 1744 objects per page (Nd) Min. = 2 Min. = 10 1745 Max. = 10 Max. = 50 1747 Client processing Ave. = 3s Ave. = 10s 1748 time (Tcp)* Min. = 1s Min. = 3s 1749 Max. = 10s Max. = 30s 1751 Server processing Ave. = 5s Ave. = 8s 1752 time (Tsp)* Min. = 1s Min. = 2s 1753 Max. = 15s Max. = 30s 1755 * The client and server processing time is distributed across the 1756 transmission / receipt of all of the main and embedded objects 1758 To be clear, the parameters in this table are reasonable guidelines 1759 for the TCP test pattern traffic generation. The test tool can use 1760 fixed parameters for simpler tests and mathematical distributions for 1761 more complex tests. However, the test pattern must be repeatable to 1762 ensure that the benchmark results can be reliably compared. 1764 - Inter-active Patterns: While Web site patterns are inter-active 1765 to a degree, they mainly emulate the downloading of various 1766 complexity web sites. Inter-active patterns are more chatty in 1767 nature since there is alot of user interaction with the servers. 1768 Examples include business applications such as Peoplesoft, Oracle 1769 and consumer applications such as Facebook, IM, etc. For the inter- 1770 active patterns, the packet capture technique was used to 1771 characterize some business applications and also the email 1772 application. 1774 In summary, an inter-active application can be described by the 1775 following parameters: 1776 - Client message size (Scm) 1777 - Number of Client messages (Nc) 1778 - Server response size (Srs) 1779 - Number of server messages (Ns) 1780 - Client processing time (Tcp) 1781 - Server processing Time (Tsp) 1782 - File size upload (Su)* 1783 - File size download (Sd)* 1785 * The file size parameters account for attachments uploaded or 1786 downloaded and may not be present in all inter-active applications 1788 Again using packet capture as a means to characterize, the following 1789 table reflects the guidelines for Simple Business Application, 1790 Complex Business Application, eCommerce, and Email Send / Receive: 1792 Simple Complex 1793 Parameter Biz. App. Biz. App eCommerce* Email 1794 -------------------------------------------------------------------- 1795 Client message Ave. = 450B Ave. = 2KB Ave. = 1KB Ave. = 200B 1796 size (Scm) Min. = 100B Min. = 500B Min. = 100B Min. = 100B 1797 Max. = 1.5KB Max. = 100KB Max. = 50KB Max. = 1KB 1799 Number of client Ave. = 10 Ave. = 100 Ave. = 20 Ave. = 10 1800 messages (Nc) Min. = 5 Min. = 50 Min. = 10 Min. = 5 1801 Max. = 25 Max. = 250 Max. = 100 Max. = 25 1803 Client processing Ave. = 10s Ave. = 30s Ave. = 15s Ave. = 5s 1804 time (Tcp)** Min. = 3s Min. = 3s Min. = 5s Min. = 3s 1805 Max. = 30s Max. = 60s Max. = 120s Max. = 45s 1807 Server response Ave. = 2KB Ave. = 5KB Ave. = 8KB Ave. = 200B 1808 size (Srs) Min. = 500B Min. = 1KB Min. = 100B Min. = 150B 1809 Max. = 100KB Max. = 1MB Max. = 50KB Max. = 750B 1811 Number of server Ave. = 50 Ave. = 200 Ave. = 100 Ave. = 15 1812 messages (Ns) Min. = 10 Min. = 25 Min. = 15 Min. = 5 1813 Max. = 200 Max. = 1000 Max. = 500 Max. = 40 1815 Server processing Ave. = 0.5s Ave. = 1s Ave. = 2s Ave. = 4s 1816 time (Tsp)** Min. = 0.1s Min. = 0.5s Min. = 1s Min. = 0.5s 1817 Max. = 5s Max. = 20s Max. = 10s Max. = 15s 1819 Complex Business Application, eCommerce, and Email Send / Receive 1820 (continued): 1822 Simple Complex 1823 Parameter Biz. App. Biz. App eCommerce* Email 1824 -------------------------------------------------------------------- 1825 File size Ave. = 50KB Ave. = 100KB Ave. = N/A Ave. = 100KB 1826 upload (Su) Min. = 2KB Min. = 10KB Min. = N/A Min. = 20KB 1827 Max. = 200KB Max. = 2MB Max. = N/A Max. = 10MB 1829 File size Ave. = 50KB Ave. = 100KB Ave. = N/A Ave. = 100KB 1830 download (Sd) Min. = 2KB Min. = 10KB Min. = N/A Min. = 20KB 1831 Max. = 200KB Max. = 2MB Max. = N/A Max. = 10MB 1833 * eCommerce used a combination of packet capture techniques and 1834 reference traffic flows from "SPECweb2009" (need proper reference) 1835 ** The client and server processing time is distributed across the 1836 transmission / receipt of all of messages. Client processing time 1837 consists mainly of the delay between user interactions (not machine 1838 processing). 1840 And again, the parameters in this table are the guidelines for the 1841 TCP test pattern traffic generation. The test tool can use fixed 1842 parameters for simpler tests and mathematical distributions for more 1843 complex tests. However, the test pattern must be repeatable to 1844 ensure that the benchmark results can be reliably compared. 1846 - SMB/CIFS File Copy: mimic a network file copy, both read and write. 1847 As opposed to FTP which is a bulk transfer and is only flow 1848 controlled via TCP, SMB/CIFS divides a file into application blocks 1849 and utilizes application level handshaking in addition to 1850 TCP flow control. 1852 In summary, an SMB/CIFS file copy can be described by the following 1853 parameters: 1854 - Client message size (Scm) 1855 - Number of client messages (Nc) 1856 - Server response size (Srs) 1857 - Number of Server messages (Ns) 1858 - Client processing time (Tcp) 1859 - Server processing time (Tsp) 1860 - Block size (Sb) 1862 The client and server messages are SMB control messages. The Block 1863 size is the data portion of th file transfer. 1865 Again using packet capture as a means to characterize the following 1866 table reflects the guidelines for SMB/CIFS file copy: 1868 SMB 1869 Parameter File Copy 1870 ------------------------------ 1871 Client message Ave. = 450B 1872 size (Scm) Min. = 100B 1873 Max. = 1.5KB 1874 Number of client Ave. = 10 1875 messages (Nc) Min. = 5 1876 Max. = 25 1877 Client processing Ave. = 1ms 1878 time (Tcp) Min. = 0.5ms 1879 Max. = 2 1880 Server response Ave. = 2KB 1881 size (Srs) Min. = 500B 1882 Max. = 100KB 1883 Number of server Ave. = 10 1884 messages (Ns) Min. = 10 1885 Max. = 200 1886 Server processing Ave. = 1ms 1887 time (Tsp) Min. = 0.5ms 1888 Max. = 2ms 1889 Block Ave. = N/A 1890 Size (Sb)* Min. = 16KB 1891 Max. = 128KB 1893 *Depending upon the tested file size, the block size will be 1894 transferred n number of times to complete the example. An example 1895 would be a 10 MB file test and 64KB block size. In this case 160 1896 blocks would be transferred after the control channel is opened 1897 between the client and server. 1899 7. Security Considerations 1901 Documents of this type do not directly affect the security of the 1902 Internet or of corporate networks as long as benchmarking is not 1903 performed on devices or systems connected to production networks. 1905 Further, benchmarking is performed on a "black-box" basis, relying 1906 solely on measurements observable external to the DUT/SUT. 1908 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 1909 benchmarking purposes. Any implications for network security arising 1910 from the DUT/SUT SHOULD be identical in the lab and in production 1911 networks. 1913 8. IANA Considerations 1915 This document does not REQUIRE an IANA registration for ports 1916 dedicated to the TCP testing described in this document. 1918 9. Acknowledgments 1920 We would like to thank Al Morton for his continuous review and 1921 invaluable input to the document. We would also like to thank 1922 Scott Bradner for providing guidance early in the drafts 1923 conception in the area of benchmarking scope of traffic management 1924 functions. Additionally, we would like to thank Tim Copley for this 1925 original input and David Taht, Gory Erg, Toke Hoiland-Jorgensen for 1926 their review and input for the AQM group. And for the formal reviews 1927 of this document, we would like to thank Gilles Forget, 1928 Vijay Gurbani, Reinhard Schrage, and Bhuvaneswaran Vengainathan 1930 10. References 1932 10.1. Normative References 1934 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1935 Requirement Levels", BCP 14, RFC2119, March 1997. 1937 [RFC1242] S. Bradner, "Benchmarking Terminology for Network 1938 Interconnection Devices," RFC1242 July 1991 1940 [RFC2544] S. Bradner, "Benchmarking Methodology for Network 1941 Interconnect Devices," RFC2544 March 1999 1943 [RFC3148] M. Mathis et al., A Framework for Defining Empirical 1944 Bulk Transfer Capacity Metrics," RFC3148 July 2001 1946 [RFC5481] A. Morton et al., "Packet Delay Variation Applicability 1947 Statement," RFC5481 March 2009 1949 [RFC6703] A. Morton et al., "Reporting IP Network Performance 1950 Metrics: Different Points of View." RFC 6703 August 2012 1952 [RFC2680] G. Almes et al., "A One-way Packet Loss Metric for IPPM," 1953 RFC2680 September 1999 1955 [RFC4689] S. Poretsky et al., "Terminology for Benchmarking 1956 Network-layer Traffic Control Mechanisms," RFC4689, 1957 October 2006 1959 [RFC4737] A. Morton et al., "Packet Reordering Metrics," RFC4737, 1960 February 2006 1962 [RFC4115] O. Aboul-Magd et al., "A Differentiated Service Two-Rate, 1963 Three-Color Marker with Efficient Handling of in-Profile Traffic." 1964 RFC4115 July 2005 1966 [RFC6349] Barry Constantine et al., "Framework for TCP Throughput 1967 Testing," RFC6349, August 2011 1969 10.2. Informative References 1971 [RFC2697] J. Heinanen et al., "A Single Rate Three Color Marker," 1972 RFC2697, September 1999 1974 [RFC2698] J. Heinanen et al., "A Two Rate Three Color Marker, " 1975 RFC2698, September 1999 1977 [AQM-RECO] Fred Baker et al., "IETF Recommendations Regarding 1978 Active Queue Management," August 2014, 1979 https://datatracker.ietf.org/doc/draft-ietf-aqm- 1980 recommendation/ 1982 [MEF-10.2] "MEF 10.2: Ethernet Services Attributes Phase 2," October 1983 2009, http://metroethernetforum.org/PDF_Documents/ 1984 technical-specifications/MEF10.2.pdf 1986 [MEF-12.1] "MEF 12.1: Carrier Ethernet Network Architecture 1987 Framework -- 1988 Part 2: Ethernet Services Layer - Base Elements," April 1989 2010, https://www.metroethernetforum.org/Assets/Technical 1990 _Specifications/PDF/MEF12.1.pdf 1992 [MEF-26] "MEF 26: External Network Network Interface (ENNI) - 1993 Phase 1,"January 2010, http://www.metroethernetforum.org 1994 /PDF_Documents/technical-specifications/MEF26.pdf 1996 [MEF-14] "Abstract Test Suite for Traffic Management Phase 1, 1997 https://www.metroethernetforum.org/Assets 1998 /Technical_Specifications/PDF/MEF_14.pdf 2000 [MEF-19] "Abstract Test Suite for UNII Type 1", 2001 https://www.metroethernetforum.org/Assets 2002 /Technical_Specifications/PDF/MEF_19.pdf 2004 [MEF-37] "Abstract Test Suite for ENNI", 2005 https://www.metroethernetforum.org/Assets 2006 /Technical_Specifications/PDF/MEF_37.pdf 2008 Authors' Addresses 2010 Barry Constantine 2011 JDSU, Test and Measurement Division 2012 Germantown, MD 20876-7100, USA 2013 Phone: +1 240 404 2227 2014 Email: barry.constantine@jdsu.com 2016 Ram Krishnan 2017 Brocade Communications 2018 San Jose, 95134, USA 2019 Phone: +001-408-406-7890 2020 Email: ramk@brocade.com