idnits 2.17.1 draft-ietf-bmwg-traffic-management-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == The page length should not exceed 58 lines per page, but there was 8 longer pages, the longest (page 25) being 78 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 77 instances of too long lines in the document, the longest one being 8 characters in excess of 72. ** There are 4 instances of lines with control characters in the document. == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- Couldn't find a document date in the document -- date freshness check skipped. Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: '1' is defined on line 1335, but no explicit reference was found in the text == Unused Reference: '2' is defined on line 1338, but no explicit reference was found in the text == Unused Reference: 'RFC2234' is defined on line 1345, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2234 (ref. '2') (Obsoleted by RFC 4234) -- Duplicate reference: RFC2119, mentioned in 'RFC2119', was also mentioned in '1'. -- Duplicate reference: RFC2234, mentioned in 'RFC2234', was also mentioned in '2'. ** Obsolete normative reference: RFC 2234 (Obsoleted by RFC 4234) Summary: 4 errors (**), 0 flaws (~~), 6 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group B. Constantine 2 Internet Draft JDSU 3 Intended status: Informational T. Copley 4 Expires: January 2015 Level-3 5 August 10, 2014 R. Krishnan 6 Brocade Communications 8 Traffic Management Benchmarking 9 draft-ietf-bmwg-traffic-management-00.txt 11 Status of this Memo 13 This Internet-Draft is submitted in full conformance with the 14 provisions of BCP 78 and BCP 79. 16 Internet-Drafts are working documents of the Internet Engineering 17 Task Force (IETF). Note that other groups may also distribute 18 working documents as Internet-Drafts. The list of current Internet- 19 Drafts is at http://datatracker.ietf.org/drafts/current/. 21 Internet-Drafts are draft documents valid for a maximum of six months 22 and may be updated, replaced, or obsoleted by other documents at any 23 time. It is inappropriate to use Internet-Drafts as reference 24 material or to cite them other than as "work in progress." 26 This Internet-Draft will expire on January 10, 2015. 28 Copyright Notice 30 Copyright (c) 2014 IETF Trust and the persons identified as the 31 document authors. All rights reserved. 33 This document is subject to BCP 78 and the IETF Trust's Legal 34 Provisions Relating to IETF Documents 35 (http://trustee.ietf.org/license-info) in effect on the date of 36 publication of this document. Please review these documents 37 carefully, as they describe your rights and restrictions with respect 38 to this document. Code Components extracted from this document must 39 include Simplified BSD License text as described in Section 4.e of 40 the Trust Legal Provisions and are provided without warranty as 41 described in the Simplified BSD License. 43 Abstract 45 This framework describes a practical methodology for benchmarking the 46 traffic management capabilities of networking devices (i.e. policing, 47 shaping, etc.). The goal is to provide a repeatable test method that 48 objectively compares performance of the device's traffic management 49 capabilities and to specify the means to benchmark traffic management 50 with representative application traffic. 52 Table of Contents 54 1. Introduction...................................................4 55 1.1. Traffic Management Overview...............................4 56 1.2. DUT Lab Configuration and Testing Overview................5 57 2. Conventions used in this document..............................7 58 3. Scope and Goals................................................8 59 4. Traffic Benchmarking Metrics...................................9 60 4.1. Metrics for Stateless Traffic Tests.......................9 61 4.2. Metrics for Stateful Traffic Tests.......................11 62 5. Tester Capabilities...........................................11 63 5.1. Stateless Test Traffic Generation........................11 64 5.2. Stateful Test Pattern Generation.........................12 65 5.2.1. TCP Test Pattern Definitions........................13 66 6. Traffic Benchmarking Methodology..............................15 67 6.1. Policing Tests...........................................15 68 6.1.1 Policer Individual Tests................................15 69 6.1.2 Policer Capacity Tests..............................16 70 6.1.2.1 Maximum Policers on Single Physical Port..........16 71 6.1.2.2 Single Policer on All Physical Ports..............17 72 6.1.2.3 Maximum Policers on All Physical Ports............17 73 6.2. Queue/Scheduler Tests....................................17 74 6.2.1 Queue/Scheduler Individual Tests........................17 75 6.2.1.1 Testing Queue/Scheduler with Stateless Traffic....17 76 6.2.1.2 Testing Queue/Scheduler with Stateful Traffic.....18 77 6.2.2 Queue / Scheduler Capacity Tests......................19 78 6.2.2.1 Multiple Queues / Single Port Active..............19 79 6.2.2.1.1 Strict Priority on Egress Port..................19 80 6.2.2.1.2 Strict Priority + Weighted Fair Queue (WFQ).....19 81 6.2.2.2 Single Queue per Port / All Ports Active..........19 82 6.2.2.3 Multiple Queues per Port, All Ports Active........20 83 6.3. Shaper tests.............................................20 84 6.3.1 Shaper Individual Tests...............................20 85 6.3.1.1 Testing Shaper with Stateless Traffic.............20 86 6.3.1.2 Testing Shaper with Stateful Traffic..............21 87 6.3.2 Shaper Capacity Tests.................................22 88 6.3.2.1 Single Queue Shaped, All Physical Ports Active....22 89 6.3.2.2 All Queues Shaped, Single Port Active.............22 90 6.3.2.3 All Queues Shaped, All Ports Active...............22 91 6.4. Concurrent Capacity Load Tests...........................24 92 7. Security Considerations.......................................24 93 8. IANA Considerations...........................................24 94 9. Conclusions...................................................24 95 10. References...................................................24 96 10.1. Normative References....................................25 97 10.2. Informative References..................................25 98 11. Acknowledgments..............................................25 100 1. Introduction 102 Traffic management (i.e. policing, shaping, etc.) is an increasingly 103 important component when implementing network Quality of Service 104 (QoS). There is currently no framework to benchmark these features 105 although some standards address specific areas. This draft provides 106 a framework to conduct repeatable traffic management benchmarks for 107 devices and systems in a lab environment. 109 Specifically, this framework defines the methods to characterize the 110 capacity of the following traffic management features in network 111 devices; classification, policing, queuing / scheduling, and 112 traffic shaping. 114 This benchmarking framework can also be used as a test procedure to 115 assist in the tuning of traffic management parameters before service 116 activation. In addition to Layer 2/3 benchmarking, Layer 4 test 117 patterns are proposed by this draft in order to more realistically 118 benchmark end-user traffic. 120 1.1. Traffic Management Overview 122 In general, a device with traffic management capabilities performs 123 the following functions: 125 - Traffic classification: identifies traffic according to various 126 configuration rules (i.e. VLAN, DSCP, etc.) and marks this traffic 127 internally to the network device. Multiple external priorities 128 (DSCP, 802.1p, etc.) can map to the same priority in the device. 129 - Traffic policing: limits the rate of traffic that enters a network 130 device according to the traffic classification. If the traffic 131 exceeds the contracted limits, the traffic is either dropped or 132 remarked and sent onto to the next network device 133 - Traffic Scheduling: provides traffic classification within the 134 network device by directing packets to various types of queues and 135 applies a dispatching algorithm to assign the forwarding sequence 136 of packets 137 - Traffic shaping: a traffic control technique that actively buffers 138 and meters the output rate in an attempt to adapt bursty traffic 139 to the configured limits 140 - Active Queue Management (AQM): monitors the status of internal 141 queues and actively drops (or re-marks) packets, which causes hosts 142 using congestion-aware protocols to back-off and in turn can 143 alleviate queue congestion. Note that AQM is outside of the scope 144 of this testing framework. 146 The following diagram is a generic model of the traffic management 147 capabilities within a network device. It is not intended to 148 represent all variations of manufacturer traffic management 149 capabilities, but provide context to this test framework. 151 |----------| |----------------| |--------------| |----------| 152 | | | | | | | | 153 |Interface | |Ingress Actions | |Egress Actions| |Interface | 154 |Input | |(classification,| |(scheduling, | |Output | 155 |Queues | | marking, | | shaping, | |Queues | 156 | |-->| policing or |-->| active queue |-->| | 157 | | | shaping) | | management | | | 158 | | | | | re-marking) | | | 159 |----------| |----------------| |--------------| |----------| 161 Figure 1: Generic Traffic Management capabilities of a Network Device 163 Ingress actions such as classification are defined in RFC 4689 and 164 include IP addresses, port numbers, DSCP, etc. In terms of marking, 165 RFC 2697 and RFC 2698 define a single rate and dual rate, three color 166 marker, respectively. 168 The MEF specifies policing and shaping in terms of Ingress and Egress 169 Subscriber/Provider Conditioning Functions in MEF12.1; Ingress and 170 Bandwidth Profile attributes in MEF 10.2 and MEF 26. 172 1.2 DUT Lab Configuration and Testing Overview 174 The following is the description of the lab set-up for the traffic 175 management tests: 177 +--------------+ +-------+ +----------+ +-----------+ 178 | Transmitting | | | | | | Receiving | 179 | Test Host | | | | | | Test Host | 180 | |-----| DUT |---->| Network |--->| | 181 | | | | | Delay | | | 182 | | | | | Emulator | | | 183 | |<----| |<----| |<---| | 184 | | | | | | | | 185 +--------------+ +-------+ +----------+ +-----------+ 187 As shown in the test diagram, the framework supports uni-directional 188 and bi-directional traffic management tests. 190 This testing framework describes the tests and metrics for each of 191 the following traffic management functions: 192 - Policing 193 - Queuing / Scheduling 194 - Shaping 196 The tests are divided into individual tests and rated capacity tests. 197 The individual tests are intended to benchmark the traffic management 198 functions according to the metrics defined in Section 4. The 199 capacity tests verify traffic management functions under full load. 200 This involves concurrent testing of multiple interfaces with the 201 specific traffic management function enabled, and doing so to the 202 capacity limit of each interface. 204 As an example: a device is specified to be capable of shaping on all 205 of it's egress ports. The individual test would first be conducted to 206 benchmark the advertised shaping function against the metrics defined 207 in section 4. Then the capacity test would be executed to test the 208 shaping function concurrently on all interfaces and with maximum 209 traffic load. 211 The Network Delay Emulator (NDE) is a requirement for the TCP 212 stateful tests, which require network delay to allow TCP to fully 213 open the TCP window. Also note that the Network Delay Emulator (NDE) 214 should be passive in nature such as a fiber spool. This is 215 recommended to eliminate the potential effects that an active delay 216 element (i.e. test impairment generator) may have on the test flows. 217 In the case that a fiber spool is not practical due to the desired 218 latency, an active NDE must be independently verified to be capable 219 of adding the configured delay without loss. In other words, the 220 DUT would be removed and the NDE performance benchmarked 221 independently. 223 Note the NDE should be used in "full pipe" delay mode. Most NDEs 224 allow for per flow delay actions, emulating QoS prioritization. For 225 this framework, the NDE's sole purpose is simply to add delay to all 226 packets (emulate network latency). So to benchmark the performance of 227 the NDE, maximum offered load should be tested against the following 228 frame sizes: 128, 256, 512, 768, 1024, 1500,and 9600 bytes. The delay 229 accuracy at each of these packet sizes can then be used to calibrate 230 the range of expected BDPs for the TCP stateful tests. 232 2. Conventions used in this document 234 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 235 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 236 document are to be interpreted as described in RFC 2119 [RFC2119]. 238 The following acronyms are used: 240 BB: Bottleneck Bandwidth 242 BDP: Bandwidth Delay Product 244 BSA: Burst Size Achieved 246 CBS: Committed Burst Size 248 CIR: Committed Information Rate 250 DUT: Device Under Test 252 EBS: Excess Burst Size 254 EIR: Excess Information Rate 256 NDE: Network Delay Emulator 258 SP: Strict Priority Queuing 260 QL: Queue Length 262 QoS: Quality of Service 264 RED: Random Early Discard 266 RTT: Round Trip Time 268 SBB: Shaper Burst Bytes 270 SBI: Shaper Burst Interval 272 SR: Shaper Rate 274 SSB: Send Socket Buffer 276 Tc: CBS Time Interval 278 Te: EBS Time Interval 279 Ti Transmission Interval 281 TTP: TCP Test Pattern 283 TTPET: TCP Test Pattern Execution Time 285 WRED: Weighted Random Early Discard 287 3. Scope and Goals 289 The scope of this work is to develop a framework for benchmarking and 290 testing the traffic management capabilities of network devices in the 291 lab environment. These network devices may include but are not 292 limited to: 293 - Switches (including Layer 2/3 devices) 294 - Routers 295 - Firewalls 296 - General Layer 4-7 appliances (Proxies, WAN Accelerators, etc.) 298 Essentially, any network device that performs traffic management as 299 defined in section 1.1 can be benchmarked or tested with this 300 framework. 302 The primary goal is to assess the maximum forwarding performance that 303 a network device can sustain without dropping or impairing packets, 304 or compromising the accuracy of multiple instances of traffic 305 management functions. This is the benchmark for comparison between 306 devices. 308 Within this framework, the metrics are defined for each traffic 309 management test but do not include pass / fail criterion, which is 310 not within the charter of BMWG. This framework provides the test 311 methods and metrics to conduct repeatable testing, which will 312 provide the means to compare measured performance between DUTs. 314 As mentioned in section 1.2, this framework describes the individual 315 tests and metrics for several management functions. It is also within 316 scope that this framework will benchmark each function in terms of 317 overall rated capacity. This involves concurrent testing of multiple 318 interfaces with the specific traffic management function enabled, up 319 to the capacity limit of each interface. 321 It is not within scope of this framework to specify the procedure for 322 testing multiple traffic management functions concurrently. The 323 multitudes of possible combinations is almost unbounded and the 324 ability to identify functional "break points" would be most times 325 impossible. 327 However, section 6.4 provides suggestions for some profiles of 328 concurrent functions that would be useful to benchmark. The key 329 requirement for any concurrent test function is that tests must 330 produce reliable and repeatable results. 332 Also, it is not within scope to perform conformance testing. Tests 333 defined in this framework benchmark the traffic management functions 334 according to the metrics defined in section 4 and do not address any 335 conformance to standards related to traffic management. Traffic 336 management specifications largely do not exist and this is a prime 337 driver for this framework; to provide an objective means to compare 338 vendor traffic management functions. 340 Another goal is to devise methods that utilize flows with 341 congestion-aware transport (TCP) as part of the traffic load and 342 still produce repeatable results in the isolated test environment. 343 This framework will derive stateful test patterns (TCP or 344 application layer) that can also be used to further benchmark the 345 performance of applicable traffic management techniques such as 346 queuing / scheduling and traffic shaping. In cases where the 347 network device is stateful in nature (i.e. firewall, etc.), 348 stateful test pattern traffic is important to test along with 349 stateless, UDP traffic in specific test scenarios (i.e. 350 applications using TCP transport and UDP VoIP, etc.) 352 And finally, this framework will provide references to open source 353 tools that can be used to provide stateless and/or stateful 354 traffic generation emulation. 356 4. Traffic Benchmarking Metrics 358 The metrics to be measured during the benchmarks are divided into two 359 (2) sections: packet layer metrics used for the stateless traffic 360 testing and segment layer metrics used for the stateful traffic 361 testing. 363 4.1. Metrics for Stateless Traffic Tests 365 For the stateless traffic tests, the metrics are defined at the layer 366 3 packet level versus layer 2 packet level for consistency. 368 Stateless traffic measurements require that sequence number and 369 time-stamp be inserted into the payload for lost packet analysis. 370 Delay analysis may be achieved by insertion of timestamps directly 371 into the packets or timestamps stored elsewhere (packet captures). 372 This framework does not specify the packet format to carry sequence 373 number or timing information. However, RFC 4689 provides 374 recommendations for sequence tracking along with definitions of 375 in-sequence and out-of-order packets. 377 The following are the metrics to be used during the stateless traffic 378 benchmarking components of the tests: 380 - Burst Size Achieved (BSA): for the traffic policing and network 381 queue tests, the tester will be configured to send bursts to test 382 either the Committed Burst Size (CBS) or Excess Burst Size (EBS) of 383 a policer or the queue / buffer size configured in the DUT. The 384 Burst Size Achieved metric is a measure of the actual burst size 385 received at the egress port of the DUT with no lost packets. As an 386 example, the configured CBS of a DUT is 64KB and after the burst test, 387 only a 63 KB can be achieved without packet loss. Then 63KB is the 388 BSA. Also, the average Packet Delay Variation (PDV see below) as 389 experienced by the packets sent at the BSA burst size should be 390 recorded. 392 - Lost Packets (LP): For all traffic management tests, the tester will 393 transmit the test packets into the DUT ingress port and the number of 394 packets received at the egress port will be measured. The difference 395 between packets transmitted into the ingress port and received at the 396 egress port is the number of lost packets as measured at the egress 397 port. These packets must have unique identifiers such that only the 398 test packets are measured. RFC 4737 and RFC 2680 describe the need to 399 to establish the time threshold to wait before a packet is declared 400 as lost. packet as lost, and this threshold MUST be reported with 401 the results. 403 - Out of Sequence (OOS): in additions to the LP metric, the test 404 packets must be monitored for sequence and the out-of-sequence (OOS) 405 packets. RFC 4689 defines the general function of sequence tracking, as 406 well as definitions for in-sequence and out-of-order packets. Out-of- 407 order packets will be counted per RFC 4737 and RFC 2680. 409 - Packet Delay (PD): the Packet Delay metric is the difference between 410 the timestamp of the received egress port packets and the packets 411 transmitted into the ingress port and specified in RFC 2285. 413 - Packet Delay Variation (PDV): the Packet Delay Variation metric is 414 the variation between the timestamp of the received egress port 415 packets and specified in RFC 5481. 417 - Shaper Rate (SR): the Shaper Rate is only applicable to the 418 traffic shaping tests. The SR represents the average egress output 419 rate (bps) over the test interval. 421 - Shaper Burst Bytes (SBB): the Shaper Burst Bytes is 422 only applicable to the traffic shaping tests. A traffic shaper will 423 emit packets in different size "trains" (bytes back-to-back). This 424 metric characterizes the method by which the shaper emits traffic. 425 Some shapers transmit larger bursts per interval, while other shapers 426 may transmit a single frame at the CIR rate (two extreme examples). 428 - Shaper Burst Interval(SBI): the interval is only applicable to the 429 traffic shaping tests and again is the time between a shaper emitted 430 bursts. 432 4.2. Metrics for Stateful Traffic Tests 434 The stateful metrics will be based on RFC 6349 TCP metrics and will 435 include: 437 - TCP Test Pattern Execution Time (TTPET): RFC 6349 defined the TCP 438 Transfer Time for bulk transfers, which is simply the measured time 439 to transfer bytes across single or concurrent TCP connections. The 440 TCP test patterns used in traffic management tests will include bulk 441 transfer and interactive applications. The interactive patterns include 442 instances such as HTTP business applications, database applications, 443 etc. The TTPET will be the measure of the time for a single execution 444 of a TCP Test Pattern (TTP). Average, minimum, and maximum times will 445 be measured or calculated. 447 An example would be an interactive HTTP TTP session which should take 448 5 seconds on a GigE network with 0.5 millisecond latency. During ten (10) 449 executions of this TTP, the TTPET results might be: average of 6.5 450 seconds, minimum of 5.0 seconds, and maximum of 7.9 seconds. 452 - TCP Efficiency: after the execution of the TCP Test Pattern, TCP 453 Efficiency represents the percentage of Bytes that were not 454 retransmitted. 456 Transmitted Bytes - Retransmitted Bytes 458 TCP Efficiency % = --------------------------------------- X 100 460 Transmitted Bytes 462 Transmitted Bytes are the total number of TCP Bytes to be transmitted 463 including the original and the retransmitted Bytes. These retransmitted 464 bytes should be recorded from the sender's TCP/IP stack perspective, 465 to avoid any misinterpretation that a reordered packet is a retransmitted 466 packet (as may be the case with packet decode interpretation). 468 - Buffer Delay: represents the increase in RTT during a TCP test 469 versus the baseline DUT RTT (non congested, inherent latency). RTT 470 and the technique to measure RTT (average versus baseline) are defined 471 in RFC 6349. Referencing RFC 6349, the average RTT is derived from 472 the total of all measured RTTs during the actual test sampled at every 473 second divided by the test duration in seconds. 475 Total RTTs during transfer 476 Average RTT during transfer = ----------------------------- 477 Transfer duration in seconds 479 Average RTT during Transfer - Baseline RTT 480 Buffer Delay % = ------------------------------------------ X 100 481 Baseline RTT 483 Note that even though this was not explicitly stated in RFC 6349, 484 retransmitted packets should not be used in RTT measurements. 486 Also, the test results should record the average RTT in millisecond 487 across the entire test duration and number of samples. 489 5. Tester Capabilities 491 The testing capabilities of the traffic management test environment 492 are divided into two (2) sections: stateless traffic testing and 493 stateful traffic testing 495 5.1. Stateless Test Traffic Generation 497 The test set must be capable of generating traffic at up to the 498 link speed of the DUT. The test set must be calibrated to verify 499 that it will not drop any packets. The test set's inherent PD and PDV 500 must also be calibrated and subtracted from the PD and PDV metrics. 501 The test set must support the encapsulation to be tested such as 502 VLAN, Q-in-Q, MPLS, etc. Also, the test set must allow control of 503 the classification techniques defined in RFC 4689 (i.e. IP address, 504 DSCP, TOS, etc classification). 506 The open source tool "iperf" can be used to generate stateless UDP 507 traffic and is discussed in Appendix A. Since iperf is a software 508 based tool, there will be performance limitations at higher link 509 speeds (e.g. GigE, 10 GigE, etc.). Careful calibration of any test 510 environment using iperf is important. At higher link speeds, it is 511 recommended to use hardware based packet test equipment. 513 5.1.1 Burst Hunt with Stateless Traffic 515 A central theme for the traffic management tests is to benchmark the 516 specified burst parameter of traffic management function, since burst 517 parameters of SLAs are specified in bytes. For testing efficiency, 518 it is recommended to include a burst hunt feature, which automates 519 the manual process of determining the maximum burst size which can 520 be supported by a traffic management function. 522 The burst hunt algorithm should start at the target burst size (maximum 523 burst size supported by the traffic management function) and will send 524 single bursts until it can determine the largest burst that can pass 525 without loss. If the target burst size passes, then the test is 526 complete. The hunt aspect occurs when the target burst size is not 527 achieved; the algorithm will drop down to a configured minimum burst 528 size and incrementally increase the burst until the maximum burst 529 supported by the DUT is discovered. The recommended granularity 530 of the incremental burst size increase is 1 KB. 532 Optionally for a policer function and if the burst size passes, the burst 533 should be increased by increments of 1 KB to verify that the policer is 534 truly configured properly (or enabled at all). 536 5.2. Stateful Test Pattern Generation 538 The TCP test host will have many of the same attributes as the TCP test 539 host defined in RFC 6349. The TCP test device may be a standard 540 computer or a dedicated communications test instrument. In both cases, 541 it must be capable of emulating both a client and a server. 543 For any test using stateful TCP test traffic, the Network Delay Emulator 544 (NDE function from the lab set-up diagram) must be used in order to provide a 545 meaningful BDP. As referenced in section 2, the target traffic rate and 546 configured RTT must be verified independently using just the NDE for all 547 stateful tests (to ensure the NDE can delay without loss). 549 The TCP test host must be capable to generate and receive stateful TCP 550 test traffic at the full link speed of the DUT. As a general rule of 551 thumb, testing TCP Throughput at rates greater than 500 Mbps may require 552 high performance server hardware or dedicated hardware based test tools. 554 The TCP test host must allow adjusting both Send and Receive Socket 555 Buffer sizes. The Socket Buffers must be large enough to fill the BDP 556 for bulk transfer TCP test application traffic. 558 Measuring RTT and retransmissions per connection will generally require 559 a dedicated communications test instrument. In the absence of 560 dedicated hardware based test tools, these measurements may need to be 561 conducted with packet capture tools, i.e. conduct TCP Throughput 562 tests and analyze RTT and retransmissions in packet captures. 564 The TCP implementation used by the test host must be specified in the 565 test results (i.e. OS version, i.e. LINUX OS kernel using TCP New Reno, 566 TCP options supported, etc.). 568 While RFC 6349 defined the means to conduct throughput tests of TCP bulk 569 transfers, the traffic management framework will extend TCP test 570 execution into interactive TCP application traffic. Examples include 571 email, HTTP, business applications, etc. This interactive traffic is 572 bi-directional and can be chatty. 574 The test device must not only support bulk TCP transfer application 575 traffic but also chatty traffic. A valid stress test SHOULD include 576 both traffic types. This is due to the non-uniform, bursty nature of 577 chatty applications versus the relatively uniform nature of bulk 578 transfers (the bulk transfer smoothly stabilizes to equilibrium state 579 under lossless conditions). 581 While iperf is an excellent choice for TCP bulk transfer testing, the 582 open source tool "Flowgrind" (referenced in Appendix A) is 583 client-server based and emulates interactive applications at the TCP 584 layer. As with any software based tool, the performance must be 585 qualified to the link speed to be tested. Hardware-based test equipment 586 should be considered for reliable results at higher links speeds (e.g. 587 1 GigE, 10 GigE). 589 5.2.1. TCP Test Pattern Definitions 591 As mentioned in the goals of this framework, techniques are defined 592 to specify TCP traffic test patterns to benchmark traffic 593 management technique(s) and produce repeatable results. Some 594 network devices such as firewalls, will not process stateless test 595 traffic which is another reason why stateful TCP test traffic must 596 be used. 598 An application could be fully emulated up to Layer 7, however this 599 framework proposes that stateful TCP test patterns be used in order 600 to provide granular and repeatable control for the benchmarks. The 601 following diagram illustrates a simple Web Browsing application 602 (HTTP). 604 GET url 606 Client ------------------------> Web 608 Web 200 OK 100ms | 610 Browser <------------------------ Server 611 In this example, the Client Web Browser (Client) requests a URL and 612 then the Web Server delivers the web page content to the Client 613 (after a Server delay of 100 millisecond). This asynchronous, "request/ 614 response" behavior is intrinsic to most TCP based applications such 615 as Email (SMTP), File Transfers (FTP and SMB), Database (SQL), Web 616 Applications (SOAP), REST, etc. The impact to the network elements is 617 due to the multitudes of Clients and the variety of bursty traffic, 618 which stresses traffic management functions. The actual emulation of 619 the specific application protocols is not required and TCP test 620 patterns can be defined to mimic the application network traffic flows 621 and produce repeatable results. 623 There are two (2) techniques recommended by this framework to develop 624 standard TCP test patterns for traffic management benchmarking. 626 The first technique involves modeling, which have been described in 627 "3GPP2 C.R1002-0 v1.0" and describe the behavior of HTTP, FTP, and 628 WAP applications at the TCP layer. The models have been defined 629 with various mathematical distributions for the Request/Response 630 bytes and inter-request gap times. The Flowgrind tool (Appendix A) 631 supports many of the distributions and is a good choice as long as 632 the processing limits of the server platform are taken into 633 consideration. 635 The second technique is to conduct packet captures of the 636 applications to test and then to statefully play the application back 637 at the TCP layer. The TCP playback includes the request byte size, 638 response byte size, and inter-message gaps at both the client and the 639 server. The advantage of this method is that very realistic test 640 patterns can be defined based on real world application traffic. 642 This framework does not specify a fixed set of TCP test patterns, but 643 does provide recommended test cases in Appendix B. Some of these examples 644 reflect those specified in "draft-ietf-bmwg-ca-bench-meth-04" which 645 suggests traffic mixes for a variety of representative application 646 profiles. Other examples are simply well known application traffic 647 types. 649 6. Traffic Benchmarking Methodology 651 The traffic benchmarking methodology uses the test set-up from 652 section 2 and metrics defined in section 4. Each test should be run 653 for a minimum test time of 5 minutes. 655 Each test should compare the network device's internal statistics 656 (available via command line management interface, SNMP, etc.) to the 657 measured metrics defined in section 4. This evaluates the accuracy 658 of the internal traffic management counters under individual test 659 conditions and capacity test conditions that are defined in each 660 subsection. 662 6.1. Policing Tests 664 The intent of the policing tests is to verify the policer performance 665 (i.e. CIR-CBS and EIR-EBS parameters). The tests will verify that the 666 network device can handle the CIR with CBS and the EIR with EBS and 667 will use back-back packet testing concepts from RFC 2544 (but adapted 668 to burst size algorithms and terminology). Also MEF-14,19,37 provide 669 some basis for specific components of this test. The burst hunt 670 algorithm defined in section 5.1.1 can also be used to automate the 671 measurement of the CBS value. 673 The tests are divided into two (2) sections; individual policer 674 tests and then full capacity policing tests. It is important to 675 benchmark the basic functionality of the individual policer then 676 proceed into the fully rated capacity of the device. This capacity may 677 include the number of policing policies per device and the number of 678 policers simultaneously active across all ports. 680 6.1.1 Policer Individual Tests 682 Policing tests should use stateless traffic. Stateful TCP test traffic 683 will generally be adversely affected by a policer in the absence of 684 traffic shaping. So while TCP traffic could be used, it is more 685 accurate to benchmark a policer with stateless traffic. 687 The policer test shall test a policer as defined by RFC 4115 or 688 MEF 10.2, depending upon the equipment's specification. As an example 689 for RFC 4115, consider a CBS and EBS of 64KB and CIR and EIR of 690 100 Mbps on a 1GigE physical link (in color-blind mode). A stateless 691 traffic burst of 64KB would be sent into the policer at the GigE rate. 692 This equates to approximately a 0.512 millisecond burst time (64 KB at 693 1 GigE). The traffic generator must space these bursts to ensure that 694 the aggregate throughput does not exceed the CIR. The Ti between the 695 bursts would equal CBS * 8 / CIR = 5.12 millisecond in this example. 697 The metrics defined in section 4.1 shall be measured at the egress 698 port and recorded. 700 In addition to verifying that the policer allows the specified CBS 701 and EBS bursts to pass, the policer test must verify that the policer 702 will police at the specified CBS/EBS values. 704 For this portion of the test, the CBS/EBS value should be incremented 705 by 1000 bytes higher than the configured CBS and that the egress port 706 measurements must show that the excess packets are dropped. 708 Additional tests beyond the simple color-blind example might include: 709 color-aware mode, configurations where EIR is greater than CIR, etc. 711 6.1.2 Policer Capacity Tests 713 The intent of the capacity tests is to verify the policer performance 714 in a scaled environment with multiple ingress customer policers on 715 multiple physical ports. This test will benchmark the maximum number 716 of active policers as specified by the device manufacturer. 718 As an example, a Layer 2 switching device may specify that each of the 719 32 physical ports can be policed using a pool of policing service 720 policies. The device may carry a single customer's traffic on each 721 physical port and a single policer is instantiated per physical port. 722 Another possibility is that a single physical port may carry multiple 723 customers, in which case many customer flows would be policed 724 concurrently on an individual physical port (separate policers per 725 customer on an individual port). 727 The specified policing function capacity is generally expressed in 728 terms of the number of policers active on each individual physical 729 port as well as the number of unique policer rates that are utilized. 730 For all of the capacity tests, the benchmarking methodology described 731 in Section 6.1.1 for a single policer should be applied to each of 732 the physical port policers. 734 6.1.2.1 Maximum Policers on Single Physical Port 736 The first policer capacity test will benchmark a single physical port, 737 maximum policers on that physical port. 739 Assume multiple categories of ingress policers at rates r1, r2,...rn. 740 There are multiple customers on a single physical port. Each customer 741 could be represented by a single tagged vlan, double tagged vlan, 742 VPLS instance etc. Each customer is mapped to a different policer. Each 743 of the policers can be of rates r1, r2,..., rn. 745 An example configuration would be 746 - Y1 customers, policer rate r1 747 - Y2 customers, policer rate r2 748 - Y3 customers, policer rate r3 749 ... 750 - Yn customers, policer rate rn 752 Some bandwidth on the physical port is dedicated for other traffic (non 753 customer traffic); this includes network control protocol traffic. There 754 is a separate policer for the other traffic. Typical deployments have 3 755 categories of policers; there may be some deployments with more or less 756 than 3 categories of ingress policers. 758 6.1.2.2 Single Policer on All Physical Ports 759 The second policer capacity test involves a single Policer function per 760 physical port with all physical ports active. In this test, there is a 761 single policer per physical port. The policer can have one of the rates 762 r1, r2,.., rn. All the physical ports in the networking device are 763 active. 765 6.1.2.3 Maximum Policers on All Physical Ports 766 Finally the third policer capacity test involves a combination of the 767 first and second capacity test, namely maximum policers active per 768 physical port and all physical ports are active . 770 6.2. Queue and Scheduler Tests 772 Queues and traffic Scheduling are closely related in that a queue's 773 priority dictates the manner in which the traffic scheduler's 774 transmits packets out of the egress port. 776 Since device queues / buffers are generally an egress function, this 777 test framework will discuss testing at the egress (although the 778 technique can be applied to ingress side queues). 780 Similar to the policing tests, the tests are divided into two 781 sections; individual queue/scheduler function tests and then full 782 capacity tests. 784 6.2.1 Queue/Scheduler Individual Tests 786 The various types of scheduling techniques include FIFO, Strict 787 Priority (SP), Weighted Fair Queueing (WFQ) along with other 788 variations. This test framework recommends to test at a minimum 789 of three techniques although it is the discretion of the tester 790 to benchmark other device scheduling algorithms. 792 6.2.1.1 Testing Queue/Scheduler with Stateless Traffic 794 A network device queue is memory based unlike a policing function, 795 which is token or credit based. However, the same concepts from 796 section 6.1 can be applied to testing network device queues. 798 The device's network queue should be configured to the desired size 799 in KB (queue length, QL) and then stateless traffic should be 800 transmitted to test this QL. 802 A queue should be able to handle repetitive bursts with the 803 transmission gaps proportional to the bottleneck bandwidth. This 804 gap is referred to as the transmission interval (Ti). Ti can 805 be defined for the traffic bursts and is based off of the QL and 806 Bottleneck Bandwidth (BB) of the egress interface. 808 Ti = QL * 8 / BB 810 Note that this equation is similar to the Ti required for transmission 811 into a policer (QL = CBS, BB = CIR). Also note that the burst hunt 812 algorithm defined in section 5.1.1 can also be used to automate the 813 measurement of the queue value. 815 The stateless traffic burst shall be transmitted at the link speed 816 and spaced within the Ti time interval. The metrics defined in section 817 4.1 shall be measured at the egress port and recorded; the primary 818 result is to verify the BSA and that no packets are dropped. 820 The scheduling function must also be characterized to benchmark the 821 device's ability to schedule the queues according to the priority. 822 An example would be 2 levels of priority including SP and FIFO 823 queueing. Under a flow load greater the egress port speed, the 824 higher priority packets should be transmitted without drops (and 825 also maintain low latency), while the lower priority (or best 826 effort) queue may be dropped. 828 6.2.1.2 Testing Queue/Scheduler with Stateful Traffic 830 To provide a more realistic benchmark and to test queues in layer 4 831 devices such as firewalls, stateful traffic testing is recommended 832 for the queue tests. Stateful traffic tests will also utilize the 833 Network Delay Emulator (NDE) from the network set-up configuration in 834 section 2. 836 The BDP of the TCP test traffic must be calibrated to the QL of the 837 device queue. Referencing RFC 6349, the BDP is equal to: 839 BB * RTT / 8 (in bytes) 841 The NDE must be configured to an RTT value which is large enough to 842 allow the BDP to be greater than QL. An example test scenario is 843 defined below: 845 - Ingress link = GigE 846 - Egress link = 100 Mbps (BB) 847 - QL = 32KB 849 RTT(min) = QL * 8 / BB and would equal 2.56 millisecond (and the 850 BDP = 32KB) 852 In this example, one (1) TCP connection with window size / SSB of 853 32KB would be required to test the QL of 32KB. This Bulk Transfer 854 Test can be accomplished using iperf as described in Appendix A. 856 Two types of TCP tests must be performed: Bulk Transfer test and Micro 857 Burst Test Pattern as documented in Appendix B. The Bulk Transfer 858 Test only bursts during the TCP Slow Start (or Congestion Avoidance) 859 state, while the Micro Burst test emulates application layer bursting 860 which may occur any time during the TCP connection. 862 Other tests types should include: Simple Web Site, Complex Web Site, 863 Business Applications, Email, SMB/CIFS File Copy (which are also 864 documented in Appendix B). 866 The test results will be recorded per the stateful metrics defined in 867 section 4.2, primarily the TCP Test Pattern Execution Time (TTPET), 868 TCP Efficiency, and Buffer Delay. 870 6.2.2 Queue / Scheduler Capacity Tests 872 The intent of these capacity tests is to benchmark queue/scheduler 873 performance in a scaled environment with multiple queues/schedulers 874 active on multiple egress physical ports. This test will benchmark 875 the maximum number of queues and schedulers as specified by the 876 device manufacturer. Each priority in the system will map to a 877 separate queue. 879 6.2.2.1 Multiple Queues / Single Port Active 881 For the first scheduler / queue capacity test, multiple queues per 882 port will be tested on a single physical port. In this case, 883 all the queues (typically 8) are active on a single physical port. 884 Traffic from multiple ingress physical ports are directed to the 885 same egress physical port which will cause oversubscription on the 886 egress physical port. 888 There are many types of priority schemes and combinations of 889 priorities that are managed by the scheduler. The following 890 sections specify the priority schemes that should be tested. 892 6.2.2.1.1 Strict Priority on Egress Port 894 For this test, Strict Priority (SP) scheduling on the egress 895 physical port should be tested and the benchmarking methodology 896 specified in section 6.2.1 should be applied here. For a given 897 priority, each ingress physical port should get a fair share of 898 the egress physical port bandwidth. 900 6.2.2.1.2 Strict Priority + Weighted Fair Queue (WFQ) on Egress Port 902 For this test, Strict Priority (SP) and Weighted Fair Queue (WFQ) 903 should be enabled simultaneously in the scheduler but on a single 904 egress port. The benchmarking methodology specified in Section 6.2.1 905 should be applied here. Additionally, the egress port bandwidth 906 sharing among weighted queues should be proportional to the assigned 907 weights. For a given priority, each ingress physical port should get 908 a fair share of the egress physical port bandwidth. 910 6.2.2.2 Single Queue per Port / All Ports Active 912 Traffic from multiple ingress physical ports are directed to the 913 same egress physical port, which will cause oversubscription on the 914 egress physical port. Also, the same amount of traffic is directed 915 to each egress physical port. 917 The benchmarking methodology specified in Section 6.2.1 should be 918 applied here. Each ingress physical port should get a fair share of 919 the egress physical port bandwidth. Additionally, each egress 920 physical port should receive the same amount of traffic. 922 6.2.2.3 Multiple Queues per Port, All Ports Active 924 Traffic from multiple ingress physical ports are directed to all 925 queues of each egress physical port, which will cause 926 oversubscription on the egress physical ports. Also, the same 927 amount of traffic is directed to each egress physical port. 929 The benchmarking methodology specified in Section 6.2.1 should be 930 applied here. For a given priority, each ingress physical port 931 should get a fair share of the egress physical port bandwidth. 932 Additionally, each egress physical port should receive the same 933 amount of traffic. 935 6.3. Shaper tests 937 A traffic shaper is memory based like a queue, but with the added 938 intelligence of an active shaping element. The same concepts from 939 section 6.2 (Queue testing) can be applied to testing network device 940 shaper. 942 Again, the tests are divided into two sections; individual shaper 943 benchmark tests and then full capacity shaper benchmark tests. 945 6.3.1 Shaper Individual Tests 947 A traffic shaper generally has three (3) components that can be 948 configured: 950 - Ingress Queue bytes 951 - Shaper Rate, bps 952 - Burst Committed (Bc) and Burst Excess (Be), bytes 954 The Ingress Queue holds burst traffic and the shaper then meters 955 traffic out of the egress port according to the Shaper Rate and 956 Bc/Be parameters. Shapers generally transmit into policers, so 957 the idea is for the emitted traffic to conform to the policer's 958 limits. 960 The stateless and stateful traffic test sections describe the 961 techniques to transmit bursts into the DUT's ingress port 962 and the metrics to benchmark at the shaper egress port. 964 6.3.1.1 Testing Shaper with Stateless Traffic 966 The stateless traffic must be burst into the DUT ingress port and 967 not exceed the Ingress Queue. The burst can be a single burst or 968 multiple bursts. If multiple bursts are transmitted, then the 969 Ti (Time interval) must be large enough so that the Shaper Rate is 970 not exceeded. An example will clarify single and multiple burst 971 test cases. 973 In the example, the shaper's ingress and egress ports are both full 974 duplex Gigabit Ethernet. The Ingress Queue is configured to be 975 512,000 bytes, the Shaper Rate = 50 Mbps, and both Bc/Be configured 976 to be 32,000 bytes. For a single burst test, the transmitting test 977 device would burst 512,000 bytes maximum into the ingress port and 978 then stop transmitting. The egress port metrics from section 4.1 979 will be recorded with particular emphasis on the LP, PDV, SBB, and 980 SBI metrics. 982 If a multiple burst test is to be conducted, then the burst bytes 983 divided by the time interval between the 512,000 byte bursts must 984 not exceed the Shaper Rate. The time interval (Ti) must adhere to 985 a similar formula as described in section 6.2.1.1 for queues, namely: 987 Ti = Ingress Queue x 8 / Shaper Rate 989 So for the example from the previous paragraph, Ti between bursts must 990 be greater than 82 millisecond (512,000 bytes x 8 / 50,000,000 bps). 991 This yields an average rate of 50 Mbps so that an Input Queue 992 would not overflow. 994 6.3.1.2 Testing Shaper with Stateful Traffic 996 To provide a more realistic benchmark and to test queues in layer 4 997 devices such as firewalls, stateful traffic testing is also 998 recommended for the shaper tests. Stateful traffic tests will also 999 utilize the Network Delay Emulator (NDE) from the network set-up 1000 configuration in section 2. 1002 The BDP of the TCP test traffic must be calculated as described in 1003 section 6.2.2. To properly stress network buffers and the traffic 1004 shaping function, the cumulative TCP window should exceed the BDP 1005 which will stress the shaper. BDP factors of 1.1 to 1.5 are 1006 recommended, but the values are the discretion of the tester and 1007 should be documented. 1009 The cumulative TCP Window Sizes* (RWND at the receiving end & CWND 1010 at the transmitting end) equates to: 1012 TCP window size* for each connection x number of connections 1014 * as described in section 3 of RFC6349, the SSB MUST be large 1015 enough to fill the BDP 1017 Example, if the BDP is equal to 256 Kbytes and a connection size of 1018 64Kbytes is used for each connection, then it would require four (4) 1019 connections to fill the BDP and 5-6 connections (over subscribe the 1020 BDP) to stress test the traffic shaping function. 1022 Two types of TCP tests must be performed: Bulk Transfer test and Micro 1023 Burst Test Pattern as documented in Appendix B. The Bulk Transfer 1024 Test only bursts during the TCP Slow Start (or Congestion Avoidance) 1025 state, while the Micro Burst test emulates application layer bursting 1026 which may any time during the TCP connection. 1028 Other tests types should include: Simple Web Site, Complex Web Site, 1029 Business Applications, Email, SMB/CIFS File Copy (which are also 1030 documented in Appendix B). 1032 The test results will be recorded per the stateful metrics defined in 1033 section 4.2, primarily the TCP Test Pattern Execution Time (TTPET), 1034 TCP Efficiency, and Buffer Delay. 1036 6.3.2 Shaper Capacity Tests 1038 The intent of these scalability tests is to verify shaper performance 1039 in a scaled environment with shapers active on multiple queues on 1040 multiple egress physical ports. This test will benchmark the maximum 1041 number of shapers as specified by the device manufacturer. 1043 For all of the capacity tests, the benchmarking methodology described 1044 in Section 6.3.1 for a single shaper should be applied to each of the 1045 physical port and/or queue shapers. 1047 6.3.2.1 Single Queue Shaped, All Physical Ports Active 1048 The first shaper capacity test involves per port shaping, all physical 1049 ports active. Traffic from multiple ingress physical ports are directed 1050 to the same egress physical port and this will cause oversubscription 1051 on the egress physical port. Also, the same amount of traffic is 1052 directed to each egress physical port. 1054 The benchmarking methodology described in Section 6.3.1 should be 1055 applied to each of the physical ports. Each ingress physical port 1056 should get a fair share of the egress physical port bandwidth. 1058 6.3.2.2 All Queues Shaped, Single Port Active 1059 The second shaper capacity test is conducted with all queues actively 1060 shaping on a single physical port. The benchmarking methodology 1061 described in per port shaping test (previous section) serves as the 1062 foundation for this. Additionally, each of the SP queues on the 1063 egress physical port is configured with a shaper. For the highest 1064 priority queue, the maximum amount of bandwidth available is limited 1065 by the bandwidth of the shaper. For the lower priority queues, the 1066 maximum amount of bandwidth available is limited by the bandwidth of 1067 the shaper and traffic in higher priority queues. 1069 6.3.2.3 All Queues Shaped, All Ports Active 1070 And for the third shaper capacity test (which is a combination of the 1071 tests in the previous two sections),all queues will be actively 1072 shaping and all physical ports active. 1074 6.4 Concurrent Capacity Load Tests 1076 As mentioned in the scope of this document, it is impossible to 1077 specify the various permutations of concurrent traffic management 1078 functions that should be tested in a device for capacity testing. 1079 However, some profiles are listed below which may be useful 1080 to test under capacity as well: 1082 - Policers on ingress and queuing on egress 1083 - Policers on ingress and shapers on egress (not intended for a 1084 flow to be policed then shaped, these would be two different 1085 flows tested at the same time) 1086 - etc. 1088 Appendix A: Open Source Tools for Traffic Management Testing 1090 This framework specifies that stateless and stateful behaviors should 1091 both be tested. Two (2) open source tools that can be used are iperf 1092 and Flowgrind to accomplish many of the tests proposed in this 1093 framework. 1095 Iperf can generate UDP or TCP based traffic; a client and server must 1096 both run the iperf software in the same traffic mode. The server is 1097 set up to listen and then the test traffic is controlled from the 1098 client. Both uni-directional and bi-directional concurrent testing 1099 are supported. 1101 The UDP mode can be used for the stateless traffic testing. The 1102 target bandwidth, packet size, UDP port, and test duration can be 1103 controlled. A report of bytes transmitted, packets lost, and delay 1104 variation are provided by the iperf receiver. 1106 The TCP mode can be used for stateful traffic testing to test bulk 1107 transfer traffic. The TCP Window size (which is actually the SSB), 1108 the number of connections, the packet size, TCP port and the test 1109 duration can be controlled. A report of bytes transmitted and 1110 throughput achieved are provided by the iperf sender. 1112 Flowgrind is a distributed network performance measurement tool. 1113 Using the flowgrind controller, tests can be setup between hosts 1114 running flowgrind. For the purposes of this traffic management 1115 testing framework, the key benefit of Flowgrind is that it can 1116 emulate non-bulk transfer applications such as HTTP, Email, etc. 1117 This is due to fact that Flowgrind supports the concept of request 1118 and response behavior while iperf does not. 1120 Traffic generation options include the request size, response size, 1121 inter-request gap, and response time gap. Additionally, various 1122 distribution types are supported including constant, normal, 1123 exponential, pareto, etc. These traffic generation parameters 1124 facilitate the emulation of some of the TCP test patterns 1125 which are discussed in Appendix B. 1127 Since these tools are software based, the host hardware must be 1128 qualified as capable of generating the target traffic loads 1129 without packet loss and within the packet delay variation threshold. 1131 Appendix B: Stateful TCP Test Patterns 1133 This framework recommends at a minimum the following TCP test patterns 1134 since they are representative of real world application traffic (section 1135 5.2.1 describes some methods to derive other application-based TCP test 1136 patterns). 1138 - Bulk Transfer: generate concurrent TCP connections whose aggregate 1139 number of in-flight data bytes would fill the BDP. Guidelines 1140 from RFC 6349 are used to create this TCP traffic pattern. 1142 - Micro Burst: generate precise burst patterns within a single or multiple 1143 TCP connections(s). The idea is for TCP to establish equilibrium and then 1144 burst application bytes at defined sizes. The test tool must allow the 1145 burst size and burst time interval to be configurable. 1147 - Web Site Patterns: The HTTP traffic model from "3GPP2 C.R1002-0 v1.0" 1148 is referenced (Table 4.1.3.2-1) to develop these TCP test patterns. In 1149 summary, the HTTP traffic model consists of the following parameters: 1150 - Main object size (Sm) 1151 - Embedded object size (Se) 1152 - Number of embedded objects per page (Nd) 1153 - Client processing time (Tcp) 1154 - Server processing time (Tsp) 1156 Web site test patterns are illustrated with the following examples: 1158 - Simple Web Site: mimic the request / response and object download 1159 behavior of a basic web site (small company). 1160 - Complex Web Site: mimic the request / response and object download 1161 behavior of a complex web site (ecommerce site). 1163 Referencing the HTTP traffic model parameters , the following table 1164 was derived (by analysis and experimentation) for Simple and Complex 1165 Web site TCP test patterns: 1167 Simple Complex 1168 Parameter Web Site Web Site 1169 ----------------------------------------------------- 1170 Main object Ave. = 10KB Ave. = 300KB 1171 size (Sm) Min. = 100B Min. = 50KB 1172 Max. = 500KB Max. = 2MB 1174 Embedded object Ave. = 7KB Ave. = 10KB 1175 size (Se) Min. = 50B Min. = 100B 1176 Max. = 350KB Max. = 1MB 1178 Number of embedded Ave. = 5 Ave. = 25 1179 objects per page (Nd) Min. = 2 Min. = 10 1180 Max. = 10 Max. = 50 1182 Client processing Ave. = 3s Ave. = 10s 1183 time (Tcp)* Min. = 1s Min. = 3s 1184 Max. = 10s Max. = 30s 1186 Server processing Ave. = 5s Ave. = 8s 1187 time (Tsp)* Min. = 1s Min. = 2s 1188 Max. = 15s Max. = 30s 1190 * The client and server processing time is distributed across the 1191 transmission / receipt of all of the main and embedded objects 1192 To be clear, the parameters in this table are reasonable guidelines 1193 for the TCP test pattern traffic generation. The test tool can use 1194 fixed parameters for simpler tests and mathematical distributions for 1195 more complex tests. However, the test pattern must be repeatable to 1196 ensure that the benchmark results can be reliably compared. 1198 - Inter-active Patterns: While Web site patterns are inter-active 1199 to a degree, they mainly emulate the downloading of various 1200 complexity web sites. Inter-active patterns are more chatty in nature 1201 since there is alot of user interaction with the servers. Examples 1202 include business applications such as Peoplesoft, Oracle and consumer 1203 applications such as Facebook, IM, etc. For the inter-active patterns, 1204 the packet capture technique was used to characterize some business 1205 applications and also the email application. 1207 In summary, an inter-active application can be described by the following 1208 parameters: 1209 - Client message size (Scm) 1210 - Number of Client messages (Nc) 1211 - Server response size (Srs) 1212 - Number of server messages (Ns) 1213 - Client processing time (Tcp) 1214 - Server processing Time (Tsp) 1215 - File size upload (Su)* 1216 - File size download (Sd)* 1218 * The file size parameters account for attachments uploaded or downloaded 1219 and may not be present in all inter-active applications 1221 Again using packet capture as a means to characterize, the following 1222 table reflects the guidelines for Simple Business Application, Complex 1223 Business Application, eCommerce, and Email Send / Receive: 1225 Simple Complex 1226 Parameter Biz. App. Biz. App eCommerce* Email 1227 -------------------------------------------------------------------- 1228 Client message Ave. = 450B Ave. = 2KB Ave. = 1KB Ave. = 200B 1229 size (Scm) Min. = 100B Min. = 500B Min. = 100B Min. = 100B 1230 Max. = 1.5KB Max. = 100KB Max. = 50KB Max. = 1KB 1232 Number of client Ave. = 10 Ave. = 100 Ave. = 20 Ave. = 10 1233 messages (Nc) Min. = 5 Min. = 50 Min. = 10 Min. = 5 1234 Max. = 25 Max. = 250 Max. = 100 Max. = 25 1236 Client processing Ave. = 10s Ave. = 30s Ave. = 15s Ave. = 5s 1237 time (Tcp)** Min. = 3s Min. = 3s Min. = 5s Min. = 3s 1238 Max. = 30s Max. = 60s Max. = 120s Max. = 45s 1240 Server response Ave. = 2KB Ave. = 5KB Ave. = 8KB Ave. = 200B 1241 size (Srs) Min. = 500B Min. = 1KB Min. = 100B Min. = 150B 1242 Max. = 100KB Max. = 1MB Max. = 50KB Max. = 750B 1244 Number of server Ave. = 50 Ave. = 200 Ave. = 100 Ave. = 15 1245 messages (Ns) Min. = 10 Min. = 25 Min. = 15 Min. = 5 1246 Max. = 200 Max. = 1000 Max. = 500 Max. = 40 1248 Server processing Ave. = 0.5s Ave. = 1s Ave. = 2s Ave. = 4s 1249 time (Tsp)** Min. = 0.1s Min. = 0.5s Min. = 1s Min. = 0.5s 1250 Max. = 5s Max. = 20s Max. = 10s Max. = 15s 1252 File size Ave. = 50KB Ave. = 100KB Ave. = N/A Ave. = 100KB 1253 upload (Su) Min. = 2KB Min. = 10KB Min. = N/A Min. = 20KB 1254 Max. = 200KB Max. = 2MB Max. = N/A Max. = 10MB 1256 File size Ave. = 50KB Ave. = 100KB Ave. = N/A Ave. = 100KB 1257 download (Sd) Min. = 2KB Min. = 10KB Min. = N/A Min. = 20KB 1258 Max. = 200KB Max. = 2MB Max. = N/A Max. = 10MB 1260 * eCommerce used a combination of packet capture techniques and 1261 reference traffic flows from "SPECweb2009" (need proper reference) 1262 ** The client and server processing time is distributed across the 1263 transmission / receipt of all of messages. Client processing time 1264 consists mainly of the delay between user interactions (not machine 1265 processing). 1267 And again, the parameters in this table are the guidelines for the 1268 TCP test pattern traffic generation. The test tool can use fixed 1269 parameters for simpler tests and mathematical distributions for more 1270 complex tests. However, the test pattern must be repeatable to ensure 1271 that the benchmark results can be reliably compared. 1273 - SMB/CIFS File Copy: mimic a network file copy, both read and write. 1274 As opposed to FTP which is a bulk transfer and is only flow controlled 1275 via TCP, SMB/CIFS divides a file into application blocks and utilizes 1276 application level handshaking in addition to TCP flow control. 1278 In summary, an SMB/CIFS file copy can be described by the following 1279 parameters: 1280 - Client message size (Scm) 1281 - Number of client messages (Nc) 1282 - Server response size (Srs) 1283 - Number of Server messages (Ns) 1284 - Client processing time (Tcp) 1285 - Server processing time (Tsp) 1286 - Block size (Sb) 1288 The client and server messages are SMB control messages. The Block size 1289 is the data portion of th file transfer. 1291 Again using packet capture as a means to characterize the following 1292 table reflects the guidelines for SMB/CIFS file copy: 1294 SMB 1295 Parameter File Copy 1296 ------------------------------ 1297 Client message Ave. = 450B 1298 size (Scm) Min. = 100B 1299 Max. = 1.5KB 1300 Number of client Ave. = 10 1301 messages (Nc) Min. = 5 1302 Max. = 25 1303 Client processing Ave. = 1ms 1304 time (Tcp) Min. = 0.5ms 1305 Max. = 2 1306 Server response Ave. = 2KB 1307 size (Srs) Min. = 500B 1308 Max. = 100KB 1309 Number of server Ave. = 10 1310 messages (Ns) Min. = 10 1311 Max. = 200 1312 Server processing Ave. = 1ms 1313 time (Tsp) Min. = 0.5ms 1314 Max. = 2ms 1315 Block Ave. = N/A 1316 Size (Sb)* Min. = 16KB 1317 Max. = 128KB 1319 *Depending upon the tested file size, the block size will be 1320 transferred n number of times to complete the example. An example 1321 would be a 10 MB file test and 64KB block size. In this case 160 1322 blocks would be transferred after the control channel is opened 1323 between the client and server. 1325 7. Security Considerations 1327 8. IANA Considerations 1329 9. Conclusions 1331 10. References 1333 10.1. Normative References 1335 [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement 1336 Levels", BCP 14, RFC 2119, March 1997. 1338 [2] Crocker, D. and Overell, P.(Editors), "Augmented BNF for Syntax 1339 Specifications: ABNF", RFC 2234, Internet Mail Consortium and 1340 Demon Internet Ltd., November 1997. 1342 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1343 Requirement Levels", BCP 14, RFC 2119, March 1997. 1345 [RFC2234] Crocker, D. and Overell, P.(Editors), "Augmented BNF for 1346 Syntax Specifications: ABNF", RFC 2234, Internet Mail 1347 Consortium and Demon Internet Ltd., November 1997. 1349 10.2. Informative References 1351 11. Acknowledgments 1353 Authors' Addresses 1355 Barry Constantine 1357 JDSU, Test and Measurement Division 1359 Germantown, MD 20876-7100, USA 1361 Phone: +1 240 404 2227 1363 Email: barry.constantine@jdsu.com 1365 Timothy Copley 1367 Level 3 Communications 1369 14605 S 50th Street 1371 Phoenix, AZ 85044 1373 Email: Timothy.copley@level3.com 1375 Ram Krishnan 1377 Brocade Communications 1379 San Jose, 95134, USA 1381 Phone: +001-408-406-7890 1383 Email: ramk@brocade.com