idnits 2.17.1 draft-ietf-bmwg-acc-bench-term-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3667, Section 5.1 on line 19. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1293. ** The document claims conformance with section 10 of RFC 2026, but uses some RFC 3978/3979 boilerplate. As RFC 3978/3979 replaces section 10 of RFC 2026, you should not claim conformance with it if you have changed to using RFC 3978/3979 boilerplate. ** The document seems to lack an RFC 3978 Section 5.1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** The document seems to lack an RFC 3978 Section 5.5 (updated by RFC 4748) Disclaimer -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document seems to lack an RFC 3979 Section 5, para. 1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document seems to lack an RFC 3979 Section 5, para. 3 IPR Disclosure Invitation -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document uses RFC 3667 boilerplate or RFC 3978-like boilerplate instead of verbatim RFC 3978 boilerplate. After 6 May 2005, submission of drafts without verbatim RFC 3978 boilerplate is not accepted. The following non-3978 patterns matched text found in the document. That text should be removed or replaced: By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. == No 'Intended status' indicated for this document; assuming Proposed Standard == It seems as if not all pages are separated by form feeds - found 0 form feeds but 26 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. (A line matching the expected section header was found, but with an unexpected indentation: ' 1. Introduction' ) ** The document seems to lack a Security Considerations section. (A line matching the expected section header was found, but with an unexpected indentation: ' 4. Security Considerations' ) ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack an Authors' Addresses Section. ** There are 6 instances of too long lines in the document, the longest one being 5 characters in excess of 72. ** There are 486 instances of lines with control characters in the document. ** The abstract seems to contain references ([2], [RFC3871], [3], [CONVMETH], [4], [IEEECQR], [NANOG25], [Br97], [CONVTERM], [1]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (October 2005) is 6765 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? '4' on line 1205 looks like a reference -- Missing reference section? 'Br97' on line 159 looks like a reference -- Missing reference section? '1' on line 1196 looks like a reference -- Missing reference section? '2' on line 1199 looks like a reference -- Missing reference section? '3' on line 1202 looks like a reference -- Missing reference section? 'RFC3871' on line 1210 looks like a reference -- Missing reference section? 'NANOG25' on line 1214 looks like a reference -- Missing reference section? 'IEEECQR' on line 1217 looks like a reference -- Missing reference section? 'CONVMETH' on line 1220 looks like a reference -- Missing reference section? 'CONVTERM' on line 1224 looks like a reference Summary: 16 errors (**), 0 flaws (~~), 3 warnings (==), 14 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group 2 INTERNET-DRAFT 3 Expires in: October 2005 4 Scott Poretsky 5 Quarry Technologies 7 Shankar Rao 8 Qwest Communications 10 February 2005 12 Terminology for Accelerated Stress Benchmarking 13 15 Intellectual Property Rights (IPR) statement: 16 By submitting this Internet-Draft, I certify that any applicable 17 patent or other IPR claims of which I am aware have been disclosed, or 18 will be disclosed, and any of which I become aware will be disclosed, 19 in accordance with RFC 3668. 21 Status of this Memo 23 This document is an Internet-Draft and is in full conformance with 24 all provisions of Section 10 of RFC2026. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF), its areas, and its working groups. Note that 27 other groups may also distribute working documents as Internet- 28 Drafts. 30 Internet-Drafts are draft documents valid for a maximum of six 31 months and may be updated, replaced, or obsoleted by other 32 documents at any time. It is inappropriate to use Internet-Drafts 33 as reference material or to cite them other than as "work in 34 progress." 36 The list of current Internet-Drafts can be accessed at 37 http://www.ietf.org/ietf/1id-abstracts.txt 38 The list of Internet-Draft Shadow Directories can be accessed at 39 http://www.ietf.org/shadow.html. 41 ABSTRACT 42 This document provides the Terminology for performing Stress 43 Benchmarking of networking devices. The three phases of the Stress 44 Test: Startup, Instability and Recovery are defined along with the 45 benchmarks and configuration terms associated with the each phase. 46 Also defined are the Benchmark Planes fundamental to stress testing 47 configuration, setup and measurement. The terminology is to be 48 used with the companion framework and methodology documents. 50 Table of Contents 51 1. Introduction ............................................... 3 52 2. Existing definitions ....................................... 3 53 3. Term definitions............................................ 4 54 3.1 General Terms............................................. 4 56 3.1.1 Benchmark Planes...................................... 4 57 3.1.2 Configuration Sets.................................... 5 58 3.1.3 Startup Conditions.................................... 5 59 3.1.4 Instability Conditions................................ 6 60 3.1.5 Aggregate Forwarding Rate............................. 6 61 3.1.6 Controlled Session Loss............................... 7 62 3.1.7 Uncontrolled Session Loss............................. 7 63 3.2 Benchmark Planes.......................................... 8 64 3.2.1 Control Plane......................................... 8 65 3.2.2 Data Plane............................................ 8 66 3.2.3 Management Plane...................................... 8 67 3.2.4 Security Plane........................................ 9 68 3.3 Startup...................................................10 69 3.3.1 Startup Phase.........................................10 70 3.3.2 Benchmarks............................................10 71 3.3.2.1 Stable Aggregate Forwarding Rate..................10 72 3.3.2.2 Stable Latency....................................11 73 3.3.2.3 Stable Session Count..............................11 74 3.3.3 Control Plane.........................................12 75 3.3.3.1 Control Plane Configuration Set...................12 76 3.3.3.2 Control Plane Startup Conditions..................13 77 3.3.4 Data Plane............................................13 78 3.3.4.1 Data Plane Configuration Set......................13 79 3.3.4.2 Traffic Profile...................................13 80 3.3.5 Management Plane......................................14 81 3.3.5.1 Management Plane Configuration Set................14 82 3.3.6 Security Plane........................................15 83 3.3.6.1 Security Plane Configuration Set..................15 84 3.3.6.2 Security Plane Startup Conditions.................16 85 3.4 Instability...............................................16 86 3.4.1 Instability Phase.....................................16 87 3.4.2 Benchmarks............................................17 88 3.4.2.1 Unstable Aggregate Forwarding Rate................17 89 3.4.2.2 Degraded Aggregate Forwarding Rate................17 90 3.4.2.3 Average Degraded Aggregate Forwarding Rate........18 91 3.4.2.4 Unstable Latency..................................18 92 3.4.2.5 Unstable Uncontrolled Sessions Lost...............19 93 3.4.3 Instability Conditions................................19 94 3.4.3.1 Control Plane Instability Conditions..............19 95 3.4.3.2 Data Plane Instability Conditions.................20 96 3.4.3.3 Management Plane Instability Conditions...........20 97 3.4.3.4 Security Plane Instability Conditions.............20 98 3.5 Recovery..................................................21 99 3.5.1 Recovery Phase........................................21 100 3.5.2 Benchmarks............................................21 101 3.5.2.1 Recovered Aggregate Forwarding Rate...............21 102 3.5.2.2 Recovered Latency.................................22 103 3.5.2.3 Recovery Time.....................................22 104 3.5.2.4 Recovered Uncontrolled Sessions Lost..............23 105 3.5.2.5 Variability Benchmarks............................23 107 4. Security Considerations.....................................24 108 5. Normative References........................................24 109 6. Informative References......................................24 110 7. Author's Address............................................25 111 Appendix 1 - White Box Benchmarks..............................25 113 1. Introduction 115 Routers in an operational network are simultaneously configured with 116 multiple protocols and security policies while forwarding traffic and 117 being managed. To accurately benchmark a router for deployment it is 118 necessary to test that router in operational conditions by 119 simultaneously configuring and scaling network protocols and security 120 policies, forwarding traffic, and managing the device. It is helpful 121 to accelerate these network operational conditions so that the 122 router under test can be benchmarked with faster test duration. 123 Testing a router in accelerated network conditions is known as 124 Accelerated Stress Testing. 126 This document provides the Terminology for performing Stress 127 Benchmarking of networking devices. The three phases of the Stress 128 Test: Startup, Instability and Recovery are defined along with the 129 benchmark and configuration terms associated with the each phase. 130 Benchmarks for stress testing are defined using the Aggregate 131 Forwarding Rate and control plane Session Count during each phase 132 of the test. Also defined are the Benchmark Planes fundamental to 133 stress testing configuration, setup and measurement. These are 134 the Control Plane, Data Plane, Management Plane and Security Plane 135 For each plane, the Configuration Set, Startup Conditions, and 136 Instability Conditions are defined. White Box benchmarks are 137 provided in Appendix 1 for additional DUT behavior measurements. 138 The terminology is to be used with the companion methodology 139 document [4]. The sequence of phases, actions, and benchmarks 140 are shown in Table 1. 142 2. Existing definitions 144 RFC 1242 "Benchmarking Terminology for Network Interconnect 145 Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching 146 Devices" should be consulted before attempting to make use of this 147 document. For the sake of clarity and continuity this RFC adopts 148 the template for definitions set out in Section 2 of RFC 1242. 149 Definitions are indexed and grouped together in sections for ease 150 of reference. 152 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 153 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 154 document are to be interpreted as described in BCP 14, RFC 2119 155 [Br97]. RFC 2119 defines the use of these key words to help make the 156 intent of standards track documents as clear as possible. While this 157 document uses these keywords, this document is not a standards track 158 document. 160 Table 1. Phase Sequence and Benchmarks 161 III. Recovery Phase II. Instability Phase I. Startup Phase 162 <-----------------<---<-------------------<----<--------------< 163 Remove Instability Achieve Configuration Apply Startup 164 Conditions Set Conditions 166 Benchmark: Benchmark: Benchmark: 167 Recovered Aggregate Unstable Aggregate Stable Aggregate 168 Forwarding Rate Forwarding Rate Forwarding Rate 170 Degraded Aggregate 171 Forwarding Rate 173 Average Degraded 174 Forwarding Rate 176 Recovered Latency Unstable Latency Startup Latency 178 Recovered Uncontrolled Recovered Uncontrolled Stable Session Count 179 Sessions Lost Sessions Lost 181 Recovery Time 183 3. Term definitions 184 3.1 General Terms 185 3.1.1 Benchmark Planes 187 Definition: 188 The features, conditions, and behavior for the Accelerated Stress 189 Benchmarking. 191 Discussion: 192 There are four Benchmark Planes: Control Plane, Data Plane, 193 Management Plane, and Security Plane as shown in Figure 1. The 194 Benchmark Planes define the Configuration, Startup Conditions, 195 Instability Conditions, and Failure Conditions used for the test. 197 Measurement units: 198 N/A 200 Issues: 201 None 203 See Also: 204 Control Plane 205 Data Plane 206 Management Plane 207 Security Plane 208 ___________ ___________ 209 | Control | | Management| 210 | Plane |___ ___| Plane | 211 | | | | | | 212 ----------- | | ----------- 213 \/ \/ ___________ 214 ___________ | Security | 215 | |<-----------| Plane | 216 | DUT | | | 217 |--->| |<---| ----------- 218 | ----------- | 219 | | 220 | ___________ | 221 | | Data | | 222 |--->| Plane |<---| 223 | | 224 ----------- 226 Figure 1. Router Accelerated Stress Benchmarking Planes 228 3.1.2 Configuration Sets 230 Definition: 231 The features and scaling limits used during the Accelerated Stress 232 Benchmarking. 234 Discussion: 235 There are four Configuration Sets: Control Plane Configuration Set, 236 Data Plane Configuration Set, Management Plane Configuration Set, 237 and Security Plane Configuration Set. 239 Measurement units: 240 N/A 242 Issues: 243 None 245 See Also: 246 Control Plane Configuration Set 247 Data Plane Configuration Set 248 Management Plane Configuration Set 249 Security Plane Configuration Set 251 3.1.3 Startup Conditions 253 Definition: 254 Test conditions that occur at the start of the Accelerated 255 Life Benchmark to establish conditions for the remainder of 256 the test. 258 Discussion: 259 Startup Conditions may cause stress on the DUT and produce 260 failure. Startup Conditions are defined for the Control 261 Plane and Security Plane. 263 Measurement units: 264 N/A 266 Issues: 267 None 269 See Also: 270 Control Plane Startup Conditions 271 Data Plane Startup Conditions 272 Management Plane Startup Conditions 273 Security Plane Startup Conditions 275 3.1.4 Instability Conditions 277 Definition: 278 Test conditions that occur during the Accelerated Stress 279 Benchmark to produce instability and stress the DUT. 281 Discussion: 282 Instability Conditions are applied to the DUT after the 283 Startup Conditions have completed. Instability Conditions 284 occur for the Control Plane, Data Plane, Management Plane, 285 and Security Plane. 287 Measurement units: 288 N/A 290 Issues: None 292 See Also: 293 Control Plane Instability Conditions 294 Data Plane Instability Conditions 295 Management Plane Instability Conditions 296 Security Plane Instability Conditions 298 3.1.5 Aggregate Forwarding Rate 300 Definition: 301 Sum of forwarding rates for all interfaces on the 302 DUT during the Startup Phase. 304 Discussion: 305 Each interface of the DUT forwards traffic at some 306 measured rate. The Aggregate Forwarding Rate is the 307 sum of forwarding rates for all interfaces on the DUT. 309 Measurement units: 310 pps 312 Issues: 313 None 315 See Also: 316 Startup Phase 318 3.1.6 Controlled Session Loss 320 Definition: 321 Control Plane sessions that are intentionally brought 322 down during the Stress test. 324 Discussion: 325 The test equipment is able to control protocol 326 session state with the DUT. 328 Measurement units: 329 None 331 Issues: 332 None 334 See Also: 335 Uncontrolled Session Loss 337 3.1.7 Uncontrolled Session Loss 339 Definition: 340 Control Plane sessions that are in the down state 341 but were not intentionally brought down during the 342 Stress test. 344 Discussion: 345 The test equipment is able to control protocol 346 session state with the DUT. The test equipment 347 is also to monitor for sessions lost with the 348 DUT which the test equipment itself did not 349 intentionally bring down. 351 Measurement units: 352 N/A 354 Issues: 355 None 357 See Also: 358 Controlled Session Loss 359 3.2 Benchmark Planes 361 3.2.1 Control Plane 362 Definition: 363 The Description of the control protocols enabled for 364 the Accelerated Stress Benchmarking. 366 Discussion: 367 The Control Plane defines the Configuration, Startup 368 Conditions, and Instability Conditions of the control 369 protocols. Control Plane protocols may include routing 370 protocols, multicast protocols, and MPLS protocols. 371 These can be enabled or disabled for a benchmark test. 373 Measurement units: 374 N/A 376 Issues: 377 None 379 See Also: 380 Benchmark Planes 381 Control Plane Configuration Set 382 Control Plane Startup Conditions 383 Control Plane Instability Conditions 385 3.2.2 Data Plane 386 Definition: 387 The data traffic profile used for the Accelerated Stress 388 Benchmarking. 390 Discussion: 391 The Data Plane defines the Configuration, Startup 392 Conditions, and Instability Conditions of the data 393 traffic. The Data Plane includes the traffic and 394 interface profile. 396 Measurement Units: 397 N/A 399 See Also: 400 Benchmark Planes 401 Data Plane Configuration Set 402 Data Plane Startup Conditions 403 Data Plane Instability Conditions 405 3.2.3 Management Plane 407 Definition: 408 The Management features and tools used for the 409 Accelerated Stress Benchmarking. 411 Discussion: 412 A key component of the Accelerated Stress Benchmarking is the 413 Management Plane to assess manageability of the router 414 under stress. The Management Plane defines the Configuration, 415 Startup Conditions, and Instability Conditions of the 416 management protocols and features. The Management Plane 417 includes SNMP, Logging/Debug, Statistics Collection, and 418 management configuration sessions such as telnet, SSH, and 419 serial console. SNMP Gets SHOULD be performed continuously. 420 Management configuration sessions should be open 421 simultaneously and be repeatedly open and closed. Open 422 management sessions should have valid and invalid 423 configuration and show commands entered. 425 Measurement units: 426 N/A 428 Issues: 429 None 431 See Also: 432 Benchmark Planes 433 Management Plane Configuration Set 434 Management Plane Startup Conditions 435 Management Plane Instability Conditions 437 3.2.4 Security Plane 439 Definition: 440 The Security features used during the Accelerated Stress 441 Benchmarking. 443 Discussion: 444 The Security Plane defines the Configuration, Startup 445 Conditions, and Instability Conditions of the security 446 features and protocols. The Security Plane includes the 447 ACLs, Firewall, Secure Protocols, and User Login. Tunnels 448 for those such as IPsec should be established and flapped. 449 Policies for Firewalls and ACLs should be repeatedly added 450 and removed from the configuration via telnet, SSH, or 451 serial management sessions. 453 Measurement units: 454 N/A 456 Issues: None 458 See Also: 459 Benchmark Planes 460 Security Plane Configuration Set 461 Security Plane Startup Conditions 462 Security Plane Instability Conditions 463 3.3 Startup 465 3.3.1 Startup Phase 467 Definition 468 The portion of the benchmarking test in which the 469 Startup Conditions are generated with the DUT. This 470 begins with the attempt to establish the first session 471 and ends when the last Control Plane session is 472 established. 474 Discussion: 475 The Startup Phase is the first Phase of the benchmarking 476 test preceding the Instability Phase and Recovery Phase. 477 It is specified by the Configuration Sets and Startup 478 Conditions for each Benchmark Plane. The Startup Phase ends 479 and Instability Phase may begin when the Configuration Sets 480 are achieved with the DUT. 482 Measurement Units: 483 None 485 Issues: 486 The 'last control plane session is established' may not 487 be a sufficient indicator that steady-state is achieved 488 and Instability Conditions can be applied to begin the 489 Instability Phase. 491 See Also: 492 Benchmark Plane 493 Configuration Sets 494 Startup Conditions 495 Instability Phase 496 Recovery Phase 498 3.3.2 Benchmarks 499 3.3.2.1 Stable Aggregate Forwarding Rate 501 Definition: 502 Average rate of traffic forwarded by the DUT during the 503 Startup Phase. 505 Discussion: 506 Stable Aggregate Forwarding Rate is the calculated 507 average of the Aggregate Forwarding Rates measured during 508 the Startup Phase. 510 Measurement units: 511 pps 512 Issues: 513 The act of the DUT establishing the Startup Conditions 514 could influence the forwarding rate in certain 515 implementations so that this "baseline" for the 516 remainder of the test is lowered. The alternative is 517 to change the definition of Stable Aggregate 518 Forwarding Rate so that it measured during the Startup 519 Phase, but after Startup Conditions are achieved. 520 The disadvantage of this definition would be that it 521 loses measurement of any impact that establishing 522 Startup Conditions would have on forwarding rate. When 523 comparing the Startup Aggregate Forwarding Rate benchmark 524 of two devices it is preferred to know the impact 525 establishing Startup Conditions has on Forwarding Rate. 526 The definition was therefore selected so that Stable 527 Aggregate Forwarding Rate is calculated from measurement 528 samples throughout the entire Startup Phase. 530 See Also: 531 Startup Phase 532 Aggregate Forwarding Rate 534 3.3.2.2 Stable Latency 536 Definition: 537 Average measured latency of traffic forwarded by the DUT 538 during the Startup Phase. 540 Discussion: 541 Stable Latency is the calculated average Latency during 542 the Startup Phase. 544 Measurement units: 545 seconds 547 Issues: 548 None 550 See Also: 551 Startup Phase 552 Stable Aggregate Forwarding Rate 554 3.3.2.3 Stable Session Count 556 Definition: 557 Total number of control plane sessions/adjacencies 558 established and maintained by the DUT during the Startup 559 Phase and prior to Instability Conditions being initiated. 561 Discussion: 562 This measurement should be made after the Control 563 Plane Startup Conditions are applied to the DUT. 565 Measurement units: 566 sessions 568 Issues: 569 None 571 See Also: 572 Startup Phase 574 3.3.3 Control Plane 576 3.3.3.1 Control Plane Configuration Set 577 Definition: 578 The routing protocols and scaling values used for the Accelerated 579 Life Benchmarking. 581 Discussion: 582 Control Plane Configuration Set is shown in Figure 2 and specifies 583 the Routing Protocols, Multicast, and MPLS configuration. Specific 584 protocols can be enabled or disabled for a benchmark test. 586 Measurement units: 587 N/A 589 Issues: 590 None 592 See Also: 593 Data Plane Configuration Set 594 Management Configuration Set 595 Security Configuration Set 597 ____________ ____________ ____________ 598 | Routing | | Multicast | | MPLS | 599 | Protocols |___ | Protocols | __| Protocols | 600 | | | | | | | | 601 ------------ | ------------ | ------------ 602 | | | 603 | | | 604 | \/ | 605 | ___________ | 606 | | | | 607 |------->| DUT |<------| 608 ``| | 609 ----------- 610 Figure 2. Control Plane Configuration Module 611 3.3.3.2 Control Plane Startup Conditions 613 Definition: 614 Control Plane conditions that occur at the start 615 of the Accelerated Stress Benchmarking to establish conditions 616 for the remainder of the test. 618 Discussion: 619 Startup Conditions may cause stress on the DUT and produce 620 failure. Startup Conditions for the Control Plane include 621 session establishment rate, number of sessions established 622 and number of routes learned. 624 Measurement units: 625 N/A 627 Issues: 628 None 630 See Also: 631 Startup Conditions 632 Security Plane Startup Conditions 633 Control Plane Configuration Set 635 3.3.4 Data Plane 636 3.3.4.1 Data Plane Configuration Set 638 Definition: 639 The data traffic profile enabled for the Accelerated Stress 640 Benchmarking. 642 Discussion: 643 Data Plane Configuration Set includes the Traffic Profile and 644 interfaces used for the Accelerated Stress Benchmarking. 646 Measurement Units: 647 N/A 649 Issues: None 651 See Also: 652 Traffic Profile 654 3.3.4.2 Traffic Profile 655 Definition 656 The characteristics of the Offered Load to the DUT used for 657 the Accelerated Stress Benchmarking. 659 Discussion 660 The Traffic Profile specifies the number of packet size(s), packet 661 rate per interface, number of flows, and encapsulation used for the 662 offered load to the DUT. 664 Measurement Units: 665 Traffic Profile is reported as follows: 667 Parameter Units 668 --------- ------ 669 Packet Size(s) bytes 670 Packet Rate(interface) array of packets per second 671 Number of Flows number 672 Encapsulation(flow) array of encapsulation type 674 Issues: 675 None 677 See Also: 678 Data Plane Configuration Set 680 3.3.5 Management Plane 681 3.3.5.1 Management Plane Configuration Set 683 Definition: 684 The router management features enabled for the 685 Accelerated Stress Test. 687 Discussion: 688 A key component of the Accelerated Stress Test is the Management 689 Configuration Set to assess manageability of the router under 690 stress. The Management Configuration Set defines the management 691 configuration of the DUT. Features that are part of the 692 Management Configuration Set include Telnet access, SNMP, 693 Logging/Debug, and Statistics Collection, and services such as 694 FTP, as shown in Figure 3. These features should be enabled 695 throughout the Stress test. SNMP Gets should be made continuously 696 with multiple FTP and Telnet sessions operating simultaneously. 697 FTP sessions should be opened and closed at varying intervals 698 and get and put files while open. Telnet sessions should be 699 opened and closed at varying intervals and enter valid and invalid 700 show and configuration commands while open. 702 Measurement units: 703 N/A 705 Issues: 706 None 708 See Also: 709 Control Plane Configuration Set 710 Data Plane Configuration Set 711 Security Plane Configuration Set 713 ____________ ____________ 714 | | | Logging/ | 715 | SNMP | __| Debug | 716 | | | | | 717 ------------ | ------------ 718 | | 719 | | 720 \/ | 721 ___________ | 722 | | | 723 | DUT |<---| 724 | | 725 ----------- 726 | 727 | 728 \/ 729 ___________ 730 | Packet | 731 | Statistics| 732 | Collector | 733 | | 734 ----------- 736 Figure 3. Management Plane Configuration Set 738 3.3.6 Security Plane 739 3.3.6.1 Security Plane Configuration Set 741 Definition: 742 Security features and scaling enabled for the Accelerated Stress 743 Test. 745 Discussion: 746 The Security Plane Configuration Set includes the configuration 747 and scaling of ACLs, Firewall, IPsec, and User Access, as shown 748 in Figure 4. Tunnels should be established and policies 749 configured. Instability is introduced by flapping tunnels and 750 configuring and removing policies. 752 ____________ ____________ ____________ 753 | | | Secure | | User | 754 |ACL/Firewall| | Protocol | __| Access | 755 | | | | | | | 756 ------------ ------------ | ------------ 757 | | | 758 | | | 759 | \/ | 760 | ___________ | 761 | | | | 762 |------->| DUT |<--------| 763 | | 764 ----------- 765 Figure 4. Security Configuration Module 766 Measurement units: 767 N/A 769 Issues: 770 None 772 See Also: 773 ACL Configuration Set 774 Secure Protocol Configuration Set 775 Password Login Configuration Set 777 3.3.6.2 Security Plane Startup Conditions 778 Definition: 779 Security Plane conditions that occur at the start 780 of the Accelerated Stress Benchmarking to establish conditions 781 for the remainder of the test. 783 Discussion: 784 Startup Conditions may cause stress on the DUT and produce 785 failure. Startup Conditions for the Security Plane include 786 session establishment rate, number of sessions established 787 and number of policies learned, and number of user access 788 sessions opened. 790 Measurement units: 791 N/A 793 Issues: 794 None 796 See Also: 797 Startup Conditions 798 Data Plane Startup Conditions 799 Management Plane Startup Conditions 800 Security Plane Startup Conditions 802 3.4 Instability 804 3.4.1 Instability Phase 806 Definition: 807 The portion of the benchmarking test in which the 808 Instability Conditions are offered to the DUT. 810 Discussion: 811 The Instability Phase is the middle Phase of 812 of the benchmarking test following the Startup 813 Phase and preceding the Recovery Phase. 815 Measurement Units: 816 None 817 Issues: 818 None 820 See Also: 821 Instability Conditions 822 Startup Phase 823 Recovery Phase 825 3.4.2 Benchmarks 826 3.4.2.1 Unstable Aggregate Forwarding Rate 828 Definition: 829 Rate of traffic forwarded by the DUT during the 830 Instability Phase. 832 Discussion: 833 Unstable Aggregated Forwarding Rate is an instantaneous 834 measurement of the Aggregate Forwarding Rate during the 835 Instability Phase. 837 Measurement units: 838 pps 840 Issues: 841 None 843 See Also: 844 Instability Conditions 845 Aggregate Forwarding Rate 847 3.4.2.2 Degraded Aggregate Forwarding Rate 849 Definition: 850 The reduction in Aggregate Forwarding Rate during the 851 Instability Phase. 853 Discussion: 854 The Degraded Aggregate Forwarding Rate is calculated 855 for each measurement of the Unstable Aggregate 856 Forwarding Rate. The Degraded Aggregate Forwarding 857 Rate is calculated by subtracting each measurement 858 of the Unstable Aggregate Forwarding Rate from the 859 Stable Aggregate Forwarding Rate, such that 861 Degraded Forwarding Rate = 862 Stable Aggregate Forwarding Rate - 863 Unstable Aggregate Forwarding Rate 865 Ideally, the Degraded Aggregate Forwarding Rate is zero. 867 Measurement Units: 868 pps 869 Issues: 870 None 872 See Also: 873 Instability Phase 874 Unstable Aggregate Forwarding Rate 876 3.4.2.3 Average Degraded Aggregate Forwarding Rate 878 Definition 879 DUT Benchmark that is the calculated average of the 880 obtained Degraded Forwarding Rates. 882 Discussion: 884 Measurement Units: 885 pps 887 Issues: 888 None 890 See Also: 891 Degraded Aggregate Forwarding Rate 893 3.4.2.4 Unstable Latency 895 Definition: 896 The average increase in measured packet latency during 897 the Instability Phase compared to the Startup Phase. 899 Discussion: 900 Latency SHOULD be measured at a fixed interval during the 901 Instability Phase. Unstable Latency is the difference 902 between Stable Latency and the average Latency measured 903 during the Instability Phase. It is expected that there 904 be an increase in average latency from the Startup Phase 905 to the Instability phase, but it is possible that the 906 difference be zero. The Unstable Latency cannot be a 907 negative number. 909 Measurement units: 910 seconds 912 Issues: 913 None 915 See Also: 916 Instability Phase 917 Stable Latency 918 3.4.2.5 Unstable Uncontrolled Sessions Lost 920 Definition: 921 Control Plane sessions that are in the down state 922 but were not intentionally brought down during the 923 Instability Phase. 925 Discussion: 926 The test equipment is able to control protocol 927 session state with the DUT. The test equipment 928 is also to monitor for sessions lost with the 929 DUT which the test equipment itself did not 930 intentionally bring down. 932 Measurement units: 933 sessions 935 Issues: 936 None 938 See Also: 939 Controlled Session Loss 940 Uncontrolled Session Loss 942 3.4.3 Instability Conditions 944 3.4.3.1 Control Plane Instability Conditions 946 Definition: 947 Control Plane conditions that occur during the Accelerated Stress 948 Benchmark to produce instability and stress the DUT. 950 Discussion: 951 Control Plane Instability Conditions are experienced by the DUT 952 after the Startup Conditions have completed. Control Plane 953 Instability Conditions experienced by the DUT include session 954 loss, route withdrawal, and route cost changes. 956 Measurement units: 957 N/A 959 Issues: 960 None 962 See Also: 963 Instability Conditions 964 Data Plane Instability Conditions 965 Management Plane Instability Conditions 966 Security Plane Instability Conditions 967 3.4.3.2 Data Plane Instability Conditions 968 Definition: 969 Data Plane conditions that occur during the Accelerated Stress 970 Benchmark to produce instability and stress the DUT. 972 Discussion: 973 Data Plane Instability Conditions are experienced by the DUT 974 after the Startup Conditions have completed. Data Plane 975 Instability Conditions experienced by the DUT include interface 976 shutdown, link loss, and overloaded links. 978 Measurement units: 979 N/A 981 Issues: 982 None 984 See Also: 985 Instability Conditions 986 Control Plane Instability Conditions 987 Management Plane Instability Conditions 988 Security Plane Instability Conditions 990 3.4.3.3 Management Plane Instability Conditions 991 Definition: 992 Management Plane conditions that occur during the Accelerated 993 Life Benchmark to produce instability and stress the DUT. 995 Discussion: 996 Management Plane Instability Conditions are experienced by the DUT 997 after the Startup Conditions have completed. Management Plane 998 Instability Conditions experienced by the DUT include repeated 999 FTP of large files. 1001 Measurement units: 1002 N/A 1004 Issues: 1005 None 1007 See Also: 1008 Instability Conditions 1009 Control Plane Instability Conditions 1010 Data Plane Instability Conditions 1011 Security Plane Instability Conditions 1013 3.4.3.4 Security Plane Instability Conditions 1015 Definition: 1016 Security Plane conditions that occur during the Accelerated 1017 Life Benchmark to produce instability and stress the DUT. 1019 Discussion: 1020 Security Plane Instability Conditions are experienced by the DUT 1021 after the Startup Conditions have completed. Security Plane 1022 Instability Conditions experienced by the DUT include session 1023 loss and uninitiated policy changes. 1025 Measurement units: 1026 N/A 1028 Issues: 1029 None 1031 See Also: 1032 Instability Conditions 1033 Control Plane Instability Conditions 1034 Data Plane Instability Conditions 1035 Management Plane Instability Conditions 1037 3.5 Recovery 1039 3.5.1 Recovery Phase 1041 Definition: 1042 The portion of the benchmarking test in which the 1043 Startup Conditions are generated with the DUT, but 1044 the Instability Conditions are no longer offered to 1045 the DUT. 1047 Discussion: 1048 The Recovery Phase is the final Phase of the 1049 benchmarking test following the Startup Phase and 1050 Instability Phase. Startup Conditions must not be 1051 Restarted. 1053 Measurement Units: 1054 None 1056 Issues: 1057 None 1059 See Also: 1060 Startup Conditions 1061 Startup Phase 1062 Instability Conditions 1063 Instability Phase 1065 3.5.2 Benchmarks 1066 3.5.2.1 Recovered Aggregate Forwarding Rate 1068 Definition 1069 Rate of traffic forwarded by the DUT during the Recovery 1070 Phase. 1072 Discussion: 1073 Recovered Aggregate Forwarding Rate is an instantaneous 1074 measurement of the Aggregate Forwarding Rate during the 1075 Recovery Phase. Ideally, each measurement of the Recovered 1076 Aggregate Forwarding Rate equals the Stable Aggregate 1077 Forwarding Rate because the Instability Conditions 1078 do not exist in both the Startup and Recovery Phases. 1080 Measurement Units: 1081 pps 1083 Issues: 1084 None 1086 See Also: 1087 Aggregate Forwarding Rate 1088 Recovery Phase 1089 Recovered Aggregate Forwarding Rate 1090 Startup Phase 1091 Stable Aggregate Forwarding Rate 1093 3.5.2.2 Recovered Latency 1095 Definition: 1096 The average increase in measured packet latency during 1097 the Recovery Phase compared to the Startup Phase. 1099 Discussion: 1100 Latency SHOULD be measured at a fixed interval during the 1101 Recovery Phase. Unstable Latency is the difference 1102 between Stable Latency and the average Latency measured 1103 during the Recovery Phase. It is expected that there 1104 be no increase in average latency from the Startup Phase 1105 to the Recovery Phase. The Recovered Latency cannot be a 1106 negative number. 1108 Measurement units: 1109 seconds 1111 Issues: None 1113 See Also: 1114 Recovery Phase 1115 Stable Latency 1117 3.5.2.3 Recovery Time 1119 Definition 1120 The amount of time for the Recovered Aggregate Forwarding 1121 Rate to become equal to the Stable Aggregate Forwarding Rate. 1123 Discussion 1124 Recovery Time is measured beginning at the instant the 1125 Instability Phase ends until the Recovered Aggregate 1126 Forwarding Rate equals the Stable Aggregate Forwarding 1127 Rate for a minimum duration of 180 consecutive seconds. 1129 Measurement Units: 1130 seconds 1132 Issues: 1133 None 1135 See Also: 1136 Recovered Aggregate Forwarding Rate 1137 Stable Aggregate Forwarding Rate 1139 3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost 1141 Definition: 1142 Control Plane sessions that are in the down state 1143 but were not intentionally brought down during the 1144 Recovery Phase. 1146 Discussion: 1147 The test equipment is able to control protocol 1148 session state with the DUT. The test equipment 1149 is also to monitor for sessions lost with the 1150 DUT which the test equipment itself did not 1151 intentionally bring down. 1153 Measurement units: 1154 sessions 1156 Issues: 1157 None 1159 See Also: 1160 Controlled Session Loss 1161 Uncontrolled Session Loss 1163 3.5.2.5 Variability Benchmarks 1165 Definition: 1166 The difference between the measured Benchmarks of the 1167 same DUT over multiple iterations. 1169 Discussion: 1170 Ideally, the benchmarks measured should be the same for 1171 multiple iterations with the same DUT. Configuration 1172 Sets Instability conditions SHOULD be held constant for 1173 this benchmark. Whether the DUT can exhibit such predictable 1174 and repeatable behavior is an important benchmark in itself. 1176 Measurement units: 1177 As applicable to each Benchmark. The results are to be 1178 presented in a table format for successive Iterations. 1179 Ideally, the differences should be zero. 1181 Issues: 1182 None 1184 See Also: 1185 Startup Period 1186 Instability Period 1187 Recovery Period 1189 4. Security Considerations 1190 Documents of this type do not directly effect the security of 1191 the Internet or of corporate networks as long as benchmarking 1192 is not performed on devices or systems connected to operating 1193 networks. 1195 5. Normative References 1196 [1] Bradner, S., Editor, "Benchmarking Terminology for Network 1197 Interconnection Devices", RFC 1242, October 1991. 1199 [2] Mandeville, R., "Benchmarking Terminology for LAN Switching 1200 Devices", RFC 2285, June 1998. 1202 [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for 1203 Network Interconnect Devices", RFC 2544, March 1999. 1205 [4] Poretsky, S. and Rao, S., "Methodology for Accelerated 1206 Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-01, 1207 work in progress, February 2005. 1209 6. Informative References 1210 [RFC3871] RFC 3871 "Operational Security Requirements for Large 1211 Internet Service Provider (ISP) IP Network Infrastructure. 1212 G. Jones, Ed.. IETF, September 2004. 1214 [NANOG25] "Core Router Evaluation for Higher Availability", Scott 1215 Poretsky, NANOG 25, June 8, 2002, Toronto, CA. 1217 [IEEECQR] "Router Stress Testing to Validate Readiness for Network 1218 Deployment", Scott Poretsky, IEEE CQR 2003. 1220 [CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data Plane 1221 Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-05, 1222 work in progress, February 2005. 1224 [CONVTERM] Poretsky, S., "Benchmarking Terminology for IGP Data Plane 1225 Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-05, 1226 work in progress, February 2005. 1228 7. Author's Address 1230 Scott Poretsky 1231 Quarry Technologies 1232 8 New England Executive Park 1233 Burlington, MA 01803 1234 USA 1235 Phone: + 1 781 395 5090 1236 EMail: sporetsky@quarrytech.com 1238 Shankar Rao 1239 Qwest Communications 1240 Denver, CO 1241 USA 1242 Phone: + 1 303 437 6643 1243 Email: shankar.rao@qwest.com 1245 Appendix 1. White Box Benchmarking Terminology 1246 Minimum Available Memory 1247 Definition: 1248 Minimum DUT Available Memory during the duration of the 1249 Accelerated Stress Test. 1251 Discussion: 1252 It is necessary to monitor DUT memory to measure this 1253 benchmark. 1255 Measurement units: 1256 bytes 1258 Issues: None 1260 See Also: 1261 Maximum CPU Utilization 1263 Maximum CPU Utilization 1264 Definition: 1265 Maximum DUT CPU utilization during the duration of the 1266 Accelerated Stress Test. 1268 Discussion: 1269 It is necessary to monitor DUT CPU Utilization to measure 1270 this benchmark. 1272 Measurement units: % 1274 Issues: None 1276 See Also: 1277 Minimum Available Memory 1278 Intellectual Property Statement 1280 The IETF takes no position regarding the validity or scope of any Intel- 1281 lectual Property Rights or other rights that might be claimed to pertain 1282 to the implementation or use of the technology described in this docu- 1283 ment or the extent to which any license under such rights might or might 1284 not be available; nor does it represent that it has made any independent 1285 effort to identify any such rights. Information on the procedures with 1286 respect to rights in RFC documents can be found in BCP 78 and BCP 79. 1288 Copies of IPR disclosures made to the IETF Secretariat and any 1289 assurances of licenses to be made available, or the result of an attempt 1290 made to obtain a general license or permission for the use of such 1291 proprietary rights by implementers or users of this specification can be 1292 obtained from the IETF on-line IPR repository at 1293 http://www.ietf.org/ipr. 1295 The IETF invites any interested party to bring to its attention any 1296 copyrights, patents or patent applications, or other proprietary rights 1297 that may cover technology that may be required to implement this stan- 1298 dard. Please address the information to the IETF at ietf-ipr@ietf.org. 1300 Disclaimer of Warranty 1302 This document and the information contained herein are provided on an 1303 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR 1304 IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 1305 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 1306 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMA- 1307 TION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 1308 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1310 Copyright Statement 1312 Copyright (C) The Internet Society (2005). This document is subject to 1313 the rights, licenses and restrictions contained in BCP 78, and except as 1314 set forth therein, the authors retain all their rights.