idnits 2.17.1 draft-ietf-bmwg-acc-bench-term-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard == It seems as if not all pages are separated by form feeds - found 0 form feeds but 24 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. (A line matching the expected section header was found, but with an unexpected indentation: ' 1. Introduction' ) ** The document seems to lack a Security Considerations section. (A line matching the expected section header was found, but with an unexpected indentation: ' 4. Security Considerations' ) ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack an Authors' Addresses Section. ** There are 531 instances of lines with control characters in the document. ** The abstract seems to contain references ([2], [3], [4], [5], [6], [1]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (January 2005) is 7041 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? '6' on line 1088 looks like a reference -- Missing reference section? '1' on line 1073 looks like a reference -- Missing reference section? '2' on line 1076 looks like a reference -- Missing reference section? '3' on line 1079 looks like a reference -- Missing reference section? '4' on line 1082 looks like a reference -- Missing reference section? '5' on line 1085 looks like a reference Summary: 7 errors (**), 0 flaws (~~), 4 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group 2 INTERNET-DRAFT 3 Expires in: January 2005 4 Scott Poretsky 5 Quarry Technologies 7 Shankar Rao 8 Qwest Communications 10 July 2004 12 Terminology for Accelerated Stress Benchmarking 14 16 Status of this Memo 18 This document is an Internet-Draft and is in full conformance with 19 all provisions of Section 10 of RFC2026. 21 Internet-Drafts are working documents of the Internet Engineering 22 Task Force (IETF), its areas, and its working groups. Note that 23 other groups may also distribute working documents as Internet- 24 Drafts. 26 Internet-Drafts are draft documents valid for a maximum of six 27 months and may be updated, replaced, or obsoleted by other 28 documents at any time. It is inappropriate to use Internet-Drafts 29 as reference material or to cite them other than as "work in 30 progress." 32 The list of current Internet-Drafts can be accessed at 33 http://www.ietf.org/ietf/1id-abstracts.txt 35 The list of Internet-Draft Shadow Directories can be accessed at 36 http://www.ietf.org/shadow.html. 38 ABSTRACT 39 This document provides the Terminology for performing Stress 40 Benchmarking of networking devices. The three phases of the Stress 41 Test: Startup, Instability and Recovery are defined along with the 42 benchmarks and configuration terms associated with the each phase. 43 Also defined are the Benchmark Planes fundamental to stress testing 44 configuration, setup and measurement. The terminology is to be 45 used with the companion framework and methodology documents. 47 Table of Contents 48 1. Introduction ............................................... 3 49 2. Existing definitions ....................................... 3 50 3. Term definitions............................................ 3 51 3.1 General Terms............................................. 3 52 3.1.1 Benchmark Planes...................................... 3 53 3.1.2 Configuration Sets.................................... 4 54 3.1.3 Startup Conditions.................................... 4 55 3.1.4 Instability Conditions................................ 5 56 3.1.5 Aggregate Forwarding Rate............................. 6 57 3.1.6 Controlled Session Loss............................... 6 58 3.1.7 Uncontrolled Session Loss............................. 6 59 3.2 Benchmark Planes.......................................... 7 60 3.2.1 Control Plane......................................... 7 61 3.2.2 Data Plane............................................ 7 62 3.2.3 Management Plane...................................... 8 63 3.2.4 Security Plane........................................ 8 64 3.3 Startup................................................... 9 65 3.3.1 Startup Phase......................................... 9 66 3.3.2 Benchmarks............................................10 67 3.3.2.1 Stable Aggregate Forwarding Rate..................10 68 3.3.2.2 Stable Session Count..............................11 69 3.3.3 Control Plane.........................................11 70 3.3.3.1 Control Plane Configuration Set...................11 71 3.3.3.2 Control Plane Startup Conditions..................12 72 3.3.4 Data Plane............................................12 73 3.3.4.1 Data Plane Configuration Set......................12 74 3.3.4.2 Traffic Profile...................................13 75 3.3.5 Management Plane......................................13 76 3.3.5.1 Management Plane Configuration Set................13 77 3.3.6 Security Plane........................................14 78 3.3.6.1 Security Plane Configuration Set..................14 79 3.3.6.2 Security Plane Startup Conditions.................15 80 3.4 Instability...............................................15 81 3.4.1 Instability Phase.....................................15 82 3.4.2 Benchmarks............................................16 83 3.4.2.1 Unstable Aggregate Forwarding Rate................16 84 3.4.2.2 Degraded Aggregate Forwarding Rate................16 85 3.4.2.3 Average Degraded Aggregate Forwarding Rate........17 86 3.4.2.4 Unstable Uncontrolled Sessions Lost...............17 87 3.4.3 Instability Conditions................................18 88 3.4.3.1 Control Plane Instability Conditions..............18 89 3.4.3.2 Data Plane Instability Conditions.................18 90 3.4.3.3 Management Plane Instability Conditions...........19 91 3.4.3.4 Security Plane Instability Conditions.............19 92 3.5 Recovery..................................................20 93 3.5.1 Recovery Phase........................................20 94 3.5.2 Benchmarks............................................20 95 3.5.2.1 Recovered Aggregate Forwarding Rate...............20 96 3.5.2.2 Recovery Time.....................................21 97 3.5.2.3 Recovered Uncontrolled Sessions Lost..............21 98 4. Security Considerations.....................................22 99 5. References..................................................22 100 6. Author's Address............................................22 101 7. Full Copyright Statement....................................23 102 Appendix 1 - White Box Benchmarks..............................23 104 1. Introduction 106 Routers in an operational network are simultaneously configured with 107 multiple protocols and security policies while forwarding traffic and 108 being managed. To accurately benchmark a router for deployment it is 109 necessary to test that router in operational conditions by 110 simultaneously configuring and scaling network protocols and security 111 policies, forwarding traffic, and managing the device. It is helpful 112 to accelerate these network operational conditions so that the 113 router under test can be benchmarked with faster test duration. 114 Testing a router in accelerated network conditions is known as 115 Accelerated Stress Testing. 117 This document provides the Terminology for performing Stress 118 Benchmarking of networking devices. The three phases of the Stress 119 Test: Startup, Instability and Recovery are defined along with the 120 benchmark and configuration terms associated with the each phase. 121 Benchmarks for stress testing are defined using the Aggregate 122 Forwarding Rate and control plane Session Count during each phase 123 of the test. Also defined are the Benchmark Planes fundamental to 124 stress testing configuration, setup and measurement. These are 125 the Control Plane, Data Plane, Management Plane and Security Plane 126 For each plane, the Configuration Set, Startup Conditions, and 127 Instability Conditions are defined. White Box benchmarks are 128 provided in Appendix 1 for additional DUT behavior measurements. 129 The terminology is to be used with the companion methodology 130 document [6]. 132 2. Existing definitions 134 RFC 1242 "Benchmarking Terminology for Network Interconnect 135 Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching 136 Devices" should be consulted before attempting to make use of this 137 document. 139 For the sake of clarity and continuity this RFC adopts the template 140 for definitions set out in Section 2 of RFC 1242. Definitions are 141 indexed and grouped together in sections for ease of reference. 143 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 144 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 145 this document are to be interpreted as described in RFC 2119. 147 3. Term definitions 148 3.1 General Terms 149 3.1.1 Benchmark Planes 151 Definition: 152 The features, conditions, and behavior for the Accelerated Stress 153 Benchmarking. 155 Discussion: 156 There are four Benchmark Planes: Control Plane, Data Plane, 157 Management Plane, and Security Plane as shown in Figure 1. The 158 Benchmark Planes define the Configuration, Startup Conditions, 159 Instability Conditions, and Failure Conditions used for the test. 161 ___________ ___________ 162 | Control | | Management| 163 | Plane |___ ___| Plane | 164 | | | | | | 165 ----------- | | ----------- 166 \/ \/ ___________ 167 ___________ | Security | 168 | |<-----------| Plane | 169 | DUT | | | 170 |--->| |<---| ----------- 171 | ----------- | 172 | | 173 | ___________ | 174 | | Data | | 175 |--->| Plane |<---| 176 | | 177 ----------- 179 Figure 1. Router Accelerated Stress Benchmarking Planes 181 Measurement units: 182 N/A 184 Issues: 185 None 187 See Also: 188 Control Plane 189 Data Plane 190 Management Plane 191 Security Plane 193 3.1.2 Configuration Sets 195 Definition: 196 The features and scaling limits used during the Accelerated Stress 197 Benchmarking. 199 Discussion: 200 There are four Configuration Sets: Control Plane Configuration Set, 201 Data Plane Configuration Set, Management Plane Configuration Set, 202 and Security Plane Configuration Set. 204 Measurement units: 205 N/A 206 Issues: 207 None 209 See Also: 210 Control Plane Configuration Set 211 Data Plane Configuration Set 212 Management Plane Configuration Set 213 Security Plane Configuration Set 215 3.1.3 Startup Conditions 217 Definition: 218 Test conditions that occur at the start of the Accelerated 219 Life Benchmark to establish conditions for the remainder of 220 the test. 222 Discussion: 223 Startup Conditions may cause stress on the DUT and produce 224 failure. Startup Conditions are defined for the Control 225 Plane and Security Plane. 227 Measurement units: 228 N/A 230 Issues: 231 None 233 See Also: 234 Control Plane Startup Conditions 235 Data Plane Startup Conditions 236 Management Plane Startup Conditions 237 Security Plane Startup Conditions 239 3.1.4 Instability Conditions 241 Definition: 242 Test conditions that occur during the Accelerated Stress 243 Benchmark to produce instability and stress the DUT. 245 Discussion: 246 Instability Conditions are applied to the DUT after the 247 Startup Conditions have completed. Instability Conditions 248 occur for the Control Plane, Data Plane, Management Plane, 249 and Security Plane. 251 Measurement units: 252 N/A 254 Issues: 255 None 256 See Also: 257 Control Plane Instability Conditions 258 Data Plane Instability Conditions 259 Management Plane Instability Conditions 260 Security Plane Instability Conditions 262 3.1.5 Aggregate Forwarding Rate 264 Definition: 265 Sum of forwarding rates for all interfaces on the 266 DUT during the Startup Phase. 268 Discussion: 269 Each interface of the DUT forwards traffic at some 270 measured rate. The Aggregate Forwarding Rate is the 271 sum of forwarding rates for all interfaces on the DUT. 273 Measurement units: 274 pps 276 Issues: 277 None 279 See Also: 280 Startup Phase 282 3.1.6 Controlled Session Loss 284 Definition: 285 Control Plane sessions that are intentionally brought 286 down during the Stress test. 288 Discussion: 289 The test equipment is able to control protocol 290 session state with the DUT. 292 Measurement units: 293 None 295 Issues: 296 None 298 See Also: 299 Uncontrolled Session Loss 301 3.1.7 Uncontrolled Session Loss 303 Definition: 304 Control Plane sessions that are in the down state 305 but were not intentionally brought down during the 306 Stress test. 308 Discussion: 309 The test equipment is able to control protocol 310 session state with the DUT. The test equipment 311 is also to monitor for sessions lost with the 312 DUT which the test equipment itself did not 313 intentionally bring down. 315 Measurement units: 316 N/A 318 Issues: 319 None 321 See Also: 322 Controlled Session Loss 324 3.2 Benchmark Planes 326 3.2.1 Control Plane 327 Definition: 328 The Description of the control protocols enabled for 329 the Accelerated Stress Benchmarking. 331 Discussion: 332 The Control Plane defines the Configuration, Startup 333 Conditions, Instability Conditions, and Failure 334 Conditions of the control protocols used for the test. 335 Control Plane protocols may include routing protocols, 336 multicast protocols, and MPLS protocols. These can be 337 enabled or disabled for a benchmark test. 339 Measurement units: 340 N/A 342 Issues: 343 None 345 See Also: 346 Benchmark Planes 347 Control Plane Configuration Set 348 Control Plane Startup Conditions 349 Control Plane Instability Conditions 350 Control Plane Failure Conditions 352 3.2.2 Data Plane 354 Definition: 355 The data traffic profile used for the Accelerated Stress 356 Benchmarking. 358 Discussion: 359 The Data Plane defines the Configuration, Startup 360 Conditions, Instability Conditions, and Failure 361 Conditions of the data traffic used for the test. 362 The Data Plane includes the traffic and interface 363 profile. 365 Measurement Units: 366 N/A 368 See Also: 369 Benchmark Planes 370 Data Plane Configuration Set 371 Data Plane Startup Conditions 372 Data Plane Instability Conditions 373 Data Plane Failure Conditions 375 3.2.3 Management Plane 377 Definition: 378 The Management features and tools used for the 379 Accelerated Stress Benchmarking. 381 Discussion: 382 A key component of the Accelerated Stress Benchmarking is the 383 Management Plane to assess manageability of the router 384 under stress. The Management Plane defines the Configuration, 385 Startup Conditions, Instability Conditions, and Failure 386 Conditions of the management protocols and features used for 387 the test. The Management Plane includes SNMP, Logging/Debug, 388 and Statistics Collection. 390 Measurement units: 391 N/A 393 Issues: 394 None 396 See Also: 397 Benchmark Planes 398 Management Plane Configuration Set 399 Management Plane Startup Conditions 400 Management Plane Instability Conditions 401 Management Plane Failure Conditions 403 3.2.4 Security Plane 405 Definition: 406 The Security features used during the Accelerated Stress 407 Benchmarking. 409 Discussion: 410 The Control Plane defines the Configuration, Startup 411 Conditions, Instability Conditions, and Failure 412 Conditions of the security features and protocols used for 413 the test. The Security Plane includes the ACLs, Firewall, 414 Secure Protocols, and User Login. 416 Measurement units: 417 N/A 419 Issues: 420 None 422 See Also: 423 Benchmark Planes 424 Security Plane Configuration Set 425 Security Plane Startup Conditions 426 Security Plane Instability Conditions 427 Security Plane Failure Conditions 429 3.3 Startup 431 3.3.1 Startup Phase 433 Definition 434 The portion of the benchmarking test in which the 435 Startup Conditions are generated with the DUT. This 436 begins with the attempt to establish the first session 437 and ends when the last Control Plane session is 438 established. 440 Discussion: 441 The Startup Phase is the first Phase of the 442 benchmarking test preceding the Instability Phase 443 and Recovery Phase. It is specified by the 444 Configuration Sets and Startup Conditions for 445 each Benchmark Plane. The Startup Phase ends 446 and Instability Phase may begin when the 447 Configuration Sets are achieved with the DUT. 449 Measurement Units: 450 None 452 Issues: 453 The 'last control plane session is established' may not 454 be a sufficient indicator that steady-state is achieved 455 and Instability Conditions can be applied to begin the 456 Instability Phase. 458 See Also: 459 Benchmark Plane 460 Configuration Sets 461 Startup Conditions 462 Instability Phase 463 Recovery Phase 465 3.3.2 Benchmarks 467 3.3.2.1 Stable Aggregate Forwarding Rate 469 Definition: 470 Average rate of traffic forwarded by the DUT during the 471 Startup Phase. 473 Discussion: 474 Stable Aggregate Forwarding Rate is the calculated 475 average the Aggregate Forwarding Rates measured during 476 the Startup Phase. It is recommended that the 477 Aggregate Forwarding Rate is measured at one-second 478 intervals until the Startup Phase ends. 480 Measurement units: 481 pps 483 Issues: 484 The act of the DUT establishing the Startup Conditions 485 could influence the forwarding rate in certain 486 implementations so that this "baseline" for the 487 remainder of the test is lowered. The alternative is 488 to change the definition of Startup Aggregate 489 Forwarding Rate so that it measured during the Startup 490 Phase, but after Startup Conditions are achieved. 491 The disadvantage of this definition would be that it 492 loses measurement of any impact establishing Startup 493 Conditions would have on forwarding rate. When 494 comparing the Startup Aggregate Forwarding Rate benchmark 495 of two devices it is preferred to know the impact 496 establishing Startup Conditions has on Forwarding Rate. 497 The definition was therefore selected so that Stable 498 Aggregate Forwarding Rate is calculated from measurement 499 samples throughout the entire Startup Phase. 501 See Also: 502 Instability Conditions 503 Aggregate Forwarding Rate 504 Stable Aggregate Forwarding Rate 505 3.3.2.2 Stable Session Count 507 Definition: 508 Total number of control plane sessions/adjacencies 509 established and maintained by the DUT prior to 510 Instability Conditions being initiated. 512 Discussion: 513 This measurement should be made after the Control 514 Plane Startup Conditions are applied to the DUT. 516 Measurement units: 517 sessions 519 Issues: 520 None 522 See Also: 523 Instability Conditions 525 3.3.3 Control Plane 527 3.3.3.1 Control Plane Configuration Set 528 Definition: 529 The routing protocols and scaling values used for the Accelerated 530 Life Benchmarking. 532 Discussion: 533 Control Plane Configuration Set is shown in Figure 2 and specifies 534 the Routing Protocols, Multicast, and MPLS configuration. Specific 535 protocols can be enabled or disabled for a benchmark test. 537 Measurement units: 538 N/A 540 Issues: 541 None 543 ____________ ____________ ____________ 544 | Routing | | Multicast | | MPLS | 545 | Protocols |___ | Protocols | __| Protocols | 546 | | | | | | | | 547 ------------ | ------------ | ------------ 548 | | | 549 | | | 550 | \/ | 551 | ___________ | 552 | | | | 553 |------->| DUT |<------| 554 ``| | 555 ----------- 556 Figure 2. Control Plane Configuration Module 557 See Also: 558 Data Plane Configuration Set 559 Management Configuration Set 560 Security Configuration Set 562 3.3.3.2 Control Plane Startup Conditions 564 Definition: 565 Control Plane conditions that occur at the start 566 of the Accelerated Stress Benchmarking to establish conditions 567 for the remainder of the test. 569 Discussion: 570 Startup Conditions may cause stress on the DUT and produce 571 failure. Startup Conditions for the Control Plane include 572 session establishment rate, number of sessions established 573 and number of routes learned. 575 Measurement units: 576 N/A 578 Issues: 579 None 581 See Also: 582 Startup Conditions 583 Security Plane Startup Conditions 584 Control Plane Configuration Set 586 3.3.4 Data Plane 588 3.3.4.1 Data Plane Configuration Set 590 Definition: 591 The data traffic profile enabled for the Accelerated Stress 592 Benchmarking. 594 Discussion: 595 Data Plane Configuration Set includes the Traffic Profile and 596 interfaces used for the Accelerated Stress Benchmarking. 598 Measurement Units: 599 N/A 601 Issues: 602 None 604 See Also: 605 Traffic Profile 606 3.3.4.2 Traffic Profile 607 Definition 608 The characteristics of the Offered Load to the DUT used for 609 the Accelerated Stress Benchmarking. 611 Discussion 612 The Traffic Profile specifies the number of packet size(s), packet 613 rate per interface, number of flows, and encapsulation used for the 614 offered load to the DUT. 616 Measurement Units: 617 Traffic Profile is reported as follows: 619 Parameter Units 620 --------- ------ 621 Packet Size(s) bytes 622 Packet Rate(interface) array of packets per second 623 Number of Flows number 624 Encapsulation(flow) array of encapsulation type 626 Issues: 627 None 629 See Also: 630 Data Plane Configuration Set 632 3.3.5 Management Plane 633 3.3.5.1 Management Plane Configuration Set 635 Definition: 636 The router management features enabled for the 637 Accelerated Stress Test. 639 Discussion: 640 A key component of the Accelerated Stress Test is the Management 641 Configuration Set to assess manageability of the router under 642 stress. The Management Configuration Set defines the management 643 configuration of the DUT. Features that are part of the 644 Management Configuration Set include SNMP, Logging/Debug, and 645 Statistics Collection, and services such as FTP, as shown in 646 Figure 3. 648 Measurement units: 649 N/A 651 Issues: 652 None 654 See Also: 655 Control Plane Configuration Set 656 Data Plane Configuration Set 657 Security Plane Configuration Set 659 ____________ ____________ 660 | | | Logging/ | 661 | SNMP | __| Debug | 662 | | | | | 663 ------------ | ------------ 664 | | 665 | | 666 \/ | 667 ___________ | 668 | | | 669 | DUT |<---| 670 | | 671 ----------- 672 | 673 | 674 \/ 675 ___________ 676 | Packet | 677 | Statistics| 678 | Collector | 679 | | 680 ----------- 682 Figure 3. Management Plane Configuration Set 684 3.3.6 Security Plane 686 3.3.6.1 Security Plane Configuration Set 688 Definition: 689 Security features and scaling enabled for the Accelerated Stress 690 Test. 692 Discussion: 693 The Security Plane Configuration Set includes the configuration 694 and scaling of ACLs, Firewall, IPsec, and User Access, as shown 695 in Figure 4. 697 ____________ ____________ ____________ 698 | | | Secure | | User | 699 |ACL/Firewall| | Protocol | __| Access | 700 | | | | | | | 701 ------------ ------------ | ------------ 702 | | | 703 | | | 704 | \/ | 705 | ___________ | 706 | | | | 707 |------->| DUT |<--------| 708 | | 709 ----------- 710 Figure 4. Security Configuration Module 711 Measurement units: 712 N/A 714 Issues: 715 None 717 See Also: 718 ACL Configuration Set 719 Secure Protocol Configuration Set 720 Password Login Configuration Set 722 3.3.6.2 Security Plane Startup Conditions 724 Definition: 725 Security Plane conditions that occur at the start 726 of the Accelerated Stress Benchmarking to establish conditions 727 for the remainder of the test. 729 Discussion: 730 Startup Conditions may cause stress on the DUT and produce 731 failure. Startup Conditions for the Security Plane include 732 session establishment rate, number of sessions established 733 and number of policies learned, and number of user access 734 sessions opened. 736 Measurement units: 737 N/A 739 Issues: 740 None 742 See Also: 743 Startup Conditions 744 Data Plane Startup Conditions 745 Management Plane Startup Conditions 746 Security Plane Startup Conditions 748 3.4 Instability 750 3.4.1 Instability Phase 752 Definition: 753 The portion of the benchmarking test in which the 754 Instability Conditions are offered to the DUT. 756 Discussion: 757 The Instability Phase is the middle Phase of 758 of the benchmarking test following the Startup 759 Phase and preceding the Recovery Phase. 761 Measurement Units: 762 None 763 Issues: 764 None 766 See Also: 767 Instability Conditions 768 Startup Phase 769 Recovery Phase 771 3.4.2 Benchmarks 773 3.4.2.1 Unstable Aggregate Forwarding Rate 775 Definition: 776 Rate of taffic forwarded by the DUT during the 777 Instability Phase. 779 Discussion: 780 Unstable Aggregated Forwarding Rate is an instantaneous 781 measurement of the Aggregate Forwarding Rate during the 782 Instability Phase. It is recommended that the 783 Unstable Aggregate Forwarding Rate is measured at 784 one-second intervals. 786 Measurement units: 787 pps 789 Issues: 790 None 792 See Also: 793 Instability Conditions 794 Aggregate Forwarding Rate 796 3.4.2.2 Degraded Aggregate Forwarding Rate 798 Definition: 799 The reduction in Aggregate Forwarding Rate during the 800 Instability Phase. 802 Discussion: 803 The Degraded Aggregate Forwarding Rate is calculated 804 for each measurement of the Unstable Aggregate 805 Forwarding Rate. The Degraded Aggregate Forwarding 806 Rate is calculated by subtracting each measurement 807 of the Unstable Aggregate Forwarding Rate from the 808 Stable Aggregate Forwarding Rate, such that 810 Degraded Forwarding Rate = 811 Stable Aggregate Forwarding Rate - 812 Unstable Aggregate Forwarding Rate 814 Ideally, the Degraded Aggregate Forwarding Rate is zero. 816 Measurement Units: 817 pps 819 Issues: 820 None 822 See Also: 823 Instability Phase 824 Unstable Aggregate Forwarding Rate 826 3.4.2.3 Average Degraded Aggregate Forwarding Rate 828 Definition 829 DUT Benchmark that is the calculated average of the 830 obtained Degraded Forwarding Rates. 832 Discussion: 834 Measurement Units: 835 pps 837 Issues: 838 None 840 See Also: 841 Degraded Aggregate Forwarding Rate 843 3.4.2.4 Unstable Uncontrolled Sessions Lost 845 Definition: 846 Control Plane sessions that are in the down state 847 but were not intentionally brought down during the 848 Instability Phase. 850 Discussion: 851 The test equipment is able to control protocol 852 session state with the DUT. The test equipment 853 is also to monitor for sessions lost with the 854 DUT which the test equipment itself did not 855 intentionally bring down. 857 Measurement units: 858 sessions 860 Issues: 861 None 863 See Also: 864 Controlled Session Loss 865 Uncontrolled Session Loss 866 3.4.3 Instability Conditions 868 3.4.3.1 Control Plane Instability Conditions 870 Definition: 871 Control Plane conditions that occur during the Accelerated Stress 872 Benchmark to produce instability and stress the DUT. 874 Discussion: 875 Control Plane Instability Conditions are experienced by the DUT 876 after the Startup Conditions have completed. Control Plane 877 Instability Conditions experienced by the DUT include session 878 loss, route withdrawal, and route cost changes. 880 Measurement units: 881 N/A 883 Issues: 884 None 886 See Also: 887 Instability Conditions 888 Data Plane Instability Conditions 889 Management Plane Instability Conditions 890 Security Plane Instability Conditions 892 3.4.3.2 Data Plane Instability Conditions 894 Definition: 895 Data Plane conditions that occur during the Accelerated Stress 896 Benchmark to produce instability and stress the DUT. 898 Discussion: 899 Data Plane Instability Conditions are experienced by the DUT 900 after the Startup Conditions have completed. Data Plane 901 Instability Conditions experienced by the DUT include interface 902 shutdown, link loss, and overloaded links. 904 Measurement units: 905 N/A 907 Issues: 908 None 910 See Also: 911 Instability Conditions 912 Control Plane Instability Conditions 913 Management Plane Instability Conditions 914 Security Plane Instability Conditions 915 3.4.3.3 Management Plane Instability Conditions 917 Definition: 918 Management Plane conditions that occur during the Accelerated 919 Life Benchmark to produce instability and stress the DUT. 921 Discussion: 922 Management Plane Instability Conditions are experienced by the DUT 923 after the Startup Conditions have completed. Management Plane 924 Instability Conditions experienced by the DUT include repeated 925 FTP of large files. 927 Measurement units: 928 N/A 930 Issues: 931 None 933 See Also: 934 Instability Conditions 935 Control Plane Instability Conditions 936 Data Plane Instability Conditions 937 Security Plane Instability Conditions 939 3.4.3.4 Security Plane Instability Conditions 941 Definition: 942 Security Plane conditions that occur during the Accelerated 943 Life Benchmark to produce instability and stress the DUT. 945 Discussion: 946 Security Plane Instability Conditions are experienced by the DUT 947 after the Startup Conditions have completed. Security Plane 948 Instability Conditions experienced by the DUT include session 949 loss and uninitiated policy changes. 951 Measurement units: 952 N/A 954 Issues: 955 None 957 See Also: 958 Instability Conditions 959 Control Plane Instability Conditions 960 Data Plane Instability Conditions 961 Management Plane Instability Conditions 962 3.5 Recovery 964 3.5.1 Recovery Phase 966 Definition: 967 The portion of the benchmarking test in which the 968 Startup Conditions are generated with the DUT, but 969 the Instability Conditions are no longer offered to 970 the DUT. 972 Discussion: 973 The Recovery Phase is the final Phase of the 974 benchmarking test following the Startup Phase and 975 Instability Phase. Startup Conditions must not be 976 Restarted. 978 Measurement Units: 979 None 981 Issues: 982 None 984 See Also: 985 Startup Conditions 986 Startup Phase 987 Instability Conditions 988 Instability Phase 990 3.5.2 Benchmarks 992 3.5.2.1 Recovered Aggregate Forwarding Rate 994 Definition 995 Rate of traffic forwarded by the DUT during the Recovery 996 Phase. 998 Discussion: 999 Recovered Aggregate Forwarding Rate is an instantaneous 1000 measurement of the Aggregate Forwarding Rate during the 1001 Recovery Phase. It is recommended that the Recovered 1002 Aggregate Forwarding Rate is measured at one-second 1003 intervals. Ideally, each measurement of the Recovered 1004 Aggregate Forwarding Rate equals the Stable Aggregate 1005 Forwarding Rate because the Instability Conditions 1006 do not exist in both the Startup and Recovery Phases. 1008 Measurement Units: 1009 pps 1011 Issues: 1012 None 1013 See Also: 1014 Aggregate Forwarding Rate 1015 Recovery Phase 1016 Recovered Aggregate Forwarding Rate 1017 Startup Phase 1018 Stable Aggregate Forwarding Rate 1020 3.5.2.2 Recovery Time 1022 Definition 1023 The amount of time for the Recovered Aggregate Forwarding 1024 Rate to become equal to the Stable Aggregate Forwarding Rate. 1026 Discussion 1027 Recovery Time is measured beginning at the instant the 1028 Instability Phase ends until the Recovered Aggregate 1029 Forwarding Rate equals the Stable Aggregate Forwarding 1030 Rate for a minimum of 180 consecutive seconds. 1032 Measurement Units: 1033 seconds 1035 Issues: 1036 None 1038 See Also: 1039 Recovered Aggregate Forwarding Rate 1040 Stable Aggregate Forwarding Rate 1042 3.5.2.3 Recovered Uncontrolled Control Plane Sessions Lost 1044 Definition: 1045 Control Plane sessions that are in the down state 1046 but were not intentionally brought down during the 1047 Recovery Phase. 1049 Discussion: 1050 The test equipment is able to control protocol 1051 session state with the DUT. The test equipment 1052 is also to monitor for sessions lost with the 1053 DUT which the test equipment itself did not 1054 intentionally bring down. 1056 Measurement units: 1057 sessions 1059 Issues: 1060 None 1062 See Also: 1063 Controlled Session Loss 1064 Uncontrolled Session Loss 1065 4. Security Considerations 1066 Documents of this type do not directly effect the security of 1067 the Internet or of corporate networks as long as benchmarking 1068 is not performed on devices or systems connected to operating 1069 networks. 1071 5. References 1073 [1] Bradner, S., Editor, "Benchmarking Terminology for Network 1074 Interconnection Devices", RFC 1242, July 1991. 1076 [2] Mandeville, R., "Benchmarking Terminology for LAN Switching 1077 Devices", RFC 2285, June 1998. 1079 [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for 1080 Network Interconnect Devices", RFC 2544, March 1999. 1082 [4] "Core Router Evaluation for Higher Availability", Scott 1083 Poretsky, NANOG 25, June 8, 2002, Toronto, CA. 1085 [5] "Router Stress Testing to Validate Readiness for Network 1086 Deployment", Scott Poretsky, IEEE CQR 2003. 1088 [6] Poretsky, S. and Rao, S., "Methodology for Accelerated 1089 Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-00, 1090 work in progress, July 2004. 1092 6. Author's Address 1094 Scott Poretsky 1095 Quarry Technologies 1096 8 New England Executive Park 1097 Burlington, MA 01803 1098 USA 1099 Phone: + 1 781 395 5090 1100 EMail: sporetsky@quarrytech.com 1102 Shankar Rao 1103 Qwest Communications 1104 Denver, CO 1105 USA 1106 Phone: + 1 303 437 6643 1107 Email: shankar.rao@qwest.com 1108 7. Full Copyright Statement 1110 Copyright (C) The Internet Society (1998). All Rights 1111 Reserved. 1113 This document and translations of it may be copied and 1114 furnished to others, and derivative works that comment on or 1115 otherwise explain it or assist in its implementation may be 1116 prepared, copied, published and distributed, in whole or in 1117 part, without restriction of any kind, provided that the above 1118 copyright notice and this paragraph are included on all such 1119 copies and derivative works. However, this document itself may 1120 not be modified in any way, such as by removing the copyright 1121 notice or references to the Internet Society or other Internet 1122 organizations, except as needed for the purpose of developing 1123 Internet standards in which case the procedures for copyrights 1124 defined in the Internet Standards process must be followed, or 1125 as required to translate it into languages other than English. 1127 The limited permissions granted above are perpetual and will 1128 not be revoked by the Internet Society or its successors or 1129 assigns. This document and the information contained herein is 1130 provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE 1131 INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, 1132 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY 1133 THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY 1134 RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS 1135 FOR A PARTICULAR PURPOSE. 1137 Appendix 1. White Box Benchmarking Terminology 1139 Minimum Available Memory 1141 Definition: 1142 Minimum DUT Available Memory during the duration of the 1143 Accelerated Stress Test. 1145 Discussion: 1146 It is necessary to monitor DUT memory to measure this 1147 benchmark. 1149 Measurement units: 1150 bytes 1152 Issues: 1153 None 1155 See Also: 1156 Maximum CPU Utilization 1157 Maximum CPU Utilization 1159 Definition: 1160 Maximum DUT CPU utilization during the duration of the 1161 Accelerated Stress Test. 1163 Discussion: 1164 It is necessary to monitor DUT CPU Utilization to measure 1165 this benchmark. 1167 Measurement units: 1168 % 1170 Issues: 1171 None 1173 See Also: 1174 Minimum Available Memory