idnits 2.17.1 draft-ietf-bmwg-acc-bench-term-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 20. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 1341. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1352. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1359. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1365. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 434 instances of lines with control characters in the document. == There are 24 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (September 2007) is 6066 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: '3' is defined on line 1247, but no explicit reference was found in the text == Unused Reference: 'RFC3871' is defined on line 1259, but no explicit reference was found in the text == Unused Reference: 'NANOG25' is defined on line 1263, but no explicit reference was found in the text == Unused Reference: 'IEEECQR' is defined on line 1266, but no explicit reference was found in the text == Outdated reference: A later version (-09) exists of draft-ietf-bmwg-acc-bench-meth-07 Summary: 2 errors (**), 0 flaws (~~), 7 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group 2 INTERNET-DRAFT 3 Expires in: September 2007 4 Intended Status: Informational 5 Scott Poretsky 6 Reef Point Systems 8 Shankar Rao 9 Qwest Communications 11 March 2007 13 Terminology for Accelerated Stress Benchmarking 14 16 Intellectual Property Rights (IPR) statement: 17 By submitting this Internet-Draft, each author represents that any 18 applicable patent or other IPR claims of which he or she is aware 19 have been or will be disclosed, and any of which he or she becomes 20 aware will be disclosed, in accordance with Section 6 of BCP 79. 22 Status of this Memo 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF), its areas, and its working groups. Note that 26 other groups may also distribute working documents as 27 Internet-Drafts. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 The list of current Internet-Drafts can be accessed at 35 http://www.ietf.org/ietf/1id-abstracts.txt. 37 The list of Internet-Draft Shadow Directories can be accessed at 38 http://www.ietf.org/shadow.html. 40 Copyright Notice 41 Copyright (C) The IETF Trust (2007). 43 ABSTRACT 44 This document provides the Terminology for performing Accelerated 45 Stress Benchmarking of networking devices. The three phases of 46 the Stress Test: Startup, Instability and Recovery are defined 47 along with the benchmarks and configuration terms associated with 48 the each phase. Also defined are the Benchmark Planes fundamental 49 to stress testing configuration, setup and measurement. The 50 terminology is to be used with the companion framework and 51 methodology documents. 53 Table of Contents 54 1. Introduction ............................................... 3 55 2. Existing definitions ....................................... 3 56 3. Term definitions............................................ 4 57 Stress Benchmarking 59 3.1 General Terms............................................. 4 60 3.1.1 Benchmark Planes...................................... 4 61 3.1.2 Configuration Sets.................................... 5 62 3.1.3 Startup Conditions.................................... 5 63 3.1.4 Instability Conditions................................ 6 64 3.1.5 Aggregate Forwarding Rate............................. 6 65 3.1.6 Controlled Session Loss............................... 7 66 3.1.7 Uncontrolled Session Loss............................. 7 67 3.2 Benchmark Planes.......................................... 8 68 3.2.1 Control Plane......................................... 8 69 3.2.2 Data Plane............................................ 8 70 3.2.3 Management Plane...................................... 8 71 3.2.4 Security Plane........................................ 9 72 3.3 Startup...................................................10 73 3.3.1 Startup Phase.........................................10 74 3.3.2 Benchmarks............................................10 75 3.3.2.1 Stable Aggregate Forwarding Rate..................10 76 3.3.2.2 Stable Latency....................................11 77 3.3.2.3 Stable Session Count..............................11 78 3.3.3 Control Plane.........................................12 79 3.3.3.1 Control Plane Configuration Set...................12 80 3.3.3.2 Control Plane Startup Conditions..................13 81 3.3.4 Data Plane............................................13 82 3.3.4.1 Data Plane Configuration Set......................13 83 3.3.4.2 Traffic Profile...................................13 84 3.3.5 Management Plane......................................14 85 3.3.5.1 Management Plane Configuration Set................14 86 3.3.6 Security Plane........................................15 87 3.3.6.1 Security Plane Configuration Set..................15 88 3.3.6.2 Security Plane Startup Conditions.................16 89 3.4 Instability...............................................16 90 3.4.1 Instability Phase.....................................16 91 3.4.2 Benchmarks............................................17 92 3.4.2.1 Unstable Aggregate Forwarding Rate................17 93 3.4.2.2 Aggregate Forwarding Rate Degradation.............17 94 3.4.2.3 Average Aggregate Forwarding Rate Degradation.....18 95 3.4.2.4 Unstable Latency..................................18 96 3.4.2.5 Unstable Uncontrolled Sessions Lost...............19 97 3.4.3 Instability Conditions................................19 98 3.4.3.1 Control Plane Instability Conditions..............19 99 3.4.3.2 Data Plane Instability Conditions.................20 100 3.4.3.3 Management Plane Instability Conditions...........20 101 3.4.3.4 Security Plane Instability Conditions.............20 102 3.5 Recovery..................................................21 103 3.5.1 Recovery Phase........................................21 104 3.5.2 Benchmarks............................................21 105 3.5.2.1 Recovered Aggregate Forwarding Rate...............21 106 3.5.2.2 Recovered Latency.................................22 107 3.5.2.3 Recovery Time.....................................22 108 3.5.2.4 Recovered Uncontrolled Sessions Lost..............23 109 3.5.2.5 Variability Benchmarks............................23 110 4. IANA Considerations.........................................24 111 Stress Benchmarking 113 5. Security Considerations.....................................24 114 6. References..................................................24 115 7. Author's Address............................................25 116 Appendix 1 - White Box Benchmarks..............................25 118 1. Introduction 120 Routers in an operational network are configured with multiple 121 protocols and security policies while simultaneously forwarding 122 traffic and being managed. To accurately benchmark a router for 123 deployment, it is necessary to test that router under operational 124 conditions by simultaneously configuring and scaling network 125 protocols and security policies, forwarding traffic, and managing 126 the device. It is useful to accelerate these network operational 127 conditions so that the router under test can be benchmarked with 128 a shorter test duration. Testing a router in accelerated network 129 conditions is known as Accelerated Stress Benchmarking. 131 This document provides the Terminology for performing Stress 132 Benchmarking of networking devices. The three phases of the Stress 133 Test: Startup, Instability and Recovery are defined along with the 134 benchmark and configuration terms associated with the each phase. 135 Benchmarks for stress testing are defined using the Aggregate 136 Forwarding Rate and control plane Session Count during each phase 137 of the test. For each plane, the Configuration Set, Startup 138 Conditions, and Instability Conditions are defined. Also defined are 139 the Benchmark Planes fundamental to stress testing configuration, 140 setup and measurement. These are the Control Plane, Data Plane, 141 Management Plane and Security Plane Multiple benchmarks are measured 142 for each Benchmark Plane during each Phase. Benchmarks can be 143 compared across multiple planes for the same DUT or at the same 144 plane for 2 or more DUTS. Benchmarks of internal DUT characteristics 145 such as memory and CPU utilization (also known as White Box 146 benchmarks) are described in Appendix 1, to allow additional 147 characterization of DUT behavior. The terminology is to be used with 148 the companion methodology document [4]. The sequence of phases, 149 actions, and benchmarks are shown in Table 1. 151 2. Existing definitions 152 RFC 1242 [1] and RFC 2285 [2] should be consulted before 153 attempting to make use of this document. For the sake of clarity 154 and continuity this RFC adopts the template for definitions set 155 out in Section 2 of RFC 1242. Definitions are indexed and grouped 156 together in sections for ease of reference. 158 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 159 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 160 document are to be interpreted as described in BCP 14, RFC 2119 161 [5]. RFC 2119 defines the use of these key words to help make the 162 intent of standards track documents as clear as possible. While this 163 document uses these keywords, it is not a standards track document. 165 Stress Benchmarking 167 Table 1. Phase Sequence and Benchmarks 168 III. Recovery Phase II. Instability Phase I. Startup Phase 169 <-----------------<---<-------------------<----<--------------< 170 Remove Instability Achieve Configuration Apply Startup 171 Conditions Set Conditions 173 Benchmark: Benchmark: Benchmark: 174 Recovered Aggregate Unstable Aggregate Stable Aggregate 175 Forwarding Rate Forwarding Rate Forwarding Rate 177 Degraded Aggregate 178 Forwarding Rate 180 Average Degraded 181 Forwarding Rate 183 Recovered Latency Unstable Latency Startup Latency 185 Recovered Uncontrolled Recovered Uncontrolled Stable Session Count 186 Sessions Lost Sessions Lost 188 Recovery Time 190 3. Term definitions 191 3.1 General Terms 192 3.1.1 Benchmark Planes 194 Definition: 195 The features, conditions, and behavior for the Accelerated Stress 196 Benchmarking. 198 Discussion: 199 There are four Benchmark Planes: Control Plane, Data Plane, 200 Management Plane, and Security Plane as shown in Figure 1. 201 Configuration, Startup Conditions, Instability Conditions, and 202 Failure Conditions used for each test are defined for each of 203 these four Benchmark Planes. 205 Measurement units: 206 N/A 208 Issues: 209 None 211 See Also: 212 Control Plane 213 Data Plane 214 Management Plane 215 Security Plane 216 Stress Benchmarking 218 ___________ ___________ 219 | Control | | Management| 220 | Plane |___ ___| Plane | 221 | | | | | | 222 ----------- | | ----------- 223 \/ \/ ___________ 224 ___________ | Security | 225 | |<-----------| Plane | 226 | DUT | | | 227 |--->| |<---| ----------- 228 | ----------- | 229 | | 230 | ___________ | 231 | | Data | | 232 |--->| Plane |<---| 233 | | 234 ----------- 236 Figure 1. Router Accelerated Stress Benchmarking Planes 238 3.1.2 Configuration Sets 240 Definition: 241 The offered load, features, and scaling limits used during the 242 Accelerated Stress Benchmarking. 244 Discussion: 245 There are four Configuration Sets: Control Plane Configuration 246 Set, Data Plane Configuration Set, Management Plane Configuration 247 Set, and Security Plane Configuration Set. The minimum 248 Configuration Set that MUST be used is discussed in the 249 Methodology document [4]. 251 Measurement units: 252 N/A 254 Issues: 255 None 257 See Also: 258 Control Plane Configuration Set 259 Data Plane Configuration Set 260 Management Plane Configuration Set 261 Security Plane Configuration Set 263 3.1.3 Startup Conditions 265 Definition: 266 Test conditions that occur at the start of the Accelerated 267 Stress Benchmark to establish conditions for the remainder of 268 the test. 270 Stress Benchmarking 272 Discussion: 273 Startup Conditions may cause stress on the DUT and produce 274 failure. Startup Conditions are defined for the Control 275 Plane and Security Plane. 277 Measurement units: 278 N/A 280 Issues: 281 None 283 See Also: 284 Control Plane Startup Conditions 285 Data Plane Startup Conditions 286 Management Plane Startup Conditions 287 Security Plane Startup Conditions 289 3.1.4 Instability Conditions 291 Definition: 292 Test conditions that occur during the Accelerated Stress 293 Benchmark to produce instability and stress the DUT. 295 Discussion: 296 Instability Conditions are applied to the DUT after the 297 Startup Conditions have completed. Instability Conditions 298 occur for the Control Plane, Data Plane, Management Plane, 299 and Security Plane. 301 Measurement units: 302 N/A 304 Issues: None 306 See Also: 307 Control Plane Instability Conditions 308 Data Plane Instability Conditions 309 Management Plane Instability Conditions 310 Security Plane Instability Conditions 312 3.1.5 Aggregate Forwarding Rate 314 Definition: 315 Sum of forwarding rates for all interfaces on the 316 DUT during the Startup Phase. 318 Discussion: 319 Each interface of the DUT forwards traffic at some 320 measured rate. The Aggregate Forwarding Rate is the 321 sum of forwarding rates for all interfaces on the DUT. 323 Stress Benchmarking 325 Measurement units: 326 pps 328 Issues: 329 None 331 See Also: 332 Startup Phase 334 3.1.6 Controlled Session Loss 336 Definition: 337 Control Plane sessions that are intentionally brought 338 down during the Stress test. 340 Discussion: 341 Controlled Session Loss is performed during the test in 342 order to stress the DUT by forcing it to tear down Control 343 Plane sessions while handling traffic. It is assumed that 344 the test equipment is able to control protocol session 345 state with the DUT and is therefore able to introduce 346 Controlled Session Loss. 348 Measurement units: 349 None 351 Issues: 352 None 354 See Also: 355 Uncontrolled Session Loss 357 3.1.7 Uncontrolled Session Loss 358 Definition: 359 Control Plane sessions that are in the down state 360 but were not intentionally brought down during the 361 Stress test. 363 Discussion: 364 The test equipment is able to control protocol 365 session state with the DUT. The test equipment 366 is also to monitor for sessions lost with the 367 DUT which the test equipment itself did not 368 intentionally bring down. 370 Measurement units: 371 N/A 373 Issues: 374 None 376 See Also: 377 Controlled Session Loss 378 Stress Benchmarking 380 3.2 Benchmark Planes 382 3.2.1 Control Plane 383 Definition: 384 The Description of the control protocols enabled for 385 the Accelerated Stress Benchmarking. 387 Discussion: 388 The Control Plane defines the Configuration, Startup 389 Conditions, and Instability Conditions of the control 390 protocols. Control Plane protocols MAY include routing 391 protocols, multicast protocols, and MPLS protocols. 392 These can be enabled or disabled for a benchmark test. 394 Measurement units: 395 N/A 397 Issues: 398 None 400 See Also: 401 Benchmark Planes 402 Control Plane Configuration Set 403 Control Plane Startup Conditions 404 Control Plane Instability Conditions 406 3.2.2 Data Plane 407 Definition: 408 The data traffic profile used for the Accelerated Stress 409 Benchmarking. 411 Discussion: 412 The Data Plane defines the Configuration, Startup 413 Conditions, and Instability Conditions of the data 414 traffic. The Data Plane includes the traffic and 415 interface profile. 417 Measurement Units: 418 N/A 420 See Also: 421 Benchmark Planes 422 Data Plane Configuration Set 423 Data Plane Startup Conditions 424 Data Plane Instability Conditions 426 3.2.3 Management Plane 428 Definition: 429 The Management features and tools used for the 430 Accelerated Stress Benchmarking. 432 Stress Benchmarking 434 Discussion: 435 A key component of the Accelerated Stress Benchmarking is the 436 Management Plane to assess manageability of the router 437 under stress. The Management Plane defines the Configuration, 438 Startup Conditions, and Instability Conditions of the 439 management protocols and features. The Management Plane 440 includes SNMP, Logging/Debug, Statistics Collection, and 441 management configuration sessions such as telnet, SSH, and 442 serial console. 444 Measurement units: 445 N/A 447 Issues: 448 None 450 See Also: 451 Benchmark Planes 452 Management Plane Configuration Set 453 Management Plane Startup Conditions 454 Management Plane Instability Conditions 456 3.2.4 Security Plane 458 Definition: 459 The Security features used during the Accelerated Stress 460 Benchmarking. 462 Discussion: 463 The Security Plane defines the Configuration, Startup 464 Conditions, and Instability Conditions of the security 465 features and protocols. The Security Plane includes the 466 ACLs, Firewall, Secure Protocols, and User Login. 468 Measurement units: 469 N/A 471 Issues: None 473 See Also: 474 Benchmark Planes 475 Security Plane Configuration Set 476 Security Plane Startup Conditions 477 Security Plane Instability Conditions 478 Stress Benchmarking 480 3.3 Startup 482 3.3.1 Startup Phase 484 Definition 485 The portion of the benchmarking test in which the 486 Startup Conditions are generated with the DUT. This 487 begins with the attempt to establish the first session 488 and ends when the last Control Plane session is 489 established. 491 Discussion: 492 The Startup Phase is the first Phase of the benchmarking 493 test preceding the Instability Phase and Recovery Phase. 494 It is specified by the Configuration Sets and Startup 495 Conditions for each Benchmark Plane. The Startup Phase ends 496 and Instability Phase MUST begin when the Configuration Sets 497 are achieved with the DUT. 499 Measurement Units: 500 None 502 Issues: 503 The 'last control plane session is established' may not 504 be a sufficient indicator that steady-state is achieved 505 and Instability Conditions can be applied to begin the 506 Instability Phase. 508 See Also: 509 Benchmark Plane 510 Configuration Sets 511 Startup Conditions 512 Instability Phase 513 Recovery Phase 515 3.3.2 Benchmarks 516 3.3.2.1 Stable Aggregate Forwarding Rate 518 Definition: 519 Average rate of traffic forwarded by the DUT during the 520 Startup Phase. 522 Discussion: 523 Stable Aggregate Forwarding Rate is the calculated 524 average of the Aggregate Forwarding Rates measured during 525 the Startup Phase. 527 Measurement units: 528 pps 529 Stress Benchmarking 531 Issues: 532 The act of the DUT establishing the Startup Conditions 533 could influence the forwarding rate in certain 534 implementations so that this "baseline" for the 535 remainder of the test is lowered. The alternative is 536 to change the definition of Stable Aggregate 537 Forwarding Rate so that it is measured during the Startup 538 Phase, but after Startup Conditions are achieved. 539 The disadvantage of this definition would be that it 540 loses measurement of any impact that establishing 541 Startup Conditions would have on forwarding rate. When 542 comparing the Startup Aggregate Forwarding Rate benchmark 543 of two devices it is preferred to know the impact 544 establishing Startup Conditions has on Forwarding Rate. 545 The definition was therefore selected so that Stable 546 Aggregate Forwarding Rate is calculated from measurement 547 samples throughout the entire Startup Phase. 549 See Also: 550 Startup Phase 551 Aggregate Forwarding Rate 553 3.3.2.2 Stable Latency 555 Definition: 556 Average measured latency of traffic forwarded by the DUT 557 during the Startup Phase. 559 Discussion: 560 Stable Latency is the calculated average Latency during 561 the Startup Phase. 563 Measurement units: 564 seconds 566 Issues: 567 None 569 See Also: 570 Startup Phase 571 Stable Aggregate Forwarding Rate 573 3.3.2.3 Stable Session Count 575 Definition: 576 Total number of control plane sessions/adjacencies 577 established and maintained by the DUT during the Startup 578 Phase and prior to Instability Conditions being initiated. 580 Discussion: 581 This measurement SHOULD be made after the Control 582 Plane Startup Conditions are applied to the DUT. 584 Stress Benchmarking 586 Measurement units: 587 sessions 589 Issues: 590 None 592 See Also: 593 Startup Phase 595 3.3.3 Control Plane 597 3.3.3.1 Control Plane Configuration Set 598 Definition: 599 The routing protocols and scaling values used for the Accelerated 600 Stress Benchmarking. 602 Discussion: 603 Control Plane Configuration Set is shown in Figure 2 and specifies 604 the Routing Protocols, Multicast, and MPLS configuration. Specific 605 protocols can be enabled or disabled for a benchmark test. 607 Measurement units: 608 N/A 610 Issues: 611 None 613 See Also: 614 Data Plane Configuration Set 615 Management Configuration Set 616 Security Configuration Set 618 ____________ ____________ ____________ 619 | Routing | | Multicast | | MPLS | 620 | Protocols |___ | Protocols | __| Protocols | 621 | | | | | | | | 622 ------------ | ------------ | ------------ 623 | | | 624 | | | 625 | \/ | 626 | ___________ | 627 | | | | 628 |------->| DUT |<------| 629 ``| | 630 ----------- 631 Figure 2. Control Plane Configuration Module 632 Stress Benchmarking 634 3.3.3.2 Control Plane Startup Conditions 636 Definition: 637 Control Plane conditions that occur at the start 638 of the Accelerated Stress Benchmarking to establish conditions 639 for the remainder of the test. 641 Discussion: 642 Startup Conditions may cause stress on the DUT and produce 643 failure. Startup Conditions for the Control Plane include 644 session establishment rate, number of sessions established 645 and number of routes learned. 647 Measurement units: 648 N/A 650 Issues: 651 None 653 See Also: 654 Startup Conditions 655 Security Plane Startup Conditions 656 Control Plane Configuration Set 658 3.3.4 Data Plane 659 3.3.4.1 Data Plane Configuration Set 661 Definition: 662 The data traffic profile enabled for the Accelerated Stress 663 Benchmarking. 665 Discussion: 666 Data Plane Configuration Set includes the Traffic Profile and 667 interfaces used for the Accelerated Stress Benchmarking. 669 Measurement Units: 670 N/A 672 Issues: None 674 See Also: 675 Traffic Profile 677 3.3.4.2 Traffic Profile 678 Definition 679 The characteristics of the Offered Load to the DUT used for 680 the Accelerated Stress Benchmarking. 682 Discussion 683 The Traffic Profile specifies the number of packet size(s), 684 packet rate per interface, number of flows, and encapsulation 685 used for the offered load to the DUT. 687 Stress Benchmarking 689 Measurement Units: 690 Traffic Profile is reported as follows: 692 Parameter Units 693 --------- ------ 694 Packet Size(s) bytes 695 Packet Rate(interface) array of packets per second 696 Number of Flows number 697 Encapsulation(flow) array of encapsulation type 699 Issues: 700 None 702 See Also: 703 Data Plane Configuration Set 705 3.3.5 Management Plane 706 3.3.5.1 Management Plane Configuration Set 708 Definition: 709 The router management features enabled for the 710 Accelerated Stress Benchmark. 712 Discussion: 713 A key component of the Accelerated Stress Benchmark is the 714 Management Configuration Set to assess manageability of the 715 router under stress. The Management Configuration Set defines 716 the management configuration of the DUT. Features that are 717 part of the Management Configuration Set include access, SNMP, 718 Logging/Debug, and Statistics Collection, and services such as 719 FTP, as shown in Figure 3. These features SHOULD be enabled 720 throughout the Stress test. 722 Measurement units: 723 N/A 725 Issues: 726 None 728 See Also: 729 Control Plane Configuration Set 730 Data Plane Configuration Set 731 Security Plane Configuration Set 732 Stress Benchmarking 733 ____________ ____________ 734 | | | Logging/ | 735 | SNMP | __| Debug | 736 | | | | | 737 ------------ | ------------ 738 | | 739 | | 740 \/ | 741 ___________ | 742 | | | 743 | DUT |<---| 744 | | 745 ----------- 746 | 747 | 748 \/ 749 ___________ 750 | Packet | 751 | Statistics| 752 | Collector | 753 | | 754 ----------- 756 Figure 3. Management Plane Configuration Set 758 3.3.6 Security Plane 759 3.3.6.1 Security Plane Configuration Set 761 Definition: 762 Security features and scaling enabled for the Accelerated Stress 763 Test. 765 Discussion: 766 The Security Plane Configuration Set includes the configuration 767 and scaling of ACLs, Firewall, IPsec, and User Access, as shown 768 in Figure 4. Tunnels SHOULD be established and policies 769 configured. Instability is introduced by flapping tunnels and 770 configuring and removing policies. 772 ____________ ____________ ____________ 773 | | | Secure | | User | 774 |ACL/Firewall| | Protocol | __| Access | 775 | | | | | | | 776 ------------ ------------ | ------------ 777 | | | 778 | | | 779 | \/ | 780 | ___________ | 781 | | | | 782 |------->| DUT |<--------| 783 | | 784 ----------- 785 Figure 4. Security Configuration Module 786 Stress Benchmarking 788 Measurement units: 789 N/A 791 Issues: 792 None 794 See Also: 795 ACL Configuration Set 796 Secure Protocol Configuration Set 797 Password Login Configuration Set 799 3.3.6.2 Security Plane Startup Conditions 800 Definition: 801 Security Plane conditions that occur at the start 802 of the Accelerated Stress Benchmarking to establish conditions 803 for the remainder of the test. 805 Discussion: 806 Startup Conditions may cause stress on the DUT and produce 807 failure. Startup Conditions for the Security Plane include 808 session establishment rate, number of sessions established 809 and number of policies learned, and number of user access 810 sessions opened. 812 Measurement units: 813 N/A 815 Issues: 816 None 818 See Also: 819 Startup Conditions 820 Data Plane Startup Conditions 821 Management Plane Startup Conditions 822 Security Plane Startup Conditions 824 3.4 Instability 826 3.4.1 Instability Phase 828 Definition: 829 The portion of the benchmarking test in which the 830 Instability Conditions are offered to the DUT. 832 Discussion: 833 The Instability Phase is the middle Phase of 834 of the benchmarking test following the Startup 835 Phase and preceding the Recovery Phase. 837 Measurement Units: 838 None 839 Stress Benchmarking 841 Issues: 842 None 844 See Also: 845 Instability Conditions 846 Startup Phase 847 Recovery Phase 849 3.4.2 Benchmarks 850 3.4.2.1 Unstable Aggregate Forwarding Rate 852 Definition: 853 Rate of traffic forwarded by the DUT during the 854 Instability Phase. 856 Discussion: 857 Unstable Aggregated Forwarding Rate is an instantaneous 858 measurement of the Aggregate Forwarding Rate during the 859 Instability Phase. 861 Measurement units: 862 pps 864 Issues: 865 None 867 See Also: 868 Instability Conditions 869 Aggregate Forwarding Rate 871 3.4.2.2 Aggregate Forwarding Rate Degradation 873 Definition: 874 The reduction in Aggregate Forwarding Rate during the 875 Instability Phase. 877 Discussion: 878 The Aggregate Forwarding Rate Degradation is calculated 879 for each measurement of the Unstable Aggregate Forwarding 880 Rate. The Aggregate Forwarding Rate Degradation is 881 calculated by subtracting each measurement of the Unstable 882 Aggregate Forwarding Rate from the Stable Aggregate 883 Forwarding Rate, such that 885 Aggregate Forwarding Rate Degradation= 886 Stable Aggregate Forwarding Rate - 887 Unstable Aggregate Forwarding Rate 889 Ideally, the Aggregate Forwarding Rate Degradation is zero. 891 Measurement Units: 892 pps 893 Stress Benchmarking 895 Issues: 896 None 898 See Also: 899 Instability Phase 900 Unstable Aggregate Forwarding Rate 902 3.4.2.3 Average Aggregate Forwarding Rate Degradation 904 Definition 905 DUT Benchmark that is the calculated average of the 906 obtained Degraded Forwarding Rates. 908 Discussion: 909 Average Aggregate Forwarding Rate Degradation= 910 (Sum (Stable Aggregate Forwarding Rate) - 911 Sum (Unstable Aggregate Forwarding Rate)) / Number of Samples 913 Measurement Units: 914 pps 916 Issues: 917 None 919 See Also: 920 Aggregate Forwarding Rate Degradation 922 3.4.2.4 Unstable Latency 924 Definition: 925 The average increase in measured packet latency during 926 the Instability Phase compared to the Startup Phase. 928 Discussion: 929 Latency SHOULD be measured at a fixed interval during the 930 Instability Phase. Unstable Latency is the difference 931 between Stable Latency and the average Latency measured 932 during the Instability Phase. It is expected that there 933 be an increase in average latency from the Startup Phase 934 to the Instability phase, but it is possible that the 935 difference be zero. The Unstable Latency cannot be a 936 negative number. 938 Measurement units: 939 seconds 941 Issues: 942 None 944 See Also: 945 Instability Phase 946 Stable Latency 947 Stress Benchmarking 949 3.4.2.5 Unstable Uncontrolled Sessions Lost 951 Definition: 952 Control Plane sessions that are in the down state 953 but were not intentionally brought down during the 954 Instability Phase. 956 Discussion: 957 The test equipment is able to control protocol 958 session state with the DUT. The test equipment 959 is also to monitor for sessions lost with the 960 DUT which the test equipment itself did not 961 intentionally bring down. 963 Measurement units: 964 sessions 966 Issues: 967 None 969 See Also: 970 Controlled Session Loss 971 Uncontrolled Session Loss 973 3.4.3 Instability Conditions 975 3.4.3.1 Control Plane Instability Conditions 977 Definition: 978 Control Plane conditions that occur during the Accelerated Stress 979 Benchmark to produce instability and stress the DUT. 981 Discussion: 982 Control Plane Instability Conditions are experienced by the DUT 983 after the Startup Conditions have completed. Control Plane 984 Instability Conditions experienced by the DUT include session 985 loss, route withdrawal, and route cost changes. 987 Measurement units: 988 N/A 990 Issues: 991 None 993 See Also: 994 Instability Conditions 995 Data Plane Instability Conditions 996 Management Plane Instability Conditions 997 Security Plane Instability Conditions 998 Stress Benchmarking 1000 3.4.3.2 Data Plane Instability Conditions 1001 Definition: 1002 Data Plane conditions that occur during the Accelerated Stress 1003 Benchmark to produce instability and stress the DUT. 1005 Discussion: 1006 Data Plane Instability Conditions are experienced by the DUT 1007 after the Startup Conditions have completed. Data Plane 1008 Instability Conditions experienced by the DUT include interface 1009 shutdown, link loss, and overloaded links. 1011 Measurement units: 1012 N/A 1014 Issues: 1015 None 1017 See Also: 1018 Instability Conditions 1019 Control Plane Instability Conditions 1020 Management Plane Instability Conditions 1021 Security Plane Instability Conditions 1023 3.4.3.3 Management Plane Instability Conditions 1024 Definition: 1025 Management Plane conditions that occur during the Accelerated 1026 Stress Benchmark to produce instability and stress the DUT. 1028 Discussion: 1029 Management Plane Instability Conditions are experienced by the DUT 1030 after the Startup Conditions have completed. Management Plane 1031 Instability Conditions experienced by the DUT include repeated 1032 FTP of large files. 1034 Measurement units: 1035 N/A 1037 Issues: 1038 None 1040 See Also: 1041 Instability Conditions 1042 Control Plane Instability Conditions 1043 Data Plane Instability Conditions 1044 Security Plane Instability Conditions 1046 3.4.3.4 Security Plane Instability Conditions 1048 Definition: 1049 Security Plane conditions that occur during the Accelerated 1050 Stress Benchmark to produce instability and stress the DUT. 1052 Stress Benchmarking 1054 Discussion: 1055 Security Plane Instability Conditions are experienced by the DUT 1056 after the Startup Conditions have completed. Security Plane 1057 Instability Conditions experienced by the DUT include session 1058 loss and uninitiated policy changes. 1060 Measurement units: 1061 N/A 1063 Issues: 1064 None 1066 See Also: 1067 Instability Conditions 1068 Control Plane Instability Conditions 1069 Data Plane Instability Conditions 1070 Management Plane Instability Conditions 1072 3.5 Recovery 1074 3.5.1 Recovery Phase 1076 Definition: 1077 The portion of the benchmarking test in which the 1078 Startup Conditions are generated with the DUT, but 1079 the Instability Conditions are no longer offered to 1080 the DUT. 1082 Discussion: 1083 The Recovery Phase is the final Phase of the 1084 benchmarking test following the Startup Phase and 1085 Instability Phase. Startup Conditions MUST NOT be 1086 Restarted. 1088 Measurement Units: 1089 None 1091 Issues: 1092 None 1094 See Also: 1095 Startup Conditions 1096 Startup Phase 1097 Instability Conditions 1098 Instability Phase 1100 3.5.2 Benchmarks 1101 3.5.2.1 Recovered Aggregate Forwarding Rate 1103 Definition 1104 Rate of traffic forwarded by the DUT during the Recovery 1105 Phase. 1107 Stress Benchmarking 1109 Discussion: 1110 Recovered Aggregate Forwarding Rate is an instantaneous 1111 measurement of the Aggregate Forwarding Rate during the 1112 Recovery Phase. Ideally, each measurement of the Recovered 1113 Aggregate Forwarding Rate equals the Stable Aggregate 1114 Forwarding Rate because the Instability Conditions 1115 do not exist in both the Startup and Recovery Phases. 1117 Measurement Units: 1118 pps 1120 Issues: 1121 None 1123 See Also: 1124 Aggregate Forwarding Rate 1125 Recovery Phase 1126 Recovered Aggregate Forwarding Rate 1127 Startup Phase 1128 Stable Aggregate Forwarding Rate 1130 3.5.2.2 Recovered Latency 1132 Definition: 1133 The average increase in measured packet latency during 1134 the Recovery Phase compared to the Startup Phase. 1136 Discussion: 1137 Latency SHOULD be measured at a fixed interval during the 1138 Recovery Phase. Unstable Latency is the difference 1139 between Stable Latency and the average Latency measured 1140 during the Recovery Phase. It is expected that there 1141 be no increase in average latency from the Startup Phase 1142 to the Recovery Phase. The Recovered Latency cannot be a 1143 negative number. 1145 Measurement units: 1146 seconds 1148 Issues: None 1150 See Also: 1151 Recovery Phase 1152 Stable Latency 1154 3.5.2.3 Recovery Time 1156 Definition 1157 The amount of time for the Recovered Aggregate Forwarding 1158 Rate to become equal to the Stable Aggregate Forwarding Rate. 1160 Stress Benchmarking 1162 Discussion 1163 Recovery Time is measured beginning at the instant the 1164 Instability Phase ends until the Recovered Aggregate 1165 Forwarding Rate equals the Stable Aggregate Forwarding 1166 Rate for a minimum duration of 180 consecutive seconds. 1168 Measurement Units: 1169 milliseconds 1171 Issues: 1172 None 1174 See Also: 1175 Recovered Aggregate Forwarding Rate 1176 Stable Aggregate Forwarding Rate 1178 3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost 1180 Definition: 1181 Control Plane sessions that are in the down state 1182 but were not intentionally brought down during the 1183 Recovery Phase. 1185 Discussion: 1186 The test equipment is able to control protocol 1187 session state with the DUT. The test equipment 1188 is also to monitor for sessions lost with the 1189 DUT which the test equipment itself did not 1190 intentionally bring down. 1192 Measurement units: 1193 sessions 1195 Issues: 1196 None 1198 See Also: 1199 Controlled Session Loss 1200 Uncontrolled Session Loss 1202 3.5.2.5 Variability Benchmarks 1204 Definition: 1205 The difference between the measured Benchmarks of the 1206 same DUT over multiple iterations. 1208 Discussion: 1209 Ideally, the measured benchmarks should be the same for multiple 1210 iterations with the same DUT. Configuration Sets and 1211 Instability Conditions MUST be held constant for this 1212 benchmark. Whether the DUT can exhibit such predictable and 1213 repeatable behavior is an important benchmark in itself. 1215 Stress Benchmarking 1217 Measurement units: 1218 As applicable to each Benchmark. The results are to be 1219 presented in a table format for successive Iterations. 1220 Ideally, the differences should be zero. 1222 Issues: 1223 None 1225 See Also: 1226 Startup Period 1227 Instability Period 1228 Recovery Period 1230 4. IANA Considerations 1231 This document requires no IANA considerations. 1233 5. Security Considerations 1234 Documents of this type do not directly affect the security of 1235 the Internet or of corporate networks as long as benchmarking 1236 is not performed on devices or systems connected to operating 1237 networks. 1239 6. References 1240 6.1 Normative References 1241 [1] Bradner, S., Editor, "Benchmarking Terminology for Network 1242 Interconnection Devices", RFC 1242, March 1991. 1244 [2] Mandeville, R., "Benchmarking Terminology for LAN Switching 1245 Devices", RFC 2285, June 1998. 1247 [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for 1248 Network Interconnect Devices", RFC 2544, March 1999. 1250 [4] Poretsky, S. and Rao, S., "Methodology Guidelines for 1251 Accelerated Stress Benchmarking", 1252 draft-ietf-bmwg-acc-bench-meth-07, work in progress, 1253 March 2007. 1255 [5] Bradner, S., "Key words for use in RFCs to Indicate 1256 Requirement Levels", RFC 2119, March 1997. 1258 6.2 Informative References 1259 [RFC3871] Jones, G., "Operational Security Requirements for Large 1260 Internet Service Provider (ISP) IP Network Infrastructure.", 1261 IETF RFC 3871 , September 2004. 1263 [NANOG25] Poretsky, S., "Core Router Evaluation for Higher 1264 Availability", NANOG 25, June 8, 2002, Toronto, CA. 1266 [IEEECQR] Poretsky, S., "Router Stress Testing to Validate 1267 Readiness for Network Deployment", IEEE CQR 2003. 1269 Stress Benchmarking 1271 7. Author's Address 1273 Scott Poretsky 1274 Reef Point Systems 1275 8 New England Executive Park 1276 Burlington, MA 01803 1277 USA 1278 Phone: + 1 781 395 5090 1279 EMail: sporetsky@reefpoint.com 1281 Shankar Rao 1282 1801 California Street 1283 8th Floor 1284 Qwest Communications 1285 Denver, CO 80202 1286 USA 1287 Phone: + 1 303 437 6643 1288 Email: shankar.rao@qwest.com 1290 Appendix 1. White Box Benchmarking Terminology 1291 Minimum Available Memory 1292 Definition: 1293 Minimum DUT Available Memory during the duration of the 1294 Accelerated Stress Benchmark. 1296 Discussion: 1297 This benchmark enables the assessment of resources in the DUT. 1298 It is necessary to monitor DUT memory to measure this benchmark. 1300 Measurement units: 1301 bytes 1303 Issues: None 1305 See Also: 1306 Maximum CPU Utilization 1308 Maximum CPU Utilization 1309 Definition: 1310 Maximum DUT CPU utilization during the duration of the 1311 Accelerated Stress Benchmark. 1313 Discussion: 1314 This benchmark enables the assessment of resources in the DUT. 1315 It is necessary to monitor DUT CPU Utilization to measure 1316 this benchmark. 1318 Measurement units: % 1320 Issues: None 1322 See Also: 1323 Minimum Available Memory 1324 Stress Benchmarking 1326 Full Copyright Statement 1328 Copyright (C) The IETF Trust (2007). 1330 This document is subject to the rights, licenses and restrictions 1331 contained in BCP 78, and except as set forth therein, the authors 1332 retain all their rights. 1334 This document and the information contained herein are provided 1335 on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 1336 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE 1337 IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL 1338 WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY 1339 WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE 1340 ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS 1341 FOR A PARTICULAR PURPOSE. 1343 Intellectual Property 1345 The IETF takes no position regarding the validity or scope of any 1346 Intellectual Property Rights or other rights that might be claimed to 1347 pertain to the implementation or use of the technology described in 1348 this document or the extent to which any license under such rights 1349 might or might not be available; nor does it represent that it has 1350 made any independent effort to identify any such rights. Information 1351 on the procedures with respect to rights in RFC documents can be 1352 found in BCP 78 and BCP 79. 1354 Copies of IPR disclosures made to the IETF Secretariat and any 1355 assurances of licenses to be made available, or the result of an 1356 attempt made to obtain a general license or permission for the use of 1357 such proprietary rights by implementers or users of this 1358 specification can be obtained from the IETF on-line IPR repository at 1359 http://www.ietf.org/ipr. 1361 The IETF invites any interested party to bring to its attention any 1362 copyrights, patents or patent applications, or other proprietary 1363 rights that may cover technology that may be required to implement 1364 this standard. Please address the information to the IETF at ietf- 1365 ipr@ietf.org. 1367 Acknowledgement 1369 Funding for the RFC Editor function is currently provided by the 1370 Internet Society.