idnits 2.17.1 draft-ietf-bmwg-acc-bench-term-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 19. -- Found old boilerplate from RFC 3978, Section 5.5 on line 1346. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1357. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1364. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1370. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 445 instances of lines with control characters in the document. == There are 24 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (December 2006) is 6342 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: '1' is defined on line 1249, but no explicit reference was found in the text == Unused Reference: '2' is defined on line 1252, but no explicit reference was found in the text == Unused Reference: '3' is defined on line 1255, but no explicit reference was found in the text == Unused Reference: 'RFC3871' is defined on line 1266, but no explicit reference was found in the text == Unused Reference: 'NANOG25' is defined on line 1270, but no explicit reference was found in the text == Unused Reference: 'IEEECQR' is defined on line 1273, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 1242 (ref. '1') ** Downref: Normative reference to an Informational RFC: RFC 2285 (ref. '2') ** Downref: Normative reference to an Informational RFC: RFC 2544 (ref. '3') == Outdated reference: A later version (-09) exists of draft-ietf-bmwg-acc-bench-meth-05 ** Downref: Normative reference to an Informational draft: draft-ietf-bmwg-acc-bench-meth (ref. '4') Summary: 8 errors (**), 0 flaws (~~), 10 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group 2 INTERNET-DRAFT 3 Expires in: December 2006 4 Scott Poretsky 5 Reef Point Systems 7 Shankar Rao 8 Qwest Communications 10 June 2006 12 Terminology for Accelerated Stress Benchmarking 13 15 Intellectual Property Rights (IPR) statement: 16 By submitting this Internet-Draft, each author represents that any 17 applicable patent or other IPR claims of which he or she is aware 18 have been or will be disclosed, and any of which he or she becomes 19 aware will be disclosed, in accordance with Section 6 of BCP 79. 21 Status of this Memo 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF), its areas, and its working groups. Note that 25 other groups may also distribute working documents as 26 Internet-Drafts. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 The list of current Internet-Drafts can be accessed at 34 http://www.ietf.org/ietf/1id-abstracts.txt. 36 The list of Internet-Draft Shadow Directories can be accessed at 37 http://www.ietf.org/shadow.html. 39 Copyright Notice 40 Copyright (C) The Internet Society (2006). 42 ABSTRACT 43 This document provides the Terminology for performing Accelerated 44 Stress Benchmarking of networking devices. The three phases of 45 the Stress Test: Startup, Instability and Recovery are defined 46 along with the benchmarks and configuration terms associated with 47 the each phase. Also defined are the Benchmark Planes fundamental 48 to stress testing configuration, setup and measurement. The 49 terminology is to be used with the companion framework and 50 methodology documents. 52 Table of Contents 53 1. Introduction ............................................... 3 54 2. Existing definitions ....................................... 3 55 3. Term definitions............................................ 4 56 3.1 General Terms............................................. 4 57 Stress Benchmarking 59 3.1.1 Benchmark Planes...................................... 4 60 3.1.2 Configuration Sets.................................... 5 61 3.1.3 Startup Conditions.................................... 5 62 3.1.4 Instability Conditions................................ 6 63 3.1.5 Aggregate Forwarding Rate............................. 6 64 3.1.6 Controlled Session Loss............................... 7 65 3.1.7 Uncontrolled Session Loss............................. 7 66 3.2 Benchmark Planes.......................................... 8 67 3.2.1 Control Plane......................................... 8 68 3.2.2 Data Plane............................................ 8 69 3.2.3 Management Plane...................................... 8 70 3.2.4 Security Plane........................................ 9 71 3.3 Startup...................................................10 72 3.3.1 Startup Phase.........................................10 73 3.3.2 Benchmarks............................................10 74 3.3.2.1 Stable Aggregate Forwarding Rate..................10 75 3.3.2.2 Stable Latency....................................11 76 3.3.2.3 Stable Session Count..............................11 77 3.3.3 Control Plane.........................................12 78 3.3.3.1 Control Plane Configuration Set...................12 79 3.3.3.2 Control Plane Startup Conditions..................13 80 3.3.4 Data Plane............................................13 81 3.3.4.1 Data Plane Configuration Set......................13 82 3.3.4.2 Traffic Profile...................................13 83 3.3.5 Management Plane......................................14 84 3.3.5.1 Management Plane Configuration Set................14 85 3.3.6 Security Plane........................................15 86 3.3.6.1 Security Plane Configuration Set..................15 87 3.3.6.2 Security Plane Startup Conditions.................16 88 3.4 Instability...............................................16 89 3.4.1 Instability Phase.....................................16 90 3.4.2 Benchmarks............................................17 91 3.4.2.1 Unstable Aggregate Forwarding Rate................17 92 3.4.2.2 Aggregate Forwarding Rate Degradation.............17 93 3.4.2.3 Average Aggregate Forwarding Rate Degradation.....18 94 3.4.2.4 Unstable Latency..................................18 95 3.4.2.5 Unstable Uncontrolled Sessions Lost...............19 96 3.4.3 Instability Conditions................................19 97 3.4.3.1 Control Plane Instability Conditions..............19 98 3.4.3.2 Data Plane Instability Conditions.................20 99 3.4.3.3 Management Plane Instability Conditions...........20 100 3.4.3.4 Security Plane Instability Conditions.............20 101 3.5 Recovery..................................................21 102 3.5.1 Recovery Phase........................................21 103 3.5.2 Benchmarks............................................21 104 3.5.2.1 Recovered Aggregate Forwarding Rate...............21 105 3.5.2.2 Recovered Latency.................................22 106 3.5.2.3 Recovery Time.....................................22 107 3.5.2.4 Recovered Uncontrolled Sessions Lost..............23 108 3.5.2.5 Variability Benchmarks............................23 109 4. IANA Considerations.........................................24 110 Stress Benchmarking 112 5. Security Considerations.....................................24 113 6. References..................................................24 114 7. Author's Address............................................25 115 Appendix 1 - White Box Benchmarks..............................25 117 1. Introduction 119 Routers in an operational network are simultaneously configured with 120 multiple protocols and security policies while forwarding traffic and 121 being managed. To accurately benchmark a router for deployment it is 122 necessary to test that router in operational conditions by 123 simultaneously configuring and scaling network protocols and security 124 policies, forwarding traffic, and managing the device. It is helpful 125 to accelerate these network operational conditions so that the 126 router under test can be benchmarked with faster test duration. 127 Testing a router in accelerated network conditions is known as 128 Accelerated Stress Benchmarking. 130 This document provides the Terminology for performing Stress 131 Benchmarking of networking devices. The three phases of the Stress 132 Test: Startup, Instability and Recovery are defined along with the 133 benchmark and configuration terms associated with the each phase. 134 Benchmarks for stress testing are defined using the Aggregate 135 Forwarding Rate and control plane Session Count during each phase 136 of the test. For each plane, the Configuration Set, Startup 137 Conditions, and Instability Conditions are defined. Also defined are 138 the Benchmark Planes fundamental to stress testing configuration, 139 setup and measurement. These are the Control Plane, Data Plane, 140 Management Plane and Security Plane Multiple benchmarks are made 141 for each Benchmark Plane during each Phase. Benchmarks can be 142 compared across multiple planes for the same DUT or at the same 143 plane for 2 or more DUTS. These benchmarks White Box benchmarks 144 are provided in Appendix 1 for additional DUT behavior 145 measurements. The terminology is to be used with the companion 146 methodology document [4]. The sequence of phases, actions, and 147 benchmarks are shown in Table 1. 149 2. Existing definitions 150 RFC 1242 "Benchmarking Terminology for Network Interconnect 151 Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching 152 Devices" should be consulted before attempting to make use of this 153 document. For the sake of clarity and continuity this RFC adopts 154 the template for definitions set out in Section 2 of RFC 1242. 155 Definitions are indexed and grouped together in sections for ease 156 of reference. 158 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 159 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 160 document are to be interpreted as described in BCP 14, RFC 2119 161 [5]. RFC 2119 defines the use of these key words to help make the 162 intent of standards track documents as clear as possible. While this 163 document uses these keywords, it is not a standards track document. 165 Stress Benchmarking 167 Table 1. Phase Sequence and Benchmarks 168 III. Recovery Phase II. Instability Phase I. Startup Phase 169 <-----------------<---<-------------------<----<--------------< 170 Remove Instability Achieve Configuration Apply Startup 171 Conditions Set Conditions 173 Benchmark: Benchmark: Benchmark: 174 Recovered Aggregate Unstable Aggregate Stable Aggregate 175 Forwarding Rate Forwarding Rate Forwarding Rate 177 Degraded Aggregate 178 Forwarding Rate 180 Average Degraded 181 Forwarding Rate 183 Recovered Latency Unstable Latency Startup Latency 185 Recovered Uncontrolled Recovered Uncontrolled Stable Session Count 186 Sessions Lost Sessions Lost 188 Recovery Time 190 3. Term definitions 191 3.1 General Terms 192 3.1.1 Benchmark Planes 194 Definition: 195 The features, conditions, and behavior for the Accelerated Stress 196 Benchmarking. 198 Discussion: 199 There are four Benchmark Planes: Control Plane, Data Plane, 200 Management Plane, and Security Plane as shown in Figure 1. The 201 Benchmark Planes define the Configuration, Startup Conditions, 202 Instability Conditions, and Failure Conditions used for the test. 204 Measurement units: 205 N/A 207 Issues: 208 None 210 See Also: 211 Control Plane 212 Data Plane 213 Management Plane 214 Security Plane 215 Stress Benchmarking 217 ___________ ___________ 218 | Control | | Management| 219 | Plane |___ ___| Plane | 220 | | | | | | 221 ----------- | | ----------- 222 \/ \/ ___________ 223 ___________ | Security | 224 | |<-----------| Plane | 225 | DUT | | | 226 |--->| |<---| ----------- 227 | ----------- | 228 | | 229 | ___________ | 230 | | Data | | 231 |--->| Plane |<---| 232 | | 233 ----------- 235 Figure 1. Router Accelerated Stress Benchmarking Planes 237 3.1.2 Configuration Sets 239 Definition: 240 The features and scaling limits used during the Accelerated Stress 241 Benchmarking. 243 Discussion: 244 There are four Configuration Sets: Control Plane Configuration Set, 245 Data Plane Configuration Set, Management Plane Configuration Set, 246 and Security Plane Configuration Set. 248 Measurement units: 249 N/A 251 Issues: 252 None 254 See Also: 255 Control Plane Configuration Set 256 Data Plane Configuration Set 257 Management Plane Configuration Set 258 Security Plane Configuration Set 260 3.1.3 Startup Conditions 262 Definition: 263 Test conditions that occur at the start of the Accelerated 264 Stress Benchmark to establish conditions for the remainder of 265 the test. 267 Stress Benchmarking 269 Discussion: 270 Startup Conditions may cause stress on the DUT and produce 271 failure. Startup Conditions are defined for the Control 272 Plane and Security Plane. 274 Measurement units: 275 N/A 277 Issues: 278 None 280 See Also: 281 Control Plane Startup Conditions 282 Data Plane Startup Conditions 283 Management Plane Startup Conditions 284 Security Plane Startup Conditions 286 3.1.4 Instability Conditions 288 Definition: 289 Test conditions that occur during the Accelerated Stress 290 Benchmark to produce instability and stress the DUT. 292 Discussion: 293 Instability Conditions are applied to the DUT after the 294 Startup Conditions have completed. Instability Conditions 295 occur for the Control Plane, Data Plane, Management Plane, 296 and Security Plane. 298 Measurement units: 299 N/A 301 Issues: None 303 See Also: 304 Control Plane Instability Conditions 305 Data Plane Instability Conditions 306 Management Plane Instability Conditions 307 Security Plane Instability Conditions 309 3.1.5 Aggregate Forwarding Rate 311 Definition: 312 Sum of forwarding rates for all interfaces on the 313 DUT during the Startup Phase. 315 Discussion: 316 Each interface of the DUT forwards traffic at some 317 measured rate. The Aggregate Forwarding Rate is the 318 sum of forwarding rates for all interfaces on the DUT. 320 Stress Benchmarking 322 Measurement units: 323 pps 325 Issues: 326 None 328 See Also: 329 Startup Phase 331 3.1.6 Controlled Session Loss 333 Definition: 334 Control Plane sessions that are intentionally brought 335 down during the Stress test. 337 Discussion: 338 The test equipment is able to control protocol 339 session state with the DUT. 341 Measurement units: 342 None 344 Issues: 345 None 347 See Also: 348 Uncontrolled Session Loss 350 3.1.7 Uncontrolled Session Loss 352 Definition: 353 Control Plane sessions that are in the down state 354 but were not intentionally brought down during the 355 Stress test. 357 Discussion: 358 The test equipment is able to control protocol 359 session state with the DUT. The test equipment 360 is also to monitor for sessions lost with the 361 DUT which the test equipment itself did not 362 intentionally bring down. 364 Measurement units: 365 N/A 367 Issues: 368 None 370 See Also: 371 Controlled Session Loss 372 Stress Benchmarking 374 3.2 Benchmark Planes 376 3.2.1 Control Plane 377 Definition: 378 The Description of the control protocols enabled for 379 the Accelerated Stress Benchmarking. 381 Discussion: 382 The Control Plane defines the Configuration, Startup 383 Conditions, and Instability Conditions of the control 384 protocols. Control Plane protocols may include routing 385 protocols, multicast protocols, and MPLS protocols. 386 These can be enabled or disabled for a benchmark test. 388 Measurement units: 389 N/A 391 Issues: 392 None 394 See Also: 395 Benchmark Planes 396 Control Plane Configuration Set 397 Control Plane Startup Conditions 398 Control Plane Instability Conditions 400 3.2.2 Data Plane 401 Definition: 402 The data traffic profile used for the Accelerated Stress 403 Benchmarking. 405 Discussion: 406 The Data Plane defines the Configuration, Startup 407 Conditions, and Instability Conditions of the data 408 traffic. The Data Plane includes the traffic and 409 interface profile. 411 Measurement Units: 412 N/A 414 See Also: 415 Benchmark Planes 416 Data Plane Configuration Set 417 Data Plane Startup Conditions 418 Data Plane Instability Conditions 420 3.2.3 Management Plane 422 Definition: 423 The Management features and tools used for the 424 Accelerated Stress Benchmarking. 426 Stress Benchmarking 428 Discussion: 429 A key component of the Accelerated Stress Benchmarking is the 430 Management Plane to assess manageability of the router 431 under stress. The Management Plane defines the Configuration, 432 Startup Conditions, and Instability Conditions of the 433 management protocols and features. The Management Plane 434 includes SNMP, Logging/Debug, Statistics Collection, and 435 management configuration sessions such as telnet, SSH, and 436 serial console. SNMP Gets SHOULD be performed continuously. 437 Management configuration sessions should be open 438 simultaneously and be repeatedly open and closed. Open 439 management sessions should have valid and invalid 440 configuration and show commands entered. 442 Measurement units: 443 N/A 445 Issues: 446 None 448 See Also: 449 Benchmark Planes 450 Management Plane Configuration Set 451 Management Plane Startup Conditions 452 Management Plane Instability Conditions 454 3.2.4 Security Plane 456 Definition: 457 The Security features used during the Accelerated Stress 458 Benchmarking. 460 Discussion: 461 The Security Plane defines the Configuration, Startup 462 Conditions, and Instability Conditions of the security 463 features and protocols. The Security Plane includes the 464 ACLs, Firewall, Secure Protocols, and User Login. Tunnels 465 for those such as IPsec should be established and flapped. 466 Policies for Firewalls and ACLs should be repeatedly added 467 and removed from the configuration via telnet, SSH, or 468 serial management sessions. 470 Measurement units: 471 N/A 473 Issues: None 475 See Also: 476 Benchmark Planes 477 Security Plane Configuration Set 478 Security Plane Startup Conditions 479 Security Plane Instability Conditions 480 Stress Benchmarking 482 3.3 Startup 484 3.3.1 Startup Phase 486 Definition 487 The portion of the benchmarking test in which the 488 Startup Conditions are generated with the DUT. This 489 begins with the attempt to establish the first session 490 and ends when the last Control Plane session is 491 established. 493 Discussion: 494 The Startup Phase is the first Phase of the benchmarking 495 test preceding the Instability Phase and Recovery Phase. 496 It is specified by the Configuration Sets and Startup 497 Conditions for each Benchmark Plane. The Startup Phase ends 498 and Instability Phase may begin when the Configuration Sets 499 are achieved with the DUT. 501 Measurement Units: 502 None 504 Issues: 505 The 'last control plane session is established' may not 506 be a sufficient indicator that steady-state is achieved 507 and Instability Conditions can be applied to begin the 508 Instability Phase. 510 See Also: 511 Benchmark Plane 512 Configuration Sets 513 Startup Conditions 514 Instability Phase 515 Recovery Phase 517 3.3.2 Benchmarks 518 3.3.2.1 Stable Aggregate Forwarding Rate 520 Definition: 521 Average rate of traffic forwarded by the DUT during the 522 Startup Phase. 524 Discussion: 525 Stable Aggregate Forwarding Rate is the calculated 526 average of the Aggregate Forwarding Rates measured during 527 the Startup Phase. 529 Measurement units: 530 pps 531 Stress Benchmarking 533 Issues: 534 The act of the DUT establishing the Startup Conditions 535 could influence the forwarding rate in certain 536 implementations so that this "baseline" for the 537 remainder of the test is lowered. The alternative is 538 to change the definition of Stable Aggregate 539 Forwarding Rate so that it measured during the Startup 540 Phase, but after Startup Conditions are achieved. 541 The disadvantage of this definition would be that it 542 loses measurement of any impact that establishing 543 Startup Conditions would have on forwarding rate. When 544 comparing the Startup Aggregate Forwarding Rate benchmark 545 of two devices it is preferred to know the impact 546 establishing Startup Conditions has on Forwarding Rate. 547 The definition was therefore selected so that Stable 548 Aggregate Forwarding Rate is calculated from measurement 549 samples throughout the entire Startup Phase. 551 See Also: 552 Startup Phase 553 Aggregate Forwarding Rate 555 3.3.2.2 Stable Latency 557 Definition: 558 Average measured latency of traffic forwarded by the DUT 559 during the Startup Phase. 561 Discussion: 562 Stable Latency is the calculated average Latency during 563 the Startup Phase. 565 Measurement units: 566 seconds 568 Issues: 569 None 571 See Also: 572 Startup Phase 573 Stable Aggregate Forwarding Rate 575 3.3.2.3 Stable Session Count 577 Definition: 578 Total number of control plane sessions/adjacencies 579 established and maintained by the DUT during the Startup 580 Phase and prior to Instability Conditions being initiated. 582 Discussion: 583 This measurement should be made after the Control 584 Plane Startup Conditions are applied to the DUT. 586 Stress Benchmarking 588 Measurement units: 589 sessions 591 Issues: 592 None 594 See Also: 595 Startup Phase 597 3.3.3 Control Plane 599 3.3.3.1 Control Plane Configuration Set 600 Definition: 601 The routing protocols and scaling values used for the Accelerated 602 Stress Benchmarking. 604 Discussion: 605 Control Plane Configuration Set is shown in Figure 2 and specifies 606 the Routing Protocols, Multicast, and MPLS configuration. Specific 607 protocols can be enabled or disabled for a benchmark test. 609 Measurement units: 610 N/A 612 Issues: 613 None 615 See Also: 616 Data Plane Configuration Set 617 Management Configuration Set 618 Security Configuration Set 620 ____________ ____________ ____________ 621 | Routing | | Multicast | | MPLS | 622 | Protocols |___ | Protocols | __| Protocols | 623 | | | | | | | | 624 ------------ | ------------ | ------------ 625 | | | 626 | | | 627 | \/ | 628 | ___________ | 629 | | | | 630 |------->| DUT |<------| 631 ``| | 632 ----------- 633 Figure 2. Control Plane Configuration Module 634 Stress Benchmarking 636 3.3.3.2 Control Plane Startup Conditions 638 Definition: 639 Control Plane conditions that occur at the start 640 of the Accelerated Stress Benchmarking to establish conditions 641 for the remainder of the test. 643 Discussion: 644 Startup Conditions may cause stress on the DUT and produce 645 failure. Startup Conditions for the Control Plane include 646 session establishment rate, number of sessions established 647 and number of routes learned. 649 Measurement units: 650 N/A 652 Issues: 653 None 655 See Also: 656 Startup Conditions 657 Security Plane Startup Conditions 658 Control Plane Configuration Set 660 3.3.4 Data Plane 661 3.3.4.1 Data Plane Configuration Set 663 Definition: 664 The data traffic profile enabled for the Accelerated Stress 665 Benchmarking. 667 Discussion: 668 Data Plane Configuration Set includes the Traffic Profile and 669 interfaces used for the Accelerated Stress Benchmarking. 671 Measurement Units: 672 N/A 674 Issues: None 676 See Also: 677 Traffic Profile 679 3.3.4.2 Traffic Profile 680 Definition 681 The characteristics of the Offered Load to the DUT used for 682 the Accelerated Stress Benchmarking. 684 Discussion 685 The Traffic Profile specifies the number of packet size(s), 686 packet rate per interface, number of flows, and encapsulation 687 used for the offered load to the DUT. 689 Stress Benchmarking 691 Measurement Units: 692 Traffic Profile is reported as follows: 694 Parameter Units 695 --------- ------ 696 Packet Size(s) bytes 697 Packet Rate(interface) array of packets per second 698 Number of Flows number 699 Encapsulation(flow) array of encapsulation type 701 Issues: 702 None 704 See Also: 705 Data Plane Configuration Set 707 3.3.5 Management Plane 708 3.3.5.1 Management Plane Configuration Set 710 Definition: 711 The router management features enabled for the 712 Accelerated Stress Benchmark. 714 Discussion: 715 A key component of the Accelerated Stress Benchmark is the 716 Management Configuration Set to assess manageability of the 717 router under stress. The Management Configuration Set defines 718 the management configuration of the DUT. Features that are 719 part of the Management Configuration Set include access, SNMP, 720 Logging/Debug, and Statistics Collection, and services such as 721 FTP, as shown in Figure 3. These features should be enabled 722 throughout the Stress test. SNMP Gets should be made 723 continuously with multiple FTP and Telnet sessions operating 724 simultaneously. FTP sessions should be opened and closed at 725 varying intervals and get and put files while open. Telnet 726 sessions should be opened and closed at varying intervals and 727 enter valid and invalid show and configuration commands while 728 open. 730 Measurement units: 731 N/A 733 Issues: 734 None 736 See Also: 737 Control Plane Configuration Set 738 Data Plane Configuration Set 739 Security Plane Configuration Set 740 Stress Benchmarking 741 ____________ ____________ 742 | | | Logging/ | 743 | SNMP | __| Debug | 744 | | | | | 745 ------------ | ------------ 746 | | 747 | | 748 \/ | 749 ___________ | 750 | | | 751 | DUT |<---| 752 | | 753 ----------- 754 | 755 | 756 \/ 757 ___________ 758 | Packet | 759 | Statistics| 760 | Collector | 761 | | 762 ----------- 764 Figure 3. Management Plane Configuration Set 766 3.3.6 Security Plane 767 3.3.6.1 Security Plane Configuration Set 769 Definition: 770 Security features and scaling enabled for the Accelerated Stress 771 Test. 773 Discussion: 774 The Security Plane Configuration Set includes the configuration 775 and scaling of ACLs, Firewall, IPsec, and User Access, as shown 776 in Figure 4. Tunnels should be established and policies 777 configured. Instability is introduced by flapping tunnels and 778 configuring and removing policies. 780 ____________ ____________ ____________ 781 | | | Secure | | User | 782 |ACL/Firewall| | Protocol | __| Access | 783 | | | | | | | 784 ------------ ------------ | ------------ 785 | | | 786 | | | 787 | \/ | 788 | ___________ | 789 | | | | 790 |------->| DUT |<--------| 791 | | 792 ----------- 793 Figure 4. Security Configuration Module 794 Stress Benchmarking 796 Measurement units: 797 N/A 799 Issues: 800 None 802 See Also: 803 ACL Configuration Set 804 Secure Protocol Configuration Set 805 Password Login Configuration Set 807 3.3.6.2 Security Plane Startup Conditions 808 Definition: 809 Security Plane conditions that occur at the start 810 of the Accelerated Stress Benchmarking to establish conditions 811 for the remainder of the test. 813 Discussion: 814 Startup Conditions may cause stress on the DUT and produce 815 failure. Startup Conditions for the Security Plane include 816 session establishment rate, number of sessions established 817 and number of policies learned, and number of user access 818 sessions opened. 820 Measurement units: 821 N/A 823 Issues: 824 None 826 See Also: 827 Startup Conditions 828 Data Plane Startup Conditions 829 Management Plane Startup Conditions 830 Security Plane Startup Conditions 832 3.4 Instability 834 3.4.1 Instability Phase 836 Definition: 837 The portion of the benchmarking test in which the 838 Instability Conditions are offered to the DUT. 840 Discussion: 841 The Instability Phase is the middle Phase of 842 of the benchmarking test following the Startup 843 Phase and preceding the Recovery Phase. 845 Measurement Units: 846 None 847 Stress Benchmarking 849 Issues: 850 None 852 See Also: 853 Instability Conditions 854 Startup Phase 855 Recovery Phase 857 3.4.2 Benchmarks 858 3.4.2.1 Unstable Aggregate Forwarding Rate 860 Definition: 861 Rate of traffic forwarded by the DUT during the 862 Instability Phase. 864 Discussion: 865 Unstable Aggregated Forwarding Rate is an instantaneous 866 measurement of the Aggregate Forwarding Rate during the 867 Instability Phase. 869 Measurement units: 870 pps 872 Issues: 873 None 875 See Also: 876 Instability Conditions 877 Aggregate Forwarding Rate 879 3.4.2.2 Aggregate Forwarding Rate Degradation 881 Definition: 882 The reduction in Aggregate Forwarding Rate during the 883 Instability Phase. 885 Discussion: 886 The Aggregate Forwarding Rate Degradation is calculated 887 for each measurement of the Unstable Aggregate Forwarding 888 Rate. The Aggregate Forwarding Rate Degradation is 889 calculated by subtracting each measurement of the Unstable 890 Aggregate Forwarding Rate from the Stable Aggregate 891 Forwarding Rate, such that 893 Aggregate Forwarding Rate Degradation= 894 Stable Aggregate Forwarding Rate - 895 Unstable Aggregate Forwarding Rate 897 Ideally, the Aggregate Forwarding Rate Degradation is zero. 899 Measurement Units: 900 pps 901 Stress Benchmarking 903 Issues: 904 None 906 See Also: 907 Instability Phase 908 Unstable Aggregate Forwarding Rate 910 3.4.2.3 Average Aggregate Forwarding Rate Degradation 912 Definition 913 DUT Benchmark that is the calculated average of the 914 obtained Degraded Forwarding Rates. 916 Discussion: 917 Average Aggregate Forwarding Rate Degradation= 918 (Sum (Stable Aggregate Forwarding Rate) - 919 Sum (Unstable Aggregate Forwarding Rate)) / Number of Samples 921 Measurement Units: 922 pps 924 Issues: 925 None 927 See Also: 928 Aggregate Forwarding Rate Degradation 930 3.4.2.4 Unstable Latency 932 Definition: 933 The average increase in measured packet latency during 934 the Instability Phase compared to the Startup Phase. 936 Discussion: 937 Latency SHOULD be measured at a fixed interval during the 938 Instability Phase. Unstable Latency is the difference 939 between Stable Latency and the average Latency measured 940 during the Instability Phase. It is expected that there 941 be an increase in average latency from the Startup Phase 942 to the Instability phase, but it is possible that the 943 difference be zero. The Unstable Latency cannot be a 944 negative number. 946 Measurement units: 947 seconds 949 Issues: 950 None 952 See Also: 953 Instability Phase 954 Stable Latency 955 Stress Benchmarking 957 3.4.2.5 Unstable Uncontrolled Sessions Lost 959 Definition: 960 Control Plane sessions that are in the down state 961 but were not intentionally brought down during the 962 Instability Phase. 964 Discussion: 965 The test equipment is able to control protocol 966 session state with the DUT. The test equipment 967 is also to monitor for sessions lost with the 968 DUT which the test equipment itself did not 969 intentionally bring down. 971 Measurement units: 972 sessions 974 Issues: 975 None 977 See Also: 978 Controlled Session Loss 979 Uncontrolled Session Loss 981 3.4.3 Instability Conditions 983 3.4.3.1 Control Plane Instability Conditions 985 Definition: 986 Control Plane conditions that occur during the Accelerated Stress 987 Benchmark to produce instability and stress the DUT. 989 Discussion: 990 Control Plane Instability Conditions are experienced by the DUT 991 after the Startup Conditions have completed. Control Plane 992 Instability Conditions experienced by the DUT include session 993 loss, route withdrawal, and route cost changes. 995 Measurement units: 996 N/A 998 Issues: 999 None 1001 See Also: 1002 Instability Conditions 1003 Data Plane Instability Conditions 1004 Management Plane Instability Conditions 1005 Security Plane Instability Conditions 1006 Stress Benchmarking 1008 3.4.3.2 Data Plane Instability Conditions 1009 Definition: 1010 Data Plane conditions that occur during the Accelerated Stress 1011 Benchmark to produce instability and stress the DUT. 1013 Discussion: 1014 Data Plane Instability Conditions are experienced by the DUT 1015 after the Startup Conditions have completed. Data Plane 1016 Instability Conditions experienced by the DUT include interface 1017 shutdown, link loss, and overloaded links. 1019 Measurement units: 1020 N/A 1022 Issues: 1023 None 1025 See Also: 1026 Instability Conditions 1027 Control Plane Instability Conditions 1028 Management Plane Instability Conditions 1029 Security Plane Instability Conditions 1031 3.4.3.3 Management Plane Instability Conditions 1032 Definition: 1033 Management Plane conditions that occur during the Accelerated 1034 Stress Benchmark to produce instability and stress the DUT. 1036 Discussion: 1037 Management Plane Instability Conditions are experienced by the DUT 1038 after the Startup Conditions have completed. Management Plane 1039 Instability Conditions experienced by the DUT include repeated 1040 FTP of large files. 1042 Measurement units: 1043 N/A 1045 Issues: 1046 None 1048 See Also: 1049 Instability Conditions 1050 Control Plane Instability Conditions 1051 Data Plane Instability Conditions 1052 Security Plane Instability Conditions 1054 3.4.3.4 Security Plane Instability Conditions 1056 Definition: 1057 Security Plane conditions that occur during the Accelerated 1058 Stress Benchmark to produce instability and stress the DUT. 1060 Stress Benchmarking 1062 Discussion: 1063 Security Plane Instability Conditions are experienced by the DUT 1064 after the Startup Conditions have completed. Security Plane 1065 Instability Conditions experienced by the DUT include session 1066 loss and uninitiated policy changes. 1068 Measurement units: 1069 N/A 1071 Issues: 1072 None 1074 See Also: 1075 Instability Conditions 1076 Control Plane Instability Conditions 1077 Data Plane Instability Conditions 1078 Management Plane Instability Conditions 1080 3.5 Recovery 1082 3.5.1 Recovery Phase 1084 Definition: 1085 The portion of the benchmarking test in which the 1086 Startup Conditions are generated with the DUT, but 1087 the Instability Conditions are no longer offered to 1088 the DUT. 1090 Discussion: 1091 The Recovery Phase is the final Phase of the 1092 benchmarking test following the Startup Phase and 1093 Instability Phase. Startup Conditions must not be 1094 Restarted. 1096 Measurement Units: 1097 None 1099 Issues: 1100 None 1102 See Also: 1103 Startup Conditions 1104 Startup Phase 1105 Instability Conditions 1106 Instability Phase 1108 3.5.2 Benchmarks 1109 3.5.2.1 Recovered Aggregate Forwarding Rate 1111 Definition 1112 Rate of traffic forwarded by the DUT during the Recovery 1113 Phase. 1115 Stress Benchmarking 1117 Discussion: 1118 Recovered Aggregate Forwarding Rate is an instantaneous 1119 measurement of the Aggregate Forwarding Rate during the 1120 Recovery Phase. Ideally, each measurement of the Recovered 1121 Aggregate Forwarding Rate equals the Stable Aggregate 1122 Forwarding Rate because the Instability Conditions 1123 do not exist in both the Startup and Recovery Phases. 1125 Measurement Units: 1126 pps 1128 Issues: 1129 None 1131 See Also: 1132 Aggregate Forwarding Rate 1133 Recovery Phase 1134 Recovered Aggregate Forwarding Rate 1135 Startup Phase 1136 Stable Aggregate Forwarding Rate 1138 3.5.2.2 Recovered Latency 1140 Definition: 1141 The average increase in measured packet latency during 1142 the Recovery Phase compared to the Startup Phase. 1144 Discussion: 1145 Latency SHOULD be measured at a fixed interval during the 1146 Recovery Phase. Unstable Latency is the difference 1147 between Stable Latency and the average Latency measured 1148 during the Recovery Phase. It is expected that there 1149 be no increase in average latency from the Startup Phase 1150 to the Recovery Phase. The Recovered Latency cannot be a 1151 negative number. 1153 Measurement units: 1154 seconds 1156 Issues: None 1158 See Also: 1159 Recovery Phase 1160 Stable Latency 1162 3.5.2.3 Recovery Time 1164 Definition 1165 The amount of time for the Recovered Aggregate Forwarding 1166 Rate to become equal to the Stable Aggregate Forwarding Rate. 1168 Stress Benchmarking 1170 Discussion 1171 Recovery Time is measured beginning at the instant the 1172 Instability Phase ends until the Recovered Aggregate 1173 Forwarding Rate equals the Stable Aggregate Forwarding 1174 Rate for a minimum duration of 180 consecutive seconds. 1176 Measurement Units: 1177 milliseconds 1179 Issues: 1180 None 1182 See Also: 1183 Recovered Aggregate Forwarding Rate 1184 Stable Aggregate Forwarding Rate 1186 3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost 1188 Definition: 1189 Control Plane sessions that are in the down state 1190 but were not intentionally brought down during the 1191 Recovery Phase. 1193 Discussion: 1194 The test equipment is able to control protocol 1195 session state with the DUT. The test equipment 1196 is also to monitor for sessions lost with the 1197 DUT which the test equipment itself did not 1198 intentionally bring down. 1200 Measurement units: 1201 sessions 1203 Issues: 1204 None 1206 See Also: 1207 Controlled Session Loss 1208 Uncontrolled Session Loss 1210 3.5.2.5 Variability Benchmarks 1212 Definition: 1213 The difference between the measured Benchmarks of the 1214 same DUT over multiple iterations. 1216 Discussion: 1217 Ideally, the measured benchmarks should be the same for multiple 1218 iterations with the same DUT. Configuration Sets and 1219 Instability Conditions SHOULD be held constant for this 1220 benchmark. Whether the DUT can exhibit such predictable and 1221 repeatable behavior is an important benchmark in itself. 1223 Stress Benchmarking 1225 Measurement units: 1226 As applicable to each Benchmark. The results are to be 1227 presented in a table format for successive Iterations. 1228 Ideally, the differences should be zero. 1230 Issues: 1231 None 1233 See Also: 1234 Startup Period 1235 Instability Period 1236 Recovery Period 1238 4. IANA Considerations 1239 This document requires no IANA considerations. 1241 5. Security Considerations 1242 Documents of this type do not directly effect the security of 1243 the Internet or of corporate networks as long as benchmarking 1244 is not performed on devices or systems connected to operating 1245 networks. 1247 6. References 1248 6.1 Normative References 1249 [1] Bradner, S., Editor, "Benchmarking Terminology for Network 1250 Interconnection Devices", RFC 1242, March 1991. 1252 [2] Mandeville, R., "Benchmarking Terminology for LAN Switching 1253 Devices", RFC 2285, June 1998. 1255 [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for 1256 Network Interconnect Devices", RFC 2544, March 1999. 1258 [4] Poretsky, S. and Rao, S., "Methodology for Accelerated 1259 Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-05, 1260 work in progress, June 2006. 1262 [5] Bradner, S., "Key words for use in RFCs to Indicate 1263 Requirement Levels", RFC 2119, March 1997. 1265 6.2 Informative References 1266 [RFC3871] RFC 3871 "Operational Security Requirements for Large 1267 Internet Service Provider (ISP) IP Network Infrastructure. 1268 G. Jones, Ed.. IETF, September 2004. 1270 [NANOG25] Poretsky, S., "Core Router Evaluation for Higher 1271 Availability", NANOG 25, June 8, 2002, Toronto, CA. 1273 [IEEECQR] Poretsky, S., "Router Stress Testing to Validate 1274 Readiness for Network Deployment", IEEE CQR 2003. 1276 Stress Benchmarking 1278 7. Author's Address 1280 Scott Poretsky 1281 Reef Point Systems 1282 8 New England Executive Park 1283 Burlington, MA 01803 1284 USA 1285 Phone: + 1 781 395 5090 1286 EMail: sporetsky@reefpoint.com 1288 Shankar Rao 1289 1801 California Street 1290 8th Floor 1291 Qwest Communications 1292 Denver, CO 80202 1293 USA 1294 Phone: + 1 303 437 6643 1295 Email: shankar.rao@qwest.com 1297 Appendix 1. White Box Benchmarking Terminology 1298 Minimum Available Memory 1299 Definition: 1300 Minimum DUT Available Memory during the duration of the 1301 Accelerated Stress Benchmark. 1303 Discussion: 1304 It is necessary to monitor DUT memory to measure this 1305 benchmark. 1307 Measurement units: 1308 bytes 1310 Issues: None 1312 See Also: 1313 Maximum CPU Utilization 1315 Maximum CPU Utilization 1316 Definition: 1317 Maximum DUT CPU utilization during the duration of the 1318 Accelerated Stress Benchmark. 1320 Discussion: 1321 It is necessary to monitor DUT CPU Utilization to measure 1322 this benchmark. 1324 Measurement units: % 1326 Issues: None 1328 See Also: 1329 Minimum Available Memory 1330 Stress Benchmarking 1332 Full Copyright Statement 1334 Copyright (C) The Internet Society (2006). 1336 This document is subject to the rights, licenses and restrictions 1337 contained in BCP 78, and except as set forth therein, the authors 1338 retain all their rights. 1340 This document and the information contained herein are provided on an 1341 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 1342 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 1343 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 1344 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 1345 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 1346 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1348 Intellectual Property 1350 The IETF takes no position regarding the validity or scope of any 1351 Intellectual Property Rights or other rights that might be claimed to 1352 pertain to the implementation or use of the technology described in 1353 this document or the extent to which any license under such rights 1354 might or might not be available; nor does it represent that it has 1355 made any independent effort to identify any such rights. Information 1356 on the procedures with respect to rights in RFC documents can be 1357 found in BCP 78 and BCP 79. 1359 Copies of IPR disclosures made to the IETF Secretariat and any 1360 assurances of licenses to be made available, or the result of an 1361 attempt made to obtain a general license or permission for the use of 1362 such proprietary rights by implementers or users of this 1363 specification can be obtained from the IETF on-line IPR repository at 1364 http://www.ietf.org/ipr. 1366 The IETF invites any interested party to bring to its attention any 1367 copyrights, patents or patent applications, or other proprietary 1368 rights that may cover technology that may be required to implement 1369 this standard. Please address the information to the IETF at ietf- 1370 ipr@ietf.org. 1372 Acknowledgement 1374 Funding for the RFC Editor function is currently provided by the 1375 Internet Society.