idnits 2.17.1 draft-ietf-bmwg-acc-bench-term-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 19. -- Found old boilerplate from RFC 3978, Section 5.5 on line 1343. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1354. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1361. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1367. ** Found boilerplate matching RFC 3978, Section 5.4, paragraph 1 (on line 1331), which is fine, but *also* found old RFC 2026, Section 10.4C, paragraph 1 text on line 40. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 9 instances of too long lines in the document, the longest one being 3 characters in excess of 72. ** There are 478 instances of lines with control characters in the document. == There are 24 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (September 2006) is 6426 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: '1' is defined on line 1247, but no explicit reference was found in the text == Unused Reference: '2' is defined on line 1250, but no explicit reference was found in the text == Unused Reference: '3' is defined on line 1253, but no explicit reference was found in the text == Unused Reference: 'RFC3871' is defined on line 1264, but no explicit reference was found in the text == Unused Reference: 'NANOG25' is defined on line 1268, but no explicit reference was found in the text == Unused Reference: 'IEEECQR' is defined on line 1271, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 1242 (ref. '1') ** Downref: Normative reference to an Informational RFC: RFC 2285 (ref. '2') ** Downref: Normative reference to an Informational RFC: RFC 2544 (ref. '3') == Outdated reference: A later version (-09) exists of draft-ietf-bmwg-acc-bench-meth-04 ** Downref: Normative reference to an Informational draft: draft-ietf-bmwg-acc-bench-meth (ref. '4') Summary: 10 errors (**), 0 flaws (~~), 10 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group 2 INTERNET-DRAFT 3 Expires in: September 2006 4 Scott Poretsky 5 Reef Point Systems 7 Shankar Rao 8 Qwest Communications 10 March 2006 12 Terminology for Accelerated Stress Benchmarking 13 15 Intellectual Property Rights (IPR) statement: 16 By submitting this Internet-Draft, each author represents that any 17 applicable patent or other IPR claims of which he or she is aware 18 have been or will be disclosed, and any of which he or she becomes 19 aware will be disclosed, in accordance with Section 6 of BCP 79. 21 Status of this Memo 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF), its areas, and its working groups. Note that 25 other groups may also distribute working documents as 26 Internet-Drafts. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 The list of current Internet-Drafts can be accessed at 34 http://www.ietf.org/ietf/1id-abstracts.txt. 36 The list of Internet-Draft Shadow Directories can be accessed at 37 http://www.ietf.org/shadow.html. 39 Copyright Notice 40 Copyright (C) The Internet Society (2006). All Rights Reserved. 42 ABSTRACT 43 This document provides the Terminology for performing Stress 44 Benchmarking of networking devices. The three phases of the Stress 45 Test: Startup, Instability and Recovery are defined along with the 46 benchmarks and configuration terms associated with the each phase. 47 Also defined are the Benchmark Planes fundamental to stress testing 48 configuration, setup and measurement. The terminology is to be 49 used with the companion framework and methodology documents. 51 Table of Contents 52 1. Introduction ............................................... 3 53 2. Existing definitions ....................................... 3 54 3. Term definitions............................................ 4 55 3.1 General Terms............................................. 4 56 Stress Benchmarking 58 3.1.1 Benchmark Planes...................................... 4 59 3.1.2 Configuration Sets.................................... 5 60 3.1.3 Startup Conditions.................................... 5 61 3.1.4 Instability Conditions................................ 6 62 3.1.5 Aggregate Forwarding Rate............................. 6 63 3.1.6 Controlled Session Loss............................... 7 64 3.1.7 Uncontrolled Session Loss............................. 7 65 3.2 Benchmark Planes.......................................... 8 66 3.2.1 Control Plane......................................... 8 67 3.2.2 Data Plane............................................ 8 68 3.2.3 Management Plane...................................... 8 69 3.2.4 Security Plane........................................ 9 70 3.3 Startup...................................................10 71 3.3.1 Startup Phase.........................................10 72 3.3.2 Benchmarks............................................10 73 3.3.2.1 Stable Aggregate Forwarding Rate..................10 74 3.3.2.2 Stable Latency....................................11 75 3.3.2.3 Stable Session Count..............................11 76 3.3.3 Control Plane.........................................12 77 3.3.3.1 Control Plane Configuration Set...................12 78 3.3.3.2 Control Plane Startup Conditions..................13 79 3.3.4 Data Plane............................................13 80 3.3.4.1 Data Plane Configuration Set......................13 81 3.3.4.2 Traffic Profile...................................13 82 3.3.5 Management Plane......................................14 83 3.3.5.1 Management Plane Configuration Set................14 84 3.3.6 Security Plane........................................15 85 3.3.6.1 Security Plane Configuration Set..................15 86 3.3.6.2 Security Plane Startup Conditions.................16 87 3.4 Instability...............................................16 88 3.4.1 Instability Phase.....................................16 89 3.4.2 Benchmarks............................................17 90 3.4.2.1 Unstable Aggregate Forwarding Rate................17 91 3.4.2.2 Aggregate Forwarding Rate Degradation.............17 92 3.4.2.3 Average Aggregate Forwarding Rate Degradation.....18 93 3.4.2.4 Unstable Latency..................................18 94 3.4.2.5 Unstable Uncontrolled Sessions Lost...............19 95 3.4.3 Instability Conditions................................19 96 3.4.3.1 Control Plane Instability Conditions..............19 97 3.4.3.2 Data Plane Instability Conditions.................20 98 3.4.3.3 Management Plane Instability Conditions...........20 99 3.4.3.4 Security Plane Instability Conditions.............20 100 3.5 Recovery..................................................21 101 3.5.1 Recovery Phase........................................21 102 3.5.2 Benchmarks............................................21 103 3.5.2.1 Recovered Aggregate Forwarding Rate...............21 104 3.5.2.2 Recovered Latency.................................22 105 3.5.2.3 Recovery Time.....................................22 106 3.5.2.4 Recovered Uncontrolled Sessions Lost..............23 107 3.5.2.5 Variability Benchmarks............................23 108 4. IANA Considerations.........................................24 109 Stress Benchmarking 111 5. Security Considerations.....................................24 112 6. References..................................................24 113 7. Author's Address............................................25 114 Appendix 1 - White Box Benchmarks..............................25 116 1. Introduction 118 Routers in an operational network are simultaneously configured with 119 multiple protocols and security policies while forwarding traffic and 120 being managed. To accurately benchmark a router for deployment it is 121 necessary to test that router in operational conditions by 122 simultaneously configuring and scaling network protocols and security 123 policies, forwarding traffic, and managing the device. It is helpful 124 to accelerate these network operational conditions so that the 125 router under test can be benchmarked with faster test duration. 126 Testing a router in accelerated network conditions is known as 127 Accelerated Stress Testing. 129 This document provides the Terminology for performing Stress 130 Benchmarking of networking devices. The three phases of the Stress 131 Test: Startup, Instability and Recovery are defined along with the 132 benchmark and configuration terms associated with the each phase. 133 Benchmarks for stress testing are defined using the Aggregate 134 Forwarding Rate and control plane Session Count during each phase 135 of the test. For each plane, the Configuration Set, Startup 136 Conditions, and Instability Conditions are defined. Also defined are 137 the Benchmark Planes fundamental to stress testing configuration, setup 138 and measurement. These are the Control Plane, Data Plane, Management 139 Plane and Security Plane Multiple benchmarks are made for each 140 Benchmark Plane during each Phase. Benchmarks can be compared across 141 multiple planes for the same DUT or at the same plane for 2 or more 142 DUTS. These benchmarks White Box benchmarks are provided in Appendix 1 143 for additional DUT behavior measurements. The terminology is to be 144 used with the companion methodology document [4]. The sequence of 145 phases, actions, and benchmarks are shown in Table 1. 147 2. Existing definitions 148 RFC 1242 "Benchmarking Terminology for Network Interconnect 149 Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching 150 Devices" should be consulted before attempting to make use of this 151 document. For the sake of clarity and continuity this RFC adopts 152 the template for definitions set out in Section 2 of RFC 1242. 153 Definitions are indexed and grouped together in sections for ease 154 of reference. 156 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 157 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 158 document are to be interpreted as described in BCP 14, RFC 2119 159 [5]. RFC 2119 defines the use of these key words to help make the 160 intent of standards track documents as clear as possible. While this 161 document uses these keywords, this document is not a standards track 162 document. 164 Stress Benchmarking 166 Table 1. Phase Sequence and Benchmarks 167 III. Recovery Phase II. Instability Phase I. Startup Phase 168 <-----------------<---<-------------------<----<--------------< 169 Remove Instability Achieve Configuration Apply Startup 170 Conditions Set Conditions 172 Benchmark: Benchmark: Benchmark: 173 Recovered Aggregate Unstable Aggregate Stable Aggregate 174 Forwarding Rate Forwarding Rate Forwarding Rate 176 Degraded Aggregate 177 Forwarding Rate 179 Average Degraded 180 Forwarding Rate 182 Recovered Latency Unstable Latency Startup Latency 184 Recovered Uncontrolled Recovered Uncontrolled Stable Session Count 185 Sessions Lost Sessions Lost 187 Recovery Time 189 3. Term definitions 190 3.1 General Terms 191 3.1.1 Benchmark Planes 193 Definition: 194 The features, conditions, and behavior for the Accelerated Stress 195 Benchmarking. 197 Discussion: 198 There are four Benchmark Planes: Control Plane, Data Plane, 199 Management Plane, and Security Plane as shown in Figure 1. The 200 Benchmark Planes define the Configuration, Startup Conditions, 201 Instability Conditions, and Failure Conditions used for the test. 203 Measurement units: 204 N/A 206 Issues: 207 None 209 See Also: 210 Control Plane 211 Data Plane 212 Management Plane 213 Security Plane 214 Stress Benchmarking 216 ___________ ___________ 217 | Control | | Management| 218 | Plane |___ ___| Plane | 219 | | | | | | 220 ----------- | | ----------- 221 \/ \/ ___________ 222 ___________ | Security | 223 | |<-----------| Plane | 224 | DUT | | | 225 |--->| |<---| ----------- 226 | ----------- | 227 | | 228 | ___________ | 229 | | Data | | 230 |--->| Plane |<---| 231 | | 232 ----------- 234 Figure 1. Router Accelerated Stress Benchmarking Planes 236 3.1.2 Configuration Sets 238 Definition: 239 The features and scaling limits used during the Accelerated Stress 240 Benchmarking. 242 Discussion: 243 There are four Configuration Sets: Control Plane Configuration Set, 244 Data Plane Configuration Set, Management Plane Configuration Set, 245 and Security Plane Configuration Set. 247 Measurement units: 248 N/A 250 Issues: 251 None 253 See Also: 254 Control Plane Configuration Set 255 Data Plane Configuration Set 256 Management Plane Configuration Set 257 Security Plane Configuration Set 259 3.1.3 Startup Conditions 261 Definition: 262 Test conditions that occur at the start of the Accelerated 263 Life Benchmark to establish conditions for the remainder of 264 the test. 266 Stress Benchmarking 268 Discussion: 269 Startup Conditions may cause stress on the DUT and produce 270 failure. Startup Conditions are defined for the Control 271 Plane and Security Plane. 273 Measurement units: 274 N/A 276 Issues: 277 None 279 See Also: 280 Control Plane Startup Conditions 281 Data Plane Startup Conditions 282 Management Plane Startup Conditions 283 Security Plane Startup Conditions 285 3.1.4 Instability Conditions 287 Definition: 288 Test conditions that occur during the Accelerated Stress 289 Benchmark to produce instability and stress the DUT. 291 Discussion: 292 Instability Conditions are applied to the DUT after the 293 Startup Conditions have completed. Instability Conditions 294 occur for the Control Plane, Data Plane, Management Plane, 295 and Security Plane. 297 Measurement units: 298 N/A 300 Issues: None 302 See Also: 303 Control Plane Instability Conditions 304 Data Plane Instability Conditions 305 Management Plane Instability Conditions 306 Security Plane Instability Conditions 308 3.1.5 Aggregate Forwarding Rate 310 Definition: 311 Sum of forwarding rates for all interfaces on the 312 DUT during the Startup Phase. 314 Discussion: 315 Each interface of the DUT forwards traffic at some 316 measured rate. The Aggregate Forwarding Rate is the 317 sum of forwarding rates for all interfaces on the DUT. 319 Stress Benchmarking 321 Measurement units: 322 pps 324 Issues: 325 None 327 See Also: 328 Startup Phase 330 3.1.6 Controlled Session Loss 332 Definition: 333 Control Plane sessions that are intentionally brought 334 down during the Stress test. 336 Discussion: 337 The test equipment is able to control protocol 338 session state with the DUT. 340 Measurement units: 341 None 343 Issues: 344 None 346 See Also: 347 Uncontrolled Session Loss 349 3.1.7 Uncontrolled Session Loss 351 Definition: 352 Control Plane sessions that are in the down state 353 but were not intentionally brought down during the 354 Stress test. 356 Discussion: 357 The test equipment is able to control protocol 358 session state with the DUT. The test equipment 359 is also to monitor for sessions lost with the 360 DUT which the test equipment itself did not 361 intentionally bring down. 363 Measurement units: 364 N/A 366 Issues: 367 None 369 See Also: 370 Controlled Session Loss 371 Stress Benchmarking 373 3.2 Benchmark Planes 375 3.2.1 Control Plane 376 Definition: 377 The Description of the control protocols enabled for 378 the Accelerated Stress Benchmarking. 380 Discussion: 381 The Control Plane defines the Configuration, Startup 382 Conditions, and Instability Conditions of the control 383 protocols. Control Plane protocols may include routing 384 protocols, multicast protocols, and MPLS protocols. 385 These can be enabled or disabled for a benchmark test. 387 Measurement units: 388 N/A 390 Issues: 391 None 393 See Also: 394 Benchmark Planes 395 Control Plane Configuration Set 396 Control Plane Startup Conditions 397 Control Plane Instability Conditions 399 3.2.2 Data Plane 400 Definition: 401 The data traffic profile used for the Accelerated Stress 402 Benchmarking. 404 Discussion: 405 The Data Plane defines the Configuration, Startup 406 Conditions, and Instability Conditions of the data 407 traffic. The Data Plane includes the traffic and 408 interface profile. 410 Measurement Units: 411 N/A 413 See Also: 414 Benchmark Planes 415 Data Plane Configuration Set 416 Data Plane Startup Conditions 417 Data Plane Instability Conditions 419 3.2.3 Management Plane 421 Definition: 422 The Management features and tools used for the 423 Accelerated Stress Benchmarking. 425 Stress Benchmarking 427 Discussion: 428 A key component of the Accelerated Stress Benchmarking is the 429 Management Plane to assess manageability of the router 430 under stress. The Management Plane defines the Configuration, 431 Startup Conditions, and Instability Conditions of the 432 management protocols and features. The Management Plane 433 includes SNMP, Logging/Debug, Statistics Collection, and 434 management configuration sessions such as telnet, SSH, and 435 serial console. SNMP Gets SHOULD be performed continuously. 436 Management configuration sessions should be open 437 simultaneously and be repeatedly open and closed. Open 438 management sessions should have valid and invalid 439 configuration and show commands entered. 441 Measurement units: 442 N/A 444 Issues: 445 None 447 See Also: 448 Benchmark Planes 449 Management Plane Configuration Set 450 Management Plane Startup Conditions 451 Management Plane Instability Conditions 453 3.2.4 Security Plane 455 Definition: 456 The Security features used during the Accelerated Stress 457 Benchmarking. 459 Discussion: 460 The Security Plane defines the Configuration, Startup 461 Conditions, and Instability Conditions of the security 462 features and protocols. The Security Plane includes the 463 ACLs, Firewall, Secure Protocols, and User Login. Tunnels 464 for those such as IPsec should be established and flapped. 465 Policies for Firewalls and ACLs should be repeatedly added 466 and removed from the configuration via telnet, SSH, or 467 serial management sessions. 469 Measurement units: 470 N/A 472 Issues: None 474 See Also: 475 Benchmark Planes 476 Security Plane Configuration Set 477 Security Plane Startup Conditions 478 Security Plane Instability Conditions 479 Stress Benchmarking 481 3.3 Startup 483 3.3.1 Startup Phase 485 Definition 486 The portion of the benchmarking test in which the 487 Startup Conditions are generated with the DUT. This 488 begins with the attempt to establish the first session 489 and ends when the last Control Plane session is 490 established. 492 Discussion: 493 The Startup Phase is the first Phase of the benchmarking 494 test preceding the Instability Phase and Recovery Phase. 495 It is specified by the Configuration Sets and Startup 496 Conditions for each Benchmark Plane. The Startup Phase ends 497 and Instability Phase may begin when the Configuration Sets 498 are achieved with the DUT. 500 Measurement Units: 501 None 503 Issues: 504 The 'last control plane session is established' may not 505 be a sufficient indicator that steady-state is achieved 506 and Instability Conditions can be applied to begin the 507 Instability Phase. 509 See Also: 510 Benchmark Plane 511 Configuration Sets 512 Startup Conditions 513 Instability Phase 514 Recovery Phase 516 3.3.2 Benchmarks 517 3.3.2.1 Stable Aggregate Forwarding Rate 519 Definition: 520 Average rate of traffic forwarded by the DUT during the 521 Startup Phase. 523 Discussion: 524 Stable Aggregate Forwarding Rate is the calculated 525 average of the Aggregate Forwarding Rates measured during 526 the Startup Phase. 528 Measurement units: 529 pps 530 Stress Benchmarking 532 Issues: 533 The act of the DUT establishing the Startup Conditions 534 could influence the forwarding rate in certain 535 implementations so that this "baseline" for the 536 remainder of the test is lowered. The alternative is 537 to change the definition of Stable Aggregate 538 Forwarding Rate so that it measured during the Startup 539 Phase, but after Startup Conditions are achieved. 540 The disadvantage of this definition would be that it 541 loses measurement of any impact that establishing 542 Startup Conditions would have on forwarding rate. When 543 comparing the Startup Aggregate Forwarding Rate benchmark 544 of two devices it is preferred to know the impact 545 establishing Startup Conditions has on Forwarding Rate. 546 The definition was therefore selected so that Stable 547 Aggregate Forwarding Rate is calculated from measurement 548 samples throughout the entire Startup Phase. 550 See Also: 551 Startup Phase 552 Aggregate Forwarding Rate 554 3.3.2.2 Stable Latency 556 Definition: 557 Average measured latency of traffic forwarded by the DUT 558 during the Startup Phase. 560 Discussion: 561 Stable Latency is the calculated average Latency during 562 the Startup Phase. 564 Measurement units: 565 seconds 567 Issues: 568 None 570 See Also: 571 Startup Phase 572 Stable Aggregate Forwarding Rate 574 3.3.2.3 Stable Session Count 576 Definition: 577 Total number of control plane sessions/adjacencies 578 established and maintained by the DUT during the Startup 579 Phase and prior to Instability Conditions being initiated. 581 Discussion: 582 This measurement should be made after the Control 583 Plane Startup Conditions are applied to the DUT. 585 Stress Benchmarking 587 Measurement units: 588 sessions 590 Issues: 591 None 593 See Also: 594 Startup Phase 596 3.3.3 Control Plane 598 3.3.3.1 Control Plane Configuration Set 599 Definition: 600 The routing protocols and scaling values used for the Accelerated 601 Life Benchmarking. 603 Discussion: 604 Control Plane Configuration Set is shown in Figure 2 and specifies 605 the Routing Protocols, Multicast, and MPLS configuration. Specific 606 protocols can be enabled or disabled for a benchmark test. 608 Measurement units: 609 N/A 611 Issues: 612 None 614 See Also: 615 Data Plane Configuration Set 616 Management Configuration Set 617 Security Configuration Set 619 ____________ ____________ ____________ 620 | Routing | | Multicast | | MPLS | 621 | Protocols |___ | Protocols | __| Protocols | 622 | | | | | | | | 623 ------------ | ------------ | ------------ 624 | | | 625 | | | 626 | \/ | 627 | ___________ | 628 | | | | 629 |------->| DUT |<------| 630 ``| | 631 ----------- 632 Figure 2. Control Plane Configuration Module 633 Stress Benchmarking 635 3.3.3.2 Control Plane Startup Conditions 637 Definition: 638 Control Plane conditions that occur at the start 639 of the Accelerated Stress Benchmarking to establish conditions 640 for the remainder of the test. 642 Discussion: 643 Startup Conditions may cause stress on the DUT and produce 644 failure. Startup Conditions for the Control Plane include 645 session establishment rate, number of sessions established 646 and number of routes learned. 648 Measurement units: 649 N/A 651 Issues: 652 None 654 See Also: 655 Startup Conditions 656 Security Plane Startup Conditions 657 Control Plane Configuration Set 659 3.3.4 Data Plane 660 3.3.4.1 Data Plane Configuration Set 662 Definition: 663 The data traffic profile enabled for the Accelerated Stress 664 Benchmarking. 666 Discussion: 667 Data Plane Configuration Set includes the Traffic Profile and 668 interfaces used for the Accelerated Stress Benchmarking. 670 Measurement Units: 671 N/A 673 Issues: None 675 See Also: 676 Traffic Profile 678 3.3.4.2 Traffic Profile 679 Definition 680 The characteristics of the Offered Load to the DUT used for 681 the Accelerated Stress Benchmarking. 683 Discussion 684 The Traffic Profile specifies the number of packet size(s), packet 685 rate per interface, number of flows, and encapsulation used for the 686 offered load to the DUT. 688 Stress Benchmarking 690 Measurement Units: 691 Traffic Profile is reported as follows: 693 Parameter Units 694 --------- ------ 695 Packet Size(s) bytes 696 Packet Rate(interface) array of packets per second 697 Number of Flows number 698 Encapsulation(flow) array of encapsulation type 700 Issues: 701 None 703 See Also: 704 Data Plane Configuration Set 706 3.3.5 Management Plane 707 3.3.5.1 Management Plane Configuration Set 709 Definition: 710 The router management features enabled for the 711 Accelerated Stress Test. 713 Discussion: 714 A key component of the Accelerated Stress Test is the Management 715 Configuration Set to assess manageability of the router under 716 stress. The Management Configuration Set defines the management 717 configuration of the DUT. Features that are part of the 718 Management Configuration Set include Telnet access, SNMP, 719 Logging/Debug, and Statistics Collection, and services such as 720 FTP, as shown in Figure 3. These features should be enabled 721 throughout the Stress test. SNMP Gets should be made continuously 722 with multiple FTP and Telnet sessions operating simultaneously. 723 FTP sessions should be opened and closed at varying intervals 724 and get and put files while open. Telnet sessions should be 725 opened and closed at varying intervals and enter valid and invalid 726 show and configuration commands while open. 728 Measurement units: 729 N/A 731 Issues: 732 None 734 See Also: 735 Control Plane Configuration Set 736 Data Plane Configuration Set 737 Security Plane Configuration Set 738 Stress Benchmarking 739 ____________ ____________ 740 | | | Logging/ | 741 | SNMP | __| Debug | 742 | | | | | 743 ------------ | ------------ 744 | | 745 | | 746 \/ | 747 ___________ | 748 | | | 749 | DUT |<---| 750 | | 751 ----------- 752 | 753 | 754 \/ 755 ___________ 756 | Packet | 757 | Statistics| 758 | Collector | 759 | | 760 ----------- 762 Figure 3. Management Plane Configuration Set 764 3.3.6 Security Plane 765 3.3.6.1 Security Plane Configuration Set 767 Definition: 768 Security features and scaling enabled for the Accelerated Stress 769 Test. 771 Discussion: 772 The Security Plane Configuration Set includes the configuration 773 and scaling of ACLs, Firewall, IPsec, and User Access, as shown 774 in Figure 4. Tunnels should be established and policies 775 configured. Instability is introduced by flapping tunnels and 776 configuring and removing policies. 778 ____________ ____________ ____________ 779 | | | Secure | | User | 780 |ACL/Firewall| | Protocol | __| Access | 781 | | | | | | | 782 ------------ ------------ | ------------ 783 | | | 784 | | | 785 | \/ | 786 | ___________ | 787 | | | | 788 |------->| DUT |<--------| 789 | | 790 ----------- 791 Figure 4. Security Configuration Module 792 Stress Benchmarking 794 Measurement units: 795 N/A 797 Issues: 798 None 800 See Also: 801 ACL Configuration Set 802 Secure Protocol Configuration Set 803 Password Login Configuration Set 805 3.3.6.2 Security Plane Startup Conditions 806 Definition: 807 Security Plane conditions that occur at the start 808 of the Accelerated Stress Benchmarking to establish conditions 809 for the remainder of the test. 811 Discussion: 812 Startup Conditions may cause stress on the DUT and produce 813 failure. Startup Conditions for the Security Plane include 814 session establishment rate, number of sessions established 815 and number of policies learned, and number of user access 816 sessions opened. 818 Measurement units: 819 N/A 821 Issues: 822 None 824 See Also: 825 Startup Conditions 826 Data Plane Startup Conditions 827 Management Plane Startup Conditions 828 Security Plane Startup Conditions 830 3.4 Instability 832 3.4.1 Instability Phase 834 Definition: 835 The portion of the benchmarking test in which the 836 Instability Conditions are offered to the DUT. 838 Discussion: 839 The Instability Phase is the middle Phase of 840 of the benchmarking test following the Startup 841 Phase and preceding the Recovery Phase. 843 Measurement Units: 844 None 845 Stress Benchmarking 847 Issues: 848 None 850 See Also: 851 Instability Conditions 852 Startup Phase 853 Recovery Phase 855 3.4.2 Benchmarks 856 3.4.2.1 Unstable Aggregate Forwarding Rate 858 Definition: 859 Rate of traffic forwarded by the DUT during the 860 Instability Phase. 862 Discussion: 863 Unstable Aggregated Forwarding Rate is an instantaneous 864 measurement of the Aggregate Forwarding Rate during the 865 Instability Phase. 867 Measurement units: 868 pps 870 Issues: 871 None 873 See Also: 874 Instability Conditions 875 Aggregate Forwarding Rate 877 3.4.2.2 Aggregate Forwarding Rate Degradation 879 Definition: 880 The reduction in Aggregate Forwarding Rate during the 881 Instability Phase. 883 Discussion: 884 The Aggregate Forwarding Rate Degradation is calculated 885 for each measurement of the Unstable Aggregate Forwarding 886 Rate. The Aggregate Forwarding Rate Degradation is 887 calculated by subtracting each measurement of the Unstable 888 Aggregate Forwarding Rate from the Stable Aggregate 889 Forwarding Rate, such that 891 Aggregate Forwarding Rate Degradation= 892 Stable Aggregate Forwarding Rate - 893 Unstable Aggregate Forwarding Rate 895 Ideally, the Aggregate Forwarding Rate Degradation is zero. 897 Measurement Units: 898 pps 899 Stress Benchmarking 901 Issues: 902 None 904 See Also: 905 Instability Phase 906 Unstable Aggregate Forwarding Rate 908 3.4.2.3 Average Aggregate Forwarding Rate Degradation 910 Definition 911 DUT Benchmark that is the calculated average of the 912 obtained Degraded Forwarding Rates. 914 Discussion: 915 Average Aggregate Forwarding Rate Degradation= 916 (Sum (Stable Aggregate Forwarding Rate) - 917 Sum (Unstable Aggregate Forwarding Rate)) / Number of Samples 919 Measurement Units: 920 pps 922 Issues: 923 None 925 See Also: 926 Aggregate Forwarding Rate Degradation 928 3.4.2.4 Unstable Latency 930 Definition: 931 The average increase in measured packet latency during 932 the Instability Phase compared to the Startup Phase. 934 Discussion: 935 Latency SHOULD be measured at a fixed interval during the 936 Instability Phase. Unstable Latency is the difference 937 between Stable Latency and the average Latency measured 938 during the Instability Phase. It is expected that there 939 be an increase in average latency from the Startup Phase 940 to the Instability phase, but it is possible that the 941 difference be zero. The Unstable Latency cannot be a 942 negative number. 944 Measurement units: 945 seconds 947 Issues: 948 None 950 See Also: 951 Instability Phase 952 Stable Latency 953 Stress Benchmarking 955 3.4.2.5 Unstable Uncontrolled Sessions Lost 957 Definition: 958 Control Plane sessions that are in the down state 959 but were not intentionally brought down during the 960 Instability Phase. 962 Discussion: 963 The test equipment is able to control protocol 964 session state with the DUT. The test equipment 965 is also to monitor for sessions lost with the 966 DUT which the test equipment itself did not 967 intentionally bring down. 969 Measurement units: 970 sessions 972 Issues: 973 None 975 See Also: 976 Controlled Session Loss 977 Uncontrolled Session Loss 979 3.4.3 Instability Conditions 981 3.4.3.1 Control Plane Instability Conditions 983 Definition: 984 Control Plane conditions that occur during the Accelerated Stress 985 Benchmark to produce instability and stress the DUT. 987 Discussion: 988 Control Plane Instability Conditions are experienced by the DUT 989 after the Startup Conditions have completed. Control Plane 990 Instability Conditions experienced by the DUT include session 991 loss, route withdrawal, and route cost changes. 993 Measurement units: 994 N/A 996 Issues: 997 None 999 See Also: 1000 Instability Conditions 1001 Data Plane Instability Conditions 1002 Management Plane Instability Conditions 1003 Security Plane Instability Conditions 1004 Stress Benchmarking 1006 3.4.3.2 Data Plane Instability Conditions 1007 Definition: 1008 Data Plane conditions that occur during the Accelerated Stress 1009 Benchmark to produce instability and stress the DUT. 1011 Discussion: 1012 Data Plane Instability Conditions are experienced by the DUT 1013 after the Startup Conditions have completed. Data Plane 1014 Instability Conditions experienced by the DUT include interface 1015 shutdown, link loss, and overloaded links. 1017 Measurement units: 1018 N/A 1020 Issues: 1021 None 1023 See Also: 1024 Instability Conditions 1025 Control Plane Instability Conditions 1026 Management Plane Instability Conditions 1027 Security Plane Instability Conditions 1029 3.4.3.3 Management Plane Instability Conditions 1030 Definition: 1031 Management Plane conditions that occur during the Accelerated 1032 Life Benchmark to produce instability and stress the DUT. 1034 Discussion: 1035 Management Plane Instability Conditions are experienced by the DUT 1036 after the Startup Conditions have completed. Management Plane 1037 Instability Conditions experienced by the DUT include repeated 1038 FTP of large files. 1040 Measurement units: 1041 N/A 1043 Issues: 1044 None 1046 See Also: 1047 Instability Conditions 1048 Control Plane Instability Conditions 1049 Data Plane Instability Conditions 1050 Security Plane Instability Conditions 1052 3.4.3.4 Security Plane Instability Conditions 1054 Definition: 1055 Security Plane conditions that occur during the Accelerated 1056 Life Benchmark to produce instability and stress the DUT. 1058 Stress Benchmarking 1060 Discussion: 1061 Security Plane Instability Conditions are experienced by the DUT 1062 after the Startup Conditions have completed. Security Plane 1063 Instability Conditions experienced by the DUT include session 1064 loss and uninitiated policy changes. 1066 Measurement units: 1067 N/A 1069 Issues: 1070 None 1072 See Also: 1073 Instability Conditions 1074 Control Plane Instability Conditions 1075 Data Plane Instability Conditions 1076 Management Plane Instability Conditions 1078 3.5 Recovery 1080 3.5.1 Recovery Phase 1082 Definition: 1083 The portion of the benchmarking test in which the 1084 Startup Conditions are generated with the DUT, but 1085 the Instability Conditions are no longer offered to 1086 the DUT. 1088 Discussion: 1089 The Recovery Phase is the final Phase of the 1090 benchmarking test following the Startup Phase and 1091 Instability Phase. Startup Conditions must not be 1092 Restarted. 1094 Measurement Units: 1095 None 1097 Issues: 1098 None 1100 See Also: 1101 Startup Conditions 1102 Startup Phase 1103 Instability Conditions 1104 Instability Phase 1106 3.5.2 Benchmarks 1107 3.5.2.1 Recovered Aggregate Forwarding Rate 1109 Definition 1110 Rate of traffic forwarded by the DUT during the Recovery 1111 Phase. 1113 Stress Benchmarking 1115 Discussion: 1116 Recovered Aggregate Forwarding Rate is an instantaneous 1117 measurement of the Aggregate Forwarding Rate during the 1118 Recovery Phase. Ideally, each measurement of the Recovered 1119 Aggregate Forwarding Rate equals the Stable Aggregate 1120 Forwarding Rate because the Instability Conditions 1121 do not exist in both the Startup and Recovery Phases. 1123 Measurement Units: 1124 pps 1126 Issues: 1127 None 1129 See Also: 1130 Aggregate Forwarding Rate 1131 Recovery Phase 1132 Recovered Aggregate Forwarding Rate 1133 Startup Phase 1134 Stable Aggregate Forwarding Rate 1136 3.5.2.2 Recovered Latency 1138 Definition: 1139 The average increase in measured packet latency during 1140 the Recovery Phase compared to the Startup Phase. 1142 Discussion: 1143 Latency SHOULD be measured at a fixed interval during the 1144 Recovery Phase. Unstable Latency is the difference 1145 between Stable Latency and the average Latency measured 1146 during the Recovery Phase. It is expected that there 1147 be no increase in average latency from the Startup Phase 1148 to the Recovery Phase. The Recovered Latency cannot be a 1149 negative number. 1151 Measurement units: 1152 seconds 1154 Issues: None 1156 See Also: 1157 Recovery Phase 1158 Stable Latency 1160 3.5.2.3 Recovery Time 1162 Definition 1163 The amount of time for the Recovered Aggregate Forwarding 1164 Rate to become equal to the Stable Aggregate Forwarding Rate. 1166 Stress Benchmarking 1168 Discussion 1169 Recovery Time is measured beginning at the instant the 1170 Instability Phase ends until the Recovered Aggregate 1171 Forwarding Rate equals the Stable Aggregate Forwarding 1172 Rate for a minimum duration of 180 consecutive seconds. 1174 Measurement Units: 1175 milliseconds 1177 Issues: 1178 None 1180 See Also: 1181 Recovered Aggregate Forwarding Rate 1182 Stable Aggregate Forwarding Rate 1184 3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost 1186 Definition: 1187 Control Plane sessions that are in the down state 1188 but were not intentionally brought down during the 1189 Recovery Phase. 1191 Discussion: 1192 The test equipment is able to control protocol 1193 session state with the DUT. The test equipment 1194 is also to monitor for sessions lost with the 1195 DUT which the test equipment itself did not 1196 intentionally bring down. 1198 Measurement units: 1199 sessions 1201 Issues: 1202 None 1204 See Also: 1205 Controlled Session Loss 1206 Uncontrolled Session Loss 1208 3.5.2.5 Variability Benchmarks 1210 Definition: 1211 The difference between the measured Benchmarks of the 1212 same DUT over multiple iterations. 1214 Discussion: 1215 Ideally, the measured benchmarks should be the same for multiple 1216 iterations with the same DUT. Configuration Sets and Instability 1217 Conditions SHOULD be held constant for this benchmark. Whether the 1218 DUT can exhibit such predictable and repeatable behavior is an 1219 important benchmark in itself. 1221 Stress Benchmarking 1223 Measurement units: 1224 As applicable to each Benchmark. The results are to be 1225 presented in a table format for successive Iterations. 1226 Ideally, the differences should be zero. 1228 Issues: 1229 None 1231 See Also: 1232 Startup Period 1233 Instability Period 1234 Recovery Period 1236 4. IANA Considerations 1237 This document requires no IANA considerations. 1239 5. Security Considerations 1240 Documents of this type do not directly effect the security of 1241 the Internet or of corporate networks as long as benchmarking 1242 is not performed on devices or systems connected to operating 1243 networks. 1245 6. References 1246 6.1 Normative References 1247 [1] Bradner, S., Editor, "Benchmarking Terminology for Network 1248 Interconnection Devices", RFC 1242, March 1991. 1250 [2] Mandeville, R., "Benchmarking Terminology for LAN Switching 1251 Devices", RFC 2285, June 1998. 1253 [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for 1254 Network Interconnect Devices", RFC 2544, March 1999. 1256 [4] Poretsky, S. and Rao, S., "Methodology for Accelerated 1257 Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-04, 1258 work in progress, March 2006. 1260 [5] Bradner, S., "Key words for use in RFCs to Indicate Requirement 1261 Levels", RFC 2119, March 1997. 1263 6.2 Informative References 1264 [RFC3871] RFC 3871 "Operational Security Requirements for Large 1265 Internet Service Provider (ISP) IP Network Infrastructure. 1266 G. Jones, Ed.. IETF, September 2004. 1268 [NANOG25] "Core Router Evaluation for Higher Availability", Scott 1269 Poretsky, NANOG 25, June 8, 2002, Toronto, CA. 1271 [IEEECQR] "Router Stress Testing to Validate Readiness for Network 1272 Deployment", Scott Poretsky, IEEE CQR 2003. 1274 Stress Benchmarking 1276 7. Author's Address 1278 Reef Point Systems 1279 8 New England Executive Park 1280 Burlington, MA 01803 1281 USA 1282 Phone: + 1 781 395 5090 1283 EMail: sporetsky@reefpoint.com 1285 Shankar Rao 1286 1801 California Street 1287 8th Floor 1288 Qwest Communications 1289 Denver, CO 80202 1290 USA 1291 Phone: + 1 303 437 6643 1292 Email: shankar.rao@qwest.com 1294 Appendix 1. White Box Benchmarking Terminology 1295 Minimum Available Memory 1296 Definition: 1297 Minimum DUT Available Memory during the duration of the 1298 Accelerated Stress Test. 1300 Discussion: 1301 It is necessary to monitor DUT memory to measure this 1302 benchmark. 1304 Measurement units: 1305 bytes 1307 Issues: None 1309 See Also: 1310 Maximum CPU Utilization 1312 Maximum CPU Utilization 1313 Definition: 1314 Maximum DUT CPU utilization during the duration of the 1315 Accelerated Stress Test. 1317 Discussion: 1318 It is necessary to monitor DUT CPU Utilization to measure 1319 this benchmark. 1321 Measurement units: % 1323 Issues: None 1325 See Also: 1326 Minimum Available Memory 1327 Stress Benchmarking 1329 Full Copyright Statement 1331 Copyright (C) The Internet Society (2006). 1333 This document is subject to the rights, licenses and restrictions 1334 contained in BCP 78, and except as set forth therein, the authors 1335 retain all their rights. 1337 This document and the information contained herein are provided on an 1338 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 1339 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 1340 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 1341 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 1342 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 1343 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1345 Intellectual Property 1347 The IETF takes no position regarding the validity or scope of any 1348 Intellectual Property Rights or other rights that might be claimed to 1349 pertain to the implementation or use of the technology described in 1350 this document or the extent to which any license under such rights 1351 might or might not be available; nor does it represent that it has 1352 made any independent effort to identify any such rights. Information 1353 on the procedures with respect to rights in RFC documents can be 1354 found in BCP 78 and BCP 79. 1356 Copies of IPR disclosures made to the IETF Secretariat and any 1357 assurances of licenses to be made available, or the result of an 1358 attempt made to obtain a general license or permission for the use of 1359 such proprietary rights by implementers or users of this 1360 specification can be obtained from the IETF on-line IPR repository at 1361 http://www.ietf.org/ipr. 1363 The IETF invites any interested party to bring to its attention any 1364 copyrights, patents or patent applications, or other proprietary 1365 rights that may cover technology that may be required to implement 1366 this standard. Please address the information to the IETF at ietf- 1367 ipr@ietf.org. 1369 Acknowledgement 1371 Funding for the RFC Editor function is currently provided by the 1372 Internet Society.