idnits 2.17.1 draft-ietf-bmwg-acc-bench-term-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3667, Section 5.1 on line 20. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1272. ** The document claims conformance with section 10 of RFC 2026, but uses some RFC 3978/3979 boilerplate. As RFC 3978/3979 replaces section 10 of RFC 2026, you should not claim conformance with it if you have changed to using RFC 3978/3979 boilerplate. ** The document seems to lack an RFC 3978 Section 5.1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** The document seems to lack an RFC 3978 Section 5.5 (updated by RFC 4748) Disclaimer -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document seems to lack an RFC 3979 Section 5, para. 1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document seems to lack an RFC 3979 Section 5, para. 3 IPR Disclosure Invitation -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document uses RFC 3667 boilerplate or RFC 3978-like boilerplate instead of verbatim RFC 3978 boilerplate. After 6 May 2005, submission of drafts without verbatim RFC 3978 boilerplate is not accepted. The following non-3978 patterns matched text found in the document. That text should be removed or replaced: By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. == No 'Intended status' indicated for this document; assuming Proposed Standard == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 59 lines == It seems as if not all pages are separated by form feeds - found 0 form feeds but 25 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. (A line matching the expected section header was found, but with an unexpected indentation: ' 1. Introduction' ) ** The document seems to lack a Security Considerations section. (A line matching the expected section header was found, but with an unexpected indentation: ' 4. Security Considerations' ) ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack an Authors' Addresses Section. ** There are 2 instances of too long lines in the document, the longest one being 2 characters in excess of 72. ** There are 489 instances of lines with control characters in the document. ** The abstract seems to contain references ([2], [3], [4], [5], [6], [1]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (April 2005) is 6948 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? '6' on line 1197 looks like a reference -- Missing reference section? '1' on line 1182 looks like a reference -- Missing reference section? '2' on line 1185 looks like a reference -- Missing reference section? '3' on line 1188 looks like a reference -- Missing reference section? '4' on line 1191 looks like a reference -- Missing reference section? '5' on line 1194 looks like a reference Summary: 16 errors (**), 0 flaws (~~), 4 warnings (==), 10 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group 3 INTERNET-DRAFT 4 Expires in: April 2005 5 Scott Poretsky 6 Quarry Technologies 8 Shankar Rao 9 Qwest Communications 11 October 2004 13 Terminology for Accelerated Stress Benchmarking 14 16 Intellectual Property Rights (IPR) statement: 17 By submitting this Internet-Draft, I certify that any applicable 18 patent or other IPR claims of which I am aware have been disclosed, or 19 will be disclosed, and any of which I become aware will be disclosed, 20 in accordance with RFC 3668. 22 Status of this Memo 24 This document is an Internet-Draft and is in full conformance with 25 all provisions of Section 10 of RFC2026. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF), its areas, and its working groups. Note that 28 other groups may also distribute working documents as Internet- 29 Drafts. 31 Internet-Drafts are draft documents valid for a maximum of six 32 months and may be updated, replaced, or obsoleted by other 33 documents at any time. It is inappropriate to use Internet-Drafts 34 as reference material or to cite them other than as "work in 35 progress." 37 The list of current Internet-Drafts can be accessed at 38 http://www.ietf.org/ietf/1id-abstracts.txt 39 The list of Internet-Draft Shadow Directories can be accessed at 40 http://www.ietf.org/shadow.html. 42 ABSTRACT 43 This document provides the Terminology for performing Stress 44 Benchmarking of networking devices. The three phases of the Stress 45 Test: Startup, Instability and Recovery are defined along with the 46 benchmarks and configuration terms associated with the each phase. 47 Also defined are the Benchmark Planes fundamental to stress testing 48 configuration, setup and measurement. The terminology is to be 49 used with the companion framework and methodology documents. 51 Table of Contents 52 1. Introduction ............................................... 3 53 2. Existing definitions ....................................... 3 54 3. Term definitions............................................ 3 55 3.1 General Terms............................................. 3 57 3.1.1 Benchmark Planes...................................... 3 58 3.1.2 Configuration Sets.................................... 4 59 3.1.3 Startup Conditions.................................... 4 60 3.1.4 Instability Conditions................................ 5 61 3.1.5 Aggregate Forwarding Rate............................. 6 62 3.1.6 Controlled Session Loss............................... 6 63 3.1.7 Uncontrolled Session Loss............................. 6 64 3.2 Benchmark Planes.......................................... 7 65 3.2.1 Control Plane......................................... 7 66 3.2.2 Data Plane............................................ 7 67 3.2.3 Management Plane...................................... 8 68 3.2.4 Security Plane........................................ 8 69 3.3 Startup................................................... 9 70 3.3.1 Startup Phase......................................... 9 71 3.3.2 Benchmarks............................................10 72 3.3.2.1 Stable Aggregate Forwarding Rate..................10 73 3.3.2.2 Stable Latency....................................10 74 3.3.2.3 Stable Session Count..............................11 75 3.3.3 Control Plane.........................................11 76 3.3.3.1 Control Plane Configuration Set...................11 77 3.3.3.2 Control Plane Startup Conditions..................12 78 3.3.4 Data Plane............................................12 79 3.3.4.1 Data Plane Configuration Set......................12 80 3.3.4.2 Traffic Profile...................................13 81 3.3.5 Management Plane......................................13 82 3.3.5.1 Management Plane Configuration Set................13 83 3.3.6 Security Plane........................................14 84 3.3.6.1 Security Plane Configuration Set..................14 85 3.3.6.2 Security Plane Startup Conditions.................15 86 3.4 Instability...............................................15 87 3.4.1 Instability Phase.....................................15 88 3.4.2 Benchmarks............................................16 89 3.4.2.1 Unstable Aggregate Forwarding Rate................16 90 3.4.2.2 Degraded Aggregate Forwarding Rate................17 91 3.4.2.3 Average Degraded Aggregate Forwarding Rate........17 92 3.4.2.4 Unstable Latency..................................17 93 3.4.2.5 Unstable Uncontrolled Sessions Lost...............18 94 3.4.3 Instability Conditions................................18 95 3.4.3.1 Control Plane Instability Conditions..............18 96 3.4.3.2 Data Plane Instability Conditions.................19 97 3.4.3.3 Management Plane Instability Conditions...........19 98 3.4.3.4 Security Plane Instability Conditions.............20 99 3.5 Recovery..................................................20 100 3.5.1 Recovery Phase........................................20 101 3.5.2 Benchmarks............................................21 102 3.5.2.1 Recovered Aggregate Forwarding Rate...............21 103 3.5.2.2 Recovered Latency.................................21 104 3.5.2.3 Recovery Time.....................................22 105 3.5.2.4 Recovered Uncontrolled Sessions Lost..............22 106 3.5.2.5 Variability Benchmarks............................23 107 4. Security Considerations.....................................23 108 5. References..................................................23 109 6. Author's Address............................................24 110 Appendix 1 - White Box Benchmarks..............................24 112 1. Introduction 114 Routers in an operational network are simultaneously configured with 115 multiple protocols and security policies while forwarding traffic and 116 being managed. To accurately benchmark a router for deployment it is 117 necessary to test that router in operational conditions by 118 simultaneously configuring and scaling network protocols and security 119 policies, forwarding traffic, and managing the device. It is helpful 120 to accelerate these network operational conditions so that the 121 router under test can be benchmarked with faster test duration. 122 Testing a router in accelerated network conditions is known as 123 Accelerated Stress Testing. 125 This document provides the Terminology for performing Stress 126 Benchmarking of networking devices. The three phases of the Stress 127 Test: Startup, Instability and Recovery are defined along with the 128 benchmark and configuration terms associated with the each phase. 129 Benchmarks for stress testing are defined using the Aggregate 130 Forwarding Rate and control plane Session Count during each phase 131 of the test. Also defined are the Benchmark Planes fundamental to 132 stress testing configuration, setup and measurement. These are 133 the Control Plane, Data Plane, Management Plane and Security Plane 134 For each plane, the Configuration Set, Startup Conditions, and 135 Instability Conditions are defined. White Box benchmarks are 136 provided in Appendix 1 for additional DUT behavior measurements. 137 The terminology is to be used with the companion methodology 138 document [6]. 140 2. Existing definitions 142 RFC 1242 "Benchmarking Terminology for Network Interconnect 143 Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching 144 Devices" should be consulted before attempting to make use of this 145 document. 147 For the sake of clarity and continuity this RFC adopts the template 148 for definitions set out in Section 2 of RFC 1242. Definitions are 149 indexed and grouped together in sections for ease of reference. 151 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 152 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 153 this document are to be interpreted as described in RFC 2119. 155 3. Term definitions 156 3.1 General Terms 157 3.1.1 Benchmark Planes 159 Definition: 160 The features, conditions, and behavior for the Accelerated Stress 161 Benchmarking. 163 Discussion: 164 There are four Benchmark Planes: Control Plane, Data Plane, 165 Management Plane, and Security Plane as shown in Figure 1. The 166 Benchmark Planes define the Configuration, Startup Conditions, 167 Instability Conditions, and Failure Conditions used for the test. 169 ___________ ___________ 170 | Control | | Management| 171 | Plane |___ ___| Plane | 172 | | | | | | 173 ----------- | | ----------- 174 \/ \/ ___________ 175 ___________ | Security | 176 | |<-----------| Plane | 177 | DUT | | | 178 |--->| |<---| ----------- 179 | ----------- | 180 | | 181 | ___________ | 182 | | Data | | 183 |--->| Plane |<---| 184 | | 185 ----------- 187 Figure 1. Router Accelerated Stress Benchmarking Planes 189 Measurement units: 190 N/A 192 Issues: 193 None 195 See Also: 196 Control Plane 197 Data Plane 198 Management Plane 199 Security Plane 201 3.1.2 Configuration Sets 203 Definition: 204 The features and scaling limits used during the Accelerated Stress 205 Benchmarking. 207 Discussion: 208 There are four Configuration Sets: Control Plane Configuration Set, 209 Data Plane Configuration Set, Management Plane Configuration Set, 210 and Security Plane Configuration Set. 212 Measurement units: 213 N/A 214 Issues: 215 None 217 See Also: 218 Control Plane Configuration Set 219 Data Plane Configuration Set 220 Management Plane Configuration Set 221 Security Plane Configuration Set 223 3.1.3 Startup Conditions 225 Definition: 226 Test conditions that occur at the start of the Accelerated 227 Life Benchmark to establish conditions for the remainder of 228 the test. 230 Discussion: 231 Startup Conditions may cause stress on the DUT and produce 232 failure. Startup Conditions are defined for the Control 233 Plane and Security Plane. 235 Measurement units: 236 N/A 238 Issues: 239 None 241 See Also: 242 Control Plane Startup Conditions 243 Data Plane Startup Conditions 244 Management Plane Startup Conditions 245 Security Plane Startup Conditions 247 3.1.4 Instability Conditions 249 Definition: 250 Test conditions that occur during the Accelerated Stress 251 Benchmark to produce instability and stress the DUT. 253 Discussion: 254 Instability Conditions are applied to the DUT after the 255 Startup Conditions have completed. Instability Conditions 256 occur for the Control Plane, Data Plane, Management Plane, 257 and Security Plane. 259 Measurement units: 260 N/A 262 Issues: 263 None 264 See Also: 265 Control Plane Instability Conditions 266 Data Plane Instability Conditions 267 Management Plane Instability Conditions 268 Security Plane Instability Conditions 270 3.1.5 Aggregate Forwarding Rate 272 Definition: 273 Sum of forwarding rates for all interfaces on the 274 DUT during the Startup Phase. 276 Discussion: 277 Each interface of the DUT forwards traffic at some 278 measured rate. The Aggregate Forwarding Rate is the 279 sum of forwarding rates for all interfaces on the DUT. 281 Measurement units: 282 pps 284 Issues: 285 None 287 See Also: 288 Startup Phase 290 3.1.6 Controlled Session Loss 292 Definition: 293 Control Plane sessions that are intentionally brought 294 down during the Stress test. 296 Discussion: 297 The test equipment is able to control protocol 298 session state with the DUT. 300 Measurement units: 301 None 303 Issues: 304 None 306 See Also: 307 Uncontrolled Session Loss 309 3.1.7 Uncontrolled Session Loss 311 Definition: 312 Control Plane sessions that are in the down state 313 but were not intentionally brought down during the 314 Stress test. 316 Discussion: 317 The test equipment is able to control protocol 318 session state with the DUT. The test equipment 319 is also to monitor for sessions lost with the 320 DUT which the test equipment itself did not 321 intentionally bring down. 323 Measurement units: 324 N/A 326 Issues: 327 None 329 See Also: 330 Controlled Session Loss 332 3.2 Benchmark Planes 334 3.2.1 Control Plane 335 Definition: 336 The Description of the control protocols enabled for 337 the Accelerated Stress Benchmarking. 339 Discussion: 340 The Control Plane defines the Configuration, Startup 341 Conditions, and Instability Conditions of the control 342 protocols. Control Plane protocols may include routing 343 protocols, multicast protocols, and MPLS protocols. 344 These can be enabled or disabled for a benchmark test. 346 Measurement units: 347 N/A 349 Issues: 350 None 352 See Also: 353 Benchmark Planes 354 Control Plane Configuration Set 355 Control Plane Startup Conditions 356 Control Plane Instability Conditions 358 3.2.2 Data Plane 359 Definition: 360 The data traffic profile used for the Accelerated Stress 361 Benchmarking. 363 Discussion: 364 The Data Plane defines the Configuration, Startup 365 Conditions, and Instability Conditions of the data 366 traffic. The Data Plane includes the traffic and 367 interface profile. 369 Measurement Units: 370 N/A 372 See Also: 373 Benchmark Planes 374 Data Plane Configuration Set 375 Data Plane Startup Conditions 376 Data Plane Instability Conditions 378 3.2.3 Management Plane 380 Definition: 381 The Management features and tools used for the 382 Accelerated Stress Benchmarking. 384 Discussion: 385 A key component of the Accelerated Stress Benchmarking is the 386 Management Plane to assess manageability of the router 387 under stress. The Management Plane defines the Configuration, 388 Startup Conditions, and Instability Conditions of the 389 management protocols and features. The Management Plane 390 includes SNMP, Logging/Debug, Statistics Collection, and 391 management configuration sessions such as telnet, SSH, and 392 serial console. SNMP Gets SHOULD be performed continuously. 393 Management configuration sessions should be open 394 simultaneously and be repeatedly open and closed. Open 395 management sessions should have valid and invalid 396 configuration and show commands entered. 398 Measurement units: 399 N/A 401 Issues: 402 None 404 See Also: 405 Benchmark Planes 406 Management Plane Configuration Set 407 Management Plane Startup Conditions 408 Management Plane Instability Conditions 410 3.2.4 Security Plane 412 Definition: 413 The Security features used during the Accelerated Stress 414 Benchmarking. 416 Discussion: 417 The Security Plane defines the Configuration, Startup 418 Conditions, and Instability Conditions of the security 419 features and protocols. The Security Plane includes the 420 ACLs, Firewall, Secure Protocols, and User Login. Tunnels 421 for those such as IPsec should be established and flapped. 422 Policies for Firewalls and ACLs should be repeatedly added 423 and removed from the configuration via telnet, SSH, or 424 serial management sessions. 426 Measurement units: 427 N/A 429 Issues: 430 None 432 See Also: 433 Benchmark Planes 434 Security Plane Configuration Set 435 Security Plane Startup Conditions 436 Security Plane Instability Conditions 438 3.3 Startup 440 3.3.1 Startup Phase 442 Definition 443 The portion of the benchmarking test in which the 444 Startup Conditions are generated with the DUT. This 445 begins with the attempt to establish the first session 446 and ends when the last Control Plane session is 447 established. 449 Discussion: 450 The Startup Phase is the first Phase of the benchmarking 451 test preceding the Instability Phase and Recovery Phase. 452 It is specified by the Configuration Sets and Startup 453 Conditions for each Benchmark Plane. The Startup Phase ends 454 and Instability Phase may begin when the Configuration Sets 455 are achieved with the DUT. 457 Measurement Units: 458 None 460 Issues: 461 The 'last control plane session is established' may not 462 be a sufficient indicator that steady-state is achieved 463 and Instability Conditions can be applied to begin the 464 Instability Phase. 466 See Also: 467 Benchmark Plane 468 Configuration Sets 469 Startup Conditions 470 Instability Phase 471 Recovery Phase 472 3.3.2 Benchmarks 473 3.3.2.1 Stable Aggregate Forwarding Rate 475 Definition: 476 Average rate of traffic forwarded by the DUT during the 477 Startup Phase. 479 Discussion: 480 Stable Aggregate Forwarding Rate is the calculated 481 average of the Aggregate Forwarding Rates measured during 482 the Startup Phase. 484 Measurement units: 485 pps 487 Issues: 488 The act of the DUT establishing the Startup Conditions 489 could influence the forwarding rate in certain 490 implementations so that this "baseline" for the 491 remainder of the test is lowered. The alternative is 492 to change the definition of Stable Aggregate 493 Forwarding Rate so that it measured during the Startup 494 Phase, but after Startup Conditions are achieved. 495 The disadvantage of this definition would be that it 496 loses measurement of any impact that establishing 497 Startup Conditions would have on forwarding rate. When 498 comparing the Startup Aggregate Forwarding Rate benchmark 499 of two devices it is preferred to know the impact 500 establishing Startup Conditions has on Forwarding Rate. 501 The definition was therefore selected so that Stable 502 Aggregate Forwarding Rate is calculated from measurement 503 samples throughout the entire Startup Phase. 505 See Also: 506 Startup Phase 507 Aggregate Forwarding Rate 509 3.3.2.2 Stable Latency 511 Definition: 512 Average measured latency of traffic forwarded by the DUT 513 during the Startup Phase. 515 Discussion: 516 Stable Latency is the calculated average Latency during 517 the Startup Phase. 519 Measurement units: 520 seconds 522 Issues: 523 None 524 See Also: 525 Startup Phase 526 Stable Aggregate Forwarding Rate 528 3.3.2.3 Stable Session Count 530 Definition: 531 Total number of control plane sessions/adjacencies 532 established and maintained by the DUT during the Startup 533 Phase and prior to Instability Conditions being initiated. 535 Discussion: 536 This measurement should be made after the Control 537 Plane Startup Conditions are applied to the DUT. 539 Measurement units: 540 sessions 542 Issues: 543 None 545 See Also: 546 Startup Phase 548 3.3.3 Control Plane 550 3.3.3.1 Control Plane Configuration Set 551 Definition: 552 The routing protocols and scaling values used for the Accelerated 553 Life Benchmarking. 555 Discussion: 556 Control Plane Configuration Set is shown in Figure 2 and specifies 557 the Routing Protocols, Multicast, and MPLS configuration. Specific 558 protocols can be enabled or disabled for a benchmark test. 560 Measurement units: 561 N/A 563 Issues: 564 None 566 See Also: 567 Data Plane Configuration Set 568 Management Configuration Set 569 Security Configuration Set 571 ____________ ____________ ____________ 572 | Routing | | Multicast | | MPLS | 573 | Protocols |___ | Protocols | __| Protocols | 574 | | | | | | | | 575 ------------ | ------------ | ------------ 576 | | | 577 | | | 578 | \/ | 579 | ___________ | 580 | | | | 581 |------->| DUT |<------| 582 ``| | 583 ----------- 584 Figure 2. Control Plane Configuration Module 586 3.3.3.2 Control Plane Startup Conditions 588 Definition: 589 Control Plane conditions that occur at the start 590 of the Accelerated Stress Benchmarking to establish conditions 591 for the remainder of the test. 593 Discussion: 594 Startup Conditions may cause stress on the DUT and produce 595 failure. Startup Conditions for the Control Plane include 596 session establishment rate, number of sessions established 597 and number of routes learned. 599 Measurement units: 600 N/A 602 Issues: 603 None 605 See Also: 606 Startup Conditions 607 Security Plane Startup Conditions 608 Control Plane Configuration Set 610 3.3.4 Data Plane 612 3.3.4.1 Data Plane Configuration Set 614 Definition: 615 The data traffic profile enabled for the Accelerated Stress 616 Benchmarking. 618 Discussion: 619 Data Plane Configuration Set includes the Traffic Profile and 620 interfaces used for the Accelerated Stress Benchmarking. 622 Measurement Units: 623 N/A 625 Issues: 626 None 628 See Also: 629 Traffic Profile 631 3.3.4.2 Traffic Profile 632 Definition 633 The characteristics of the Offered Load to the DUT used for 634 the Accelerated Stress Benchmarking. 636 Discussion 637 The Traffic Profile specifies the number of packet size(s), packet 638 rate per interface, number of flows, and encapsulation used for the 639 offered load to the DUT. 641 Measurement Units: 642 Traffic Profile is reported as follows: 644 Parameter Units 645 --------- ------ 646 Packet Size(s) bytes 647 Packet Rate(interface) array of packets per second 648 Number of Flows number 649 Encapsulation(flow) array of encapsulation type 651 Issues: 652 None 654 See Also: 655 Data Plane Configuration Set 657 3.3.5 Management Plane 658 3.3.5.1 Management Plane Configuration Set 660 Definition: 661 The router management features enabled for the 662 Accelerated Stress Test. 664 Discussion: 665 A key component of the Accelerated Stress Test is the Management 666 Configuration Set to assess manageability of the router under 667 stress. The Management Configuration Set defines the management 668 configuration of the DUT. Features that are part of the 669 Management Configuration Set include Telnet access, SNMP, 670 Logging/Debug, and Statistics Collection, and services such as 671 FTP, as shown in Figure 3. These features should be enabled 672 throughout the Stress test. SNMP Gets should be made continuously 673 with multiple FTP and Telnet sessions operating simultaneously. 675 FTP sessions should be opened and closed at varying intervals 676 and get and put files while open. Telnet sessions should be 677 opened and closed at varying intervals and enter valid and invalid 678 show and configuration commands while open. 680 Measurement units: 681 N/A 683 Issues: 684 None 686 See Also: 687 Control Plane Configuration Set 688 Data Plane Configuration Set 689 Security Plane Configuration Set 691 ____________ ____________ 692 | | | Logging/ | 693 | SNMP | __| Debug | 694 | | | | | 695 ------------ | ------------ 696 | | 697 | | 698 \/ | 699 ___________ | 700 | | | 701 | DUT |<---| 702 | | 703 ----------- 704 | 705 | 706 \/ 707 ___________ 708 | Packet | 709 | Statistics| 710 | Collector | 711 | | 712 ----------- 714 Figure 3. Management Plane Configuration Set 716 3.3.6 Security Plane 718 3.3.6.1 Security Plane Configuration Set 720 Definition: 721 Security features and scaling enabled for the Accelerated Stress 722 Test. 724 Discussion: 725 The Security Plane Configuration Set includes the configuration 726 and scaling of ACLs, Firewall, IPsec, and User Access, as shown 727 in Figure 4. Tunnels should be established and policies 728 configured. Instability is introduced by flapping tunnels and 729 configuring and removing policies. 731 ____________ ____________ ____________ 732 | | | Secure | | User | 733 |ACL/Firewall| | Protocol | __| Access | 734 | | | | | | | 735 ------------ ------------ | ------------ 736 | | | 737 | | | 738 | \/ | 739 | ___________ | 740 | | | | 741 |------->| DUT |<--------| 742 | | 743 ----------- 744 Figure 4. Security Configuration Module 746 Measurement units: 747 N/A 749 Issues: 750 None 752 See Also: 753 ACL Configuration Set 754 Secure Protocol Configuration Set 755 Password Login Configuration Set 757 3.3.6.2 Security Plane Startup Conditions 758 Definition: 759 Security Plane conditions that occur at the start 760 of the Accelerated Stress Benchmarking to establish conditions 761 for the remainder of the test. 763 Discussion: 764 Startup Conditions may cause stress on the DUT and produce 765 failure. Startup Conditions for the Security Plane include 766 session establishment rate, number of sessions established 767 and number of policies learned, and number of user access 768 sessions opened. 770 Measurement units: 771 N/A 773 Issues: 774 None 775 See Also: 776 Startup Conditions 777 Data Plane Startup Conditions 778 Management Plane Startup Conditions 779 Security Plane Startup Conditions 781 3.4 Instability 783 3.4.1 Instability Phase 785 Definition: 786 The portion of the benchmarking test in which the 787 Instability Conditions are offered to the DUT. 789 Discussion: 790 The Instability Phase is the middle Phase of 791 of the benchmarking test following the Startup 792 Phase and preceding the Recovery Phase. 794 Measurement Units: 795 None 797 Issues: 798 None 800 See Also: 801 Instability Conditions 802 Startup Phase 803 Recovery Phase 805 3.4.2 Benchmarks 807 3.4.2.1 Unstable Aggregate Forwarding Rate 809 Definition: 810 Rate of traffic forwarded by the DUT during the 811 Instability Phase. 813 Discussion: 814 Unstable Aggregated Forwarding Rate is an instantaneous 815 measurement of the Aggregate Forwarding Rate during the 816 Instability Phase. 818 Measurement units: 819 pps 821 Issues: 822 None 824 See Also: 825 Instability Conditions 826 Aggregate Forwarding Rate 827 3.4.2.2 Degraded Aggregate Forwarding Rate 829 Definition: 830 The reduction in Aggregate Forwarding Rate during the 831 Instability Phase. 833 Discussion: 834 The Degraded Aggregate Forwarding Rate is calculated 835 for each measurement of the Unstable Aggregate 836 Forwarding Rate. The Degraded Aggregate Forwarding 837 Rate is calculated by subtracting each measurement 838 of the Unstable Aggregate Forwarding Rate from the 839 Stable Aggregate Forwarding Rate, such that 841 Degraded Forwarding Rate = 842 Stable Aggregate Forwarding Rate - 843 Unstable Aggregate Forwarding Rate 845 Ideally, the Degraded Aggregate Forwarding Rate is zero. 847 Measurement Units: 848 pps 850 Issues: 851 None 853 See Also: 854 Instability Phase 855 Unstable Aggregate Forwarding Rate 857 3.4.2.3 Average Degraded Aggregate Forwarding Rate 859 Definition 860 DUT Benchmark that is the calculated average of the 861 obtained Degraded Forwarding Rates. 863 Discussion: 865 Measurement Units: 866 pps 868 Issues: 869 None 871 See Also: 872 Degraded Aggregate Forwarding Rate 874 3.4.2.4 Unstable Latency 876 Definition: 877 The average increase in measured packet latency during 878 the Instability Phase compared to the Startup Phase. 880 Discussion: 881 Latency SHOULD be measured at a fixed interval during the 882 Instability Phase. Unstable Latency is the difference 883 between Stable Latency and the average Latency measured 884 during the Instability Phase. It is expected that there 885 be an increase in average latency from the Startup Phase 886 to the Instability phase, but it is possible that the 887 difference be zero. The Unstable Latency cannot be a 888 negative number. 890 Measurement units: 891 seconds 893 Issues: 894 None 896 See Also: 897 Instability Phase 898 Stable Latency 900 3.4.2.5 Unstable Uncontrolled Sessions Lost 902 Definition: 903 Control Plane sessions that are in the down state 904 but were not intentionally brought down during the 905 Instability Phase. 907 Discussion: 908 The test equipment is able to control protocol 909 session state with the DUT. The test equipment 910 is also to monitor for sessions lost with the 911 DUT which the test equipment itself did not 912 intentionally bring down. 914 Measurement units: 915 sessions 917 Issues: 918 None 920 See Also: 921 Controlled Session Loss 922 Uncontrolled Session Loss 924 3.4.3 Instability Conditions 926 3.4.3.1 Control Plane Instability Conditions 928 Definition: 929 Control Plane conditions that occur during the Accelerated Stress 930 Benchmark to produce instability and stress the DUT. 932 Discussion: 933 Control Plane Instability Conditions are experienced by the DUT 934 after the Startup Conditions have completed. Control Plane 935 Instability Conditions experienced by the DUT include session 936 loss, route withdrawal, and route cost changes. 938 Measurement units: 939 N/A 941 Issues: 942 None 944 See Also: 945 Instability Conditions 946 Data Plane Instability Conditions 947 Management Plane Instability Conditions 948 Security Plane Instability Conditions 950 3.4.3.2 Data Plane Instability Conditions 952 Definition: 953 Data Plane conditions that occur during the Accelerated Stress 954 Benchmark to produce instability and stress the DUT. 956 Discussion: 957 Data Plane Instability Conditions are experienced by the DUT 958 after the Startup Conditions have completed. Data Plane 959 Instability Conditions experienced by the DUT include interface 960 shutdown, link loss, and overloaded links. 962 Measurement units: 963 N/A 965 Issues: 966 None 968 See Also: 969 Instability Conditions 970 Control Plane Instability Conditions 971 Management Plane Instability Conditions 972 Security Plane Instability Conditions 974 3.4.3.3 Management Plane Instability Conditions 975 Definition: 976 Management Plane conditions that occur during the Accelerated 977 Life Benchmark to produce instability and stress the DUT. 979 Discussion: 980 Management Plane Instability Conditions are experienced by the DUT 981 after the Startup Conditions have completed. Management Plane 982 Instability Conditions experienced by the DUT include repeated 983 FTP of large files. 985 Measurement units: 986 N/A 988 Issues: 989 None 991 See Also: 992 Instability Conditions 993 Control Plane Instability Conditions 994 Data Plane Instability Conditions 995 Security Plane Instability Conditions 997 3.4.3.4 Security Plane Instability Conditions 999 Definition: 1000 Security Plane conditions that occur during the Accelerated 1001 Life Benchmark to produce instability and stress the DUT. 1003 Discussion: 1004 Security Plane Instability Conditions are experienced by the DUT 1005 after the Startup Conditions have completed. Security Plane 1006 Instability Conditions experienced by the DUT include session 1007 loss and uninitiated policy changes. 1009 Measurement units: 1010 N/A 1012 Issues: 1013 None 1015 See Also: 1016 Instability Conditions 1017 Control Plane Instability Conditions 1018 Data Plane Instability Conditions 1019 Management Plane Instability Conditions 1021 3.5 Recovery 1023 3.5.1 Recovery Phase 1025 Definition: 1026 The portion of the benchmarking test in which the 1027 Startup Conditions are generated with the DUT, but 1028 the Instability Conditions are no longer offered to 1029 the DUT. 1031 Discussion: 1032 The Recovery Phase is the final Phase of the 1033 benchmarking test following the Startup Phase and 1034 Instability Phase. Startup Conditions must not be 1035 Restarted. 1037 Measurement Units: 1038 None 1040 Issues: 1041 None 1043 See Also: 1044 Startup Conditions 1045 Startup Phase 1046 Instability Conditions 1047 Instability Phase 1049 3.5.2 Benchmarks 1051 3.5.2.1 Recovered Aggregate Forwarding Rate 1053 Definition 1054 Rate of traffic forwarded by the DUT during the Recovery 1055 Phase. 1057 Discussion: 1058 Recovered Aggregate Forwarding Rate is an instantaneous 1059 measurement of the Aggregate Forwarding Rate during the 1060 Recovery Phase. Ideally, each measurement of the Recovered 1061 Aggregate Forwarding Rate equals the Stable Aggregate 1062 Forwarding Rate because the Instability Conditions 1063 do not exist in both the Startup and Recovery Phases. 1065 Measurement Units: 1066 pps 1068 Issues: 1069 None 1071 See Also: 1072 Aggregate Forwarding Rate 1073 Recovery Phase 1074 Recovered Aggregate Forwarding Rate 1075 Startup Phase 1076 Stable Aggregate Forwarding Rate 1078 3.5.2.2 Recovered Latency 1080 Definition: 1081 The average increase in measured packet latency during 1082 the Recovery Phase compared to the Startup Phase. 1084 Discussion: 1085 Latency SHOULD be measured at a fixed interval during the 1086 Recovery Phase. Unstable Latency is the difference 1087 between Stable Latency and the average Latency measured 1088 during the Recovery Phase. It is expected that there 1089 be no increase in average latency from the Startup Phase 1090 to the Recovery Phase. The Recovered Latency cannot be a 1091 negative number. 1093 Measurement units: 1094 seconds 1096 Issues: 1097 None 1099 See Also: 1100 Recovery Phase 1101 Stable Latency 1103 3.5.2.3 Recovery Time 1105 Definition 1106 The amount of time for the Recovered Aggregate Forwarding 1107 Rate to become equal to the Stable Aggregate Forwarding Rate. 1109 Discussion 1110 Recovery Time is measured beginning at the instant the 1111 Instability Phase ends until the Recovered Aggregate 1112 Forwarding Rate equals the Stable Aggregate Forwarding 1113 Rate for a minimum duration of 180 consecutive seconds. 1115 Measurement Units: 1116 seconds 1118 Issues: 1119 None 1121 See Also: 1122 Recovered Aggregate Forwarding Rate 1123 Stable Aggregate Forwarding Rate 1125 3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost 1127 Definition: 1128 Control Plane sessions that are in the down state 1129 but were not intentionally brought down during the 1130 Recovery Phase. 1132 Discussion: 1133 The test equipment is able to control protocol 1134 session state with the DUT. The test equipment 1135 is also to monitor for sessions lost with the 1136 DUT which the test equipment itself did not 1137 intentionally bring down. 1139 Measurement units: 1140 sessions 1141 Issues: 1142 None 1144 See Also: 1145 Controlled Session Loss 1146 Uncontrolled Session Loss 1148 3.5.2.5 Variability Benchmarks 1150 Definition: 1151 The difference between the measured Benchmarks of the 1152 same DUT over multiple iterations. 1154 Discussion: 1155 Ideally, the benchmarks measured should be the same for 1156 multiple iterations with the same DUT. Configuration 1157 Sets Instability conditions SHOULD be held constant for 1158 this benchmark. Whether the DUT can exhibit such predictable 1159 and repeatable behavior is an important benchmark in itself. 1161 Measurement units: 1162 As applicable to each Benchmark. The results are to be 1163 presented in a table format for successive Iterations. 1164 Ideally, the differences should be zero. 1166 Issues: 1167 None 1169 See Also: 1170 Startup Period 1171 Instability Period 1172 Recovery Period 1174 4. Security Considerations 1175 Documents of this type do not directly effect the security of 1176 the Internet or of corporate networks as long as benchmarking 1177 is not performed on devices or systems connected to operating 1178 networks. 1180 5. References 1182 [1] Bradner, S., Editor, "Benchmarking Terminology for Network 1183 Interconnection Devices", RFC 1242, October 1991. 1185 [2] Mandeville, R., "Benchmarking Terminology for LAN Switching 1186 Devices", RFC 2285, June 1998. 1188 [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for 1189 Network Interconnect Devices", RFC 2544, March 1999. 1191 [4] "Core Router Evaluation for Higher Availability", Scott 1192 Poretsky, NANOG 25, June 8, 2002, Toronto, CA. 1194 [5] "Router Stress Testing to Validate Readiness for Network 1195 Deployment", Scott Poretsky, IEEE CQR 2003. 1197 [6] Poretsky, S. and Rao, S., "Methodology for Accelerated 1198 Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-01, 1199 work in progress, October 2004. 1201 6. Author's Address 1203 Scott Poretsky 1204 Quarry Technologies 1205 8 New England Executive Park 1206 Burlington, MA 01803 1207 USA 1208 Phone: + 1 781 395 5090 1209 EMail: sporetsky@quarrytech.com 1211 Shankar Rao 1212 Qwest Communications 1213 Denver, CO 1214 USA 1215 Phone: + 1 303 437 6643 1216 Email: shankar.rao@qwest.com 1218 Appendix 1. White Box Benchmarking Terminology 1220 Minimum Available Memory 1222 Definition: 1223 Minimum DUT Available Memory during the duration of the 1224 Accelerated Stress Test. 1226 Discussion: 1227 It is necessary to monitor DUT memory to measure this 1228 benchmark. 1230 Measurement units: 1231 bytes 1233 Issues: 1234 None 1236 See Also: 1237 Maximum CPU Utilization 1239 Maximum CPU Utilization 1241 Definition: 1242 Maximum DUT CPU utilization during the duration of the 1243 Accelerated Stress Test. 1245 Discussion: 1246 It is necessary to monitor DUT CPU Utilization to measure 1247 this benchmark. 1249 Measurement units: % 1251 Issues: 1252 None 1254 See Also: 1255 Minimum Available Memory 1257 Intellectual Property Statement 1259 The IETF takes no position regarding the validity or scope of any Intel- 1260 lectual Property Rights or other rights that might be claimed to pertain 1261 to the implementation or use of the technology described in this docu- 1262 ment or the extent to which any license under such rights might or might 1263 not be available; nor does it represent that it has made any independent 1264 effort to identify any such rights. Information on the procedures with 1265 respect to rights in RFC documents can be found in BCP 78 and BCP 79. 1267 Copies of IPR disclosures made to the IETF Secretariat and any 1268 assurances of licenses to be made available, or the result of an attempt 1269 made to obtain a general license or permission for the use of such 1270 proprietary rights by implementers or users of this specification can be 1271 obtained from the IETF on-line IPR repository at 1272 http://www.ietf.org/ipr. 1274 The IETF invites any interested party to bring to its attention any 1275 copyrights, patents or patent applications, or other proprietary rights 1276 that may cover technology that may be required to implement this stan- 1277 dard. Please address the information to the IETF at ietf-ipr@ietf.org. 1279 Disclaimer of Warranty 1281 This document and the information contained herein are provided on an 1282 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR 1283 IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 1284 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 1285 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMA- 1286 TION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 1287 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1289 Copyright Statement 1291 Copyright (C) The Internet Society (2004). This document is subject to 1292 the rights, licenses and restrictions contained in BCP 78, and except as 1293 set forth therein, the authors retain all their rights.