Network Working Group S. Poretsky Internet Draft NextPoint Networks Expires: August 2008 Intended Status: Informational Shankar Rao Qwest Communications February 25, 2008 Terminology for Accelerated Stress Benchmarking Intellectual Property Rights (IPR) statement: By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Status of this Memo Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The IETF Trust (2008). ABSTRACT This document provides the Terminology for performing Accelerated Stress Benchmarking of networking devices. The three phases of the Stress Test: Startup, Instability and Recovery are defined along with the benchmarks and configuration terms associated with the each phase. Also defined are the Benchmark Planes fundamental to stress testing configuration, setup and measurement. The terminology is to be used with the companion framework and methodology documents. Table of Contents 1. Introduction ............................................... 3 2. Existing definitions ....................................... 3 3. Term definitions............................................ 4 Poretsky and Rao [Page 1] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.1 General Terms............................................. 4 3.1.1 Benchmark Planes...................................... 4 3.1.2 Configuration Sets.................................... 5 3.1.3 Startup Conditions.................................... 5 3.1.4 Instability Conditions................................ 6 3.1.5 Aggregate Forwarding Rate............................. 6 3.1.6 Discontinued Sessions................................. 7 3.1.7 Uncontrolled Session Loss............................. 7 3.2 Benchmark Planes.......................................... 8 3.2.1 Control Plane......................................... 8 3.2.2 Data Plane............................................ 8 3.2.3 Management Plane...................................... 8 3.2.4 Security Plane........................................ 9 3.3 Startup...................................................10 3.3.1 Startup Phase.........................................10 3.3.2 Benchmarks............................................10 3.3.2.1 Stable Aggregate Forwarding Rate..................10 3.3.2.2 Stable Latency....................................11 3.3.2.3 Stable Session Count..............................11 3.3.3 Control Plane.........................................12 3.3.3.1 Control Plane Configuration Set...................12 3.3.3.2 Control Plane Startup Conditions..................13 3.3.4 Data Plane............................................13 3.3.4.1 Data Plane Configuration Set......................13 3.3.4.2 Traffic Profile...................................14 3.3.5 Management Plane......................................14 3.3.5.1 Management Plane Configuration Set................14 3.3.6 Security Plane........................................15 3.3.6.1 Security Plane Configuration Set..................15 3.3.6.2 Security Plane Startup Conditions.................16 3.4 Instability...............................................17 3.4.1 Instability Phase.....................................17 3.4.2 Benchmarks............................................17 3.4.2.1 Unstable Aggregate Forwarding Rate................17 3.4.2.2 Aggregate Forwarding Rate Degradation.............18 3.4.2.3 Average Aggregate Forwarding Rate Degradation.....18 3.4.2.4 Unstable Latency..................................19 3.4.2.5 Unstable Uncontrolled Sessions Lost...............19 3.4.3 Instability Conditions................................20 3.4.3.1 Control Plane Instability Conditions..............20 3.4.3.2 Data Plane Instability Conditions.................20 3.4.3.3 Management Plane Instability Conditions...........21 3.4.3.4 Security Plane Instability Conditions.............21 3.5 Recovery..................................................22 3.5.1 Recovery Phase........................................22 3.5.2 Benchmarks............................................22 3.5.2.1 Recovered Aggregate Forwarding Rate...............22 3.5.2.2 Recovered Latency.................................23 3.5.2.3 Recovery Time.....................................23 3.5.2.4 Recovered Uncontrolled Sessions Lost..............24 3.5.2.5 Variability Benchmarks............................24 4. IANA Considerations.........................................25 Poretsky and Rao [Page 2] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 5. Security Considerations.....................................25 6. Acknowledgements............................................25 7. References..................................................25 8. Author's Address............................................26 Appendix 1 - White Box Benchmarks..............................26 1. Introduction Routers in an operational network are configured with multiple protocols and security policies while simultaneously forwarding traffic and being managed. To accurately benchmark a router for deployment, it is necessary to test that router under operational conditions by simultaneously configuring and scaling network protocols and security policies, forwarding traffic, and managing the device in a lab environment. It is useful to accelerate these network operational conditions so that the router under test can be benchmarked in a lab environment with a shorter test duration. Testing a router in accelerated network conditions is known as Accelerated Stress Benchmarking. This document provides the Terminology for performing Stress Benchmarking of networking devices. The three phases of the Stress Test: Startup, Instability and Recovery are defined along with the benchmark and configuration terms associated with the each phase. Benchmarks for stress testing are defined using the Aggregate Forwarding Rate and control plane Session Count during each phase of the test. For each plane, the Configuration Set, Startup Conditions, and Instability Conditions are defined. Also defined are the Benchmark Planes fundamental to stress testing configuration, setup and measurement. These are the Control Plane, Data Plane, Management Plane and Security Plane. Multiple benchmarks are measured for each Benchmark Plane during each Phase. Benchmarks can be compared across multiple planes for the same DUT or at the same plane for 2 or more DUTS. Benchmarks of internal DUT characteristics such as memory and CPU utilization (also known as White Box benchmarks) are described in Appendix 1, to allow additional characterization of DUT behavior. The terminology is to be used with the companion methodology document [4]. The sequence of phases, actions, and benchmarks are shown in Table 1. 2. Existing definitions RFC 1242 [1] and RFC 2285 [2] should be consulted before attempting to make use of this document. For the sake of clarity and continuity this RFC adopts the template for definitions set out in Section 2 of RFC 1242. Definitions are indexed and grouped together in sections for ease of reference. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [5]. RFC 2119 defines the use of these key words to help make the intent of standards track documents as clear as possible. While this document uses these keywords, it is not a standards track document. Poretsky and Rao [Page 3] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking Table 1. Phase Sequence and Benchmarks III. Recovery Phase II. Instability Phase I. Startup Phase <-----------------<---<-------------------<----<--------------< Remove Instability Achieve Configuration Apply Startup Conditions Set and Apply Conditions Instability Conditions Benchmark: Benchmark: Benchmark: Recovered Aggregate Unstable Aggregate Stable Aggregate Forwarding Rate Forwarding Rate Forwarding Rate Degraded Aggregate Forwarding Rate Average Degraded Forwarding Rate Recovered Latency Unstable Latency Startup Latency Recovered Uncontrolled Recovered Uncontrolled Stable Session Count Sessions Lost Sessions Lost Recovery Time 3. Term definitions 3.1 General Terms 3.1.1 Benchmark Planes Definition: The features, conditions, and behavior for the Accelerated Stress Benchmarking. Discussion: There are four Benchmark Planes: Control Plane, Data Plane, Management Plane, and Security Plane as shown in Figure 1. Configuration, Startup Conditions, Instability Conditions, and Failure Conditions used for each test are defined for each of these four Benchmark Planes. Measurement units: N/A Issues: None See Also: Control Plane Data Plane Management Plane Security Plane Poretsky and Rao [Page 4] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking ___________ ___________ | Control | | Management| | Plane |___ ___| Plane | | | | | | | ----------- | | ----------- \/ \/ ___________ ___________ | Security | | |<-----------| Plane | | DUT | | | |--->| |<---| ----------- | ----------- | | | | ___________ | | | Data | | |--->| Plane |<---| | | ----------- Figure 1. Router Accelerated Stress Benchmarking Planes 3.1.2 Configuration Sets Definition: The offered load, features, and scaling limits used during the Accelerated Stress Benchmarking. Discussion: There are four Configuration Sets: Control Plane Configuration Set, Data Plane Configuration Set, Management Plane Configuration Set, and Security Plane Configuration Set. The minimum Configuration Set that MUST be used is discussed in the Methodology document [4]. Measurement units: N/A Issues: None See Also: Control Plane Configuration Set Data Plane Configuration Set Management Plane Configuration Set Security Plane Configuration Set 3.1.3 Startup Conditions Definition: Test conditions applied at the start of the Accelerated Stress Benchmark to establish conditions for the remainder of the test. Poretsky and Rao [Page 5] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking Discussion: Startup Conditions may cause stress on the DUT and produce failure. Startup Conditions are defined for the Control Plane and Security Plane. Measurement units: N/A Issues: None See Also: Control Plane Startup Conditions Data Plane Startup Conditions Management Plane Startup Conditions Security Plane Startup Conditions 3.1.4 Instability Conditions Definition: Test conditions applied during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Instability Conditions are applied to the DUT after the Startup Conditions have completed. Instability Conditions occur for the Control Plane, Data Plane, Management Plane, and Security Plane. Measurement units: N/A Issues: None See Also: Control Plane Instability Conditions Data Plane Instability Conditions Management Plane Instability Conditions Security Plane Instability Conditions 3.1.5 Aggregate Forwarding Rate Definition: Sum of forwarding rates for all interfaces on the DUT. Discussion: Each interface of the DUT forwards traffic at some measured rate. The Aggregate Forwarding Rate is the sum of forwarding rates for all interfaces on the DUT. Poretsky and Rao [Page 6] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking Measurement units: pps Issues: None See Also: Startup Phase 3.1.6 Discontinued Sessions Definition: Control Plane sessions that are intentionally brought down during the Stress test. Discussion: Discontinued Sessions is performed during the test in order to stress the DUT by forcing it to tear down Control Plane sessions while handling traffic. It is assumed that the test equipment is able to control protocol session state with the DUT and is therefore able to introduce Discontinued Sessions. Measurement units: None Issues: None See Also: Uncontrolled Session Loss 3.1.7 Uncontrolled Session Loss Definition: Control Plane sessions that are in the down state but were not intentionally brought down during the Stress test. Discussion: The test equipment is able to control protocol session state with the DUT. The test equipment is also to monitor for sessions lost with the DUT which the test equipment itself did not intentionally bring down. Measurement units: N/A Issues: None See Also: Discontinued Sessions Poretsky and Rao [Page 7] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.2 Benchmark Planes 3.2.1 Control Plane Definition: The Description of the control protocols enabled for the Accelerated Stress Benchmarking. Discussion: The Control Plane defines the Configuration, Startup Conditions, and Instability Conditions of the control protocols. Control Plane protocols MAY include routing protocols, multicast protocols, and MPLS protocols. These can be enabled or disabled for a benchmark test. Measurement units: N/A Issues: None See Also: Benchmark Planes Control Plane Configuration Set Control Plane Startup Conditions Control Plane Instability Conditions 3.2.2 Data Plane Definition: The data traffic profile used for the Accelerated Stress Benchmarking. Discussion: The Data Plane defines the Configuration, Startup Conditions, and Instability Conditions of the data traffic. The Data Plane includes the traffic and interface profile. Issues: None Measurement Units: N/A See Also: Benchmark Planes Data Plane Configuration Set Data Plane Startup Conditions Data Plane Instability Conditions 3.2.3 Management Plane Definition: The Management features and tools used for the Accelerated Stress Benchmarking. Poretsky and Rao [Page 8] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking Discussion: A key component of the Accelerated Stress Benchmarking is the Management Plane to assess manageability of the router under stress. The Management Plane defines the Configuration, Startup Conditions, and Instability Conditions of the management protocols and features. The Management Plane includes SNMP, Logging/Debug, Statistics Collection, and management configuration sessions such as telnet, SSH, and serial console. Measurement units: N/A Issues: None See Also: Benchmark Planes Management Plane Configuration Set Management Plane Startup Conditions Management Plane Instability Conditions 3.2.4 Security Plane Definition: The Security features used during the Accelerated Stress Benchmarking. Discussion: The Security Plane defines the Configuration, Startup Conditions, and Instability Conditions of the security features and protocols. The Security Plane includes the ACLs, Firewall, Secure Protocols, and User Login. Measurement units: N/A Issues: None See Also: Benchmark Planes Security Plane Configuration Set Security Plane Startup Conditions Security Plane Instability Conditions Poretsky and Rao [Page 9] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.3 Startup 3.3.1 Startup Phase Definition The step of the benchmarking test in which the Startup Conditions are generated with the DUT. This begins with the attempt to establish the first session and ends when the last Control Plane session is established. Discussion: The Startup Phase is the first Phase of the benchmarking test preceding the Instability Phase and Recovery Phase. It is specified by the Configuration Sets and Startup Conditions for each Benchmark Plane. The Startup Phase ends and Instability Phase MUST begin when the Configuration Sets are achieved with the DUT. The DUT MUST be stable and without failure during the Startup Phase to continue to the Instability Phase. If there is failure during the Startup Phase then the test MUST be restarted with new Startup Conditions. Measurement Units: None Issues: None See Also: Benchmark Plane Configuration Sets Startup Conditions Instability Phase Recovery Phase 3.3.2 Benchmarks 3.3.2.1 Stable Aggregate Forwarding Rate Definition: Sum of forwarding rates for all interfaces on the DUT during the Startup Phase. Discussion: The Stable Aggregate Forwarding Rate is calculated from measurement samples throughout the entire Startup Phase. Stable Aggregate Forwarding Rate is the calculated average of the samples measured during the Startup Phase. It is RECOMMENDED that the sample measurements be made on every DUT interface every 1 second. Measurement units: pps Poretsky and Rao [Page 10] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking Issues: The act of the DUT establishing the Startup Conditions could influence the forwarding rate in certain implementations so that this "baseline" for the remainder of the test is lowered. The alternative is to change the definition of Stable Aggregate Forwarding Rate so that it is measured after Startup Conditions are achieved. The disadvantage of that definition would be that it loses measurement of any impact that establishing Startup Conditions would have on forwarding rate. When comparing the Startup Aggregate Forwarding Rate benchmark of two devices it is preferred to know the impact establishing Startup Conditions has on Forwarding Rate. See Also: Startup Phase Aggregate Forwarding Rate 3.3.2.2 Stable Latency Definition: Average measured latency of traffic forwarded by the DUT during the Startup Phase. Discussion: Stable Latency is the calculated average Latency during the Startup Phase. Measurement units: seconds Issues: None See Also: Startup Phase Stable Aggregate Forwarding Rate 3.3.2.3 Stable Session Count Definition: Total number of control plane sessions/adjacencies established and maintained by the DUT during the Startup Phase and prior to Instability Conditions being initiated. Discussion: This measurement SHOULD be made after the Control Plane Startup Conditions are applied to the DUT. Poretsky and Rao [Page 11] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking Measurement units: sessions Issues: None See Also: Startup Phase 3.3.3 Control Plane 3.3.3.1 Control Plane Configuration Set Definition: The control protocols and scaling values used for the Accelerated Stress Benchmarking. Discussion: Control Plane Configuration Set is represented in Figure 2 and specifies protocol configurations for protocols such as, but not limitied to, Routing, Multicast, SIP and MPLS. Specific protocols can be enabled or disabled for a benchmark test. Measurement units: N/A Issues: None See Also: Data Plane Configuration Set Management Configuration Set Security Configuration Set ____________ ____________ ____________ | Routing | | Multicast | | MPLS | | Protocols |___ | Protocols | __| Protocols | | | | | | | | | ------------ | ------------ | ------------ | | | | | | | \/ | | ___________ | | | | | ____________ +------->| DUT |<------+ | SIP |----------->| | | Sessions | ----------- ------------ Figure 2. Control Plane Configuration Module Poretsky and Rao [Page 12] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.3.3.2 Control Plane Startup Conditions Definition: Control Plane conditions that occur at the start of the Accelerated Stress Benchmarking to establish conditions for the remainder of the test. Discussion: Startup Conditions may cause stress on the DUT and produce failure. Startup Conditions for the Control Plane include session establishment rate, number of sessions established and number of routes learned. Measurement units: N/A Issues: None See Also: Startup Conditions Security Plane Startup Conditions Control Plane Configuration Set 3.3.4 Data Plane 3.3.4.1 Data Plane Configuration Set Definition: The data traffic profile and interfaces that are enabled for the Accelerated Stress Benchmarking. Discussion: Data Plane Configuration Set includes the Traffic Profile and interfaces used for the Accelerated Stress Benchmarking. The interface type(s) and number of interfaces for each interface type MUST be reported. Measurement Units: N/A Issues: None See Also: Traffic Profile Poretsky and Rao [Page 13] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.3.4.2 Traffic Profile Definition The characteristics of the Offered Load to the DUT on each interface for the Accelerated Stress Benchmarking. Discussion The Traffic Profile specifies the number of packet size(s), packet rate, number of flows, and encapsulation on a per-interface basis used for the offered load to the DUT. Measurement Units: Traffic Profile is reported as follows: Parameter Units --------- ------ Packet Size(s) bytes Packet Rate(interface) array of packets per second Aggregate Offered Load pps Number of Flows number of flows Traffic Type array of (RTP, UDP, TCP, other) Encapsulation(flow) array of encapsulation type Mirroring enabled/disabled Issues: None See Also: Data Plane Configuration Set 3.3.5 Management Plane 3.3.5.1 Management Plane Configuration Set Definition: The router management features enabled for the Accelerated Stress Benchmark. Discussion: A key component of the Accelerated Stress Benchmark is the Management Configuration Set to assess manageability of the router under stress. The Management Configuration Set defines the management configuration of the DUT. Features that are part of the Management Configuration Set include access, SNMP, Logging/Debug, and Statistics Collection, and services such as FTP, as shown in Figure 3. These features SHOULD be enabled throughout the Stress test. Poretsky and Rao [Page 14] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking Measurement units: N/A Issues: None See Also: Control Plane Configuration Set Data Plane Configuration Set Security Plane Configuration Set ____________ ____________ | | | Logging/ | | SNMP | __| Debug | | | | | | ------------ | ------------ | | | | \/ | ___________ | | | | | DUT |<---| | | ----------- | | \/ ___________ | Packet | | Statistics| | Collector | | | ----------- Figure 3. Management Plane Configuration Set 3.3.6 Security Plane 3.3.6.1 Security Plane Configuration Set Definition: Security features and scaling enabled for the Accelerated Stress Test. Discussion: The Security Plane Configuration Set includes the configuration and scaling of ACLs, Firewall, IPsec, and User Access, as shown in Figure 4. Tunnels SHOULD be established and policies configured. Instability is introduced by flapping tunnels and configuring and removing policies. Poretsky and Rao [Page 15] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking ____________ ____________ ____________ | | | Secure | | User | |ACL/Firewall| | Protocol | __| Access | | | | | | | | ------------ ------------ | ------------ | | | | | | | \/ | | ___________ | | | | | |------->| DUT |<--------| | | ----------- Figure 4. Security Configuration Module Measurement units: N/A Issues: None See Also: ACL Configuration Set Secure Protocol Configuration Set Password Login Configuration Set 3.3.6.2 Security Plane Startup Conditions Definition: Security Plane conditions that occur at the start of the Accelerated Stress Benchmarking to establish conditions for the remainder of the test. Discussion: Startup Conditions may cause stress on the DUT and produce failure. Startup Conditions for the Security Plane include session establishment rate, number of sessions established and number of policies learned, and number of user access sessions opened. Measurement units: N/A Issues: None See Also: Startup Conditions Data Plane Startup Conditions Management Plane Startup Conditions Security Plane Startup Conditions Poretsky and Rao [Page 16] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.4 Instability 3.4.1 Instability Phase Definition: The step of the benchmarking test in which the Instability Conditions are offered to the DUT. Discussion: The Instability Phase is the middle phase of the benchmarking test following the Startup Phase and preceding the Recovery Phase. The Startup Phase MUST complete without failure to begin the Instability Phase. Measurement Units: None Issues: None See Also: Instability Conditions Startup Phase Recovery Phase 3.4.2 Benchmarks 3.4.2.1 Unstable Aggregate Forwarding Rate Definition: Rate of traffic forwarded by the DUT during the Instability Phase. Discussion: Unstable Aggregated Forwarding Rate is an instantaneous measurement of the Aggregate Forwarding Rate during the Instability Phase. Measurement units: pps Issues: None See Also: Instability Conditions Aggregate Forwarding Rate Poretsky and Rao [Page 17] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.4.2.2 Aggregate Forwarding Rate Degradation Definition: The reduction in Aggregate Forwarding Rate during the Instability Phase. Discussion: The Aggregate Forwarding Rate Degradation is calculated for each measurement of the Unstable Aggregate Forwarding Rate. The Aggregate Forwarding Rate Degradation is calculated by subtracting each measurement of the Unstable Aggregate Forwarding Rate from the Stable Aggregate Forwarding Rate, such that Aggregate Forwarding Rate Degradation= Stable Aggregate Forwarding Rate - Unstable Aggregate Forwarding Rate Ideally, the Aggregate Forwarding Rate Degradation is zero. Measurement Units: pps Issues: None See Also: Instability Phase Unstable Aggregate Forwarding Rate 3.4.2.3 Average Aggregate Forwarding Rate Degradation Definition DUT Benchmark that is the calculated average of the obtained Degraded Forwarding Rates. Discussion: Average Aggregate Forwarding Rate Degradation= (Sum (Stable Aggregate Forwarding Rate) - Sum (Unstable Aggregate Forwarding Rate)) / Number of Samples Measurement Units: pps Issues: None See Also: Aggregate Forwarding Rate Degradation Poretsky and Rao [Page 18] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.4.2.4 Unstable Latency Definition: The average increase in measured packet latency during the Instability Phase compared to the Startup Phase. Discussion: Latency SHOULD be measured at a fixed interval during the Instability Phase. Unstable Latency is the difference between Stable Latency and the average Latency measured during the Instability Phase. It is expected that there be an increase in average latency from the Startup Phase to the Instability phase, but it is possible that the difference be zero. The Unstable Latency cannot be a negative number. Measurement units: seconds Issues: None See Also: Instability Phase Stable Latency 3.4.2.5 Unstable Uncontrolled Sessions Lost Definition: Control Plane sessions that are in the down state but were not intentionally brought down during the Instability Phase. Discussion: The test equipment is able to control protocol session state with the DUT. The test equipment is also to monitor for sessions lost with the DUT which the test equipment itself did not intentionally bring down. Measurement units: sessions Issues: None See Also: Discontinued Sessions Uncontrolled Session Loss Poretsky and Rao [Page 19] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.4.3 Instability Conditions 3.4.3.1 Control Plane Instability Conditions Definition: Control Plane conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Control Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Control Plane Instability Conditions experienced by the DUT include session loss, route withdrawal, and route cost changes. Measurement units: N/A Issues: None See Also: Instability Conditions Data Plane Instability Conditions Management Plane Instability Conditions Security Plane Instability Conditions 3.4.3.2 Data Plane Instability Conditions Definition: Data Plane conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Data Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Data Plane Instability Conditions experienced by the DUT include interface shutdown, link loss, and overloaded links. Measurement units: N/A Issues: None See Also: Instability Conditions Control Plane Instability Conditions Management Plane Instability Conditions Security Plane Instability Conditions Poretsky and Rao [Page 20] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.4.3.3 Management Plane Instability Conditions Definition: Management Plane conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Management Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Management Plane Instability Conditions experienced by the DUT include repeated FTP of large files. Measurement units: N/A Issues: None See Also: Instability Conditions Control Plane Instability Conditions Data Plane Instability Conditions Security Plane Instability Conditions 3.4.3.4 Security Plane Instability Conditions Definition: Security Plane conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Security Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Security Plane Instability Conditions experienced by the DUT include session loss and uninitiated policy changes. Measurement units: N/A Issues: None See Also: Instability Conditions Control Plane Instability Conditions Data Plane Instability Conditions Management Plane Instability Conditions Poretsky and Rao [Page 21] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.5 Recovery 3.5.1 Recovery Phase Definition: The step of the benchmarking test in which the Startup Conditions are generated with the DUT, but the Instability Conditions are no longer offered to the DUT. Discussion: The Recovery Phase is the final Phase of the benchmarking test following the Startup Phase and Instability Phase. Startup Conditions MUST NOT be Restarted. Measurement Units: None Issues: None See Also: Startup Conditions Startup Phase Instability Conditions Instability Phase 3.5.2 Benchmarks 3.5.2.1 Recovered Aggregate Forwarding Rate Definition Rate of traffic forwarded by the DUT during the Recovery Phase. Discussion: Recovered Aggregate Forwarding Rate is an instantaneous measurement of the Aggregate Forwarding Rate during the Recovery Phase. Ideally, each measurement of the Recovered Aggregate Forwarding Rate equals the Stable Aggregate Forwarding Rate because the Instability Conditions do not exist in both the Startup and Recovery Phases. Measurement Units: pps Issues: None See Also: Aggregate Forwarding Rate Recovery Phase Recovered Aggregate Forwarding Rate Startup Phase Stable Aggregate Forwarding Rate Poretsky and Rao [Page 22] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.5.2.2 Recovered Latency Definition: The average increase in measured packet latency during the Recovery Phase compared to the Startup Phase. Discussion: Latency SHOULD be measured at a fixed interval during the Recovery Phase. Unstable Latency is the difference between Stable Latency and the average Latency measured during the Recovery Phase. It is expected that there be no increase in average latency from the Startup Phase to the Recovery Phase. The Recovered Latency cannot be a negative number. Measurement units: seconds Issues: None See Also: Recovery Phase Stable Latency 3.5.2.3 Recovery Time Definition The amount of time for the Recovered Aggregate Forwarding Rate to become equal to the Stable Aggregate Forwarding Rate. Discussion Recovery Time is measured beginning at the instant the Instability Phase ends until the Recovered Aggregate Forwarding Rate equals the Stable Aggregate Forwarding Rate for a minimum duration of 180 consecutive seconds. Measurement Units: milliseconds Issues: None See Also: Recovered Aggregate Forwarding Rate Stable Aggregate Forwarding Rate Poretsky and Rao [Page 23] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost Definition: Control Plane sessions that are in the down state but were not intentionally brought down during the Recovery Phase. Discussion: The test equipment is able to control protocol session state with the DUT. The test equipment is also to monitor for sessions lost with the DUT which the test equipment itself did not intentionally bring down. Measurement units: sessions Issues: None See Also: Discontinued Sessions Uncontrolled Session Loss 3.5.2.5 Variability Benchmarks Definition: The difference between the measured Benchmarks of the same DUT over multiple iterations. Discussion: Ideally, the measured benchmarks should be the same for multiple iterations with the same DUT. Configuration Sets and Instability Conditions MUST be held constant for this benchmark. Whether the DUT can exhibit such predictable and repeatable behavior is an important benchmark in itself. Measurement units: As applicable to each Benchmark. The results are to be presented in a table format for successive Iterations. Ideally, the differences should be zero. Issues: None See Also: Startup Period Instability Period Recovery Period Poretsky and Rao [Page 24] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 4. IANA Considerations This document requires no IANA considerations. 5. Security Considerations Documents of this type do not directly affect the security of the Internet or of corporate networks as long as benchmarking is not performed on devices or systems connected to operating networks. 6. Acknowledgements The authors would like to thank the BMWG and particularly Al Morton, Jay Karthik, and George Jones for their contributions. 7. References 7.1 Normative References [1] Bradner, S., Editor, "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, March 1991. [2] Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, June 1998. [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999. [4] Poretsky, S. and Rao, S., "Methodology Guidelines for Accelerated Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-09, work in progress, February 2008. [5] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, March 1997. 7.2 Informative References [RFC3871] Jones, G., "Operational Security Requirements for Large Internet Service Provider (ISP) IP Network Infrastructure.", IETF RFC 3871 , September 2004. [NANOG25] Poretsky, S., "Core Router Evaluation for Higher Availability", NANOG 25, June 8, 2002, Toronto, CA. [IEEECQR] Poretsky, S., "Router Stress Testing to Validate Readiness for Network Deployment", IEEE CQR 2003. Poretsky and Rao [Page 25] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking 8. Author's Address Scott Poretsky NextPoint Networks 3 Federal Street Billerica, MA 01821 USA Phone: + 1 508 439 9008 EMail: sporetsky@nextpointnetworks.com Shankar Rao 1801 California Street 8th Floor Qwest Communications Denver, CO 80202 USA Phone: + 1 303 437 6643 Email: shankar.rao@qwest.com Appendix 1. White Box Benchmarking Terminology Minimum Available Memory Definition: Minimum DUT Available Memory during the duration of the Accelerated Stress Benchmark. Discussion: This benchmark enables the assessment of resources in the DUT. It is necessary to monitor DUT memory to measure this benchmark. Measurement units: bytes Issues: None See Also: Maximum CPU Utilization Maximum CPU Utilization Definition: Maximum DUT CPU utilization during the duration of the Accelerated Stress Benchmark. Discussion: This benchmark enables the assessment of resources in the DUT. It is necessary to monitor DUT CPU Utilization to measure this benchmark. Measurement units: % Issues: None See Also: Minimum Available Memory Poretsky and Rao [Page 26] INTERNET-DRAFT Terminology for Accelerated February 2008 Stress Benchmarking Full Copyright Statement Copyright (C) The IETF Trust (2008). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf- ipr@ietf.org. Acknowledgement Funding for the RFC Editor function is currently provided by the Internet Society. Poretsky and Rao [Page 27]