Network Working Group INTERNET-DRAFT Expires in: April 2004 Scott Poretsky Quarry Technologies Shankar Rao Qwest Communications Ray Piatt Cable and Wireless October 2003 Terminology for Accelerated Stress Benchmarking Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. ABSTRACT This terminology document provides the terms to be used for benchmarking a router or switch under Accelerated Stress conditions. Terminology is provided for performing this benchmark using the Contol Plane, Data Plane, Management Plane, and Security Plane of the Device Under Test. Terms are defined for Configuration, Startup Conditions, Instability Conditions, Failure Conditions, and Benchmarks. Poretsky, Rao, Piatt [Page 1] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 Table of Contents 1. Introduction ............................................... 3 2. Existing definitions ....................................... 3 3. Term definitions............................................ 3 3.1 Benchmark Planes.......................................... 3 3.1.1 Control Plane........................................... 4 3.1.2 Data Plane.............................................. 5 3.1.3 Management Plane........................................ 5 3.1.4 Security Plane.......................................... 6 3.2 Configuration Sets........................................ 6 3.2.1 Control Plane Configuration Set......................... 7 3.2.2 Data Plane Configuration Set............................ 7 3.2.2.1 Traffic Profile....................................... 8 3.2.3 Management Plane Configuration Set...................... 8 3.2.4 Security Plane Configuration Set........................ 9 3.3 Startup Conditions........................................10 3.3.1 Control Plane Startup Conditions........................10 3.3.2 Security Plane Startup Conditions.......................11 3.4 Instability Conditions....................................11 3.4.1 Control Plane Instability Conditions....................12 3.4.2 Data Plane Instability Conditions.......................12 3.4.3 Management Plane Instability Conditions.................13 3.4.4 Security Plane Instability Conditions...................13 3.5 Failure Conditions........................................14 3.5.1 Unexpected Packet Loss..................................14 3.5.2 Unexpected Session Loss.................................14 3.5.3 Misrouted Packets.......................................15 3.5.4 Access Denial...........................................15 3.5.5 Errored Management Value................................16 3.5.6 Security Plane Failure..................................16 3.6 Benchmarks................................................17 3.6.1 Success Threshold.......................................17 3.6.2 Accelerated-Life Test Duration..........................17 4. Security Considerations.....................................18 5. References..................................................18 6. Author's Address............................................18 7. Full Copyright Statement....................................19 Appendix 1 - White Box Benchmarks..............................20 Poretsky, Rao, Piatt [Page 2] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 1. Introduction Routers in an operational network are simultaneously configured with multiple protocols and security policies while forwarding traffic and being managed. To accurately benchmark a router for deployment it is necessary to test that router in operational conditions by simultaneously configuring and scaling network protocols and security policies, forwarding traffic, and managing the device. It is helpful to accelerate these network operational conditions so that the router under test can be benchmarked with faster test duration. Testing a router in accelerated network conditions is known as Accelerated Stress Testing. This document defines the Benchmark Planes used for the Accelerated Life Benchmarking. These are the Control Plane, Data Plane, Management Plane, and Security Plane. For each plane, the Configuration Set, Startup Conditions, and Instability Conditions. Terms are also defined for observing Failure Conditions and reporting Benchmarks. White Box benchmarks are provided in Appendix 1 for additional DUT behavior measurements. 2. Existing definitions RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching Devices" should be consulted before attempting to make use of this document. For the sake of clarity and continuity this RFC adopts the template for definitions set out in Section 2 of RFC 1242. Definitions are indexed and grouped together in sections for ease of reference. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 3. Term definitions 3.1 Benchmark Planes Definition: The features, conditions, and behavior for the Accelerated Stress Benchmarking. Discussion: There are four Benchmark Planes: Control Plane, Data Plane, Management Plane, and Security Plane as shown in Figure 1. The Benchmark Planes define the Configuration, Startup Conditions, Instability Conditions, and Failure Conditions used for the test. Poretsky, Rao, Piatt [Page 3] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 ___________ ___________ | Control | | Management| | Plane |___ ___| Plane | | | | | | | ----------- | | ----------- \/ \/ ___________ ___________ | Security | | |<-----------| Plane | | DUT | | | |--->| |<---| ----------- | ----------- | | | | ___________ | | | Data | | |--->| Plane |<---| | | ----------- Figure 1. Router Accelerated Stress Benchmarking Planes Measurement units: N/A Issues: None See Also: Control Plane Data Plane Management Plane Security Plane 3.1.1 Control Plane Definition: The Description of the control protocols enabled for the Accelerated Stress Benchmarking. Discussion: The Control Plane defines the Configuration, Startup Conditions, Instability Conditions, and Failure Conditions of the control protocols used for the test. Control Plane protocols may include routing protocols, multicast protocols, and MPLS protocols. These can be enabled or disabled for a benchmark test. Measurement units: N/A Issues: None See Also: Benchmark Planes Control Plane Configuration Set Control Plane Startup Conditions Control Plane Instability Conditions Control Plane Failure Conditions Poretsky, Rao, Piatt [Page 4] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 3.1.2 Data Plane Definition: The data traffic profile used for the Accelerated Stress Benchmarking. Discussion: The Data Plane defines the Configuration, Startup Conditions, Instability Conditions, and Failure Conditions of the data traffic used for the test. The Data Plane includes the traffic and interface profile. Measurement Units: N/A See Also: Benchmark Planes Data Plane Configuration Set Data Plane Startup Conditions Data Plane Instability Conditions Data Plane Failure Conditions 3.1.3 Management Plane Definition: The Management features and tools used for the Accelerated Stress Benchmarking. Discussion: A key component of the Accelerated Stress Benchmarking is the Management Plane to assess manageability of the router under stress. The Management Plane defines the Configuration, Startup Conditions, Instability Conditions, and Failure Conditions of the management protocols and features used for the test. The Management Plane includes SNMP, Logging/Debug, and Statistics Collection. Measurement units: N/A Issues: None See Also: Benchmark Planes Management Plane Configuration Set Management Plane Startup Conditions Management Plane Instability Conditions Management Plane Failure Conditions Poretsky, Rao, Piatt [Page 5] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 3.1.4 Security Plane Definition: The Security features used during the Accelerated Stress Benchmarking. Discussion: The Control Plane defines the Configuration, Startup Conditions, Instability Conditions, and Failure Conditions of the security features and protocols used for the test. The Security Plane includes the ACLs, Firewall, Secure Protocols, and User Login. Measurement units: N/A Issues: None See Also: Benchmark Planes Security Plane Configuration Set Security Plane Startup Conditions Security Plane Instability Conditions Security Plane Failure Conditions 3.2 Configuration Sets Definition: The features and scaling limits used during the Accelerated Stress Benchmarking. Discussion: There are four Configuration Sets: Control Plane Configuration Set, Data Plane Configuration Set, Management Plane Configuration Set, and Security Plane Configuration Set. Measurement units: N/A Issues: None See Also: Control Plane Configuration Set Data Plane Configuration Set Management Plane Configuration Set Security Plane Configuration Set Poretsky, Rao, Piatt [Page 6] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 3.2.1 Control Plane Configuration Set Definition: The routing protocols and scaling values used for the Accelerated Life Benchmarking. Discussion: Control Plane Configuration Set is shown in Figure 2 and specifies the Routing Protocols, Multicast, and MPLS configuration. Specific protocols can be enabled or disabled for a benchmark test. Measurement units: None Issues: None ____________ ____________ ____________ | Routing | | Multicast | | MPLS | | Protocols |___ | Protocols | __| Protocols | | | | | | | | | ------------ | ------------ | ------------ | | | | | | | \/ | | ___________ | | | | | |------->| DUT |<------| | | ----------- Figure 2. Control Plane Configuration Module See Also: Data Plane Configuration Set Management Configuration Set Security Configuration Set 3.2.2 Data Plane Configuration Set Definition: The data traffic profile enabled for the Accelerated Stress Benchmarking. Discussion: Data Plane Configuration Set includes the Traffic Profile and interfaces used for the Accelerated Stress Benchmarking. Measurement Units: N/A Issues: None See Also: Traffic Profile Poretsky, Rao, Piatt [Page 7] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 3.2.2.1 Traffic Profile Definition The characteristics of the Offered Load to the DUT used for the Accelerated Stress Benchmarking. Discussion The Traffic Profile specifies the number of packet size(s), packet rate per interface, number of flows, and encapsulation used for the offered load to the DUT. Measurement Units: Traffic Profile is reported as follows: Parameter Units --------- ------ Packet Size(s) bytes Packet Rate(interface) array of packets per second Number of Flows number Encapsulation(flow) array of encapsulation type Issues: None See Also: Data Plane Configuration Set 3.2.3 Management Plane Configuration Set Definition: The router management features enabled for the Accelerated Stress Test. Discussion: A key component of the Accelerated Stress Test is the Management Configuration Set to assess manageability of the router under stress. The Management Configuration Set defines the management configuration of the DUT. Features that are part of the Management Configuration Set include SNMP, Logging/Debug, and Statistics Collection, and services such as FTP, as shown in Figure 3. Measurement units: N/A Issues: None See Also: Control Plane Configuration Set Data Plane Configuration Set Security Plane Configuration Set Poretsky, Rao, Piatt [Page 8] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 ____________ ____________ | | | Logging/ | | SNMP | __| Debug | | | | | | ------------ | ------------ | | | | \/ | ___________ | | | | | DUT |<---| | | ----------- | | \/ ___________ | Packet | | Statistics| | Collector | | | ----------- Figure 3. Management Plane Configuration Set 3.2.4 Security Plane Configuration Set Definition: Security features and scaling enabled for the Accelerated Stress Test. Discussion: The Security Plane Configuration Set includes the configuration and scaling of ACLs, Firewall, IPsec, and User Access, as shown in Figure 4. ____________ ____________ ____________ | | | Secure | | User | |ACL/Firewall| | Protocol | __| Access | | | | | | | | ------------ ------------ | ------------ | | | | | | | \/ | | ___________ | | | | | |-------> | DUT |<--------| | | ----------- Figure 4. Security Configuration Module Poretsky, Rao, Piatt [Page 9] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 Measurement units: N/A Issues: None See Also: ACL Configuration Set Secure Protocol Configuration Set Password Login Configuration Set 3.3 Startup Conditions Definition: Test conditions that occur at the start of the Accelerated Life Benchmark to establish conditions for the remainder of the test. Discussion: Startup Conditions may cause stress on the DUT and produce failure. Startup Conditions are defined for the Control Plane and Security Plane. Measurement units: N/A Issues: None See Also: Control Plane Startup Conditions Data Plane Startup Conditions Management Plane Startup Conditions Security Plane Startup Conditions 3.3.1 Control Plane Startup Conditions Definition: Control Plane conditions that occur at the start of the Accelerated Stress Benchmarking to establish conditions for the remainder of the test. Discussion: Startup Conditions may cause stress on the DUT and produce failure. Startup Conditions for the Control Plane include session establishment rate, number of sessions established and number of routes learned. Measurement units: N/A Poretsky, Rao, Piatt [Page 10] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 Issues: None See Also: Startup Conditions Security Plane Startup Conditions Control Plane Configuration Set 3.3.2 Security Plane Startup Conditions Definition: Security Plane conditions that occur at the start of the Accelerated Stress Benchmarking to establish conditions for the remainder of the test. Discussion: Startup Conditions may cause stress on the DUT and produce failure. Startup Conditions for the Security Plane include session establishment rate, number of sessions established and number of policies learned, and number of user access sessions opened. Measurement units: N/A Issues: None See Also: Startup Conditions Data Plane Startup Conditions Management Plane Startup Conditions Security Plane Startup Conditions 3.4 Instability Conditions Definition: Test conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Instability Conditions occur for the Control Plane, Data Plane, Management Plane, and Security Plane. Measurement units: N/A Issues: None Poretsky, Rao, Piatt [Page 11] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 See Also: Control Plane Instability Conditions Data Plane Instability Conditions Management Plane Instability Conditions Security Plane Instability Conditions 3.4.1 Control Plane Instability Conditions Definition: Control Plane conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Control Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Control Plane Instability Conditions experienced by the DUT include session loss, route withdrawal, and route cost changes. Measurement units: N/A Issues: None See Also: Instability Conditions Data Plane Instability Conditions Management Plane Instability Conditions Security Plane Instability Conditions 3.4.2 Data Plane Instability Conditions Definition: Data Plane conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Data Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Data Plane Instability Conditions experienced by the DUT include interface shutdown, link loss, and overloaded links. Measurement units: N/A Issues: None See Also: Instability Conditions Control Plane Instability Conditions Management Plane Instability Conditions Security Plane Instability Conditions Poretsky, Rao, Piatt [Page 12] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 3.4.3 Management Plane Instability Conditions Definition: Management Plane conditions that occur during the Accelerated Life Benchmark to produce instability and stress the DUT. Discussion: Management Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Management Plane Instability Conditions experienced by the DUT include repeated FTP of large files. Measurement units: N/A Issues: None See Also: Instability Conditions Control Plane Instability Conditions Data Plane Instability Conditions Security Plane Instability Conditions 3.4.4 Security Plane Instability Conditions Definition: Security Plane conditions that occur during the Accelerated Life Benchmark to produce instability and stress the DUT. Discussion: Security Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Security Plane Instability Conditions experienced by the DUT include session loss and policy changes. Measurement units: N/A Issues: None See Also: Instability Conditions Control Plane Instability Conditions Data Plane Instability Conditions Management Plane Instability Conditions Poretsky, Rao, Piatt [Page 13] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 3.5 Failure Conditions 3.5.1 Unexpected Session Loss Definition: Session Loss due to Control Plane or Data Plane failure during the Accelerated Stress Benchmarking. Discussion: Session loss will be observed due to Convergence Events produced by the Instability Conditions. This is not considered a failure during the Accelerated Stress Benchmarking. It is expected behavior. Unexpected packet loss occurs due to Data Plane or Control Plane failure. Measurement units: sessions Issues: None See Also: Unexpected Packet Loss Misrouted Packets Access Denial Errored Management Values Security Plane Failure 3.5.2 Unexpected Packet Loss Definition: Packet Loss due to Control Plane or Data Plane failure during the Accelerated Stress Benchmarking. Discussion: Packet loss will be observed due to Convergence Events produced by the Instability Conditions. This is not considered a failure during the Accelerated Stress Benchmarking. It is expected behavior. Unexpected packet loss occurs due to Data Plane or Control Plane failure. Measurement units: packets Issues: None See Also: Unexpected Session Loss Misrouted Packets Access Denial Errored Management Values Security Plane Failure Poretsky, Rao, Piatt [Page 14] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 3.5.3 Misrouted Packets Definition: Packets routed to the incorrect next-hop or LSP due to Control Plane failure during the Accelerated Stress Benchmarking. Discussion: Packets will be rerouted to new next-hops and LSPs due to Convergence Events produced by the Instability Conditions. This is not considered a failure during the Accelerated Stress Benchmarking. It is expected behavior. Unexpected Misrouted packets occur due to Control Plane failure. Measurement units: packets Issues: None See Also: Unexpected Packet Loss Unexpected Session Loss Access Denial Errored Management Values Security Plane Failure 3.5.4 Access Denial Definition: Inability to management the DUT due to the inability to gain remote access to the DUT. Discussion: Handling of Instability Conditions may cause a Control Plane failure of the DUT externally observable as the inability to remotely manage the DUT. Whether using SNMP, Telnet, SSH, or FTP, failure to manage the device constitues a failure. Measurement units: Failed session Issues: None See Also: Unexpected Packet Loss Unexpected Session Loss Misrouted Packets Errored Management Value Security Plane Failure Poretsky, Rao, Piatt [Page 15] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 3.5.5 Errored Management Value Definition: Incorrect SNMP object value or statistic during the Accelerated Life Benchmark. Discussion: Management of a router relies upon accurate reporting of object values and statistics. The Accelerated Stress Benchmarkinging validates that the DUT is able to report correct values when experiencing Instability Conditions. Measurement units: Name of incorrect object or statistic Issues: None See Also: Unexpected Packet Loss Unexpected Session Loss Misrouted Packets Access Denial Security Plane Failure Success Threshold 3.5.6 Security Plane Failure Definition: Violation of Security Policy for packets directed to the DUT or routed through the DUT. Discussion: Security Policy is configured to permit and deny access to packets destined for the DUT and packets to be routed through the DUT. A violation of this security policy is when either a packet to be denied is permitted or a packet to be permitted is denied. Measurement units: packets Issues: None See Also: Unexpected Packet Loss Unexpected Session Loss Misrouted Packets Access Denial Errored Management Value Poretsky, Rao, Piatt [Page 16] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 3.6 Benchmarks 3.6.1 Success Threshold Definition: The intended number of hours for the DUT to operate without failure during the Accelerated Stress Benchmarking. Discussion: Success Threshold is a user determined value of time. It is necessary to define the Success Threshold because it is possible for the DUT to experience zero failures for given Configuration Sets, Startup Conditions, and Instability Conditions Measurement units: Hours Issues: None See Also: Accelerated-Life Test Duration 3.6.2 Accelerated-Life Test Duration Definition: The run-time for the Accelerated Stress Test without the DUT exhibiting an error in the control plane, data plane, management plane, or security plane. Discussion: For a successful test, the Test Duration will equal the Success Threshold. In the event of any failure, the Test Duration is less than the Success Threshold. Reporting of the Test Duration MUST inlcude the Failure Condition from those defined in this document. Measurement units: Hours Issues: None See Also: Success Threshold Poretsky, Rao, Piatt [Page 17] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 4. Security Considerations Documents of this type do not directly effect the security of the Internet or of corporate networks as long as benchmarking is not performed on devices or systems connected to operating networks. 5. References [1] Bradner, S., Editor, "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, July 1991. [2] Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, June 1998. [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999. [4] "Core Router Evaluation for Higher Availability", Scott Poretsky, NANOG 25, June 8, 2002, Toronto, CA. [5] "Router Stress Testing to Validate Readiness for Network Deployment", Scott Poretsky, IEEE CQR 2003. 6. Author's Address Scott Poretsky Quarry Technologies 8 New England Executive Park Burlington, MA 01803 USA Phone: + 1 781 395 5090 EMail: sporetsky@quarrytech.com Shankar Rao 950 17th Street Suite 1900 Qwest Communications Denver, CO 80210 USA Phone: + 1 303 437 6643 Email: srao@qwest.net Ray Piatt Cable and Wireless 11700 Plaza America Drive Reston, VA 20190 USA Phone: + 1 703 292 2113 Email: rpiatt@cw.net Poretsky, Rao, Piatt [Page 18] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 7. Full Copyright Statement Copyright (C) The Internet Society (1998). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Poretsky, Rao, Piatt [Page 19] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003 Appendix 1. White Box Benchmarking Terminology Appendix 1.1 Minimum Available Memory Definition: Minimum DUT Available Memory during the duration of the Accelerated Stress Test. Discussion: It is necessary to monitor DUT memory to measure this benchmark. Measurement units: bytes Issues: None See Also: Maximum CPU Utilization Appendix 1.2 Maximum CPU Utilization Definition: Maximum DUT CPU utilization during the duration of the Accelerated Stress Test. Discussion: It is necessary to monitor DUT CPU Utilization to measure this benchmark. Measurement units: % Issues: None See Also: Minimum Available Memory Poretsky, Rao, Piatt [Page 20]