Network Working Group INTERNET-DRAFT Expires in: April 2004 Scott Poretsky Quarry Technologies Shankar Rao Qwest Communications Ray Piatt Cable and Wireless October 2003 Framework for Accelerated Stress Benchmarking Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This document provides a framework for executing the Accelerated Stress Benchmarking. It is intended that this framework be applied with the Terminology document when using the Methodology document. Discussion to specify and apply Startup Conditions, Configuration Sets, and Instability Conditions is provided with examples. The motivation and benefits of stress testing are also discussed. Poretsky, Rao, Piatt [Page 1] INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003 Table of Contents 1. Introduction ................................................ 2 2. Existing definitions ........................................ 2 3. Motivation for Accelerated Stress Benchmarking............... 2 4. Application of Configuration Sets............................ 3 5. Application of Startup Conditions............................ 5 6. Application of Instability Conditions........................ 6 7. Service Provider Application of Accelerated Stress Testing... 6 8. Security Considerations...................................... 6 9. References................................................... 6 10. Author's Address............................................ 6 11. Full Copyright Statement.................................... 7 1. Introduction This document provides the motivation and framework to perform Accelerated Stress Benchmarking. The terminology to be used for Accelerated Stress Benchmarking is defined in [1] and the methodology is provided in [2]. This document discusses how to apply the terminology to the benchmarking for producing effective reproducible tests. Configuration Sets, Startup Conditions, and Instability Conditions are defined [1] and examples are provided in this document. 2. Existing definitions RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching Devices" should be consulted before attempting to make use of this document. For the sake of clarity and continuity this RFC adopts the template for definitions set out in Section 2 of RFC 1242. Definitions are indexed and grouped together in sections for ease of reference. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 3. Motivation for Accelerated Stress Benchmarking Router testing benchmarks have consistently been made in a monolithic fashion in which a single protocol or behavior is measured in an isolated environment. It is important to know the limits for a router/switch's (hereby referred to as Router) behavior for each protocol, however this does not produce a reliable benchmark of the router's behavior in a deployed network. Routers in an operational network are simultaneously configured with multiple protocols and security policies while forwarding traffic and being managed. To accurately benchmark a router for deployment it is necessary to test that router in operational conditions by simultaneously configuring the network protocols and security policies, sourcing traffic, and managing the router. The benchmarks are externally Poretsky, Rao, Piatt [Page 2] INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003 observable as control plane or data plane errors at the DUT. It is helpful to accelerate these network operational conditions so that the DUT can be benchmarked with faster test duration. Accelerated Stress Testing of routers provides the following benefits: 1. Evaluation of multiple protocols enabled simultaneously as configured in deployed networks 2. Evaluation of System and Software Stability 3. Evaluation of Manageability under stressful conditions 4. Identification of Software Coding bugs such as: a. Memory Leaks b. Suboptimal CPU Utilization c. Coding Logic These benefits produce three advantages for netowrk operations: 1. Increased stability of routers and protocols 2. Hardened routers to DoS attacks 3. Verified manageability under stress 4. Application of Configuration Sets Configuration Sets are defined in [1] for the Control Plane, Data Plane, Management Plane, and Security Plane. It is intended that the user of these documents specify the specific parameters of the Configuration Set based upon applicability to the device and network. Example Configuration Sets are provided below. 4.1 Control Plane Configuration Sets Key protocols for the Control Plane are Routing Protocols, MPLS Signaling Protocols, and Multicast Protocols. Examples for these are as follow: Example Routing Protocol Configuration Set- PARAMETER UNITS BGP Enabled/Disabled Number of EBGP Peers Peers Number of IBGP Peers Peers Number of BGP Route Instances Routes Number of BGP Installed Routes Routes MBGP Enabled/Disabled Number of MBGP Route Instances Routes Number of MBGP Installed Routes Routes ISIS Enabled/Disabled ISIS-TE Enabled/Disabled Number of ISIS Adjacencies Adjacencies Number of ISIS Routes Routes Number of Nodes per Area Nodes Poretsky, Rao, Piatt [Page 3] INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003 OSPF Enabled/Disabled OSPF-TE Enabled/Disabled Number of OSPF Adjacencies Adjacencies Number of OSPF Routes Routes Number of Nodes per Area Nodes Example MPLS Protocol Configuration Set- PARAMETER UNITS MPLS-TE Number of Ingress Tunnels Tunnels Number of Mid-Point Tunnels Tunnels Number of Egress Tunnels Tunnels LDP Number of Sessions Sessions Number of FECs FECs Example Multicast Protocol Configuration Set- PARAMETER UNITS PIM-SM Enabled/Disabled RP Enabled/Disabled Number of Multicast Groups Groups MSDP Enabled/Disabled 4.2 Data Plane Configuration Set The Data Plane Configuration Set includes the Traffic Profile as defined in [1]. The example configuration set is as follows: Example Data Plane Configuration Set- PARAMETER UNITS Traffic Forwarding Enabled/Disabled Aggregate Offered Load bps (or pps) Number of Ingress Interfaces number Number of Ingress Interfaces number TRAFFIC PROFILE Packet Size(s) bytes Packet Rate(interface) array of packets per second Number of Flows number Encapsulation(flow) array of encapsulation type Poretsky, Rao, Piatt [Page 4] INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003 4.3 Management Configuration Set The Management Configuration Set can include SNMP, Logging, Debug, Telnet, FTP, SSH, and RADIUS parameters. An example is as follows: Example Management Configuration Set- PARAMETER UNITS SNMP GET Rate SNMP Gets/minute Logging Enabled/Disabled Protocol Debug Enabled/Disabled Telnet Rate Sessions/Hour FTP Rate Sessions/Hour Concurrent Telnet Sessions Sessions Concurrent FTP Session Sessions Packet Statistics Collector Enabled/Disabled Statistics Sampling Rate X:1 packets 4.4 Security Configuration Set The Security Configuration Set can include Packet Filters and Access session restrictions. An example is as follows: Example Security Configuration Set - PARAMETER UNITS Packet Filters Enabled/Disabled Number of Filters For-Me number Number of Filter Rules For-Me number Number of Traffic Filters number Number of Traffic Filter Rules number SSH Enabled/Disabled Number of simultaneous SSH sessions number RADIUS Enabled/Disabled TACACS Enabled/Disabled 5. Application of Startup Conditions Startup conditions are the conditions that must be met in order for Accelerated Stress benchmarking to begin. Startup Conditions specify how a particular Configuration Set should be obtained. Example Startup Conditions include: PARAMETER UNITS Routing Session Establishment Rate sessions per minute User Config Session Establishment Rate number per minute Security Session Establishment Rate number per minute Routes Learned Rate routes per minute MPLS LSPs Establishment Rate number per minute Poretsky, Rao, Piatt [Page 5] INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003 6. Application of Instability Conditions Test conditions that occur during the Accelerated Stress Test should simulate instability in an operational network. Repeating these conditions should stress the SUT. Example Instability Conditions are provided below: PARAMETER UNITS Interface Shutdown Cycling Rate interfaces per minute BGP Session Loss Rate sessions per minute BGP Route Flap Rate routes per minutes IGP Route Flap Rate routes per minutes Route Convergence from Better Next-Hop routes per minutes LSP Reroute Rate LSP per minute Overloaded Links number Amount Links Overloaded % of bandwidth FTP Rate Mb/minute IPsec Session Loss sessions per minute Filter Policy Changes policies per minute SSH Session Re-Start SSH sessions per minute 7. Accelerated Stress Benchmarking Application The Accelerated Stress Benchmarking test can be applied in service provider test environments to benchmark DUTs under stress in an environment that is reflective of an operational network. A particular Configuration Set is defined and the DUT is benchmarked using this and the Instability Conditions. Varying ConfigurationSets and/or Instability Conditions for repeated iterations can provide a characterization of the DUT to help determine future network deployments. 8. Security Considerations Documents of this type do not directly effect the security of the Internet or of corporate networks as long as benchmarking is not performed on devices or systems connected to operating networks. 9. References [1] Poretsky, Scott, Rao, Shankar, and Piatt, Ray, "Terminology for Accelerated Stress Benchmarking, draft-ietf-bmwg-acc-bench-term- 01, work in progress, October 2003. 10. Author's Address Scott Poretsky Quarry Technologies 8 New England Executive Park Burlington, MA 01803 USA Phone: + 1 781 395 5090 EMail: sporetsky@quarrytech.com Poretsky, Rao, Piatt [Page 6] INTERNET-DRAFT Framework for Accelerated Stress Benchmarking October 2003 Shankar Rao 950 17th Street Suite 1900 Qwest Communications Denver, CO 80210 USA Phone: + 1 303 437 6643 Email: srao@qwest.net Ray Piatt Cable and Wireless 11700 Plaza America Drive Reston, VA 20190 USA Phone: + 1 703 292 2113 Email: rpiatt@cw.net 11. Full Copyright Statement Copyright (C) The Internet Society (1998). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Poretsky, Rao, Piatt [Page 7]