idnits 2.17.1 draft-ietf-bmwg-acc-bench-framework-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard == It seems as if not all pages are separated by form feeds - found 0 form feeds but 7 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. (A line matching the expected section header was found, but with an unexpected indentation: ' 1. Introduction' ) ** The document seems to lack a Security Considerations section. (A line matching the expected section header was found, but with an unexpected indentation: ' 8. Security Considerations' ) ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack an Authors' Addresses Section. ** There are 133 instances of lines with control characters in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == Line 98 has weird spacing: '...ecurity polic...' == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (April 2004) is 7309 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? '1' on line 283 looks like a reference -- Missing reference section? '2' on line 65 looks like a reference Summary: 6 errors (**), 0 flaws (~~), 5 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group 2 INTERNET-DRAFT 3 Expires in: April 2004 4 Scott Poretsky 5 Quarry Technologies 7 Shankar Rao 8 Qwest Communications 10 Ray Piatt 11 Cable and Wireless 13 October 2003 15 Framework for Accelerated Stress Benchmarking 16 18 Status of this Memo 20 This document is an Internet-Draft and is in full conformance with 21 all provisions of Section 10 of RFC2026. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF), its areas, and its working groups. Note that 25 other groups may also distribute working documents as Internet- 26 Drafts. 28 Internet-Drafts are draft documents valid for a maximum of six 29 months and may be updated, replaced, or obsoleted by other 30 documents at any time. It is inappropriate to use Internet-Drafts 31 as reference material or to cite them other than as "work in 32 progress." 34 The list of current Internet-Drafts can be accessed at 35 http://www.ietf.org/ietf/1id-abstracts.txt 37 The list of Internet-Draft Shadow Directories can be accessed at 38 http://www.ietf.org/shadow.html. 40 Abstract 41 This document provides a framework for executing the Accelerated 42 Stress Benchmarking. It is intended that this framework be applied 43 with the Terminology document when using the Methodology document. 44 Discussion to specify and apply Startup Conditions, Configuration 45 Sets, and Instability Conditions is provided with examples. The 46 motivation and benefits of stress testing are also discussed. 48 Table of Contents 49 1. Introduction ................................................ 2 50 2. Existing definitions ........................................ 2 51 3. Motivation for Accelerated Stress Benchmarking............... 2 52 4. Application of Configuration Sets............................ 3 53 5. Application of Startup Conditions............................ 5 54 6. Application of Instability Conditions........................ 6 55 7. Service Provider Application of Accelerated Stress Testing... 6 56 8. Security Considerations...................................... 6 57 9. References................................................... 6 58 10. Author's Address............................................ 6 59 11. Full Copyright Statement.................................... 7 61 1. Introduction 62 This document provides the motivation and framework to perform 63 Accelerated Stress Benchmarking. The terminology to be used 64 for Accelerated Stress Benchmarking is defined in [1] and the 65 methodology is provided in [2]. This document discusses how to 66 apply the terminology to the benchmarking for producing effective 67 reproducible tests. Configuration Sets, Startup Conditions, and 68 Instability Conditions are defined [1] and examples are provided 69 in this document. 71 2. Existing definitions 72 RFC 1242 "Benchmarking Terminology for Network Interconnect 73 Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching 74 Devices" should be consulted before attempting to make use of this 75 document. 77 For the sake of clarity and continuity this RFC adopts the template 78 for definitions set out in Section 2 of RFC 1242. Definitions are 79 indexed and grouped together in sections for ease of reference. 80 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 81 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 82 this document are to be interpreted as described in RFC 2119. 84 3. Motivation for Accelerated Stress Benchmarking 86 Router testing benchmarks have consistently been made in a 87 monolithic fashion in which a single protocol or behavior is 88 measured in an isolated environment. It is important to know the 89 limits for a router/switch's (hereby referred to as Router) behavior 90 for each protocol, however this does not produce a reliable benchmark 91 of the router's behavior in a deployed network. Routers in an 92 operational network are simultaneously configured with multiple 93 protocols and security policies while forwarding traffic and being 94 managed. 96 To accurately benchmark a router for deployment it is necessary to 97 test that router in operational conditions by simultaneously 98 configuring the network protocols and security policies, sourcing 99 traffic, and managing the router. The benchmarks are externally 100 observable as control plane or data plane errors at the DUT. It is 101 helpful to accelerate these network operational conditions so that 102 the DUT can be benchmarked with faster test duration. Accelerated 103 Stress Testing of routers provides the following benefits: 105 1. Evaluation of multiple protocols enabled simultaneously as 106 configured in deployed networks 107 2. Evaluation of System and Software Stability 108 3. Evaluation of Manageability under stressful conditions 109 4. Identification of Software Coding bugs such as: 110 a. Memory Leaks 111 b. Suboptimal CPU Utilization 112 c. Coding Logic 114 These benefits produce three advantages for netowrk operations: 115 1. Increased stability of routers and protocols 116 2. Hardened routers to DoS attacks 117 3. Verified manageability under stress 119 4. Application of Configuration Sets 121 Configuration Sets are defined in [1] for the Control Plane, Data 122 Plane, Management Plane, and Security Plane. It is intended that 123 the user of these documents specify the specific parameters of the 124 Configuration Set based upon applicability to the device and 125 network. Example Configuration Sets are provided below. 127 4.1 Control Plane Configuration Sets 129 Key protocols for the Control Plane are Routing Protocols, MPLS 130 Signaling Protocols, and Multicast Protocols. Examples for these 131 are as follow: 133 Example Routing Protocol Configuration Set- 135 PARAMETER UNITS 136 BGP Enabled/Disabled 137 Number of EBGP Peers Peers 138 Number of IBGP Peers Peers 139 Number of BGP Route Instances Routes 140 Number of BGP Installed Routes Routes 142 MBGP Enabled/Disabled 143 Number of MBGP Route Instances Routes 144 Number of MBGP Installed Routes Routes 146 ISIS Enabled/Disabled 147 ISIS-TE Enabled/Disabled 148 Number of ISIS Adjacencies Adjacencies 149 Number of ISIS Routes Routes 150 Number of Nodes per Area Nodes 151 OSPF Enabled/Disabled 152 OSPF-TE Enabled/Disabled 153 Number of OSPF Adjacencies Adjacencies 154 Number of OSPF Routes Routes 155 Number of Nodes per Area Nodes 157 Example MPLS Protocol Configuration Set- 159 PARAMETER UNITS 160 MPLS-TE 161 Number of Ingress Tunnels Tunnels 162 Number of Mid-Point Tunnels Tunnels 163 Number of Egress Tunnels Tunnels 165 LDP 166 Number of Sessions Sessions 167 Number of FECs FECs 169 Example Multicast Protocol Configuration Set- 171 PARAMETER UNITS 172 PIM-SM Enabled/Disabled 173 RP Enabled/Disabled 174 Number of Multicast Groups Groups 175 MSDP Enabled/Disabled 177 4.2 Data Plane Configuration Set 179 The Data Plane Configuration Set includes the Traffic Profile 180 as defined in [1]. The example configuration set is as follows: 182 Example Data Plane Configuration Set- 184 PARAMETER UNITS 185 Traffic Forwarding Enabled/Disabled 186 Aggregate Offered Load bps (or pps) 187 Number of Ingress Interfaces number 188 Number of Ingress Interfaces number 190 TRAFFIC PROFILE 191 Packet Size(s) bytes 192 Packet Rate(interface) array of packets per second 193 Number of Flows number 194 Encapsulation(flow) array of encapsulation type 195 4.3 Management Configuration Set 197 The Management Configuration Set can include SNMP, Logging, Debug, 198 Telnet, FTP, SSH, and RADIUS parameters. An example is as follows: 200 Example Management Configuration Set- 202 PARAMETER UNITS 203 SNMP GET Rate SNMP Gets/minute 204 Logging Enabled/Disabled 205 Protocol Debug Enabled/Disabled 206 Telnet Rate Sessions/Hour 207 FTP Rate Sessions/Hour 208 Concurrent Telnet Sessions Sessions 209 Concurrent FTP Session Sessions 210 Packet Statistics Collector Enabled/Disabled 211 Statistics Sampling Rate X:1 packets 213 4.4 Security Configuration Set 215 The Security Configuration Set can include Packet Filters and 216 Access session restrictions. An example is as follows: 218 Example Security Configuration Set - 220 PARAMETER UNITS 221 Packet Filters Enabled/Disabled 222 Number of Filters For-Me number 223 Number of Filter Rules For-Me number 224 Number of Traffic Filters number 225 Number of Traffic Filter Rules number 226 SSH Enabled/Disabled 227 Number of simultaneous SSH sessions number 228 RADIUS Enabled/Disabled 229 TACACS Enabled/Disabled 231 5. Application of Startup Conditions 233 Startup conditions are the conditions that must be met in order 234 for Accelerated Stress benchmarking to begin. Startup Conditions 235 specify how a particular Configuration Set should be obtained. 236 Example Startup Conditions include: 238 PARAMETER UNITS 239 Routing Session Establishment Rate sessions per minute 240 User Config Session Establishment Rate number per minute 241 Security Session Establishment Rate number per minute 242 Routes Learned Rate routes per minute 243 MPLS LSPs Establishment Rate number per minute 244 6. Application of Instability Conditions 246 Test conditions that occur during the Accelerated Stress Test 247 should simulate instability in an operational network. 248 Repeating these conditions should stress the SUT. Example 249 Instability Conditions are provided below: 251 PARAMETER UNITS 252 Interface Shutdown Cycling Rate interfaces per minute 253 BGP Session Loss Rate sessions per minute 254 BGP Route Flap Rate routes per minutes 255 IGP Route Flap Rate routes per minutes 256 Route Convergence from Better Next-Hop routes per minutes 257 LSP Reroute Rate LSP per minute 258 Overloaded Links number 259 Amount Links Overloaded % of bandwidth 260 FTP Rate Mb/minute 261 IPsec Session Loss sessions per minute 262 Filter Policy Changes policies per minute 263 SSH Session Re-Start SSH sessions per minute 265 7. Accelerated Stress Benchmarking Application 266 The Accelerated Stress Benchmarking test can be applied in 267 service provider test environments to benchmark DUTs under 268 stress in an environment that is reflective of an operational 269 network. A particular Configuration Set is defined and the 270 DUT is benchmarked using this and the Instability Conditions. 271 Varying ConfigurationSets and/or Instability Conditions for 272 repeated iterations can provide a characterization of the DUT 273 to help determine future network deployments. 275 8. Security Considerations 276 Documents of this type do not directly effect the security of 277 the Internet or of corporate networks as long as benchmarking 278 is not performed on devices or systems connected to operating 279 networks. 281 9. References 283 [1] Poretsky, Scott, Rao, Shankar, and Piatt, Ray, "Terminology for 284 Accelerated Stress Benchmarking, draft-ietf-bmwg-acc-bench-term- 285 01, work in progress, October 2003. 287 10. Author's Address 289 Scott Poretsky 290 Quarry Technologies 291 8 New England Executive Park 292 Burlington, MA 01803 293 USA 294 Phone: + 1 781 395 5090 295 EMail: sporetsky@quarrytech.com 297 Shankar Rao 298 950 17th Street 299 Suite 1900 300 Qwest Communications 301 Denver, CO 80210 302 USA 303 Phone: + 1 303 437 6643 304 Email: srao@qwest.net 306 Ray Piatt 307 Cable and Wireless 308 11700 Plaza America Drive 309 Reston, VA 20190 310 USA 311 Phone: + 1 703 292 2113 312 Email: rpiatt@cw.net 314 11. Full Copyright Statement 316 Copyright (C) The Internet Society (1998). All Rights 317 Reserved. 319 This document and translations of it may be copied and 320 furnished to others, and derivative works that comment on or 321 otherwise explain it or assist in its implementation may be 322 prepared, copied, published and distributed, in whole or in 323 part, without restriction of any kind, provided that the above 324 copyright notice and this paragraph are included on all such 325 copies and derivative works. However, this document itself may 326 not be modified in any way, such as by removing the copyright 327 notice or references to the Internet Society or other Internet 328 organizations, except as needed for the purpose of developing 329 Internet standards in which case the procedures for copyrights 330 defined in the Internet Standards process must be followed, or 331 as required to translate it into languages other than English. 333 The limited permissions granted above are perpetual and will 334 not be revoked by the Internet Society or its successors or 335 assigns. This document and the information contained herein is 336 provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE 337 INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, 338 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY 339 THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY 340 RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS 341 FOR A PARTICULAR PURPOSE.