idnits 2.17.1 draft-ietf-bmwg-acc-bench-meth-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 20. -- Found old boilerplate from RFC 3978, Section 5.5 on line 498. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 509. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 516. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 522. ** The document claims conformance with section 10 of RFC 2026, but uses some RFC 3978/3979 boilerplate. As RFC 3978/3979 replaces section 10 of RFC 2026, you should not claim conformance with it if you have changed to using RFC 3978/3979 boilerplate. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard == It seems as if not all pages are separated by form feeds - found 0 form feeds but 11 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. (A line matching the expected section header was found, but with an unexpected indentation: ' 1. Introduction' ) ** The document seems to lack a Security Considerations section. (A line matching the expected section header was found, but with an unexpected indentation: ' 5. Security Considerations' ) ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack an Authors' Addresses Section. ** There are 6 instances of too long lines in the document, the longest one being 5 characters in excess of 72. ** There are 188 instances of lines with control characters in the document. ** The abstract seems to contain references ([2], [RFC3871], [3], [CONVMETH], [4], [IEEECQR], [NANOG25], [5], [6], [1]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (January 2006) is 6675 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? '4' on line 439 looks like a reference -- Missing reference section? '6' on line 447 looks like a reference -- Missing reference section? '1' on line 430 looks like a reference -- Missing reference section? '2' on line 433 looks like a reference -- Missing reference section? '3' on line 436 looks like a reference -- Missing reference section? '5' on line 443 looks like a reference -- Missing reference section? 'RFC3871' on line 452 looks like a reference -- Missing reference section? 'NANOG25' on line 456 looks like a reference -- Missing reference section? 'IEEECQR' on line 459 looks like a reference -- Missing reference section? 'CONVMETH' on line 462 looks like a reference Summary: 11 errors (**), 0 flaws (~~), 3 warnings (==), 17 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group 2 INTERNET-DRAFT 3 Expires in: January 2006 4 Scott Poretsky 5 Reef Point Systems 7 Shankar Rao 8 Qwest Communications 10 July 2005 12 Methodology Guidelines for 13 Accelerated Stress Benchmarking 14 16 Intellectual Property Rights (IPR) statement: 17 By submitting this Internet-Draft, each author represents that any 18 applicable patent or other IPR claims of which he or she is aware 19 have been or will be disclosed, and any of which he or she becomes 20 aware will be disclosed, in accordance with Section 6 of BCP 79. 22 Status of this Memo 24 This document is an Internet-Draft and is in full conformance with 25 all provisions of Section 10 of RFC2026. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF), its areas, and its working groups. Note that 29 other groups may also distribute working documents as Internet- 30 Drafts. 32 Internet-Drafts are draft documents valid for a maximum of six 33 months and may be updated, replaced, or obsoleted by other 34 documents at any time. It is inappropriate to use Internet-Drafts 35 as reference material or to cite them other than as "work in 36 progress." 38 The list of current Internet-Drafts can be accessed at 39 http://www.ietf.org/ietf/1id-abstracts.txt 40 The list of Internet-Draft Shadow Directories can be accessed at 41 http://www.ietf.org/shadow.html. 43 ABSTRACT 44 Routers in an operational network are simultaneously configured with 45 multiple protocols and security policies while forwarding traffic and 46 being managed. To accurately benchmark a router for deployment it is 47 necessary that the router be tested in these simultaneous 48 operational conditions, which is known as Stress Testing. This 49 document provides the Methodology Guidelines for performing Stress 50 Benchmarking of networking devices. Descriptions of Test Topology, 51 Benchmarks and Reporting Format are provided in addition to procedures 52 for conducting various test cases. The methodology is to be used with 53 the companion terminology document [4]. These guidelines can be used 54 as the basis for additional methodology documents that benchmark 55 specific network technologies under accelerated stress. 57 Table of Contents 58 1. Introduction ............................................... 2 59 2. Existing definitions ....................................... 3 60 3. Test Setup.................................................. 3 61 3.1 Test Topologies............................................ 3 62 3.2 Test Considerations........................................ 3 63 3.3 Reporting Format........................................... 4 64 3.3.1 Configuration Sets....................................... 5 65 3.3.2 Startup Conditions....................................... 6 66 3.3.3 Instability Conditions................................... 6 67 3.3.4 Benchmarks............................................... 7 68 4. Example Test Case Procedure................................. 7 69 5. Security Considerations..................................... 9 70 6. Normative References........................................ 9 71 7. Informative References......................................10 72 8. Author's Address............................................10 74 1. Introduction 75 Router testing benchmarks have consistently been made in a monolithic 76 fashion wherein a single protocol or behavior is measured in an 77 isolated environment. It is important to know the limits for a 78 networking device's behavior for each protocol in isolation, however 79 this does not produce a reliable benchmark of the device's behavior 80 in an operational network. 82 Routers in an operational network are simultaneously configured with 83 multiple protocols and security policies while forwarding traffic 84 and being managed. To accurately benchmark a router for deployment 85 it is necessary to test that router in operational conditions by 86 simultaneously configuring and scaling network protocols and security 87 policies, forwarding traffic, and managing the device. It is helpful 88 to accelerate these network operational conditions with Instability 89 Conditions [4] so that the networking devices are stress tested. 91 This document provides the Methodology for performing Stress 92 Benchmarking of networking devices. Descriptions of Test Topology, 93 Benchmarks and Reporting Format are provided in addition to 94 procedures for conducting various test cases. The methodology is 95 to be used with the companion terminology document [4]. 97 Stress Testing of networking devices provides the following benefits: 98 1. Evaluation of multiple protocols enabled simultaneously as 99 configured in deployed networks 100 2. Evaluation of System and Software Stability 101 3. Evaluation of Manageability under stressful conditions 102 4. Identification of Buffer Overflow conditions 103 5. Identification of Software Coding bugs such as: 104 a. Memory Leaks 105 b. Suboptimal CPU Utilization 106 c. Coding Logic 107 These benefits produce significant advantages for network operations: 108 1. Increased stability of routers and protocols 109 2. Hardened routers to DoS attacks 110 3. Verified manageability under stress 111 4. Planning router resources for growth and scale 113 2. Existing definitions 114 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 115 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 116 document are to be interpreted as described in BCP 14, RFC 2119 117 [6]. RFC 2119 defines the use of these key words to help make the 118 intent of standards track documents as clear as possible. While this 119 document uses these keywords, this document is not a standards track 120 document. 122 Terms related to Accelerated Stress Benchmarking are defined in [4]. 124 3. Test Setup 125 3.1 Test Topologies 126 Figure 1 shows the physical configuration to be used for the 127 methodologies provided in this document. The number of interfaces 128 between the tester and DUT will scale depending upon the number of 129 control protocol sessions and traffic forwarding interfaces. A 130 separate device may be required to externally manage the device in 131 the case that the test equipment does not support such 132 functionality. Figure 2 shows the logical configuration for the 133 stress test methodologies. Each plane may be emulated by single or 134 multiple test equipment. 136 3.2 Test Considerations 137 The Accelerated Stress Benchmarking test can be applied in 138 service provider test environments to benchmark DUTs under 139 stress in an environment that is reflective of an operational 140 network. A particular Configuration Set is defined and the 141 DUT is benchmarked using this configuration set and the 142 Instability Conditions. Varying Configuration Sets and/or 143 Instability Conditions applied in an iterative fashion can 144 provide an accurate characterization of the DUT 145 to help determine future network deployments. 147 ___________ 148 | DUT | 149 ___|Management | 150 | | | 151 | ----------- 152 \/ 153 ___________ 154 | | 155 | DUT | 156 |--->| |<---| 157 xN | ----------- | xN 158 interfaces | | interfaces 159 | ___________ | 160 | | | | 161 |--->| Tester |<---| 162 | | 163 ----------- 165 Figure 1. Physical Configuration 167 ___________ ___________ 168 | Control | | Management| 169 | Plane |___ ___| Plane | 170 | | | | | | 171 ----------- | | ----------- 172 \/ \/ ___________ 173 ___________ | Security | 174 | |<-----------| Plane | 175 | DUT | | | 176 |--->| |<---| ----------- 177 | ----------- | 178 | | 179 | ___________ | 180 | | Data | | 181 |--->| Plane |<---| 182 | | 183 ----------- 185 Figure 2. Logical Configuration 187 3.3 Reporting Format 189 Each methodology requires reporting of information for test 190 repeatability when benchmarking the same or different devices. 191 The information that are the Configuration Sets, Instability 192 Conditions, and Benchmarks, as defined in [4]. Example 193 reporting formats for each are provided below. 195 3.3.1 Configuration Sets 197 Configuration Sets may include and are not limited to the following 198 examples. 200 Example Routing Protocol Configuration Set- 201 PARAMETER UNITS 202 BGP Enabled/Disabled 203 Number of EBGP Peers Peers 204 Number of IBGP Peers Peers 205 Number of BGP Route Instances Routes 206 Number of BGP Installed Routes Routes 207 MBGP Enabled/Disabled 208 Number of MBGP Route Instances Routes 209 Number of MBGP Installed Routes Routes 210 IGP Enabled/Disabled 211 IGP-TE Enabled/Disabled 212 Number of IGP Adjacencies Adjacencies 213 Number of IGP Routes Routes 214 Number of Nodes per Area Nodes 216 Example MPLS Protocol Configuration Set- 217 PARAMETER UNITS 218 MPLS-TE Enabled/Disabled 219 Number of Ingress Tunnels Tunnels 220 Number of Mid-Point Tunnels Tunnels 221 Number of Egress Tunnels Tunnels 222 LDP Enabled/Disabled 223 Number of Sessions Sessions 224 Number of FECs FECs 226 Example Multicast Protocol Configuration Set- 227 PARAMETER UNITS 228 PIM-SM Enabled/Disabled 229 RP Enabled/Disabled 230 Number of Multicast Groups Groups 231 MSDP Enabled/Disabled 233 Example Data Plane Configuration Set- 234 PARAMETER UNITS 235 Traffic Forwarding Enabled/Disabled 236 Aggregate Offered Load bps (or pps) 237 Number of Ingress Interfaces number 238 Number of Egress Interfaces number 240 TRAFFIC PROFILE 241 Packet Size(s) bytes 242 Offered Load (interface) array of bps 243 Number of Flows number 244 Encapsulation(flow) array of encapsulation type 245 Management Configuration Set- 246 PARAMETER UNITS 247 SNMP GET Rate SNMP Gets/minute 248 Logging Enabled/Disabled 249 Protocol Debug Enabled/Disabled 250 Telnet Rate Sessions/Hour 251 FTP Rate Sessions/Hour 252 Concurrent Telnet Sessions Sessions 253 Concurrent FTP Session Sessions 254 Packet Statistics Collector Enabled/Disabled 255 Statistics Sampling Rate X:1 packets 257 Security Configuration Set - 258 PARAMETER UNITS 259 Packet Filters Enabled/Disabled 260 Number of Filters For-Me number 261 Number of Filter Rules For-Me number 262 Number of Traffic Filters number 263 Number of Traffic Filter Rules number 264 IPsec tunnels number 265 SSH Enabled/Disabled 266 Number of simultaneous SSH sessions number 267 RADIUS Enabled/Disabled 268 TACACS Enabled/Disabled 270 3.3.2 Startup Conditions 271 Startup Conditions may include and are not limited to the following 272 examples: 273 PARAMETER UNITS 274 EBGP peering sessions negotiated Total EBGP Sessions 275 IBGP peering sessions negotiated Total IBGP Sessions 276 BGP routes learned rate BGP Routes per Second 277 ISIS adjacencies established Total ISIS Adjacencies 278 ISIS routes learned rate ISIS Routes per Second 279 IPsec tunnels negotiated Total IPsec Tunnels 280 IPsec tunnel establishment rate IPsec tunnels per second 282 3.3.3 Instability Conditions 283 Instability Conditions may include and are not limited to the 284 following examples: 285 PARAMETER UNITS 286 Interface Shutdown Cycling Rate interfaces per minute 287 BGP Session Flap Rate sessions per minute 288 BGP Route Flap Rate routes per minutes 289 IGP Route Flap Rate routes per minutes 290 LSP Reroute Rate LSP per minute 291 Overloaded Links number 292 Amount Links Overloaded % of bandwidth 293 FTP Rate Mb/minute 294 IPsec Tunnel Flap Rate tunnels per minute 295 Filter Policy Changes policies per hour 296 SSH Session Restart SSH sessions per hour 297 Telnet Session Restart Telnet session per hour 299 3.3.4 Benchmarks 301 Benchmarks are as defined in [1] and listed as follow: 303 PARAMETER UNITS PHASE 304 Stable Aggregate Forwarding Rate pps Startup 305 Stable Latency seconds Startup 306 Stable Session Count sessions Startup 307 Unstable Aggregate Forwarding Rate pps Instability 308 Degraded Aggregate Forwarding Rate pps Instability 309 Ave. Degraded Aggregate Forwarding Rate pps Instability 310 Unstable Latency seconds Instability 311 Unstable Uncontrolled Sessions Lost sessions Instability 312 Recovered Aggregate Forwarding Rate pps Recovery 313 Recovered Latency seconds Recovery 314 Recovery Time seconds Recovery 315 Recovered Uncontrolled Sessions Lost sessions Recovery 317 4. Example Test Case Procedure 318 1. Report Configuration Set 320 BGP Enabled 321 10 EBGP Peers 322 30 IBGP Peers 323 500K BGP Route Instances 324 160K BGP FIB Routes 326 ISIS Enabled 327 ISIS-TE Disabled 328 30 ISIS Adjacencies 329 10K ISIS Level-1 Routes 330 250 ISIS Nodes per Area 332 MPLS Disabled 333 IP Multicast Disabled 335 IPsec Enabled 336 10K IPsec tunnels 337 640 Firewall Policies 338 100 Firewall Rules per Policy 340 Traffic Forwarding Enabled 341 Aggregate Offered Load 10Gbps 342 30 Ingress Interfaces 343 30 Egress Interfaces 344 Packet Size(s) = 64, 128, 256, 512, 1024, 1280, 1518 bytes 345 Forwarding Rate[1..30] = 1Gbps 346 10000 Flows 347 Encapsulation[1..5000] = IPv4 348 Encapsulation[5001.10000] = IPsec 350 Logging Enabled 351 Protocol Debug Disabled 352 SNMP Enabled 353 SSH Enabled 354 20 Concurrent SSH Sessions 355 FTP Enabled 356 RADIUS Enabled 357 TACACS Disabled 358 Packet Statistics Collector Enabled 360 2. Begin Startup Conditions with the DUT 362 10 EBGP peering sessions negotiated 363 30 EBGP peering sessions negotiated 364 1K BGP routes learned per second 365 30 ISIS Adjacencies 366 1K ISIS routes learned per second 367 10K IPsec tunnels negotiated 369 3. Establish Configuration Sets with the DUT 371 4. Report Stability Benchmarks as follow: 373 Stable Aggregate Forwarding Rate 374 Stable Latency 375 Stable Session Count 377 It is RECOMMENDED that the benchmarks be measured and 378 recorded at one-second intervals. 380 5. Apply Instability Conditions 382 Interface Shutdown Cycling Rate = 1 interface every 5 minutes 383 BGP Session Flap Rate = 1 session every 10 minutes 384 BGP Route Flap Rate = 100 routes per minute 385 ISIS Route Flap Rate = 100 routes per minute 386 IPsec Tunnel Flap Rate = 1 tunnel per minute 387 Overloaded Links = 5 of 30 388 Amount Links Overloaded = 20% 389 SNMP GETs = 1 per sec 390 SSH Restart Rate = 10 sessions per hour 391 FTP Restart Rate = 10 transfers per hour 392 FTP Transfer Rate = 100 Mbps 393 Statistics Sampling Rate = 1:1 packets 395 6. Apply Instability Condition specific to test case. 397 7. Report Instability Benchmarks as follow: 398 Unstable Aggregate Forwarding Rate 399 Degraded Aggregate Forwarding Rate 400 Ave. Degraded Aggregate Forwarding Rate 401 Unstable Latency 402 Unstable Uncontrolled Sessions Lost 404 It is RECOMMENDED that the benchmarks be measured and 405 recorded at one-second intervals. 407 8. Stop applying all Instability Conditions 409 9. Report Recovery Benchmarks as follow: 411 Recovered Aggregate Forwarding Rate 412 Recovered Latency 413 Recovery Time 414 Recovered Uncontrolled Sessions Lost 416 It is RECOMMENDED that the benchmarks be measured and 417 recorded at one-second intervals. 419 10. Optional - Change Configuration Set and/or Instability 420 Conditions for next iteration 422 5. Security Considerations 423 Documents of this type do not directly affect the security of 424 the Internet or of corporate networks as long as benchmarking 425 is not performed on devices or systems connected to operating 426 networks. 428 6. Normative References 430 [1] Bradner, S., Editor, "Benchmarking Terminology for Network 431 Interconnection Devices", RFC 1242, July 1991. 433 [2] Mandeville, R., "Benchmarking Terminology for LAN Switching 434 Devices", RFC 2285, June 1998. 436 [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for 437 Network Interconnect Devices", RFC 2544, March 1999. 439 [4] Poretsky, S. and Rao, S., "Terminology for Accelerated 440 Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-05, 441 work in progress, July 2005. 443 [5] Poretsky, S., "Benchmarking Terminology for IGP Data Plane 444 Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-06, 445 work in progress, July 2005. 447 [6] Bradner, S., "Key words for use in RFCs to Indicate Requirement 448 Levels", RFC 2119, March 1997. 450 7. Informative References 452 [RFC3871] RFC 3871 "Operational Security Requirements for Large 453 Internet Service Provider (ISP) IP Network Infrastructure. 454 G. Jones, Ed.. IETF, September 2004. 456 [NANOG25] "Core Router Evaluation for Higher Availability", Scott 457 Poretsky, NANOG 25, June 8, 2002, Toronto, CA. 459 [IEEECQR] "Router Stress Testing to Validate Readiness for Network 460 Deployment", Scott Poretsky, IEEE CQR 2003. 462 [CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data Plane 463 Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-05, 464 work in progress, July 2005. 466 8. Author's Address 468 Scott Poretsky 469 Reef Point Systems 470 8 New England Executive Park 471 Burlington, MA 01803 472 USA 473 Phone: + 1 781 395 5090 474 EMail: sporetsky@reefpoint.com 476 Shankar Rao 477 1801 California Street 478 8th Floor 479 Qwest Communications 480 Denver, CO 80202 481 USA 482 Phone: + 1 303 437 6643 483 Email: shankar.rao@qwest.com 484 Full Copyright Statement 486 Copyright (C) The Internet Society (2005). 488 This document is subject to the rights, licenses and restrictions 489 contained in BCP 78, and except as set forth therein, the authors 490 retain all their rights. 492 This document and the information contained herein are provided on an 493 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 494 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 495 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 496 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 497 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 498 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 500 Intellectual Property 502 The IETF takes no position regarding the validity or scope of any 503 Intellectual Property Rights or other rights that might be claimed to 504 pertain to the implementation or use of the technology described in 505 this document or the extent to which any license under such rights 506 might or might not be available; nor does it represent that it has 507 made any independent effort to identify any such rights. Information 508 on the procedures with respect to rights in RFC documents can be 509 found in BCP 78 and BCP 79. 511 Copies of IPR disclosures made to the IETF Secretariat and any 512 assurances of licenses to be made available, or the result of an 513 attempt made to obtain a general license or permission for the use of 514 such proprietary rights by implementers or users of this 515 specification can be obtained from the IETF on-line IPR repository at 516 http://www.ietf.org/ipr. 518 The IETF invites any interested party to bring to its attention any 519 copyrights, patents or patent applications, or other proprietary 520 rights that may cover technology that may be required to implement 521 this standard. Please address the information to the IETF at ietf- 522 ipr@ietf.org. 524 Acknowledgement 526 Funding for the RFC Editor function is currently provided by the 527 Internet Society.