idnits 2.17.1 draft-ietf-bmwg-acc-bench-meth-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3667, Section 5.1 on line 26. -- Found old boilerplate from RFC 3978, Section 5.1 on line 20. -- Found old boilerplate from RFC 3978, Section 5.5 on line 524. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 535. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 542. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 548. ** Found boilerplate matching RFC 3978, Section 5.1 (on line 20), which is fine, but *also* found old RFC 3667, Section 5.1 text on line 26. ** Found boilerplate matching RFC 3978, Section 5.4, paragraph 1 (on line 512), which is fine, but *also* found old RFC 2026, Section 10.4C, paragraph 1 text on line 44. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 61 lines == It seems as if not all pages are separated by form feeds - found 0 form feeds but 11 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 1 character in excess of 72. ** There are 25 instances of lines with control characters in the document. ** The abstract seems to contain references ([4]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == Line 329 has weird spacing: '...ns Lost ses...' -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (April 2006) is 6579 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: '2' is defined on line 453, but no explicit reference was found in the text == Unused Reference: '3' is defined on line 456, but no explicit reference was found in the text == Unused Reference: '5' is defined on line 465, but no explicit reference was found in the text == Unused Reference: 'RFC3871' is defined on line 475, but no explicit reference was found in the text == Unused Reference: 'NANOG25' is defined on line 479, but no explicit reference was found in the text == Unused Reference: 'IEEECQR' is defined on line 482, but no explicit reference was found in the text == Unused Reference: 'CONVMETH' is defined on line 485, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 1242 (ref. '1') ** Downref: Normative reference to an Informational RFC: RFC 2285 (ref. '2') ** Downref: Normative reference to an Informational RFC: RFC 2544 (ref. '3') == Outdated reference: A later version (-13) exists of draft-ietf-bmwg-acc-bench-term-07 ** Downref: Normative reference to an Informational draft: draft-ietf-bmwg-acc-bench-term (ref. '4') == Outdated reference: A later version (-23) exists of draft-ietf-bmwg-igp-dataplane-conv-term-08 ** Downref: Normative reference to an Informational draft: draft-ietf-bmwg-igp-dataplane-conv-term (ref. '5') == Outdated reference: A later version (-23) exists of draft-ietf-bmwg-igp-dataplane-conv-meth-08 Summary: 13 errors (**), 0 flaws (~~), 15 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group 2 INTERNET-DRAFT 3 Expires in: April 2006 4 Scott Poretsky 5 Reef Point Systems 7 Shankar Rao 8 Qwest Communications 10 October 2005 12 Methodology Guidelines for 13 Accelerated Stress Benchmarking 14 16 Intellectual Property Rights (IPR) statement: 17 By submitting this Internet-Draft, each author represents that any 18 applicable patent or other IPR claims of which he or she is aware 19 have been or will be disclosed, and any of which he or she becomes 20 aware will be disclosed, in accordance with Section 6 of BCP 79. 22 Status of this Memo 23 By submitting this Internet-Draft, I certify that any applicable 24 patent or other IPR claims of which I am aware have been disclosed, 25 and any of which I become aware will be disclosed, in accordance with 26 RFC 3668. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF), its areas, and its working groups. Note that 30 other groups may also distribute working documents as 31 Internet-Drafts. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 The list of current Internet-Drafts can be accessed at 39 http://www.ietf.org/ietf/1id-abstracts.txt. 40 The list of Internet-Draft Shadow Directories can be accessed at 41 http://www.ietf.org/shadow.html. 43 Copyright Notice 44 Copyright (C) The Internet Society (2005). All Rights Reserved. 46 ABSTRACT 47 Routers in an operational network are simultaneously configured 48 With multiple protocols and security policies while forwarding 49 traffic and being managed. To accurately benchmark a router for 50 deployment it is necessary that the router be tested in these 51 simultaneous operational conditions, which is known as Stress 52 Testing. This document provides the Methodology Guidelines for 53 performing Stress Benchmarking of networking devices. 54 Descriptions of Test Topology, Benchmarks and Reporting Format 55 are provided in addition to procedures for conducting various 56 test cases. The methodology is to be used with the companion 57 terminology document [4]. These guidelines can be used as the 58 basis for additional methodology documents that benchmark specific 59 network technologies under accelerated stress. 61 Stress Benchmarking 62 Table of Contents 63 1. Introduction ............................................... 2 64 2. Existing definitions ....................................... 3 65 3. Test Setup.................................................. 3 66 3.1 Test Topologies............................................ 3 67 3.2 Test Considerations........................................ 3 68 3.3 Reporting Format........................................... 4 69 3.3.1 Configuration Sets....................................... 5 70 3.3.2 Startup Conditions....................................... 6 71 3.3.3 Instability Conditions................................... 6 72 3.3.4 Benchmarks............................................... 7 73 4. Example Test Case Procedure................................. 7 74 5. IANA Considerations......................................... 8 75 6. Security Considerations..................................... 9 76 7. Normative References........................................ 9 77 8. Informative References......................................10 78 9. Author's Address............................................10 80 1. Introduction 81 Router testing benchmarks have consistently been made in a monolithic 82 fashion wherein a single protocol or behavior is measured in an 83 isolated environment. It is important to know the limits for a 84 networking device's behavior for each protocol in isolation, however 85 this does not produce a reliable benchmark of the device's behavior 86 in an operational network. 88 Routers in an operational network are simultaneously configured with 89 multiple protocols and security policies while forwarding traffic 90 and being managed. To accurately benchmark a router for deployment 91 it is necessary to test that router in operational conditions by 92 simultaneously configuring and scaling network protocols and security 93 policies, forwarding traffic, and managing the device. It is helpful 94 to accelerate these network operational conditions with Instability 95 Conditions [4] so that the networking devices are stress tested. 97 This document provides the Methodology for performing Stress 98 Benchmarking of networking devices. Descriptions of Test Topology, 99 Benchmarks and Reporting Format are provided in addition to 100 procedures for conducting various test cases. The methodology is 101 to be used with the companion terminology document [4]. 103 Stress Testing of networking devices provides the following benefits: 104 1. Evaluation of multiple protocols enabled simultaneously as 105 configured in deployed networks 106 2. Evaluation of System and Software Stability 107 3. Evaluation of Manageability under stressful conditions 108 4. Identification of Buffer Overflow conditions 109 5. Identification of Software Coding bugs such as: 110 a. Memory Leaks 111 b. Suboptimal CPU Utilization 112 c. Coding Logic 113 Stress Benchmarking 115 These benefits produce significant advantages for network operations: 116 1. Increased stability of routers and protocols 117 2. Hardened routers to DoS attacks 118 3. Verified manageability under stress 119 4. Planning router resources for growth and scale 121 2. Existing definitions 122 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 123 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 124 document are to be interpreted as described in BCP 14, RFC 2119 125 [6]. RFC 2119 defines the use of these key words to help make the 126 intent of standards track documents as clear as possible. While this 127 document uses these keywords, this document is not a standards track 128 document. 130 Terms related to Accelerated Stress Benchmarking are defined in [4]. 132 3. Test Setup 133 3.1 Test Topologies 134 Figure 1 shows the physical configuration to be used for the 135 methodologies provided in this document. The number of interfaces 136 between the tester and DUT will scale depending upon the number of 137 control protocol sessions and traffic forwarding interfaces. A 138 separate device may be required to externally manage the device in 139 the case that the test equipment does not support such 140 functionality. Figure 2 shows the logical configuration for the 141 stress test methodologies. Each plane may be emulated by single or 142 multiple test equipment. 144 3.2 Test Considerations 145 The Accelerated Stress Benchmarking test can be applied in 146 service provider test environments to benchmark DUTs under 147 stress in an environment that is reflective of an operational 148 network. A particular Configuration Set is defined and the 149 DUT is benchmarked using this configuration set and the 150 Instability Conditions. Varying Configuration Sets and/or 151 Instability Conditions applied in an iterative fashion can 152 provide an accurate characterization of the DUT 153 to help determine future network deployments. 155 Stress Benchmarking 157 ___________ 158 | DUT | 159 ___|Management | 160 | | | 161 | ----------- 162 \/ 163 ___________ 164 | | 165 | DUT | 166 |--->| |<---| 167 xN | ----------- | xN 168 interfaces | | interfaces 169 | ___________ | 170 | | | | 171 |--->| Tester |<---| 172 | | 173 ----------- 175 Figure 1. Physical Configuration 177 ___________ ___________ 178 | Control | | Management| 179 | Plane |___ ___| Plane | 180 | | | | | | 181 ----------- | | ----------- 182 \/ \/ ___________ 183 ___________ | Security | 184 | |<-----------| Plane | 185 | DUT | | | 186 |--->| |<---| ----------- 187 | ----------- | 188 | | 189 | ___________ | 190 | | Data | | 191 |--->| Plane |<---| 192 | | 193 ----------- 195 Figure 2. Logical Configuration 197 3.3 Reporting Format 199 Each methodology requires reporting of information for test 200 repeatability when benchmarking the same or different devices. 201 The information that are the Configuration Sets, Instability 202 Conditions, and Benchmarks, as defined in [4]. Example 203 reporting formats for each are provided below. 205 Stress Benchmarking 207 3.3.1 Configuration Sets 209 Configuration Sets may include and are not limited to the following 210 examples. 212 Example Routing Protocol Configuration Set- 213 PARAMETER UNITS 214 BGP Enabled/Disabled 215 Number of EBGP Peers Peers 216 Number of IBGP Peers Peers 217 Number of BGP Route Instances Routes 218 Number of BGP Installed Routes Routes 219 MBGP Enabled/Disabled 220 Number of MBGP Route Instances Routes 221 Number of MBGP Installed Routes Routes 222 IGP Enabled/Disabled 223 IGP-TE Enabled/Disabled 224 Number of IGP Adjacencies Adjacencies 225 Number of IGP Routes Routes 226 Number of Nodes per Area Nodes 228 Example MPLS Protocol Configuration Set- 229 PARAMETER UNITS 230 MPLS-TE Enabled/Disabled 231 Number of Tunnels as Ingress Tunnels 232 Number of Tunnels as Mid-Point Tunnels 233 Number of Tunnels as Egress Tunnels 234 LDP Enabled/Disabled 235 Number of Sessions Sessions 236 Number of FECs FECs 238 Example Multicast Protocol Configuration Set- 239 PARAMETER UNITS 240 PIM-SM Enabled/Disabled 241 RP Enabled/Disabled 242 Number of Multicast Groups Groups 243 MSDP Enabled/Disabled 245 Example Data Plane Configuration Set- 246 PARAMETER UNITS 247 Traffic Forwarding Enabled/Disabled 248 Aggregate Offered Load bps (or pps) 249 Number of Ingress Interfaces number 250 Number of Egress Interfaces number 252 TRAFFIC PROFILE 253 Packet Size(s) bytes 254 Offered Load (interface) array of bps 255 Number of Flows number 256 Encapsulation(flow) array of encapsulation type 257 Stress Benchmarking 259 Management Configuration Set- 260 PARAMETER UNITS 261 SNMP GET Rate SNMP Gets/minute 262 Logging Enabled/Disabled 263 Protocol Debug Enabled/Disabled 264 Telnet Rate Sessions/Hour 265 FTP Rate Sessions/Hour 266 Concurrent Telnet Sessions Sessions 267 Concurrent FTP Session Sessions 268 Packet Statistics Collector Enabled/Disabled 269 Statistics Sampling Rate X:1 packets 271 Security Configuration Set - 272 PARAMETER UNITS 273 Packet Filters Enabled/Disabled 274 Number of Filters For-Me number 275 Number of Filter Rules For-Me number 276 Number of Traffic Filters number 277 Number of Traffic Filter Rules number 278 IPsec tunnels number 279 SSH Enabled/Disabled 280 Number of simultaneous SSH sessions number 281 RADIUS Enabled/Disabled 282 TACACS Enabled/Disabled 284 3.3.2 Startup Conditions 285 Startup Conditions may include and are not limited to the following 286 examples: 287 PARAMETER UNITS 288 EBGP peering sessions negotiated Total EBGP Sessions 289 IBGP peering sessions negotiated Total IBGP Sessions 290 BGP routes learned rate BGP Routes per Second 291 ISIS adjacencies established Total ISIS Adjacencies 292 ISIS routes learned rate ISIS Routes per Second 293 IPsec tunnels negotiated Total IPsec Tunnels 294 IPsec tunnel establishment rate IPsec tunnels per second 296 3.3.3 Instability Conditions 297 Instability Conditions may include and are not limited to the 298 following examples: 299 PARAMETER UNITS 300 Interface Shutdown Cycling Rate interfaces per minute 301 BGP Session Flap Rate sessions per minute 302 BGP Route Flap Rate routes per minutes 303 IGP Route Flap Rate routes per minutes 304 LSP Reroute Rate LSP per minute 305 Overloaded Links number 306 Amount Links Overloaded % of bandwidth 307 FTP Rate Mb/minute 308 IPsec Tunnel Flap Rate tunnels per minute 309 Filter Policy Changes policies per hour 310 SSH Session Restart SSH sessions per hour 311 Telnet Session Restart Telnet session per hour 312 Stress Benchmarking 314 3.3.4 Benchmarks 316 Benchmarks are as defined in [1] and listed as follow: 317 PARAMETER UNITS PHASE 318 Stable Aggregate Forwarding Rate pps Startup 319 Stable Latency seconds Startup 320 Stable Session Count sessions Startup 321 Unstable Aggregate Forwarding Rate pps Instability 322 Degraded Aggregate Forwarding Rate pps Instability 323 Ave. Degraded Aggregate Forwarding Rate pps Instability 324 Unstable Latency seconds Instability 325 Unstable Uncontrolled Sessions Lost sessions Instability 326 Recovered Aggregate Forwarding Rate pps Recovery 327 Recovered Latency seconds Recovery 328 Recovery Time seconds Recovery 329 Recovered Uncontrolled Sessions Lost sessions Recovery 331 4. Example Test Case Procedure 332 1. Report Configuration Set 334 BGP Enabled 335 10 EBGP Peers 336 30 IBGP Peers 337 500K BGP Route Instances 338 160K BGP FIB Routes 340 ISIS Enabled 341 ISIS-TE Disabled 342 30 ISIS Adjacencies 343 10K ISIS Level-1 Routes 344 250 ISIS Nodes per Area 346 MPLS Disabled 347 IP Multicast Disabled 349 IPsec Enabled 350 10K IPsec tunnels 351 640 Firewall Policies 352 100 Firewall Rules per Policy 354 Traffic Forwarding Enabled 355 Aggregate Offered Load 10Gbps 356 30 Ingress Interfaces 357 30 Egress Interfaces 358 Packet Size(s) = 64, 128, 256, 512, 1024, 1280, 1518 bytes 359 Forwarding Rate[1..30] = 1Gbps 360 10000 Flows 361 Encapsulation[1..5000] = IPv4 362 Encapsulation[5001.10000] = IPsec 363 Stress Benchmarking 365 Logging Enabled 366 Protocol Debug Disabled 367 SNMP Enabled 368 SSH Enabled 369 20 Concurrent SSH Sessions 370 FTP Enabled 371 RADIUS Enabled 372 TACACS Disabled 373 Packet Statistics Collector Enabled 375 2. Begin Startup Conditions with the DUT 377 10 EBGP peering sessions negotiated 378 30 EBGP peering sessions negotiated 379 1K BGP routes learned per second 380 30 ISIS Adjacencies 381 1K ISIS routes learned per second 382 10K IPsec tunnels negotiated 384 3. Establish Configuration Sets with the DUT 386 4. Report Stability Benchmarks as follow: 388 Stable Aggregate Forwarding Rate 389 Stable Latency 390 Stable Session Count 392 It is RECOMMENDED that the benchmarks be measured and 393 recorded at one-second intervals. 395 5. Apply Instability Conditions 397 Interface Shutdown Cycling Rate = 1 interface every 5 minutes 398 BGP Session Flap Rate = 1 session every 10 minutes 399 BGP Route Flap Rate = 100 routes per minute 400 ISIS Route Flap Rate = 100 routes per minute 401 IPsec Tunnel Flap Rate = 1 tunnel per minute 402 Overloaded Links = 5 of 30 403 Amount Links Overloaded = 20% 404 SNMP GETs = 1 per sec 405 SSH Restart Rate = 10 sessions per hour 406 FTP Restart Rate = 10 transfers per hour 407 FTP Transfer Rate = 100 Mbps 408 Statistics Sampling Rate = 1:1 packets 410 6. Apply Instability Condition specific to test case. 412 Stress Benchmarking 414 7. Report Instability Benchmarks as follow: 415 Unstable Aggregate Forwarding Rate 416 Degraded Aggregate Forwarding Rate 417 Ave. Degraded Aggregate Forwarding Rate 418 Unstable Latency 419 Unstable Uncontrolled Sessions Lost 421 It is RECOMMENDED that the benchmarks be measured and 422 recorded at one-second intervals. 424 8. Stop applying all Instability Conditions 426 9. Report Recovery Benchmarks as follow: 428 Recovered Aggregate Forwarding Rate 429 Recovered Latency 430 Recovery Time 431 Recovered Uncontrolled Sessions Lost 433 It is RECOMMENDED that the benchmarks be measured and 434 recorded at one-second intervals. 436 10. Optional - Change Configuration Set and/or Instability 437 Conditions for next iteration 439 5. IANA Considerations 440 This document requires no IANA considerations. 442 6. Security Considerations 443 Documents of this type do not directly affect the security of 444 the Internet or of corporate networks as long as benchmarking 445 is not performed on devices or systems connected to operating 446 networks. 448 7. Normative References 450 [1] Bradner, S., Editor, "Benchmarking Terminology for Network 451 Interconnection Devices", RFC 1242, October 1991. 453 [2] Mandeville, R., "Benchmarking Terminology for LAN Switching 454 Devices", RFC 2285, June 1998. 456 [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for 457 Network Interconnect Devices", RFC 2544, March 1999. 459 [4] Poretsky, S. and Rao, S., "Terminology for Accelerated 460 Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-07, 461 work in progress, October 2005. 463 Stress Benchmarking 465 [5] Poretsky, S., "Benchmarking Terminology for IGP Data Plane 466 Route Convergence", 467 draft-ietf-bmwg-igp-dataplane-conv-term-08, work in 468 progress, October 2005. 470 [6] Bradner, S., "Key words for use in RFCs to Indicate 471 Requirement Levels", RFC 2119, March 1997. 473 8. Informative References 475 [RFC3871] RFC 3871 "Operational Security Requirements for Large 476 Internet Service Provider (ISP) IP Network Infrastructure. 477 G. Jones, Ed.. IETF, September 2004. 479 [NANOG25] "Core Router Evaluation for Higher Availability", 480 Scott Poretsky, NANOG 25, June 8, 2002, Toronto, CA. 482 [IEEECQR] "Router Stress Testing to Validate Readiness for 483 Network Deployment", Scott Poretsky, IEEE CQR 2003. 485 [CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data 486 Plane Route Convergence", 487 draft-ietf-bmwg-igp-dataplane-conv-meth-08, work in progress, 488 October 2005. 490 9. Author's Address 492 Scott Poretsky 493 Reef Point Systems 494 8 New England Executive Park 495 Burlington, MA 01803 496 USA 497 Phone: + 1 781 395 5090 498 EMail: sporetsky@reefpoint.com 500 Shankar Rao 501 1801 California Street 502 8th Floor 503 Qwest Communications 504 Denver, CO 80202 505 USA 506 Phone: + 1 303 437 6643 507 Email: shankar.rao@qwest.com 508 Stress Benchmarking 510 Full Copyright Statement 512 Copyright (C) The Internet Society (2005). 514 This document is subject to the rights, licenses and restrictions 515 contained in BCP 78, and except as set forth therein, the authors 516 retain all their rights. 518 This document and the information contained herein are provided on an 519 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 520 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 521 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 522 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 523 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 524 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 526 Intellectual Property 528 The IETF takes no position regarding the validity or scope of any 529 Intellectual Property Rights or other rights that might be claimed to 530 pertain to the implementation or use of the technology described in 531 this document or the extent to which any license under such rights 532 might or might not be available; nor does it represent that it has 533 made any independent effort to identify any such rights. Information 534 on the procedures with respect to rights in RFC documents can be 535 found in BCP 78 and BCP 79. 537 Copies of IPR disclosures made to the IETF Secretariat and any 538 assurances of licenses to be made available, or the result of an 539 attempt made to obtain a general license or permission for the use of 540 such proprietary rights by implementers or users of this 541 specification can be obtained from the IETF on-line IPR repository at 542 http://www.ietf.org/ipr. 544 The IETF invites any interested party to bring to its attention any 545 copyrights, patents or patent applications, or other proprietary 546 rights that may cover technology that may be required to implement 547 this standard. Please address the information to the IETF at ietf- 548 ipr@ietf.org. 550 Acknowledgement 552 Funding for the RFC Editor function is currently provided by the 553 Internet Society.