idnits 2.17.1 draft-ietf-bmwg-reset-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year (Using the creation date from RFC1242, updated by this document, for RFC5378 checks: 1991-07-01) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (December 20, 2010) is 4848 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-23) exists of draft-ietf-bmwg-igp-dataplane-conv-meth-21 Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Benchmarking Methodology WG Rajiv Asati 2 Internet Draft Cisco 3 Updates: 1242, 2544 (if approved) Carlos Pignataro 4 Intended status: Informational Cisco 5 Expires: June 2011 Fernando Calabria 6 Cisco 7 Cesar Olvera 8 Consulintel 10 December 20, 2010 12 Device Reset Characterization 13 draft-ietf-bmwg-reset-06 15 Abstract 17 An operational forwarding device may need to be re-started 18 (automatically or manually) for a variety of reasons, an event that 19 we call a "reset" in this document. Since there may be an 20 interruption in the forwarding operation during a reset, it is 21 useful to know how long a device takes to resume the forwarding 22 operation. 24 This document specifies a methodology for characterizing reset (and 25 reset time) during benchmarking of forwarding devices, and provides 26 clarity and consistency in reset test procedures beyond what's 27 specified in RFC2544. It therefore updates RFC2544. This document 28 also defines the benchmarking term "Reset Time" and only in this 29 updates RFC1242. 31 Status of this Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF), its areas, and its working groups. Note that 38 other groups may also distribute working documents as Internet- 39 Drafts. 41 Internet-Drafts are draft documents valid for a maximum of six 42 months and may be updated, replaced, or obsoleted by other documents 43 at any time. It is inappropriate to use Internet-Drafts as 44 reference material or to cite them other than as "work in progress." 46 The list of current Internet-Drafts can be accessed at 47 http://www.ietf.org/ietf/1id-abstracts.txt 49 The list of Internet-Draft Shadow Directories can be accessed at 50 http://www.ietf.org/shadow.html 52 This Internet-Draft will expire on June 20, 2011. 54 Copyright Notice 56 Copyright (c) 2010 IETF Trust and the persons identified as the 57 document authors. All rights reserved. 59 This document is subject to BCP 78 and the IETF Trust's Legal 60 Provisions Relating to IETF Documents 61 (http://trustee.ietf.org/license-info) in effect on the date of 62 publication of this document. Please review these documents 63 carefully, as they describe your rights and restrictions with 64 respect to this document. Code Components extracted from this 65 document must include Simplified BSD License text as described in 66 Section 4.e of the Trust Legal Provisions and are provided without 67 warranty as described in the Simplified BSD License. 69 Table of Contents 71 1. Key Words to Reflect Requirements..............................4 72 2. Introduction...................................................4 73 2.1. Scope.....................................................4 74 2.2. Reset Time................................................5 75 2.3. Reset Time Measurement Methods............................6 76 2.4. Reporting Format..........................................7 77 3. Test Requirements..............................................8 78 4. Reset Test.....................................................9 79 4.1. Hardware Reset Test......................................10 80 4.1.1. Routing Processor (RP) / Routing Engine Reset.......10 81 4.1.1.1. RP Reset for a single-RP device (REQUIRED).....11 82 4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL) 83 ........................................................11 84 4.1.2. Line Card (LC) Removal and Insertion (REQUIRED).....13 85 4.2. Software Reset Test......................................14 86 4.2.1. Operating System (OS) Reset (REQUIRED)..............14 87 4.2.2. Process Reset (OPTIONAL)............................15 88 4.3. Power Interruption Test..................................16 89 4.3.1. Power Interruption (REQUIRED).......................16 90 5. Security Considerations.......................................17 91 6. IANA Considerations...........................................17 92 7. Acknowledgments...............................................17 93 8. References....................................................19 94 8.1. Normative References.....................................19 95 8.2. Informative References...................................19 96 Authors' Addresses...............................................20 98 1. Key Words to Reflect Requirements 100 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 101 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 102 document are to be interpreted as described in BCP 14, RFC 2119 103 [RFC2119]. RFC 2119 defines the use of these key words to help make 104 the intent of standards track documents as clear as possible. While 105 this document uses these keywords, this document is not a standards 106 track document. 108 2. Introduction 110 An operational forwarding device (or one of its components) may need 111 to be re-started for a variety of reasons, an event that we call a 112 "reset" in this document. Since there may be an interruption in the 113 forwarding operation during a reset, it is useful to know how long a 114 device takes to resume the forwarding operation. In other words, it 115 is desired to know the duration of the recovery time following the 116 reset (reset time, see Section 2.2). 118 However, the answer to this question is no longer simple and 119 straight-forward as the modern forwarding devices employ many 120 hardware advancements (distributed forwarding, etc.) and software 121 advancements (graceful restart, etc.) that influence the recovery 122 time after the reset. 124 2.1. Scope 126 This document specifies a methodology for characterizing reset (and 127 reset time) during benchmarking of forwarding devices, and provides 128 clarity and consistency in reset procedures beyond what is specified 129 in [RFC2544]. Software upgrades involve additional benchmarking 130 complexities and are outside the scope of this document. 132 These procedures may be used by other benchmarking documents such as 133 [RFC2544], [RFC5180], [RFC5695], etc., and is expected that other 134 protocol-specific benchmarking documents would reference this 135 document for reset recovery time characterization. Specific Routing 136 Information Base (RIB) and Forwarding Information Base (FIB) scaling 137 considerations are outside the scope of this document and can be 138 quite complex to characterize. However, other documents can 139 characterize specific dynamic protocols scaling and interactions and 140 leverage and augment the reset tests defined in this document. 142 This document updates Section 26.6 of [RFC2544], and defines the 143 benchmarking term "Reset Time" updating [RFC1242]. 145 This document focuses only on the reset criterion of benchmarking, 146 and presumes that it would be beneficial to [RFC5180], [RFC5695], 147 and other IETF Benchmarking Methodology Working Group (BMWG) 148 efforts. 150 2.2. Reset Time 152 Definition 154 Reset Time is the total time when a device is determined to be out 155 of operation, and includes the time to perform the reset and the 156 time to recover from it. 158 Discussion 160 During a period of time after a reset or power up, network devices 161 may not accept and forward frames. The duration of this period of 162 forwarding unavailability can be useful in evaluating devices. In 163 addition, some network devices require some form of reset when 164 specific setup variables are modified. If the reset period were 165 long it might discourage network managers from modifying these 166 variables on production networks. 168 The events characterized in this document are entire reset events. 169 That is, the recovery period measured includes the time to perform 170 the reset and the time to recover from it. Some reset events will 171 be atomic (such as pressing a reset button) while others (such as 172 power cycling) may comprise multiple actions with a recognized 173 interval between them. In both cases, the duration considered is 174 from the start of the event until full recovery of forwarding 175 after the completion of the reset events. 177 Measurement units 179 Time in milliseconds, providing sufficient resolution to 180 distinguish between different trials and different 181 implementations. See Section 2.4. 183 Issues 185 There are various types of Reset: Hardware resets, software 186 resets, and power interruption. See Section 4. 188 See Also 190 This definition updates [RFC1242]. 192 2.3. Reset Time Measurement Methods 194 The 'reset time' is the time during which the traffic forwarding is 195 temporarily interrupted following a reset event. Strictly speaking, 196 this is the time over which one or more frames are lost. This 197 definition is similar to that of 'Loss of connectivity period' 198 defined in [IGPConv] section 4. 200 There are two accepted methods to measure the 'reset time': 202 1. Frame-Loss Method - This method requires test tool capability 203 to monitor the number of lost frames. In this method, the 204 offered stream rate (frames per second) must be known. The 205 reset time is calculated per the below equation: 207 Frames_lost (packets) 208 Reset_time = ------------------------------------- 209 Offered_rate (packets per second) 211 2. Time-Stamp Method - This method requires test tool capability 212 to timestamp each frame. In this method, the test tool 213 timestamps each transmitted frame and monitors the received 214 frame's timestamp. During the test, the test tool would record 215 the timestamp (Timestamp A) of the frame that was last 216 received prior to the reset interruption and the timestamp 217 (Timestamp B) of the first frame after the interruption 218 stopped. The difference between Timestamp B and Timestamp A is 219 the reset time. 221 The tester / operator MAY use either method for reset time 222 measurement depending on the test tool capability. However, the 223 Frame-loss method SHOULD be used if the test tool is capable of (a) 224 counting the number of lost frames per stream, and (b) transmitting 225 test frame despite the physical link status, whereas Time-stamp 226 method SHOULD be used if the test tool is capable of (a) time- 227 stamping each frame, (b) monitoring received frame's timestamp, and 228 (c) transmitting frames only if the physical link status is UP. That 229 is, specific test tool capabilities may dictate which method to use. 230 If the test tool supports both methods based on its capabilities, 231 the tester / operator SHOULD use the one that provides more 232 accuracy. 234 2.4. Reporting Format 236 All reset results are reported in a simple statement including the 237 frame loss (if measured) and reset times. 239 For each test case, it is RECOMMENDED that the following parameters 240 be reported in these units: 242 Parameter Units or Examples 244 ----------------------------------------------------------------- 246 Throughput Frames per second and bits per 247 second 249 Loss (average) Frames 251 Reset Time (average) Milliseconds 253 Number of trials Integer count 255 Protocol IPv4, IPv6, MPLS, etc. 257 Frame Size Octets 259 Port Media Ethernet, GigE (Gigabit Ethernet), 260 POS (Packet over SONET), etc. 262 Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc. 264 Interface Encap. Ethernet, Ethernet VLAN, 265 PPP, HDLC, etc. 267 For mixed protocol environments, frames SHOULD be distributed 268 between all the different protocols. The distribution MAY 269 approximate the network conditions of deployment. In all cases the 270 details of the mixed protocol distribution MUST be included in the 271 reporting. 273 Additionally, the DUT (Device Under Test) / SUT (System Under Test) 274 and test bed provisioning, port and Line Card arrangement, 275 configuration, and deployed methodologies that may influence the 276 overall reset time MUST be listed. (Refer to the additional factors 277 listed in Section 3). 279 The reporting of results MUST regard repeatability considerations 280 from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple 281 trials and report average results. 283 3. Test Requirements 285 Tests SHOULD first be performed such that the forwarding state re- 286 establishment is independent from an external source (i.e., using 287 static address resolution, routing and forwarding configuration, and 288 not dynamic protocols). However tests MAY subsequently be performed 289 using dynamic protocols that the forwarding state depends on (e.g., 290 dynamic Interior Gateway Protocols (IGP), ARP, PPP Control 291 Protocols, etc.) The considerations in this Section apply. 293 In order to provide consistence and fairness while benchmarking a 294 set of different DUTs, the Network tester / operator MUST (a) use 295 identical control and data plane information during testing, (b) 296 document & report any factors that may influence the overall time 297 after reset / convergence. 299 Some of these factors include: 301 1. Type of reset - Hardware (line-card crash, etc.) vs. Software 302 (protocol reset, process crash, etc.) or even complete power 303 failures 305 2. Manual vs. Automatic reset 307 3. Scheduled vs. non-scheduled reset 309 4. Local vs. Remote reset 311 5. Scale - Number of line cards present vs. in-use 313 6. Scale - Number of physical and logical interfaces 315 7. Scale - Number of routing protocol instances 316 8. Scale - Number of Routing Table entries 318 9. Scale - Number of Route Processors available 320 10. Performance - Redundancy strategy deployed for route 321 processors and line cards 323 11. Performance - Interface encapsulation as well as achievable 324 Throughput [RFC2544] 326 12. Any other internal or external factor that may influence reset 327 time after a hardware or software reset 329 The reset time is one of the key characterization results reported 330 after each test run. While the reset time during a reset test event 331 may be zero, there may still be effects on traffic, such as 332 transient delay variation or increased latency. However, that is not 333 covered and deemed outside the scope of this document. In this case, 334 only "no loss" is reported. 336 4. Reset Test 338 This section contains the description of the tests that are related 339 to the characterization of the time needed for DUTs (Device Under 340 Test) / SUTs (System Under Test) to recover from a reset. There are 341 three types of reset considered in this document: 343 1. Hardware resets 345 2. Software resets 347 3. Power interruption 349 Different types of reset have potentially different impact on the 350 forwarding behavior of the device. As an example, a software reset 351 (of a routing process) might not result in forwarding interruption, 352 whereas a hardware reset (of a line card) most likely will. 354 Section 4.1 describes various hardware resets, whereas Section 4.2 355 describes various software resets. Additionally, Section 4.3 356 describes power interruption tests. These sections define and 357 characterize these resets. 359 Additionally, since device specific implementations may vary for 360 hardware and software type resets, it is desirable to classify each 361 test case as "REQUIRED" or "OPTIONAL". 363 4.1. Hardware Reset Test 365 A test designed to characterize the time it takes a DUT to recover 366 from the hardware reset. 368 A "hardware reset" generally involves the re-initialization of one 369 or more physical components in the DUT, but not the entire DUT. 371 A hardware reset is executed by the operator for example by physical 372 removal of a hardware component, by pressing on a "reset" button for 373 the component, or could even be triggered from the command line 374 interface. 376 Reset procedures that do not require the physical removal and 377 insertion of a hardware component are RECOMMENDED. These include 378 using the Command Line Interface (CLI) or a physical switch or 379 button. If such procedures cannot be performed (e.g., for lack of 380 platform support, or because the corresponding Test Case calls for 381 them), human operation time SHOULD be minimized across different 382 platforms and Test Cases as much as possible, and variation in human 383 operator time SHOULD also be minimized across different vendors 384 products as much as practical, by having the same person perform the 385 operation, and by practicing the operation. Additionally, the time 386 between removal and insertion SHOULD be recorded and reported. 388 For routers that do not contain separate Routing Processor and Line 389 Card modules, the hardware reset tests are not performed since they 390 are not relevant; instead, the power interruption tests MUST be 391 performed (see Section 4.3) in these cases. 393 4.1.1. Routing Processor (RP) / Routing Engine Reset 395 The Routing Processor (RP) is the DUT module that is primarily 396 concerned with Control Plane functions. 398 4.1.1.1. RP Reset for a single-RP device (REQUIRED) 400 Objective 402 To characterize time needed for a DUT to recover from a Route 403 processor hardware reset in a single RP environment. 405 Procedure 407 First, ensure that the RP is in a permanent state to which it will 408 return to after the reset, by performing some or all of the 409 following operational tasks: save the current DUT configuration, 410 specify boot parameters, ensure the appropriate software files are 411 available, or perform additional Operating System or hardware 412 related task. 414 Second, ensure that the DUT is able to forward the traffic for at 415 least 15 seconds before any test activities are performed. The 416 traffic should use the minimum frame size possible on the media 417 used in the testing and rate should be sufficient for the DUT to 418 attain the maximum forwarding throughput. This enables a finer 419 granularity in the reset time measurement. 421 Third, perform the Route Processor (RP) hardware reset at this 422 point. This entails for example physically removing the RP to 423 later re-insert it, or triggering a hardware reset by other means 424 (e.g., command line interface, physical switch, etc.) 426 Finally, the characterization is completed by recording the frame 427 loss or time stamps (as reported by the test tool) and calculating 428 the reset time (as defined in Section 2.3). 430 Reporting format 432 The reporting format is defined in Section 2.4. 434 4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL) 436 Objective 438 To characterize time needed for "secondary" Route Processor 439 (sometimes referred to as "backup" RP) of a DUT to become active 440 after a "primary" (or "active") Route Processor hardware reset. 441 This process is often referred to as "RP Switchover". The 442 characterization in this test should be done for the default DUT 443 behavior as well as a DUT's non-default configuration that 444 minimizes frame loss, if exists. 446 Procedure 448 This test characterizes "RP Switchover". Many implementations 449 allow for optimized switchover capabilities that minimize the 450 downtime during the actual switchover. This test consists of two 451 sub-cases from a switchover characteristics standpoint: First, a 452 default behavior (with no switchover-specific configurations); and 453 potentially second, a non-default behavior with switchover 454 configuration to minimize frame loss. Therefore, the procedures 455 hereby described are executed twice, and reported separately. 457 First, ensure that the RPs are in a permanent state such that the 458 secondary will be activated to the same state as the active is, by 459 performing some or all of the following operational tasks: save 460 the current DUT configuration, specify boot parameters, ensure the 461 appropriate software files are available, or perform additional 462 Operating System or hardware related task. 464 Second, ensure that the DUT is able to forward the traffic for at 465 least 15 seconds before any test activities are performed. The 466 traffic should use the minimum frame size possible on the media 467 used in the testing and rate should be sufficient for the DUT to 468 attain the maximum forwarding throughput. This enables a finer 469 granularity in the reset time measurement. 471 Third, perform the primary Route Processor (RP) hardware reset at 472 this point. This entails for example physically removing the RP, 473 or triggering a hardware reset by other means (e.g., command line 474 interface, physical switch, etc.) It is up to the operator to 475 decide if the primary RP needs to be re-inserted after a grace 476 period or not. 478 Finally, the characterization is completed by recording the frame 479 loss or time stamps (as reported by the test tool) and calculating 480 the reset time (as defined in Section 2.3). 482 Reporting format 484 The reset results are potentially reported twice, one for the 485 default switchover behavior (i.e., the DUT without any switchover- 486 specific enhanced configuration) and the other for the switchover- 487 specific behavior if it exists (i.e., the DUT configured for 488 optimized switchover capabilities that minimize the downtime 489 during the actual switchover). 491 The reporting format is defined in Section 2.4, and also includes 492 any specific redundancy scheme in place. 494 4.1.2. Line Card (LC) Removal and Insertion (REQUIRED) 496 The Line Card (LC) is the DUT component that is responsible with 497 packet forwarding. 499 Objective 501 To characterize time needed for a DUT to recover from a Line Card 502 removal and insertion event. 504 Procedure 506 For this test, the Line Card that is being hardware-reset MUST be 507 on the forwarding path and all destinations MUST be directly 508 connected. 510 First, complete some or all of the following operational tasks: 511 save the current DUT configuration, specify boot parameters, 512 ensure the appropriate software files are available, or perform 513 additional Operating System or hardware related task. 515 Second, ensure that the DUT is able to forward the traffic for at 516 least 15 seconds before any test activities are performed. The 517 traffic should use the minimum frame size possible on the media 518 used in the testing and rate should be sufficient for the DUT to 519 attain the maximum forwarding throughput. This enables a finer 520 granularity in the reset time measurement. 522 Third, perform the Line Card (LC) hardware reset at this point. 523 This entails for example physically removing the LC to later re- 524 insert it, or triggering a hardware reset by other means (e.g., 525 CLI, physical switch, etc.). 527 Finally, the characterization is completed by recording the frame 528 loss or time stamps (as reported by the test tool) and calculating 529 the reset time (as defined in Section 2.3). 531 Reporting Format 533 The reporting format is defined in Section 2.4. 535 4.2. Software Reset Test 537 To characterize time needed for a DUT to recover from the software 538 reset. 540 In contrast to a "hardware reset", a "software reset" involves only 541 the re-initialization of the execution, data structures, and partial 542 state within the software running on the DUT module(s). 544 A software reset is initiated for example from the DUT's CLI. 546 4.2.1. Operating System (OS) Reset (REQUIRED) 548 Objective 550 To characterize time needed for a DUT to recover from an Operating 551 System (OS) software reset. 553 Procedure 555 First, complete some or all of the following operational tasks: 556 save the current DUT configuration, specify software boot 557 parameters, ensure the appropriate software files are available, 558 or perform additional Operating System task. 560 Second, ensure that the DUT is able to forward the traffic for at 561 least 15 seconds before any test activities are performed. The 562 traffic should use the minimum frame size possible on the media 563 used in the testing and rate should be sufficient for the DUT to 564 attain the maximum forwarding throughput. This enables a finer 565 granularity in the reset time measurement. 567 Third, trigger an Operating System re-initialization in the DUT, 568 by operational means such as use of the DUT's CLI or other 569 management interface. 571 Finally, the characterization is completed by recording the frame 572 loss or time stamps (as reported by the test tool) and calculating 573 the reset time (as defined in Section 2.3). 575 Reporting format 577 The reporting format is defined in Section 2.4. 579 4.2.2. Process Reset (OPTIONAL) 581 Objective 583 To characterize time needed for a DUT to recover from a software 584 process reset. 586 Such time period may depend upon the number and types of process 587 running in the DUT and which ones are tested. Different 588 implementations of forwarding devices include various common 589 processes. A process reset should be performed only in the 590 processes most relevant to the tester and most impactful to 591 forwarding. 593 Procedure 595 First, complete some or all of the following operational tasks: 596 save the current DUT configuration, specify software parameters or 597 environmental variables, or perform additional Operating System 598 task. 600 Second, ensure that the DUT is able to forward the traffic for at 601 least 15 seconds before any test activities are performed. The 602 traffic should use the minimum frame size possible on the media 603 used in the testing and rate should be sufficient for the DUT to 604 attain the maximum forwarding throughput. This enables a finer 605 granularity in the reset time measurement. 607 Third, trigger a process reset for each process running in the DUT 608 and considered for testing from a management interface (e.g., by 609 means of the CLI, etc.) 611 Finally, the characterization is completed by recording the frame 612 loss or time stamps (as reported by the test tool) and calculating 613 the reset time (as defined in Section 2.3). 615 Reporting format 617 The reporting format is defined in Section 2.4, and is used for 618 each process running in the DUT and tested. Given the 619 implementation nature of this test, details of the actual process 620 tested should be included along with the statement. 622 4.3. Power Interruption Test 624 "Power interruption" refers to the complete loss of power on the 625 DUT. It can be viewed as a special case of a hardware reset, 626 triggered by the loss of the power supply to the DUT or its 627 components, and is characterized by the re-initialization of all 628 hardware and software in the DUT. 630 4.3.1. Power Interruption (REQUIRED) 632 Objective 634 To characterize time needed for a DUT to recover from a complete 635 loss of electric power or complete power interruption. This test 636 simulates a complete power failure or outage, and should be 637 indicative of the DUT/SUTs behavior during such event. 639 Procedure 641 First, ensure that the entire DUT is at a permanent state to which 642 it will return to after the power interruption, by performing some 643 or all of the following operational tasks: save the current DUT 644 configuration, specify boot parameters, ensure the appropriate 645 software files are available, or perform additional Operating 646 System or hardware related task. 648 Second, ensure that the DUT is able to forward the traffic for at 649 least 15 seconds before any test activities are performed. The 650 traffic should use the minimum frame size possible on the media 651 used in the testing and rate should be sufficient for the DUT to 652 attain the maximum forwarding throughput. This enables a finer 653 granularity in the reset time measurement. 655 Third, interrupt the power (AC or DC) that feeds the corresponding 656 DUTs power supplies at this point. This entails for example 657 physically removing the power supplies in the DUT to later re- 658 insert them, or simply disconnecting or switching off their power 659 feeds (AC or DC as applicable). The actual power interruption 660 should last at least 15 seconds. 662 Finally, the characterization is completed by recording the frame 663 loss or time stamps (as reported by the test tool) and calculating 664 the reset time (as defined in Section 2.3). 666 For easier comparison with other testing, the 15 seconds are 667 removed from the reported reset time. 669 Reporting format 671 The reporting format is defined in Section 2.4. 673 5. Security Considerations 675 Benchmarking activities, as described in this memo, are limited to 676 technology characterization using controlled stimuli in a laboratory 677 environment, with dedicated address space and the constraints 678 specified in the sections above. 680 The benchmarking network topology will be an independent test setup 681 and MUST NOT be connected to devices that may forward the test 682 traffic into a production network or misroute traffic to the test 683 management network. 685 Furthermore, benchmarking is performed on a "black-box" basis, 686 relying solely on measurements observable external to the DUT/SUT. 688 Special capabilities SHOULD NOT exist in the DUT/SUT specifically 689 for benchmarking purposes. Any implications for network security 690 arising from the DUT/SUT SHOULD be identical in the lab and in 691 production networks. 693 There are no specific security considerations within the scope of 694 this document. 696 6. IANA Considerations 698 There is no IANA consideration for this document. 700 7. Acknowledgments 702 The authors would like to thank Ron Bonica, who motivated us to 703 write this document. The authors would also like to thank Al Morton, 704 Andrew Yourtchenko, David Newman, John E. Dawson, Timmons C. Player, 705 Jan Novak, Steve Maxwell, Ilya Varlashkin, and Sarah Banks for 706 providing thorough review, useful suggestions, and valuable input. 708 This document was prepared using 2-Word-v2.0.template.dot. 710 8. References 712 8.1. Normative References 714 [RFC1242] Bradner, S., "Benchmarking terminology for network 715 interconnection devices", RFC 1242, July 1991. 717 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 718 Requirement Levels", BCP 14, RFC 2119, March 1997. 720 [RFC2544] Bradner, S. and McQuaid, J., "Benchmarking Methodology for 721 Network Interconnect Devices", RFC 2544, March 1999. 723 8.2. Informative References 725 [IGPConv] Poretsky, S., Imhoff, B., and K. Michielsen, "Benchmarking 726 Methodology for Link-State IGP Data Plane Route 727 Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-21 728 (work in progress), May 2010. 730 [RFC5180] Popoviciu, C., et al, "IPv6 Benchmarking Methodology for 731 Network Interconnect Devices", RFC 5180, May 2008. 733 [RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding 734 Benchmarking Methodology for IP Flows", RFC 5695, November 735 2009. 737 Authors' Addresses 739 Rajiv Asati 740 Cisco Systems 741 7025-6 Kit Creek Road 742 RTP, NC 27709 743 USA 745 Email: rajiva@cisco.com 747 Carlos Pignataro 748 Cisco Systems 749 7200-12 Kit Creek Road 750 RTP, NC 27709 751 USA 753 Email: cpignata@cisco.com 755 Fernando Calabria 756 Cisco Systems 757 7200-12 Kit Creek Road 758 RTP, NC 27709 759 USA 761 Email: fcalabri@cisco.com 763 Cesar Olvera 764 Consulintel 765 Joaquin Turina, 2 766 Pozuelo de Alarcon, Madrid, E-28224 767 Spain 769 Email: cesar.olvera@consulintel.es