idnits 2.17.1 draft-ietf-bmwg-issu-meth-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 99 has weird spacing: '... goal of e...' -- The document date (March 9, 2015) is 3328 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC 5880' is mentioned on line 197, but not defined == Missing Reference: 'RFC2544' is mentioned on line 661, but not defined == Unused Reference: '1' is defined on line 685, but no explicit reference was found in the text == Unused Reference: '2' is defined on line 688, but no explicit reference was found in the text == Unused Reference: 'RFC2234' is defined on line 695, but no explicit reference was found in the text == Unused Reference: '3' is defined on line 701, but no explicit reference was found in the text == Unused Reference: 'Fab1999' is defined on line 705, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2234 (ref. '2') (Obsoleted by RFC 4234) -- Duplicate reference: RFC2119, mentioned in 'RFC2119', was also mentioned in '1'. -- Duplicate reference: RFC2234, mentioned in 'RFC2234', was also mentioned in '2'. ** Obsolete normative reference: RFC 2234 (Obsoleted by RFC 4234) Summary: 2 errors (**), 0 flaws (~~), 9 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Benchmarking Working Group Sarah Banks 2 Internet Draft VSS Monitoring 3 Intended status: Informational Fernando Calabria 4 Expires: September 9, 2015 Cisco 5 Gery Czirjak 6 Ramdas Machat 7 Juniper 8 March 9, 2015 10 ISSU Benchmarking Methodology 11 draft-ietf-bmwg-issu-meth-00 13 Abstract 15 Modern forwarding devices attempt to minimize any control and data 16 plane disruptions while performing planned software changes, by 17 implementing a technique commonly known as In Service Software 18 Upgrade (ISSU) This document specifies a set of common methodologies 19 and procedures designed to characterize the overall behavior of a 20 Device Under Test (DUT), subject to an ISSU event. 22 Status of this Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF), its areas, and its working groups. Note that 29 other groups may also distribute working documents as Internet- 30 Drafts. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 The list of current Internet-Drafts can be accessed at 38 http://www.ietf.org/1id-abstracts.html 40 The list of Internet-Draft Shadow Directories can be accessed at 41 http://www.ietf.org/shadow.html 43 This Internet-Draft will expire on September 6, 2015. 45 Copyright Notice 47 Copyright (c) 2015 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (http://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with 55 respect to this document. Code Components extracted from this 56 document must include Simplified BSD License text as described in 57 Section 4.e of the Trust Legal Provisions and are provided without 58 warranty as described in the Simplified BSD License. 60 This document is subject to BCP 78 and the IETF Trust's Legal 61 Provisions Relating to IETF Documents 62 (http://trustee.ietf.org/license-info) in effect on the date of 63 publication of this document. Please review these documents 64 carefully, as they describe your rights and restrictions with 65 respect to this document. 67 Table of Contents 69 1. Introduction...................................................3 70 2. Conventions used in this document..............................4 71 3. Generic ISSU Process, phased approach..........................5 72 3.1. Software Download.........................................5 73 3.2. Software Staging..........................................6 74 3.3. Upgrade Run...............................................6 75 3.4. Upgrade Acceptance........................................7 76 4. Test Methodology...............................................7 77 4.1. Test Topology.............................................7 78 4.2. Load Model................................................8 79 5. ISSU Test Methodology..........................................9 80 5.1. Pre-ISSU recommended verifications........................9 81 5.2. Software Staging.........................................10 82 5.3. Upgrade Run..............................................11 83 5.4. Post ISSU verification...................................11 84 5.5. ISSU under negative stimuli..............................12 85 6. ISSU Abort and Rollback.......................................12 86 7. Final Report - Data Presentation - Analysis...................13 87 7.1. Data collection considerations...........................15 88 8. Security Considerations.......................................15 89 9. IANA Considerations...........................................16 90 10. References...................................................16 91 10.1. Normative References....................................16 92 10.2. Informative References..................................16 93 11. Acknowledgments..............................................16 95 1. Introduction 96 As required by most Service Provider (SP) network operators, ISSU 97 functionality has been implemented by modern forwarding devices to 98 upgrade or downgrade from one software version to another with a 99 goal of eliminating the downtime of the router and/or the outage 100 of service. However, it is noted that while most operators desire 101 complete elimination of downtime, minimization of downtime and 102 service degradation is often the expectation. 104 The ISSU operation may apply in terms of an atomic version change of 105 the entire system software or it may be applied in a more modular 106 sense such as for a patch or maintenance upgrade. The procedure 107 described herein may be used to verify either approach, as may be 108 supported by the vendor hardware and software. 110 In support of this document, the desired behavior for an ISSU 111 operation can be summarized as follows: 113 - The software is successfully migrated, from one version to a 114 successive version or vice versa. 116 - There are no control plane interruptions throughout the process. 117 That is, the upgrade/downgrade could be accomplished while the device 118 remains "in service". It is noted however, that most service 119 providers will still undertake such actions in a maintenance window 120 (even in redundant environments) to minimize any risk. 122 - Interruptions to the forwarding plane are minimal to none. 124 - The total time to accomplish the upgrade is minimized, again to 125 reduce potential network outage exposure (e.g. an external failure 126 event might impact the network as it operates with reduced 127 redundancy). 129 This document provides a set of procedures to characterize a given 130 forwarding device's ISSU behavior quantitatively, from the 131 perspective of meeting the above expectations. 133 Different hardware configurations may be expected to be benchmarked, 134 but a typical configuration for a forwarding device that supports 135 ISSU consists of at least one pair of Routing Processors (RP's) that 136 operate in a redundant fashion, and single or multiple Forwarding 137 Engines (Line Cards) that may or may not be redundant, as well as 138 fabric cards or other components as applicable. This does not 139 preclude the possibility that a device in question can perform ISSU 140 functions through the operation of independent process components, 141 which may be upgraded without impact to the overall operation of the 142 device. As an example, perhaps the software module involved in SNMP 143 functions can be upgraded without impacting other operations. 145 The concept of a multi-chassis deployment may also be characterized 146 by the current set of proposed methodologies, but the implementation 147 specific details (i.e. process placement and others) are beyond the 148 scope of the current document. 150 Since most modern forwarding devices, where ISSU would be 151 applicable, do consist of redundant RP's and hardware-separated 152 control plane and data plane functionality, this document will focus 153 on methodologies which would be directly applicable to those 154 platforms. It is anticipated that the concepts and approaches 155 described herein may be readily extended to accommodate other device 156 architectures as well. 158 2. Conventions used in this document 160 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 161 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 162 document are to be interpreted as described in RFC-2119 [RFC2119]. 164 In this document, these words will appear with that interpretation 165 only when in ALL CAPS. Lower case uses of these words are not to be 166 interpreted as carrying RFC-2119 significance. 168 In this document, the characters ">>" preceding an indented line(s) 169 indicates a compliance requirement statement using the key words 170 listed above. This convention aids reviewers in quickly identifying 171 or finding the explicit compliance requirements of this RFC. 173 3. Generic ISSU Process, phased approach 175 ISSU may be viewed as the behavior of a device when exposed to a 176 planned change in its software functionality. This may mean changes 177 to the core operating system, separate processes or daemons or even 178 of firmware logic in programmable hardware devices (e.g. CPLD/FPGA). 179 The goal of an ISSU implementation is to permit such actions with 180 minimal or no disruption to the primary operation of the device in 181 question. 183 ISSU may be user initiated through direct interaction with the 184 device or activated through some automated process on a management 185 system or even on the device itself. For the purposes of this 186 document, we will focus on the model where the ISSU action is 187 initiated by direct user intervention. 189 The ISSU process can be viewed as a series of different phases or 190 activities, as defined below. For each of these phases, the test 191 operator MUST record the outcome as well as any relevant 192 observations (defined further in the present document). Note that, a 193 given vendor implementation may or may not permit the abortion of 194 the in-progress ISSU at particular stages. There may also be certain 195 restrictions as to ISSU availability given certain functional 196 configurations (for example, ISSU in the presence of Bidirectional 197 Failure Detection (BFD) [RFC 5880] may not be supported). It is 198 incumbent upon the test operator to ensure that the DUT is 199 appropriately configured to provide the appropriate test 200 environment. As with any properly orchestrated test effort, the test 201 plan document should reflect these and other relevant details and 202 SHOULD be written with close attention to the expected production- 203 operating environment. The combined analysis of the results of each 204 phase will characterize the overall ISSU process with the main goal 205 of being able to identify and quantify any disruption in service 206 (from the data and control plane perspective) allowing operators to 207 plan their maintenance activities with greater precision. 209 3.1. Software Download 211 In this first phase, the requested software package may be 212 downloaded to the router and is typically stored onto a device. The 213 downloading of software may be performed automatically by the device 214 as part of the upgrade process, or it may be initiated separately. 215 Such separation allows an administrator to download the new code 216 inside or outside of a maintenance window; it is anticipated that 217 downloading new code and saving it to disk on the router will not 218 impact operations. In the case where the software can be downloaded 219 outside of the actual upgrade process, the administrator SHOULD do 220 so; downloading software can skew timing results based on factors 221 that are often not comparative in nature. Internal compatibility 222 verification may be performed by the software running on the DUT, to 223 verify the checksum of the files downloaded as well as any other 224 pertinent checks. Depending upon vendor implementation, these 225 mechanisms may extend to include verification that the downloaded 226 module(s) meet a set of identified pre-requisites such as hardware 227 or firmware compatibility or minimum software requirements. Where 228 such mechanisms are made available by the product, they should be 229 verified, by the tester, with the perspective of avoiding 230 operational issues in production. Verification should include both 231 positive verification (ensuring that an ISSU action should be 232 permitted) as well as negative tests (creation of scenarios where 233 the verification mechanisms would report exceptions). 235 3.2. Software Staging 237 In this second phase, the requested software package is loaded in 238 the pertinent components of a given forwarding device (typically the 239 RP in standby state). Internal compatibility verification may be 240 performed by the software running on the DUT, as part of the upgrade 241 process itself, to verify the checksum of the files downloaded as 242 well as any other pertinent checks. Depending upon vendor 243 implementation, these mechanisms may extend to include verification 244 that the downloaded module(s) meet a set of identified pre- 245 requisites such as hardware or firmware compatibility or minimum 246 software requirements. Where such mechanisms are made available by 247 the product, they should be verified, by the tester (again with the 248 perspective of avoiding operational issues in production). In this 249 case, the execution of these checks is within scope of the upgrade 250 time, and SHOULD be included in the testing results. Once the new 251 software is downloaded to the pertinent components of the DUT, the 252 upgrade begins and the DUT begins to prepare itself for upgrade. 253 Depending on the vendor implementation, it is expected that 254 redundant hardware pieces within the DUT are upgraded, including the 255 backup or secondary RP. 257 3.3. Upgrade Run 259 In this phase, a switchover of RPs may take place, where one RP is 260 now upgraded with the new version of software. More importantly, the 261 "Upgrade Run" phase is where the internal changes made to 262 information and state stored on the router, on disk and in memory, 263 are either migrated to the "new" version of code, or 264 transformed/rebuilt to meet the standards of the new version of 265 code, and pushed onto the appropriate pieces of hardware. It is 266 within this phase that any outage(s) on the control or forwarding 267 plane may be expected to be observed. This is the critical phase of 268 the ISSU, where the control plane should not be impacted and any 269 interruptions to the forwarding plane should be minimal to none. For 270 some implementations, the above two steps may be concatenated into 271 one monolithic operation. In such case, the calculation of the 272 respective ISSU time intervals may need to be adapted accordingly. 274 If any control or data plane interruptions are observed within this 275 stage, they should be recorded as part of the results document. 277 3.4. Upgrade Acceptance 279 In this phase, the new version of software MUST be running in all 280 the physical nodes of the logical forwarding device. (RP's and LC's 281 as applicable). At this point, configuration control is returned to 282 the operator and normal device operation i.e. outside of ISSU- 283 oriented operation, is resumed. 285 4. Test Methodology 287 As stated by https://tools.ietf.org/html/rfc6815, the Test Topology 288 Setup must be part of an ITE (Isolated Test Environment) 290 The reporting of results MUST take into account the repeatability 291 considerations from Section 4 of [RFC2544]. It is RECOMMENDED to 292 perform multiple trials and report average results. The results are 293 reported in a simple statement including the measured frame loss and 294 ISSU impact times. 296 4.1. Test Topology 298 The hardware configuration of the DUT (Device Under test) SHOULD be 299 identical to the one expected to be or currently deployed in 300 production in order for the benchmark to have relevance. This would 301 include the number of RP's, hardware version, memory and initial 302 software release, any common chassis components, such as fabric 303 hardware in the case of a fabric-switching platform and the specific 304 LC's (version, memory, interfaces type, rate etc.) 306 For the Control and Data plane, differing configuration approached 307 MAY be utilized. The recommended approach relies on "mimicking" the 308 existing production data and control plane information, in order to 309 emulate all the necessary Layer1 through Layer3 communications and, 310 if appropriate, the upper layer characteristics of the network, as 311 well as end to end traffic/communication pairs. In other words, 312 design a representative load model of the production environment and 313 deploy a collapsed topology utilizing test tools and/or external 314 devices, where the DUT will be tested. Note that, the negative 315 impact of ISSU operations is likely to impact scaled, dynamic 316 topologies to a greater extent than simpler, static environments. As 317 such, this methodology (based upon production environments) is 318 advised for most test scenarios. 320 The second, more simplistic approach is to deploy an ITE "Isolated 321 Testing Environment" as described in some of the existing standards 322 for benchmarking methodologies (e.g. RFC2544/RFC6815) in which end- 323 points are "directly" connected to the DUT. In this manner control 324 plane information is kept to a minimum (only connected interfaces) 325 and only a basic data plane of sources and destinations is applied. 326 If this methodology is selected, care must be taken to understand 327 that the systemic behavior of the ITE may not be identical to that 328 experienced by a device in a production network role. That is, 329 control plane validation may be minimal to none if this methodology. 331 4.2. Load Model 333 In consideration of the defined test topology, a load model must be 334 developed to exercise the DUT while the ISSU event is introduced. 335 This applied load should be defined in such a manner as to provide a 336 granular, repeatable verification of the ISSU impact on transit 337 traffic. Sufficient traffic load (rate) should be applied to permit 338 timing extrapolations at a minimum granularity of 100 milliseconds 339 e.g. 100Mbps for a 10Gbps interface. The use of steady traffic 340 streams rather than bursty loads is preferred to simplify analysis. 342 The traffic should be patterned to provide a broad range of source 343 and destination pairs, which resolve to a variety of FIB (forwarding 344 information base) prefix lengths. If the production network 345 environment includes multicast traffic or VPN's (L2, L3 or IPSec) it 346 is critical to include these in the model. 348 For mixed protocol environments (e.g. IPv4 and IPv6), frames SHOULD 349 be distributed between the different protocols. The distribution 350 SHOULD approximate the network conditions of deployment. In all 351 cases, the details of the mixed protocol distribution MUST be 352 included in the reporting. 354 The feature, protocol timing and other relevant configurations 355 should be matched to the expected production environment. Deviations 356 from the production templates may be deemed necessary by the test 357 operator (for example, certain features may not support ISSU or the 358 test bed may not be able to accommodate such). However, the impact 359 of any such divergence should be clearly understood and the 360 differences MUST be recorded in the results documentation. It is 361 recommended that an NMS system be deployed, preferably similar to 362 that utilized in production. This will allow for monitoring of the 363 DUT while it is being tested both in terms of supporting the system 364 resource impact analysis as well as from the perspective of 365 detecting interference with non-transit (management) traffic as a 366 result of the ISSU operation. Additionally, a DUT management session 367 other than snmp-based, typical of usage in production, should be 368 established to the DUT and monitored for any disruption. It is 369 suggested that the actual test exercise be managed utilizing direct 370 console access to the DUT, if at all possible to avoid the 371 possibility that a network interruption impairs execution of the 372 test exercise. 374 All in all, the load model should attempt to simulate the production 375 network environment to the greatest extent possible in order to 376 maximize the applicability of the results generated. 378 5. ISSU Test Methodology 380 As previously described, for the purposes of this test document, the 381 ISSU process is divided into three main phases. The following 382 methodology assumes that a suitable test topology has been 383 constructed per section 4. A description of the methodology to be 384 applied for each of the above phases follows: 386 5.1. Pre-ISSU recommended verifications 388 1. Verify that enough hardware and software resources are available 389 to complete the Load operation (enough disk space). 391 2. Verify that the redundancy states between RPs and other nodes are 392 as expected (e.g. redundancy on, RP's synchronized). 394 3. Verify that the device, if running NSR capable routing protocols, 395 is in a "ready" state; that is, that the sync between RPs is 396 complete and the system is ready for failover, if necessary. 398 4. Gather a configuration snapshot of the device and all of its 399 applicable components. 401 5. Verify that the node is operating in a "steady" state (that is, 402 no critical or maintenance function is being currently performed). 404 6. Note any other operational characteristics that the tester may 405 deem applicable to the specific implementation deployed. 407 5.2. Software Staging 409 1. Establish all relevant protocol adjacencies and stabilize routing 410 within the test topology. In particular, ensure that the scaled 411 levels of the dynamic protocols are dimensioned as specified by the 412 test topology plan. 414 2. Clear relevant logs and interface counters to simplify analysis. 415 If possible, set logging timestamps to a highly granular mode. If 416 the topology includes management systems, ensure that the 417 appropriate polling levels have been applied, sessions established 418 and that the responses are per expectation. 420 3. Apply the traffic loads as specified in the load model previously 421 developed for this exercise. 423 4. Document an operational baseline for the test bed with relevant 424 data supporting the above steps (include all relevant load 425 characteristics of interest in the topology e.g. routing load, 426 traffic volumes, memory and CPU utilization) 428 5. Note the start time (T0) and begin the code change process 429 utilizing the appropriate mechanisms as expected to be used in 430 production (e.g. active download with TFTP/FTP/SCP/etc. or direct 431 install from local or external storage facility). In order to ensure 432 that ISSU process timings are not skewed by the lack of a network 433 wide synchronization source, the use of a network NTP source is 434 encouraged. 436 6. Take note of any logging information and command line interface 437 (CLI) prompts as needed (this detail will be vendor-specific) 438 Respond to any DUT prompts in a timely manner. 440 7. Monitor the DUT for the reload of secondary RP to the new 441 software level. Once the secondary has stabilized on the new code, 442 note the completion time. The duration of these steps will be 443 recorded as "T1". 445 8. Review system logs for any anomalies, check that relevant dynamic 446 protocols have remained stable and note traffic loss if any. Verify 447 that deployed management systems have not identified any unexpected 448 behavior. 450 5.3. Upgrade Run 452 The following assumes that the software load step and upgrade step 453 are discretely controllable. If not, maintain the afore-mentioned 454 timer and monitor for completion of the ISSU as described below. 456 1. Note the start time and initiate the actual upgrade procedure 458 2. Monitor the operation of the secondary route processor while it 459 initializes with the new software and assumes mastership of the DUT. 460 At this point, pay particular attention to any indications of 461 control plane disruption, traffic impact or other anomalous 462 behavior. Once the DUT has converged upon the new code and returned 463 to normal operation note the completion time and log the duration of 464 this step as T2. 466 3. Review the syslog data in the DUT and neighboring devices for any 467 behavior, which would be disruptive in a production environment 468 (linecard reloads, control plane flaps etc.). Examine the traffic 469 generators for any indication of traffic loss over this interval. If 470 the Test Set reported any traffic loss, note the number of frames 471 lost as "TP_frames". If the test set also provides outage duration, 472 note this as TP_time (alternatively this may be calculated as 473 TP/offered pps (packets per second) load). 475 4. Verify the DUT status observations as per any NMS systems 476 managing the DUT and its neighboring devices. Document the observed 477 CPU and memory statistics both during the ISSU upgrade event and 478 after and ensure that memory and CPU have returned to an expected 479 (previously baselined) level. 481 5.4. Post ISSU verification 483 The following describes a set of post-ISSU verification tasks that 484 are not directly part of the ISSU process, but are recommended for 485 execution in order to validate a successful upgrade: 487 1. Configuration delta analysis 489 Examine the post-ISSU configurations to determine if any changes 490 have occurred either through process error or due to differences 491 in the implementation of the upgraded code. 493 2. Exhaustive control plane analysis 494 Review the details of the RIB and FIB to assess whether any 495 unexpected changes have been introduced in the forwarding paths. 497 3. Verify that both RPs are up and that the redundancy mechanism for 498 the control plane is enabled and fully synchronized. 500 4. Verify that no control plane (protocol) events or flaps were 501 detected. 503 5. Verify that no L1 and or L2 interface flaps were observed. 505 6. Document the hitless operation or presence of an outage based 506 upon the counter values provided by the Test Set. 508 5.5. ISSU under negative stimuli 510 As an OPTIONAL Test Case, the operator may want to perform an ISSU 511 test while the DUT is under stress by introducing route churn to any 512 or all of the involved phases of the ISSU process. 514 One approach relies on the operator to gather statistical 515 information from the production environment and determine a specific 516 number of routes to flap every 'fixed' or 'variable' interval. 517 Alternatively, the operator may wish to simply pre-select a fixed 518 number of prefixes to flap. As an example, an operator may decide to 519 flap 1% of all the BGP routes every minute and restore them 1 minute 520 afterwards. The tester may wish to apply this negative stimulus 521 throughout the entire ISSU process or most importantly, during the 522 run phase. It is important to ensure that these routes, which are 523 introduced solely for stress proposes, must not overlap the ones 524 (per the Load Model) specifically leveraged to calculate the TP 525 (recorded outage). Furthermore, there SHOULD NOT be 'operator 526 induced' control plane - protocol adjacency flaps for the duration 527 of the test process as it may adversely affect the characterization 528 of the entire test exercise. For example, triggering IGP adjacency 529 events may force re-computation of underlying routing tables with 530 attendant impact to the perceived ISSU timings. While not 531 recommended, if such trigger events are desired by the test 532 operator, care should be taken to avoid the introduction of 533 unexpected anomalies within the test harness. 535 6. ISSU Abort and Rollback 537 Where a vendor provides such support, the ISSU process could be 538 aborted for any reason by the operator. However, the end results and 539 behavior may depend on the specific phase where the process was 540 aborted. While this is implementation dependent, as a general 541 recommendation, if the process is aborted during the "Software 542 Download" or "Software Staging" phases, no impact to service or 543 device functionality should be observed. In contrast, if the process 544 is aborted during the "Upgrade Run" or "Upgrade Accept" phases, the 545 system may reload and revert back to the previous software release 546 and as such, this operation may be service affecting. Where vendor 547 support is available, the abort/rollback functionality should be 548 verified and the impact, if any, quantified generally following the 549 procedures provided above. 551 7. Final Report - Data Presentation - Analysis 553 All ISSU impact results are summarized in a simple statement 554 describing the "ISSU Disruption Impact" including the measured frame 555 loss and impact time, where impact time is defined as the time frame 556 determined per the TP reported outage. These are considered to be 557 the primary data points of interest. 559 However, the entire ISSU operational impact should also be 560 considered in support of planning for maintenance and as such 561 additional reporting points are included. 563 Software download/secondary update T1 565 Upgrade/Run T2 567 ISSU Traffic Disruption (Frame Loss) TP_frames 569 ISSU Traffic Impact Time (milliseconds) TP Time 571 ISSU Housekeeping Interval T 573 (Time for both RP's up on new code and fully synced - Redundancy 574 restored) 576 Total ISSU Maintenance Window T4 (sum of T1+T2+T3) 578 The results reporting MUST provide the following information: 580 DUT hardware and software detail 582 Test Topology definition and diagram (especially as related to the 583 ISSU operation) 585 Load Model description including protocol mixes and any divergence 586 from the production environment 587 Time Results as per above 589 Anomalies Observed during ISSU 591 Anomalies Observed in post-ISSU analysis 593 It is RECOMMENDED that the following parameters be reported in these 594 units: 596 Parameter Units or Examples 598 --------------------------------------------------------------- 599 Traffic Load Frames per second and bits per Second 601 Disruption (average) Frames 603 Impact Time (average) Milliseconds 605 Number of trials Integer count 607 Protocols IPv4, IPv6, MPLS, etc. 609 Frame Size Octets 611 Port Media Ethernet, Gigabit Ethernet (GbE), 613 Packet over SONET (POS), etc. 615 Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc. 617 Interface Encap. Ethernet, Ethernet VLAN, 619 PPP, High-Level Data Link 621 Control(HDLC),etc. 623 Number of Prefixes Integer count 625 flapped (ON Interval) (Optional # of prefixes / Time 626 (minutes) 628 flapped (OFF Interval) (Optional # of prefixes / Time 629 (minutes) 631 Document any configuration deltas, which are observed after the 632 ISSU upgrade has taken effect. Note differences, which are driven 633 by changes in the patch or release level as well as items, which 634 are aberrant changes due to software faults. In either of these 635 cases, any unexpected behavioral changes should be analyzed and a 636 determination made as to the impact of the change (be it 637 functional variances or operational impacts to existing scripts or 638 management mechanisms. 640 7.1. Data collection considerations 642 When a DUT is undergoing an ISSU operation, it's worth noting that 643 the DUT's data collection and reporting of data, such as counters, 644 interface statistics, log messages, etc., might not be accurate. As 645 such, one SHOULD NOT rely on the DUTs data collection methods, but 646 rather, SHOULD use the test tools and equipment to collect data used 647 for reporting in Section 7. Care and consideration should be paid in 648 testing or adding new test cases, such that the desired data can be 649 collected from the test tools themselves, or other external 650 equipment, outside of the DUT itself. 652 8. Security Considerations 654 This Applicability Statement intends to help preserve the security 655 of the Internet by clarifying that the scope of this document and 656 other BMWG memos are all limited to testing in a laboratory ITE, 657 thus avoiding accidental Denial-of-Service attacks or congestion due 658 to high traffic volume test streams. All benchmarking activities are 659 limited to technology characterization using controlled stimuli in a 660 laboratory environment, with dedicated address space and the other 661 constraints [RFC2544]. 663 The benchmarking network topology will be an independent test setup 664 and MUST NOT be connected to devices that may forward the test 665 traffic into a production network or misroute traffic to the test 666 management network. 668 Further, benchmarking is performed on a "black-box" basis, relying 669 solely on measurements observable external to the device under test/ 670 system under test (DUT/SUT). 672 Special capabilities SHOULD NOT exist in the DUT/SUT specifically 673 for benchmarking purposes. Any implications for network security 674 arising from the DUT/SUT SHOULD be identical in the lab and in 675 production networks. 677 9. IANA Considerations 679 There are no IANA actions required by this memo. 681 10. References 683 10.1. Normative References 685 [1] Bradner, S., "Key words for use in RFCs to Indicate 686 Requirement Levels", BCP 14, RFC 2119, March 1997. 688 [2] Crocker, D. and Overell, P.(Editors), "Augmented BNF for 689 Syntax Specifications: ABNF", RFC 2234, Internet Mail 690 Consortium and Demon Internet Ltd., November 1997. 692 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 693 Requirement Levels", BCP 14, RFC 2119, March 1997. 695 [RFC2234] Crocker, D. and Overell, P.(Editors), "Augmented BNF for 696 Syntax Specifications: ABNF", RFC 2234, Internet Mail 697 Consortium and Demon Internet Ltd., November 1997. 699 10.2. Informative References 701 [3] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in TCP 702 and Its Effect on Busy Servers", Proc. Infocom 1999 pp. 1573- 703 1583. 705 [Fab1999] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in 706 TCP and Its Effect on Busy Servers", Proc. Infocom 1999 707 pp. 1573-1583. 709 11. Acknowledgments 711 The authors wish to thanks Vibin Thomas for his valued review and 712 feedback. 714 Authors' Addresses 716 Sarah Banks 717 VSS Monitoring 718 Email: sbanks@encrypted.net 720 Fernando Calabria 721 Cisco Systems 722 Email: fcalabri@cisco.com 724 Gery Czirjak 725 Juniper Networks 726 Email: gczirjak@juniper.net 728 Ramdas Machat 729 Juniper Networks 730 Email: rmachat@juniper.net