idnits 2.17.1 draft-ietf-bmwg-2544-as-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. -- The draft header indicates that this document updates RFC2544, but the abstract doesn't seem to directly say this. It does mention RFC2544 though, so this could be OK. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 326 has weird spacing: '...Bradner serv...' == Line 328 has weird spacing: '... Dubray ser...' (Using the creation date from RFC2544, updated by this document, for RFC5378 checks: 1999-03-01) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (September 18, 2012) is 4236 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group S. Bradner 3 Internet-Draft Harvard University 4 Updates: 2544 (if approved) K. Dubray 5 Intended status: Informational Juniper Networks 6 Expires: March 22, 2013 J. McQuaid 7 Turnip Video 8 A. Morton 9 AT&T Labs 10 September 18, 2012 12 RFC 2544 Applicability Statement: 13 Use on Production Networks Considered Harmful 14 draft-ietf-bmwg-2544-as-07 16 Abstract 18 Benchmarking Methodology Working Group (BMWG) has been developing key 19 performance metrics and laboratory test methods since 1990, and 20 continues this work at present. The methods described in RFC 2544 21 are intended to generate traffic that overloads network device 22 resources in order to assess their capacity. Overload of shared 23 resources would likely be harmful to user traffic performance on a 24 production network, and there are further negative consequences 25 identified with production application of the methods. This memo 26 clarifies the scope of RFC 2544 and other IETF BMWG benchmarking work 27 for isolated test environments only, and encourages new standards 28 activity for measurement methods applicable outside that scope. 30 Status of this Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF). Note that other groups may also distribute 37 working documents as Internet-Drafts. The list of current Internet- 38 Drafts is at http://datatracker.ietf.org/drafts/current/. 40 Internet-Drafts are draft documents valid for a maximum of six months 41 and may be updated, replaced, or obsoleted by other documents at any 42 time. It is inappropriate to use Internet-Drafts as reference 43 material or to cite them other than as "work in progress." 45 This Internet-Draft will expire on March 22, 2013. 47 Copyright Notice 48 Copyright (c) 2012 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (http://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with respect 56 to this document. Code Components extracted from this document must 57 include Simplified BSD License text as described in Section 4.e of 58 the Trust Legal Provisions and are provided without warranty as 59 described in the Simplified BSD License. 61 Table of Contents 63 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 64 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 65 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . 4 66 3. The Concept of an Isolated Test Environment . . . . . . . . . 4 67 4. Why RFC 2544 Methods are intended only for ITE . . . . . . . . 4 68 4.1. Experimental Control and Accuracy . . . . . . . . . . . . 4 69 4.2. Containing Damage . . . . . . . . . . . . . . . . . . . . 5 70 5. Advisory on RFC 2544 Methods in Production Networks . . . . . 5 71 6. Considering Performance Testing in Production Networks . . . . 6 72 7. Security Considerations . . . . . . . . . . . . . . . . . . . 7 73 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 74 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 8 75 10. Appendix - Example of RFC 2544 method failure in 76 production network measurement . . . . . . . . . . . . . . . . 8 77 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 9 78 11.1. Normative References . . . . . . . . . . . . . . . . . . . 9 79 11.2. Informative References . . . . . . . . . . . . . . . . . . 10 80 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 10 82 1. Introduction 84 This memo clarifies the scope and use of IETF Benchmarking 85 Methodology Working Group (BMWG) tests including [RFC2544], which 86 discusses and defines several tests that may be used to characterize 87 the performance of a network interconnecting device. All readers of 88 this memo must read and fully understand [RFC2544]. 90 Benchmarking methodologies (beginning with [RFC2544]) have always 91 relied on test conditions that can only be produced and replicated 92 reliably in the laboratory. These methodologies are not appropriate 93 for inclusion in wider specifications such as: 95 1. Validation of telecommunication service configuration, such as 96 the Committed Information Rate (CIR). 98 2. Validation of performance metrics in a telecommunication Service 99 Level Agreement (SLA), such as frame loss and latency. 101 3. Telecommunication service activation testing, where traffic that 102 shares network resources with the test might be adversely 103 affected. 105 Above, we distinguish "telecommunication service" (where a network 106 service provider contracts with a customer to transfer information 107 between specified interfaces at different geographic locations) from 108 the generic term "service". Below, we use the adjective "production" 109 to refer to networks carrying live user traffic. [RFC2544] used the 110 term "real-world" to refer to production networks and to 111 differentiate them from test networks. 113 Although RFC 2544 has been held up as the standard reference for such 114 testing, we believe that the actual methods used vary from [RFC2544] 115 in significant ways. Since the only citation is to [RFC2544], the 116 modifications are opaque to the standards community and to users in 117 general. 119 Since applying the test traffic and methods described in [RFC2544] on 120 a production network risks causing overload in shared resources there 121 is direct risk of harming user traffic if the methods are misused in 122 this way. Therefore, IETF BMWG developed this Applicability 123 Statement for [RFC2544] to directly address the situation. 125 1.1. Requirements Language 127 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 128 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 129 document are to be interpreted as described in RFC 2119 [RFC2119]. 131 2. Scope and Goals 133 This memo clarifies the scope of [RFC2544] with the goal to provide 134 guidance to the industry on its applicability, which is limited to 135 laboratory testing. 137 3. The Concept of an Isolated Test Environment 139 An Isolated Test Environment (ITE) used with [RFC2544] methods (as 140 illustrated in Figures 1 through 3 of [RFC2544]) has the ability to: 142 o contain the test streams to paths within the desired set-up 144 o prevent non-test traffic from traversing the test set-up 146 These features allow unfettered experimentation, while at the same 147 time protecting lab equipment management/control LANs and other 148 production networks from the unwanted effects of the test traffic. 150 4. Why RFC 2544 Methods are intended only for ITE 152 The following sections discuss some of the reasons why [RFC2544] 153 methods are applicable only for isolated laboratory use, and the 154 consequences of applying these methods outside the lab environment. 156 4.1. Experimental Control and Accuracy 158 All of the tests described in RFC 2544 require that the tester and 159 device under test are the only devices on the networks that are 160 transmitting data. The presence of other traffic (unwanted on the 161 ITE network) would mean that the specified test conditions have not 162 been achieved and flawed results are a likely consequence. 164 If any other traffic appears and the amount varies over time, the 165 repeatability of any test result will likely depend to some degree on 166 the amount and variation of the other traffic. 168 The presence of other traffic makes accurate, repeatable, and 169 consistent measurements of the performance of the device under test 170 very unlikely, since the complete details of test conditions will not 171 be reported. 173 For example, the RFC 2544 Throughput Test attempts to characterize a 174 maximum reliable load, thus there will be testing above the maximum 175 that causes packet/frame loss. Any other sources of traffic on the 176 network will cause packet loss to occur at a tester data rate lower 177 than the rate that would be achieved without the extra traffic. 179 4.2. Containing Damage 181 [RFC2544] methods, specifically to determine Throughput as defined in 182 [RFC1242] and other benchmarks, may overload the resources of the 183 device under test, and may cause failure modes in the device under 184 test. Since failures can become the root cause of more wide-spread 185 failure, it is clearly desirable to contain all test traffic within 186 the ITE. 188 In addition, such testing can have a negative effect on any traffic 189 that shares resources with the test stream(s) since, in most cases, 190 the traffic load will be close to the capacity of the network links. 192 Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private 193 IPv4 address range for testing: 195 "...The network addresses 198.18.0.0 through 198.19.255.255 have been 196 assigned to the BMWG by the IANA for this purpose. This assignment 197 was made to minimize the chance of conflict in case a testing device 198 were to be accidentally connected to part of the Internet. The 199 specific use of the addresses is detailed below." 201 In other words, devices operating on the Internet may be configured 202 to discard any traffic they observe in this address range, as it is 203 intended for laboratory ITE use only. Thus, if testers using the 204 assigned testing address ranges are connected to the Internet and 205 test packets are forwarded across the Internet, it is likely that the 206 packets will be discarded and the test will not work. 208 We note that a range of IPv6 addresses has been assigned to BMWG for 209 laboratory test purposes, in [RFC5180] (as amended by errata). 211 See the Security Considerations Section below for further 212 considerations on containing damage. 214 5. Advisory on RFC 2544 Methods in Production Networks 216 The tests in [RFC2544] were designed to measure the performance of 217 network devices, not of networks, and certainly not production 218 networks carrying user traffic on shared resources. There will be 219 undesirable consequences when applying these methods outside the 220 isolated test environment. 222 One negative consequence stems from reliance on frame loss as an 223 indicator of resource exhaustion in [RFC2544] methods. In practice, 224 link-layer and physical-layer errors prevent production networks from 225 operating loss-free. The [RFC2544] methods will not correctly assess 226 Throughput when loss from uncontrolled sources is present. Frame 227 loss occurring at the SLA levels of some networks could affect every 228 iteration of Throughput testing (when each step includes sufficient 229 packets to experience facility-related loss). Flawed results waste 230 the time and resources of the testing service user and of the service 231 provider when called to dispute the measurement. These are 232 additional examples of harm that compliance with this advisory should 233 help to avoid. See the Appendix for an example. 235 The methods described in [RFC2544] are intended to generate traffic 236 that overloads network device resources in order to assess their 237 capacity. Overload of shared resources would likely be harmful to 238 user traffic performance on a production network. These tests MUST 239 NOT be used on production networks and as discussed above. The tests 240 will not produce a reliable or accurate benchmarking result on a 241 production network. 243 [RFC2544] methods have never been validated on a network path, even 244 when that path is not part of a production network and carrying no 245 other traffic. It is unknown whether the tests can be used to 246 measure valid and reliable performance of a multi-device, multi- 247 network path. It is possible that some of the tests may prove valid 248 in some path scenarios, but that work has not been done or has not 249 been shared with the IETF community. Thus, such testing is contra- 250 indicated by the BMWG. 252 6. Considering Performance Testing in Production Networks 254 The IETF has addressed the problem of production network performance 255 measurement by chartering a different working group: IP Performance 256 Metrics (IPPM). This working group has developed a set of standard 257 metrics to assess the quality, performance, and reliability of 258 Internet packet transfer services. These metrics can be measured by 259 network operators, end users, or independent testing groups. We note 260 that some IPPM metrics differ from RFC 2544 metrics with similar 261 names, and there is likely to be confusion if the details are 262 ignored. 264 IPPM has not yet standardized methods for raw capacity measurement of 265 Internet paths. Such testing needs to adequately consider the strong 266 possibility for degradation to any other traffic that may be present 267 due to congestion. There are no specific methods proposed for 268 activation of a packet transfer service in IPPM at this time. Thus, 269 individuals who need to conduct capacity tests on production networks 270 should actively participate in standards development to ensure their 271 methods receive appropriate industry review and agreement, in the 272 IETF or in alternate standards development organizations. 274 Other standards may help to fill gaps in telecommunication service 275 testing. For example, the IETF has many standards intended to assist 276 with network operation, administration and maintenance (OAM), and 277 ITU-T Study Group 12 has a Recommendation on service activation test 278 methodology [Y.1564]. 280 The world will not spin off axis while waiting for appropriate and 281 standardized methods to emerge from the consensus process. 283 7. Security Considerations 285 This Applicability Statement intends to help preserve the security of 286 the Internet by clarifying that the scope of [RFC2544] and other BMWG 287 memos are all limited to testing in a laboratory ITE, thus avoiding 288 accidental Denial of Service attacks or congestion due to high 289 traffic volume test streams. 291 All Benchmarking activities are limited to technology 292 characterization using controlled stimuli in a laboratory 293 environment, with dedicated address space and the other constraints 294 [RFC2544]. 296 The benchmarking network topology will be an independent test setup 297 and MUST NOT be connected to devices that may forward the test 298 traffic into a production network, or misroute traffic to the test 299 management network. 301 Further, benchmarking is performed on a "black-box" basis, relying 302 solely on measurements observable external to the device under test/ 303 system under test (DUT/SUT). 305 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 306 benchmarking purposes. Any implications for network security arising 307 from the DUT/SUT SHOULD be identical in the lab and in production 308 networks. 310 8. IANA Considerations 312 This memo makes no requests of IANA. 314 9. Acknowledgements 316 Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, Curtis 317 Villamizar, David Newman, and Adrian Farrel for suggesting 318 improvements to this memo. 320 Specifically, Al Morton would like to thank his co-authors, who 321 constitute the complete set of Chairmen-Emeritus of the BMWG, for 322 returning from other pursuits to develop this statement and see it 323 through to approval. This has been a rare privilege; one that likely 324 will not be matched in the IETF again: 326 Scott Bradner served as Chairman from 1990 to 1993 327 Jim McQuaid served as Chairman from 1993 to 1995 328 Kevin Dubray served as Chairman from 1995 to 2006 330 It's all about the band. 332 10. Appendix - Example of RFC 2544 method failure in production network 333 measurement 335 This Appendix provides an example illustrating how [RFC2544] methods 336 applied on production networks can easily produce a form of harm from 337 flawed and misleading results. 339 The [RFC2544] Throughput benchmarking method usually includes the 340 following steps: 342 a. Set the offered traffic level, less than max of the ingress 343 link(s). 345 b. Send the test traffic through the device under test (DUT) and 346 count all frames successfully transferred. 348 c. If all frames are received, increment traffic level and repeat 349 step b. 351 d. If one or more frames are lost, the level is in the DUT-overload 352 region (Step b may be repeated at a reduced traffic level to more 353 exactly determine the maximum rate at which none of the frames are 354 dropped by the DUT, defined as the Throughput [RFC1242]). 356 e. Report the Throughput values, the x-y of graph of frame size and 357 Throughput, and other information in accordance with [RFC2544]. 359 In this method, frame loss is the sole indicator of overload and 360 therefore the determining factor in the measurement of Throughput 361 using the [RFC2544] methodology (even though the results may not 362 report frame loss per se). 364 Frame loss is subject to many factors in addition to operating above 365 the Throughput traffic level. These factors include optical 366 interference (which may be due to dirty interfaces, cross-over from 367 other signals, fiber bend and temperature, etc.) and electrical 368 interference (caused by local sources of radio signals, electrical 369 spikes, solar particles, etc.). In the laboratory environment many 370 of these issues can be carefully controlled through cleaning and 371 isolation. Since [RFC2544] methodologies are primarily intended to 372 test devices and not paths, the total length of path, the number of 373 interfaces, and compound risk of random frame loss can be kept to a 374 minimum. 376 In a production network, however, there will be many interfaces and 377 many kilometres of path under test. This considerably increases the 378 risk of random frame loss. 380 The risk of frame loss caused by outside effects is significantly 381 higher in production networks, and significantly higher with long 382 paths (both those with long physical path lengths, and those with 383 large numbers of interfaces in the path). Thus, the risk of falsely 384 low reported Throughput using an [RFC2544] methodology test is 385 considerably increased in a production network. 387 Therefore, to successfully conduct tests with similar objectives to 388 those in [RFC2544] in a production network, it will be necessary to 389 develop modifications to the methodologies defined in [RFC2544] and 390 standards to describe them. 392 11. References 394 11.1. Normative References 396 [RFC1242] Bradner, S., "Benchmarking terminology for network 397 interconnection devices", RFC 1242, July 1991. 399 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 400 Requirement Levels", BCP 14, RFC 2119, March 1997. 402 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 403 Network Interconnect Devices", RFC 2544, March 1999. 405 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. 406 Dugatkin, "IPv6 Benchmarking Methodology for Network 407 Interconnect Devices", RFC 5180, May 2008. 409 11.2. Informative References 411 [Y.1564] ITU-T Recommendation Y.1564, "Ethernet Service Activation 412 Test Methodology", March 2011. 414 Authors' Addresses 416 Scott Bradner 417 Harvard University 418 29 Oxford St. 419 Cambridge, MA 02138 420 USA 422 Phone: +1 617 495 3864 423 Fax: 424 Email: sob@harvard.edu 425 URI: http://www.sobco.com 427 Kevin Dubray 428 Juniper Networks 430 Phone: 431 Fax: 432 Email: kdubray@juniper.net 433 URI: 435 Jim McQuaid 436 Turnip Video 437 6 Cobbleridge Court 438 Durham, North Carolina 27713 439 USA 441 Phone: +1 919-619-3220 442 Fax: 443 Email: jim@turnipvideo.com 444 URI: www.turnipvideo.com 445 Al Morton 446 AT&T Labs 447 200 Laurel Avenue South 448 Middletown,, NJ 07748 449 USA 451 Phone: +1 732 420 1571 452 Fax: +1 732 368 1192 453 Email: acmorton@att.com 454 URI: http://home.comcast.net/~acmacm/