idnits 2.17.1 draft-ietf-bmwg-2544-as-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 11, 2012) is 4270 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group S. Bradner 3 Internet-Draft Harvard University 4 Intended status: Informational K. Dubray 5 Expires: February 12, 2013 Juniper Networks 6 J. McQuaid 7 Turnip Video 8 A. Morton 9 AT&T Labs 10 August 11, 2012 12 RFC 2544 Applicability Statement: Use on Production Networks Considered 13 Harmful 14 draft-ietf-bmwg-2544-as-05 16 Abstract 18 Benchmarking Methodology Working Group (BMWG) has been developing key 19 performance metrics and laboratory test methods since 1990, and 20 continues this work at present. Recent application of the methods 21 beyond their intended scope is cause for concern. The methods 22 described in RFC 2544, where overload is a possible outcome, would no 23 doubt be harmful to user traffic performance on a production network. 24 This memo clarifies the scope of RFC 2544 and other benchmarking work 25 for the IETF community. 27 Status of this Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at http://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on February 12, 2013. 44 Copyright Notice 46 Copyright (c) 2012 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (http://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 62 1.1. Requirements Language . . . . . . . . . . . . . . . . . . . 3 63 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . . 4 64 3. The Concept of an Isolated Test Environment . . . . . . . . . . 4 65 4. Why RFC 2544 Methods are intended for ITE . . . . . . . . . . . 4 66 4.1. Experimental Control and Accuracy . . . . . . . . . . . . . 4 67 4.2. Containing Damage . . . . . . . . . . . . . . . . . . . . . 5 68 5. Advisory on RFC 2544 Methods in Production Networks . . . . . . 5 69 6. What to do without RFC 2544? . . . . . . . . . . . . . . . . . 6 70 7. Security Considerations . . . . . . . . . . . . . . . . . . . . 7 71 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7 72 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 7 73 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 8 74 10.1. Normative References . . . . . . . . . . . . . . . . . . . 8 75 10.2. Informative References . . . . . . . . . . . . . . . . . . 8 76 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 8 78 1. Introduction 80 This memo clarifies the scope of RFC 2544 [RFC2544], which discusses 81 and defines several tests that may be used to characterize the 82 performance of a network interconnecting device, and other 83 benchmarking work for the IETF community. 85 Benchmarking Methodologies (beginning with [RFC2544]) have always 86 relied on test conditions that can only be produced and replicated 87 reliably in the laboratory. Thus it was unfortunate to find that 88 this foundation methodology was being cited in several unintended 89 specifications and products performing applications such as: 91 1. Validation of telecommunication service configuration, such as 92 the Committed Information Rate (CIR). 94 2. Validation of performance metrics in a telecommunication Service 95 Level Agreement (SLA), such as frame loss and latency. 97 3. Telecommunication service activation testing, where traffic that 98 shares network resources with the test might be adversely 99 affected. 101 Above, we distinguish "telecommunication service" (where a network 102 service provider contracts with a customer to transfer information 103 between specified interfaces at different geographic locations) from 104 the generic term "service". Below, we use the adjective "production" 105 to refer to networks carrying live user traffic. [RFC2544] used the 106 term "real-world" to refer to production networks and to 107 differentiate them from test networks. 109 Although RFC 2544 is held up as the standard reference for such 110 testing, we believe that the actual methods used vary from RFC 2544 111 in significant ways. Since the only citation is to RFC 2544, the 112 modifications are opaque to the standards community and to users in 113 general (an undesirable situation). There is risk of harm to user 114 traffic from applying the test traffic and methods described in 115 [RFC2544] on a production network, because overload in shared 116 resources is a possible outcome. 118 To directly address this situation, the past and present Chairs of 119 the IETF Benchmarking Methodology Working Group (BMWG) have prepared 120 this Applicability Statement for RFC 2544. 122 1.1. Requirements Language 124 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 125 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 126 document are to be interpreted as described in RFC 2119 [RFC2119]. 128 2. Scope and Goals 130 This memo clarifies the scope of [RFC2544], with the goal to provide 131 guidance to the community on its applicability, which is limited to 132 laboratory testing. 134 3. The Concept of an Isolated Test Environment 136 An Isolated Test Environment (ITE) used with [RFC2544] methods (as 137 illustrated in Figures 1 through 3 of [RFC2544]) has the ability to: 139 o contain the test streams to paths within the desired set-up 141 o prevent non-test traffic from traversing the test set-up 143 These features allow unfettered experimentation, while at the same 144 time protecting lab equipment management/control LANs and other 145 production networks from the unwanted effects of the test traffic. 147 4. Why RFC 2544 Methods are intended for ITE 149 The following sections discuss some of the reasons why RFC 2544 150 [RFC2544] methods were intended only for isolated laboratory use, and 151 the difficulties of applying these methods outside the lab 152 environment. 154 4.1. Experimental Control and Accuracy 156 All of the tests described in RFC 2544 require that the tester and 157 device under test are the only devices on the networks that are 158 transmitting data. The presence of other unwanted traffic on the 159 network would mean that the specified test conditions have not been 160 achieved. 162 If any unwanted traffic appears and the amount varies over time, the 163 repeatability of any test result will likely depend to some degree on 164 the unwanted traffic. 166 The presence of unwanted or unknown traffic makes accurate, 167 repeatable, and consistent measurements of the performance of the 168 device under test very unlikely, since the complete details of test 169 conditions will not be reported. 171 For example, the RFC 2544 Throughput Test attempts to characterize a 172 maximum reliable load, thus there will be testing above the maximum 173 that causes packet/frame loss. Any other sources of traffic on the 174 network will cause packet loss to occur at a tester data rate lower 175 than the rate that would be achieved without the extra traffic. 177 4.2. Containing Damage 179 RFC 2544 methods, specifically to determine Throughput as defined in 180 [RFC1242] and other benchmarks, may overload the resources of the 181 device under test, and may cause failure modes in the device under 182 test. Since failures can become the root cause of more wide-spread 183 failure, it is clearly desirable to contain all test traffic within 184 the ITE. 186 In addition, such testing can have a negative effect on any traffic 187 that shares resources with the test stream(s) since, in most cases, 188 the traffic load will be close to the capacity of the network links. 190 Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private 191 IPv4 address range for testing: 193 "...The network addresses 198.18.0.0 through 198.19.255.255 have been 194 assigned to the BMWG by the IANA for this purpose. This assignment 195 was made to minimize the chance of conflict in case a testing device 196 were to be accidentally connected to part of the Internet. The 197 specific use of the addresses is detailed below." 199 In other words, devices operating on the Internet may be configured 200 to discard any traffic they observe in this address range, as it is 201 intended for laboratory ITE use only. Thus, testers using the 202 assigned testing address ranges MUST NOT be connected to the 203 Internet. 205 We note that a range of IPv6 addresses has been assigned to BMWG for 206 laboratory test purposes, in [RFC5180] (as amended by errata). Also, 207 the strong statements in the Security Considerations Section of this 208 memo make the scope even more clear; this is now a standard fixture 209 of all BMWG memos. 211 5. Advisory on RFC 2544 Methods in Production Networks 213 The tests in [RFC2544] were designed to measure the performance of 214 network devices, not of networks, and certainly not production 215 networks carrying user traffic on shared resources. There will be 216 unanticipated difficulties when applying these methods outside the 217 lab environment. 219 One such difficulty stems from reliance on frame loss as an indicator 220 of resource exhaust in RFC 2544 methods. In practice, link-layer and 221 physical-layer errors prevent production networks from operating 222 loss-free. The RFC 2544 methods will not correctly assess Throughput 223 when loss from uncontrolled sources is present. Frame loss occurring 224 at the SLA levels of some networks could affect every iteration of 225 Throughput testing (when each step includes sufficient packets to 226 experience facility-related loss). Flawed results waste the time and 227 resources of the testing service user, and of the service provider 228 when called to dispute the measurement. These are additional 229 examples of harm that compliance with this advisory should help to 230 avoid. 232 Operating test equipment on production networks according to the 233 methods described in [RFC2544], where overload is a possible outcome, 234 would no doubt be harmful to user traffic performance. These tests 235 MUST NOT be used on production networks and as discussed above, the 236 tests will never produce a reliable or accurate benchmarking result 237 on a production network. 239 [RFC2544] methods have never been validated on a network path, even 240 when that path is not part of a production network and carrying no 241 other traffic. It is unknown whether the tests can be used to 242 measure valid and reliable performance of a multi-device, multi- 243 network path. It is possible that some of the tests may prove valid 244 in some path scenarios, but that work has not been done or has not 245 been shared with the IETF community. Thus, such testing is contra- 246 indicated by the BMWG. 248 6. What to do without RFC 2544? 250 The IETF has addressed the problem of production network performance 251 measurement by chartering a different working group: IP Performance 252 Metrics (IPPM). This working group has developed a set of standard 253 metrics to assess the quality, performance, and reliability of 254 Internet packet transfer services. These metrics can be measured by 255 network operators, end users, or independent testing groups. We note 256 that some IPPM metrics differ from RFC 2544 metrics with similar 257 names, and there is likely to be confusion if the details are 258 ignored. 260 IPPM has not yet standardized methods for raw capacity measurement of 261 Internet paths. Such testing needs to adequately consider the strong 262 possibility for degradation to any other traffic that may be present 263 due to congestion. There are no specific methods proposed for 264 activation of a packet transfer service in IPPM. 266 Other standards may help to fill gaps in telecommunication service 267 testing. For example, the IETF has many standards intended to assist 268 with network operation, administration and maintenance (OAM), and 269 ITU-T Study Group 12 has a recommendation on service activation test 270 methodology [Y.1564]. 272 The world will not spin off axis while waiting for appropriate and 273 standardized methods to emerge from the consensus process. 275 7. Security Considerations 277 This Applicability Statement intends to help preserve the security of 278 the Internet by clarifying that the scope of [RFC2544] and other BMWG 279 memos are all limited to testing in a laboratory ITE, thus avoiding 280 accidental Denial of Service attacks or congestion due to high 281 traffic volume test streams. 283 All Benchmarking activities are limited to technology 284 characterization using controlled stimuli in a laboratory 285 environment, with dedicated address space and the other constraints 286 [RFC2544]. 288 The benchmarking network topology will be an independent test setup 289 and MUST NOT be connected to devices that may forward the test 290 traffic into a production network, or misroute traffic to the test 291 management network. 293 Further, benchmarking is performed on a "black-box" basis, relying 294 solely on measurements observable external to the device under test/ 295 system under test (DUT/SUT). 297 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 298 benchmarking purposes. Any implications for network security arising 299 from the DUT/SUT SHOULD be identical in the lab and in production 300 networks. 302 8. IANA Considerations 304 This memo makes no requests of IANA. 306 9. Acknowledgements 308 Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, Curtis 309 Villamizar, and David Newman for reading and suggesting improvements 310 to this memo. 312 10. References 314 10.1. Normative References 316 [RFC1242] Bradner, S., "Benchmarking terminology for network 317 interconnection devices", RFC 1242, July 1991. 319 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 320 Requirement Levels", BCP 14, RFC 2119, March 1997. 322 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 323 Network Interconnect Devices", RFC 2544, March 1999. 325 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. 326 Dugatkin, "IPv6 Benchmarking Methodology for Network 327 Interconnect Devices", RFC 5180, May 2008. 329 10.2. Informative References 331 [Y.1564] ITU-T Recommendation Y.1564, "Ethernet Service Activation 332 Test Methodology", March 2011. 334 Authors' Addresses 336 Scott Bradner 337 Harvard University 338 29 Oxford St. 339 Cambridge, MA 02138 340 USA 342 Phone: +1 617 495 3864 343 Fax: 344 Email: sob@harvard.edu 345 URI: http://www.sobco.com 347 Kevin Dubray 348 Juniper Networks 350 Phone: 351 Fax: 352 Email: kdubray@juniper.net 353 URI: 355 Jim McQuaid 356 Turnip Video 357 6 Cobbleridge Court 358 Durham, North Carolina 27713 359 USA 361 Phone: +1 919-619-3220 362 Fax: 363 Email: jim@turnipvideo.com 364 URI: www.turnipvideo.com 366 Al Morton 367 AT&T Labs 368 200 Laurel Avenue South 369 Middletown,, NJ 07748 370 USA 372 Phone: +1 732 420 1571 373 Fax: +1 732 368 1192 374 Email: acmorton@att.com 375 URI: http://home.comcast.net/~acmacm/