idnits 2.17.1 draft-ietf-bmwg-2544-as-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 6, 2011) is 4644 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2679' is defined on line 302, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) Summary: 1 error (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group S. Bradner 3 Internet-Draft Harvard University 4 Intended status: Informational K. Dubray 5 Expires: February 7, 2012 Juniper Networks 6 J. McQuaid 7 Turnip Video 8 A. Morton 9 AT&T Labs 10 August 6, 2011 12 RFC 2544 Applicability Statement: Use on Real-World Networks Considered 13 Harmful 14 draft-ietf-bmwg-2544-as-00 16 Abstract 18 Benchmarking Methodology Working Group (BMWG) has been developing key 19 performance metrics and laboratory test methods since 1990, and 20 continues this work at present. Recent application of the methods 21 beyond their intended scope is cause for concern. This memo 22 clarifies the scope of RFC 2544 and other benchmarking work for the 23 IETF community. 25 Status of this Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at http://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on February 7, 2012. 42 Copyright Notice 44 Copyright (c) 2011 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 1.1. Requirements Language . . . . . . . . . . . . . . . . . . . 3 61 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . . 3 62 3. The Concept of an Isolated Test Environment . . . . . . . . . . 4 63 4. Why RFC 2544 Methods are intended for ITE . . . . . . . . . . . 4 64 4.1. Experimental Control, Repeatability, and Accuracy . . . . . 4 65 4.2. Containment of Implementation Failure Impact . . . . . . . 5 66 5. Advisory on RFC 2544 Methods in Real-world Networks . . . . . . 5 67 6. What to do without RFC 2544? . . . . . . . . . . . . . . . . . 6 68 7. Security Considerations . . . . . . . . . . . . . . . . . . . . 6 69 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7 70 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 7 71 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 7 72 10.1. Normative References . . . . . . . . . . . . . . . . . . . 7 73 10.2. Informative References . . . . . . . . . . . . . . . . . . 8 74 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 8 76 1. Introduction 78 This memo clarifies the scope of RFC 2544 [RFC2544], and other 79 benchmarking work for the IETF community. 81 Benchmarking Methodologies (beginning with [RFC2544]) have always 82 relied on test conditions that can only be reliably produced in the 83 laboratory. Thus it was surprising to find that this foundation 84 methodology was being cited in several unintended applications, such 85 as: 87 1. Validation of telecommunication service configuration, such as 88 the Committed Information Rate (CIR). 90 2. Validation of performance metrics in a telecommunication Service 91 Level Agreement (SLA), such as frame loss and latency. 93 3. As an integral part of telecommunication service activation 94 testing, where traffic that shares network resources with the 95 test might be adversely affected. 97 Above, we distinguish "telecommunication service" (where a network 98 service provider contracts with a customer to transfer information 99 between specified interfaces at different geographic locations in the 100 real world) from the generic term "service". Also, we use the term 101 "real-world networks" to refer to production networks carrying live 102 user traffic. 104 Although RFC 2544 is held up as the standard reference for such 105 testing, we believe that the actual methods used vary from RFC 2544 106 in significant ways. Since the only citation is to RFC 2544, the 107 modifications are opaque to the standards community and to users in 108 general (an undesirable situation). 110 To directly address this situation, the past and present Chairs of 111 the IETF Benchmarking Methodology Working Group (BMWG) have prepared 112 this Applicability Statement for RFC 2544. 114 1.1. Requirements Language 116 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 117 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 118 document are to be interpreted as described in RFC 2119 [RFC2119]. 120 2. Scope and Goals 122 This memo clarifies the scope of [RFC2544], with the goal to provide 123 guidance to the community on its applicability, which is limited to 124 laboratory testing. 126 3. The Concept of an Isolated Test Environment 128 An Isolated Test Environment (ITE) used with [RFC2544] methods (as 129 illustrated in Figures 1 through 3 of [RFC2544])has the ability to: 131 o contain the test streams to paths within the desired set-up 133 o prevent non-test traffic from traversing the test set-up 135 These features allow unfettered experimentation, while at the same 136 time protecting equipment management LANs and other production 137 networks from the unwanted effects of the test traffic. 139 4. Why RFC 2544 Methods are intended for ITE 141 The following sections discuss some of the reasons why RFC 2544 142 [RFC2544] methods were intended only for isolated laboratory use, and 143 the difficulties of applying these methods outside the lab 144 environment. 146 4.1. Experimental Control, Repeatability, and Accuracy 148 All of the tests described in RFC 2544 assume that the tester and 149 device under test are the only devices on the networks that are 150 transmitting data. The presence of other unwanted traffic on the 151 network would mean that the specified test conditions have not been 152 achieved. 154 Assuming that the unwanted traffic appears in variable amounts over 155 time, the repeatability of any test result will likely depend to some 156 degree on the unwanted traffic. 158 The presence of unwanted or unknown traffic makes accurate, 159 repeatable, and consistent measurements of the performance of the 160 device under test very unlikely, since the actual test conditions 161 will not be reported. 163 For example, the RFC 2544 Throughput Test attempts to characterize a 164 maximum reliable load, thus there will be testing above the maximum 165 that causes packet/frame loss. Any other sources of traffic on the 166 network will cause packet loss to occur at a tester data rate lower 167 than the rate that would be achieved without the extra traffic. 169 4.2. Containment of Implementation Failure Impact 171 RFC 2544 methods, specifically to determine Throughput as defined in 172 [RFC1242] and other benchmarks, are designed to overload the 173 resources of the device under test, and may cause failure modes in 174 the device under test. Since failures can become the root cause of 175 more wide-spread failure, it is clearly desirable to contain all DUT 176 traffic within the ITE. 178 In addition, such testing can have a negative affect on any traffic 179 which shares resources with the test stream(s) since, in most cases, 180 the traffic load will be close to the capacity of the network links. 182 Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private 183 IPv4 address range for testing: 185 "...The network addresses 198.18.0.0 through 198.19.255.255 have been 186 assigned to the BMWG by the IANA for this purpose. This assignment 187 was made to minimize the chance of conflict in case a testing device 188 were to be accidentally connected to part of the Internet. The 189 specific use of the addresses is detailed below." 191 In other words, devices operating on the Internet may be configured 192 to discard any traffic they observe in this address range, as it is 193 intended for laboratory ITE use only. Thus, testers using the 194 assigned testing address ranges MUST NOT be connected to the 195 Internet. 197 We note that a range of IPv6 addresses has been assigned to BMWG for 198 laboratory test purposes, in [RFC5180]. Also, the strong statements 199 in the Security Considerations Section of this memo make the scope 200 even more clear; this is now a standard fixture of all BMWG memos. 202 5. Advisory on RFC 2544 Methods in Real-world Networks 204 The tests in [RFC2544] were designed to measure the performance of 205 network devices, not of networks, and certainly not production 206 networks carrying user traffic on shared resources. There will be 207 unanticipated difficulties when applying these methods outside the 208 lab environment. 210 Operating test equipment on real-world networks according to the 211 methods described in [RFC2544], where overload is a required outcome, 212 would no doubt be harmful to user traffic performance. These tests 213 MUST NOT be used on active networks. And as discussed above, the 214 tests will never produce a reliable or accurate benchmarking result. 216 [RFC2544] methods have never been validated on a network path, even 217 when that path is not part of a production network and carrying no 218 other traffic. It is unknown whether the tests can be used to 219 measure valid and reliable performance of a multi-device, multi- 220 network path. It is possible that some of the tests may prove to be 221 valid in some path scenarios, but that work has not been done or has 222 not been shared with the IETF community. Thus, such testing is 223 contra-indicated by the BMWG. 225 6. What to do without RFC 2544? 227 The IETF has addressed the problem of real-world network performance 228 measurement by chartering a different working group: IP Performance 229 Metrics (IPPM). This working group has developed a set of standard 230 metrics to assess the quality, performance, and reliability of 231 Internet packet transfer services. These metrics can be measured by 232 network operators, end users, or independent testing groups. We note 233 that some IPPM metrics differ from RFC 2544 metrics with similar 234 names, and there is likely to be confusion if the details are 235 ignored. 237 IPPM has not standardized methods for raw capacity measurement of 238 Internet paths. Such testing needs to adequately consider the strong 239 possibility for degradation to any other traffic that may be present 240 due to congestion. There are no specific methods proposed for 241 activation of a packet transfer service in IPPM. 243 Other standards may help to fill gaps in telecommunication service 244 testing. For example, the IETF has many standards intended to assist 245 with network operation, administration and maintenance (OAM), the 246 ITU-T Study Group 12 has a recommendation on service activation test 247 methodology. 249 The world will not spin off axis while waiting for appropriate and 250 standardized methods to emerge from the consensus process. 252 7. Security Considerations 254 This Applicability Statement is also intended to help preserve the 255 security of the Internet by clarifying that the scope of [RFC2544] 256 and other BMWG memos are all limited to testing in laboratory ITE, 257 thus avoiding accidental Denial of Service attacks or congestion due 258 to high traffic volume test streams. 260 All Benchmarking activities are limited to technology 261 characterization using controlled stimuli in a laboratory 262 environment, with dedicated address space and the other constraints 263 [RFC2544]. 265 The benchmarking network topology will be an independent test setup 266 and MUST NOT be connected to devices that may forward the test 267 traffic into a production network, or misroute traffic to the test 268 management network. 270 Further, benchmarking is performed on a "black-box" basis, relying 271 solely on measurements observable external to the device under test/ 272 system under test (DUT/SUT). 274 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 275 benchmarking purposes. Any implications for network security arising 276 from the DUT/SUT SHOULD be identical in the lab and in production 277 networks. 279 8. IANA Considerations 281 This memo makes no requests of IANA, and hopes that IANA will leave 282 it alone as well. 284 9. Acknowledgements 286 Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, and Curtis 287 Villamizar for reading and suggesting improvements for this memo. 289 10. References 291 10.1. Normative References 293 [RFC1242] Bradner, S., "Benchmarking terminology for network 294 interconnection devices", RFC 1242, July 1991. 296 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 297 Requirement Levels", BCP 14, RFC 2119, March 1997. 299 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 300 Network Interconnect Devices", RFC 2544, March 1999. 302 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 303 Delay Metric for IPPM", RFC 2679, September 1999. 305 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. 306 Dugatkin, "IPv6 Benchmarking Methodology for Network 307 Interconnect Devices", RFC 5180, May 2008. 309 10.2. Informative References 311 Authors' Addresses 313 Scott Bradner 314 Harvard University 315 29 Oxford St. 316 Cambridge, MA 02138 317 USA 319 Phone: +1 617 495 3864 320 Fax: 321 Email: sob@harvard.edu 322 URI: http://www.sobco.com 324 Kevin Dubray 325 Juniper Networks 327 Phone: 328 Fax: 329 Email: kdubray@juniper.net 330 URI: 332 Jim McQuaid 333 Turnip Video 334 6 Cobbleridge Court 335 Durham, North Carolina 27713 336 USA 338 Phone: +1 919-619-3220 339 Fax: 340 Email: jim@turnipvideo.com 341 URI: www.turnipvideo.com 342 Al Morton 343 AT&T Labs 344 200 Laurel Avenue South 345 Middletown,, NJ 07748 346 USA 348 Phone: +1 732 420 1571 349 Fax: +1 732 368 1192 350 Email: acmorton@att.com 351 URI: http://home.comcast.net/~acmacm/