idnits 2.17.1 draft-lencse-bmwg-rfc2544-bis-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (May 20, 2020) is 1435 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Duplicate reference: RFC2544, mentioned in 'LEN2020C', was also mentioned in 'RFC2544'. -- Duplicate reference: RFC8219, mentioned in 'SIITPERF', was also mentioned in 'RFC8219'. Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working Group G. Lencse 3 Internet-Draft BUTE 4 Intended status: Informational K. Shima 5 Expires: November 21, 2020 IIJ-II 6 May 20, 2020 8 An Upgrade to Benchmarking Methodology for Network Interconnect Devices 9 draft-lencse-bmwg-rfc2544-bis-00 11 Abstract 13 RFC 2544 has defined a benchmarking methodology for network 14 interconnect devices. We recommend a few upgrades to it for 15 producing more reasonable results. The recommended upgrades can be 16 classified into two categories: the application of the novelties of 17 RFC 8219 for the legacy RFC 2544 use cases and the following new 18 ones. Checking a reasonably small timeout individually for every 19 single frame in the throughput and frame loss rate benchmarking 20 procedures. Performing a statistically relevant number of tests for 21 all benchmarking procedures. Addition of an optional non-zero frame 22 loss acceptance criterion for the throughput measurement procedure 23 and defining its reporting format. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on November 21, 2020. 42 Copyright Notice 44 Copyright (c) 2020 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 61 2. Recommendation to Backport the Novelties of RFC8219 . . . . . 3 62 3. Improved Throughput and Frame Loss Rate Measurement 63 Procedures using Individual Frame Timeout . . . . . . . . . . 3 64 4. Requirement of Statistically Relevant Number of Tests . . . . 4 65 5. An Optional Non-zero Frame Loss Acceptance Criterion for the 66 Throughput Measurement Procedure . . . . . . . . . . . . . . 5 67 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 6 68 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 6 69 8. Security Considerations . . . . . . . . . . . . . . . . . . . 7 70 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 7 71 9.1. Normative References . . . . . . . . . . . . . . . . . . 7 72 9.2. Informative References . . . . . . . . . . . . . . . . . 7 73 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 8 74 A.1. 00 . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 75 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8 77 1. Introduction 79 [RFC2544] has defined a benchmarking methodology for network 80 interconnect devices. [RFC5180] addressed IPv6 specificities and 81 also added technology updates, but declared IPv6 transition 82 technologies out of its scope. [RFC8219] addressed the IPv6 83 transition technologies, and it added further measurement procedures 84 (e.g. for packet delay variation (PDV) and inter packet delay 85 variation (IPDV)). It has also recommended to perform multiple tests 86 (at least 20), and it proposed median as summarizing function and 1st 87 and 99th percentiles as the measure of variation of the results of 88 the multiple tests. This is a significant change compared to 89 [RFC2544], which always used only average as summarizing function. 90 [RFC8219] also redefined the latency measurement procedure with the 91 requirement of marking at least 500 frames with identifying tags for 92 latency measurements, instead of using only a single one. However, 93 all these improvements apply only for the IPv6 transition 94 technologies, and no update was made to [RFC2544] / [RFC5180], which 95 we believe to be desirable. 97 Moreover, [RFC8219] has reused the throughput and frame loss rate 98 benchmarking procedures from [RFC2544] with no changes. When we 99 tested their feasibility with a few SIIT [RFC7915] implementations, 100 we have pointed out three possible improvements in [LEN2020A]: 102 o Checking a reasonably small timeout individually for every single 103 frame with the throughput and frame loss rate benchmarking 104 procedures. 106 o Performing a statistically relevant number of tests for these two 107 benchmarking procedures. 109 o Addition of an optional non-zero frame loss acceptance criterion 110 for the throughput benchmarking procedure and defining its 111 reporting format. 113 1.1. Requirements Language 115 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 116 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 117 "OPTIONAL" in this document are to be interpreted as described in 118 BCP14 [RFC2119] [RFC8174] when, and only when, they appear in all 119 capitals, as shown here. 121 2. Recommendation to Backport the Novelties of RFC8219 123 Besides addressing IPv6 transition technologies, [RFC8219] has also 124 made several technological upgrades reflecting the current state of 125 the art of networking technologies and benchmarking. But all the 126 novelties mentioned in Section 1 of this document currently apply 127 only for the benchmarking of IPv6 transition technologies. We 128 contend that they could be simply backported to the benchmarking of 129 network interconnect devices. For example, siitperf [SIITPERF], our 130 [RFC8219] compliant DPDK-based software Tester was designed for 131 benchmarking different SIIT [RFC7915] (also called stateless NAT64) 132 implementations, but if it is configured to have the same IP version 133 on both sides, it can be used to test IPv4 or IPv6 (or dual stack) 134 routers [LEN2020B]. We highly recommend the backporting of the 135 latency, PDV and IPDV benchmarking measurement procedures of 136 [RFC8219]. 138 3. Improved Throughput and Frame Loss Rate Measurement Procedures using 139 Individual Frame Timeout 141 The throughput measurement procedure defined in [RFC2544] only counts 142 the number of the sent and received test frames, but it does not 143 identify the test frames individually. On the one hand, this 144 approach allows the Tester to send always the very same test frame to 145 the DUT, which was very likely an important advantage in 1999. 146 However, on the other hand, thus the Tester cannot check if the order 147 of the frames is kept, or if the frames arrive back to the Tester 148 within a given timeout time. (Perhaps none of them was an issue of 149 hardware based network interconnect devices in 1999. But today 150 network packet forwarding and manipulation is often implemented in 151 software having larger buffers and producing potentially higher 152 latencies.) 154 Whereas real-time applications are obviously time sensitive, other 155 applications like HTTP or FTP are often considered throughput hungry 156 and time insensitive. However, we have demonstrated that when we 157 applied 100ms delay to 1% of the test frames, the throughput of HTTP 158 download dropped by more that 50% [LEN2020C]. Therefore, an advanced 159 throughput measurement procedure that checks the timeout time for 160 every single test frame may produce more reasonable results. We have 161 shown that this measurement is now feasible [LEN2020B]. In this 162 case, we used 64-bit integers to identify the test frames and 163 measured the latency of the frames as required by the PDV measurement 164 procedure in Section 7.3.1. of [RFC8219]. In our particular test, we 165 used 10ms as frame timeout, which could be a suitable value, but we 166 recommend further studies do determine the recommended timeout value. 168 We recommend that the reported results of the improved throughput and 169 frame loss rate measurements SHOULD include the applied timeout 170 value. 172 4. Requirement of Statistically Relevant Number of Tests 174 Section 4 of [RFC2544] says that: "Furthermore, selection of the 175 tests to be run and evaluation of the test data must be done with an 176 understanding of generally accepted testing practices regarding 177 repeatability, variance and statistical significance of small numbers 178 of trials." It is made a stronger requirement (by using a "MUST") in 179 Section 3 of [RFC5180] stating that: "Test execution and results 180 analysis MUST be performed while observing generally accepted testing 181 practices regarding repeatability, variance, and statistical 182 significance of small numbers of trials." But no practical 183 guidelines are provided concerning the minimally necessary number of 184 tests. 186 [RFC8219] mentions at four different places that the tests must be 187 repeated at least 20 times. These places are the benchmarking 188 procedures for: 190 o latency (Section 7.2) 192 o packet delay variation (Section 7.3.1) 193 o inter packet delay variation (Section 7.3.2) 195 o DNS64 performance (Section 9.2). 197 We believe that a similar guideline for the minimal number of tests 198 would be helpful for the throughput and frame loss rate benchmarking 199 procedures. We consider 20 as an affordable number of minimum 200 repetitions of the frame loss rate measurements. However, as for 201 throughput measurements, we contend that the binary search may 202 require rather high number of steps in certain situations (e.g. tens 203 of millions of frames per second rate and high resolution) that the 204 requirement of at least 20 repetitions of the binary search would 205 result in unreasonably high measurement execution times. Therefore, 206 we recommend to use an algorithm that checks the statistical 207 properties of the results of the tests and it may stop before 20 208 repetitions, if the results are consistent, but it may require more 209 than 20 repetitions, if the results are scattered. (The algorithm is 210 yet to be developed.) 212 5. An Optional Non-zero Frame Loss Acceptance Criterion for the 213 Throughput Measurement Procedure 215 When we defined the measurement procedure for DNS64 performance in 216 Section 9.2 of [RFC8219], we followed both spirit and wording of the 217 [RFC2544] throughput measurement procedure including the requirement 218 for absolutely zero packet loss. We have elaborated our underlying 219 considerations in our research paper [LEN2017] as follows: 221 1. Our goal is a well-defined performance metric, which can be 222 measured simply and efficiently. Allowing any packet loss would 223 result in a need for scanning/trying a large range of rates to 224 discover the highest rate of successfully processed DNS queries. 226 2. Even if users may tolerate a low loss rate (please note the DNS 227 uses UDP with no guarantee for delivery), it cannot be 228 arbitrarily high, thus, we could not avoid defining a limit. 229 However, any other limits than zero percent would be hardly 230 defensible. 232 3. Other benchmarking procedures use the same criteria of zero 233 packet loss and this is the standard in IETF Benchmarking 234 Methodology Working Group. 236 On the one hand, we still consider our arguments valid, however, on 237 the other hand, we are aware of different arguments for the 238 justification of an optional non-zero frame loss acceptance 239 criterion, too: 241 o Frame loss is present in our networks from the very beginning and 242 our applications are well prepared to handle frame loss. They can 243 definitely tolerate some low frame loss rates like 0.01% (1 frame 244 from 10,000 frames). 246 o It is a wide-spread practice among benchmarking professionals to 247 allow a certain low rate of frame loss for a long time [TOL2001] 248 and commercially available network performance testers allow to 249 specify a parameter usually called as "Loss Tolerance" to express 250 a zero or non-zero acceptance criterion for throughput 251 measurements. 253 o Today network packet forwarding and manipulation is often 254 implemented in software. They do not work the same as the 255 hardware-based forwarding devices, and may be affected by other 256 processes running in the same host hardware. So it is not 257 feasible to require 0% of frame loss in such forwarding devices. 259 o Forwarding devices (especially but not necessarily only the 260 software-based ones) may today also have larger buffers and thus 261 they may produce potentially higher latencies. As we have shown 262 in Section 3, late packets are not really useful for the 263 applications, and thus they are to be considered as lost ones. 264 For being strict with the latency during throughput measurements 265 (e.g. 10ms timeout), we should make up with the loss tolerance to 266 provide meaningful benchmarking results. 268 o Likely due to the high frame loss rates can be experienced in WiFi 269 networks, the latest development direction of TCP congestion 270 control algorithms considers loss no more a sign of congestion 271 (e.g. TCP BBR). 273 So we felt the necessity of having options to allow frame loss. 274 Therefore, we recommend that throughput measurement with some low 275 tolerated frame loss rates like 0.001% or 0.01% be a recognized 276 optional test for network interconnect devices. To avoid the 277 possibility of gaming, our recommendation is that the results of such 278 tests MUST clearly state the applied loss tolerance rate. 280 6. Acknowledgements 282 The authors would like to thank ... (TBD) 284 7. IANA Considerations 286 This document does not make any request to IANA. 288 8. Security Considerations 290 We have no further security considerations beyond that of [RFC8219]. 291 Perhaps they should be cited here so that they be applied not only 292 for the benchmarking of IPv6 transition technologies, but also for 293 the benchmarking of all network interconnect devices. 295 9. References 297 9.1. Normative References 299 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 300 Requirement Levels", BCP 14, RFC 2119, 301 DOI 10.17487/RFC2119, March 1997, 302 . 304 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 305 Network Interconnect Devices", RFC 2544, 306 DOI 10.17487/RFC2544, March 1999, 307 . 309 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. 310 Dugatkin, "IPv6 Benchmarking Methodology for Network 311 Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May 312 2008, . 314 [RFC7915] Bao, C., Li, X., Baker, F., Anderson, T., and F. Gont, 315 "IP/ICMP Translation Algorithm", RFC 7915, 316 DOI 10.17487/RFC7915, June 2016, 317 . 319 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 320 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 321 May 2017, . 323 [RFC8219] Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking 324 Methodology for IPv6 Transition Technologies", RFC 8219, 325 DOI 10.17487/RFC8219, August 2017, 326 . 328 9.2. Informative References 330 [LEN2017] Lencse, G., Georgescu, M., and Y. Kadobayashi, 331 "Benchmarking Methodology for DNS64 Servers", Computer 332 Communications, vol. 109, no. 1, pp. 162-175, DOI: 333 10.1016/j.comcom.2017.06.004, Sep 2017, 334 . 337 [LEN2020A] 338 Lencse, G. and K. Shima, "Performance analysis of SIIT 339 implementations: Testing and improving the 340 methodology", Computer Communications, vol. 156, no. 1, 341 pp. 54-67, DOI: 10.1016/j.comcom.2020.03.034, Apr 2020, 342 . 345 [LEN2020B] 346 Lencse, G., "Design and Implementation of a Software 347 Tester for Benchmarking Stateless NAT64 Gateways", under 348 second review in IEICE Transactions on Communications, 349 . 352 [LEN2020C] 353 Lencse, G., Shima, K., and A. Kovacs, "Gaming with the 354 Throughput and the Latency Benchmarking Measurement 355 Procedures of RFC 2544", under review in International 356 Journal of Advances in Telecommunications, 357 Electrotechnics, Signals and Systems, 358 . 361 [SIITPERF] 362 Lencse, G. and Y. Kadobayashi, "Siitperf: An RFC 8219 363 compliant SIIT (stateless NAT64) tester written in C++ 364 using DPDK", source code, available from GitHub, 2019, 365 . 367 [TOL2001] Tolly, K., "The real meaning of zero-loss testing", IT 368 World Canada, 2001, 369 . 372 Appendix A. Change Log 374 A.1. 00 376 Initial version. 378 Authors' Addresses 379 Gabor Lencse 380 Budapest University of Technology and Economics 381 Magyar Tudosok korutja 2. 382 Budapest H-1117 383 Hungary 385 Email: lencse@hit.bme.hu 387 Keiichi Shima 388 IIJ Innovation Institute 389 Iidabashi Grand Bloom, 2-10-2 Fujimi 390 Chiyoda-ku, Tokyo 102-0071 391 Japan 393 Email: keiichi@iijlab.net