idnits 2.17.1 draft-lencse-bmwg-benchmarking-stateful-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 2 instances of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The exact meaning of the all-uppercase expression 'MAY NOT' is not defined in RFC 2119. If it is intended as a requirements expression, it should be rewritten using one of the combinations defined in RFC 2119; otherwise it should not be all-uppercase. == The expression 'MAY NOT', while looking like RFC 2119 requirements text, is not defined in RFC 2119, and should not be used. Consider using 'MUST NOT' instead (if that is what you mean). Found 'MAY NOT' in this paragraph: Pseudorandom reading order of the state table MAY NOT be used with unidirectional traffic from the Responder to the Initiator, because if a four tuple is not used until timeout time, then its connection is deleted from the connection tracking table of the DUT and a later use of the given four tuple will cause frame loss. There is no such problem, when bidirectional traffic is used, because then the state table of the Responder is periodically refreshed. -- The document date (27 August 2021) is 944 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-15) exists of draft-ietf-bmwg-ngfw-performance-09 -- Duplicate reference: RFC4814, mentioned in 'LEN2020', was also mentioned in 'RFC4814'. -- Duplicate reference: RFC8219, mentioned in 'SIITPERF', was also mentioned in 'RFC8219'. Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working Group G.L. Lencse 3 Internet-Draft Szechenyi Istvan University 4 Intended status: Informational K.S. Shima 5 Expires: 28 February 2022 IIJ Innovation Institute 6 27 August 2021 8 Benchmarking Methodology for Stateful NATxy Gateways using RFC 4814 9 Pseudorandom Port Numbers 10 draft-lencse-bmwg-benchmarking-stateful-01 12 Abstract 14 RFC 2544 has defined a benchmarking methodology for network 15 interconnect devices. RFC 5180 addressed IPv6 specificities and it 16 also provided a technology update, but excluded IPv6 transition 17 technologies. RFC 8219 addressed IPv6 transition technologies, 18 including stateful NAT64. However, none of them discussed how to 19 apply RFC 4814 pseudorandom port numbers to any stateful NAT (NAT44, 20 NAT64, NAT66) technologies. We discuss why using pseudorandom port 21 numbers with stateful NAT gateways is a hard problem and recommend a 22 solution. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on 28 February 2022. 41 Copyright Notice 43 Copyright (c) 2021 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 48 license-info) in effect on the date of publication of this document. 49 Please review these documents carefully, as they describe your rights 50 and restrictions with respect to this document. Code Components 51 extracted from this document must include Simplified BSD License text 52 as described in Section 4.e of the Trust Legal Provisions and are 53 provided without warranty as described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 59 2. Pseudorandom Port Numbers and Stateful Translation . . . . . 3 60 3. Test Setup and Terminology . . . . . . . . . . . . . . . . . 4 61 4. Recommended Benchmarking Method . . . . . . . . . . . . . . . 5 62 4.1. Restricted Port Number Ranges . . . . . . . . . . . . . . 6 63 4.2. Preliminary Test Phase . . . . . . . . . . . . . . . . . 6 64 4.3. Control of the Connection Tracking Table Entries . . . . 7 65 4.4. Measurement of the Maximum Connection Establishment 66 Rate . . . . . . . . . . . . . . . . . . . . . . . . . . 8 67 4.5. Real Test Phase . . . . . . . . . . . . . . . . . . . . . 9 68 4.6. Writing and Reading Order of the State Table . . . . . . 10 69 4.7. Peculiarities of Stateful Testing . . . . . . . . . . . . 10 70 4.7.1. Timeout Budget . . . . . . . . . . . . . . . . . . . 10 71 4.7.2. Special Warning Against Non-zero Frame Loss 72 Testing . . . . . . . . . . . . . . . . . . . . . . . 11 73 5. Implementation and Experience . . . . . . . . . . . . . . . . 11 74 6. Limitations of using UDP as Transport Layer Protocol . . . . 11 75 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 76 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 77 9. Security Considerations . . . . . . . . . . . . . . . . . . . 12 78 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 79 10.1. Normative References . . . . . . . . . . . . . . . . . . 12 80 10.2. Informative References . . . . . . . . . . . . . . . . . 13 81 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 14 82 A.1. 00 . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 83 A.2. 01 . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 84 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 14 86 1. Introduction 88 [RFC2544] has defined a comprehensive benchmarking methodology for 89 network interconnect devices, which is still in use. It was mainly 90 IP version independent, but it used IPv4 in its examples. [RFC5180] 91 addressed IPv6 specificities and also added technology updates, but 92 declared IPv6 transition technologies out of its scope. [RFC8219] 93 addressed the IPv6 transition technologies, including stateful NAT64. 94 It has reused several benchmarking procedures from [RFC2544] (e.g. 95 throughput, frame loss rate), it has redefined the latency 96 measurement, and added further ones, e.g. the PDV (packet delay 97 variation) measurement. 99 However, none of them discussed, how to apply [RFC4814] pseudorandom 100 port numbers, when benchmarking stateful NATxy (NAT44, NAT64, NAT66) 101 gateways. We are not aware of any other RFCs that address this 102 question. 104 First, we discuss why using pseudorandom port numbers with stateful 105 NATxy gateways is a hard problem. 107 Then we recommend a solution. 109 1.1. Requirements Language 111 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 112 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 113 "OPTIONAL" in this document are to be interpreted as described in 114 BCP14 [RFC2119] [RFC8174] when, and only when, they appear in all 115 capitals, as shown here. 117 2. Pseudorandom Port Numbers and Stateful Translation 119 In its appendix, [RFC2544] has defined a frame format for test frames 120 including specific source and destination port numbers. [RFC4814] 121 recommends to use pseudorandom and uniformly distributed values for 122 both source and destination port numbers. However, stateful NATxy 123 (NAT44, NAT64, NAT66) solutions use the port numbers to identify 124 connections. The usage of pseudorandom port numbers causes different 125 problems depending on the direction. 127 * As for the private to public direction, pseudorandom source and 128 destination port numbers could be used, however, this approach 129 would be a denial of service attack against the stateful NATxy 130 gateway, because it would exhaust its connection tracking table. 131 To that end, let us see some calculations using the 132 recommendations of RFC 4814: 134 - The recommended source port range is: 1024-65535, thus its size 135 is: 64512. 137 - The recommended destination port range is: 1-49151, thus its 138 size is: 49151. 140 - The number of source and destination port number combinations 141 is: 3,170,829,312. 143 We note that section 12 of [RFC2544] also requires testing with 144 256 destination networks, which further increases the number of 145 connection tracking table entries. 147 * As for the public to private direction, the stateful DUT (Device 148 Under Test) would drop any packets that do not belong to an 149 existing connection, therefore, the direct usage of pseudorandom 150 port numbers from the above-mentioned ranges is not feasible. 152 3. Test Setup and Terminology 154 Our methodology works with any IP version. We use IPv4 in the Test 155 Setup shown in Figure 1 to facilitate its easy understanding based on 156 the well-known stateful NAT44 (also called NAPT: Network Address and 157 Port Translation) solution. 159 +--------------------------------------+ 160 10.0.0.2 |Initiator Responder| 198.19.0.2 161 +-------------| Tester |<------------+ 162 | private IPv4| [state table]| public IPv4 | 163 | +--------------------------------------+ | 164 | | 165 | +--------------------------------------+ | 166 | 10.0.0.1 | DUT: | 198.19.0.1 | 167 +------------>| Sateful NATxy gateway |-------------+ 168 private IPv4| [connection tracking table] | public IPv4 169 +--------------------------------------+ 171 Figure 1: Test Setup for benchmarking stateful NATxy gateways 173 As for transport layer protocol, [RFC2544] recommended testing with 174 UDP, and it was kept also in [RFC8219]. For the general 175 recommendation, we also keep UDP, thus the port numbers in the 176 following text are to be understood as UDP port numbers. We discuss 177 the limitation of this approach in Section 6. 179 We define the most important elements of our proposed benchmarking 180 system as follows. 182 * Connection tracking table: The stateful NATxy gateway uses a 183 connection tracking table to be able to perform the stateful 184 translation in the public to private direction. Its size, policy 185 and content are unknown for the Tester. 187 * Four tuple: The four numbers that identify a connection are source 188 IP address, source port number, destination IP address, 189 destination port number. 191 * State table: The Responder of the Tester extracts the four tuple 192 from each received test frame and stores it in its state table. 193 Recommendation is given for writing and reading order of the state 194 table in Section 4.6. 196 * Initiator: The port of the Tester that may initiate a connection 197 through the stateful DUT in the private to public direction. 198 Theoretically, it can use any source and destination port numbers 199 from the ranges recommended by [RFC4814]: if the used four tuple 200 does not belong to an existing connection, the DUT will register a 201 new connection into its connection tracking table. 203 * Responder: The port of the Tester that may not initiate a 204 connection through the stateful DUT in the public to private 205 direction. It may send only frames that belong to an existing 206 connection. To that end, it uses four tuples that have been 207 previously extracted from the received test frames and stored in 208 its state table. 210 * Preliminary test phase: Test frames are sent only by the Initiator 211 to the Responder through the DUT to fill both the connection 212 tracking table of the DUT and the state table of the Responder. 213 This is a newly introduced operation phase for stateful NATxy 214 benchmarking. The necessity of this phase is explained in 215 Section 4.2. 217 * Real test phase: The actual test (e.g. throughput, latency, etc.) 218 is performed in this phase after the completion of the preliminary 219 test phase. Test frames are sent as required (e.g. bidirectional 220 test or unidirectional test in any of the two directions). 222 4. Recommended Benchmarking Method 223 4.1. Restricted Port Number Ranges 225 The Initiator SHOULD use restricted ranges for source and destination 226 port numbers to avoid the denial of service attack like event against 227 the connection tracking table of the DUT described in Section 2. The 228 size of the source port number range SHOULD be larger (e.g. in the 229 order of a few times ten thousand), whereas the size of the 230 destination port number range SHOULD be smaller (may vary from a few 231 to several hundreds or thousands as needed). The rationale is that 232 source and destination port numbers that can be observed in the 233 Internet traffic are not symmetrical. Whereas source port numbers 234 may be random, there are a few very popular destination port numbers 235 (e.g. 443, 80, etc., see [IIR2020]) and others hardly occur. And we 236 have found that their role is also asymmetric in the Linux kernel 237 routing hash function [LEN2020]. 239 The product of the sizes of the two ranges can be used as a 240 parameter. The performance of the stateful NATxy gateway MAY be 241 examined as a function of this parameter. 243 4.2. Preliminary Test Phase 245 The preliminary phase serves two purposes: 247 1. The connection tracking table of the DUT is filled. It is 248 important, because its maximum connection establishment rate may 249 be lower than its maximum frame forwarding rate (that is 250 throughput). 252 2. The state table of the Responder is filled with valid four 253 tuples. It is a precondition for the Responder to be able to 254 transmit frames that belong to connections exist in the 255 connection tracking table of the DUT. 257 Whereas the above two things are always necessary before the real 258 test phase, the preliminary phase can be used without the real test 259 phase. It is done so, when the maximum connection establishment rate 260 is measured (as described in Section 4.4). 262 A preliminary test phase MUST be performed before all tests performed 263 in the real test phase. In this phase, the following things happen: 265 1. The Initiator sends test frames to the Responder through the DUT 266 at a specific frame rate. 268 2. The DUT performs the stateful translation of the test frames and 269 it also stores the new combinations in its connection tracking 270 table. 272 3. The Responder receives the translated test frames and updates its 273 state table with the received four tuples. The responder 274 transmits no test frames during the preliminary phase. 276 When the preliminary test phase is performed in preparation to the 277 real test phase, the applied frame rate and the duration of the 278 preliminary phase SHOULD be carefully selected so that: 280 * The applied frame rate be safely lower than the maximum connection 281 establishment rate. 283 * The initial transient of the filling of the connection tracking 284 table of the DUT be finished. 286 * Enough four tuples be stored in the state table of the Responder 287 so that it can generate frames with the proper distribution of the 288 four tuples. 290 * The connections do not time out in the DUT even during the 291 beginning of the real test phase. 293 4.3. Control of the Connection Tracking Table Entries 295 The number of the entries in the connection tracking table of the DUT 296 MAY be controlled by using all different source port number 297 destination port number combinations. 299 Let NF and NC denote the number of test frames to send and the number 300 of all possible source port number destination port number 301 combinations, respectively. 303 1. If NF and NC are in the same order of magnitude, then the all 304 different source port number destination port number combinations 305 may be computing efficiently generated by preparing a random 306 permutation of the previously enumerated all possible source port 307 number destination port number combinations using Dustenfeld's 308 random shuffle algorithm [DUST1964]. 310 2. If NF is at least an order of magnitude less than NC, then a 311 simpler solution may be used: the Initiator registers in a table 312 and then it checks if a given source port number destination port 313 number combination was already used (and if yes, then it MUST 314 generate a new one). 316 3. If NF is at least two orders of magnitude less than NC, then 317 mostly different source port number destination port number 318 combinations can be generated without any specific provision. 320 Important warning: in normal (non-NAT) router testing, the port 321 number selection algorithm (whether it is pseudo-random or 322 enumerated) does not affect final results. However, our experience 323 with iptables shows that if the connection tracking table is filled 324 using port number enumeration in increasing order, then the maximum 325 connection establishment rate of iptables degrades significantly 326 compared to its performance using pseudorandom port numbers 327 [LEN2021]. 329 [RFC4814] REQUIRES pseudorandom port numbers, which we believe is a 330 good approximation of the distribution of the source port numbers a 331 NATxy gateway on the Internet may face with. 333 The enumeration of the source port number destination port number 334 combinations in increasing order MAY be used as an additional 335 measurement to discover worst case performance. 337 4.4. Measurement of the Maximum Connection Establishment Rate 339 The maximum connection establishment rate is an important 340 characteristic of the stateful NATxy gateway and its determination is 341 necessary for the safe execution of the preliminary test phase 342 (without frame loss) before the real test phase. 344 The measurement procedure of the maximum connection establishment 345 rate is very similar to the throughput measurement procedure defined 346 in [RFC2544]. 348 Procedure: The Initiator sends a specific number of test frames using 349 all (or mostly) different source port number destination port number 350 combinations at a specific rate through the DUT. The Responder 351 counts the frames that are successfully translated by the DUT. If 352 the count of offered frames is equal to the count of received frames, 353 the rate of the offered stream is raised and the test is rerun. If 354 fewer frames are received than were transmitted, the rate of the 355 offered stream is reduced and the test is rerun. 357 The maximum connection establishment rate is the fastest rate at 358 which the count of test frames successfully translated by the DUT is 359 equal to the number of test frames sent to it by the Initiator. 361 Notes: 363 1. All different source port number destination port number 364 combinations SHOULD be used, if the results of the measurement 365 are published. Mostly different source port number destination 366 port number combinations MAY be used, if the results of the 367 measurement are not published, but they are used only to 368 determine a good enough frame rate for the preliminary test phase 369 preparing the test system for the real test phase. 371 2. In practice, we RECOMMEND the usage of binary search. 373 3. As for the successful translation, the Responder MAY (or SHOULD?) 374 check that the source IP address is different than the original 375 source IP address set by the Initiator. 377 4.5. Real Test Phase 379 As for the traffic direction, there are three possible cases during 380 the real test phase: 382 * bidirectional traffic: The Initiator sends test frames to the 383 Responder and the Responder sends test frames to the Initiator. 385 * unidirectional traffic from the Initiator to the Responder: The 386 Initiator sends test frames to the Responder but the Responder 387 does not send test frames to the Initiator. 389 * unidirectional traffic from the Responder to the Initiator: The 390 Responder sends test frames to the Initiator but the Initiator 391 does not send test frames to the Responder. 393 If the Initiator sends test frames, then it uses pseudorandom source 394 port numbers and destination port numbers from the restricted port 395 number ranges. The responder receives the test frames, updates its 396 state table and processes the test frames as required by the given 397 measurement procedure (e.g. only counts them for throughput test, 398 handles timestamps for latency or PDV tests, etc.). 400 If the Responder sends test frames, then it uses the four tuples from 401 its state table. The reading order of the state table may follow 402 different policies (discussed in Section 4.6). The Initiator 403 receives the test frames, and processes them as required by the given 404 measurement procedure. 406 As for the actual measurement procedures, we RECOMMEND to use the 407 updated ones from Section 7 of [RFC8219]. 409 4.6. Writing and Reading Order of the State Table 411 As for writing policy of the state table of the Responder, we 412 RECOMMEND round robin, because it ensures that its entries are 413 automatically kept fresh and thus there is no need to handle timeout. 415 The Responder can read its state table in various orders. We 416 RECOMMEND one of the following ones: 418 * round robin 420 * pseudorandom (with restriction!) 422 * random permutation (no position is repeated until all positions 423 are used). 425 Pseudorandom reading order of the state table MAY NOT be used with 426 unidirectional traffic from the Responder to the Initiator, because 427 if a four tuple is not used until timeout time, then its connection 428 is deleted from the connection tracking table of the DUT and a later 429 use of the given four tuple will cause frame loss. There is no such 430 problem, when bidirectional traffic is used, because then the state 431 table of the Responder is periodically refreshed. 433 We do not see any problem in the round robin reading order, because 434 the state table is filled using pseudorandom port numbers. 436 4.7. Peculiarities of Stateful Testing 438 Stateful testing involves some issues not present in stateless 439 testing. 441 4.7.1. Timeout Budget 443 Even though we do black box testing, one MUST consider timeout and 444 carefully manage timeout budget. For example, if the frame rate is 445 high enough, then every single entry of the state table of the 446 Responder is refreshed within timeout time and it prevents frame 447 sending with a stale four tuple. If the entries of the state table 448 are not refreshed (due to testing with single directional traffic 449 from the Responder to the Initiator) then using all four tuples 450 within timeout time can keep all connection tracking table entries of 451 the DUT alive. 453 Special care should be taken for the lower frame rate in the 454 preliminary phase. 456 If the binary search (or the decreasing of the applied frame rates 457 during the frame loss rate test) results in a frame rate that is too 458 low to prevent the deletion of the connection tracking table entries 459 of the DUT due to timeout, then it results in the failure of the 460 consecutive tests (the binary search of the throughput test counts 461 down to zero). 463 4.7.2. Special Warning Against Non-zero Frame Loss Testing 465 Several network performance tester vendors include a parameter called 466 "Loss Tolerance" (or similar) for the throughput test and several 467 benchmarking professionals actually use nonzero values [TOL2001]. If 468 frames are lost during stateful testing (especially if it happens 469 during a test with unidirectional traffic from the Responder to the 470 Initiator) the refreshing of the corresponding connection tracking 471 table element of the DUT is not ensured and it may result in the loss 472 of further frames (not due to the low performance of the DUT, but due 473 to using a stale four tuple). 475 5. Implementation and Experience 477 The "stateful" branch of siitperf [SIITPERF] is an implementation of 478 this concept. 480 Our experience is partially documented in a paper currently under 481 review [LEN2021]. 483 6. Limitations of using UDP as Transport Layer Protocol 485 Stateful NATxy solutions handle TCP and UDP differently, e.g. 486 iptables uses 30s timeout for UDP and 60s timeout for TCP. Thus 487 benchmarking results produced using UDP do not necessarily 488 characterize the performance of a NATxy gateway well enough, when 489 they are used for forwarding Internet traffic. As for the given 490 example, timeout values of the DUT may be adjusted, but it requires 491 extra consideration. 493 Other differences in handling UDP or TCP are also possible. Thus we 494 recommend that further investigations are to be performed in this 495 field. 497 As a mitigation of this problem, we recommend that testing with 498 protocols usig TCP (like HTTP and HTTPS) can be performed as 499 described in [I-D.ietf-bmwg-ngfw-performance]. This approach also 500 solves the potential problem of protocol helpers may be present in 501 the stateful DUT. 503 7. Acknowledgements 505 The authors would like to thank Edwin Cordeiro, Lukasz Bromirski and 506 Sandor Repas for their comments. 508 8. IANA Considerations 510 This document does not make any request to IANA. 512 9. Security Considerations 514 We have no further security considerations beyond that of [RFC8219]. 515 Perhaps they should be cited here so that they be applied not only 516 for the benchmarking of IPv6 transition technologies, but also for 517 the benchmarking of stateful NATxy gateways. 519 10. References 521 10.1. Normative References 523 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 524 Requirement Levels", BCP 14, RFC 2119, 525 DOI 10.17487/RFC2119, March 1997, 526 . 528 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 529 Network Interconnect Devices", RFC 2544, 530 DOI 10.17487/RFC2544, March 1999, 531 . 533 [RFC4814] Newman, D. and T. Player, "Hash and Stuffing: Overlooked 534 Factors in Network Device Benchmarking", RFC 4814, 535 DOI 10.17487/RFC4814, March 2007, 536 . 538 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. 539 Dugatkin, "IPv6 Benchmarking Methodology for Network 540 Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May 541 2008, . 543 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 544 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 545 May 2017, . 547 [RFC8219] Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking 548 Methodology for IPv6 Transition Technologies", RFC 8219, 549 DOI 10.17487/RFC8219, August 2017, 550 . 552 10.2. Informative References 554 [DUST1964] Durstenfeld, R., "Algorithm 235: Random 555 permutation", Communications of the ACM, vol. 7, no. 7, 556 p.420., DOI 10.1145/364520.364540, July 1964, 557 . 559 [I-D.ietf-bmwg-ngfw-performance] 560 Balarajah, B., Rossenhoevel, C., and B. Monkman, 561 "Benchmarking Methodology for Network Security Device 562 Performance", Work in Progress, Internet-Draft, draft- 563 ietf-bmwg-ngfw-performance-09, 21 May 2021, 564 . 567 [IIR2020] Kurahashi, T., Matsuzaki, Y., Sasaki, T., Saito, T., and 568 F. Tsutsuji, "Periodic observation report: Internet trends 569 as seen from IIJ infrastructure - 2020", Internet 570 Infrastructure Review, vol. 49, December 2020, 571 . 574 [LEN2020] Lencse, G., "Adding RFC 4814 Random Port Feature to 575 Siitperf: Design, Implementation and Performance 576 Estimation", International Journal of Advances in 577 Telecommunications, Electrotechnics, Signals and Systems, 578 vol 9, no 3, pp. 18-26., DOI 10.11601/ijates.v9i3.291, 579 2020, . 582 [LEN2021] Lencse, G., "Design and Implementation of a Software 583 Tester for Benchmarking Stateful NAT64 Gateways: Theory 584 and Practice of Extending Siitperf for Stateful 585 Tests", under review in Computer Communications, may be 586 revised or removed without notice, 2021, 587 . 590 [SIITPERF] Lencse, G., "Siitperf: An RFC 8219 compliant SIIT 591 (stateless NAT64) tester written in C++ using 592 DPDK", source code, available from GitHub, 2019-2021, 593 . 595 [TOL2001] Tolly, K., "The real meaning of zero-loss testing", IT 596 World Canada, 2001, 597 . 600 Appendix A. Change Log 602 A.1. 00 604 Initial version. 606 A.2. 01 608 Updates based on the comments received on the BMWG mailing list and 609 minor corrections. 611 Authors' Addresses 613 Gabor Lencse 614 Szechenyi Istvan University 615 Gyor 616 Egyetem ter 1. 617 H-9026 618 Hungary 620 Email: lencse@sze.hu 622 Keiichi Shima 623 IIJ Innovation Institute 624 Iidabashi Grand Bloom, 2-10-2 Fujimi, Tokyo 625 102-0071 626 Japan 628 Email: keiichi@iijlab.net