idnits 2.17.1 draft-ietf-bmwg-mlrsearch-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The abstract seems to contain references ([RFC2544]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (February 09, 2021) is 1171 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC8174' is defined on line 790, but no explicit reference was found in the text Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Working Group M. Konstantynowicz, Ed. 3 Internet-Draft V. Polak, Ed. 4 Intended status: Informational Cisco Systems 5 Expires: August 13, 2021 February 09, 2021 7 Multiple Loss Ratio Search for Packet Throughput (MLRsearch) 8 draft-ietf-bmwg-mlrsearch-00 10 Abstract 12 This document proposes changes to [RFC2544], specifically to packet 13 throughput search methodology, by defining a new search algorithm 14 referred to as Multiple Loss Ratio search (MLRsearch for short). 15 Instead of relying on binary search with pre-set starting offered 16 load, it proposes a novel approach discovering the starting point in 17 the initial phase, and then searching for packet throughput based on 18 defined packet loss ratio (PLR) input criteria and defined final 19 trial duration time. One of the key design principles behind 20 MLRsearch is minimizing the total test duration and searching for 21 multiple packet throughput rates (each with a corresponding PLR) 22 concurrently, instead of doing it sequentially. 24 The main motivation behind MLRsearch is the new set of challenges and 25 requirements posed by NFV (Network Function Virtualization), 26 specifically software based implementations of NFV data planes. 27 Using [RFC2544] in the experience of the authors yields often not 28 repetitive and not replicable end results due to a large number of 29 factors that are out of scope for this draft. MLRsearch aims to 30 address this challenge in a simple way of getting the same result 31 sooner, so more repetitions can be done to describe the 32 replicability. 34 Status of This Memo 36 This Internet-Draft is submitted in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF). Note that other groups may also distribute 41 working documents as Internet-Drafts. The list of current Internet- 42 Drafts is at https://datatracker.ietf.org/drafts/current/. 44 Internet-Drafts are draft documents valid for a maximum of six months 45 and may be updated, replaced, or obsoleted by other documents at any 46 time. It is inappropriate to use Internet-Drafts as reference 47 material or to cite them other than as "work in progress." 48 This Internet-Draft will expire on August 13, 2021. 50 Copyright Notice 52 Copyright (c) 2021 IETF Trust and the persons identified as the 53 document authors. All rights reserved. 55 This document is subject to BCP 78 and the IETF Trust's Legal 56 Provisions Relating to IETF Documents 57 (https://trustee.ietf.org/license-info) in effect on the date of 58 publication of this document. Please review these documents 59 carefully, as they describe your rights and restrictions with respect 60 to this document. Code Components extracted from this document must 61 include Simplified BSD License text as described in Section 4.e of 62 the Trust Legal Provisions and are provided without warranty as 63 described in the Simplified BSD License. 65 Table of Contents 67 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 2 68 2. MLRsearch Background . . . . . . . . . . . . . . . . . . . . 4 69 3. MLRsearch Overview . . . . . . . . . . . . . . . . . . . . . 5 70 4. Sample Implementation . . . . . . . . . . . . . . . . . . . . 8 71 4.1. Input Parameters . . . . . . . . . . . . . . . . . . . . 8 72 4.2. Initial Phase . . . . . . . . . . . . . . . . . . . . . . 9 73 4.3. Non-Initial Phases . . . . . . . . . . . . . . . . . . . 10 74 5. FD.io CSIT Implementation . . . . . . . . . . . . . . . . . . 12 75 5.1. Additional details . . . . . . . . . . . . . . . . . . . 12 76 5.1.1. FD.io CSIT Input Parameters . . . . . . . . . . . . . 14 77 5.2. Example MLRsearch Run . . . . . . . . . . . . . . . . . . 14 78 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 79 7. Security Considerations . . . . . . . . . . . . . . . . . . . 16 80 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 81 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 17 82 9.1. Normative References . . . . . . . . . . . . . . . . . . 17 83 9.2. Informative References . . . . . . . . . . . . . . . . . 17 84 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 17 86 1. Terminology 88 o Frame size: size of an Ethernet Layer-2 frame on the wire, 89 including any VLAN tags (dot1q, dot1ad) and Ethernet FCS, but 90 excluding Ethernet preamble and inter-frame gap. Measured in 91 bytes. 93 o Packet size: same as frame size, both terms used interchangeably. 95 o Device Under Test (DUT): In software networking, "device" denotes 96 a specific piece of software tasked with packet processing. Such 97 device is surrounded with other software components (such as 98 operating system kernel). It is not possible to run devices 99 without also running the other components, and hardware resources 100 are shared between both. For purposes of testing, the whole set 101 of hardware and software components is called "system under test" 102 (SUT). As SUT is the part of the whole test setup performance of 103 which can be measured by [RFC2544] methods, this document uses SUT 104 instead of [RFC2544] DUT. Device under test (DUT) can be re- 105 introduced when analysing test results using whitebox techniques, 106 but this document sticks to blackbox testing. 108 o System Under Test (SUT): System under test (SUT) is a part of the 109 whole test setup whose performance is to be benchmarked. The 110 complete test setup contains other parts, whose performance is 111 either already established, or not affecting the benchmarking 112 result. 114 o Bi-directional throughput tests: involve packets/frames flowing in 115 both transmit and receive directions over every tested interface 116 of SUT/DUT. Packet flow metrics are measured per direction, and 117 can be reported as aggregate for both directions and/or separately 118 for each measured direction. In most cases bi-directional tests 119 use the same (symmetric) load in both directions. 121 o Uni-directional throughput tests: involve packets/frames flowing 122 in only one direction, i.e. either transmit or receive direction, 123 over every tested interface of SUT/DUT. Packet flow metrics are 124 measured and are reported for measured direction. 126 o Packet Loss Ratio (PLR): ratio of packets received relative to 127 packets transmitted over the test trial duration, calculated using 128 formula: PLR = ( pkts_transmitted - pkts_received ) / 129 pkts_transmitted. For bi-directional throughput tests aggregate 130 PLR is calculated based on the aggregate number of packets 131 transmitted and received. 133 o Packet Throughput Rate: maximum packet offered load DUT/SUT 134 forwards within the specified Packet Loss Ratio (PLR). In many 135 cases the rate depends on the frame size processed by DUT/SUT. 136 Hence packet throughput rate MUST be quoted with specific frame 137 size as received by DUT/SUT during the measurement. For bi- 138 directional tests, packet throughput rate should be reported as 139 aggregate for both directions. Measured in packets-per-second 140 (pps) or frames-per-second (fps), equivalent metrics. 142 o Bandwidth Throughput Rate: a secondary metric calculated from 143 packet throughput rate using formula: bw_rate = pkt_rate * 144 (frame_size + L1_overhead) * 8, where L1_overhead for Ethernet 145 includes preamble (8 Bytes) and inter-frame gap (12 Bytes). For 146 bi-directional tests, bandwidth throughput rate should be reported 147 as aggregate for both directions. Expressed in bits-per-second 148 (bps). 150 o Non Drop Rate (NDR): maximum packet/bandwith throughput rate 151 sustained by DUT/SUT at PLR equal zero (zero packet loss) specific 152 to tested frame size(s). MUST be quoted with specific packet size 153 as received by DUT/SUT during the measurement. Packet NDR 154 measured in packets-per-second (or fps), bandwidth NDR expressed 155 in bits-per-second (bps). 157 o Partial Drop Rate (PDR): maximum packet/bandwith throughput rate 158 sustained by DUT/SUT at PLR greater than zero (non-zero packet 159 loss) specific to tested frame size(s). MUST be quoted with 160 specific packet size as received by DUT/SUT during the 161 measurement. Packet PDR measured in packets-per-second (or fps), 162 bandwidth PDR expressed in bits-per-second (bps). 164 o Maximum Receive Rate (MRR): packet/bandwidth rate regardless of 165 PLR sustained by DUT/SUT under specified Maximum Transmit Rate 166 (MTR) packet load offered by traffic generator. MUST be quoted 167 with both specific packet size and MTR as received by DUT/SUT 168 during the measurement. Packet MRR measured in packets-per-second 169 (or fps), bandwidth MRR expressed in bits-per-second (bps). 171 o Trial: a single measurement step. See [RFC2544] section 23. 173 o Trial duration: amount of time over which packets are transmitted 174 in a single measurement step. 176 2. MLRsearch Background 178 Multiple Loss Ratio search (MLRsearch) is a packet throughput search 179 algorithm suitable for deterministic systems (as opposed to 180 probabilistic systems). MLRsearch discovers multiple packet 181 throughput rates in a single search, with each rate associated with a 182 distinct Packet Loss Ratio (PLR) criteria. 184 For cases when multiple rates need to be found, this property makes 185 MLRsearch more efficient in terms of time execution, compared to 186 traditional throughput search algorithms that discover a single 187 packet rate per defined search criteria (e.g. a binary search 188 specified by [RFC2544]). MLRsearch reduces execution time even 189 further by relying on shorter trial durations of intermediate steps, 190 with only the final measurements conducted at the specified final 191 trial duration. This results in the shorter overall search execution 192 time when compared to a traditional binary search, while guaranteeing 193 the same results for deterministic systems. 195 In practice two rates with distinct PLRs are commonly used for packet 196 throughput measurements of NFV systems: Non Drop Rate (NDR) with 197 PLR=0 and Partial Drop Rate (PDR) with PLR>0. The rest of this 198 document describes MLRsearch for NDR and PDR. If needed, MLRsearch 199 can be adapted to discover more throughput rates with different pre- 200 defined PLRs. 202 Similarly to other throughput search approaches like binary search, 203 MLRsearch is effective for SUTs/DUTs with PLR curve that is 204 continuously flat or increasing with growing offered load. It may 205 not be as effective for SUTs/DUTs with abnormal PLR curves. 207 MLRsearch relies on traffic generator to qualify the received packet 208 stream as error-free, and invalidate the results if any disqualifying 209 errors are present e.g. out-of-sequence frames. 211 MLRsearch can be applied to both uni-directional and bi-directional 212 throughput tests. 214 For bi-directional tests, MLRsearch rates and ratios are aggregates 215 of both directions, based on the following assumptions: 217 o Traffic transmitted by traffic generator and received by SUT/DUT 218 has the same packet rate in each direction, in other words the 219 offered load is symmetric. 221 o SUT/DUT packet processing capacity is the same in both directions, 222 resulting in the same packet loss under load. 224 3. MLRsearch Overview 226 The main properties of MLRsearch: 228 o MLRsearch is a duration aware multi-phase multi-rate search 229 algorithm: 231 * Initial Phase determines promising starting interval for the 232 search. 234 * Intermediate Phases progress towards defined final search 235 criteria. 237 * Final Phase executes measurements according to the final search 238 criteria. 240 * Final search criteria are defined by following inputs: 242 + PLRs associated with NDR and PDR. 244 + Final trial duration. 246 + Measurement resolution. 248 o Initial Phase: 250 * Measure MRR over initial trial duration. 252 * Measured MRR is used as an input to the first intermediate 253 phase. 255 o Multiple Intermediate Phases: 257 * Trial duration: 259 + Start with initial trial duration in the first intermediate 260 phase. 262 + Converge geometrically towards the final trial duration. 264 * Track two values for NDR and two for PDR: 266 + The values are called lower_bound and upper_bound. 268 + Each value comes from a specific trial measurement: 270 - Most recent for that transmit rate. 272 - As such the value is associated with that measurement's 273 duration and loss. 275 + A bound can be valid or invalid: 277 - Valid lower_bound must conform with PLR search criteria. 279 - Valid upper_bound must not conform with PLR search 280 criteria. 282 - Example of invalid NDR lower_bound is if it has been 283 measured with non-zero loss. 285 - Invalid bounds are not real boundaries for the searched 286 value: 288 o They are needed to track interval widths. 290 - Valid bounds are real boundaries for the searched value. 292 - Each non-initial phase ends with all bounds valid. 294 - Bound can become invalid if it re-measured at a longer 295 trial duration in a sub-sequent phase. 297 * Search: 299 + Start with a large (lower_bound, upper_bound) interval 300 width, that determines measurement resolution. 302 + Geometrically converge towards the width goal of the phase. 304 + Each phase halves the previous width goal. 306 - First measurement of the next phase will be internal 307 search which always gives a valid bound and brings the 308 width to the new goal. 310 - Only one bound then needs to be re-measured with new 311 duration. 313 * Use of internal and external searches: 315 + External search: 317 - Measures at transmit rates outside the (lower_bound, 318 upper_bound) interval. 320 - Activated when a bound is invalid, to search for a new 321 valid bound by multiplying (for example doubling) the 322 interval width. 324 - It is a variant of "exponential search". 326 + Internal search: 328 - A "binary search" that measures at transmit rates within 329 the (lower_bound, upper_bound) valid interval, halving 330 the interval width. 332 o Final Phase: 334 * Executed with the final test trial duration, and the final 335 width goal that determines resolution of the overall search. 337 o Intermediate Phases together with the Final Phase are called Non- 338 Initial Phases. 340 The main benefits of MLRsearch vs. binary search include: 342 o In general MLRsearch is likely to execute more trials overall, but 343 likely less trials at a set final trial duration. 345 o In well behaving cases, e.g. when results do not depend on trial 346 duration, it greatly reduces (>50%) the overall duration compared 347 to a single PDR (or NDR) binary search over duration, while 348 finding multiple drop rates. 350 o In all cases MLRsearch yields the same or similar results to 351 binary search. 353 o Note: both binary search and MLRsearch are susceptible to 354 reporting non-repeatable results across multiple runs for very bad 355 behaving cases. 357 Caveats: 359 o Worst case MLRsearch can take longer than a binary search e.g. in 360 case of drastic changes in behaviour for trials at varying 361 durations. 363 4. Sample Implementation 365 Following is a brief description of a sample MLRsearch 366 implementation, which is a simlified version of the existing 367 implementation. 369 4.1. Input Parameters 371 1. *maximum_transmit_rate* - Maximum Transmit Rate (MTR) of packets 372 to be used by external traffic generator implementing MLRsearch, 373 limited by the actual Ethernet link(s) rate, NIC model or traffic 374 generator capabilities. 376 2. *minimum_transmit_rate* - minimum packet transmit rate to be used 377 for measurements. MLRsearch fails if lower transmit rate needs 378 to be used to meet search criteria. 380 3. *final_trial_duration* - required trial duration for final rate 381 measurements. 383 4. *initial_trial_duration* - trial duration for initial MLRsearch 384 phase. 386 5. *final_relative_width* - required measurement resolution 387 expressed as (lower_bound, upper_bound) interval width relative 388 to upper_bound. 390 6. *packet_loss_ratio* - maximum acceptable PLR search criterion for 391 PDR measurements. 393 7. *number_of_intermediate_phases* - number of phases between the 394 initial phase and the final phase. Impacts the overall MLRsearch 395 duration. Less phases are required for well behaving cases, more 396 phases may be needed to reduce the overall search duration for 397 worse behaving cases. 399 4.2. Initial Phase 401 1. First trial measures at configured maximum transmit rate (MTR) 402 and discovers maximum receive rate (MRR). 404 * IN: trial_duration = initial_trial_duration. 406 * IN: offered_transmit_rate = maximum_transmit_rate. 408 * DO: single trial. 410 * OUT: measured loss ratio. 412 * OUT: MRR = measured receive rate. If loss ratio is zero, MRR 413 is set below MTR so that interval width is equal to the width 414 goal of the first intermediate phase. 416 2. Second trial measures at MRR and discovers MRR2. 418 * IN: trial_duration = initial_trial_duration. 420 * IN: offered_transmit_rate = MRR. 422 * DO: single trial. 424 * OUT: measured loss ratio. 426 * OUT: MRR2 = measured receive rate. If loss ratio is zero, 427 MRR2 is set above MRR so that interval width is equal to the 428 width goal of the first intermediate phase. MRR2 could end up 429 being equal to MTR (for example if both measurements so far 430 had zero loss), which was already measured, step 3 is skipped 431 in that case. 433 3. Third trial measures at MRR2. 435 * IN: trial_duration = initial_trial_duration. 437 * IN: offered_transmit_rate = MRR2. 439 * DO: single trial. 441 * OUT: measured loss ratio. 443 4.3. Non-Initial Phases 445 1. Main loop: 447 1. IN: trial_duration for the current phase. Set to 448 initial_trial_duration for the first intermediate phase; to 449 final_trial_duration for the final phase; or to the element 450 of interpolating geometric sequence for other intermediate 451 phases. For example with two intermediate phases, 452 trial_duration of the second intermediate phase is the 453 geometric average of initial_trial_duration and 454 final_trial_duration. 456 2. IN: relative_width_goal for the current phase. Set to 457 final_relative_width for the final phase; doubled for each 458 preceding phase. For example with two intermediate phases, 459 the first intermediate phase uses quadruple of 460 final_relative_width and the second intermediate phase uses 461 double of final_relative_width. 463 3. IN: ndr_interval, pdr_interval from the previous main loop 464 iteration or the previous phase. If the previous phase is 465 the initial phase, both intervals are formed by a (correctly 466 ordered) pair of MRR2 and MRR. Note that the initial phase 467 is likely to create intervals with invalid bounds. 469 4. DO: According to the procedure described in point 2., either 470 exit the phase (by jumping to 1.7.), or calculate new 471 transmit rate to measure with. 473 5. DO: Perform the trial measurement at the new transmit rate 474 and trial_duration, compute its loss ratio. 476 6. DO: Update the bounds of both intervals, based on the new 477 measurement. The actual update rules are numerous, as NDR 478 external search can affect PDR interval and vice versa, but 479 the result agrees with rules of both internal and external 480 search. For example, any new measurement below an invalid 481 lower_bound becomes the new lower_bound, while the old 482 measurement (previously acting as the invalid lower_bound) 483 becomes a new and valid upper_bound. Go to next iteration 484 (1.3.), taking the updated intervals as new input. 486 7. OUT: current ndr_interval and pdr_interval. In the final 487 phase this is also considered to be the result of the whole 488 search. For other phases, the next phase loop is started 489 with the current results as an input. 491 2. New transmit rate (or exit) calculation (for point 1.4.): 493 1. If there is an invalid bound then prepare for external 494 search: 496 + IF the most recent measurement at NDR lower_bound transmit 497 rate had the loss higher than zero, then the new transmit 498 rate is NDR lower_bound decreased by two NDR interval 499 widths. 501 + Else, IF the most recent measurement at PDR lower_bound 502 transmit rate had the loss higher than PLR, then the new 503 transmit rate is PDR lower_bound decreased by two PDR 504 interval widths. 506 + Else, IF the most recent measurement at NDR upper_bound 507 transmit rate had no loss, then the new transmit rate is 508 NDR upper_bound increased by two NDR interval widths. 510 + Else, IF the most recent measurement at PDR upper_bound 511 transmit rate had the loss lower or equal to PLR, then the 512 new transmit rate is PDR upper_bound increased by two PDR 513 interval widths. 515 2. Else, if interval width is higher than the current phase 516 goal: 518 + IF NDR interval does not meet the current phase width 519 goal, prepare for internal search. The new transmit rate 520 is a in the middle of NDR lower_bound and NDR upper_bound. 522 + IF PDR interval does not meet the current phase width 523 goal, prepare for internal search. The new transmit rate 524 is a in the middle of PDR lower_bound and PDR upper_bound. 526 3. Else, if some bound has still only been measured at a lower 527 duration, prepare to re-measure at the current duration (and 528 the same transmit rate). The order of priorities is: 530 + NDR lower_bound, 532 + PDR lower_bound, 534 + NDR upper_bound, 536 + PDR upper_bound. 538 4. Else, do not prepare any new rate, to exit the phase. This 539 ensures that at the end of each non-initial phase all 540 intervals are valid, narrow enough, and measured at current 541 phase trial duration. 543 5. FD.io CSIT Implementation 545 The only known working implementation of MLRsearch is in the open- 546 source code running in Linux Foundation FD.io CSIT project 547 [FDio-CSIT-MLRsearch] as part of a Continuous Integration / 548 Continuous Development (CI/CD) framework. 550 MLRsearch is also available as a Python package in [PyPI-MLRsearch]. 552 5.1. Additional details 554 This document so far has been describing a simplified version of 555 MLRsearch algorithm. The full algorithm as implemented in CSIT 556 contains additional logic, which makes some of the details (but not 557 general ideas) above incorrect. Here is a short description of the 558 additional logic as a list of principles, explaining their main 559 differences from (or additions to) the simplified description, but 560 without detailing their mutual interaction. 562 1. Logarithmic transmit rate. 564 * In order to better fit the relative width goal, the interval 565 doubling and halving is done differently. 567 * For example, the middle of 2 and 8 is 4, not 5. 569 2. Optimistic maximum rate. 571 * The increased rate is never higher than the maximum rate. 573 * Upper bound at that rate is always considered valid. 575 3. Pessimistic minimum rate. 577 * The decreased rate is never lower than the minimum rate. 579 * If a lower bound at that rate is invalid, a phase stops 580 refining the interval further (until it gets re-measured). 582 4. Conservative interval updates. 584 * Measurements above the current upper bound never update a 585 valid upper bound, even if drop ratio is low. 587 * Measurements below the current lower bound always update any 588 lower bound if drop ratio is high. 590 5. Ensure sufficient interval width. 592 * Narrow intervals make external search take more time to find a 593 valid bound. 595 * If the new transmit increased or decreased rate would result 596 in width less than the current goal, increase/decrease more. 598 * This can happen if the measurement for the other interval 599 makes the current interval too narrow. 601 * Similarly, take care the measurements in the initial phase 602 create wide enough interval. 604 6. Timeout for bad cases. 606 * The worst case for MLRsearch is when each phase converges to 607 intervals way different than the results of the previous 608 phase. 610 * Rather than suffer total search time several times larger than 611 pure binary search, the implemented tests fail themselves when 612 the search takes too long (given by argument _timeout_). 614 7. Pessimistic external search. 616 * Valid bound becoming invalid on re-measurement with higher 617 duration is frequently a sign of SUT behaving in non- 618 deterministic way (from blackbox point of view). If the final 619 width interval goal is too narrow compared to width of rate 620 region where SUT is non-deterministic, it is quite likely that 621 there will be multiple invalid bounds before the external 622 search finds a valid one. 624 * In this case, external search can be sped up by increasing 625 interval width more rapidly. As only powers of two ensure the 626 subsequent internal search will not result in needlessly 627 narrow interval, a parameter _doublings_ is introduced to 628 control the pessimism of external search. For example three 629 doublings result in interval width being multiplied by eight 630 in each external search iteration. 632 5.1.1. FD.io CSIT Input Parameters 634 1. *maximum_transmit_rate* - Typical values: 2 * 14.88 Mpps for 64B 635 10GE link rate, 2 * 18.75 Mpps for 64B 40GE NIC (specific model). 637 2. *minimum_transmit_rate* - Value: 2 * 10 kpps (traffic generator 638 limitation). 640 3. *final_trial_duration* - Value: 30 seconds. 642 4. *initial_trial_duration* - Value: 1 second. 644 5. *final_relative_width* - Value: 0.005 (0.5%). 646 6. *packet_loss_ratio* - Value: 0.005 (0.5%). 648 7. *number_of_intermediate_phases* - Value: 2. The value has been 649 chosen based on limited experimentation to date. More 650 experimentation needed to arrive to clearer guidelines. 652 8. *timeout* - Limit for the overall search duration (for one 653 search). If MLRsearch oversteps this limit, it immediatelly 654 declares the test failed, to avoid wasting even more time on a 655 misbehaving SUT. Value: 600 (seconds). 657 9. *doublings* - Number of dublings when computing new interval 658 width in external search. Value: 2 (interval width is 659 quadroupled). Value of 1 is best for well-behaved SUTs, but 660 value of 2 has been found to decrease overall search time for 661 worse-behaved SUT configurations, contributing more to the 662 overall set of different SUT configurations tested. 664 5.2. Example MLRsearch Run 666 The following table shows data from a real test run in CSIT (using 667 the default input values as above). The first column is the phase, 668 the second is the trial measurement performed (aggregate 669 bidirectional offered load in megapackets per second, and trial 670 duration in seconds). Each of last four columns show one bound as 671 updated after the measurement (duration truncated to save space). 673 Loss ratio is not shown, but invalid bounds are marked with a plus 674 sign. 676 +-------+-----------+-----------+-----------+-----------+-----------+ 677 | Phase | Trial | NDR lower | NDR upper | PDR lower | PDR upper | 678 +-------+-----------+-----------+-----------+-----------+-----------+ 679 | init. | 37.50 | N/A | 37.50 1. | N/A | 37.50 1. | 680 | | 1.00 | | | | | 681 | | | | | | | 682 | init. | 10.55 | +10.55 1. | 37.50 1. | +10.55 1. | 37.50 1. | 683 | | 1.00 | | | | | 684 | | | | | | | 685 | init. | 9.437 | +9.437 1. | 10.55 1. | +9.437 1. | 10.55 1. | 686 | | 1.00 | | | | | 687 | | | | | | | 688 | int 1 | 6.053 | 6.053 1. | 9.437 1. | 6.053 1. | 9.437 1. | 689 | | 1.00 | | | | | 690 | | | | | | | 691 | int 1 | 7.558 | 7.558 1. | 9.437 1. | 7.558 1. | 9.437 1. | 692 | | 1.00 | | | | | 693 | | | | | | | 694 | int 1 | 8.446 | 8.446 1. | 9.437 1. | 8.446 1. | 9.437 1. | 695 | | 1.00 | | | | | 696 | | | | | | | 697 | int 1 | 8.928 | 8.928 1. | 9.437 1. | 8.928 1. | 9.437 1. | 698 | | 1.00 | | | | | 699 | | | | | | | 700 | int 1 | 9.179 | 8.928 1. | 9.179 1. | 9.179 1. | 9.437 1. | 701 | | 1.00 | | | | | 702 | | | | | | | 703 | int 1 | 9.052 | 9.052 1. | 9.179 1. | 9.179 1. | 9.437 1. | 704 | | 1.00 | | | | | 705 | | | | | | | 706 | int 1 | 9.307 | 9.052 1. | 9.179 1. | 9.179 1. | 9.307 1. | 707 | | 1.00 | | | | | 708 | | | | | | | 709 | int 2 | 9.115 | 9.115 5. | 9.179 1. | 9.179 1. | 9.307 1. | 710 | | 5.48 | | | | | 711 | | | | | | | 712 | int 2 | 9.243 | 9.115 5. | 9.179 1. | 9.243 5. | 9.307 1. | 713 | | 5.48 | | | | | 714 | | | | | | | 715 | int 2 | 9.179 | 9.115 5. | 9.179 5. | 9.243 5. | 9.307 1. | 716 | | 5.48 | | | | | 717 | | | | | | | 718 | int 2 | 9.307 | 9.115 5. | 9.179 5. | 9.243 5. | +9.307 5. | 719 | | 5.48 | | | | | 720 | | | | | | | 721 | int 2 | 9.687 | 9.115 5. | 9.179 5. | 9.307 5. | 9.687 5. | 722 | | 5.48 | | | | | 723 | | | | | | | 724 | int 2 | 9.495 | 9.115 5. | 9.179 5. | 9.307 5. | 9.495 5. | 725 | | 5.48 | | | | | 726 | | | | | | | 727 | int 2 | 9.401 | 9.115 5. | 9.179 5. | 9.307 5. | 9.401 5. | 728 | | 5.48 | | | | | 729 | | | | | | | 730 | final | 9.147 | 9.115 5. | 9.147 30 | 9.307 5. | 9.401 5. | 731 | | 30.0 | | | | | 732 | | | | | | | 733 | final | 9.354 | 9.115 5. | 9.147 30 | 9.307 5. | 9.354 30 | 734 | | 30.0 | | | | | 735 | | | | | | | 736 | final | 9.115 | +9.115 30 | 9.147 30 | 9.307 5. | 9.354 30 | 737 | | 30.0 | | | | | 738 | | | | | | | 739 | final | 8.935 | 8.935 30 | 9.115 30 | 9.307 5. | 9.354 30 | 740 | | 30.0 | | | | | 741 | | | | | | | 742 | final | 9.025 | 9.025 30 | 9.115 30 | 9.307 5. | 9.354 30 | 743 | | 30.0 | | | | | 744 | | | | | | | 745 | final | 9.070 | 9.070 30 | 9.115 30 | 9.307 5. | 9.354 30 | 746 | | 30.0 | | | | | 747 | | | | | | | 748 | final | 9.307 | 9.070 30 | 9.115 30 | 9.307 30 | 9.354 30 | 749 | | 30.0 | | | | | 750 +-------+-----------+-----------+-----------+-----------+-----------+ 752 6. IANA Considerations 754 No requests of IANA. 756 7. Security Considerations 758 Benchmarking activities as described in this memo are limited to 759 technology characterization of a DUT/SUT using controlled stimuli in 760 a laboratory environment, with dedicated address space and the 761 constraints specified in the sections above. 763 The benchmarking network topology will be an independent test setup 764 and MUST NOT be connected to devices that may forward the test 765 traffic into a production network or misroute traffic to the test 766 management network. 768 Further, benchmarking is performed on a "black-box" basis, relying 769 solely on measurements observable external to the DUT/SUT. 771 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 772 benchmarking purposes. Any implications for network security arising 773 from the DUT/SUT SHOULD be identical in the lab and in production 774 networks. 776 8. Acknowledgements 778 Many thanks to Alec Hothan of OPNFV NFVbench project for thorough 779 review and numerous useful comments and suggestions. 781 9. References 783 9.1. Normative References 785 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 786 Network Interconnect Devices", RFC 2544, 787 DOI 10.17487/RFC2544, March 1999, 788 . 790 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 791 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 792 May 2017, . 794 9.2. Informative References 796 [FDio-CSIT-MLRsearch] 797 "FD.io CSIT Test Methodology - MLRsearch", February 2020, 798 . 802 [PyPI-MLRsearch] 803 "MLRsearch 0.3.0, Python Package Index", February 2020, 804 . 806 Authors' Addresses 808 Maciek Konstantynowicz (editor) 809 Cisco Systems 811 Email: mkonstan@cisco.com 812 Vratko Polak (editor) 813 Cisco Systems 815 Email: vrpolak@cisco.com