idnits 2.17.1 draft-ietf-ippm-capacity-metric-method-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (March 9, 2020) is 1507 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC7680' is mentioned on line 235, but not defined == Unused Reference: 'RFC1242' is defined on line 837, but no explicit reference was found in the text == Unused Reference: 'RFC2889' is defined on line 860, but no explicit reference was found in the text == Unused Reference: 'RFC5180' is defined on line 874, but no explicit reference was found in the text == Unused Reference: 'RFC6201' is defined on line 879, but no explicit reference was found in the text == Unused Reference: 'RFC6412' is defined on line 884, but no explicit reference was found in the text == Unused Reference: 'RFC6985' is defined on line 904, but no explicit reference was found in the text == Unused Reference: 'RFC8468' is defined on line 926, but no explicit reference was found in the text == Unused Reference: 'RFC8239' is defined on line 939, but no explicit reference was found in the text == Unused Reference: 'TST009' is defined on line 943, but no explicit reference was found in the text == Unused Reference: 'VSPERF-b2b' is defined on line 950, but no explicit reference was found in the text == Unused Reference: 'VSPERF-BSLV' is defined on line 956, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 1242 ** Downref: Normative reference to an Informational RFC: RFC 2330 ** Downref: Normative reference to an Informational RFC: RFC 2544 ** Downref: Normative reference to an Informational RFC: RFC 2889 ** Downref: Normative reference to an Informational RFC: RFC 3148 ** Downref: Normative reference to an Informational RFC: RFC 5136 ** Downref: Normative reference to an Informational RFC: RFC 5180 ** Downref: Normative reference to an Informational RFC: RFC 6201 ** Downref: Normative reference to an Informational RFC: RFC 6412 ** Downref: Normative reference to an Informational RFC: RFC 6815 ** Downref: Normative reference to an Informational RFC: RFC 6985 ** Downref: Normative reference to an Informational RFC: RFC 7312 ** Downref: Normative reference to an Informational RFC: RFC 7799 ** Downref: Normative reference to an Experimental RFC: RFC 8337 ** Downref: Normative reference to an Informational RFC: RFC 8468 Summary: 15 errors (**), 0 flaws (~~), 14 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Morton 3 Internet-Draft AT&T Labs 4 Updates: ???? (if approved) R. Geib 5 Intended status: Standards Track Deutsche Telekom 6 Expires: September 10, 2020 L. Ciavattone 7 AT&T Labs 8 March 9, 2020 10 Metrics and Methods for IP Capacity 11 draft-ietf-ippm-capacity-metric-method-01 13 Abstract 15 This memo revisits the problem of Network Capacity metrics first 16 examined in RFC 5136. The memo specifies a more practical Maximum 17 IP-layer Capacity metric definition catering for measurement 18 purposes, and outlines the corresponding methods of measurement. 20 Requirements Language 22 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 23 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 24 "OPTIONAL" in this document are to be interpreted as described in BCP 25 14[RFC2119] [RFC8174] when, and only when, they appear in all 26 capitals, as shown here. 28 Status of This Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at https://datatracker.ietf.org/drafts/current/. 38 Internet-Drafts are draft documents valid for a maximum of six months 39 and may be updated, replaced, or obsoleted by other documents at any 40 time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress." 43 This Internet-Draft will expire on September 10, 2020. 45 Copyright Notice 47 Copyright (c) 2020 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (https://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with respect 55 to this document. Code Components extracted from this document must 56 include Simplified BSD License text as described in Section 4.e of 57 the Trust Legal Provisions and are provided without warranty as 58 described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 63 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . 4 64 3. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 4 65 4. General Parameters and Definitions . . . . . . . . . . . . . 5 66 5. IP-Layer Capacity Singleton Metric Definitions . . . . . . . 6 67 5.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 6 68 5.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 6 69 5.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 6 70 5.4. Related Round-Trip Delay and Loss Definitions . . . . . . 8 71 5.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 8 72 5.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 8 73 6. Maximum IP-Layer Capacity Metric Definitions (Statistic) . . 8 74 6.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 8 75 6.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 8 76 6.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 9 77 6.4. Related Round-Trip Delay and Loss Definitions . . . . . . 10 78 6.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 10 79 6.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 11 80 7. IP-Layer Sender Bit Rate Singleton Metric Definitions . . . . 11 81 7.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 11 82 7.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 12 83 7.3. Metric Definition . . . . . . . . . . . . . . . . . . . . 12 84 7.4. Discussion . . . . . . . . . . . . . . . . . . . . . . . 12 85 7.5. Reporting the Metric . . . . . . . . . . . . . . . . . . 12 86 8. Method of Measurement . . . . . . . . . . . . . . . . . . . . 13 87 8.1. Load Rate Adjustment Algorithm . . . . . . . . . . . . . 13 88 8.2. Measurement Qualification or Verification . . . . . . . . 14 89 8.3. Measurement Considerations . . . . . . . . . . . . . . . 15 90 9. Reporting Formats . . . . . . . . . . . . . . . . . . . . . . 16 91 10. Security Considerations . . . . . . . . . . . . . . . . . . . 18 92 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 93 12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 18 94 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 95 13.1. Normative References . . . . . . . . . . . . . . . . . . 18 96 13.2. Informative References . . . . . . . . . . . . . . . . . 20 97 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21 99 1. Introduction 101 The IETF's efforts to define Network and Bulk Transport Capacity have 102 been chartered and progressed for over twenty years. Over that time, 103 the performance community has seen development of Informative 104 definitions in [RFC3148] for Framework for Bulk Transport Capacity 105 (BTC), RFC 5136 for Network Capacity and Maximum IP-layer Capacity, 106 and the Experimental metric definitions and methods in [RFC8337], 107 Model-Based Metrics for BTC. 109 This memo revisits the problem of Network Capacity metrics examined 110 first in [RFC3148] and later in [RFC5136]. Maximum IP-Layer Capacity 111 and [RFC3148] Bulk Transfer Capacity (goodput) are different metrics. 112 Max IP-layer Capacity is like the theoretical goal for goodput. 113 There are many metrics in [RFC5136], such as Available Capacity. 114 Measurements depend on the network path under test and the use case. 115 Here, the main use case is to assess the maximum capacity of the 116 access network, with specific performance criteria used in the 117 measurement. 119 This memo recognizes the importance of a definition of a Maximum IP- 120 layer Capacity Metric at a time when access speeds have increased 121 dramatically; a definition that is both practical and effective for 122 the performance community's needs, including Internet users. The 123 metric definition is intended to use Active Methods of Measurement 124 [RFC7799], and a method of measurement is included. 126 The most direct active measurement of IP-layer Capacity would use IP 127 packets, but in practice a transport header is needed to traverse 128 address and port translators. UDP offers the most direct assessment 129 possibility, and in the [copycat][copycat] measurement study to 130 investigate whether UDP is viable as a general Internet transport 131 protocol, the authors found that a high percentage of paths tested 132 support UDP transport. A number of liaisons have been exchanged on 133 this topic [ refs to ITU-T SG12, ETSI STQ, BBF liaisons ], discussing 134 the laboratory and field tests that support the UDP-based approach to 135 IP-layer Capacity measurement. 137 This memo also recognizes the many updates to the IP Performance 138 Metrics Framework [RFC2330] published over twenty years, and makes 139 use of [RFC7312] for Advanced Stream and Sampling Framework, and RFC 140 8468 [RFC8468]IPv4, IPv6, and IPv4-IPv6 Coexistence Updates. 142 2. Scope and Goals 144 The scope of this memo is to define a metric and corresponding method 145 to unambiguously perform Active measurements of Maximum IP-Layer 146 Capacity, along with related metrics and methods. 148 The main goal is to harmonize the specified metric and method across 149 the industry, and this memo is the vehicle through which working 150 group (and eventually, IETF) consensus will be captured and 151 communicated to achieve broad agreement, and possibly result in 152 changes in the specifications of other Standards Development 153 Organizations (SDO) (through the SDO's normal contribution process, 154 or through liaison exchange). 156 A local goal is to aid efficient test procedures where possible, and 157 to recommend reporting with additional interpretation of the results. 158 Also, to foster the development of protocol support for this metric 159 and method of measurement (all active testing protocols currently 160 defined by the IPPM WG are UDP-based, meeting a key requirement of 161 these methods). 163 3. Motivation 165 As with any problem that has been worked for many years in various 166 SDOs without any special attempts at coordination, various solutions 167 for metrics and methods have emerged. 169 There are five factors that have changed (or begun to change) in the 170 2013-2019 time frame, and the presence of any one of them on the path 171 requires features in the measurement design to account for the 172 changes: 174 1. Internet access is no longer the bottleneck for many users. 176 2. Both speed and latency are important to user's satisfaction. 178 3. UDP's growing role in Transport, in areas where TCP once 179 dominated. 181 4. Content and applications moving physically closer to users. 183 5. Less emphasis on ISP gateway measurements, possibly due to less 184 traffic crossing ISP gateways in future. 186 4. General Parameters and Definitions 188 This section lists the REQUIRED input factors to specify a Sender or 189 Receiver metric. 191 o Src, the address of a host (such as the globally routable IP 192 address). 194 o Dst, the address of a host (such as the globally routable IP 195 address). 197 o i, the limit on the number of Hops a specific packet may visit as 198 it traverses from the host at Src to the host at Dst (such as the 199 TTL or Hop Limit). 201 o MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops). 203 o T0, the time at the start of measurement interval, when packets 204 are first transmitted from the Source. 206 o I, the duration of a measurement interval (default 10 sec) 208 o dt, the duration of N equal sub-intervals in I (default 1 sec) 210 o Tmax, a maximum waiting time for test packets to arrive at the 211 destination, set sufficiently long to disambiguate packets with 212 long delays from packets that are discarded (lost), such that the 213 distribution of one-way delay is not truncated. 215 o F, the number of different flows synthesized by the method 216 (default 1 flow) 218 o flow, the stream of packets with the same n-tuple of designated 219 header fields that (when held constant) result in identical 220 treatment in a multi-path decision (such as the decision taken in 221 load balancing). Note: The IPv6 flow label MAY be included in the 222 flow definition when routers have complied with [RFC6438] 223 guidelines at the Tunnel End Points (TEP), and the source of the 224 measurement is a TEP. 226 o Type-P, the complete description of the packets for which this 227 assessment applies (including the flow-defining fields). Note 228 that the UDP transport layer is one requirement specified below. 229 Type-P is a parallel concept to "population of interest" defined 230 in ITU-T Rec. Y.1540. 232 o PM, a list of fundamental metrics, such as loss, delay, and 233 reordering, and corresponding Target performance threshold. At 234 least one fundamental metric and Target performance threshold MUST 235 be supplied (such as One-way IP Packet Loss [RFC7680] equal to 236 zero). 238 A non-Parameter which is required for several metrics is defined 239 below: 241 o T, the host time of the *first* test packet's *arrival* as 242 measured at MP(Dst). There may be other packets sent between 243 source and destination hosts that are excluded, so this is the 244 time of arrival of the first packet used for measurement of the 245 metric. 247 Note that time stamps, sequnce numbers, etc. will be established by 248 the test protocol. 250 5. IP-Layer Capacity Singleton Metric Definitions 252 This section sets requirements for the following components to 253 support the Maximum IP-layer Capacity Metric. 255 5.1. Formal Name 257 Type-P-IP-Capacity, or informally called IP-layer Capacity. 259 Note that Type-P depends on the chosen method. 261 5.2. Parameters 263 This section lists the REQUIRED input factors to specify the metric, 264 beyond those listed in Section 4. 266 No additional Parameters are needed. 268 5.3. Metric Definitions 270 This section defines the REQUIRED aspects of the measureable IP-layer 271 Capacity metric (unless otherwise indicated) for measurements between 272 specified Source and Destination hosts: 274 Define the IP-layer capacity, C(T,I,PM), to be the number of IP-layer 275 bits (including header and data fields) in packets that can be 276 transmitted from the Src host and correctly received by the Dst host 277 during one contiguous sub-interval, dt. 279 The number of these IP-layer bits is designated n0[dtn,dtn+1] for a 280 specific dt. 282 When the packet size is known and of fixed size, the packet count 283 during a single sub-interval dt multiplied by the total bits in IP 284 header and data fields is equal to n0[dtn,dtn+1]. 286 Anticipating a Sample of Singletons, the interval dt SHOULD be set to 287 a natural number m so that T+I = T + m*dt with dtn+1 - dtn = dt and 288 with 1 <= n <= m. 290 Parameter PM represents other performance metrics [see section 291 Related Round-Trip Delay and Loss Definitions below]; their 292 measurement results SHALL be collected during measurement of IP-layer 293 Capacity and associated with the corresponding dtn for further 294 evaluation and reporting. 296 Mathematically, this definition can be represented as: 298 ( n0[dtn,dtn+1] ) 299 C(T,I,PM) = ------------------------- 300 dt 302 Equation for IP-Layer Capacity 304 and: 306 o n0 is the total number of IP-layer header and payload bits that 307 can be transmitted in Standard Formed packets from the Src host 308 and correctly received by the Dst host during one contiguous sub- 309 interval, dt in length, during the interval [T, T+I], 311 o C(T,I,PM) the IP-Layer Capacity, corresponds to the value of n0 312 measured in any sub-interval ending at dtn (meaning T + n*dt), 313 divided by the length of sub-interval, dt. 315 o all sub-intervals SHOULD be of equal duration. Choosing dt as 316 non-overlapping consecutive time intervals allows for a simple 317 implementation. 319 o The bit rate of the physical interface of the measurement device 320 must be higher than that of the link whose C(T,I,PM) is to be 321 measured. 323 Measurements according to these definitions SHALL use UDP transport 324 layer. 326 5.4. Related Round-Trip Delay and Loss Definitions 328 RTD[dtn-1,dtn] is defined as a sample of the [RFC2681] Round-trip 329 Delay between the Src host and the Dst host over the interval 330 [T,T+I]. The statistics used to to summarize RTD[dtn-1,dtn] MAY 331 include the minimum, maximum, and mean. 333 RTL[dtn-1,dtn] is defined as a sample of the [RFC6673] Round-trip 334 Loss between the Src host and the Dst host over the interval [T,T+I]. 335 The statistics used to to summarize RTL[dtn-1,dtn] MAY include the 336 lost packet count and the lost packet ratio. 338 5.5. Discussion 340 See the corresponding section for Maximum IP-Layer Capacity. 342 5.6. Reporting the Metric 344 The IP-Layer Capacity SHALL be reported with meaningful resolution, 345 in units of Megabits per second (Mbps). 347 The Related Round Trip Delay and/or Loss metric measurements for the 348 same Singleton SHALL be reported, also with meaningful resolution for 349 the values measured. 351 Individual Capacity measurements MAY be reported in a manner 352 consistent with the Maximum IP-Layer Capacity, see Section 9. 354 6. Maximum IP-Layer Capacity Metric Definitions (Statistic) 356 This section sets requirements for the following components to 357 support the Maximum IP-layer Capacity Metric. 359 6.1. Formal Name 361 Type-P-Max-IP-Capacity, or informally called Maximum IP-layer 362 Capacity. 364 Note that Type-P depends on the chosen method. 366 6.2. Parameters 368 This section lists the REQUIRED input factors to specify the metric, 369 beyond those listed in Section 4. 371 No additional Parameters or definitions are needed. 373 6.3. Metric Definitions 375 This section defines the REQUIRED aspects of the Maximum IP-layer 376 Capacity metric (unless otherwise indicated) for measurements between 377 specified Source and Destination hosts: 379 Define the Maximum IP-layer capacity, Maximum_C(T,I,PM), to be the 380 maximum number of IP-layer bits n0[dtn,dtn+1] that can be transmitted 381 in packets from the Src host and correctly received by the Dst host, 382 over all dt length intervals in [T, T+I], and meeting the PM 383 criteria. Equivalently the Maximum of a Sample of size m of 384 C(T,I,PM) collected during the interval [T, T+I] and meeting the PM 385 criteria. 387 The interval dt SHOULD be set to a natural number m so that T+I = T + 388 m*dt with dtn+1 - dtn = dt and with 1 <= n <= m. 390 Parameter PM represents the other performance metrics [see section 391 Related Round-Trip Delay and Loss Definitions below] and their 392 measurement results for the maximum IP-layer capacity. At least one 393 target performance threshold (PM criterion) MUST be defined. If more 394 than one target performance threshold is defined, then the sub- 395 interval with maximum number of bits transmitted MUST meet all the 396 target performance thresholds. 398 Mathematically, this definition can be represented as: 400 max ( n0[dtn,dtn+1] ) 401 [T,T+I] 402 Maximum_C(T,I,PM) = ------------------------- 403 dt 404 where: 405 T T+I 406 _________________________________________ 407 | | | | | | | | | | | 408 dtn=1 2 3 4 5 6 7 8 9 10 n+1 409 n=m 411 Equation for Maximum Capacity 413 and: 415 o n0 is the total number of IP-layer header and payload bits that 416 can be transmitted in Standard Formed packets from the Src host 417 and correctly received by the Dst host during one contiguous sub- 418 interval, dt in length, during the interval [T, T+I], 420 o Maximum _C(T,I,PM) the Maximum IP-Layer Capacity, corresponds to 421 the maximum value of n0 measured in any sub-interval ending at dtn 422 (meaning T + n*dt), divided by the constant length of all sub- 423 intervals, dt. 425 o all sub-intervals SHOULD be of equal duration. Choosing dt as 426 non-overlapping consecutive time intervals allows for a simple 427 implementation. 429 o The bit rate of the physical interface of the measurement systems 430 must be higher than that of the link whose Maximum _C(T,I,PM) is 431 to be measured (the bottleneck link). 433 In this definition, the m sub-intervals can be viewed as trials when 434 the Src host varies the transmitted packet rate, searching for the 435 maximum n0 that meets the PM criteria measured at the Dst host in a 436 test of duration, I. When the transmitted packet rate is held 437 constant at the Src host, the m sub-intervals may also be viewed as 438 trials to evaluate the stability of n0 and metric(s) in the PM list 439 over all dt-length intervals in I. 441 Measurements according to these definitions SHALL use UDP transport 442 layer. 444 6.4. Related Round-Trip Delay and Loss Definitions 446 RTD[dtn,dtn+1] is defined as a sample of the [RFC2681] Round-trip 447 Delay between the Src host and the Dst host over the interval 448 [T,T+I], and corresponds to the dt interval containing 449 Maximum_C(T,I,PM). The statistics used to to summarize 450 RTD[dtn,dtn+1] MAY include the minimum, maximum, and mean. 452 RTL[dtn,dtn+1] is defined as a sample of the [RFC6673] Round-trip 453 Loss between the Src host and the Dst host over the interval [T,T+I] 454 and corresponds to the dt interval containing Maximum_C(T,I,PM). The 455 statistics used to to summarize RTL[dtn-1,dtn] MAY include the lost 456 packet count and the lost packet ratio. 458 6.5. Discussion 460 If traffic conditioning applies along a path for which Maximum 461 _C(T,I,PM) is to be determined, different values for dt SHOULD be 462 picked and measurements be executed during multiple intervals [T, 463 T+I]. Any single interval dt SHOULD be chosen so that is an integer 464 multiple of increasing values k times serialisation delay of a path 465 MTU at the physical interface speed where traffic conditioning is 466 expected. This should avoid taking configured burst tolerance 467 singletons as a valid Maximum _C(T,I,PM) result. 469 A Maximum_C(T,I,PM) without any indication of bottleneck congestion, 470 be that an increasing latency, packet loss or ECN marks during a 471 measurement interval I, is likely to underestimate Maximum_C(T,I,PM). 473 6.6. Reporting the Metric 475 The Maximum IP-Layer Capacity SHALL be reported with meaningful 476 resolution, in units of Megabits per second. 478 The Related Round Trip Delay and/or Loss metric measurements for the 479 same Singleton SHALL be reported, also with meaningful resolution for 480 the values measured. 482 When there are demonstrated and repeatable Capacity modes in the 483 Sample, then the Maximum IP-Layer Capacity SHALL be reported for each 484 mode, along with the relative time from the beginning of the stream 485 that the mode was observed to be present. Bimodal Maxima have been 486 observed with some services, sometimes called a "turbo mode" 487 intending to deliver short transfers more quickly, or reduce the 488 initial buffering time for some video streams. Note that modes 489 lasting less than dt duration will not be detected. 491 Some transmission technologies have multiple methods of operation 492 that may be activated when channel conditions degrade or improve, and 493 these transmission methods may determine the Maximum IP-Layer 494 Capacity. Examples include line-of-sight microwave modulator 495 constellations, or cellular modem technologies where the changes may 496 be initiated by a user moving from one coverage area to another. 497 Operation in the different transmission methods may be observed over 498 time, but the modes of Maximum IP-Layer Capacity will not be 499 activated deterministically as with the "turbo mode" described in the 500 paragraph above. 502 7. IP-Layer Sender Bit Rate Singleton Metric Definitions 504 This section sets requirements for the following components to 505 support the IP-layer Sender Bitrate Metric. 507 7.1. Formal Name 509 Type-P-IP-Sender-Bit-Rate, or informally called IP-layer Sender 510 Bitrate. 512 Note that Type-P depends on the chosen method. 514 7.2. Parameters 516 This section lists the REQUIRED input factors to specify the metric, 517 beyond those listed in Section 4. 519 o S, the duration of the measurement interval at the Source 521 o st, the nominal duration of N sub-intervals in S (default = 0.05 522 seconds) 524 S SHALL be longer than I, primarily to account for on-demand 525 activation of the path, or any preamble to testing required. 527 st SHOULD be much smaller than the sub-interval dt. The st parameter 528 does not have relevance when the Source is transmitting at a fixed 529 rate throughout S. 531 7.3. Metric Definition 533 This section defines the REQUIRED aspects of the IP-layer Sender 534 Bitrate metric (unless otherwise indicated) for measurements at the 535 specified Source on packets addressed for the intended Destination 536 host and matching the required Type-P: 538 Define the IP-layer Sender Bit Rate, B(S,st), to be the number of IP- 539 layer bits (including header and data fields) that are transmitted 540 from the Source during one contiguous sub-interval, st, during the 541 test interval S (where S SHALL be longer than I), and where the 542 fixed-size packet count during that single sub-interval st also 543 provides the number of IP-layer bits in any interval: n0[stn-1,stn]. 545 Measurements according to these definitions SHALL use UDP transport 546 layer. Any feedback from Dst host to Src host received by Src host 547 during an interval [stn-1,stn] MUST NOT result in an adaptation of 548 the Src host traffic conditioning during this interval. 550 7.4. Discussion 552 Both the Sender and Receiver or (source and destination) bit rates 553 SHOULD be assessed as part of a measurement. 555 7.5. Reporting the Metric 557 The IP-Layer Sender Bit Rate SHALL be reported with meaningful 558 resolution, in units of Megabits per second. 560 Individual IP-Layer Sender Bit Rate measurements are discussed 561 further in Section 9. 563 8. Method of Measurement 565 The duration of a test, I, MUST be constrained in a production 566 network, since this is an active test method and it will likely cause 567 congestion on the Src to Dst host path during a test. 569 Additional Test methods and configurations may be provided in this 570 section, after review and further testing. 572 8.1. Load Rate Adjustment Algorithm 574 A table is pre-built defining all the offered load rates that will be 575 supported (R1 - Rn, in ascending order). Each rate is defined as 576 datagrams of size S, sent as a burst of count C, every time interval 577 T. While it is advantageous to use datagrams of as large a size as 578 possible, it may be prudent to use a slightly smaller maximum that 579 allows for secondary protocol headers and/or tunneling without 580 resulting in IP-layer fragmentation. 582 At the beginning of a test, the sender begins sending at rate R1 and 583 the receiver starts a feedback timer at interval F (while awaiting 584 inbound datagrams). As datagrams are received they are checked for 585 sequence number anomalies (loss, out-of-order, duplication, etc.) and 586 the delay variation is measured (one-way or round-trip). This 587 information is accumulated until the feedback timer F expires and a 588 status feedback message is sent from the receiver back to the sender, 589 to communicate this information. The accumulated statistics are then 590 reset by the receiver for the next feedback interval. As feedback 591 messages are received back at the sender, they are evaluated to 592 determine how to adjust the current offered load rate (Rx). 594 If the feedback indicates that there were no sequence number 595 anomalies AND the delay variation was below the lower threshold, the 596 offered load rate is increased. If congestion has not been confirmed 597 up to this point, the offered load rate is increased by more than one 598 rate (e.g., Rx+10). This allows the offered load to quickly reach a 599 near-maximum rate. Conversely, if congestion has been previously 600 confirmed, the offered load rate is only increased by one (Rx+1). 602 If the feedback indicates that sequence number anomalies were 603 detected OR the delay variation was above the upper threshold, the 604 offered load rate is decreased. If congestion is confirmed by the 605 current feedback message being processed, the offered load rate is 606 decreased by more than one rate (e.g., Rx-30). This one-time 607 reduction is intended to compensate for the fast initial ramp-up. In 608 all other cases, the offered load rate is only decreased by one (Rx- 609 1). 611 If the feedback indicates that there were no sequence number 612 anomalies AND the delay variation was above the lower threshold, but 613 below the upper threshold, the offered load rate is not changed. 614 This allows time for recent changes in the offered load rate to 615 stabilize, and the feedback to represent current conditions more 616 accurately. 618 Lastly, the method for confirming congestion is that there were 619 sequence number anomalies OR the delay variation was above the upper 620 threshold for two consecutive feedback intervals. The algorithm 621 described above is also presented in ITU-T Rec. Y.1540, 2020 622 version[Y.1540], in Annexes A and B. 624 8.2. Measurement Qualification or Verification 626 When assessing a Maximum rate as the metric specifies, artificially 627 high (optimistic) values might be measured until some buffer on the 628 path is filled. Other causes include bursts of back-to-back packets 629 with idle intervals delivered by a path, while the measurement 630 interval (dt) is small and aligned with the bursts. The artificial 631 values might result in an un-sustainable Maximum Capacity observed 632 when the method of measurement is searching for the Maximum, and that 633 would not do. This situation is different from the bi-modal service 634 rates (discussed under Reporting), which are characterized by a 635 multi-second duration (much longer than the measured RTT) and 636 repeatable behavior. 638 There are many ways that the Method of Measurement could handle this 639 false-max issue. The default value for measurement of singletons (dt 640 = 1 second) has proven to a be of practical value during tests of 641 this method, allows the bimodal service rates to be characterized, 642 and it has an obvious alignment with the reporting units (Mbps). 644 Another approach comes from Section 24 of RFC 2544[RFC2544] and its 645 discussion of Trial duration, where relatively short trials conducted 646 as part of the search are followed by longer trials to make the final 647 determination. In the production network, measurements of singletons 648 and samples (the terms for trials and tests of Lab Benchmarking) must 649 be limited in duration because they may be service-affecting. But 650 there is sufficient value in repeating a sample with a fixed sending 651 rate determined by the previous search for the Max IP-layer Capacity, 652 to qualify the result in terms of the other performance metrics 653 measured at the same time. 655 A qualification measurement for the search result is a subsequent 656 measurement, sending at a fixed 99.x % of the Max IP-layer Capacity 657 for I, or an indefinite period. The same Max Capacity Metric is 658 applied, and the Qualification for the result is a sample without 659 packet loss or a growing minimum delay trend in subsequent singletons 660 (or each dt of the measurement interval, I). Samples exhibiting 661 losses or increasing queue occupation require a repeated search and/ 662 or test at reduced fixed sender rate for qualification. 664 Here, as with any Active Capacity test, the test duration must be 665 kept short. 10 second tests for each direction of transmission are 666 common today. The default measurement interval specified here is I = 667 10 seconds). In combination with a fast search method and user- 668 network coordination, the concerns raised in RFC 6815[RFC6815] are 669 alleviated. The method for assessing Max IP Capacity is different 670 from classic [RFC2544] methods: they use short term load adjustment 671 and are sensitive to loss and delay, like other congestion control 672 algorithms used on the Internet every day. 674 8.3. Measurement Considerations 676 In general, the wide-spread measurements that this memo encourages 677 will encounter wide-spread behaviors. The bimodal IP Capacity 678 behaviors already discussed in Section 6.6 are good examples. 680 In general, it is RECOMMENDED to locate test endpoints as close to 681 the intended measured link(s) as practical (this is not always 682 possible for reasons of scale; there is a limit on number of test 683 endpoints coming from many perspecitves, management and measurement 684 traffic for example). 686 The path measured may be state-full based on many factors, and the 687 Parameter "Time of day" when a test starts may not be enough 688 information. Repeatable testing may require the time from the 689 beginning of a measured flow, and how the flow is constructed 690 including how much traffic has already been sent on that flow when a 691 state-change is observed, because the state-change may be based on 692 time or bytes sent or both. 694 Many different traffic shapers and on-demand access technologies may 695 be encountered, as anticipated in [RFC7312], and play a key role in 696 measurement results. Methods MUST be prepared to provide a short 697 preamble transmission to activate on-demand access, and to discard 698 the preamble from subsequent test results. 700 Conditions which might be encountered during measurement, where 701 packet losses may occur independently from the measurement sending 702 rate: 704 1. Congestion of an interconnection or backbone interface may appear 705 as packet losses distributed over time in the test stream, due to 706 much higher rate interfaces in the backbone. 708 2. Packet loss due to use of Random Early Detection (RED) or other 709 active queue management. 711 3. There may be only small delay variation independent of sending 712 rate under these conditions, too. 714 4. Persistent competing traffic on measurement paths that include 715 shared media may cause random packet losses in the test stream. 717 It is possible to mitigate these conditions using the flexibility of 718 the load-rate adjusting algorithm described in Section 8.1 above 719 (tuning specific parameters). 721 In general, results depend on the sending stream characteristics; the 722 measurement community has known this for a long time, and needs to 723 keep it front of mind. Although the default is a single flow (F=1) 724 for testing, use of multiple flows may be advantageous for the 725 following reasons: 727 1. the test hosts may be able to create higher load than with a 728 single flow, or parallel test hosts may be used to generate 1 729 flow each. 731 2. there may be link aggregation present (flow-based load balancing) 732 and multiple flows are needed to occupy each member of the 733 aggregate. 735 Each flow would be controlled using its own implementation of the 736 Load Adjustment (Search) Algorithm. 738 As testing continues, implementers should expect some evolution in 739 the methods. 741 9. Reporting Formats 743 The singleton IP-Layer Capacity results SHOULD be accompanied by the 744 context under which they were measured. 746 o timestamp (especially the time when the maximum was observed in 747 dtn) 749 o source and destination (by IP or other meaningful ID) 751 o other inner parameters of the test case (Section 4) 753 o outer parameters, such as "done in motion" or other factors 754 belonging to the context of the measurement 756 o result validity (indicating cases where the process was somehow 757 interrupted or the attempt failed) 759 o a field where unusual circumstances could be documented, and 760 another one for "ignore/mask out" purposes in further processing 762 The Maximum IP-Layer Capacity results SHOULD be reported in the 763 format of a table with a row for each of the test Phases and Number 764 of Flows. There SHOULD be columns for the phases with number of 765 flows, and for the resultant Maximum IP-Layer Capacity results for 766 the aggregate and each flow tested. 768 As mentioned in Section 6.6, bi-modal (or multi-modal) maxima SHALL 769 be reported for each mode separately. 771 +--------------+----------------------+-----------+-----------------+ 772 | Phase, # | Max IP-Layer | Loss | RTT min, max, | 773 | Flows | Capacity, Mbps | Ratio | msec | 774 +--------------+----------------------+-----------+-----------------+ 775 | Search,1 | 967.31 | 0.0002 | 30, 58 | 776 | Verify,1 | 966.00 | 0.0000 | 30, 38 | 777 +--------------+----------------------+-----------+-----------------+ 779 Maximum IP-layer Capacity Results 781 Static and configuration parameters: 783 The sub-interval time, dt, MUST accompany a report of Maximum IP- 784 Layer Capacity results, and the remaining Parameters from Section 4, 785 General Parameters. 787 The PM list metrics corresponding to the sub-interval where the 788 Maximum Capacity occurred MUST accompany a report of Maximum IP-Layer 789 Capacity results, for each test phase. 791 The IP-Layer Sender Bit rate results SHOULD be reported in the format 792 of a table with a row for each of the test Phases, sub-intervals (st) 793 and Number of Flows. There SHOULD be columns for the phases with 794 number of flows, and for the resultant IP-Layer Sender Bit rate 795 results for the aggregate and each flow tested. 797 +------------------------+-------------+-----------------------+----+ 798 | Phase, Flow or | st, sec | Sender Bit Rate, Mbps | ?? | 799 | Aggregate | | | | 800 +------------------------+-------------+-----------------------+----+ 801 | Search,1 | 0.00 - 0.05 | 345 | __ | 802 | Search,2 | 0.00 - 0.05 | 289 | __ | 803 | Search,Agg | 0.00 - 0.05 | 634 | __ | 804 +------------------------+-------------+-----------------------+----+ 806 IP-layer Sender Bit Rate Results 808 Static and configuration parameters: 810 The subinterval time, st, MUST accompany a report of Sender IP-Layer 811 Bit Rate results. 813 Also, the values of the remaining Parameters from Section 4, General 814 Parameters, MUST be reported. 816 10. Security Considerations 818 Active metrics and measurements have a long history of security 819 considerations [add references to LMAP Framework, etc.]. 821 823 11. IANA Considerations 825 This memo makes no requests of IANA. 827 12. Acknowledgements 829 Thanks to Joachim Fabini, Matt Mathis, Ignacio Alvarez-Hamelin, and 830 Wolfgang Balzer for their extensive comments on the memo and related 831 topics. 833 13. References 835 13.1. Normative References 837 [RFC1242] Bradner, S., "Benchmarking Terminology for Network 838 Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, 839 July 1991, . 841 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 842 Requirement Levels", BCP 14, RFC 2119, 843 DOI 10.17487/RFC2119, March 1997, 844 . 846 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 847 "Framework for IP Performance Metrics", RFC 2330, 848 DOI 10.17487/RFC2330, May 1998, 849 . 851 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 852 Network Interconnect Devices", RFC 2544, 853 DOI 10.17487/RFC2544, March 1999, 854 . 856 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 857 Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, 858 September 1999, . 860 [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology 861 for LAN Switching Devices", RFC 2889, 862 DOI 10.17487/RFC2889, August 2000, 863 . 865 [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining 866 Empirical Bulk Transfer Capacity Metrics", RFC 3148, 867 DOI 10.17487/RFC3148, July 2001, 868 . 870 [RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity", 871 RFC 5136, DOI 10.17487/RFC5136, February 2008, 872 . 874 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. 875 Dugatkin, "IPv6 Benchmarking Methodology for Network 876 Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May 877 2008, . 879 [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera, 880 "Device Reset Characterization", RFC 6201, 881 DOI 10.17487/RFC6201, March 2011, 882 . 884 [RFC6412] Poretsky, S., Imhoff, B., and K. Michielsen, "Terminology 885 for Benchmarking Link-State IGP Data-Plane Route 886 Convergence", RFC 6412, DOI 10.17487/RFC6412, November 887 2011, . 889 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 890 for Equal Cost Multipath Routing and Link Aggregation in 891 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 892 . 894 [RFC6673] Morton, A., "Round-Trip Packet Loss Metrics", RFC 6673, 895 DOI 10.17487/RFC6673, August 2012, 896 . 898 [RFC6815] Bradner, S., Dubray, K., McQuaid, J., and A. Morton, 899 "Applicability Statement for RFC 2544: Use on Production 900 Networks Considered Harmful", RFC 6815, 901 DOI 10.17487/RFC6815, November 2012, 902 . 904 [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet 905 Sizes for Additional Testing", RFC 6985, 906 DOI 10.17487/RFC6985, July 2013, 907 . 909 [RFC7312] Fabini, J. and A. Morton, "Advanced Stream and Sampling 910 Framework for IP Performance Metrics (IPPM)", RFC 7312, 911 DOI 10.17487/RFC7312, August 2014, 912 . 914 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 915 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 916 May 2016, . 918 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 919 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 920 May 2017, . 922 [RFC8337] Mathis, M. and A. Morton, "Model-Based Metrics for Bulk 923 Transport Capacity", RFC 8337, DOI 10.17487/RFC8337, March 924 2018, . 926 [RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. 927 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for 928 the IP Performance Metrics (IPPM) Framework", RFC 8468, 929 DOI 10.17487/RFC8468, November 2018, 930 . 932 13.2. Informative References 934 [copycat] Edleine, K., Kuhlewind, K., Trammell, B., and B. Donnet, 935 "copycat: Testing Differential Treatment of New Transport 936 Protocols in the Wild (ANRW '17)", July 2017, 937 . 939 [RFC8239] Avramov, L. and J. Rapp, "Data Center Benchmarking 940 Methodology", RFC 8239, DOI 10.17487/RFC8239, August 2017, 941 . 943 [TST009] Morton, R. A., "ETSI GS NFV-TST 009 V3.1.1 (2018-10), 944 "Network Functions Virtualisation (NFV) Release 3; 945 Testing; Specification of Networking Benchmarks and 946 Measurement Methods for NFVI"", October 2018, 947 . 950 [VSPERF-b2b] 951 Morton, A., "Back2Back Testing Time Series (from CI)", 952 June 2017, . 956 [VSPERF-BSLV] 957 Morton, A. and S. Rao, "Evolution of Repeatability in 958 Benchmarking: Fraser Plugfest (Summary for IETF BMWG)", 959 July 2018, 960 . 964 [Y.1540] Y.1540, I. R., "Internet protocol data communication 965 service - IP packet transfer and availability performance 966 parameters", January 2020, 967 . 969 Authors' Addresses 971 Al Morton 972 AT&T Labs 973 200 Laurel Avenue South 974 Middletown,, NJ 07748 975 USA 977 Phone: +1 732 420 1571 978 Fax: +1 732 368 1192 979 Email: acm@research.att.com 981 Ruediger Geib 982 Deutsche Telekom 983 Heinrich Hertz Str. 3-7 984 Darmstadt 64295 985 Germany 987 Phone: +49 6151 5812747 988 Email: Ruediger.Geib@telekom.de 989 Len Ciavattone 990 AT&T Labs 991 200 Laurel Avenue South 992 Middletown,, NJ 07748 993 USA 995 Email: lencia@att.com