idnits 2.17.1 draft-morton-ippm-capacity-metric-method-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (October 18, 2019) is 1651 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC7680' is mentioned on line 243, but not defined == Unused Reference: 'RFC1242' is defined on line 756, but no explicit reference was found in the text == Unused Reference: 'RFC2889' is defined on line 779, but no explicit reference was found in the text == Unused Reference: 'RFC5180' is defined on line 793, but no explicit reference was found in the text == Unused Reference: 'RFC6201' is defined on line 798, but no explicit reference was found in the text == Unused Reference: 'RFC6412' is defined on line 803, but no explicit reference was found in the text == Unused Reference: 'RFC6985' is defined on line 823, but no explicit reference was found in the text == Unused Reference: 'RFC8468' is defined on line 845, but no explicit reference was found in the text == Unused Reference: 'RFC8239' is defined on line 858, but no explicit reference was found in the text == Unused Reference: 'TST009' is defined on line 862, but no explicit reference was found in the text == Unused Reference: 'VSPERF-b2b' is defined on line 869, but no explicit reference was found in the text == Unused Reference: 'VSPERF-BSLV' is defined on line 875, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 1242 ** Downref: Normative reference to an Informational RFC: RFC 2330 ** Downref: Normative reference to an Informational RFC: RFC 2544 ** Downref: Normative reference to an Informational RFC: RFC 2889 ** Downref: Normative reference to an Informational RFC: RFC 3148 ** Downref: Normative reference to an Informational RFC: RFC 5136 ** Downref: Normative reference to an Informational RFC: RFC 5180 ** Downref: Normative reference to an Informational RFC: RFC 6201 ** Downref: Normative reference to an Informational RFC: RFC 6412 ** Downref: Normative reference to an Informational RFC: RFC 6815 ** Downref: Normative reference to an Informational RFC: RFC 6985 ** Downref: Normative reference to an Informational RFC: RFC 7312 ** Downref: Normative reference to an Informational RFC: RFC 7799 ** Downref: Normative reference to an Experimental RFC: RFC 8337 ** Downref: Normative reference to an Informational RFC: RFC 8468 Summary: 15 errors (**), 0 flaws (~~), 14 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Morton 3 Internet-Draft AT&T Labs 4 Updates: ???? (if approved) R. Geib 5 Intended status: Standards Track Deutsche Telekom 6 Expires: April 20, 2020 L. Ciavattone 7 AT&T Labs 8 October 18, 2019 10 Metrics and Methods for IP Capacity 11 draft-morton-ippm-capacity-metric-method-00 13 Abstract 15 This memo revisits the problem of Network Capacity metrics first 16 examined in RFC 5136. The memo specifies a more practical Maximum 17 IP-layer Capacity metric definition catering for measurement 18 purposes, and outlines the corresponding methods of measurement. 20 Requirements Language 22 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 23 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 24 "OPTIONAL" in this document are to be interpreted as described in BCP 25 14[RFC2119] [RFC8174] when, and only when, they appear in all 26 capitals, as shown here. 28 Status of This Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at https://datatracker.ietf.org/drafts/current/. 38 Internet-Drafts are draft documents valid for a maximum of six months 39 and may be updated, replaced, or obsoleted by other documents at any 40 time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress." 43 This Internet-Draft will expire on April 20, 2020. 45 Copyright Notice 47 Copyright (c) 2019 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (https://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with respect 55 to this document. Code Components extracted from this document must 56 include Simplified BSD License text as described in Section 4.e of 57 the Trust Legal Provisions and are provided without warranty as 58 described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 63 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . 4 64 3. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 4 65 4. General Parameters and Definitions . . . . . . . . . . . . . 5 66 5. IP-Layer Capacity Singleton Metric Definitions . . . . . . . 6 67 5.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 6 68 5.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 6 69 5.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 6 70 5.4. Related Round-Trip Delay and Loss Definitions . . . . . . 8 71 5.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 8 72 5.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 8 73 6. Maximum IP-Layer Capacity Metric Definitions (Statistic) . . 8 74 6.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 8 75 6.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 8 76 6.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 9 77 6.4. Related Round-Trip Delay and Loss Definitions . . . . . . 10 78 6.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 10 79 6.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 11 80 7. IP-Layer Sender Bit Rate Singleton Metric Definitions . . . . 11 81 7.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 11 82 7.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 11 83 7.3. Metric Definition . . . . . . . . . . . . . . . . . . . . 12 84 7.4. Discussion . . . . . . . . . . . . . . . . . . . . . . . 12 85 7.5. Reporting the Metric . . . . . . . . . . . . . . . . . . 12 86 8. Method of Measurement . . . . . . . . . . . . . . . . . . . . 12 87 8.1. Load Rate Adjustment Algorithm (from udpst) . . . . . . . 12 88 8.2. Measurement Qualification or Verification . . . . . . . . 13 89 8.3. Measurement Considerations . . . . . . . . . . . . . . . 14 90 9. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . 15 91 10. Security Considerations . . . . . . . . . . . . . . . . . . . 16 92 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 93 12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 16 94 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 16 95 13.1. Normative References . . . . . . . . . . . . . . . . . . 16 96 13.2. Informative References . . . . . . . . . . . . . . . . . 18 97 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 19 99 1. Introduction 101 The IETF's efforts to define Network and Bulk Transport Capacity have 102 been chartered and progressed for over twenty years. Over that time, 103 the performance community has seen development of Informative 104 definitions in [RFC3148] for Framework for Bulk Transport Capacity 105 (BTC), RFC 5136 for Network Capacity and Maximum IP-layer Capacity, 106 and the Experimental metric definitions and methods in [RFC8337], 107 Model-Based Metrics for BTC. 109 This memo revisits the problem of Network Capacity metrics examined 110 first in [RFC3148] and later in [RFC5136]. Maximum IP-Layer Capacity 111 and [RFC3148] Bulk Transfer Capacity (goodput) are different metrics. 112 Max IP-layer Capacity is like the theoretical goal for goodput. 113 There are many metrics in [RFC5136], such as Available Capacity. 114 Measurements depend on the network path under test and the use case. 115 Here, the main use case is to assess the maximum capacity of the 116 access network, with specific performance criteria used in the 117 measurement. 119 This memo recognizes the importance of a definition of a Maximum IP- 120 layer Capacity Metric at a time when access speeds have increased 121 dramatically; a definition that is both practical and effective for 122 the performance community's needs, including Internet users. The 123 metric definition is intended to use Active Methods of Measurement 124 [RFC7799], and a method of measurement is included. 126 The most direct active measurement of IP-layer Capacity would use IP 127 packets, but in practice a transport header is needed to traverse 128 address and port translators. UDP offers the most direct assessment 129 possibility, and in the [copycat][copycat] measurement study to 130 investigate whether UDP is viable as a general Internet transport 131 protocol, the authors found that a high percentage of paths tested 132 support UDP transport. A number of liaisons have been exchanged on 133 this topic [ refs to ITU-T SG12, ETSI STQ, BBF liaisons ], discussing 134 the laboratory and field tests that support the UDP-based approach to 135 IP-layer Capacity measurement. 137 This memo also recognizes the many updates to the IP Performance 138 Metrics Framework [RFC2330] published over twenty years, and makes 139 use of [RFC7312] for Advanced Stream and Sampling Framework, and RFC 140 8468 [RFC8468]IPv4, IPv6, and IPv4-IPv6 Coexistence Updates. 142 NOTE: The text contains Author comments, in brackets [RG: , acm: ] 144 2. Scope and Goals 146 The scope of this memo is to define a metric and corresponding method 147 to unambiguously perform Active measurements of Maximum IP-Layer 148 Capacity, along with related metrics and methods. 150 The main goal is to harmonize the specified metric and method across 151 the industry, and this memo is the vehicle through which working 152 group (and eventually, IETF) consensus will be captured and 153 communicated to achieve broad agreement, and possibly result in 154 changes in the specifications of other Standards Development 155 Organizations (SDO) (through the SDO's normal contribution process, 156 or through liaison exchange). 158 A local goal is to aid efficient test procedures where possible, and 159 to recommend reporting with additional interpretation of the results. 160 Also, to foster the development of protocol support for this metric 161 and method of measurement (all active testing protocols currently 162 defined by the IPPM WG are UDP-based, meeting a key requirement of 163 these methods). 165 3. Motivation 167 As with any problem that has been worked for many years in various 168 SDOs without any special attempts at coordination, various solutions 169 for metrics and methods have emerged. 171 There are five factors that have changed (or begun to change) in the 172 2013-2019 time frame, and the presence of any one of them on the path 173 requires features in the measurement design to account for the 174 changes: 176 1. Internet access is no longer the bottleneck for many users. 178 2. Both speed and latency are important to user's satisfaction. 180 3. UDP's growing role in Transport, in areas where TCP once 181 dominated. 183 4. Content and applications moving physically closer to users. 185 5. Less emphasis on ISP gateway measurements, possibly due to less 186 traffic crossing ISP gateways in future. 188 4. General Parameters and Definitions 190 This section lists the REQUIRED input factors to specify a Sender or 191 Receiver metric. 193 o Src, the address of a host (such as the globally routable IP 194 address). 196 o Dst, the address of a host (such as the globally routable IP 197 address). 199 o i, the limit on the number of Hops a specific packet may visit as 200 it traverses from the host at Src to the host at Dst (such as the 201 TTL or Hop Limit). 203 o MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops). 205 o T0, the time at the start of measurement interval, when packets 206 are first transmitted from the Source. 208 o I, the duration of a measurement interval 210 o dt, the duration of N equal sub-intervals in I 212 o Ts, the host time of a transmitted test packet as measured at 213 MP(Src), meaning Measurement Point at the Source. 215 o Ta, the host time of a test packet's *arrival* as measured at 216 MP(Dst), assigned to packets that arrive within a "reasonable" 217 time (see parameter below). 219 o Tmax, a maximum waiting time for test packets to arrive at the 220 destination, set sufficiently long to disambiguate packets with 221 long delays from packets that are discarded (lost), such that the 222 distribution of one-way delay is not truncated. 224 o F, the number of different flows synthesized by the method. 226 o flow, the stream of packets with the same n-tuple of designated 227 header fields that (when held constant) result in identical 228 treatment in a multi-path decision (such as the decision taken in 229 load balancing). Note: The IPv6 flow label MAY be included in the 230 flow definition when routers have complied with [RFC6438] 231 guidelines at the Tunnel End Points (TEP), and the source of the 232 measurement is a TEP. 234 o Type-P, the complete description of the packets for which this 235 assessment applies (including the flow-defining fields). Note 236 that the UDP transport layer is one requirement specified below. 237 Type-P is a parallel concept to "population of interest" defined 238 in ITU-T Rec. Y.1540. 240 o PM, a list of fundamental metrics, such as loss, delay, and 241 reordering, and corresponding Target performance threshold. At 242 least one fundamental metric and Target performance threshold MUST 243 be supplied (such as One-way IP Packet Loss [RFC7680] equal to 244 zero). 246 A non-Parameter which is required for several metrics is defined 247 below: 249 o T, the host time of the *first* test packet's *arrival* as 250 measured at MP(Dst). There may be other packets sent between 251 source and destination hosts that are excluded, so this is the 252 time of arrival of the first packet used for measurement of the 253 metric. 255 5. IP-Layer Capacity Singleton Metric Definitions 257 This section sets requirements for the following components to 258 support the Maximum IP-layer Capacity Metric. 260 5.1. Formal Name 262 Type-P-IP-Capacity, or informally called IP-layer Capacity. 264 Note that Type-P depends on the chosen method. 266 5.2. Parameters 268 This section lists the REQUIRED input factors to specify the metric, 269 beyond those listed in Section 4. 271 No additional Parameters are needed. 273 5.3. Metric Definitions 275 This section defines the REQUIRED aspects of the measureable IP-layer 276 Capacity metric (unless otherwise indicated) for measurements between 277 specified Source and Destination hosts: 279 Define the IP-layer capacity, C(T,I,PM), to be the number of IP-layer 280 bits (including header and data fields) in packets that can be 281 transmitted from the Src host and correctly received by the Dst host 282 during one contiguous sub-interval, dt. 284 The number of these IP-layer bits is designated n0[dtn-1,dtn] for a 285 specific dtn. 287 When the packet size is known and of fixed size, the packet count 288 during a single sub-interval dt multiplied by the total bits in IP 289 header and data fields is equal to n0[dtn-1,dtn]. 291 Anticipating a Sample of Singletons, the interval dt SHOULD be set to 292 a natural number m so that T+I = T + m*dt with dtn - dtn-1 = dt and 293 with 0 < n <= m. 295 Parameter PM represents other performance metrics [see section 296 Related Round-Trip Delay and Loss Definitions below]; their 297 measurement results SHALL be collected during measurement of IP-layer 298 Capacity and associated with the corresponding dtn for further 299 evaluation and reporting. 301 Mathematically, this definition can be represented as: 303 ( n0[dtn-1,dtn] ) 304 C(T,I,PM) = ------------------------- 305 dt 307 Equation for IP-Layer Capacity 309 and: 311 o n0 is the total number of IP-layer header and payload bits that 312 can be transmitted in Standard Formed packets from the Src host 313 and correctly received by the Dst host during one contiguous sub- 314 interval, dt in length, during the interval [T, T+I], 316 o C(T,I,PM) the IP-Layer Capacity, corresponds to the value of n0 317 measured in any sub-interval ending at dtn (meaning T + n*dt), 318 divided by the length of sub-interval, dt. 320 o all sub-intervals SHOULD be of equal duration. Choosing dt as 321 non-overlapping consecutive time intervals allows for a simple 322 implementation. 324 o The bit rate of the physical interface of the measurement device 325 must be higher than that of the link whose C(T,I,PM) is to be 326 measured. 328 Measurements according to these definitions SHALL use UDP transport 329 layer. 331 5.4. Related Round-Trip Delay and Loss Definitions 333 RTD[dtn-1,dtn] is defined as a sample of the [RFC2681] Round-trip 334 Delay between the Src host and the Dst host over the interval 335 [T,T+I]. The statistics used to to summarize RTD[dtn-1,dtn] MAY 336 include the minimum, maximum, and mean. 338 RTL[dtn-1,dtn] is defined as a sample of the [RFC6673] Round-trip 339 Loss between the Src host and the Dst host over the interval [T,T+I]. 340 The statistics used to to summarize RTL[dtn-1,dtn] MAY include the 341 lost packet count and the lost packet ratio. 343 5.5. Discussion 345 [To be added. Same as for the Maximum_C or move the entire text 346 here?] 348 5.6. Reporting the Metric 350 The IP-Layer Capacity SHALL be reported with meaningful resolution, 351 in units of Megabits per second. 353 The Related Round Trip Delay and/or Loss metric measurements for the 354 same Singleton SHALL be reported, also with meaningful resolution for 355 the values measured. 357 Individual Capacity measurements MAY be reported in a manner 358 consistent with the Maximum IP-Layer Capacity, see Section 9. 360 6. Maximum IP-Layer Capacity Metric Definitions (Statistic) 362 This section sets requirements for the following components to 363 support the Maximum IP-layer Capacity Metric. 365 6.1. Formal Name 367 Type-P-Max-IP-Capacity, or informally called Maximum IP-layer 368 Capacity. 370 Note that Type-P depends on the chosen method. 372 6.2. Parameters 374 This section lists the REQUIRED input factors to specify the metric, 375 beyond those listed in Section 4. 377 No additional Parameters or definitions are needed. 379 6.3. Metric Definitions 381 This section defines the REQUIRED aspects of the Maximum IP-layer 382 Capacity metric (unless otherwise indicated) for measurements between 383 specified Source and Destination hosts: 385 Define the Maximum IP-layer capacity, Maximum_C(T,I,PM), to be the 386 maximum number of IP-layer bits n0[dtn-1,dtn] that can be transmitted 387 in packets from the Src host and correctly received by the Dst host, 388 over all dt length intervals in [T, T+I], and meeting the PM 389 criteria. Equivalently the Maximum of a Sample of size m of 390 C(T,I,PM) collected during the interval [T, T+I] and meeting the PM 391 criteria. 393 The interval dt SHOULD be set to a natural number m so that T+I = T + 394 m*dt with dtn - dtn-1 = dt and with 0 < n <= m. 396 Parameter PM represents the other performance metrics [see section 397 Related Round-Trip Delay and Loss Definitions below] and their 398 measurement results for the maximum IP-layer capacity. At least one 399 target performance threshold (PM criterion) MUST be defined. If more 400 than one target performance threshold is defined, then the sub- 401 interval with maximum number of bits transmitted MUST meet all the 402 target performance thresholds. 404 Mathematically, this definition can be represented as: 406 max ( n0[dtn-1,dtn] ) 407 [T,T+I] 408 Maximum_C(T,I,PM) = ------------------------- 409 dt 410 where: 411 T T+I 412 _________________________________________ 413 | | | | | | | | | | | 414 dtn=0 1 2 3 4 5 6 7 8 9 m=10 416 Equation for Maximum Capacity 418 and: 420 o n0 is the total number of IP-layer header and payload bits that 421 can be transmitted in Standard Formed packets from the Src host 422 and correctly received by the Dst host during one contiguous sub- 423 interval, dt in length, during the interval [T, T+I], 425 o Maximum _C(T,I,PM) the Maximum IP-Layer Capacity, corresponds to 426 the maximum value of n0 measured in any sub-interval ending at dtn 427 (meaning T + n*dt), divided by the constant length of all sub- 428 intervals, dt. 430 o all sub-intervals SHOULD be of equal duration. Choosing dt as 431 non-overlapping consecutive time intervals allows for a simple 432 implementation. 434 o The bit rate of the physical interface of the measurement systems 435 must be higher than that of the link whose Maximum _C(T,I,PM) is 436 to be measured (the bottleneck link). 438 In this definition, the m sub-intervals can be viewed as trials when 439 the Src host varies the transmitted packet rate, searching for the 440 maximum n0 that meets the PM criteria measured at the Dst host in a 441 test of duration, I. When the transmitted packet rate is held 442 constant at the Src host, the m sub-intervals may also be viewed as 443 trials to evaluate the stability of n0 and metric(s) in the PM list 444 over all dt-length intervals in I. 446 Measurements according to these definitions SHALL use UDP transport 447 layer. 449 6.4. Related Round-Trip Delay and Loss Definitions 451 RTD[dtn-1,dtn] is defined as a sample of the [RFC2681] Round-trip 452 Delay between the Src host and the Dst host over the interval 453 [T,T+I], and corresponds to the dt interval containing 454 Maximum_C(T,I,PM). The statistics used to to summarize RTD[dtn- 455 1,dtn] MAY include the minimum, maximum, and mean. 457 RTL[dtn-1,dtn] is defined as a sample of the [RFC6673] Round-trip 458 Loss between the Src host and the Dst host over the interval [T,T+I] 459 and corresponds to the dt interval containing Maximum_C(T,I,PM). The 460 statistics used to to summarize RTL[dtn-1,dtn] MAY include the lost 461 packet count and the lost packet ratio. 463 6.5. Discussion 465 If traffic conditioning applies along a path for which Maximum 466 _C(T,I,PM) is to be determined, different values for dt SHOULD be 467 picked and measurements be executed during multiple intervals [T, 468 T+I]. Any single interval dt SHOULD be chosen so that is an integer 469 multiple of increasing values k times serialisation delay of a path 470 MTU at the physical interface speed where traffic conditioning is 471 expected. This should avoid taking configured burst tolerance 472 singletons as a valid Maximum _C(T,I,PM) result. 474 A Maximum_C(T,I,PM) without any indication of bottleneck congestion, 475 be that an increasing latency, packet loss or ECN marks during a 476 measurement interval I, is likely to underestimate Maximum_C(T,I,PM). 478 6.6. Reporting the Metric 480 The Maximum IP-Layer Capacity SHALL be reported with meaningful 481 resolution, in units of Megabits per second. 483 The Related Round Trip Delay and/or Loss metric measurements for the 484 same Singleton SHALL be reported, also with meaningful resolution for 485 the values measured. 487 When there are demonstrated and repeatable modes in the Sample, then 488 the Maximum IP-Layer Capacity SHALL be reported for each mode, along 489 with the relative time from the beginning of the stream that the mode 490 was observed to be present. Bimodal Maxima have been observed with 491 some services, sometimes called a "turbo" mode" intending to deliver 492 short transfers more quickly, or reduce the initial buffering time 493 for some video streams. 495 7. IP-Layer Sender Bit Rate Singleton Metric Definitions 497 This section sets requirements for the following components to 498 support the IP-layer Sender Bitrate Metric. 500 7.1. Formal Name 502 Type-P-IP-Sender-Bit-Rate, or informally called IP-layer Sender 503 Bitrate. 505 Note that Type-P depends on the chosen method. 507 7.2. Parameters 509 This section lists the REQUIRED input factors to specify the metric, 510 beyond those listed in Section 4. 512 o S, the duration of the measurement interval at the Source 514 o st, the nominal duration of N sub-intervals in S 516 S SHALL be longer than I, primarily to account for on-demand 517 activation of the path, or any preamble to testing required. 519 st SHOULD be much smaller than the sub-interval dt. The st parameter 520 does not have relevance when the Source is transmitting at a fixed 521 rate throughout S. 523 7.3. Metric Definition 525 This section defines the REQUIRED aspects of the IP-layer Sender 526 Bitrate metric (unless otherwise indicated) for measurements at the 527 specified Source on packets addressed for the intended Destination 528 host and matching the required Type-P: 530 Define the IP-layer Sender Bit Rate, B(S,st), to be the number of IP- 531 layer bits (including header and data fields) that are transmitted 532 from the Source during one contiguous sub-interval, st, during the 533 test interval S (where S SHALL be longer than I), and where the 534 fixed-size packet count during that single sub-interval st also 535 provides the number of IP-layer bits in any interval: n0[stn-1,stn]. 537 Measurements according to these definitions SHALL use UDP transport 538 layer. Any feedback from Dst host to Src host received by Src host 539 during an interval [stn-1,stn] MUST NOT result in an adaptation of 540 the Src host traffic conditioning during this interval. 542 7.4. Discussion 544 Both the Sender and Receiver or (source and destination) bit rates 545 SHOULD be assessed as part of a measurement. 547 7.5. Reporting the Metric 549 The IP-Layer Sender Bit Rate SHALL be reported with meaningful 550 resolution, in units of Megabits per second. 552 Individual IP-Layer Sender Bit Rate measurements are discussed 553 further in Section 9. 555 8. Method of Measurement 557 The duration of a test, I, MUST be constrained in a production 558 network, since this is an active test method and it will likely cause 559 congestion on the Src to Dst host path during a test. 561 Additional Test methods and configurations may be provided in this 562 section, after review and further testing. 564 8.1. Load Rate Adjustment Algorithm (from udpst) 566 A table is pre-built defining all the offered load rates that will be 567 supported (R1 - Rn, in ascending order). Each rate is defined as 568 datagrams of size S, sent as a burst of count C, every time interval 569 T. While it is advantageous to use datagrams of as large a size as 570 possible, it may be prudent to use a slightly smaller maximum that 571 allows for secondary protocol headers and/or tunneling without 572 resulting in IP-layer fragmentation. 574 At the beginning of a test, the sender begins sending at rate R1 and 575 the receiver starts a feedback timer at interval F (while awaiting 576 inbound datagrams). As datagrams are received they are checked for 577 sequence number anomalies (loss, out-of-order, duplication, etc.) and 578 the delay variation is measured (one-way or round-trip). This 579 information is accumulated until the feedback timer F expires and a 580 status feedback message is sent from the receiver back to the sender, 581 to communicate this information. The accumulated statistics are then 582 reset by the receiver for the next feedback interval. As feedback 583 messages are received back at the sender, they are evaluated to 584 determine how to adjust the current offered load rate (Rx). 586 If the feedback indicates that there were no sequence number 587 anomalies AND the delay variation was below the lower threshold, the 588 offered load rate is increased. If congestion has not been confirmed 589 up to this point, the offered load rate is increased by more than one 590 rate (e.g., Rx+10). This allows the offered load to quickly reach a 591 near-maximum rate. Conversely, if congestion has been previously 592 confirmed, the offered load rate is only increased by one (Rx+1). 594 If the feedback indicates that sequence number anomalies were 595 detected OR the delay variation was above the upper threshold, the 596 offered load rate is decreased. If congestion is confirmed by the 597 current feedback message being processed, the offered load rate is 598 decreased by more than one rate (e.g., Rx-30). This one-time 599 reduction is intended to compensate for the fast initial ramp-up. In 600 all other cases, the offered load rate is only decreased by one (Rx- 601 1). 603 If the feedback indicates that there were no sequence number 604 anomalies AND the delay variation was above the lower threshold, but 605 below the upper threshold, the offered load rate is not changed. 606 This allows time for recent changes in the offered load rate to 607 stabilize, and the feedback to represent current conditions more 608 accurately. 610 Lastly, the method for confirming congestion is that there were 611 sequence number anomalies OR the delay variation was above the upper 612 threshold for two consecutive feedback intervals. 614 8.2. Measurement Qualification or Verification 616 When assessing a Maximum rate as the metric specifies, artificially 617 high (optimistic) values might be measured until some buffer on the 618 path is filled. The artificial values might result in the Maximum 619 Capacity observed when the method of measurement is searching for the 620 Maximum, and that would not do. This situation is different from the 621 bi-modal service rates (discussed under Reporting), which are 622 characterized by a multi-second duration (much longer than the 623 measured RTT) and repeatable behavior. 625 There are many ways that the Method of Measurement could handle this 626 false-max issue, and the simplest seems to come from Section 24 of 627 RFC 2544[RFC2544] and its discussion of Trial duration, where 628 relatively short trials conducted as part of the search are followed 629 by longer trials to make the final determination. 631 In the production network, measurements of singletons and samples 632 (the terms for trials and tests of Lab Benchmarking) must be limited 633 in duration because they may be service-affecting. But there is 634 sufficient value in repeating a sample with a fixed sending rate 635 determined by the previous search for the Max IP-layer Capacity, to 636 qualify the result in terms of the other performance metrics measured 637 at the same time. 639 A qualification measurement for the search result is a subsequent 640 measurement, sending at a fixed 99.x % of the Max IP-layer Capacity 641 for I, or an indefinite period. The same Max Capacity Metric is 642 applied, and the Qualification for the result is a sample without 643 packet loss or a growing minimum delay trend in subsequent singletons 644 (or each dt of the measurement interval, I). Samples exhibiting 645 losses or increasing queue occupation require a repeated search and/ 646 or test at reduced fixed sender rate for qualification. 648 Here, as with any Active Capacity test, the test duration must be 649 kept short. 10 second tests for each direction of transmission are 650 common today. In combination with a fast search method and user- 651 network coordination, the concerns raised in RFC 6815[RFC6815] are 652 alleviated. The method for assessing Max IP Capacity is different 653 from classic [RFC2544] methods: they use short term load adjustment 654 and are sensitive to loss and delay, like other congestion control 655 algorithms used on the Internet every day. 657 8.3. Measurement Considerations 659 In general, the wide-spread measurements that this memo encourages 660 will encounter wide-spread behaviors. The bimodal IP Capacity 661 behavior is a good example. 663 The path measured may be state-full based on many factors, and the 664 Parameter "Time of day" when a test starts may not be enough enough 665 information. Repeatable testing may require the time from the 666 beginning of a measured flow, and how the flow is constructed 667 including how much traffic has already been sent on that flow when a 668 state-change is observed, because the state-change may be based on 669 time or bytes sent or both. 671 Many different traffic shapers and on-demand access technology may be 672 encountered, as anticipated in [RFC7312], and play a key role in 673 measurement results. Methods MUST be prepared to provide a short 674 preamble transmission to activate on-demand access, and to discard 675 the preamble from subsequent test results. 677 In general, results depend on the sending stream characteristics; the 678 measurement community has known this for a long time, and to keep it 679 front of mind. 681 As testing continues, implementers should expect some evolution in 682 the methods. 684 9. Reporting 686 The Maximum IP-Layer Capacity results SHOULD be reported in the 687 format of a table with a row for each of the test Phases and Number 688 of Flows. There SHOULD be columns for the phases with number of 689 flows, and for the resultant Maximum IP-Layer Capacity results for 690 the aggregate and each flow tested. 692 The PM list metrics corresponding to the sub-interval where the 693 Maximum Capacity occurred MUST accompany a report of Maximum IP-Layer 694 Capacity results, for each test phase. 696 +--------------+----------------------+-----------+-----------------+ 697 | Phase, # | Max IP-Layer | Loss | RTT min, max, | 698 | Flows | Capacity, Mbps | Ratio | msec | 699 +--------------+----------------------+-----------+-----------------+ 700 | Search,1 | 967.31 | 0.0002 | 30, 58 | 701 | Verify,1 | 966.00 | 0.0000 | 30, 38 | 702 +--------------+----------------------+-----------+-----------------+ 704 Maximum IP-layer Capacity Results 706 Static and configuration parameters: 708 The sub-interval time, dt, MUST accompany a report of Maximum IP- 709 Layer Capacity results, and the remaining Parameters from Section 4, 710 General Parameters. 712 The IP-Layer Sender Bit rate results SHOULD be reported in the format 713 of a table with a row for each of the test Phases, sub-intervals (st) 714 and Number of Flows. There SHOULD be columns for the phases with 715 number of flows, and for the resultant IP-Layer Sender Bit rate 716 results for the aggregate and each flow tested. 718 +------------------------+-------------+-----------------------+----+ 719 | Phase, Flow or | st, sec | Sender Bit Rate, Mbps | ?? | 720 | Aggregate | | | | 721 +------------------------+-------------+-----------------------+----+ 722 | Search,1 | 0.00 - 1.00 | 345 | __ | 723 | Search,2 | 0.00 - 1.00 | 289 | __ | 724 | Search,Agg | 0.00 - 1.00 | 634 | __ | 725 +------------------------+-------------+-----------------------+----+ 727 IP-layer Sender Bit Rate Results 729 Static and configuration parameters: 731 The subinterval time, st, MUST accompany a report of Sender IP-Layer 732 Bit Rate results. 734 Also, the remaining Parameters from Section 4, General Parameters. 736 10. Security Considerations 738 Active metrics and measurements have a long history of security 739 considerations [add references to LMAP Framework, etc.]. 741 743 11. IANA Considerations 745 This memo makes no requests of IANA. 747 12. Acknowledgements 749 Thanks to Joachim Fabini, Matt Mathis, and Ignacio Alvarez-Hamelin 750 for their extensive comments on the memo and related topics. 752 13. References 754 13.1. Normative References 756 [RFC1242] Bradner, S., "Benchmarking Terminology for Network 757 Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, 758 July 1991, . 760 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 761 Requirement Levels", BCP 14, RFC 2119, 762 DOI 10.17487/RFC2119, March 1997, 763 . 765 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 766 "Framework for IP Performance Metrics", RFC 2330, 767 DOI 10.17487/RFC2330, May 1998, 768 . 770 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 771 Network Interconnect Devices", RFC 2544, 772 DOI 10.17487/RFC2544, March 1999, 773 . 775 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 776 Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, 777 September 1999, . 779 [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology 780 for LAN Switching Devices", RFC 2889, 781 DOI 10.17487/RFC2889, August 2000, 782 . 784 [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining 785 Empirical Bulk Transfer Capacity Metrics", RFC 3148, 786 DOI 10.17487/RFC3148, July 2001, 787 . 789 [RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity", 790 RFC 5136, DOI 10.17487/RFC5136, February 2008, 791 . 793 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. 794 Dugatkin, "IPv6 Benchmarking Methodology for Network 795 Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May 796 2008, . 798 [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera, 799 "Device Reset Characterization", RFC 6201, 800 DOI 10.17487/RFC6201, March 2011, 801 . 803 [RFC6412] Poretsky, S., Imhoff, B., and K. Michielsen, "Terminology 804 for Benchmarking Link-State IGP Data-Plane Route 805 Convergence", RFC 6412, DOI 10.17487/RFC6412, November 806 2011, . 808 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 809 for Equal Cost Multipath Routing and Link Aggregation in 810 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 811 . 813 [RFC6673] Morton, A., "Round-Trip Packet Loss Metrics", RFC 6673, 814 DOI 10.17487/RFC6673, August 2012, 815 . 817 [RFC6815] Bradner, S., Dubray, K., McQuaid, J., and A. Morton, 818 "Applicability Statement for RFC 2544: Use on Production 819 Networks Considered Harmful", RFC 6815, 820 DOI 10.17487/RFC6815, November 2012, 821 . 823 [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet 824 Sizes for Additional Testing", RFC 6985, 825 DOI 10.17487/RFC6985, July 2013, 826 . 828 [RFC7312] Fabini, J. and A. Morton, "Advanced Stream and Sampling 829 Framework for IP Performance Metrics (IPPM)", RFC 7312, 830 DOI 10.17487/RFC7312, August 2014, 831 . 833 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 834 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 835 May 2016, . 837 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 838 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 839 May 2017, . 841 [RFC8337] Mathis, M. and A. Morton, "Model-Based Metrics for Bulk 842 Transport Capacity", RFC 8337, DOI 10.17487/RFC8337, March 843 2018, . 845 [RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. 846 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for 847 the IP Performance Metrics (IPPM) Framework", RFC 8468, 848 DOI 10.17487/RFC8468, November 2018, 849 . 851 13.2. Informative References 853 [copycat] Edleine, K., Kuhlewind, K., Trammell, B., and B. Donnet, 854 "copycat: Testing Differential Treatment of New Transport 855 Protocols in the Wild (ANRW '17)", July 2017, 856 . 858 [RFC8239] Avramov, L. and J. Rapp, "Data Center Benchmarking 859 Methodology", RFC 8239, DOI 10.17487/RFC8239, August 2017, 860 . 862 [TST009] Morton, R. A., "ETSI GS NFV-TST 009 V3.1.1 (2018-10), 863 "Network Functions Virtualisation (NFV) Release 3; 864 Testing; Specification of Networking Benchmarks and 865 Measurement Methods for NFVI"", October 2018, 866 . 869 [VSPERF-b2b] 870 Morton, A., "Back2Back Testing Time Series (from CI)", 871 June 2017, . 875 [VSPERF-BSLV] 876 Morton, A. and S. Rao, "Evolution of Repeatability in 877 Benchmarking: Fraser Plugfest (Summary for IETF BMWG)", 878 July 2018, 879 . 883 Authors' Addresses 885 Al Morton 886 AT&T Labs 887 200 Laurel Avenue South 888 Middletown,, NJ 07748 889 USA 891 Phone: +1 732 420 1571 892 Fax: +1 732 368 1192 893 Email: acm@research.att.com 894 Ruediger Geib 895 Deutsche Telekom 896 Heinrich Hertz Str. 3-7 897 Darmstadt 64295 898 Germany 900 Phone: +49 6151 5812747 901 Email: Ruediger.Geib@telekom.de 903 Len Ciavattone 904 AT&T Labs 905 200 Laurel Avenue South 906 Middletown,, NJ 07748 907 USA 909 Email: lencia@att.com