idnits 2.17.1 draft-morton-ippm-capacity-metric-method-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (November 4, 2019) is 1627 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC7680' is mentioned on line 238, but not defined == Unused Reference: 'RFC1242' is defined on line 775, but no explicit reference was found in the text == Unused Reference: 'RFC2889' is defined on line 798, but no explicit reference was found in the text == Unused Reference: 'RFC5180' is defined on line 812, but no explicit reference was found in the text == Unused Reference: 'RFC6201' is defined on line 817, but no explicit reference was found in the text == Unused Reference: 'RFC6412' is defined on line 822, but no explicit reference was found in the text == Unused Reference: 'RFC6985' is defined on line 842, but no explicit reference was found in the text == Unused Reference: 'RFC8468' is defined on line 864, but no explicit reference was found in the text == Unused Reference: 'RFC8239' is defined on line 877, but no explicit reference was found in the text == Unused Reference: 'TST009' is defined on line 881, but no explicit reference was found in the text == Unused Reference: 'VSPERF-b2b' is defined on line 888, but no explicit reference was found in the text == Unused Reference: 'VSPERF-BSLV' is defined on line 894, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 1242 ** Downref: Normative reference to an Informational RFC: RFC 2330 ** Downref: Normative reference to an Informational RFC: RFC 2544 ** Downref: Normative reference to an Informational RFC: RFC 2889 ** Downref: Normative reference to an Informational RFC: RFC 3148 ** Downref: Normative reference to an Informational RFC: RFC 5136 ** Downref: Normative reference to an Informational RFC: RFC 5180 ** Downref: Normative reference to an Informational RFC: RFC 6201 ** Downref: Normative reference to an Informational RFC: RFC 6412 ** Downref: Normative reference to an Informational RFC: RFC 6815 ** Downref: Normative reference to an Informational RFC: RFC 6985 ** Downref: Normative reference to an Informational RFC: RFC 7312 ** Downref: Normative reference to an Informational RFC: RFC 7799 ** Downref: Normative reference to an Experimental RFC: RFC 8337 ** Downref: Normative reference to an Informational RFC: RFC 8468 Summary: 15 errors (**), 0 flaws (~~), 14 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Morton 3 Internet-Draft AT&T Labs 4 Updates: ???? (if approved) R. Geib 5 Intended status: Standards Track Deutsche Telekom 6 Expires: May 7, 2020 L. Ciavattone 7 AT&T Labs 8 November 4, 2019 10 Metrics and Methods for IP Capacity 11 draft-morton-ippm-capacity-metric-method-01 13 Abstract 15 This memo revisits the problem of Network Capacity metrics first 16 examined in RFC 5136. The memo specifies a more practical Maximum 17 IP-layer Capacity metric definition catering for measurement 18 purposes, and outlines the corresponding methods of measurement. 20 Requirements Language 22 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 23 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 24 "OPTIONAL" in this document are to be interpreted as described in BCP 25 14[RFC2119] [RFC8174] when, and only when, they appear in all 26 capitals, as shown here. 28 Status of This Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at https://datatracker.ietf.org/drafts/current/. 38 Internet-Drafts are draft documents valid for a maximum of six months 39 and may be updated, replaced, or obsoleted by other documents at any 40 time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress." 43 This Internet-Draft will expire on May 7, 2020. 45 Copyright Notice 47 Copyright (c) 2019 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (https://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with respect 55 to this document. Code Components extracted from this document must 56 include Simplified BSD License text as described in Section 4.e of 57 the Trust Legal Provisions and are provided without warranty as 58 described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 63 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . 4 64 3. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 4 65 4. General Parameters and Definitions . . . . . . . . . . . . . 5 66 5. IP-Layer Capacity Singleton Metric Definitions . . . . . . . 6 67 5.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 6 68 5.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 6 69 5.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 6 70 5.4. Related Round-Trip Delay and Loss Definitions . . . . . . 8 71 5.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 8 72 5.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 8 73 6. Maximum IP-Layer Capacity Metric Definitions (Statistic) . . 8 74 6.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 8 75 6.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 8 76 6.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 9 77 6.4. Related Round-Trip Delay and Loss Definitions . . . . . . 10 78 6.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 10 79 6.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 11 80 7. IP-Layer Sender Bit Rate Singleton Metric Definitions . . . . 11 81 7.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 11 82 7.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 11 83 7.3. Metric Definition . . . . . . . . . . . . . . . . . . . . 12 84 7.4. Discussion . . . . . . . . . . . . . . . . . . . . . . . 12 85 7.5. Reporting the Metric . . . . . . . . . . . . . . . . . . 12 86 8. Method of Measurement . . . . . . . . . . . . . . . . . . . . 12 87 8.1. Load Rate Adjustment Algorithm (from udpst) . . . . . . . 13 88 8.2. Measurement Qualification or Verification . . . . . . . . 14 89 8.3. Measurement Considerations . . . . . . . . . . . . . . . 15 90 9. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . 16 91 10. Security Considerations . . . . . . . . . . . . . . . . . . . 17 92 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 93 12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 94 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 17 95 13.1. Normative References . . . . . . . . . . . . . . . . . . 17 96 13.2. Informative References . . . . . . . . . . . . . . . . . 19 97 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 99 1. Introduction 101 The IETF's efforts to define Network and Bulk Transport Capacity have 102 been chartered and progressed for over twenty years. Over that time, 103 the performance community has seen development of Informative 104 definitions in [RFC3148] for Framework for Bulk Transport Capacity 105 (BTC), RFC 5136 for Network Capacity and Maximum IP-layer Capacity, 106 and the Experimental metric definitions and methods in [RFC8337], 107 Model-Based Metrics for BTC. 109 This memo revisits the problem of Network Capacity metrics examined 110 first in [RFC3148] and later in [RFC5136]. Maximum IP-Layer Capacity 111 and [RFC3148] Bulk Transfer Capacity (goodput) are different metrics. 112 Max IP-layer Capacity is like the theoretical goal for goodput. 113 There are many metrics in [RFC5136], such as Available Capacity. 114 Measurements depend on the network path under test and the use case. 115 Here, the main use case is to assess the maximum capacity of the 116 access network, with specific performance criteria used in the 117 measurement. 119 This memo recognizes the importance of a definition of a Maximum IP- 120 layer Capacity Metric at a time when access speeds have increased 121 dramatically; a definition that is both practical and effective for 122 the performance community's needs, including Internet users. The 123 metric definition is intended to use Active Methods of Measurement 124 [RFC7799], and a method of measurement is included. 126 The most direct active measurement of IP-layer Capacity would use IP 127 packets, but in practice a transport header is needed to traverse 128 address and port translators. UDP offers the most direct assessment 129 possibility, and in the [copycat][copycat] measurement study to 130 investigate whether UDP is viable as a general Internet transport 131 protocol, the authors found that a high percentage of paths tested 132 support UDP transport. A number of liaisons have been exchanged on 133 this topic [ refs to ITU-T SG12, ETSI STQ, BBF liaisons ], discussing 134 the laboratory and field tests that support the UDP-based approach to 135 IP-layer Capacity measurement. 137 This memo also recognizes the many updates to the IP Performance 138 Metrics Framework [RFC2330] published over twenty years, and makes 139 use of [RFC7312] for Advanced Stream and Sampling Framework, and RFC 140 8468 [RFC8468]IPv4, IPv6, and IPv4-IPv6 Coexistence Updates. 142 NOTE: The text contains a few Author comments, in brackets [RG: , 143 acm: ] 145 2. Scope and Goals 147 The scope of this memo is to define a metric and corresponding method 148 to unambiguously perform Active measurements of Maximum IP-Layer 149 Capacity, along with related metrics and methods. 151 The main goal is to harmonize the specified metric and method across 152 the industry, and this memo is the vehicle through which working 153 group (and eventually, IETF) consensus will be captured and 154 communicated to achieve broad agreement, and possibly result in 155 changes in the specifications of other Standards Development 156 Organizations (SDO) (through the SDO's normal contribution process, 157 or through liaison exchange). 159 A local goal is to aid efficient test procedures where possible, and 160 to recommend reporting with additional interpretation of the results. 161 Also, to foster the development of protocol support for this metric 162 and method of measurement (all active testing protocols currently 163 defined by the IPPM WG are UDP-based, meeting a key requirement of 164 these methods). 166 3. Motivation 168 As with any problem that has been worked for many years in various 169 SDOs without any special attempts at coordination, various solutions 170 for metrics and methods have emerged. 172 There are five factors that have changed (or begun to change) in the 173 2013-2019 time frame, and the presence of any one of them on the path 174 requires features in the measurement design to account for the 175 changes: 177 1. Internet access is no longer the bottleneck for many users. 179 2. Both speed and latency are important to user's satisfaction. 181 3. UDP's growing role in Transport, in areas where TCP once 182 dominated. 184 4. Content and applications moving physically closer to users. 186 5. Less emphasis on ISP gateway measurements, possibly due to less 187 traffic crossing ISP gateways in future. 189 4. General Parameters and Definitions 191 This section lists the REQUIRED input factors to specify a Sender or 192 Receiver metric. 194 o Src, the address of a host (such as the globally routable IP 195 address). 197 o Dst, the address of a host (such as the globally routable IP 198 address). 200 o i, the limit on the number of Hops a specific packet may visit as 201 it traverses from the host at Src to the host at Dst (such as the 202 TTL or Hop Limit). 204 o MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops). 206 o T0, the time at the start of measurement interval, when packets 207 are first transmitted from the Source. 209 o I, the duration of a measurement interval (default 10 sec) 211 o dt, the duration of N equal sub-intervals in I (default 1 sec) 213 o Tmax, a maximum waiting time for test packets to arrive at the 214 destination, set sufficiently long to disambiguate packets with 215 long delays from packets that are discarded (lost), such that the 216 distribution of one-way delay is not truncated. 218 o F, the number of different flows synthesized by the method 219 (default 1 flow) 221 o flow, the stream of packets with the same n-tuple of designated 222 header fields that (when held constant) result in identical 223 treatment in a multi-path decision (such as the decision taken in 224 load balancing). Note: The IPv6 flow label MAY be included in the 225 flow definition when routers have complied with [RFC6438] 226 guidelines at the Tunnel End Points (TEP), and the source of the 227 measurement is a TEP. 229 o Type-P, the complete description of the packets for which this 230 assessment applies (including the flow-defining fields). Note 231 that the UDP transport layer is one requirement specified below. 232 Type-P is a parallel concept to "population of interest" defined 233 in ITU-T Rec. Y.1540. 235 o PM, a list of fundamental metrics, such as loss, delay, and 236 reordering, and corresponding Target performance threshold. At 237 least one fundamental metric and Target performance threshold MUST 238 be supplied (such as One-way IP Packet Loss [RFC7680] equal to 239 zero). 241 A non-Parameter which is required for several metrics is defined 242 below: 244 o T, the host time of the *first* test packet's *arrival* as 245 measured at MP(Dst). There may be other packets sent between 246 source and destination hosts that are excluded, so this is the 247 time of arrival of the first packet used for measurement of the 248 metric. 250 Note that time stamps, sequnce numbers, etc. will be established by 251 the test protocol. 253 5. IP-Layer Capacity Singleton Metric Definitions 255 This section sets requirements for the following components to 256 support the Maximum IP-layer Capacity Metric. 258 5.1. Formal Name 260 Type-P-IP-Capacity, or informally called IP-layer Capacity. 262 Note that Type-P depends on the chosen method. 264 5.2. Parameters 266 This section lists the REQUIRED input factors to specify the metric, 267 beyond those listed in Section 4. 269 No additional Parameters are needed. 271 5.3. Metric Definitions 273 This section defines the REQUIRED aspects of the measureable IP-layer 274 Capacity metric (unless otherwise indicated) for measurements between 275 specified Source and Destination hosts: 277 Define the IP-layer capacity, C(T,I,PM), to be the number of IP-layer 278 bits (including header and data fields) in packets that can be 279 transmitted from the Src host and correctly received by the Dst host 280 during one contiguous sub-interval, dt. 282 The number of these IP-layer bits is designated n0[dtn-1,dtn] for a 283 specific dtn. 285 When the packet size is known and of fixed size, the packet count 286 during a single sub-interval dt multiplied by the total bits in IP 287 header and data fields is equal to n0[dtn-1,dtn]. 289 Anticipating a Sample of Singletons, the interval dt SHOULD be set to 290 a natural number m so that T+I = T + m*dt with dtn - dtn-1 = dt and 291 with 0 < n <= m. 293 Parameter PM represents other performance metrics [see section 294 Related Round-Trip Delay and Loss Definitions below]; their 295 measurement results SHALL be collected during measurement of IP-layer 296 Capacity and associated with the corresponding dtn for further 297 evaluation and reporting. 299 Mathematically, this definition can be represented as: 301 ( n0[dtn-1,dtn] ) 302 C(T,I,PM) = ------------------------- 303 dt 305 Equation for IP-Layer Capacity 307 and: 309 o n0 is the total number of IP-layer header and payload bits that 310 can be transmitted in Standard Formed packets from the Src host 311 and correctly received by the Dst host during one contiguous sub- 312 interval, dt in length, during the interval [T, T+I], 314 o C(T,I,PM) the IP-Layer Capacity, corresponds to the value of n0 315 measured in any sub-interval ending at dtn (meaning T + n*dt), 316 divided by the length of sub-interval, dt. 318 o all sub-intervals SHOULD be of equal duration. Choosing dt as 319 non-overlapping consecutive time intervals allows for a simple 320 implementation. 322 o The bit rate of the physical interface of the measurement device 323 must be higher than that of the link whose C(T,I,PM) is to be 324 measured. 326 Measurements according to these definitions SHALL use UDP transport 327 layer. 329 5.4. Related Round-Trip Delay and Loss Definitions 331 RTD[dtn-1,dtn] is defined as a sample of the [RFC2681] Round-trip 332 Delay between the Src host and the Dst host over the interval 333 [T,T+I]. The statistics used to to summarize RTD[dtn-1,dtn] MAY 334 include the minimum, maximum, and mean. 336 RTL[dtn-1,dtn] is defined as a sample of the [RFC6673] Round-trip 337 Loss between the Src host and the Dst host over the interval [T,T+I]. 338 The statistics used to to summarize RTL[dtn-1,dtn] MAY include the 339 lost packet count and the lost packet ratio. 341 5.5. Discussion 343 See the corresponding section for Maximum IP-Layer Capacity. 345 5.6. Reporting the Metric 347 The IP-Layer Capacity SHALL be reported with meaningful resolution, 348 in units of Megabits per second (Mbps). 350 The Related Round Trip Delay and/or Loss metric measurements for the 351 same Singleton SHALL be reported, also with meaningful resolution for 352 the values measured. 354 Individual Capacity measurements MAY be reported in a manner 355 consistent with the Maximum IP-Layer Capacity, see Section 9. 357 6. Maximum IP-Layer Capacity Metric Definitions (Statistic) 359 This section sets requirements for the following components to 360 support the Maximum IP-layer Capacity Metric. 362 6.1. Formal Name 364 Type-P-Max-IP-Capacity, or informally called Maximum IP-layer 365 Capacity. 367 Note that Type-P depends on the chosen method. 369 6.2. Parameters 371 This section lists the REQUIRED input factors to specify the metric, 372 beyond those listed in Section 4. 374 No additional Parameters or definitions are needed. 376 6.3. Metric Definitions 378 This section defines the REQUIRED aspects of the Maximum IP-layer 379 Capacity metric (unless otherwise indicated) for measurements between 380 specified Source and Destination hosts: 382 Define the Maximum IP-layer capacity, Maximum_C(T,I,PM), to be the 383 maximum number of IP-layer bits n0[dtn-1,dtn] that can be transmitted 384 in packets from the Src host and correctly received by the Dst host, 385 over all dt length intervals in [T, T+I], and meeting the PM 386 criteria. Equivalently the Maximum of a Sample of size m of 387 C(T,I,PM) collected during the interval [T, T+I] and meeting the PM 388 criteria. 390 The interval dt SHOULD be set to a natural number m so that T+I = T + 391 m*dt with dtn - dtn-1 = dt and with 0 < n <= m. 393 Parameter PM represents the other performance metrics [see section 394 Related Round-Trip Delay and Loss Definitions below] and their 395 measurement results for the maximum IP-layer capacity. At least one 396 target performance threshold (PM criterion) MUST be defined. If more 397 than one target performance threshold is defined, then the sub- 398 interval with maximum number of bits transmitted MUST meet all the 399 target performance thresholds. 401 Mathematically, this definition can be represented as: 403 max ( n0[dtn-1,dtn] ) 404 [T,T+I] 405 Maximum_C(T,I,PM) = ------------------------- 406 dt 407 where: 408 T T+I 409 _________________________________________ 410 | | | | | | | | | | | 411 dtn=0 1 2 3 4 5 6 7 8 9 m=10 413 Equation for Maximum Capacity 415 and: 417 o n0 is the total number of IP-layer header and payload bits that 418 can be transmitted in Standard Formed packets from the Src host 419 and correctly received by the Dst host during one contiguous sub- 420 interval, dt in length, during the interval [T, T+I], 422 o Maximum _C(T,I,PM) the Maximum IP-Layer Capacity, corresponds to 423 the maximum value of n0 measured in any sub-interval ending at dtn 424 (meaning T + n*dt), divided by the constant length of all sub- 425 intervals, dt. 427 o all sub-intervals SHOULD be of equal duration. Choosing dt as 428 non-overlapping consecutive time intervals allows for a simple 429 implementation. 431 o The bit rate of the physical interface of the measurement systems 432 must be higher than that of the link whose Maximum _C(T,I,PM) is 433 to be measured (the bottleneck link). 435 In this definition, the m sub-intervals can be viewed as trials when 436 the Src host varies the transmitted packet rate, searching for the 437 maximum n0 that meets the PM criteria measured at the Dst host in a 438 test of duration, I. When the transmitted packet rate is held 439 constant at the Src host, the m sub-intervals may also be viewed as 440 trials to evaluate the stability of n0 and metric(s) in the PM list 441 over all dt-length intervals in I. 443 Measurements according to these definitions SHALL use UDP transport 444 layer. 446 6.4. Related Round-Trip Delay and Loss Definitions 448 RTD[dtn-1,dtn] is defined as a sample of the [RFC2681] Round-trip 449 Delay between the Src host and the Dst host over the interval 450 [T,T+I], and corresponds to the dt interval containing 451 Maximum_C(T,I,PM). The statistics used to to summarize RTD[dtn- 452 1,dtn] MAY include the minimum, maximum, and mean. 454 RTL[dtn-1,dtn] is defined as a sample of the [RFC6673] Round-trip 455 Loss between the Src host and the Dst host over the interval [T,T+I] 456 and corresponds to the dt interval containing Maximum_C(T,I,PM). The 457 statistics used to to summarize RTL[dtn-1,dtn] MAY include the lost 458 packet count and the lost packet ratio. 460 6.5. Discussion 462 If traffic conditioning applies along a path for which Maximum 463 _C(T,I,PM) is to be determined, different values for dt SHOULD be 464 picked and measurements be executed during multiple intervals [T, 465 T+I]. Any single interval dt SHOULD be chosen so that is an integer 466 multiple of increasing values k times serialisation delay of a path 467 MTU at the physical interface speed where traffic conditioning is 468 expected. This should avoid taking configured burst tolerance 469 singletons as a valid Maximum _C(T,I,PM) result. 471 A Maximum_C(T,I,PM) without any indication of bottleneck congestion, 472 be that an increasing latency, packet loss or ECN marks during a 473 measurement interval I, is likely to underestimate Maximum_C(T,I,PM). 475 6.6. Reporting the Metric 477 The Maximum IP-Layer Capacity SHALL be reported with meaningful 478 resolution, in units of Megabits per second. 480 The Related Round Trip Delay and/or Loss metric measurements for the 481 same Singleton SHALL be reported, also with meaningful resolution for 482 the values measured. 484 When there are demonstrated and repeatable modes in the Sample, then 485 the Maximum IP-Layer Capacity SHALL be reported for each mode, along 486 with the relative time from the beginning of the stream that the mode 487 was observed to be present. Bimodal Maxima have been observed with 488 some services, sometimes called a "turbo" mode" intending to deliver 489 short transfers more quickly, or reduce the initial buffering time 490 for some video streams. 492 7. IP-Layer Sender Bit Rate Singleton Metric Definitions 494 This section sets requirements for the following components to 495 support the IP-layer Sender Bitrate Metric. 497 7.1. Formal Name 499 Type-P-IP-Sender-Bit-Rate, or informally called IP-layer Sender 500 Bitrate. 502 Note that Type-P depends on the chosen method. 504 7.2. Parameters 506 This section lists the REQUIRED input factors to specify the metric, 507 beyond those listed in Section 4. 509 o S, the duration of the measurement interval at the Source 511 o st, the nominal duration of N sub-intervals in S (default = 0.05 512 seconds) 514 S SHALL be longer than I, primarily to account for on-demand 515 activation of the path, or any preamble to testing required. 517 st SHOULD be much smaller than the sub-interval dt. The st parameter 518 does not have relevance when the Source is transmitting at a fixed 519 rate throughout S. 521 7.3. Metric Definition 523 This section defines the REQUIRED aspects of the IP-layer Sender 524 Bitrate metric (unless otherwise indicated) for measurements at the 525 specified Source on packets addressed for the intended Destination 526 host and matching the required Type-P: 528 Define the IP-layer Sender Bit Rate, B(S,st), to be the number of IP- 529 layer bits (including header and data fields) that are transmitted 530 from the Source during one contiguous sub-interval, st, during the 531 test interval S (where S SHALL be longer than I), and where the 532 fixed-size packet count during that single sub-interval st also 533 provides the number of IP-layer bits in any interval: n0[stn-1,stn]. 535 Measurements according to these definitions SHALL use UDP transport 536 layer. Any feedback from Dst host to Src host received by Src host 537 during an interval [stn-1,stn] MUST NOT result in an adaptation of 538 the Src host traffic conditioning during this interval. 540 7.4. Discussion 542 Both the Sender and Receiver or (source and destination) bit rates 543 SHOULD be assessed as part of a measurement. 545 7.5. Reporting the Metric 547 The IP-Layer Sender Bit Rate SHALL be reported with meaningful 548 resolution, in units of Megabits per second. 550 Individual IP-Layer Sender Bit Rate measurements are discussed 551 further in Section 9. 553 8. Method of Measurement 555 The duration of a test, I, MUST be constrained in a production 556 network, since this is an active test method and it will likely cause 557 congestion on the Src to Dst host path during a test. 559 Additional Test methods and configurations may be provided in this 560 section, after review and further testing. 562 8.1. Load Rate Adjustment Algorithm (from udpst) 564 A table is pre-built defining all the offered load rates that will be 565 supported (R1 - Rn, in ascending order). Each rate is defined as 566 datagrams of size S, sent as a burst of count C, every time interval 567 T. While it is advantageous to use datagrams of as large a size as 568 possible, it may be prudent to use a slightly smaller maximum that 569 allows for secondary protocol headers and/or tunneling without 570 resulting in IP-layer fragmentation. 572 At the beginning of a test, the sender begins sending at rate R1 and 573 the receiver starts a feedback timer at interval F (while awaiting 574 inbound datagrams). As datagrams are received they are checked for 575 sequence number anomalies (loss, out-of-order, duplication, etc.) and 576 the delay variation is measured (one-way or round-trip). This 577 information is accumulated until the feedback timer F expires and a 578 status feedback message is sent from the receiver back to the sender, 579 to communicate this information. The accumulated statistics are then 580 reset by the receiver for the next feedback interval. As feedback 581 messages are received back at the sender, they are evaluated to 582 determine how to adjust the current offered load rate (Rx). 584 If the feedback indicates that there were no sequence number 585 anomalies AND the delay variation was below the lower threshold, the 586 offered load rate is increased. If congestion has not been confirmed 587 up to this point, the offered load rate is increased by more than one 588 rate (e.g., Rx+10). This allows the offered load to quickly reach a 589 near-maximum rate. Conversely, if congestion has been previously 590 confirmed, the offered load rate is only increased by one (Rx+1). 592 If the feedback indicates that sequence number anomalies were 593 detected OR the delay variation was above the upper threshold, the 594 offered load rate is decreased. If congestion is confirmed by the 595 current feedback message being processed, the offered load rate is 596 decreased by more than one rate (e.g., Rx-30). This one-time 597 reduction is intended to compensate for the fast initial ramp-up. In 598 all other cases, the offered load rate is only decreased by one (Rx- 599 1). 601 If the feedback indicates that there were no sequence number 602 anomalies AND the delay variation was above the lower threshold, but 603 below the upper threshold, the offered load rate is not changed. 604 This allows time for recent changes in the offered load rate to 605 stabilize, and the feedback to represent current conditions more 606 accurately. 608 Lastly, the method for confirming congestion is that there were 609 sequence number anomalies OR the delay variation was above the upper 610 threshold for two consecutive feedback intervals. 612 8.2. Measurement Qualification or Verification 614 When assessing a Maximum rate as the metric specifies, artificially 615 high (optimistic) values might be measured until some buffer on the 616 path is filled. Other causes include bursts of back-to-back packets 617 with idle intervals delivered by a path, while the measurement 618 interval (dt) is small and aligned with the bursts. The artificial 619 values might result in an un-sustainable Maximum Capacity observed 620 when the method of measurement is searching for the Maximum, and that 621 would not do. This situation is different from the bi-modal service 622 rates (discussed under Reporting), which are characterized by a 623 multi-second duration (much longer than the measured RTT) and 624 repeatable behavior. 626 There are many ways that the Method of Measurement could handle this 627 false-max issue. The default value for measurement of singletons (dt 628 = 1 second) has proven to a be of practical value during tests of 629 this method, allows the bimodal service rates to be characterized, 630 and it has an obvious alignment with the reporting units (Mbps). 632 Another approach comes from Section 24 of RFC 2544[RFC2544] and its 633 discussion of Trial duration, where relatively short trials conducted 634 as part of the search are followed by longer trials to make the final 635 determination. In the production network, measurements of singletons 636 and samples (the terms for trials and tests of Lab Benchmarking) must 637 be limited in duration because they may be service-affecting. But 638 there is sufficient value in repeating a sample with a fixed sending 639 rate determined by the previous search for the Max IP-layer Capacity, 640 to qualify the result in terms of the other performance metrics 641 measured at the same time. 643 A qualification measurement for the search result is a subsequent 644 measurement, sending at a fixed 99.x % of the Max IP-layer Capacity 645 for I, or an indefinite period. The same Max Capacity Metric is 646 applied, and the Qualification for the result is a sample without 647 packet loss or a growing minimum delay trend in subsequent singletons 648 (or each dt of the measurement interval, I). Samples exhibiting 649 losses or increasing queue occupation require a repeated search and/ 650 or test at reduced fixed sender rate for qualification. 652 Here, as with any Active Capacity test, the test duration must be 653 kept short. 10 second tests for each direction of transmission are 654 common today. The default measurement interval specified here is I = 655 10 seconds). In combination with a fast search method and user- 656 network coordination, the concerns raised in RFC 6815[RFC6815] are 657 alleviated. The method for assessing Max IP Capacity is different 658 from classic [RFC2544] methods: they use short term load adjustment 659 and are sensitive to loss and delay, like other congestion control 660 algorithms used on the Internet every day. 662 8.3. Measurement Considerations 664 In general, the wide-spread measurements that this memo encourages 665 will encounter wide-spread behaviors. The bimodal IP Capacity 666 behavior is a good example. 668 The path measured may be state-full based on many factors, and the 669 Parameter "Time of day" when a test starts may not be enough enough 670 information. Repeatable testing may require the time from the 671 beginning of a measured flow, and how the flow is constructed 672 including how much traffic has already been sent on that flow when a 673 state-change is observed, because the state-change may be based on 674 time or bytes sent or both. 676 Many different traffic shapers and on-demand access technology may be 677 encountered, as anticipated in [RFC7312], and play a key role in 678 measurement results. Methods MUST be prepared to provide a short 679 preamble transmission to activate on-demand access, and to discard 680 the preamble from subsequent test results. 682 In general, results depend on the sending stream characteristics; the 683 measurement community has known this for a long time, and to keep it 684 front of mind. Although the default is a single flow (F=1) for 685 testing, use of multiple flows may be advantageous for the following 686 reasons: 688 1. the test hosts may be able to create higher load than with a 689 single flow, or parallel test hosts may be used to generate 1 690 flow each. 692 2. there may be link aggregation present (flow-based load balancing) 693 and multiple flows are need to occupy each member of the 694 aggregate. 696 Each flow would be controlled using its own implementation of the 697 Load Adjustment (Search) Algorithm. 699 As testing continues, implementers should expect some evolution in 700 the methods. 702 9. Reporting 704 The Maximum IP-Layer Capacity results SHOULD be reported in the 705 format of a table with a row for each of the test Phases and Number 706 of Flows. There SHOULD be columns for the phases with number of 707 flows, and for the resultant Maximum IP-Layer Capacity results for 708 the aggregate and each flow tested. 710 The PM list metrics corresponding to the sub-interval where the 711 Maximum Capacity occurred MUST accompany a report of Maximum IP-Layer 712 Capacity results, for each test phase. 714 +--------------+----------------------+-----------+-----------------+ 715 | Phase, # | Max IP-Layer | Loss | RTT min, max, | 716 | Flows | Capacity, Mbps | Ratio | msec | 717 +--------------+----------------------+-----------+-----------------+ 718 | Search,1 | 967.31 | 0.0002 | 30, 58 | 719 | Verify,1 | 966.00 | 0.0000 | 30, 38 | 720 +--------------+----------------------+-----------+-----------------+ 722 Maximum IP-layer Capacity Results 724 Static and configuration parameters: 726 The sub-interval time, dt, MUST accompany a report of Maximum IP- 727 Layer Capacity results, and the remaining Parameters from Section 4, 728 General Parameters. 730 The IP-Layer Sender Bit rate results SHOULD be reported in the format 731 of a table with a row for each of the test Phases, sub-intervals (st) 732 and Number of Flows. There SHOULD be columns for the phases with 733 number of flows, and for the resultant IP-Layer Sender Bit rate 734 results for the aggregate and each flow tested. 736 +------------------------+-------------+-----------------------+----+ 737 | Phase, Flow or | st, sec | Sender Bit Rate, Mbps | ?? | 738 | Aggregate | | | | 739 +------------------------+-------------+-----------------------+----+ 740 | Search,1 | 0.00 - 0.05 | 345 | __ | 741 | Search,2 | 0.00 - 0.05 | 289 | __ | 742 | Search,Agg | 0.00 - 0.05 | 634 | __ | 743 +------------------------+-------------+-----------------------+----+ 745 IP-layer Sender Bit Rate Results 747 Static and configuration parameters: 749 The subinterval time, st, MUST accompany a report of Sender IP-Layer 750 Bit Rate results. 752 Also, the values of the remaining Parameters from Section 4, General 753 Parameters, MUST be reported. 755 10. Security Considerations 757 Active metrics and measurements have a long history of security 758 considerations [add references to LMAP Framework, etc.]. 760 762 11. IANA Considerations 764 This memo makes no requests of IANA. 766 12. Acknowledgements 768 Thanks to Joachim Fabini, Matt Mathis, and Ignacio Alvarez-Hamelin 769 for their extensive comments on the memo and related topics. 771 13. References 773 13.1. Normative References 775 [RFC1242] Bradner, S., "Benchmarking Terminology for Network 776 Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, 777 July 1991, . 779 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 780 Requirement Levels", BCP 14, RFC 2119, 781 DOI 10.17487/RFC2119, March 1997, 782 . 784 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 785 "Framework for IP Performance Metrics", RFC 2330, 786 DOI 10.17487/RFC2330, May 1998, 787 . 789 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 790 Network Interconnect Devices", RFC 2544, 791 DOI 10.17487/RFC2544, March 1999, 792 . 794 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 795 Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, 796 September 1999, . 798 [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology 799 for LAN Switching Devices", RFC 2889, 800 DOI 10.17487/RFC2889, August 2000, 801 . 803 [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining 804 Empirical Bulk Transfer Capacity Metrics", RFC 3148, 805 DOI 10.17487/RFC3148, July 2001, 806 . 808 [RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity", 809 RFC 5136, DOI 10.17487/RFC5136, February 2008, 810 . 812 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. 813 Dugatkin, "IPv6 Benchmarking Methodology for Network 814 Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May 815 2008, . 817 [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera, 818 "Device Reset Characterization", RFC 6201, 819 DOI 10.17487/RFC6201, March 2011, 820 . 822 [RFC6412] Poretsky, S., Imhoff, B., and K. Michielsen, "Terminology 823 for Benchmarking Link-State IGP Data-Plane Route 824 Convergence", RFC 6412, DOI 10.17487/RFC6412, November 825 2011, . 827 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 828 for Equal Cost Multipath Routing and Link Aggregation in 829 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 830 . 832 [RFC6673] Morton, A., "Round-Trip Packet Loss Metrics", RFC 6673, 833 DOI 10.17487/RFC6673, August 2012, 834 . 836 [RFC6815] Bradner, S., Dubray, K., McQuaid, J., and A. Morton, 837 "Applicability Statement for RFC 2544: Use on Production 838 Networks Considered Harmful", RFC 6815, 839 DOI 10.17487/RFC6815, November 2012, 840 . 842 [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet 843 Sizes for Additional Testing", RFC 6985, 844 DOI 10.17487/RFC6985, July 2013, 845 . 847 [RFC7312] Fabini, J. and A. Morton, "Advanced Stream and Sampling 848 Framework for IP Performance Metrics (IPPM)", RFC 7312, 849 DOI 10.17487/RFC7312, August 2014, 850 . 852 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 853 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 854 May 2016, . 856 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 857 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 858 May 2017, . 860 [RFC8337] Mathis, M. and A. Morton, "Model-Based Metrics for Bulk 861 Transport Capacity", RFC 8337, DOI 10.17487/RFC8337, March 862 2018, . 864 [RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. 865 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for 866 the IP Performance Metrics (IPPM) Framework", RFC 8468, 867 DOI 10.17487/RFC8468, November 2018, 868 . 870 13.2. Informative References 872 [copycat] Edleine, K., Kuhlewind, K., Trammell, B., and B. Donnet, 873 "copycat: Testing Differential Treatment of New Transport 874 Protocols in the Wild (ANRW '17)", July 2017, 875 . 877 [RFC8239] Avramov, L. and J. Rapp, "Data Center Benchmarking 878 Methodology", RFC 8239, DOI 10.17487/RFC8239, August 2017, 879 . 881 [TST009] Morton, R. A., "ETSI GS NFV-TST 009 V3.1.1 (2018-10), 882 "Network Functions Virtualisation (NFV) Release 3; 883 Testing; Specification of Networking Benchmarks and 884 Measurement Methods for NFVI"", October 2018, 885 . 888 [VSPERF-b2b] 889 Morton, A., "Back2Back Testing Time Series (from CI)", 890 June 2017, . 894 [VSPERF-BSLV] 895 Morton, A. and S. Rao, "Evolution of Repeatability in 896 Benchmarking: Fraser Plugfest (Summary for IETF BMWG)", 897 July 2018, 898 . 902 Authors' Addresses 904 Al Morton 905 AT&T Labs 906 200 Laurel Avenue South 907 Middletown,, NJ 07748 908 USA 910 Phone: +1 732 420 1571 911 Fax: +1 732 368 1192 912 Email: acm@research.att.com 914 Ruediger Geib 915 Deutsche Telekom 916 Heinrich Hertz Str. 3-7 917 Darmstadt 64295 918 Germany 920 Phone: +49 6151 5812747 921 Email: Ruediger.Geib@telekom.de 923 Len Ciavattone 924 AT&T Labs 925 200 Laurel Avenue South 926 Middletown,, NJ 07748 927 USA 929 Email: lencia@att.com