<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="std" docName="draft-morton-ippm-capacity-metric-method-01"
     ipr="trust200902" updates="????">
  <front>
    <title abbrev="IP Capacity Metrics/Methods">Metrics and Methods for IP
    Capacity</title>

    <author fullname="Al Morton" initials="A." surname="Morton">
      <organization>AT&amp;T Labs</organization>

      <address>
        <postal>
          <street>200 Laurel Avenue South</street>

          <city>Middletown,</city>

          <region>NJ</region>

          <code>07748</code>

          <country>USA</country>
        </postal>

        <phone>+1 732 420 1571</phone>

        <facsimile>+1 732 368 1192</facsimile>

        <email>acm@research.att.com</email>

        <uri/>
      </address>
    </author>

    <author fullname="Ruediger Geib" initials="R." surname="Geib">
      <organization>Deutsche Telekom</organization>

      <address>
        <postal>
          <street>Heinrich Hertz Str. 3-7</street>

          <city>Darmstadt</city>

          <region/>

          <code>64295</code>

          <country>Germany</country>
        </postal>

        <phone>+49 6151 5812747</phone>

        <facsimile/>

        <email>Ruediger.Geib@telekom.de</email>

        <uri/>
      </address>
    </author>

    <author fullname="Len Ciavattone" initials="L." surname="Ciavattone">
      <organization>AT&amp;T Labs</organization>

      <address>
        <postal>
          <street>200 Laurel Avenue South</street>

          <city>Middletown,</city>

          <region>NJ</region>

          <code>07748</code>

          <country>USA</country>
        </postal>

        <phone/>

        <facsimile/>

        <email>lencia@att.com</email>

        <uri/>
      </address>
    </author>

    <date day="4" month="November" year="2019"/>

    <abstract>
      <t>This memo revisits the problem of Network Capacity metrics first
      examined in RFC 5136. The memo specifies a more practical Maximum
      IP-layer Capacity metric definition catering for measurement purposes,
      and outlines the corresponding methods of measurement.</t>
    </abstract>

    <note title="Requirements Language">
      <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
      "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
      "OPTIONAL" in this document are to be interpreted as described in BCP
      14<xref target="RFC2119"/> <xref target="RFC8174"/> when, and only when,
      they appear in all capitals, as shown here.</t>

      <t/>
    </note>
  </front>

  <middle>
    <section title="Introduction">
      <t>The IETF's efforts to define Network and Bulk Transport Capacity have
      been chartered and progressed for over twenty years. Over that time, the
      performance community has seen development of Informative definitions in
      <xref target="RFC3148"/> for Framework for Bulk Transport Capacity
      (BTC), RFC 5136 for Network Capacity and Maximum IP-layer Capacity, and
      the Experimental metric definitions and methods in <xref
      target="RFC8337"/>, Model-Based Metrics for BTC.</t>

      <t>This memo revisits the problem of Network Capacity metrics examined
      first in <xref target="RFC3148"/> and later in <xref target="RFC5136"/>.
      Maximum IP-Layer Capacity and <xref target="RFC3148"/> Bulk Transfer
      Capacity (goodput) are different metrics. Max IP-layer Capacity is like
      the theoretical goal for goodput. There are many metrics in <xref
      target="RFC5136"/>, such as Available Capacity. Measurements depend on
      the network path under test and the use case. Here, the main use case is
      to assess the maximum capacity of the access network, with specific
      performance criteria used in the measurement.</t>

      <t>This memo recognizes the importance of a definition of a Maximum
      IP-layer Capacity Metric at a time when access speeds have increased
      dramatically; a definition that is both practical and effective for the
      performance community's needs, including Internet users. The metric
      definition is intended to use Active Methods of Measurement <xref
      target="RFC7799"/>, and a method of measurement is included.</t>

      <t>The most direct active measurement of IP-layer Capacity would use IP
      packets, but in practice a transport header is needed to traverse
      address and port translators. UDP offers the most direct assessment
      possibility, and in the [copycat]<xref target="copycat"/> measurement
      study to investigate whether UDP is viable as a general Internet
      transport protocol, the authors found that a high percentage of paths
      tested support UDP transport. A number of liaisons have been exchanged
      on this topic [ refs to ITU-T SG12, ETSI STQ, BBF liaisons ], discussing
      the laboratory and field tests that support the UDP-based approach to
      IP-layer Capacity measurement.</t>

      <t>This memo also recognizes the many updates to the IP Performance
      Metrics Framework <xref target="RFC2330"/> published over twenty years,
      and makes use of <xref target="RFC7312"/> for Advanced Stream and
      Sampling Framework, and RFC 8468 <xref target="RFC8468"/>IPv4, IPv6, and
      IPv4-IPv6 Coexistence Updates.</t>

      <t>NOTE: The text contains a few Author comments, in brackets [RG: ,
      acm: ]</t>
    </section>

    <section title="Scope and Goals">
      <t>The scope of this memo is to define a metric and corresponding method
      to unambiguously perform Active measurements of Maximum IP-Layer
      Capacity, along with related metrics and methods.</t>

      <t>The main goal is to harmonize the specified metric and method across
      the industry, and this memo is the vehicle through which working group
      (and eventually, IETF) consensus will be captured and communicated to
      achieve broad agreement, and possibly result in changes in the
      specifications of other Standards Development Organizations (SDO)
      (through the SDO's normal contribution process, or through liaison
      exchange).</t>

      <t>A local goal is to aid efficient test procedures where possible, and
      to recommend reporting with additional interpretation of the results.
      Also, to foster the development of protocol support for this metric and
      method of measurement (all active testing protocols currently defined by
      the IPPM WG are UDP-based, meeting a key requirement of these
      methods).</t>
    </section>

    <section title="Motivation">
      <t>As with any problem that has been worked for many years in various
      SDOs without any special attempts at coordination, various solutions for
      metrics and methods have emerged.</t>

      <t>There are five factors that have changed (or begun to change) in the
      2013-2019 time frame, and the presence of any one of them on the path
      requires features in the measurement design to account for the
      changes:</t>

      <t><list style="numbers">
          <t>Internet access is no longer the bottleneck for many users.</t>

          <t>Both speed and latency are important to user's satisfaction.</t>

          <t>UDP's growing role in Transport, in areas where TCP once
          dominated.</t>

          <t>Content and applications moving physically closer to users.</t>

          <t>Less emphasis on ISP gateway measurements, possibly due to less
          traffic crossing ISP gateways in future.</t>
        </list></t>
    </section>

    <section title="General Parameters and Definitions">
      <t>This section lists the REQUIRED input factors to specify a Sender or
      Receiver metric.<list style="symbols">
          <t>Src, the address of a host (such as the globally routable IP
          address).</t>

          <t>Dst, the address of a host (such as the globally routable IP
          address).</t>

          <t>i, the limit on the number of Hops a specific packet may visit as
          it traverses from the host at Src to the host at Dst (such as the
          TTL or Hop Limit).</t>

          <t>MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops).</t>

          <t>T0, the time at the start of measurement interval, when packets
          are first transmitted from the Source.</t>

          <t>I, the duration of a measurement interval (default 10 sec)</t>

          <t>dt, the duration of N equal sub-intervals in I (default 1
          sec)</t>

          <t>Tmax, a maximum waiting time for test packets to arrive at the
          destination, set sufficiently long to disambiguate packets with long
          delays from packets that are discarded (lost), such that the
          distribution of one-way delay is not truncated.</t>

          <t>F, the number of different flows synthesized by the method
          (default 1 flow)</t>

          <t>flow, the stream of packets with the same n-tuple of designated
          header fields that (when held constant) result in identical
          treatment in a multi-path decision (such as the decision taken in
          load balancing). Note: The IPv6 flow label MAY be included in the
          flow definition when routers have complied with <xref
          target="RFC6438"/> guidelines at the Tunnel End Points (TEP), and
          the source of the measurement is a TEP.</t>

          <t>Type-P, the complete description of the packets for which this
          assessment applies (including the flow-defining fields). Note that
          the UDP transport layer is one requirement specified below. Type-P
          is a parallel concept to "population of interest" defined in ITU-T
          Rec. Y.1540.</t>

          <t>PM, a list of fundamental metrics, such as loss, delay, and
          reordering, and corresponding Target performance threshold. At least
          one fundamental metric and Target performance threshold MUST be
          supplied (such as One-way IP Packet Loss [RFC7680] equal to
          zero).</t>
        </list>A non-Parameter which is required for several metrics is
      defined below:</t>

      <t><list style="symbols">
          <t>T, the host time of the *first* test packet's *arrival* as
          measured at MP(Dst). There may be other packets sent between source
          and destination hosts that are excluded, so this is the time of
          arrival of the first packet used for measurement of the metric.</t>
        </list>Note that time stamps, sequnce numbers, etc. will be
      established by the test protocol.</t>
    </section>

    <section title="IP-Layer Capacity Singleton Metric Definitions">
      <t>This section sets requirements for the following components to
      support the Maximum IP-layer Capacity Metric.</t>

      <section title="Formal Name">
        <t>Type-P-IP-Capacity, or informally called IP-layer Capacity.</t>

        <t>Note that Type-P depends on the chosen method.</t>
      </section>

      <section title="Parameters">
        <t>This section lists the REQUIRED input factors to specify the
        metric, beyond those listed in Section 4.</t>

        <t>No additional Parameters are needed.</t>
      </section>

      <section title="Metric Definitions">
        <t>This section defines the REQUIRED aspects of the measureable
        IP-layer Capacity metric (unless otherwise indicated) for measurements
        between specified Source and Destination hosts:</t>

        <t>Define the IP-layer capacity, C(T,I,PM), to be the number of
        IP-layer bits (including header and data fields) in packets that can
        be transmitted from the Src host and correctly received by the Dst
        host during one contiguous sub-interval, dt.</t>

        <t>The number of these IP-layer bits is designated n0[dtn-1,dtn] for a
        specific dtn.</t>

        <t>When the packet size is known and of fixed size, the packet count
        during a single sub-interval dt multiplied by the total bits in IP
        header and data fields is equal to n0[dtn-1,dtn].</t>

        <t>Anticipating a Sample of Singletons, the interval dt SHOULD be set
        to a natural number m so that T+I = T + m*dt with dtn - dtn-1 = dt and
        with 0 &lt; n &lt;= m.</t>

        <t>Parameter PM represents other performance metrics [see section
        Related Round-Trip Delay and Loss Definitions below]; their
        measurement results SHALL be collected during measurement of IP-layer
        Capacity and associated with the corresponding dtn for further
        evaluation and reporting.</t>

        <t>Mathematically, this definition can be represented as:</t>

        <t><figure title="Equation for IP-Layer Capacity">
            <artwork align="center"><![CDATA[
                        ( n0[dtn-1,dtn] )
        C(T,I,PM) = -------------------------
                               dt

]]></artwork>
          </figure>and:<list style="symbols">
            <t>n0 is the total number of IP-layer header and payload bits that
            can be transmitted in Standard Formed packets from the Src host
            and correctly received by the Dst host during one contiguous
            sub-interval, dt in length, during the interval [T, T+I],</t>

            <t>C(T,I,PM) the IP-Layer Capacity, corresponds to the value of n0
            measured in any sub-interval ending at dtn (meaning T + n*dt),
            divided by the length of sub-interval, dt.</t>

            <t>all sub-intervals SHOULD be of equal duration. Choosing dt as
            non-overlapping consecutive time intervals allows for a simple
            implementation.</t>

            <t>The bit rate of the physical interface of the measurement
            device must be higher than that of the link whose C(T,I,PM) is to
            be measured.</t>
          </list></t>

        <t>Measurements according to these definitions SHALL use UDP transport
        layer.</t>
      </section>

      <section title="Related Round-Trip Delay and Loss Definitions">
        <t>RTD[dtn-1,dtn] is defined as a sample of the <xref
        target="RFC2681"/> Round-trip Delay between the Src host and the Dst
        host over the interval [T,T+I]. The statistics used to to summarize
        RTD[dtn-1,dtn] MAY include the minimum, maximum, and mean.</t>

        <t>RTL[dtn-1,dtn] is defined as a sample of the <xref
        target="RFC6673"/> Round-trip Loss between the Src host and the Dst
        host over the interval [T,T+I]. The statistics used to to summarize
        RTL[dtn-1,dtn] MAY include the lost packet count and the lost packet
        ratio.</t>
      </section>

      <section title="Discussion">
        <t>See the corresponding section for Maximum IP-Layer Capacity.</t>
      </section>

      <section title="Reporting the Metric">
        <t>The IP-Layer Capacity SHALL be reported with meaningful resolution,
        in units of Megabits per second (Mbps).</t>

        <t>The Related Round Trip Delay and/or Loss metric measurements for
        the same Singleton SHALL be reported, also with meaningful resolution
        for the values measured.</t>

        <t>Individual Capacity measurements MAY be reported in a manner
        consistent with the Maximum IP-Layer Capacity, see Section 9.</t>
      </section>
    </section>

    <section title="Maximum IP-Layer Capacity Metric Definitions (Statistic)">
      <t>This section sets requirements for the following components to
      support the Maximum IP-layer Capacity Metric.</t>

      <section title="Formal Name">
        <t>Type-P-Max-IP-Capacity, or informally called Maximum IP-layer
        Capacity.</t>

        <t>Note that Type-P depends on the chosen method.</t>
      </section>

      <section title="Parameters">
        <t>This section lists the REQUIRED input factors to specify the
        metric, beyond those listed in Section 4.</t>

        <t>No additional Parameters or definitions are needed.</t>
      </section>

      <section title="Metric Definitions">
        <t>This section defines the REQUIRED aspects of the Maximum IP-layer
        Capacity metric (unless otherwise indicated) for measurements between
        specified Source and Destination hosts:</t>

        <t>Define the Maximum IP-layer capacity, Maximum_C(T,I,PM), to be the
        maximum number of IP-layer bits n0[dtn-1,dtn] that can be transmitted
        in packets from the Src host and correctly received by the Dst host,
        over all dt length intervals in [T, T+I], and meeting the PM criteria.
        Equivalently the Maximum of a Sample of size m of C(T,I,PM) collected
        during the interval [T, T+I] and meeting the PM criteria.</t>

        <t>The interval dt SHOULD be set to a natural number m so that T+I = T
        + m*dt with dtn - dtn-1 = dt and with 0 &lt; n &lt;= m.</t>

        <t>Parameter PM represents the other performance metrics [see section
        Related Round-Trip Delay and Loss Definitions below] and their
        measurement results for the maximum IP-layer capacity. At least one
        target performance threshold (PM criterion) MUST be defined. If more
        than one target performance threshold is defined, then the
        sub-interval with maximum number of bits transmitted MUST meet all the
        target performance thresholds.</t>

        <t>Mathematically, this definition can be represented as:</t>

        <t><figure title="Equation for Maximum Capacity">
            <artwork align="center"><![CDATA[
                        max  ( n0[dtn-1,dtn] )
                       [T,T+I]
  Maximum_C(T,I,PM) = -------------------------
                                 dt
 where:
    T                                      T+I
    _________________________________________
    |   |   |   |   |   |   |   |   |   |   |
dtn=0   1   2   3   4   5   6   7   8   9   m=10


]]></artwork>
          </figure>and:<list style="symbols">
            <t>n0 is the total number of IP-layer header and payload bits that
            can be transmitted in Standard Formed packets from the Src host
            and correctly received by the Dst host during one contiguous
            sub-interval, dt in length, during the interval [T, T+I],</t>

            <t>Maximum _C(T,I,PM) the Maximum IP-Layer Capacity, corresponds
            to the maximum value of n0 measured in any sub-interval ending at
            dtn (meaning T + n*dt), divided by the constant length of all
            sub-intervals, dt.</t>

            <t>all sub-intervals SHOULD be of equal duration. Choosing dt as
            non-overlapping consecutive time intervals allows for a simple
            implementation.</t>

            <t>The bit rate of the physical interface of the measurement
            systems must be higher than that of the link whose Maximum
            _C(T,I,PM) is to be measured (the bottleneck link).</t>
          </list></t>

        <t>In this definition, the m sub-intervals can be viewed as trials
        when the Src host varies the transmitted packet rate, searching for
        the maximum n0 that meets the PM criteria measured at the Dst host in
        a test of duration, I. When the transmitted packet rate is held
        constant at the Src host, the m sub-intervals may also be viewed as
        trials to evaluate the stability of n0 and metric(s) in the PM list
        over all dt-length intervals in I.</t>

        <t>Measurements according to these definitions SHALL use UDP transport
        layer.</t>
      </section>

      <section title="Related Round-Trip Delay and Loss Definitions">
        <t>RTD[dtn-1,dtn] is defined as a sample of the <xref
        target="RFC2681"/> Round-trip Delay between the Src host and the Dst
        host over the interval [T,T+I], and corresponds to the dt interval
        containing Maximum_C(T,I,PM). The statistics used to to summarize
        RTD[dtn-1,dtn] MAY include the minimum, maximum, and mean.</t>

        <t>RTL[dtn-1,dtn] is defined as a sample of the <xref
        target="RFC6673"/> Round-trip Loss between the Src host and the Dst
        host over the interval [T,T+I] and corresponds to the dt interval
        containing Maximum_C(T,I,PM). The statistics used to to summarize
        RTL[dtn-1,dtn] MAY include the lost packet count and the lost packet
        ratio.</t>
      </section>

      <section title="Discussion">
        <t>If traffic conditioning applies along a path for which Maximum
        _C(T,I,PM) is to be determined, different values for dt SHOULD be
        picked and measurements be executed during multiple intervals [T,
        T+I]. Any single interval dt SHOULD be chosen so that is an integer
        multiple of increasing values k times serialisation delay of a path
        MTU at the physical interface speed where traffic conditioning is
        expected. This should avoid taking configured burst tolerance
        singletons as a valid Maximum _C(T,I,PM) result.</t>

        <t>A Maximum_C(T,I,PM) without any indication of bottleneck
        congestion, be that an increasing latency, packet loss or ECN marks
        during a measurement interval I, is likely to underestimate
        Maximum_C(T,I,PM).</t>
      </section>

      <section title="Reporting the Metric">
        <t>The Maximum IP-Layer Capacity SHALL be reported with meaningful
        resolution, in units of Megabits per second.</t>

        <t>The Related Round Trip Delay and/or Loss metric measurements for
        the same Singleton SHALL be reported, also with meaningful resolution
        for the values measured.</t>

        <t>When there are demonstrated and repeatable modes in the Sample,
        then the Maximum IP-Layer Capacity SHALL be reported for each mode,
        along with the relative time from the beginning of the stream that the
        mode was observed to be present. Bimodal Maxima have been observed
        with some services, sometimes called a "turbo" mode" intending to
        deliver short transfers more quickly, or reduce the initial buffering
        time for some video streams.</t>
      </section>
    </section>

    <section title="IP-Layer Sender Bit Rate Singleton Metric Definitions">
      <t>This section sets requirements for the following components to
      support the IP-layer Sender Bitrate Metric.</t>

      <section title="Formal Name">
        <t>Type-P-IP-Sender-Bit-Rate, or informally called IP-layer Sender
        Bitrate.</t>

        <t>Note that Type-P depends on the chosen method.</t>
      </section>

      <section title="Parameters">
        <t>This section lists the REQUIRED input factors to specify the
        metric, beyond those listed in Section 4.</t>

        <t><list style="symbols">
            <t>S, the duration of the measurement interval at the Source</t>

            <t>st, the nominal duration of N sub-intervals in S (default =
            0.05 seconds)</t>
          </list></t>

        <t>S SHALL be longer than I, primarily to account for on-demand
        activation of the path, or any preamble to testing required.</t>

        <t>st SHOULD be much smaller than the sub-interval dt. The st
        parameter does not have relevance when the Source is transmitting at a
        fixed rate throughout S.</t>
      </section>

      <section title="Metric Definition">
        <t>This section defines the REQUIRED aspects of the IP-layer Sender
        Bitrate metric (unless otherwise indicated) for measurements at the
        specified Source on packets addressed for the intended Destination
        host and matching the required Type-P:</t>

        <t>Define the IP-layer Sender Bit Rate, B(S,st), to be the number of
        IP-layer bits (including header and data fields) that are transmitted
        from the Source during one contiguous sub-interval, st, during the
        test interval S (where S SHALL be longer than I), and where the
        fixed-size packet count during that single sub-interval st also
        provides the number of IP-layer bits in any interval:
        n0[stn-1,stn].</t>

        <t>Measurements according to these definitions SHALL use UDP transport
        layer. Any feedback from Dst host to Src host received by Src host
        during an interval [stn-1,stn] MUST NOT result in an adaptation of the
        Src host traffic conditioning during this interval.</t>
      </section>

      <section title="Discussion">
        <t>Both the Sender and Receiver or (source and destination) bit rates
        SHOULD be assessed as part of a measurement.</t>
      </section>

      <section title="Reporting the Metric">
        <t>The IP-Layer Sender Bit Rate SHALL be reported with meaningful
        resolution, in units of Megabits per second.</t>

        <t>Individual IP-Layer Sender Bit Rate measurements are discussed
        further in Section 9.</t>
      </section>
    </section>

    <section title="Method of Measurement">
      <t>The duration of a test, I, MUST be constrained in a production
      network, since this is an active test method and it will likely cause
      congestion on the Src to Dst host path during a test.</t>

      <t>Additional Test methods and configurations may be provided in this
      section, after review and further testing.</t>

      <section title="Load Rate Adjustment Algorithm (from udpst)">
        <t>A table is pre-built defining all the offered load rates that will
        be supported (R1 &ndash; Rn, in ascending order). Each rate is defined
        as datagrams of size S, sent as a burst of count C, every time
        interval T. While it is advantageous to use datagrams of as large a
        size as possible, it may be prudent to use a slightly smaller maximum
        that allows for secondary protocol headers and/or tunneling without
        resulting in IP-layer fragmentation.</t>

        <t>At the beginning of a test, the sender begins sending at rate R1
        and the receiver starts a feedback timer at interval F (while awaiting
        inbound datagrams). As datagrams are received they are checked for
        sequence number anomalies (loss, out-of-order, duplication, etc.) and
        the delay variation is measured (one-way or round-trip). This
        information is accumulated until the feedback timer F expires and a
        status feedback message is sent from the receiver back to the sender,
        to communicate this information. The accumulated statistics are then
        reset by the receiver for the next feedback interval. As feedback
        messages are received back at the sender, they are evaluated to
        determine how to adjust the current offered load rate (Rx).</t>

        <t>If the feedback indicates that there were no sequence number
        anomalies AND the delay variation was below the lower threshold, the
        offered load rate is increased. If congestion has not been confirmed
        up to this point, the offered load rate is increased by more than one
        rate (e.g., Rx+10). This allows the offered load to quickly reach a
        near-maximum rate. Conversely, if congestion has been previously
        confirmed, the offered load rate is only increased by one (Rx+1).</t>

        <t>If the feedback indicates that sequence number anomalies were
        detected OR the delay variation was above the upper threshold, the
        offered load rate is decreased. If congestion is confirmed by the
        current feedback message being processed, the offered load rate is
        decreased by more than one rate (e.g., Rx-30). This one-time reduction
        is intended to compensate for the fast initial ramp-up. In all other
        cases, the offered load rate is only decreased by one (Rx-1).</t>

        <t>If the feedback indicates that there were no sequence number
        anomalies AND the delay variation was above the lower threshold, but
        below the upper threshold, the offered load rate is not changed. This
        allows time for recent changes in the offered load rate to stabilize,
        and the feedback to represent current conditions more accurately.</t>

        <t>Lastly, the method for confirming congestion is that there were
        sequence number anomalies OR the delay variation was above the upper
        threshold for two consecutive feedback intervals.</t>
      </section>

      <section title="Measurement Qualification or Verification">
        <t>When assessing a Maximum rate as the metric specifies, artificially
        high (optimistic) values might be measured until some buffer on the
        path is filled. Other causes include bursts of back-to-back packets
        with idle intervals delivered by a path, while the measurement
        interval (dt) is small and aligned with the bursts. The artificial
        values might result in an un-sustainable Maximum Capacity observed
        when the method of measurement is searching for the Maximum, and that
        would not do. This situation is different from the bi-modal service
        rates (discussed under Reporting), which are characterized by a
        multi-second duration (much longer than the measured RTT) and
        repeatable behavior.</t>

        <t>There are many ways that the Method of Measurement could handle
        this false-max issue. The default value for measurement of singletons
        (dt = 1 second) has proven to a be of practical value during tests of
        this method, allows the bimodal service rates to be characterized, and
        it has an obvious alignment with the reporting units (Mbps).</t>

        <t>Another approach comes from Section 24 of RFC 2544<xref
        target="RFC2544"/> and its discussion of Trial duration, where
        relatively short trials conducted as part of the search are followed
        by longer trials to make the final determination. In the production
        network, measurements of singletons and samples (the terms for trials
        and tests of Lab Benchmarking) must be limited in duration because
        they may be service-affecting. But there is sufficient value in
        repeating a sample with a fixed sending rate determined by the
        previous search for the Max IP-layer Capacity, to qualify the result
        in terms of the other performance metrics measured at the same
        time.</t>

        <t>A qualification measurement for the search result is a subsequent
        measurement, sending at a fixed 99.x % of the Max IP-layer Capacity
        for I, or an indefinite period. The same Max Capacity Metric is
        applied, and the Qualification for the result is a sample without
        packet loss or a growing minimum delay trend in subsequent singletons
        (or each dt of the measurement interval, I). Samples exhibiting losses
        or increasing queue occupation require a repeated search and/or test
        at reduced fixed sender rate for qualification.</t>

        <t>Here, as with any Active Capacity test, the test duration must be
        kept short. 10 second tests for each direction of transmission are
        common today. The default measurement interval specified here is I =
        10 seconds). In combination with a fast search method and user-network
        coordination, the concerns raised in RFC 6815<xref target="RFC6815"/>
        are alleviated. The method for assessing Max IP Capacity is different
        from classic <xref target="RFC2544"/> methods: they use short term
        load adjustment and are sensitive to loss and delay, like other
        congestion control algorithms used on the Internet every day.</t>
      </section>

      <section title="Measurement Considerations">
        <t>In general, the wide-spread measurements that this memo encourages
        will encounter wide-spread behaviors. The bimodal IP Capacity behavior
        is a good example.</t>

        <t>The path measured may be state-full based on many factors, and the
        Parameter "Time of day" when a test starts may not be enough enough
        information. Repeatable testing may require the time from the
        beginning of a measured flow, and how the flow is constructed
        including how much traffic has already been sent on that flow when a
        state-change is observed, because the state-change may be based on
        time or bytes sent or both.</t>

        <t>Many different traffic shapers and on-demand access technology may
        be encountered, as anticipated in <xref target="RFC7312"/>, and play a
        key role in measurement results. Methods MUST be prepared to provide a
        short preamble transmission to activate on-demand access, and to
        discard the preamble from subsequent test results.</t>

        <t>In general, results depend on the sending stream characteristics;
        the measurement community has known this for a long time, and to keep
        it front of mind. Although the default is a single flow (F=1) for
        testing, use of multiple flows may be advantageous for the following
        reasons:</t>

        <t><list style="numbers">
            <t>the test hosts may be able to create higher load than with a
            single flow, or parallel test hosts may be used to generate 1 flow
            each.</t>

            <t>there may be link aggregation present (flow-based load
            balancing) and multiple flows are need to occupy each member of
            the aggregate.</t>
          </list>Each flow would be controlled using its own implementation of
        the Load Adjustment (Search) Algorithm.</t>

        <t>As testing continues, implementers should expect some evolution in
        the methods.</t>
      </section>
    </section>

    <section title="Reporting">
      <t>The Maximum IP-Layer Capacity results SHOULD be reported in the
      format of a table with a row for each of the test Phases and Number of
      Flows. There SHOULD be columns for the phases with number of flows, and
      for the resultant Maximum IP-Layer Capacity results for the aggregate
      and each flow tested.</t>

      <t>The PM list metrics corresponding to the sub-interval where the
      Maximum Capacity occurred MUST accompany a report of Maximum IP-Layer
      Capacity results, for each test phase.</t>

      <t/>

      <texttable style="full" title="Maximum IP-layer Capacity Results">
        <ttcol>Phase, # Flows</ttcol>

        <ttcol>Max IP-Layer Capacity, Mbps</ttcol>

        <ttcol>Loss Ratio</ttcol>

        <ttcol>RTT min, max, msec</ttcol>

        <c>Search,1</c>

        <c>967.31</c>

        <c>0.0002</c>

        <c>30, 58</c>

        <c>Verify,1</c>

        <c>966.00</c>

        <c>0.0000</c>

        <c>30, 38</c>
      </texttable>

      <t>Static and configuration parameters:</t>

      <t>The sub-interval time, dt, MUST accompany a report of Maximum
      IP-Layer Capacity results, and the remaining Parameters from Section 4,
      General Parameters.</t>

      <t>The IP-Layer Sender Bit rate results SHOULD be reported in the format
      of a table with a row for each of the test Phases, sub-intervals (st)
      and Number of Flows. There SHOULD be columns for the phases with number
      of flows, and for the resultant IP-Layer Sender Bit rate results for the
      aggregate and each flow tested.</t>

      <texttable style="full" title="IP-layer Sender Bit Rate Results">
        <ttcol>Phase, Flow or Aggregate</ttcol>

        <ttcol>st, sec</ttcol>

        <ttcol>Sender Bit Rate, Mbps</ttcol>

        <ttcol>??</ttcol>

        <c>Search,1</c>

        <c>0.00 - 0.05</c>

        <c>345</c>

        <c>__</c>

        <c>Search,2</c>

        <c>0.00 - 0.05</c>

        <c>289</c>

        <c>__</c>

        <c>Search,Agg</c>

        <c>0.00 - 0.05</c>

        <c>634</c>

        <c>__</c>
      </texttable>

      <t>Static and configuration parameters:</t>

      <t>The subinterval time, st, MUST accompany a report of Sender IP-Layer
      Bit Rate results.</t>

      <t>Also, the values of the remaining Parameters from Section 4, General
      Parameters, MUST be reported.</t>

      <t/>
    </section>

    <section title="Security Considerations">
      <t>Active metrics and measurements have a long history of security
      considerations [add references to LMAP Framework, etc.].</t>

      <t>&lt;There are certainly some new ones for Capacity testing&gt;</t>
    </section>

    <section anchor="IANA" title="IANA Considerations">
      <t>This memo makes no requests of IANA.</t>
    </section>

    <section title="Acknowledgements">
      <t>Thanks to Joachim Fabini, Matt Mathis, and Ignacio Alvarez-Hamelin
      for their extensive comments on the memo and related topics.</t>
    </section>
  </middle>

  <back>
    <references title="Normative References">
      <?rfc include='reference.RFC.2544'?>

      <?rfc include='reference.RFC.2330'?>

      <?rfc include='reference.RFC.1242'?>

      <?rfc include='reference.RFC.2889'?>

      <?rfc include="reference.RFC.2119"?>

      <?rfc include='reference.RFC.3148'?>

      <?rfc include='reference.RFC.5136'?>

      <?rfc include='reference.RFC.6201'?>

      <?rfc include='reference.RFC.6412'?>

      <?rfc include='reference.RFC.6815'?>

      <?rfc include='reference.RFC.7312'?>

      <?rfc include='reference.RFC.7799'?>

      <?rfc include='reference.RFC.8468'?>

      <?rfc include='reference.RFC.8337'?>

      <?rfc include='reference.RFC.5180'?>

      <?rfc include='reference.RFC.6985'?>

      <?rfc include='reference.RFC.8174'?>

      <?rfc include='reference.RFC.6438'?>

      <?rfc include='reference.RFC.6673'?>

      <?rfc ?>

      <?rfc include='reference.RFC.2681'?>

      <?rfc ?>

      <?rfc ?>

      <?rfc ?>

      <?rfc ?>
    </references>

    <references title="Informative References">
      <reference anchor="copycat"
                 target="https://irtf.org/anrw/2017/anrw17-final5.pdf">
        <front>
          <title>copycat: Testing Differential Treatment of New Transport
          Protocols in the Wild (ANRW '17)</title>

          <author fullname="Korian Edeline" initials="K." surname="Edleine">
            <organization/>
          </author>

          <author fullname="Mirja Kuhlewind" initials="K." surname="Kuhlewind">
            <organization/>
          </author>

          <author fullname="Brian Trammell" initials="B." surname="Trammell">
            <organization/>
          </author>

          <author fullname="Benoit Donnet" initials="B." surname="Donnet">
            <organization/>
          </author>

          <date day="15" month="July" year="2017"/>
        </front>
      </reference>

      <reference anchor="VSPERF-b2b"
                 target="https://wiki.opnfv.org/display/vsperf/Traffic+Generator+Testing#TrafficGeneratorTesting-AppendixB:Back2BackTestingTimeSeries(fromCI)">
        <front>
          <title>Back2Back Testing Time Series (from CI)</title>

          <author fullname="Al Morton" initials="A." surname="Morton">
            <organization/>
          </author>

          <date month="June" year="2017"/>
        </front>
      </reference>

      <reference anchor="VSPERF-BSLV"
                 target="https://datatracker.ietf.org/meeting/102/materials/slides-102-bmwg-evolution-of-repeatability-in-benchmarking-fraser-plugfest-summary-for-ietf-bmwg-00">
        <front>
          <title>Evolution of Repeatability in Benchmarking: Fraser Plugfest
          (Summary for IETF BMWG)</title>

          <author fullname="Al Morton" initials="A." surname="Morton">
            <organization>AT&amp;T Labs</organization>
          </author>

          <author fullname="Sridhar Rao" initials="S." surname="Rao">
            <organization>Spirent Communications</organization>
          </author>

          <date month="July" year="2018"/>
        </front>
      </reference>

      <reference anchor="TST009"
                 target="https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.01.01_60/gs_NFV-TST009v030101p.pdf">
        <front>
          <title>ETSI GS NFV-TST 009 V3.1.1 (2018-10), "Network Functions
          Virtualisation (NFV) Release 3; Testing; Specification of Networking
          Benchmarks and Measurement Methods for NFVI"</title>

          <author fullname="Rapporteur: Al Morton">
            <organization>ETSI Network Function Virtualization
            ISG</organization>
          </author>

          <date month="October" year="2018"/>
        </front>
      </reference>

      <?rfc include='reference.RFC.8239'?>

      <?rfc ?>
    </references>
  </back>
</rfc>
