2.8.4 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 58th IETF Meeting in Minneapolis, Minnesota USA. It may now be out-of-date.

Last Modified: 2003-11-03

Henk Uijterwaal <henk@ripe.net>
Matthew Zekauskas <matt@internet2.edu>
Transport Area Director(s):
Allison Mankin <mankin@psg.com>
Jon Peterson <jon.peterson@neustar.biz>
Transport Area Advisor:
Allison Mankin <mankin@psg.com>
Technical Advisor(s):
Andy Bierman <abierman@cisco.com>
Mailing Lists:
General Discussion: ippm@ietf.org
To Subscribe: https://www1.ietf.org/mailman/listinfo/ippm
In Body: subscribe
Archive: http://www.ietf.org/mail-archive/working-groups/ippm/current/maillist.html
Description of Working Group:
Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:

- connectivity

- one-way delay and loss

- round-trip delay and loss

- delay variation

- loss patterns

- packet reordering

- bulk transport capacity

- link bandwidth capacity

This is the cumulative set, including the metricsalready completed and published.

The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.

The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.

The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:
Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
Done  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
Done  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
Done  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
Feb 02  Create initial draft on the definitions of link bandwidth capacity.
Done  First draft for AS on one-way delay and loss.
Mar 02  Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS).
Done  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
Done  Create initial draft on a MIB for reporting IPPM metrics.
Apr 02  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
Apr 02  Create initial draft on comparing ITU and IETF IP performance metrics.
Done  Create initial draft on a packet reordering metric.
Jun 02  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
Jul 02  Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard.
Oct 02  Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay).
Oct 02  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
Dec 02  Submit AS for one-way delay and loss to the IESG for PS.
Dec 02  Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC.
Feb 03  Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard.
Mar 03  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
Mar 03  Discuss rechartering or ending working group.
  • - draft-ietf-ippm-owdp-07.txt
  • - draft-ietf-ippm-owdp-reqs-06.txt
  • - draft-ietf-ippm-metrics-registry-05.txt
  • - draft-ietf-ippm-reporting-mib-04.txt
  • - draft-ietf-ippm-reordering-04.txt
  • Request For Comments:
    RFC2330 I Framework for IP Performance Metrics
    RFC2678 E IPPM Metrics for Measuring Connectivity
    RFC2679 PS A One-way Delay Metric for IPPM
    RFC2680 PS A One-way Packet Loss Metric for IPPM
    RFC2681 PS A Round-trip Delay Metric for IPPM
    RFC3148 I A Framework for Defining Empirical Bulk Transfer Capacity Metrics
    RFC3357 I One-way Loss Pattern Sample Metrics
    RFC3393 PS IP Packet Delay Variation Metric for IPPM
    RFC3432 PS Network performance measurement for periodic streams

    Current Meeting Report

    IETF IP Performance Metrics WG (ippm)
    Thursday, November 13, 2003 at 13:00 to 15:00
    The meeting was moderated by the working group chairs, Matt Zekauskas and 
    Henk Uijterwaal.  Al Morton and Ruediger Geib took notes, which were 
    edited into these minutes by the chairs.
    1. Agenda Bashing
    2. Packet Reordering
    3. Reordering Density
    4. IPPM-MIB
    5. OWAMP
    6. Status, Milestones & Futures
    1. Agenda Bashing
       -- The chairs
    There was no bashing, but Matt welcomed Henk Uijterwaal as the new 
    co-chair, replacing Merike Kaeo who is now focusing on security work.
    2. Packet Reordering
       -- Al Morton
    Al Morton presented the changes since the last meeting to the 
    reordering draft (see slides).  The major changes are (1) continued 
    clarifications started in the last version, including changing the sense of 
    the base metric so it reports "true" when a packet has been 
    reordered, moving alternate methods of detecting reordering (time or bytes 
    instead of sequence number) to a separate section, (2) adding the 
    concept of steady-state versus transient reordering, and (3) adding text to 
    cover packet fragmentation.  The fragmentation text needs work, and will 
    likely have the details moved to a separate section.  The draft is still 
    missing an appendix on applicability promised by Greg Wright.  Matt asked 
    which appendices, if any, were going to be normative (and we should so 
    Jerry Perser asked if we could drop IPPM from the title, since in his view it 
    was more general (that is, there was some applicability to BMWG as well).  
    There was no objection from the assembled group.
    Matt (chair hat off) asked if adding fragmentation added too much 
    complexity; none of the other metrics look at individual fragments. Jerry 
    felt that either we should specify the metric so fragmentation does not 
    occur, or we should check whether fragments are reordered. Al Morton noted 
    that he has seen systems "in the wild" that don't handle reordered 
    fragments well.  Stanislav Shalunov noted that some versions of Linux 
    actually send fragments in the reverse order because they get a slight 
    performance improvement during reassembly; a note should be added.
    3. Reordering Density
       -- Anura Jayasumana
    The next presentation was by Anura Jayasumana on reordering density (an 
    individual submission).  This is the second presentation we had on this 
    topic, but the first by an author; there was a presentation at the last 
    IETF meeting by Jerry McCollum from Agilent, who is familiar with the 
    work. One can think of the reorder density function like a normalized 
    histogram of buffer occupancy.
    The major problem that the group has with this work is that lost packets 
    show up in the reorder density, so you cannot completely separate losses 
    from reordering events using this technique.
    An example showing a lost packet was given.  Stanislav Shalunov noted that it 
    appears that loss is counted as reordering.  Anura stated that it was 
    correct.  Stanislav asked if a sequence had no reordering but just loss 
    would the result be non-zero?  Anura said that was correct, that there is a 
    problem with packet loss, the current definition was a compromise, and 
    ensured the computation could be done in order n time and constant space.
    One interesting claim made by the author is that they have been able to 
    characterize the nature of loss based on distributions of reorder 
    density. It could also be a basis for TCP adjusting it's sensitivity to 
    reordering on the fly, and the distribution could be used when modeling a 
    network.  (See also the "Properties" slides.)
    One major change from the previous proposal was the addition of "late" and 
    "early" reordering density.  Together with the original definition, Anura 
    felt that this characterization might also be useful in solving 
    concatenation of partial path reordering metrics to give a reordering 
    metric for the full path.
    This led back to a discussion of packet loss.  Jerry Perser asked if lost 
    packets were early or late?  Anura said that was recognized with the help of a 
    place label.  Al Morton again brought up that this was the biggest issue in 
    this draft; the current working group draft makes reordering 
    orthogonal with loss, and there is a clear definition of "early" and 
    "late".  The definitions of in and out of order and early and late need to 
    merge to add this to the draft.  Anura said that it depends on what you are 
    trying to achieve with reordering measurements: perform TCP or network 
    diagnosis?  Predict network behavior?  The use of density would add 
    value; a simple number describing reordering doesn't provide enough 
    information.  The definition of loss would have to be agreed.
    Stanislav asserted that n-reordering makes the definition of loss 
    orthogonal to reordering. n-reordering metric doesn't change in the 
    presence of loss.  Not every router sends a stream of numbers with 
    monotonically increasing sequence numbers.  Anura thought there was a 
    problem with n-reordering and fast routers. Stanislav said that a linear 
    buffer would be sufficient.  Anura said that n squared would hit. 
    Stanislav said not for n-reordering.  Anura said that then duplicates are an 
    issue. Al said that you either save information on packets which have 
    arrived or those missing, usually the latter. Anura thought that a moving 
    window is better than keeping information on packets.  Jerry Perser noted 
    that test instruments can't track millions of duplicate packets.  Anura 
    said that with reordering density you only need to keep track for a fixed 
    window.  Stanislav wanted to get back to loss.  Anura said they 
    recognize loss within window.  Jerry wondered why it isn't called loss and 
    reordering density.  Anura said it isn't a loss density.  Jerry said that he 
    had a problem claiming reordered packets when only one is lost.  For 
    example, look at slide 6; Jerry said packet 4 is in sequence.  Anura said 
    that according to his definition packet 4 is reordered [3 was lost].  
    Anura wanted to know about packets arriving much later; suddenly "2" comes in 
    and your whole view changes.  Is this appropriate?  Applications would 
    presumably treat them as lost.  Are they lost or reordered?  Stanislav said 
    the definition of reordering is only based on a sequence of arrivals.
    Jerry wanted to know if the authors checked the resources you require to 
    implement reordering density (say for an oc192 link) -- what was the 
    fastest speed tested?   Anura said that results are forthcoming; they have 
    software on Linux for testing, and the fastest speed tested so far was fast 
    The authors would like this draft used as an alternative 
    characterization metric in the current draft or published as a separate 
    informational draft; publishing elsewhere was mentioned as another 
    possibility.  Henk asked if we should merge drafts or proceed 
    separately?  Matt said that his sense was that the current authors seem to be 
    disinclined to merge with loss mixed with reordering. Al Morton said that 
    was true, and there were no other comments.
    4. IPPM-MIB
       -- Emile Stephan
    Next, Emile Stephan presented the latest version of the MIB.  He wanted to 
    know the status of the registry -- it will be presented to the ADs as soon as 
    the (apparently now expired) version is refreshed by the I-D editor after 
    the meeting.  RMON would like the OIDs for the metrics.  Bob Cole asked 
    what happens for new metrics?  Matt said OIDs should be included in those 
    As far as the MIB goes, Emile presented some recent (minor) changes, and 
    then tried to walk the room through a sample setup showing how all the 
    tables were organized and how views of the data are consumed (see 
    slides).  He tried to do a demo showing the MIB presenting results from a 
    commercial measurement system, but the demo hiccupped in the middle.  (He 
    later showed it to the chairs.)  He noted that Andy Bierman had sent a 
    comment to the list today, and that he hasn't had time to consider it in 
    detail, but he will respond.
    Matt tried to start a discussion based on Andy's initial paragraph to the 
    list, which Matt summarized as that the MIB was "too complex". Henk asked 
    who had read the draft and if anyone was interested in implementing it or 
    had an alternative.  Only the chairs and Andy admitted to reading it.  
    There was therefore no point in an extended discussion, but Andy noted that 
    there was a higher-level question: what should be in an SNMP agent.  There 
    are roles for an SNMP agent, and roles for the management station.  In his 
    mind, there are too many features, and a lot in this draft is out of 
    scope.  The draft ideas are basically two years old, and there are no 
    other implementations. If no one is going to implement this (or even read 
    it) then there must be some action at the WG-level...  Emile noted that an 
    operator would not like to get flooded by "unsolicited" measurement 
    results; filters are required close to the controller of the 
    measurement system, not in the back office.
    The feedback from Henk's customers is that the MIB should be simple, 
    light-weight, and give users the freedom to do their own aggregation and 
    statistics computation.  The draft seems to be too France Telecom 
    specific.  Emile noted that this desire was in contradiction with a 
    previous concern that the buffer for all results might be too big and 
    therefore some aggregation might be required.  There is a 
    contradiction between a desire to report only raw data and a need for 
    aggregation.  Henk wondered if perhaps the draft could be limited to just 
    the two tables it needs to provide raw data and remove everything else?  
    Emile noted that the report tables are optional. Bob Cole wondered if the 
    compliance section was clear that only two tables were required.  Al said 
    that he would read a document limited to core functionality.  Andy felt the 
    document itself was too complex?  Does it want to specify a MIB for a 
    single aggregation box or 1000 agents doing measurements?  Matt said that he 
    would like to see comments on this issue on the mailing list.
    5. OWAMP
       -- Stanislav Shalunov
    Finally Stanislav Shalunov presented the updates to OWAMP.  The 
    approval of the requirements document was announced to the group. 
    Updates to the protocol document are based on a sample open source 
    implementation, and include test vectors for the random number 
    generator used in the generation of packet sending schedules, so that the 
    receiver knows accurately when the sender was supposed to send packets 
    (which allows losses to be detected at the receiver without having to 
    correlate sent packets with the sender).
    Matt asked how close Stanislav felt this document was to be forwarded to the 
    IESG.  One big issue is that the security has not been reviewed by 
    experts. That needs to be done.  Matt: What about a comparison with IPMP? 
    Stanislav said that there are two differences: OWAMP includes control, and 
    OWAMP tries to be "sneaky" and be not recognizable by routers.  Matt noted 
    that there was an ongoing discussion on IPMP on the IRTF/IMRG mailing list if 
    any working group members were interested in following IPMP.
    6. Milestones and Futures
       -- The Chairs
    Matt Zekauskas closed the meeting noting the milestones that have been met, 
    those that are in process, and those that will be dropped if no one steps up 
    to work on them.  (see slides) These were largely a re-iteration of the 
    conclusions from the Vienna meeting; they need to be run through the 
    mailing list. Matt and Henk will send email to the list to solicit input.


    Packet Reordering Metric for IPPM
    Reorder Density Function A Metric for Packet Reordering
    IPPM REPORTING MIB & ISP measurement infrastructure
    A One-way Active Measurement Protocol Requirements
    IETF IPPM WG Status and Futures
    IPPM REPORTING MIB in proxy mode