Current Meeting Report
Slides
Jabber Logs


2.8.4 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 55th IETF Meeting in Altanta, Georgia USA. It may now be out-of-date.

Last Modifield: 05/06/2002

Chair(s):
Merike Kaeo <kaeo@merike.com>
Matthew Zekauskas <matt@advanced.org>
Transport Area Director(s):
Scott Bradner <sob@harvard.edu>
A. Mankin <mankin@isi.edu>
Transport Area Advisor:
A. Mankin <mankin@isi.edu>
Technical Advisor(s):
Andy Bierman <abierman@cisco.com>
Mailing Lists:
General Discussion: ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: http://www.advanced.org/IPPM/archive/
Description of Working Group:
Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:

- connectivity

- one-way delay and loss

- round-trip delay and loss

- delay variation

- loss patterns

- packet reordering

- bulk transport capacity

- link bandwidth capacity

This is the cumulative set, including the metricsalready completed and published.

The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.

The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.

The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:
Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
FEB 02  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
FEB 02  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
FEB 02  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
FEB 02  Create initial draft on the definitions of link bandwidth capacity.
FEB 02  First draft for AS on one-way delay and loss.
MAR 02  Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS).
MAR 02  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
MAR 02  Create initial draft on a MIB for reporting IPPM metrics.
APR 02  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
APR 02  Create initial draft on comparing ITU and IETF IP performance metrics.
APR 02  Create initial draft on a packet reordering metric.
JUN 02  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
JUL 02  Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard.
OCT 02  Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay).
OCT 02  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
DEC 02  Submit AS for one-way delay and loss to the IESG for PS.
DEC 02  Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC.
FEB 03  Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard.
MAR 03  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
MAR 03  Discuss rechartering or ending working group.
Internet-Drafts:
  • - draft-ietf-ippm-ipdv-09.txt
  • - draft-ietf-ippm-npmps-07.txt
  • - draft-ietf-ippm-owdp-04.txt
  • - draft-ietf-ippm-owdp-reqs-03.txt
  • - draft-ietf-ippm-metrics-registry-01.txt
  • - draft-ietf-ippm-reordering-00.txt
  • - draft-ietf-ippm-reporting-mib-00.txt
  • - draft-ietf-ippm-owmetric-as-00.txt
  • Request For Comments:
    RFCStatusTitle
    RFC2330 I Framework for IP Performance Metrics
    RFC2678 E IPPM Metrics for Measuring Connectivity
    RFC2680 PS A One-way Packet Loss Metric for IPPM
    RFC2681 PS A Round-trip Delay Metric for IPPM
    RFC2679 PS A One-way Delay Metric for IPPM
    RFC3148 I A Framework for Defining Empirical Bulk Transfer Capacity Metrics
    RFC3357 I One-way Loss Pattern Sample Metrics

    Current Meeting Report

    IP Performance Metrics WG (ippm)
    Tuesday, November 19 at 17:30 - 20:00
    ====================================
    
    The meeting was moderated by the working group chairs, Matt Zekauskas and 
    Merike Kaeo.  Dave Plonka and Al Morton took notes, which were edited into 
    these minutes by the chairs.
    
    
    AGENDA:
    1. Agenda bashing
    2. Announcements
    3. Discussion on Packet Reordering
    4. IPPM Reporting MIB and Metrics Registry discussion
    5. Discussion on One-way Active Measurements Protocol
    6. One-Way Metric Applicability Statement
    7. Potential work: "IP Measurement Protocol" (IPMP)
    8. Potential work: IPPM Spatial Metrics Measurement
    9. Available Bandwidth Measurement in IP Networks
    
    IETF home page:  
    http://www.ietf.org/html.charters/ippm-charter.html
    IPPM home page: http://www.advanced.org/IPPM/
    
    
    2. Announcements 
    
    Mark Allman introduced the IRTF Internet Measurement RG. In general IMRG is a 
    forum for measurement research that isn't yet ready for IETF.  It 
    intends to foster interaction amongst communities: IETF, operators, 
    researchers.  The IMRG is an *open* IRTF group, anyone can join the list: 
    http://imrg.grc.nasa.gov/imrg/ (linked from irtf.org).
    
    David Martin asked if IMRG looking at control aspects - what kinds of 
    measurement?  Mark responded that it's open; lots of kinds: passive, 
    active, lots of options to investigate.
    
    Al Morton presented on IPPM and ITU-T performance status.  Since Will 
    Leland's comparison in March, 1999, some terms have changed and 
    concepts and definitions have gotten closer.  One new definition is IP 
    severe loss block ratio, prompted by a desire to measure the effect of 
    re-routing.  The one activity where the two groups continue to differ is in 
    the area of specifying objectives for performance; the ITU "Numerical 
    performance objectives" are beyond IPPM's scope.
    
    Matt asked how Al participates in ITU-T meetings - in essence why he is 
    qualified to make such a report.  Al responded that he attends ITU-T 
    meetings, is the designated editor for recommendations in Study Group 13 
    where his company has interest (such as Y.1540 and Y.1541).  He also 
    contributes to the work of Study Group 12.
    
    Al has considered the possibility of an Informational RFC comparing ITU and 
    IPPM terminology but suggests that that is becoming less and less 
    necessary.
    
    An audience member (Hidega Tiku, from France Telecom) believed that a 
    broader mapping is possible than just to the Study Group 13 documents 
    (Y.1540 and Y.1541), she believes there is relevance to 30 ITU groups.  Al 
    believed that the closest parallel was SG 13, and would like to work with 
    her to understand.  Another audience member (Chuck Dvorak, vice chair of SG 
    12, the lead study group in all ITU-T QoS activities) confirmed that 
    IPPM's closest working counterpart is the SG 13 work that Al 
    described, other groups may have interest in IPPM, but do not define IP 
    metrics. Matt invited them to work together and bring conclusions to the 
    mailing list.
    
    
    3. Discussion on Packet Reordering -- Al Morton
    
      
    http://www.ietf.org/internet-drafts/dra
    ft-ietf-ippm-reordering-01.txt
    
    Al reported on status of current draft and presented some examples.  He 
    expressed that more input is desired as to whether the examples are 
    sufficient.  Major issues include integrating the definition of 
    n-reordering (clarification, description of how it fits in with 
    non-reversing sequence), specific advice concerning duplicates, adding a 
    metric of distance between reordering events, and unifying notation.
    
    Stanislav Shalunov pointed out that if you want to consider all 
    duplicate packets as not out of order, this is a reasonable 
    definition, but the implementation would require O(N) space to keep track of 
    all the packet numbers.  Thus, the metric is "fine but expensive".  Al 
    stated that we could just allow all duplicate packets to be declared out of 
    order and consider duplicates at the end of the test.
    
    Matt Z. asked for a clarification on Al's use of "clarification" with 
    respect to the n-reordering metric.  Was he looking to change the 
    definition?  Al responded that he wasn't quite sure yet, but that the goal 
    was to understand when particular metrics were applicable (and possibly add 
    some caveats, but didn't want to resort to that, yet);  
    n-reordering may better related to understanding reordering with espect to 
    TCP, while other sample metrics in section 5, such as Late Time, may be 
    better related to understanding reordering with respect to sizing 
    buffers for streaming media.  Jerry Perser (another author) echoed this, and a 
    general difficulty understanding the n-reordering results.
    
    Matt Mathis noted that n=3 for reordering isn't a constant for TCP's;  it's a 
    parameter to at least one stack.  He would also like a better 
    understanding how Web100's reordering statistic is related to 
    n-reordering, and how the metric is related to SACK blocks.
    
    
    4. IPPM Reporting MIB and Metrics Registry discussion -- Emile Stephan
    
      
    http://www.ietf.org/internet-drafts/dra
    ft-ietf-ippm-reporting-mib-01.txt
      
    http://www.ietf.org/internet-drafts/dra
    ft-ietf-ippm-metrics-registry-01.txt
    
    Jesse Jewitt discussed changes made from IPPM reporting MIB from version 00 
    to 01.
    
    Henk Uijterwaal started a discussion around the use of the word 
    "stratum" in the document - in particular, if it is the same as the NTP 
    stratum level.  It is apparently similar, but not exactly the same as 
    NTP's usage (the exact similarities and differences were not clear from the 
    discussion).  In addition, Henk worried that any notion of stratum might not 
    be sufficient, a notion of accuracy was also important.  Matt stated that 
    since there was confusion, the document should contain a 
    clarification.  Henk agreed to take the discussion to the mailing list.
    
    A complete list of enumerated open issues is contained in the slides.  The 
    authors are working on a prototype.
    
    Matt asked if there was anything learned from implementing the 
    prototype.  Jesse responded yes; Emile would provide input before or at the 
    next meeting.
    
    Emile then gave update on registry.  Emile is looking for some 
    procedural guidance to obtaining a MIB root value (how one is obtained from 
    IANA).  There was a discussion about MIB OID lengths; the length would vary 
    depending on where it was rooted.  Emile was wanted to keep the lengths 
    reasonable for comparison.
    
    Emile is also looking for guidance on suggesting procedures to specify how 
    new draft metrics add a section to create a new registry entry.
    
    Finally, Bob Cole noted that there are four "types" of metrics 
    mentioned in the registry: singleton, sample, statistical, and 
    "ambiguous statistical" like percentile.  He wondered if it was 
    appropriate to have entries for statistics that required parameters.  RMON 
    points to just singleton metrics.  He felt that the 'ambiguous 
    statistical' metrics that require parameters might not be 
    appropriate for the registry, or that maybe the registry should be split to 
    avoid an explosion.  It was noted that all the metrics require 
    parameters;  Bob would take the discussion to the mailing list to talk 
    about details.
    
    
    5. Discussion on One-Way Active Measurements Protocol -- Stanislav 
    Shalunov
    
      
    http://www.ietf.org/internet-drafts/dra
    ft-ietf-ippm-owdp-05.txt
    
    Stanislav Shalunov gave update on OWAMP.  Gave list of updates from 
    version 04 to 05 that were based on implementation experience.  Matt asked 
    for a clarification on the units of accuracy; it is a scaled power of two, 
    with a huge range but not much resolution.  Al Morton noted one spot where 
    the periodic sampling now included in the draft had not been added.
    
    The two significant discussion points were the error in timestamps when 
    packets are lost and reported errors for the start session command.  The 
    protocol needs a way to report the error in the sending timestamps when the 
    test packets are lost (or at least account for the error).  It 
    currently reports a time sent that isn't derived from explicit packet 
    data, and the send error isn't specified.  Al suggested setting the send 
    time error to zero as a distinguished value; Stanislav agreed it should be 
    set to something.  Second, the reason a session start command is 
    rejected should be stated (or enumerated).  There will be at least a 
    number that can be correlated to a specific reason.
    
    
    6. One-Way Metric Applicability Statement -- Henk Uijterwaal
    
      
    http://www.ietf.org/internet-drafts/dra
    ft-ietf-ippm-owmetric-as-01.txt
    
    Henk gave update on the applicability statement status.  There hasn't been a 
    lot of input, and we really need input from operators.  Merike queried the 
    room -- about half the people in the room were operators.  However, few 
    people had read draft. Folks were asked to read document and send 
    comments to mailing list.  Merike mentioned that Henk gave a pitch at the 
    last NANOG, and many there thought that this document would be useful.
    
    
    7. Potential work: another iteration on  "IP Measurement Protocol" (IPMP) -- 
    Jon Bennett
    
      
    http://www.ietf.org/internet-drafts/dra
    ft-bennett-ippm-ipmp-00.txt
    
    Jon Bennett gave presentation on his version of "IPMP" - an IP 
    measurement protocol, first presented last year by Tony McGregor.  He 
    talked about the motivation, and then gave an overview of changes that were 
    modifications to make the protocol more palatable to hardware 
    implementers (in his opinion) and add some functionality.  IPMP is 
    explicitly designed to be easily recognizable by intermediate systems, so 
    that they can add measurement data as the packet is routed.  After an 
    initial stumble, Jon is working with the authors of the McGregor draft to 
    create a merged document.
    
    David Martin wondered if this is essentially a DDoS engine, since it is not 
    required to have authentication.  Jon noted that there are a number of 
    security/authentication options and other considerations to prevent it from 
    being used as a DoS; in addition it doesn't make any new attacks 
    available.  It does allow you to redirect, but not hide the original 
    source IP address.
    
    Matt queried to the room to see if there was more interest than was shown on 
    the mailing list since Tony presented last year.  Only a couple people 
    admitted to reading the draft.  Merike asked if there were any vendors 
    willing to implement it, or if providers in the room were 
    interested.  Jon noted that Motorola was looking towards 
    implementing it, which could get it in ~200,000 cable modems.  Nevil 
    Brownlee noted that Tony McGregor does have a working 
    implementation (in end hosts), and it's deployed in the NLANR AMP 
    infrastructure (~100 hosts); Nevil also noted that the focus on 
    turnaround time and minimal IPMP processing was to have IPMP be a good 
    measure of the forwarding of "real" traffic along a forwarding path.
    
    A service provider indicated that it would be nice to have, but he was 
    concerned that IPMP is awfully complex to implement well; if a vendor 
    can't decrement TTL properly, how would they get this right?
    
    
    8. Potential work: IPPM spatial metrics measurement -- Emile Stephan
    
    
    http://www.ietf.org/internet-drafts/draf
    t-stephan-ippm-spatial-metrics-00.txt
    
    Emile presented a draft on "spatial metrics" -- obtaining 
    measurement data from intermediate points to help break apart path 
    measurement data to component path segments.  The advantage of this 
    document would be understanding and standardizing the accuracy and 
    uncertainty in the results.  Ruediger Geib noted that this assumes that you 
    understand packet routing a-priori, and something like 
    load-balanced induced changes in routing would cause problems. Emile 
    thought it could be found by combining passive measurements with the 
    active traffic.  Ruediger also wondered if the metric involved 
    collecting many individual results, perhaps it involved more of a 
    database and measurement system, and wasn't an individual result itself.
    
    
    9. Potential work: Available Bandwidth Measurement in IP Networks -- 
    Tricha Anjali
    
    
    http://www.ietf.org/internet-drafts/draf
    t-anjali-ippm-avail-band-measurement-00.txt
    
    Tricha Anjali presented work on available bandwidth measurements.  She 
    started with explanations of the various definitions, and then talked 
    about her technique, which involves the use of TCP, but not allowing it to 
    reach steady state.  Each measurement is a sample; the samples are 
    massaged to appear uniform, and then a linear filter is used to make 
    predictions.  Additional data (such as SNMP measurements) can be used to 
    provide additional accuracy.  The details are in the slides and draft.
    
    Mark Allman noted that a RFC defining all the terminology used in 
    bandwidth measurement would be useful, and he has attempted to help write 
    one; other authors would be welcome. Mark asked the chairs if the notion of 
    predicting the "future bandwidth" in WG scope?  Matt responded that his 
    feeling was that it was not, and it was more of a research topic.
    
    Mark Allman also questioned letting TCP go until a loss occurs, since the 
    size of queue skews the bandwidth metric because of the way TCP bursts.  
    Tricha would investigate.
    
    Matt asked if there are there simulations or actual measurements showing how 
    accurate this method is; Tricha responded not yet.
    
    Nevil Brownlee comment about method: if you assume there is nothing on the 
    link, and you use one TCP stream to test, I don't see how you can add one 
    more TCP stream to an unknown number, and understand the affect on the 
    other traffic.  Matt Mathis noted it was a Heisenberg problem: you don't 
    know how much the test traffic causes the other traffic to slow down.  This 
    seems to be a fundamental problem with this class of method.  Mark Allman 
    noted the measurement indeed could affect other traffic since it goes 
    until it induces loss.
    

    Slides

    Agenda
    IRTF Internet Measurement RG
    IPPM and ITU-T IP Performance Status