2.7.4 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 59th IETF Meeting in Seoul, Korea. It may now be out-of-date.

Last Modified: 2004-02-11

Chair(s):
Henk Uijterwaal <henk@ripe.net>
Matthew Zekauskas <matt@internet2.edu>
Transport Area Director(s):
Allison Mankin <mankin@psg.com>
Jon Peterson <jon.peterson@neustar.biz>
Transport Area Advisor:
Allison Mankin <mankin@psg.com>
Technical Advisor(s):
Andy Bierman <abierman@cisco.com>
Mailing Lists:
General Discussion: ippm@ietf.org
To Subscribe: https://www1.ietf.org/mailman/listinfo/ippm
In Body: subscribe
Archive: http://www.ietf.org/mail-archive/working-groups/ippm/current/maillist.html
Description of Working Group:
Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:

- connectivity

- one-way delay and loss

- round-trip delay and loss

- delay variation

- loss patterns

- packet reordering

- bulk transport capacity

- link bandwidth capacity

This is the cumulative set, including the metricsalready completed and published.

The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.

The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.

The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:
Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
Done  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
Done  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
Done  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
Feb 02  Create initial draft on the definitions of link bandwidth capacity.
Done  First draft for AS on one-way delay and loss.
Mar 02  Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS).
Done  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
Done  Create initial draft on a MIB for reporting IPPM metrics.
Apr 02  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
Apr 02  Create initial draft on comparing ITU and IETF IP performance metrics.
Done  Create initial draft on a packet reordering metric.
Jun 02  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
Jul 02  Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard.
Oct 02  Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay).
Oct 02  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
Dec 02  Submit AS for one-way delay and loss to the IESG for PS.
Dec 02  Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC.
Feb 03  Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard.
Mar 03  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
Mar 03  Discuss rechartering or ending working group.
Internet-Drafts:
  • - draft-ietf-ippm-owdp-07.txt
  • - draft-ietf-ippm-owdp-reqs-06.txt
  • - draft-ietf-ippm-metrics-registry-05.txt
  • - draft-ietf-ippm-reporting-mib-05.txt
  • - draft-ietf-ippm-reordering-05.txt
  • Request For Comments:
    RFCStatusTitle
    RFC2330 I Framework for IP Performance Metrics
    RFC2678 E IPPM Metrics for Measuring Connectivity
    RFC2679 PS A One-way Delay Metric for IPPM
    RFC2680 PS A One-way Packet Loss Metric for IPPM
    RFC2681 PS A Round-trip Delay Metric for IPPM
    RFC3148 I A Framework for Defining Empirical Bulk Transfer Capacity Metrics
    RFC3357 I One-way Loss Pattern Sample Metrics
    RFC3393 PS IP Packet Delay Variation Metric for IPPM
    RFC3432 PS Network performance measurement for periodic streams

    Current Meeting Report

    IETF IP Performance Metrics WG (ippm)
    Tuesday, March 2, 2004 at 15:45 to 16:45
    
    ========================================
    
    The meeting was moderated by the working group chairs, Matt Zekauskas and 
    Henk Uijterwaal.  Rick Whitner and Matt Zekauskas took notes, which were 
    edited into these minutes by the chairs.
    
    AGENDA
    1. Agenda Bashing
    2. Milestone Updates
    3. One-slide review of reordering metrics (no authors could attend)
    4. IPPM-MIB
    5. "Needs of IPPM metrics for non-e2e measures"
    
    
    1. Agenda Bashing
       -- The Chairs
    
       There were no updates suggested.
    
    
    
    2. Milestone updates
       -- Henk Uijterwaal for the Chairs
    
    Henk reviewed the milestones that are still open, and what the chairs feel 
    should be done about them.
    
    We found one person who volunteered to write the bandwidth capacity 
    definitions document, but he hasn't yet confirmed by email.
    
    To be deleted, unless someone steps up to do the work: 
      * Sensitivity analysis of one-way delay and loss parameters
      * Comparison of ITU and IETF IP performance metrics
      * Applicability statement for one-way delay and loss
      * Bulk Transfer Capacity metric, CAP
    
    The OWAMP protocol document is mature, has one implementation, and 
    reflects the current requirements document.  Stanislav Shalunov feels it is 
    ready, except that it needs more people to review, and it also needs a 
    thorough security review.  Looking for WG last call around July 2004.
    
    The packet reordering draft has undergone another revision (more later). It 
    needs some clarifications, but has all major comments included at this 
    point.  One issue that hasn't been settled, however: what to do (if 
    anything) with the reordering density draft.  If there are no major 
    shifts, we will be looking for a working group last call around 
    IETF-60.
    
    Finally, there is the issue of advancing metrics.  We are still stalled on a 
    document describing metric progression; in addition the NEWTRK working 
    group is looking at revising the standards track.  The one thing that 
    seems would be useful no matter what the outcome are 
    implementation reports.  We will be placing a call to the list that asks for 
    implementation reports. What we believe would be useful are:
       * name of implementation
       * the organization (if any) creating the implementation
       * origin of code
       * the platform the code runs on
       * a contact
       * the metrics and features implemented
       * metrics and features not implemented, and why
       * tests done on the implementation, such as comparison with another 
    implementation; verification of results with passive measurements; error 
    estimation, clock source.
    
    
    
    3. One-slide review of the reordering draft
       --Henk Uijterwaal, slide from Al Morton
    
    The biggest changes are reporting fragmentation and including examples of 
    'reordering free runs' from Jon Bennett (see slide). The given 
    pseudo-code has become complex, and Al is thinking about designating it 
    informative-only.  Al would really like people to read the draft and 
    comment on the mailing list.
    
    Matt Z asked if anyone had any additional comments on including 
    fragmentation in the draft, instead of our traditional focus on 
    complete packets only.  The fragmentation was added to allow the metric to be 
    used with router testing (so the same metric could be used within 
    BMWG-style benchmarking).  No one commented.
    
    
    4. IPPM reporting MIB
       --Emile Stephan
    
    Emile reported on the changes since previous version, principally 
    simplification by deleting a number of groups (see slides); 
    thresholds added in AggrMeasureTabls; the textual convention 
    IppmOwnerIndex added to clear up owner namespace; GMTTimestamp reverted 
    back to GMT format
    
    Emile then gave some usage scenarios, giving a slide each on the data 
    model, the proxy architecture, how metrics are aggregated, and how 
    history size is limited.
    
    Emile asked for feedback on history size mechanism.  Matt Z. commented that 
    his understanding is that people don't want this kind of filtering, they 
    want that to be part of the management application; when a threshold is 
    specified, adds complexity; Emile replied that he only provides place for 
    the threshold, not the actual value.  It's not mandatory, and provides a 
    balance between memory size and capacity.
    
    Emile then went on to present the three open issues he sees with the MIB.
    
    Open issue #1 - memory optimization; SNMP objects use much more memory than 
    the value itself (like 10x).  Polling shortlived entries requires too much 
    bandwidth and makes the table unstable.  How to limit the size of the 
    history, how to reduce the poling freq?  Proposal: optionally store 
    results in file when buffer of a measure wraps and give application a URL of 
    the file.
    
    Open issue #2 - NetMeasureTable access; concern over the access 
    attributes of the table.  One opinion is that the read-only 
    characteristics of table violates the spirit of SNMP; should be 
    read-create.  Emile would like some opinions.
    
    Matt asked if anyone in the room had read the document other than the 
    Chairs.  Allison Mankin was in the room and had read the document, but 
    otherwise noone volunteered that they had read the document. Thus, there was 
    nobody in the meeting who had read the document and could provide 
    feedback.
    
    Last issue is to specify the history buffers mechanism management, then 
    Emile feels it is ready for first (last) call.  Also Emile recommends 
    implementing in SNMPv3.  Without v3, management access won't be secure 
    enough.
    
    
    5. Needs of IPPM metrics for none2e measures 
       - Emile Stephan
    
    The idea is that we should want precise one-way delay using the same 
    methodologies across groups doing end to end and intermediate 
    (possibly passive) delay measurements.  Want aggregated one-way delay for 
    both passive and active measurement.
    
    Passive measurement can be used to decompose end to end delay; the psamp wg 
    is standardizing hash function for one-way delay measurement  between two 
    intermediary devices, and we need a common definition  or this metric, and 
    also what it means to concatenate segments measured passively.  
    Proposal: define an IPPM spatial one-way delay.
    
    With regard to active measurement we need to be able to concatenate 
    measurement results, and provide standard reporting.  Proposal: 
    standardize an IPPM aggregated one-way delay.  We need to span 
    measurement systems, currently no interoperability.
    
    Matt Mathis asked Emile about his motivation: is it oriented towards 
    representation for easy interoperability, or do the semantics need to be 
    fixed to make this scheme work?  There are two problems.  Different 
    implementations of one way delay don't constrain measurement 
    properties in ways that are comparable, for example by using different bins 
    to report statistics.  Secondly, if you don't represent the metrics in the 
    same way it is cumbersome to do analysis.
    
    Emile responded that he thought about this and produced an individual 
    draft on spatial metrics; and looked at instantaneous 
    concatenation with concatenation based on the metrics performed.  if you 
    don't know the methodology, you can't concatenate.
    
    Matt said that it bears on the first question.  He was afraid of issues of 
    representation, which is cumbersome to do, but doesn't change anything 
    we've already done.
    
    Emile said implementation issues are out of scope.
    
    Matt also pointed out that in the framework RFC we have a notion of an 
    "A-frame".  It has always been the vision of the group (the "holy 
    grail") of being able to concatenate metrics (including BTC). This 
    problem has proven to be much harder than anticipated initially.
    
    Emile noted that PSAMP will provide standards that essentially measure 
    spatial delay.
    
    Matt noted that delay was easy to concatenate.
    
    Email pointed out that these tools will be available, so we should 
    clearly distinguish what happens with these metrics.
    
    Matt noted that if we are getting into composition of metrics, as we 
    outlined in the framework draft, it's a huge amount of work, at least as 
    great as the work already done on the metrics themselves. It might be 
    useful to "stick one toe into the pool".  This is a large problem space, and 
    maybe we just don't want to go there.
    
    Emile replied that he believes we have to address the boundaries now, 
    i.e., what can be concatenated and what can't. With IPFIX and PSAMP we will 
    have tools, people will share and combine -- we should say how.
    
    Matt replied that there are other kinds of concatenation that would be 
    extremely valuable.  If we wanted to make a metric which was an 
    aggregate across a path across other provider, starting with a 
    collection of endpoints.  There would be a large number of 
    singletons, and large number of paths across provider.  In this example the 
    issue is how to define a set of endpoints that make sense to measure.  This 
    is different.
    
    
    Email said that in his draft, that would be based on composition of 
    metrics.
    
    Matt Mathis replied that this is another holy grail.  What is the 
    typical one-way delay across a provider.  For example, how well does 
    routing policy match geography.  this is still an open question.  The 
    point is that if you are going to think about concatenation, then you need to 
    consider other kinds of composition.  It might be warranted to do one, and 
    then decide not to do others now.  Again, this is a much bigger space than 
    the group has already tackled.
    
    Matt Z. said that the composition thing is a research topic; it's not good 
    enough to say you want to do the composition, we need to have an idea of how 
    to do the composition.  On the other hand, an applicability statement of the 
    one-way metric to passive measurement could be useful.
    
    No other comments.  Meeting closed.
     

    Slides

    Agenda
    IPPM REPORTING MIB Simplification
    Needs of IPPM metrics for non e2e measures