2.8.7 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 64th IETF Meeting in Vancouver, British Columbia Canada. It may now be out-of-date.

Last Modified: 2005-10-31


Henk Uijterwaal <henk@ripe.net>
Matthew Zekauskas <matt@internet2.edu>

Transport Area Director(s):

Allison Mankin <mankin@psg.com>
Jon Peterson <jon.peterson@neustar.biz>

Transport Area Advisor:

Allison Mankin <mankin@psg.com>

Technical Advisor(s):

Andy Bierman <ietf@andybierman.com>

Mailing Lists:

General Discussion: ippm@ietf.org
To Subscribe: https://www1.ietf.org/mailman/listinfo/ippm
In Body: subscribe
Archive: http://www.ietf.org/mail-archive/web/ippm/index.html

Description of Working Group:

Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied
to the quality, performance, and reliability of Internet data delivery
services. These metrics will be designed such that they can be
performed by network operators, end users, or independent testing
groups. It is important that the metrics not represent a value
judgment (i.e. define "good" and "bad"), but rather provide unbiased
quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as
NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and
procedures for accurately measuring and documenting these metrics.
The metrics are:

  - connectivity

  - one-way delay and loss

  - round-trip delay and loss

  - delay variation

  - loss patterns

  - packet reordering

  - bulk transport capacity

  - link bandwidth capacity

This is the cumulative set, including the metricsalready completed
and published.

The working group will closely review and then be guided by an
IESG document on how metrics advance along the standards track within
the IETF.  This document will also be relevant to the work of the
benchmarking working group (BMWG).  The first draft of this document
was discussed at IETF 51. Additionally, the WG will produce Proposed
Standard AS documents, comparable to applicability statements in RFC
2026, that will focus on procedures for measuring the individual
metrics and how these metrics characterize features that are important
to different service classes, such as bulk transport, periodic
or multimedia streams. It is specifically out of scope for this
group to actually characterize traffic, for example to characterize a
voice-over-IP stream. Each AS document will discuss the performance
characteristics that are pertinent to a specified service class;
clearly identify the set of metrics that aid in the description of
those characteristics; specify the methodologies required to collect
said metrics; and lastly, present the requirements for the common,
unambiguous reporting of testing results. Specific topics of these
AS documents must be approved by the Area Directors as charter

The WG will produce a protocol to enable communication among test
equipment that implements the one-way metrics. The intent is to create
a protocol that provides a base level of functionality that will allow
different manufacturer's equipment that implements the metrics
according to a standard to interoperate. A protocol requirements
document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM
metrics, such as one-way delay and loss, to facilitate the
communication of metrics to existing network management systems. Thus,
the group will create a MIB that contains predominantly read only
variables.  If, after the protocol requirements document is finished,
the group decides that it is appropriate to add variables that control
the underlying measurements that the metrics report, such a control
structure may be added as a separate document, subject to review by
the IESG.

The intent of the WG is to cooperate with other appropriate standards
bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote
consistent approaches and metrics. Within the IETF process, IPPM
metrics definitions will be subject to as rigorous a scrutiny for
usefulness, clarity, and accuracy as other protocol standards. The
IPPM WG will interact with other areas of IETF activity whose scope
intersect with the requirement of these specific metrics. These
include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:

Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
Done  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
Done  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
Done  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
Done  First draft for AS on one-way delay and loss.
Done  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
Done  Create initial draft on a MIB for reporting IPPM metrics.
Done  Create initial draft on a packet reordering metric.
Done  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
Done  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
Dec 2005  Collect implementation reports for RFCs 2678-2681
Apr 2006  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
Jun 2006  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
Jun 2006  Develop new charter text


  • draft-ietf-ippm-owdp-15.txt
  • draft-ietf-ippm-reordering-10.txt
  • draft-ietf-ippm-implement-01.txt
  • draft-ietf-ippm-bw-capacity-01.txt
  • draft-ietf-ippm-twamp-00.txt

    Request For Comments:

    RFC2330 I Framework for IP Performance Metrics
    RFC2678 E IPPM Metrics for Measuring Connectivity
    RFC2679 PS A One-way Delay Metric for IPPM
    RFC2680 PS A One-way Packet Loss Metric for IPPM
    RFC2681 PS A Round-trip Delay Metric for IPPM
    RFC3148 I A Framework for Defining Empirical Bulk Transfer Capacity Metrics
    RFC3357 I One-way Loss Pattern Sample Metrics
    RFC3393 PS IP Packet Delay Variation Metric for IPPM
    RFC3432 PS Network performance measurement for periodic streams
    RFC3763 I A One-way Active Measurement Protocol Requirements
    RFC4148 BCP IP Performance Metrics (IPPM) metrics registry

    Current Meeting Report

    IP Performance Metrics WG (ippm)
    Monday, November 7, 2005 - 09:00--11:30
    The meeting was chaired by Henk Uijterwaal and Matt Zekauskas.  Benoit Claise
    and Al Morton took notes, which were edited into these minutes by the
    1. Administrivia: Agenda, Status of Drafts and Milestones
    2. Temporal Aggregation (and Composition Framework)
    3. Spatial Composition Draft and an overall Framework Proposal
    4. Spatial and Multicast Metrics
    5. Jitter Definitions discussion
    6. Traceroute Metrics and Data Model
    7. TWAMP Draft
    8. Recent ITU Liaison statements
    Loki Jorgenson was unable to attend, thus he did not present his
    opinions on the ITU's IP Performance documents (specifically, Y.1541).
    1. Administrivia: Agenda, Status of Drafts and Milestones
       --Henk Uijterwaal for both chairs
    Henk opened the meeting with the usual working group information
    boilerplate.  He then talked about the state of the existing drafts,
    in particular, that OWAMP has been stuck in a security review (we're
    going to talk with Russ Housley; unfortunately, Stanislav Shalunov
    couldn't make the Vancouver meeting); and also we're going to need
    advice from the ADs as to what (if anything) to do with the
    implementation reports we've gathered for the extant IPPM metrics.
    Henk did another revision of the implementation report, taking comments
    into account.  There is work going on in the background on the capacity
    draft, but there is no report today.
    [After the meeting Russ and Stanislav had a productive email email
    exchange solving most of the issues; Stanislav is to work on the last two
    after the conference he is attending is over.  Henk also sent out
    email on the implementation reports after the meeting.]
    Henk then discussed the state of the reordering drafts.  There are
    two drafts, a group draft and a individual submission from a group
    at Colorado State University.  The latter has been presented but
    hasn't drawn much interest from the working group.  The group draft
    has been reviewed seriously by a number of people,  and it has been
    stable for about a year.
    He also discussed the claim from the CSU group that the byte-offset
    definition in the WG draft is derived from reorder-buffer-density.
    Henk noted that both he and Matt had reviewed this claim and found
    that the editor was reflecting consensus from the working group.  It
    appears that the two efforts appeared to be working in parallel
    towards the same idea from different directions.  Also, the ideas had
    been publicly discussed at IPPM meetings and elsewhere, and that
    therefore it did not appear that byte-offset was directly derived from
    reorder-buffer-density (and in any case it is hard to analyze years
    after the fact).
    Henk proposed that a stable reference to the Colorado work be added
    to the draft discussing reordering approaches, that we ensure there
    is an acknowledgement to the individuals that participated in meetings
    and email discussions (we believe this exists already), and that
    we will go to WGLC on the group's draft.  Unless the reorder buffer
    density draft gets any outside support we will not pursue making
    the Colorado draft a working group document.  This decision will be
    posted to the mailing list shortly.  Those present in the room agreed
    with this approach.
    The dates on the milestones are being revised.  Authors should give
    Henk good estimates for their milestones.
    2. Temporal Aggregation (and Composition Framework)
        --Steven Van den Berghe
    Steven Van den Berghe (who has been doing some work for GEANT2's JRA1 group)
    presented a short draft on thoughts on a unified framework for thinking
    about temporal and spatial composition of IPPM metrics, and then some
    examples of temporal aggregation, and concerns that need to be addressed.
    One example of temporal aggregation is taking a measurement every five seconds,
    and then creating a meaningful summary of an hour. [See slides.]
    Kaynam Hedayat asked if Steven had thought about jitter.  Steven says
    that they had, and have some ideas, but haven't worked them out.
    Another participant asked about how precedence, or treatment of packets
    would fit in.  The underlying measurements take that into account,
    as part of Type-P.  You can say something about one forwarding behavior
    directly; aggregating packets with different forwarding behaviors would
    take more work, and may not be possible.
    Emile Stephan asked about error propagation, and what error should
    be reported.  Steven said that error propagation needs to be worked
    out as well.
    Matt asked the group if they thought this would be useful; there
    were a few (4-5) nodding heads, including some new participants.
    Matt stated that personally, he felt the useful work was the work
    that wasn't completed -- error propagation, jitter propagation,
    considering values that are not averages.
    3. Spatial Composition Draft and an overall Framework Proposal
       -- Al Morton
    Al Morton presented spatial composition, and a more formal proposal for
    a framework for composition (that included discussion with Steven).
    This draft's example just talks about composition of finite one-way delays.
    The framework is such that you could also include the multimetrics draft
    as a different approach within the framework.  [See slides.]  Al would
    like comment on the overall framework, and then the individual drafts.
    Roman Krzanowski stated that from his perspective the composition and
    decomposition of metrics has been on his mind for a while, and that
    providers have no guidance, and he thought we should work on drafts
    like this to get some consensus.
    4. Spatial and Multicast Metrics
       -- Emile Stephan
    Emile Stephan then gave an update on the multimetrics draft.  This is looking
    at measurements that are taken on the same stream at multiple points.
    The latest version adds a new metric that more clearly defines a two point
    one way delay -- one where you passively measure (possibly a synthetic stream)
    at two points.  He also discussed where this draft fit with the other two,
    and in a general composition framework.  He was looking for feedback as
    to whether a purely passive metric was the right approach, compared to
    something like an applicability statement for the current one-way delay
    metric. [See slides.]
    The result of the discussion on these three drafts (Van den Berghe, Morton,
    Stefan) is that there will be four drafts: A framework draft with common
    text, and the three discussing temporal, spatial, and multi-point composition.
    5. Jitter Definitions discussion
       --Roman Krzanowski
    Roman Krzanowski then presented a problem he saw working on an MIT
    study of metrics used by multiple operators, both in the US and
    abroad.  [See slides.]  The basic issue was with jitter, and that US
    operators tend to use inter-packet delay variation, where the European
    operators tend to use delay variation from a single reference point.
    The issue is not one of definition -- the IPPM IPDV metric can capture
    either, but one of "best practice" or recommendations.  He is going to
    see if there is enough interest (among US and European operators)
    generate an applicability statement -- especially with respect to
    sharing measurement results.  
    Matt said that we'd like to see some interest, and if so set a
    deadline to see progress or declare failure since previous efforts to
    get providers to agree on sharing results had not been fruitful within
    the IETF.  Roman said he would summarize the discussion, send it to
    the group, and a few select people working for service providers, and
    see if we can generate some enthusiasm.
    6. Traceroute Metrics and Data Model
       --Martin Stiemerling
    Martin Stiemerling presented the individual traceroute metric and data model
    draft.  They have been talking to folks in the GGF Network Measurement
    Working Group.  This draft shows a complete traceroute schema in XML,
    and an alternative one based on the GGF work, and a preliminary sketch
    of how to combine approaches.  Issues to be resolved include (1) seeing
    if the GGF intends to create a standard (2) see if a merged approach with
    the GGF structure and IETF information model makes sense (3) if a merged
    effort succeeds, should the work be published in the IETF, the GGF, or
    both?   There is a naming issue - the GGF has it's own way of naming things,
    this draft tries to use existing DISMAN names; a common namespace is desirable.
    The authors were going to compare the GGF document series with the IETF one
    to see if the GGF is even a possible alternative, but the authors strongly
    prefer something that uses IETF naming.  
    There were about 10 bobbing heads when asked if people thought this
    was important; it seems clear that the work would be welcome, the
    question is if anything should be done within IPPM.
    7. TWAMP Draft  [no slides]
       --Keynam Heyadat
    Next, Keynam Heyadat presented the TWAMP draft, and current status.
    There have been some changes based on implementation experience, mainly
    relating to time stamps, and some text cleanup.   The timestamps and
    sequence numbers are now in fixed locations so that a hardware implementation
    is possible.  There was also a correction to the draft; TWAMP packets
    fit in ATM cells only in unauthenticated mode.
    Henk wanted to make sure the authors are aware of the OWAMP security
    considerations so that TWAMP does not run into the same issues (TWAMP
    is based on OWAMP).
    Keynam stated that he knows of four implementations that are underway;
    three of which are substantially done.  He feels that after there is
    some interoperability experience with them, the draft may be close to
    being done.  He thought doing a report on implementation experience
    would be a good idea.
    8. Recent ITU Liaison statements
       --Al Morton
    Finally Al pointed out the recent Liaison Statements from the ITU to the
    group.  He showed how to find them, and gave a synopsis of what they were.
    Start at the IETF home page, select liaison activities, and then
    liaison statements.
    The two of interest were one to NSIS, which includes Y1541 (so we can
    get it "for free") and the most recent one which talks about a potential
    project, G.chirp, which uses packet chirps to characterize available bandwidth
    (and eventually the download time for web pages).  It's an experimental
    thing, but a stable draft is expected about January of 2007.  People should
    look at them, and let the group know if they have any comments.  SG12 doesn't
    meet until June of next year, so there is plenty of time to comment.
    The NSIS statement: 'Communication of Recommendation Y.1541, "Network
    Performance Objectives for IP-based Services"', September 2005.
    From ITU-T Study Group 12, Question 17 to NSIS.
    "Question 17/12 has been following the development of the Next Steps In
     Signalling (NSIS) framework, requirements and protocols, and notes
     with particular interest the work on QSpec Templates and QoS
     Models. We understand your desire to reference Recommendation Y.1541
     (published 05/02), and so communicate it to you such that a copy can
     be referenced and accessed by the IETF community in a persistent way."
    NSIS is working on signalling protocols that could communicate
    requests for QOS, beyond IP stuff in Y.1541: bandwidth, priority, etc.
    This liaison is sent to them, so that they would have a persistent
    URL in order to access the recommendation easily for all members
    of the IETF.
    There are two copies available in the liaison.  There is a version
    that shows some revisions recently added; classes with additional
    performance objectives, for high bandwidth and high sensitivity to
    loss applications, such as "big TCPs", and emulation of TDM services
    over IP networks.  A third kind of thing that these classes support is
    transport of IPTV.
    The G.chirp statement: "Liaison statement on the Development of New
    Recommendation on measuring file download and session time", November 2005.
    From ITU-T SG12, Questions 13 and 17, to IPPM.
    "Q.13/12 would like to inform IETF IPPM Working Group that we have
     started developing a new Recommendation G.chirp, which provides a
     means for measuring file download and session time in a simplified
     way. The technical content can be found in the attached document
     (COM12-C11). We expect to produce a stable draft in January 2007.  It
     is very much appreciated if your group gives any suggestions or
     comments on this topic."
    G.chirp is active sending sequence which intends to characterize available
    bandwidth, and eventually file download time.  It is an 
    experimental thing that was proposed by TNO of The Netherlands.
    The method is described in the contribution.  It sends packets in
    closer spacing using an algorithm for spacing, and attempts to assess
    the download times from that.
    A formal reply could be prepared back to SG 12 and the questions.
    If folks would like to try out, TNO probably has some intellectual
    property covering it.  However, trying it out experimentally and
    publishing results would probably be reasonable.
    The most powerful way to communicate back to the ITU is via additional
    liaison statements.  If the group looks and thinks it is great idea,
    or bad idea, it is best if a response comes back formally.
    However, given that certain people participate in both the IETF and
    ITU, any response is useful, even if it is informal.
    The meeting was then closed.


    Agenda, Draft Status, Milestones
    Temporal Aggregation (& Composition Framework)
    Spatial Composition Draft and an overall Framework Proposal
    Spatial and Multicast Metrics
    Jitter Definitions Discussion
    Traceroute Metrics and Data Model