2.7.4 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 47th IETF Meeting in Adelaide, Australia. It may now be out-of-date. Last Modified: 09-Mar-00


Will Leland <wel@research.telcordia.com>
Matthew Zekauskas <matt@advanced.org>

Transport Area Director(s):

Scott Bradner <sob@harvard.edu>
Vern Paxson <vern@aciri.org>

Transport Area Advisor:

Vern Paxson <vern@aciri.org>

Mailing Lists:

General Discussion:ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: http://www.advanced.org/IPPM/archive/

Description of Working Group:

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgement (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will define specific metrics, cultivate technology for the accurate measurement and documentation of these metrics, and promote the sharing of effective tools and procedures for measuring these metrics. It will also offer a forum for sharing information about the implementation and application of these metrics, but actual implementations and applications are understood to be beyond the scope of this working group.

Goals and Milestones:

Oct 97


Submit drafts of standard metrics for connectivity and treno-bulk-throughput.

Nov 97


Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.

Feb 98


Submit documents on delay and loss to IESG for publication as Informational RFCs.

Apr 98


Submit a document on connectivity to IESG for publication as an Informational RFC.

Sep 98


Submit a document on bulk-throughput to IESG for publication as an Informational RFC.


Request For Comments:







Framework for IP Performance Metrics



IPPM Metrics for Measuring Connectivity



A One-way Delay Metric for IPPM



A One-way Packet Loss Metric for IPPM



A Round-trip Delay Metric for IPPM

Current Meeting Report

IP Performance Metrics WG (ippm)
Monday, 27 March 2000, at 19:30 - 22:00

The meeting was chaired by the working group chairs, Will Leland and Matt Zekauskas, and was reported by Paul Love.

1. Status
2. Loss patterns
3. IPDV experience
4. Periodic streams
5. Bulk transport capacity
6. Futures

IETF home page: http://www.ietf.org/html.charters/ippm-charter.html
IPPM home page: http://www.advanced.org/IPPM/

1. Status, Will Leland (Telcordia)

IPPM opened with a status report. Since the meeting in Oslo there were minor changes to loss patterns and IPDV, a new draft on delay for periodic streams, and the expiration of the Treno bulk transport capacity metric.

2. Loss Patterns, Rayadurgam Ravikanth (Nokia)

Ravikanth noted that minor changes have been made since Oslo (a draft was released in October addressing the Oslo comments), and that no further discussion had occurred on the mailing list.
If there are no further changes, one more editorial pass will occur just after IETF, and then the authors wanted it to be submitted for consideration as an Experimental RFC.

A question from the floor asked if the draft had been implemented, with any experience using the metric. Other than some internal use by the authors, there's been no published work using it. There was a presentation of the ideas by example at the LA IETF - see those minutes.

A question from the floor asked what it means for a loss to be "noticeable". The mathematical definition seemed clear, but the motivation was not. The explanation was that a stretch of time might be a problem if repeated small numbers of losses, even though no small group would be a problem. The number of losses, the number of intervening successful packets are parameters and not specified in the draft. It depends on the application what would be considered noticeable. For details, see Rajeev's thesis referenced in the draft. Another audience member asked if there could be some guidance on how to use the metric, perhaps pulling the gist of the thesis description into the draft. Ravikanth thought this would be a good idea.

With no other comments, the authors will add another example, make final editorial changes, and will ship a draft in about two weeks following IETF. We will then "last call" the document as experimental.

3. IPDV, Matt Zekauskas (Advanced Network & Services), presenting

See slides.

Matt first gave a summary of the changes to the document. Then he presented Phil Chimento's slides on the IPDV draft.

Phil had experimental results to show using the metric applied to Surveyor data. There was solid consensus that the I-D has made the right decision to condition on received packets, that is, reporting losses but not including them in the IPDV values.

Phil noted that the histograms were first cuts, with the histogram buckets evenly distributed over the entire min-max range. Because there are extended tails, Matt Mathis suggested that the bucket size should be allowed to increase as one moves away from the center of the peak.

Another observation made on the sample data was that the min and max IPDV value often occurred close together. This is consistent with spikes developing in queues along the path.

There was extensive discussion of whether IPDV's differential approach was better than deviation from an expected (or first) value. Dr. Guven Mercankosk was concerned about glitches in display of real-time media streams. He felt that the variations reported by the differential approach do not reflect the buffer size required. (The buffers would be too small.) In the views of the chairs, the rest of the audience felt that measurements represent a point in time and do not guarantee future variation. One example raised was a shift in route during a session. In this case, an arbitrarily large buffer might be required to guarantee perfect display. A histogram of the differential variation would show one outlier; a histogram of the "expected" approach would be bimodal. The requirement depends on the application. There was consensus that the differential approach was a sufficient initial approach. Experience with real applications using this metric will be required. The chairs encouraged Dr. Mercankosk to present his case to the mailing list so that the author (and the working group in general) could respond.

A suggestion was made to provide some information on usefulness of IPDV, and any obvious limitations in the document.

There was also discussion of the relationship between the delay variation measured by the two methods (differential versus expected). It was agreed that while they are related, you cannot directly derive one from the other without having the complete time series of variation measurements (and a known starting point).

At several points during the meeting, Dr. Mercankosk suggested that interpreting the metrics will sometimes require knowledge of the background traffic. The example given was configuring an application based on measurements taken at the start of a session. Someone mentioned that if you need a hard guarantee, then you need QoS. The relevance of background traffic to defining metrics was not well understood (since if more aspects of the network should be captured then additional metrics should be defined), and Dr. Mercankosk was encouraged to present the concern to the mailing list.

4. Periodic Streams, Chuck Powers (Motorola), presenting

Chuck Powers presented the new I-D on "periodic streams" for the authors, Vilho Raisanen and Glenn Grotefeld. The chairs suggested that the key questions were if a periodic streams metric was needed, and if the proposed metric was a good approach. The meeting easily agreed that periodic traffic is increasingly important and can exhibit delay behavior not seen by Poisson samples. Voice over IP, streaming audio and streaming video have regular traffic.

This document is parallel to the one-way delay document, but for periodic flows. The mapping to the Framework needs to be improved. The mailing list notion of a "sample of samples" was discussed. The "sample of samples" refers to multiple streams that are sent over time to better characterize what an arbitrary periodic stream might have encountered. A Poisson schedule of the periodic streams was agreed to be a good approach.

Dr. Mercankosk thought that the sample of samples would indeed provide more information about the underlying network, but again called for better characterization of background traffic.

In response to a question by Will Leland, Matt Mathis and Vern Paxson noted that periodic streams see distinct loss behavior in addition to delay behavior. For example, periodic streams may synchronize with slot times for cable modem access. Another example is what is known as a "clock slip" in telephony, which causes a point-to-point link to drop bits periodically. This led to a discussion of periodic versions of all the existing metrics. There was some consensus that periodic streams needed to be considered for all IPPM metrics, but no volunteers to address this larger question.

5. Bulk Transport Capacity, Will Leland & Matt Zekauskas

See slides.

The Co-Chairs then described the progress, or lack thereof, with a bulk transport capacity metric. The BTC framework was updated in October, but the Treno draft has now expired.

Mark Allman now has concrete plans to produce a draft of 'cap' as a BTC metric by the end of the summer. 'cap' is a user-level TCP emulation tool, originally built to easily experiment with changing TCP features. Mark is planning to perform experiments that tests the relationship of the values cap produces to the implementation of TCP in FreeBSD.

Matt Mathis described what blocked Treno progress: Treno does not tolerate any packet reordering, unlike most TCPs. His concern was that a metric that allowed any packet reordering might create a situation where vendors designed for that reordering. ISPs that use such equipment might pass TCP throughput tests. However, reordering is cumulative. A path that included a concatenation of ISPs, each of which reordered a few packets, would produce a stream of reordered packets that TCP would not tolerate.

There is a high demand for measuring TCP throughput and many poorly designed ad hoc approaches are used today. One is web browser download time, which conflates DNS lookup time, server load and network throughput. It is also at the mercy of the end-host TCP stacks. Another approach is to just bless one stack, perhaps the Windows 2000 implementation. Both approaches are vulnerable to future software changes.

There was wide consensus that a well-founded metric, even if imperfect, is needed quickly. The chairs asked if anyone had an alternative approach they would advocate, but none was suggested.

6. WG Future, Will Leland & Matt Zekauskas

See slides.

The meeting closed with a wide-ranging discussion of future directions for the group. To date, the focus has been on defining a small set of unambiguous useful metrics that focus on transport, not applications.

Will noted that the original reason behind IPPM was to develop accurate measurements for service level agreements and specifications. Others noted that these have a "net to net" rather than "host to host" focus.

A related concern was the perceived bias of IPPM metrics toward active measurements. The applicability of the existing delay and loss metrics to passive observation of actual traffic has not been resolved.

Another issue raised was metrics for the emerging distinct classes of service in IP. Ruediger Geib mentioned the idea of back-to-back active measurements where each has a different class of service, in order to quantify the difference made by that class of service between two points.

Other potential directions raised during the discussion were comparison of metrics, metrics for network availability, MIBs for these metrics, and some way of defining meaningful "weather maps", rather than the ad-hoc maps used today.

The consensus was that the working group needs to continue to actively develop further metrics. The working group provides a forum to debate new metrics; without it ad-hoc metrics evolve without carefulscrutiny. Will closed the meeting by noting that there were several good directions mentioned today. We will be looking for volunteers to define them; if there are no volunteers the WG would go dormant.


Bulk Transport Capacity
Group Future
Initial Analysis of some IPDV
measurements in Europe
IPDV Changes
Network Performance Measurement for
Periodic Streams