2.6.4 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 48th IETF Meeting in Pittsburgh, Pennsylvania. It may now be out-of-date. Last Modified: 17-Jul-00


Will Leland <wel@research.telcordia.com>
Matthew Zekauskas <matt@advanced.org>

Transport Area Director(s):

Scott Bradner <sob@harvard.edu>
Allison Mankin <mankin@east.isi.edu>

Transport Area Advisor:

Allison Mankin <mankin@east.isi.edu>

Mailing Lists:

General Discussion:ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: http://www.advanced.org/IPPM/archive/

Description of Working Group:

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgement (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will define specific metrics, cultivate technology for the accurate measurement and documentation of these metrics, and promote the sharing of effective tools and procedures for measuring these metrics. It will also offer a forum for sharing information about the implementation and application of these metrics, but actual implementations and applications are understood to be beyond the scope of this working group.

Goals and Milestones:

Oct 97


Submit drafts of standard metrics for connectivity and treno-bulk-throughput.

Nov 97


Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.

Feb 98


Submit documents on delay and loss to IESG for publication as Informational RFCs.

Apr 98


Submit a document on connectivity to IESG for publication as an Informational RFC.

Sep 98


Submit a document on bulk-throughput to IESG for publication as an Informational RFC.


Request For Comments:







Framework for IP Performance Metrics



IPPM Metrics for Measuring Connectivity



A One-way Delay Metric for IPPM



A One-way Packet Loss Metric for IPPM



A Round-trip Delay Metric for IPPM

Current Meeting Report

IP Performance Metrics WG (ippm)
Tuesday, 1 August 2000, 13:00-14:00

The meeting was chaired by the working group chairs, Will Leland and Matt Zekauskas, and was reported by Paul Love.

1. Status
2. ITU & QoS Issues
3. Network performance measurement for periodic streams
4. Bulk Transport Capacity experiments
5. Discussion on interest for a one-way delay protocol

IETF home page: http://www.ietf.org/html.charters/ippm-charter.html
IPPM home page: http://www.advanced.org/IPPM/

1. Status update & agenda bashing -- The Chairs

If you have any comments on the loss patterns or IP delay variation drafts, please take them to the list, as we would like to last-call them soon. The rest of the drafts are being discussed today.

A member asked if IPPM would produce metrics more relevant to voice-over-IP. One reason for concern is that ITU is pursuing VoIP monitoring, and there are holes in what existing standards cover. We noted that the proposed periodic streams metrics was inspired by VoIP and other streaming media issues. We asked him to comment on the periodic streams document, and to bring his concerns to the mailing list.

2. ITU & QoS issues - Lubomir Citkusev (Telcordia)

We were then given a brief report notifying us of work defining service classes in ITU 13/13. IPPM was invited to read and comment on the current drafts on "layered approaches" (documents (X.146R & Y.1541) under consideration in the ITU. (See the slides for details.) This work tries to be independent of "higher layers" (the applications).

3. Network performance measurement for periodic streams
- Glenn Grotefeld (Motorola)

The changes to the draft were presented; it has undergone two revisions since Adelaide. See the slides for details. Some highlights: A mapping of NPMPS to IPPM concepts was presented. The ITU concept of measurement points was used, which avoids requiring implementation on source or destination, for potentially increased accuracy or so the source or sink could be a dumb device. The key difference from the one-way delay metric to this draft: emphasis on samples. A singleton and sample discussion added to draft, in particular the concept of a "sample of samples" discussed at Adelaide was introduced. The authors would like to go for Last Call after this meeting (as a Proposed Standard).

Discussion on draft

Do we understand the samples? Will the number of samples affect measurement? Might not periodic streams induce congestion on the network? You have to be careful, any active measurement could induce congestion. This is addressed in the text.

Related question - could different samples at the same time take different paths? What about samples taken at different times that take the same path? We are measuring a "black box", we can't tell what path, and we might not care (if path changes it influences the measurement, which we want to capture). There is also path comments in the draft and in the framework: if you know, or can tell, then report the path.

The RMON WG is looking at ongoing streams. What they want is a real time measurement of the state of streams, and what they are looking at is a "sliding window" to give recent information. Is that related here? Or are there better statistics to be captured to assist their work? (The current draft is draft-ietf-rmonmib-apm-01.txt, Application Performance Measurement MIB. It is trying to capture stream performance ("Continuing real-time statistics on a stream's behavior"). RMON has in the past concentrated on passive measurements.)

There has been a continuing bias toward active measurements. How do our metrics apply to passive measurements? Thought is needed on whether our proposed standards are adequate (we think so, but it has never been systematically worked through). The thought was that this draft would be applicable. We asked if there were statistics that RMON would find useful that might be added to the draft -- "text donations are appreciated".

Periodic streams mentions out-of-sequence, but does not define rigorously, and there are a few different possible definitions. For example, count whenever a packet with sequence number N is followed by a packet with sequence number less than N. This is a purely local definition. On the other hand, you could use a global definition, such as the minimum number of transpositions that would place the sequence in order (but then you must look at the entire sequence).

Where is this draft headed? Informational or Standards? The authors prefer Standards track. Will Leland expressed the opinion that Informational would be less useful, since the intent is to promote consistent implementation and experience.

There was also concern that this draft only covered delay, and not loss or delay variation. The authors felt they were both covered, but perhaps this could be clarified.

Matt Zekauskas asked if this going the right way for VoIP? (to the person asking at the beginning of the meeting whether we could do more for VoIP) The response was "no" but it wasn't clarified. He will take up the issue on the mailing list.

Henk Uijterwaal stated that before we make this a standard we should be sure it measures something useful and of interest; it isn't a good idea to make something standards track without experience using it. Matt replied that technically it is OK to make it a Proposed Standard; it could loop there if revisions are necessary based on experience. Implementations are preferred, however. It can die at proposed standard if there is no implementation. The chairs will discuss it with the ADs.

4. Preliminary Empirical Study of BTC Tools - Mark Allman (NASA Glenn & BBN)

This comparison is a preliminary empirical comparison. Background: we have a Framework draft, a TReno Draft from Matt Mathis, and Mark has been developing cap but there is no draft yet.

The framework draft and the TReno draft have expired. Mark's work suggests the framework is pretty good, so he will make minor revisions and resubmit for working group last-call.

The goal is to measure the throughput a flow would get in the given network environment, but without relying on some one specific TCP implementation. In other words: Is it possible to design a reproducible tool that measures conformant transport protocols (ones that have congestion control) like TCP? It seems that we should be able to do so, but there was no empirical evidence.

The cap tool requires both a sender and receiver. Acks are cumulative, just as in TCP, which is important in tracking performance. Because a sender and receiver are necessary, you have to run something at both end points. A good result of running code on both ends is that ack loss and data loss can be differentiated (which TReno cannot as it is only run on the source host). TReno is more widely usable (because it is sender-only), but doesn't appear to be as accurate.

These preliminary results are from 1000 measurements total taken on 31 machines over 3 days. (2000 were scheduled, however only results from 1000 were available.) The measurement hosts were using a standard version of FreeBSD's TCP, but with 200k buffer size.

Each experiment picked one pair of tests to run, and then ran them sequentially. The pairs were FreeBSD TCP versus FreeBSD TCP (TCP vs TCP); TCP versus cap; and TCP versus TReno.

So TCP vs TCP is the standard to measure the other comparisons against. It turns out that TCP vs cap isn't bad, but TCP vs TReno is a good bit worse. The slopes of TCP versus cap are similar to TCP versus TCP.

TReno tends to underestimate, cap is better (see slides for quantitative numbers). However, cap could be improved further; for example, its emulation of the TCP heartbeat appears to be off (occurring every 2 seconds versus every 1 second in FreeBSD TCP).

Also found a bug in FreeBSD TCP where the system takes a series of RTOs instead of going to slow-start.

Summary: BTC with sender/receiver seems to be possible to do. However, sender only is still an open question in Mark's mind. Needs to dig into this more. Mark would love to make cap do sender side only (if all the time in the world). Mark would like to see a couple of different TCP stacks run against each other. See their variations hen get a feel how cap related to this.

What does this mean for IPPM? Mark thinks framework is good and would like to push forward after some light editing. Draft on tap as in the TReno doc.

- Discussion -

Can't track any one TCP - must be vendor neutral. Aren't all other metrics are independent of protocol? Can you free throughput from TCP? We don't know what that would mean; TCP is the Internet bulk-transport protocol, and is very widely used. It's also what people want to measure. Because there is no standard, people are doing their own ad-hoc measures.

One audience member suggested that IPPM should consider other high throughput protocols. We should develop other tools within the framework, or a real protocol free metric.

More questions were asked about what it meant for cap to be a metric, and what our plans for cap should be. How should it be used operationally? Are we blessing a particular implementation of cap? The intent of the metric is to define the congestion control/response parameters so precisely that any implementation (not just *this* code) will measure the same thing. To use cap, in particular, you simply install it on a pair of boxes and run. The Framework addresses the question of multiple metrics: we can have many bulk-transport capacity metrics within the framework. We now have empirical evidence that cap may provide what we need, and is a way to move forward.

5. Discussion on interest for a one-way delay protocol

The following two presentations were informal discussions; not current working group activities.

5a. Open Active Measurement Protocols -- Ben Teitelbaum (Advanced Network & Services)

The basic question presented was "how can someone develop or use their own measurement tools and yet utilize other existing machines (such as Surveyor) for measurements?". Should we define open protocols for performing IPPM metric measurements? (Note that if we decide to make this an IPPM work item, the charter must be amended.)

Ben presented a model that can be thought of as a simple version of the NIMI work. Four classes of protocols needed - establish the session, exchange test packets, manage the tests, and deliver the data.

The basic protocol requirements were outlined - see slides.
- The establishment protocol is assumed to be over TCP for its reliability.
- All measurements can be performed over the open Internet, so they need to be invisible to others to prevent malicious or inadvertent alteration. For example, use port negotiation or hopping rather than static ports. Perhaps encryption should be an option (or a requirement for session establishment).
- Un-padded message should be small - is it important to fit in an ATM cell?

Ben is writing a draft and looking for help.

[Note: possibility of needing test packet formats that include sequence numbers & timestamps was discussed at the BMWG session on July 31st. It was mentioned there that this might be an opportunity for common test packet format design across several working groups.]

- Discussion -

Bob Cole and Dan Romascani said that some of us are doing draft on synthetic source generators and have a need for test packet formats and test management. They are loosely associated with RMON, and expect to be a future work item in RMON. For example, there is a MIB for controlling source & sync generation. (See draft-ietf-kalbfleisch-sspmmib-00.txt; there is also a framework document.)

There was rough consensus that protocol development should be pursued. A rough draft will be produced soon, and the chairs will approach the ADs based on the draft.

5b. Measurement Protocol for One-Way Performance Metrics
- Lijun Quian (Rutgers)

Lijun Quian asked if he could present a personal submission on a protocol that could be used for traffic engineering. There was no general dissent, so in the final three minutes he described <draft-elwalid-icmp-ext-00.txt>, which defined a ICMP protocol extension to measure one-way delay for the purpose of traffic engineering (in particular selecting MPLS LSPs).

See the slides.

A sample implementation is available from Bell Labs.
The measurement is fault tolerant of reverse path loss.

- Discussion -

Those present voiced problems with using ICMP, and whether we should be doing anything that just supports MPLS. The authors noted that a similar UDP-based approach would also work.

The meeting ended over time, with the Chairs noting that further discussion should be taken to the mailing list.


Preliminary Empirical Study of BTC Tools
IP QoS classes: Layered Approach
Network Performance Measurements for Periodic Streams (npmps)
Measurement Protocol for One-Way Performance Metrics
Open Active Measurement Protocols