2.7.3 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 43rd IETF Meeting in Orlando, Florida. It may now be out-of-date. Last Modified: 25-Nov-98


Will Leland <wel@bellcore.com>
Matthew Zekauskas <matt@advanced.org>

Transport Area Director(s):

Scott Bradner <sob@harvard.edu>
Vern Paxson <vern@ee.lbl.gov>

Transport Area Advisor:

Vern Paxson <vern@ee.lbl.gov>

Mailing Lists:

General Discussion:ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: http://www.advanced.org/IPPM/archive/

Description of Working Group:

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgement (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will define specific metrics, cultivate technology for the accurate measurement and documentation of these metrics, and promote the sharing of effective tools and procedures for measuring these metrics. It will also offer a forum for sharing information about the implementation and application of these metrics, but actual implementations and applications are understood to be beyond the scope of this working group.

Goals and Milestones:

Oct 97


Submit drafts of standard metrics for connectivity and treno-bulk-throughput.

Nov 97


Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.

Feb 98


Submit documents on delay and loss to IESG for publication as Informational RFCs.

Apr 98


Submit a document on connectivity to IESG for publication as an Informational RFC.

Sep 98


Submit a document on bulk-throughput to IESG for publication as an Informational RFC.


Request For Comments:







Framework for IP Performance Metrics

Current Meeting Report

IP Performance Metrics WG (ippm)
Tuesday, December 8 at 1415-1645

The meeting was chaired by the working group chairs, Will Leland and Matt Zekauskas, and was reported by Paul Love.


1. Introduction (5min)
2. Report from T1A1 - Garry Couch (30min)
3. Standards coordination discussion - Garry Couch (15min)
4. One-way delay & loss - Matt Zekauskas (2min)
5. Round-trip delay - Guy Almes (10min)
6. Bulk transfer framework - Matt Mathis (20min)
7. IPDV - Carlo Demichelis (10min)
8. Derived metrics - Rajeev Koodli (10min)
9. Wrap up & futures - Will Leland (10min)

IETF Home page: http://www.ietf.org/html.charters/ippm-charter.html
IPPM Home page: http://www.advanced.org/IPPM/

1. Introduction - Will Leland, Bellcore and Matt Zekauskas, Advanced Network & Services

Will opened the meeting by welcoming Matt Zekauskas as a new co-chair, replacing Guy Almes. The WG thanked Guy for his miny contributions as co-chair, and then Matt reported on the status of the RFCs and Internet-Drafts of the IPPM WG.

In addition to the Framework document, which is an Informational RFC, the Connectivity draft has been approved as an Experimental RFC. We are trying to advance the metrics in a manner parallel to the way protocols are advanced within the IETF. Thus, we are particularly pleased that Connectivity was approved as Experimental. The one-way loss and delay drafts are basically done. They are still in WG last call because we are waiting for input from T1A1. We anticipate they will be in IESG last call in a month. There is a new round-trip draft, which follows the one-way drafts and we anticipate moving quickly. There seems to be less consensus on the IP Delay Variation draft.

2. Status of the T1A1.3 WG, and Standards Coordination - Garry Couch, Chair of T1A1.3 (and also from Bellcore)

2.1 Introduction, and coordination

In the past year, the T1A1.3 working group has been working on IP performance. It has produced a document that is now numbered as I.380. It is one step away from approval, which is expected at the next meeting in February. At this point, only minor wording fixes can be made to the body of I.380, but the appendices are not normative (that is, not formally part of the standard) and can still be changed. They tried to put much of the technical detail, especially where IPPM's approach was not defined yet, in appendices to allow easier revision.

However, there is the appearance of duplicate effort between T1A1.3 and IPPM, which is not needed (nor wanted by industry). If we don't plan to coordinate work efforts it will cause confusion so let's coordinate.

2.2 T1A1.3 performance effort history, with an eye toward IP performance

T1A1.3 is charged with defining performance standards for digital networks and services. Historically, this has been networks like SONET, ATM, and X.25. The ITU is a treaty organization under the UN. Its member's are governments, although a new class of membership has been recently added to handle representation of multinationals like AT&T. ITU-T defines telecommunications standards. T1A1.3 is the US forum for positions on digital performance going into ITU-T.

T1A1.3 is working to develop performance measures that are:

For example, they don't just note that a network loss event occurred, but try to understand the cause of a loss and don't charge the loss to the network if the loss is caused by a violation of the network design specifications. Thus, they look at what protocol data units (PDUs, "packets") are in the network, what layer causes a loss, if PDUs are not lost do they arrive completely in time (a "success"), or are they errored? Their framework could observe "spontaneous PDU creation" and count that.

This approach is shaped by the expectation that T1A1.3 will be defining specific levels, or values, of metrics that define an "acceptable service". They want users to be able to test the network and see if it's meeting its performance commitments.

Two previous (non-IP) examples are work with ATM (I.356) and Frame Relay (X.146). In the case of ATM, they looked at loss, delay and delay variation. This included two point cell delay variation (2pt CDV), cell loss ratios (CLR), and cell transfer delay (CTD) with four classes of service.

2.3 T1A1.3 IP performance efforts

The intent is to follow IPPM definitions as closely as possible. Some wording is changed because terms have very specific meanings with the ITU, and T1A1.3 did not want to introduce confusion. For example, "data link" is changed to "link".

The purpose of I.380 is to allow various providers to talk in a common language about performance, thus it does not concentrate on performance within a network.

I.380 considers routers, hosts, links, circuit sections, but not circuit links because of the focus on the total result for a given network, not in internal pieces. Measurements are taken at "measurement points". There is also a notion of network ensembles - collections of networks put together.

An example was given with a source host, gateway router into net A, which was interconnected with three other networks, one of which had a gateway router to a destination host.

Definitions in I.380 are based on consideration of all fragments and all allowable input and output routers. Therefore, losses (for example) are charged to where fragment(s) go astray.

Possible IP packet transfer outcomes are in section 5, and include successful, errored, loss (or misdirected), and spurious (spontaneous creation). Possible IP packet transfer performance outcomes are considered in section 6, and define the set of packets considered (the populations of interest), delay, mean delay, delay variation, error ratio, loss ratio, spurious creation rate, and flow-related parameters. Both delay variation and flow related parameters are described in general, but the details are relegated to appendices since they don't yet have a good handle on these definitions. Because the details are in appendices, they are not normative and may change in the future.

I.380 also defines IP service availability. There is an IP service availability function (that would allow, for example, the claim that if it's less than 75% ask for your money back). The IP service availability parameters include percent IP service unavailability and percent IP service availability. All of these assume a 24x7 target schedule. Again, the details are in an appendix.

The IP packet delay variation (jitter) appendix considers two methods to limit jitter: interval-based and quantile-based. The interval-based method sets a limit of some percentage of packets that fall outside a delay variation interval, e.g., [-30 msec, +30 msec]. The quantile-based method bounds the delay distribution, saying for example (99.5 percentile) - (0.5 percentile) < b.

2.4 Future efforts within T1A1.3

Two potential future areas were specified. First was the general quality of IP services; they may look at TCP, UDP, RSVP, etc., in addition to IP. Second, the group may look at effects on ATM and frame relay performance on IP performance.

T1A1.3 will also impact other working groups within T1A1:
1 - which is survivability of the telephone network
5 - which is looking at multimedia specification for IP

2.5 Coordination between IPPM and T1A1.3

From the above discussion, it appears that we are right at the edge of stepping on each other's toes, so we really need coordination between our two efforts. Both Garry and Will Leland work in the same company, so they can easily share thoughts.

One possibility is to consider complementary approaches that build on the respective groups' strength:

- measurement methods is the IPPM area
- controlled or active mode of measurements
- relative QoS

- abstract definitions
- "uncontrolled" or passive measurement methods
- absolute QoS

Garry's presentation closed with a call for cooperation & coordination between the two groups

T1A1.3 work can be located by visiting the T1 web page at www.T1.org. The last "official", rapporteurs, version of the draft is located by looking at T1A1 files, then T1A1.3, then 1998. The version is called 8a13041c.doc, and there is a PDF version with a PDF suffix. A better, more up-to-date version was given to the IPPM chairs, and is currently at http://www.advanced.org/IPPM/T1A1/I.380-8dec98.doc (also .pdf).

3. Q&A, and coordination of standards discussion - Garry Couch

Another complementary area was raised: we can't do "real numbers" (define "good"), while T1A1.3 can put up actual criteria values. This was seconded by Scott Bradner - IPPM does "how to measure", not "what measurements mean".

T1A1.3 also considers how long and how often to measure, and the population of interest -- what points to measure. They define more possibilities for error. They also have more material on sampling techniques. They use 5 minute intervals as the basic test. They have looked into how to get rid of biases; it was done a while ago for the packet case. However, they won't do anything specific for IP if they can reference an IPPM work.

There was a discussion about periodic sampling. Some felt that there was too much of a desire to avoid synchronous measures in IPPM work -- all the emphasis on Poisson samples. IPPM started with Poisson samples to avoid synchronization with network events. (The current metrics do not prohibit periodic samples; one can submit periodic samples for standardization following the existing text.) T1A1 has a historic emphasis on periodic samples representing aggregate behavior over successive intervals, as has been conventional in traditional telephony and datacomm. IPPM has a base idea of "delay observed at this sample point", contrasted with the delay seen within the Nth 5-minute interval of day X.

4. One-way delay and loss - Matt Zekauskas

The one-way delay and loss drafts are now in working group last call, awaiting comment from T1A1.3 and ITU folks. The minor changes from the last meeting were quickly reviewed, mainly wording changes to alleviate confusion about calibration reported at the last meeting, and the addition of RFC-2119-style MUST/SHOULD wording.

5. Round-trip delay - Guy Almes, Advanced Network & Services

The new round-trip delay draft was presented. It follows the one-way delay draft very closely, so we hope to move it along quickly. Two points not in the slides were mentioned: instrumentation at the "far end" can be very simple, and the clocks do not have to be synchronized. The clocks do not have to be synchronized because only the source clock need be considered - it measures from the time the test packet is sent out to the time the response packet is received.

The major comment from the floor had to do with differing Type-P values for forward and reverse paths. For example, it might be necessary to define round-trip delay on paths related to QoS. One of the strengths of one-way delay is that it applies to multicast and DiffServ. We may also want 4k bytes outbound with a 40 byte return. It was noted that we should be careful to understand if this is two different one-way delays or an expanded round-trip delay.

6. Bulk transfer framework - Matt Mathis, Pittsburgh Supercomputer Center

Matt showed a slide showing why a single bulk transfer capacity metric is hard or impossible. The slide shows the throughput of three TCP implementations, one using Reno, one using New-reno, and one using SACK. Of these, only SACK was monotonic; the other two had a knee after which additional capacity actually lowered net throughput. The problem is that in this example TCP opens its window wider as the raw throughput increases. However, there is a router in the path which "hiccups" - in this case, its route cache is flushed (a CSU with clock drift may also cause this phenomenon). This hiccup causes a clustered loss, and TCP implementations without SACK cannot tolerate clustered losses.

A framework was presented to cover this case that includes a three-pronged approach. First, there will be multiple bulk transfer metrics. Metrics may only be valid in some region. Second, these metrics will be tightly specified for example, by giving an implementation. Third, ancillary metrics will show under what region given bulk-transfer metric is valid... for example, one might measure path pathologies.

There was a discussion of the problem of capturing idiosyncrasies in metrics: you don't want router vendors designing for metric that captures some idiosyncratic part of current networks, rather than for a monotonic metric.

6a. Comments, Q&A, and discussion

Vern Paxson asked if this is a real world problem? Matt contended that routers which exhibit the above behavior do still exist. They have been mostly eradicated from backbone networks, but still exist on lots of campuses, and CSU clock variation can also cause the problem. Vern also mentioned that the non-monotonic behavior is important to users (someone would like to know if buying a larger capacity connection could produce worse performance).

In addition, Vern pointed out that the essence of bulk transfer is how much you can do within the rules. There is a lot of potential for the analytic models to pay off, but they aren't here today. A bulk transfer metric would be a big benefit if it can be gotten out the door soon. The implication is that you might not want a complex complete solution. Matt contended that the framework will help us to learn to understand multiple bulk transfer capacity metrics and learn which applies where.

7. IP Packet Delay Variation Metric; Changes in draft - Carlo Demichelis, CSELT

Carlo presented the latest changes in the IPDV metric, which are all in the slides. There was a discussion of asymmetry at the end: is it of interest? If not, section A.5 will be deleted.

8. Derived metrics - Rajeev Koodli, Nokia

So far IPPM has produced a framework and some base metrics. This is a proposal for work on new metrics that build on these base metrics. For example, a metric on loss processes would be useful for voice-over-IP and streamed media. This would include "best current practices" that apply existing metrics to new areas. Methodologies for extremely low-loss low-delay networks may be different. In addition, the idea of a framework for these "derived metrics" was presented.

8a. Comments and Q&A

There was discussion on the purpose of less-generic metrics. There was consensus against a metric that was less-generic in the sense of not generally applicable. Rajeev pointed out that he really wasn't arguing for very specific metrics; they would still have wide applicability, but would be capturing new features, for example a loss process rather than just loss. Matt Mathis pointed out that the tail of a delay curve is very useful, but not a basic metric. For example, there are applications for which if a packet arrives more than N ms late, it is as good as lost. Perhaps they should be termed "synthetic" in place of derived or less-generic.

It was pointed out that the Type-P parameter fits many needs: it can represent DiffServ traffic, for example.

9. Wrap up & futures - Will Leland

Since there is little time left, please use the mailing list for discussion. A new charter paragraph that includes cooperation with other standards bodies will be posted soon. In addition, new goals and milestones will be posted to the list for debate. In particular we need some more experimentation.

Will then raised some new potential directions. First, best current practices documents are definitely needed. As with protocols, we would like at least two implementations of the metrics, and these need to yield the same results. The result should be a best current practices and comparison document. Second, should we be concerned with multicast? Please discuss on the list if you think this is a needed direction. Finally, is IPPM the appropriate forum for protocol definitions? Should IPPM define message protocols, so different implementations of a tool for measuring a given metric could interoperate? Would such work be useful? Is it premature? Should IPPM or a new working group do the work?

The meeting was closed with an invitation to comment on these topics on the mailing list.


Report from T1A1
One-way Delay and Packet Loss Update
Round-trip Delay
Bulk Transfer Framework
Derived Metrics