2.7.2 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 40th IETF Meeting in Washington, DC. It may now be out-of-date. Last Modified: 10-Nov-97

Chair(s):

Guy Almes <almes@advanced.org>
Vern Paxson <vern@ee.lbl.gov>

Transport Area Director(s):

Scott Bradner <sob@harvard.edu>
Allyn Romanow <allyn.romanow@eng.sun.com>

Transport Area Advisor:

Scott Bradner <sob@harvard.edu>

Mailing Lists:

General Discussion:ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: ftp://ftp.advanced.org/pub/IPPM/archive

Description of Working Group:

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgement (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will define specific metrics, cultivate technology for the accurate measurement and documentation of these metrics, and promote the sharing of effective tools and procedures for measuring these metrics. It will also offer a forum for sharing information about the implementation and application of these metrics, but actual implementations and applications are understood to be beyond the scope of this working group.

Goals and Milestones:

Oct 97

  

Submit drafts of standard metrics for connectivity and treno-bulk-throughput.

Nov 97

  

Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.

Feb 98

  

Submit documents on delay and loss to IESG for publication as Informational RFCs.

Apr 98

  

Submit a document on connectivity to IESG for publication as an Informational RFC.

Sep 98

  

Submit a document on bulk-throughput to IESG for publication as an Informational RFC.

Internet-Drafts:

No Request For Comments

Current Meeting Report

Minutes of the IP Performance Metrics (ippm) Working Group

Reported by Paul Love and Guy Almes

1. Overview and Agenda Bashing

The meeting was chaired by WG co-chairs Guy Almes and Vern Paxson, and was very well attended. This was our first meeting as a formal working group within the Transport Area, and was also our first meeting co-chaired by Vern Paxson.

Proposed agenda:

Welcome: we are now a working group. (10 min)
Final changes to Framework document and Last Call. (10 min)
Update on draft metrics (delay, loss, connectivity). (15 min)
Experiences with draft metrics. (20 min)
Surveyor: Guy Almes
Poip: Vern Paxson
DOE PingER effort: David Martin. (20 min)
End-User Perspective on Internet Performance:
Venkat Rangan/Jim Goetz. (20 min)
Impact of loss characteristics on real-time applications:
Rajeev Koodli/Rayadurgam Ravikanth. (20min)
Report from ANSI T1A1.3 liaison. (10 min)

The agenda was agreeable to the participants, and it was noted that the meeting would be somewhat packed.

2. Final Changes to Framework Document and Last Call: Vern Paxson

[slides included in the proceedings]

Vern Paxson reviewed the changes made to the Framework since our Munich meeting. Unless there are significant objections, we will soon submit the Framework as an RFC.

Key changes include the following:

Pros

Cons

Given this discussion, Vern issued a Last Call. We intend to submit to publish the Framework as an informational RFC. The last call was sent out last week via email. That email last call resulted in two improvements in the Framework:

There being no further comments during the meeting, the Framework was passed off to the IESG (pending a set of final edits). Scott Bradner, the Area Director overseeing our work, expressed his appreciation at this completed milestone.

3. Revisions to the Connectivity ID: Vern Paxson

[slides included in proceedings]

Vern reported on recent updates to the Connectivity Metric:

He also mentioned the comment from Jeff Sedayao that One-way connectivity, which had been thought to be only of theoretical interest, might be useful for dealing with certain security issues; this comment will be added in the future.

4. Revisions to the One-way Packet Loss and One-way Delay Metrics: Guy Almes

[slides included in proceedings]

Guy reported on recent updates to the metrics for One-way Packet Loss and One-way Delay:

With regard to the latter point, he mentioned one specific reservation with wire time: in the case of contention-based networks, such as the classical CSMA/CD-based ethernet, time spent by the sending host in waiting for the network to become available should be regarded as a form of queuing delay in the first hop across that contention-based network and thus included in one-way delay. If a strict notion of wire-time is used, however, this waiting time will not be included. This observation will be included in future versions of the one-way delay metric draft, and may influence future versions of the Framework. He was careful, however, to stress that this problem only occurs when the first hop on a path is over a contention-based network; as networks become increasingly switched base, this problem will occur less often.

Guy then commented on two alternatives to Poisson sampling:

There are no current plans to include either in the one-way delay or packet loss metrics.

Christian Huitema strongly argued for the need to state the "error bars" for measurement results.

5. Experiences with Delay/Loss Metrics

5.a. Poip (Poisson Ping): Vern Paxson

[slides included in the proceedings]

Vern Paxson reported on the development of Poip. Key features include:

Wire time API given for wire_init, wire_done, wire_add_fds, wire_num_filter_drops and wire_activity.

Experience with the goodness-of-fit testing of measurement times shows that scheduled times pass the test, but that user-level timestamps with 10 msec granularity fail when hundreds of samples are tested.

Vern then showed some measurements of wire time vs. send time from Poip testing. Generally, the differences between the two fall into about three discrete values between 100 usec and 200 usec. However, occasionally network events occur that cause widely varying differences. Two that were identified were a large midnight batch job, and nightly backups over the network. These events serve as reminders that sometimes wire times can differ significantly from application-layer perceptions, due to external factors.

5.b. Surveyor Project: Guy Almes

[slides included in the proceedings]

The Surveyor project is a joint effort of Advanced Network & Services and the 23 universities of the Common Solutions Group. The current emphasis is on ongoing operational measurement and archiving of one-way delay and packet loss along all the end-to-end paths between pairs of campuses.

At each campus there is a dedicated 200 MHz Pentium-based measurement machine that measures one-way delay and packet loss with a lambda of 2 packets/sec.

These measurement machines upload their results in an ongoing basis to a database server which stores them indefinitely.

The results can then be accessed via the web using a (currently immature) set of analysis/visualization tools.

The slides show several examples of one-way delay and loss along several paths. In each slide, one 24-hour (GMT) period is indicated. In the delay graphs, the minimum, 50th percentile, and 90th percentile of delay are displayed for each one-minute period. In the loss graphs, the percentage of packet loss is displayed for each one-minute period. The delay figures are believed to be accurate to about 100 usec.

One challenge is to relate these frequent measurements of delay and loss to dynamic changes in route. It is not practical, for example, to both measure delay accurately and know the route taken. Without making too much of it, the similarities with the Heisenberg effect can be considered.

Current work includes broadening the deployment to other sites and to improving the analysis and visualization tools.

6. Three Presentations on Related Research

6.a. Report on the DoE Energy Research PingER Network Monitoring Effort: David Martin

[slides included in the proceedings]

David Martin reported on joint work with colleagues at Fermilab, SLAC, and 15 participating DoE HEPnet sites.

HEPnet has sites of interest around world. Since the network has moved from single-purpose to a world of NAPs, ISPs, etc., there is a need to measure performance.

The talk emphasized the following points:

A set of example screen captures were then shown to give a feel for the use of the tools.

6.b. End-user Perspective on Internet Performance: Venkat Rangan

[slides included in the proceedings]

Venkat presented a set of tools by VitalSigns that aims to:

The technical means used emphasize passively observing:

VitalSigns has a white paper at: www.vitalsigns.com/products/vista/wp/index.html
describing the tools in more detail.

6.c. Impact of Loss Characteristics on Real-Time Applications: Rajeev Koodli/Rayadurgam Ravikanth

[slides included in the proceedings]

This talk, based on research at Nokia's Boston laboratories, advanced some new ideas on understanding how packet loss impacts real-time applications.

Using the current notions of singletons and samples of packet loss, it was noted that the only statistic defined so far was percentage. The talk presented two ways to treat the time-series aspects of packet loss.

First, it was noted that some real-time applications are sensitive to packet loss in such a fashion that isolated losses can be withstood, but bunches of packets lost would degrade quality. For example, an audio application using forward error correction might be able to tolerate isolated packet losses provided that 5 successful packet transmission occur between successive packet losses. Based on this observation, the notion of loss constraint was introduced as the minimum number of successful packet transmissions between lost packets. Thus, if a given test shows that there are always 5 successful transmissions between packet losses, then the stream is said to succeed with a loss constraint of 5.

Second, it was noted that the loss period, defined as the time duration of a burst of losses, might be important for real-time applications. Bursts of losses longer than a critical threshold might have an especially severe negative impact on such applications.

This talk was the first attempt with the IPPM effort to treat the time-series aspects of packet loss. It triggered many questions.

Vern Paxson noted that the significance of loss period would depend on the rate at which the application was attempting to send packets.

Christian Huitema noted that the work was very specific to a particular kind of real-time application, and may not be suitable to standardize for an IETF effort, and asked how could this be generalized to archive information with more general usefulness.

Vern Paxson asked whether anything is known about what burst lengths are known to be problematic for specific applications. The presenters answered that they did not know for audio, but that the loss of two frames in a row was perceived in video.

7. Report from ANSI T1A1.3 Liaison: Vern Paxson

[slides included in proceedings]

Vern Paxson reported on his work as liaison to the ANSI T1A1.3 effort.

T1A1.3 is an ANSI working group in "Performance of Digital networks and services". It has recently initiated work on "Internet Service Performance Specification", and will likely forward the results eventually to the ITU. This work is rooted in ANSI experience going back to efforts to specify the quality of X.25 services (X134 - X139), and has considerable overlap with IETF IPPM work.

Based on their experiences and traditions, the ANSI effort uses terminology different from that used by IPPM. As one example, they allow themselves to define (theoretically) "observable events" that, while well defined, have no practical methodology. Also unlike the IPPM work, they may well specify specific values as criteria for rating networks as acceptable. They will likely emphasize passive methodologies and will likely include link-layer (in addition to network-layer) notions.

For more information, refer to: www.t1.org/t1a1/t1a1.htm

Slides

Agenda and Update
Experience from the Surveyor Project
Paxson Presentation
Impact of Loss Characteristics on Real-Time Applications
End-User Perspective on Internet Performance
Report on the DOE/ER PingER Network Monitoring Effort

Attendees List

go to list

Previous PageNext Page