NOTE: This charter is a snapshot of the 51st IETF Meeting in London, England. It may now be out-of-date. Last Modified: 31-Jul-01
Merike Kaeo <kaeo@merike.com>
Matthew Zekauskas <matt@advanced.org>
Scott Bradner <sob@harvard.edu>
Allison Mankin <mankin@isi.edu>
Allison Mankin <mankin@isi.edu>
General Discussion:ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: http://www.advanced.org/IPPM/archive/
The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery
services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgement (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.
Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.
The IPPM WG will define specific metrics, cultivate technology for the accurate measurement and documentation of these metrics, and promote the sharing of effective tools and procedures for measuring these metrics. It will also offer a forum for sharing information about the implementation and application of these metrics, but actual implementations and applications are understood to be beyond the scope of this working group.
Oct 97 |
|
Submit drafts of standard metrics for connectivity and treno-bulk-throughput. |
Nov 97 |
|
Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC. |
Feb 98 |
|
Submit documents on delay and loss to IESG for publication as Informational RFCs. |
Apr 98 |
|
Submit a document on connectivity to IESG for publication as an Informational RFC. |
Sep 98 |
|
Submit a document on bulk-throughput to IESG for publication as an Informational RFC. |
· IP Packet Delay Variation Metric for IPPM
· A Framework for Defining Empirical Bulk Transfer Capacity Metrics
· One-way Loss Pattern Sample Metrics
· Network performance measurement for periodic streams
· A One-way Delay Measurement Protocol
· A Bulk Transfer Capacity Methodology for Cooperating Hosts
RFC |
Status |
Title |
RFC2330 |
Framework for IP Performance Metrics | |
RFC2678 |
E |
IPPM Metrics for Measuring Connectivity |
RFC2679 |
PS |
A One-way Delay Metric for IPPM |
RFC2680 |
PS |
A One-way Packet Loss Metric for IPPM |
RFC2681 |
PS |
A Round-trip Delay Metric for IPPM |
Minutes of the IP Performance Metrics WG (ippm)
Tuesday, 07 August 2001, 13:00 - 14:00
===================================================
The meeting was chaired by the working group chairs, Matt Zekauskas and Merike Kaeo. Reported by James Stevens and Paul Love; these minutes were edited by the chairs.
AGENDA
1. Agenda bashing
2. Small case-study on advancing metrics through the IETF standards process; comparing RIPE-NCC and Surveyor data as per the internet-draft.
3. One-way Active Measurements Protcol Requirements
4. Measuring Packet Reordering
5. Potential new work items
Sampling Methodology
MIB for IPPM
IETF home page: http://www.ietf.org/html.charters/ippm-charter.html
IPPM home page: http://www.advanced.org/IPPM/
2. Comparing Metrics based on draft-bradner-metricstest-00.txt
--Matt Zekauskas
Matt presented a small case-study on advancing metrics through the IETF standards process, comparing RIPE-NCC and Surveyor data as per the I-D. He noted some issues: choosing which values you used for the mean & standard deviation, the fact that you might end up with a standard deviation that is less than the calibration error, that you need to be explicit about which values are included, or not included (for example, holes due to GPS loss), and that comparing singleton measurements for streams of delay values is not well defined.
Questions were raised on how were intervals in which no data from metrics was recorded dealt with. This had the potential to make the data set recorded appear in statistical terms better than it actually was. This was highlighted as a measurement implementation issue e.g. GPS fading due to rain, which could coincide with network degradation on a microwave link.
3. One-way Active Measurements Protocol Requirements
--Merike Kaeo
Although we received no comments on the requirements draft, several people in the meeting indicated that they did read the draft.
There was some discussion on post-processing of data, and whether the requirements draft should forbid it. The current draft appears to forbid it, but several commented that post-processing, at least in some sense, is required to detect loss and errors. Section 4.1 of the draft is ambiguous -- does it forbid it or require it? Matt pointed out that his understanding of the authors intent was that significant post-processing should not be required (but be allowed).
Merike requested that those reviewing the drafts provide sample text for revisions rather than just pointing out that revisions are needed.
The scalability and security of protocol were discussed . Comments were solicited, and Merike noted that security aspects would need to be reviewed by relevant IETF experts.
The types of measurement machines that could be controlled can vary in their capability (for example, precision). There should be some ability for the control node to know the capabilities. There was brief discussion to put this in general terms in the requirements, and work out details in the protocol.
A participant noted that there was no need to restrict the measurements to be one-way delay, or even one-way. Merike noted that was why the name was proposed to be changed from one-way delay protocol requirements to one-way active measurement requirements.
There was some discussion of additional feature reporting that should be added. The following were mentioned:
- Any known time-varying aspect of metrics or the implementation should be reported at the outset.
- The ability to have a maximum number of measurement packets or sequence numbers should be provided.
- The ToS/DSCP should be reported.
It was also noted that in some cases the requirements document switched to implementation instead of requirements, for example, specifiying UDP is not approprate; it is one solution, not a requirement.
Another participant noted that it would be useful to specify that parameters (say of a Poisson stream) should change over time, for example, set changes based on diurnal or weekly patterns in traffic streams. Right now, you must start a new session.
There was also a comment that there should be some way for vendors to include optional vendor-specific information.
Another participant asked about the timeframe for progressing this document; the hope is that the authors can get a revision out in the next month. The requirements document should be finalized before work on OWDP can officially continue.
Those that commented should send notes and comments to the list to provoke more discussion and allow us to reach consensus.
The working group had unanimous consensus that the name for the OWDP should be made more generic (so as to also include metrics for IPDV, Loss, etc). Proposed was the One-Way Active Measurement Protocol.
Matt briefly presented a slide from Stanislav Shalunov asking if the new send schedule proposed for OWDP was sufficient for the requirements. A pointer to the slide will be posted to the mailing list, and working group members were requested to comment on it.
4. The Packet Reordering Problem
-- Dale Pullin, California Institute of Technology
Dale Pullin presented his work on a metric for packet reordering that would result in a single number representing the "orderedness". He focussed on a using a value known as "Minimum Longest Ascending Sequence" (MLAS). A website (http://www.tiac.net/users/cri/mlas.html) contains the basic algorithm. However, Dale noted that the sample code on the website contains some errors. His metric, omega, is the length of the MLAS, divided by the number of elements. It can be computed in NlogN time in O(N) space.
Matt Mathis noted that for sequences which are nearly always in order, the MLAS algorithm doesn't seem to characterize the measure of the number of packets not in the expected location. He also wondered that in a practical sense, was an NlogN algorithm really needed?
Comment by Henk that MLAS algorithm represented a measure of orderness, not distance between out-of-order packets, and wondered if this showed anything interesting about real network behavior. For example, if a sequence has 10 elements, and the first two elements are flipped, omega is .9; If you flip the first and last elements, omega is still .9. MLAS provides fraction of packets that are out of sequence, but does not state how far out they are. Dale thought this could be determined if needed.
Clarification was asked for what was meant by the "terminal element": it is the last element.
There was a question regarding similarity of this algorithm and the results it provides with other sort based algorithms. In those algorithms the measure is number of sequential inversions (SI). Is this different from that, or just a different way to come up the same thing. Which is better? SI can be done in linear time. Has thought been given to the relative value of information provided?
Matt Mathis commented that the number of backward steps might be sufficient, i.e. that the simple number of swaps metric might be better suited for TCP (detecting when TCP will have major problems). He also pointed out that he thought that the presented algorithm would see more reordering than one that just did backward swaps.
Dale will check this and follow up, but verified that the MLAS algorithm was different. Dale also commented that he investigated some faster algorithms, but found that these were not always representative.
Someone asked if the algorithm could be used incrementally, rather than operating on an entire sequence at once. Dale confirmed that MLAS algorithm could take updates on packet arrivals for use in "real-time".
The group felt some out-of-order metric (packet reordering metric) was needed, but that this particular metric would require some relationship to other network phenomena (like bulk transport capacity) to be really useful, before it would be taken up as a WG document.
5. Other potential new work
Matt Z. briefly presented an outline slide on Sampling Methodology work by Mihailo Zilovic and Lubomir Citkusev. They have performed some simulations to see what parameters might be suitable for detecting the underlying network traffic. The initial observation is that the sample size required depends on the traffic type. There will be a post to the mailing list; please comment on it.
Finally, there was a personal submission on developing a MIB for the IPPM metrics. Emile Stephan wanted to present a personal draft on a MIB for IPPM. An outline and introduction was given, but time constraints prevented full discussion. The chairs requested that consideration of whether this belongs in the IPPM WG needs to be addressed (including the relationship to work in RMONMIB and maybe other ops MIB activity) and that comments to mailing list are required on acceptance for this work
Agenda
Comparing Metrics Using 'bradner-metricstest-00'
One-Way Active Measurement Protocol Requirements
One-Way [mumble] Measurement Protocol: General Issues
On the Packet Ordering Problem
Sampling Methodology for Packet Delay Measurements
'draft-stephan-ippm-mib-00'