NOTE: This charter is a snapshot of the 45th IETF Meeting in Oslo, Norway. It may now be out-of-date. Last Modified: 24-Jun-99
Will Leland <firstname.lastname@example.org>
Matthew Zekauskas <email@example.com>
Transport Area Director(s):
Scott Bradner <firstname.lastname@example.org>
Vern Paxson <email@example.com>
Transport Area Advisor:
Vern Paxson <firstname.lastname@example.org>
To Subscribe: email@example.com
Description of Working Group:
The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgement (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.
Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.
The IPPM WG will define specific metrics, cultivate technology for the accurate measurement and documentation of these metrics, and promote the sharing of effective tools and procedures for measuring these metrics. It will also offer a forum for sharing information about the implementation and application of these metrics, but actual implementations and applications are understood to be beyond the scope of this working group.
Goals and Milestones:
Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Submit documents on delay and loss to IESG for publication as Informational RFCs.
Submit a document on connectivity to IESG for publication as an Informational RFC.
Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
Request For Comments:
Framework for IP Performance Metrics
IPPM Metrics for Measuring Connectivity
Tuesday, 13 July 1999 at 14:15-15:15
The meeting was chaired by the working group chairs, Will Leland and Matt Zekauskas, and was reported by Paul Love.
1. Introduction & Status
3. Loss pattern update
4. Futures and Milestones
IETF Home page: http://www.ietf.org/html.charters/ippm-charter.html
IPPM Home page: http://www.advanced.org/IPPM/
1. Introduction & Status, Matt Zekauskas (Advanced Network & Services)
Within the last two weeks a number of metrics were approved as Proposed Standard. Vern Paxson and Scott Bradner will write a document that explains the meaning of a metric to be on "standards track" (similar to the document for MIBs). Connectivity (RFC 2498) will be reclassified as a Proposed Standard, and the one-way delay, loss, and round-trip delay drafts were all approved as Proposed Standards.
Since the last meeting, the bulk transfer framework, IP delay variation (jitter) and loss pattern drafts were all updated. If anyone has comments on the bulk transfer framework, please take them to the mailing list; it seems to the chairs that the bulk transfer document is close to being done (at least until we have experience with some individual bulk transfer metrics). The IPDV document has been picked up by Phil Chimento (see below). The loss patterns document has been updated, but mostly cosmetically, and will be discussed today.
2. IPDV Update, Phil Chimento (University of Twente)
Since the last meeting, Phil Chimento has taken over the IP Delay Variation draft after Carlo Demichelis switched positions; Phil has released version 3 of the draft. The major changes are that the "discontinuous samples" definition was deleted from the draft, and the old discussion of IPDV mean value was also deleted. A new section was added to clarify distributions of variation, and a new statistic was added for jitter that follows Van Jacobson's definition from DiffServ. In addition, a new section on the relationship with other standards (ITU-T) was added, as well as a short section on security. The next version will clean up the discussion of clock skew and the effects of taking IPDV measurements when the clocks are not synchronized.
Phil then went through a discussion on the components of delay (see slide) to ensure that we all understood what delay variation really means, as presented in this draft. The specific emphasis was on the relationship between pair-wise delay variation versus change in delay from some absolute: even where each pair-wise delay variation is small, the total variation can be large.
There was also a discussion of the ITU's delay variation measures for 1-point and 2-point delay variation of ATM cells, and the non-normative appendix to the IP measurement specification which discusses 2-point delay for IP. The ITU's specification is based on absolute variation from a reference, either a particular value (e.g., the first) or the average of a set of values.
Finally, Phil turned to questions for the working group -- essentially is the IPDV metric headed in the right direction?
In particular, what is the purpose of IPDV?
- Jitter Estimation? If so, in the EE sense (deviation from absolute) or the Computer Science sense (relative differences; for example, Van's definition from the expedited forwarding specification, or min & max differentiation with distribution)?
- Sizing Buffers?
- Traffic characterization?
- Estimation of queuing delay?
What would you do if you had reams and reams of numbers?
One ISP in the audience said that their need was for measuring real time voice characteristics and capacity planning. These kinds of uses imply that summaries and trends over time are important. Another audience member mentioned that delay variation metrics can serve to determine how large a token bucket is required to provide unimpeded access to a packet stream that is intended to be constant (or nearly constant) rate. Will mentioned that such use relates to so-called "R/S statistics" developed in hydrology for sizing dams.
3. Loss pattern update
- Introduction by Matt Zekauskas,
- Update by Steven Glass (Nokia)
Matt's introduction gave the basics about loss patterns and why they are interesting, and about the loss pattern metrics which build on one-way loss streams. [See slides.]
Steve pointed out the changes to the document, which include many minor changes. Section 3.3 added a clarifying example, but it may have a typo. There is a new Loss-distance-stream metric.
Finally, Steve led a discussion about potential controversial items and whether the metrics are ready to be submitted to the IESG. It is felt that the draft needs polish, but it's complete and pretty close to being ready to go.
A question from the audience asked about the effect of the number of drops causing retransmits and timeouts that cause retransmits (specifically, TCP). This particular item is out of scope for the group, except as part of a bulk transfer capacity metric. Matt Z. noted that Matt Mathis has talked about a "TCP MIB" -- exporting TCP variables. However, that is out-of-scope for us.
There was a discussion about "Noticeable Rate" -- some applications are sensitive to even two adjacent packet losses.
There was also a discussion about the fact that a burst loss is considered terminated by a single good packet arrival. A burst followed by a single good packet and another burst would seem to be better characterized as one long burst. Will noted that what is being measured could provide the raw data for constructing a "less sensitive" metric. Would some sort of "errored period" be a meaningful thing to define?
Finally, there was a discussion about what to do with this draft. Is it useful enough to move forward? Should it be standards track? Consensus from the group was that it should be cleaned up and moved forward as an Experimental RFC.
4. Milestones and futures, Matt Zekauskas and Will Leland (Telcordia).
There wasn't much time left, but Matt presented the current thinking on milestones. [See the slides.] Key to the table: The columns are target dates -- months in 1999 and 1Q 2000. The work items are on the left. Within the table, N means new draft, R means revised draft, S means submit to the IESG. "IPPM and ITU" is shorthand for a draft that compares the IPPM metrics with the ITU metrics based on Will Leland's presentation at the last IETF in Minneapolis. The Delay-Loss BCP is a best-current-practices document for the one-way delay and loss metrics, based on at least the experience with the Advanced Network & Services implementation and the RIPE NCC implementation. The milestones are fairly aggressive, but the goal is to have all the basic metrics done by 1Q 2000.
There were two questions from the floor: should Internet weather maps be discussed here, and should we define MIBs for metrics. Both topics are apparently in-scope, as long as the maps are restricted to the specific metrics under discussion. However, they are not currently specifically part of the charter. There has been a preference in the IETF for MIBs to be developed by the working group that originates the related standards, so MIBs for metrics naturally fall under our responsibility. We should look for help from people already experienced in MIB design and use.
In continued informal discussions after the official WG slot ended, the broad question was raised about the applicability of IPPM's proposed metrics for SLAs: are our current metrics of use for SLAs? Are there other metrics within our scope that could be useful for SLA definitions? The Chairs urged that this discussion be raised on the mailing list.