Last Modifield: 05/06/2002
The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.
Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.
The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:
- connectivity
- one-way delay and loss
- round-trip delay and loss
- delay variation
- loss patterns
- packet reordering
- bulk transport capacity
- link bandwidth capacity
This is the cumulative set, including the metricsalready completed and published.
The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.
The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.
The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.
The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.
Done | Submit drafts of standard metrics for connectivity and treno-bulk-throughput. | |
Done | Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC. | |
Done | Submit documents on delay and loss to IESG for publication as Informational RFCs. | |
Done | Submit a document on connectivity to IESG for publication as an Informational RFC. | |
Done | Submit a document on bulk-throughput to IESG for publication as an Informational RFC. | |
FEB 02 | Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC. | |
FEB 02 | Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC. | |
FEB 02 | Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC. | |
FEB 02 | Create initial draft on the definitions of link bandwidth capacity. | |
FEB 02 | First draft for AS on one-way delay and loss. | |
MAR 02 | Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS). | |
MAR 02 | Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC. | |
MAR 02 | Create initial draft on a MIB for reporting IPPM metrics. | |
APR 02 | Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document. | |
APR 02 | Create initial draft on comparing ITU and IETF IP performance metrics. | |
APR 02 | Create initial draft on a packet reordering metric. | |
JUN 02 | Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC. | |
JUL 02 | Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard. | |
OCT 02 | Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay). | |
OCT 02 | Submit draft on a packet reordering metric to the IESG for Proposed Standard. | |
DEC 02 | Submit AS for one-way delay and loss to the IESG for PS. | |
DEC 02 | Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC. | |
FEB 03 | Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard. | |
MAR 03 | Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS. | |
MAR 03 | Discuss rechartering or ending working group. |
RFC | Status | Title |
---|---|---|
RFC2330 | I | Framework for IP Performance Metrics |
RFC2678 | E | IPPM Metrics for Measuring Connectivity |
RFC2680 | PS | A One-way Packet Loss Metric for IPPM |
RFC2681 | PS | A Round-trip Delay Metric for IPPM |
RFC2679 | PS | A One-way Delay Metric for IPPM |
RFC3148 | I | A Framework for Defining Empirical Bulk Transfer Capacity Metrics |
RFC3357 | I | One-way Loss Pattern Sample Metrics |
IP Performance Metrics WG (ippm) Tuesday, November 19 at 17:30 - 20:00 ==================================== The meeting was moderated by the working group chairs, Matt Zekauskas and Merike Kaeo. Dave Plonka and Al Morton took notes, which were edited into these minutes by the chairs. AGENDA: 1. Agenda bashing 2. Announcements 3. Discussion on Packet Reordering 4. IPPM Reporting MIB and Metrics Registry discussion 5. Discussion on One-way Active Measurements Protocol 6. One-Way Metric Applicability Statement 7. Potential work: "IP Measurement Protocol" (IPMP) 8. Potential work: IPPM Spatial Metrics Measurement 9. Available Bandwidth Measurement in IP Networks IETF home page: http://www.ietf.org/html.charters/ippm-charter.html IPPM home page: http://www.advanced.org/IPPM/ 2. Announcements Mark Allman introduced the IRTF Internet Measurement RG. In general IMRG is a forum for measurement research that isn't yet ready for IETF. It intends to foster interaction amongst communities: IETF, operators, researchers. The IMRG is an *open* IRTF group, anyone can join the list: http://imrg.grc.nasa.gov/imrg/ (linked from irtf.org). David Martin asked if IMRG looking at control aspects - what kinds of measurement? Mark responded that it's open; lots of kinds: passive, active, lots of options to investigate. Al Morton presented on IPPM and ITU-T performance status. Since Will Leland's comparison in March, 1999, some terms have changed and concepts and definitions have gotten closer. One new definition is IP severe loss block ratio, prompted by a desire to measure the effect of re-routing. The one activity where the two groups continue to differ is in the area of specifying objectives for performance; the ITU "Numerical performance objectives" are beyond IPPM's scope. Matt asked how Al participates in ITU-T meetings - in essence why he is qualified to make such a report. Al responded that he attends ITU-T meetings, is the designated editor for recommendations in Study Group 13 where his company has interest (such as Y.1540 and Y.1541). He also contributes to the work of Study Group 12. Al has considered the possibility of an Informational RFC comparing ITU and IPPM terminology but suggests that that is becoming less and less necessary. An audience member (Hidega Tiku, from France Telecom) believed that a broader mapping is possible than just to the Study Group 13 documents (Y.1540 and Y.1541), she believes there is relevance to 30 ITU groups. Al believed that the closest parallel was SG 13, and would like to work with her to understand. Another audience member (Chuck Dvorak, vice chair of SG 12, the lead study group in all ITU-T QoS activities) confirmed that IPPM's closest working counterpart is the SG 13 work that Al described, other groups may have interest in IPPM, but do not define IP metrics. Matt invited them to work together and bring conclusions to the mailing list. 3. Discussion on Packet Reordering -- Al Morton http://www.ietf.org/internet-drafts/dra ft-ietf-ippm-reordering-01.txt Al reported on status of current draft and presented some examples. He expressed that more input is desired as to whether the examples are sufficient. Major issues include integrating the definition of n-reordering (clarification, description of how it fits in with non-reversing sequence), specific advice concerning duplicates, adding a metric of distance between reordering events, and unifying notation. Stanislav Shalunov pointed out that if you want to consider all duplicate packets as not out of order, this is a reasonable definition, but the implementation would require O(N) space to keep track of all the packet numbers. Thus, the metric is "fine but expensive". Al stated that we could just allow all duplicate packets to be declared out of order and consider duplicates at the end of the test. Matt Z. asked for a clarification on Al's use of "clarification" with respect to the n-reordering metric. Was he looking to change the definition? Al responded that he wasn't quite sure yet, but that the goal was to understand when particular metrics were applicable (and possibly add some caveats, but didn't want to resort to that, yet); n-reordering may better related to understanding reordering with espect to TCP, while other sample metrics in section 5, such as Late Time, may be better related to understanding reordering with respect to sizing buffers for streaming media. Jerry Perser (another author) echoed this, and a general difficulty understanding the n-reordering results. Matt Mathis noted that n=3 for reordering isn't a constant for TCP's; it's a parameter to at least one stack. He would also like a better understanding how Web100's reordering statistic is related to n-reordering, and how the metric is related to SACK blocks. 4. IPPM Reporting MIB and Metrics Registry discussion -- Emile Stephan http://www.ietf.org/internet-drafts/dra ft-ietf-ippm-reporting-mib-01.txt http://www.ietf.org/internet-drafts/dra ft-ietf-ippm-metrics-registry-01.txt Jesse Jewitt discussed changes made from IPPM reporting MIB from version 00 to 01. Henk Uijterwaal started a discussion around the use of the word "stratum" in the document - in particular, if it is the same as the NTP stratum level. It is apparently similar, but not exactly the same as NTP's usage (the exact similarities and differences were not clear from the discussion). In addition, Henk worried that any notion of stratum might not be sufficient, a notion of accuracy was also important. Matt stated that since there was confusion, the document should contain a clarification. Henk agreed to take the discussion to the mailing list. A complete list of enumerated open issues is contained in the slides. The authors are working on a prototype. Matt asked if there was anything learned from implementing the prototype. Jesse responded yes; Emile would provide input before or at the next meeting. Emile then gave update on registry. Emile is looking for some procedural guidance to obtaining a MIB root value (how one is obtained from IANA). There was a discussion about MIB OID lengths; the length would vary depending on where it was rooted. Emile was wanted to keep the lengths reasonable for comparison. Emile is also looking for guidance on suggesting procedures to specify how new draft metrics add a section to create a new registry entry. Finally, Bob Cole noted that there are four "types" of metrics mentioned in the registry: singleton, sample, statistical, and "ambiguous statistical" like percentile. He wondered if it was appropriate to have entries for statistics that required parameters. RMON points to just singleton metrics. He felt that the 'ambiguous statistical' metrics that require parameters might not be appropriate for the registry, or that maybe the registry should be split to avoid an explosion. It was noted that all the metrics require parameters; Bob would take the discussion to the mailing list to talk about details. 5. Discussion on One-Way Active Measurements Protocol -- Stanislav Shalunov http://www.ietf.org/internet-drafts/dra ft-ietf-ippm-owdp-05.txt Stanislav Shalunov gave update on OWAMP. Gave list of updates from version 04 to 05 that were based on implementation experience. Matt asked for a clarification on the units of accuracy; it is a scaled power of two, with a huge range but not much resolution. Al Morton noted one spot where the periodic sampling now included in the draft had not been added. The two significant discussion points were the error in timestamps when packets are lost and reported errors for the start session command. The protocol needs a way to report the error in the sending timestamps when the test packets are lost (or at least account for the error). It currently reports a time sent that isn't derived from explicit packet data, and the send error isn't specified. Al suggested setting the send time error to zero as a distinguished value; Stanislav agreed it should be set to something. Second, the reason a session start command is rejected should be stated (or enumerated). There will be at least a number that can be correlated to a specific reason. 6. One-Way Metric Applicability Statement -- Henk Uijterwaal http://www.ietf.org/internet-drafts/dra ft-ietf-ippm-owmetric-as-01.txt Henk gave update on the applicability statement status. There hasn't been a lot of input, and we really need input from operators. Merike queried the room -- about half the people in the room were operators. However, few people had read draft. Folks were asked to read document and send comments to mailing list. Merike mentioned that Henk gave a pitch at the last NANOG, and many there thought that this document would be useful. 7. Potential work: another iteration on "IP Measurement Protocol" (IPMP) -- Jon Bennett http://www.ietf.org/internet-drafts/dra ft-bennett-ippm-ipmp-00.txt Jon Bennett gave presentation on his version of "IPMP" - an IP measurement protocol, first presented last year by Tony McGregor. He talked about the motivation, and then gave an overview of changes that were modifications to make the protocol more palatable to hardware implementers (in his opinion) and add some functionality. IPMP is explicitly designed to be easily recognizable by intermediate systems, so that they can add measurement data as the packet is routed. After an initial stumble, Jon is working with the authors of the McGregor draft to create a merged document. David Martin wondered if this is essentially a DDoS engine, since it is not required to have authentication. Jon noted that there are a number of security/authentication options and other considerations to prevent it from being used as a DoS; in addition it doesn't make any new attacks available. It does allow you to redirect, but not hide the original source IP address. Matt queried to the room to see if there was more interest than was shown on the mailing list since Tony presented last year. Only a couple people admitted to reading the draft. Merike asked if there were any vendors willing to implement it, or if providers in the room were interested. Jon noted that Motorola was looking towards implementing it, which could get it in ~200,000 cable modems. Nevil Brownlee noted that Tony McGregor does have a working implementation (in end hosts), and it's deployed in the NLANR AMP infrastructure (~100 hosts); Nevil also noted that the focus on turnaround time and minimal IPMP processing was to have IPMP be a good measure of the forwarding of "real" traffic along a forwarding path. A service provider indicated that it would be nice to have, but he was concerned that IPMP is awfully complex to implement well; if a vendor can't decrement TTL properly, how would they get this right? 8. Potential work: IPPM spatial metrics measurement -- Emile Stephan http://www.ietf.org/internet-drafts/draf t-stephan-ippm-spatial-metrics-00.txt Emile presented a draft on "spatial metrics" -- obtaining measurement data from intermediate points to help break apart path measurement data to component path segments. The advantage of this document would be understanding and standardizing the accuracy and uncertainty in the results. Ruediger Geib noted that this assumes that you understand packet routing a-priori, and something like load-balanced induced changes in routing would cause problems. Emile thought it could be found by combining passive measurements with the active traffic. Ruediger also wondered if the metric involved collecting many individual results, perhaps it involved more of a database and measurement system, and wasn't an individual result itself. 9. Potential work: Available Bandwidth Measurement in IP Networks -- Tricha Anjali http://www.ietf.org/internet-drafts/draf t-anjali-ippm-avail-band-measurement-00.txt Tricha Anjali presented work on available bandwidth measurements. She started with explanations of the various definitions, and then talked about her technique, which involves the use of TCP, but not allowing it to reach steady state. Each measurement is a sample; the samples are massaged to appear uniform, and then a linear filter is used to make predictions. Additional data (such as SNMP measurements) can be used to provide additional accuracy. The details are in the slides and draft. Mark Allman noted that a RFC defining all the terminology used in bandwidth measurement would be useful, and he has attempted to help write one; other authors would be welcome. Mark asked the chairs if the notion of predicting the "future bandwidth" in WG scope? Matt responded that his feeling was that it was not, and it was more of a research topic. Mark Allman also questioned letting TCP go until a loss occurs, since the size of queue skews the bandwidth metric because of the way TCP bursts. Tricha would investigate. Matt asked if there are there simulations or actual measurements showing how accurate this method is; Tricha responded not yet. Nevil Brownlee comment about method: if you assume there is nothing on the link, and you use one TCP stream to test, I don't see how you can add one more TCP stream to an unknown number, and understand the affect on the other traffic. Matt Mathis noted it was a Heisenberg problem: you don't know how much the test traffic causes the other traffic to slow down. This seems to be a fundamental problem with this class of method. Mark Allman noted the measurement indeed could affect other traffic since it goes until it induces loss. |