Last Modified: 2004-02-11
The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.
Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.
The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:
- connectivity
- one-way delay and loss
- round-trip delay and loss
- delay variation
- loss patterns
- packet reordering
- bulk transport capacity
- link bandwidth capacity
This is the cumulative set, including the metricsalready completed and published.
The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.
The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.
The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.
The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.
Done | Submit drafts of standard metrics for connectivity and treno-bulk-throughput. | |
Done | Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC. | |
Done | Submit documents on delay and loss to IESG for publication as Informational RFCs. | |
Done | Submit a document on connectivity to IESG for publication as an Informational RFC. | |
Done | Submit a document on bulk-throughput to IESG for publication as an Informational RFC. | |
Done | Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC. | |
Done | Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC. | |
Done | Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC. | |
Feb 02 | Create initial draft on the definitions of link bandwidth capacity. | |
Done | First draft for AS on one-way delay and loss. | |
Mar 02 | Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS). | |
Done | Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC. | |
Done | Create initial draft on a MIB for reporting IPPM metrics. | |
Apr 02 | Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document. | |
Apr 02 | Create initial draft on comparing ITU and IETF IP performance metrics. | |
Done | Create initial draft on a packet reordering metric. | |
Jun 02 | Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC. | |
Jul 02 | Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard. | |
Oct 02 | Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay). | |
Oct 02 | Submit draft on a packet reordering metric to the IESG for Proposed Standard. | |
Dec 02 | Submit AS for one-way delay and loss to the IESG for PS. | |
Dec 02 | Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC. | |
Feb 03 | Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard. | |
Mar 03 | Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS. | |
Mar 03 | Discuss rechartering or ending working group. |
RFC | Status | Title |
---|---|---|
RFC2330 | I | Framework for IP Performance Metrics |
RFC2678 | E | IPPM Metrics for Measuring Connectivity |
RFC2679 | PS | A One-way Delay Metric for IPPM |
RFC2680 | PS | A One-way Packet Loss Metric for IPPM |
RFC2681 | PS | A Round-trip Delay Metric for IPPM |
RFC3148 | I | A Framework for Defining Empirical Bulk Transfer Capacity Metrics |
RFC3357 | I | One-way Loss Pattern Sample Metrics |
RFC3393 | PS | IP Packet Delay Variation Metric for IPPM |
RFC3432 | PS | Network performance measurement for periodic streams |
IETF IP Performance Metrics WG (ippm) Tuesday, March 2, 2004 at 15:45 to 16:45 ======================================== The meeting was moderated by the working group chairs, Matt Zekauskas and Henk Uijterwaal. Rick Whitner and Matt Zekauskas took notes, which were edited into these minutes by the chairs. AGENDA 1. Agenda Bashing 2. Milestone Updates 3. One-slide review of reordering metrics (no authors could attend) 4. IPPM-MIB 5. "Needs of IPPM metrics for non-e2e measures" 1. Agenda Bashing -- The Chairs There were no updates suggested. 2. Milestone updates -- Henk Uijterwaal for the Chairs Henk reviewed the milestones that are still open, and what the chairs feel should be done about them. We found one person who volunteered to write the bandwidth capacity definitions document, but he hasn't yet confirmed by email. To be deleted, unless someone steps up to do the work: * Sensitivity analysis of one-way delay and loss parameters * Comparison of ITU and IETF IP performance metrics * Applicability statement for one-way delay and loss * Bulk Transfer Capacity metric, CAP The OWAMP protocol document is mature, has one implementation, and reflects the current requirements document. Stanislav Shalunov feels it is ready, except that it needs more people to review, and it also needs a thorough security review. Looking for WG last call around July 2004. The packet reordering draft has undergone another revision (more later). It needs some clarifications, but has all major comments included at this point. One issue that hasn't been settled, however: what to do (if anything) with the reordering density draft. If there are no major shifts, we will be looking for a working group last call around IETF-60. Finally, there is the issue of advancing metrics. We are still stalled on a document describing metric progression; in addition the NEWTRK working group is looking at revising the standards track. The one thing that seems would be useful no matter what the outcome are implementation reports. We will be placing a call to the list that asks for implementation reports. What we believe would be useful are: * name of implementation * the organization (if any) creating the implementation * origin of code * the platform the code runs on * a contact * the metrics and features implemented * metrics and features not implemented, and why * tests done on the implementation, such as comparison with another implementation; verification of results with passive measurements; error estimation, clock source. 3. One-slide review of the reordering draft --Henk Uijterwaal, slide from Al Morton The biggest changes are reporting fragmentation and including examples of 'reordering free runs' from Jon Bennett (see slide). The given pseudo-code has become complex, and Al is thinking about designating it informative-only. Al would really like people to read the draft and comment on the mailing list. Matt Z asked if anyone had any additional comments on including fragmentation in the draft, instead of our traditional focus on complete packets only. The fragmentation was added to allow the metric to be used with router testing (so the same metric could be used within BMWG-style benchmarking). No one commented. 4. IPPM reporting MIB --Emile Stephan Emile reported on the changes since previous version, principally simplification by deleting a number of groups (see slides); thresholds added in AggrMeasureTabls; the textual convention IppmOwnerIndex added to clear up owner namespace; GMTTimestamp reverted back to GMT format Emile then gave some usage scenarios, giving a slide each on the data model, the proxy architecture, how metrics are aggregated, and how history size is limited. Emile asked for feedback on history size mechanism. Matt Z. commented that his understanding is that people don't want this kind of filtering, they want that to be part of the management application; when a threshold is specified, adds complexity; Emile replied that he only provides place for the threshold, not the actual value. It's not mandatory, and provides a balance between memory size and capacity. Emile then went on to present the three open issues he sees with the MIB. Open issue #1 - memory optimization; SNMP objects use much more memory than the value itself (like 10x). Polling shortlived entries requires too much bandwidth and makes the table unstable. How to limit the size of the history, how to reduce the poling freq? Proposal: optionally store results in file when buffer of a measure wraps and give application a URL of the file. Open issue #2 - NetMeasureTable access; concern over the access attributes of the table. One opinion is that the read-only characteristics of table violates the spirit of SNMP; should be read-create. Emile would like some opinions. Matt asked if anyone in the room had read the document other than the Chairs. Allison Mankin was in the room and had read the document, but otherwise noone volunteered that they had read the document. Thus, there was nobody in the meeting who had read the document and could provide feedback. Last issue is to specify the history buffers mechanism management, then Emile feels it is ready for first (last) call. Also Emile recommends implementing in SNMPv3. Without v3, management access won't be secure enough. 5. Needs of IPPM metrics for none2e measures - Emile Stephan The idea is that we should want precise one-way delay using the same methodologies across groups doing end to end and intermediate (possibly passive) delay measurements. Want aggregated one-way delay for both passive and active measurement. Passive measurement can be used to decompose end to end delay; the psamp wg is standardizing hash function for one-way delay measurement between two intermediary devices, and we need a common definition or this metric, and also what it means to concatenate segments measured passively. Proposal: define an IPPM spatial one-way delay. With regard to active measurement we need to be able to concatenate measurement results, and provide standard reporting. Proposal: standardize an IPPM aggregated one-way delay. We need to span measurement systems, currently no interoperability. Matt Mathis asked Emile about his motivation: is it oriented towards representation for easy interoperability, or do the semantics need to be fixed to make this scheme work? There are two problems. Different implementations of one way delay don't constrain measurement properties in ways that are comparable, for example by using different bins to report statistics. Secondly, if you don't represent the metrics in the same way it is cumbersome to do analysis. Emile responded that he thought about this and produced an individual draft on spatial metrics; and looked at instantaneous concatenation with concatenation based on the metrics performed. if you don't know the methodology, you can't concatenate. Matt said that it bears on the first question. He was afraid of issues of representation, which is cumbersome to do, but doesn't change anything we've already done. Emile said implementation issues are out of scope. Matt also pointed out that in the framework RFC we have a notion of an "A-frame". It has always been the vision of the group (the "holy grail") of being able to concatenate metrics (including BTC). This problem has proven to be much harder than anticipated initially. Emile noted that PSAMP will provide standards that essentially measure spatial delay. Matt noted that delay was easy to concatenate. Email pointed out that these tools will be available, so we should clearly distinguish what happens with these metrics. Matt noted that if we are getting into composition of metrics, as we outlined in the framework draft, it's a huge amount of work, at least as great as the work already done on the metrics themselves. It might be useful to "stick one toe into the pool". This is a large problem space, and maybe we just don't want to go there. Emile replied that he believes we have to address the boundaries now, i.e., what can be concatenated and what can't. With IPFIX and PSAMP we will have tools, people will share and combine -- we should say how. Matt replied that there are other kinds of concatenation that would be extremely valuable. If we wanted to make a metric which was an aggregate across a path across other provider, starting with a collection of endpoints. There would be a large number of singletons, and large number of paths across provider. In this example the issue is how to define a set of endpoints that make sense to measure. This is different. Email said that in his draft, that would be based on composition of metrics. Matt Mathis replied that this is another holy grail. What is the typical one-way delay across a provider. For example, how well does routing policy match geography. this is still an open question. The point is that if you are going to think about concatenation, then you need to consider other kinds of composition. It might be warranted to do one, and then decide not to do others now. Again, this is a much bigger space than the group has already tackled. Matt Z. said that the composition thing is a research topic; it's not good enough to say you want to do the composition, we need to have an idea of how to do the composition. On the other hand, an applicability statement of the one-way metric to passive measurement could be useful. No other comments. Meeting closed. |