IP Performance Metrics WG (ippm)
Monday, August 1, 2005 - 10:30--12:30
The meeting was chaired by Henk Uijterwaal and Matt Zekauskas. Dave Plonka
and Al Morton took notes, which were edited into these minutes by the
1. Administrivia: Agenda, Status of Drafts and Milestones
2. Implementation reports update
3. Capacity draft
5. Traceroute storage
6. Multi-point metrics
7. Composition of metrics
8. Two-way Active Measurement Protocol
1. Administrivia: Agenda, Status of Drafts and Milestones
Henk Uijterwaal opened up the meeting by reviewing the agenda, and
then the status of outstanding drafts (see slides). There were no
audience requests to change the agenda (although the multimetrics and
composition requests were flipped at the request of the speakers
later, and the eighth item was added just before the meeting). The
metrics registry is in AUTH48; The OWAMP draft has been submitted to
the IESG, there have been some minor changes in response to IESG
comments (including minor on-the-wire changes); the draft is currently
being discussed with the security AD; the authors are currently
waiting for feedback; other drafts are discussed below.
2. Henk Uijterwaal: Implementation reports update
Henk then segued into the implementation reports draft. Henk has some
small updates pending for the draft, then it appears to be done. Matt
said he would try to get another system (Internet2's "owamp" code) into
the draft although there appear to be enough implementations to recommend
advancement. The chairs will discuss with the proper disposition of
this draft with the AD.
3. Matt Zekauskas for Phil Chimento and Joe Ishac: Capacity draft
Matt gave a presentation by Phil Chimento and Joe Ishac on the
capacity draft. The purpose of this draft is to provide clear
and precise capacity definitions. Two outstanding issues brought
up on the mailing list along with proposed changes (also on the
mailing list) were shown.
Matt personally expressed happiness with the direction of the current
Al Morton asked about the concept of "valid IP bits" - what does this mean
for payload? It assumes that a simple match of byte count; Al doesn't think
this is enough [to say its valid]. Matt said that he thought that since
we are showing capacity and utilization, any bits are probably fine.
Al suggested comparing the definition with the rest of the framework.
Emile Stephan wondered if we would have some metrics as the result of
this draft. Matt stated that he thought we *could*, following the
precedent of bulk measurement. Emile thought that what was proposed
was massively intrusive and the capacity definitions are not compatible
with the framework. Phil (via Jabber) stated that the document is
only there to set definitions.
Farooq Bari asked if the authors intended to do anything above IP
(such as TCP). Matt said that he believes no; Phil confirmed via
jabber that the current document is only intended to be definitions.
The same audience member thought it might be interesting to see TCP
capacity addressed because of retransmissions.
4. Al Morton: Update on reordering metric draft
Al Morton gave an update on the reordering draft, and walked through
a histogram example to show the difference between the proposed
byte offset with the Colorado State reordering density metric
(in response to a PDF posted by Nischal Piratla to the list the
night before the meeting).
The comparison was a direct comparison between Reordering Byte Offset
in the current draft, compared with Reordering Buffer Density (RBD). Al
felt that Byte Offset is more directly focused on reordering, since
it gives the buffer size that would accommodate different proportions
of reordered packets, while RBD gives buffer occupancy over all
packets in the stream.
About five people in the audience raised their hands when Al asked
who had read the draft.
Henk asked how we should proceed with this draft? All comments have
been respond to except the PDF posted last night. Should we declare
it finished and move to last call? There was no audience response.
Henk said he would ask again on the mailing list. Al said that
in summary, many people have read the draft, and said they were mostly
satisfied, and any comments that were made were incorporated. He feels
that the comment last night is a comparison between the metrics, not
saying what we're doing is wrong, but others may be better in some
respects. Most are happy with what we have, and a few folks have
other ideas. Henk agreed with the summary, and felt that there was
rough consensus now to move to Last Call. [Note: on Jabber,
Mark Allman asked how the byte offset and RBD metrics "square"; Henk
had closed the topic and said the question would be taken off-line.]
5. Juergen Quittek - Traceroute Metrics and Data Model (see slides)
Juergen Quittek talked about a traceroute data model and exchange
format that they would like to see standardized (hence taken up
by the working group). They surveyed other measurement repositories;
CAIDA has 7 years of data, but said they suggested that their
format not be used. The proposed works is more of an exchange format
than a metric; the draft is now in it's second revision. The names
in the information model align with the other traceroute work in
the IETF in the DISMAN traceroute MIB. So far they prefer an XML
Henk asked for comments. Jerome Durand said that the Global Grid Forum
Network Measurement Working Group had common ways to represent metrics
in XML. Juergen said that he wasn't aware of the work; Jerome said he
would post a pointer to the list. (Comment from Matt: the GGF operates
in a manner similar to the IETF.)
Emile Stephan thought that a common representation for traceroute was
a good idea, and he wondered why we don't use an IPFIX representation
rather than ASCII/XML (i.e. use the IPFIX data model). Juergen stated
that it is not necessarily obvious to use passive IPFIX representation
for an active traceroute measurement, but agreed it is an open point.
Emile also asked if there is already a MIB, what is the purpose of
this document? Juergen responded that the MIB doesn't suggest how
to store it locally, nor how to send the results to a location other
than where the measurement originated.
Juergen also noted that generalizing IPFIX was denounced by the ops ADs
(in light of current workload). Juergen asked if it could be made a working
Someone on jabber asked how to interpret the symbols on slide 6:
+ means well suited, - means not suited, 0 means "don't care".
Matt Mathis felt that this would be useful to find landmarks for measuring
performance in multi-provider environments. He believes this work is critical,
and should be done somewhere. He mentioned that he was aware of the GGF work,
but it might be useful to do the work twice as a sanity check. He just wanted
to make an observation, not advocate the work be done in the WG at this
Dave Plonka felt that transport of traceroute is probably in scope for IPPM,
but his inclination is to say storage format is not. He also said that
the IPFIX working group will touch on it this afternoon.
Matt and Henk both echoed that they felt it is unclear whether the storage
(serialization) format is in scope. Henk agreed that it is important
that this work is done somewhere. Emile thought that there should be
a discussion with PSAMP and IPFIX and see if there was a general solution
for the export of measurement data. Juergen said that PSAMP and IPFIX
have a binary export format; IPFIX does not define how to store the info,
i.e. files are not exchangeable (necessarily), which is why they didn't go
to IPFIX. Matt thought that there was general agreement the work should
be done somewhere, but the question is where.
6. Lei Lang: Multi-point metrics
Lei Liang gave a presentation provocatively titled "IPPM Metrics for
IPTV performance and QoS measurement" (see slides). Simplistically,
the multimetrics draft looks at broadcast multicast streams, and
reports on performance, either "one-to-many" (where you report on
end-to-end results) or "spatial" (where to also report on performance
to intermediate points) (which could also be used for a unicast
stream). They used broadcast IPTV as a motivator.
Keyam Hedayat said that he had not read the draft, but one of the most
important IPTV metrics is channel change time. Many people claim
proprietary solutions for < less than 10ms change times for example;
this has to do with multicast join performance. Have you considered
this is a metric? Lei said that they had not considered this.
Emile thought such a metric was very needed; he wondered if one
could start with a BMWG multicast benchmark.
Farooq Bari asked if the spatial metrics on slide 5 could be defined in
generic form, not specific to IPTV? If so, why not do that instead?
Why would IETF want to standardize service-specific metrics for
QoS, if general ones are sufficient? Lei was initially confused
by the question; Matt restated as: "why are you developing an
IPTV-specific metric, rather than something general?"; Matt then
answered his own question, and said that from the presentation, they
are general metrics and IPTV was just the motivation. Lei agreed.
Someone else said that we should be careful about using the term QoS,
since it has a specific other meaning(s) elsewhere in IETF.
Emile said that to clarify: this work defines two kinds of metrics,
spatial and also one-to-group metrics.
Al Morton noted that with respect to channel changing time, the IGMP
join flows up tree toward source, and this work assumes there is a
single source packet going down the path, so that may be out of scope.
Matt Mathis thought this seems service provider oriented, and one way
(one-to-many), not symmetric. He wondered, especially in the one-to-group
case is there a straightforward way to generalize it for "subscriber use":
many-to-one. For example, measuring the best path (say based on loss
rate) from a large number of web servers to a user. Lei thought it
could be done. Matt Mathis asked for clarification: do you mean
group-to-one measurements will be easy to define, but they are not
in the draft? When evaluating service providers, that would be useful.
There was concern that 1-many might not be part of the IPPM framework,
which usually follows a single source packet through the network.
Matt Mathis noted that the time interval you care about might be
hours or a day... if shopping for an ISP, you want to know how well
connected to the rest of the world it is. If the time interval was
a day, you might not care about synchronization.
Al asked if this is of interest to IPPM? Henk deferred that question
until after the next presentation, which seems related.
7. Al Morton: Composition of Metrics
Al Morton gave a presentation on composing metrics -- multiple
independent measurements along a given path (see slides).
Concatenation in space and aggregation in space and time was
considered. (Al noted that he wrote the draft in an evening; many
ideas came from the five co-authors listed at the front of the draft,
but he did not have a chance to review the draft with them before the
-00 deadline.) The presentation co-author (Maurizio Molina) is active
in the GEANT2 Joint Research Activity devoted to measurement (JRA1),
and they have a milestone on metric composition. JRA1 has
concentrated more on temporal composition; this draft focuses mostly
on spatial composition.
Al asked if there interest to take this up as a work item?
He see this as a separate item from the previous multimetrics presentation.
Emile said that he doesn't agree with the average summation example, but
won't make a big issue at this time. He asked for clarification,
are we only going to define spatial or both spatial and temporal
compositions? Al said that this draft just focuses on spatial
composition; JRA1 is working on temporal composition.
Matt Mathis thought his sort of work is exactly what the group was
thinking of with the AFRAME concept in the framework. One obvious
metric that is missing is reordering... the mathematics of compositions
of reordering is complex. It may be too complex to add to this composition
work. May need to separate metrics that have certain engineering properties
and others that have certain mathematical properties. There is a fundamental
difference between the composition of symmetrical and non-symmetrical
metrics. Reordering (in TCP, for instance) is not symmetrical with
respect to composition. This may be useful to think through to formalize
the metrics you wish to. Al also mentioned steady-state versus transient
reordering; composition is likely easier with a steady-state assumption.
Matt also mentioned a related problem - what data thinning does. Say
two streams mix, the aggregate is reordered, and then the two streams
separate. What does this mean?
Al said he left reordering out of the draft because he still feels it's
a research topic.
Henk felt that there is interest in both the previous work (Lei's
presentation) and this. We will ask the mailing list, but thinks
the group should pick up; the chairs will talk to the AD about adding
Steven Van Den Berghe felt it would be nice to have a new framework,
and fit both drafts into this composition framework. He thought the
same issues will be recurring in both drafts. Al said that right now
there isn't much framework (for spatial); it could be copied to both
drafts. Steven was encouraged to send in a text contribution.
8. Kaynam Hedayat: Two-way Active Measurement Protocol (no slides)
Kaynam gave a brief update on the TWAMP draft. He said that
the draft itself hasn't changed, although he just reissued it
in anticipation of being a working group document. There
has been an implementation of the draft; if anyone out there
has another implementation and would like to test interoperation
With that, the meeting was closed.