[ippm] 00 version of IPPM minutes from IETF 68

[Matthew J Zekauskas <matt at internet2.edu>]

I have not uploaded this version.  It's a bit draftier than my usual
first draft; you'll note some questions usually within [brackets].
However, I don't have much time to work on it for the next week, so I
thought I would toss it out for comment.  I have not spell checked it,
so there are probably spelling errors too.... don't concentrate on
those, but let me know if I made any factual errors.

[Recall: all the presentations have been uploaded and are available at
<https://datatracker.ietf.org/public/proceeding_interim.cgi?meeting_num=68>,
search for "ippm".]

--Matt

ps: Al: I made this before your additional notes reached me.


IP Performance Metrics WG (ippm)
Wednesday, March 21, 2007 - 13:00--15:00
========================================

The meeting was chaired by Henk Uijterwaal and Matt Zekauskas. 
Al Morton, Ruediger Geib, and Martin Swany took notes, which were
edited into these minutes by the chairs.


AGENDA:
0. Administrivia
1. Status of milestones and drafts
2. Drafts about to go to last call
    a. Bandwidth Capacity Draft
    b. TWAMP
    c. Traceroutes draft
3. Work in progress
    a. Composition drafts
    b. Multimetrics draft
    c. Reporting drafts
    d. Delay Variation AS draft 
4. Possible new work
    a. Application Loss Pattern Metrics
    b. Duplication metrics
5. AOB



0. Administrivia

Henk Uijterwaal opened the meeting asking for agenda changes.  There
was one agenda modification -- move multimetrics up so Lei Lang could
attend another session.  [In addition, before the meeting, a few
people noted that the conflict between ippm and opsarea was draining
some group members.]




1. Status of milestones and drafts

Since the last meeting, we have one new RFC - 4373.  The chairs need
eed to resolve Implementation draft status with ADs.  The capacity
draft has addressed (some ops area) AD comments, we expect another
draft followed by a second last call soon.

The milesotnes almost all are overdue, but there are plans to resolve.




2. Drafts about to go to last call
    a. Bandwidth Capacity Draft
       draft-ietf-ippm-bw-capacity-04.txt

None of the authors were able to attend, but Henk reported we expect a
-05 version soon, and that will be follwoed by a short WGLC.


    b. TWAMP (Kaynam Hedayat)
       draft-ietf-ippm-twamp-03.txt

Kaynam reported that the authors felt the draft was ready for Last
Call.  Matt Z said that he had re-read the draft with a fine-toothed
comb, and he had several comments, that he intended to also put in a
mesage to the list.  First, TTL values are not kept by TWAMP.  This
was a late addition to OWAMP.  Several in the room agreed it should be
added, and Kaynam agreed.

Second, TWAMP Lite has implications on the security section.  It's
certainly not intended behavior, but since there need be no control
handshake, the reflector could reflect traffic anonymously, which is
something that can't be done with a full TWAMP implementation, or
OWAMP implementation.  Thus it isn't covered by a blanket "see OWAMP
for security concerns".  Perhaps requirements need to be placed on the
undocumented protocol, or implementations of TWAMP Lite.

Third, Matt wasn't sure the one-paragraph security section would make
it through the IESG.  This led to a general discussion about the
section.  It was generally agreed that duplication was not generally
helpful, so truly duplicate concerns could be included by reference.
Lars agreed.  The strategy will be to add anything new for TWAMP Lite,
and do another general review that there isn't anything else that
should be added.

Fourth, TWAMP adds a new number to the Request field.  Should this be
in a registry?  Matt didn't envision any further additions, and asked
for comment.  Lars thought that if we have a protocol field that has
multiple RFCs defining values we need a registry.  Matt will help
Kaynam on what to put in the IANA Considerations section.




    c. Traceroutes draft (Jurgen Quittek)
       draft-ietf-ippm-storetraceroutes-03

Slides are available, but Jurgen is not in the room.  We will revisit
this later.




3. Work in progress
    b. Multimetrics draft (Lei Liang)
       * draft-ietf-ippm-multimetrics-02.txt

Lei reported on current draft revisions.  There is a new section on
the impact of packet loss on the statistics.  There are many new
one-to-group metrics in a new section.  There is a new discussion of
measurement methods, in particular on scalability and reporting
considerations.

Ruediger Geib thought there should be more emphasis on temporal
statistics.  He also asked if the authors were aware of the MINC
project at ATT from a few years ago.  Nick Duffield had some
interesting multicast results.  Ruediger will provide a link to the
MINC work and provide comments on the mailing list.

Alan Clark noted that RTCP-XR was developed to support some of the
Minc project metrics, and thegroup should also take a look at that.

Al said that the authors were looking for more volunteers reading and
commenting the draft. Juergen Quittek promised to provide a review.




    a. Composition drafts (Al Morton, 10')
       * draft-ietf-ippm-framework-compagg-03
       * draft-ietf-ippm-spatial-composition-02

Al Morton reported on the composition drafts.  He started with the
framework.  See the slides.  The drafts in "green" are in good shape.
The spatial aggregation draft has been reorganized.   For the temporal
draft, the authors would like more readers (and contributors).  Work
on the framework is on hold while the detailed draft is being worked
on.  There has been a single comment from Ruediger on the terminology
section, but it's minor.

He then went on to the spatial composition draft, which got in just after
the deadline, and was just released, so probably not many had read.  The
common stuff has been pulled up front.  There is a new metric for delay
variation that is an approximate convolution.  There's also another one
which is similar to one in Y.1541 (and there is a reference).  Al would
really appreciate some thought and comment on these two, and what should
be done.  Benoit Claise? and Alan Clark both volunteered to read.




    c. Reporting drafts
       * draft-ietf-ippm-reporting-01.txt 
       * draft-morton-ippm-reporting-metrics-01.txt (Al Morton)

Matt Z reported that Stanislav has switched jobs, and will no longer
be able to edit the main reporting document.  However, he has been working
with Martin Swany, who has volunteered to edit.  Martin reports that
he wants to get a draft (with some pending revisions) out by May.
Martin will get together with Al Morton at the meeting to ensure all
his concerns are addressed.

Al then gave a short talk on his reporting metrics draft.  It has
not been updated from last time.  He summarized the draft (see slides).
Al again noted that an important issue is the treatment of delay of
lost packets.  In this draft lost packets are treated as having undefined
(instead of infinite) delay, thus allowing for orthogonalithy needed
for some of the composition drafts.  

People must read this and comment; if this is not acceptable it throws
the other work into question.  This is what most implementors are
doing anyway in the field; if providers or test equipment vendors are
doing this (and apparently many are) Al would like them to speak up on
the mailing list.

Alan Clark noted that he agreed with most of the points on Al's slide;
the industry needs to move away from "average jitter".  On the long
loss threshold, Alan noted that it depends on what you want to
measure.  If you have test boxes, a long delay is fine.  If tests are
embedded in appliances, you may need to keep the threashold fairly
high.  Al noted that his was definitely a tradeoff.  Alan thought that
from a practical perspective, if you were interested in application
performance that the time would be small.  And if you embed metrics
into routers, or session boarder controllers, there is finite memory,
so the threshold may be limited.  Al noted this also applies to the
delay variation applicability statement, next.  Dave McDysan wanted
to make sure Al wanted comments from providers, and said he would
provide some.




    d. Delay Variation AS draft (Al Morton, 15')
       * draft-morton-ippm-delay-var-as-02.txt


Al then moved on to the delay varation applicability statement draft.
Benoit Cliase? commented on, and then supplied text for enough
material that Al made him a co-author.  There is a lot of new material
in this draft.  Al saw an applicability statement as a potential
"point of entry" to the IPPM literature, and tried to provide some
intraductory material and pointers.  Section 7, and later, have the
new material.

There was discussion about the relative merits of packet delay
variation (PDV) versus inter-packet delay variation (IPDV).  Al has
found that PDV has better properties (see slides, and draft text).  He
would like feedback.  Alan Clark agreed that PDV is generally more
useful than IPDV, except in one case.  If applied to applications
traffic, and in particular IPTV, the effect of smoothing of IPTV
streams (in particular the timestamps in RTP) causes strong delay
varation to be introduced.  This effect is better separated from
network caused congestion by IPDV.  IPDV tells you more of what the
network has done.  The application effect is caused by using CBR with
large I-frames.  Alan will send a reference to the list.

Kaynam thought there was a lot of confusion in the industry, and this
draft was good work.  He agreed with Alan, that PDV is generally a
more useful metric, simply because applicatoins use PDV to
characterize jitter buffers.  He doesn't see major benefits to IPDV.
Dave McDysan said that he agreed.  PDV is operationally more useful.
It maps to de-jitter buffers.  However, Dave felt that one thing not
in the draft is a relation to routing impairment; routing loop
detection can be performed using delay varation.  Al noted that
reordering is another way to see routing loop effects.

Al asked for comments on making this a working group draft.

A new participant asked if congestion control designers consider delay
variation along with delay, since variation gets high towards the end
of slow-start periods.  Lars noted that TCP does this, but it's not
used from a TCP-friendly rate control point of view.

Kaynam said it would try to get Roman Krzanowski to comment, since he
was the instigator of the whole effort.  He reiterated that he thought
it was a good draft.




4. Possible new work
    a. Application Loss Pattern Metrics (Kaynam Hedayat)
       draft-venna-ippm-app-loss-metrics-00

This is a draft written after a presentation at the last IETF.  Kaynam
felt there was a need for yard-stick metrics in the industry, and this
tries to show the applicability of RFC 3357 to the "media layer".  The
purpose is to get a "media errored seconds" metric.

Ruediger Geib read from a TMN [[expand?]] Forum said that packet-level
measurements cannot provide estimates of video quality, so what was
the purpose of this document.  Kaynam says that's not the purpose of
this document, here we are just trying to apply RFC 3357 metrics at the
media layer.

Alan Clark disagreed with the draft contents, but not the concept.  He
felt the group (or some group) should look at how IPPM metrics apply
to the performance of all the applications and protocols above, like
TCP.  This document focuses too much on video metrics; and the there
are other efforts defining metrics in the IETF , some in AVT.  Further
TV streams carry frames of differnet importnace so simple frame loss
doesn't characterize the impact of packet loss sufficiently.  There
are also efforts outside the IETF to produce metrics.  Kaynam said the
draft is just on media stream, not the video result.  It is not meant
to be video-specific, but uses video as an example.  He will see about
making that clear.  He also felt that AVT should not be defining
metrics, but instead defining how to report metrics.  There is a
"missing bridge" between IPPM and groups like AVT, this document is
one attempt.  Alan noted that AVT does have the expertise in IETF on
application performance; Kaynam said there needs to be a bridge
between the groups and sort out the charters.  Henk agreed that
coordination needs to take place.  Lars as AD said that the current
charter excludes this work, in his opinion; we would need to
re-charter.  Further, we are behind on some work, so we should focus
on that.  Maybe a BOF on this is the right thing rather than putting
the effort in either IPPM or AVT.  Kaynam noted that RTCP-XR
applies these metrics.  Al Morton noted that people were also asking
about "SIP performance parameters", and that is another kind of
application performance.  Ruediger felt that a purely application
performance working group wouldn't be successful.  Lars also noted
that FEC[FRAME?] has some related work, since that deals with packet loss
impact, and protects applications from them.  Alan Clark felt it
was a good idea.

Jim Van Loo asked if there are no standards definitions of application
correctness?  Kaynam said he know of no standard ones. but there are
some out there.  One is the Media Distribution Index, which is an RFC,
but not a standard, which was felt to not be very good.  In Alan's
opinion, another MDI is not what was needed.  By the time we tried to
replace it with an RFC, there will be some no-reference QOE metrics
for video already in the standards of ITU-T (he estimated by 2008).






    b. Duplication metrics (Henk Uijterwaal)

Henk presented thoughts based on the fact that as we were working on
the reporting metrics draft, we found that there was no definition for
packet duplication, and the reporting metrics draft wasn't the right
place to define one.  Henk proposes a definition that is basically a
count of copies of the same packet, corrected for loss.  Al thought
this was useful.  He did note that as a summary metric the number of
duplicated packets was ambiguous.  For example, was one packet
duplicated 100 times, or were 100 packets duplicated once?  Henk
reported that the original idea was a simple single number that would
give a simple indication of quality.  Al noted that Y.1540
distinguishes Replicated Packets from Duplicated Packets; Al will send
the ITU-T draft definitions to the list.  It was felt that the ideas
were useful enough to turn into a draft.







2. Drafts about to go to last call, revisited
    c. Traceroutes draft (Jurgen Quittek)
       draft-ietf-ippm-storetraceroutes-03

Juergen Quittek reported on the traceroute storage draft.  The draft
is now compliant with OGF standards.  However, a description of traceroute
is required.  A new version will come in april, and he hopes for a WGLC then.

Lars mentioned his comments to the list, that the schema doesn't
describe how traceroute is parameterized, you have to know how the
traceroute works - needs the foundation.  The draft also needs to explain
terms like traceroute probe.  Add terminology as a section or while
explaining how traceroute works.






5. AOB.


Al Morton reported on current events within ITU-T, and in particular
IP performance work being performed within Question 17 of Study
Group 12.  The IP performance document, Y.1540, is being revised.
Replication and duplication is being added, and delay variation is
being refined.  In addition multicast performance is being considered.
The charter of Q17 is wider than IPPM, and they can look beyond the IP
layer.  They are looking at session setup and teardown for IGMP.
There is also some work related to ethernet performance.  They have
been encouraged to rationalize Ethernet performance parameters.  One
set has been defined by the Metro Ethernet Forum, but thier definition
is opposite everyone else's use.  The MEF will be working on this too.
MEF is very percentile-oriented, and includes pointers to
distributions.

Al will look into the possiblity of setting up a Liason FTP area
so that IPPM members could see, and potentially comment on the work;
if not, the work could be shown to IPPM through the liason process.

Alan Clarke wanted to add a few points.  He believes SG12 is also
taking bursty gap metrics from [???] and putting them in Y.1540.
Within the AVT group, there is some interest in expressing sparse
bursts of loss.  Perhaps that is something else that could be added
to IPPM documents.  If this was of interest, he would be willing
to write a draft.

There was also question in the AVT group yesterday relating to
G.1050, and Alan had sent that to the IPPM list.  This relates
to work that Alan did four years ago, using time-series models
to represent packet delay variation; it generates nice models of
packet behavior.  There is an open implementation that can be
used for experimentation, and the technique was picked up by
the ITU.

There were no other comments, and the meeting was closed.

_______________________________________________
ippm mailing list
ippm at ietf.org 
https://www1.ietf.org/mailman/listinfo/ippm