IP Performance Metrics WG (ippm) Wednesday, March 21, 2007 - 13:00--15:00 ======================================== The meeting was chaired by Henk Uijterwaal and Matt Zekauskas. Al Morton, Ruediger Geib, and Martin Swany took notes, which were edited into these minutes by the chairs. AGENDA: 0. Administrivia 1. Status of milestones and drafts 2. Drafts about to go to last call a. Bandwidth Capacity Draft b. TWAMP c. Traceroutes draft 3. Work in progress a. Composition drafts b. Multimetrics draft c. Reporting drafts d. Delay Variation AS draft 4. Possible new work a. Application Loss Pattern Metrics b. Duplication metrics 5. AOB 0. Administrivia Henk Uijterwaal opened the meeting asking for agenda changes. There were two agenda modifications to accommodate schedules -- move multimetrics up and delay traceroutes to the end. [In addition, before the meeting, a few people noted that the conflict between ippm and opsarea was draining some group members.] 1. Status of milestones and drafts Since the last meeting, we have one new RFC - 4373. The chairs need to resolve Implementation draft status with ADs. The capacity draft has addressed (some ops area) AD comments, we expect another draft followed by a second last call soon. The milestones almost all are overdue, and will be updated and resolved based on the results of this meeting. 2. Drafts about to go to last call a. Bandwidth Capacity Draft draft-ietf-ippm-bw-capacity-04.txt None of the authors were able to attend, but Henk reported we expect a -05 version soon, and that will be followed by a short WGLC. b. TWAMP (Kaynam Hedayat) draft-ietf-ippm-twamp-03.txt Kaynam reported that the authors felt the draft was ready for Last Call. Matt Z said that he had re-read the draft with a fine-toothed comb, and he had several comments, that he intended to also put in a message to the list. First, TTL values are not kept by TWAMP. This was a late addition to OWAMP. Several in the room agreed it should be added, and Kaynam agreed. Second, TWAMP Lite has implications on the security section. It's certainly not intended behavior, but since there need be no control handshake, the reflector could reflect traffic anonymously, which is something that can't be done with a full TWAMP implementation, or OWAMP implementation. Thus it isn't covered by a blanket "see OWAMP for security concerns". Perhaps requirements need to be placed on the undocumented protocol, or implementations of TWAMP Lite. Third, Matt wasn't sure the one-paragraph security section would make it through the IESG. This led to a general discussion about the section. It was generally agreed that duplication was not generally helpful, so truly duplicate concerns could be included by reference. Lars agreed. The strategy will be to add anything new for TWAMP Lite, and do another general review that there isn't anything else that should be added. Fourth, TWAMP adds a new number to the Request field. Should this be in a registry? Matt didn't envision any further additions, and asked for comment. Lars thought that if we have a protocol field that has multiple RFCs defining values we need a registry. Matt will help Kaynam on what to put in the IANA Considerations section. 3. Work in progress b. Multimetrics draft (Lei Liang) * draft-ietf-ippm-multimetrics-02.txt Lei reported on current draft revisions. There is a new section on the impact of packet loss on the statistics. There are many new one-to-group metrics in a new section. There is a new discussion of measurement methods, in particular on scalability and reporting considerations. Ruediger Geib thought there should be more emphasis on temporal statistics. He also asked if the authors were aware of the MINC project at ATT from a few years ago. Nick Duffield had some interesting multicast results. Ruediger will provide a link to the MINC work and provide comments on the mailing list. Alan Clark noted that RTCP-XR was developed to support some of the Minc project metrics, and the group should also take a look at that. Al said that the authors were looking for more volunteers reading and commenting the draft. Juergen Quittek promised to provide a review. a. Composition drafts (Al Morton, 10') * draft-ietf-ippm-framework-compagg-03 * draft-ietf-ippm-spatial-composition-02 Al Morton reported on the composition drafts. He started with the framework. See the slides. The drafts in "green" are in good shape. The spatial composition draft has been reorganized. For the temporal draft, there is currently an author team of two people, and they would like more people to join them, as well as more readers. Work on the framework is on hold while the detailed draft is being worked on. Ruediger offered to share a minor comment on the terminology section on the list. Al then went on to the spatial composition draft, which was submitted after the deadline, and just posted, so probably not many had read. The common stuff has been pulled up front. There is a new metric for delay variation that is an approximate convolution. There's also another one which is similar to one in Y.1541 (and there is a reference). Al would really appreciate some thought and comment on these two, and what should be done. Benoit Claise and Alan Clark both volunteered to read. c. Reporting drafts * draft-ietf-ippm-reporting-01.txt * draft-morton-ippm-reporting-metrics-01.txt (Al Morton) Matt Z reported that Stanislav has switched jobs, and will no longer be able to edit the main reporting document. However, he has been working with Martin Swany, who has volunteered to edit. Martin reports that he wants to get a draft (with some pending revisions) out by May. Martin will get together with Al Morton to ensure all his concerns are addressed. Al then gave a short talk on his reporting metrics draft. It has not been updated from last time. He summarized the draft (see slides). Al again noted that an important issue is the treatment of delay of lost packets. In this draft lost packets are treated as having undefined (instead of infinite) delay, thus allowing for orthogonality needed for some of the composition drafts. People must read this and comment; if this is not acceptable it throws the other work into question. This is what most implementors are doing anyway in the field; if providers or test equipment vendors are doing this (and apparently many are) Al would like them to speak up on the mailing list. Alan Clark noted that he agreed with most of the points on Al's slide 3; especially that the industry needs to move away from "average jitter". On the long loss threshold, Alan noted that it depends on what you want to measure. If you have test boxes, a long delay is fine. If tests are embedded in appliances, you may need to keep the threshold fairly high. Al noted that his was definitely a tradeoff. Alan thought that from a practical perspective, if you were interested in application performance that the time would be small. And if you embed metrics into routers, or session boarder controllers, there is finite memory, so the threshold may be limited. Al noted this also applies to the delay variation applicability statement, next. Dave McDysan wanted to make sure Al wanted comments from providers, and said he would provide some. d. Delay Variation AS draft (Al Morton, 15') * draft-morton-ippm-delay-var-as-02.txt Al then moved on to the delay variation applicability statement draft. Benoit Claise commented on, and then supplied text for enough material that Al made him a co-author. There is a lot of new material in this draft. Benoit saw an applicability statement as a potential "point of entry" to the IPPM literature, and Al tried to provide some introductory material and pointers. Section 7, and later, have most of the new material. There was discussion about the relative merits of packet delay variation (PDV) versus inter-packet delay variation (IPDV). Al has found that PDV has better properties (see slides, and draft text). He would like feedback. Alan Clark agreed that PDV is generally more useful than IPDV, except in one case. If applied to applications traffic, and in particular IPTV, the effect of smoothing of IPTV streams (in particular the timestamps in RTP) causes strong delay variation to be introduced. This effect is better separated from network caused congestion by IPDV. IPDV tells you more of what the network has done. The application effect is caused by using CBR with large I-frames. Alan will send a reference to the list. Kaynam thought there was a lot of confusion in the industry, and this draft was good work. He agreed with Alan, that PDV is generally a more useful metric, simply because applications use PDV to characterize jitter buffers. He doesn't see major benefits to IPDV. Dave McDysan said that he agreed. PDV is operationally more useful. It maps to de-jitter buffers. However, Dave felt that one thing not in the draft is a relation to routing impairment; routing loop detection can be performed using delay variation. Al noted that the type of reordering seen is a unique indicator of routing loops. Al asked for comments on making this a working group draft. A new participant asked if congestion control designers consider delay variation along with delay, since variation gets high towards the end of slow-start periods. Lars noted that TCP does this, but it's not used from a TCP-friendly rate control point of view. Kaynam said he would try to get Roman Krzanowski to comment, since he was the instigator of the whole effort. He reiterated that he thought it was a good draft. 4. Possible new work a. Application Loss Pattern Metrics (Kaynam Hedayat) draft-venna-ippm-app-loss-metrics-00 This is a draft written after a presentation at the last IETF. Kaynam felt there was a need for yard-stick metrics in the industry, and this tries to show the applicability of RFC 3357 to the "media layer". The purpose is to get a "media errored seconds" metric. Ruediger Geib read from a TMF Forum (TeleManagement Forum, http://www.tmforum.org/) publication that said that packet-level measurements cannot provide estimates of video quality, so what was the purpose of this document. Kaynam says that's not the purpose of this document, here we are just trying to apply RFC 3357 metrics at the media layer. Alan Clark disagreed with the draft contents, but not the concept. He felt the group (or some group) should look at how IPPM metrics apply to the performance of all the applications and protocols above, like TCP. This document focuses too much on video metrics; and the there are other efforts defining metrics in the IETF , some in AVT. Further TV streams carry frames of different importance so simple frame loss doesn't characterize the impact of packet loss sufficiently. There are also efforts outside the IETF to produce metrics. Kaynam said the draft is just on media stream, not the video result. It is not meant to be video-specific, but uses video as an example. He will see about making that clear. He also felt that AVT should not be defining metrics, but instead defining how to report metrics. There is a "missing bridge" between IPPM and groups like AVT, this document is one attempt. Alan noted that AVT does have the expertise in IETF on application performance; Kaynam said there needs to be a bridge between the groups and sort out the charters. Henk agreed that coordination needs to take place. Lars as AD said that the current IPPM charter excludes this work, in his opinion; we would need to re-charter. Further, we are behind on some work, so we should focus on that. Maybe a BOF on this is the right thing rather than putting the effort in either IPPM or AVT. Kaynam noted that RTCP-XR applies these metrics. Al Morton noted that people were also asking about "SIP performance parameters", and that is another kind of application performance. Ruediger felt that a purely application performance working group wouldn't be successful. Lars also noted that FECFRAME has some related work, since that deals with packet loss impact, and protects applications from them. Alan Clark felt it was a good idea. Jim Van Loo asked if there are no standards definitions of application correctness? Kaynam said he know of no standard ones. but there are some out there. One is the Media Distribution Index, which is an RFC, but not a standard, which was felt to not be very good. In Alan's opinion, another MDI is not what was needed. By the time we tried to replace it with an RFC, there will be some no-reference QOE metrics for video already in the standards of ITU-T (he estimated by 2008). b. Duplication metrics (Henk Uijterwaal) Henk presented thoughts based on the fact that as we were working on the reporting metrics draft, we found that there was no definition for packet duplication, and the reporting metrics draft wasn't the right place to define one. Henk proposes a definition that is basically a count of copies of the same packet, corrected for loss. Al thought this was useful. He did note that as a summary metric the number of duplicated packets was ambiguous. For example, was one packet duplicated 100 times, or were 100 packets duplicated once? Henk reported that the original idea was a simple single number that would give a simple indication of quality. Al noted that Y.1540 distinguishes Replicated Packets from Duplicated Packets; Al will send the ITU-T draft definitions to the list. It was felt that the ideas were useful enough to turn into a draft. 2. Drafts about to go to last call, revisited c. Traceroutes draft (Jurgen Quittek) draft-ietf-ippm-storetraceroutes-03 Juergen Quittek reported on the traceroute storage draft. The draft is now compliant with OGF standards. However, a description of traceroute is required. A new version will come in April, and he hopes for a WGLC then. Lars mentioned his comments to the list, that the schema doesn't describe how traceroute is parameterized, you have to know how the traceroute works - needs the foundation. The draft also needs to explain terms like traceroute probe. Add terminology as a section or while explaining how traceroute works. 5. AOB. Al Morton reported on current events within ITU-T, and in particular IP performance work being performed within Question 17 of Study Group 12. The IP performance document, Y.1540, is being revised. Replication and duplication is being added, and delay variation is being refined. In addition multicast performance is being considered. The charter of Q17 is wider than IPPM, and they can look beyond the IP layer. They are looking at session setup and teardown for IGMP. There is also some work related to Ethernet performance. They have been encouraged to rationalize Ethernet performance parameters. One set has been defined by the Metro Ethernet Forum, but their definition is opposite from everyone else's use. The MEF will be working on this too. The MEF metrics are very percentile-oriented, and include pointers to distributions. Al will look into the possibility of setting up a Liaison FTP area so that IPPM members could see, and potentially comment on the work in Question 17; if not, the work could be shown to IPPM through the liaison process. Alan Clarke wanted to add a few points. He believes SG12 is also taking bursty gap metrics from RFC 3611 and putting them in Y.1540. Within the AVT group, there is some interest in expressing sparse bursts of loss. Perhaps that is something else that could be added to IPPM documents. If this was of interest, he would be willing to write a draft. There was also question in the AVT group yesterday relating to G.1050, and Alan had sent that to the IPPM list. This relates to work that Alan did four years ago, using time-series models to represent packet delay variation; it generates nice models of packet behavior. There is an open implementation that can be used for experimentation, and the technique was picked up by the ITU. There were no other comments, and the meeting was closed.