0=-=-=- IP Performance Metrics WG (ippm) Tuesday, July 24, 09:00--11:30 CDT This session was chaired by Henk Uijterwaal and Matt Zekauskas. Al Morton acted as scribe, and his notes (along with notes taken by Matt) were edited into these minutes by the Chairs. AGENDA: 1. Administrivia 2. TWAMP updates, and next steps 3. Duplication Draft Update 4. Composition Framework and Spatial Composition Additions 5. Multimetrics Draft Developments 6. Delay Variation Draft Developments 7. Any Other Business 1. Administrivia Agenda bashing, Scribe, Minutes, Blue Sheets Status of drafts and milestones Henk opened the meeting. There were no changes to the agenda. During the discussion of drafts and milestones, Henk menioned that we are still sitting on the implementation draft awaiting guidance on progressing metrics along the standards track. While initially the thought was we should just publish the docoument as informational, Lars skimmed draft-bradner-metricstest-02.txt, and it said metrics would progress when test equipment from different vendors had results in controlled experiment that were verifyably equivalent. However, the definition of "verifyably equivalent" is still open, and the results of that definition might imply that changes need to be made to the draft. Therefore we will let the draft sit, and encourage this group (and bmwg) to contribute to the definition of "verifyably equivalent". Henk asked if Al had any idea of realistic dates for his composition drafts, since the milestones for those drafts had passed. Al thought that the framework was in good shape, but didn't want to progress it before one of the composition drafts was ready. The big barrier is that it needs more review by working group members. We need some rigorous reviews. December 2007 is probably too soon, mid-2008 is a more realistic target. Folks in the group are requested to read and commend on the mailing list. Emile said that the multimetrics draft is more-or-less complete and has been stable. It too needs review, and he thought a WGLC would be the means to coerce folks to review. However, he has been speaking with Jurgen Quittek, and Jurgen has a series of comments he is going to send in. Therefore, he requests that the working group review the document after the next revision. Al offered to help with readability going in to the next version as well. The big issue with the draft are the passive aspects, which were discussed during the draft discussion below. 2. TWAMP updates, and next steps --Kaynam Hedayat http://tools.ietf.org/id/draft-ietf-ippm-twamp-04.txt http://tools.ietf.org/wg/ippm/draft-ietf-ippm-twamp/ Kaynam presented a single slide on TWAMP status. At the last meeting they agreed to add sneder TTL into test packets, as well as clarifying security text with respect to TWAMP-Light, and both have been done. In addition, text has been clarified for sender port and receiver port. Someone brought up setting DSCP on the ML. What does the group feel we should do? Matt asked if Kaynam had seen a use for it, and he said he did, but more for one-way than two-way tests. It seems like a fine idea, but should we make such a change now? No one felt strongly that we should do so. Therefore, no further changes for DSCP are proposed. This question was also asked on the list before the meeting, with no response. There were additional clarifications that were suggested on the ML and a new revision is in the works to address those. Matt mentioned that he was supposed to help the authors come up with some IANA text for OWAMP-Control commands, since this document adds a new value. He said he would do that by the end of the week. 3. Duplication Draft Update --Henk Uijterwaal http://tools.ietf.org/id/draft-ietf-ippm-duplicate-01.txt http://tools.ietf.org/wg/ippm/draft-ietf-ippm-duplicate/ Henk next presented updates on the duplication draft. Based on previous comments, a new version was created in April. There are two statistics, and two ways to express duplication. This draft spun off from the original reporting draft, which wanted to reference duplication but there was no definition. There is a placeholder Y.1540 section. However, references have been made along the way, so Henk does not see the need to keep the section. Al agreed, although he also stated that it might not be a bad idea to see where we could harmonize the definitions, and make the differnces go away (Y.1540 is currently under revision, see the final topic.) Al had reviewed the draft and made a few comments. He thought the names for some metrics might be change to be more descriptive. For example, "type-P-one-way-packet-duplication-average" was new in this version and didn't seem to capture the meaning of the metric. Al also noted that the real singleton here is not "duplication", but an arrival counter. It starts at 0, and goes to 1, 2, 3, etc, for each packet. Including "duplication" in the name seems like forming a conclusion that we haven't reached yet. WE should evaluate the arrival counter and see if a packet only once or more than once. Only those that arrive more than once should be called duplicates. Al offered to send his comments to the list. They may be found here: http://www1.ietf.org/mail-archive/ippm/current/msg01174.html 4. Composition Framework and Spatial Composition Additions -- Al Morton http://tools.ietf.org/id/draft-ietf-ippm-framework-compagg-04.txt http://tools.ietf.org/wg/ippm/draft-ietf-ippm-framework-compagg/ http://tools.ietf.org/id/draft-ietf-ippm-spatial-composition-04.txt http://tools.ietf.org/wg/ippm/draft-ietf-ippm-spatial-composition/ Al started by reviewing the composition framework. Overall, it's been fairly stable; the definitions have been refined, and a maintenance update has been performed. However, there has not been much feedback recently. Al said he is declaring this stable, pending feedback or one of the composition drafts to be complete. Al noted the Metro Ethernet Forum is specifying some end-to-end performance parameters, and they are going to need a solution to this problem as well. Jurgen Quittek and Emile Stephan offered to read the draft. Al then did an overview of the spatial composition draft, and the metrics that are defined there. The metric names have been simplified, allowing for easier exposition. Scott Bradner and Loki Jorgenson offered to read this draft. Matt admitted he had not read the draft recently, but the description triggered a thought -- are there auxilliary metrics or parameters to specifcy when a composition is valid? For example, min composition works as long as the path is relatively stable and uncongested. If the path is congested, the min will jump around. Al thought that it might just mean the "error bars" are larger, but you can still get an estimate of the minimum. Craig White remarked: Given a large enough sample size, the minimum will tend to be the one that occurs the most frequently, as long as the path is relatively stable. Even if the path is not stable, you do get some bimodal distributions (for example if a SONET ring goes to its protect side, and then back. Al said that bimodal distributions are dealt with in the document. You don't want to mix them together because the minimum for one is different than the other. The intent is to measure for a long time, if you see some path changes, segregate the measurements into different intervals. Emile said that in his opinion you can't consider making an aggregation that consists of measurements where the path changes -- the result would be unusable. Perhaps we need to say that in the framework or in the specific metrics. Al said that path stability needs to be mentioned when we talk about measuremetn validity; he thought it went to an overall section since it applies to all parameters. 5. Multimetrics Draft Developments -- Emile Stephan http://tools.ietf.org/id/draft-ietf-ippm-multimetrics-04.txt http://tools.ietf.org/wg/ippm/draft-ietf-ippm-multimetrics/ Emile started with a quick review of the multimetrics draft, starting with the spatial metrics, and then multicast metrics. For the spatial metrics, there has been some harmonization and discussion of scalability aspects. These metrics are used for either path composition or by the end user. Emile felt that these descriptions are now stable, and he didn't plan to make more changes except to clean up wording. The segment metrics have been renamed so that it is clear that they look at performance between any two hops on the network. They are purely passive. One participant had an issue with purely passive measurements, especially those based only on user packets. Did we want to discuss this now or later? Jurgen said that he had read the document, and think we run into problems if we focus on passive measurements. A lot of the terminology has been inherited from other drafts, and doesn't apply to passive measurements. If there are active probes, there is one sender and one receiver. If you passively examine user traffic, there are many senders and receivers that pass by. Emile thought that each flow had one sender and one receiver. There is some time when the packet was sent, even if it is not measurable. You want to know the performance between two points, you don't need to know the original sending time, you just need the two points. The terminology could be clarified. This would also provide psamp with some purely passive metrics; the section could be removed if it doesn't make sense. Emile thought that 80% of the wording is OK, and we need strong input, such as that Jurgen is giving. He would like a WGLC to force people to read the document. Al said that he wanted to make a pass over the document first, to improve grammar. He thought the document could benefit form an editorial pass to make it more readable, and as a co-author felt that he was responsible for that. After that, he agreed a WGLC would be a good idea. Al also wanted to know the groups point of view with respect to purely passive metrics. There was some discussion among the authors. The IPPM charter says that the "metrics will not characterize traffic", and thus would not be used to characterize VOIP traffic, for example. He reads this as passive is out of bounds for the working group. However, other people are entitled to give interpretations of that. Emile siad that it is not easy to mix passive and active metrics, but that we needed some ground truth. If we want to compose metrics, we will need to make approximations, and we need to include passive metrics in the algorithms. The spatial definition relies on passive technieuqs in nodes the packet cros. If there is an active source that provides packets that can be observed along the path, that was still an active measurement. The discussion is about qualifying the stream you are using for the basis of measurement. One opinion was that we haven't done this yet, and that there is enough material to create a separate draft on that topic. Lars (our area director) agreed that the charter text has an underlying notion that the metrics are based on active sampling. The working group could of course decide to take on passive measurements. The previous presentation goes in that direction. But he thought it was out of scope in the current charter, and he would like to see us finish the metrics and related documents on active techniques first. He thought passive measuremetns were a pretty large piece of new work -- in short, he agreed with Al. Emile reiterated his opinion that ignoring passive metrics is a mistake, and that the best way to have some "ground-truth" measurements is to have somee passive measurements to compare to. Alan Clark said he agreed with both positions, in that if you look at composition and define it using active measurements, the basic metrics are equally applicable to passive techniques. Work in composing delay, delay variation, and others, can be picked up and used with passive measurements. He felt that once the basic ideas have been developed, that the group should look at passive applications. However, it is difficult to make passive measurements of an IP stream, because you have to decode protocols, and that runs the risk of overlapping with other groups. Al Morton thought that was an interesting point; that the working group has focused on IP, dabbled a bit in TCP, but has focussed on active live measurements of the IP layer of the stack, and that's our core charter. If you work on passive metrics, and start to look at things like sequence numbers in higher layers, that definitely goes beyond our charter. Al also wanted to note that there are other gaps, places where people want to grow the group beyond the charter, both here and in BMWG. Interested folks should follow the "application performance metrics" BOF developments. Jurgen said that he agreed with Emile, that it would be good to have a methodology to apply. However, if the working group wanted to pursue this, it would take more than just a paragraph; there are many things to consier. There should be a framework for how we do passive measurements in general, before we apply them to any specific document. Emile feld we had to fix this before going further in composition, and that something was needed now to promote comparable metrics in this space. He also agreed that squeezing it in as a subsection of a draft isn't the right place for this to be. There was a comment that at least with the active measurements, we would be combining metrics where we understand the characteristics; if we combine passive measurements, there may be less we can rely on to combine them meaningfully. Emile noted that companies and people will do this combination anyway, so we should be working to defining standard metrics. There was a bit of further discussion on how large this effort might be and where the right place to do it would be; there was no definitive conclusion. Next, Karsten Fleischhauer presented a multimetrics use case provided by Ruediger Geib, who could not attend this meeting. The goal in this case it to improve multicast monitoring within a provider's network. It would give an operator increased understanding of what was happening with multicast, and an easy view of where failures occur. Several simple multicast trees, and failures were presented, along with showing how the metrics, especially the segment metrics, would improve understanding. (See slides for details.) Emile pointed out that the current draft does not mix segment metrics and multicast metrics, which these examples seem to do. Karsten said that his understanding is that there are no requests to change the draft, the example is just showing how the metrics can be used together. 6. Delay Variation Draft Developments -- Al Morton http://tools.ietf.org/wg/ippm/draft-morton-ippm-delay-var-as-03.txt Next Al Morton presented the delay variation applicability statement draft, which is not yet a working group item, but has had extensive review in the group. He spent some time presenting the problem space and showing how the different metric formulations are useful for different problems. See the slides for details. There was a recent section that made suggestions on what could be done to reduce variation. This provoked a discussion of whether the section was appropriate for the draft. Scott Bradner mentioned that there was a different set of expertise involved in fixing a problem than how to describe and measure the problem clearly. We would need to reach out to operators in different environments including enterprise and ISP for validation. He thought this section was better in a different document, or even a Wiki to keep advice up to date. Al noted that Roman Krzanowski, who first asked for this work, worked for an operator, and that Al himself was "tier-5 support" at his company. Wenji Wu asked about applicability of this work, combining it with loss to indicate congestion. Al said that the draft was not trying to combine delay variation with packet loss in this analysis; loss affects the delay variation metric, however. In the extreme case of losing every other packet you cannot produce a inter-packet delay variation result. Wenji said that if you have the minimum delay, and an indication of the maximum delay, then you have an indication of the minimum latency from source to destination, and an indication of queueing. Have there been thoughts to standardize this application? Al said that the draft includes an estimate of queue occupation, in orther words, this is a task that is already identified. Al asked that Wenji read the draft and comment. Lars made the comment that there are transport protocols that are trying to infer congestion by looking at both delay and loss changes. They are experimental. There are other causes for loss than congestion, and other causes for delay than congestion, for example a wireless link changing it's rate. He said that there is ongoing research here but there is nothing baked enough to become a metric. is research on this,but not ready. Al noted that one thing the draft could talk about with respect to minimum delay is that it is an indication of path changes and could be used to separate a bimodal distribution. If we can detect that a path has changed positivly by TTL changes, and correlate that with bursts of loss and delay change (and possibly by reordering caused by "microloops" during a change) you may be able to reset the minimum delay at that point. If path changes are frequent, delay variation is probably not your biggest problem. Resetting the minimum at appropriate points would be good for long-term measurements. Alan said that at the last meeting he raised a possible case where inter-packet delay variation might be useful, and that is the packet video buffer smoothing problem. Video packets have timestamps in them, but the rate is highly variable due to big complete frames being mixed with packets indicating changes since the last complete frame. There is a smoothing buffer to try and even things out. You need to take fragmentation and MTU size into account in this case. He's thought about it a bit, but has no conclusion yet. Alan also noted another interesting problem, not specific to video, is where the bandwith of test signal, whatever it may be, is sufficient to cause some delay variation itself. Can you separate this test signal induction (due to bandwidth limits in the network) from those from other sources in the network? It doesn't matter what the absolute value of variation is in this case. He has been doing some simulation that might apply, with scene characteristics of video streams. In summary, Al said that he'll go through the comparisions again, and felt he was still trying to acheive consensus, but that he has not seen much resistance to the basic results. Speaking to slide 9, Alan Clarke made the observation that rather than calling these guidelines in the document, we should call them measurement consderations; to him guidelines are directing test vendors to do something rather than giving them advice about things to watch out for. Packet Delay Variation is one of the most difficult things to measure and understand properly, so he thinks a section like this is extremely useful. Scott Bradner noted that his earlier reaction to the guidance in the appendix did not apply here, measurement considerations are good to state. Al concluded noting that there had been significant support in the room, comments from group members both here and on the mailing list and that this draft had already been cited to satisfy an item the charter, so it's time to make this document a working group item. Henk took a quick poll of the room; about 1/2 indicated support for the document. When asked who thought it was a bad idea to continue with this document, there was no opposition. Therefore Henk said the Chairs would verify the sense of the room on the mailing list, and assuming no major change would work with the area director to make this a working group draft. An group member thought the document should be broadend from just focusing on delay variation. Al felt that it was better to stay focused in order to finish the document and complete the milestone. [Later note by MZ: we have had a couple of other starts at applicability statements that did not have enough support to follow through to completion; others are welcome to make suggestions, but we should just focus on finishing the document in an area where there is a need and support from the working group.] 7. Any Other Business Henk mentioned a passive measurement draft that had been sent to the list: draft-kikuchi-passive-measure-00.txt. No group members had read the draft; given the earlier discussion about passive measurements and the upcoming "Application Performance Metrics BOF", it seems that the draft is currently out of scope for this working group but potentially in scope for the results of the BOF, Henk will send out a note to that effect. Al Morton made a comment about his ITU-T liaison work. Years ago, Al was put in charge of the IP Performance question in SG 12. We have been working to reduce the differences between the work done there and here, harmonizing the results as much as possible. The scope of SG 12 is different and wider -- it produces numerical objectives based on performance perameters (from metrics); this group is focussed on the metrics. One way to reduce the differences between the groups is to share information so that each group can comment on the other groups doucments. Following the model of several other ITU-T study groups, SG 12 has created an easily accessable FTP site. See: www1.ietf.org/mail-archive/web/ippm/current/msg1163.html and www.itu.int/ITU-T/studygroups/com12/packet/index.html. Scott Bradner noted that working documents are what are covered by the FTP site; right now finished documents are available for free. Brian Carpenter noted that the plan to distribute finished documents for free was approved last January for one year, so it is not yet clear what will happen in 2008. SG 12 would specifically like feedback on two documents that are now in the repository. The first is a revised version of Y.1540, the IP performance metrics document. This revision has more metrics, with reordering pulled into the normative text, and new text on duplication and replication. We should try to harmonize on terminology if we can. At the moment the group is grappling with the problem of how to include a markov model description of burst loss. There are two competing proposals of how to define states (like burst or gap, alternatively stable or un-stable). The current burst/gap proposal needs to evaluate packets in sending order and requires an evaluation interval whose length has a dependece on the application of interests. An alternative proposal might be to use arrival order instad, so it would include the effects of reordering. The other document is a multicast metrics draft. Within the ITU the scope is expanded so that it covers the IGMP aspects of multicast, not just packet transfer performance. This is a complement to Y.1540. It goes "above and below" to look at session establishment and teardown of IGMP. It would be useful to get feedback from IETF folks. If one wanted to post a comment document, they could send it to Al Morton, acmorton at att.com. The IPPM chairs and also the Transport AD also have the ability to post to the site. If there is a document that is relvant to the whole community we will post it to the web site. Someone asked about the intellectual property rules of documents posted to that space. Al said he did not know, but would look into it. Alan noted that one of the differences between the ITU and IETF; here people are encouraged to disclose IPR as early as possible. Within the ITU, the onus on members is to formally offer to license IPR, if any exists, prior to any recommendation being approved. There is less pressure to disclose early. Scott Bradner replied that this is accurate, except... IETF has no requirment to say what licensing is but ITU does, everything else is exactly right APM application performance metrics. tomorrow afternoon 1510