Performance Metrics for Other Layers WG (pmol) Thursday, November 20, 2008, 1740-1930 Afternoon Session III Room: Duluth Co-Chairs: Al Morton <acmorton@att.com> Alan Clark <alan@telchemy.com> This report of the PMOL Working Group was prepared by Al Morton, using the detailed notes of Juergen Quittek and the Audio record. There were about 15 people in attendance (late on Thursday night), and one participated remotely. Briefly Summarizing: both the Framework and SIP metrics drafts will enter WGLC shortly. There was a discussion of Networked Application Performance monitoring, and after discussion there was interest expressed to have a more detailed problem statement to refine the scope of work and allow the topic to be discussed with other areas, such as the Applications area and the Transport area. The discussion of the future of PMOL examined what we have learned in the last year and what form of PMOL entity would serve best. This discussion is intended to continue on the list, using the notes below as the beginning of the thread. 1. WG Status ============ http://tools.ietf.org/wg/pmol/ Al: SIP performance ID has completed one WG Last Call, another is needed. Metric framework ID may soon be ready for WG Last Call. The WG is a bit behind its milestone deadlines, but making progress. 2. SIP performance metrics ID ============================= http://tools.ietf.org/html/draft-ietf-pmol-sip-perf-metrics-02 Daryl Malas: Changes Since last version: - Separated failed registration metric from request delay metric - Updated session request delay. Also here failure was separated from just delay - calculation of percentage values has been clarified - Re-tries of the same attempt (same Call-ID = one attempt) - Ineffective attempts – focus on UAC and leave out metrics that might be defined at a registrar, for example. Discussion of options on slide 3 – need some change here, but Daryl seeks additional wording – the solution seems to be removing the mention of the Registrar in the definition of failed register request. Additional feedback on timer section: Al: There is a different interpretation of our text describing timer accuracy. The error of the estimation of the difference (offset) between separate timers should be reported also. We can adopt this comment, but need to be sure that we also require that the “timer offset” is reported with the metrics (instead of just minimizing the offset). Next steps are - incorporate changes and comments in the recent feedback - hold another WGLC Loki (via Jabber): If the offset is reported, does the error (or certainty) also need to be reported? What if the UTC is not well known? Al: You should report the offset and you should minimize the error. Offset may be an average offset derived from NTP measurements (and this could be traceable to UTC). We need to look at the modified text and see if it is satisfactory. Daryl: Vendors are asking, when will this be done? 3. Framework and Guidelines Draft ================================= http://tools.ietf.org/html/draft-ietf-pmol-metrics-framework-01 Alan: Incorporated comments from last meeting summarized in the slides. No comments so far on version -01. Dan (as contributor): I believe this document is ready for WGLC. Al: RFC numbers and procedures on IETF procedures, 2026, 2418, and some Web pages. Al: Question for the author of the SIP metrics I-D: Is your document In-line with the current version of the framework? Daryl: I checked read the Framework coming in to the Dublin meeting (72) and It was pretty-well aligned, in my opinion. Some things were removed at the 00 version to align it. Al: I also discussed a comment to clarify the reference to delay variation from RFC3550, and will supply text for that as part of the WG LC. Al: How many in the room have read the ID? Lots of hands up, and also agreement with entering WGLC. 4. Discussion of Future of PMOL =============================== - Is there a continued need for PMOL, as a WG or as a Directorate? - Presentation on Application Performance Monitoring Alan: Presentation on Networked Application Performance Monitoring Shall we use a layered approach for measuring performance of applications? When measuring an application the contributions of different layers should be individually visible. For example, performance of SIP relies on the underlying performance of lower layers. Slides 3-5 describe the layered approach. The goal is NOT to measure the performance of each layer independently. Would this be a potential work item for the PMOL WG? Daryl: The approach seems to be very useful and applicable. However, there are already metrics defined for several of the lower layers and many of them are for good reasons independent of each other. When I measure SIP metrics, I want to receive SIP performance values and not properties of underlying layers. We shouldn’t address anything below layer 5. Alan: part of the value is to be able to look at the lower layers, but not re-invent any metrics that exist, just provide a comprehensive view of the performance. Mike Patton: When we started IPPM, we found that there are different metrics for users of service and for providers of a service. Need to define Audience for this work. Alan: You may use layered metrics and not report to the users all the details of underlying layers. DNS example is good – is an HTTP Transaction due to DNS delays, or something else. Daryl: There are delay-sensitive apps, on those that are not. The actual DNS systems used may be different for each user’s HTTP transaction. This is a very broad area, and the scope needs to be more tightly defined. Alan: Agree, you could apply this to passive monitoring as well. Juergen Quittek: What would we work on? Standardizing a diagnosis tool? I’m not sure if this is definition of metric? Or is it a way to collect the metrics? Alan: Idea is to monitor a real or synthetic transaction, and measure perf. I’m suggesting standardization of the set of metrics we would monitor for each application. Mike Patton: We have lots of metrics, and some need to be defined. Just measure all of the key ones. ... Discussion clarified some aspects of TCP metrics ... [00:42:20] <Loki> Does this imply Emodel-like implementations for applications (other than VoIP, such as HTTP in this case)? [00:42:54] <Loki> Or simply measures at two or more different layers that are compared (e.g. HTTP and IP)? [00:46:04] <Loki> In the jargon of the market, this would suggest metrics for Quality of Application (QoA) - how well is the application working, and moreover distinguishing application performance due to application considerations from underlying network considerations? Is this the idea? Alan: Much explanation – summarizing: Why do different component transactions, when you want to know the higher-layer transaction performance? Juergen: Do we need to measure all the lower layers for everything? Alan: For SIP, we would need very few of the lower layers. Juergen: Procedurally, we could look at each application and determine what lower-layer measurements would be useful. Alan: That’s for discussion, but this work is envisaged to cover a few basic applications, and there are some missing raw metrics but also many aggregate metrics that need definition. [00:54:46] <Loki> Would work done in PMOL address, for example, categories of application? E.g. transaction-like (HTTP), data transfer -like (FTP).... etc.? [00:56:24] <Loki> Each might be defined by their dependencies on lower layers, and by typical operation types, for example? [00:58:11] <Loki> For example, HTTP is oriented around a page, use of multiple sessions/transactions, small transfers, TCP performance in early slow start, etc. Al/Alan: This seems like a correct summary. Daryl: Can you take HTTP, develop the metrics that don’t exist, and just Measure them. Do you need to create an Index, like an R-value? Alan: I’m not advocating that we create an R-value for every application. Benoit Claise: Is this work to measure performance at all layers with active probing? Alan: This could be done active or passive, and in most cases you are passively monitoring what comes back. As part of an HTTP test, you may be using metrics from IPPM, etc. Al: Need to get a sense of the room for interest. But first a clarification question. Benoit: It would be fine to work on HTTP, but... How many more applications would we do, beyond HTTP? Is this a large effort with open scope? Where shall we stop? Alan: The goal was to quantify the 4 or 5 most common app protocols (HTTP, POP/SMTP, FTP), and define the metrics as independently as possible. [00:54:46] <Loki> Would work done in PMOL address, for example, categories of application? E.g. transaction-like (HTTP), data transfer -like (FTP).... etc.? [00:56:24] <Loki> Each might be defined by their dependencies on lower layers, and by typical operation types, for example? [00:58:11] <Loki> For example, HTTP is oriented around a page, use of multiple sessions/transactions, small transfers, TCP performance in early slow start, etc. Daryl: Has same concern as Benoit, proposal for SIP metrics came to fill an obvious gap – or is this for completeness. Alan: HTTP is measured widely, but there aren’t standard metrics, so you have the problem of results comparison without standards. Dan: This was an interesting discussion and this is also a very important problem in the industry. In the IETF there is room for this kind of issues. but we need to find out where to start. Action: need a problem statement that could be reviewed by the applications area, and possibly the transport area as well. <<< Change of Subject to Future of PMOL >>>> I would like to discuss where this WG is going. Does PMOL need to continue as a working group? The numbers of people participating now is much lower than at the start. There is the option of continuing as a directorate. There is also the option of doing nothing. What have we learned during this year? Al: I think there’s a fourth option: Finish the work of PMOL WG. Then, when a sufficient number of Proposals emerge, have a BOF to refine them and propose another short-lived working group to deal with those items. Daryl: The work on SIP performance metric started within SIPPING. There was a lot of interest on the draft there. However, there was a lack of expertise in metrics. In PMOL there is the expertise, but since the mailing list is separate from SIPPING, there is little feedback to the work now. Wouldn’t a directorate be a better form, where expertise is available, but metrics work would remain in the WGs. Alan: We should consider not separating the discussions from the original mailing lists. Mike Patton: There are tools available at the IETF to copy new ID announcements to more than one mailing list. Alan: We had that, but it did not help in this case. Daryl: But we did cross-post, and still it was forgotten once it left RAI area. Dan: It does not matter in which group work is done at the IETF. What is important is how much time did it take, and is the work of good quality. Daryl: SIP will not be the last time this need for a metric emerges, and that’s What the framework draft is good for: it provides the guidance we didn’t have and now protocol WG can look at drafts for metrics and know how to evaluate and develop them. I think a Directorate could work with that model. Alan: For some protocols, there’s not WG, like FTP for example. Also, AVT has not taken up metric definition, we could help them there. Mike: PMOL should provide opportunity for application metrics where there is no specific matching WG, But if there is one, application metrics should be developed within the specific WG. Al: Let's see what proposals come up. If they do have a home WG then there would be no need for PMOL. If not, then PMOL can provide a home. Dan: The issue will come up in the IESG when the last ID of the current charter is complete. “What is the PMOL Entity?” will be asked. We have started the discussion here. Let’s use the minutes of the meeting to start a thread on the list. Perhaps we can draw the attention of OPSAREA and APP Area, in perhaps co-chairing a future BOF. Al: Thanks for the good summary, and thanks to all for staying late and participating in this discussion. 5. AOB