Performance Metrics for Other Layers WG (pmol) Minutes of the March 24, 2009 1710-1810 Afternoon Session III Imperial A Co-Chairs: Al Morton <acmorton@att.com> Alan Clark <alan@telchemy.com> Reported by Al Morton, based on detailed minutes provided by Mike Hamilton as official note-taker. The meeting was chaired by Al Morton. Draft of 4/5/2009 Summary 27 people attended the PMOL session. Although the WG is behind schedule on its milestones, it seems possible to produce two mature drafts that have reached WG consensus in the next few months. The SIP metrics draft now appears to have satisfied all commentors, and consensus will be tested with a short WG Last Call (with CC to SIPPING working group). There were a few comments on the Framework Draft, including a request to make applicability to Passive and/or Active measurement methods more clear, and references to definitions of QoE and Application Performance. A revised draft is needed, then another WG Last Call. The PMOL WG needs to continue long enough to see these drafts through IESG review, but would probably not need to meet again unless there are significant comments that require discussion to resolve. ACTION ITEMS: 1. WGLC on SIP metrics Draft (cc SIPPING) 2. Revised Framework and Guidelines Draft, then WGLC DETAILED MINUTES 0. Agenda Bashing --Starting about 3 minutes late due to previous meeting running over --Just meeting for 1 hour --Al asking for short but sweet comments --Mentioning mailing list --Mike Hamilton as note taker --Matt Z as Jabber scribe. 1. WG Status http://tools.ietf.org/wg/pmol/ --AD/IESG Review None --WG Last Call sip-perf-metrics-03 WGLC ending on July 25, 08 metric-framework-02 WGLC ending Jan 2 --No other ID's PMOL Deliverables --framework and guidelines memo audience, motivation, necessary elements of metrics Milestones --Jun 2008 Perf metrics draft to IESG review for consideration as proposed standard 2. SIP Metrics Draft: Daryl Malas (review comment resolutions) http://tools.ietf.org/html/draft-ietf-pmol-sip-perf-metrics-03 Changes since IETF 73 Made some editorial changes the draft from "word" to "XML". Added ACM as co-author for contributions to Section 3 and other general input Since ACM is also co-chair, Alan will have to call consensus Talking about draft having gone through WGLC a few times Dan Romanascu Talking --Depends on how many changes since last draft whether to WGLC --Consensus will need to be met on mailing list also, so not to worry about Alan's absence (Back to Daryl) Major Updates since 73 --Section 3 two major changes Time of Day accuracy- conform to RFC 2330 Clarified time interval accuracy to align to 2330 --Section 4 Added call ID detail for identifying retransmissions Further clarified output values based on the input values One sentence in beginning --Metrics Clarified start and stop of interval - really just a few additional words to make specific Went and review RFC's out of BMWG looking to align terminology with existing metrics drafts Clarified registration request attempt Changed session request delay units of measurement from milliseconds TO seconds Most common units in industry Defined how SDT is calculated at the UAS for successful and failed situations Whole new paragraph explaining. Was previously pretty ambiguous, now an exact definition Other minor clarifications Two metrics had the same text Hops-per-request metric defined some new variables and added verbiage around them. Didn't change the metric Already noticed a few typos and missed grammatical errors. Q&A Dan R Q: Do you have any positive feedback or any feedback from the SIP committee A: Mario has provided feedback (Ericcson) and commented that he agreed with Pedro Caplan (Acme Packet) - caught the cut-and-paste error The main two people, but also several cable providers have reviewed the document Copy sipping on WGLC notice Also got comments from a professor Al: Several others have also reviewed Daryl: Once we complete WGLC, would like to have expert review of it from SIP experts AL: one comment from a colleague - Sometimes we count the number of retries. I think that would show up as a mode of session request delay. If you had systems with one retry, all of their delays offset by the retry timeout. If two retries, 2xretry timeout. Al is pulling up the freehand drawing tool drawing at a tri- modal histogram showing #established invites cross time - since we only look at delay, this is one way to derive the information on number of retries involved with successful requests. Daryl: There is not a metric associated with counting the number of retries per session. You don't count retries when looking at the duration of the session. Metric says - Start from the initial invite to when the invite is accepted. If the timer expires, the delay is when that timer expires. Daryl: Obviously users could define additional metrics beyond what the IETF has specified. But, at least we have some well-defined set of metrics. Two peering SIP providers may use additional metrics, but they may be drastically different from each other. The purpose of this isn't to cover all metrics. Al: Next steps - another short WGLC on this draft. 3. Framework and Guidelines Draft: (Al M. leading discussion) http://tools.ietf.org/html/draft-ietf-pmol-metrics-framework-02 Al: No updates on the Metrics Framework Draft, How many have read? About three quarters of those in attendance. Comments: Loki Jorgenson - There was an early discussion regarding scope of active/passive and was watching draft for language if either were excluded or included. Should be clear about this. Are we OK with that broadness. Al: Do we think that the current framework is sufficient to cover both? Should we restrict the scope to one or continue to go forward. Dan: My opinion is that the framework for the definition of metrics. Applicability to active/passive monitoring is irrelevant. Maybe there should be some text that clarifies specifying the metrics apply to a certain type of measurement. Loki: Certainly comfortable with a section stating it's applicable to both active and passive. There is no clear suggestion that it was addressed. A simple declaration would make it stronger. CONCLUSION: Add a statement on relevance to active and passive measurement in the draft. Loki: There are a number of references that I found confusing and was hoping Alan would be able to clarify. References to QoS (roots in G.1000 ITU), QoE(a little bit more ambiguous, but some verbiage), application performance. These things feel like they should be clearer. QoS is the clearest. Hunch is that in the context of VoIP, the jump from QoS to QoE to Application performance is straightforward. But in other applications (video) there are larger gaps between them. Wondering if App Perf and QoE need additional clarification. Al: Thinks it reasonable to define QoE in the near-term. If you can live with the QoS definition in G.1000, you'll probably be OK with the QoE definition (in P.10/G.100 - those references supplied here). I would guess G.1010 would have a synonym for application performance. If we have definitions we can reference, then a quick look at the doc would help to illustrate. Loki: Will take a look at G.1010 (and P.10/G.100) and make sure they're acceptable. Jonathan Clairidge: Happened to go to IPFIX session. Looks something like ETSI TCN docs. Al: we haven't done that here. Al to Jonathan: Are the ETSI TCN specs publicly available? Jon: Just looking at them on their website. Al: Suggesting that if you have some familiarity with those docs, maybe we can dig some references out for this document. We can then asses and is a reasonable way to proceed. (Jon subsequently withdrew his comment via e-mail). Al: Any other comments on the framework draft. I'll mention quickly I posted a comment on the list and suggested a rewrite. Alan had already agreed on it in a hallway discussion at IETF-73. We have got some feedback to provide in the notes, if we can get another revised version in short order, we have another WGLC. CONCLUSION: Revise then WGLC. 4. Discussion of Future of PMOL - Further thoughts following discussion at IETF-73: Is there a continued need for PMOL, as a WG or as a Directorate? Al: Our last topic is the future of PMOL. Appears we'll just try to finish up work program. If we have any additional comments or thoughts on usefulness of the WG. Any thoughts on the future of PMOL. Dan R: If there are no comments on further suggestions here or on the mailing list, next steps would be to finish up the last 2 WGLCs. Would mean the WG would enter IESG review. Expect the framework to get some scrutiny. The WG should be around for such inspection. We would probably end PMOL after that. If the documents are approved, they reference PMOL. Looks to go more on the directorate direction. We still have time to consider. Al: Basically ask for volunteers to be directorate members. Thanking Dan for capturing that. 5. AOB Al: Thanks to all here, and to those who have contributed to PMOL along the way.