[PMOL] comments on Framework Metrics draft

["Loki Jorgenson" <ljorgenson at apparentnetworks.com>]

All - I am aware that the WG last call closed January 2, 2009 for the Framework for Performance Metrics Development draft (v0.1).   I am not sure of the utility of any comments past the expiry time.  I have not seen any mention of it being submitted.

 

Regardless, on a recent re-reading, I noted some additional concerns.   I am posting them in brief for consideration:

 

-------------------------------------

 

Quality of Service/Experience:

 

There are a variety of references within the document to "quality of service" (QoS) and several more for "quality of experience" (QoE).  Further, there are references to "application performance".  None of these have been defined and therefore have been accepted with denotative meaning.  However, these words have broader connotative meaning within the field and it would be difficult to interpret certain passages without additional reference.

 

>>> I would suggest that they should be more precisely defined.  I can provide some input to this if it is seen as desirable.

 

Related point - there are repeated references to "application performance" as well as references to "application performance models" (ex. 3.5.3).  These seem distinct and yet related to QoS and QoE.  Some pundits (including myself) have begun referring to "quality of application" (QoA) to distinguish between the effects attributed to the network layers (QoS), the impacts of the application design and implementation (e.g. codec), and the experience of the end-user (QoE) which derives from other influences besides QoS and QoA.   QoA seems most pertinent here insofar as PMOL aims to capture aspects of what may be considered application design (choice of buffer, codec, protocol).

 

>>> I would suggest considering the addition of QoA and using it as part of the definition set above.  Again, I can provide some input if desirable.

 

Related point - there is a general suggestion that a proposed metric should be correlated against one of "quality of service", "quality of experience", or perhaps "application performance/quality of application" (see 3.2 (ii)) as a "test of usefulness".   Similarly, 3.4.2 (ii) refers to "how this relates to the performance of the system being measured".  Also, 4.2 refers to metric assessment on the basis of "correlation with application performance/user experience".

 

However there are no further directions on how that correlation is done, what the criteria are, or how correlation should be described within a PE entity definition.  For example using the language referred to above, it may be that a given metric is considered identical to QoE (e.g. MOS) and thus directly comparable to end-user experience - correlation would then be exact with respect to the G.107 implementation and correlate approximately with standard means for generating user opinion scores.  Alternately, a given metric may be considered only partially related (e.g. rate of discards given a fixed buffer size) and could be described as having some well-defined correlation to QoE or a QoE-specific metric (perhaps simply a graph of discards to MOS).

 

>>> I would recommend a section in the informative part of the metric definition describing any anticipated relationships or correlations to a quality assessment, and, where possible, how to confirm that correlation.

 

3.3.1 - the phrase "each of which has a greater time span that the original metrics" causes some confusion on first read.  Upon reflection, and by virtue of the example provided, the meaning of the qualifier can be determined correctly.  However, it can be incorrectly read that (each or all of)  the summarized metrics span a greater time than the source metrics that generated the summary - which of course they don't - at best they span the same overall period.  What is intended was something like "each individual metric value within the summarized set spans a period of time greater than or equal to each metric value from the original set".  However, that still may not be the case where the periods for each metric value are unequal within the original set and/or where the summarized values may aggregate those original values unequally.

 

3.3.2 - The reference to an index would benefit from an example.  An obvious one would be MOS or R-value from the Emodel.

 

3.4.2 (iii) Measurement Method - The second line suggests "It is important to also define exception cases...".   I suggest that this should be replaced with "This SHOULD also define exception cases and how these are handled".

 

3.4.2 (iv) Units of Measurement - "must" --> "MUST"

 

3.4.2 (v) Measurement Timing - "must" --> "MUST"

 

3.4.3 (i) Implementation - "may" --> "MAY"

 

3.4.3 (iv) Reporting Model - the initial statement should rephrased as a "SHOULD" statement.  Such as "The Reporting Model definition SHOULD describe any relationship(s) between the metric and the reporting model"

 

3.4.4. The "Verification" section described in 3.4.3 (ii) is missing.

 

3.4.4  The layout could be improved for readability - the "Normative" and "Informative" headers appear as peers of the other aspects.

 

3.4.5 Examples - the "Verification" section appears named as "Conformance Testing" - this should be made consistent with the balance of the document.

 

3.5.3 Relationship .... - as mentioned above, this section seems centrally important and yet under-represented in the definition of a PE.  Simply listing this as a "dependency" seems inconsistent with the other statements throughout the document.  I would recommend that QoS/QoE be more explicitly defined, with appropriate examples, possibly with addition of QoA (if there is support for this), and some sketch at least of how correlation may be established or can be described.

Loki Jorgenson
Chief Scientist
Apparent Networks
The Hudson House
Suite 400 - 321 Water Street
Vancouver, BC, Canada, V6B 1B8

e   ljorgenson at ApparentNetworks.com
t   604 433 2333 ext 105
f   604 433 2311
m   604 250-4642
w   www.ApparentNetworks.com