[PMOL] Fwd: comments on Framework Metrics draft

[Al Morton <acmorton at att.com>]

Forwarded on behalf of Loki,

> Subject: comments on Framework
> Metrics draft
> Date: Wed, 4 Mar 2009 13:26:09 -0800
> X-MS-Has-Attach: 
> X-MS-TNEF-Correlator: 
> thread-topic: comments on Framework Metrics draft
> thread-index: Acmc6yDA5SVndh2KQbGG9V2tieZapgAJIcGQ
> From: "Loki Jorgenson"
> <ljorgenson at apparentnetworks.com>
> To: <pmol at ietf.org>
> Cc: "Alan Clark" <alan.d.clark at telchemy.com>, "Al
> Morton" <acmorton at att.com>
> X-Brightmail-Tracker: AAAAAA==
>
> All - I am aware that the WG last
> call closed January 2, 2009 for the Framework for Performance Metrics
> Development draft (v0.1).   I am not sure of the utility of any
> comments past the expiry time.  I have not seen any mention of it
> being submitted.
>  
> Regardless, on a recent re-reading,
> I noted some additional concerns.   I am posting them in brief
> for consideration:
>  
> -------------------------------------
>  
> Quality of Service/Experience:
>  
> There are a variety of references
> within the document to "quality of service" (QoS) and several
> more for "quality of experience" (QoE).  Further, there
> are references to "application performance".  None of
> these have been defined and therefore have been accepted with denotative
> meaning.  However, these words have broader connotative meaning
> within the field and it would be difficult to interpret certain passages
> without additional reference.
>  
> >>> I would suggest that
> they should be more precisely defined.  I can provide some input to
> this if it is seen as desirable.
>  
> Related point - there are repeated
> references to "application performance" as well as references
> to "application performance models" (ex. 3.5.3).  These
> seem distinct and yet related to QoS and QoE.  Some pundits
> (including myself) have begun referring to "quality of
> application" (QoA) to distinguish between the effects attributed to
> the network layers (QoS), the impacts of the application design and
> implementation (e.g. codec), and the experience of the end-user (QoE)
> which derives from other influences besides QoS and QoA.   QoA
> seems most pertinent here insofar as PMOL aims to capture aspects of what
> may be considered application design (choice of buffer, codec,
> protocol).
>  
> >>> I would suggest
> considering the addition of QoA and using it as part of the definition
> set above.  Again, I can provide some input if desirable.
>  
> Related point - there is a general
> suggestion that a proposed metric should be correlated against one of
> "quality of service", "quality of experience", or
> perhaps "application performance/quality of application" (see
> 3.2 (ii)) as a "test of usefulness".   Similarly,
> 3.4.2 (ii) refers to "how this relates to the performance of the
> system being measured".  Also, 4.2 refers to metric assessment
> on the basis of "correlation with application performance/user
> experience".
>  
> However there are no further
> directions on how that correlation is done, what the criteria are, or how
> correlation should be described within a PE entity definition.  For
> example using the language referred to above, it may be that a given
> metric is considered identical to QoE (e.g. MOS) and thus directly
> comparable to end-user experience - correlation would then be exact with
> respect to the G.107 implementation and correlate approximately with
> standard means for generating user opinion scores.  Alternately, a
> given metric may be considered only partially related (e.g. rate of
> discards given a fixed buffer size) and could be described as having some
> well-defined correlation to QoE or a QoE-specific metric (perhaps simply
> a graph of discards to MOS).
>  
> >>> I would recommend a
> section in the informative part of the metric definition describing any
> anticipated relationships or correlations to a quality assessment, and,
> where possible, how to confirm that correlation.
>  
> 3.3.1 - the phrase "each of
> which has a greater time span that the original metrics" causes some
> confusion on first read.  Upon reflection, and by virtue of the
> example provided, the meaning of the qualifier can be determined
> correctly.  However, it can be incorrectly read that (each or all
> of)  the summarized metrics span a greater time than the source
> metrics that generated the summary - which of course they don't - at best
> they span the same overall period.  What is intended was something
> like "each individual metric value within the summarized set spans a
> period of time greater than or equal to each metric value from the
> original set".  However, that still may not be the case where
> the periods for each metric value are unequal within the original set
> and/or where the summarized values may aggregate those original values
> unequally.
>  
> 3.3.2 - The reference to an index
> would benefit from an example.  An obvious one would be MOS or
> R-value from the Emodel.
>  
> 3.4.2 (iii) Measurement Method - The
> second line suggests "It is important to also define exception
> cases...".   I suggest that this should be replaced with
> "This SHOULD also define exception cases and how these are
> handled".
>  
> 3.4.2 (iv) Units of Measurement -
> "must" --> "MUST"
>  
> 3.4.2 (v) Measurement Timing -
> "must" --> "MUST"
>  
> 3.4.3 (i) Implementation -
> "may" --> "MAY"
>  
> 3.4.3 (iv) Reporting Model - the
> initial statement should rephrased as a "SHOULD"
> statement.  Such as "The Reporting Model definition SHOULD
> describe any relationship(s) between the metric and the reporting
> model"
>  
> 3.4.4. The "Verification"
> section described in 3.4.3 (ii) is missing.
>  
> 3.4.4  The layout could be
> improved for readability - the "Normative" and
> "Informative" headers appear as peers of the other
> aspects.
>  
> 3.4.5 Examples - the
> "Verification" section appears named as "Conformance
> Testing" - this should be made consistent with the balance of the
> document.
>  
> 3.5.3 Relationship .... - as
> mentioned above, this section seems centrally important and yet
> under-represented in the definition of a PE.  Simply listing this as
> a "dependency" seems inconsistent with the other statements
> throughout the document.  I would recommend that QoS/QoE be more
> explicitly defined, with appropriate examples, possibly with addition of
> QoA (if there is support for this), and some sketch at least of how
> correlation may be established or can be described.
>
> Loki Jorgenson
> Chief Scientist
> Apparent Networks
> The Hudson House
> Suite 400 - 321 Water Street
> Vancouver, BC, Canada, V6B 1B8
>
> e   ljorgenson at ApparentNetworks.com
> t   604 433 2333 ext 105
> f   604 433 2311
> m   604 250-4642
> w  
>
> www.ApparentNetworks.com
>
>