[PMOL] Comments on G.1010 for framework metrics draft

["Loki Jorgenson" <ljorgenson at apparentnetworks.com>]

In San Francisco at 74, Al Morton responded to my concerns about the ambiguity of terms in the framework draft.   Specifically, they included "application performance" as they pertained to QoS (better defined) and QoE (somewhat defined).  He suggested that a reference ITU-T G.1010 (2001) might suffice.  I said I would review and comment.

 

Having reviewed G.1010, I suggest that a reference to G.1010 should appear in the framework draft.  It provides a useful list of categories that at least suggests how application performance relates to network QoS. 

 

That said, I still see a clear need for something like a framework definition for application performance (quality of application) and how it relates to network QoS.  I won't address that here.

 

In particular, I suggest that section 3.2 be revised somewhat to refer directly to "application performance" as it related to QoS.  Suggested changes below (in CAPS) in -02 version of text (with additional references suggested to G.107 and P.800 for illustration):

 

3.2.  Definitions of a Metric
   A metric is a measure of an observable behavior of an application,
   protocol or other system.  The definition of a metric often assumes
   some implicit or explicit underlying statistical process, and a
   metric is an estimate of a parameter of this process.  If the assumed
   statistical process closely models the behavior of the system then
   the metric is "better" in the sense that it more accurately
   characterizes the state or behavior of the system.

   A metric should serve some defined purpose.  This may include the
   measurement of capacity, quantifying how bad some problem is,
   measurement of service level, problem diagnosis or location and other
   such uses.  A metric may also be an input to some other process, for
   example the computation of a composite metric or a model or
   simulation of a system AS MIGHT BE EMPLOYED TO ASSESS THE 
   PERFORMANCE OF A SPECIFIC APPLICATION - FOR EXAMPLE THE EMODEL FOR VOIP (G107).

   Tests of the "usefulness" of a metric include:

      (i) the degree to which its absence would cause significant loss
      of information on the behavior or state OR PERFORMANCE of the application or
      system being measured

      (ii) the correlation between the metric and NETWORK QoS [G1000], APPLICATION-SPECIFIC
      PERFORMANCE (G1010), OR experience delivered to the user (person or other application) (P800)

      (iii) the degree to which the metric is able to support the
      identification and location of problems affecting service quality.

   For example, consider a distributed application operating over a
   network connection that is subject to packet loss.  A Packet Loss
   Rate (PLR) metric is defined as the mean packet loss rate over some
   time period.  If the application performs poorly over network
   connections with high packet loss rate and always performs well when
   the packet loss rate is zero then the PLR metric is useful to some
   degree.  Some applications are sensitive to short periods of high
   loss (bursty loss) and are relatively insensitive to isolated packet
   loss events; for this type of application there would be very weak
   correlation between PLR and application performance.  A "better"
   metric would consider both the packet loss rate and the distribution
   of loss events.  If application performance is degraded when the PLR
   exceeds some rate then a useful metric may be a measure of the
   duration and frequency of periods during which the PLR exceeds that
   rate.

Loki Jorgenson
Chief Scientist
Apparent Networks
The Hudson House
Suite 400 - 321 Water Street
Vancouver, BC, Canada, V6B 1B8

e   ljorgenson at ApparentNetworks.com
t   604 433 2333 ext 105
f   604 433 2311
m   604 250-4642
w   www.ApparentNetworks.com