RE: [AVT] Suggestion for how to handle derived/computedmetricindraft-avt-rtcpxr-*

["Even, Roni" <roni.even at polycom.co.il>]

Alan,
This will be helpful.
Still my concern is that EP implementers will not know what to calculate
since their product may use different types of codecs.
Roni

> -----Original Message-----
> From: Alan Clark [mailto:alan.d.clark at telchemy.com]
> Sent: Thursday, July 26, 2007 10:59 PM
> To: Even, Roni; 'David R Oran'; 'AVT WG'
> Subject: RE: [AVT] Suggestion for how to handle
> derived/computedmetricindraft-avt-rtcpxr-*
> 
> 
> Roni
> 
> Certainly there is a difference between voice and audio codecs. In the
> case
> of voice codecs (both narrowband and wideband) the methods for
objectively
> measuring the quality of the call and for taking codec characteristics
> into
> account is well understood - there are admittedly several approaches
> however
> there are existing standards that address means for calculating
objective
> estimates of MOS and even for conformance testing of algorithms that
do
> this.
> 
> The approach used for voice quality is fairly straightforward.  There
are
> several parameters that are defined on a codec specific basis -
notably Ie
> and Bpl, where Ie identifies the quality degradation due to the
encoder
> and
> Bpl the robustness of the PLC algorithm against packet loss/ discard.
The
> endpoint incorporates a small table of these values for each voice
codec
> that it supports.  Based on the codec selected for the call - the
> appropriate parameters are used.  Some other parameters, for example
> signal
> level, noise level, echo, delay, are less codec dependant.  A good
source
> of
> information on how to do this using G.107 is ITU-T Recommendation
G.799.1,
> which is a VoIP trunking gateway standard that incorporates a clear
> definition of how to compute the R factors and MOS scores required by
RTCP
> XR.
> 
> Methods for measuring audio quality are less heavily standardized at
this
> time, although there are ITU standards such as G.1070 (for IP
> videoconferencing) that incorporate voice, video and multimedia
quality
> metrics and ITU-R BS-1387 (which is a PESQ-like algorithm for audio).
> There
> is ongoing work within the industry related to IPTV audio stream
> measurement
> that is occurring in at least three different standards organizations.
> 
> I'd be happy to put together an informational document that outlines
> existing approaches to objective voice/ audio/ video performance
> measurement
> and reviews some of the current industry activities - if that would be
> useful?
> 
> Regards
> 
> Alan
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Even, Roni [mailto:roni.even at polycom.co.il]
> Sent: Thursday, July 26, 2007 3:30 PM
> To: Alan Clark; David R Oran; AVT WG
> Subject: RE: [AVT] Suggestion for how to handle
> derived/computedmetricindraft-avt-rtcpxr-*
> 
> Alan,
> What bothers me is that in my view many people building audio and
video
> EPs
> will use codecs that come from the DSP manufacturer or from a open
source
> library. It will be difficult to them to choose the right metrics. I
am
> not
> the biggest expert on quality assessments but I think that there is a
> difference between the current voice codecs (3, 7Khz) and the new
audio
> ones
> (14 and 20Khz) which are now used in video conferencing and are coming
to
> VOIP applications in term of the quality assessment.
> 
> Now how do you expect a non-educated EP developer to know what is the
> right
> way to calculate the quality. How will he calculate quality if his
system
> may use G.711 in one call and AAC-LD in a second call. It needs either
to
> be
> very easy to understand or just ask him for raw data.
> 
> Roni
> 
> > -----Original Message-----
> > From: Alan Clark [mailto:alan.d.clark at telchemy.com]
> > Sent: Tuesday, July 24, 2007 1:08 AM
> > To: 'David R Oran'; 'AVT WG'
> > Subject: RE: [AVT] Suggestion for how to handle derived/computed
> > metricindraft-avt-rtcpxr-*
> >
> > Dave
> >
> > I agree with the general tenor of your thoughts.  I also agree that
we
> > should not mandate that endpoints have to implement any particular
set
> of
> > derived metrics - these should be optional and should be implemented
> based
> > on their usefulness.
> >
> > For cases in which there is a fairly well defined metric class (e.g.
> MOS
> > for
> > speech) then I suggest that we do define a separate draft for that
> with a
> > registry of algorithms that could be (optionally) used - it is
easier
> to
> > update a registry with information that a new version of ITU-T G.xxx
> is
> > available than it would be to generate a new draft.
> >
> > Regards
> >
> > Alan
> >
> >
> > -----Original Message-----
> > From: David R Oran [mailto:oran at cisco.com]
> > Sent: Monday, July 23, 2007 5:51 PM
> > To: AVT WG
> > Subject: [AVT] Suggestion for how to handle derived/computed metric
> > indraft-avt-rtcpxr-*
> >
> > At the Mic during the AVT session I sketched a vague idea for how to
> > resolve the conflict between the desire on the part of many folks to
> > only
> carry
> > raw/primitive instrumented measurements in RTCP-XR, and other
equally
> > vocal, who argued for carrying the derived/computed metrics as well.
> >
> > I won't repeat the pro/con arguments here. Let's assume we want to
> carry
> > both.
> >
> > The chairs asked me to write up my vague thoughts as the beginnings
of
> a
> > concrete proposal.
> >
> > Some particular technical difficulties with the derived metrics that
> makes
> > them different from the primitive measurements is:
> > - there are competing standards for how to compute them
> > - the results are not generally aggregatable, comparable, or self-
> > describing.
> > - the set of interesting derived metrics is not small and seems to
be
> > growing, so it will be difficult to nail these down for stable RTCP
> > reporting.
> >
> > This makes it problematic to mix primitive and derived metrics in
the
> same
> > metric blocks in RTCP.
> >
> > So, my suggestion is to do the following.
> >
> > - A standards body who defines one or more consistent derived
metrics
> may
> > create (by bringing a draft to IETF) a separate metric block for
those
> > metrics.
> > - The block and its contained derived/computed metrics would be
> completely
> > self-describing in terms of exactly what version of what standard
> defines
> > the block.
> > - The definition of the metric would explicit contain the input
> variables,
> > in order to ensure that those inputs are separately available as
> primitive
> > metrics in the RTCP-XR. This increases the probability that any
metric
> > that is computable by the endpoint is also computable by a central
> > element receiving and processing RTCP-XR reports.
> > - If there are competing/overlapping blocks, an endpoint need not do
> all
> > of
> > them and there would be no confusion about exactly which (e.g."video
> MOS")
> > calculation was done.
> >
> > As an aside, I would rather not see further updates to the existing
> drafts
> > until we resolve this issue, since this is in fact major surgery to
> the
> > documents that is best done before polishing the text and individual
> > metric definitions.
> >
> > Dave Oran.
> >
> > _______________________________________________
> > Audio/Video Transport Working Group
> > avt at ietf.org
> > https://www1.ietf.org/mailman/listinfo/avt
> >
> >
> > _______________________________________________
> > Audio/Video Transport Working Group
> > avt at ietf.org
> > https://www1.ietf.org/mailman/listinfo/avt
> 
> _______________________________________________
> Audio/Video Transport Working Group
> avt at ietf.org
> https://www1.ietf.org/mailman/listinfo/avt


_______________________________________________
Audio/Video Transport Working Group
avt at ietf.org
https://www1.ietf.org/mailman/listinfo/avt