[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [AVT] Suggestion for how to handle derived/computedmetricindraft-avt-rtcpxr-*
Roni
Certainly there is a difference between voice and audio codecs. In the case
of voice codecs (both narrowband and wideband) the methods for objectively
measuring the quality of the call and for taking codec characteristics into
account is well understood - there are admittedly several approaches however
there are existing standards that address means for calculating objective
estimates of MOS and even for conformance testing of algorithms that do
this.
The approach used for voice quality is fairly straightforward. There are
several parameters that are defined on a codec specific basis - notably Ie
and Bpl, where Ie identifies the quality degradation due to the encoder and
Bpl the robustness of the PLC algorithm against packet loss/ discard. The
endpoint incorporates a small table of these values for each voice codec
that it supports. Based on the codec selected for the call - the
appropriate parameters are used. Some other parameters, for example signal
level, noise level, echo, delay, are less codec dependant. A good source of
information on how to do this using G.107 is ITU-T Recommendation G.799.1,
which is a VoIP trunking gateway standard that incorporates a clear
definition of how to compute the R factors and MOS scores required by RTCP
XR.
Methods for measuring audio quality are less heavily standardized at this
time, although there are ITU standards such as G.1070 (for IP
videoconferencing) that incorporate voice, video and multimedia quality
metrics and ITU-R BS-1387 (which is a PESQ-like algorithm for audio). There
is ongoing work within the industry related to IPTV audio stream measurement
that is occurring in at least three different standards organizations.
I'd be happy to put together an informational document that outlines
existing approaches to objective voice/ audio/ video performance measurement
and reviews some of the current industry activities - if that would be
useful?
Regards
Alan
-----Original Message-----
From: Even, Roni [mailto:roni.even at polycom.co.il]
Sent: Thursday, July 26, 2007 3:30 PM
To: Alan Clark; David R Oran; AVT WG
Subject: RE: [AVT] Suggestion for how to handle
derived/computedmetricindraft-avt-rtcpxr-*
Alan,
What bothers me is that in my view many people building audio and video EPs
will use codecs that come from the DSP manufacturer or from a open source
library. It will be difficult to them to choose the right metrics. I am not
the biggest expert on quality assessments but I think that there is a
difference between the current voice codecs (3, 7Khz) and the new audio ones
(14 and 20Khz) which are now used in video conferencing and are coming to
VOIP applications in term of the quality assessment.
Now how do you expect a non-educated EP developer to know what is the right
way to calculate the quality. How will he calculate quality if his system
may use G.711 in one call and AAC-LD in a second call. It needs either to be
very easy to understand or just ask him for raw data.
Roni
> -----Original Message-----
> From: Alan Clark [mailto:alan.d.clark at telchemy.com]
> Sent: Tuesday, July 24, 2007 1:08 AM
> To: 'David R Oran'; 'AVT WG'
> Subject: RE: [AVT] Suggestion for how to handle derived/computed
> metricindraft-avt-rtcpxr-*
>
> Dave
>
> I agree with the general tenor of your thoughts. I also agree that we
> should not mandate that endpoints have to implement any particular set
of
> derived metrics - these should be optional and should be implemented
based
> on their usefulness.
>
> For cases in which there is a fairly well defined metric class (e.g.
MOS
> for
> speech) then I suggest that we do define a separate draft for that
with a
> registry of algorithms that could be (optionally) used - it is easier
to
> update a registry with information that a new version of ITU-T G.xxx
is
> available than it would be to generate a new draft.
>
> Regards
>
> Alan
>
>
> -----Original Message-----
> From: David R Oran [mailto:oran at cisco.com]
> Sent: Monday, July 23, 2007 5:51 PM
> To: AVT WG
> Subject: [AVT] Suggestion for how to handle derived/computed metric
> indraft-avt-rtcpxr-*
>
> At the Mic during the AVT session I sketched a vague idea for how to
> resolve the conflict between the desire on the part of many folks to
> only
carry
> raw/primitive instrumented measurements in RTCP-XR, and other equally
> vocal, who argued for carrying the derived/computed metrics as well.
>
> I won't repeat the pro/con arguments here. Let's assume we want to
carry
> both.
>
> The chairs asked me to write up my vague thoughts as the beginnings of
a
> concrete proposal.
>
> Some particular technical difficulties with the derived metrics that
makes
> them different from the primitive measurements is:
> - there are competing standards for how to compute them
> - the results are not generally aggregatable, comparable, or self-
> describing.
> - the set of interesting derived metrics is not small and seems to be
> growing, so it will be difficult to nail these down for stable RTCP
> reporting.
>
> This makes it problematic to mix primitive and derived metrics in the
same
> metric blocks in RTCP.
>
> So, my suggestion is to do the following.
>
> - A standards body who defines one or more consistent derived metrics
may
> create (by bringing a draft to IETF) a separate metric block for those
> metrics.
> - The block and its contained derived/computed metrics would be
completely
> self-describing in terms of exactly what version of what standard
defines
> the block.
> - The definition of the metric would explicit contain the input
variables,
> in order to ensure that those inputs are separately available as
primitive
> metrics in the RTCP-XR. This increases the probability that any metric
> that is computable by the endpoint is also computable by a central
> element receiving and processing RTCP-XR reports.
> - If there are competing/overlapping blocks, an endpoint need not do
all
> of
> them and there would be no confusion about exactly which (e.g."video
MOS")
> calculation was done.
>
> As an aside, I would rather not see further updates to the existing
drafts
> until we resolve this issue, since this is in fact major surgery to
the
> documents that is best done before polishing the text and individual
> metric definitions.
>
> Dave Oran.
>
> _______________________________________________
> Audio/Video Transport Working Group
> avt at ietf.org
> https://www1.ietf.org/mailman/listinfo/avt
>
>
> _______________________________________________
> Audio/Video Transport Working Group
> avt at ietf.org
> https://www1.ietf.org/mailman/listinfo/avt
_______________________________________________
Audio/Video Transport Working Group
avt at ietf.org
https://www1.ietf.org/mailman/listinfo/avt
_______________________________________________
Audio/Video Transport Working Group
avt at ietf.org
https://www1.ietf.org/mailman/listinfo/avt