idnits 2.17.1 draft-valin-codec-results-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 16, 2011) is 4698 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '9' on line 400 -- Looks like a reference, but probably isn't: '10' on line 400 -- Looks like a reference, but probably isn't: '11' on line 400 -- Looks like a reference, but probably isn't: '12' on line 400 -- Looks like a reference, but probably isn't: '13' on line 401 Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 codec JM. Valin 3 Internet-Draft Octasic Inc. 4 Intended status: Informational K. Vos 5 Expires: December 18, 2011 Skype Technologies S.A. 6 J. Skoglund 7 Google 8 June 16, 2011 10 Summary of Opus listening test results 11 draft-valin-codec-results-00 13 Abstract 15 This document describes and examines listening test results obtained 16 for the Opus codec and how they relate to the requirements. 18 Status of this Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at http://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on December 18, 2011. 35 Copyright Notice 37 Copyright (c) 2011 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (http://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 53 2. Pre-Opus listening tests . . . . . . . . . . . . . . . . . . . 4 54 2.1. SILK Dynastat listening test . . . . . . . . . . . . . . . 4 55 2.2. SILK Deutsche Telekom test . . . . . . . . . . . . . . . . 4 56 2.3. SILK Nokia test . . . . . . . . . . . . . . . . . . . . . 4 57 2.4. CELT 0.3.2 listening test . . . . . . . . . . . . . . . . 5 58 2.5. CELT 0.5.0 listening test . . . . . . . . . . . . . . . . 5 59 3. Opus listening tests on non-final bit-stream . . . . . . . . . 6 60 3.1. First hybrid mode test . . . . . . . . . . . . . . . . . . 6 61 3.2. Broadcom stereo music test . . . . . . . . . . . . . . . . 6 62 4. Opus listening tests on final bit-stream . . . . . . . . . . . 8 63 4.1. Google listening tests . . . . . . . . . . . . . . . . . . 8 64 4.1.1. Google narrowband listening test . . . . . . . . . . . 8 65 4.1.2. Google wideband and fullband listening test . . . . . 9 66 4.1.3. Google stereo music listening test . . . . . . . . . . 10 67 4.2. HydrogenAudio stereo music listening test . . . . . . . . 12 68 4.3. Nokia Interspeech 2011 listening test . . . . . . . . . . 12 69 5. In-the-field testing . . . . . . . . . . . . . . . . . . . . . 13 70 6. Conclusion on the requirements . . . . . . . . . . . . . . . . 14 71 6.1. Comparison to Speex (narrowband) . . . . . . . . . . . . . 14 72 6.2. Comparison to iLBC . . . . . . . . . . . . . . . . . . . . 14 73 6.3. Comparison to Speex (wideband) . . . . . . . . . . . . . . 14 74 6.4. Comparison to G.722.1 . . . . . . . . . . . . . . . . . . 14 75 6.5. Comparison to G.722.1C . . . . . . . . . . . . . . . . . . 15 76 6.6. Comparison to AMR-NB . . . . . . . . . . . . . . . . . . . 15 77 6.7. Comparison to AMR-WB . . . . . . . . . . . . . . . . . . . 15 78 7. Security Considerations . . . . . . . . . . . . . . . . . . . 16 79 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 80 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 18 81 10. Informative References . . . . . . . . . . . . . . . . . . . . 19 82 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 84 1. Introduction 86 This document describes and examines listening test results obtained 87 for the Opus codec. Some of the test results presented are based on 88 older versions of the codec or on older versions of the SILK or CELT 89 components. While they do not necessarily represent the exact 90 quality of the current version, they are nonetheless useful for 91 validating the technology used and as an indication of a lower bound 92 on quality (based on the assumption that the codec has been improved 93 since they were performed). 95 Throughout this document, all statements about one codec being better 96 than or worse than another codec are based on 95% confidence. When 97 no statistically significant difference can be shown with 95% 98 confidence, then two codecs are said to be "tied". 100 In addition to the results summarized in this draft, Opus has been 101 subjected to many informal subjective listening tests, as well as 102 objective testing. 104 2. Pre-Opus listening tests 106 Several listening tests have been performed on the SILK and CELT 107 codecs prior to them being merged as part of the Opus codec. 109 2.1. SILK Dynastat listening test 111 The original (pre-Opus) SILK codec was characterized in a Dynastat 112 listening test [SILK-Dynastat]. The test included 32 conditions with 113 4 male and 4 female talkers. The test signals were wideband speech 114 with and without office background noise at 15 dB SNR. Packet loss 115 was tested at 2, 5, and 10% loss rates. The bitrates ranged from 116 8.85 kb/s to 64 kb/s. The codecs included in the test were SILK-WB, 117 AMR-WB, Speex-WB and G.722 (which ran at 64 kb/s). 119 The results showed that for clean speech (1) SILK out-performs AMR-WB 120 at all bit-rates except 8.85 kb/s (which was a tie); (2) SILK out- 121 performs Speex at all bit-rates; and (3) SILK running at 18.25 kb/s 122 and above out-performs G.722 at 64 kbps. For noisy speech, tested at 123 18.25 kb/s, SILK is tied with AMR-WB, and out-performs Speex. For 2, 124 5 and 10% packet loss, tested at 18.25 kb/s, SILK out-performs both 125 AMR-WB and Speex in all conditions. 127 2.2. SILK Deutsche Telekom test 129 In 2010 Deutsche Telekom published results [Wustenhagen2010] of their 130 evaulation of super-wideband speech and audio codecs. The test 131 included the version of SILK submitted to the IETF. The results 132 showed that for clean speech (item "speechsample") SILK was tied with 133 AMR-WB and G.718, and out-performed Speex. For noisy speech (item 134 "arbeit") SILK out-performed AMR-WB and G.718 at 12 and 24 kb/s, and 135 Speex at all bitrates. At bitrates above 24 kb/s SILK and G.718 were 136 tied. 138 2.3. SILK Nokia test 140 In 2010, Anssi Ramo from Nokia presented [Ramo2010] the results of a 141 listening test focusing on open-source codecs at Interspeech 2010. 142 The methodology used was a 9-scale ACR MOS test with clean and noisy 143 speech samples. 145 It was noted in the test that: 147 "Especially at around 16 kbit/s or above Silk is better than AMR-WB 148 at comparable bitrates. This is due to the fact that Silk wideband 149 is critically sampled up to 8 kHz instead of ITU- T or 3GPP defined 7 150 kHz. This added bandwidth (from 7 to 8 kHz) shows up in the results 151 favourable to Silk. It seems that Silk provides quite artifact free 152 voice quality for the whole 16- 24 kbit/s range with WB signals. At 153 32 and 40 kbit/s Silk is SWB and competes quite equally against 154 G.718B or G.722.1C although having a slightly narrower bandwidth than 155 the ITU-T standardized codecs." 157 2.4. CELT 0.3.2 listening test 159 The first listening tests conducted on CELT version 0.3.2 in 2009 and 160 published in 2010 [valin2010] included AAC-LD (Apple), G.722.1C and 161 MP3 (Lame). Two MUSHRA tests were conducted: a 48 kb/s test and a 64 162 kb/s test, both at a 44.1 kHz sampling rate. CELT was used with 256- 163 sample frames (5.8 ms). All codecs used constant bit-rate (CBR). 164 The algorithmic delay was 8.7 ms for CELT, 34.8 ms for AAC-LD, 40 ms 165 for G.722.1C and more than 100 ms for MP3. 167 The 48 kb/s test included two clean speech samples (one male, one 168 female) from the EBU SQAM database, four clean speech files (two 169 male, two female) from the NTT multi-lingual speech database for 170 telephonometry, and two music samples. In this test, CELT out- 171 performed AAC-LD, G.722.1C and MP3. 173 The 64 kb/s test included two clean speech samples (one male, one 174 female) from the EBU SQAM database, and six music files. In this 175 test, AAC-LD out-performed CELT, but CELT out-performed both MP3 and 176 G.722.1C (running at its highest rate of 48 kb/s). 178 2.5. CELT 0.5.0 listening test 180 Another CELT listening test was conducted in 2009 on version 0.5.0 181 and presented at EUSIPCO 2009 [valin2009]. In that test, CELT was 182 compared to G.722.1C and to the Fraunhofer Ultra Low-Delay (ULD) 183 codec on 9 audio samples: 2 clean speech samples and 7 music samples. 184 At 64 kb/s with 5.3 ms frames, CELT clearly out-performed G.722.1C 185 running at 48 kb/s with 20 ms frames. Also, at 96 kb/s and equal 186 frame size (2.7 ms), CELT clearly out-performed the ULD codec. 188 3. Opus listening tests on non-final bit-stream 190 The following listening tests were conducted on the Opus codec on 191 versions prior to the bit-stream freeze. While Opus has evolved 192 since these tests were conducted, the results should be considered as 193 a _lower bound_ on the quality of the final codec. 195 3.1. First hybrid mode test 197 In July 2010, the Opus codec authors conducted a preliminary MUSHRA 198 listening test to evaluate the quality of the recently created 199 "hybrid" mode combining the SILK and CELT codecs. That test was 200 conducted at 32 kb/s and compared the following codecs: 202 o Opus hybrid mode (fullband) 204 o G.719 (fullband) 206 o CELT (fullband) 208 o SILK (wideband) 210 o BroadVoice32 (wideband) 212 The test material consisted of two English speech samples from the 213 EBU SQAM (one male, one female) database and six speech samples 214 (three male, three female) from the NTT multi-lingual speech database 215 for telephonometry. Although only eight listeners participated to 216 the test, the difference between the Opus hybrid mode and all other 217 codecs was large enough to obtain 95% confidence that the Opus hybrid 218 mode provided better quality than all other codecs tested. This test 219 is of interest because it shows that the hybrid clearly out-performs 220 the codecs that it combines (SILK and CELT). It also out-performs 221 G.719, which is the only fullband interactive codec standardized by 222 the ITU-T. These results were presented [Maastricht-78] at the 78th 223 IETF meeting Maastricht. 225 3.2. Broadcom stereo music test 227 In December 2010, Broadcom conducted an ITU-R BS.1116-style 228 subjective listening test comparing different configurations of the 229 CELT-only mode of the IETF Opus codec along with MP3 and AAC-LC. The 230 test included stereo 10 audio samples sampled at 44.1 kHz and 231 distributed as follows: 233 o 2 pure speech 234 o 2 vocal 236 o 2 solo instruments 238 o 1 rock-and-roll 240 o 1 pop 242 o 1 classical orchestra 244 o 1 jazz 246 A total of 17 listeners participated to the test. The results of the 247 test are a available on the testing slides presented at the Prague 248 meeting [Prague-80]. Although at the time, Opus was not properly 249 optimised for 44.1 kHz audio, the quality of the Opus codec at 96 250 kb/s with 22 ms frame was significantly better than MP3 and only 251 slightly worse than AAC-LC. Even in ultra low-delay mode (5.4 ms), 252 Opus still outperformed MP3. The test also confirmed the usefulness 253 of the prefilter/postfilter contribution by Raymond Chen, showing 254 that this contribution significantly improves quality for small 255 frames (long frames were not tested with the prefilter/postfilter 256 disabled). 258 4. Opus listening tests on final bit-stream 260 The following tests were performed on the Opus codec _after_ the bit- 261 stream was finalized. 263 4.1. Google listening tests 265 The tests followed the MUSHRA test methodology. Two anchors were 266 used, one lowpass-filtered at 3.5 kHz and one lowpass-filtered at 7.0 267 kHz. Both trained and untrained listeners participated in the tests. 268 The reference signals were manually normalized to the same subjective 269 levels according to the experimenters' opinion. Experiments with 270 automatic normalization with respect to both level and loudness (in 271 Adobe Audition) did not result in signals having equal subjective 272 loudness. The sample magnitude levels were kept lower than 2^14 to 273 provide headroom for possible amplification through the codecs. 274 However, the normalization exercise was not repeated with the 275 processed sequences as neither the experimenters nor any of the 276 subjects (which included expert listeners) noticed any significant 277 level differences between the conditions in the tests. The only 278 post-processing performed was to remove noticeable delays in the MP3 279 files, as one could identify the MP3 samples when switching between 280 conditions when the MP3 had the longer delay. The testing tool Step 281 from ARL was used for tests and all listeners were instructed to to 282 carefully listen through the conditions before starting the grading. 283 The results of the tests are a available on the testing slides 284 presented at the Prague meeting [Prague-80]. 286 4.1.1. Google narrowband listening test 288 The test sequences in Test 1 were mono recordings (between 2 and 6 289 seconds long) of 4 different male and 4 different female speakers 290 sampled at 48 kHz in low background noise. 17 listeners were 291 presented with 6 stimuli according to Table 1 for each test sequence. 292 The corresponding bit rate for the reference is 48000 (sampling 293 frequency in Hz) x 16 (bits/sample) = 768 kbps. Since the anchors 294 are low-pass filtered they can also be downsampled for transmission 295 which corresponds to lower bit rates. Three narrowband codecs were 296 compared in this test: Opus NB, the royalty-free iLBC, and the 297 royalty-free Speex. The codecs all have an encoder frame length of 298 20 ms. Both Opus and Speex had variable rate whereas iLBC operated 299 at a fixed bit rate. 301 +-----------+----------------------+----------------+ 302 | Type | Signal bandwidth | Bitrate | 303 +-----------+----------------------+----------------+ 304 | Reference | 24 kHz (Fullband) | | 305 | | | | 306 | Anchor 1 | 3.5 kHz (Narrowband) | | 307 | | | | 308 | Anchor 2 | 7 kHz (Wideband) | | 309 | | | | 310 | iLBC | 4 kHz (Narrowband) | 15.2 kbps, CBR | 311 | | | | 312 | Opus NB | 4 kHz (Narrowband) | 11 kbps, VBR | 313 | | | | 314 | Speex NB | 3.5 kHz (Narrowband) | 11 kbps, VBR | 315 +-----------+----------------------+----------------+ 317 Test 1 stimuli 319 Table 1 321 The overall results of the narrowband test, i.e., averaged over all 322 listeners for all sequences, are presented in the Prague meeting 323 slides [Prague-80]. The results suggest that Opus at 11 kbps is 324 superior to both iLBC at 15 kpbs and Speex at 11 kbps. T-tests 325 performed by Greg Maxwell confirm that there is indeed a 326 statistically significant difference. Note also that Opus has a 327 slightly higher average score than the 3.5 kHz anchor, likely due to 328 the higher bandwidth of Opus. 330 4.1.2. Google wideband and fullband listening test 332 The eight test sequences for Test 1 were also used in Test 2. 16 333 listeners rated the stimuli listed in Table 2. In this test 334 comparisons were made between four wideband codecs: Opus WB, the 335 royalty-free Speex, the royalty-free ITU-T G.722.1, AMR-WB (ITU-T 336 G.722.2), and two fullband codecs: Opus FB and the royalty-free ITU-T 337 G.719. All six codecs utilize 20 ms encoding frames. Opus used 338 variable bitrate, while other codecs used constant bit rate. 340 +-----------+----------------------+-----------------+ 341 | Type | Signal bandwidth | Bitrate | 342 +-----------+----------------------+-----------------+ 343 | Reference | 24 kHz (Fullband) | | 344 | | | | 345 | Anchor 1 | 3.5 kHz (Narrowband) | | 346 | | | | 347 | Anchor 2 | 7 kHz (Wideband) | | 348 | | | | 349 | G.722.1 | 7 kHz (Wideband) | 24 kbps, CBR | 350 | | | | 351 | Speex WB | 7 kHz (Wideband) | 23.8 kbps, CBR | 352 | | | | 353 | AMR-WB | 7 kHz (Wideband) | 19.85 kbps, CBR | 354 | | | | 355 | Opus WB | 8 kHz (Wideband) | 19.85 kbps, VBR | 356 | | | | 357 | G.719 | ~20 kHz (Fullband) | 32 kbps, CBR | 358 | | | | 359 | Opus FB | ~20 kHz (Fullband) | 32 kbps, CBR | 360 +-----------+----------------------+-----------------+ 362 Test 2 stimuli 364 Table 2 366 The results from Test 2 are depicted in the Prague meeting slides 367 [Prague-80]. Opus at 32 kbps is almost transparent, although there 368 is a small, but statistically significant, difference from the 369 fullband reference material. Opus at 20 kbps is significantly better 370 than all the other codecs, including AMR-WB and the fullband G.719, 371 and both low-pass anchors. 373 4.1.3. Google stereo music listening test 375 The sequences in this test were excerpts from 10 different stereo 376 music files: 378 o Rock/RnB (Boz Scaggs) 380 o Soft Rock (Steely Dan) 382 o Rock (Queen) 384 o Jazz (Harry James) 386 o Classical (Purcell) 387 o Electronica (Matmos) 389 o Piano (Moonlight Sonata) 391 o Vocals (Suzanne Vega) 393 o Glockenspiel 395 o Castanets 397 These sequences were originally recorded at a sampling frequency of 398 44.1 kHz and were upsampled to 48 kHz prior to processing. Test 3 399 included comparisons between six codecs (c.f., Table 3): Opus at 400 three rates, G.719, AAC-LC [9] (Nero 1.5.1 [10]), and MP3 [11,12] 401 (Lame 3.98.4 [13]). G.719 is a mono codec, so the two channels were 402 each coded independently at 32 kbps. 9 listeners participated in Test 403 3, and the results are depicted in the Prague meeting slides 404 [Prague-80]. The codecs operated at constant (or comparable) bit 405 rate. 407 +-----------+-------------------+-------------+---------------------+ 408 | Type | Signal bandwidth | Frame size | Bitrate | 409 | | | (ms) | | 410 +-----------+-------------------+-------------+---------------------+ 411 | Reference | 22 kHz (Fullband) | - | (1536 kbps) | 412 | | | | | 413 | Anchor 1 | 3.5 kHz | - | (256 kbps) | 414 | | (Narrowband) | | | 415 | | | | | 416 | Anchor 2 | 7 kHz (Wideband) | - | (512 kbps) | 417 | | | | | 418 | MP3 | 16 kHz (Super | >100 | 96 kbps, CBR | 419 | | wideband) | | | 420 | | | | | 421 | AAC-LC | ~20 kHz | 21 | 64 kbps, CBR (bit | 422 | | (Fullband) | | reservoir) | 423 | | | | | 424 | G.719 | ~20 kHz | 20 | 64 kbps (2x32), CBR | 425 | | (Fullband) | | | 426 | | | | | 427 | Opus FB | ~20 kHz | 20 | 64 kbps, | 428 | | (Fullband) | | constrained VBR | 429 | | | | | 430 | Opus FB | ~20 kHz | 10 | 80 kbps, | 431 | | (Fullband) | | constrained VBR | 432 | | | | | 433 | Opus FB | ~20 kHz | 5 | 128 kbps, | 434 | | (Fullband) | | constrained VBR | 435 +-----------+-------------------+-------------+---------------------+ 437 Test 3 stimuli 439 Table 3 441 The results indicate that all codecs had comparable performance, 442 except for G.719, which had a considerably lower score. T-tests by 443 Greg Maxwell verified that the low-delay Opus at 128 kbps had a 444 significantly higher performance and that G.719 had a significantly 445 lower performance than the other four. 447 4.2. HydrogenAudio stereo music listening test 449 In March 2011, the HydrogenAudio community conducted a listening test 450 comparing codec performance on stereo audio at 64 kb/s [ha-test]. 451 The Opus codec was compared to the Apple and Nero implementations of 452 HE-AAC, as well as to the Vorbis codec. The test included 30 audio 453 samples, including known "hard to code" samples from previous 454 HydrogenAudio listening tests. 456 A total of 33 listeners participated in the test, 10 of which 457 provided results for all the audio samples. The results of test 458 showed that Opus out-performed both HE-AAC implementations as well as 459 Vorbis. 461 4.3. Nokia Interspeech 2011 listening test 463 In 2011, Anssi Ramo from Nokia submitted [Ramo2011] the results of a 464 second listening test, focusing specifically on the Opus codec, to 465 Interspeech 2011. As in the previous test, the methodology used was 466 a 9-scale ACR MOS test with clean and noisy speech samples. 468 The results show Opus clearly out-performing both G.722.1C and G.719 469 on clean speech at 24 kb/s and above, while on noisy speech all 470 codecs and bit-rates above 24 kb/s are very close. It is also found 471 that the Opus hybrid mode at 28 kb/s has quality that is very close 472 to the recent G.718B standard at the same rate. At 20 kb/s, the Opus 473 wideband mode also out-performs AMR-WB, while the situation is 474 reversed for 12 kb/s and below. The only narrowband rate tested is 6 475 kb/s, which is below what Opus targets and unsurprisingly shows 476 poorer quality than AMR-NB at 5.9 kb/s. 478 5. In-the-field testing 480 Various versions of Opus (or SILK/CELT components) are currently in 481 use in production in the following applications: 483 o Skype: VoIP client used by hundreds of millions of people 485 o Steam: Gaming distribution and communications platform with over 486 30 million users 488 o Mumble: Gaming VoIP client with more than 200 thousand users 490 o Soundjack: Client for live network music performances 492 o Freeswitch: Open-source telephony platform 494 o Ekiga: Open-source VoIP client 496 o CHNC: Radio station using CELT for its studio-transmitter link 498 6. Conclusion on the requirements 500 The requirements call for the Opus codec to be better than Speex and 501 iLBC in narrowband mode, better than Speex and G.722.1 in wideband 502 mode, and better than G.722.1C in super-wideband/fullband mode. 504 6.1. Comparison to Speex (narrowband) 506 The Opus codec was compared to Speex in narrowband mode in the Google 507 narrowband test (Section 4.1.1). This test showed that Opus at 11 508 kb/s was significantly better than Speex at the same rate. In fact, 509 Opus at 11 kb/s was tied with the 3.5 low-pass of the original. 510 Considering the results, we conclude that the Opus codec is better 511 than the Speex codec. 513 6.2. Comparison to iLBC 515 The Opus codec was compared to iLBC in the Google narrowband test 516 (Section 4.1.1). This test showed that Opus at 11 kb/s was 517 significantly better than iLBC running at 15 kb/s. Considering the 518 results, we conclude that the Opus codec is better than the iLBC 519 codec. 521 6.3. Comparison to Speex (wideband) 523 The Opus codec was compared to Speex in wideband mode in the Google 524 wideband and fullband test (Section 4.1.2). This test showed that 525 Opus at 20 kb/s was significantly better than Speex at at 24 kb/s. 526 In fact, Opus at 20 kb/s was better than the 7 kHz low-pass of the 527 original. These results are consistent with an earlier Dynastat test 528 (Section 2.1) that also concluded that SILK had significantly higher 529 quality than Speex in wideband mode at the same bit-rate. 530 Considering the results, we conclude that the Opus codec is better 531 than the Speex codec for wideband. 533 6.4. Comparison to G.722.1 535 In the Google wideband and fullband test (Section 4.1.2), Opus at 20 536 kb/s was shown to significantly out-perform G.722.1 operating at 24 537 kb/s. An indirect comparison point also comes from the Nokia 538 Interspeech 2011 listening test (Section 4.3) that shows Opus out- 539 performing AMR-WB at 20 kb/s, while AMR-WB is known to out-perform 540 G.722.1. Considering these results, we conclude that the Opus codec 541 is better than the G.722.1 codec for wideband. 543 6.5. Comparison to G.722.1C 545 Opus has been compared to G.722.1C in multiple listening tests. As 546 early as 2008, an old version of the CELT codec (Section 2.4) using 547 very short frames was found to have higher quality than G.722.1C at 548 48 kb/s. More recently, the Nokia Interspeech 2011 listening test 549 (Section 4.3) showed that Opus out-performed G.722.1C at 24 kb/s, 32 550 kb/s, and 48 kb/s. We thus conclude that the Opus codec is better 551 than the G.722.1C codec for superwideband/fullband audio. 553 6.6. Comparison to AMR-NB 555 In the Google narrowband test (Section 4.1.1), Opus was shown to out- 556 perform AMR-NB at 12 kb/s. On the other hand, in the Nokia 557 Interspeech 2011 listening test (Section 4.3), AMB-NB was found to 558 have better quality than Opus at 6 kb/s. This indicates that Opus is 559 better than AMR-NB at higher rates and worse at lower rates, which is 560 to be expected given Opus' emphasis on higher quality and higher 561 rates. 563 6.7. Comparison to AMR-WB 565 In the Google wideband and fullband test (Section 4.1.2), Opus at 20 566 kb/s was shown to out-perform AMR-WB at the same rate. This was also 567 confirmed by the Nokia Interspeech 2011 listening test (Section 4.3), 568 with also found AMR-WB to out-perform Opus at 12 kb/s and below. As 569 with AMR-NB, we conclude that Opus is better than AMR-WB at higher 570 rates and worse at lower rates. 572 7. Security Considerations 574 No security considerations. 576 8. IANA Considerations 578 This document has no actions for IANA. 580 9. Acknowledgments 582 The authors would like to thank Anssi Ramo and the HydrogenAudio 583 community, who conducted some of the Opus listening test cited in 584 this draft. 586 10. Informative References 588 [valin2010] 589 Valin, J., Terriberry, T., Montgomery, C., and G. Maxwell, 590 "A High-Quality Speech and Audio Codec With Less Than 10 591 ms delay", 2010. 593 [valin2009] 594 Valin, J., Terriberry, T., and G. Maxwell, "A High-Quality 595 Speech and Audio Codec With Less Than 10 ms delay", 2010. 597 [Wustenhagen2010] 598 Wuestenhagen, U., Feiten, B., Kroll, J., Raake, A., and M. 599 Waeltermann, "Evaluation of Super-Wideband Speech and 600 Audio Codecs", 2010. 602 [Ramo2010] 603 Ramo, A. and H. Toukomaa, "Voice Quality Evaluation of 604 Recent Open Source Codecs", 2010. 606 [Ramo2011] 607 Ramo, A. and H. Toukomaa, "Voice Quality Characterization 608 of IETF Opus Codec", 2011. 610 [Maastricht-78] 611 Valin, J. and K. Vos, "Codec Prototype", 2010. 613 [Prague-80] 614 Chen, R., Terriberry, T., Maxwell, G., Skoglund, J., and 615 H. Nguyet, "Testing results", 2011. 617 [SILK-Dynastat] 618 Skype, "SILK Datasheet", 2009. 620 [ha-test] Dyakonov, "Results of the public multiformat listening 621 test @ 64 kbps", 20011. 623 Authors' Addresses 625 Jean-Marc Valin 626 Octasic Inc. 627 4101, Molson Street 628 Montreal, Quebec 629 Canada 631 Email: jmvalin@jmvalin.ca 633 Koen Vos 634 Skype Technologies S.A. 635 Stadsgarden 6 636 Stockholm, 11645 637 Sweden 639 Email: koen.vos@skype.net 641 Jan Skoglund 642 Google 644 Email: jks@google.com