IETF 82 CODEC Minutes ----------------------------------------------------------------- 0. Liaisons (Chairs - 15 min) Cullen Liaison documents available, treating their comments as LC comments. No questions ??? - study group 16 meets next week or week after. ==> ACTION: Any responses to Study group 16 should be sent by early next week. JDR - has already sent comments. ------------------------------------------------------------------------- 1. Listening results (Christian - 15 min) draft-ietf-codec-results-00 Paul Coverdale - If there are any results for Opus ??? AMR is the lowest score A few changes have been made to silk stereo that may improve performance. Jean Marc - the rates for the stereo test were lower than usual testing, which explains the quality. Paul C. - Did opus have different delays, were they consistent. Jean Marc - ??? the delay in AMRWB+ is ??? Opus was running at 20ms. Cullen, to the group - Comments? Group silent. ==> DECISION: Cullen - Take the following issues to the list: Shall we run objective tests on speech and audio quality as well as runtime and memory consumption Merging with draft-hoene-quality-01 to include some more textural description.s Christian says ok. ------------------------------------------------------------------------- 2. Opus: (Jean-Marc - as long as it takes) Discuss any WGLC comments draft-ietf-codec-opus-10 Jean-Marc covered updates to the draft. Time stretching - added pointers to algorithms. == Compliance == Eric Norvell - the bitstream is still normative even in an informational draft Jean-Marc - ?? Stephan Wegner - I don't care if the decoder source file or the human readable text is normative. Both work. There is a conformance spec that includes tools and description of an algorithm, how to use the tools, and what do to with the results. Jean-Narc - it's in the draft. Stephan - We agree, but has to be defined here. (something about lawyers) Jean-Marc - the open question - How sensitive is the tool? Stephan - pick any metric, but don't leave it open. ?? - I care how sensitive it is. The fix and floating point both passed, got green light. There will exist decoders that sound equally good but won't pass. Jean-Marc - It would be unlikely for that to happen. Stephan - If decoder implementors run the tests and they are red-lighted, then they will tweak their source code. Jean-Marc - do we want to make more optimal encoders by setting the bar lower? ?? - ?? Beyond legal, ?? Stephan - if someone comes up with the new decoder, they can write a draft and we can get ipr declarations. Christian Hoene - make the test lest strict and use peak. Jean-Marc - can't use peak. Can't use proprietary standard as compliance tool. Eric Norvell - if the test doesn't allow imperceptible differences, then the test is too tight. Jean-Marc - who in the room cares? Cullen - There's legal issues - need a red/green light. As for the other error condition, would it be better to allow things that are compliant but sound like crap (broad tolerance). Or narrow - noncompliance, but sounds fine. Jean-Marc - I have another option. Float and Fixed point pass the test - put the bar there - we catch errors that are perceptible. Allow errors for float to fixed point conversions. Schwarz - from proposed to standard. can we relax the standard if it's too tight in practice? J-M - yes. Stephan - It's acceptable to say you are compliant when you pass either the fixed or float or pay lots of money that says you're compliant. So long as there's an open source choice, that's ok. Cullen - Anyone have an analyzer and are willing to ? Christian - I can add conformance test for peak within a few weeks. Having one test vector for fixed and float make sense. J-M - ?? we allow a range around ?? (something about fixed and float) Eric - it doesn't need to be a quality test. Some implementation degrade the quality. Cullen - we all agree there has to be a clean test. Stephan - except for ?? patent, it only covers the decoder. J-M - The Skype patent covers the decoder. Christian says to Eric - I do not agree. Eric - which part? ?? - I support a looser test for the decoder. Cullen - there are three issues: - Should the error margin for the test be broad or narrow? - How do we do those tests? - like use peak J-M - on the second one - only unless someone writes it. Cullen - lets talk about peak. Christian can define a peak-based metric. J-M - don't know if its feasible. Ben Schwarz - can the test boundary be moved after the proposed standard is released? can we change the conformance test? Cullen - We would need new IPR statements. J-M - the IPR statements cover future changes. Stephan - IPR comes into picture if one of these 4 companies sue each other. I'm pretty sure we are fine. Cullen - I was wrong previously. We can do this. Christian - peak has be used for years. The current tool hasn't been tested by third parties. J-M - The tool was written for the codec. Stephan - this is marketing. If a company wants to trust a testing tool that is more well known than what we came up with. That's their choice. Gregory Maxwell - The tool was tweaked to fail real bugs. Don't know about peak. J-M - we need to run peak with real data. Eric - The tool should be used for verifying conformance, not catching bugs. ?? - maybe we need two tools - one is for finding bugs, one tells you if your implementation is conferment. Why use a tool that excludes good sounding implementations? Cullen - We are not writing tools to find bugs. J-M - We need to experiment. Cullen - No, we don't. What are we trying do here? Tools for checking IPR. Anyone proposing super narrow for IPR check? Ben Schwarz - Super narrow. Matthew Coffman describes an Apple II - 6502 cassette port implementation. 7 out of 10 words is a reasonable pass. Ben Schwarz - optimize for known objective function. A too narrow test can always be broadened, but not vice versa. Stephan - I'm in favor of wider tests. Diffuses the risks of implementation. J-M - if you pass the test vectors, you pass the test. Ben Schwarz - How am I supposed to optimize the encoder if all decoders sound different? Stephan - This is about compliance. gregory - can relax it by 1db. Christian - What about one test for legal, and wider test for interop? Stephan - bad idea. If you confuse the people in this room, you will confuse the jury. Cullen - take a hum on three options: - Wide - a broad metric, bad sounding could pass the test. - Narrow - the results were audibly the same, float and fixed points still pass. - Don't know/Don't care Ralph - what's the option in the current draft? J-M - Narrow Cullen - Those in favor of wide approach, hum now. Group hums (2 on jabber) Cullen - Make the test very narrow, hum. Group hums (7 on jabber) ==> CONCLUSION Cullen - On the list, people support narrow. In the room, wide. Weak consensus. Take to the list. J-M - I will try to find a compromise and take to the list. -------------------------------------------------------------------------- Opus Testing Tim Terriberry presenting. presentation will be on the website after the wg meeting. Lennox - did you test on any platforms that didn't have 16-bit byte? Or 9 bit bytes? ?? - no. (?there was some port tested briefly?) No further questions. -------------------------------------------------------------------------- Issues found during WGLC Koen Vos presenting == Issue 1 == Tim - We found these issues in the second implementation Tim - We've run dtmf tones to see if they are decoded correctly. == Issue 2 == Lennox - packet loss during DTMF is worth testing. Vos - I'll test packet loss during DTMF. Cullen - does anyone object to the solutions? No objections. ==> ACTION: Cullen - make the changes that were presented. == Opus testing with network traces == Paul C - on packet loss, how many frames? Vos - One frame per packet Paul - What if you put more frame in a packet, has a bigger impact. Is error correction only good for one frame. Vos - Only need FEC is you lose 2 packets in a row. J-M - if you pack 2 frames per packet, your burst will be shorter. Cullen, to the group - any questions? None. Cullen - summarizing for the minutes: ==> ACTION: Update draft with WGLC comments. ==> ACTION: Get more testing info for DTMF. Paul - what about modems or hard of hearing? J-M - I would put that into test that we do after standardization. Brian Rosen - Test TTY tone when doing DTMF. It's the only tone based signaling used by deaf folks. Modems aren't interesting for accessibly. We have problems getting TTY through. If you can get DTMF though you can get TTY. Cullen - do you have files? Do we send you files? Brian - I have a guy who could do it, have to ask. Christian - DTMF is required, fax and modem excluded. Gregory - assume they don't work, is that a blocker? Laughter. Stephan - I can make files with various error percentages available for testing. However, media coding people need to sanity check them first.