NETVC WG Session at IETF 101 in London Monday, March 19, 2018 15:50-17:20 Area Director: Adam Roach Chairs: Mo Zanaty, Natasha Rooney Note Taker: Matthew A. Miller, Nathan Egge Jabber Scribe: Jonathan Lennox ---------------------------------------------------------------------------------------- Notes from Matt Miller ---------------------------------------------------------------------------------------- Administrivia, Status of Requirements and Testing Documents (Charis, 10 minutes) Testing and Evaluation Criteria (Thomas Daede, 10 minutes) * (Mo Zanaty) MSSSIM vs SSIM? - updated version of VMAF, draft has not been updated to address that - if the results have a large discrepency, perform subjective testing * (Mo Zanaty) close (publish) the doc now? - once the open issue is cleared out - (Tim Terriberry) there is some issues on rate control, but it doesn't have to hold publication * does this get a WGLC? - never did a WGLC because it wasn't supposed to stop changing ACTIONS: * Thomas Daede to publish an update * WGLC once that update is published Thor/AV1 Update and Comparisons (Steinar Midtskogen, 20 minutes) * (Stephen?) what's the relationship between Thor and AV1? - when AV1 started: VP9 + parts of Daala and Thor * (Mo Zanaty) Is the plan to keep Thor going here? - It depends on what happens with AV1 * (Mo Zanaty) What's the pratical (real-time encoding) gap from H264 (720p @ 30fps)? - possible with Thor with a few cores - none of AV1 implementations can do this yet - checked the references Jonnathan was using, and there's a 2% gain ACTIONS: N/A The xvc Video Codec (Jonatan Samuelsson, 30 minutes) * (Mo Zanaty) did you design those based on suspecting IPR encumberence, or by the technology - want make every piece disableable - whatever is practical to separate is separated; not just IPR encumberence or technology * (Tim Terriberry) there are other ways to mitigate risk, such as paying a bunch of money to laywers, or agreements with companies to have royalty-free terms. Thse strategy here is to mitigate the future by trying to turn things off. And trying to make money on the high-end. This is ok if it weren't for past images (e.g., Firefox). To be widely deployable is you need to do that risk mitigation work up front. - We send out requests for patent declarations - IOW, we have to do all the IPR work * (Tim Terriberry) have you actually filed an IPR declaration to license royalty-free? The charter works on the royalty-free baseline, and how good that performance is. - "I'm not sure ..." - This group doesn't require royalty-free? + if the compression is really good and the terms very reasonable, maybe: if the compression is not good ad the term heavy, NO) - We can't "legally" say it must be royalty-free - Different participants have different thresholds for licensing, but favoring the largest deployment * (Thomas Daede) Any comments test conditions or feedback on xvc testing tools and frameworks? - our results were based on the uncompressed framework - we have tested with others, but we have seen much more divergence - w/r/t HEVC, it's easier to talk about feature disparity * (Thomas Daede) Is this on Github? - work on version 1.1 is on Github, but version 2.0 is not released yet * (Ben Schwartz) How you think the management of a dynamic codec will work with a standards organization? - Think the IETF is more agile that others, so can update as needed - we guarantee disabling tools within 60 days (and licensing is with the Corporation) * (Cullen Jennings) I like the enabling/disabling of tools idea; allow companies to enable/disable things as needed * IPR codec litigtation is for many reason; if for money then it starts near expiration to maximize gains * (Harald) Have you actually had experience with dealing with IPR holders - we have not had any reports - (Stephen ??) My experience has been somewhat different (more specifics vs "you [defendent] figure it out") - (Haralld) I like the idea, but am curious how it works in practice (we don't know yet) * (Stephen) Most work on a additive from baseline, but this is substractive from baseline. Maybe this group will take this, which I don't know. But then the question is if we can get the granularity to balance performance/trolls * (Ben Schwartz) Cullen has been speaking about other architectures on media, and using runtime plugins of codecs. If xvc is not a good fit for netvc, it might be good to participate in whatever the "new media" work ends up doing * (Mo Zanaty) You are in continuous dialog -- do you think enough vendors will agree to baseline terms - no, but there's not much use then there's not much interest in defining the licensing terms - we would not make a decision on something close to the charter objectives. We cannot adopt something without a rough idea of what the performant baseline is * Do you think you can figure out the rf baseline before the next meeting? - we have some idea what it looks like, but we haven't gotten commitements - it's very difficult for us to say it is royalty-free * Are the current toolsets closer to the HEVC + extras? - (missed) * You may need to check on the latest to re-evaluate your baseline (correction: only 2% off, not 80% off) ACTIONS: * WG to review the document and have more discussion ---------------------------------------------------------------------------------------- Notes from Nathan Egge: ---------------------------------------------------------------------------------------- --- Milestone's and Status (Mo Zanaty, Natasha Rooney) --- Mo: It seems unlikely that we will have a submitted codec even if we extend for 4 months, let us discuss later. Natasha Rooney: We will have a short meeting today. Mo: No Daala update today (removed from the agenda). --- Testing (Thomas Daede) --- Thomas: Draft has not had updates since the last meeting. Mo: I don't see VMAF and there was also a discussion about dropping MS SSIM. Thomas: Netflix has created an updated version of VMAF (called harmonic). This update resolves issues related to the score saturating. Mo: Does the testing infrastructure have new VMAF integrated? Thomas: It does not. Mo: So the draft and infrastructure needs updating before we can close this out. Mo: What about the Fast SSIM v. MS SSIM issue? Thomas: These produce different results so we don't want to drop one of them. The draft says if you see drastically different results, you should use some other kind of testing like subjective. Mo: So after the VMAF issue, does anyone have any other issues we need to address before we close (publish) the draft. Thomas: If we are going to add VMAF leave it open until next meeting. Tim: If I recall, we were going to add tests specifically for rate control. If we are not going to address that, we could potentially just publish. We could even publish without having another working group meeting. Mo: If anyone feels like we cannot publish now, please speak now (or on the mailing list). I think the preference of the editor will be to close it out once we resolve VMAF issue. Jonathan Lennox: Has this been through WG last call. Mo: No, it has not. The intent was to keep it open while it was changing. We will certainly have a long WG last call period. Mo: For the notes: Leave the document open waiting for the VMAF revision. Then have a WG last call barring any other comments on the list asking to keep it open. Mo: As a tool to help evaluate results, when all the metrics agree it is a no brainer. But when you see there is divergence or significant differences in the metrics, is it useful to try to categorize that on specific content. So we can tell when metrics should be ignored, or taken with a grain of salt. Can we look at test clips and figure out how to classify things? Is this worth trying to do? Thomas: We could, we would need to compare this to subjective results to get a good correlation there. There have only been a few subjective tests on loop filters, so we could make some conclusions there. But no one has done subjective tests on other tools. Mo: This could be good for developers, as certain classes of content have different metrics. Thomas: I have also considered defining a subset of metrics, or a priority to some metrics, and eliminate some that are not useful. In particular, the PSNR Cb and PSNR Cr are redundant with CIEDE 2000. All of the APSNR ones are mostly redundant. --- Thor Update (Steinar Midtskogen) --- Steinar: I will also be pretty short, not much has happened since last time we met. No new tools, only optimizations and bug fixes. Stephan Wenger: What is the relationship between Thor and AV1? Steinar: When the Aliance for Open Media was formed, there were three candidates: VP9, Thor and Daala. So AV1 is VP9 and parts of Daala and Thor. Some of the development on AV1 made it back into Thor. Stephan Wenger: So what is the plan for Thor? Is this group just going to rubber stamp AV1? Mo: Let's hold off on that discussion until after the XVC presentation. As Steinar shows here, AV1 could potentially fit the bill, but practically the complexity is not there. What is the gap to make it practical for real time? Steinar: (interrupted) Mo: To get a 30fps at reasonable resolutions. Steinar: For AV1 there are not any current implementations doing that, which I am aware of. For Thor, I think it is possible with existing CPUs. Mo: You are showing frames per minute for AV1, so we are two orders of magnitude off real-time. Steinar: It is certainly possible to get real-time encoding for Thor. With AV1 it is not proven yet. It will probably take time for implementations to be able to do that. Steinar: I'd also like to note that I picked a bad reference on our last update and wanted to make sure that Jonatan was using as a bad reference. Since February there has been a 2% gain so it seems like he has not picked a bad reference. --- XVC Video Codec (Jonatan Samuelsson) --- Jonatan: Developed by Divideon, open source, exploring possibility of having a RF baseline. Mo: Question about the selection of the tools controlled by the flags, did you design those by speculation on the areas that may be effected by patents, or did you look at coding tools when making the choices. Jonatan: Good question, we tried to have as many flags as possible with as many options as possible, so you can turn off as many things as possible. Mo: So it is not based on IPR or technology, it is based on turning off as many things as are possible. Jonatan: Yes we have flags that turn off things which are not at risk, like bi-prediction. Tim: (On slide 6) Your argument is that high performance tools are newer and have high risk and that low performance tools are older and have low risk and have based XVC on this. However, there are other ways to mitigate risk, e.g. paying lawyers to do patent research, or paying to license other technologies. I just want to be clear that you have done neither of these things. Your strategy is to just turn off the components. Stephan: It is actually worse than that, these people [Divideon] plan to make some money on this at the high operating points. Tim: We'll get to that in a second. I think that strategy would be fine if we didn't live in a world with past damages. For example, if I have been shipping this codec for months, I'm liable for past damages. Tim: In order to make this viable you will need to do some of the other risk mitigation strategies up front. Jonatan: One of the reasons we brought this codec to the IETF is to get more information, and maybe ask for patent declarations on the technology. Tim: I can understand that, but it sounds like what you are asking is for us to do all that work to determine if it is viable for RF use cases. Tim: The other question is, you have talked about the possibility of there being an RF baseline. Have you actually filed an IP declaration and agreed to license your technology on a RF basis. Stephan: I don't know to what extent you are willing to disclose anything. At this point you have not published a draft and are under no obligation to do so. Tim don't push you luck. [Note, there is a draft here: https://datatracker.ietf.org/doc/draft-samuelsson-netvc-xvc/] Tim (let me clarify): If there are things that you don't want to contribute on a RF basis, we would have to take those out of the current codec to understand what the performance is. This group is interested in only the RF baseline. And having metrics for how much the RF baseline would be impacted by making those changes, would be necessary to understand. Jonatan: My understanding is that we are not only considering royalty free codecs in this working group. Mo: The charter does not exclude royalty bearing codecs, however you would need to demonstrate gains to make it worth including. We don't want anything worse than HEVC or other codecs risk-wise. Tim: If you go read the charter we talk about how we will follow BCP 79 (...) in order to prefer selections that are RF. We cannot make RF a codec requirement as the IETF is unable to make that determination, it would require a court to do so. Stephan: Tim: As someone who ships open source code that others can redistribute, the only acceptable license to use would be no-money. Other people might have other business models. Thomas Daede: I was very happy with your result slides, I thought you did a good job following the testing draft procedure. I was curious if you had any comments about the testing draft, and if you had any problems following it with XVC. Thomas Daede: I am also interested in how you got the results. Were there any coding tools that specifically give you the gains, or is it just the sum of the parts? Jonatan: When we did this testing we used AWCY and it was very good to work with. When we have tested with other methodologies you do get some outliers, there are individual sequences where AV1 does better than XVC (some screen content sequences). We also see some difference depending on which metric we are looking at. I don't have any specific comment on the test conditions (your first question). Jonatan: When it comes to what is actually bringing the performance gains, I don't have enough insight into AV1 to comment on that. Compared to HM or HEVC, it is much easier to compare. I have a list (slide ?), these are features that do not exist in HEVC, and this is where we see the performance benefit. Thomas: I have one other question, did you do your testing with version 2.0 or 1.0 of XVC Jonatan: The testing is with all these tools enabled. Thomas: Are all these tools up on github? Jonatan: We have not released 2.0 yet because we have not determined which tools will go in it. Ben Schwartz: I wanted to ask how you imagine mapping a highly dynamic codec (something that changes frequently over time) into an IETF RFC which does not change much over time. Jonatan: That is a very good question. I guess I might be more agile than other standards organizations and simply update more frequently. We don't envision that though, we think it will be somewhat rare that you would need to remove tools. Mo: From the chairs, I don't think there would be technical problem (we have many WGs that have lived too long). I don't see a problem with having drafts coming in over time. Ben: Your company's website has listed 60 days as the time-frame for disclosure of tools. And that you will be in charge of change control. Jonatan: Yes, we will disclose within 60 days. Cullen: One of the things I like about this is the ability to turn off tools individually. One of the things I am involved in is looking at if we think patents are obvious or not, so it could be useful on an individual company basis to be able to turn off features. Cullen: Another comment, for companies, when it is just a money case, typically the optimal strategy is to wait until a few years before the patent is expiring. So you may have to wait many many years to see if there is a problem. Harald Alvestrand: I am curious to see how your process has worked. When I have run into this in the past, I have had comments to the effect of "I have patents in your technology" full stop. When I ask "Well tell me which technology?", they have said "That is your problem" Jonatan: We would still do the disclosure (interrupted) Stephan: I have had experience in this area. These days patent disclosures have to be more specific. I think that it would be very useful to be able to turn off this one tool with a nasty patent troll. However, video codec engineers will tell you it is not easy to do. Stephan: Standards groups however would still like to have this ability because it will discourage patent trolls from coming after a standard. Stephan: Those who worked on H263 you may recall we ran out of letters when making all the different variations of the format. Within a vendor cross compatiblity worked, but across vendors there was fall back to the baseline. Assuming that this group would want something like this, the question is can we get the granularity right. On the one hand, broad enough to not hurt the performance, but narrow enough to rule out the troll. This is what 263 did not get right. Ben Schwartz: I thought Cullen would ask this, but he didn't so I will. Cullen has been speaking in other WG's about new architectures for media and formats, and has reinvigorated the idea of runtime toggleable codecs, e.g., being able to download the codec on the fly and use it for that session. If XVC is not something that this WG is interested in, you may want to consider other working groups. Mo: From the floor, you mentioned something in the draft about continuous dialog with patent holders. Do you have a rough feel for if you will have enough of those committing to RF baseline that performance would be at least Thor like? Jonatan: What I can say is it is kind of a chicken and egg problem. The feedback I get is that as long as there is no interest in using XVC, there is no commitment to make the claims to define the licensing terms. The response that we get so far, many organizations want to lay low to see where this is going first. See who is going to use it. Mo: From a chair point of view, we would not make a decision to hum on something that will not meet the charter requirements. Unless we know what the performance is for an RF baseline, we cannot make a decision to use this over something like Thor. Is it possible that you will go off and make that determination before the next meeting? Jonatan: We have some ideas of what it would look like, but the question is would it really be Royalty Free. It is very difficult for our company to be able to say that this is Royalty Free. Mo: Fair answer. Would you say that the list of RF tools, does it look like JVT or HM + something different. Jonatan: A lot of tools resemble HEVC and some of the additional tools look like JVT. The basic technology is very much on that track. Mo: Double check with Steinar that the blip in performance that happened when someone tried to make AV1 faster is not what you were comparing against? It looked like your dates may overlap. Tim: If you heard Steinar in his presentation, he went back and had a look and that blip was 2% worse than the tip. but not 20% Mo: Thanks Tim. I think there are a lot of good things in XVC that could contribute to NETVC or the IETF. In particular having a regular cadence of releases might line up nicely with browser releases and that would let us regularly update the codec in WebRTC. The idea of having a dynamically updated codec makes sense in the RT case, but the big question is what performance can you get that is still RF.