Notes for RMCAT minutes, as heard by Brian Trammell (thanks!) Magnus Westerlund (RTCWeb update): ================================== Trying to finish core specification before end 2014. Mandating circuit breakers. Trying to provide tools in RTP to deploy proprietary solutions until RMCAT complete. Two interesting thigs coming up: interaction with the data channel, prioritizations. Need input from people working on CC. Evaluating Congestion Control for Interactive Real-time Media Varun Singh -- draft-ietf-rmcat-eval-criteria-00 ============================================================= Lars: Does the working group feel it is okay to split out appendix B? (Yes.) Reviewers: Mo, Zahed, Markku Test Cases for Evaluating RMCAT Proposals Varun Singh -- draft-sarker-rmcat-eval-test-00 ============================================== (Media source description) Varun: Anything to add to media source description? Mirja: Did you integrate all input from last time? (Yes.) Lars: Have you run this by RTCWeb? (Just the video encoder.) Rong Pan: How quickly does it produce a new rate -- sometimes the encoder can't conform to the rate wanted to Varun: This is modeled... Gorry: Haven't read the draft but surely there's a difference between increasing and decreasing the rate? Varun: Just specify in media source characteristics. (Test cases) Varun: Any more test cases you'd like? Fill out using guidelines in previous slides ???: Are you planning to consider data channel test cases? Varun: Considering data channel cross traffic. Do we need to do this beyond TCP? Lars: One thing to talk about TCP/SCTP is whether channel is sender limited Varun: Goes into desc. of competing traffic. Zahed: Ask the working group: is this what happens in the Internet? Please provide input. Lars: Send input soon. List is pretty exhaustive. ???: SCTP congestion control has CWV so less aggressive than TCP. Randell (Jabber): I can help with DataChannel details [Lars: more action items for Randall?] Michael Ramalho: Hybrid of 4+5, long lived data pull from a source, full out and bursty alternating. Rong Pan: Any plan to test larger scale, as in a conference situation? Varun: Scope is unicast. We chose multiple of 10 media sources. Rong: When there's enough multiplexing, some characteristics change... Lars: Each scenario we put on the list we commit to evaluating later. Brant Hirshnen: Consideration of different QoS on the flows? Zahed: Not really in the basic test cases. Mirja: We're chartered for e2e congestion control, no network-based things Rong: We run the *** evaluations, it's easy for us to run 100 flows. Lars: These are the basic. You can do extra credit. Randall (Jabber) via Lars: Agree with Michael R - NetFlix does that too even without explicit DASH protocol (up to 10 seconds on, then off). They'll be moving to DASH (in JS) using MediaSource Extensions Matt Mathis: Shouldn't be too much TCP vs RMCAT. If you're competing with TCP you can't meet first order requirements for RMCAT. We have to avoid this problem through other mechanisms Mirja: We have a coexist with TCP requirement Matt: The only way to do that is push on the queue and increase delay beyond TCP. Lars: We should maybe evaluate in AQM context Matt: Only helps with a short RTT TCP flow. Bernhard Aboba: Requirements document talks about these two contradictory things Mirja: Network mechanisms out of scope in this WG. Would solve problem but not here. Gorry: The aim of building this framework is to use it on the in-ter-net. We will compete with TCP. We will have FIFO. It's okay to look at this. Everything Matt said is true. I like this list.  People can do other stuff. Varun: AQM is there in the topology description Dave Taht: AQM and WebRTC are required for each other to work. randall jesup: you can compete to a degree with short TCP flows.  We do include AQM in the requirements.  And yes, we can't wave a wand and make TCP play nice, but we can evaluate how *badly* it gets hurt, and how well it recovers.  We need to look at it.  We will lose to low-RTT TCP flows over enough time.  We know this. Matt: need to play nicely with on-off flows. Big difference if TCP has always data to sent Arat: Just because we have a test case doesn't mean that a problem is a fatal error. randall jesup: we need to know how badly each alg gets hurt Colin: RTCweb usage draft doesn't motivate support for ECN. Varun: Remove from draft? Colin: If it's useful, put in RTC draft. Otherwise remove Gorry: ECN important for the future. But we don't need it here. Zahed: I know we're focusing on RTCweb Lars: We added ECN before we knew what RTC was Colin: Need manageable scope Mo: Balancing act with reqs and evaluation: candidates shouldn't fall apart, want to make sure alg survives new deployments, whether or not we have test cases Lars: Phase one, experimental candidates, phase two, one standards track. Don't have to do everything in phase one. Does this give implementors a good idea as to how to evaluate? Mirja: everyone happy with this? (hands go up) Matt: Social engineering path: users discover apps don't work together. Charter has requirement for instrumentation. If it doesn't work when competing with TCP, tell the user that. Lars: we seem to believe this is a pretty good list of tests. I would propose that people implement their own algorithm as well as competing algorithms. Do this and get more time to talk. Bernhard: WiFi test case? Does it do rate adaptation? Typical models in simulators do a bad job of this. Michael Welzl: Something on the wishlist: something we also had in the old days of TCP, don't just run each other's schemes, also provide an online unified simulation/evaluation environment Mirja: please send a link when you've done this Evaluation Test Cases for Interactive Real-Time Media over Cellular Networks Zahed Sarker - draft-sarker-rmcat-cellular-eval-test-cases-00 ============================================================================ How many have read this? 5 Varun (on slide 9, description structure): clarification on radio environment, how do I model this? Zahed: You need an LTE simulator Varun: Is there an open source one? Zahed: No Varun: How do I do this test? Zahed: I guess the tools exist, many papers. Varun: Do we use traces? Can we decompose this into something simple? Paul ???: Will need to define the entire test processing chain. Zahed: I agree. Lars: I suspect everyone has different evaluation capabilities. This is why I want you to run other people's algorithms, this would get us better comparability. Varun: We need clarifications on mobility modeling Zahed: in 3gpp spec, I can link to them. Vijay: Re simulator, ns3 has LENA. Varun (on slide 10): Best effort bearer, is that ack mode or unack mode. Does RTC say something about this? Zahed: I want unack, there are deployments with ack. I expect unack. Mo: In general, are there aspects of LTE that would make a solution candidate work better if the app knew it was on LTE. Is there benefit to knowing? Lars: Want to push back on link-type-specific rmcat algorithms. We want an algorithm that works reasonable well everywhere. LTE networks have lots of knobs, not clear results portable between operators. Paul ???: Cell throughput, useful for operator but not useful for user. Raises issue for one-size-fits-all solution: this metric is only useful in wireless environments. Michael Ramalho: haven't dealt with link layer, but we talk about channel delay. Mobile is incredibly important going forward. If we can infer mobile is bottleneck, we'd be doing the industry a disservice by not adapting to that. Lars: Solution candidates optimized for one case are not good. Glor.. Ti...: this is repetition of section 5.1 or previous draft, why is this separate document Zahed: in basic test, we're looking at internet behavior, here we're looking at a single link type, these are two different discussions. Randall Jesup (jabber): I struggle with the complexity too; I agree we need to know how well it does on "generic" LTE/etc cases Magnus: Very important to find things that work in most cases. LTE has knobs. Wifi has variants. We're going to have to pick cases and see how well they work Lars: agree, would be unhappy with three algs, one good for lte, one for wifi, one for fixed. premature optimization is the root of all evil. ??? (Qualcomm): Definitely support scenario, maybe unify with other document, and have basic/advanced test cases, third stage could be *** quality? Lars: let's not talk about how many docs. Mo: bad idea for any link specific candidates, but useful to abstract out differences in these different secnarios, let the app know that. propose for wireless networks to distill their environment down to a few factors. Jorg: for which you need a lot of statistics... trying to simplify things, 3g eval of CC had scenarios such as elevators, buildings, outdoors... we shouldn't go into models, but similar abstractions as building blocks for looking at mobility between regimes... multiple documents should use one set of traffic models. Zahed: traffic models are same as other doc. Mirja: This document is useful. But not adopting it soon. People can already use this scenarios for testing. RMCAT Application Interaction [slides] Mo Zanaty -- draft-zanaty-rmcat-app-interaction-00 ================================================== Zahed: I don't get what you're saying with the conceptual model, the config should go everywhere, not just to codec. Mo: Implementation model, typically congestion control is in the RTP stack. Zahed: Is the codec controlling the RTP stack? The config only goes to the codec? Mo: First thing to get agreement on is model Z: what is goal of the document Lars: it's a milestone, mo wrote a doc (Brian clarifies) Mo: config should branch out to all components, not just through the RTP stack Colin: the way I thought this would work, I would put CC between RTP and codec so it can mediate Mo: it's on the bottom because it does transmission shaping Colin: clearly implementation is fuzzier. wise to split RTP and RTCP conceptually. algs will treat feedback differently. Mo: separating feedback from media useful Bernard: there's feedback you get and your response, those are distinct things. Varun: having pacer/shaper separate is useful. Brad Hirschmann: draw this in separate control and media planes with CC in control plane Bernard: codec also does control. Varun: (on config-codec interactions) this is a config for the whole application, not just for codec... Colin: you want to negotiate RTCP XRs. Mo: good point Michael: what we'll need for coupled congestion control is the rate the codec is able to use, will send to list. Mirja-from-the-floor: this seems like a list of all things you could exchange, what I'd like to see is a smaller list of things you really have to exchange, there is a risk people might misuse too much information Colin: we want a way to tell the codec "i can tolerate a burst of this size", for iframes etc. Zahed: we discussed responsiveness in the usecase, this is maybe more important, how frequently can you change or how much time you need to adjust. Mo: captured in elasticity, maybe have as separate. Roni: not just from codec to cc, also from app. Mo: codec is fuzzy boundary, might not be a codec at all, e.g. in switching mixer Lars: isn't most of the CC-UDP interaction here just something the CC should do? Mo: if the underlying stack provides this features cc can use them. Mirja: feedback on the list from app designers and cc developers Lars: good first start, going broad, feedback should include which features to flesh out more Jorg: is there a list of three or four interaction that are most important? Mo: already in prio order on codec-cc slide. Progress on Practical Shared Bottleneck Detection for Coupled Congestion Control David Hayes ================================================================================ Michael Ramalho: your delay plot looks like a 2-sided dist, packet nets have a 1-sided dist. Mirja: how did you generate the different traffic loads? David: using Tmix traffic generators Matt: can't imagine these dual bottlenecks being common David: these should be harder than we'd normally find Zahed: how fast can this react? David: Slower at startup. If the flows are already up and running, the change in the stats come out more quickly. Order of seconds, not milliseconds. Much faster with smaller queues. Zahed: Cross sources? David: None are greedy sources. Trace-based Tmix sources. Lots of short flows, big flows, all tcp controlled. Packets sent at varying rates. Mo: Any faster convergance if bottleneck is closer? David: No. Stats faster with shorter queues. Can be faster with higher error rate. MPTCP might be willing to make this tradeoff Rudiger: Isn't the number of datapoints very important? You're measuring correlation. David: Take summary stats, if different by threshold, divide into group, not quite correlation, grouping problem. Varun: For specific video source, are the packet sizes equal? David: Not going out of my way to model video. Update on Google Congestion Control Algorithm for Real-Time Communication Stefan Holmer -- draft-alvestrand-rmcat-congestion-02 ========================================================================= Zahed (on gcc vs tcp): what kind of TCP source is this? is this one object fetch? How does TCP adapt? Mirja: i.e. these are greedy sources? Stefan: yes Michael Ramalho: bottleneck queue is 350ms? Stefan: I'm not sure. These are RTTs. Michael: The dynamics of TCP are going up and down consistent with 400MS RTT. Alg is workig as designed. Did you try on other TCPs, with different RTTs? Mirja: More eval next time. Michael: With a lot of TCPs, you'll fill the link and tail drop. Do you lose bandwidth again Stefan: DV will be small Lars: Folks that have alg proposals, talk to Varun and Zahed, pick scenarios with quick result generation, more presentation time if you simulate other people's algorithm Mirja: No candidates without external evaluation (half-joking) Lars: Will summarize to list.