Notes -- ICCRG Meeting begins at 4:25pm Thursday 21 July 2016 

1) Congestion control issues in 4G/5G networks (Ingemar Johansson)

Ingemar: don't rely too much on RTT measurements since they can be noisy...
Ingemar: [explains the LTE protocol stack, slide 3]
Lars: if you could touch on ARQ schemes on 5G networks...e.g. if they're intelligent
Ingemar: [implicit OK]
Ingermar: [explains different kinds of scheduling mechanisms]
Lars: Question from Simone Ferlin on Meetecho i would like to know if the delivery is deterministic related for TTI for example
Ingermar: every 1ms you make a desicion, and it might take 1-10 ms depending on the load and how it's scheduled. resource allocation also plays a role. 
Joachim Fabini: confirm that depending on your absolute time of sending your message you can lower your end-to-end delay, when you use 10 ms TTI
Ingermar: ...explains that ACK are piled up in uplink, you might get an impression that downlink is congested which in fact is the uplink that delays the packets... 
Ingermar: one shouldn't rely heavily in inter-packet arrival either...
Ingemar: with DASH if you don't do packet pacing you get large chunks (with high cwnd) if you don't do packet pacing 
Joanna Kulik: is that for uplink or downlink?  For quic we're pacing uploads.
Ingemar: I only show for downlink
Lars: A suggestion, there was PILC group, Performance Implications of Link Layer Characteristics, described to link layer developers what TCP would like to see; maybe do an inverse document? 
Ingemar: if you consider L4S you have a higher chance of getting bursts (with ECN marks)...the solution here is to do the packet pacing.
Stuart:  I am happy to hear you talking about ECN...how much of it is theoretical and how much of it is likely to go into deployment
Ingemar: I am quite sure but I can't tell
Stuart: in today's networks...often people mean the network is broken when they talk about congestion. congestion is not broken, it is the normal state of the Internet and if you don't lose a packet it has to be retransmitted which adds delay. Else TCP will go faster and faster and then you'll get big queues with delay, and then the delay-based scheme will get the wrong implication and that's why i have been advocating ECN since it lets you tell the sender "please so down" without the cost of throwing a packet away. These issues you're raising about getting false reductions in cwnd become less of a problem. You won't get false reduction in cwnd often. One of the down-side of the pacing is if you don't have the block of packets for FEC, then it's gonna be difficult whereas the ECN gets it right... the right feedback without excessive buffering
		Stuart: with .11ac you have the packet batching. you cant batch if you don't have a batch. pacing is good for some things but bad for batching. and FEC is a similar batching operation
Ingemar: depends on how pacing is implemented...here i am relying on Linux TCP stack, in slow start you pace at a rate that is 50% higher than the link bandwidth, you have back-to-back packets.
Ingemar: I agree with you about ECN...
Stuart: 50% of the connections in IOS8 use ECN. Hope in the future all connections do it. 
Chair (Jana): can we continue to the end of presentation? 
Ingemar: AIMD CC have problem following throughput...
Ingemar: There is an example code in the draft I published recently 
Lars: From Jabber why TCP cubic? should the user get bandwidth in the order of GB? 
Ingemar: for the fun of it, could do other congestion controls as well
Lars: it seems that physical layer do a lot of care-feeding for the upper layers to mask the packet loss... is it too sensitive to packet loss? Shouldn't it be solved in the transport layer? What is the loss distribution going to look like if we turn of all link layer retransmission?
Ingermar: interesting debate [he doesn't get to asnwer, time is running, Jana (chair) ends the session] 


2) Performance-Oriented Congestion Control (PCC) (Mo Dong)
[Audio problem at the beginning] 
Mo: all the logos in the slide 3 aren't happy with TCP even though it's been there etc. 1988 and they use their own prop. solutions
Mo: TCP fails to achieve a constant high performance even with so many years of research...
Mo: all CC mechanisms [e.g. reno, cubic, fast, etc on the slide] do hardwire mapping between between the event and action 
Mo: the real network scenario doesn't match the assume network scenario e.g. in LTE networks
Mo: if loss is purely random we should actually increase the rate to increase performance
Mo: there is no hardwire mapping that can achieve optimal performane. Let's throw away the event-controlled mapping which TCP is based on. threat network as a backbox and do experiment with it. Send with different rates and get different results. 
Mo: [uses a utilility function based on sending with different rates]. Try at rate 1 and get Utility 1 and so on. Then compare U1 and U2. and if U2 is bigger than U1 then more in the direction of R2. 
Stuart: is the U (utility function) merely the rate that arrives at the receivers? or you take into account other accounts? e.g. shared radio spectrum...if i care about the resources i am using, then it's not the same. 
Mo: we'll get into that in following slides. 
Mo: we're proposing the Performance-oriented CC (PCC). 
Mo: selfishly maximizing utilizing is not necessarily bad...do we converge to a Nash equlibrium? 
Mo: AIMD is basically a hack to achieve asymptotic fairness. 
[Mo's voice is disconnected]
Mo: PCC doesn't need AIMD because it looks at real performance. 
Mo: Game Theory force will push the two senders together without using the AIMD hack.
Mo: PCC is smashing all the flow's rate together (meaning the have almost equal rates) slides 23. 
Mo: [shows a long list of things they have done, since they don't get to talk about it, e.g. fairness, nash eq. proof, etc]
Mo: PCC v2, for fast convergence to optimal in tuny casuality window, etc but we're working on it and it's coming.
Mo: our code is open source, and there's a webpage on the slides.

3) Verus (Yasir Zaki)
Yasir: I'd like to present adaptive CC for cellular networks. we want to provide good fairness. Verus uses the delay feedback, it's proactive and has small signalling overhead. The main idea, was that we didn't want to model the channel or prediction. We build on TCP concepts, e.g. use of slow-start. We use the delay-curve which is the reflection of the relationship between the delay and sending window. in every 5ms epoch we decide how much the cwnd is going to change. 
Yasir: [shows the delay-curve concept on slides]
Yasir: Verus dynamically learns the delay curve by updating the delay feedback.
Yasir: you rebuild the curve every 1 sec. 
Yasir: [shows similar experiments that was inspired by PCC] 
Yasir: both TCP CUBIC and Vegas suffer from high delay whereas Sprout achieved good delay. 
Yasir: open source Verus over UDP on github. Verus sockets is ongoing work. also use as adaptation layer over CUBIC and as a CC module within QUIC (planned)
Yasir: while CUBIC is widely used, what happens if you run your CC on top of CUBIC? it's odd but it adapts useful since CUBIC opens a large cwnd and the Verus CC can be used as an adaptation layer. [shows verus over cubic performs good in delay in slides] 
Yasir: we use two-dimentional markov-chain to model Verus.  We use the markov model as another CC protocol. 
Ingermar: interesting presentation but what about complexities e.g. with CUBIC? 
Yasir: less or similar complexities compared to cubic. There's not lots of computation in here, you just have to maintain the delay curve and the protocol is simple  
[presentation ends here]

Jana (generic comment): we have ppl from google, apple etc. please talk to them and tell them if these stuff isn't useful and if not, tell why? 

4) Assembling a Pantheon of Congestion Control (Keith Winstein) 
Keith: measuring all ppl's CC scheme in both emulation and in real-life. throughput of cellular network is highly variable and therefore skype doesn't work well while on the road. 
Lars: remote attendees can only see Slide 1.
Keith: there's lots of bufferbloat. we measured the cellular networks (e.g. skype, google hangouts and facetime) while driving and we put them in an emulator called MahiMahi. 
Keith: [shows skype over verizon LTE throughput and delay plots] 
Keith: we built a protocol called Sprout that tries to keep the delay less than 100ms. model variation in link speed, infer the current link speed, predict future link speed...
Keith: packet inter-arrivals time looks like a poisson process. 
Keith: Andrew McGregor was right and it's not poisson and it's flickery. 
Keith: [shows Sprout results with Skype]. Sprout makes a cautious forecast and stays below the 100ms and fixes the delay problem although you may not have a resonable video quality [during the delay-spike time].
Keith: the effect of delay concious CCs would be the same effect as if running AQM with normal CCs (e.g. CUBIC).
Keith: [shows MahiMahi emulator demo]
Keith: 3000 attempts in student competition for beating sprout. 
Aaron: [about the student competition results] is that on all links? 
Keith: it's on one particular emulated link.    
Keith: SCReam has the best delay-scheme [but with low throughput].

5) Open Q&A and discussion with the panel
[all speakers are now behind the mic]
San Jei from Verizon: i think the presenters starting with Ericsson was really good. one tool we use for CC which is "keep adding capacity". we're definately looking for some of these innovations that come through like ECN, any scheme we'd like to engage and do experiments and trials with the tools you have. 
Keith: it seems to us quite clear that the carriers and chip vendors should deploy ECN and AQMs and should have been done 10 years ago. What are the obstacles of deploying PIE and CoDel. Can you help us understand how we can deploy AQM and ECN? 
San Jei: many of these solutions are in the network and they're working and depends on the region where the network is deployed and there's no barrier in technology. 
[guy in blue shirt with beard]: physical layer has time-slot in that is periodic. the data is accumulated until it's transmitted. i have so many uncertanity factors in PHY and then TCP comes on the top and tries to be smart. If we had done the PHY to be smart (e.g. with UDP on the top) perhaps we could do better. 
Ingemar: is that FEC you're after? 
[guy in blue shirt with beard]: many schemes that are replicated across layers!
[guy in blue shirt with beard]: [to Keith] if you load the traffic if you put on the network, the network reacts. this is missing from emulation.
Keith: i want to encourage us to have an empirical understanding of the things. i don't think this is an academic problem. the way is to find if it's accurage enough calibrate the emulation based on experiments in real life. i am not sure if the emulation is faithful enough until we measure it.
Michael Abrahamason: as far as i know there's no ecn and it's just fifo with bufferbloat. i really love the whole sprout presentation when it says take the whole things in account (e.g. ecn marking, delay, packet loss, etc). i think you're compl. in the right direction. make sure there's a little bit of buffering for the scheduler. but 100ms or even 50ms is little bit too much! when cubic is there with 4s delay there's not much you can do as long as someone else is in the queue. 
Jana: why AQM or ECN isn't deployed? 
Mikael Abrahamsson: the common thinking is that the packet loss is bad. i know vendors that have 400 packets in the queue. 
Jana: i think we agree on the problem. who and how we address this problem?
Dirk Kutscher: one reason why aqm/ecn and better e2e cc isn't deployed, there's an allergic reaction to non-managed networks. some of those functions that e2e cc and aqm could do is often done by traffic management. we need to demonstrate that better technology works much better, is easier to operate and cheaper. we should do more experiments, feed in more models.
Ingermar: operators are looking too much at speed test (throughput)  
Roland Bless: there are natural limits on what you can achieve with AQM. you have to deploy AQM to get better delay with loss-based CC. the delay-based CCs have problems with co-existing with the currently deployed schemes, they get suppressed by Cubic etc so we can't use them easily together. If you have multiple flows AQMs can achieve good performance, high util. and low delay. if you want to do that for few flows, you have to change CC scheme. 
Jana: low delay and high utilization, that's just by aqm not a change in e2e cc.
Roland: yes, is it many flows, some flows..?
Keith: the cost and benefits of AQM is well-studied since it's already been told by Sally, Jim and others. Huge improvement of the status quo. This is a trillion dollar industry, strange if they're not deploying this because they want to win speedtest
Joanna Kulik: fairness is another problem? our own flows against our own flows. when we put two flows against each other, one just sort of eats the other. do you have a metric in mahimahi to check how the same algorithm works against itself? 
Keith: we are not doing that yet. 
Yasir: we experiment with fairness, with CUBIC and Sprout and they seem to be fair with each other and they don't seem to eat each other. 
Aaron: what about reno? 
Yasir: we checked, but not entirely sure.
Andrew McGregor: we run a system that is very analogous to 4G/5G system. admission control for the network is a hard problem, and it's beyond AQM.   
Brian: what happens if we turn off the mac layer loss masking? 
Ingemar: i wouldn't like to call it loss-masking! it's a retransmission in mac layer. it's a sort of optimal use of capacity? 
Brian: what happen if you turn it off? 
Ingemar: you get 10% [loss rate?], at the most 1-2 retransmission. i never really considered to turn it off. 
Brian: maybe try it. 
Brian: all session talking about CC for mobile. assumption is the bottleneck link on the mobile network but it goes over the network so there's something probably on a datacenter. bottleneck isn't always in mobile network. 
Ingemar: you have a shared spectrum which you need to use optimally and get the most of rate spectrum with a given power. 8-16ms extra delay with retransmission and i don't know if that's harmful for e-2-e performance. if you are looking for really-low latency we need to look at other solutions. 
Michael: ask the speakers to be concise
Koen: i didn't see any experiments related to immediate ECN and DCTCP-like CC. 
Yasir: our target was cellular mobile because of unpredictiblity and high change of throughput. 
Keith: we accept pull requests to add scenarios to text and add schemes (referring to his presentation github)
Michael Abrahamson: in the future, ppl are gonna be behind lossy, bloated, lossy links and transport needs to do this, AQM needs to be in the network, some are gonna fixed and some gonna be lacking, you can't push it somewhere else, everybody has to work together, i have no idea on how to do that. 
Stuart: an earlier comment, all users care about throughput, because that's all they see when they look at speedtest...there's some truth about that, quick shout-out for dslreports.com where they show rtt as it varies during download and upload.
Christian Huitema (Microsoft): follow-on comment that cubic is making it messy for everybody; prisoner dilemma, you want to be nice, but you know if you're going to be nice you're gonna lose, and that's a dilemma. known solution default is punishing. could someone invent CC that is mostly nice but [gets the most performance]