LEDBAT Hiroshma - Note takers: Alissa Cooper & Eric Burger LEDBAT document: Need to document need for real randomness in algorithm. Both in terms of implementation (don’t count on disk rotation latency, etc.) and theory (non-convergence). Document late-comers ADVANTAGE Explain where constants come from. Give a single, specific value with a range. Stas TCP congestion control designed to maximize throughput, but not address latency if you ignore one characteristic, may be fine, may not be algorithms can be absed on either characteristic or a combination TCP based on throughput Stas approach: keep latency fixed draft has new name: draft-ietf-ledbat-congestion were several open issues, the one non-trivial one was addressed: non- congestive constraint non-congestive constraint: app may be rate-limited itself in this case, congestion may not be encoutered for an extended period, queuing delay is zero congestion control would continue to climb forever if congestion is later encountered, app will have to start from a point where it thinks it can send much more than it should not only will delay be high, but app will encounter loss --> sub-optimal sender should never make assumptions to deviant from what it has done in the past proposed solution: flight size flight size (fs) = amnt of outstanding data (sent but not yet acknowledged by the recipient) fs only meaninful to the sender fs is effective window *right now* keep congestion window on a leash attached to fs cwnd <= ALLOWED_INCREASE + TETHER*fs where TETHER is your leeway parameter tether needs to allow for at least one more packet to allow you to probe, so can't be < 2 but tether < 2 may be too generous once you've ramped up, so we add additive constant (ALLOWED_INCREASE) so tether must be > 1, Stas recommends at least 25% increase ALLOWED_INCREASE and TEHTER are constant A_I must be at least 1 packet, but not more than 3 packets TETHER must be > 1, should be between 1.25 and 2 fs is implemented in all BitTorrent products that use uTP known document issues: framing is out of scope -- discussed during charter discussion need to capture discussion about late comer's advantage (lack thereof) need to discuss whether we have fairness between ledbat flows need to discuss choice of parameter values, e.g., when there are ranges, what tradeoffs are between either end of range check for loose ends in pseudocode and spec -- need new eyes on this Document why no framing in LEDBAT document framing out of scope: wire representatin is out of scope but need to be able to carry two quantities: time stamp and delay without time stamp, can't measure one-way delay recipient needs to be able to convey info about one-way delay to sender two quantities go in opposite directions (sender->recipient and vice versa) Recommendations document: Split out DIFFSERV into a different document (if people care about DIFFSERV) Be absolutely sure to clarify difference between practices that work around bugs versus practices that are good to follow. Specifically, approaches to work around broken HTTP implementations (e.g., no pipelining) Please send text; formulate things in terms of “It would be in your interest NOT to do xxx” instead of “You SHOULD do yyy.” Low Extra Delay Background Transport: Stanislav Shalunov ===== Framing ----- Rich Woundy: Transport level may or may not have transport identifiers. Depending on transport protocol, you will need to be able to identify stream. Slav: need it for fine-grain monitoring. However, "this is the last latency I have seen" should be good enough. Reording should not be much of an issue. Rich: worth highlighting order issue. If you get stuff in the order 3-2-1, you will think packet 3 has great latency, 2 has normal latency, 1 has awful latency. Slav: if measurement error is small, should not impact algorithm. If lots of variance, then ledbat algorithm has failed. Murari: if I am doing framing, why would I make it sequence-number specific? Slav: not a stream-domain protocol. Lars: Not doing framing, but should note how to frame: good, bad, recommended. Should be in the document. Not normative, but suggestions. (ACTION) Late comer's (non)-advantage ----- Lars: implementation depends on side effects on user's operating system, and looks like only might work by serendipity. It seems to be just a hack that might not work all the time. Slav: if your machine is so good, then you can introduce noise / random fluctuations in inter-packet transmission time. Stewart: As world moves to flash disk, high-performance CPUs, timers are more and more precise. Cannot depend on the end system introducing random noise. Murari: problem is non-existent on many systems. Rich: Assumption about a backoff time is random. Need to clearly state expectation that things back off with a random component. The random component is an implementation detail. (ACTION) Lars: Better to have something explicit in the document. Slav: Need to flush queue to figure out RTT on second stream to really know. Lars: That is too heavy weight (Slav clearly agrees) Bob Briscoe: Late-comer's advantage: please add it to the draft as described on the list. Disadvantage: late-comer does not know base delay. Advantage: if it does know base delay, other flows will not necessarily get out of the way of the new flow. Draft says randomness fixes this, but that means randomness needs to be ensured (ACTION) LNS Fairness ----- Bob: Need to write down assumptions. Random walks with a lot of flows competing with each other; do not see how ones will a small amount of bandwidth will ever get large or vice versa. Need to state why randomness helps and importance of true randomness in the algorithm. (ACTION) Andrew: randomness protocol adds needs an estimator to know what randomness is available so it knows how much randomness to add. Bob: May be we see randomness in networks because today’s protocols have randomness built in. If on a ledbat-only network, no randomness. E.g., data center Slav: experience is randomness still there. Stuart: every network does NOT have randomness. Slav: every network I have TRIED has randomness, but not necessarily true for all networks. Stuart: MacOS 8.6 (?) had this problem; throughput cut in half. No randomness, so everyone had the same collisions repeating. Bob: just need to note this in spec. Parameter Values ----- current reqs may be unecessarily strict, target and gain both fixed target must be 25 ms or more (and should be < ??) gain must be 1 mss/rtt, but what if you ramp up slower? Bob: very uncomfortable about target delay at 25ms Murari: if we are not strict (MUST), then bad actors can set values better for themselves but worse for the network Lars: bad actors can always chose, e.g., low TARGET values Stas: in BT, there is competition, several connections share same bottleneck (home uplink). 4-5 uploading connections split it up based onr andom walk. 2 <= base history <= 10: higher values give better measurements of base delay if neither of the other factors is active but there may be situations to deviate from sweet spot (perfect clocks, no rerouting) 1 packet <= noice filter <= cwnd/2 (should): at low end, you get faster reaction and lower delay in control loop -> more stable, converge faster. probably not a good idea to make noice filter more than 1 cwnd (1 RTT) bc you will double your response time. Lars: This is the "Magic Numbers" slide. Numbers do not seem to have any justification. OK for experimental protocol. Need a lot of data on what we think will be eventual PS solution. Slav: can get rid of the NOISE_FILTER number; almost always will be 1 packet, which means it is not considered in the calculation. Today, these are ranges because we do NOT know what the value should really be. That is why we are doing the experiment. Jana: these values came from somewhere. How about: “Here is a mechanism, and here is a constant. Here are the implications of moving the values.” Should explain all of the constants this way. (ACTION) How can I see if the protocol works if different implementations use different parameters. Where is the failure, if any? Implementers will just copy and paste from spec, so why not give preferred values? Lars: “It should be this number, but here is the range.” Andrew: Bottom 5 numbers are consequences of the choice of the first number. However, the first number seems to be pulled out from nowhere. Slav: Yes, it really IS a magic number. Implementation ----- Rich: Has anyone done an ns2 or ns3 model. Slav: some told in private, but nothing public. Murari: one implementation known of. careful proofreading: need implementors to read (esp section 5) best way to read is to implement Rich: might be interesting to take the model and run it against different network topologies. Vijay Gurbani: have implemented BT in NS 3. doesn't have docsis but does have a wireless model. Stas: and don't need to simulate in context of BT, because that has been done already. General Questions ----- Jana: Do we have data we can look at data from BitTorrent implementation? Slav: Can distribute test executable for people. Not open source. Jana: Not using TCP mechanisms. There should be discussion about burstiness considerations. Slav: spent lots of time looking at rate-based, window-based, etc. methods. Not sure how much should be in the spec. Since spec is for implementers, not researchers. Do not need it. Lars: actually, you do need to figure out, e.g., when to send feedback. Perhaps “Framing Considerations”? Jana: Need to have some mechanism for spreading out (not bursting) data. Slav: Thought about it a lot. There may be something in their implementation, but not there today. Discuss off-line. Bob: Note: we do not know that BitTorrent implementation of ledbat is ledbat. Slav: Feel free to write an implementation. LEDBAT Applications, Practices, and Recommendations. (Reinaldo Penno) ===== Lars: What do you mean by a control connection? Is it a limited application connection? Reinaldo: It is not a bulk data connection. Vijay: Control connection could be, e.g., SIP; data could be RTP. HTTP does not have that distinction. FTP does. Control connection exchanges parameters to setup bulk data transfer connection. Lars: this is not the obvious place to discuss DIFFSERV. Might want to split out to another document. (ACTION) Recommendations ----- Bob: Scope of draft is about application design. One aspect is whether you give user control. Not in document. Could be important to document. Six connections: “A number between 4 and 8.” Jana: How does application know first connections are not saturated. Reinaldo: Because the application does not see packet loss. Stuart: Need to look at studies, but on his MacBook he can saturate his GE interface with a single TCP connection. TCP fast retransmit works better with one connection than with multiple connections. Reinaldo: multiple connections good for head-of-line blocking. Described in the draft. Jana: purpose of multiple connections was for head-of-line blocking, not for getting more-than-fair network bandwidth. Might get more bandwidth, but not guaranteed and not the purpose. Should not be a consideration for this work group. Slav: Set your Firefox browser to 32 connections per server. Looks like win is from parallizing latency from server response. Bob: Would it be better to say things like “It would be in your interest NOT to xxx” instead of saying “You SHOULD do yyy.” (ACTION) Jana: +1. Need to wordsmith carefully, as implementors can misconstrue document to say, “I can always open 6 connections.” ACTION: Reinaldo: Please send text (ACTION) Concurrent Connections ---- Stuart: Is question how to generically deal with performance, or how to deal with HTTP servers that cannot do pipelined requests? Good HTTP guidance, but not general guidance. You get better performance with a server that does pipelining with a single connection than multiple connections. Slav: Does no harm Murari: about documenting current practices. Stuart: Important to highlight HTTP issue, not an IP issue. Reinaldo: Important to document why using multiple connections for HTTP. Stuart: Useful to describe what is being done to work around bugs versus truly best practices. **** Bob: Often confusion about problems with half-open connections on operation systems. Murari: Anyone do not want to adopt this document as a WG document? RESULT: silence Murari: Anyone want to adopt this? RESULT: consensus Jana: +1 to Stuart about being careful distinguishing current practice versus recommendations **** Goal: have parameter values by Anaheim (IETF 77). Please review pseudo code. Discuss on list. At least one implementation report by Anaheim. Also: do not have implications of opening multiple connections on devices with low power constraints.