idnits 2.17.1 draft-ietf-ledbat-congestion-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (September 25, 2012) is 4203 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Experimental ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 LEDBAT WG S. Shalunov 3 Internet-Draft G. Hazel 4 Intended status: Experimental BitTorrent Inc 5 Expires: March 29, 2013 J. Iyengar 6 Franklin and Marshall College 7 M. Kuehlewind 8 University of Stuttgart 9 September 25, 2012 11 Low Extra Delay Background Transport (LEDBAT) 12 draft-ietf-ledbat-congestion-10.txt 14 Abstract 16 LEDBAT is an experimental delay-based congestion control algorithm 17 that seeks to utilize the available bandwidth on an end-to-end path 18 while limiting the consequent increase in queueing delay on that 19 path. LEDBAT uses changes in one-way delay measurements to limit 20 congestion that the flow itself induces in the network. LEDBAT is 21 designed for use by background bulk-transfer applications to be no 22 more aggressive than standard TCP congestion control (as specified in 23 RFC5681) and to yield in the presence of competing flows, thus 24 limiting interference with the network performance of competing 25 flows. 27 Status of this Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at http://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on March 29, 2013. 44 Copyright Notice 46 Copyright (c) 2012 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (http://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Requirements notation . . . . . . . . . . . . . . . . . . . . 4 62 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 63 2.1. Design Goals . . . . . . . . . . . . . . . . . . . . . . . 4 64 2.2. Applicability . . . . . . . . . . . . . . . . . . . . . . 5 65 3. LEDBAT Congestion Control . . . . . . . . . . . . . . . . . . 6 66 3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . 6 67 3.2. Preliminaries . . . . . . . . . . . . . . . . . . . . . . 6 68 3.3. Receiver-Side Operation . . . . . . . . . . . . . . . . . 7 69 3.4. Sender-Side Operation . . . . . . . . . . . . . . . . . . 7 70 3.4.1. An Overview . . . . . . . . . . . . . . . . . . . . . 7 71 3.4.2. The Complete Sender Algorithm . . . . . . . . . . . . 8 72 3.5. Parameter Values . . . . . . . . . . . . . . . . . . . . . 11 73 4. Understanding LEDBAT Mechanisms . . . . . . . . . . . . . . . 13 74 4.1. Delay Estimation . . . . . . . . . . . . . . . . . . . . . 13 75 4.1.1. Estimating Base Delay . . . . . . . . . . . . . . . . 13 76 4.1.2. Estimating Queueing Delay . . . . . . . . . . . . . . 13 77 4.2. Managing the Congestion Window . . . . . . . . . . . . . . 14 78 4.2.1. Window Increase: Probing For More Bandwidth . . . . . 14 79 4.2.2. Window Decrease: Responding To Congestion . . . . . . 14 80 4.3. Choosing The Queuing Delay Target . . . . . . . . . . . . 14 81 5. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 15 82 5.1. Framing and ACK Frequency Considerations . . . . . . . . . 15 83 5.2. Competing With TCP Flows . . . . . . . . . . . . . . . . . 15 84 5.3. Competing With Non-TCP Flows . . . . . . . . . . . . . . . 16 85 5.4. Fairness Among LEDBAT Flows . . . . . . . . . . . . . . . 16 86 6. Open Areas for Experimentation . . . . . . . . . . . . . . . . 17 87 6.1. Network Effects and Monitoring . . . . . . . . . . . . . . 17 88 6.2. Parameter Values . . . . . . . . . . . . . . . . . . . . . 18 89 6.3. Filters . . . . . . . . . . . . . . . . . . . . . . . . . 18 90 6.4. Framing . . . . . . . . . . . . . . . . . . . . . . . . . 19 91 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 92 8. Security Considerations . . . . . . . . . . . . . . . . . . . 19 93 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 20 94 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20 95 10.1. Normative References . . . . . . . . . . . . . . . . . . . 20 96 10.2. Informative References . . . . . . . . . . . . . . . . . . 20 97 Appendix A. Measurement Errors . . . . . . . . . . . . . . . . . 21 98 A.1. Clock Offset . . . . . . . . . . . . . . . . . . . . . . . 21 99 A.2. Clock Skew . . . . . . . . . . . . . . . . . . . . . . . . 22 100 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23 102 1. Requirements notation 104 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 105 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 106 document are to be interpreted as described in [RFC2119]. 108 2. Introduction 110 TCP congestion control [RFC5681] seeks to share bandwidth at a 111 bottleneck link equitably among flows competing at the bottleneck, 112 and it is the predominant congestion control mechanism used on the 113 Internet. However, not all applications seek an equitable share of 114 network throughput. "Background" applications, such as software 115 updates or file-sharing applications, seek to operate without 116 interfering with the performance of more interactive and delay- 117 and/or bandwidth-sensitive "foreground" applications. Standard TCP 118 congestion control, as specified in [RFC5681], may be too aggressive 119 for use with such background applications. 121 LEDBAT is an experimental delay-based congestion control mechanism 122 that reacts early to congestion in the network, thus enabling 123 "background" applications to use the network while avoiding 124 interference with the network performance of competing flows. A 125 LEDBAT sender uses one-way delay measurements to estimate the amount 126 of queueing on the data path, controls the LEDBAT flow's congestion 127 window based on this estimate, and minimizes interference with 128 competing flows by adding low extra queueing delay on the end-to-end 129 path. 131 Delay-based congestion control protocols, such as TCP-Vegas 132 [Bra94][Low02], are generally designed to achieve more, not less 133 throughput than standard TCP, and often outperform TCP under 134 particular network settings. For further discussion on Lower-than- 135 Best-Effort (BE) transport protocols see [RFC6297]. In contrast, 136 LEDBAT is designed to be no more aggressive than TCP [RFC5681]; 137 LEDBAT is a "scavenger" congestion control mechanism that seeks to 138 utilize all available bandwidth and yields quickly when competing 139 with standard TCP at a bottleneck link. 141 In the rest of this document, we refer to [RFC5681]-specified 142 congestion control as "standard TCP". 144 2.1. Design Goals 146 LEDBAT congestion control seeks to achieve the following goals: 148 1. to utilize end-to-end available bandwidth, and to maintain low 149 queueing delay when no other traffic is present, 150 2. to add limited queuing delay to that induced by concurrent flows, 151 and 152 3. to yield quickly to standard TCP flows that share the same 153 bottleneck link. 155 2.2. Applicability 157 LEDBAT is a "scavenger" congestion control mechanism that is 158 motivated primarily by for background bulk-transfer applications, 159 such as large file transfers (as with file-sharing applications) and 160 software updates. It can be used with any application that seeks to 161 minimize its impact on the network and on other interactive delay- 162 and/or bandwidth-sensitive network applications. LEDBAT is expected 163 to work well when the sender and/or receiver is connected via a 164 residential access network. 166 LEDBAT can be used as part of a transport protocol or as part of an 167 application, as long as the data transmission mechanisms are capable 168 of carrying timestamps and acknowledging data frequently. LEDBAT can 169 be used with TCP, SCTP, and DCCP, with appropriate extensions where 170 necessary, and with proprietary application protocols, such as those 171 built on top of UDP for P2P applications. 173 When used with an ECN-capable framing protocol, LEDBAT should react 174 to an ECN mark as it would to a loss, as specified in [RFC3168]. 176 LEDBAT is designed to reduce build-up of a standing queue by long- 177 lived LEDBAT flows at a link with a tail-drop FIFO queue, so as to 178 avoid persistently delaying other flows sharing the queue. If Active 179 Queue Management (AQM) is configured to drop or ECN-mark packets 180 before the LEDBAT flow starts reacting to persistent queue build-up, 181 LEDBAT reverts to standard TCP behavior rather than yielding to other 182 TCP flows. However, such an AQM is still desirable since it keeps 183 queuing delay low, achieving an outcome that is in line with LEDBAT's 184 goals. Additionally, a LEDBAT transport that supports ECN enjoys the 185 advantages that an ECN-capable TCP enjoys over an ECN-agnostic TCP; 186 avoiding losses and possible retransmissions. Weighted Fair Queuing 187 (WFQ), as employed by some home gateways, seeks to isolate and 188 protect delay-sensitive flows from delays due to standing queues 189 built up by concurrent long-lived flows. Consequently, while it 190 prevents LEDBAT from yielding to other TCP flows, it again achieves 191 an outcome that is in line with LEDBAT's goals [Sch10]. 193 3. LEDBAT Congestion Control 195 3.1. Overview 197 A standard TCP sender increases its congestion window until a loss 198 occurs [RFC5681] or an ECN mark is received [RFC3168], which, in the 199 absence of any AQM and link errors in the network, occurs only when 200 the queue at the bottleneck link on the end-to-end path overflows. 201 Since packet loss or marking at the bottleneck link is expected to be 202 preceded by an increase in the queueing delay at the bottleneck link, 203 LEDBAT congestion control uses this increase in queueing delay as an 204 early signal of congestion, enabling it to respond to congestion 205 earlier than standard TCP, and enabling it to yield bandwidth to a 206 competing TCP flow. 208 LEDBAT employs one-way delay measurements to estimate queueing delay. 209 When the estimated queueing delay is less than a pre-determined 210 target, LEDBAT infers that the network is not yet congested, and 211 increases its sending rate to utilize any spare capacity in the 212 network. When the estimated queueing delay becomes greater than the 213 pre-determined target, LEDBAT decreases its sending rate as a 214 response to potential congestion in the network. 216 3.2. Preliminaries 218 A LEDBAT sender uses a congestion window (cwnd) to gate the amount of 219 data that the sender can send into the network in one roundtrip time 220 (RTT). A sender MAY maintain its cwnd in bytes or in packets; this 221 document uses cwnd in bytes. LEDBAT requires that each data segment 222 carries a "timestamp" from the sender, based on which the receiver 223 computes the one-way delay from the sender and sends this computed 224 value back to the sender. 226 In addition to the LEDBAT mechanism described below, we note that a 227 slow start mechanism can be used as specified in [RFC5681]. Since 228 slow start leads to faster increase in the window than that specified 229 in LEDBAT, conservative congestion control implementations employing 230 LEDBAT may skip slow start altogether and start with an initial 231 window of INIT_CWND * MSS. (INIT_CWND is described later in 232 Section 3.5.) 234 The term "MSS", or the sender's Maximum Segment Size, used in this 235 document refers to the size of the largest segment that the sender 236 can transmit. The value of MSS can be based on the path MTU 237 discovery [RFC4821] algorithm and/or on other factors. 239 3.3. Receiver-Side Operation 241 A LEDBAT receiver calculates the one-way delay from the sender to the 242 receiver based on its own system time and timestamps in the received 243 data packets. The receiver then feeds the computed one-way delay 244 back to the sender in the next acknowledgement. A LEDBAT receiver 245 operates as follows: 247 on data_packet: 248 remote_timestamp = data_packet.timestamp 249 acknowledgement.delay = local_timestamp() - remote_timestamp 250 # fill in other fields of acknowledgement 251 acknowlegement.send() 253 A receiver may choose to delay sending an ACK and may combine 254 acknowledgements for more than one data packet into a single ACK 255 packet, as is with delayed ACKs in standard TCP, for example. In 256 such cases, the receiver MAY bundle all the delay samples into one 257 ACK packet and MUST transmit the samples in the order generated. 258 When multiple delay samples are bundled within a single ACK, the 259 sender applies these bundled delay samples at once during its cwnd 260 adjustment (discussed in the next section). Since the sender's 261 adjustment may be sensitive to the order in which the delay samples 262 are applied, the computed delay samples should be available to the 263 sender in the order they were generated at the receiver. 265 3.4. Sender-Side Operation 267 3.4.1. An Overview 269 As a first approximation, a LEDBAT sender operates as shown below; 270 the complete algorithm is specified later in Section 3.4.2. TARGET 271 is the maximum queueing delay that LEDBAT itself may introduce in the 272 network, and GAIN determines the rate at which the cwnd responds to 273 changes in queueing delay; both constants are specified later. 274 off_target is a normalized value representing the difference between 275 the measured current queueing delay and the pre-determined TARGET 276 delay. off_target can be positive or negative, consequently, cwnd 277 increases or decreases in proportion to off_target. 279 on initialization: 280 base_delay = +INFINITY 282 on acknowledgement: 283 current_delay = acknowledgement.delay 284 base_delay = min(base_delay, current_delay) 285 queuing_delay = current_delay - base_delay 286 off_target = (TARGET - queuing_delay) / TARGET 287 cwnd += GAIN * off_target * bytes_newly_acked * MSS / cwnd 289 The simplified mechanism above ignores multiple delay samples in an 290 acknowledgment, noise filtering, base delay expiration, and sender 291 idle times, which we now take into account in our complete sender 292 algorithm below. 294 3.4.2. The Complete Sender Algorithm 296 update_current_delay() maintains a list of one-way delay 297 measurements, of which a filtered value is used as an estimate of the 298 current end-to-end delay. update_base_delay() maintains a list of 299 one-way delay minima over a number of one-minute intervals, to 300 measure and to track changes in the base delay of the end-to-end 301 path. Both of these lists are maintained per LEDBAT flow. 303 We note this algorithm assumes that slight random fluctuations exist 304 in inter-packet arrival times at the bottleneck queue, to allow a 305 LEDBAT sender to correctly measure the base delay. See section 306 Section 5.4 for a more complete discussion. 308 The full sender-side algorithm is given below: 310 on initialization: 311 # cwnd is the amount of data that is allowed to be 312 # outstanding in an RTT and is defined in bytes. 313 # CTO is the Congestion Timeout value. 315 create current_delays list with CURRENT_FILTER elements 316 create base_delays list with BASE_HISTORY number of elements 317 initialize elements in base_delays to +INFINITY 318 initialize elements in current_delays according to FILTER() 319 last_rollover = -INFINITY # More than a minute in the past 320 flightsize = 0 321 cwnd = INIT_CWND * MSS 322 CTO = 1 second 324 on acknowledgment: 325 # flightsize is the amount of data outstanding before this ACK 326 # was received and is updated later; 327 # bytes_newly_acked is the number of bytes that this ACK 328 # newly acknowledges, and it MAY be set to MSS. 330 for each delay sample in the acknowledgment: 331 delay = acknowledgement.delay 332 update_base_delay(delay) 333 update_current_delay(delay) 335 queuing_delay = FILTER(current_delays) - MIN(base_delays) 336 off_target = (TARGET - queuing_delay) / TARGET 337 cwnd += GAIN * off_target * bytes_newly_acked * MSS / cwnd 338 max_allowed_cwnd = flightsize + ALLOWED_INCREASE * MSS 339 cwnd = min(cwnd, max_allowed_cwnd) 340 cwnd = max(cwnd, MIN_CWND * MSS) 341 flightsize = flightsize - bytes_newly_acked 342 update_CTO() 344 on data loss: 345 # at most once per RTT 346 cwnd = min (cwnd, max (cwnd/2, MIN_CWND * MSS)) 347 if data lost is not to be retransmitted: 348 flightsize = flightsize - bytes_not_to_be_retransmitted 350 if no ACKs are received within a CTO: 351 # extreme congestion, or significant RTT change. 352 # set cwnd to 1MSS and backoff the congestion timer. 353 cwnd = 1 * MSS 354 CTO = 2 * CTO 356 update_CTO() 357 # implements an RTT estimation mechanism using data 358 # transmission times and ACK reception times, 359 # which is used to implement a congestion timeout (CTO). 360 # If implementing LEDBAT in TCP, sender SHOULD use 361 # mechanisms described in RFC 6298 [RFC6298], 362 # and the CTO would be the same as the RTO. 364 update_current_delay(delay) 365 # Maintain a list of CURRENT_FILTER last delays observed. 366 delete first item in current_delays list 367 append delay to current_delays list 369 update_base_delay(delay) 370 # Maintain BASE_HISTORY delay-minima. 371 # Each minimum is measured over a period of a minute. 372 # 'now' is the current system time 373 if round_to_minute(now) != round_to_minute(last_rollover) 374 last_rollover = now 375 delete first item in base_delays list 376 append delay to base_delays list 377 else 378 base_delays.tail = MIN(base_delays.tail, delay) 380 The LEDBAT sender seeks to extract the actual delay estimate from the 381 current_delay samples by implementing FILTER() to eliminate any 382 outliers. Different types of filters MAY be used for FILTER() --- a 383 NULL filter, that does not filter at all, is a reasonable candidate 384 as well, since LEDBAT's use of a linear controller for cwnd increase 385 and decrease may allow it to recover quickly from errors induced by 386 bad samples. Another example of a filter is the Exponentially- 387 Weighted Moving Average (EWMA) function, with weights that enable 388 agile tracking of changing network delay. A simple MIN filter 389 applied over a small window (much smaller than BASE_HISTORY) may also 390 provide robustness to large delay peaks, as may occur with delayed 391 ACKs in TCP. Care should be taken that the filter used, while 392 providing robustness to noise, remains sensitive to persistent 393 congestion signals. 395 We note that when multiple delay samples are bundled within a single 396 ACK, the sender's resulting cwnd may be slightly different than when 397 the samples are sent individually in separate ACKs. The cwnd is 398 adjusted based on the total number of bytes ACKed and the final 399 filtered value of queueing_delay, irrespective of the number of delay 400 samples in an ACK. 402 To implement an approximate minimum over the past few minutes, a 403 LEDBAT sender stores BASE_HISTORY separate minima---one each for the 404 last BASE_HISTORY-1 minutes, and one for the running current minute. 405 At the end of the current minute, the window moves---the earliest 406 minimum is dropped and the latest minimum is added. If the 407 connection is idle for a given minute, no data is available for the 408 one-way delay and, therefore, a value of +INFINITY has to be stored 409 in the list. If the connection has been idle for BASE_HISTORY 410 minutes, all minima in the list are thus set to +INFINITY and 411 measurement begins anew. LEDBAT thus requires that during idle 412 periods, an implementation must maintain the base delay list. 414 LEDBAT restricts cwnd growth after a period of inactivity. When the 415 sender is application-limited, the sender's cwnd is clamped down 416 using max_allowed_cwnd to a little more than flightsize. To be TCP- 417 friendly, LEDBAT halves its cwnd on data loss. 419 LEDBAT uses a congestion timeout (CTO) to avoid transmitting data 420 during periods of heavy congestion, and to avoid congestion collapse. 421 A CTO is used to detect heavy congestion indicated by loss of all 422 outstanding data or acknowledgments, resulting in reduction of the 423 cwnd to 1 MSS and an exponential backoff of the CTO interval. This 424 backoff of the CTO value avoids sending more data into an overloaded 425 queue, and also allows the sender to cope with sudden changes in the 426 RTT of the path. The function of a CTO is similar to that of an 427 retransmission timeout (RTO) in TCP [RFC6298], but since LEDBAT 428 separates reliability from congestion control, a retransmission need 429 not be triggered by a CTO. LEDBAT, however does not preclude a CTO 430 from triggering retransmissions, as could be the case if LEDBAT 431 congestion control were to be used with TCP framing and reliability. 433 The CTO is a gating mechanism that ensures exponential backoff of 434 sending rate under heavy congestion, and it may be implemented with 435 or without a timer. An implementation choosing to avoid timers may 436 consider using a "next-time-to-send" variable, set based on the CTO, 437 to control the earliest time a sender may transmit without receiving 438 any ACKs. A maximum value MAY be placed on the CTO, and if placed, 439 it MUST be at least 60 seconds. 441 3.5. Parameter Values 443 TARGET MUST be 100 milliseconds or less, and this choice of value is 444 explained further in Section 4.3. Note that using the same TARGET 445 value across LEDBAT flows enables equitable sharing of the bottleneck 446 bandwidth. A flow with a higher TARGET value than other competing 447 LEDBAT flows may get a larger share of the bottleneck bandwidth. It 448 is possible to consider the use of different TARGET values for 449 implementing a relative priority between two competing LEDBAT flows 450 by setting a higher TARGET value for the higher-priority flow. 452 ALLOWED_INCREASE SHOULD be 1, and it MUST be greater than 0. An 453 ALLOWED_INCREASE of 0 results in no cwnd growth at all, and an 454 ALLOWED_INCREASE of 1 allows and limits the cwnd increase based on 455 flightsize in the previous RTT. An ALLOWED_INCREASE greater than 1 456 MAY be used when interactions between LEDBAT and the framing protocol 457 provide a clear reason for doing so. 459 GAIN MUST be set to 1 or less. A GAIN of 1 limits the maximum cwnd 460 ramp-up to the same rate as TCP Reno in Congestion Avoidance. While 461 this document specifies the use of the same GAIN for both cwnd 462 increase (when off_target is greater than zero) and decrease (when 463 off_target is less than zero), implementations MAY use a higher GAIN 464 for cwnd decrease than for the increase; our justification follows. 465 When a competing non-LEDBAT flow increases its sending rate, the 466 LEDBAT sender may only measure a small amount of additional delay and 467 decrease the sending rate slowly. To ensure no impact on a competing 468 non-LEDBAT flow, the LEDBAT flow should decrease its sending rate at 469 least as quickly as the competing flow increases its sending rate. A 470 higher decrease-GAIN MAY be used to allow the LEDBAT flow to decrease 471 its sending rate faster than the competing flow's increase rate. 473 The size of the base_delays list, BASE_HISTORY, SHOULD be 10. If the 474 actual base delay decreases, due to a route change for instance, a 475 LEDBAT sender adapts immediately, irrespective of the value of 476 BASE_HISTORY. If the actual base delay increases however, a LEDBAT 477 sender will take BASE_HISTORY minutes to adapt and may wrongly infer 478 a little more extra delay than intended (TARGET) in the meanwhile. A 479 value for BASE_HISTORY is thus a tradeoff: a higher value may yield a 480 more accurate measurement when the base delay is unchanging, and a 481 lower value results in a quicker response to actual increase in base 482 delay. 484 A LEDBAT sender uses the current_delays list to maintain only delay 485 measurements made within a RTT amount of time in the past, seeking to 486 eliminate noise spikes in its measurement of the current one-way 487 delay through the network. The size of this list, CURRENT_FILTER, 488 may be variable, and depends on the FILTER() function as well as the 489 number of successful measurements made within a RTT amount of time in 490 the past. The sender should seek to gather enough delay samples in 491 each RTT so as to have statistical confidence in the measurements. 492 While the number of delay samples required for such confidence will 493 vary depending on network conditions, we recommend that the sender 494 SHOULD use at least 4 samples in each RTT, unless the number of 495 samples is lower due to a small congestion window. Thus, subject to 496 congestion window constraints, the number of delay samples in each 497 RTT SHOULD be at least 4. The value of CURRENT_FILTER will depend on 498 the filter being employed, but CURRENT_FILTER MUST be limited such 499 that samples in the list are not older than an RTT in the past. 501 INIT_CWND and MIN_CWND SHOULD both be 2. An INIT_CWND of 2 should 502 help seed FILTER() at the sender when there are no samples at the 503 beginning of a flow, and a MIN_CWND of 2 allows FILTER() to use more 504 than a single instantaneous delay estimate while not being too 505 aggressive. Slight deviations may be warranted, for example, when 506 these values of INIT_CWND and MIN_CWND interact poorly with the 507 framing protocol. However, INIT_CWND and MIN_CWND MUST be no larger 508 than the corresponding values specified for TCP [RFC5681]. 510 4. Understanding LEDBAT Mechanisms 512 This section describes the delay estimation and window management 513 mechanisms used in LEDBAT. 515 4.1. Delay Estimation 517 LEDBAT estimates congestion in the direction of the data flow, and to 518 avoid measuring additional delay from e.g. queue build-up on the 519 reverse path (or ACK path) or reordering, LEDBAT uses one-way delay 520 estimates. LEDBAT assumes measurements are done with data packets, 521 thus avoiding the need for separate measurement packets and avoiding 522 the pitfall of measurement packets being treated differently from the 523 data packets in the network. 525 End-to-end delay can be decomposed into transmission (or 526 serialization) delay, propagation (or speed-of-light) delay, queueing 527 delay, and processing delay. On any given path, barring some noise, 528 all delay components except for queueing delay are constant. To 529 observe an increase in the queueing delay in the network, a LEDBAT 530 sender separates the queueing delay component from the rest of the 531 end-to-end delay, as described below. 533 4.1.1. Estimating Base Delay 535 Since queuing delay is always additive to the end-to-end delay, 536 LEDBAT estimates the sum of the constant delay components, which we 537 call "base delay", to be the minimum delay observed on the end-to-end 538 path. 540 To respond to true changes in the base delay, as can be caused by a 541 route change, LEDBAT uses only recent measurements in estimating the 542 base delay. The duration of the observation window itself is a 543 tradeoff between robustness of measurement and responsiveness to 544 change---a larger observation window increases the chances that the 545 true base delay will be detected (as long as the true base delay is 546 unchanged), whereas a smaller observation window results in faster 547 response to true changes in the base delay. 549 4.1.2. Estimating Queueing Delay 551 Assuming that the base delay is constant (in the absence of any route 552 changes), the queueing delay is represented by the variable component 553 of the measured end-to-end delay. LEDBAT measures queueing delay as 554 simply the difference between an end-to-end delay measurement and the 555 current estimate of base delay. The queueing delay should be 556 filtered (depending on the usage scenario) to eliminate noise in the 557 delay estimation, such as due to spikes in processing delay at a node 558 on the path. 560 4.2. Managing the Congestion Window 562 LEDBAT uses a simple linear controller to determine the sending rate 563 as a function of the delay estimate, where the response of the sender 564 is proportional to the difference between the current queueing delay 565 estimate and the target. 567 4.2.1. Window Increase: Probing For More Bandwidth 569 When the queuing delay is smaller than a delay target value, as 570 specified by the TARGET parameter in this document, a LEDBAT sender 571 will increase its congestion window proportionally to the relative 572 difference between the current queueing delay and the delay target. 573 As the current queuing delay gets closer to TARGET, LEDBAT's window 574 growth gets slower. To compete fairly with concurrent TCP flows, we 575 set the highest rate of LEDBAT's window growth (when the current 576 queueing delay estimate is zero) to be the same as TCP's (increase of 577 one packet per RTT). In other words, a LEDBAT flow thus never ramps 578 up faster than a competing TCP flow over the same path. The TARGET 579 value specifies the maximum extra queuing delay that LEDBAT will 580 induce. If the current queuing delay equals the TARGET value, LEDBAT 581 tries to maintain this extra delay. This is done by a very slow 582 increase rate (1 packet all couple of RTTs) in this state. 584 4.2.2. Window Decrease: Responding To Congestion 586 When a sender's queueing delay estimate is higher than the target, 587 the LEDBAT flow's rate should be reduced. LEDBAT's linear controller 588 allows the sender to decrease the window proportional to the 589 difference between the target and the current queueing delay. 591 Unlike TCP-like loss-based congestion control, LEDBAT seeks to avoid 592 losses and so a LEDBAT sender is not expected to normally rely on 593 losses to determine the sending rate. However, when data loss does 594 occur, LEDBAT must respond as standard TCP does; even if the queueing 595 delay estimates indicate otherwise, a loss is assumed to be a strong 596 indication of congestion. Thus, to deal with severe congestion when 597 packets are dropped in the network, and to provide a fallback against 598 incorrect queuing delay estimates, a LEDBAT sender halves its 599 congestion window when a loss event is detected. As with TCP New- 600 Reno, LEDBAT reduces its cwnd by half at most once per RTT. 602 4.3. Choosing The Queuing Delay Target 604 The International Telecommunication Union's (ITU's) Recommendation 605 G.114 defines a one-way delay of 150 ms to be acceptable for most 606 user voice applications [g114]. Thus the delay induced by LEDBAT 607 must be well below 150 ms to limit its impact on concurrent delay- 608 sensitive traffic sharing the same bottleneck queue. A target that 609 is too low, on the other hand, increases the sensitivity of the 610 sender's algorithm to noise in the one-way delays and in the delay 611 measurement process, and may lead to reduced throughput for the 612 LEDBAT flow and to under-utilization of the bottleneck link. 614 Our recommendation of 100 ms or less as the target is a tradeoff 615 between these considerations. Anecdotal evidence indicates that this 616 value works well---LEDBAT has been implemented and successfully 617 deployed with a target value of 100 ms in two Bittorrent 618 implementations: as the exclusive congestion control mechanism in 619 BitTorrent Delivery Network Accelerator (DNA), and as an experimental 620 mechanism in uTorrent [uTorrent]. 622 5. Discussion 624 5.1. Framing and ACK Frequency Considerations 626 While the actual framing and wire format of the protocols using 627 LEDBAT are outside the scope of this document, we briefly consider 628 the data framing and ACK frequency needs of LEDBAT mechanisms. 630 To compute the data path's one-way delay, our discussion of LEDBAT 631 assumes a framing that allows the sender to timestamp packets and for 632 the receiver to convey the measured one-way delay back to the sender 633 in ACK packets. LEDBAT does not require this particular method, but 634 it does require unambiguous delay estimates using data and ACK 635 packets. 637 A LEDBAT receiver may send an ACK as frequently as one for every data 638 packet received or less frequently; LEDBAT does require that the 639 receiver MUST transmit at least one ACK in every RTT. 641 5.2. Competing With TCP Flows 643 LEDBAT is designed to respond to congestion indications earlier than 644 loss-based standard TCP [RFC5681]. A LEDBAT flow gets more 645 aggressive as the queueing delay estimate gets lower; since the 646 queueing delay estimate is non-negative, LEDBAT is most aggressive 647 when the queueing delay estimate is zero. In this case, LEDBAT ramps 648 up its congestion window at the same rate as standard TCP [RFC5681]. 649 LEDBAT may reduce its rate earlier than standard TCP and always 650 halves its congestion window on loss. Thus, in the worst case, where 651 the delay estimates are completely and consistently off, a LEDBAT 652 flow falls back to standard TCP behavior, and is no more aggressive 653 than standard TCP [RFC5681]. 655 5.3. Competing With Non-TCP Flows 657 While LEDBAT yields to all high-load flows, both TCP and non-TCP, 658 LEDBAT may not yield to low-load and latency-sensitive traffic that 659 do not induce a measurable delay at the bottleneck queue, such as 660 VoIP traffic. While such flows will experience additional delay due 661 to any concurrent LEDBAT flows, the TARGET delay sets a limit to the 662 total amount of additional delay that all the concurrent LEDBAT flows 663 will jointly induce. If the TARGET delay is higher than what the 664 bottleneck queue can sustain, the LEDBAT flows should experience loss 665 and will fall back to standard loss-based TCP behavior. Thus, in the 666 worst case, LEDBAT will add no more latency than standard TCP when 667 competing with non-TCP flows. In the common case however, we expect 668 LEDBAT flows to add TARGET amount of delay, which ought to be within 669 the delay tolerance for most latency-sensitive applications, 670 including VoIP applications. 672 5.4. Fairness Among LEDBAT Flows 674 The primary design goals of LEDBAT are focussed on the aggregate 675 behavior of LEDBAT flows when they compete with standard TCP. Since 676 LEDBAT is designed for background traffic, we consider link 677 utilization to be more important than fairness amongst LEDBAT flows. 678 Nevertheless, we now consider fairness issues that might arise 679 amongst competing LEDBAT flows. 681 LEDBAT as described so far lacks a mechanism specifically designed to 682 equalize utilization amongst LEDBAT flows. Anecdotally observed 683 behavior of existing implementations indicates that a rough 684 equalization does occur since in most enviroments some amount of 685 randomness in the inter-packet transmission times exist, as explained 686 further below. 688 Delay-based congestion control systems suffer from the possibility of 689 late-comers incorrectly measuring and using a higher base-delay than 690 an active flow that started earlier. Consider that a bottleneck is 691 saturated by a single LEDBAT flow, and the flow therefore maintains 692 the bottleneck queue at TARGET delay. When a new LEDBAT flow arrives 693 at the bottleneck, it might incorrectly include the steady queueing 694 delay in its measurement of the base delay on the path. The new flow 695 has an inflated estimate of the base delay, and may now effectively 696 build on top of the existing, already maximal, queueing delay. As 697 the late-comer flow builds up, the old flow sees the true queueing 698 delay and backs off, while the late-comer keeps building up, using up 699 the entire link's capacity, and effectively shutting the old flow 700 out. This advantage is called the "late-comer's advantage". 702 In the worst case, if the first flow yields at the same rate as the 703 new flow increases its sending rate, the new flow will see constant 704 end-to-end delay, which it assumes is the base delay, until the first 705 flow backs off completely. As a result, by the time the second flow 706 stops increasing its cwnd, it would have added twice the target 707 queueing delay to the network. 709 This advantage can be reduced if the first flow yields and empties 710 the bottleneck queue faster than the incoming flow increases its 711 occupancy in the queue. In such a case, the late-comer might measure 712 correctly a delay that is closer to the base delay. While such a 713 reduction might be achieved through a multiplicative decrease of the 714 congestion window, this may cause strong fluctuations in flow 715 throughput during the flow's steady state. Thus we do not recommend 716 a multiplicative decrease scheme. 718 We note that in certain use-case scenarios, it is possible for a 719 later LEDBAT flow to gain an unfair advantage over an existing one 720 [Car10]. In practice, this concern ought to be alleviated by the 721 burstiness of network traffic: all that's needed to measure the base 722 delay is one small gap in transmission schedules between the LEDBAT 723 flows. These gaps can occur for a number of reasons such as latency 724 introduced due to application sending patterns, OS scheduling at the 725 sender, processing delay at the sender or any network node, and link 726 contention. When such a gap occurs in the first sender's 727 transmission while the late-comer is starting, base delay is 728 immediately correctly measured. With a small number of LEDBAT flows, 729 system noise may sufficiently regulate the late-comer's advantage. 731 6. Open Areas for Experimentation 733 We now outline some areas that need experimentation in the Internet 734 and under different network scenarios. These experiments should help 735 the community understand LEDBAT's dynamics and should help towards 736 further standardization of LEDBAT and LEDBAT-related documents. 738 6.1. Network Effects and Monitoring 740 Further study is required to fully understand the behavior and 741 convergence properties of LEDBAT in networks with non-tail-drop, non- 742 FIFO queues, in networks with frequent route changes, and in networks 743 with network-level load balancing. These studies should have two 744 broad goals: (i) to understand the effects of different network 745 mechanisms on LEDBAT, and (ii) to understand the impact of LEDBAT on 746 the network. 748 Network mechanisms and dynamics can influence LEDBAT flows in 749 unintended ways. For instance, frequent route changes that result in 750 increasing base delays may, in the worst case, throttle a LEDBAT 751 flow's throughput significantly. The influence of different network 752 traffic management mechanisms on LEDBAT throughput should be studied. 754 An increasing number of LEDBAT flows in the network will likely 755 result in operator-visible network effects as well and should thus be 756 studied. For instance, as long as the bottleneck queue in a network 757 is larger than TARGET (in terms of delay), we expect that both the 758 average queueing delay and loss rate in the network should reduce as 759 LEDBAT traffic increasingly dominates the traffic mix in the network. 760 Note that for bottleneck queues that are smaller than TARGET, LEDBAT 761 will appear to behave very similar to standard TCP and it's flow- 762 level behavior may not be distinguishable from that of standard TCP. 764 We note that a network operator may be able to verify the operation 765 of a LEDBAT flow by monitoring per-flow behavior and queues in the 766 network---when the queueing delay at a bottleneck queue is above 767 TARGET as specified in this document, LEDBAT flows should be expected 768 to back off and reduce their sending rate. 770 6.2. Parameter Values 772 The throughput and response of LEDBAT to the proposed parameter 773 values of TARGET, decrease-GAIN, BASE_HISTORY, INIT_CWND, and 774 MIN_CWND should be evaluated with different types of competing 775 traffic in different network settings, including with different AQM 776 schemes at the bottleneck queue. TARGET controls LEDBAT's added 777 latency, while decrease-GAIN controls LEDBAT's response to competing 778 traffic. Since LEDBAT is intended to be minimally intrusive to 779 competing traffic, the impact of TARGET and decrease-GAIN on delay- 780 sensitive traffic should be studied. TARGET also impacts the growth 781 rate of the congestion window when off_target is smaller than 1. 782 This impact of TARGET on the rate of cwnd growth should be studied. 783 The amount of history maintained by the base delay estimator, 784 BASE_HISTORY, influences the responsiveness of LEDBAT to changing 785 network conditions. LEDBAT's responsiveness and throughput should be 786 evaluated in the wide area and under conditions where abrupt changes 787 in base delay might occur, such as with route changes and with 788 cellular handovers. The impact and efficacy of these parameters 789 should be carefully studied with tests over the Internet. 791 6.3. Filters 793 LEDBAT's effectiveness depends on a sender's ability to accurately 794 estimate end-to-end queueing delay from delay samples. Consequently, 795 the filtering algorithm used for this estimation, FILTER(), is an 796 important candidate for experiments. This document suggests the use 797 of NULL, EWMA and MIN filters for estimating the current delay; the 798 efficacy of these and other possible filters for this estimate should 799 be investigated. FILTER() may also impact cwnd dynamics when delay 800 samples are bundled in ACKs, since cwnd adaption is done once per ACK 801 irrespective of the number of delay samples in the ACK. This impact 802 should be studied when the different filters are considered. 804 6.4. Framing 806 This document defines only a congestion control algorithm and assumes 807 that framing mechanisms for exchanging delay information exist within 808 the protocol in which LEDBAT is being implemented. If implemented in 809 a new protocol, both the sender and receiver may be LEDBAT-aware, but 810 if implemented in an existing protocol which is capable of providing 811 one-way delay information, LEDBAT may be implemented as a sender- 812 side-only modification. In either case, the parent protocol may 813 interact with LEDBAT's algorithms; for instance, the rate of ACK 814 feedback to the data sender may be dictated by other protocol 815 parameters, but will interact with the LEDBAT flow's dynamics. 816 Careful experimentation is necessary to understand and integrate 817 LEDBAT into both new and existing protocols. 819 7. IANA Considerations 821 There are no IANA considerations for this document. 823 8. Security Considerations 825 LEDBAT's aggressiveness is contingent on the delay estimates and on 826 the TARGET delay value. If these parameter values at the sender are 827 compromised such that delay estimates are artificially set to zero 828 and the TARGET delay value is set to +INFINITY, the LEDBAT algorithm 829 deteriorates to TCP-like behavior. Thus, while LEDBAT is sensitive 830 to these parameters, the algorithm is fundamentally limited in the 831 worst case to be as aggressive as standard TCP. 833 An man-in-the-middle may be able to change queueing delay on a 834 network path, and/or modify the timestamps transmitted by a LEDBAT 835 sender and/or modify the delays reported by a LEDBAT receiver, thus 836 causing a LEDBAT flow to back off even when there's no congestion. A 837 protocol using LEDBAT ought to minimize the risk of such man-in-the- 838 middle attacks by at least authenticating the timestamp field in the 839 data packets and the delay field in the ACK packets. 841 LEDBAT is not known to introduce any new concerns with privacy, 842 integrity, or other security issues for flows that use it. LEDBAT is 843 compatible with use of IPsec and TLS/DTLS. 845 9. Acknowledgements 847 We thank folks in the LEDBAT working group for their comments and 848 feedback. Special thanks to Murari Sridharan and Rolf Winter for 849 their patient and untiring shepherding. 851 10. References 853 10.1. Normative References 855 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 856 Requirement Levels", BCP 14, RFC 2119, March 1997. 858 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 859 of Explicit Congestion Notification (ECN) to IP", 860 RFC 3168, September 2001. 862 [RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU 863 Discovery", RFC 4821, March 2007. 865 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 866 Control", RFC 5681, September 2009. 868 [RFC6298] Paxson, V., Allman, M., Chu, J., and M. Sargent, 869 "Computing TCP's Retransmission Timer", RFC 6298, 870 June 2011. 872 10.2. Informative References 874 [Bra94] Brakmo, L., O'Malley, S., and L. Peterson, "TCP Vegas: New 875 techniques for congestion detection and avoidance", 876 Proceedings of SIGCOMM '94, pages 24-35, August 1994. 878 [Car10] Carofiglio, G., Muscariello, L., Rossi, D., Testa, C., and 879 S. Valenti, "Rethinking Low Extra Delay Background 880 Transport Protocols", arXiv:1010.5623v1, September 2010. 882 [Low02] Low, S., Peterson, L., and L. Wang, "Understanding TCP 883 Vegas: A Duality Model", JACM 49 (2), March 2002. 885 [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network 886 Time Protocol Version 4: Protocol and Algorithms 887 Specification", RFC 5905, June 2010. 889 [RFC6297] Welzl, M. and D. Ros, "A Survey of Lower-than-Best-Effort 890 Transport Protocols", RFC 6297, June 2011. 892 [Sch10] Schneider, J., Wagner, J., Winter, R., and H. Kolbe, "Out 893 of my Way -- Evaluating Low Extra Delay Background 894 Transport in an ADSL Access Network", Proceedings of 22nd 895 International Teletraffic Congress (ITC22), September 896 2010. 898 [g114] "SERIES G: TRANSMISSION SYSTEMS AND MEDIA, DIGITAL SYSTEMS 899 AND NETWORKS; International telephone connections and 900 circuits - General; Recommendations on the transmission 901 quality for an entire international telephone connection; 902 One-way transmission time", ITU-T Recommendation G.114, 903 05/2003. 905 [uTorrent] 906 Hazel, G., "uTorrent Transport Protocol library", 907 http://github.com/bittorrent/libutp, July 2012. 909 Appendix A. Measurement Errors 911 LEDBAT measures and uses one-way delays, and we now consider 912 measurement errors in timestamp generation and use. In this Section, 913 we use the same locally-linear clock model and the same terminology 914 as Network Time Protocol (NTP) [RFC5905]. In particular, NTP uses 915 the terms "offset" to refer to the difference between measured time 916 and true time, and "skew" to refer to difference of clock rate from 917 the true rate. A clock thus has two time measurement errors: a fixed 918 offset from the true time, and a skew. We now consider these errors 919 in the context of LEDBAT. 921 A.1. Clock Offset 923 The offset of the clocks, both the sender's and the receiver's, shows 924 up as a fixed error in LEDBAT's one-way delay measurement. The 925 offset in the measured one-way delay is simply the difference in 926 offsets between the receiver's and the sender's clocks. LEDBAT 927 however does not use this estimate directly, but uses the difference 928 between the measured one-way delay and a measured base delay. Since 929 the offset error (difference of clock offsets) is the same for the 930 measured one-way delay and the base delay, the offsets cancel each 931 other out in the queuing delay estimate, which LEDBAT uses for its 932 window computations. Clock offset error thus has no impact on 933 LEDBAT. 935 A.2. Clock Skew 937 Clock skew generally shows up as a linearly changing error in a time 938 estimate. Similar to the offset, the skew of LEDBAT's one-way delay 939 estimate is thus the difference between the two clocks' skews. 940 Unlike the offset however, skew does not cancel out when the queuing 941 delay estimate is computed, since it causes the two clocks' offsets 942 to change over time. 944 While the offset could be large, with some clocks off by minutes or 945 even hours or more, skew is typically small. Typical skews of 946 untrained clocks seem to be around 100-200 PPM [RFC5905], where a 947 skew of 100 PPM translates to an error accumulation of 6 milliseconds 948 per minute. This accumulation is limited in LEDBAT, since any error 949 accumulation is limited to the amount of history maintained by the 950 base delay estimator, as dictated by the BASE_HISTORY parameter. The 951 effects of clock skew error on LEDBAT should generally be 952 insignificant unless the skew is unusually high, or unless extreme 953 values have been chosen for TARGET (extremely low) and BASE_HISTORY 954 (extremely large). Nevertheless, we now consider the possible impact 955 of skew on LEDBAT behavior. 957 Clock skew can manifest in two ways: the sender's clock can be faster 958 than the receiver's clock, or the receiver's clock can be faster than 959 the sender's clock. In the first case, the measured one-way delay 960 will decrease as the sender's clock drifts forward. While this drift 961 can lead to an artificially low estimate of the queueing delay, the 962 drift should also lead to a lower base delay measurement, which 963 consequently absorbs the erroneous reduction in the one-way delay 964 estimates. 966 In the second case, the one-way delay estimate will artifically 967 increase with time. This increase can reduce a LEDBAT flow's 968 throughput unnecessarily. In this case, a skew correction mechanism 969 can be beneficial. 971 We now discuss an example clock skew correction mechanism. In this 972 example, the receiver sends back raw (sending and receiving) 973 timestamps. Using this information, the sender can estimate one-way 974 delays in both directions, and the sender can also compute and 975 maintain an estimate of the base delay as would be observed by the 976 receiver. If the sender detects the receiver reducing its estimate 977 of the base delay, it may infer that this reduction is due to clock 978 drift. The sender then compensates by increasing its base delay 979 estimate by the same amount. To apply this mechanism, timestamps 980 need to be transmitted in both directions. 982 We now outline a few other ideas that can be used for skew 983 correction. 984 o Skew correction with faster virtual clock: 985 Since having a faster clock on the sender will result in 986 continuous updates of the base delay, a faster virtual clock can 987 be used for sender timestamping. This virtual clock can be 988 computed from the default machine clock through a linear 989 transformation. For instance, with a 500 PPM speed-up the 990 sender's clock is very likely to be faster than a receiver's 991 clock. Consequently, LEDBAT will benefit from the implicit 992 correction when updating the base delay. 994 o Skew correction with estimating drift: 995 A LEDBAT sender maintains a history of base delay minima. This 996 history can provide a base to compute the clock skew difference 997 between the two hosts. The slope of a linear function fitted to 998 the set of minima base delays gives an estimate of the clock skew. 999 This estimation can be used to correct the clocks. If the other 1000 endpoint is doing the same, the clock should be corrected by half 1001 of the estimated skew amount. 1003 o Byzantine skew correction: 1004 When it is known that each host maintains long-lived connections 1005 to a number of different other hosts, a byzantine scheme can be 1006 used to estimate the skew with respect to the true time. Namely, 1007 a host calculates the skew difference for each of the peer hosts 1008 as described with the previous approach, then take the median of 1009 the skew differences. While this scheme is not universally 1010 applicable, it combines well with other schemes, since it is 1011 essentially a clock training mechanism. The scheme also corrects 1012 fast, since state is preserved between connections. 1014 Authors' Addresses 1016 Stanislav Shalunov 1017 BitTorrent Inc 1018 612 Howard St, Suite 400 1019 San Francisco, CA 94105 1020 USA 1022 Email: shalunov@shlang.com 1023 URI: http://shlang.com 1024 Greg Hazel 1025 BitTorrent Inc 1026 612 Howard St, Suite 400 1027 San Francisco, CA 94105 1028 USA 1030 Email: greg@bittorrent.com 1032 Janardhan Iyengar 1033 Franklin and Marshall College 1034 415 Harrisburg Ave. 1035 Lancaster, PA 17603 1036 USA 1038 Email: jiyengar@fandm.edu 1040 Mirja Kuehlewind 1041 University of Stuttgart 1042 Stuttgart 1043 DE 1045 Email: mirja.kuehlewind@ikr.uni-stuttgart.de