idnits 2.17.1 draft-irtf-tmrg-tools-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 13. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 1386. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1397. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1404. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1410. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (9 December 2006) is 6345 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Obsolete informational reference (is this intentional?): RFC 3484 (ref. 'IMLGK03') (Obsoleted by RFC 6724) Summary: 1 error (**), 0 flaws (~~), 3 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force S. Floyd 2 INTERNET-DRAFT E. Kohler 3 draft-irtf-tmrg-tools-03.txt Editors 4 Expires: June 2007 9 December 2006 6 Tools for the Evaluation of Simulation and Testbed Scenarios 8 Status of this Memo 10 By submitting this Internet-Draft, each author represents that any 11 applicable patent or other IPR claims of which he or she is aware 12 have been or will be disclosed, and any of which he or she becomes 13 aware will be disclosed, in accordance with Section 6 of BCP 79. 15 Internet-Drafts are working documents of the Internet Engineering 16 Task Force (IETF), its areas, and its working groups. Note that 17 other groups may also distribute working documents as Internet- 18 Drafts. 20 Internet-Drafts are draft documents valid for a maximum of six 21 months and may be updated, replaced, or obsoleted by other documents 22 at any time. It is inappropriate to use Internet-Drafts as 23 reference material or to cite them other than as "work in progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt. 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 This Internet-Draft will expire on June 2007. 33 Abstract 35 This document describes tools for the evaluation of simulation and 36 testbed scenarios used in research on Internet congestion control 37 mechanisms. We believe that research in congestion control 38 mechanisms has been seriously hampered by the lack of good models 39 underpinning analysis, simulation, and testbed experiments, and that 40 tools for the evaluation of simulation and testbed scenarios can 41 help in the construction of better scenarios, based on better 42 underlying models. One use of the tools described in this document 43 is in comparing key characteristics of test scenarios with known 44 characteristics from the diverse and ever-changing real world. 45 Tools characterizing the aggregate traffic on a link include the 46 distribution of per-packet round-trip times, the distribution of 47 connection sizes, and the like. Tools characterizing end-to-end 48 paths include drop rates as a function of packet size and of burst 49 size, the synchronization ratio between two end-to-end TCP flows, 50 and the like. For each characteristic, we describe what aspects of 51 the scenario determine this characteristic, how the characteristic 52 can affect the results of simulations and experiments for the 53 evaluation of congestion control mechanisms, and what is known about 54 this characteristic in the real world. We also explain why the use 55 of such tools can add considerable power to our understanding and 56 evaluation of simulation and testbed scenarios. 58 Table of Contents 60 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 5 61 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 6 62 3. Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 63 3.1. Characterizing Aggregate Traffic on a Link . . . . . . . 6 64 3.2. Characterizing an End-to-End Path. . . . . . . . . . . . 6 65 3.3. Other Characteristics. . . . . . . . . . . . . . . . . . 7 66 4. The Distribution of Per-packet Round-trip Times . . . . . . . 7 67 5. The Distribution of Connection Sizes. . . . . . . . . . . . . 8 68 6. The Distribution of Packet Sizes. . . . . . . . . . . . . . . 9 69 7. The Ratio Between Forward-path and Reverse-path Traf- 70 fic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 71 8. The Distribution of Per-Packet Peak Flow Rates. . . . . . . . 11 72 9. The Distribution of Transport Protocols.. . . . . . . . . . . 12 73 10. The Synchronization Ratio. . . . . . . . . . . . . . . . . . 12 74 11. Drop or Mark Rates as a Function of Packet Size. . . . . . . 13 75 12. Drop Rates as a Function of Burst Size.. . . . . . . . . . . 15 76 13. Drop Rates as a Function of Sending Rate.. . . . . . . . . . 18 77 14. Congestion Control Mechanisms for Traffic, along 78 with Sender and Receiver Buffer Sizes. . . . . . . . . . . . . . 18 79 15. Characterization of Congested Links in Terms of 80 Bandwidth and Typical Levels of Congestion . . . . . . . . . . . 18 81 15.1. Bandwidth . . . . . . . . . . . . . . . . . . . . . . . 19 82 15.2. Queue Management Mechanisms . . . . . . . . . . . . . . 19 83 15.3. Typical Levels of Congestion. . . . . . . . . . . . . . 19 84 16. Characterization of Challenging Lower Layers.. . . . . . . . 19 85 16.1. Error Losses. . . . . . . . . . . . . . . . . . . . . . 19 86 16.2. Packet Reordering . . . . . . . . . . . . . . . . . . . 20 87 16.3. Delay Variation . . . . . . . . . . . . . . . . . . . . 20 88 16.4. Bandwidth Variation . . . . . . . . . . . . . . . . . . 21 89 16.5. Bandwidth and Latency Asymmetry . . . . . . . . . . . . 22 90 16.6. Queue Management Mechanisms . . . . . . . . . . . . . . 23 91 17. Network Changes Affecting Congestion . . . . . . . . . . . . 23 92 17.1. Routing Changes: Routing Loops . . . . . . . . . . . . 24 93 17.2. Routing Changes: Fluttering. . . . . . . . . . . . . . 24 94 17.3. Routing Changes: Routing Asymmetry . . . . . . . . . . 25 95 17.4. Link Disconnections and Intermittent Link Con- 96 nectivity . . . . . . . . . . . . . . . . . . . . . . . . . . 25 97 17.5. Changes in Wireless Links: Mobility . . . . . . . . . . 26 98 18. Using the Tools Presented in this Document . . . . . . . . . 27 99 19. Related Work . . . . . . . . . . . . . . . . . . . . . . . . 27 100 20. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . 27 101 21. Security Considerations. . . . . . . . . . . . . . . . . . . 27 102 22. IANA Considerations. . . . . . . . . . . . . . . . . . . . . 27 103 23. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 27 104 Informative References . . . . . . . . . . . . . . . . . . . . . 27 105 Editors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 106 Full Copyright Statement . . . . . . . . . . . . . . . . . . . . 31 107 Intellectual Property. . . . . . . . . . . . . . . . . . . . . . 32 108 TO BE DELETED BY THE RFC EDITOR UPON PUBLICATION: 110 Changes from draft-irtf-tmrg-tools-02.txt: 111 * Added sections on Challenging Lower Layers and Network 112 Changes affecting Congestion. Contributed by Jasani Rohan, 113 with Julie Tarr, Tony Desimone, Christou Christos, and 114 Vemulapalli Archana. 116 * Minor editing. 118 Changes from draft-irtf-tmrg-tools-01.txt: 119 * Added section on "Drop Rates as a Function of Sending Rate." 121 * Added a number of new references. 123 END OF SECTION TO BE DELETED. 125 1. Introduction 127 This document discusses tools for the evaluation of simulation and 128 testbed scenarios used in research on Internet congestion control 129 mechanisms. These tools include but are not limited to measurement 130 tools; the tools discussed in this document are largely ways of 131 characterizing aggregate traffic on a link, or characterizing the 132 end-to-end path. One use of these tools is for understanding key 133 characteristics of test scenarios; many characteristics, such as the 134 distribution of per-packet round-trip times on the link, don't come 135 from a single input parameter but are determined by a range of 136 inputs. A second use of the tools is to compare key characteristics 137 of test scenarios with what is known of the same characteristics of 138 the past and current Internet, and with what can be conjectured 139 about these characteristics of future networks. This paper follows 140 the general approach from "Internet Research Needs Better Models" 141 [FK02]. 143 As an example of the power of tools for characterizing scenarios, a 144 great deal is known about the distribution of connection sizes on a 145 link, or equivalently, the distribution of per-packet sequence 146 numbers. It has been conjectured that a heavy-tailed distribution 147 of connection sizes is an invariant feature of Internet traffic. A 148 test scenario with mostly long-lived traffic, or with a mix with 149 only long-lived and very short flows, does not have a realistic 150 distribution of connection sizes, and can give unrealistic results 151 in simulations or experiments evaluating congestion control 152 mechanisms. For instance, the distribution of connection sizes 153 makes clear the fraction of traffic on a link from medium-sized 154 connections, e.g., with packet sequence numbers from 100 to 1000. 156 These medium-sized connections can slow-start up to a large 157 congestion window, possibly coming to an abrupt stop soon 158 afterwards, contributing significantly to the burstiness of the 159 aggregate traffic, and to the problems facing congestion control. 161 In the sections below we will discuss a number of tools for 162 describing and evaluating scenarios, show how these characteristics 163 can affect the results of research on congestion control mechanisms, 164 and summarize what is known about these characteristics in real- 165 world networks. 167 2. Conventions 169 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 170 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 171 document are to be interpreted as described in [RFC2119]. 173 3. Tools 175 The tools or characteristics that we discuss are the following. 177 3.1. Characterizing Aggregate Traffic on a Link 179 o Distribution of per-packet round-trip times. 181 o Distribution of connection sizes. 183 o Distribution of packet sizes. 185 o Ratio between forward-path and reverse-path traffic. 187 o Distribution of peak flow rates. 189 o Distribution of transport protocols. 191 3.2. Characterizing an End-to-End Path 193 o Synchronization ratio. 195 o Drop rates as a function of packet size. 197 o Drop rates as a function of burst size. 199 o Drop rates as a function of sending rate. 201 o Degree of packet drops. 203 o Range of queueing delay. 205 3.3. Other Characteristics 207 o Congestion control mechanisms for traffic, along with sender and 208 receiver buffer sizes. 210 o Characterization of congested links in terms of bandwidth and 211 typical levels of congestion (in terms of packet drop rates). 213 o Characterization of congested links in terms of buffer size. 215 o Characterization of challenging lower layers in terms of 216 reordering, delay variation, packet corruption, and the like. 218 o Characterization of network changes affecting congestion, such as 219 routing changes or link outages. 221 Below we will discuss each characteristic in turn, giving the 222 definition, the factors determining that characteristic, the effect 223 on congestion control metrics, and what is known so far from 224 measurement studies in the Internet. 226 4. The Distribution of Per-packet Round-trip Times 228 Definition: The distribution of per-packet round-trip times on a 229 link is defined formally by assigning to each packet the most recent 230 round trip time measured for that end-to-end connection. In 231 practice, coarse-grained information is generally sufficient, even 232 though it has been shown that there is significant variability in 233 round-trip times within a TCP connection [AKSJ03], and it is 234 sufficient to assign to each packet the first round-trip time 235 measurement for that connection, or to assign the current round-trip 236 time estimate maintained by the TCP connection. 238 Determining factors: The distribution of per-packet round-trip times 239 on a link is determined by end-to-end propagation delays, by 240 queueing delays along end-to-end paths, and by the congestion 241 control mechanisms used by the traffic. For example, for a scenario 242 using TCP, TCP connections with smaller round-trip times will 243 receive a proportionally larger fraction of traffic than competing 244 TCP connections with larger round-trip times, all else being equal, 245 due to the dynamics of TCP favoring flows with smaller round-trip 246 times. This will generally shift the distribution of per-packet 247 RTTs lower relative to the distribution of per-connection RTTs, 248 since short-RTT connections will have more packets. 250 Effect on congestion control metrics: The distribution of per-packet 251 round-trip times on a link affects the burstiness of the aggregate 252 traffic, and therefore can affect congestion control performance in 253 a range of areas such as delay/throughput tradeoffs. The 254 distribution of per-packet round-trip times can also affect metrics 255 of fairness, degree of oscillations, and the like. For example, 256 long-term oscillations of queueing delay are more likely to occur in 257 scenarios with a narrow range of round-trip times [FK02]. 259 Measurements: The distribution of per-packet round-trip times for 260 TCP traffic on a link can be measured from a packet trace with the 261 passive TCP round-trip time estimator from Jiang and Dovrolis 262 [JD02]. [Add pointers to other estimators, such as ones mentioned 263 in JD02. Add a pointer to Mark Allman's loss detection tool.] 264 Their paper shows the distribution of per-packet round-trip times 265 for TCP packets for a number of different links. For the links 266 measured, the percent of packets with round-trip times at most 267 100 ms ranged from 30% to 80%, and the percent of packets with 268 round-trip times at most 200 ms ranged from 55% to 90%, depending on 269 the link. 271 In the NS simulator, the distribution of per-packet round-trip times 272 for TCP packets on a link can be reported by the queue monitor, 273 using TCP's estimated round-trip time added to packet headers. This 274 is illustrated in the validation test "./test-all-simple stats3" in 275 the directory tcl/test. 277 Scenarios: [FK02] shows a relatively simple scenario, with a 278 dumbbell topology with four access links on each end, that gives a 279 fairly realistic range of round-trip times. [Look for the other 280 citations to add.] 282 5. The Distribution of Connection Sizes 284 Definition: Instead of the connection-based measurement of the 285 distribution of connection sizes (the total number of bytes or of 286 data packets in a connection), we consider the packet-based 287 measurement of the distribution of packet sequence numbers. The 288 distribution of packet sequence numbers on a link is defined by 289 giving each packet a sequence number, where the first packet in a 290 connection has sequence number 1, the second packet has sequence 291 number 2, and so on. The distribution of packet sequence numbers 292 can be derived in a straightforward manner from the distribution of 293 connection sizes, and vice versa; however, the distribution of 294 connection sizes is more suited for traffic generators, and the 295 distribution of packet sequence numbers is more suited for measuring 296 and illustrating the packets actually seen on a link over a fixed 297 interval of time. There has been a considerably body of research 298 over the last ten years on the heavy-tailed distribution of 299 connection sizes for traffic on the Internet. [CBC95] [Add 300 citations.] 302 Determining factors: The distribution of connection sizes is largely 303 determined by the traffic generators used in a scenario. For 304 example, is there a single traffic generator characterized by a 305 distribution of connection sizes? A mix of long-lived and web 306 traffic, with the web traffic characterized by a distribution of 307 connection sizes? Or something else? 309 Effect on congestion control metrics: The distribution of packet 310 sequence numbers affects the burstiness of aggregate traffic on a 311 link, thereby affecting all congestion control metrics for which 312 this is a factor. As an example, [FK02] illustrates that the 313 traffic mix can affect the queue dynamics on a congested link. 314 [Find more to cite, about the effect of the distribution of packet 315 sequence numbers on congestion control metrics.] 317 [Add a paragraph about the impact of medium-size flows.] 319 [Add a paragraph about the impact of flows starting and stopping.] 321 [Add a warning about scenarios that use only long-lived flows, or a 322 mix of long-lived and very short flows.] 324 Measurements: [Cite some of the literature.] 326 Traffic generators: Some of the available traffic generators are 327 listed on the web site for "Traffic Generators for Internet Traffic" 328 [TG]. This includes pointers to traffic generators for peer-to-peer 329 traffic, traffic from online games, and traffic from Distributed 330 Denial of Service (DDoS) attacks. 332 In the NS simulator, the distribution of packet sequence numbers for 333 TCP packets on a link can be reported by the queue monitor at a 334 router. This is illustrated in the validation test "./test-all- 335 simple stats3" in the directory tcl/test. 337 6. The Distribution of Packet Sizes 339 Definition: The distribution of packet sizes is defined in a 340 straightforward way, using packet sizes in bytes. 342 Determining factors: The distribution of packet sizes is determined 343 by the traffic mix, the path MTUs, and by the packet sizes used by 344 the transport-level senders. 346 The distribution of packet sizes on a link is also determined by the 347 mix of forward-path TCP traffic and reverse-path TCP traffic in that 348 scenario, for a scenario characterized by a `forward path' (e.g., 349 left to right on a particular link) and a `reverse path' (e.g., 350 right to left on the same link). For such a scenario, the forward- 351 path TCP traffic contributes data packets to the forward link and 352 acknowledgment packets to the reverse link, while the reverse-path 353 TCP traffic contributes small acknowledgment packets to the forward 354 link. The ratio between TCP data and TCP ACK packets on a link can 355 be used as some indication of the ratio between forward-path and 356 reverse-path TCP traffic. 358 Effect on congestion control metrics: The distribution of packet 359 sizes on a link is an indicator of the ratio of forward-path and 360 reverse-path TCP traffic in that network. The amount of reverse- 361 path traffic determines the loss and queueing delay experienced by 362 acknowledgement packets on the reverse path, significantly affecting 363 the burstiness of the aggregate traffic on the forward path. [In 364 what other ways does the distribution of packet sizes affect 365 congestion control metrics?] 367 Measurements: There has been a wealth of measurements over time on 368 the packet size distribution of traffic [A00], [HMTG01]. These 369 measurements are generally consistent with a model of roughly 10% of 370 the TCP connections using an MSS of roughly 500 bytes, and with the 371 other 90% of TCP connections using an MSS of 1460 bytes. 373 7. The Ratio Between Forward-path and Reverse-path Traffic 375 Definition: For a scenario characterized by a `forward path' (e.g., 376 left to right on a particular link) and a `reverse path' (e.g., 377 right to left on the same link), the ratio between forward-path and 378 reverse-path traffic can be defined as the ratio between the 379 forward-path traffic in bps, and the reverse-path traffic in bps. 381 Determining factors: The ratio between forward-path and reverse-path 382 traffic is defined largely by the traffic mix. 384 Effect on congestion control metrics: Zhang, Shenker and Clark have 385 shown in 1991 that for TCP, the amount of reverse-path traffic 386 affects the ACK compression and packet drop rate for TCP 387 acknowledgement packets, significantly affecting the burstiness of 388 TCP traffic on the forward path [ZSC91]. The queueing delay on the 389 reverse path also affects the performance of delay-based congestion 390 control mechanisms, if the delay is computed based on round-trip 391 times. This has been shown by Grieco and Mascolo in [GM04] and by 392 Prasad, Jain, and Dovrolis in [PJD04]. 394 Measurements: There is a need for measurements on the range of 395 ratios between forward-path and reverse-path traffic for congested 396 links. In particular, for TCP traffic traversing congested link X, 397 what is the likelihood that the acknowledgement traffic will 398 encounter congestion (i.e., queueing delay, packet drops) somewhere 399 on the reverse path as well? 401 As discussed in Section 6, the distribution of packet sizes on a 402 link can be used as an indicator of the ratio of forward-path and 403 reverse-path TCP traffic in that network. 405 8. The Distribution of Per-Packet Peak Flow Rates 407 Definition: The distribution of peak flow rates is defined by 408 assigning to each packet the peak sending rate in bytes per second 409 of that connection, where the peak sending rate is defined over 410 0.1-second intervals. The distribution of peak flow rates gives 411 some indication of the ratio of "alpha" and "beta" traffic on a 412 link, where alpha traffic on a congested link is defined as traffic 413 with that link at the main bottleneck, while the beta traffic on the 414 link has a primary bottleneck elsewhere along its path [RSB01]. 416 Determining factors: The distribution of peak flow rates is 417 determined by flows with bottlenecks elsewhere along their end-to- 418 end path, e.g., flows with low-bandwidth access links. The 419 distribution of peak flow rates is also affected by applications 420 with limited sending rates. 422 Effect on congestion control metrics: The distribution of peak flow 423 rates affects the burstiness of aggregate traffic, with low-peak- 424 rate traffic decreasing the aggregate burstiness, and adding to the 425 traffic's tractability. 427 Measurements: [RSB01]. The distribution of peak rates can be 428 expected to change over time, as there is an increasing number of 429 high-bandwidth access links to the home, and of high-bandwidth 430 Ethernet links at work and at other institutions. 432 Simulators: [For NS, add a pointer to the DelayBox, 433 "http://dirt.cs.unc.edu/delaybox/", for more easily simulating low- 434 bandwidth access links for flows.] 436 Testbeds: In testbeds, Dummynet [Dummynet] and NISTNet [NISTNet] 437 provide convenient ways to emulate paths with different limited peak 438 rates. 440 9. The Distribution of Transport Protocols. 442 Definition: The distribution of transport protocols on a congested 443 link is straightforward, with each packet given its associated 444 transport protocol (e.g., TCP, UDP). The distribution is often 445 given both in terms of packets and in terms of bytes. 447 For UDP packets, it might be more helpful to classify them in terms 448 of the port number, or the assumed application (e.g., DNS, RIP, 449 games, Windows Media, RealAudio, RealVideo, etc.) [MAWI]). Other 450 traffic includes ICMP, IPSEC, and the like. In the future there 451 could be traffic from SCTP, DCCP, or from other transport protocols. 453 Effect on congestion control metrics: The distribution of transport 454 protocols affects metrics relating to the effectiveness of AQM 455 mechanisms on a link. 457 Measurements: In the past, TCP traffic has typically consisted of 458 90% to 95% of the bytes on a link [UW02], [UA01]. [Get updated 459 citations for this.] Measurement studies show that TCP traffic from 460 web servers almost always uses conformant TCP congestion control 461 procedures [MAF05]. 463 10. The Synchronization Ratio 465 Definition: The synchronization ratio is defined as the degree of 466 synchronization of loss events between two TCP flows on the same 467 path. Thus, the synchronization ratio is defined as a 468 characteristic of an end-to-end path. When one TCP flow of a pair 469 has a loss event, the synchronization ratio is given by the fraction 470 of those loss events for which the second flow has a loss event 471 within one round-trip time. Each connection in a flow pair has a 472 separate synchronization ratio, and the overall synchronization 473 ratio of the pair of flows is the higher of the two ratios. When 474 measuring the synchronization ratio, it is preferable to start the 475 two TCP flows at slightly different times, with large receive 476 windows. 478 Determining factors: The synchronization ratio is determined largely 479 by the traffic mix on the congested link, and by the AQM mechanism 480 (or lack of AQM mechanism). 482 Different types of TCP flows are also likely to have different 483 synchronization measures. E.g., Two HighSpeed TCP flows might have 484 higher synchronization measures that two Standard TCP flows on the 485 same path, because of their more aggressive window increase rates. 486 Raina, Towsley, and Wischik [RTW05] have discussed the relationships 487 between synchronization and TCP's increase and decrease parameters. 489 Effect on congestion control metrics: The synchronization ratio 490 affects convergence times for high-bandwidth TCPs. Convergence 491 times are known to be poor for some high-bandwidth protocols in 492 environments with high levels of synchronization. [Cite the papers 493 by Leith and Shorten.] However, it is not clear if these 494 environments with high levels of synchronization are realistic. 496 Wischik and McKeown [WM05] have shown that the level of 497 synchronization affects the buffer requirements at congested 498 routers. Baccelli and Hong [BH02] have a model showing the effect 499 of the synchronization ratio on aggregate throughput. 501 Measurements: Grenville Armitage and Qiang Fu have performed initial 502 experiments of synchronization in the Internet, using Standard TCP 503 flows, and have found very low levels of synchronization. 505 In a discussion of the relationship between stability and 506 desynchronization, Raina, Towsley, and Wischik [RTW05] report that 507 "synchronization has been reported again and again in simulations". 508 In contrast, synchronization has not been reported again and again 509 in the real-world Internet. 511 Appenzeller, Keslassy, and McKeown in [AKM04] report the following: 512 "Flows are not synchronized in a backbone router carrying thousands 513 of flows with varying RTTs. Small variations in RTT or processing 514 time are sufficient to prevent synchronization [QZK01]; and the 515 absence of synchronization has been demonstrated in real networks 516 [F02,IMD01]." 518 [Appenzeller et al, Sizing Router Buffers, reports that 519 synchronization is rare as the number of competing flows increases. 520 Kevin Jeffay has some results on synchronization also.] 522 Needed: We need measurements of the synchronization ratio for flows 523 that use high-bandwidth protocols over high-bandwidth paths, given 524 typical levels of competing traffic and with typical queueing 525 mechanisms at routers (whatever these are), to see if there are 526 higher levels of synchronization with high-bandwidth protocols such 527 as HighSpeed TCP, Fast TCP, and the like, which are more aggressive 528 than Standard TCP. The assumption would be that in many 529 environments, high-bandwidth protocols have higher levels of 530 synchronization than flows using Standard TCP. 532 11. Drop or Mark Rates as a Function of Packet Size 534 Definition: Drop rates as a function of packet size are defined by 535 the actual drop rates for different packets on an end-to-end path or 536 on a congested link over a particular time interval. In some cases, 537 e.g., Drop-Tail queues in units of packets, general statements can 538 be made; e.g., that large and small packets will experience the same 539 packet drop rates. However, in other cases, e.g., Drop-Tail queues 540 in units of bytes, no such general statement can be made, and the 541 drop rate as a function of packet size will be determined in part by 542 the traffic mix at the congested link at that point of time. 544 Determining factors: The drop rate as a function of packet size is 545 determined in part by the queue architecture. E.g., is the Drop- 546 Tail queue in units of packets, of bytes, of 60-byte buffers, or of 547 a mix of buffer sizes? Is the AQM mechanism in packet mode, 548 dropping each packet with the same probability, or in byte mode, 549 with the probability of dropping or marking a packet being 550 proportional to the packet size in bytes. 552 The effect of packet size on drop rate would also be affected by the 553 presence of preferential scheduling for small packets, or by 554 differential scheduling for packets from different flows (e.g., per- 555 flow scheduling, or differential scheduling for UDP and TCP 556 traffic). 558 In many environments, the drop rate as a function of packet size 559 will be heavily affected by the traffic mix at a particular time. 560 For example, is the traffic mix dominated by large packets, or by 561 smaller ones? In some cases, the overall packet drop rate could 562 also affect the relative drop rates for different packet sizes. 564 In wireless networks, the drop rate as a function of packet size is 565 also determined by the packet corruption rate as a function of 566 packet size. [Cite Deborah Pinck's papers on Satellite-Enhanced 567 Personal Communications Experiments and on Experimental Results from 568 Internetworking Data Applications Over Various Wireless Networks 569 Using a Single Flexible Error Control Protocol.] [Cite the general 570 literature.] 572 Effect on congestion control metrics: The drop rate as a function of 573 packet size has a significant effect on the performance of 574 congestion control for VoIP and other small-packet flows. 575 [Citation: "TFRC for Voice: the VoIP Variant", draft-ietf-dccp-tfrc- 576 voip-02.txt, and earlier papers.] The drop rate as a function of 577 packet size also has an effect on TCP performance, as it affects the 578 drop rates for TCP's SYN and ACK packets. [Citation: Jeffay and 579 others.] 581 Measurements: We need measurements of the drop rate as a function of 582 packet size over a wide range of paths, or for a wide range of 583 congested links. For tests of relative drop rates on end-to-end 584 packets, one possibility would be to run successive TCP connections 585 with 200-byte, 512-byte, and 1460-byte packets, and to compare the 586 packet drop rates. The ideal test would include running TCP 587 connections on the reverse path, to measure the drop rates for the 588 small ACK packets on the forward path. It would also be useful to 589 characterize the difference in drop rates for 200-byte TCP packets 590 and 200-byte UDP packets, even though some of this difference could 591 be due to the relative burstiness of the different connections. 593 Ping experiments could also be used to get measurements of drop 594 rates as a function size, but it would be necessary to make sure 595 that the ping sending rates were adjusted to be TCP-friendly. 597 [Cite the known literature on drop rates as a function of packet 598 size.] 600 Our conjecture is that there is a wide range of behaviors for this 601 characteristic in the real world. Routers include Drop-Tail queues 602 in packets, bytes, and buffer sizes in between; these will have 603 quite different drop rates as a function of packet size. Some 604 routers include RED in byte mode (the default for RED in Linux) and 605 some have RED in packet mode (Cisco, I believe). This also affects 606 drop rates as a function of packet size. 608 Some routers on congested access links use per-flow scheduling. In 609 this case, does the per-flow scheduling have the goal of fairness in 610 *bytes* per second or in *packets* per second? What effect does the 611 per-flow scheduling have on the drop rate as a function of packet 612 size, for packets in different flows (e.g., a small-packet VoIP flow 613 competing against a large-packet TCP flow) or for packets within the 614 same flow (small ACK packets and large data packets on a two-way TCP 615 connection). 617 12. Drop Rates as a Function of Burst Size. 619 Definition: Burst-tolerance, or drop rates as a function of burst 620 size, can be defined in terms of an end-to-end path, or in terms of 621 aggregate traffic on a congested link. 623 The burst-tolerance of an end-to-end path is defined in terms of 624 connections with different degrees of burstiness within a round-trip 625 time. When packets are sent in bursts of N packets, does the drop 626 rate vary as a function of N? For example, if the TCP sender sends 627 small bursts of K packets, for K less than the congestion window, 628 how does the size of K affect the loss rate? Similarly, for a ping 629 tool sending pings at a certain rate in packets per second, one 630 could see how the clustering of the ping packets in clusters of size 631 K affects the packet drop rate. As always with such ping 632 experiments, it would be important to adjust the sending rate to 633 maintain a longer-term sending rate that was TCP-friendly. 635 Determining factors: The burst-tolerance is determined largely by 636 the AQM mechanisms for the congested routers on a path, and by the 637 traffic mix. For a Drop-Tail queue with only a small number of 638 competing flows, the burst-tolerance is likely to be low, and for 639 AQM mechanisms where the packet drop rate is a function of the 640 average queue size rather than the instantaneous queue size, the 641 burst tolerance should be quite high. 643 Effect on congestion control metrics: The burst-tolerance of the 644 path or congested link can affect fairness between competing flows 645 with different round-trip times; for example, Standard TCP flows 646 with longer round-trip times are likely to have a more bursty 647 arrival pattern at the congested link that that of Standard TCP 648 flows with shorter round-trip times. As a result, in environment 649 with low burst tolerance (e.g., scenarios with Drop-Tail queues), 650 longer-round-trip-time TCP connections can see higher packet drop 651 rates than other TCP connections, and receive an even smaller 652 fraction of the link bandwidth than they would otherwise. [FJ92] 653 (Section 3.2). We note that some TCP traffic is inherently bursty, 654 e.g., Standard TCP without rate-based pacing, particularly in the 655 presence of dropped ACK packets or of ACK compression. The burst- 656 tolerance of a router can also affect the delay-throughput tradeoffs 657 and packet drop rates of the path or of the congested link. 659 Measurements: One could measure the burst-tolerance of an end-to-end 660 path by running successive TCP connections, forcing bursts of size 661 at least K by dropping an appropriate fraction of the ACK packets to 662 the TCP receiver. Alternately, if one had control of the TCP 663 sender, one could modify the TCP sender to send bursts of K packets 664 when the congestion window was K or more segments. 666 Blanton and Allman in [BA05] consider the TCP micro-bursts that 667 result from the receipt of a single acknowledgement packet or from 668 application-layer dynamics, and consider bursts of four or more 669 packets. They consider four traces, and plot the probability of at 670 least one packet from a burst being lost, as a function of burst 671 size. Considering only connections with both bursts and packet 672 losses, the probability of packet loss when the TCP connection was 673 bursting was somewhat higher than the probability of packet loss 674 when the TCP connection was not bursting in three of the four 675 traces. For each trace, the paper shows the aggregate probability 676 of loss as a function of the burst size in packets. Because these 677 are aggregate statistics, it cannot be determined if there is a 678 correlation between the burst size and the TCP connection's sending 679 rate. 681 [Look at: M. Allman and E. Blanton, "Notes on Burst Mitigation for 682 Transport Protocols", ACM Computer Communication Review, vol. 35(2), 683 (2005).] 685 Making inferences about the AQM mechanism for the congested router 686 on an end-to-end path: One potential use of measurement tools for 687 determining the burst-tolerance of an end-to-end path would be to 688 make inferences about the presence or absence of an AQM mechanism at 689 the congested link or links. As a simple test, one could run a TCP 690 connection until the connection comes out of slow-start. If the 691 receive window of the TCP connection was sufficiently high that the 692 connection exited slow-start with packet drops or marks instead of 693 because of the limitation of the receive window, one could record 694 the congestion window at the end of slow-start, and the number of 695 packets dropped from this window. A high packet drop rate might be 696 more typical of a Drop-Tail queue with small-scale statistical 697 multiplexing on the congested link, and a single packet drop coming 698 out of slow-start would suggest an AQM mechanism at the congested 699 link. 701 The synchronization measure could also add information about the 702 likely presence or absence of AQM on the congested link(s) of an 703 end-to-end path, with paths with higher levels of synchronization 704 being more likely to have Drop-Tail queues with small-scale 705 statistical multiplexing on the congested link(s). 707 Lui and Crovella in [LC01] use loss pairs to infer the queue size 708 when packets are dropped. A loss pair consists of two packets sent 709 back-to-back, where one of the two packets is dropped in the 710 network. The round-trip time of the surviving packet is used to 711 estimate the round-trip time when the companion packet was dropped 712 in the network. For a path with Drop-Tail queueing at the congested 713 link, this round-trip time can be used to estimate the queue size, 714 given estimates of the link bandwidth and minimum round-trip time. 715 For a path with AQM at the congested link, trial pairs are also 716 considered, where a trial pair is any pair of packets sent back-to- 717 back. [LC01] uses the ratio between the number of loss pairs and 718 the number of trial pairs for each round-trip range to estimate the 719 drop probability of the AQM mechanism at the congested link as a 720 function of queue size. [LC01] uses loss pairs in simulation 721 settings with a minimum of noise in terms of queueing delays 722 elsewhere on the forward or reverse path. 724 [Cite the relevant literature about tools for determining the AQM 725 mechanism on an end-to-end path.] 727 13. Drop Rates as a Function of Sending Rate. 729 Definition: Drop rates as a function of sending rate is defined in 730 terms of the drop behavior of a flow in the end-to-end path. That 731 is, does the sending rate of an individual flow affect its own 732 packet drop rate, or its packet drop rate largely independent of the 733 sending rate of the flow? 735 Determining factors: The sending rate of the flow affects its own 736 packet drop rate in an environment with small-scale statistical 737 multiplexing on the congested link. The packet drop rate is largely 738 independent of the sending rate in an environment with large-scale 739 statistical multiplexing, with many competing small flows at the 740 congested link. Thus, the behavior of drop rates as a function of 741 sending rate is a rough measure of the level of statistical 742 multiplexing on the congested links of an end-to-end path. 744 Effect on congestion control metrics: The level of statistical 745 multiplexing at the congested link can affect the performance of 746 congestion control mechanisms in transport protocols. For example, 747 delay-based congestion control is often better suited to 748 environments small-scale statistical multiplexing at the congested 749 link, where the transport protocol responds to the delay caused by 750 its own sending rate. 752 Measurements: In a simulation or testbed, the level of statistical 753 multiplexing on the congested link can be observed directly. In the 754 Internet, the level of statistical multiplexing on the congested 755 links of an end-to-end path can be inferred indirectly through per- 756 flow measurements, by observing whether the packet drop rate varies 757 as a function of the sending rate of the flow. 759 14. Congestion Control Mechanisms for Traffic, along with Sender and 760 Receiver Buffer Sizes. 762 Effect on congestion control metrics: Please don't evaluate AQM 763 mechanisms by using Reno TCP, or evaluate new transport protocols by 764 comparing them with the performance of Reno TCP! For an 765 explanation, see [FK02] (Section 3.4). 767 Measurements: See [MAF05]. 769 15. Characterization of Congested Links in Terms of Bandwidth and 770 Typical Levels of Congestion 771 15.1. Bandwidth 773 15.2. Queue Management Mechanisms 775 15.3. Typical Levels of Congestion 777 [Pointers to the current state of our knowledge.] 779 16. Characterization of Challenging Lower Layers. 781 With an increasing number of wireless networks connecting to the 782 wired Internet, more and more end-to-end paths will contain a 783 combination of wired and wireless links. These wireless links 784 exhibit new characteristics which congestion control mechanisms will 785 need to cope with. The main characteristics, detailed in subsequent 786 sections, include error losses, packet reordering, delay variation, 787 bandwidth variation, and bandwidth and latency asymmetry. 789 16.1. Error Losses 791 Definition: Packet losses due to corruption rarely occur on wired 792 links, but occur on wireless links due to random/transient errors 793 and/or extended burst errors. If packet errors cannot be detected 794 and discarded within the network through error detection schemes or 795 recovered through error recovery schemes such as Forward Error 796 Correction (FEC) and Automatic Repeat Request (ARQ), the corrupted 797 packet is discarded, resulting in an error loss. 799 Determining Factors: Error losses are primarily caused by the 800 degradation of the quality of a wireless link (multipath, fade, 801 etc.). Link errors can be characterized by the type of errors that 802 occur (e.g., random, burst), the length of time they occur, and the 803 frequency at which they occur. These characteristics are highly 804 dependent on the wireless channel conditions and are influenced by 805 the distance between two nodes on a wireless link, the type and 806 orientation of antennas, encoding algorithms, and other factors 807 [22]. Therefore, error losses are significantly influenced by these 808 link errors. 810 Effect on congestion control metrics: Since error losses can be 811 unrelated to congestion, congestion control mechanisms should 812 recover from these types of losses differently than from congestion 813 losses. If congestion control mechanisms misinterpret error losses 814 as congestion losses, then they respond inappropriately, reducing 815 the sending rate too much [23]. As a result, an unnecessary 816 reduction in the sending rate can occur, when in reality the 817 available bandwidth has not changed. This can result in a reduction 818 in throughput and underutilization of the channel. However, error 819 recovery mechanisms such as FEC or ARQ are heavily used in cellular 820 networks to reduce the impact of error losses [IMLGK03]. 822 Measurements: In 3G cellular networks, error recovery mechanisms 823 have reduced the rate of error losses to under 1%, making their 824 impact marginal [CR04]. 826 16.2. Packet Reordering 828 Definition: Due to the connectionless nature of IP, packets can 829 arrive out of order at their destination. Packet reordering events 830 can occur at varying times and to varying degrees. For example, a 831 particular channel may reorder one out of ten packets and the 832 reordered packet arrives three packets out of order. 834 Determining Factors: For the most part, packet reordering on 835 wireless links rarely occurs. However, packet re-ordering can occur 836 due to link layer error recovery. Extensive packet reordering has 837 been shown to occur with particular handoff mechanisms, and is 838 definitely detrimental to transport performance [GF04]. 840 Effects on congestion control metrics: With TCP, packet reordering 841 can cause the receiver to wait for the arrival of packets that are 842 out of order, since the receiver must reassemble the packets in the 843 correct order before passing them up to the application. With TCP 844 and other transport protocols, packet reordering can also result in 845 the sender incorrectly inferring packet loss, triggering packet 846 retransmissions and congestion control responses. 848 Measurements: Measurements by Zhou and Mieghem show that reordering 849 happens quite often in the Internet, but few streams have more than 850 two reordered packets [ZM04]. For further measurements, see 851 [ANP06][BPS99][LC05]. 853 16.3. Delay Variation 855 Definition: Delay Variation occurs when selected packets of a given 856 flow experience a difference in the One-Way-Delay across a network. 857 Delay variation can be caused by a variation in propagation, 858 transmission and queueing delay that can occur across links or 859 network nodes. 861 Determining Factors: Delay and delay variation is introduced due to 862 various features of wireless links [IMLGK03]. The delay experience 863 by subsequent packets of a given flow can change due to On-Demand 864 Resource Allocation, which allocates a wireless channel to a user 865 based on current bandwidth availability. In addition, FEC and ARQ, 866 which are commonly used to combat error loss on wireless links, can 867 introduce delay into the channel, depending on the degree of error 868 loss that occurs. These mechanisms either resend packets that have 869 been corrupted or attempt to recover the actual corrupted packet, 870 which both add delay to the channel. 872 Effect on congestion control metrics: A spike in delay can have a 873 negative impact on transport protocols [AAR03][AGR00][CR04]. 874 Transport protocols use timers for loss recovery and for congestion 875 control, which are set according to the RTT. Delay spikes can 876 trigger spurious timeouts that cause unnecessary retransmissions and 877 incorrect congestion control responses. if these delay spikes 878 continue, they can inflate the retransmission timeout, increasing 879 the wait before a dropped packet is recovered. Delay-based 880 congestion control mechanisms (e.g. TCP Vegas, TCP Westwood, etc.) 881 use end-to-end delay to control the sending rate of the sender. 882 Delay-based congestion control mechanisms use delay to indicate when 883 there is congestion in the network. When delay variation occurs for 884 reasons other than queueing delay, delay based congestion control 885 mechanisms can reduce the sending rate unnecessarily. Rate-based 886 protocols can perform poorly as they do not adjust the sending rate 887 after a change in the RTT, possibly creating unnecessary congestion 888 [GF04]. 890 Measurements: Cellular links, particularly GPRS and CDMA2000, can 891 have one-way latencies varying from 100 to 500 ms [IMLGK03]. The 892 length of a delay variation event can vary from three to fifteen 893 seconds and the frequency at which delay variation events occur can 894 be anywhere from 40 to 400 seconds. GEO satellite links tend not 895 see much variation in delay, while LEO satellite links can see 896 significant variability in delay due to the constant motion of 897 satellites and multiple hops. The delay variation of LEO satellite 898 links can be from 40 to 400ms [GK98]. 900 16.4. Bandwidth Variation 902 Definition: The bandwidth of a wireless channel can vary over time 903 during a single session, as wireless networks can change the 904 available bandwidth allotted to a user. Therefore, a user may have 905 a low-bandwidth channel for part of their session and a high- 906 bandwidth channel the remainder of the session. The bandwidth of 907 the channel can vary abruptly or gradually, at various intervals, 908 and these variations can occur at different times. On-demand 909 Resource Allocation is one of the mechanisms used to dynamically 910 allocate resources to users according to system load and traffic 911 priority. For instance, in GPRS a radio channel is allocated when 912 data arrives toward the user, and released when the queue size falls 913 below a certain threshold [GPAR02][W01]. 915 Determining Factors: The amount of bandwidth of a given channel 916 that is allocated by the wireless network to a user can vary based 917 on a number of factors. Factors such as wireless conditions or the 918 amount of users connected to the base station both affect the 919 available capacity [IMLGK03]. In certain satellite systems, 920 technologies such as On-Demand Resource Allocation constantly adjust 921 the available bandwidth for a given user every second, which can 922 cause significant bandwidth variation during a session. On-Demand 923 Resource Allocation is designed into most 2.5 and 3G wireless 924 networks, but it can be implemented differently from network to 925 network, resulting in different impacts on the link. 927 Effect on congestion control metrics: In the absence of congestion, 928 congestion control mechanisms increase the sending rate gradually 929 over multiple round-trip times. If the bandwidth of a wireless 930 channel suddenly increases and this increases the bandwidth 931 available on the end-to-end path, the transport protocol might not 932 be able to increase its sending rate quickly enough to use the 933 newly-available bandwidth [24]. If the bandwidth of a wireless 934 channel suddenly decreases and this decreases the bandwidth 935 available on the end-to-end path, the sender might not decrease its 936 sending rate quickly enough, resulting in transient congestion. 937 Frequent changes of bandwidth on a wireless channel can result in 938 the average transmission rate of the channel being limited by the 939 amount of bandwidth available during times where the channel has the 940 lowest bandwidth. Persistent delay variation can inflate the 941 retransmission timeout, increasing the wait before a dropped packet 942 is recovered, ultimately leading to channel underutilization 943 [GPAR02][W01]. 945 Measurements: Further references on the measurements for the amount 946 of bandwidth variation are needed. On-demand channel allocation can 947 be modeled by introducing an additional delay when a packet arrives 948 to a queue that has been empty longer than the channel hold time 949 (i.e., propagation delay). The delay value represents the channel 950 allocation delay, and the hold time represents the duration of 951 channel holding after transmitting a data packet [GPAR02][W01]. See 952 also [NM01]. 954 16.5. Bandwidth and Latency Asymmetry 956 Definition: The bandwidth in the forward direction or uplink can be 957 different than the bandwidth in the reverse direction or downlink. 958 Similar to bandwidth asymmetry, latency in the forward direction or 959 uplink can be different than latency in the reverse direction or 960 downlink. For example, bandwidth asymmetry occurs in wireless 961 networks where the channel from the mobile to base station (uplink) 962 has a fraction of the bandwidth of the channel from the base station 963 to the mobile channel (downlink). 965 Determining Factors: Bandwidth and latency asymmetry can occur for 966 a variety of reasons. Mobile devices that must transmit at lower 967 power levels to conserve power have low bandwidth and high latency 968 transmission, while base stations can transmit at higher power 969 levels, resulting in higher bandwidth and lower latency [IMLGK03]. 970 In addition, because applications such as HTTP require significantly 971 more bandwidth on the downlink as opposed to the uplink, wireless 972 networks have been designed with asymmetry to accommodate these 973 applications. Coupled with these design constraints, the 974 environmental conditions can add increased asymmetry [HK99]. 976 Effect on congestion control metrics: TCP's congestion control 977 algorithms rely on ACK-clocking, with the reception of ACKs 978 controlling sending rates. If ACKs are dropped or delayed in the 979 reverse direction, then the sending rate in the forward direction 980 can be reduced. In addition, excessive delay can result in a 981 retransmit timeout and a corresponding reduction in the sending rate 982 [HK99][23]. 984 Measurements: For cellular networks the downlink bandwidth 985 typically does not exceed three to six times the uplink bandwidth 986 [IMLGK03]. However, different cellular networks (e.g. IS-95, 987 CDMA2000, etc.) have different ratios of bandwidth and latency 988 asymmetry. 990 16.6. Queue Management Mechanisms 992 In wireless networks, queueing delay typically occurs at the end 993 points of a wireless connection (i.e. mobile device, base station) 994 [GF04]. 996 Measurements: For current cellular and WLAN links, the queue can 997 plausibly be modeled with Drop-Tail queueing with a configurable 998 maximum size in packets. The use of RED may be more appropriate for 999 modeling satellite or future cellular and WLAN links [GF04]. 1001 17. Network Changes Affecting Congestion 1003 Changes in the network can have a significant impact on the 1004 performance of congestion control algorithms. These changes can 1005 include events such as the unnecessary duplication of packets, 1006 topology changes due to node mobility, and temporary link 1007 disconnections. These types of network changes can be broadly 1008 categorized as routing changes, link disconnections and intermittent 1009 link connectivity, and mobility. 1011 17.1. Routing Changes: Routing Loops 1013 Definition: A routing loop is a network event in which packets 1014 continue to be routed in an endless circle until the packets are 1015 eventually dropped [P96]. 1017 Determining factors: Routing loops can occur when the network 1018 experiences a change in connectivity which is not immediately 1019 propagated to all of the routers [H95]. Network and node mobility 1020 are examples of network events that can cause a change in 1021 connectivity. 1023 Effect on Congestion Control: Loops can rapidly lead to congestion, 1024 as a router will route packets towards a destination, but the 1025 forwarded packets end up being routed back to the router. 1026 Furthermore, network congestion due to a routing loop with multicast 1027 packets will be more severe than with unicast packets because each 1028 router replicates multicast packets thereby causing congestion more 1029 rapidly [P96]. 1031 Measurements: Routing dynamics that lead to temporary route loss or 1032 forwarding loops are also called routing failures. ICMP response 1033 messages, measured by traceroutes and pings, can be used to identify 1034 routing failures [WMWGB06]. 1036 17.2. Routing Changes: Fluttering 1038 Definition: The term fluttering is used to describe rapidly- 1039 oscillating routing. Fluttering occurs when a router alternates 1040 between multiple next-hop routers in order to split the load among 1041 the links to those routers. While fluttering can provide benefits 1042 as a way to balance load in a network, it also creates problems for 1043 TCP [P96][AP99]. 1045 Determining factors: Multi-path routing in the Internet can cause 1046 route fluttering. Route fluttering can result in significant out- 1047 of-order packet delivery and/or frequent abrupt end-to-end RTT 1048 variation [P97]. 1050 Effect on Congestion Control Metrics: When two routes have different 1051 propagation delays, packets will often arrive at the destination 1052 out-of-order, depending on whether they arrived via the shorter 1053 route or the longer route. Whenever a TCP endpoint receives an out- 1054 of-order packet, this triggers the transmission of a duplicate 1055 acknowledgement to inform the sender that the receiver has a hole in 1056 its sequence space. If three out-of-order packets arrive in a row, 1057 then the receiver will generate three duplicate acknowledgements for 1058 the segment that was not received. These duplicate acknowledgements 1059 will trigger fast retransmit by the sender, leading it to reduce the 1060 sending rate and needlessly retransmit data. Thus, out-of-order 1061 delivery can result in unnecessary reductions in the sending rate 1062 and also in redundant network traffic, due to extra acknowledgements 1063 and possibly unnecessary data retransmissions [P96][AP99]. 1065 Measurements: Two metrics [WMWGB06] can be used to measure the 1066 degree of out-of-order delivery: the number of reordering packets 1067 and the reordering offset. The number of reordering packets is 1068 simply the number of packets that are considered out of order. The 1069 reordering offset for an out-of-order packet is the difference 1070 between the actual arrival order and the expected arrival order. 1071 See also [LMJ96]. 1073 17.3. Routing Changes: Routing Asymmetry 1075 Definition: Routing asymmetry occurs when packets traveling between 1076 two end-points follow different routes in the forward and reverse 1077 directions. The two routes could have different characteristics in 1078 terms of bandwidth, delay, levels of congestion, etc. [P96]. 1080 Determining factors: Some of the main causes for asymmetry are 1081 policy routing, traffic engineering, and the absence of a unique 1082 shortest path between a pair of hosts. While the lack of a unique 1083 shortest path is one potential contributor to asymmetric routing 1084 within domains, the principal source of asymmetries in backbone 1085 routers is policy routing. Another cause of routing asymmetry is 1086 adaptive routing, in which a router shifts traffic from a highly 1087 loaded link to a less loaded one, or load balances across multiple 1088 paths [P96]. 1090 Effect on Congestion Control: When delay-based congestion control 1091 is used, asymmetry can introduce problems in estimating the one-way 1092 latency between hosts. 1094 Measurements: Further references are needed. 1096 17.4. Link Disconnections and Intermittent Link Connectivity 1098 Definition: A link disconnection is a period when the link loses all 1099 frames, until the link is restored. Intermittent Link Connectivity 1100 occurs when the link is disconnected regularly and for short periods 1101 of time. This is a common characteristic of wireless links, 1102 particularly those with highly mobile nodes [AP99]. 1104 Determining factors: In a wireless environment, link disconnections 1105 and intermittent link connectivity could occur when a mobile device 1106 leaves the range of a base station, which can lead to signal 1107 degradation or failure in a handoff [AP99]. 1109 Effect on Congestion Control Metrics: If a link disconnection lasts 1110 longer than the TCP RTO, and results in a path disconnection for 1111 that period of time, the TCP sender will perform a retransmit 1112 timeout, resending a packet and reducing the sending rate. TCP will 1113 continue this pattern, with longer and longer retransmit timeouts, 1114 up to a retry limit, until an acknowledgement is received. TCP only 1115 determines that connectivity has been restored after a (possibly 1116 long) retransmit timeout followed by the successful receipt of an 1117 ACK. Thus a link disconnection can result in a long delay in 1118 sending accompanied by a significant reduction in the sending rate 1119 [AP99]. 1121 Measurements: End-to-end performance under realistic topology and 1122 routing policies can be studied; [WMWGB06] suggests controlling 1123 routing events by injecting well-designed routing updates at known 1124 times to emulate link failures and repairs. 1126 17.5. Changes in Wireless Links: Mobility 1128 Definition: Network and node mobility, both wired and wireless, 1129 allows users to roam from one network to another seamlessly without 1130 losing service. 1132 Determining factors: Mobility is a key attribute of wireless 1133 networks. Mobility can determined by the presence of intersystem 1134 handovers, an intrinsic property of most wireless links [HS03]. 1136 Effect on Congestion Control Metrics: Mobility presents a major 1137 challenge to transport protocols through the packet losses and delay 1138 introduced by handovers. In addition to delay and losses, handovers 1139 can also cause a significant change in link bandwidth and latency. 1140 Host mobility increases packet delay and delay variation, and also 1141 degrades the throughput of TCP connections in wireless environments. 1142 Also, in the event of a handoff, slowly-responsive congestion 1143 control can require considerable time to adapt to changes. For 1144 example a flow under-utilizes a fast link after a handover from a 1145 slow link [HS03]. 1147 Measurements: Further references are needed to specify how mobility 1148 is actually measured [JEAS03]. 1150 18. Using the Tools Presented in this Document 1152 [To be done.] 1154 19. Related Work 1156 [Cite "On the Effective Evaluation of TCP" by Allman and Falk.] 1158 20. Conclusions 1160 [To be done.] 1162 21. Security Considerations 1164 There are no security considerations in this document. 1166 22. IANA Considerations 1168 There are no IANA considerations in this document. 1170 23. Acknowledgements 1172 Thanks to Xiaoliang (David) Wei for feedback and contributions to 1173 this document. The sections on "Challenging Lower Layers" and 1174 "Network Changes Affecting Congestion" are contributions from Jasani 1175 Rohan with Julie Tarr, Tony Desimone, Christou Christos, and 1176 Vemulapalli Archana. 1178 Informative References 1180 [RFC2119] S. Bradner. Key Words For Use in RFCs to Indicate 1181 Requirement Levels. RFC 2119. 1183 [MAWI] M.W. Group, Mawi working group traffic archive, URL 1184 "http://tracer.csl.sony.jp/mawi/". 1186 [A00] M. Allman, A Web Server's View of the Transport Layer, 1187 Computer Communication Review, 30(5), October 200. 1189 [AAR03] Alhussein A. Abouzeid, Sumit Roy, Stochastic Modeling of TCP 1190 in Networks with Abrupt Delay Variations, Wireless Networks, V.9 1191 N.5, 2003. 1193 [AGR00] M. Allman, J. Griner, and A. Richard. TCP Behavior in 1194 Networks with Dynamic Propagation Delay. Globecom 2000, November 1195 2000. 1197 [AKM04] B. Appenzeller, I. Keslassy, and N. McKeown, Sizing Router 1198 Buffers, SIGCOMM 2004. 1200 [AKSJ03] J. Aikat, J. Kaur, F.D. Smith, and K. Jeffay, Variability 1201 in TCP Roundtrip Times, ACM SIGCOMM Internet Measurement 1202 Conference, Maimi, FL, October 2003, pp. 279-284. 1204 [ANP06] On Monitoring of End-to-End Packet Reordering over the 1205 Internet, B. Ye, A. P. Jayasumana, and N. M. Piratla, 2006. 1207 [AP99] M. Allman and V. Paxson. On Estimating End-to-end Network 1208 Path Properties. SIGCOMM, September 1999. 1210 [BH02] F. Baccelli and D. Hong, AIMD, Fairness and Fractal Scaling 1211 of TCP Traffic, Infocom 2002. 1213 [BA05] E. Blanton and M. Allman, "On the Impack to Bursting on TCP 1214 Performance", Passive and Active Measurement Workshop, March 1215 2005. 1217 [BPS99] Bennett, J. C. R., Partridge, C. and Shectman, N., "Packet 1218 Reordering is Not Pathological Network Behavior," Trans. on 1219 Networking IEEE/ACM, Dec. 1999, pp.789-798. 1221 [CBC95] C. Cunha, A. Bestavros, and M. Crovella, "Characteristics of 1222 WWW Client-based Traces", BU Technical Report BUCS-95-010, 1995. 1224 [CR04] M. Chan and R. Ramjee, (2004) Improving TCP/IP Performance 1225 over Third Generation Wireless Networks. IEEE Infocom 2004. 1227 [Dummynet] L. Rizzo, Dummynet, URL 1228 "http://info.iet.unipi.it/~luigi/ip_dummynet/". 1230 [F02] C. J. Fraleigh, Provisioning Internet Backbone Networks to 1231 Support Latency Sensitive Applications. PhD thesis, Stanford 1232 University, Department of Electrical Engineering, June 2002. 1234 [FJ92] S. Floyd and V. Jacobson, On Traffic Phase Effects in Packet- 1235 Switched Gateways, Internetworking: Research and Experience, V.3 1236 N.3, September 1992, p.115-156. 1238 [FK02] S. Floyd and E. Kohler, Internet Research Needs Better 1239 Models, Hotnets-I, October 2002. 1241 [GF04] A. Gurtov and S. Floyd. Modeling Wireless Links for Transport 1242 Protocols. ACM CCR, 34(2):85-96, April 2004. 1244 [GK98] B. Gavish and J. Kalvenes. The Impact of Satellite Altitude 1245 on the Performance of LEOS based Communication Systems. Wireless 1246 Networks, 4(2):199--212, 1998. 1248 [GM04] L. Grieco and S. Mascolo, Performance Evaluation and 1249 Comparison of Westwood+, New Reno, and Vegas TCP Congestion 1250 Control, CCR, April 2004. 1252 [GPAR02] A. Gurtov, M. Passoja, O. Aalto, and M. Raitola. Multi- 1253 layer Protocol Tracing in a GPRS Network. In Proc. of the IEEE 1254 Vehicular Technology Conference (VTC02 Fall), Sept. 2002. 1256 [H95] Huitema, C., (1995) Routing in the Internet. Prentice Hall 1257 PTR, 1995. 1259 [HK99] T. Henderson and R. Katz, (1999) Transport Protocols for 1260 Internet-Compatible Satellite Networks. IEEE Journal on Selected 1261 Areas in Communications, Vol. 17, No. 2, pp. 345-359, February 1262 1999. 1264 [HMTG01] C. Hollot, V. Misra, D. Towsley, and W. Gong, On Designing 1265 Improved Controllers for AQM Routers Supporting TCP Flows, IEEE 1266 Infocom, 2001. 1268 [HS03] R. Hsieh and A. Seneviratne. A Comparison of Mechanisms for 1269 Improving Mobile IP Handoff Latency for End-to-end TCP. 1270 MOBICOM, Sept. 2003. 1272 [IMD01] G. Iannaccone, M. May, and C. Diot, Aggregate Traffic 1273 Performance with Active Queue Management and Drop From Tail. 1274 SIGCOMM Comput. Commun. Rev., 31(3):4-13, 2001. 1276 [IMLGK03] H. Inamura, G. Montenegro, R. Ludwig, A. Gurtov and F. 1277 Khafizov. TCP over Second (2.5G) and Third (3G) Generation 1278 Wireless Networks. RFC 3484, IETF, February 2003. 1280 [JD02] H. Jiang and C. Dovrolis, Passive Estimation of TCP Round- 1281 trip Times, Computer Communication Review, 32(3), July 2002. 1283 [JEAS03] A. Jardosh, E. Belding-Royer, K. Almeroth, and S. Suri. 1284 Towards Realistic Mobility Models for Mobile Ad Hoc Networks. 1285 MOBICOM, Sept. 2003. 1287 [LC01] J. Liu and Mark Crovella, Using Loss Pairs to Discover 1288 Network Properties, ACM SIGCOMM Internet Measurement Workshop, 1289 2001. 1291 [LC05] X. Luo and R. K. C. Chang, Novel Approaches to End-to-end 1292 Packet Reordering Measurement, 2005. 1294 [LMJ96] Labovitz, C., Malan, G.R., and Jahanian, F., (1996) Internet 1295 Routing Instability. Proceedings of SIGCOMM 96. 1297 [MAF05] A. Medina, M. Allman, and A. Floyd. Measuring the Evolution 1298 of Transport Protocols in the Internet. Computer Communication 1299 Review, April 2005. 1301 [NISTNet] NIST Net, URL "http://snad.ncsl.nist.gov/itg/nistnet/". 1303 [NM01] J. Neale and A. Mohsen. Impact of CF-DAMA on TCP via 1304 Satellite Performance. Globecom, Nov. 2001. 1306 [P96] Paxson, V., (1996) End-to-end Routing Behavior in the 1307 Internet. Proceedings of SIGCOMM 96, pp. 25-38, August 1992. 1309 [P97] V. Paxson. End-to-end Routing Behavior in the Internet. 1310 IEEE/ACM Transactions on Networking, 5(5):60115, October 1997. 1312 [PJD04] R. Prasad, M. Jain, and C. Dovrolis, On the Effectiveness of 1313 Delay-Based Congestion Avoidance, PFLDnet 2004, February 2004. 1315 [QZK01] L. Qiu, Y. Zhang, and S. Keshav, Understanding the 1316 Performance of Many TCP Flows, Comput. Networks, 1317 37(3-4):277-306, 2001. 1319 [RSB01] R. Riedi, S. Sarvotham, and R. Varaniuk, Connection-level 1320 Analysis and Modeling of Network Traffic, SIGCOMM Internet 1321 Measurement Workshop, 2001. 1323 [RTW05] G. Raina, D. Towsley, and D. Wischik, Control Theory for 1324 Buffer Sizing, CCR, July 2005. 1326 [TG] Traffic Generators for Internet Traffic Web Page, URL 1327 "http://www.icir.org/models/trafficgenerators.html". 1329 [UA01] U. of Auckland, Auckland-vi trace data, June 2001. URL 1330 "http://wans.cs.waikato.ac.nz/wand/wits/auck/6/". 1332 [UW02] UW-Madison, Network Performance Statistics, October 2002. 1333 URL "http://wwwstats.net.wisc.edu/". 1335 [W01] B. Walke. Mobile Radio Networks, Networking and Protocols (2. 1336 Ed.). Wiley & Sons, 2001. 1338 [WM05] D. Wischik and N. McKeown, Buffer sizes for Core Routers, 1339 CCR, July 2005. URL 1340 "http://yuba.stanford.edu/~nickm/papers/BufferSizing.pdf". 1342 [WMWGB06] F. Wang, Z. M. Mao, J. Wang, L. Gao and R. Bush. A 1343 Measurement Study on the Impact of Routing Events on End-to-End 1344 Internet Path Performance, SIGCOMM, 2006. 1346 [ZM04] X. Zhou and P. Van Mieghem, Reordering of IP Packets in 1347 Internet, PAM 2004. 1349 [ZSC91] L. Zhang, S. Shenker, and D.D. Clark, Observations and 1350 Dynamics of a Congestion Control Algorithm: the Effects of Two- 1351 way Traffic, SIGCOMM 1991. 1353 [22] 1355 [23] 1357 [24] 1359 Editors' Addresses 1361 Sally Floyd 1362 ICSI Center for Internet Research 1363 1947 Center Street, Suite 600 1364 Berkeley, CA 94704 1365 USA 1367 Eddie Kohler 1368 4531C Boelter Hall 1369 UCLA Computer Science Department 1370 Los Angeles, CA 90095 1371 USA 1373 Full Copyright Statement 1375 Copyright (C) The IETF Trust 2006. This document is subject to the 1376 rights, licenses and restrictions contained in BCP 78, and except as 1377 set forth therein, the authors retain all their rights. 1379 This document and the information contained herein are provided on 1380 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 1381 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE 1382 IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL 1383 WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY 1384 WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE 1385 ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS 1386 FOR A PARTICULAR PURPOSE. 1388 Intellectual Property 1390 The IETF takes no position regarding the validity or scope of any 1391 Intellectual Property Rights or other rights that might be claimed 1392 to pertain to the implementation or use of the technology described 1393 in this document or the extent to which any license under such 1394 rights might or might not be available; nor does it represent that 1395 it has made any independent effort to identify any such rights. 1396 Information on the procedures with respect to rights in RFC 1397 documents can be found in BCP 78 and BCP 79. 1399 Copies of IPR disclosures made to the IETF Secretariat and any 1400 assurances of licenses to be made available, or the result of an 1401 attempt made to obtain a general license or permission for the use 1402 of such proprietary rights by implementers or users of this 1403 specification can be obtained from the IETF on-line IPR repository 1404 at http://www.ietf.org/ipr. 1406 The IETF invites any interested party to bring to its attention any 1407 copyrights, patents or patent applications, or other proprietary 1408 rights that may cover technology that may be required to implement 1409 this standard. Please address the information to the IETF at ietf- 1410 ipr@ietf.org.