idnits 2.17.1 draft-irtf-tmrg-metrics-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 13. -- Found old boilerplate from RFC 3978, Section 5.5 on line 787. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 798. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 805. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 811. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (24 June 2006) is 6508 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'MTZ04' is mentioned on line 722, but not defined == Outdated reference: A later version (-07) exists of draft-ietf-dccp-tfrc-voip-02 -- Obsolete informational reference (is this intentional?): RFC 3448 (Obsoleted by RFC 5348) Summary: 3 errors (**), 0 flaws (~~), 5 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force Sally Floyd 2 INTERNET-DRAFT Editor 3 draft-irtf-tmrg-metrics-03.txt 24 June 2006 4 Expires: December 2006 6 Metrics for the Evaluation of Congestion Control Mechanisms 8 Status of this Memo 10 By submitting this Internet-Draft, each author represents that any 11 applicable patent or other IPR claims of which he or she is aware 12 have been or will be disclosed, and any of which he or she becomes 13 aware will be disclosed, in accordance with Section 6 of BCP 79. 15 Internet-Drafts are working documents of the Internet Engineering 16 Task Force (IETF), its areas, and its working groups. Note that 17 other groups may also distribute working documents as Internet- 18 Drafts. 20 Internet-Drafts are draft documents valid for a maximum of six 21 months and may be updated, replaced, or obsoleted by other documents 22 at any time. It is inappropriate to use Internet-Drafts as 23 reference material or to cite them other than as "work in progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt. 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 This Internet-Draft will expire on December 2006. 33 Abstract 35 This document discusses the metrics to be considered in an 36 evaluation of new or modified congestion control mechanisms for the 37 Internet. This includes metrics for the evaluation of new transport 38 protocols, of proposed modifications to TCP, of application-level 39 congestion control, and of Active Queue Management (AQM) mechanisms 40 in the router. This document is intended to be the first in a 41 series of documents aimed at improving the models that we use in the 42 evaluation of transport protocols. 44 This document is a product of the Transport Modeling Research Group 45 (TRMG), and has received detailed feedback from several members of 46 the Research Group (RG). We are not aware of any controversies 47 regarding the content of this document. 49 Table of Contents 51 1. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 52 2. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 5 53 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 54 3.1. Throughput, Delay, and Loss Rates. . . . . . . . . . . . 7 55 3.1.1. Throughput. . . . . . . . . . . . . . . . . . . . . 7 56 3.1.2. Delay . . . . . . . . . . . . . . . . . . . . . . . 8 57 3.1.3. Packet Loss Rates . . . . . . . . . . . . . . . . . 8 58 3.2. Response Times and Minimizing Oscillations . . . . . . . 9 59 3.2.1. Response to Changes . . . . . . . . . . . . . . . . 9 60 3.2.2. Minimizing Oscillations . . . . . . . . . . . . . . 10 61 3.3. Fairness and Convergence . . . . . . . . . . . . . . . . 10 62 3.4. Robustness for Challenging Environments. . . . . . . . . 12 63 3.5. Robustness to Failures and to Misbehaving 64 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 65 3.6. Deployability. . . . . . . . . . . . . . . . . . . . . . 13 66 3.7. Metrics for Specific Types of Transport. . . . . . . . . 14 67 3.8. User-Based Metrics . . . . . . . . . . . . . . . . . . . 14 68 4. Metrics in the IP Performance Metrics (IPPM) Working 69 Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 70 5. Comments on Methodology . . . . . . . . . . . . . . . . . . . 14 71 6. Security Considerations . . . . . . . . . . . . . . . . . . . 15 72 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 73 8. Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . 15 74 Informative References . . . . . . . . . . . . . . . . . . . . . 15 75 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 18 76 Full Copyright Statement . . . . . . . . . . . . . . . . . . . . 18 77 Intellectual Property. . . . . . . . . . . . . . . . . . . . . . 18 79 1. Conventions 81 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 82 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 83 document are to be interpreted as described in [RFC 2119]. TO BE 84 DELETED BY THE RFC EDITOR UPON PUBLICATION: 86 Changes from draft-irtf-tmrg-metrics-02.txt: * Added a few sentences 87 to the Abstract about the 88 status of the document. 90 Changes from draft-irtf-tmrg-metrics-01.txt: * Added a discussion 91 about the metrics in IPPM. 93 Changes from draft-irtf-tmrg-metrics-01c.txt: 95 * Added to the discussion of network-based, flow-based, 96 and user-based metrics, based on email from Dado Colussi, 97 Sean Moore, Damon Wischik, Dah Ming Chiu, and others. 99 * Changed "packet drop rate" to "packet loss rate". 100 Suggestion from Nelson Fonseca. 102 * Added a discussion of the Colussi et al. paper on a new 103 definition of fairness. 105 * Added a discussion of the Chiu and Tan paper on redefining 106 fairness for inelastic traffic. 108 Changes from draft-irtf-tmrg-metrics-01b.txt: 110 * Added a discussion of goodput vs. throughput. 111 Suggestion from Nelson Fonseca. 113 Changes from draft-irtf-tmrg-metrics-01a.txt: 115 * Added to the discussion of packet drop rate metrics. 116 Suggestions from Janardhan Iyengar, Sean Moore, 117 Armando Caro, and Nelson Fonseca. 119 * Added a sentence about throughput used as a metric for 120 transfer times for very short flows. 121 Response to email from Seam Moore. 123 Changes from draft-irtf-tmrg-metrics-00.txt: 125 * Added a list of relevant congestion control mechanisms to 126 the abstract. Suggestion from Sean Moore. 128 * Added to the Introduction. Suggestion from Dado Colussi. 130 * Added a sentence about jitter to the discussion of minimizing 131 oscillations. Suggestion from Wesley Eddy. 133 * Added a note about convergence between existing flows after 134 a change in bandwidth. Suggestion from Wesley Eddy. 136 * Added to the section on deployability. Suggestion from 137 Wesley Eddy. 139 Changes from draft-floyd-transport-metrics-00.txt: 141 * Added metrics for: 142 - robustness in challenging environments, 143 - deployability, 144 - robustness to failures and to misbehaving users 146 * Added a discussion of fairness and packet size. 148 2. Introduction 150 As a step towards improving our methodologies for evaluating 151 congestion control mechanisms, in this document we discuss some of 152 the metrics to be considered. We also consider the relationship 153 between metrics, e.g., the well-known tradeoff between throughput 154 and delay. 156 We consider metrics for aggregate traffic (taking into account the 157 effect of flows on competing traffic in the network) as well as the 158 heterogeneous goals of different applications or transport protocols 159 (e.g., of high throughput for bulk data transfer, and of low delay 160 for interactive voice or video). Different transport protocols or 161 AQM mechanisms might have goals of optimizing different sets of 162 metrics, with one transport protocol optimized for per-flow 163 throughput and another optimized for robustness over wireless links, 164 and with different degrees of attention to fairness with competing 165 traffic. We hope this document will be used as a step in evaluating 166 proposed congestion control mechanisms for a wide range of metrics, 167 noting that Mechanism X is good at optimizing Metric A, but pays the 168 price with poor performance for Metric B. The goal would be to have 169 a broad view of both the strengths and weaknesses of newly-proposed 170 congestion control mechanisms. 172 Subsequent documents are planned to present sets of simulation and 173 testbed scenarios for the evaluation of transport protocols and of 174 congestion control mechanisms, based on the best current practice of 175 the research community. These are not intended to be complete or 176 final benchmark test suites, but simply to be one step of many to be 177 used by researchers in evaluating congestion control mechanisms. 178 Subsequent documents are also planned on the methodologies in using 179 these sets of scenarios. 181 This is work from the Transport Modeling Research Group (TMRG) in 182 the IRTF (Internet Research Task Force). 184 3. Metrics 186 The metrics that we discuss are the following: 188 o Throughput; 190 o Delay; 192 o Packet loss rates; 194 o Response to sudden changes or to transient events; 196 o Minimizing oscillations in throughput or in delay; 198 o Fairness and convergence times; 200 o Robustness for challenging environments; 202 o Robustness to failures and to misbehaving users; 204 o Deployability; 206 o Metrics for specific types of transport. 208 o User-based metrics. 210 We consider each of these below. Many of the metrics have network- 211 based, flow-based, and user-based interpretations. For example, 212 network-based metrics can consider aggregate bandwidth and aggregate 213 drop rates, flow-based metrics can consider end-to-end transfer 214 times for file transfers or end-to-end delay and packet drop rates 215 for interactive traffic, and user-based metrics can consider user 216 wait time or user satisfaction with the multimedia experience. Our 217 main goal in this document is to explain the set of metrics that can 218 be relevant, and not to legislate on the more appropriate 219 methodology for using each general metric. 221 For some of the metrics, such as fairness between flows, there is 222 not a clear agreement in the network community about the desired 223 goals. In these cases, the document attempts to present the range 224 of approaches. 226 3.1. Throughput, Delay, and Loss Rates 228 Because of the clear tradeoffs between throughput, delay, and loss 229 rates, it can be useful to consider the three metrics together. 231 An alternative would be to consider a separate metric such as power, 232 defined in this context as throughput over delay, that combines 233 throughput and delay. However, we do not propose in this document a 234 clear target in terms of the tradeoffs between throughput and delay; 235 we are simply proposing that the evaluation of transport protocols 236 include an exploration of the competing metrics. 238 3.1.1. Throughput 240 Throughput can be measured as a router-based metric of aggregate 241 link throughput, as a flow-based metric of per-connection transfer 242 times, and as user-based metrics of utility functions or user wait 243 times. It is a clear goal of most congestion control mechanisms to 244 maximize throughput, subject to application demand and to the 245 constraints of the other metrics. 247 Throughput is sometimes distinguished from goodput, where throughput 248 is the link or flow throughput in bytes per second, and goodput, 249 also measured in bytes per second, is the subset of throughput 250 consisting of useful traffic. That is, `goodput' excludes duplicate 251 packets, packets that will be dropped downstream, packet fragments 252 or ATM cells that are dropped at the receiver because they can't be 253 re-assembled into complete packets, and the like. 255 We note that maximizing throughput is of concern in a wide range of 256 environments, from highly-congested networks to under-utilized ones, 257 and from long-lived flows to very short ones. As an example, 258 throughput has been used as one of the metrics for evaluating Quick- 259 Start, a proposal to allow flows to start-up faster than slow-start, 260 where throughput has been evaluated in terms of the transfer times 261 for connections with a range of transfer sizes [QuickStart]. 263 In some contexts, it might be sufficient to consider the aggregate 264 throughput or the mean per-flow throughput, while in other contexts 265 it might be necessary to consider the distribution of per-flow 266 throughput. Some researchers evaluate transport protocols in terms 267 of maximizing the aggregate user utility, where a user's utility is 268 generally defined as a function of the user's throughput [KMT98]. 270 Individual applications can have application-specific needs in terms 271 of throughput. For example, real-time video traffic can have highly 272 variable bandwidth demands; VoIP traffic is sensitive to the amount 273 of bandwidth received immediately after idle periods. Thus, user 274 metrics for throughput can be more complex than simply the per- 275 connection transfer time. 277 3.1.2. Delay 279 Like throughput, delay can be measured as a router-based metric of 280 queueing delay over time, or as a flow-based metric in terms of per- 281 packet transfer times. For reliable transfer, the per-packet 282 transfer time includes the possible delay of retransmitting a lost 283 packet. 285 Users of bulk data transfer applications might care about per-packet 286 transfer times only in so far as they affect the per-connection 287 transfer time. On the other end of the spectrum, for users of 288 streaming media, per-packet delay can be a significant concern. 289 Note that in some cases the average delay might not capture the 290 metric of interest to the users; for example, some users might care 291 about the worst-case delay, or about the tail of the delay 292 distribution. 294 3.1.3. Packet Loss Rates 296 Packet loss rates can be measured as a network-based or as a flow- 297 based metric. 299 When evaluating the effect of packet losses or ECN marks (Explicit 300 Congestion Notification, RFC 3168) on the performance of a 301 congestion control mechanism for an individual flow, researchers 302 often use both the packet loss/mark rate for that connection, and 303 the congestion event rate (also called the loss event rate), where a 304 congestion event or loss event consists of one or more lost or 305 marked packets in one round-trip time [RFC 3448]. 307 Some users might care about packet loss rates only in so far as they 308 affect per-connection transfer times, while other users might care 309 about packet loss rates directly. RFC 3611, RTP Control Protocol 310 Extended Reports, describes a VoIP performance-reporting standard 311 called RTCP XR, which includes a set of burst metrics. In RFC 3611, 312 a burst is defined as the maximal sequence starting and ending with 313 a lost packet, and not including a sequence of Gmin or more packets 314 that are not lost [RFC 3611]. The burst metrics in RFC 3611 consist 315 of the burst density (the fraction of packets in bursts), gap 316 density (the fraction of packets in the gaps between bursts), burst 317 duration (the mean duration of bursts in seconds), and gap duration 318 (the mean duration of gaps in seconds). 320 In some cases it is useful to distinguish between packets dropped at 321 routers due to congestion, and packets lost in the network due to 322 corruption. 324 One network-related reason to avoid high steady-state packet loss 325 rates is to avoid congestion collapse in environments containing 326 paths with multiple congested links. In such environments, high 327 packet loss rates could result in congested links wasting scarce 328 bandwidth by carrying packets that will only be dropped downstream, 329 before being delivered to the receiver. 331 3.2. Response Times and Minimizing Oscillations 333 In this section we consider response times and oscillations 334 together, as there are well-known tradeoffs between improving 335 response times and minimizing oscillations. In addition, the 336 scenarios that illustrate the dangers of poor response times are 337 often quite different from the scenarios that illustrate the dangers 338 of unnecessary oscillations. 340 3.2.1. Response to Changes 342 One of the key concerns in the design of congestion control 343 mechanisms has been the response times to sudden congestion in the 344 network. On the one hand, congestion control mechanisms should 345 respond reasonably promptly to sudden congestion from routing or 346 bandwidth changes, or from a burst of competing traffic. At the 347 same time, congestion control mechanisms should not respond too 348 severely to transient changes, e.g., to a sudden increase in delay 349 that will dissipate in less than the connection's round-trip time. 351 Evaluating the response to sudden or transient changes can be of 352 particular concern for slowly-responding congestion control 353 mechanisms such as equation-based congestion control [RFC 3448], and 354 for AIMD (Additive Increase Multiplicative Decrease) or related 355 mechanisms using parameters that make them more slowly-responding 356 that TCP [BB01, BBFS01]. 358 In addition to the responsiveness and smoothness of aggregate 359 traffic, one can consider the tradeoffs between responsiveness, 360 smoothness, and aggressiveness for an individual connection [FHP00]. 361 In this case smoothness can be defined by the largest reduction in 362 the sending rate in one round-trip time, in a deterministic 363 environment with a packet drop exactly every 1/p packets. The 364 responsiveness is defined as the number of round-trip times of 365 sustained congested required for the sender to halve the sending 366 rate, and the aggressiveness is defined as the maximum increase in 367 the sending rate in one round-trip time, in packets per second, in 368 the absence of congestion. 370 3.2.2. Minimizing Oscillations 372 One goal is that of stability, in terms of minimizing oscillations 373 of queueing delay or of throughput. Scenarios illustrating 374 oscillations are often dominated by long-lived connections, perhaps 375 with a small number of changes in the level of congestion. 377 Minimizing oscillations in queueing delay or throughput has related 378 per-flow metrics of minimizing jitter in round-trip times and loss 379 rates. 381 An orthogonal goal for some congestion control mechanisms, e.g., for 382 equation-based congestion control, is to minimize the oscillations 383 in the sending rate for an individual connection, given an 384 environment with a fixed, steady-state packet drop rate. (As is 385 well known, TCP congestion control is characterized by a pronounced 386 oscillation in the sending rate, with the sender halving the sending 387 rate in response to congestion.) One metric for the level of 388 oscillations is the smoothness metric given above. 390 3.3. Fairness and Convergence 392 Another set of metrics are those of fairness and of convergence 393 times. Fairness can be considered between flows of the same 394 protocol, and between flows using different protocols (e.g., 395 fairness between TCP and a new transport protocol). 397 There are a number of different fairness measures. These include 398 max-min fairness [HG86], proportional fairness [KMT98, K01], the 399 fairness index proposed in [JCH84], and the product measure, a 400 variant of network power [BJ81]. 402 Max-min fairness: In order to satisfy the max-min fairness criteria, 403 the smallest throughput rate must be as large as possible. Given 404 this condition, the next-smallest throughput rate must be as large 405 as possible, and so on. Thus, the max-min fairness gives absolute 406 priority to the smallest flows. 408 Epsilon-fairness: A metric related to max-min fairness is epsilon- 409 fairness, where a rate allocation is defined as epsilon-fair if 411 min_i x_i / max_i x_i >= 1 - epsilon. 413 where x_i is the resource allocation to the i-th user. Epsilon- 414 fairness measures the worst-case ratio between any two throughput 415 rates [ZKL04]. 417 Proportional fairness: In contrast, an allocation x is defined as 418 proportionally fair if for any other feasible allocation x*, the 419 aggregate of proportional changes is zero or negative: 421 sum_i (x*_i - x_i)/x_i <= 0. 423 "This criterion favours smaller flows, but less emphatically than 424 max-min fairness" [K01]. 426 Jain's fairness index: The fairness index in [JCH84] is 428 (( sum_i x_i )^2) / (n * sum_i (x_i)^2 ) , 430 where there are n users. This fairness index ranges from 0 to 1, 431 and is maximum when all users receive the same allocation. This 432 index is k/n when k users equally share the resource, and the other 433 n-k users receive zero allocation. 435 The product measure: The product measure 437 product_i x_i , 439 the product of the throughput of the individual connections, is also 440 used as a measure of fairness. For our purposes, let x_i be the 441 throughput for the i-th connection. (In other contexts x_i is taken 442 as the power of the i-th connection, and the product measure is 443 referred to as network power.) The product measure is particularly 444 sensitive to segregation; the product measure is zero if any 445 connection receives zero throughput. In [MS91] it is shown that for 446 a network with many connections and one shared gateway, the product 447 measure is maximized when all connections receive the same 448 throughput. 450 In [CRM05], Colussi et al. propose a new definition of fairness, 451 that "a set of TCP fair flows do not cause more congestion than a 452 set of TCP flows would cause", where congestion is defined in terms 453 of queueing delay, queueing delay variation, the congestion event 454 rate [e.g., loss event rate], and the packet loss rate. 456 Chiu and Tan in [CT06] argue for redefining the notion of fairness 457 when studying traffic controls for inelastic traffic, proposing that 458 inelastic flows adopt other traffic controls such as admission 459 control. 461 Fairness and the number of congested links: Some of these fairness 462 metrics are discussed in more detail in [F91]. We note that there 463 is not a clear consensus for the fairness goals, in particular for 464 fairness between flows that traverse different numbers of congested 465 links [F91]. 467 Fairness and round-trip times: One goal cited in a number of new 468 transport protocols has been that of fairness between flows with 469 different round-trip times [KHR02, XHR04]. We note that there is not 470 a consensus in the networking community about the desirability of 471 this goal, or about the implications and interactions between this 472 goal and other metrics [FJ92] (Section 3.3). 474 Fairness and packet size: One fairness issue is that of the relative 475 fairness for flows with different packet sizes. Many file transfer 476 applications will use the maximum packet size possible; in 477 contrast, low-bandwidth VoIP flows are likely to send small packets, 478 sending a new packet every 10 to 40 ms., to limit delay. Should a 479 small-packet VoIP connection receive the same sending rate in bytes 480 per second as a large-packet TCP connection in the same environment, 481 or should it receive the same sending rate in *packets* per second? 482 This fairness issue has been discussed in more detail in [FK04], 483 with [FK05] also describing the ways that packet size can effect the 484 packet drop rate experienced by a flow. 486 Convergence times: Convergence times concern the time for 487 convergence to fairness between an existing flow and a newly- 488 starting one, and are a special concern for environments with high- 489 bandwidth flows. Convergence times also concern the time for 490 convergence to fairness between two existing flows after a sudden 491 change such as a change in link capacity on a wireless link. As 492 with fairness, convergence times can matter both between flows of 493 the same protocol, and between flows using different protocols 494 [SLFK03]. One metric used for convergence times is the delta-fair 495 convergence time, defined as the time taken for two flows with the 496 same round-trip time to go from shares of 100/101-th and 1/101-th of 497 the link bandwidth, to having close to fair sharing with shares of 498 (1+delta)/2 and (1-delta)/2 of the link bandwidth [BBFS01]. A 499 similar metric for convergence times measures the convergence time 500 as the number of round-trip times for two flows to reach epsilon- 501 fairness, when starting from a maximally-unfair state [ZKL04]. ' 503 3.4. Robustness for Challenging Environments 505 While congestion control mechanisms are generally evaluated first 506 over environments with static routing in a network of two-way point- 507 to-point links, some environments bring up more challenging problems 508 (e.g., corrupted packets, variable bandwidth, mobility) as well as 509 new metrics to be considered (e.g., energy consumption). 511 Robustness for challenging environments: Robustness needs to be 512 explored for paths with reordering, corruption, variable bandwidth, 513 asymmetric routing, router configuration changes, mobility, and the 514 like. In general, Internet architecture has valued robustness over 515 efficiency, e.g., when there are tradeoffs between robustness and 516 the throughput, delay, and fairness metrics described above. 518 Energy consumption: In mobile environments the energy consumption 519 for the mobile end-node can be a key metric that is affected by the 520 transport protocol [TM02]. 522 Goodput: For wireless networks, goodput can be a useful metric, 523 where goodput is defined as the fraction of useful data from all of 524 the data delivered. High goodput indicates an efficient use of the 525 radio spectrum and lower interference to other users [GF04]. 527 3.5. Robustness to Failures and to Misbehaving Users 529 One goal is for congestion control mechanisms to be robust to 530 misbehaving users, such as receivers that `lie' to data senders 531 about the congestion experienced along the path or otherwise attempt 532 to bypass the congestion control mechanisms of the sender [SCWA99]. 533 Another goal is for congestion control mechanisms to be as robust as 534 possible to failures, such as failures of routers in using explicit 535 feedback to end-nodes or failures of end-nodes to follow the 536 prescribed protocols, 538 3.6. Deployability 540 One metric for congestion control mechanisms is their deployability 541 in the current Internet. Metrics related to deployability include 542 the ease of failure diagnosis, and the overhead in terms of packet 543 header size or added complexity at end-nodes or routers. 545 One key aspect of deployability concerns the range of deployment 546 needed for a new congestion control mechanism. Consider the 547 following possible deployment requirements: 549 * Only at the sender (e.g., NewReno in TCP); 550 * Only at the receiver (e.g., delayed acknowledgements in TCP); 551 * Both the sender and receiver (e.g., SACK TCP); 552 * At a single router (e.g., RED); 553 * All of the routers along the end-to-end path; 554 * Both end nodes and all routers along the path (e.g., XCP). 556 Another deployability issue concerns the complexity of the code. 557 Roughly how many lines of code are required to implement the 558 mechanism in software? Is floating point math required? We note 559 that we don't suggest these questions as ways to reduce the 560 deployability metric to a single number; we suggest them as issues 561 that could be considered in evaluating the deployability of a 562 proposed congestion control mechanism. 564 3.7. Metrics for Specific Types of Transport 566 In some cases modified metrics are needed for evaluting transport 567 protocols intended for QoS-enabled environments or for below-best- 568 effort traffic [VKD02, KK03]. 570 3.8. User-Based Metrics 572 An alternate approach that has been proposed for the evaluation of 573 congestion control mechanisms would be to evaluate in terms of user 574 metrics such as user satisfaction, or in terms of application- 575 specific utility functions. Such an approach would require the 576 definition of a range of user metrics or of application-specific 577 utility functions for the range of applications under consideration 578 (e.g., FTP, HTTP, VoIP). 580 4. Metrics in the IP Performance Metrics (IPPM) Working Group 582 The IPPM Working Group [IPPM] was established to define performance 583 metrics to be used by network operators, end users, or independent 584 testing groups. The metrics include metrics for connectivity, delay 585 and loss, delay variation, loss patterns, packet reordering, bulk 586 transfer capacity, and link capacity. The IPPM documents give 587 concrete, well-defined metrics, along with a methodology for 588 measuring the metric. The metrics discussed in this document have a 589 different purpose from the IPPM metrics; in this document we are 590 discussing metrics as used in analysis, simulations, and experiments 591 for the evaluation of congestion control mechanisms. Further, we 592 are discussing these metrics in a general sense, rather than looking 593 for specific concrete definitions for each metric. However, there 594 are many cases where the metric definitions from IPPM could be 595 useful, particularly for specific issues of how to measure these 596 metrics in simulations or in testbeds. 598 5. Comments on Methodology 600 The types of scenarios that are used to test specific metrics, and 601 the range of parameters that it is useful to consider, will be 602 discussed in separate documents, e.g., along with specific scenarios 603 for use in evaluating congestion control mechanisms. 605 We note that it can be important to evaluate metrics over a wide 606 range of environments, with a range of link bandwidths, congestion 607 levels, and levels of statistical multiplexing. It is also 608 important to evaluate congestion control mechanisms in a range of 609 scenarios, including typical ranges of connection sizes and round- 610 trip times [FK02]. It is also useful to compare metrics for new or 611 modified transport protocols with those of the current standards for 612 TCP. 614 Li et al. in "Experimental Evaluation of TCP Protocols for High- 615 Speed Networks" [LLS05] focus on the performance of TCP in high- 616 speed networks, and consider metrics for aggregate throughput, loss 617 rates, fairness (including fairness between flows with different 618 round-trip times), response times (including convergence times), and 619 incremental deployment. 621 More general references on methodology include [J91]. Papers that 622 discuss the range of metrics for evaluating congestion control 623 include [MTZ04]. 625 6. Security Considerations 627 There are no security considerations in this document. 629 7. IANA Considerations 631 There are no IANA considerations in this document. 633 8. Acknowledgements 635 Thanks to Armando Caro, Dah Ming Chiu, Dado Colussi, Wesley Eddy, 636 Nelson Fonseca, Janardhan Iyengar, Doug Leith, Saverio Mascolo, Sean 637 Moore, and Damon Wischik, and members of the Transport Modeling 638 Research Group for feedback and contributions. 640 Informative References 642 [BB01] D. Bansal and H. Balakrishnan, Binomial Congestion Control 643 Algorithms, IEEE Infocom, April 2001. 645 [BBFS01] D. Bansal, H. Balakrishnan, S. Floyd, and S. Shenker, 646 Dynamic Behavior of Slowly-Responsive Congestion Control 647 Algorithms, SIGCOMM 2001. 649 [BJ81] K. Bharath-Kumar and J. Jeffrey, A New Approach to 650 Performance-Oriented Flow Control, IEEE Transactions on 651 Communications, Vol.COM-29 N.4, April 1981. 653 [CRM05] A New Approach to TCP-Fairness, Report C-2005-49, University 654 of Helsinki, Finland, 2005. 656 [CT06] D. Chiu and A. Tam, Redefining Fairness in the Study of TCP- 657 friendly Traffic Controls, Technical Report, 2006. 659 [F91] S. Floyd, Connections with Multiple Congested Gateways in 660 Packet-Switched Networks Part 1: One-way Traffic, Computer 661 Communication Review, Vol.21, No.5, October 1991, p. 30-47. 663 [FK05] S. Floyd and E. Kohler, TFRC for Voice: the VoIP Variant, 664 draft-ietf-dccp-tfrc-voip-02.txt, internet draft, work in 665 progress, July 2005. 667 [FHP00] S. Floyd, M. Handley, and J. Padhye, A Comparison of 668 Equation-Based and AIMD Congestion Control, May 2000. URL 669 "http://www.icir.org/tfrc/". 671 [FJ92] S. Floyd and V. Jacobson, On Traffic Phase Effects in Packet- 672 Switched Gateways, Internetworking: Research and Experience, V.3 673 N.3, September 1992, p.115-156. 675 [FK04] S. Floyd and J. Kempf, IAB Concerns Regarding Congestion 676 Control for Voice Traffic in the Internet, RFC 3714, March 2004. 678 [FK02] S. Floyd and E. Kohler, Internet Research Needs Better 679 Models, Hotnets-I. October 2002. 681 [GF04] A. Gurtov and S. Floyd, Modeling Wireless Links for Transport 682 Protocols, ACM CCR, 34(2):85-96, April 2004. 684 [HG86] E. Hahne and R. Gallager, Round Robin Scheduling for Fair 685 Flow Control in Data Communications Networks, IEEE International 686 Conference on Communications, June 1986. 688 [IPPM] IP Performance Metrics (IPPM) Working Group, URL 689 "http://www.ietf.org/html.charters/ippm-charter.html". 691 [J91] R. Jain, The Art of Computer Systems Performance Analysis: 692 Techniques for Experimental Design, Measurement, Simulation, and 693 Modeling, John Wiley & Sons, 1991. 695 [JCH84] R. Jain, D.M. Chiu, and W. Hawe, A Quantitative Measure of 696 Fairness and Discrimination for Resource Allocation in Shared 697 Systems, DEC TR-301, Littleton, MA: Digital Equipment 698 Corporation, 1984. 700 [K01] F. Kelly, Mathematical Modelling of the Internet, "Mathematics 701 Unlimited - 2001 and Beyond" (Editors B. Engquist and W. 702 Schmid), Springer-Verlag, Berlin, pp. 685-702, 2001. 704 [KHR02] D. Katabi, M. Handley, and C. Rohrs, Congestion Control for 705 High Bandwidth-Delay Product Networks, ACM Sigcomm, 2002. 707 [KK03] A. Kuzmanovic and E. W. Knightly, TCP-LP: A Distributed 708 Algorithm for Low Priority Data Transfer, IEEE INFOCOM 2003, 709 April 2003. 711 [KMT98] F. Kelly, A. Maulloo and D. Tan, Rate Control in 712 Communication Networks: Shadow Prices, Proportional Fairness and 713 Stability. Journal of the Operational Research Society 49, pp. 714 237-252, 1998. 716 [LLS05] Y-T. Li, D. Leith, and R. Shorten, Experimental Evaluation 717 of TCP Protocols for High-Speed Networks, Hamilton Institute, 718 2005. URL "http://www.hamilton.ie/net/eval/results-HI2005.pdf". 720 [MS91] D. Mitra and J. Seery, Dynamic Adaptive Windows for High 721 Speed Data Networks with Multiple Paths and Propagation Delays, 722 INFOCOM '91, pp 39-48. [MTZ04] L. Mamatas, V. Tsaoussidis, and 723 C. Zhang, Approaches to Congestion Control in Packet Networks, 724 2004. 726 [QuickStart] Quick-Start Web Page, URL 727 "http://www.icir.org/floyd/quickstart.html". 729 [RFC 2119] S. Bradner. Key Words For Use in RFCs to Indicate 730 Requirement Levels. RFC 2119. 732 BIBREF RFC3168 "RFC 3168" K Ramakrishnan, S. Floyd, and D. Black, 733 The Addition of Explicit Congestion Notification (ECN) to IP, 734 RFC 3168, September 2001. 736 [RFC 3448] M. Handley, S. Floyd, J. Padhye, and J. Widmer, TCP 737 Friendly Rate Control (TFRC): Protocol Specification, RFC 3448, 738 Proposed Standard, January 2003. 740 [RFC 3611] T. Friedman, R. Caceres, and A. Clark, RTP Control 741 Protocol Extended Reports (RTCP XR), RFC 3611, November 2003. 743 [SLFK03] R.N. Shorten, D.J. Leith, J. Foy, and R. Kilduff, Analysis 744 and Design of Congestion Control in Synchronised Communication 745 Networks. Proc. 12th Yale Workshop on Adaptive & Learning 746 Systems, May 2003. 748 [SCWA99] TCP Congestion Control with a Misbehaving Receiver, ACM 749 Computer Communications Review, October 1999. 751 [TM02] V. Tsaoussidis and I. Matta, Open Issues of TCP for Mobile 752 Computing, Journal of Wireless Communications and Mobile 753 Computing: Special Issue on Reliable Transport Protocols for 754 Mobile Computing, February 2002. 756 [VKD02] A. Venkataramani, R. Kokku, and M. Dahlin, TCP Nice: A 757 Mechanism for Background Transfers, Fifth USENIX Symposium on 758 Operating System Design and Implementation (OSDI), 2002. 760 [XHR04] L. Xu, K. Harfoush, and I. Rhee, Binary Increase Congestion 761 Control for Fast, Long Distance Networks, Infocom 2004. 763 [ZKL04] Y. Zhang, S.-R. Kang, and D. Loguinov, Delayed Stability and 764 Performance of Distributed Congestion Control, ACM SIGCOMM, 765 August 2004. 767 Authors' Addresses 769 Sally Floyd 770 ICSI Center for Internet Research 771 1947 Center Street, Suite 600 772 Berkeley, CA 94704 773 USA 775 Full Copyright Statement 777 Copyright (C) The Internet Society 2006. This document is subject 778 to the rights, licenses and restrictions contained in BCP 78, and 779 except as set forth therein, the authors retain all their rights. 781 This document and the information contained herein are provided on 782 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 783 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE 784 INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR 785 IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 786 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 787 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 789 Intellectual Property 791 The IETF takes no position regarding the validity or scope of any 792 Intellectual Property Rights or other rights that might be claimed 793 to pertain to the implementation or use of the technology described 794 in this document or the extent to which any license under such 795 rights might or might not be available; nor does it represent that 796 it has made any independent effort to identify any such rights. 797 Information on the procedures with respect to rights in RFC 798 documents can be found in BCP 78 and BCP 79. 800 Copies of IPR disclosures made to the IETF Secretariat and any 801 assurances of licenses to be made available, or the result of an 802 attempt made to obtain a general license or permission for the use 803 of such proprietary rights by implementers or users of this 804 specification can be obtained from the IETF on-line IPR repository 805 at http://www.ietf.org/ipr. 807 The IETF invites any interested party to bring to its attention any 808 copyrights, patents or patent applications, or other proprietary 809 rights that may cover technology that may be required to implement 810 this standard. Please address the information to the IETF at ietf- 811 ipr@ietf.org.