idnits 2.17.1 draft-irtf-tmrg-metrics-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 13. -- Found old boilerplate from RFC 3978, Section 5.5 on line 779. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 790. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 797. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 803. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (1 June 2006) is 6533 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'MTZ04' is mentioned on line 714, but not defined == Outdated reference: A later version (-07) exists of draft-ietf-dccp-tfrc-voip-02 -- Obsolete informational reference (is this intentional?): RFC 3448 (Obsoleted by RFC 5348) Summary: 3 errors (**), 0 flaws (~~), 5 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force Sally Floyd 2 INTERNET-DRAFT Editor 3 draft-irtf-tmrg-metrics-02.txt 1 June 2006 4 Expires: December 2006 6 Metrics for the Evaluation of Congestion Control Mechanisms 8 Status of this Memo 10 By submitting this Internet-Draft, each author represents that any 11 applicable patent or other IPR claims of which he or she is aware 12 have been or will be disclosed, and any of which he or she becomes 13 aware will be disclosed, in accordance with Section 6 of BCP 79. 15 Internet-Drafts are working documents of the Internet Engineering 16 Task Force (IETF), its areas, and its working groups. Note that 17 other groups may also distribute working documents as Internet- 18 Drafts. 20 Internet-Drafts are draft documents valid for a maximum of six 21 months and may be updated, replaced, or obsoleted by other documents 22 at any time. It is inappropriate to use Internet-Drafts as 23 reference material or to cite them other than as "work in progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt. 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 This Internet-Draft will expire on December 2006. 33 Abstract 35 This document discusses the metrics to be considered in an 36 evaluation of new or modified congestion control mechanisms for the 37 Internet. This includes metrics for the evaluation of new transport 38 protocols, of proposed modifications to TCP, of application-level 39 congestion control, and of Active Queue Management (AQM) mechanisms 40 in the router. This document is intended to be the first in a 41 series of documents aimed at improving the models that we use in the 42 evaluation of transport protocols. 44 Table of Contents 46 1. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 47 2. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 5 48 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 49 3.1. Throughput, Delay, and Loss Rates. . . . . . . . . . . . 7 50 3.1.1. Throughput. . . . . . . . . . . . . . . . . . . . . 7 51 3.1.2. Delay . . . . . . . . . . . . . . . . . . . . . . . 8 52 3.1.3. Packet Loss Rates . . . . . . . . . . . . . . . . . 8 53 3.2. Response Times and Minimizing Oscillations . . . . . . . 9 54 3.2.1. Response to Changes . . . . . . . . . . . . . . . . 9 55 3.2.2. Minimizing Oscillations . . . . . . . . . . . . . . 10 56 3.3. Fairness and Convergence . . . . . . . . . . . . . . . . 10 57 3.4. Robustness for Challenging Environments. . . . . . . . . 12 58 3.5. Robustness to Failures and to Misbehaving 59 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 60 3.6. Deployability. . . . . . . . . . . . . . . . . . . . . . 13 61 3.7. Metrics for Specific Types of Transport. . . . . . . . . 14 62 3.8. User-Based Metrics . . . . . . . . . . . . . . . . . . . 14 63 4. Metrics in the IP Performance Metrics (IPPM) Working 64 Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 65 5. Comments on Methodology . . . . . . . . . . . . . . . . . . . 14 66 6. Security Considerations . . . . . . . . . . . . . . . . . . . 15 67 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 68 8. Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . 15 69 Informative References . . . . . . . . . . . . . . . . . . . . . 15 70 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 18 71 Full Copyright Statement . . . . . . . . . . . . . . . . . . . . 18 72 Intellectual Property. . . . . . . . . . . . . . . . . . . . . . 18 74 1. Conventions 76 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 77 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 78 document are to be interpreted as described in [RFC 2119]. 80 TO BE DELETED BY THE RFC EDITOR UPON PUBLICATION: 82 Changes from draft-irtf-tmrg-metrics-01.txt: * Added a discussion 83 about the metrics in IPPM. 85 Changes from draft-irtf-tmrg-metrics-01c.txt: 87 * Added to the discussion of network-based, flow-based, 88 and user-based metrics, based on email from Dado Colussi, 89 Sean Moore, Damon Wischik, Dah Ming Chiu, and others. 91 * Changed "packet drop rate" to "packet loss rate". 92 Suggestion from Nelson Fonseca. 94 * Added a discussion of the Colussi et al. paper on a new 95 definition of fairness. 97 * Added a discussion of the Chiu and Tan paper on redefining 98 fairness for inelastic traffic. 100 Changes from draft-irtf-tmrg-metrics-01b.txt: 102 * Added a discussion of goodput vs. throughput. 103 Suggestion from Nelson Fonseca. 105 Changes from draft-irtf-tmrg-metrics-01a.txt: 107 * Added to the discussion of packet drop rate metrics. 108 Suggestions from Janardhan Iyengar, Sean Moore, 109 Armando Caro, and Nelson Fonseca. 111 * Added a sentence about throughput used as a metric for 112 transfer times for very short flows. 113 Response to email from Seam Moore. 115 Changes from draft-irtf-tmrg-metrics-00.txt: 117 * Added a list of relevant congestion control mechanisms to 118 the abstract. Suggestion from Sean Moore. 120 * Added to the Introduction. Suggestion from Dado Colussi. 122 * Added a sentence about jitter to the discussion of minimizing 123 oscillations. Suggestion from Wesley Eddy. 125 * Added a note about convergence between existing flows after 126 a change in bandwidth. Suggestion from Wesley Eddy. 128 * Added to the section on deployability. Suggestion from 129 Wesley Eddy. 131 Changes from draft-floyd-transport-metrics-00.txt: 133 * Added metrics for: 134 - robustness in challenging environments, 135 - deployability, 136 - robustness to failures and to misbehaving users 138 * Added a discussion of fairness and packet size. 140 2. Introduction 142 As a step towards improving our methodologies for evaluating 143 congestion control mechanisms, in this document we discuss some of 144 the metrics to be considered. We also consider the relationship 145 between metrics, e.g., the well-known tradeoff between throughput 146 and delay. 148 We consider metrics for aggregate traffic (taking into account the 149 effect of flows on competing traffic in the network) as well as the 150 heterogeneous goals of different applications or transport protocols 151 (e.g., of high throughput for bulk data transfer, and of low delay 152 for interactive voice or video). Different transport protocols or 153 AQM mechanisms might have goals of optimizing different sets of 154 metrics, with one transport protocol optimized for per-flow 155 throughput and another optimized for robustness over wireless links, 156 and with different degrees of attention to fairness with competing 157 traffic. We hope this document will be used as a step in evaluating 158 proposed congestion control mechanisms for a wide range of metrics, 159 noting that Mechanism X is good at optimizing Metric A, but pays the 160 price with poor performance for Metric B. The goal would be to have 161 a broad view of both the strengths and weaknesses of newly-proposed 162 congestion control mechanisms. 164 Subsequent documents are planned to present sets of simulation and 165 testbed scenarios for the evaluation of transport protocols and of 166 congestion control mechanisms, based on the best current practice of 167 the research community. These are not intended to be complete or 168 final benchmark test suites, but simply to be one step of many to be 169 used by researchers in evaluating congestion control mechanisms. 170 Subsequent documents are also planned on the methodologies in using 171 these sets of scenarios. 173 This is work from the Transport Modeling Research Group (TMRG) in 174 the IRTF (Internet Research Task Force). 176 3. Metrics 178 The metrics that we discuss are the following: 180 o Throughput; 182 o Delay; 184 o Packet loss rates; 186 o Response to sudden changes or to transient events; 188 o Minimizing oscillations in throughput or in delay; 190 o Fairness and convergence times; 192 o Robustness for challenging environments; 194 o Robustness to failures and to misbehaving users; 196 o Deployability; 198 o Metrics for specific types of transport. 200 o User-based metrics. 202 We consider each of these below. Many of the metrics have network- 203 based, flow-based, and user-based interpretations. For example, 204 network-based metrics can consider aggregate bandwidth and aggregate 205 drop rates, flow-based metrics can consider end-to-end transfer 206 times for file transfers or end-to-end delay and packet drop rates 207 for interactive traffic, and user-based metrics can consider user 208 wait time or user satisfaction with the multimedia experience. Our 209 main goal in this document is to explain the set of metrics that can 210 be relevant, and not to legislate on the more appropriate 211 methodology for using each general metric. 213 For some of the metrics, such as fairness between flows, there is 214 not a clear agreement in the network community about the desired 215 goals. In these cases, the document attempts to present the range 216 of approaches. 218 3.1. Throughput, Delay, and Loss Rates 220 Because of the clear tradeoffs between throughput, delay, and loss 221 rates, it can be useful to consider the three metrics together. 223 An alternative would be to consider a separate metric such as power, 224 defined in this context as throughput over delay, that combines 225 throughput and delay. However, we do not propose in this document a 226 clear target in terms of the tradeoffs between throughput and delay; 227 we are simply proposing that the evaluation of transport protocols 228 include an exploration of the competing metrics. 230 3.1.1. Throughput 232 Throughput can be measured as a router-based metric of aggregate 233 link throughput, as a flow-based metric of per-connection transfer 234 times, and as user-based metrics of utility functions or user wait 235 times. It is a clear goal of most congestion control mechanisms to 236 maximize throughput, subject to application demand and to the 237 constraints of the other metrics. 239 Throughput is sometimes distinguished from goodput, where throughput 240 is the link or flow throughput in bytes per second, and goodput, 241 also measured in bytes per second, is the subset of throughput 242 consisting of useful traffic. That is, `goodput' excludes duplicate 243 packets, packets that will be dropped downstream, packet fragments 244 or ATM cells that are dropped at the receiver because they can't be 245 re-assembled into complete packets, and the like. 247 We note that maximizing throughput is of concern in a wide range of 248 environments, from highly-congested networks to under-utilized ones, 249 and from long-lived flows to very short ones. As an example, 250 throughput has been used as one of the metrics for evaluating Quick- 251 Start, a proposal to allow flows to start-up faster than slow-start, 252 where throughput has been evaluated in terms of the transfer times 253 for connections with a range of transfer sizes [QuickStart]. 255 In some contexts, it might be sufficient to consider the aggregate 256 throughput or the mean per-flow throughput, while in other contexts 257 it might be necessary to consider the distribution of per-flow 258 throughput. Some researchers evaluate transport protocols in terms 259 of maximizing the aggregate user utility, where a user's utility is 260 generally defined as a function of the user's throughput [KMT98]. 262 Individual applications can have application-specific needs in terms 263 of throughput. For example, real-time video traffic can have highly 264 variable bandwidth demands; VoIP traffic is sensitive to the amount 265 of bandwidth received immediately after idle periods. Thus, user 266 metrics for throughput can be more complex than simply the per- 267 connection transfer time. 269 3.1.2. Delay 271 Like throughput, delay can be measured as a router-based metric of 272 queueing delay over time, or as a flow-based metric in terms of per- 273 packet transfer times. For reliable transfer, the per-packet 274 transfer time includes the possible delay of retransmitting a lost 275 packet. 277 Users of bulk data transfer applications might care about per-packet 278 transfer times only in so far as they affect the per-connection 279 transfer time. On the other end of the spectrum, for users of 280 streaming media, per-packet delay can be a significant concern. 281 Note that in some cases the average delay might not capture the 282 metric of interest to the users; for example, some users might care 283 about the worst-case delay, or about the tail of the delay 284 distribution. 286 3.1.3. Packet Loss Rates 288 Packet loss rates can be measured as a network-based or as a flow- 289 based metric. 291 When evaluating the effect of packet losses or ECN marks (Explicit 292 Congestion Notification, RFC 3168) on the performance of a 293 congestion control mechanism for an individual flow, researchers 294 often use both the packet loss/mark rate for that connection, and 295 the congestion event rate (also called the loss event rate), where a 296 congestion event or loss event consists of one or more lost or 297 marked packets in one round-trip time [RFC 3448]. 299 Some users might care about packet loss rates only in so far as they 300 affect per-connection transfer times, while other users might care 301 about packet loss rates directly. RFC 3611, RTP Control Protocol 302 Extended Reports, describes a VoIP performance-reporting standard 303 called RTCP XR, which includes a set of burst metrics. In RFC 3611, 304 a burst is defined as the maximal sequence starting and ending with 305 a lost packet, and not including a sequence of Gmin or more packets 306 that are not lost [RFC 3611]. The burst metrics in RFC 3611 consist 307 of the burst density (the fraction of packets in bursts), gap 308 density (the fraction of packets in the gaps between bursts), burst 309 duration (the mean duration of bursts in seconds), and gap duration 310 (the mean duration of gaps in seconds). 312 In some cases it is useful to distinguish between packets dropped at 313 routers due to congestion, and packets lost in the network due to 314 corruption. 316 One network-related reason to avoid high steady-state packet loss 317 rates is to avoid congestion collapse in environments containing 318 paths with multiple congested links. In such environments, high 319 packet loss rates could result in congested links wasting scarce 320 bandwidth by carrying packets that will only be dropped downstream, 321 before being delivered to the receiver. 323 3.2. Response Times and Minimizing Oscillations 325 In this section we consider response times and oscillations 326 together, as there are well-known tradeoffs between improving 327 response times and minimizing oscillations. In addition, the 328 scenarios that illustrate the dangers of poor response times are 329 often quite different from the scenarios that illustrate the dangers 330 of unnecessary oscillations. 332 3.2.1. Response to Changes 334 One of the key concerns in the design of congestion control 335 mechanisms has been the response times to sudden congestion in the 336 network. On the one hand, congestion control mechanisms should 337 respond reasonably promptly to sudden congestion from routing or 338 bandwidth changes, or from a burst of competing traffic. At the 339 same time, congestion control mechanisms should not respond too 340 severely to transient changes, e.g., to a sudden increase in delay 341 that will dissipate in less than the connection's round-trip time. 343 Evaluating the response to sudden or transient changes can be of 344 particular concern for slowly-responding congestion control 345 mechanisms such as equation-based congestion control [RFC 3448], and 346 for AIMD (Additive Increase Multiplicative Decrease) or related 347 mechanisms using parameters that make them more slowly-responding 348 that TCP [BB01, BBFS01]. 350 In addition to the responsiveness and smoothness of aggregate 351 traffic, one can consider the tradeoffs between responsiveness, 352 smoothness, and aggressiveness for an individual connection [FHP00]. 353 In this case smoothness can be defined by the largest reduction in 354 the sending rate in one round-trip time, in a deterministic 355 environment with a packet drop exactly every 1/p packets. The 356 responsiveness is defined as the number of round-trip times of 357 sustained congested required for the sender to halve the sending 358 rate, and the aggressiveness is defined as the maximum increase in 359 the sending rate in one round-trip time, in packets per second, in 360 the absence of congestion. 362 3.2.2. Minimizing Oscillations 364 One goal is that of stability, in terms of minimizing oscillations 365 of queueing delay or of throughput. Scenarios illustrating 366 oscillations are often dominated by long-lived connections, perhaps 367 with a small number of changes in the level of congestion. 369 Minimizing oscillations in queueing delay or throughput has related 370 per-flow metrics of minimizing jitter in round-trip times and loss 371 rates. 373 An orthogonal goal for some congestion control mechanisms, e.g., for 374 equation-based congestion control, is to minimize the oscillations 375 in the sending rate for an individual connection, given an 376 environment with a fixed, steady-state packet drop rate. (As is 377 well known, TCP congestion control is characterized by a pronounced 378 oscillation in the sending rate, with the sender halving the sending 379 rate in response to congestion.) One metric for the level of 380 oscillations is the smoothness metric given above. 382 3.3. Fairness and Convergence 384 Another set of metrics are those of fairness and of convergence 385 times. Fairness can be considered between flows of the same 386 protocol, and between flows using different protocols (e.g., 387 fairness between TCP and a new transport protocol). 389 There are a number of different fairness measures. These include 390 max-min fairness [HG86], proportional fairness [KMT98, K01], the 391 fairness index proposed in [JCH84], and the product measure, a 392 variant of network power [BJ81]. 394 Max-min fairness: In order to satisfy the max-min fairness criteria, 395 the smallest throughput rate must be as large as possible. Given 396 this condition, the next-smallest throughput rate must be as large 397 as possible, and so on. Thus, the max-min fairness gives absolute 398 priority to the smallest flows. 400 Epsilon-fairness: A metric related to max-min fairness is epsilon- 401 fairness, where a rate allocation is defined as epsilon-fair if 403 min_i x_i / max_i x_i >= 1 - epsilon. 405 where x_i is the resource allocation to the i-th user. Epsilon- 406 fairness measures the worst-case ratio between any two throughput 407 rates [ZKL04]. 409 Proportional fairness: In contrast, an allocation x is defined as 410 proportionally fair if for any other feasible allocation x*, the 411 aggregate of proportional changes is zero or negative: 413 sum_i (x*_i - x_i)/x_i <= 0. 415 "This criterion favours smaller flows, but less emphatically than 416 max-min fairness" [K01]. 418 Jain's fairness index: The fairness index in [JCH84] is 420 (( sum_i x_i )^2) / (n * sum_i (x_i)^2 ) , 422 where there are n users. This fairness index ranges from 0 to 1, 423 and is maximum when all users receive the same allocation. This 424 index is k/n when k users equally share the resource, and the other 425 n-k users receive zero allocation. 427 The product measure: The product measure 429 product_i x_i , 431 the product of the throughput of the individual connections, is also 432 used as a measure of fairness. For our purposes, let x_i be the 433 throughput for the i-th connection. (In other contexts x_i is taken 434 as the power of the i-th connection, and the product measure is 435 referred to as network power.) The product measure is particularly 436 sensitive to segregation; the product measure is zero if any 437 connection receives zero throughput. In [MS91] it is shown that for 438 a network with many connections and one shared gateway, the product 439 measure is maximized when all connections receive the same 440 throughput. 442 In [CRM05], Colussi et al. propose a new definition of fairness, 443 that "a set of TCP fair flows do not cause more congestion than a 444 set of TCP flows would cause", where congestion is defined in terms 445 of queueing delay, queueing delay variation, the congestion event 446 rate [e.g., loss event rate], and the packet loss rate. 448 Chiu and Tan in [CT06] argue for redefining the notion of fairness 449 when studying traffic controls for inelastic traffic, proposing that 450 inelastic flows adopt other traffic controls such as admission 451 control. 453 Fairness and the number of congested links: Some of these fairness 454 metrics are discussed in more detail in [F91]. We note that there 455 is not a clear consensus for the fairness goals, in particular for 456 fairness between flows that traverse different numbers of congested 457 links [F91]. 459 Fairness and round-trip times: One goal cited in a number of new 460 transport protocols has been that of fairness between flows with 461 different round-trip times [KHR02, XHR04]. We note that there is not 462 a consensus in the networking community about the desirability of 463 this goal, or about the implications and interactions between this 464 goal and other metrics [FJ92] (Section 3.3). 466 Fairness and packet size: One fairness issue is that of the relative 467 fairness for flows with different packet sizes. Many file transfer 468 applications will use the maximum packet size possible; in 469 contrast, low-bandwidth VoIP flows are likely to send small packets, 470 sending a new packet every 10 to 40 ms., to limit delay. Should a 471 small-packet VoIP connection receive the same sending rate in bytes 472 per second as a large-packet TCP connection in the same environment, 473 or should it receive the same sending rate in *packets* per second? 474 This fairness issue has been discussed in more detail in [FK04], 475 with [FK05] also describing the ways that packet size can effect the 476 packet drop rate experienced by a flow. 478 Convergence times: Convergence times concern the time for 479 convergence to fairness between an existing flow and a newly- 480 starting one, and are a special concern for environments with high- 481 bandwidth flows. Convergence times also concern the time for 482 convergence to fairness between two existing flows after a sudden 483 change such as a change in link capacity on a wireless link. As 484 with fairness, convergence times can matter both between flows of 485 the same protocol, and between flows using different protocols 486 [SLFK03]. One metric used for convergence times is the delta-fair 487 convergence time, defined as the time taken for two flows with the 488 same round-trip time to go from shares of 100/101-th and 1/101-th of 489 the link bandwidth, to having close to fair sharing with shares of 490 (1+delta)/2 and (1-delta)/2 of the link bandwidth [BBFS01]. A 491 similar metric for convergence times measures the convergence time 492 as the number of round-trip times for two flows to reach epsilon- 493 fairness, when starting from a maximally-unfair state [ZKL04]. ' 495 3.4. Robustness for Challenging Environments 497 While congestion control mechanisms are generally evaluated first 498 over environments with static routing in a network of two-way point- 499 to-point links, some environments bring up more challenging problems 500 (e.g., corrupted packets, variable bandwidth, mobility) as well as 501 new metrics to be considered (e.g., energy consumption). 503 Robustness for challenging environments: Robustness needs to be 504 explored for paths with reordering, corruption, variable bandwidth, 505 asymmetric routing, router configuration changes, mobility, and the 506 like. In general, Internet architecture has valued robustness over 507 efficiency, e.g., when there are tradeoffs between robustness and 508 the throughput, delay, and fairness metrics described above. 510 Energy consumption: In mobile environments the energy consumption 511 for the mobile end-node can be a key metric that is affected by the 512 transport protocol [TM02]. 514 Goodput: For wireless networks, goodput can be a useful metric, 515 where goodput is defined as the fraction of useful data from all of 516 the data delivered. High goodput indicates an efficient use of the 517 radio spectrum and lower interference to other users [GF04]. 519 3.5. Robustness to Failures and to Misbehaving Users 521 One goal is for congestion control mechanisms to be robust to 522 misbehaving users, such as receivers that `lie' to data senders 523 about the congestion experienced along the path or otherwise attempt 524 to bypass the congestion control mechanisms of the sender [SCWA99]. 525 Another goal is for congestion control mechanisms to be as robust as 526 possible to failures, such as failures of routers in using explicit 527 feedback to end-nodes or failures of end-nodes to follow the 528 prescribed protocols, 530 3.6. Deployability 532 One metric for congestion control mechanisms is their deployability 533 in the current Internet. Metrics related to deployability include 534 the ease of failure diagnosis, and the overhead in terms of packet 535 header size or added complexity at end-nodes or routers. 537 One key aspect of deployability concerns the range of deployment 538 needed for a new congestion control mechanism. Consider the 539 following possible deployment requirements: 541 * Only at the sender (e.g., NewReno in TCP); 542 * Only at the receiver (e.g., delayed acknowledgements in TCP); 543 * Both the sender and receiver (e.g., SACK TCP); 544 * At a single router (e.g., RED); 545 * All of the routers along the end-to-end path; 546 * Both end nodes and all routers along the path (e.g., XCP). 548 Another deployability issue concerns the complexity of the code. 549 Roughly how many lines of code are required to implement the 550 mechanism in software? Is floating point math required? We note 551 that we don't suggest these questions as ways to reduce the 552 deployability metric to a single number; we suggest them as issues 553 that could be considered in evaluating the deployability of a 554 proposed congestion control mechanism. 556 3.7. Metrics for Specific Types of Transport 558 In some cases modified metrics are needed for evaluting transport 559 protocols intended for QoS-enabled environments or for below-best- 560 effort traffic [VKD02, KK03]. 562 3.8. User-Based Metrics 564 An alternate approach that has been proposed for the evaluation of 565 congestion control mechanisms would be to evaluate in terms of user 566 metrics such as user satisfaction, or in terms of application- 567 specific utility functions. Such an approach would require the 568 definition of a range of user metrics or of application-specific 569 utility functions for the range of applications under consideration 570 (e.g., FTP, HTTP, VoIP). 572 4. Metrics in the IP Performance Metrics (IPPM) Working Group 574 The IPPM Working Group [IPPM] was established to define performance 575 metrics to be used by network operators, end users, or independent 576 testing groups. The metrics include metrics for connectivity, delay 577 and loss, delay variation, loss patterns, packet reordering, bulk 578 transfer capacity, and link capacity. The IPPM documents give 579 concrete, well-defined metrics, along with a methodology for 580 measuring the metric. The metrics discussed in this document have a 581 different purpose from the IPPM metrics; in this document we are 582 discussing metrics as used in analysis, simulations, and experiments 583 for the evaluation of congestion control mechanisms. Further, we 584 are discussing these metrics in a general sense, rather than looking 585 for specific concrete definitions for each metric. However, there 586 are many cases where the metric definitions from IPPM could be 587 useful, particularly for specific issues of how to measure these 588 metrics in simulations or in testbeds. 590 5. Comments on Methodology 592 The types of scenarios that are used to test specific metrics, and 593 the range of parameters that it is useful to consider, will be 594 discussed in separate documents, e.g., along with specific scenarios 595 for use in evaluating congestion control mechanisms. 597 We note that it can be important to evaluate metrics over a wide 598 range of environments, with a range of link bandwidths, congestion 599 levels, and levels of statistical multiplexing. It is also 600 important to evaluate congestion control mechanisms in a range of 601 scenarios, including typical ranges of connection sizes and round- 602 trip times [FK02]. It is also useful to compare metrics for new or 603 modified transport protocols with those of the current standards for 604 TCP. 606 Li et al. in "Experimental Evaluation of TCP Protocols for High- 607 Speed Networks" [LLS05] focus on the performance of TCP in high- 608 speed networks, and consider metrics for aggregate throughput, loss 609 rates, fairness (including fairness between flows with different 610 round-trip times), response times (including convergence times), and 611 incremental deployment. 613 More general references on methodology include [J91]. Papers that 614 discuss the range of metrics for evaluating congestion control 615 include [MTZ04]. 617 6. Security Considerations 619 There are no security considerations in this document. 621 7. IANA Considerations 623 There are no IANA considerations in this document. 625 8. Acknowledgements 627 Thanks to Armando Caro, Dah Ming Chiu, Dado Colussi, Wesley Eddy, 628 Nelson Fonseca, Janardhan Iyengar, Doug Leith, Saverio Mascolo, Sean 629 Moore, and Damon Wischik, and members of the Transport Modeling 630 Research Group for feedback and contributions. 632 Informative References 634 [BB01] D. Bansal and H. Balakrishnan, Binomial Congestion Control 635 Algorithms, IEEE Infocom, April 2001. 637 [BBFS01] D. Bansal, H. Balakrishnan, S. Floyd, and S. Shenker, 638 Dynamic Behavior of Slowly-Responsive Congestion Control 639 Algorithms, SIGCOMM 2001. 641 [BJ81] K. Bharath-Kumar and J. Jeffrey, A New Approach to 642 Performance-Oriented Flow Control, IEEE Transactions on 643 Communications, Vol.COM-29 N.4, April 1981. 645 [CRM05] A New Approach to TCP-Fairness, Report C-2005-49, University 646 of Helsinki, Finland, 2005. 648 [CT06] D. Chiu and A. Tam, Redefining Fairness in the Study of TCP- 649 friendly Traffic Controls, Technical Report, 2006. 651 [F91] S. Floyd, Connections with Multiple Congested Gateways in 652 Packet-Switched Networks Part 1: One-way Traffic, Computer 653 Communication Review, Vol.21, No.5, October 1991, p. 30-47. 655 [FK05] S. Floyd and E. Kohler, TFRC for Voice: the VoIP Variant, 656 draft-ietf-dccp-tfrc-voip-02.txt, internet draft, work in 657 progress, July 2005. 659 [FHP00] S. Floyd, M. Handley, and J. Padhye, A Comparison of 660 Equation-Based and AIMD Congestion Control, May 2000. URL 661 "http://www.icir.org/tfrc/". 663 [FJ92] S. Floyd and V. Jacobson, On Traffic Phase Effects in Packet- 664 Switched Gateways, Internetworking: Research and Experience, V.3 665 N.3, September 1992, p.115-156. 667 [FK04] S. Floyd and J. Kempf, IAB Concerns Regarding Congestion 668 Control for Voice Traffic in the Internet, RFC 3714, March 2004. 670 [FK02] S. Floyd and E. Kohler, Internet Research Needs Better 671 Models, Hotnets-I. October 2002. 673 [GF04] A. Gurtov and S. Floyd, Modeling Wireless Links for Transport 674 Protocols, ACM CCR, 34(2):85-96, April 2004. 676 [HG86] E. Hahne and R. Gallager, Round Robin Scheduling for Fair 677 Flow Control in Data Communications Networks, IEEE International 678 Conference on Communications, June 1986. 680 [IPPM] IP Performance Metrics (IPPM) Working Group, URL 681 "http://www.ietf.org/html.charters/ippm-charter.html". 683 [J91] R. Jain, The Art of Computer Systems Performance Analysis: 684 Techniques for Experimental Design, Measurement, Simulation, and 685 Modeling, John Wiley & Sons, 1991. 687 [JCH84] R. Jain, D.M. Chiu, and W. Hawe, A Quantitative Measure of 688 Fairness and Discrimination for Resource Allocation in Shared 689 Systems, DEC TR-301, Littleton, MA: Digital Equipment 690 Corporation, 1984. 692 [K01] F. Kelly, Mathematical Modelling of the Internet, "Mathematics 693 Unlimited - 2001 and Beyond" (Editors B. Engquist and W. 694 Schmid), Springer-Verlag, Berlin, pp. 685-702, 2001. 696 [KHR02] D. Katabi, M. Handley, and C. Rohrs, Congestion Control for 697 High Bandwidth-Delay Product Networks, ACM Sigcomm, 2002. 699 [KK03] A. Kuzmanovic and E. W. Knightly, TCP-LP: A Distributed 700 Algorithm for Low Priority Data Transfer, IEEE INFOCOM 2003, 701 April 2003. 703 [KMT98] F. Kelly, A. Maulloo and D. Tan, Rate Control in 704 Communication Networks: Shadow Prices, Proportional Fairness and 705 Stability. Journal of the Operational Research Society 49, pp. 706 237-252, 1998. 708 [LLS05] Y-T. Li, D. Leith, and R. Shorten, Experimental Evaluation 709 of TCP Protocols for High-Speed Networks, Hamilton Institute, 710 2005. URL "http://www.hamilton.ie/net/eval/results-HI2005.pdf". 712 [MS91] D. Mitra and J. Seery, Dynamic Adaptive Windows for High 713 Speed Data Networks with Multiple Paths and Propagation Delays, 714 INFOCOM '91, pp 39-48. [MTZ04] L. Mamatas, V. Tsaoussidis, and 715 C. Zhang, Approaches to Congestion Control in Packet Networks, 716 2004. 718 [QuickStart] Quick-Start Web Page, URL 719 "http://www.icir.org/floyd/quickstart.html". 721 [RFC 2119] S. Bradner. Key Words For Use in RFCs to Indicate 722 Requirement Levels. RFC 2119. 724 BIBREF RFC3168 "RFC 3168" K Ramakrishnan, S. Floyd, and D. Black, 725 The Addition of Explicit Congestion Notification (ECN) to IP, 726 RFC 3168, September 2001. 728 [RFC 3448] M. Handley, S. Floyd, J. Padhye, and J. Widmer, TCP 729 Friendly Rate Control (TFRC): Protocol Specification, RFC 3448, 730 Proposed Standard, January 2003. 732 [RFC 3611] T. Friedman, R. Caceres, and A. Clark, RTP Control 733 Protocol Extended Reports (RTCP XR), RFC 3611, November 2003. 735 [SLFK03] R.N. Shorten, D.J. Leith, J. Foy, and R. Kilduff, Analysis 736 and Design of Congestion Control in Synchronised Communication 737 Networks. Proc. 12th Yale Workshop on Adaptive & Learning 738 Systems, May 2003. 740 [SCWA99] TCP Congestion Control with a Misbehaving Receiver, ACM 741 Computer Communications Review, October 1999. 743 [TM02] V. Tsaoussidis and I. Matta, Open Issues of TCP for Mobile 744 Computing, Journal of Wireless Communications and Mobile 745 Computing: Special Issue on Reliable Transport Protocols for 746 Mobile Computing, February 2002. 748 [VKD02] A. Venkataramani, R. Kokku, and M. Dahlin, TCP Nice: A 749 Mechanism for Background Transfers, Fifth USENIX Symposium on 750 Operating System Design and Implementation (OSDI), 2002. 752 [XHR04] L. Xu, K. Harfoush, and I. Rhee, Binary Increase Congestion 753 Control for Fast, Long Distance Networks, Infocom 2004. 755 [ZKL04] Y. Zhang, S.-R. Kang, and D. Loguinov, Delayed Stability and 756 Performance of Distributed Congestion Control, ACM SIGCOMM, 757 August 2004. 759 Authors' Addresses 761 Sally Floyd 762 ICSI Center for Internet Research 763 1947 Center Street, Suite 600 764 Berkeley, CA 94704 765 USA 767 Full Copyright Statement 769 Copyright (C) The Internet Society 2006. This document is subject 770 to the rights, licenses and restrictions contained in BCP 78, and 771 except as set forth therein, the authors retain all their rights. 773 This document and the information contained herein are provided on 774 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 775 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE 776 INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR 777 IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 778 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 779 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 781 Intellectual Property 783 The IETF takes no position regarding the validity or scope of any 784 Intellectual Property Rights or other rights that might be claimed 785 to pertain to the implementation or use of the technology described 786 in this document or the extent to which any license under such 787 rights might or might not be available; nor does it represent that 788 it has made any independent effort to identify any such rights. 789 Information on the procedures with respect to rights in RFC 790 documents can be found in BCP 78 and BCP 79. 792 Copies of IPR disclosures made to the IETF Secretariat and any 793 assurances of licenses to be made available, or the result of an 794 attempt made to obtain a general license or permission for the use 795 of such proprietary rights by implementers or users of this 796 specification can be obtained from the IETF on-line IPR repository 797 at http://www.ietf.org/ipr. 799 The IETF invites any interested party to bring to its attention any 800 copyrights, patents or patent applications, or other proprietary 801 rights that may cover technology that may be required to implement 802 this standard. Please address the information to the IETF at ietf- 803 ipr@ietf.org.