idnits 2.17.1 draft-irtf-tmrg-metrics-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 13. -- Found old boilerplate from RFC 3978, Section 5.5 on line 868. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 879. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 886. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 892. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (23 November 2006) is 6364 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'MTZ04' is mentioned on line 796, but not defined == Outdated reference: A later version (-07) exists of draft-ietf-dccp-tfrc-voip-02 -- Obsolete informational reference (is this intentional?): RFC 3448 (Obsoleted by RFC 5348) Summary: 3 errors (**), 0 flaws (~~), 5 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force Sally Floyd 2 INTERNET-DRAFT Editor 3 draft-irtf-tmrg-metrics-05.txt 23 November 2006 4 Expires: May 2007 6 Metrics for the Evaluation of Congestion Control Mechanisms 8 Status of this Memo 10 By submitting this Internet-Draft, each author represents that any 11 applicable patent or other IPR claims of which he or she is aware 12 have been or will be disclosed, and any of which he or she becomes 13 aware will be disclosed, in accordance with Section 6 of BCP 79. 15 Internet-Drafts are working documents of the Internet Engineering 16 Task Force (IETF), its areas, and its working groups. Note that 17 other groups may also distribute working documents as Internet- 18 Drafts. 20 Internet-Drafts are draft documents valid for a maximum of six 21 months and may be updated, replaced, or obsoleted by other documents 22 at any time. It is inappropriate to use Internet-Drafts as 23 reference material or to cite them other than as "work in progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt. 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 This Internet-Draft will expire on May 2007. 33 Abstract 35 This document discusses the metrics to be considered in an 36 evaluation of new or modified congestion control mechanisms for the 37 Internet. This includes metrics for the evaluation of new transport 38 protocols, of proposed modifications to TCP, of application-level 39 congestion control, and of Active Queue Management (AQM) mechanisms 40 in the router. This document is the first in a series of documents 41 aimed at improving the models that we use in the evaluation of 42 transport protocols. 44 This document is a product of the Transport Modeling Research Group 45 (TRMG), and has received detailed feedback from many members of the 46 Research Group (RG). As the document tries to make clear, there is 47 not necessarily a consensus within the research community, (or the 48 IETF community, the vendor community, the operations community, or 49 any other community) about the metrics that congestion control 50 mechanisms should be designed to optimize, in terms of tradeoffs 51 between throughput and delay, fairness between competing flows, and 52 the like. However, we believe that there is a clear consensus that 53 congestion control mechanisms should be evaluated in terms of 54 tradeoffs between a range of metrics, rather than in terms of 55 optimizing for a single metric. 57 Table of Contents 59 1. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 60 2. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 5 61 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 62 3.1. Throughput, Delay, and Loss Rates. . . . . . . . . . . . 7 63 3.1.1. Throughput. . . . . . . . . . . . . . . . . . . . . 7 64 3.1.2. Delay . . . . . . . . . . . . . . . . . . . . . . . 8 65 3.1.3. Packet Loss Rates . . . . . . . . . . . . . . . . . 8 66 3.2. Response Times and Minimizing Oscillations . . . . . . . 9 67 3.2.1. Response to Changes . . . . . . . . . . . . . . . . 9 68 3.2.2. Minimizing Oscillations . . . . . . . . . . . . . . 10 69 3.3. Fairness and Convergence . . . . . . . . . . . . . . . . 11 70 3.4. Robustness for Challenging Environments. . . . . . . . . 14 71 3.5. Robustness to Failures and to Misbehaving 72 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 73 3.6. Deployability. . . . . . . . . . . . . . . . . . . . . . 14 74 3.7. Metrics for Specific Types of Transport. . . . . . . . . 15 75 3.8. User-Based Metrics . . . . . . . . . . . . . . . . . . . 15 76 4. Metrics in the IP Performance Metrics (IPPM) Working 77 Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 78 5. Comments on Methodology . . . . . . . . . . . . . . . . . . . 16 79 6. Security Considerations . . . . . . . . . . . . . . . . . . . 16 80 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 81 8. Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . 16 82 Informative References . . . . . . . . . . . . . . . . . . . . . 16 83 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 84 Full Copyright Statement . . . . . . . . . . . . . . . . . . . . 20 85 Intellectual Property. . . . . . . . . . . . . . . . . . . . . . 20 87 1. Conventions 89 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 90 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 91 document are to be interpreted as described in [RFC 2119]. 93 TO BE DELETED BY THE RFC EDITOR UPON PUBLICATION: 95 Changes from draft-irtf-tmrg-metrics-04.txt: 97 * Added text to the abstract. 99 Changes from draft-irtf-tmrg-metrics-03.txt: 101 * Added a paragraph about sudden changes due to mobility 102 and the heterogeneity of wireless access types. 103 Suggestion from Andras Veres. 105 * Add covariance as one of the metrics for oscillations. 106 Suggestion from Saverio Mascolo, original text 107 contribution from Injong Rhee. 109 Changes from draft-irtf-tmrg-metrics-02.txt: 111 * Added a few sentences to the Abstract about the 112 status of the document. 114 Changes from draft-irtf-tmrg-metrics-01.txt: 116 * Added a discussion about the metrics in IPPM. 118 Changes from draft-irtf-tmrg-metrics-01c.txt: 120 * Added to the discussion of network-based, flow-based, 121 and user-based metrics, based on email from Dado Colussi, 122 Sean Moore, Damon Wischik, Dah Ming Chiu, and others. 124 * Changed "packet drop rate" to "packet loss rate". 125 Suggestion from Nelson Fonseca. 127 * Added a discussion of the Colussi et al. paper on a new 128 definition of fairness. 130 * Added a discussion of the Chiu and Tan paper on redefining 131 fairness for inelastic traffic. 133 Changes from draft-irtf-tmrg-metrics-01b.txt: 135 * Added a discussion of goodput vs. throughput. 136 Suggestion from Nelson Fonseca. 138 Changes from draft-irtf-tmrg-metrics-01a.txt: 140 * Added to the discussion of packet drop rate metrics. 141 Suggestions from Janardhan Iyengar, Sean Moore, 142 Armando Caro, and Nelson Fonseca. 144 * Added a sentence about throughput used as a metric for 145 transfer times for very short flows. 146 Response to email from Seam Moore. 148 Changes from draft-irtf-tmrg-metrics-00.txt: 150 * Added a list of relevant congestion control mechanisms to 151 the abstract. Suggestion from Sean Moore. 153 * Added to the Introduction. Suggestion from Dado Colussi. 155 * Added a sentence about jitter to the discussion of minimizing 156 oscillations. Suggestion from Wesley Eddy. 158 * Added a note about convergence between existing flows after 159 a change in bandwidth. Suggestion from Wesley Eddy. 161 * Added to the section on deployability. Suggestion from 162 Wesley Eddy. 164 Changes from draft-floyd-transport-metrics-00.txt: 166 * Added metrics for: 167 - robustness in challenging environments, 168 - deployability, 169 - robustness to failures and to misbehaving users 171 * Added a discussion of fairness and packet size. 173 2. Introduction 175 As a step towards improving our methodologies for evaluating 176 congestion control mechanisms, in this document we discuss some of 177 the metrics to be considered. We also consider the relationship 178 between metrics, e.g., the well-known tradeoff between throughput 179 and delay. 181 We consider metrics for aggregate traffic (taking into account the 182 effect of flows on competing traffic in the network) as well as the 183 heterogeneous goals of different applications or transport protocols 184 (e.g., of high throughput for bulk data transfer, and of low delay 185 for interactive voice or video). Different transport protocols or 186 AQM mechanisms might have goals of optimizing different sets of 187 metrics, with one transport protocol optimized for per-flow 188 throughput and another optimized for robustness over wireless links, 189 and with different degrees of attention to fairness with competing 190 traffic. We hope this document will be used as a step in evaluating 191 proposed congestion control mechanisms for a wide range of metrics, 192 noting that Mechanism X is good at optimizing Metric A, but pays the 193 price with poor performance for Metric B. The goal would be to have 194 a broad view of both the strengths and weaknesses of newly-proposed 195 congestion control mechanisms. 197 Subsequent documents are planned to present sets of simulation and 198 testbed scenarios for the evaluation of transport protocols and of 199 congestion control mechanisms, based on the best current practice of 200 the research community. These are not intended to be complete or 201 final benchmark test suites, but simply to be one step of many to be 202 used by researchers in evaluating congestion control mechanisms. 203 Subsequent documents are also planned on the methodologies in using 204 these sets of scenarios. 206 This is work from the Transport Modeling Research Group (TMRG) in 207 the IRTF (Internet Research Task Force). 209 3. Metrics 211 The metrics that we discuss are the following: 213 o Throughput; 215 o Delay; 217 o Packet loss rates; 219 o Response to sudden changes or to transient events; 221 o Minimizing oscillations in throughput or in delay; 223 o Fairness and convergence times; 225 o Robustness for challenging environments; 227 o Robustness to failures and to misbehaving users; 228 o Deployability; 230 o Metrics for specific types of transport. 232 o User-based metrics. 234 We consider each of these below. Many of the metrics have network- 235 based, flow-based, and user-based interpretations. For example, 236 network-based metrics can consider aggregate bandwidth and aggregate 237 drop rates, flow-based metrics can consider end-to-end transfer 238 times for file transfers or end-to-end delay and packet drop rates 239 for interactive traffic, and user-based metrics can consider user 240 wait time or user satisfaction with the multimedia experience. Our 241 main goal in this document is to explain the set of metrics that can 242 be relevant, and not to legislate on the more appropriate 243 methodology for using each general metric. 245 For some of the metrics, such as fairness between flows, there is 246 not a clear agreement in the network community about the desired 247 goals. In these cases, the document attempts to present the range 248 of approaches. 250 3.1. Throughput, Delay, and Loss Rates 252 Because of the clear tradeoffs between throughput, delay, and loss 253 rates, it can be useful to consider the three metrics together. 255 An alternative would be to consider a separate metric such as power, 256 defined in this context as throughput over delay, that combines 257 throughput and delay. However, we do not propose in this document a 258 clear target in terms of the tradeoffs between throughput and delay; 259 we are simply proposing that the evaluation of transport protocols 260 include an exploration of the competing metrics. 262 3.1.1. Throughput 264 Throughput can be measured as a router-based metric of aggregate 265 link throughput, as a flow-based metric of per-connection transfer 266 times, and as user-based metrics of utility functions or user wait 267 times. It is a clear goal of most congestion control mechanisms to 268 maximize throughput, subject to application demand and to the 269 constraints of the other metrics. 271 Throughput is sometimes distinguished from goodput, where throughput 272 is the link or flow throughput in bytes per second, and goodput, 273 also measured in bytes per second, is the subset of throughput 274 consisting of useful traffic. That is, `goodput' excludes duplicate 275 packets, packets that will be dropped downstream, packet fragments 276 or ATM cells that are dropped at the receiver because they can't be 277 re-assembled into complete packets, and the like. 279 We note that maximizing throughput is of concern in a wide range of 280 environments, from highly-congested networks to under-utilized ones, 281 and from long-lived flows to very short ones. As an example, 282 throughput has been used as one of the metrics for evaluating Quick- 283 Start, a proposal to allow flows to start-up faster than slow-start, 284 where throughput has been evaluated in terms of the transfer times 285 for connections with a range of transfer sizes [QuickStart]. 287 In some contexts, it might be sufficient to consider the aggregate 288 throughput or the mean per-flow throughput, while in other contexts 289 it might be necessary to consider the distribution of per-flow 290 throughput. Some researchers evaluate transport protocols in terms 291 of maximizing the aggregate user utility, where a user's utility is 292 generally defined as a function of the user's throughput [KMT98]. 294 Individual applications can have application-specific needs in terms 295 of throughput. For example, real-time video traffic can have highly 296 variable bandwidth demands; VoIP traffic is sensitive to the amount 297 of bandwidth received immediately after idle periods. Thus, user 298 metrics for throughput can be more complex than simply the per- 299 connection transfer time. 301 3.1.2. Delay 303 Like throughput, delay can be measured as a router-based metric of 304 queueing delay over time, or as a flow-based metric in terms of per- 305 packet transfer times. For reliable transfer, the per-packet 306 transfer time includes the possible delay of retransmitting a lost 307 packet. 309 Users of bulk data transfer applications might care about per-packet 310 transfer times only in so far as they affect the per-connection 311 transfer time. On the other end of the spectrum, for users of 312 streaming media, per-packet delay can be a significant concern. 313 Note that in some cases the average delay might not capture the 314 metric of interest to the users; for example, some users might care 315 about the worst-case delay, or about the tail of the delay 316 distribution. 318 3.1.3. Packet Loss Rates 320 Packet loss rates can be measured as a network-based or as a flow- 321 based metric. 323 When evaluating the effect of packet losses or ECN marks (Explicit 324 Congestion Notification, RFC 3168) on the performance of a 325 congestion control mechanism for an individual flow, researchers 326 often use both the packet loss/mark rate for that connection, and 327 the congestion event rate (also called the loss event rate), where a 328 congestion event or loss event consists of one or more lost or 329 marked packets in one round-trip time [RFC 3448]. 331 Some users might care about packet loss rates only in so far as they 332 affect per-connection transfer times, while other users might care 333 about packet loss rates directly. RFC 3611, RTP Control Protocol 334 Extended Reports, describes a VoIP performance-reporting standard 335 called RTCP XR, which includes a set of burst metrics. In RFC 3611, 336 a burst is defined as the maximal sequence starting and ending with 337 a lost packet, and not including a sequence of Gmin or more packets 338 that are not lost [RFC 3611]. The burst metrics in RFC 3611 consist 339 of the burst density (the fraction of packets in bursts), gap 340 density (the fraction of packets in the gaps between bursts), burst 341 duration (the mean duration of bursts in seconds), and gap duration 342 (the mean duration of gaps in seconds). 344 In some cases it is useful to distinguish between packets dropped at 345 routers due to congestion, and packets lost in the network due to 346 corruption. 348 One network-related reason to avoid high steady-state packet loss 349 rates is to avoid congestion collapse in environments containing 350 paths with multiple congested links. In such environments, high 351 packet loss rates could result in congested links wasting scarce 352 bandwidth by carrying packets that will only be dropped downstream, 353 before being delivered to the receiver. 355 3.2. Response Times and Minimizing Oscillations 357 In this section we consider response times and oscillations 358 together, as there are well-known tradeoffs between improving 359 response times and minimizing oscillations. In addition, the 360 scenarios that illustrate the dangers of poor response times are 361 often quite different from the scenarios that illustrate the dangers 362 of unnecessary oscillations. 364 3.2.1. Response to Changes 366 One of the key concerns in the design of congestion control 367 mechanisms has been the response times to sudden congestion in the 368 network. On the one hand, congestion control mechanisms should 369 respond reasonably promptly to sudden congestion from routing or 370 bandwidth changes, or from a burst of competing traffic. At the 371 same time, congestion control mechanisms should not respond too 372 severely to transient changes, e.g., to a sudden increase in delay 373 that will dissipate in less than the connection's round-trip time. 375 Congestion control mechanisms also have to contend with sudden 376 changes in the bandwidth-delay product due to mobility. Such 377 bandwith-delay product changes are expected to become more frequent 378 and to have greater impact than path changes today. As a result of 379 both mobility and of the heterogenity of wireless access types 380 (802.11b,a,g, WIMAX, WCDMA, HS-WCDMA, E-GPRS, Bluetooth, etc.), both 381 the bandwidth and the round-trip delay can change suddenly, 382 sometimes by several orders of magnitude. 384 Evaluating the response to sudden or transient changes can be of 385 particular concern for slowly-responding congestion control 386 mechanisms such as equation-based congestion control [RFC 3448], and 387 for AIMD (Additive Increase Multiplicative Decrease) or related 388 mechanisms using parameters that make them more slowly-responding 389 that TCP [BB01] [BBFS01]. 391 In addition to the responsiveness and smoothness of aggregate 392 traffic, one can consider the tradeoffs between responsiveness, 393 smoothness, and aggressiveness for an individual connection [FHP00]. 394 In this case smoothness can be defined by the largest reduction in 395 the sending rate in one round-trip time, in a deterministic 396 environment with a packet drop exactly every 1/p packets. The 397 responsiveness is defined as the number of round-trip times of 398 sustained congested required for the sender to halve the sending 399 rate, and the aggressiveness is defined as the maximum increase in 400 the sending rate in one round-trip time, in packets per second, in 401 the absence of congestion. 403 3.2.2. Minimizing Oscillations 405 One goal is that of stability, in terms of minimizing oscillations 406 of queueing delay or of throughput. In practice, stability is 407 frequently associated with rate fluctuations or variance. Rate 408 variations can result in fluctuations in router queue size and 409 therefore of queue overflows. These queue overflows can cause loss 410 synchronizations across co-existing flows and periodic under- 411 utilization of link capacity, both of which are considered to be 412 general signs of network instability. Thus, measuring the rate 413 variations of flows is often used to measure the stability of 414 transport protocols. To measure rate variations, [JWL04], [RX05], 415 and [FHPW00] use the coefficient of variation (CoV) of per-flow 416 transmission rates and [WCL05] suggests the use of standard 417 deviations of per-flow rates. Since rate variations are a function 418 of time scales, it makes sense to measure these rate variations over 419 various time scales. 421 Measuring per-flow rate variations, however, is only one aspect of 422 transport protocol stability. A realistic experiment setting always 423 involves multiple flows of the transport protocol being observed, 424 along with a significant amount of cross traffic, with rates varying 425 over time, on both the forward and reverse paths. As a congestion 426 control protocol must adapt its rate to the varying rates of 427 competing traffic, just measuring the per-flow statistics of a 428 subset of the traffic could be misleading because it measures the 429 rate fluctuations due in part to the adaptation to competing traffic 430 on the path. Thus, per-flow statistics are most meaningful if they 431 are accompanied by the statistics measured at the network level. As 432 a complementary metric to the per-flow statistics, [HKLRX05] uses 433 measurements of the rate variations of the aggregate flows observed 434 in bottleneck routers over various time scales. 436 Minimizing oscillations in queueing delay or throughput has related 437 per-flow metrics of minimizing jitter in round-trip times and loss 438 rates. 440 An orthogonal goal for some congestion control mechanisms, e.g., for 441 equation-based congestion control, is to minimize the oscillations 442 in the sending rate for an individual connection, given an 443 environment with a fixed, steady-state packet drop rate. (As is 444 well known, TCP congestion control is characterized by a pronounced 445 oscillation in the sending rate, with the sender halving the sending 446 rate in response to congestion.) One metric for the level of 447 oscillations is the smoothness metric given above. 449 3.3. Fairness and Convergence 451 Another set of metrics are those of fairness and of convergence 452 times. Fairness can be considered between flows of the same 453 protocol, and between flows using different protocols (e.g., 454 fairness between TCP and a new transport protocol). 456 There are a number of different fairness measures. These include 457 max-min fairness [HG86], proportional fairness [KMT98] [K01], the 458 fairness index proposed in [JCH84], and the product measure, a 459 variant of network power [BJ81]. 461 Max-min fairness: In order to satisfy the max-min fairness criteria, 462 the smallest throughput rate must be as large as possible. Given 463 this condition, the next-smallest throughput rate must be as large 464 as possible, and so on. Thus, the max-min fairness gives absolute 465 priority to the smallest flows. 467 Epsilon-fairness: A metric related to max-min fairness is epsilon- 468 fairness, where a rate allocation is defined as epsilon-fair if 469 min_i x_i / max_i x_i >= 1 - epsilon. 471 where x_i is the resource allocation to the i-th user. Epsilon- 472 fairness measures the worst-case ratio between any two throughput 473 rates [ZKL04]. 475 Proportional fairness: In contrast, an allocation x is defined as 476 proportionally fair if for any other feasible allocation x*, the 477 aggregate of proportional changes is zero or negative: 479 sum_i (x*_i - x_i)/x_i <= 0. 481 "This criterion favours smaller flows, but less emphatically than 482 max-min fairness" [K01]. 484 Jain's fairness index: The fairness index in [JCH84] is 486 (( sum_i x_i )^2) / (n * sum_i (x_i)^2 ) , 488 where there are n users. This fairness index ranges from 0 to 1, 489 and is maximum when all users receive the same allocation. This 490 index is k/n when k users equally share the resource, and the other 491 n-k users receive zero allocation. 493 The product measure: The product measure 495 product_i x_i , 497 the product of the throughput of the individual connections, is also 498 used as a measure of fairness. For our purposes, let x_i be the 499 throughput for the i-th connection. (In other contexts x_i is taken 500 as the power of the i-th connection, and the product measure is 501 referred to as network power.) The product measure is particularly 502 sensitive to segregation; the product measure is zero if any 503 connection receives zero throughput. In [MS91] it is shown that for 504 a network with many connections and one shared gateway, the product 505 measure is maximized when all connections receive the same 506 throughput. 508 In [CRM05], Colussi et al. propose a new definition of fairness, 509 that "a set of TCP fair flows do not cause more congestion than a 510 set of TCP flows would cause", where congestion is defined in terms 511 of queueing delay, queueing delay variation, the congestion event 512 rate [e.g., loss event rate], and the packet loss rate. 514 Chiu and Tan in [CT06] argue for redefining the notion of fairness 515 when studying traffic controls for inelastic traffic, proposing that 516 inelastic flows adopt other traffic controls such as admission 517 control. 519 Fairness and the number of congested links: Some of these fairness 520 metrics are discussed in more detail in [F91]. We note that there 521 is not a clear consensus for the fairness goals, in particular for 522 fairness between flows that traverse different numbers of congested 523 links [F91]. 525 Fairness and round-trip times: One goal cited in a number of new 526 transport protocols has been that of fairness between flows with 527 different round-trip times [KHR02] [XHR04]. We note that there is 528 not a consensus in the networking community about the desirability 529 of this goal, or about the implications and interactions between 530 this goal and other metrics [FJ92] (Section 3.3). 532 Fairness and packet size: One fairness issue is that of the relative 533 fairness for flows with different packet sizes. Many file transfer 534 applications will use the maximum packet size possible; in 535 contrast, low-bandwidth VoIP flows are likely to send small packets, 536 sending a new packet every 10 to 40 ms., to limit delay. Should a 537 small-packet VoIP connection receive the same sending rate in bytes 538 per second as a large-packet TCP connection in the same environment, 539 or should it receive the same sending rate in *packets* per second? 540 This fairness issue has been discussed in more detail in [FK04], 541 with [FK05] also describing the ways that packet size can effect the 542 packet drop rate experienced by a flow. 544 Convergence times: Convergence times concern the time for 545 convergence to fairness between an existing flow and a newly- 546 starting one, and are a special concern for environments with high- 547 bandwidth flows. Convergence times also concern the time for 548 convergence to fairness between two existing flows after a sudden 549 change such as a change in link capacity on a wireless link. As 550 with fairness, convergence times can matter both between flows of 551 the same protocol, and between flows using different protocols 552 [SLFK03]. One metric used for convergence times is the delta-fair 553 convergence time, defined as the time taken for two flows with the 554 same round-trip time to go from shares of 100/101-th and 1/101-th of 555 the link bandwidth, to having close to fair sharing with shares of 556 (1+delta)/2 and (1-delta)/2 of the link bandwidth [BBFS01]. A 557 similar metric for convergence times measures the convergence time 558 as the number of round-trip times for two flows to reach epsilon- 559 fairness, when starting from a maximally-unfair state [ZKL04]. ' 561 3.4. Robustness for Challenging Environments 563 While congestion control mechanisms are generally evaluated first 564 over environments with static routing in a network of two-way point- 565 to-point links, some environments bring up more challenging problems 566 (e.g., corrupted packets, variable bandwidth, mobility) as well as 567 new metrics to be considered (e.g., energy consumption). 569 Robustness for challenging environments: Robustness needs to be 570 explored for paths with reordering, corruption, variable bandwidth, 571 asymmetric routing, router configuration changes, mobility, and the 572 like. In general, Internet architecture has valued robustness over 573 efficiency, e.g., when there are tradeoffs between robustness and 574 the throughput, delay, and fairness metrics described above. 576 Energy consumption: In mobile environments the energy consumption 577 for the mobile end-node can be a key metric that is affected by the 578 transport protocol [TM02]. 580 Goodput: For wireless networks, goodput can be a useful metric, 581 where goodput is defined as the fraction of useful data from all of 582 the data delivered. High goodput indicates an efficient use of the 583 radio spectrum and lower interference to other users [GF04]. 585 3.5. Robustness to Failures and to Misbehaving Users 587 One goal is for congestion control mechanisms to be robust to 588 misbehaving users, such as receivers that `lie' to data senders 589 about the congestion experienced along the path or otherwise attempt 590 to bypass the congestion control mechanisms of the sender [SCWA99]. 591 Another goal is for congestion control mechanisms to be as robust as 592 possible to failures, such as failures of routers in using explicit 593 feedback to end-nodes or failures of end-nodes to follow the 594 prescribed protocols, 596 3.6. Deployability 598 One metric for congestion control mechanisms is their deployability 599 in the current Internet. Metrics related to deployability include 600 the ease of failure diagnosis, and the overhead in terms of packet 601 header size or added complexity at end-nodes or routers. 603 One key aspect of deployability concerns the range of deployment 604 needed for a new congestion control mechanism. Consider the 605 following possible deployment requirements: 607 * Only at the sender (e.g., NewReno in TCP); 608 * Only at the receiver (e.g., delayed acknowledgements in TCP); 609 * Both the sender and receiver (e.g., SACK TCP); 610 * At a single router (e.g., RED); 611 * All of the routers along the end-to-end path; 612 * Both end nodes and all routers along the path (e.g., XCP). 614 Another deployability issue concerns the complexity of the code. 615 Roughly how many lines of code are required to implement the 616 mechanism in software? Is floating point math required? We note 617 that we don't suggest these questions as ways to reduce the 618 deployability metric to a single number; we suggest them as issues 619 that could be considered in evaluating the deployability of a 620 proposed congestion control mechanism. 622 3.7. Metrics for Specific Types of Transport 624 In some cases modified metrics are needed for evaluting transport 625 protocols intended for QoS-enabled environments or for below-best- 626 effort traffic [VKD02] [KK03]. 628 3.8. User-Based Metrics 630 An alternate approach that has been proposed for the evaluation of 631 congestion control mechanisms would be to evaluate in terms of user 632 metrics such as user satisfaction, or in terms of application- 633 specific utility functions. Such an approach would require the 634 definition of a range of user metrics or of application-specific 635 utility functions for the range of applications under consideration 636 (e.g., FTP, HTTP, VoIP). 638 4. Metrics in the IP Performance Metrics (IPPM) Working Group 640 The IPPM Working Group [IPPM] was established to define performance 641 metrics to be used by network operators, end users, or independent 642 testing groups. The metrics include metrics for connectivity, delay 643 and loss, delay variation, loss patterns, packet reordering, bulk 644 transfer capacity, and link capacity. The IPPM documents give 645 concrete, well-defined metrics, along with a methodology for 646 measuring the metric. The metrics discussed in this document have a 647 different purpose from the IPPM metrics; in this document we are 648 discussing metrics as used in analysis, simulations, and experiments 649 for the evaluation of congestion control mechanisms. Further, we 650 are discussing these metrics in a general sense, rather than looking 651 for specific concrete definitions for each metric. However, there 652 are many cases where the metric definitions from IPPM could be 653 useful, particularly for specific issues of how to measure these 654 metrics in simulations or in testbeds. 656 5. Comments on Methodology 658 The types of scenarios that are used to test specific metrics, and 659 the range of parameters that it is useful to consider, will be 660 discussed in separate documents, e.g., along with specific scenarios 661 for use in evaluating congestion control mechanisms. 663 We note that it can be important to evaluate metrics over a wide 664 range of environments, with a range of link bandwidths, congestion 665 levels, and levels of statistical multiplexing. It is also 666 important to evaluate congestion control mechanisms in a range of 667 scenarios, including typical ranges of connection sizes and round- 668 trip times [FK02]. It is also useful to compare metrics for new or 669 modified transport protocols with those of the current standards for 670 TCP. 672 Li et al. in "Experimental Evaluation of TCP Protocols for High- 673 Speed Networks" [LLS05] focus on the performance of TCP in high- 674 speed networks, and consider metrics for aggregate throughput, loss 675 rates, fairness (including fairness between flows with different 676 round-trip times), response times (including convergence times), and 677 incremental deployment. 679 More general references on methodology include [J91]. Papers that 680 discuss the range of metrics for evaluating congestion control 681 include [MTZ04]. 683 6. Security Considerations 685 There are no security considerations in this document. 687 7. IANA Considerations 689 There are no IANA considerations in this document. 691 8. Acknowledgements 693 Thanks to Armando Caro, Dah Ming Chiu, Dado Colussi, Wesley Eddy, 694 Nelson Fonseca, Janardhan Iyengar, Doug Leith, Saverio Mascolo, Sean 695 Moore, Injong Rhee, Andras Veres, and Damon Wischik, and members of 696 the Transport Modeling Research Group for feedback and 697 contributions. 699 Informative References 701 [BB01] D. Bansal and H. Balakrishnan, Binomial Congestion Control 702 Algorithms, IEEE Infocom, April 2001. 704 [BBFS01] D. Bansal, H. Balakrishnan, S. Floyd, and S. Shenker, 705 Dynamic Behavior of Slowly-Responsive Congestion Control 706 Algorithms, SIGCOMM 2001. 708 [BJ81] K. Bharath-Kumar and J. Jeffrey, A New Approach to 709 Performance-Oriented Flow Control, IEEE Transactions on 710 Communications, Vol.COM-29 N.4, April 1981. 712 [CRM05] A New Approach to TCP-Fairness, Report C-2005-49, University 713 of Helsinki, Finland, 2005. 715 [CT06] D. Chiu and A. Tam, Redefining Fairness in the Study of TCP- 716 friendly Traffic Controls, Technical Report, 2006. 718 [F91] S. Floyd, Connections with Multiple Congested Gateways in 719 Packet-Switched Networks Part 1: One-way Traffic, Computer 720 Communication Review, Vol.21, No.5, October 1991, p. 30-47. 722 [FK05] S. Floyd and E. Kohler, TFRC for Voice: the VoIP Variant, 723 draft-ietf-dccp-tfrc-voip-02.txt, internet draft, work in 724 progress, July 2005. 726 [FHP00] S. Floyd, M. Handley, and J. Padhye, A Comparison of 727 Equation-Based and AIMD Congestion Control, May 2000. URL 728 "http://www.icir.org/tfrc/". 730 [FHPW00] S. Floyd, M. Handley, J. Padhye, and J. Widmer, Equation- 731 Based Congestion Control for Unicast Applications, SIGCOMM 2000, 732 August 2000. 734 [FJ92] S. Floyd and V. Jacobson, On Traffic Phase Effects in Packet- 735 Switched Gateways, Internetworking: Research and Experience, V.3 736 N.3, September 1992, p.115-156. 738 [FK04] S. Floyd and J. Kempf, IAB Concerns Regarding Congestion 739 Control for Voice Traffic in the Internet, RFC 3714, March 2004. 741 [FK02] S. Floyd and E. Kohler, Internet Research Needs Better 742 Models, Hotnets-I. October 2002. 744 [GF04] A. Gurtov and S. Floyd, Modeling Wireless Links for Transport 745 Protocols, ACM CCR, 34(2):85-96, April 2004. 747 [HKLRX05] S. Ha, Y. Kim, L. Le, I. Rhee, and L. Xu, A Step Toward 748 Realistic Evaluation of High-speed TCP Protocols, technnical 749 report, North Carolina State University, January 2006. 751 [HG86] E. Hahne and R. Gallager, Round Robin Scheduling for Fair 752 Flow Control in Data Communications Networks, IEEE International 753 Conference on Communications, June 1986. 755 [IPPM] IP Performance Metrics (IPPM) Working Group, URL 756 "http://www.ietf.org/html.charters/ippm-charter.html". 758 [J91] R. Jain, The Art of Computer Systems Performance Analysis: 759 Techniques for Experimental Design, Measurement, Simulation, and 760 Modeling, John Wiley & Sons, 1991. 762 [JCH84] R. Jain, D.M. Chiu, and W. Hawe, A Quantitative Measure of 763 Fairness and Discrimination for Resource Allocation in Shared 764 Systems, DEC TR-301, Littleton, MA: Digital Equipment 765 Corporation, 1984. 767 [JWL04] C. Jin, D. Wei, and S. Low, FAST TCP: Motivation, 768 Architecture, Algorithms, Performance, IEEE INFOCOM, March 2004. 770 [K01] F. Kelly, Mathematical Modelling of the Internet, "Mathematics 771 Unlimited - 2001 and Beyond" (Editors B. Engquist and W. 772 Schmid), Springer-Verlag, Berlin, pp. 685-702, 2001. 774 [KHR02] D. Katabi, M. Handley, and C. Rohrs, Congestion Control for 775 High Bandwidth-Delay Product Networks, ACM Sigcomm, 2002. 777 [HKLRX05] S. Ha, Y. Kim, L. Le, I. Rhee and L. Xu, A Step toward 778 Realistic Performance Evaluation of High-Speed TCP Variants, 779 2005. 781 [KK03] A. Kuzmanovic and E. W. Knightly, TCP-LP: A Distributed 782 Algorithm for Low Priority Data Transfer, IEEE INFOCOM 2003, 783 April 2003. 785 [KMT98] F. Kelly, A. Maulloo and D. Tan, Rate Control in 786 Communication Networks: Shadow Prices, Proportional Fairness and 787 Stability. Journal of the Operational Research Society 49, pp. 788 237-252, 1998. 790 [LLS05] Y-T. Li, D. Leith, and R. Shorten, Experimental Evaluation 791 of TCP Protocols for High-Speed Networks, Hamilton Institute, 792 2005. URL "http://www.hamilton.ie/net/eval/results-HI2005.pdf". 794 [MS91] D. Mitra and J. Seery, Dynamic Adaptive Windows for High 795 Speed Data Networks with Multiple Paths and Propagation Delays, 796 INFOCOM '91, pp 39-48. [MTZ04] L. Mamatas, V. Tsaoussidis, and 797 C. Zhang, Approaches to Congestion Control in Packet Networks, 798 2004. 800 [QuickStart] Quick-Start Web Page, URL 801 "http://www.icir.org/floyd/quickstart.html". 803 [RFC 2119] S. Bradner. Key Words For Use in RFCs to Indicate 804 Requirement Levels. RFC 2119. 806 BIBREF RFC3168 "RFC 3168" K Ramakrishnan, S. Floyd, and D. Black, 807 The Addition of Explicit Congestion Notification (ECN) to IP, 808 RFC 3168, September 2001. 810 [RFC 3448] M. Handley, S. Floyd, J. Padhye, and J. Widmer, TCP 811 Friendly Rate Control (TFRC): Protocol Specification, RFC 3448, 812 Proposed Standard, January 2003. 814 [RFC 3611] T. Friedman, R. Caceres, and A. Clark, RTP Control 815 Protocol Extended Reports (RTCP XR), RFC 3611, November 2003. 817 [RX05] I. Rhee and L. Xu, CUBIC: A New TCP-Friendly High-Speed TCP 818 Variant, PFLDnet 2005. 820 [SLFK03] R.N. Shorten, D.J. Leith, J. Foy, and R. Kilduff, Analysis 821 and Design of Congestion Control in Synchronised Communication 822 Networks. Proc. 12th Yale Workshop on Adaptive & Learning 823 Systems, May 2003. 825 [SCWA99] TCP Congestion Control with a Misbehaving Receiver, ACM 826 Computer Communications Review, October 1999. 828 [TM02] V. Tsaoussidis and I. Matta, Open Issues of TCP for Mobile 829 Computing, Journal of Wireless Communications and Mobile 830 Computing: Special Issue on Reliable Transport Protocols for 831 Mobile Computing, February 2002. 833 [WCL05] D. X. Wei, P. Cao and S. H. Low, Time for a TCP Benchmark 834 Suite?, Technical Report, Caltech CS, Stanford EAS, Caltech, 835 2005. 837 [VKD02] A. Venkataramani, R. Kokku, and M. Dahlin, TCP Nice: A 838 Mechanism for Background Transfers, Fifth USENIX Symposium on 839 Operating System Design and Implementation (OSDI), 2002. 841 [XHR04] L. Xu, K. Harfoush, and I. Rhee, Binary Increase Congestion 842 Control for Fast, Long Distance Networks, Infocom 2004. 844 [ZKL04] Y. Zhang, S.-R. Kang, and D. Loguinov, Delayed Stability and 845 Performance of Distributed Congestion Control, ACM SIGCOMM, 846 August 2004. 848 Authors' Addresses 850 Sally Floyd 851 ICSI Center for Internet Research 852 1947 Center Street, Suite 600 853 Berkeley, CA 94704 854 USA 856 Full Copyright Statement 858 Copyright (C) The Internet Society 2006. This document is subject 859 to the rights, licenses and restrictions contained in BCP 78, and 860 except as set forth therein, the authors retain all their rights. 862 This document and the information contained herein are provided on 863 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 864 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE 865 INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR 866 IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 867 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 868 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 870 Intellectual Property 872 The IETF takes no position regarding the validity or scope of any 873 Intellectual Property Rights or other rights that might be claimed 874 to pertain to the implementation or use of the technology described 875 in this document or the extent to which any license under such 876 rights might or might not be available; nor does it represent that 877 it has made any independent effort to identify any such rights. 878 Information on the procedures with respect to rights in RFC 879 documents can be found in BCP 78 and BCP 79. 881 Copies of IPR disclosures made to the IETF Secretariat and any 882 assurances of licenses to be made available, or the result of an 883 attempt made to obtain a general license or permission for the use 884 of such proprietary rights by implementers or users of this 885 specification can be obtained from the IETF on-line IPR repository 886 at http://www.ietf.org/ipr. 888 The IETF invites any interested party to bring to its attention any 889 copyrights, patents or patent applications, or other proprietary 890 rights that may cover technology that may be required to implement 891 this standard. Please address the information to the IETF at ietf- 892 ipr@ietf.org.