idnits 2.17.1 draft-irtf-tmrg-metrics-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 13. -- Found old boilerplate from RFC 3978, Section 5.5 on line 858. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 869. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 876. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 882. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (7 August 2006) is 6472 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'MTZ04' is mentioned on line 786, but not defined == Outdated reference: A later version (-07) exists of draft-ietf-dccp-tfrc-voip-02 -- Obsolete informational reference (is this intentional?): RFC 3448 (Obsoleted by RFC 5348) Summary: 3 errors (**), 0 flaws (~~), 5 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force Sally Floyd 2 INTERNET-DRAFT Editor 3 draft-irtf-tmrg-metrics-04.txt 7 August 2006 4 Expires: February 2007 6 Metrics for the Evaluation of Congestion Control Mechanisms 8 Status of this Memo 10 By submitting this Internet-Draft, each author represents that any 11 applicable patent or other IPR claims of which he or she is aware 12 have been or will be disclosed, and any of which he or she becomes 13 aware will be disclosed, in accordance with Section 6 of BCP 79. 15 Internet-Drafts are working documents of the Internet Engineering 16 Task Force (IETF), its areas, and its working groups. Note that 17 other groups may also distribute working documents as Internet- 18 Drafts. 20 Internet-Drafts are draft documents valid for a maximum of six 21 months and may be updated, replaced, or obsoleted by other documents 22 at any time. It is inappropriate to use Internet-Drafts as 23 reference material or to cite them other than as "work in progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt. 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 This Internet-Draft will expire on February 2007. 33 Abstract 35 This document discusses the metrics to be considered in an 36 evaluation of new or modified congestion control mechanisms for the 37 Internet. This includes metrics for the evaluation of new transport 38 protocols, of proposed modifications to TCP, of application-level 39 congestion control, and of Active Queue Management (AQM) mechanisms 40 in the router. This document is intended to be the first in a 41 series of documents aimed at improving the models that we use in the 42 evaluation of transport protocols. 44 This document is a product of the Transport Modeling Research Group 45 (TRMG), and has received detailed feedback from many members of the 46 Research Group (RG). We are not aware of any controversies 47 regarding the content of this document. 49 Table of Contents 51 1. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 52 2. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 5 53 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 54 3.1. Throughput, Delay, and Loss Rates. . . . . . . . . . . . 7 55 3.1.1. Throughput. . . . . . . . . . . . . . . . . . . . . 7 56 3.1.2. Delay . . . . . . . . . . . . . . . . . . . . . . . 8 57 3.1.3. Packet Loss Rates . . . . . . . . . . . . . . . . . 8 58 3.2. Response Times and Minimizing Oscillations . . . . . . . 9 59 3.2.1. Response to Changes . . . . . . . . . . . . . . . . 9 60 3.2.2. Minimizing Oscillations . . . . . . . . . . . . . . 10 61 3.3. Fairness and Convergence . . . . . . . . . . . . . . . . 11 62 3.4. Robustness for Challenging Environments. . . . . . . . . 13 63 3.5. Robustness to Failures and to Misbehaving 64 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 65 3.6. Deployability. . . . . . . . . . . . . . . . . . . . . . 14 66 3.7. Metrics for Specific Types of Transport. . . . . . . . . 15 67 3.8. User-Based Metrics . . . . . . . . . . . . . . . . . . . 15 68 4. Metrics in the IP Performance Metrics (IPPM) Working 69 Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 70 5. Comments on Methodology . . . . . . . . . . . . . . . . . . . 15 71 6. Security Considerations . . . . . . . . . . . . . . . . . . . 16 72 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 73 8. Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . 16 74 Informative References . . . . . . . . . . . . . . . . . . . . . 16 75 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 19 76 Full Copyright Statement . . . . . . . . . . . . . . . . . . . . 20 77 Intellectual Property. . . . . . . . . . . . . . . . . . . . . . 20 79 1. Conventions 81 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 82 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 83 document are to be interpreted as described in [RFC 2119]. 85 TO BE DELETED BY THE RFC EDITOR UPON PUBLICATION: 87 Changes from draft-irtf-tmrg-metrics-03.txt: 89 * Added a paragraph about sudden changes due to mobility 90 and the heterogeneity of wireless access types. 91 Suggestion from Andras Veres. 93 * Add covariance as one of the metrics for oscillations. 94 Suggestion from Saverio Mascolo, original text 95 contribution from Injong Rhee. 97 Changes from draft-irtf-tmrg-metrics-02.txt: 99 * Added a few sentences to the Abstract about the 100 status of the document. 102 Changes from draft-irtf-tmrg-metrics-01.txt: 104 * Added a discussion about the metrics in IPPM. 106 Changes from draft-irtf-tmrg-metrics-01c.txt: 108 * Added to the discussion of network-based, flow-based, 109 and user-based metrics, based on email from Dado Colussi, 110 Sean Moore, Damon Wischik, Dah Ming Chiu, and others. 112 * Changed "packet drop rate" to "packet loss rate". 113 Suggestion from Nelson Fonseca. 115 * Added a discussion of the Colussi et al. paper on a new 116 definition of fairness. 118 * Added a discussion of the Chiu and Tan paper on redefining 119 fairness for inelastic traffic. 121 Changes from draft-irtf-tmrg-metrics-01b.txt: 123 * Added a discussion of goodput vs. throughput. 124 Suggestion from Nelson Fonseca. 126 Changes from draft-irtf-tmrg-metrics-01a.txt: 128 * Added to the discussion of packet drop rate metrics. 129 Suggestions from Janardhan Iyengar, Sean Moore, 130 Armando Caro, and Nelson Fonseca. 132 * Added a sentence about throughput used as a metric for 133 transfer times for very short flows. 134 Response to email from Seam Moore. 136 Changes from draft-irtf-tmrg-metrics-00.txt: 138 * Added a list of relevant congestion control mechanisms to 139 the abstract. Suggestion from Sean Moore. 141 * Added to the Introduction. Suggestion from Dado Colussi. 143 * Added a sentence about jitter to the discussion of minimizing 144 oscillations. Suggestion from Wesley Eddy. 146 * Added a note about convergence between existing flows after 147 a change in bandwidth. Suggestion from Wesley Eddy. 149 * Added to the section on deployability. Suggestion from 150 Wesley Eddy. 152 Changes from draft-floyd-transport-metrics-00.txt: 154 * Added metrics for: 155 - robustness in challenging environments, 156 - deployability, 157 - robustness to failures and to misbehaving users 159 * Added a discussion of fairness and packet size. 161 2. Introduction 163 As a step towards improving our methodologies for evaluating 164 congestion control mechanisms, in this document we discuss some of 165 the metrics to be considered. We also consider the relationship 166 between metrics, e.g., the well-known tradeoff between throughput 167 and delay. 169 We consider metrics for aggregate traffic (taking into account the 170 effect of flows on competing traffic in the network) as well as the 171 heterogeneous goals of different applications or transport protocols 172 (e.g., of high throughput for bulk data transfer, and of low delay 173 for interactive voice or video). Different transport protocols or 174 AQM mechanisms might have goals of optimizing different sets of 175 metrics, with one transport protocol optimized for per-flow 176 throughput and another optimized for robustness over wireless links, 177 and with different degrees of attention to fairness with competing 178 traffic. We hope this document will be used as a step in evaluating 179 proposed congestion control mechanisms for a wide range of metrics, 180 noting that Mechanism X is good at optimizing Metric A, but pays the 181 price with poor performance for Metric B. The goal would be to have 182 a broad view of both the strengths and weaknesses of newly-proposed 183 congestion control mechanisms. 185 Subsequent documents are planned to present sets of simulation and 186 testbed scenarios for the evaluation of transport protocols and of 187 congestion control mechanisms, based on the best current practice of 188 the research community. These are not intended to be complete or 189 final benchmark test suites, but simply to be one step of many to be 190 used by researchers in evaluating congestion control mechanisms. 191 Subsequent documents are also planned on the methodologies in using 192 these sets of scenarios. 194 This is work from the Transport Modeling Research Group (TMRG) in 195 the IRTF (Internet Research Task Force). 197 3. Metrics 199 The metrics that we discuss are the following: 201 o Throughput; 203 o Delay; 205 o Packet loss rates; 207 o Response to sudden changes or to transient events; 209 o Minimizing oscillations in throughput or in delay; 211 o Fairness and convergence times; 213 o Robustness for challenging environments; 215 o Robustness to failures and to misbehaving users; 217 o Deployability; 219 o Metrics for specific types of transport. 221 o User-based metrics. 223 We consider each of these below. Many of the metrics have network- 224 based, flow-based, and user-based interpretations. For example, 225 network-based metrics can consider aggregate bandwidth and aggregate 226 drop rates, flow-based metrics can consider end-to-end transfer 227 times for file transfers or end-to-end delay and packet drop rates 228 for interactive traffic, and user-based metrics can consider user 229 wait time or user satisfaction with the multimedia experience. Our 230 main goal in this document is to explain the set of metrics that can 231 be relevant, and not to legislate on the more appropriate 232 methodology for using each general metric. 234 For some of the metrics, such as fairness between flows, there is 235 not a clear agreement in the network community about the desired 236 goals. In these cases, the document attempts to present the range 237 of approaches. 239 3.1. Throughput, Delay, and Loss Rates 241 Because of the clear tradeoffs between throughput, delay, and loss 242 rates, it can be useful to consider the three metrics together. 244 An alternative would be to consider a separate metric such as power, 245 defined in this context as throughput over delay, that combines 246 throughput and delay. However, we do not propose in this document a 247 clear target in terms of the tradeoffs between throughput and delay; 248 we are simply proposing that the evaluation of transport protocols 249 include an exploration of the competing metrics. 251 3.1.1. Throughput 253 Throughput can be measured as a router-based metric of aggregate 254 link throughput, as a flow-based metric of per-connection transfer 255 times, and as user-based metrics of utility functions or user wait 256 times. It is a clear goal of most congestion control mechanisms to 257 maximize throughput, subject to application demand and to the 258 constraints of the other metrics. 260 Throughput is sometimes distinguished from goodput, where throughput 261 is the link or flow throughput in bytes per second, and goodput, 262 also measured in bytes per second, is the subset of throughput 263 consisting of useful traffic. That is, `goodput' excludes duplicate 264 packets, packets that will be dropped downstream, packet fragments 265 or ATM cells that are dropped at the receiver because they can't be 266 re-assembled into complete packets, and the like. 268 We note that maximizing throughput is of concern in a wide range of 269 environments, from highly-congested networks to under-utilized ones, 270 and from long-lived flows to very short ones. As an example, 271 throughput has been used as one of the metrics for evaluating Quick- 272 Start, a proposal to allow flows to start-up faster than slow-start, 273 where throughput has been evaluated in terms of the transfer times 274 for connections with a range of transfer sizes [QuickStart]. 276 In some contexts, it might be sufficient to consider the aggregate 277 throughput or the mean per-flow throughput, while in other contexts 278 it might be necessary to consider the distribution of per-flow 279 throughput. Some researchers evaluate transport protocols in terms 280 of maximizing the aggregate user utility, where a user's utility is 281 generally defined as a function of the user's throughput [KMT98]. 283 Individual applications can have application-specific needs in terms 284 of throughput. For example, real-time video traffic can have highly 285 variable bandwidth demands; VoIP traffic is sensitive to the amount 286 of bandwidth received immediately after idle periods. Thus, user 287 metrics for throughput can be more complex than simply the per- 288 connection transfer time. 290 3.1.2. Delay 292 Like throughput, delay can be measured as a router-based metric of 293 queueing delay over time, or as a flow-based metric in terms of per- 294 packet transfer times. For reliable transfer, the per-packet 295 transfer time includes the possible delay of retransmitting a lost 296 packet. 298 Users of bulk data transfer applications might care about per-packet 299 transfer times only in so far as they affect the per-connection 300 transfer time. On the other end of the spectrum, for users of 301 streaming media, per-packet delay can be a significant concern. 302 Note that in some cases the average delay might not capture the 303 metric of interest to the users; for example, some users might care 304 about the worst-case delay, or about the tail of the delay 305 distribution. 307 3.1.3. Packet Loss Rates 309 Packet loss rates can be measured as a network-based or as a flow- 310 based metric. 312 When evaluating the effect of packet losses or ECN marks (Explicit 313 Congestion Notification, RFC 3168) on the performance of a 314 congestion control mechanism for an individual flow, researchers 315 often use both the packet loss/mark rate for that connection, and 316 the congestion event rate (also called the loss event rate), where a 317 congestion event or loss event consists of one or more lost or 318 marked packets in one round-trip time [RFC 3448]. 320 Some users might care about packet loss rates only in so far as they 321 affect per-connection transfer times, while other users might care 322 about packet loss rates directly. RFC 3611, RTP Control Protocol 323 Extended Reports, describes a VoIP performance-reporting standard 324 called RTCP XR, which includes a set of burst metrics. In RFC 3611, 325 a burst is defined as the maximal sequence starting and ending with 326 a lost packet, and not including a sequence of Gmin or more packets 327 that are not lost [RFC 3611]. The burst metrics in RFC 3611 consist 328 of the burst density (the fraction of packets in bursts), gap 329 density (the fraction of packets in the gaps between bursts), burst 330 duration (the mean duration of bursts in seconds), and gap duration 331 (the mean duration of gaps in seconds). 333 In some cases it is useful to distinguish between packets dropped at 334 routers due to congestion, and packets lost in the network due to 335 corruption. 337 One network-related reason to avoid high steady-state packet loss 338 rates is to avoid congestion collapse in environments containing 339 paths with multiple congested links. In such environments, high 340 packet loss rates could result in congested links wasting scarce 341 bandwidth by carrying packets that will only be dropped downstream, 342 before being delivered to the receiver. 344 3.2. Response Times and Minimizing Oscillations 346 In this section we consider response times and oscillations 347 together, as there are well-known tradeoffs between improving 348 response times and minimizing oscillations. In addition, the 349 scenarios that illustrate the dangers of poor response times are 350 often quite different from the scenarios that illustrate the dangers 351 of unnecessary oscillations. 353 3.2.1. Response to Changes 355 One of the key concerns in the design of congestion control 356 mechanisms has been the response times to sudden congestion in the 357 network. On the one hand, congestion control mechanisms should 358 respond reasonably promptly to sudden congestion from routing or 359 bandwidth changes, or from a burst of competing traffic. At the 360 same time, congestion control mechanisms should not respond too 361 severely to transient changes, e.g., to a sudden increase in delay 362 that will dissipate in less than the connection's round-trip time. 364 Congestion control mechanisms also have to contend with sudden 365 changes in the bandwidth-delay product due to mobility. Such 366 bandwith-delay product changes are expected to become more frequent 367 and to have greater impact than path changes today. As a result of 368 both mobility and of the heterogenity of wireless access types 369 (802.11b,a,g, WIMAX, WCDMA, HS-WCDMA, E-GPRS, Bluetooth, etc.), both 370 the bandwidth and the round-trip delay can change suddenly, 371 sometimes by several orders of magnitude. 373 Evaluating the response to sudden or transient changes can be of 374 particular concern for slowly-responding congestion control 375 mechanisms such as equation-based congestion control [RFC 3448], and 376 for AIMD (Additive Increase Multiplicative Decrease) or related 377 mechanisms using parameters that make them more slowly-responding 378 that TCP [BB01] [BBFS01]. 380 In addition to the responsiveness and smoothness of aggregate 381 traffic, one can consider the tradeoffs between responsiveness, 382 smoothness, and aggressiveness for an individual connection [FHP00]. 383 In this case smoothness can be defined by the largest reduction in 384 the sending rate in one round-trip time, in a deterministic 385 environment with a packet drop exactly every 1/p packets. The 386 responsiveness is defined as the number of round-trip times of 387 sustained congested required for the sender to halve the sending 388 rate, and the aggressiveness is defined as the maximum increase in 389 the sending rate in one round-trip time, in packets per second, in 390 the absence of congestion. 392 3.2.2. Minimizing Oscillations 394 One goal is that of stability, in terms of minimizing oscillations 395 of queueing delay or of throughput. In practice, stability is 396 frequently associated with rate fluctuations or variance. Rate 397 variations can result in fluctuations in router queue size and 398 therefore of queue overflows. These queue overflows can cause loss 399 synchronizations across co-existing flows and periodic under- 400 utilization of link capacity, both of which are considered to be 401 general signs of network instability. Thus, measuring the rate 402 variations of flows is often used to measure the stability of 403 transport protocols. To measure rate variations, [JWL04], [RX05], 404 and [FHPW00] use the coefficient of variation (CoV) of per-flow 405 transmission rates and [WCL05] suggests the use of standard 406 deviations of per-flow rates. Since rate variations are a function 407 of time scales, it makes sense to measure these rate variations over 408 various time scales. 410 Measuring per-flow rate variations, however, is only one aspect of 411 transport protocol stability. A realistic experiment setting always 412 involves multiple flows of the transport protocol being observed, 413 along with a significant amount of cross traffic, with rates varying 414 over time, on both the forward and reverse paths. As a congestion 415 control protocol must adapt its rate to the varying rates of 416 competing traffic, just measuring the per-flow statistics of a 417 subset of the traffic could be misleading because it measures the 418 rate fluctuations due in part to the adaptation to competing traffic 419 on the path. Thus, per-flow statistics are most meaningful if they 420 are accompanied by the statistics measured at the network level. As 421 a complementary metric to the per-flow statistics, [HKLRX05] uses 422 measurements of the rate variations of the aggregate flows observed 423 in bottleneck routers over various time scales. 425 Minimizing oscillations in queueing delay or throughput has related 426 per-flow metrics of minimizing jitter in round-trip times and loss 427 rates. 429 An orthogonal goal for some congestion control mechanisms, e.g., for 430 equation-based congestion control, is to minimize the oscillations 431 in the sending rate for an individual connection, given an 432 environment with a fixed, steady-state packet drop rate. (As is 433 well known, TCP congestion control is characterized by a pronounced 434 oscillation in the sending rate, with the sender halving the sending 435 rate in response to congestion.) One metric for the level of 436 oscillations is the smoothness metric given above. 438 3.3. Fairness and Convergence 440 Another set of metrics are those of fairness and of convergence 441 times. Fairness can be considered between flows of the same 442 protocol, and between flows using different protocols (e.g., 443 fairness between TCP and a new transport protocol). 445 There are a number of different fairness measures. These include 446 max-min fairness [HG86], proportional fairness [KMT98] [K01], the 447 fairness index proposed in [JCH84], and the product measure, a 448 variant of network power [BJ81]. 450 Max-min fairness: In order to satisfy the max-min fairness criteria, 451 the smallest throughput rate must be as large as possible. Given 452 this condition, the next-smallest throughput rate must be as large 453 as possible, and so on. Thus, the max-min fairness gives absolute 454 priority to the smallest flows. 456 Epsilon-fairness: A metric related to max-min fairness is epsilon- 457 fairness, where a rate allocation is defined as epsilon-fair if 459 min_i x_i / max_i x_i >= 1 - epsilon. 461 where x_i is the resource allocation to the i-th user. Epsilon- 462 fairness measures the worst-case ratio between any two throughput 463 rates [ZKL04]. 465 Proportional fairness: In contrast, an allocation x is defined as 466 proportionally fair if for any other feasible allocation x*, the 467 aggregate of proportional changes is zero or negative: 469 sum_i (x*_i - x_i)/x_i <= 0. 471 "This criterion favours smaller flows, but less emphatically than 472 max-min fairness" [K01]. 474 Jain's fairness index: The fairness index in [JCH84] is 476 (( sum_i x_i )^2) / (n * sum_i (x_i)^2 ) , 478 where there are n users. This fairness index ranges from 0 to 1, 479 and is maximum when all users receive the same allocation. This 480 index is k/n when k users equally share the resource, and the other 481 n-k users receive zero allocation. 483 The product measure: The product measure 485 product_i x_i , 487 the product of the throughput of the individual connections, is also 488 used as a measure of fairness. For our purposes, let x_i be the 489 throughput for the i-th connection. (In other contexts x_i is taken 490 as the power of the i-th connection, and the product measure is 491 referred to as network power.) The product measure is particularly 492 sensitive to segregation; the product measure is zero if any 493 connection receives zero throughput. In [MS91] it is shown that for 494 a network with many connections and one shared gateway, the product 495 measure is maximized when all connections receive the same 496 throughput. 498 In [CRM05], Colussi et al. propose a new definition of fairness, 499 that "a set of TCP fair flows do not cause more congestion than a 500 set of TCP flows would cause", where congestion is defined in terms 501 of queueing delay, queueing delay variation, the congestion event 502 rate [e.g., loss event rate], and the packet loss rate. 504 Chiu and Tan in [CT06] argue for redefining the notion of fairness 505 when studying traffic controls for inelastic traffic, proposing that 506 inelastic flows adopt other traffic controls such as admission 507 control. 509 Fairness and the number of congested links: Some of these fairness 510 metrics are discussed in more detail in [F91]. We note that there 511 is not a clear consensus for the fairness goals, in particular for 512 fairness between flows that traverse different numbers of congested 513 links [F91]. 515 Fairness and round-trip times: One goal cited in a number of new 516 transport protocols has been that of fairness between flows with 517 different round-trip times [KHR02] [XHR04]. We note that there is 518 not a consensus in the networking community about the desirability 519 of this goal, or about the implications and interactions between 520 this goal and other metrics [FJ92] (Section 3.3). 522 Fairness and packet size: One fairness issue is that of the relative 523 fairness for flows with different packet sizes. Many file transfer 524 applications will use the maximum packet size possible; in 525 contrast, low-bandwidth VoIP flows are likely to send small packets, 526 sending a new packet every 10 to 40 ms., to limit delay. Should a 527 small-packet VoIP connection receive the same sending rate in bytes 528 per second as a large-packet TCP connection in the same environment, 529 or should it receive the same sending rate in *packets* per second? 530 This fairness issue has been discussed in more detail in [FK04], 531 with [FK05] also describing the ways that packet size can effect the 532 packet drop rate experienced by a flow. 534 Convergence times: Convergence times concern the time for 535 convergence to fairness between an existing flow and a newly- 536 starting one, and are a special concern for environments with high- 537 bandwidth flows. Convergence times also concern the time for 538 convergence to fairness between two existing flows after a sudden 539 change such as a change in link capacity on a wireless link. As 540 with fairness, convergence times can matter both between flows of 541 the same protocol, and between flows using different protocols 542 [SLFK03]. One metric used for convergence times is the delta-fair 543 convergence time, defined as the time taken for two flows with the 544 same round-trip time to go from shares of 100/101-th and 1/101-th of 545 the link bandwidth, to having close to fair sharing with shares of 546 (1+delta)/2 and (1-delta)/2 of the link bandwidth [BBFS01]. A 547 similar metric for convergence times measures the convergence time 548 as the number of round-trip times for two flows to reach epsilon- 549 fairness, when starting from a maximally-unfair state [ZKL04]. ' 551 3.4. Robustness for Challenging Environments 553 While congestion control mechanisms are generally evaluated first 554 over environments with static routing in a network of two-way point- 555 to-point links, some environments bring up more challenging problems 556 (e.g., corrupted packets, variable bandwidth, mobility) as well as 557 new metrics to be considered (e.g., energy consumption). 559 Robustness for challenging environments: Robustness needs to be 560 explored for paths with reordering, corruption, variable bandwidth, 561 asymmetric routing, router configuration changes, mobility, and the 562 like. In general, Internet architecture has valued robustness over 563 efficiency, e.g., when there are tradeoffs between robustness and 564 the throughput, delay, and fairness metrics described above. 566 Energy consumption: In mobile environments the energy consumption 567 for the mobile end-node can be a key metric that is affected by the 568 transport protocol [TM02]. 570 Goodput: For wireless networks, goodput can be a useful metric, 571 where goodput is defined as the fraction of useful data from all of 572 the data delivered. High goodput indicates an efficient use of the 573 radio spectrum and lower interference to other users [GF04]. 575 3.5. Robustness to Failures and to Misbehaving Users 577 One goal is for congestion control mechanisms to be robust to 578 misbehaving users, such as receivers that `lie' to data senders 579 about the congestion experienced along the path or otherwise attempt 580 to bypass the congestion control mechanisms of the sender [SCWA99]. 581 Another goal is for congestion control mechanisms to be as robust as 582 possible to failures, such as failures of routers in using explicit 583 feedback to end-nodes or failures of end-nodes to follow the 584 prescribed protocols, 586 3.6. Deployability 588 One metric for congestion control mechanisms is their deployability 589 in the current Internet. Metrics related to deployability include 590 the ease of failure diagnosis, and the overhead in terms of packet 591 header size or added complexity at end-nodes or routers. 593 One key aspect of deployability concerns the range of deployment 594 needed for a new congestion control mechanism. Consider the 595 following possible deployment requirements: 597 * Only at the sender (e.g., NewReno in TCP); 598 * Only at the receiver (e.g., delayed acknowledgements in TCP); 599 * Both the sender and receiver (e.g., SACK TCP); 600 * At a single router (e.g., RED); 601 * All of the routers along the end-to-end path; 602 * Both end nodes and all routers along the path (e.g., XCP). 604 Another deployability issue concerns the complexity of the code. 605 Roughly how many lines of code are required to implement the 606 mechanism in software? Is floating point math required? We note 607 that we don't suggest these questions as ways to reduce the 608 deployability metric to a single number; we suggest them as issues 609 that could be considered in evaluating the deployability of a 610 proposed congestion control mechanism. 612 3.7. Metrics for Specific Types of Transport 614 In some cases modified metrics are needed for evaluting transport 615 protocols intended for QoS-enabled environments or for below-best- 616 effort traffic [VKD02] [KK03]. 618 3.8. User-Based Metrics 620 An alternate approach that has been proposed for the evaluation of 621 congestion control mechanisms would be to evaluate in terms of user 622 metrics such as user satisfaction, or in terms of application- 623 specific utility functions. Such an approach would require the 624 definition of a range of user metrics or of application-specific 625 utility functions for the range of applications under consideration 626 (e.g., FTP, HTTP, VoIP). 628 4. Metrics in the IP Performance Metrics (IPPM) Working Group 630 The IPPM Working Group [IPPM] was established to define performance 631 metrics to be used by network operators, end users, or independent 632 testing groups. The metrics include metrics for connectivity, delay 633 and loss, delay variation, loss patterns, packet reordering, bulk 634 transfer capacity, and link capacity. The IPPM documents give 635 concrete, well-defined metrics, along with a methodology for 636 measuring the metric. The metrics discussed in this document have a 637 different purpose from the IPPM metrics; in this document we are 638 discussing metrics as used in analysis, simulations, and experiments 639 for the evaluation of congestion control mechanisms. Further, we 640 are discussing these metrics in a general sense, rather than looking 641 for specific concrete definitions for each metric. However, there 642 are many cases where the metric definitions from IPPM could be 643 useful, particularly for specific issues of how to measure these 644 metrics in simulations or in testbeds. 646 5. Comments on Methodology 648 The types of scenarios that are used to test specific metrics, and 649 the range of parameters that it is useful to consider, will be 650 discussed in separate documents, e.g., along with specific scenarios 651 for use in evaluating congestion control mechanisms. 653 We note that it can be important to evaluate metrics over a wide 654 range of environments, with a range of link bandwidths, congestion 655 levels, and levels of statistical multiplexing. It is also 656 important to evaluate congestion control mechanisms in a range of 657 scenarios, including typical ranges of connection sizes and round- 658 trip times [FK02]. It is also useful to compare metrics for new or 659 modified transport protocols with those of the current standards for 660 TCP. 662 Li et al. in "Experimental Evaluation of TCP Protocols for High- 663 Speed Networks" [LLS05] focus on the performance of TCP in high- 664 speed networks, and consider metrics for aggregate throughput, loss 665 rates, fairness (including fairness between flows with different 666 round-trip times), response times (including convergence times), and 667 incremental deployment. 669 More general references on methodology include [J91]. Papers that 670 discuss the range of metrics for evaluating congestion control 671 include [MTZ04]. 673 6. Security Considerations 675 There are no security considerations in this document. 677 7. IANA Considerations 679 There are no IANA considerations in this document. 681 8. Acknowledgements 683 Thanks to Armando Caro, Dah Ming Chiu, Dado Colussi, Wesley Eddy, 684 Nelson Fonseca, Janardhan Iyengar, Doug Leith, Saverio Mascolo, Sean 685 Moore, Injong Rhee, Andras Veres, and Damon Wischik, and members of 686 the Transport Modeling Research Group for feedback and 687 contributions. 689 Informative References 691 [BB01] D. Bansal and H. Balakrishnan, Binomial Congestion Control 692 Algorithms, IEEE Infocom, April 2001. 694 [BBFS01] D. Bansal, H. Balakrishnan, S. Floyd, and S. Shenker, 695 Dynamic Behavior of Slowly-Responsive Congestion Control 696 Algorithms, SIGCOMM 2001. 698 [BJ81] K. Bharath-Kumar and J. Jeffrey, A New Approach to 699 Performance-Oriented Flow Control, IEEE Transactions on 700 Communications, Vol.COM-29 N.4, April 1981. 702 [CRM05] A New Approach to TCP-Fairness, Report C-2005-49, University 703 of Helsinki, Finland, 2005. 705 [CT06] D. Chiu and A. Tam, Redefining Fairness in the Study of TCP- 706 friendly Traffic Controls, Technical Report, 2006. 708 [F91] S. Floyd, Connections with Multiple Congested Gateways in 709 Packet-Switched Networks Part 1: One-way Traffic, Computer 710 Communication Review, Vol.21, No.5, October 1991, p. 30-47. 712 [FK05] S. Floyd and E. Kohler, TFRC for Voice: the VoIP Variant, 713 draft-ietf-dccp-tfrc-voip-02.txt, internet draft, work in 714 progress, July 2005. 716 [FHP00] S. Floyd, M. Handley, and J. Padhye, A Comparison of 717 Equation-Based and AIMD Congestion Control, May 2000. URL 718 "http://www.icir.org/tfrc/". 720 [FHPW00] S. Floyd, M. Handley, J. Padhye, and J. Widmer, Equation- 721 Based Congestion Control for Unicast Applications, SIGCOMM 2000, 722 August 2000. 724 [FJ92] S. Floyd and V. Jacobson, On Traffic Phase Effects in Packet- 725 Switched Gateways, Internetworking: Research and Experience, V.3 726 N.3, September 1992, p.115-156. 728 [FK04] S. Floyd and J. Kempf, IAB Concerns Regarding Congestion 729 Control for Voice Traffic in the Internet, RFC 3714, March 2004. 731 [FK02] S. Floyd and E. Kohler, Internet Research Needs Better 732 Models, Hotnets-I. October 2002. 734 [GF04] A. Gurtov and S. Floyd, Modeling Wireless Links for Transport 735 Protocols, ACM CCR, 34(2):85-96, April 2004. 737 [HKLRX05] S. Ha, Y. Kim, L. Le, I. Rhee, and L. Xu, A Step Toward 738 Realistic Evaluation of High-speed TCP Protocols, technnical 739 report, North Carolina State University, January 2006. 741 [HG86] E. Hahne and R. Gallager, Round Robin Scheduling for Fair 742 Flow Control in Data Communications Networks, IEEE International 743 Conference on Communications, June 1986. 745 [IPPM] IP Performance Metrics (IPPM) Working Group, URL 746 "http://www.ietf.org/html.charters/ippm-charter.html". 748 [J91] R. Jain, The Art of Computer Systems Performance Analysis: 749 Techniques for Experimental Design, Measurement, Simulation, and 750 Modeling, John Wiley & Sons, 1991. 752 [JCH84] R. Jain, D.M. Chiu, and W. Hawe, A Quantitative Measure of 753 Fairness and Discrimination for Resource Allocation in Shared 754 Systems, DEC TR-301, Littleton, MA: Digital Equipment 755 Corporation, 1984. 757 [JWL04] C. Jin, D. Wei, and S. Low, FAST TCP: Motivation, 758 Architecture, Algorithms, Performance, IEEE INFOCOM, March 2004. 760 [K01] F. Kelly, Mathematical Modelling of the Internet, "Mathematics 761 Unlimited - 2001 and Beyond" (Editors B. Engquist and W. 762 Schmid), Springer-Verlag, Berlin, pp. 685-702, 2001. 764 [KHR02] D. Katabi, M. Handley, and C. Rohrs, Congestion Control for 765 High Bandwidth-Delay Product Networks, ACM Sigcomm, 2002. 767 [HKLRX05] S. Ha, Y. Kim, L. Le, I. Rhee and L. Xu, A Step toward 768 Realistic Performance Evaluation of High-Speed TCP Variants, 769 2005. 771 [KK03] A. Kuzmanovic and E. W. Knightly, TCP-LP: A Distributed 772 Algorithm for Low Priority Data Transfer, IEEE INFOCOM 2003, 773 April 2003. 775 [KMT98] F. Kelly, A. Maulloo and D. Tan, Rate Control in 776 Communication Networks: Shadow Prices, Proportional Fairness and 777 Stability. Journal of the Operational Research Society 49, pp. 778 237-252, 1998. 780 [LLS05] Y-T. Li, D. Leith, and R. Shorten, Experimental Evaluation 781 of TCP Protocols for High-Speed Networks, Hamilton Institute, 782 2005. URL "http://www.hamilton.ie/net/eval/results-HI2005.pdf". 784 [MS91] D. Mitra and J. Seery, Dynamic Adaptive Windows for High 785 Speed Data Networks with Multiple Paths and Propagation Delays, 786 INFOCOM '91, pp 39-48. [MTZ04] L. Mamatas, V. Tsaoussidis, and 787 C. Zhang, Approaches to Congestion Control in Packet Networks, 788 2004. 790 [QuickStart] Quick-Start Web Page, URL 791 "http://www.icir.org/floyd/quickstart.html". 793 [RFC 2119] S. Bradner. Key Words For Use in RFCs to Indicate 794 Requirement Levels. RFC 2119. 796 BIBREF RFC3168 "RFC 3168" K Ramakrishnan, S. Floyd, and D. Black, 797 The Addition of Explicit Congestion Notification (ECN) to IP, 798 RFC 3168, September 2001. 800 [RFC 3448] M. Handley, S. Floyd, J. Padhye, and J. Widmer, TCP 801 Friendly Rate Control (TFRC): Protocol Specification, RFC 3448, 802 Proposed Standard, January 2003. 804 [RFC 3611] T. Friedman, R. Caceres, and A. Clark, RTP Control 805 Protocol Extended Reports (RTCP XR), RFC 3611, November 2003. 807 [RX05] I. Rhee and L. Xu, CUBIC: A New TCP-Friendly High-Speed TCP 808 Variant, PFLDnet 2005. 810 [SLFK03] R.N. Shorten, D.J. Leith, J. Foy, and R. Kilduff, Analysis 811 and Design of Congestion Control in Synchronised Communication 812 Networks. Proc. 12th Yale Workshop on Adaptive & Learning 813 Systems, May 2003. 815 [SCWA99] TCP Congestion Control with a Misbehaving Receiver, ACM 816 Computer Communications Review, October 1999. 818 [TM02] V. Tsaoussidis and I. Matta, Open Issues of TCP for Mobile 819 Computing, Journal of Wireless Communications and Mobile 820 Computing: Special Issue on Reliable Transport Protocols for 821 Mobile Computing, February 2002. 823 [WCL05] D. X. Wei, P. Cao and S. H. Low, Time for a TCP Benchmark 824 Suite?, Technical Report, Caltech CS, Stanford EAS, Caltech, 825 2005. 827 [VKD02] A. Venkataramani, R. Kokku, and M. Dahlin, TCP Nice: A 828 Mechanism for Background Transfers, Fifth USENIX Symposium on 829 Operating System Design and Implementation (OSDI), 2002. 831 [XHR04] L. Xu, K. Harfoush, and I. Rhee, Binary Increase Congestion 832 Control for Fast, Long Distance Networks, Infocom 2004. 834 [ZKL04] Y. Zhang, S.-R. Kang, and D. Loguinov, Delayed Stability and 835 Performance of Distributed Congestion Control, ACM SIGCOMM, 836 August 2004. 838 Authors' Addresses 840 Sally Floyd 841 ICSI Center for Internet Research 842 1947 Center Street, Suite 600 843 Berkeley, CA 94704 844 USA 846 Full Copyright Statement 848 Copyright (C) The Internet Society 2006. This document is subject 849 to the rights, licenses and restrictions contained in BCP 78, and 850 except as set forth therein, the authors retain all their rights. 852 This document and the information contained herein are provided on 853 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 854 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE 855 INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR 856 IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 857 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 858 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 860 Intellectual Property 862 The IETF takes no position regarding the validity or scope of any 863 Intellectual Property Rights or other rights that might be claimed 864 to pertain to the implementation or use of the technology described 865 in this document or the extent to which any license under such 866 rights might or might not be available; nor does it represent that 867 it has made any independent effort to identify any such rights. 868 Information on the procedures with respect to rights in RFC 869 documents can be found in BCP 78 and BCP 79. 871 Copies of IPR disclosures made to the IETF Secretariat and any 872 assurances of licenses to be made available, or the result of an 873 attempt made to obtain a general license or permission for the use 874 of such proprietary rights by implementers or users of this 875 specification can be obtained from the IETF on-line IPR repository 876 at http://www.ietf.org/ipr. 878 The IETF invites any interested party to bring to its attention any 879 copyrights, patents or patent applications, or other proprietary 880 rights that may cover technology that may be required to implement 881 this standard. Please address the information to the IETF at ietf- 882 ipr@ietf.org.