idnits 2.17.1 draft-ietf-rmcat-eval-criteria-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 20, 2015) is 3082 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC5681' is defined on line 507, but no explicit reference was found in the text == Unused Reference: 'SA4-EVAL' is defined on line 520, but no explicit reference was found in the text == Unused Reference: 'SA4-LR' is defined on line 524, but no explicit reference was found in the text == Unused Reference: 'TCP-eval-suite' is defined on line 527, but no explicit reference was found in the text == Outdated reference: A later version (-18) exists of draft-ietf-avtcore-rtp-circuit-breakers-09 == Outdated reference: A later version (-11) exists of draft-ietf-rmcat-wireless-tests-00 == Outdated reference: A later version (-10) exists of draft-ietf-rmcat-eval-test-00 == Outdated reference: A later version (-02) exists of draft-zhu-rmcat-video-traffic-source-00 Summary: 0 errors (**), 0 flaws (~~), 9 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RMCAT WG V. Singh 3 Internet-Draft callstats.io 4 Intended status: Informational J. Ott 5 Expires: April 22, 2016 Aalto University 6 October 20, 2015 8 Evaluating Congestion Control for Interactive Real-time Media 9 draft-ietf-rmcat-eval-criteria-04 11 Abstract 13 The Real-time Transport Protocol (RTP) is used to transmit media in 14 telephony and video conferencing applications. This document 15 describes the guidelines to evaluate new congestion control 16 algorithms for interactive point-to-point real-time media. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at http://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on April 22, 2016. 35 Copyright Notice 37 Copyright (c) 2015 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (http://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 54 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 55 3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 4 56 4. List of Network Parameters . . . . . . . . . . . . . . . . . 5 57 4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 5 58 4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 5 59 4.3. DropTail Router Queue Length . . . . . . . . . . . . . . 6 60 4.4. Loss generation model . . . . . . . . . . . . . . . . . . 6 61 4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 6 62 4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 7 63 4.5.2. Approximately Random Subject to No-Reordering Bounded 64 PDV (NR-RPVD) . . . . . . . . . . . . . . . . 8 65 5. WiFi or Cellular Links . . . . . . . . . . . . . . . . . . . 9 66 6. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 9 67 6.1. TCP taffic model . . . . . . . . . . . . . . . . . . . . 9 68 6.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 9 69 7. Security Considerations . . . . . . . . . . . . . . . . . . . 9 70 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 71 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 10 72 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 10 73 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 10 74 11.1. Normative References . . . . . . . . . . . . . . . . . . 10 75 11.2. Informative References . . . . . . . . . . . . . . . . . 11 76 Appendix A. Application Trade-off . . . . . . . . . . . . . . . 12 77 A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 12 78 Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 12 79 B.1. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 12 80 B.2. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 12 81 B.3. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 12 82 B.4. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 13 83 B.5. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 13 84 B.6. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 13 85 B.7. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 13 86 B.8. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 13 87 B.9. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 14 88 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 14 90 1. Introduction 92 This memo describes the guidelines to help with evaluating new 93 congestion control algorithms for interactive point-to-point real 94 time media. The requirements for the congestion control algorithm 95 are outlined in [I-D.ietf-rmcat-cc-requirements]). This document 96 builds upon previous work at the IETF: Specifying New Congestion 97 Control Algorithms [RFC5033] and Metrics for the Evaluation of 98 Congestion Control Algorithms [RFC5166]. 100 The guidelines proposed in the document are intended to help prevent 101 a congestion collapse, promote fair capacity usage and optimize the 102 media flow's throughput. Furthermore, the proposed algorithms are 103 expected to operate within the envelope of the circuit breakers 104 defined in [I-D.ietf-avtcore-rtp-circuit-breakers]. 106 This document only provides broad-level criteria for evaluating a new 107 congestion control algorithm. The minimal requirement for RMCAT 108 proposals is to produce or present results for the test scenarios 109 described in [I-D.ietf-rmcat-eval-test] (Basic Test Cases). The 110 results of the evaluation are not expected to be included within the 111 internet-draft but should be cited in the document. 113 2. Terminology 115 The terminology defined in RTP [RFC3550], RTP Profile for Audio and 116 Video Conferences with Minimal Control [RFC3551], RTCP Extended 117 Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback 118 (RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506] 119 apply. 121 3. Metrics 123 Each experiment is expected to log every incoming and outgoing packet 124 (the RTP logging format is described in Section 3.1). The logging 125 can be done inside the application or at the endpoints using PCAP 126 (packet capture, e.g., tcpdump, wireshark). The following are 127 calculated based on the information in the packet logs: 129 1. Sending rate, Receiver rate, Goodput (measured at 200ms 130 intervals) 132 2. Packets sent, Packets received 134 3. Bytes sent, bytes received 136 4. Packet delay 138 5. Packets lost, Packets discarded (from the playout or de-jitter 139 buffer) 141 6. If using, retransmission or FEC: post-repair loss 143 7. Fairness or Unfairness: Experiments testing the performance of 144 an RMCAT proposal against any cross-traffic must define its 145 expected criteria for fairness. The "unfairness" test guideline 146 (measured at 1s intervals) is: 147 1. Does not trigger the circuit breaker. 148 2. No RMCAT stream achieves more than 3 times the average 149 throughput of the RMCAT stream with the lowest average 150 throughput, for a case when the competing streams have similar 151 RTTs. 152 3. RTT should not grow by a factor of 3 for the existing flows 153 when a new flow is added. 154 For example, see the test scenarios described in 155 [I-D.ietf-rmcat-eval-test]. 157 8. Convergence time: The time taken to reach a stable rate at 158 startup, after the available link capacity changes, or when new 159 flows get added to the bottleneck link. 161 9. Instability or oscillation in the sending rate: The frequency or 162 number of instances when the sending rate oscillates between an 163 high watermark level and a low watermark level, or vice-versa in 164 a defined time window. For example, the watermarks can be set 165 at 4x interval: 500 Kbps, 2 Mbps, and a time window of 500ms. 167 10. Bandwidth Utilization, defined as ratio of the instantaneous 168 sending rate to the instantaneous bottleneck capacity. This 169 metric is useful only when an RMCAT flow is by itself or 170 competing with similar cross-traffic. 172 From the logs the statistical measures (min, max, mean, standard 173 deviation and variance) for the whole duration or any specific part 174 of the session can be calculated. Also the metrics (sending rate, 175 receiver rate, goodput, latency) can be visualized in graphs as 176 variation over time, the measurements in the plot are at 1 second 177 intervals. Additionally, from the logs it is possible to plot the 178 histogram or CDF of packet delay. 180 [Open issue (1): Using Jain-fairness index (JFI) for measuring self- 181 fairness between RTP flows? measured at what intervals? visualized as 182 a CDF or a timeseries? Additionally: Use JFI for comparing fairness 183 between RTP and long TCP flows? ] 185 3.1. RTP Log Format 187 The log file is tab or comma separated containing the following 188 details: 190 Send or receive timestamp (unix) 191 RTP payload type 192 SSRC 193 RTP sequence no 194 RTP timestamp 195 marker bit 196 payload size 198 If the congestion control implements, retransmissions or FEC, the 199 evaluation should report both packet loss (before applying error- 200 resilience) and residual packet loss (after applying error- 201 resilience). 203 4. List of Network Parameters 205 The implementors initially are encouraged to choose evaluation 206 settings from the following values: 208 4.1. One-way Propagation Delay 210 Experiments are expected to verify that the congestion control is 211 able to work in challenging situations, for example over trans- 212 continental and/or satellite links. Typical values are: 214 1. Very low latency: 0-1ms 216 2. Low latency: 50ms 218 3. High latency: 150ms 220 4. Extreme latency: 300ms 222 4.2. End-to-end Loss 224 To model lossy links, the experiments can choose one of the following 225 loss rates, the fractional loss is the ratio of packets lost and 226 packets sent. 228 1. no loss: 0% 230 2. 1% 232 3. 5% 234 4. 10% 236 5. 20% 238 4.3. DropTail Router Queue Length 240 The router queue length is measured as the time taken to drain the 241 FIFO queue. It has been noted in various discussions that the queue 242 length in the current deployed Internet varies significantly. While 243 the core backbone network has very short queue length, the home 244 gateways usually have larger queue length. Those various queue 245 lengths can be categorized in the following way: 247 1. QoS-aware (or short): 70ms 249 2. Nominal: 300-500ms 251 3. Buffer-bloated: 1000-2000ms 253 Here the size of the queue is measured in bytes or packets and to 254 convert the queue length measured in seconds to queue length in 255 bytes: 257 QueueSize (in bytes) = QueueSize (in sec) x Throughput (in bps)/8 259 4.4. Loss generation model 261 [Editor's note : Describes the model for generating packet losses, 262 for example, losses can be generated using traces, or using the 263 Gilbert-Elliot model, or randomly (uncorrelated loss).] 265 4.5. Jitter models 267 This section defines jitter model for the purposes of this document. 268 When jitter is to be applied to both the RMCAT flow and any competing 269 flow (such as a TCP competing flow), the competing flow will use the 270 jitter definition below that does not allow for re-ordering of 271 packets on the competing flow (see NR-RBPDV definition below). 273 Jitter is an overloaded term in communications. Its meaning is 274 typically associated with the variation of a metric (e.g., delay) 275 with respect to some reference metric (e.g., average delay or minimum 276 delay). For example, RFC 3550 jitter is a smoothed estimate of 277 jitter which is particularly meaningful if the underlying packet 278 delay variation was caused by a Gaussian random process. 280 Because jitter is an overloaded term, we instead use the term Packet 281 Delay Variation (PDV) to describe the variation of delay of 282 individual packets in the same sense as the IETF IPPM WG has defined 283 PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has 284 defined IP Packet Delay Variation (IPDV) in their documents (e.g., 285 Y.1540). 287 Most PDV distributions in packet network systems are one-sided 288 distributions (the measurement of which with a finite number of 289 measurement samples result in one-sided histograms). In the usual 290 packet network transport case there is typically one packet that 291 transited the network with the minimum delay, then a majority of 292 packets also transit the system within some variation from this 293 minimum delay, and then a minority of the packets transits the 294 network with delays higher than the median or average transit time 295 (these are outliers). Although infrequent, outliers can cause 296 significant deleterious operation in adaptive systems and should be 297 considered in RMCAT adaptation designs. 299 In this section we define two different bounded PDV characteristics, 300 1) Random Bounded PDV and 2) Approximately Random Subject to No- 301 Reordering Bounded PDV. 303 4.5.1. Random Bounded PDV (RBPDV) 305 The RBPDV probability distribution function (pdf) is specified to be 306 of some mathematically describable function which includes some 307 practical minimum and maximum discrete values suitable for testing. 308 For example, the minimum value, x_min, might be specified as the 309 minimum transit time packet and the maximum value, x_max, might be 310 idefined to be two standard deviations higher than the mean. 312 Since we are typically interested in the distribution relative to the 313 mean delay packet, we define the zero mean PVD sample, z(n), to be 314 z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random 315 variable x and x_mean is the mean of x. 317 We assume here that s(n) is the original source time of packet n and 318 the post-jitter induced emmission time, j(n), for packet n is j(n) = 319 {[z(n) + x_mean] + s(n)}. It follows that the separation in the post- 320 jitter time of packets n and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}. 321 Since the first term is always a positive quantity, we note that 322 packet reordering at the receiver is possible whenever the second 323 term is greater than the first. Said another way, whenever the 324 difference in possible zero mean PDV sample delays (i.e., [x_max- 325 x_min]) exceeds the inter-departure time of any two sent packets, we 326 have the possibility of packet re-ordering. 328 There are important use cases in real networks where packets can 329 become re-ordered such as in load balancing topologies and during 330 route changes. However, for the vast majority of cases there is no 331 packet re-ordering because most of the time packets follow the same 332 path. Due to this, if a packet becomes overly delayed, the packets 333 after it on that flow are also delayed. This is especially true for 334 mobile wireless links where there are per-flow queues prior to base 335 station scheduling. Owing to this important use case, we define 336 another PDV profile similar to the above, but one that does not allow 337 for re-ordering within a flow. 339 4.5.2. Approximately Random Subject to No-Reordering Bounded PDV (NR- 340 RPVD) 342 No Reordering RPDV, NR-RPVD, is defined similarly to the above with 343 one important exception. Let serial(n) be defined as the 344 serialization delay of packet n at the lowest bottleneck link rate 345 (or other appropriate rate) in a given test. Then we produce all the 346 post-jitter values for j(n) for n = 1, 2, ... N, where N is the 347 length of the source sequence s to be offset-ed. The exception can 348 be stated as follows: We revisit all j(n) beginning from index n=2, 349 and if j(n) is determined to be less than [j(n-1)+serial(n-1)], we 350 redefine j(n) to be equal to [j(n-1)+serial(n-1)] and continue for 351 all remaining n (i.e., n = 3, 4, .. N). This models the case where 352 the packet n is sent immediately after packet (n-1) at the bottleneck 353 link rate. Although this is generally the theoretical minimum in 354 that it assumes that no other packets from other flows are in-between 355 packet n and n+1 at the bottleneck link, it is a reasonable 356 assumption for per flow queuing. 358 We note that this assumption holds for some important exception 359 cases, such as packets immediately following outliers. There are a 360 multitude of software controlled elements common on end-to-end 361 Internet paths (such as firewalls, ALGs and other middleboxes) which 362 stop processing packets while servicing other functions (e.g., 363 garbage collection). Often these devices do not drop packets, but 364 rather queue them for later processing and cause many of the 365 outliers. Thus NR-RPVD models this particular use case (assuming 366 serial(n+1) is defined appropriately for the device causing the 367 outlier) and thus is believed to be important for adaptation 368 development for RMCAT. 370 [Editor's Note: It may require to define test distributions as well. 371 Example test distribution may include- 373 1 - Two-sided: Uniform PDV Distribution. Two quantities to define: 374 x_min and x_max. 376 2 - Two-sided: Truncated Gaussian PDV Distribution. Four quantities 377 to define: the appropriate x_min and x_max for test (e.g., +/- two 378 sigma values), the standard deviation, and the mean. 380 3 - One Sided: Truncated Gaussian PDV Distribution. Quantities to 381 define: three sigma value, the standard deviation, and the mean] 383 5. WiFi or Cellular Links 385 [I-D.fu-rmcat-wifi-test-case] describes methods to evaluate the 386 congestion control in WiFi network, alternatively 387 [I-D.ietf-rmcat-wireless-tests] describes mechanisms to emulate and 388 simulate cellular networks. 390 6. Traffic Models 392 6.1. TCP taffic model 394 Long-lived TCP flows will download data throughout the session and 395 are expected to have infinite amount of data to send or receive. For 396 example, to 398 Each short TCP flow is modeled as a sequence of file downloads 399 interleaved with idle periods. Not all short TCPs start at the same 400 time, i.e., some start in the ON state while others start in the OFF 401 state. 403 The short TCP flows can be modelled as follows: 30 connections start 404 simultaneously fetching small (30-50 KB) amounts of data. This 405 covers the case where the short TCP flows are not fetching a video 406 file. 408 The idle period between bursts of starting a group of TCP flows is 409 typically derived from an exponential distribution with the mean 410 value of 10 seconds. 412 [These values were picked based on the data available at 413 http://httparchive.org/interesting.php as of October 2015]. 415 6.2. RTP Video model 417 [I-D.zhu-rmcat-video-traffic-source] describes two types of video 418 traffic models for evaluating RMCAT candidate algorithms. The first 419 model statistically characterizes the behavior of a video encoder. 420 Whereas the second model uses video traces. 422 7. Security Considerations 424 Security issues have not been discussed in this memo. 426 8. IANA Considerations 428 There are no IANA impacts in this memo. 430 9. Contributors 432 The content and concepts within this document are a product of the 433 discussion carried out in the Design Team. 435 Michael Ramalho provided the text for the Jitter model. 437 10. Acknowledgements 439 Much of this document is derived from previous work on congestion 440 control at the IETF. 442 The authors would like to thank Harald Alvestrand, Anna Brunstrom, 443 Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde, 444 Stefan Holmer, Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers 445 O'Hanlon, Colin Perkins, Michael Ramalho, Zaheduzzaman Sarker, 446 Timothy B. Terriberry, Michael Welzl, and Mo Zanaty for providing 447 valuable feedback on earlier versions of this draft. Additionally, 448 also thank the participants of the design team for their comments and 449 discussion related to the evaluation criteria. 451 11. References 453 11.1. Normative References 455 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 456 Jacobson, "RTP: A Transport Protocol for Real-Time 457 Applications", STD 64, RFC 3550, July 2003. 459 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 460 Video Conferences with Minimal Control", STD 65, RFC 3551, 461 July 2003. 463 [RFC3611] Friedman, T., Caceres, R., and A. Clark, "RTP Control 464 Protocol Extended Reports (RTCP XR)", RFC 3611, November 465 2003. 467 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 468 "Extended RTP Profile for Real-time Transport Control 469 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July 470 2006. 472 [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size 473 Real-Time Transport Control Protocol (RTCP): Opportunities 474 and Consequences", RFC 5506, April 2009. 476 [I-D.ietf-rmcat-cc-requirements] 477 Jesup, R. and Z. Sarker, "Congestion Control Requirements 478 for Interactive Real-Time Media", draft-ietf-rmcat-cc- 479 requirements-09 (work in progress), December 2014. 481 [I-D.ietf-avtcore-rtp-circuit-breakers] 482 Perkins, C. and V. Singh, "Multimedia Congestion Control: 483 Circuit Breakers for Unicast RTP Sessions", draft-ietf- 484 avtcore-rtp-circuit-breakers-09 (work in progress), March 485 2015. 487 [I-D.ietf-rmcat-wireless-tests] 488 Sarker, Z. and I. Johansson, "Evaluation Test Cases for 489 Interactive Real-Time Media over Wireless Networks", 490 draft-ietf-rmcat-wireless-tests-00 (work in progress), 491 June 2015. 493 [I-D.fu-rmcat-wifi-test-case] 494 Fu, J., Zhu, X., Ramalho, M., and W. Tan, "Evaluation Test 495 Cases for Interactive Real-Time Media over Wi-Fi 496 Networks", draft-fu-rmcat-wifi-test-case-01 (work in 497 progress), July 2015. 499 11.2. Informative References 501 [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion 502 Control Algorithms", BCP 133, RFC 5033, August 2007. 504 [RFC5166] Floyd, S., "Metrics for the Evaluation of Congestion 505 Control Mechanisms", RFC 5166, March 2008. 507 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 508 Control", RFC 5681, September 2009. 510 [I-D.ietf-rmcat-eval-test] 511 Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test 512 Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- 513 eval-test-00 (work in progress), August 2014. 515 [I-D.zhu-rmcat-video-traffic-source] 516 Zhu, X., Cruz, S., and Z. Sarker, "Modeling Video Traffic 517 Sources for RMCAT Evaluations", draft-zhu-rmcat-video- 518 traffic-source-00 (work in progress), October 2014. 520 [SA4-EVAL] 521 R1-081955, 3GPP., "LTE Link Level Throughput Data for SA4 522 Evaluation Framework", 3GPP R1-081955, 5 2008. 524 [SA4-LR] S4-050560, 3GPP., "Error Patterns for MBMS Streaming over 525 UTRAN and GERAN", 3GPP S4-050560, 5 2008. 527 [TCP-eval-suite] 528 Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier, 529 R., Gang, W., Eggert, L., Ha, S., and I. Rhee, "Towards a 530 Common TCP Evaluation Suite", Proc. PFLDnet. 2008, August 531 2008. 533 Appendix A. Application Trade-off 535 Application trade-off is yet to be defined. see RMCAT requirements 536 [I-D.ietf-rmcat-cc-requirements] document. Perhaps each experiment 537 should define the application's expectation or trade-off. 539 A.1. Measuring Quality 541 No quality metric is defined for performance evaluation, it is 542 currently an open issue. However, there is consensus that congestion 543 control algorithm should be able to show that it is useful for 544 interactive video by performing analysis using a real codec and video 545 sequences. 547 Appendix B. Change Log 549 Note to the RFC-Editor: please remove this section prior to 550 publication as an RFC. 552 B.1. Changes in draft-ietf-rmcat-eval-criteria-04 554 o Removed the guidelines section, as most of the sections are now 555 covered: wireless tests, video model, etc. 557 o Improved Short TCP model based on the suggestion to use 558 httparchive.org. 560 B.2. Changes in draft-ietf-rmcat-eval-criteria-03 562 o Keep-alive version. 564 o Moved link parameters and traffic models from eval-test 566 B.3. Changes in draft-ietf-rmcat-eval-criteria-02 568 o Incorporated fairness test as a working test. 570 o Updated text on mimimum evaluation requirements. 572 B.4. Changes in draft-ietf-rmcat-eval-criteria-01 574 o Removed Appendix B. 576 o Removed Section on Evaluation Parameters. 578 B.5. Changes in draft-ietf-rmcat-eval-criteria-00 580 o Updated references. 582 o Resubmitted as WG draft. 584 B.6. Changes in draft-singh-rmcat-cc-eval-04 586 o Incorporate feedback from IETF 87, Berlin. 588 o Clarified metrics: convergence time, bandwidth utilization. 590 o Changed fairness criteria to fairness test. 592 o Added measuring pre- and post-repair loss. 594 o Added open issue of measuring video quality to appendix. 596 o clarified use of DropTail and AQM. 598 o Updated text in "Minimum Requirements for Evaluation" 600 B.7. Changes in draft-singh-rmcat-cc-eval-03 602 o Incorporate the discussion within the design team. 604 o Added a section on evaluation parameters, it describes the flow 605 and network characteristics. 607 o Added Appendix with self-fairness experiment. 609 o Changed bottleneck parameters from a proposal to an example set. 611 o 613 B.8. Changes in draft-singh-rmcat-cc-eval-02 615 o Added scenario descriptions. 617 B.9. Changes in draft-singh-rmcat-cc-eval-01 619 o Removed QoE metrics. 621 o Changed stability to steady-state. 623 o Added measuring impact against few and many flows. 625 o Added guideline for idle and data-limited periods. 627 o Added reference to TCP evaluation suite in example evaluation 628 scenarios. 630 Authors' Addresses 632 Varun Singh 633 Nemu Dialogue Systems Oy 634 Runeberginkatu 4c A 4 635 Helsinki 00100 636 Finland 638 Email: varun@callstats.io 639 URI: http://www.callstats.io/ 641 Joerg Ott 642 Aalto University 643 School of Electrical Engineering 644 Otakaari 5 A 645 Espoo, FIN 02150 646 Finland 648 Email: jo@comnet.tkk.fi