idnits 2.17.1 draft-ietf-rmcat-eval-criteria-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (September 22, 2016) is 2773 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC5681' is defined on line 531, but no explicit reference was found in the text == Unused Reference: 'SA4-LR' is defined on line 550, but no explicit reference was found in the text == Unused Reference: 'TCP-eval-suite' is defined on line 553, but no explicit reference was found in the text == Outdated reference: A later version (-18) exists of draft-ietf-avtcore-rtp-circuit-breakers-14 == Outdated reference: A later version (-11) exists of draft-ietf-rmcat-wireless-tests-01 == Outdated reference: A later version (-10) exists of draft-ietf-rmcat-eval-test-03 == Outdated reference: A later version (-07) exists of draft-ietf-rmcat-video-traffic-model-00 == Outdated reference: A later version (-09) exists of draft-ietf-netvc-testing-03 Summary: 0 errors (**), 0 flaws (~~), 9 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RMCAT WG V. Singh 3 Internet-Draft callstats.io 4 Intended status: Informational J. Ott 5 Expires: March 26, 2017 Technical University of Munich 6 S. Holmer 7 Google 8 September 22, 2016 10 Evaluating Congestion Control for Interactive Real-time Media 11 draft-ietf-rmcat-eval-criteria-06 13 Abstract 15 The Real-time Transport Protocol (RTP) is used to transmit media in 16 telephony and video conferencing applications. This document 17 describes the guidelines to evaluate new congestion control 18 algorithms for interactive point-to-point real-time media. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at http://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on March 26, 2017. 37 Copyright Notice 39 Copyright (c) 2016 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (http://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 57 3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 5 58 4. List of Network Parameters . . . . . . . . . . . . . . . . . 5 59 4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 5 60 4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 5 61 4.3. DropTail Router Queue Length . . . . . . . . . . . . . . 6 62 4.4. Loss generation model . . . . . . . . . . . . . . . . . . 6 63 4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 6 64 4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 7 65 4.5.2. Approximately Random Subject to No-Reordering Bounded 66 PDV (NR-RPVD) . . . . . . . . . . . . . . . . 8 67 4.5.3. Recommended distribution . . . . . . . . . . . . . . 9 68 5. WiFi or Cellular Links . . . . . . . . . . . . . . . . . . . 9 69 6. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 9 70 6.1. TCP taffic model . . . . . . . . . . . . . . . . . . . . 9 71 6.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 10 72 6.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 10 73 7. Security Considerations . . . . . . . . . . . . . . . . . . . 10 74 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 75 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 10 76 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 10 77 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 11 78 11.1. Normative References . . . . . . . . . . . . . . . . . . 11 79 11.2. Informative References . . . . . . . . . . . . . . . . . 12 80 Appendix A. Application Trade-off . . . . . . . . . . . . . . . 13 81 A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 13 82 Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 13 83 B.1. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 13 84 B.2. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 13 85 B.3. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 13 86 B.4. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 13 87 B.5. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 13 88 B.6. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 14 89 B.7. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 14 90 B.8. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 14 91 B.9. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 14 92 B.10. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 14 93 B.11. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 15 94 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 15 96 1. Introduction 98 This memo describes the guidelines to help with evaluating new 99 congestion control algorithms for interactive point-to-point real 100 time media. The requirements for the congestion control algorithm 101 are outlined in [I-D.ietf-rmcat-cc-requirements]). This document 102 builds upon previous work at the IETF: Specifying New Congestion 103 Control Algorithms [RFC5033] and Metrics for the Evaluation of 104 Congestion Control Algorithms [RFC5166]. 106 The guidelines proposed in the document are intended to help prevent 107 a congestion collapse, promote fair capacity usage and optimize the 108 media flow's throughput. Furthermore, the proposed algorithms are 109 expected to operate within the envelope of the circuit breakers 110 defined in [I-D.ietf-avtcore-rtp-circuit-breakers]. 112 This document only provides broad-level criteria for evaluating a new 113 congestion control algorithm. The minimal requirement for RMCAT 114 proposals is to produce or present results for the test scenarios 115 described in [I-D.ietf-rmcat-eval-test] (Basic Test Cases). 116 Additionally, proponents may produce evaluation results for the 117 wireless test scenarios [I-D.ietf-rmcat-wireless-tests]. 119 2. Terminology 121 The terminology defined in RTP [RFC3550], RTP Profile for Audio and 122 Video Conferences with Minimal Control [RFC3551], RTCP Extended 123 Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback 124 (RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506] 125 apply. 127 3. Metrics 129 Each experiment is expected to log every incoming and outgoing packet 130 (the RTP logging format is described in Section 3.1). The logging 131 can be done inside the application or at the endpoints using PCAP 132 (packet capture, e.g., tcpdump, wireshark). The following are 133 calculated based on the information in the packet logs: 135 1. Sending rate, Receiver rate, Goodput (measured at 200ms 136 intervals) 138 2. Packets sent, Packets received 140 3. Bytes sent, bytes received 142 4. Packet delay 143 5. Packets lost, Packets discarded (from the playout or de-jitter 144 buffer) 146 6. If using, retransmission or FEC: post-repair loss 148 7. Fairness or Unfairness: Experiments testing the performance of 149 an RMCAT proposal against any cross-traffic must define its 150 expected criteria for fairness. The "unfairness" test guideline 151 (measured at 1s intervals) is: 152 1. Does not trigger the circuit breaker. 153 2. No RMCAT stream achieves more than 3 times the average 154 throughput of the RMCAT stream with the lowest average 155 throughput, for a case when the competing streams have similar 156 RTTs. 157 3. RTT should not grow by a factor of 3 for the existing flows 158 when a new flow is added. 159 For example, see the test scenarios described in 160 [I-D.ietf-rmcat-eval-test]. 162 8. Convergence time: The time taken to reach a stable rate at 163 startup, after the available link capacity changes, or when new 164 flows get added to the bottleneck link. 166 9. Instability or oscillation in the sending rate: The frequency or 167 number of instances when the sending rate oscillates between an 168 high watermark level and a low watermark level, or vice-versa in 169 a defined time window. For example, the watermarks can be set 170 at 4x interval: 500 Kbps, 2 Mbps, and a time window of 500ms. 172 10. [Editor's note: Section 3, in [I-D.ietf-netvc-testing] contains 173 objective Metrics for evaluating codecs.] 175 11. Bandwidth Utilization, defined as ratio of the instantaneous 176 sending rate to the instantaneous bottleneck capacity. This 177 metric is useful only when an RMCAT flow is by itself or 178 competing with similar cross-traffic. 180 From the logs the statistical measures (min, max, mean, standard 181 deviation and variance) for the whole duration or any specific part 182 of the session can be calculated. Also the metrics (sending rate, 183 receiver rate, goodput, latency) can be visualized in graphs as 184 variation over time, the measurements in the plot are at 1 second 185 intervals. Additionally, from the logs it is possible to plot the 186 histogram or CDF of packet delay. 188 [Open issue (1): Using Jain-fairness index (JFI) for measuring self- 189 fairness between RTP flows? measured at what intervals? visualized as 190 a CDF or a timeseries? Additionally: Use JFI for comparing fairness 191 between RTP and long TCP flows? ] 193 3.1. RTP Log Format 195 The log file is tab or comma separated containing the following 196 details: 198 Send or receive timestamp (unix) 199 RTP payload type 200 SSRC 201 RTP sequence no 202 RTP timestamp 203 marker bit 204 payload size 206 If the congestion control implements, retransmissions or FEC, the 207 evaluation should report both packet loss (before applying error- 208 resilience) and residual packet loss (after applying error- 209 resilience). 211 4. List of Network Parameters 213 The implementors initially are encouraged to choose evaluation 214 settings from the following values: 216 4.1. One-way Propagation Delay 218 Experiments are expected to verify that the congestion control is 219 able to work in challenging situations, for example over trans- 220 continental and/or satellite links. Typical values are: 222 1. Very low latency: 0-1ms 224 2. Low latency: 50ms 226 3. High latency: 150ms 228 4. Extreme latency: 300ms 230 4.2. End-to-end Loss 232 To model lossy links, the experiments can choose one of the following 233 loss rates, the fractional loss is the ratio of packets lost and 234 packets sent. 236 1. no loss: 0% 237 2. 1% 239 3. 5% 241 4. 10% 243 5. 20% 245 4.3. DropTail Router Queue Length 247 The router queue length is measured as the time taken to drain the 248 FIFO queue. It has been noted in various discussions that the queue 249 length in the current deployed Internet varies significantly. While 250 the core backbone network has very short queue length, the home 251 gateways usually have larger queue length. Those various queue 252 lengths can be categorized in the following way: 254 1. QoS-aware (or short): 70ms 256 2. Nominal: 300-500ms 258 3. Buffer-bloated: 1000-2000ms 260 Here the size of the queue is measured in bytes or packets and to 261 convert the queue length measured in seconds to queue length in 262 bytes: 264 QueueSize (in bytes) = QueueSize (in sec) x Throughput (in bps)/8 266 4.4. Loss generation model 268 [Open Issue: Describes the model for generating packet losses, for 269 example, losses can be generated using traces, or using the Gilbert- 270 Elliot model, or randomly (uncorrelated loss).] 272 4.5. Jitter models 274 This section defines jitter models for the purposes of this document. 275 When jitter is to be applied to both the RMCAT flow and any competing 276 flow (such as a TCP competing flow), the competing flow will use the 277 jitter definition below that does not allow for re-ordering of 278 packets on the competing flow (see NR-RBPDV definition below). 280 Jitter is an overloaded term in communications. Its meaning is 281 typically associated with the variation of a metric (e.g., delay) 282 with respect to some reference metric (e.g., average delay or minimum 283 delay). For example, RFC 3550 jitter is a smoothed estimate of 284 jitter which is particularly meaningful if the underlying packet 285 delay variation was caused by a Gaussian random process. 287 Because jitter is an overloaded term, we instead use the term Packet 288 Delay Variation (PDV) to describe the variation of delay of 289 individual packets in the same sense as the IETF IPPM WG has defined 290 PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has 291 defined IP Packet Delay Variation (IPDV) in their documents (e.g., 292 Y.1540). 294 Most PDV distributions in packet network systems are one-sided 295 distributions (the measurement of which with a finite number of 296 measurement samples result in one-sided histograms). In the usual 297 packet network transport case there is typically one packet that 298 transited the network with the minimum delay, then a majority of 299 packets also transit the system within some variation from this 300 minimum delay, and then a minority of the packets transit the network 301 with delays higher than the median or average transit time (these are 302 outliers). Although infrequent, outliers can cause significant 303 deleterious operation in adaptive systems and should be considered in 304 RMCAT adaptation designs. 306 In this section we define two different bounded PDV characteristics, 307 1) Random Bounded PDV and 2) Approximately Random Subject to No- 308 Reordering Bounded PDV. 310 [Open issue: which one is used in evaluations? Are both used?] 312 4.5.1. Random Bounded PDV (RBPDV) 314 The RBPDV probability distribution function (pdf) is specified to be 315 of some mathematically describable function which includes some 316 practical minimum and maximum discrete values suitable for testing. 317 For example, the minimum value, x_min, might be specified as the 318 minimum transit time packet and the maximum value, x_max, might be 319 idefined to be two standard deviations higher than the mean. 321 Since we are typically interested in the distribution relative to the 322 mean delay packet, we define the zero mean PDV sample, z(n), to be 323 z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random 324 variable x and x_mean is the mean of x. 326 We assume here that s(n) is the original source time of packet n and 327 the post-jitter induced emmission time, j(n), for packet n is j(n) = 328 {[z(n) + x_mean] + s(n)}. It follows that the separation in the post- 329 jitter time of packets n and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}. 330 Since the first term is always a positive quantity, we note that 331 packet reordering at the receiver is possible whenever the second 332 term is greater than the first. Said another way, whenever the 333 difference in possible zero mean PDV sample delays (i.e., [x_max- 334 x_min]) exceeds the inter-departure time of any two sent packets, we 335 have the possibility of packet re-ordering. 337 There are important use cases in real networks where packets can 338 become re-ordered such as in load balancing topologies and during 339 route changes. However, for the vast majority of cases there is no 340 packet re-ordering because most of the time packets follow the same 341 path. Due to this, if a packet becomes overly delayed, the packets 342 after it on that flow are also delayed. This is especially true for 343 mobile wireless links where there are per-flow queues prior to base 344 station scheduling. Owing to this important use case, we define 345 another PDV profile similar to the above, but one that does not allow 346 for re-ordering within a flow. 348 4.5.2. Approximately Random Subject to No-Reordering Bounded PDV (NR- 349 RPVD) 351 No Reordering RPDV, NR-RPVD, is defined similarly to the above with 352 one important exception. Let serial(n) be defined as the 353 serialization delay of packet n at the lowest bottleneck link rate 354 (or other appropriate rate) in a given test. Then we produce all the 355 post-jitter values for j(n) for n = 1, 2, ... N, where N is the 356 length of the source sequence s to be offset-ed. The exception can 357 be stated as follows: We revisit all j(n) beginning from index n=2, 358 and if j(n) is determined to be less than [j(n-1)+serial(n-1)], we 359 redefine j(n) to be equal to [j(n-1)+serial(n-1)] and continue for 360 all remaining n (i.e., n = 3, 4, .. N). This models the case where 361 the packet n is sent immediately after packet (n-1) at the bottleneck 362 link rate. Although this is generally the theoretical minimum in 363 that it assumes that no other packets from other flows are in-between 364 packet n and n+1 at the bottleneck link, it is a reasonable 365 assumption for per flow queuing. 367 We note that this assumption holds for some important exception 368 cases, such as packets immediately following outliers. There are a 369 multitude of software controlled elements common on end-to-end 370 Internet paths (such as firewalls, ALGs and other middleboxes) which 371 stop processing packets while servicing other functions (e.g., 372 garbage collection). Often these devices do not drop packets, but 373 rather queue them for later processing and cause many of the 374 outliers. Thus NR-RPVD models this particular use case (assuming 375 serial(n+1) is defined appropriately for the device causing the 376 outlier) and thus is believed to be important for adaptation 377 development for RMCAT. 379 4.5.3. Recommended distribution 381 It is recommended that z(n) is distributed according to a truncated 382 Gaussian: 384 z(n) ~ |max(min(N(0, std^2), N_STD * std), -N_STD * std)| 386 where N(0, std^2) is the Gaussian distribution with zero mean and 387 standard deviation std. Recommended values: 389 o std = 5 ms 391 o N_STD = 3 393 5. WiFi or Cellular Links 395 [I-D.ietf-rmcat-wireless-tests] describes the test cases to simulate 396 networks with wireless links. The document describes mechanism to 397 simulate both cellular and WiFi networks. 399 6. Traffic Models 401 6.1. TCP taffic model 403 Long-lived TCP flows will download data throughout the session and 404 are expected to have infinite amount of data to send or receive. For 405 example, to 407 Each short TCP flow is modeled as a sequence of file downloads 408 interleaved with idle periods. Not all short TCPs start at the same 409 time, i.e., some start in the ON state while others start in the OFF 410 state. 412 The short TCP flows can be modelled as follows: 30 connections start 413 simultaneously fetching small (30-50 KB) amounts of data. This 414 covers the case where the short TCP flows are not fetching a video 415 file. 417 The idle period between bursts of starting a group of TCP flows is 418 typically derived from an exponential distribution with the mean 419 value of 10 seconds. 421 [These values were picked based on the data available at 422 http://httparchive.org/interesting.php as of October 2015]. 424 6.2. RTP Video model 426 [I-D.ietf-rmcat-video-traffic-model] describes two types of video 427 traffic models for evaluating RMCAT candidate algorithms. The first 428 model statistically characterizes the behavior of a video encoder. 429 Whereas the second model uses video traces. 431 For example, test sequences are available at: [xiph-seq] and 432 [HEVC-seq]. The currently chosen video streams are: Foreman and 433 FourPeople. 435 6.3. Background UDP 437 Background UDP flow is modeled as a constant bit rate (CBR) flow. It 438 will download data at a particular CBR rate for the complete session, 439 or will change to particular CBR rate at predefined intervals. The 440 inter packet interval is calculated based on the CBR and the packet 441 size (is typically set to the path MTU size, the default value can be 442 1500 bytes). 444 7. Security Considerations 446 Security issues have not been discussed in this memo. 448 8. IANA Considerations 450 There are no IANA impacts in this memo. 452 9. Contributors 454 The content and concepts within this document are a product of the 455 discussion carried out in the Design Team. 457 Michael Ramalho provided the text for the Jitter model. 459 10. Acknowledgements 461 Much of this document is derived from previous work on congestion 462 control at the IETF. 464 The authors would like to thank Harald Alvestrand, Anna Brunstrom, 465 Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde, 466 Stefan Holmer, Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers 467 O'Hanlon, Colin Perkins, Michael Ramalho, Zaheduzzaman Sarker, 468 Timothy B. Terriberry, Michael Welzl, and Mo Zanaty for providing 469 valuable feedback on earlier versions of this draft. Additionally, 470 also thank the participants of the design team for their comments and 471 discussion related to the evaluation criteria. 473 11. References 475 11.1. Normative References 477 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 478 Jacobson, "RTP: A Transport Protocol for Real-Time 479 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 480 July 2003, . 482 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 483 Video Conferences with Minimal Control", STD 65, RFC 3551, 484 DOI 10.17487/RFC3551, July 2003, 485 . 487 [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., 488 "RTP Control Protocol Extended Reports (RTCP XR)", RFC 489 3611, DOI 10.17487/RFC3611, November 2003, 490 . 492 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 493 "Extended RTP Profile for Real-time Transport Control 494 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI 495 10.17487/RFC4585, July 2006, 496 . 498 [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size 499 Real-Time Transport Control Protocol (RTCP): Opportunities 500 and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 501 2009, . 503 [I-D.ietf-rmcat-cc-requirements] 504 Jesup, R. and Z. Sarker, "Congestion Control Requirements 505 for Interactive Real-Time Media", draft-ietf-rmcat-cc- 506 requirements-09 (work in progress), December 2014. 508 [I-D.ietf-avtcore-rtp-circuit-breakers] 509 Perkins, C. and V. Varun, "Multimedia Congestion Control: 510 Circuit Breakers for Unicast RTP Sessions", draft-ietf- 511 avtcore-rtp-circuit-breakers-14 (work in progress), March 512 2016. 514 [I-D.ietf-rmcat-wireless-tests] 515 Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and 516 M. Ramalho, "Evaluation Test Cases for Interactive Real- 517 Time Media over Wireless Networks", draft-ietf-rmcat- 518 wireless-tests-01 (work in progress), November 2015. 520 11.2. Informative References 522 [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion 523 Control Algorithms", BCP 133, RFC 5033, DOI 10.17487/ 524 RFC5033, August 2007, 525 . 527 [RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion 528 Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March 529 2008, . 531 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 532 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 533 . 535 [I-D.ietf-rmcat-eval-test] 536 Sarker, Z., Varun, V., Zhu, X., and M. Ramalho, "Test 537 Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- 538 eval-test-03 (work in progress), March 2016. 540 [I-D.ietf-rmcat-video-traffic-model] 541 Zhu, X., Cruz, S., and Z. Sarker, "Modeling Video Traffic 542 Sources for RMCAT Evaluations", draft-ietf-rmcat-video- 543 traffic-model-00 (work in progress), January 2016. 545 [I-D.ietf-netvc-testing] 546 Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec 547 Testing and Quality Measurement", draft-ietf-netvc- 548 testing-03 (work in progress), July 2016. 550 [SA4-LR] S4-050560, 3GPP., "Error Patterns for MBMS Streaming over 551 UTRAN and GERAN", 3GPP S4-050560, 5 2008. 553 [TCP-eval-suite] 554 Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier, 555 R., Gang, W., Eggert, L., Ha, S., and I. Rhee, "Towards a 556 Common TCP Evaluation Suite", Proc. PFLDnet. 2008, August 557 2008. 559 [xiph-seq] 560 Daede, T., "Video Test Media Set", 561 https://people.xiph.org/~tdaede/sets/ , . 563 [HEVC-seq] 564 HEVC, , "Test Sequences", 565 http://www.netlab.tkk.fi/~varun/test_sequences/ , . 567 Appendix A. Application Trade-off 569 Application trade-off is yet to be defined. see RMCAT requirements 570 [I-D.ietf-rmcat-cc-requirements] document. Perhaps each experiment 571 should define the application's expectation or trade-off. 573 A.1. Measuring Quality 575 No quality metric is defined for performance evaluation, it is 576 currently an open issue. However, there is consensus that congestion 577 control algorithm should be able to show that it is useful for 578 interactive video by performing analysis using a real codec and video 579 sequences. 581 Appendix B. Change Log 583 Note to the RFC-Editor: please remove this section prior to 584 publication as an RFC. 586 B.1. Changes in draft-ietf-rmcat-eval-criteria-06 588 o Updated Jitter. 590 B.2. Changes in draft-ietf-rmcat-eval-criteria-05 592 o Improved text surrounding wireless tests, video sequences, and 593 short-TCP model. 595 B.3. Changes in draft-ietf-rmcat-eval-criteria-04 597 o Removed the guidelines section, as most of the sections are now 598 covered: wireless tests, video model, etc. 600 o Improved Short TCP model based on the suggestion to use 601 httparchive.org. 603 B.4. Changes in draft-ietf-rmcat-eval-criteria-03 605 o Keep-alive version. 607 o Moved link parameters and traffic models from eval-test 609 B.5. Changes in draft-ietf-rmcat-eval-criteria-02 611 o Incorporated fairness test as a working test. 613 o Updated text on mimimum evaluation requirements. 615 B.6. Changes in draft-ietf-rmcat-eval-criteria-01 617 o Removed Appendix B. 619 o Removed Section on Evaluation Parameters. 621 B.7. Changes in draft-ietf-rmcat-eval-criteria-00 623 o Updated references. 625 o Resubmitted as WG draft. 627 B.8. Changes in draft-singh-rmcat-cc-eval-04 629 o Incorporate feedback from IETF 87, Berlin. 631 o Clarified metrics: convergence time, bandwidth utilization. 633 o Changed fairness criteria to fairness test. 635 o Added measuring pre- and post-repair loss. 637 o Added open issue of measuring video quality to appendix. 639 o clarified use of DropTail and AQM. 641 o Updated text in "Minimum Requirements for Evaluation" 643 B.9. Changes in draft-singh-rmcat-cc-eval-03 645 o Incorporate the discussion within the design team. 647 o Added a section on evaluation parameters, it describes the flow 648 and network characteristics. 650 o Added Appendix with self-fairness experiment. 652 o Changed bottleneck parameters from a proposal to an example set. 654 o 656 B.10. Changes in draft-singh-rmcat-cc-eval-02 658 o Added scenario descriptions. 660 B.11. Changes in draft-singh-rmcat-cc-eval-01 662 o Removed QoE metrics. 664 o Changed stability to steady-state. 666 o Added measuring impact against few and many flows. 668 o Added guideline for idle and data-limited periods. 670 o Added reference to TCP evaluation suite in example evaluation 671 scenarios. 673 Authors' Addresses 675 Varun Singh 676 CALLSTATS I/O Oy 677 Runeberginkatu 4c A 4 678 Helsinki 00100 679 Finland 681 Email: varun@callstats.io 682 URI: https://www.callstats.io/about 684 Joerg Ott 685 Technical University of Munich 686 Faculty of Informatics 687 Boltzmannstrasse 3 688 Garching bei Muenchen, DE 85748 689 Germany 691 Email: ott@in.tum.de 693 Stefan Holmer 694 Google 695 Kungsbron 2 696 Stockholm 11122 697 Sweden 699 Email: holmer@google.com