idnits 2.17.1 draft-ietf-rmcat-eval-criteria-12.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 27, 2020) is 1521 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-11) exists of draft-ietf-rmcat-wireless-tests-08 Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RMCAT WG V. Singh 3 Internet-Draft callstats.io 4 Intended status: Informational J. Ott 5 Expires: August 30, 2020 Technical University of Munich 6 S. Holmer 7 Google 8 February 27, 2020 10 Evaluating Congestion Control for Interactive Real-time Media 11 draft-ietf-rmcat-eval-criteria-12 13 Abstract 15 The Real-time Transport Protocol (RTP) is used to transmit media in 16 telephony and video conferencing applications. This document 17 describes the guidelines to evaluate new congestion control 18 algorithms for interactive point-to-point real-time media. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at https://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on August 30, 2020. 37 Copyright Notice 39 Copyright (c) 2020 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (https://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 57 3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 5 58 4. List of Network Parameters . . . . . . . . . . . . . . . . . 6 59 4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 6 60 4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 6 61 4.3. Drop Tail Router Queue Length . . . . . . . . . . . . . . 6 62 4.4. Loss generation model . . . . . . . . . . . . . . . . . . 7 63 4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 7 64 4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 8 65 4.5.2. Approximately Random Subject to No-Reordering Bounded 66 PDV (NR-RPVD) . . . . . . . . . . . . . . . . 9 67 4.5.3. Recommended distribution . . . . . . . . . . . . . . 10 68 5. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 10 69 5.1. TCP traffic model . . . . . . . . . . . . . . . . . . . . 10 70 5.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 11 71 5.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 11 72 6. Security Considerations . . . . . . . . . . . . . . . . . . . 11 73 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 74 8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12 75 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12 76 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 77 10.1. Normative References . . . . . . . . . . . . . . . . . . 12 78 10.2. Informative References . . . . . . . . . . . . . . . . . 13 79 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 14 80 A.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 14 81 A.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 14 82 A.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 15 83 A.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 15 84 A.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 15 85 A.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 15 86 A.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 15 87 A.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 15 88 A.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 15 89 A.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 16 90 A.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 16 91 A.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 16 92 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 16 94 1. Introduction 96 This memo describes the guidelines to help with evaluating new 97 congestion control algorithms for interactive point-to-point real 98 time media. The requirements for the congestion control algorithm 99 are outlined in [I-D.ietf-rmcat-cc-requirements]). This document 100 builds upon previous work at the IETF: Specifying New Congestion 101 Control Algorithms [RFC5033] and Metrics for the Evaluation of 102 Congestion Control Algorithms [RFC5166]. 104 The guidelines proposed in the document are intended to help prevent 105 a congestion collapse, promote fair capacity usage and optimize the 106 media flow's throughput. Furthermore, the proposed algorithms are 107 expected to operate within the envelope of the circuit breakers 108 defined in RFC8083 [RFC8083]. 110 This document only provides the broad set of network parameters and 111 and traffic models for evaluating a new congestion control algorithm. 112 The minimal requirements for congestion control proposals is to 113 produce or present results for the test scenarios described in 114 [I-D.ietf-rmcat-eval-test] (Basic Test Cases), which also defines the 115 specifics for the test cases. Additionally, proponents may produce 116 evaluation results for the wireless test scenarios 117 [I-D.ietf-rmcat-wireless-tests]. 119 This document does not cover application-specific implications of 120 congestion control algorithms and how those could be evaluated. 121 Therefore, no quality metrics are defined for performance evaluation; 122 quality metrics and algorithms to infer those vary between media 123 types. Metrics and algorithms to assess, e.g., quality of experience 124 evolve continuously so that determining suitable choices is left for 125 future work. However, there is consensus that each congestion 126 control algorithm should be able to show that it is useful for 127 interactive video by performing analysis using a real codecs and 128 video sequences and state-of-the-art quality metrics. 130 Beyond optimizing individual metrics, real-time applications may have 131 further options to trade off performance, e.g., across multiple 132 media; refer to the RMCAT requirements 133 [I-D.ietf-rmcat-cc-requirements] document. Such trade-offs may be 134 defined in the future. 136 2. Terminology 138 The terminology defined in RTP [RFC3550], RTP Profile for Audio and 139 Video Conferences with Minimal Control [RFC3551], RTCP Extended 140 Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback 141 (RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506] 142 apply. 144 3. Metrics 146 This document specifies testing criteria for evaluating congestion 147 control algorithms for RTP media flows. Proposed algorithms are to 148 prove their performance by means of simulation and/or emulation 149 experiments for all the cases described. 151 Each experiment is expected to log every incoming and outgoing packet 152 (the RTP logging format is described in Section 3.1). The logging 153 can be done inside the application or at the endpoints using PCAP 154 (packet capture, e.g., tcpdump, wireshark). The following metrics 155 are calculated based on the information in the packet logs: 157 1. Sending rate, Receiver rate, Goodput (measured at 200ms 158 intervals) 160 2. Packets sent, Packets received 162 3. Bytes sent, bytes received 164 4. Packet delay 166 5. Packets lost, Packets discarded (from the playout or de-jitter 167 buffer) 169 6. If using, retransmission or FEC: post-repair loss 171 7. Self-Fairness and Fairness with respect to cross traffic: 172 Experiments testing a given congestion control proposal must 173 report on relative ratios of the average throughput (measured at 174 coarser time intervals) obtained by each RTP media stream. In 175 the presence of background cross-traffic such as TCP, the report 176 must also include the relative ratio between average throughput 177 of RTP media streams and cross-traffic streams. 178 During static periods of a test (i.e., when bottleneck bandwidth 179 is constant and no arrival/departure of streams), these report 180 on relative ratios serve as an indicator of how fair the RTP 181 streams share bandwidth amongst themselves and against cross- 182 traffic streams. The throughput measurement interval should be 183 set at a few values (for example, at 1s, 5s, and 20s) in order 184 to measure fairness across different time scales. 185 As a general guideline, the relative ratio between congestion 186 controlled RTP flows with the same priority level and similar 187 path RTT should be bounded between (0.333 and 3.) For example, 188 see the test scenarios described in [I-D.ietf-rmcat-eval-test]. 190 8. Convergence time: The time taken to reach a stable rate at 191 startup, after the available link capacity changes, or when new 192 flows get added to the bottleneck link. 194 9. Instability or oscillation in the sending rate: The frequency or 195 number of instances when the sending rate oscillates between an 196 high watermark level and a low watermark level, or vice-versa in 197 a defined time window. For example, the watermarks can be set 198 at 4x interval: 500 Kbps, 2 Mbps, and a time window of 500ms. 200 10. Bandwidth Utilization, defined as ratio of the instantaneous 201 sending rate to the instantaneous bottleneck capacity. This 202 metric is useful only when a congestion controlled RTP flow is 203 by itself or competing with similar cross-traffic. 205 Note that the above metrics are all objective application-independent 206 metrics. Refer to Section 3, in [I-D.ietf-netvc-testing] for 207 objective metrics for evaluating codecs. 209 From the logs the statistical measures (min, max, mean, standard 210 deviation and variance) for the whole duration or any specific part 211 of the session can be calculated. Also the metrics (sending rate, 212 receiver rate, goodput, latency) can be visualized in graphs as 213 variation over time, the measurements in the plot are at 1 second 214 intervals. Additionally, from the logs it is possible to plot the 215 histogram or CDF of packet delay. 217 3.1. RTP Log Format 219 Having a common log format simplifies running analyses across and 220 comparing different measurements. The log file should be tab or 221 comma separated containing the following details: 223 Send or receive timestamp (unix) 224 RTP payload type 225 SSRC 226 RTP sequence no 227 RTP timestamp 228 marker bit 229 payload size 231 If the congestion control implements, retransmissions or FEC, the 232 evaluation should report both packet loss (before applying error- 233 resilience) and residual packet loss (after applying error- 234 resilience). 236 4. List of Network Parameters 238 The implementors initially are encouraged to choose evaluation 239 settings from the following values: 241 4.1. One-way Propagation Delay 243 Experiments are expected to verify that the congestion control is 244 able to work across a broad range of path characteristics, also 245 including challenging situations, for example over trans-continental 246 and/or satellite links. Tests thus account for the following 247 different latencies: 249 1. Very low latency: 0-1ms 251 2. Low latency: 50ms 253 3. High latency: 150ms 255 4. Extreme latency: 300ms 257 4.2. End-to-end Loss 259 Many paths in the Internet today are largely lossless but, with 260 wireless networks and interference, towards remote regions, or in 261 scenarios featuring high/fast mobility, media flows may exhibit 262 substantial packet loss. This variety needs to be reflected 263 appropriately by the tests. 265 To model a wide range of lossy links, the experiments can choose one 266 of the following loss rates, the fractional loss is the ratio of 267 packets lost and packets sent. 269 1. no loss: 0% 271 2. 1% 273 3. 5% 275 4. 10% 277 5. 20% 279 4.3. Drop Tail Router Queue Length 281 Routers should be configured to use Drop Trail queues in the 282 experiments due to their (still) prevalent nature. Experimentation 283 with AQM schemes is encouraged but not mandatory. 285 The router queue length is measured as the time taken to drain the 286 FIFO queue. It has been noted in various discussions that the queue 287 length in the current deployed Internet varies significantly. While 288 the core backbone network has very short queue length, the home 289 gateways usually have larger queue length. Those various queue 290 lengths can be categorized in the following way: 292 1. QoS-aware (or short): 70ms 294 2. Nominal: 300-500ms 296 3. Buffer-bloated: 1000-2000ms 298 Here the size of the queue is measured in bytes or packets and to 299 convert the queue length measured in seconds to queue length in 300 bytes: 302 QueueSize (in bytes) = QueueSize (in sec) x Throughput (in bps)/8 304 4.4. Loss generation model 306 Many models for generating packet loss are available, some yield 307 correlated, others independent losses; losses can also be extracted 308 from packet traces. As a (simple) minimum loss model with minimal 309 parameterization (i.e., the loss rate), independent random losses 310 must be used in the evaluation. 312 It is known that independent loss models may reflect reality poorly 313 and hence more sophisticated loss models could be considered. 314 Suitable models for correlated losses includes the Gilbert-Elliot 315 model and losses generated by modeling a queue including its 316 (different) drop behaviors. 318 4.5. Jitter models 320 This section defines jitter models for the purposes of this document. 321 When jitter is to be applied to both the congestion controlled RTP 322 flow and any competing flow (such as a TCP competing flow), the 323 competing flow will use the jitter definition below that does not 324 allow for re-ordering of packets on the competing flow (see NR-RBPDV 325 definition below). 327 Jitter is an overloaded term in communications. It is is typically 328 used to refer to the variation of a metric (e.g., delay) with respect 329 to some reference metric (e.g., average delay or minimum delay). For 330 example, RFC 3550 jitter is computed as the smoothed difference in 331 packet arrival times relative to their respective expected arrival 332 times, which is particularly meaningful if the underlying packet 333 delay variation was caused by a Gaussian random process. 335 Because jitter is an overloaded term, we use the term Packet Delay 336 Variation (PDV) instead to describe the variation of delay of 337 individual packets in the same sense as the IETF IPPM WG has defined 338 PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has 339 defined IP Packet Delay Variation (IPDV) in their documents (e.g., 340 Y.1540). 342 Most PDV distributions in packet network systems are one-sided 343 distributions, the measurement of which with a finite number of 344 measurement samples results in one-sided histograms. In the usual 345 packet network transport case, there is typically one packet that 346 transited the network with the minimum delay; a (large) number of 347 packets transit the network within some (smaller) positive variation 348 from this minimum delay, and a (small) number of the packets transit 349 the network with delays higher than the median or average transit 350 time (these are outliers). Although infrequent, outliers can cause 351 significant deleterious operation in adaptive systems and should be 352 considered in rate adaptation designs for RTP congestion control. 354 In this section we define two different bounded PDV characteristics, 355 1) Random Bounded PDV and 2) Approximately Random Subject to No- 356 Reordering Bounded PDV. 358 The former, 1) Random Bounded PDV is presented for information only, 359 while the latter, 2) Approximately Random Subject to No-Reordering 360 Bounded PDV, must be used in the evaluation. 362 4.5.1. Random Bounded PDV (RBPDV) 364 The RBPDV probability distribution function (PDF) is specified to be 365 of some mathematically describable function which includes some 366 practical minimum and maximum discrete values suitable for testing. 367 For example, the minimum value, x_min, might be specified as the 368 minimum transit time packet and the maximum value, x_max, might be 369 defined to be two standard deviations higher than the mean. 371 Since we are typically interested in the distribution relative to the 372 mean delay packet, we define the zero mean PDV sample, z(n), to be 373 z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random 374 variable x and x_mean is the mean of x. 376 We assume here that s(n) is the original source time of packet n and 377 the post-jitter induced emission time, j(n), for packet n is: 379 j(n) = {[z(n) + x_mean] + s(n)}. 381 It follows that the separation in the post-jitter time of packets n 382 and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}. Since the first term is 383 always a positive quantity, we note that packet reordering at the 384 receiver is possible whenever the second term is greater than the 385 first. Said another way, whenever the difference in possible zero 386 mean PDV sample delays (i.e., [x_max-x_min]) exceeds the inter- 387 departure time of any two sent packets, we have the possibility of 388 packet re-ordering. 390 There are important use cases in real networks where packets can 391 become re-ordered such as in load balancing topologies and during 392 route changes. However, for the vast majority of cases there is no 393 packet re-ordering because most of the time packets follow the same 394 path. Due to this, if a packet becomes overly delayed, the packets 395 after it on that flow are also delayed. This is especially true for 396 mobile wireless links where there are per-flow queues prior to base 397 station scheduling. Owing to this important use case, we define 398 another PDV profile similar to the above, but one that does not allow 399 for re-ordering within a flow. 401 4.5.2. Approximately Random Subject to No-Reordering Bounded PDV (NR- 402 RPVD) 404 No Reordering RPDV, NR-RPVD, is defined similarly to the above with 405 one important exception. Let serial(n) be defined as the 406 serialization delay of packet n at the lowest bottleneck link rate 407 (or other appropriate rate) in a given test. Then we produce all the 408 post-jitter values for j(n) for n = 1, 2, ... N, where N is the 409 length of the source sequence s to be offset-ed. The exception can 410 be stated as follows: We revisit all j(n) beginning from index n=2, 411 and if j(n) is determined to be less than [j(n-1)+serial(n-1)], we 412 redefine j(n) to be equal to [j(n-1)+serial(n-1)] and continue for 413 all remaining n (i.e., n = 3, 4, .. N). This models the case where 414 the packet n is sent immediately after packet (n-1) at the bottleneck 415 link rate. Although this is generally the theoretical minimum in 416 that it assumes that no other packets from other flows are in-between 417 packet n and n+1 at the bottleneck link, it is a reasonable 418 assumption for per flow queuing. 420 We note that this assumption holds for some important exception 421 cases, such as packets immediately following outliers. There are a 422 multitude of software controlled elements common on end-to-end 423 Internet paths (such as firewalls, ALGs and other middleboxes) which 424 stop processing packets while servicing other functions (e.g., 425 garbage collection). Often these devices do not drop packets, but 426 rather queue them for later processing and cause many of the 427 outliers. Thus NR-RPVD models this particular use case (assuming 428 serial(n+1) is defined appropriately for the device causing the 429 outlier) and thus is believed to be important for adaptation 430 development for congestion controlled RTP streams. 432 4.5.3. Recommended distribution 434 Whether Random Bounded PDV or Approximately Random Subject to No- 435 Reordering Bounded PDV, it is recommended that z(n) is distributed 436 according to a truncated Gaussian for the above jitter models: 438 z(n) ~ |max(min(N(0, std^2), N_STD * std), -N_STD * std)| 440 where N(0, std^2) is the Gaussian distribution with zero mean and 441 standard deviation std. Recommended values: 443 o std = 5 ms 445 o N_STD = 3 447 5. Traffic Models 449 5.1. TCP traffic model 451 Long-lived TCP flows will download data throughout the session and 452 are expected to have infinite amount of data to send or receive. 453 This roughly applies, for example, when downloading software 454 distributions. 456 Each short TCP flow is modeled as a sequence of file downloads 457 interleaved with idle periods. Not all short TCP flows start at the 458 same time, i.e., some start in the ON state while others start in the 459 OFF state. 461 The short TCP flows can be modeled as follows: 30 connections start 462 simultaneously fetching small (30-50 KB) amounts of data, evenly 463 distributed. This covers the case where the short TCP flows are 464 fetching web page resources rather than video files. 466 The idle period between bursts of starting a group of TCP flows is 467 typically derived from an exponential distribution with the mean 468 value of 10 seconds. 470 [These values were picked based on the data available at 471 http://httparchive.org/interesting.php as of October 2015]. 473 Many different TCP congestion control schemes are deployed today. 474 Therefore, experimentation with a range of different schemes, 475 especially including CUBIC, is encouraged. Experiments must document 476 in detail which congestion control schemes they tested against and 477 which parameters were used. 479 5.2. RTP Video model 481 [RFC8593] describes two types of video traffic models for evaluating 482 candidate algorithms for RTP congestion control. The first model 483 statistically characterizes the behavior of a video encoder, whereas 484 the second model uses video traces. 486 Sample video test sequences are available at: [xiph-seq] and 487 [HEVC-seq]. The following two video streams are the recommended 488 minimum for testing: Foreman and FourPeople. 490 5.3. Background UDP 492 Background UDP flow is modeled as a constant bit rate (CBR) flow. It 493 will download data at a particular CBR rate for the complete session, 494 or will change to particular CBR rate at predefined intervals. The 495 inter packet interval is calculated based on the CBR and the packet 496 size (is typically set to the path MTU size, the default value can be 497 1500 bytes). 499 Note that new transport protocols such as QUIC may use UDP but, due 500 to their congestion control algorithms, will exhibit behavior 501 conceptually similar in nature to TCP flows above and can thus be 502 subsumed by the above, including the division into short- and long- 503 lived flows. As QUIC evolves independently of TCP congestion control 504 algorithms, its future congestion control should be considered as 505 competing traffic as appropriate. 507 6. Security Considerations 509 This document specifies evaluation criteria and parameters for 510 assessing and comparing the performance of congestion control 511 protocols and algorithms for real-time communication. This memo 512 itself is thus not subject to security considerations but the 513 protocols and algorithms evaluated may be. In particular, successful 514 operation under all tests defined in this document may suffice for a 515 comparative evaluation but must not be interpreted that the protocol 516 is free of risks when deployed on the Internet as briefly described 517 in the following by example. 519 Such evaluations are expected to be carried out in controlled 520 environments for limited numbers of parallel flows. As such, these 521 evaluations are by definition limited and will not be able to 522 systematically consider possible interactions or very large groups of 523 communicating nodes under all possible circumstances, so that careful 524 protocol design is advised to avoid incidentally contributing traffic 525 that could lead to unstable networks, e.g., (local) congestion 526 collapse. 528 This specification focuses on assessing the regular operation of the 529 protocols and algorithms under considerations. It does not suggest 530 checks against malicious use of the protocols -- by the sender, the 531 receiver, or intermediate parties, e.g., through faked, dropped, 532 replicated, or modified congestion signals. It is up to the protocol 533 specifications themselves to ensure that authenticity, integrity, 534 and/or plausibility of received signals are checked and the 535 appropriate actions (or non-actions) are taken. 537 7. IANA Considerations 539 There are no IANA impacts in this memo. 541 8. Contributors 543 The content and concepts within this document are a product of the 544 discussion carried out in the Design Team. 546 Michael Ramalho provided the text for the Jitter model. 548 9. Acknowledgments 550 Much of this document is derived from previous work on congestion 551 control at the IETF. 553 The authors would like to thank Harald Alvestrand, Anna Brunstrom, 554 Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde, 555 Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers O'Hanlon, Colin 556 Perkins, Michael Ramalho, Zaheduzzaman Sarker, Timothy B. 557 Terriberry, Michael Welzl, Mo Zanaty, and Xiaoqing Zhu for providing 558 valuable feedback on earlier versions of this draft. Additionally, 559 also thank the participants of the design team for their comments and 560 discussion related to the evaluation criteria. 562 10. References 564 10.1. Normative References 566 [I-D.ietf-rmcat-cc-requirements] 567 Jesup, R. and Z. Sarker, "Congestion Control Requirements 568 for Interactive Real-Time Media", draft-ietf-rmcat-cc- 569 requirements-09 (work in progress), December 2014. 571 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 572 Jacobson, "RTP: A Transport Protocol for Real-Time 573 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 574 July 2003, . 576 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 577 Video Conferences with Minimal Control", STD 65, RFC 3551, 578 DOI 10.17487/RFC3551, July 2003, 579 . 581 [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., 582 "RTP Control Protocol Extended Reports (RTCP XR)", 583 RFC 3611, DOI 10.17487/RFC3611, November 2003, 584 . 586 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 587 "Extended RTP Profile for Real-time Transport Control 588 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 589 DOI 10.17487/RFC4585, July 2006, 590 . 592 [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size 593 Real-Time Transport Control Protocol (RTCP): Opportunities 594 and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 595 2009, . 597 [RFC8083] Perkins, C. and V. Singh, "Multimedia Congestion Control: 598 Circuit Breakers for Unicast RTP Sessions", RFC 8083, 599 DOI 10.17487/RFC8083, March 2017, 600 . 602 [RFC8593] Zhu, X., Mena, S., and Z. Sarker, "Video Traffic Models 603 for RTP Congestion Control Evaluations", RFC 8593, 604 DOI 10.17487/RFC8593, May 2019, 605 . 607 10.2. Informative References 609 [HEVC-seq] 610 HEVC, "Test Sequences", 611 http://www.netlab.tkk.fi/~varun/test_sequences/ . 613 [I-D.ietf-netvc-testing] 614 Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec 615 Testing and Quality Measurement", draft-ietf-netvc- 616 testing-09 (work in progress), January 2020. 618 [I-D.ietf-rmcat-eval-test] 619 Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test 620 Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- 621 eval-test-10 (work in progress), May 2019. 623 [I-D.ietf-rmcat-wireless-tests] 624 Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and 625 M. Ramalho, "Evaluation Test Cases for Interactive Real- 626 Time Media over Wireless Networks", draft-ietf-rmcat- 627 wireless-tests-08 (work in progress), July 2019. 629 [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion 630 Control Algorithms", BCP 133, RFC 5033, 631 DOI 10.17487/RFC5033, August 2007, 632 . 634 [RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion 635 Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March 636 2008, . 638 [xiph-seq] 639 Daede, T., "Video Test Media Set", 640 https://people.xiph.org/~tdaede/sets/ . 642 Appendix A. Change Log 644 Note to the RFC-Editor: please remove this section prior to 645 publication as an RFC. 647 A.1. Changes in draft-ietf-rmcat-eval-criteria-07 649 Updated the draft according to the discussion at IETF-101. 651 o Updated the discussion on fairness. Thanks to Xiaoqing Zhu for 652 providing text. 654 o Fixed a simple loss model and provided pointers to more 655 sophisticated ones. 657 o Fixed the choice of the jitter model. 659 A.2. Changes in draft-ietf-rmcat-eval-criteria-06 661 o Updated Jitter. 663 A.3. Changes in draft-ietf-rmcat-eval-criteria-05 665 o Improved text surrounding wireless tests, video sequences, and 666 short-TCP model. 668 A.4. Changes in draft-ietf-rmcat-eval-criteria-04 670 o Removed the guidelines section, as most of the sections are now 671 covered: wireless tests, video model, etc. 673 o Improved Short TCP model based on the suggestion to use 674 httparchive.org. 676 A.5. Changes in draft-ietf-rmcat-eval-criteria-03 678 o Keep-alive version. 680 o Moved link parameters and traffic models from eval-test 682 A.6. Changes in draft-ietf-rmcat-eval-criteria-02 684 o Incorporated fairness test as a working test. 686 o Updated text on mimimum evaluation requirements. 688 A.7. Changes in draft-ietf-rmcat-eval-criteria-01 690 o Removed Appendix B. 692 o Removed Section on Evaluation Parameters. 694 A.8. Changes in draft-ietf-rmcat-eval-criteria-00 696 o Updated references. 698 o Resubmitted as WG draft. 700 A.9. Changes in draft-singh-rmcat-cc-eval-04 702 o Incorporate feedback from IETF 87, Berlin. 704 o Clarified metrics: convergence time, bandwidth utilization. 706 o Changed fairness criteria to fairness test. 708 o Added measuring pre- and post-repair loss. 710 o Added open issue of measuring video quality to appendix. 712 o clarified use of DropTail and AQM. 714 o Updated text in "Minimum Requirements for Evaluation" 716 A.10. Changes in draft-singh-rmcat-cc-eval-03 718 o Incorporate the discussion within the design team. 720 o Added a section on evaluation parameters, it describes the flow 721 and network characteristics. 723 o Added Appendix with self-fairness experiment. 725 o Changed bottleneck parameters from a proposal to an example set. 727 o 729 A.11. Changes in draft-singh-rmcat-cc-eval-02 731 o Added scenario descriptions. 733 A.12. Changes in draft-singh-rmcat-cc-eval-01 735 o Removed QoE metrics. 737 o Changed stability to steady-state. 739 o Added measuring impact against few and many flows. 741 o Added guideline for idle and data-limited periods. 743 o Added reference to TCP evaluation suite in example evaluation 744 scenarios. 746 Authors' Addresses 748 Varun Singh 749 CALLSTATS I/O Oy 750 Runeberginkatu 4c A 4 751 Helsinki 00100 752 Finland 754 Email: varun.singh@iki.fi 755 URI: https://www.callstats.io/about 756 Joerg Ott 757 Technical University of Munich 758 Faculty of Informatics 759 Boltzmannstrasse 3 760 Garching bei Muenchen, DE 85748 761 Germany 763 Email: ott@in.tum.de 765 Stefan Holmer 766 Google 767 Kungsbron 2 768 Stockholm 11122 769 Sweden 771 Email: holmer@google.com