idnits 2.17.1 draft-ietf-rmcat-eval-criteria-14.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 19, 2020) is 1499 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RMCAT WG V. Singh 3 Internet-Draft callstats.io 4 Intended status: Informational J. Ott 5 Expires: September 20, 2020 Technical University of Munich 6 S. Holmer 7 Google 8 March 19, 2020 10 Evaluating Congestion Control for Interactive Real-time Media 11 draft-ietf-rmcat-eval-criteria-14 13 Abstract 15 The Real-time Transport Protocol (RTP) is used to transmit media in 16 telephony and video conferencing applications. This document 17 describes the guidelines to evaluate new congestion control 18 algorithms for interactive point-to-point real-time media. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at https://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on September 20, 2020. 37 Copyright Notice 39 Copyright (c) 2020 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (https://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 57 3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 5 58 4. List of Network Parameters . . . . . . . . . . . . . . . . . 6 59 4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 6 60 4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 6 61 4.3. Drop Tail Router Queue Length . . . . . . . . . . . . . . 7 62 4.4. Loss generation model . . . . . . . . . . . . . . . . . . 7 63 4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 7 64 4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 8 65 4.5.2. Approximately Random Subject to No-Reordering Bounded 66 PDV (NR-RPVD) . . . . . . . . . . . . . . . . 9 67 4.5.3. Recommended distribution . . . . . . . . . . . . . . 10 68 5. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 10 69 5.1. TCP traffic model . . . . . . . . . . . . . . . . . . . . 10 70 5.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 11 71 5.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 11 72 6. Security Considerations . . . . . . . . . . . . . . . . . . . 12 73 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 74 8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12 75 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12 76 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 77 10.1. Normative References . . . . . . . . . . . . . . . . . . 13 78 10.2. Informative References . . . . . . . . . . . . . . . . . 14 79 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 15 80 A.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 15 81 A.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 15 82 A.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 15 83 A.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 15 84 A.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 15 85 A.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 15 86 A.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 16 87 A.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 16 88 A.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 16 89 A.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 16 90 A.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 16 91 A.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 17 92 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 17 94 1. Introduction 96 This memo describes the guidelines to help with evaluating new 97 congestion control algorithms for interactive point-to-point real 98 time media. The requirements for the congestion control algorithm 99 are outlined in [I-D.ietf-rmcat-cc-requirements]). This document 100 builds upon previous work at the IETF: Specifying New Congestion 101 Control Algorithms [RFC5033] and Metrics for the Evaluation of 102 Congestion Control Algorithms [RFC5166]. 104 The guidelines proposed in the document are intended to help prevent 105 a congestion collapse, promote fair capacity usage and optimize the 106 media flow's throughput. Furthermore, the proposed congestion 107 control algorithms are expected to operate within the envelope of the 108 circuit breakers defined in RFC8083 [RFC8083]. 110 This document only provides the broad set of network parameters and 111 and traffic models for evaluating a new congestion control algorithm. 112 The minimal requirements for congestion control proposals is to 113 produce or present results for the test scenarios described in 114 [I-D.ietf-rmcat-eval-test] (Basic Test Cases), which also defines the 115 specifics for the test cases. Additionally, proponents may produce 116 evaluation results for the wireless test scenarios 117 [I-D.ietf-rmcat-wireless-tests]. 119 This document does not cover application-specific implications of 120 congestion control algorithms and how those could be evaluated. 121 Therefore, no quality metrics are defined for performance evaluation; 122 quality metrics and algorithms to infer those vary between media 123 types. Metrics and algorithms to assess, e.g., quality of experience 124 evolve continuously so that determining suitable choices is left for 125 future work. However, there is consensus that each congestion 126 control algorithm should be able to show that it is useful for 127 interactive video by performing analysis using a real codecs and 128 video sequences and state-of-the-art quality metrics. 130 Beyond optimizing individual metrics, real-time applications may have 131 further options to trade off performance, e.g., across multiple 132 media; refer to the RMCAT requirements 133 [I-D.ietf-rmcat-cc-requirements] document. Such trade-offs may be 134 defined in the future. 136 2. Terminology 138 The terminology defined in RTP [RFC3550], RTP Profile for Audio and 139 Video Conferences with Minimal Control [RFC3551], RTCP Extended 140 Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback 141 (RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506] 142 apply. 144 3. Metrics 146 This document specifies testing criteria for evaluating congestion 147 control algorithms for RTP media flows. Proposed algorithms are to 148 prove their performance by means of simulation and/or emulation 149 experiments for all the cases described. 151 Each experiment is expected to log every incoming and outgoing packet 152 (the RTP logging format is described in Section 3.1). The logging 153 can be done inside the application or at the endpoints using PCAP 154 (packet capture, e.g., tcpdump [tcpdump], wireshark [wireshark]). 155 The following metrics are calculated based on the information in the 156 packet logs: 158 1. Sending rate, Receiver rate, Goodput (measured at 200ms 159 intervals) 161 2. Packets sent, Packets received 163 3. Bytes sent, bytes received 165 4. Packet delay 167 5. Packets lost, Packets discarded (from the playout or de-jitter 168 buffer) 170 6. If using, retransmission or FEC: post-repair loss 172 7. Self-Fairness and Fairness with respect to cross traffic: 173 Experiments testing a given congestion control proposal must 174 report on relative ratios of the average throughput (measured at 175 coarser time intervals) obtained by each RTP media stream. In 176 the presence of background cross-traffic such as TCP, the report 177 must also include the relative ratio between average throughput 178 of RTP media streams and cross-traffic streams. 179 During static periods of a test (i.e., when bottleneck bandwidth 180 is constant and no arrival/departure of streams), these report 181 on relative ratios serve as an indicator of how fair the RTP 182 streams share bandwidth amongst themselves and against cross- 183 traffic streams. The throughput measurement interval should be 184 set at a few values (for example, at 1s, 5s, and 20s) in order 185 to measure fairness across different time scales. 186 As a general guideline, the relative ratio between congestion 187 controlled RTP flows with the same priority level and similar 188 path RTT should be bounded between (0.333 and 3.) For example, 189 see the test scenarios described in [I-D.ietf-rmcat-eval-test]. 191 8. Convergence time: The time taken to reach a stable rate at 192 startup, after the available link capacity changes, or when new 193 flows get added to the bottleneck link. 195 9. Instability or oscillation in the sending rate: The frequency or 196 number of instances when the sending rate oscillates between an 197 high watermark level and a low watermark level, or vice-versa in 198 a defined time window. For example, the watermarks can be set 199 at 4x interval: 500 Kbps, 2 Mbps, and a time window of 500ms. 201 10. Bandwidth Utilization, defined as ratio of the instantaneous 202 sending rate to the instantaneous bottleneck capacity. This 203 metric is useful only when a congestion controlled RTP flow is 204 by itself or competing with similar cross-traffic. 206 Note that the above metrics are all objective application-independent 207 metrics. Refer to Section 3, in [I-D.ietf-netvc-testing] for 208 objective metrics for evaluating codecs. 210 From the logs the statistical measures (min, max, mean, standard 211 deviation and variance) for the whole duration or any specific part 212 of the session can be calculated. Also the metrics (sending rate, 213 receiver rate, goodput, latency) can be visualized in graphs as 214 variation over time, the measurements in the plot are at 1 second 215 intervals. Additionally, from the logs it is possible to plot the 216 histogram or CDF of packet delay. 218 3.1. RTP Log Format 220 Having a common log format simplifies running analyses across and 221 comparing different measurements. The log file should be tab or 222 comma separated containing the following details: 224 Send or receive timestamp (Unix): . -- sec.usec decimal 225 RTP payload type -- decimal 226 SSRC -- hexadecimal 227 RTP sequence no -- decimal 228 RTP timestamp -- decimal 229 marker bit 0|1 -- character 230 Payload size -- # bytes, decimal 232 Each line of the log file should be terminated with CRLF, CR, or LF 233 characters. Empty lines are disregarded. 235 If the congestion control implements retransmissions or FEC, the 236 evaluation should report both packet loss (before applying error- 237 resilience) and residual packet loss (after applying error- 238 resilience). 240 These data should suffice to compute the media-encoding independent 241 metrics described above. Use of a common log will allow simplified 242 post-processing and analysis across different implementations. 244 4. List of Network Parameters 246 The implementors initially are encouraged to choose evaluation 247 settings from the following values: 249 4.1. One-way Propagation Delay 251 Experiments are expected to verify that the congestion control is 252 able to work across a broad range of path characteristics, also 253 including challenging situations, for example over trans-continental 254 and/or satellite links. Tests thus account for the following 255 different latencies: 257 1. Very low latency: 0-1ms 259 2. Low latency: 50ms 261 3. High latency: 150ms 263 4. Extreme latency: 300ms 265 4.2. End-to-end Loss 267 Many paths in the Internet today are largely lossless but, with 268 wireless networks and interference, towards remote regions, or in 269 scenarios featuring high/fast mobility, media flows may exhibit 270 substantial packet loss. This variety needs to be reflected 271 appropriately by the tests. 273 To model a wide range of lossy links, the experiments can choose one 274 of the following loss rates, the fractional loss is the ratio of 275 packets lost and packets sent. 277 1. no loss: 0% 279 2. 1% 281 3. 5% 282 4. 10% 284 5. 20% 286 4.3. Drop Tail Router Queue Length 288 Routers should be configured to use Drop Trail queues in the 289 experiments due to their (still) prevalent nature. Experimentation 290 with AQM schemes is encouraged but not mandatory. 292 The router queue length is measured as the time taken to drain the 293 FIFO queue. It has been noted in various discussions that the queue 294 length in the current deployed Internet varies significantly. While 295 the core backbone network has very short queue length, the home 296 gateways usually have larger queue length. Those various queue 297 lengths can be categorized in the following way: 299 1. QoS-aware (or short): 70ms 301 2. Nominal: 300-500ms 303 3. Buffer-bloated: 1000-2000ms 305 Here the size of the queue is measured in bytes or packets and to 306 convert the queue length measured in seconds to queue length in 307 bytes: 309 QueueSize (in bytes) = QueueSize (in sec) x Throughput (in bps)/8 311 4.4. Loss generation model 313 Many models for generating packet loss are available, some yield 314 correlated, others independent losses; losses can also be extracted 315 from packet traces. As a (simple) minimum loss model with minimal 316 parameterization (i.e., the loss rate), independent random losses 317 must be used in the evaluation. 319 It is known that independent loss models may reflect reality poorly 320 and hence more sophisticated loss models could be considered. 321 Suitable models for correlated losses includes the Gilbert-Elliot 322 model [gilbert-elliott] and losses generated by modeling a queue 323 including its (different) drop behaviors. 325 4.5. Jitter models 327 This section defines jitter models for the purposes of this document. 328 When jitter is to be applied to both the congestion controlled RTP 329 flow and any competing flow (such as a TCP competing flow), the 330 competing flow will use the jitter definition below that does not 331 allow for re-ordering of packets on the competing flow (see NR-RBPDV 332 definition below). 334 Jitter is an overloaded term in communications. It is typically used 335 to refer to the variation of a metric (e.g., delay) with respect to 336 some reference metric (e.g., average delay or minimum delay). For 337 example, RFC 3550 jitter is computed as the smoothed difference in 338 packet arrival times relative to their respective expected arrival 339 times, which is particularly meaningful if the underlying packet 340 delay variation was caused by a Gaussian random process. 342 Because jitter is an overloaded term, we use the term Packet Delay 343 Variation (PDV) instead to describe the variation of delay of 344 individual packets in the same sense as the IETF IPPM WG has defined 345 PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has 346 defined IP Packet Delay Variation (IPDV) in their documents (e.g., 347 Y.1540). 349 Most PDV distributions in packet network systems are one-sided 350 distributions, the measurement of which with a finite number of 351 measurement samples results in one-sided histograms. In the usual 352 packet network transport case, there is typically one packet that 353 transited the network with the minimum delay; a (large) number of 354 packets transit the network within some (smaller) positive variation 355 from this minimum delay, and a (small) number of the packets transit 356 the network with delays higher than the median or average transit 357 time (these are outliers). Although infrequent, outliers can cause 358 significant deleterious operation in adaptive systems and should be 359 considered in rate adaptation designs for RTP congestion control. 361 In this section we define two different bounded PDV characteristics, 362 1) Random Bounded PDV and 2) Approximately Random Subject to No- 363 Reordering Bounded PDV. 365 The former, 1) Random Bounded PDV is presented for information only, 366 while the latter, 2) Approximately Random Subject to No-Reordering 367 Bounded PDV, must be used in the evaluation. 369 4.5.1. Random Bounded PDV (RBPDV) 371 The RBPDV probability distribution function (PDF) is specified to be 372 of some mathematically describable function which includes some 373 practical minimum and maximum discrete values suitable for testing. 374 For example, the minimum value, x_min, might be specified as the 375 minimum transit time packet and the maximum value, x_max, might be 376 defined to be two standard deviations higher than the mean. 378 Since we are typically interested in the distribution relative to the 379 mean delay packet, we define the zero mean PDV sample, z(n), to be 380 z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random 381 variable x and x_mean is the mean of x. 383 We assume here that s(n) is the original source time of packet n and 384 the post-jitter induced emission time, j(n), for packet n is: 386 j(n) = {[z(n) + x_mean] + s(n)}. 388 It follows that the separation in the post-jitter time of packets n 389 and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}. Since the first term is 390 always a positive quantity, we note that packet reordering at the 391 receiver is possible whenever the second term is greater than the 392 first. Said another way, whenever the difference in possible zero 393 mean PDV sample delays (i.e., [x_max-x_min]) exceeds the inter- 394 departure time of any two sent packets, we have the possibility of 395 packet re-ordering. 397 There are important use cases in real networks where packets can 398 become re-ordered such as in load balancing topologies and during 399 route changes. However, for the vast majority of cases there is no 400 packet re-ordering because most of the time packets follow the same 401 path. Due to this, if a packet becomes overly delayed, the packets 402 after it on that flow are also delayed. This is especially true for 403 mobile wireless links where there are per-flow queues prior to base 404 station scheduling. Owing to this important use case, we define 405 another PDV profile similar to the above, but one that does not allow 406 for re-ordering within a flow. 408 4.5.2. Approximately Random Subject to No-Reordering Bounded PDV (NR- 409 RPVD) 411 No Reordering RPDV, NR-RPVD, is defined similarly to the above with 412 one important exception. Let serial(n) be defined as the 413 serialization delay of packet n at the lowest bottleneck link rate 414 (or other appropriate rate) in a given test. Then we produce all the 415 post-jitter values for j(n) for n = 1, 2, ... N, where N is the 416 length of the source sequence s to be offset-ed. The exception can 417 be stated as follows: We revisit all j(n) beginning from index n=2, 418 and if j(n) is determined to be less than [j(n-1)+serial(n-1)], we 419 redefine j(n) to be equal to [j(n-1)+serial(n-1)] and continue for 420 all remaining n (i.e., n = 3, 4, .. N). This models the case where 421 the packet n is sent immediately after packet (n-1) at the bottleneck 422 link rate. Although this is generally the theoretical minimum in 423 that it assumes that no other packets from other flows are in-between 424 packet n and n+1 at the bottleneck link, it is a reasonable 425 assumption for per flow queuing. 427 We note that this assumption holds for some important exception 428 cases, such as packets immediately following outliers. There are a 429 multitude of software controlled elements common on end-to-end 430 Internet paths (such as firewalls, ALGs and other middleboxes) which 431 stop processing packets while servicing other functions (e.g., 432 garbage collection). Often these devices do not drop packets, but 433 rather queue them for later processing and cause many of the 434 outliers. Thus NR-RPVD models this particular use case (assuming 435 serial(n+1) is defined appropriately for the device causing the 436 outlier) and thus is believed to be important for adaptation 437 development for congestion controlled RTP streams. 439 4.5.3. Recommended distribution 441 Whether Random Bounded PDV or Approximately Random Subject to No- 442 Reordering Bounded PDV, it is recommended that z(n) is distributed 443 according to a truncated Gaussian for the above jitter models: 445 z(n) ~ |max(min(N(0, std^2), N_STD * std), -N_STD * std)| 447 where N(0, std^2) is the Gaussian distribution with zero mean and 448 standard deviation std. Recommended values: 450 o std = 5 ms 452 o N_STD = 3 454 5. Traffic Models 456 5.1. TCP traffic model 458 Long-lived TCP flows will download data throughout the session and 459 are expected to have infinite amount of data to send or receive. 460 This roughly applies, for example, when downloading software 461 distributions. 463 Each short TCP flow is modeled as a sequence of file downloads 464 interleaved with idle periods. Not all short TCP flows start at the 465 same time, i.e., some start in the ON state while others start in the 466 OFF state. 468 The short TCP flows can be modeled as follows: 30 connections start 469 simultaneously fetching small (30-50 KB) amounts of data, evenly 470 distributed. This covers the case where the short TCP flows are 471 fetching web page resources rather than video files. 473 The idle period between bursts of starting a group of TCP flows is 474 typically derived from an exponential distribution with the mean 475 value of 10 seconds. 477 [These values were picked based on the data available at 478 http://httparchive.org/interesting.php as of October 2015]. 480 Many different TCP congestion control schemes are deployed today. 481 Therefore, experimentation with a range of different schemes, 482 especially including CUBIC, is encouraged. Experiments must document 483 in detail which congestion control schemes they tested against and 484 which parameters were used. 486 5.2. RTP Video model 488 [RFC8593] describes two types of video traffic models for evaluating 489 candidate algorithms for RTP congestion control. The first model 490 statistically characterizes the behavior of a video encoder, whereas 491 the second model uses video traces. 493 Sample video test sequences are available at [xiph-seq]. The 494 following two video streams are the recommended minimum for testing: 495 Foreman (CIF sequence) and FourPeople (720p); both come as raw video 496 data to be encoded dynamically. As these video sequences are short 497 (300 and 600 frames, respectively, they shall be stitched together 498 repeatedly until the desired length is reached. 500 5.3. Background UDP 502 Background UDP flow is modeled as a constant bit rate (CBR) flow. It 503 will download data at a particular CBR rate for the complete session, 504 or will change to particular CBR rate at predefined intervals. The 505 inter packet interval is calculated based on the CBR and the packet 506 size (is typically set to the path MTU size, the default value can be 507 1500 bytes). 509 Note that new transport protocols such as QUIC may use UDP but, due 510 to their congestion control algorithms, will exhibit behavior 511 conceptually similar in nature to TCP flows above and can thus be 512 subsumed by the above, including the division into short- and long- 513 lived flows. As QUIC evolves independently of TCP congestion control 514 algorithms, its future congestion control should be considered as 515 competing traffic as appropriate. 517 6. Security Considerations 519 This document specifies evaluation criteria and parameters for 520 assessing and comparing the performance of congestion control 521 protocols and algorithms for real-time communication. This memo 522 itself is thus not subject to security considerations but the 523 protocols and algorithms evaluated may be. In particular, successful 524 operation under all tests defined in this document may suffice for a 525 comparative evaluation but must not be interpreted that the protocol 526 is free of risks when deployed on the Internet as briefly described 527 in the following by example. 529 Such evaluations are expected to be carried out in controlled 530 environments for limited numbers of parallel flows. As such, these 531 evaluations are by definition limited and will not be able to 532 systematically consider possible interactions or very large groups of 533 communicating nodes under all possible circumstances, so that careful 534 protocol design is advised to avoid incidentally contributing traffic 535 that could lead to unstable networks, e.g., (local) congestion 536 collapse. 538 This specification focuses on assessing the regular operation of the 539 protocols and algorithms under considerations. It does not suggest 540 checks against malicious use of the protocols -- by the sender, the 541 receiver, or intermediate parties, e.g., through faked, dropped, 542 replicated, or modified congestion signals. It is up to the protocol 543 specifications themselves to ensure that authenticity, integrity, 544 and/or plausibility of received signals are checked and the 545 appropriate actions (or non-actions) are taken. 547 7. IANA Considerations 549 There are no IANA impacts in this memo. 551 8. Contributors 553 The content and concepts within this document are a product of the 554 discussion carried out in the Design Team. 556 Michael Ramalho provided the text for the Jitter model. 558 9. Acknowledgments 560 Much of this document is derived from previous work on congestion 561 control at the IETF. 563 The authors would like to thank Harald Alvestrand, Anna Brunstrom, 564 Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde, 565 Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers O'Hanlon, Colin 566 Perkins, Michael Ramalho, Zaheduzzaman Sarker, Timothy B. 567 Terriberry, Michael Welzl, Mo Zanaty, and Xiaoqing Zhu for providing 568 valuable feedback on earlier versions of this draft. Additionally, 569 also thank the participants of the design team for their comments and 570 discussion related to the evaluation criteria. 572 10. References 574 10.1. Normative References 576 [I-D.ietf-rmcat-cc-requirements] 577 Jesup, R. and Z. Sarker, "Congestion Control Requirements 578 for Interactive Real-Time Media", draft-ietf-rmcat-cc- 579 requirements-09 (work in progress), December 2014. 581 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 582 Jacobson, "RTP: A Transport Protocol for Real-Time 583 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 584 July 2003, . 586 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 587 Video Conferences with Minimal Control", STD 65, RFC 3551, 588 DOI 10.17487/RFC3551, July 2003, 589 . 591 [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., 592 "RTP Control Protocol Extended Reports (RTCP XR)", 593 RFC 3611, DOI 10.17487/RFC3611, November 2003, 594 . 596 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 597 "Extended RTP Profile for Real-time Transport Control 598 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 599 DOI 10.17487/RFC4585, July 2006, 600 . 602 [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size 603 Real-Time Transport Control Protocol (RTCP): Opportunities 604 and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 605 2009, . 607 [RFC8083] Perkins, C. and V. Singh, "Multimedia Congestion Control: 608 Circuit Breakers for Unicast RTP Sessions", RFC 8083, 609 DOI 10.17487/RFC8083, March 2017, 610 . 612 [RFC8593] Zhu, X., Mena, S., and Z. Sarker, "Video Traffic Models 613 for RTP Congestion Control Evaluations", RFC 8593, 614 DOI 10.17487/RFC8593, May 2019, 615 . 617 10.2. Informative References 619 [gilbert-elliott] 620 Hasslinger, G. and O. Hohlfeld, "The Gilbert-Elliott Model 621 for Packet Loss in Real Time Services on the Internet", 622 14th GI/ITG Conference - Measurement, Modelling and 623 Evalutation of Computer and Communication Systems , 3 624 2008. 626 [I-D.ietf-netvc-testing] 627 Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec 628 Testing and Quality Measurement", draft-ietf-netvc- 629 testing-09 (work in progress), January 2020. 631 [I-D.ietf-rmcat-eval-test] 632 Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test 633 Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- 634 eval-test-10 (work in progress), May 2019. 636 [I-D.ietf-rmcat-wireless-tests] 637 Sarker, Z., Zhu, X., and J. Fu, "Evaluation Test Cases for 638 Interactive Real-Time Media over Wireless Networks", 639 draft-ietf-rmcat-wireless-tests-11 (work in progress), 640 March 2020. 642 [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion 643 Control Algorithms", BCP 133, RFC 5033, 644 DOI 10.17487/RFC5033, August 2007, 645 . 647 [RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion 648 Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March 649 2008, . 651 [tcpdump] "Homepage of tcpdump and libpcap", 652 https://www.tcpdump.org/index.html . 654 [wireshark] 655 "Homepage of Wireshark", https://www.wireshark.org . 657 [xiph-seq] 658 Daede, T., "Video Test Media Set", 659 https://media.xiph.org/video/derf/ . 661 Appendix A. Change Log 663 Note to the RFC-Editor: please remove this section prior to 664 publication as an RFC. 666 A.1. Changes in draft-ietf-rmcat-eval-criteria-07 668 Updated the draft according to the discussion at IETF-101. 670 o Updated the discussion on fairness. Thanks to Xiaoqing Zhu for 671 providing text. 673 o Fixed a simple loss model and provided pointers to more 674 sophisticated ones. 676 o Fixed the choice of the jitter model. 678 A.2. Changes in draft-ietf-rmcat-eval-criteria-06 680 o Updated Jitter. 682 A.3. Changes in draft-ietf-rmcat-eval-criteria-05 684 o Improved text surrounding wireless tests, video sequences, and 685 short-TCP model. 687 A.4. Changes in draft-ietf-rmcat-eval-criteria-04 689 o Removed the guidelines section, as most of the sections are now 690 covered: wireless tests, video model, etc. 692 o Improved Short TCP model based on the suggestion to use 693 httparchive.org. 695 A.5. Changes in draft-ietf-rmcat-eval-criteria-03 697 o Keep-alive version. 699 o Moved link parameters and traffic models from eval-test 701 A.6. Changes in draft-ietf-rmcat-eval-criteria-02 703 o Incorporated fairness test as a working test. 705 o Updated text on mimimum evaluation requirements. 707 A.7. Changes in draft-ietf-rmcat-eval-criteria-01 709 o Removed Appendix B. 711 o Removed Section on Evaluation Parameters. 713 A.8. Changes in draft-ietf-rmcat-eval-criteria-00 715 o Updated references. 717 o Resubmitted as WG draft. 719 A.9. Changes in draft-singh-rmcat-cc-eval-04 721 o Incorporate feedback from IETF 87, Berlin. 723 o Clarified metrics: convergence time, bandwidth utilization. 725 o Changed fairness criteria to fairness test. 727 o Added measuring pre- and post-repair loss. 729 o Added open issue of measuring video quality to appendix. 731 o clarified use of DropTail and AQM. 733 o Updated text in "Minimum Requirements for Evaluation" 735 A.10. Changes in draft-singh-rmcat-cc-eval-03 737 o Incorporate the discussion within the design team. 739 o Added a section on evaluation parameters, it describes the flow 740 and network characteristics. 742 o Added Appendix with self-fairness experiment. 744 o Changed bottleneck parameters from a proposal to an example set. 746 o 748 A.11. Changes in draft-singh-rmcat-cc-eval-02 750 o Added scenario descriptions. 752 A.12. Changes in draft-singh-rmcat-cc-eval-01 754 o Removed QoE metrics. 756 o Changed stability to steady-state. 758 o Added measuring impact against few and many flows. 760 o Added guideline for idle and data-limited periods. 762 o Added reference to TCP evaluation suite in example evaluation 763 scenarios. 765 Authors' Addresses 767 Varun Singh 768 CALLSTATS I/O Oy 769 Runeberginkatu 4c A 4 770 Helsinki 00100 771 Finland 773 Email: varun.singh@iki.fi 774 URI: https://www.callstats.io/about 776 Joerg Ott 777 Technical University of Munich 778 Faculty of Informatics 779 Boltzmannstrasse 3 780 Garching bei Muenchen, DE 85748 781 Germany 783 Email: ott@in.tum.de 785 Stefan Holmer 786 Google 787 Kungsbron 2 788 Stockholm 11122 789 Sweden 791 Email: holmer@google.com