idnits 2.17.1 draft-ietf-rmcat-eval-criteria-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 203: '...ements. The log file SHOULD be tab or...' RFC 2119 keyword, line 264: '... Routers SHOULD be configured to use...' RFC 2119 keyword, line 460: '...encouraged. Experiments MUST document...' RFC 2119 keyword, line 489: '...ngestion control SHOULD be considered ...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 2, 2019) is 1752 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC5681' is defined on line 623, but no explicit reference was found in the text == Unused Reference: 'SA4-LR' is defined on line 627, but no explicit reference was found in the text == Unused Reference: 'TCP-eval-suite' is defined on line 630, but no explicit reference was found in the text == Outdated reference: A later version (-11) exists of draft-ietf-rmcat-wireless-tests-07 == Outdated reference: A later version (-09) exists of draft-ietf-netvc-testing-08 Summary: 1 error (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RMCAT WG V. Singh 3 Internet-Draft callstats.io 4 Intended status: Informational J. Ott 5 Expires: January 3, 2020 Technical University of Munich 6 S. Holmer 7 Google 8 July 2, 2019 10 Evaluating Congestion Control for Interactive Real-time Media 11 draft-ietf-rmcat-eval-criteria-09 13 Abstract 15 The Real-time Transport Protocol (RTP) is used to transmit media in 16 telephony and video conferencing applications. This document 17 describes the guidelines to evaluate new congestion control 18 algorithms for interactive point-to-point real-time media. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at https://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on January 3, 2020. 37 Copyright Notice 39 Copyright (c) 2019 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (https://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 57 3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 5 58 4. List of Network Parameters . . . . . . . . . . . . . . . . . 5 59 4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 5 60 4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 6 61 4.3. Drop Tail Router Queue Length . . . . . . . . . . . . . . 6 62 4.4. Loss generation model . . . . . . . . . . . . . . . . . . 7 63 4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 7 64 4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 8 65 4.5.2. Approximately Random Subject to No-Reordering Bounded 66 PDV (NR-RPVD) . . . . . . . . . . . . . . . . 9 67 4.5.3. Recommended distribution . . . . . . . . . . . . . . 9 68 5. WiFi or Cellular Links . . . . . . . . . . . . . . . . . . . 10 69 6. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 10 70 6.1. TCP traffic model . . . . . . . . . . . . . . . . . . . . 10 71 6.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 10 72 6.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 11 73 7. Security Considerations . . . . . . . . . . . . . . . . . . . 11 74 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 75 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12 76 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12 77 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 78 11.1. Normative References . . . . . . . . . . . . . . . . . . 12 79 11.2. Informative References . . . . . . . . . . . . . . . . . 13 80 Appendix A. Application Trade-off . . . . . . . . . . . . . . . 14 81 A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 14 82 Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 14 83 B.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 15 84 B.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 15 85 B.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 15 86 B.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 15 87 B.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 15 88 B.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 15 89 B.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 15 90 B.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 16 91 B.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 16 92 B.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 16 93 B.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 16 94 B.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 16 95 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 17 97 1. Introduction 99 This memo describes the guidelines to help with evaluating new 100 congestion control algorithms for interactive point-to-point real 101 time media. The requirements for the congestion control algorithm 102 are outlined in [I-D.ietf-rmcat-cc-requirements]). This document 103 builds upon previous work at the IETF: Specifying New Congestion 104 Control Algorithms [RFC5033] and Metrics for the Evaluation of 105 Congestion Control Algorithms [RFC5166]. 107 The guidelines proposed in the document are intended to help prevent 108 a congestion collapse, promote fair capacity usage and optimize the 109 media flow's throughput. Furthermore, the proposed algorithms are 110 expected to operate within the envelope of the circuit breakers 111 defined in RFC8083 [RFC8083]. 113 This document only provides broad-level criteria for evaluating a new 114 congestion control algorithm. The minimal requirement for congestion 115 control proposals is to produce or present results for the test 116 scenarios described in [I-D.ietf-rmcat-eval-test] (Basic Test Cases). 117 Additionally, proponents may produce evaluation results for the 118 wireless test scenarios [I-D.ietf-rmcat-wireless-tests]. 120 2. Terminology 122 The terminology defined in RTP [RFC3550], RTP Profile for Audio and 123 Video Conferences with Minimal Control [RFC3551], RTCP Extended 124 Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback 125 (RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506] 126 apply. 128 3. Metrics 130 This document specifies testing criteria for evaluating congestion 131 control algorithms for RTP media flows. Proposed algorithms are to 132 prove their performance by means of simulation and/or emulation 133 experiments for all the cases described. 135 Each experiment is expected to log every incoming and outgoing packet 136 (the RTP logging format is described in Section 3.1). The logging 137 can be done inside the application or at the endpoints using PCAP 138 (packet capture, e.g., tcpdump, wireshark). The following are 139 calculated based on the information in the packet logs: 141 1. Sending rate, Receiver rate, Goodput (measured at 200ms 142 intervals) 144 2. Packets sent, Packets received 145 3. Bytes sent, bytes received 147 4. Packet delay 149 5. Packets lost, Packets discarded (from the playout or de-jitter 150 buffer) 152 6. If using, retransmission or FEC: post-repair loss 154 7. Self-Fairness and Fairness with respect to cross traffic: 155 Experiments testing a given congestion control proposal must 156 report on relative ratios of the average throughput (measured at 157 coarser time intervals) obtained by each RTP media stream. In 158 the presence of background cross-traffic such as TCP, the report 159 must also include the relative ratio between average throughput 160 of RTP media streams and cross-traffic streams. 161 During static periods of a test (i.e., when bottleneck bandwidth 162 is constant and no arrival/departure of streams), these report 163 on relative ratios serve as an indicator of how fair the RTP 164 streams share bandwidth amongst themselves and against cross- 165 traffic streams. The throughput measurement interval should be 166 set at a few values (for example, at 1s, 5s, and 20s) in order 167 to measure fairness across different time scales. 168 As a general guideline, the relative ratio between congestion 169 controlled RTP flows with the same priority level and similar 170 path RTT should be bounded between (0.333 and 3.) For example, 171 see the test scenarios described in [I-D.ietf-rmcat-eval-test]. 173 8. Convergence time: The time taken to reach a stable rate at 174 startup, after the available link capacity changes, or when new 175 flows get added to the bottleneck link. 177 9. Instability or oscillation in the sending rate: The frequency or 178 number of instances when the sending rate oscillates between an 179 high watermark level and a low watermark level, or vice-versa in 180 a defined time window. For example, the watermarks can be set 181 at 4x interval: 500 Kbps, 2 Mbps, and a time window of 500ms. 183 10. Bandwidth Utilization, defined as ratio of the instantaneous 184 sending rate to the instantaneous bottleneck capacity. This 185 metric is useful only when a congestion controlled RTP flow is 186 by itself or competing with similar cross-traffic. 188 Note that the above metrics are all objective application-independent 189 metrics. Refer to Section 3, in [I-D.ietf-netvc-testing] for 190 objective metrics for evaluating codecs. 192 From the logs the statistical measures (min, max, mean, standard 193 deviation and variance) for the whole duration or any specific part 194 of the session can be calculated. Also the metrics (sending rate, 195 receiver rate, goodput, latency) can be visualized in graphs as 196 variation over time, the measurements in the plot are at 1 second 197 intervals. Additionally, from the logs it is possible to plot the 198 histogram or CDF of packet delay. 200 3.1. RTP Log Format 202 Having a common log format simplifies running analyses across and 203 comparing different measurements. The log file SHOULD be tab or 204 comma separated containing the following details: 206 Send or receive timestamp (unix) 207 RTP payload type 208 SSRC 209 RTP sequence no 210 RTP timestamp 211 marker bit 212 payload size 214 If the congestion control implements, retransmissions or FEC, the 215 evaluation should report both packet loss (before applying error- 216 resilience) and residual packet loss (after applying error- 217 resilience). 219 4. List of Network Parameters 221 The implementors initially are encouraged to choose evaluation 222 settings from the following values: 224 4.1. One-way Propagation Delay 226 Experiments are expected to verify that the congestion control is 227 able to work across a broad range of path characteristics, also 228 including challenging situations, for example over trans-continental 229 and/or satellite links. Tests thus account for the following 230 different latencies: 232 1. Very low latency: 0-1ms 234 2. Low latency: 50ms 236 3. High latency: 150ms 238 4. Extreme latency: 300ms 240 4.2. End-to-end Loss 242 Many paths in the Internet today are largely lossless but, with 243 wireless networks and interference, towards remote regions, or in 244 scenarios featuring high/fast mobility, media flows may exhibit 245 substantial packet loss. This variety needs to be reflected 246 appropriately by the tests. 248 To model a wide range of lossy links, the experiments can choose one 249 of the following loss rates, the fractional loss is the ratio of 250 packets lost and packets sent. 252 1. no loss: 0% 254 2. 1% 256 3. 5% 258 4. 10% 260 5. 20% 262 4.3. Drop Tail Router Queue Length 264 Routers SHOULD be configured to use Drop Trail queues in the 265 experiments due to their (still) prevalent nature. Experimentation 266 with AQM schemes is encouraged but not mandatory. 268 The router queue length is measured as the time taken to drain the 269 FIFO queue. It has been noted in various discussions that the queue 270 length in the current deployed Internet varies significantly. While 271 the core backbone network has very short queue length, the home 272 gateways usually have larger queue length. Those various queue 273 lengths can be categorized in the following way: 275 1. QoS-aware (or short): 70ms 277 2. Nominal: 300-500ms 279 3. Buffer-bloated: 1000-2000ms 281 Here the size of the queue is measured in bytes or packets and to 282 convert the queue length measured in seconds to queue length in 283 bytes: 285 QueueSize (in bytes) = QueueSize (in sec) x Throughput (in bps)/8 287 4.4. Loss generation model 289 Many models for generating packet loss are available, some yield 290 correlated, others independent losses; losses can also be extracted 291 from packet traces. As a (simple) minimum loss model with minimal 292 parameterization (i.e., the loss rate), independent random losses 293 must be used in the evaluation. 295 It is known that independent loss models may reflect reality poorly 296 and hence more sophisticated loss models could be considered. 297 Suitable models for correlated losses includes the Gilbert-Elliot 298 model and losses generated by modeling a queue including its 299 (different) drop behaviors. 301 4.5. Jitter models 303 This section defines jitter models for the purposes of this document. 304 When jitter is to be applied to both the congestion controlled RTP 305 flow and any competing flow (such as a TCP competing flow), the 306 competing flow will use the jitter definition below that does not 307 allow for re-ordering of packets on the competing flow (see NR-RBPDV 308 definition below). 310 Jitter is an overloaded term in communications. Its meaning is 311 typically associated with the variation of a metric (e.g., delay) 312 with respect to some reference metric (e.g., average delay or minimum 313 delay). For example, RFC 3550 jitter is a smoothed estimate of 314 jitter which is particularly meaningful if the underlying packet 315 delay variation was caused by a Gaussian random process. 317 Because jitter is an overloaded term, we instead use the term Packet 318 Delay Variation (PDV) to describe the variation of delay of 319 individual packets in the same sense as the IETF IPPM WG has defined 320 PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has 321 defined IP Packet Delay Variation (IPDV) in their documents (e.g., 322 Y.1540). 324 Most PDV distributions in packet network systems are one-sided 325 distributions (the measurement of which with a finite number of 326 measurement samples result in one-sided histograms). In the usual 327 packet network transport case there is typically one packet that 328 transited the network with the minimum delay, then a majority of 329 packets also transit the system within some variation from this 330 minimum delay, and then a minority of the packets transit the network 331 with delays higher than the median or average transit time (these are 332 outliers). Although infrequent, outliers can cause significant 333 deleterious operation in adaptive systems and should be considered in 334 rate adaptation designs for RTP congestion control. 336 In this section we define two different bounded PDV characteristics, 337 1) Random Bounded PDV and 2) Approximately Random Subject to No- 338 Reordering Bounded PDV. 340 The former, 1) Random Bounded PDV is presented for information only, 341 while the latter, 2) Approximately Random Subject to No-Reordering 342 Bounded PDV, must be used in the evaluation. 344 4.5.1. Random Bounded PDV (RBPDV) 346 The RBPDV probability distribution function (PDF) is specified to be 347 of some mathematically describable function which includes some 348 practical minimum and maximum discrete values suitable for testing. 349 For example, the minimum value, x_min, might be specified as the 350 minimum transit time packet and the maximum value, x_max, might be 351 defined to be two standard deviations higher than the mean. 353 Since we are typically interested in the distribution relative to the 354 mean delay packet, we define the zero mean PDV sample, z(n), to be 355 z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random 356 variable x and x_mean is the mean of x. 358 We assume here that s(n) is the original source time of packet n and 359 the post-jitter induced emission time, j(n), for packet n is j(n) = 360 {[z(n) + x_mean] + s(n)}. It follows that the separation in the post- 361 jitter time of packets n and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}. 362 Since the first term is always a positive quantity, we note that 363 packet reordering at the receiver is possible whenever the second 364 term is greater than the first. Said another way, whenever the 365 difference in possible zero mean PDV sample delays (i.e., [x_max- 366 x_min]) exceeds the inter-departure time of any two sent packets, we 367 have the possibility of packet re-ordering. 369 There are important use cases in real networks where packets can 370 become re-ordered such as in load balancing topologies and during 371 route changes. However, for the vast majority of cases there is no 372 packet re-ordering because most of the time packets follow the same 373 path. Due to this, if a packet becomes overly delayed, the packets 374 after it on that flow are also delayed. This is especially true for 375 mobile wireless links where there are per-flow queues prior to base 376 station scheduling. Owing to this important use case, we define 377 another PDV profile similar to the above, but one that does not allow 378 for re-ordering within a flow. 380 4.5.2. Approximately Random Subject to No-Reordering Bounded PDV (NR- 381 RPVD) 383 No Reordering RPDV, NR-RPVD, is defined similarly to the above with 384 one important exception. Let serial(n) be defined as the 385 serialization delay of packet n at the lowest bottleneck link rate 386 (or other appropriate rate) in a given test. Then we produce all the 387 post-jitter values for j(n) for n = 1, 2, ... N, where N is the 388 length of the source sequence s to be offset-ed. The exception can 389 be stated as follows: We revisit all j(n) beginning from index n=2, 390 and if j(n) is determined to be less than [j(n-1)+serial(n-1)], we 391 redefine j(n) to be equal to [j(n-1)+serial(n-1)] and continue for 392 all remaining n (i.e., n = 3, 4, .. N). This models the case where 393 the packet n is sent immediately after packet (n-1) at the bottleneck 394 link rate. Although this is generally the theoretical minimum in 395 that it assumes that no other packets from other flows are in-between 396 packet n and n+1 at the bottleneck link, it is a reasonable 397 assumption for per flow queuing. 399 We note that this assumption holds for some important exception 400 cases, such as packets immediately following outliers. There are a 401 multitude of software controlled elements common on end-to-end 402 Internet paths (such as firewalls, ALGs and other middleboxes) which 403 stop processing packets while servicing other functions (e.g., 404 garbage collection). Often these devices do not drop packets, but 405 rather queue them for later processing and cause many of the 406 outliers. Thus NR-RPVD models this particular use case (assuming 407 serial(n+1) is defined appropriately for the device causing the 408 outlier) and thus is believed to be important for adaptation 409 development for congestion controlled RTP streams. 411 4.5.3. Recommended distribution 413 Whether Random Bounded PDV or Approximately Random Subject to No- 414 Reordering Bounded PDV, it is recommended that z(n) is distributed 415 according to a truncated Gaussian for the above jitter models: 417 z(n) ~ |max(min(N(0, std^2), N_STD * std), -N_STD * std)| 419 where N(0, std^2) is the Gaussian distribution with zero mean and 420 standard deviation std. Recommended values: 422 o std = 5 ms 424 o N_STD = 3 426 5. WiFi or Cellular Links 428 [I-D.ietf-rmcat-wireless-tests] describes the test cases to simulate 429 networks with wireless links. The document describes mechanism to 430 simulate both cellular and WiFi networks. 432 6. Traffic Models 434 6.1. TCP traffic model 436 Long-lived TCP flows will download data throughout the session and 437 are expected to have infinite amount of data to send or receive. 438 This roughly applies, for example, when downloading software 439 distributions. 441 Each short TCP flow is modeled as a sequence of file downloads 442 interleaved with idle periods. Not all short TCP flows start at the 443 same time, i.e., some start in the ON state while others start in the 444 OFF state. 446 The short TCP flows can be modeled as follows: 30 connections start 447 simultaneously fetching small (30-50 KB) amounts of data. This 448 covers the case where the short TCP flows are not fetching a video 449 file. 451 The idle period between bursts of starting a group of TCP flows is 452 typically derived from an exponential distribution with the mean 453 value of 10 seconds. 455 [These values were picked based on the data available at 456 http://httparchive.org/interesting.php as of October 2015]. 458 Many different TCP congestion control schemes are deployed today. 459 Therefore, experimentation with a range of different schemes, 460 especially including CUBIC, is encouraged. Experiments MUST document 461 in detail which congestion control schemes they tested against and 462 which parameters were used. 464 6.2. RTP Video model 466 [I-D.ietf-rmcat-video-traffic-model] describes two types of video 467 traffic models for evaluating candidate algorithms for RTP congestion 468 control. The first model statistically characterizes the behavior of 469 a video encoder. Whereas the second model uses video traces. 471 For example, test sequences are available at: [xiph-seq] and 472 [HEVC-seq]. The currently chosen video streams are: Foreman and 473 FourPeople. 475 6.3. Background UDP 477 Background UDP flow is modeled as a constant bit rate (CBR) flow. It 478 will download data at a particular CBR rate for the complete session, 479 or will change to particular CBR rate at predefined intervals. The 480 inter packet interval is calculated based on the CBR and the packet 481 size (is typically set to the path MTU size, the default value can be 482 1500 bytes). 484 Note that new transport protocols such as QUIC may use UDP but, due 485 to their congestion control algorithms, will exhibit behavior 486 conceptually similar in nature to TCP flows above and can thus be 487 subsumed by the above, including the division into short- and long- 488 lived flows. As QUIC evolves independently of TCP congestion control 489 algorithms, its future congestion control SHOULD be considered as 490 competing traffic as appropriate. 492 7. Security Considerations 494 This document specifies evaluation criteria and parameters for 495 assessing and comparing the performance of congestion control 496 protocola and algorithm for real-time communication. This memo 497 itself is thus not subject to security considerations but the 498 protocols and algorithms evaluated may be. In particular, successful 499 operation under all tests defined in this document may suffice for a 500 comparative evaluation but must not be interpreted that the protocol 501 is free of risks when deployed on the Internet as briefly described 502 in the following by example. 504 Such evaluations are expected to be carried out in controlled 505 environments for limited numbers of parallel flows. As such, these 506 evaluations are by definition limited and will not be able to 507 systematically consider possible interactions or very large groups of 508 communicating nodes under all possible circumstances, so that careful 509 protocol design is advised to avoid incidentally contributing traffic 510 that could lead to unstable networks, e.g., (local) congestion 511 collapse. 513 This specification focuses on assessing the regular operation of the 514 protocols and algorithms under considerations. It does not suggest 515 checks against malicious use of the protocols -- by the sender, the 516 receiver, or intermediate parties, e.g., through faked, dropped, 517 replicated, or modified congestion signals. It is up to the protocol 518 specifications themselves to ensure that authenticity, integrity, 519 and/or plausibility of received signals are checked and the 520 appropriate actions (or non-actions) are taken. 522 8. IANA Considerations 524 There are no IANA impacts in this memo. 526 9. Contributors 528 The content and concepts within this document are a product of the 529 discussion carried out in the Design Team. 531 Michael Ramalho provided the text for the Jitter model. 533 10. Acknowledgments 535 Much of this document is derived from previous work on congestion 536 control at the IETF. 538 The authors would like to thank Harald Alvestrand, Anna Brunstrom, 539 Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde, 540 Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers O'Hanlon, Colin 541 Perkins, Michael Ramalho, Zaheduzzaman Sarker, Timothy B. 542 Terriberry, Michael Welzl, Mo Zanaty, and Xiaoqing Zhu for providing 543 valuable feedback on earlier versions of this draft. Additionally, 544 also thank the participants of the design team for their comments and 545 discussion related to the evaluation criteria. 547 11. References 549 11.1. Normative References 551 [I-D.ietf-rmcat-cc-requirements] 552 Jesup, R. and Z. Sarker, "Congestion Control Requirements 553 for Interactive Real-Time Media", draft-ietf-rmcat-cc- 554 requirements-09 (work in progress), December 2014. 556 [I-D.ietf-rmcat-wireless-tests] 557 Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and 558 M. Ramalho, "Evaluation Test Cases for Interactive Real- 559 Time Media over Wireless Networks", draft-ietf-rmcat- 560 wireless-tests-07 (work in progress), July 2019. 562 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 563 Jacobson, "RTP: A Transport Protocol for Real-Time 564 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 565 July 2003, . 567 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 568 Video Conferences with Minimal Control", STD 65, RFC 3551, 569 DOI 10.17487/RFC3551, July 2003, 570 . 572 [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., 573 "RTP Control Protocol Extended Reports (RTCP XR)", 574 RFC 3611, DOI 10.17487/RFC3611, November 2003, 575 . 577 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 578 "Extended RTP Profile for Real-time Transport Control 579 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 580 DOI 10.17487/RFC4585, July 2006, 581 . 583 [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size 584 Real-Time Transport Control Protocol (RTCP): Opportunities 585 and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 586 2009, . 588 [RFC8083] Perkins, C. and V. Singh, "Multimedia Congestion Control: 589 Circuit Breakers for Unicast RTP Sessions", RFC 8083, 590 DOI 10.17487/RFC8083, March 2017, 591 . 593 11.2. Informative References 595 [HEVC-seq] 596 HEVC, "Test Sequences", 597 http://www.netlab.tkk.fi/~varun/test_sequences/ . 599 [I-D.ietf-netvc-testing] 600 Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec 601 Testing and Quality Measurement", draft-ietf-netvc- 602 testing-08 (work in progress), January 2019. 604 [I-D.ietf-rmcat-eval-test] 605 Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test 606 Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- 607 eval-test-10 (work in progress), May 2019. 609 [I-D.ietf-rmcat-video-traffic-model] 610 Zhu, X., Cruz, S., and Z. Sarker, "Video Traffic Models 611 for RTP Congestion Control Evaluations", draft-ietf-rmcat- 612 video-traffic-model-07 (work in progress), February 2019. 614 [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion 615 Control Algorithms", BCP 133, RFC 5033, 616 DOI 10.17487/RFC5033, August 2007, 617 . 619 [RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion 620 Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March 621 2008, . 623 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 624 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 625 . 627 [SA4-LR] S4-050560, 3GPP., "Error Patterns for MBMS Streaming over 628 UTRAN and GERAN", 3GPP S4-050560, 5 2008. 630 [TCP-eval-suite] 631 Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier, 632 R., Gang, W., Eggert, L., Ha, S., and I. Rhee, "Towards a 633 Common TCP Evaluation Suite", Proc. PFLDnet. 2008, August 634 2008. 636 [xiph-seq] 637 Daede, T., "Video Test Media Set", 638 https://people.xiph.org/~tdaede/sets/ . 640 Appendix A. Application Trade-off 642 Application trade-off is yet to be defined. see RMCAT requirements 643 [I-D.ietf-rmcat-cc-requirements] document. Perhaps each experiment 644 should define the application's expectation or trade-off. 646 A.1. Measuring Quality 648 No quality metric is defined for performance evaluation, it is 649 currently an open issue. However, there is consensus that congestion 650 control algorithm should be able to show that it is useful for 651 interactive video by performing analysis using a real codec and video 652 sequences. 654 Appendix B. Change Log 656 Note to the RFC-Editor: please remove this section prior to 657 publication as an RFC. 659 B.1. Changes in draft-ietf-rmcat-eval-criteria-07 661 Updated the draft according to the discussion at IETF-101. 663 o Updated the discussion on fairness. Thanks to Xiaoqing Zhu for 664 providing text. 666 o Fixed a simple loss model and provided pointers to more 667 sophisticated ones. 669 o Fixed the choice of the jitter model. 671 B.2. Changes in draft-ietf-rmcat-eval-criteria-06 673 o Updated Jitter. 675 B.3. Changes in draft-ietf-rmcat-eval-criteria-05 677 o Improved text surrounding wireless tests, video sequences, and 678 short-TCP model. 680 B.4. Changes in draft-ietf-rmcat-eval-criteria-04 682 o Removed the guidelines section, as most of the sections are now 683 covered: wireless tests, video model, etc. 685 o Improved Short TCP model based on the suggestion to use 686 httparchive.org. 688 B.5. Changes in draft-ietf-rmcat-eval-criteria-03 690 o Keep-alive version. 692 o Moved link parameters and traffic models from eval-test 694 B.6. Changes in draft-ietf-rmcat-eval-criteria-02 696 o Incorporated fairness test as a working test. 698 o Updated text on mimimum evaluation requirements. 700 B.7. Changes in draft-ietf-rmcat-eval-criteria-01 702 o Removed Appendix B. 704 o Removed Section on Evaluation Parameters. 706 B.8. Changes in draft-ietf-rmcat-eval-criteria-00 708 o Updated references. 710 o Resubmitted as WG draft. 712 B.9. Changes in draft-singh-rmcat-cc-eval-04 714 o Incorporate feedback from IETF 87, Berlin. 716 o Clarified metrics: convergence time, bandwidth utilization. 718 o Changed fairness criteria to fairness test. 720 o Added measuring pre- and post-repair loss. 722 o Added open issue of measuring video quality to appendix. 724 o clarified use of DropTail and AQM. 726 o Updated text in "Minimum Requirements for Evaluation" 728 B.10. Changes in draft-singh-rmcat-cc-eval-03 730 o Incorporate the discussion within the design team. 732 o Added a section on evaluation parameters, it describes the flow 733 and network characteristics. 735 o Added Appendix with self-fairness experiment. 737 o Changed bottleneck parameters from a proposal to an example set. 739 o 741 B.11. Changes in draft-singh-rmcat-cc-eval-02 743 o Added scenario descriptions. 745 B.12. Changes in draft-singh-rmcat-cc-eval-01 747 o Removed QoE metrics. 749 o Changed stability to steady-state. 751 o Added measuring impact against few and many flows. 753 o Added guideline for idle and data-limited periods. 755 o Added reference to TCP evaluation suite in example evaluation 756 scenarios. 758 Authors' Addresses 760 Varun Singh 761 CALLSTATS I/O Oy 762 Runeberginkatu 4c A 4 763 Helsinki 00100 764 Finland 766 Email: varun@callstats.io 767 URI: https://www.callstats.io/about 769 Joerg Ott 770 Technical University of Munich 771 Faculty of Informatics 772 Boltzmannstrasse 3 773 Garching bei Muenchen, DE 85748 774 Germany 776 Email: ott@in.tum.de 778 Stefan Holmer 779 Google 780 Kungsbron 2 781 Stockholm 11122 782 Sweden 784 Email: holmer@google.com