idnits 2.17.1 draft-ietf-rmcat-eval-test-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 6 instances of too long lines in the document, the longest one being 4 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 23, 2019) is 1799 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: '300ms' is mentioned on line 1012, but not defined == Missing Reference: '1000ms' is mentioned on line 1012, but not defined == Outdated reference: A later version (-14) exists of draft-ietf-rmcat-eval-criteria-08 == Outdated reference: A later version (-11) exists of draft-ietf-rmcat-wireless-tests-06 == Outdated reference: A later version (-09) exists of draft-ietf-rmcat-coupled-cc-08 Summary: 1 error (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group Z. Sarker 3 Internet-Draft Ericsson AB 4 Intended status: Informational V. Singh 5 Expires: November 24, 2019 callstats.io 6 X. Zhu 7 M. Ramalho 8 Cisco Systems 9 May 23, 2019 11 Test Cases for Evaluating RMCAT Proposals 12 draft-ietf-rmcat-eval-test-10 14 Abstract 16 The Real-time Transport Protocol (RTP) is used to transmit media in 17 multimedia telephony applications. These applications are typically 18 required to implement congestion control. This document describes 19 the test cases to be used in the performance evaluation of such 20 congestion control algorithms in a controlled environment. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on November 24, 2019. 39 Copyright Notice 41 Copyright (c) 2019 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (https://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 57 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 58 3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 59 4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 8 60 4.1. Evaluation metrics . . . . . . . . . . . . . . . . . . . 8 61 4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 62 4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 63 5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 64 5.1. Variable Available Capacity with a Single Flow . . . . . 10 65 5.2. Variable Available Capacity with Multiple Flows . . . . . 13 66 5.3. Congested Feedback Link with Bi-directional Media Flows . 14 67 5.4. Competing Media Flows with same Congestion Control 68 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 17 69 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19 70 5.6. Media Flow Competing with a Long TCP Flow . . . . . . . . 21 71 5.7. Media Flow Competing with Short TCP Flows . . . . . . . . 23 72 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25 73 6. Other potential test cases . . . . . . . . . . . . . . . . . 27 74 6.1. Media Flows with Priority . . . . . . . . . . . . . . . . 27 75 6.2. Explicit Congestion Notification Usage . . . . . . . . . 27 76 6.3. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 28 77 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 30 78 8. Security Considerations . . . . . . . . . . . . . . . . . . . 30 79 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 80 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 81 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 30 82 11.1. Normative References . . . . . . . . . . . . . . . . . . 30 83 11.2. Informative References . . . . . . . . . . . . . . . . . 32 84 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32 86 1. Introduction 88 This memo describes a set of test cases for evaluating congestion 89 control algorithm proposals in controlled environments for real-time 90 interactive media. It is based on the guidelines enumerated in 91 [I-D.ietf-rmcat-eval-criteria] and the requirements discussed in 92 [I-D.ietf-rmcat-cc-requirements]. The test cases cover basic usage 93 scenarios and are described using a common structure, which allows 94 for additional test cases to be added to those described herein to 95 accommodate other topologies and/or the modelling of different path 96 characteristics. The described test cases in this memo should be 97 used to evaluate any proposed congestion control algorithm for real- 98 time interactive media. 100 2. Terminology 102 The terminology defined in RTP [RFC3550], RTP Profile for Audio and 103 Video Conferences with Minimal Control [RFC3551], RTCP Extended 104 Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback 105 (RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506] 106 apply. 108 3. Structure of Test cases 110 All the test cases in this document follow a basic structure allowing 111 implementers to describe a new test scenario without repeatedly 112 explaining common attributes. The structure includes a general 113 description section that describes the test case and its motivation. 114 Additionally the test case defines a set of attributes that 115 characterize the testbed, for example, the network path between 116 communicating peers and the diverse traffic sources. 118 o Define the test case: 120 * General description: describes the motivation and the goals of 121 the test case. 123 * Expected behavior: describes the desired rate adaptation 124 behavior. 126 * Define a list of metrics to evaluate the desired behavior: this 127 indicates the minimum set of metrics (e.g., link utilization, 128 media sending rate) that a proposed algorithm needs to measure 129 to validate the expected rate adaptation behavior. It should 130 also indicate the time granularity (e.g., averaged over 10ms, 131 100ms, or 1s) for measuring certain metrics. Typical 132 measurement interval is 200ms. 134 o Define testbed topology: every test case needs to define an 135 evaluation testbed topology. Figure 1 shows such an evaluation 136 topology. In this evaluation topology, S1..Sn are traffic 137 sources. These sources generate media traffic and use the 138 congestion control algorithm(s) under investigation. R1..Rn are 139 the corresponding receivers. A test case can have one or more 140 such traffic sources (S) and their corresponding receivers (R). 141 The path from the source to destination is denoted as "forward" 142 and the path from a destination to a source is denoted as 143 "backward". The following basic structure of the test case has 144 been described from the perspective of media generating endpoints 145 attached on the left-hand side of Figure 1. In this setup, the 146 media flows are transported in forward direction and corresponding 147 feedback/control messages are transported in the backward 148 direction. However, it is also possible to set up the test with 149 media in both forward and backward directions. In that case, 150 unless otherwise specified by the test case, it is expected that 151 the backward path does not introduce any congestion related 152 impairments and has enough capacity to accommodate both media and 153 feedback/control messages. It should be noted that depending on 154 the test cases it is possible to have different path 155 characteristics in either of the directions. 157 +---+ +---+ 158 |S1 |====== \ Forward --> / =======|R1 | 159 +---+ \\ // +---+ 160 \\ // 161 +---+ +-----+ +-----+ +---+ 162 |S2 |=======| A |------------------------------>| B |=======|R2 | 163 +---+ | |<------------------------------| | +---+ 164 +-----+ +-----+ 165 (...) // \\ (...) 166 // <-- Backward \\ 167 +---+ // \\ +---+ 168 |Sn |====== / \ ======|Rn | 169 +---+ +---+ 171 Figure 1: Example of A Testbed Topology 173 In a testbed environment with real equipments, there may exist a 174 significant amount of unwanted traffic on the portions of the 175 network path between the endpoints. Some of this traffic may be 176 generated by other processes on the endpoints themselves (e.g., 177 discovery protocols) or by other endpoints not presently under 178 test. Such unwanted traffic should be removed or avoided to the 179 greatest extent possible. 181 o Define testbed attributes: 183 * Duration: defines the duration of the test in seconds. 185 * Path characteristics: defines the end-to-end transport level 186 path characteristics of the testbed for a particular test case. 187 Two sets of attributes describe the path characteristics, one 188 for the forward path and the other for the backward path. The 189 path characteristics for a particular path direction is 190 applicable to all the Sources "S" sending traffic on that path. 191 If only one attribute is specified, it is used for both path 192 directions, however, unless specified the reverse path has no 193 capacity restrictions and no path loss. 195 + Path direction: forward or backward. 197 + Minimum bottleneck-link capacity: defines minimum capacity 198 of the end-to-end path 200 + Reference bottleneck capacity: defines a reference value for 201 the bottleneck capacity for test cases with time-varying 202 bottleneck capacities. All bottleneck capacities will be 203 specified as a ratio with respect to the reference capacity 204 value. 206 + One-way propagation delay: describes the end-to-end latency 207 along the path when network queues are empty, i.e., the time 208 it takes for a packet to go from the sender to the receiver 209 without encountering any queuing delay. 211 + Maximum end-to-end jitter: defines the maximum jitter that 212 can be observed along the path. 214 + Bottleneck queue type: for example, "tail drop" [RFC7567], 215 Flow Queue -CoDel (FQ-CoDel)[RFC8290], or Proportional 216 Integral controller Enhanced (PIE)[RFC8033]. 218 + Bottleneck queue size: defines the size of queue in terms of 219 queuing time when the queue is full (in milliseconds). 221 + Path loss ratio: characterizes the non-congested, additive, 222 losses to be generated on the end-to-end path. This must 223 describe the loss pattern or loss model used to generate the 224 losses. 226 * Application-related: defines the traffic source behavior for 227 implementing the test case 229 + Media traffic Source: defines the characteristics of the 230 media sources. When using more than one media source, the 231 different attributes are enumerated separately for each 232 different media source. 234 - Media type: Video/Voice 236 - Media flow direction: forward, backward or both. 238 - Number of media sources: defines the total number of 239 media sources 241 - Media codec: Constant Bit Rate (CBR) or Variable Bit Rate 242 (VBR) 244 - Media source behavior: describes the media encoder 245 behavior. It defines the main parameters that affect the 246 adaptation behavior. This may include but is not limited 247 to: 249 o Adaptability: describes the adaptation options. For 250 example, in the case of video it defines the following 251 ranges of adaptation: bit rate, frame rate, video 252 resolution. Similarly, in the case of voice, it 253 defines the range of bit rate adaptation, the sampling 254 rate variation, and the variation in packetization 255 interval. 257 o Output variation : for a VBR encoder it defines the 258 encoder output variation from the average target rate 259 over a particular measurement interval. For example, 260 on average the encoder output may vary between 5% to 261 15% above or below the average target bit rate when 262 measured over a 100 ms time window. The time interval 263 over which the variation is specified must be 264 provided. 266 o Responsiveness to a new bit rate request: the lag in 267 time between a new bit rate request from the 268 congestion control algorithm and actual rate changes 269 in encoder output. Depending on the encoder, this 270 value may be specified in absolute time (e.g. 10ms to 271 1000ms) or other appropriate metric (e.g. next frame 272 interval time). 274 More detailed discussions on expected media source 275 behavior, including those from synthetic video traffic 276 sources, is at [I-D.ietf-rmcat-video-traffic-model]. 278 - Media content: describes the chosen video scenario. For 279 example, video test sequences are available at: 280 [xiph-seq] and [HEVC-seq]. Different video scenarios 281 give different distribution of video frames produced by 282 the video encoder. Hence, it is important to specify the 283 media content used in a particular test. If a synthetic 284 video traffic souce [I-D.ietf-rmcat-video-traffic-model] 285 is used, then the synthetic video traffic source needs to 286 configure according to the characteristics of the media 287 content specified. 289 - Media timeline: describes the point when the media source 290 is introduced and removed from the testbed. For example, 291 the media source may start transmitting immediately when 292 the test case begins, or after a few seconds. 294 - Startup behavior: the media starts at a defined bit rate, 295 which may be the minimum, maximum bit rate, or a value in 296 between (in Kbps). 298 + Competing traffic source: describes the characteristics of 299 the competing traffic source, the different types of 300 competing flows are enumerated in 301 [I-D.ietf-rmcat-eval-criteria]. 303 - Traffic direction: forward, backward or both. 305 - Type of sources: defines the types of competing traffic 306 sources. Types of competing traffic flows are listed in 307 [I-D.ietf-rmcat-eval-criteria]. For example, the number 308 of TCP flows connected to a web browser, the mean size 309 and distribution of the content downloaded. 311 - Number of sources: defines the total number of competing 312 sources of each media type per traffic direction. 314 - Congestion control: enumerates the congestion control 315 used by each type of competing traffic. 317 - Traffic timeline: describes when the competing traffic 318 starts and ends in the test case. 320 * Additional attributes: describes attributes essential for 321 implementing a test case which are not included in the above 322 structure. These attributes must be well defined, so that the 323 other implementers of that particular test case are able to 324 implement it easily. 326 Any attribute can have a set of values (enclosed within "[]"). Each 327 member value of such a set must be treated as different value for the 328 same attribute. It is desired to run separate tests for each such 329 attribute value. 331 The test cases described in this document follow the above structure. 333 4. Recommended Evaluation Settings 335 This section describes recommended test case settings and could be 336 overwritten by the respective test cases. 338 4.1. Evaluation metrics 340 To evaluate the performance of the candidate algorithms the 341 implementers must log enough information to visualize the following 342 metrics at a fine enough time granularity: 344 1. Flow level: 346 A. End-to-end delay for the congestion controlled media flow(s). 347 For example - end-to-end delay observed on IP packet level, 348 video frame level. 350 B. Variation in sending bit rate and throughput. Mainly 351 observing the frequency and magnitude of oscillations. 353 C. Packet losses observed at the receiving endpoint. 355 D. Feedback message overhead. 357 E. Convergence time - time to reach steady state for the 358 congestion controlled media flow(s). Each occurrence of 359 convergence during the test period need to be presented. 361 2. Transport level: 363 A. Bandwidth utilization. 365 B. Queue length (milliseconds at specified path capacity). 367 4.2. Path characteristics 369 Each path between a sender and receiver as described in Figure 1 have 370 the following characteristics unless otherwise specified in the test 371 case. 373 o Path direction: forward and backward. 375 o Reference bottleneck capacity: 1Mbps. 377 o One-Way propagation delay: 50ms. Implementers are encouraged to 378 run the experiment with additional propagation delays mentioned in 379 [I-D.ietf-rmcat-eval-criteria] 381 o Maximum end-to-end jitter: 30ms. Jitter models are described in 382 [I-D.ietf-rmcat-eval-criteria] 384 o Bottleneck queue type: "tail drop". Implementers are encouraged 385 to run the experiment with other AQM schemes, such as FQ-CoDel and 386 PIE. 388 o Bottleneck queue size: 300ms. 390 o Path loss ratio: 0%. 392 Examples of additional network parameters are discussed in 393 [I-D.ietf-rmcat-eval-criteria]. 395 For test cases involving time-varying bottleneck capacity, all 396 capacity values are specified as a ratio with respect to a reference 397 capacity value, so as to allow flexible scaling of capacity values 398 along with media source rate range. There exist two different 399 mechanisms for inducing path capacity variation: a) by explicitly 400 modifying the value of physical link capacity; or b) by introducing 401 background non-adaptive UDP traffic with time-varying traffic rate. 402 Implementers are encouraged to run the experiments with both 403 mechanisms for test cases specified in Section 5.1, Section 5.2, and 404 Section 5.3. 406 4.3. Media source 408 Unless otherwise specified, each test case will include one or more 409 media sources as described below. 411 o Media type: Video 413 * Media codec: VBR 415 * Media source behavior: 417 + Adaptability: 419 - Bit rate range: 150 Kbps - 1.5 Mbps. In real-life 420 applications the bit rate range can vary a lot depending 421 on the provided service, for example, the maximum bit 422 rate can be up to 4Mbps. However, for running tests to 423 evaluate the congestion control algorithms it is more 424 important to have a look at how they are reacting to 425 certain amount of bandwidth change. Also it is possible 426 that the media traffic generator used in a particular 427 simulator or testbed is not capable of generating higher 428 bit rate. Hence we have selected a suitable bit rate 429 range typical of consumer-grade video conferencing 430 applications in designing the test case. If a different 431 bit rate range is used in the test cases, then the end- 432 to-end path capacity values will also need to be scaled 433 accordingly. 435 - Frame resolution: 144p - 720p (or 1080p). This 436 resolution range is selected based on the bit rate range. 437 If a different bit rate range is used in the test cases 438 then the frame resolution range also need to be selected 439 suitably. 441 - Frame rate: 10fps - 30fps. This frame rate range is 442 selected based on the bit rate range. If a different bit 443 rate range is used in the test cases then the frame rate 444 range also need to be adjusted suitably. 446 + Variation from target bit rate: +/-5%. Unless otherwise 447 specified in the test case(s), bit rate variation should be 448 calculated over one (1) second period of time. 450 + Responsiveness to new bit rate request: 100ms 452 * Media content: The media content should represent a typical 453 video conversational scenario with head and shoulder movement. 454 We recommend to use Foreman video sequence[xiph-seq]. 456 * Media startup behavior: 150Kbps. It should be noted that 457 applications can use smart ways to select an optimal startup 458 bit rate value for a certain network condition. In such cases 459 the candidate proposals may show the effectiveness of such 460 smart approach as an additional information for the evaluation 461 process. 463 o Media type: Audio 465 * Media codec: CBR 467 * Media bit rate: 20Kbps 469 5. Basic Test Cases 471 5.1. Variable Available Capacity with a Single Flow 473 In this test case the minimum bottleneck-link capacity between the 474 two endpoints varies over time. This test is designed to measure the 475 responsiveness of the candidate algorithm. This test tries to 476 address the requirements in [I-D.ietf-rmcat-cc-requirements], which 477 requires the algorithm to adapt the flow(s) and provide lower end-to- 478 end latency when there exists: 480 o an intermediate bottleneck 482 o change in available capacity (e.g., due to interface change, 483 routing change, abrupt arrival/departure of background non- 484 adaptive traffic). 486 o maximum media bit rate is greater than link capacity. In this 487 case, when the application tries to ramp up to its maximum bit 488 rate, since the link capacity is limited to a value lower, the 489 congestion control scheme is expected to stabilize the sending bit 490 rate close to the available bottleneck capacity. 492 It should be noted that the exact variation in available capacity due 493 to any of the above depends on the underlying technologies. Hence, 494 we describe a set of known factors, which may be extended to devise a 495 more specific test case targeting certain behaviors in a certain 496 network environment. 498 Expected behavior: the candidate algorithm is expected to detect the 499 path capacity constraint, converge to the bottleneck link's capacity 500 and adapt the flow to avoid unwanted media rate oscillation when the 501 sending bit rate is approaching the bottleneck link's capacity. Such 502 oscillations might occur when the media flow(s) attempts to reach its 503 maximum bit rate but overshoots the usage of the available bottleneck 504 capacity then to rectify, it reduces the bit rate and starts to ramp 505 up again. 507 Evaluation metrics : as described in Section 4.1. 509 Testbed topology: One media source S1 is connected to the 510 corresponding R1. The media traffic is transported over the forward 511 path and corresponding feedback/control traffic is transported over 512 the backward path. 514 Forward --> 515 +---+ +-----+ +-----+ +---+ 516 |S1 |=======| A |------------------------------>| B |=======|R1 | 517 +---+ | |<------------------------------| | +---+ 518 +-----+ +-----+ 519 <-- Backward 521 Figure 2: Testbed Topology for Limited Link Capacity 523 Testbed attributes: 525 o Test duration: 100s 527 o Path characteristics: as described in Section 4.2 529 o Application-related: 531 * Media Traffic: 533 + Media type: Video 535 - Media direction: forward. 537 - Number of media sources: one (1) 539 - Media timeline: 541 o Start time: 0s. 543 o End time: 99s. 545 + Media type: Audio 547 - Media direction: forward. 549 - Number of media sources: one (1) 551 - Media timeline: 553 o Start time: 0s. 555 o End time: 99s. 557 * Competing traffic: 559 + Number of sources : zero (0) 561 o Test Specific Information: 563 * One-way propagation delay: [ 50 ms, 100 ms]. on the forward 564 path direction 566 * This test uses bottleneck path capacity variation as listed in 567 Table 1 569 * When using background non-adaptive UDP traffic to induce time- 570 varying bottleneck , the physical path capacity remains at 571 4Mbps and the UDP traffic source rate changes over time as (4 - 572 (Y x r)), where r is the Reference bottleneck capacity in Mbps 573 and Y is the path capacity ratio specified in Table 1 575 +--------------------+--------------+-----------+-------------------+ 576 | Variation pattern | Path | Start | Path capacity | 577 | index | direction | time | ratio | 578 +--------------------+--------------+-----------+-------------------+ 579 | One | Forward | 0s | 1.0 | 580 | Two | Forward | 40s | 2.5 | 581 | Three | Forward | 60s | 0.6 | 582 | Four | Forward | 80s | 1.0 | 583 +--------------------+--------------+-----------+-------------------+ 585 Table 1: Path capacity variation pattern for forward direction 587 5.2. Variable Available Capacity with Multiple Flows 589 This test case is similar to Section 5.1. However in addition this 590 test will also consider persistent network load due to competing 591 traffic. 593 Expected behavior: the candidate algorithm is expected to detect the 594 variation in available capacity and adapt the media stream(s) 595 accordingly. The flows stabilize around their maximum bit rate as 596 the maximum link capacity is large enough to accommodate the flows. 597 When the available capacity drops, the flows adapt by decreasing 598 their sending bit rate, and when congestion disappears, the flows are 599 again expected to ramp up. 601 Evaluation metrics : as described in Section 4.1. 603 Testbed Topology: Two (2) media sources S1 and S2 are connected to 604 their corresponding destinations R1 and R2. The media traffic is 605 transported over the forward path and corresponding feedback/control 606 traffic is transported over the backward path. 608 +---+ +---+ 609 |S1 |===== \ / =======|R1 | 610 +---+ \\ Forward --> // +---+ 611 \\ // 612 +-----+ +-----+ 613 | A |------------------------------>| B | 614 | |<------------------------------| | 615 +-----+ +-----+ 616 // \\ 617 // <-- Backward \\ 618 +---+ // \\ +---+ 619 |S2 |====== / \ ======|R2 | 620 +---+ +---+ 622 Figure 3: Testbed Topology for Variable Available Capacity 624 Testbed attributes: 626 Testbed attributes are similar as described in Section 5.1 except the 627 test specific capacity variation setup. 629 Test Specific Information: This test uses path capacity variation as 630 listed in Table 2 with a corresponding end time of 125 seconds. The 631 reference bottleneck capacity is 2Mbps. When using background non- 632 adaptive UDP traffic to induce time-varying bottleneck for congestion 633 controlled media flows, the physical path capacity is 4Mbps and the 634 UDP traffic source rate changes over time as (4 - (Y x r)), where r 635 is the Reference bottleneck capacity in Mbps and Y is the path 636 capacity ratio specified in Table 2. 638 +--------------------+--------------+-----------+-------------------+ 639 | Variation pattern | Path | Start | Path capacity | 640 | index | direction | time | ratio | 641 +--------------------+--------------+-----------+-------------------+ 642 | One | Forward | 0s | 2.0 | 643 | Two | Forward | 25s | 1.0 | 644 | Three | Forward | 50s | 1.75 | 645 | Four | Forward | 75s | 0.5 | 646 | Five | Forward | 100s | 1.0 | 647 +--------------------+--------------+-----------+-------------------+ 649 Table 2: Path capacity variation pattern for forward direction 651 5.3. Congested Feedback Link with Bi-directional Media Flows 653 Real-time interactive media uses RTP hence it is assumed that RTCP, 654 RTP header extension or such would be used by the congestion control 655 algorithm in the backchannel. Due to the asymmetric nature of the 656 link between communicating peers it is possible for a participating 657 peer to not receive such feedback information due to an impaired or 658 congested backchannel (even when the forward channel might not be 659 impaired). This test case is designed to observe the candidate 660 congestion control behavior in such an event. 662 Expected behavior: It is expected that the candidate algorithms are 663 able to cope with the lack of feedback information and adapt to 664 minimize the performance degradation of media flows in the forward 665 channel. 667 It should be noted that for this test case: logs are compared with 668 the reference case, i.e, when the backward channel has no 669 impairments. 671 Evaluation metrics : as described in Section 4.1. 673 Testbed topology: One (1) media source S1 is connected to 674 corresponding R1, but both endpoints are additionally receiving and 675 sending data, respectively. The media traffic (S1->R1) is 676 transported over the forward path and corresponding feedback/control 677 traffic is transported over the backward path. Likewise media 678 traffic (S2->R2) is transported over the backward path and 679 corresponding feedback/control traffic is transported over the 680 forward path. 682 +---+ +---+ 683 |S1 |===== \ Forward --> / =======|R1 | 684 +---+ \\ // +---+ 685 \\ // 686 +-----+ +-----+ 687 | A |------------------------------>| B | 688 | |<------------------------------| | 689 +-----+ +-----+ 690 // \\ 691 // <-- Backward \\ 692 +---+ // \\ +---+ 693 |R2 |===== / \ ======|S2 | 694 +---+ +---+ 696 Figure 4: Testbed Topology for Congested Feedback Link 698 Testbed attributes: 700 o Test duration: 100s 702 o Path characteristics: 704 * Reference bottleneck capacity: 1Mbps. 706 o Application-related: 708 * Media Source: 710 + Media type: Video 712 - Media direction: forward and backward 714 - Number of media sources: two (2) 716 - Media timeline: 718 o Start time: 0s. 720 o End time: 99s. 722 + Media type: Audio 724 - Media direction: forward and backward 726 - Number of media sources: two (2) 728 - Media timeline: 730 o Start time: 0s. 732 o End time: 99s. 734 * Competing traffic: 736 + Number of sources : zero (0) 738 o Test Specific Information: this test uses path capacity variations 739 to create congested feedback link. Table 3 lists the variation 740 patterns applied to the forward path and Table 4 lists the 741 variation patterns applied to the backward path. When using 742 background non-adaptive UDP traffic to induce time-varying 743 bottleneck for congestion controlled media flows, the physical 744 path capacity is 4Mbps for both directions and the UDP traffic 745 source rate changes over time as (4-x)Mbps in each direction, 746 where x is the bottleneck capacity specified in Table 4. 748 +--------------------+--------------+-----------+-------------------+ 749 | Variation pattern | Path | Start | Path capacity | 750 | index | direction | time | ratio | 751 +--------------------+--------------+-----------+-------------------+ 752 | One | Forward | 0s | 2.0 | 753 | Two | Forward | 20s | 1.0 | 754 | Three | Forward | 40s | 0.5 | 755 | Four | Forward | 60s | 2.0 | 756 +--------------------+--------------+-----------+-------------------+ 758 Table 3: Path capacity variation pattern for forward direction 760 +--------------------+--------------+-----------+-------------------+ 761 | Variation pattern | Path | Start | Path capacity | 762 | index | direction | time | ratio | 763 +--------------------+--------------+-----------+-------------------+ 764 | One | Backward | 0s | 2.0 | 765 | Two | Backward | 35s | 0.8 | 766 | Three | Backward | 70s | 2.0 | 767 +--------------------+--------------+-----------+-------------------+ 769 Table 4: Path capacity variation pattern for backward direction 771 5.4. Competing Media Flows with same Congestion Control Algorithm 773 In this test case, more than one media flow share the bottleneck link 774 and each of them uses the same congestion control algorithm. This is 775 a typical scenario where a real-time interactive application sends 776 more than one media flow to the same destination and these flows are 777 multiplexed over the same port. In such a scenario it is likely that 778 the flows will be routed via the same path and need to share the 779 available bandwidth amongst themselves. For the sake of simplicity 780 it is assumed that there are no other competing traffic sources in 781 the bottleneck link and that there is sufficient capacity to 782 accommodate all the flows individually. While this appears to be a 783 variant of the test case defined in Section 5.2, it focuses on the 784 capacity sharing aspect of the candidate algorithm. The previous 785 test case, on the other hand, measures adaptability, stability, and 786 responsiveness of the candidate algorithm. 788 Expected behavior: It is expected that the competing flows will 789 converge to an optimum bit rate to accommodate all the flows with 790 minimum possible latency and loss. Specifically, the test introduces 791 three media flows at different time instances, when the second flow 792 appears there should still be room to accommodate another flow on the 793 bottleneck link. Lastly, when the third flow appears the bottleneck 794 link should be saturated. 796 Evaluation metrics : as described in Section 4.1. 798 Testbed topology: Three media sources S1, S2, S3 are connected to R1, 799 R2, R3 respectively. The media traffic is transported over the 800 forward path and corresponding feedback/control traffic is 801 transported over the backward path. 803 +---+ +---+ 804 |S1 |===== \ Forward --> / =======|R1 | 805 +---+ \\ // +---+ 806 \\ // 807 +---+ +-----+ +-----+ +---+ 808 |S2 |=======| A |------------------------------>| B |=======|R2 | 809 +---+ | |<------------------------------| | +---+ 810 +-----+ +-----+ 811 // <-- Backward \\ 812 +---+ // \\ +---+ 813 |S3 |===== / \ ======|R3 | 814 +---+ +---+ 816 Figure 5: Testbed Topology for Multiple congestion controlled media 817 Flows 819 Testbed attributes: 821 o Test duration: 120s 823 o Path characteristics: 825 * Reference bottleneck capacity: 3.5Mbps 827 * Path capacity ratio: 1.0 829 o Application-related: 831 * Media Source: 833 + Media type: Video 835 - Media direction: forward. 837 - Number of media sources: three (3) 839 - Media timeline: new media flows are added sequentially, 840 at short time intervals. See test specific setup below. 842 + Media type: Audio 843 - Media direction: forward. 845 - Number of media sources: three (3) 847 - Media timeline: new media flows are added sequentially, 848 at short time intervals. See test specific setup below. 850 * Competing traffic: 852 + Number of sources : zero (0) 854 o Test Specific Information: Table 5 defines the media timeline for 855 both media type. 857 +---------+------------+------------+----------+ 858 | Flow ID | Media type | Start time | End time | 859 +---------+------------+------------+----------+ 860 | 1 | Video | 0s | 119s | 861 | 2 | Video | 20s | 119s | 862 | 3 | Video | 40s | 119s | 863 | 4 | Audio | 0s | 119s | 864 | 5 | Audio | 20s | 119s | 865 | 6 | Audio | 40s | 119s | 866 +---------+------------+------------+----------+ 868 Table 5: Media Timeline for Video and Audio media sources 870 5.5. Round Trip Time Fairness 872 In this test case, multiple media flows share the bottleneck link, 873 but the one-way propagation delay for each flow is different. For 874 the sake of simplicity it is assumed that there are no other 875 competing traffic sources in the bottleneck link and that there is 876 sufficient capacity to accommodate all the flows. While this appears 877 to be a variant of test case 5.2, it focuses on the capacity sharing 878 aspect of the candidate algorithm under different RTTs. 880 Expected behavior: It is expected that the competing flows will 881 converge to bit rates to accommodate all the flows with minimum 882 possible latency and loss. The effectiveness of the algorithm 883 depends on how fast and fairly the competing flows converge to their 884 steady states irrespective of the RTT observed. 886 Evaluation metrics : as described in Section 4.1. 888 Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to 889 their corresponding media sinks R1,R2,..,R5. The media traffic is 890 transported over the forward path and corresponding feedback/control 891 traffic is transported over the backward path. The topology is the 892 same as in Section 5.4. 894 Testbed attributes: 896 o Test duration: 300s 898 o Path characteristics: 900 * Reference bottleneck capacity: 4Mbps 902 * Path capacity ratio: 1.0 904 * One-Way propagation delay for each flow: 10ms for S1-R1, 25ms 905 for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms S5-R5. 907 o Application-related: 909 * Media Source: 911 + Media type: Video 913 - Media direction: forward 915 - Number of media sources: five (5) 917 - Media timeline: new media flows are added sequentially, 918 at short time intervals. See test specific setup below. 920 + Media type: Audio 922 - Media direction: forward. 924 - Number of media sources: five (5) 926 - Media timeline: new media flows are added sequentially, 927 at short time intervals. See test specific setup below. 929 * Competing traffic: 931 + Number of sources : zero (0) 933 o Test Specific Information: Table 6 defines the media timeline for 934 both media type. 936 +---------+------------+------------+----------+ 937 | Flow IF | Media type | Start time | End time | 938 +---------+------------+------------+----------+ 939 | 1 | Video | 0s | 299s | 940 | 2 | Video | 10s | 299s | 941 | 3 | Video | 20s | 299s | 942 | 4 | Video | 30s | 299s | 943 | 5 | Video | 40s | 299s | 944 | 6 | Audio | 0 | 299s | 945 | 7 | Audio | 10s | 299s | 946 | 8 | Audio | 20s | 299s | 947 | 9 | Audio | 30s | 299s | 948 | 10 | Audio | 40s | 299s | 949 +---------+------------+------------+----------+ 951 Table 6: Media Timeline for Video and Audio media sources 953 5.6. Media Flow Competing with a Long TCP Flow 955 In this test case, one or more media flows share the bottleneck link 956 with at least one long lived TCP flow. Long lived TCP flows download 957 data throughout the session and are expected to have infinite amount 958 of data to send and receive. This is a scenario where a multimedia 959 application co-exists with a large file download. The test case 960 measures the adaptivity of the candidate algorithm to competing 961 traffic. It addresses the requirement 3 in 962 [I-D.ietf-rmcat-cc-requirements]. 964 Expected behavior: depending on the convergence observed in test case 965 5.1 and 5.2, the candidate algorithm may be able to avoid congestion 966 collapse. In the worst case, the media stream will fall to the 967 minimum media bit rate. 969 Evaluation metrics : following metrics in addition to as described in 970 Section 4.1. 972 1. Flow level: 974 A. TCP throughput. 976 B. Loss for the TCP flow 978 Testbed topology: One (1) media source S1 is connected to the 979 corresponding media sink, R1. In addition, there is a long-live TCP 980 flow sharing the same bottleneck link. The media traffic is 981 transported over the forward path and corresponding feedback/control 982 traffic is transported over the backward path. The TCP traffic goes 983 over the forward path from, S_tcp with acknowledgment packets go over 984 the backward path from, R_tcp. 986 +--+ +--+ 987 |S1|===== \ Forward --> / =====|R1| 988 +--+ \\ // +--+ 989 \\ // 990 +-----+ +-----+ 991 | A |---------------------------->| B | 992 | |<----------------------------| | 993 +-----+ +-----+ 994 // <-- Backward \\ 995 +-----+ // \\ +-----+ 996 |S_tcp|=== / \ ===|R_tcp| 997 +-----+ +-----+ 999 Figure 6: Testbed Topology for TCP vs congestion controlled media 1000 Flows 1002 Testbed attributes: 1004 o Test duration: 120s 1006 o Path characteristics: 1008 * Reference bottleneck capacity: 2Mbps 1010 * Path capacity ratio: 1.0 1012 * Bottleneck queue size: [300ms, 1000ms] 1014 o Application-related: 1016 * Media Source: 1018 + Media type: Video 1020 - Media direction: forward 1022 - Number of media sources: one (1) 1024 - Media timeline: 1026 o Start time: 5s. 1028 o End time: 119s. 1030 + Media type: Audio 1031 - Media direction: forward 1033 - Number of media sources: one (1) 1035 - Media timeline: 1037 o Start time: 5s. 1039 o End time: 119s. 1041 * Additionally, implementers are encouraged to run the experiment 1042 with multiple media sources. 1044 * Competing traffic: 1046 + Number and Types of sources : one (1) and long-lived TCP 1048 + Traffic direction : forward 1050 + Congestion control: default TCP congestion control[RFC5681]. 1051 Implementers are also encouraged to run the experiment with 1052 alternative TCP congestion control algorithm. 1054 + Traffic timeline: 1056 - Start time: 0s. 1058 - End time: 119s. 1060 o Test Specific Information: none 1062 5.7. Media Flow Competing with Short TCP Flows 1064 In this test case, one or more congestion controlled media flow 1065 shares the bottleneck link with multiple short-lived TCP flows. 1066 Short-lived TCP flows resemble the on/off pattern observed in the web 1067 traffic, wherein clients (for example, browsers) connect to a server 1068 and download a resource (typically a web page, few images, text 1069 files, etc.) using several TCP connections. This scenario shows the 1070 performance of a multimedia application when several browser windows 1071 are active. The test case measures the adaptivity of the candidate 1072 algorithm to competing web traffic, it addresses the requirements 1.E 1073 in [I-D.ietf-rmcat-cc-requirements]. 1075 Depending on the number of short TCP flows, the cross-traffic either 1076 appears as a short burst flow or resembles a long TCP flow. The 1077 intention of this test is to observe the impact of short-term burst 1078 on the behavior of the candidate algorithm. 1080 Expected behavior: The candidate algorithm is expected to avoid flow 1081 starvation during the presence of short and bursty competing TCP 1082 flows, streaming at least at the minimum media bit rate. After 1083 competing TCP flows terminate, the media streams are expected to be 1084 robust enough to eventually recover to previous steady state 1085 behavior, and at the very least, avoid persistent starvation. 1087 Evaluation metrics : following metrics in addition to as described in 1088 Section 4.1. 1090 1. Flow level: 1092 A. Variation in the sending rate of the TCP flow. 1094 B. TCP throughput. 1096 Testbed topology: The topology described here is same as the one 1097 described in Figure 6. 1099 Testbed attributes: 1101 o Test duration: 300s 1103 o Path characteristics: 1105 * Reference bottleneck capacity: 2.0Mbps 1107 * Path capacity ratio: 1.0 1109 o Application-related: 1111 * Media source: 1113 + Media type: Video 1115 - Media direction: forward 1117 - Number of media sources: two (2) 1119 - Media timeline: 1121 o Start time: 5s. 1123 o End time: 299s. 1125 + Media type: Audio 1127 - Media direction: forward 1128 - Number of media sources: two (2) 1130 - Media timeline: 1132 o Start time: 5s. 1134 o End time: 299s. 1136 * Competing traffic: 1138 + Number and Types of sources : ten (10), short-lived TCP 1139 flows. 1141 + Traffic direction : forward 1143 + Congestion algorithm: default TCP Congestion control 1144 [RFC5681]. Implementers are also encouraged to run the 1145 experiment with alternative TCP congestion control 1146 algorithm. 1148 + Traffic timeline: each short TCP flow is modeled as a 1149 sequence of file downloads interleaved with idle periods. 1150 Not all short TCP flows start at the same time, 2 of them 1151 start in the ON state while rest of the 8 flows start in an 1152 OFF state. For description of short TCP flow model see test 1153 specific information below. 1155 o Test Specific Information: 1157 * Short-TCP traffic model: The short TCP model to be used in this 1158 test is described in [I-D.ietf-rmcat-eval-criteria]. 1160 5.8. Media Pause and Resume 1162 In this test case, more than one real-time interactive media flows 1163 share the link bandwidth and all flows reach to a steady state by 1164 utilizing the link capacity in an optimum way. At this stage one of 1165 the media flows is paused for a moment. This event will result in 1166 more available bandwidth for the rest of the flows as they are on a 1167 shared link. When the paused media flow resumes it would no longer 1168 have the same bandwidth share on the link. It has to make its way 1169 through the other existing flows in the link to achieve a fair share 1170 of the link capacity. This test case is important specially for 1171 real-time interactive media which consists of more than one media 1172 flows and can pause/resume media flows at any point of time during 1173 the session. This test case directly addresses the requirement 1174 number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a 1175 variation of test case defined in Section 5.4. However, it is 1176 different as the candidate algorithms can use different strategies to 1177 increase its efficiency, for example in terms of fairness, 1178 convergence time, reduce oscillation etc, by capitalizing the fact 1179 that they have previous information of the link. 1181 Expected behavior: During the period where the third stream is 1182 paused, the two remaining flows are expected to increase their rates 1183 and reach the maximum media bit rate. When the third stream resumes, 1184 all three flows are expected to converge to the same original fair 1185 share of rates prior to the media pause/resume event. 1187 Evaluation metrics : following metrics in addition to as described in 1188 Section 4.1. 1190 1. Flow level: 1192 A. Variation in sending bit rate and throughput. Mainly 1193 observing the frequency and magnitude of oscillations. 1195 Testbed Topology: Same as test case defined in Section 5.4 1197 Testbed attributes: The general description of the testbed parameters 1198 are same as Section 5.4 with changes in the test specific setup as 1199 below- 1201 o Other test specific setup: 1203 * Media flow timeline: 1205 + Flow ID: one (1) 1207 + Start time: 0s 1209 + Flow duration: 119s 1211 + Pause time: not required 1213 + Resume time: not required 1215 * Media flow timeline: 1217 + Flow ID: two (2) 1219 + Start time: 0s 1221 + Flow duration: 119s 1223 + Pause time: at 40s 1224 + Resume time: at 60s 1226 * Media flow timeline: 1228 + Flow ID: three (3) 1230 + Start time: 0s 1232 + Flow duration:119s 1234 + Pause time: not required 1236 + Resume time: not required 1238 6. Other potential test cases 1240 It has been noticed that there are other interesting test cases 1241 besides the basic test cases listed above. In many aspects, these 1242 additional test cases can help further evaluation of the candidate 1243 algorithm. They are listed as below. 1245 6.1. Media Flows with Priority 1247 In this test case media flows will have different priority levels. 1248 This will be an extension of Section 5.4 where the same test will be 1249 run with different priority levels imposed on each of the media 1250 flows. For example, the first flow (S1) is assigned a priority of 2 1251 whereas the remaining two flows (S2 and S3) are assigned a priority 1252 of 1. The candidate algorithm must reflect the relative priorities 1253 assigned to each media flow. In this case, the first flow (S1) must 1254 arrive at a steady-state rate approximately twice of that of the 1255 other two flows (S2 and S3). 1257 The candidate algorithm can use a coupled congestion control 1258 mechanism [I-D.ietf-rmcat-coupled-cc] or use a weighted priority 1259 scheduler for the bandwidth distribution according to the respective 1260 media flow priority or use. 1262 6.2. Explicit Congestion Notification Usage 1264 This test case requires to run all the basic test cases with the 1265 availability of Explicit Congestion Notification (ECN) [RFC6679] 1266 feature enabled. The goal of this test is to exhibit that the 1267 candidate algorithms do not fail when ECN signals are available. 1268 With ECN signals enabled the algorithms are expected to perform 1269 better than their delay-based variants. 1271 6.3. Multiple Bottlenecks 1273 In this test case one congestion controlled media flow, S1->R1, 1274 traverses a path with multiple bottlenecks. As illustrated in 1275 Figure 7, the first flow (S1->R1) competes with the second congestion 1276 controlled media flow (S2->R2) over the link between A and B which is 1277 close to the sender side; again, that flow (S1->R1) competes with the 1278 third congestion controlled media flow (S3->R3) over the link between 1279 C and D which is close to the receiver side. The goal of this test 1280 is to ensure that the candidate algorithms work properly in the 1281 presence of multiple bottleneck links on the end to end path. 1283 Expected behavior: The candidate algorithm is expected to achieve 1284 full utilization at both bottleneck links without starving any of the 1285 three congestion controlled media flows and ensuring fair share of 1286 the available bandwidth at each bottlenecks. 1288 Forward ----> 1290 +---+ +---+ +---+ +---+ 1291 |S2 | |R2 | |S3 | |R3 | 1292 +---+ +---+ +---+ +---+ 1293 | | | | 1294 | | | | 1295 +---+ +-----+ +-----+ +-----+ +-----+ +---+ 1296 |S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 | 1297 +---+ | |<------| |<-----| |<----| | +---+ 1298 +-----+ +-----+ +-----+ +-----+ 1300 1st 2nd 1301 Bottleneck (A->B) Bottleneck (C->D) 1303 <------ Backward 1305 Figure 7: Testbed Topology for Multiple Bottlenecks 1307 Testbed topology: Three media sources S1, S2, and S3 are connected to 1308 respective destinations R1, R2, and R3. For all three flows the 1309 media traffic is transported over the forward path and corresponding 1310 feedback/control traffic is transported over the backward path. 1312 Testbed attributes: 1314 o Test duration: 300s 1316 o Path characteristics: 1318 * Reference bottleneck capacity: 2Mbps. 1320 * Path capacity ratio between A and B: 1.0 1322 * Path capacity ratio between B and C: 4.0. 1324 * Path capacity ratio between C and D: 0.75. 1326 * One-Way propagation delay: 1328 1. Between S1 and R1: 100ms 1330 2. Between S2 and R2: 40ms 1332 3. Between S3 and R3: 40ms 1334 o Application-related: 1336 * Media Source: 1338 + Media type: Video 1340 - Media direction: Forward 1342 - Number of media sources: Three (3) 1344 - Media timeline: 1346 o Start time: 0s. 1348 o End time: 299s. 1350 + Media type: Audio 1352 - Media direction: Forward 1354 - Number of media sources: Three (3) 1356 - Media timeline: 1358 o Start time: 0s. 1360 o End time: 299s. 1362 * Competing traffic: 1364 + Number of sources : Zero (0) 1366 7. Wireless Access Links 1368 Additional wireless network (both cellular network and WiFi network) 1369 specific test cases are defined in [I-D.ietf-rmcat-wireless-tests]. 1371 8. Security Considerations 1373 The security considerations in [I-D.ietf-rmcat-eval-criteria] and the 1374 relevant congestion control algorithms apply. The principles for 1375 congestion control are described in [RFC2914], and in particular any 1376 new method must implement safeguards to avoid congestion collapse of 1377 the Internet. 1379 The evaluation of the test cases are intended to be run in a 1380 controlled lab environment. Hence, the applications, simulators and 1381 network nodes ought to be well-behaved and should not impact the 1382 desired results. Moreover, proper measures must be taken to avoid 1383 leaking non-responsive traffic from unproven congestion avoidance 1384 techniques onto the open Internet. 1386 9. IANA Considerations 1388 There are no IANA impacts in this memo. 1390 10. Acknowledgements 1392 Much of this document is derived from previous work on congestion 1393 control at the IETF. 1395 The content and concepts within this document are a product of the 1396 discussion carried out in the Design Team. 1398 11. References 1400 11.1. Normative References 1402 [I-D.ietf-rmcat-cc-requirements] 1403 Jesup, R. and Z. Sarker, "Congestion Control Requirements 1404 for Interactive Real-Time Media", draft-ietf-rmcat-cc- 1405 requirements-09 (work in progress), December 2014. 1407 [I-D.ietf-rmcat-eval-criteria] 1408 Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion 1409 Control for Interactive Real-time Media", draft-ietf- 1410 rmcat-eval-criteria-08 (work in progress), November 2018. 1412 [I-D.ietf-rmcat-video-traffic-model] 1413 Zhu, X., Cruz, S., and Z. Sarker, "Video Traffic Models 1414 for RTP Congestion Control Evaluations", draft-ietf-rmcat- 1415 video-traffic-model-07 (work in progress), February 2019. 1417 [I-D.ietf-rmcat-wireless-tests] 1418 Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and 1419 M. Ramalho, "Evaluation Test Cases for Interactive Real- 1420 Time Media over Wireless Networks", draft-ietf-rmcat- 1421 wireless-tests-06 (work in progress), December 2018. 1423 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1424 Jacobson, "RTP: A Transport Protocol for Real-Time 1425 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 1426 July 2003, . 1428 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 1429 Video Conferences with Minimal Control", STD 65, RFC 3551, 1430 DOI 10.17487/RFC3551, July 2003, 1431 . 1433 [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., 1434 "RTP Control Protocol Extended Reports (RTCP XR)", 1435 RFC 3611, DOI 10.17487/RFC3611, November 2003, 1436 . 1438 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 1439 "Extended RTP Profile for Real-time Transport Control 1440 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 1441 DOI 10.17487/RFC4585, July 2006, 1442 . 1444 [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size 1445 Real-Time Transport Control Protocol (RTCP): Opportunities 1446 and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 1447 2009, . 1449 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1450 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1451 . 1453 [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., 1454 and K. Carlberg, "Explicit Congestion Notification (ECN) 1455 for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 1456 2012, . 1458 11.2. Informative References 1460 [HEVC-seq] 1461 HEVC, "Test Sequences", 1462 http://www.netlab.tkk.fi/~varun/test_sequences/ . 1464 [I-D.ietf-rmcat-coupled-cc] 1465 Islam, S., Welzl, M., and S. Gjessing, "Coupled congestion 1466 control for RTP media", draft-ietf-rmcat-coupled-cc-08 1467 (work in progress), January 2019. 1469 [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, 1470 RFC 2914, DOI 10.17487/RFC2914, September 2000, 1471 . 1473 [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF 1474 Recommendations Regarding Active Queue Management", 1475 BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, 1476 . 1478 [RFC8033] Pan, R., Natarajan, P., Baker, F., and G. White, 1479 "Proportional Integral Controller Enhanced (PIE): A 1480 Lightweight Control Scheme to Address the Bufferbloat 1481 Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017, 1482 . 1484 [RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, 1485 J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler 1486 and Active Queue Management Algorithm", RFC 8290, 1487 DOI 10.17487/RFC8290, January 2018, 1488 . 1490 [xiph-seq] 1491 Xiph.org, "Video Test Media", 1492 http://media.xiph.org/video/derf/ . 1494 Authors' Addresses 1495 Zaheduzzaman Sarker 1496 Ericsson AB 1497 Torshamnsgatan 23 1498 Stockholm, SE 164 83 1499 Sweden 1501 Phone: +46 10 717 37 43 1502 Email: zaheduzzaman.sarker@ericsson.com 1504 Varun Singh 1505 Nemu Dialogue Systems Oy 1506 Runeberginkatu 4c A 4 1507 Helsinki 00100 1508 Finland 1510 Email: varun.singh@iki.fi 1511 URI: http://www.callstats.io/ 1513 Xiaoqing Zhu 1514 Cisco Systems 1515 12515 Research Blvd 1516 Austing, TX 78759 1517 USA 1519 Email: xiaoqzhu@cisco.com 1521 Michael A. Ramalho 1522 Cisco Systems, Inc. 1523 6310 Watercrest Way Unit 203 1524 Lakewood Ranch, FL 34202-5211 1525 USA 1527 Phone: +1 919 476 2038 1528 Email: mramalho@cisco.com