idnits 2.17.1 draft-ietf-rmcat-eval-test-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 9 instances of too long lines in the document, the longest one being 4 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 26, 2018) is 1949 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: '300ms' is mentioned on line 1012, but not defined == Missing Reference: '1000ms' is mentioned on line 1012, but not defined == Outdated reference: A later version (-14) exists of draft-ietf-rmcat-eval-criteria-08 == Outdated reference: A later version (-07) exists of draft-ietf-rmcat-video-traffic-model-06 == Outdated reference: A later version (-11) exists of draft-ietf-rmcat-wireless-tests-05 Summary: 1 error (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group Z. Sarker 3 Internet-Draft Ericsson AB 4 Intended status: Informational V. Singh 5 Expires: May 30, 2019 callstats.io 6 X. Zhu 7 M. Ramalho 8 Cisco Systems 9 November 26, 2018 11 Test Cases for Evaluating RMCAT Proposals 12 draft-ietf-rmcat-eval-test-08 14 Abstract 16 The Real-time Transport Protocol (RTP) is used to transmit media in 17 multimedia telephony applications. These applications are typically 18 required to implement congestion control. This document describes 19 the test cases to be used in the performance evaluation of such 20 congestion control algorithms in a controlled environment. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on May 30, 2019. 39 Copyright Notice 41 Copyright (c) 2018 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (https://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 57 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 58 3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 59 4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 7 60 4.1. Evaluation metrics . . . . . . . . . . . . . . . . . . . 8 61 4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 62 4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 63 5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 64 5.1. Variable Available Capacity with a Single Flow . . . . . 10 65 5.2. Variable Available Capacity with Multiple Flows . . . . . 13 66 5.3. Congested Feedback Link with Bi-directional Media Flows . 14 67 5.4. Competing Media Flows with same Congestion Control 68 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 17 69 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19 70 5.6. Media Flow Competing with a Long TCP Flow . . . . . . . . 21 71 5.7. Media Flow Competing with Short TCP Flows . . . . . . . . 23 72 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25 73 6. Other potential test cases . . . . . . . . . . . . . . . . . 27 74 6.1. Media Flows with Priority . . . . . . . . . . . . . . . . 27 75 6.2. Explicit Congestion Notification Usage . . . . . . . . . 27 76 6.3. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 27 77 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 30 78 8. Security Considerations . . . . . . . . . . . . . . . . . . . 30 79 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 80 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 81 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 30 82 11.1. Normative References . . . . . . . . . . . . . . . . . . 30 83 11.2. Informative References . . . . . . . . . . . . . . . . . 31 84 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32 86 1. Introduction 88 This memo describes a set of test cases for evaluating congestion 89 control algorithm proposals for real-time interactive media. It is 90 based on the guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] 91 and the requirements discussed in [I-D.ietf-rmcat-cc-requirements]. 92 The test cases cover basic usage scenarios and are described using a 93 common structure, which allows for additional test cases to be added 94 to those described herein to accommodate other topologies and/or the 95 modelling of different path characteristics. The described test 96 cases in this memo SHOULD be used to evaluate any proposed congestion 97 control algorithm for real-time interactive media. 99 2. Terminology 101 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 102 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 103 document are to be interpreted as described in RFC2119 [RFC2119]. 105 In addition, the terminology defined in RTP [RFC3550], RTP Profile 106 for Audio and Video Conferences with Minimal Control [RFC3551], RTCP 107 Extended Report (XR) [RFC3611], Extended RTP Profile for RTCP-based 108 Feedback (RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP 109 [RFC5506] apply. 111 3. Structure of Test cases 113 All the test cases in this document follow a basic structure allowing 114 implementers to describe a new test scenario without repeatedly 115 explaining common attributes. The structure includes a general 116 description section that describes the test case and its motivation. 117 Additionally the test case defines a set of attributes that 118 characterize the testbed, for example, the network path between 119 communicating peers and the diverse traffic sources. 121 o Define the test case: 123 * General description: describes the motivation and the goals of 124 the test case. 126 * Expected behavior: describes the desired rate adaptation 127 behavior. 129 * Define a list of metrics to evaluate the desired behavior: this 130 indicates the minimum set of metrics (e.g., link utilization, 131 media sending rate) that a proposed algorithm needs to measure 132 to validate the expected rate adaptation behavior. It should 133 also indicate the time granularity (e.g., averaged over 10ms, 134 100ms, or 1s) for measuring certain metrics. Typical 135 measurement interval is 200ms. 137 o Define testbed topology: every test case needs to define an 138 evaluation testbed topology. Figure 1 shows such an evaluation 139 topology. In this evaluation topology, S1..Sn are traffic 140 sources. These sources generate media traffic and use the 141 congestion control algorithm(s) under investigation. R1..Rn are 142 the corresponding receivers. A test case can have one or more 143 such traffic sources (S) and their corresponding receivers (R). 145 The path from the source to destination is denoted as "forward" 146 and the path from a destination to a source is denoted as 147 "backward". The following basic structure of the test case has 148 been described from the perspective of media generating endpoints 149 attached on the left-hand side of Figure 1. In this setup, the 150 media flows are transported in forward direction and corresponding 151 feedback/control messages are transported in the backward 152 direction. However, it is also possible to set up the test with 153 media in both forward and backward directions. In that case, 154 unless otherwise specified by the test case, it is expected that 155 the backward path does not introduce any congestion related 156 impairments and has enough capacity to accommodate both media and 157 feedback/control messages. It should be noted that depending on 158 the test cases it is possible to have different path 159 characteristics in either of the directions. 161 +---+ +---+ 162 |S1 |====== \ Forward --> / =======|R1 | 163 +---+ \\ // +---+ 164 \\ // 165 +---+ +-----+ +-----+ +---+ 166 |S2 |=======| A |------------------------------>| B |=======|R2 | 167 +---+ | |<------------------------------| | +---+ 168 +-----+ +-----+ 169 (...) // \\ (...) 170 // <-- Backward \\ 171 +---+ // \\ +---+ 172 |Sn |====== / \ ======|Rn | 173 +---+ +---+ 175 Figure 1: Example of A Testbed Topology 177 In a testbed environment with real equipments, there may exist a 178 significant amount of unwanted traffic on the portions of the 179 network path between the endpoints. Some of this traffic may be 180 generated by other processes on the endpoints themselves (e.g., 181 discovery protocols) or by other endpoints not presently under 182 test. Such unwanted traffic should be removed or avoided to the 183 greatest extent possible. 185 o Define testbed attributes: 187 * Duration: defines the duration of the test in seconds. 189 * Path characteristics: defines the end-to-end transport level 190 path characteristics of the testbed for a particular test case. 191 Two sets of attributes describe the path characteristics, one 192 for the forward path and the other for the backward path. The 193 path characteristics for a particular path direction is 194 applicable to all the Sources "S" sending traffic on that path. 195 If only one attribute is specified, it is used for both path 196 directions, however, unless specified the reverse path has no 197 capacity restrictions and no path loss. 199 + Path direction: forward or backward. 201 + Bottleneck-link capacity: defines minimum capacity of the 202 end-to-end path 204 + Reference bottleneck capacity: defines a reference value for 205 the bottleneck capacity for test cases with time-varying 206 bottleneck capacities. All bottleneck capacities will be 207 specified as a ratio with respect to the reference capacity 208 value. 210 + One-way propagation delay: describes the end-to-end latency 211 along the path when network queues are empty, i.e., the time 212 it takes for a packet to go from the sender to the receiver 213 without encountering any queuing delay. 215 + Maximum end-to-end jitter: defines the maximum jitter that 216 can be observed along the path. 218 + Bottleneck queue type: for example, Droptail, FQ-CoDel, or 219 PIE. 221 + Bottleneck queue size: defines the size of queue in terms of 222 queuing time when the queue is full (in milliseconds). 224 + Path loss ratio: characterizes the non-congested, additive, 225 losses to be generated on the end-to-end path. MUST 226 describe the loss pattern or loss model used to generate the 227 losses. 229 * Application-related: defines the traffic source behavior for 230 implementing the test case 232 + Media traffic Source: defines the characteristics of the 233 media sources. When using more than one media source, the 234 different attributes are enumerated separately for each 235 different media source. 237 - Media type: Video/Voice 239 - Media flow direction: forward, backward or both. 241 - Number of media sources: defines the total number of 242 media sources 244 - Media codec: Constant Bit Rate (CBR) or Variable Bit Rate 245 (VBR) 247 - Media source behavior: describes the media encoder 248 behavior. It defines the main parameters that affect the 249 adaptation behavior. This may include but is not limited 250 to: 252 o Adaptability: describes the adaptation options. For 253 example, in the case of video it defines the following 254 ranges of adaptation: bit rate, frame rate, video 255 resolution. Similarly, in the case of voice, it 256 defines the range of bit rate adaptation, the sampling 257 rate variation, and the variation in packetization 258 interval. 260 o Output variation : for a VBR encoder it defines the 261 encoder output variation from the average target rate 262 over a particular measurement interval. For example, 263 on average the encoder output may vary between 5% to 264 15% above or below the average target bit rate when 265 measured over a 100 ms time window. The time interval 266 over which the variation is specified MUST be 267 provided. 269 o Responsiveness to a new bit rate request: the lag in 270 time between a new bit rate request from the 271 congestion control algorithm and actual rate changes 272 in encoder output. Depending on the encoder, this 273 value may be specified in absolute time (e.g. 10ms to 274 1000ms) or other appropriate metric (e.g. next frame 275 interval time). 277 More detailed discussions on expected media source 278 behavior, including those from synthetic video traffic 279 sources, is at [I-D.ietf-rmcat-video-traffic-model]. 281 - Media content: describes the chosen media sequences; For 282 example, test sequences are available at: [xiph-seq] and 283 [HEVC-seq]. 285 - Media timeline: describes the point when the media source 286 is introduced and removed from the testbed. For example, 287 the media source may start transmitting immediately when 288 the test case begins, or after a few seconds. 290 - Startup behavior: the media starts at a defined bit rate, 291 which may be the minimum, maximum bit rate, or a value in 292 between (in Kbps). 294 + Competing traffic source: describes the characteristics of 295 the competing traffic source, the different types of 296 competing flows are enumerated in 297 [I-D.ietf-rmcat-eval-criteria]. 299 - Traffic direction: forward, backward or both. 301 - Type of sources: defines the types of competing traffic 302 sources. Types of competing traffic flows are listed in 303 [I-D.ietf-rmcat-eval-criteria]. For example, the number 304 of TCP flows connected to a web browser, the mean size 305 and distribution of the content downloaded. 307 - Number of sources: defines the total number of competing 308 sources of each media type per traffic direction. 310 - Congestion control: enumerates the congestion control 311 used by each type of competing traffic. 313 - Traffic timeline: describes when the competing traffic 314 starts and ends in the test case. 316 * Additional attributes: describes attributes essential for 317 implementing a test case which are not included in the above 318 structure. These attributes MUST be well defined, so that the 319 other implementers of that particular test case are able to 320 implement it easily. 322 Any attribute can have a set of values (enclosed within "[]"). Each 323 member value of such a set MUST be treated as different value for the 324 same attribute. It is desired to run separate tests for each such 325 attribute value. 327 The test cases described in this document follow the above structure. 329 4. Recommended Evaluation Settings 331 This section describes recommended test case settings and could be 332 overwritten by the respective test cases. 334 4.1. Evaluation metrics 336 To evaluate the performance of the candidate algorithms the 337 implementers MUST log enough information to visualize the following 338 metrics at a fine enough time granularity: 340 1. Flow level: 342 A. End-to-end delay for the congestion controlled media flow(s). 344 B. Variation in sending bit rate and goodput. Mainly observing 345 the frequency and magnitude of oscillations. 347 C. Packet losses observed at the receiving endpoint. 349 D. Feedback message overhead. 351 E. Convergence time - time to reach steady state for the 352 congestion controlled media flow(s). 354 2. Transport level: 356 A. Bandwidth utilization. 358 B. Queue length (milliseconds at specified path capacity): 360 + average over the length of the session. 362 + 5 and 95 percentile. 364 + median, maximum, minimum. 366 4.2. Path characteristics 368 Each path between a sender and receiver as described in Figure 1 have 369 the following characteristics unless otherwise specified in the test 370 case. 372 o Path direction: forward and backward. 374 o Reference bottleneck capacity: 1Mbps. 376 o One-Way propagation delay: 50ms. Implementers are encouraged to 377 run the experiment with additional propagation delays mentioned in 378 [I-D.ietf-rmcat-eval-criteria] 380 o Maximum end-to-end jitter: 30ms. Jitter models are described in 381 [I-D.ietf-rmcat-eval-criteria] 383 o Bottleneck queue type: Drop tail. Implementers are encouraged to 384 run the experiment with other AQM schemes, such as FQ-CoDel and 385 PIE. 387 o Bottleneck queue size: 300ms. 389 o Path loss ratio: 0%. 391 Examples of additional network parameters are discussed in 392 [I-D.ietf-rmcat-eval-criteria]. 394 For test cases involving time-varying bottleneck capacity, all 395 capacity values are specified as a ratio with respect to a reference 396 capacity value, so as to allow flexible scaling of capacity values 397 along with media source rate range. There exist two different 398 mechanisms for inducing path capacity variation: a) by explicitly 399 modifying the value of physical link capacity; or b) by introducing 400 background non-adaptive UDP traffic with time-varying traffic rate. 401 Implementers are encouraged to run the experiments with both 402 mechanisms for test cases specified in Section 5.1, Section 5.2, and 403 Section 5.3. 405 4.3. Media source 407 Unless otherwise specified, each test case will include one or more 408 media sources as described below. 410 o Media type: Video 412 * Media codec: VBR 414 * Media source behavior: 416 + Adaptability: 418 - Bit rate range: 150 Kbps - 1.5 Mbps. In real-life 419 applications the bit rate range can vary a lot depending 420 on the provided service, for example, the maximum bit 421 rate can be up to 4Mbps. However, for running tests to 422 evaluate the congestion control algorithms it is more 423 important to have a look at how they are reacting to 424 certain amount of bandwidth change. Also it is possible 425 that the media traffic generator used in a particular 426 simulator or testbed is not capable of generating higher 427 bit rate. Hence we have selected a suitable bit rate 428 range typical of consumer-grade video conferencing 429 applications in designing the test case. If a different 430 bit rate range is used in the test cases, then the end- 431 to-end path capacity values will also need to be scaled 432 accordingly. 434 - Frame resolution: 144p - 720p (or 1080p). This 435 resolution range is selected based on the bit rate range. 436 If a different bit rate range is used in the test cases 437 then the frame resolution range also need to be selected 438 suitably. 440 - Frame rate: 10fps - 30fps. This frame rate range is 441 selected based on the bit rate range. If a different bit 442 rate range is used in the test cases then the frame rate 443 range also need to be adjusted suitably. 445 + Variation from target bit rate: +/-5%. Unless otherwise 446 specified in the test case(s), bit rate variation SHOULD be 447 calculated over one (1) second period of time. 449 + Responsiveness to new bit rate request: 100ms 451 * Media content: The media content should represent a typical 452 video conversational scenario with head and shoulder movement. 453 We recommend to use Foreman video sequence. 455 * Media startup behavior: 150Kbps. It should be noted that 456 applications can use smart ways to select an optimal startup 457 bit rate value for a certain network condition. In such cases 458 the candidate proposals MAY show the effectiveness of such 459 smart approach as an additional information for the evaluation 460 process. 462 o Media type: Audio 464 * Media codec: CBR 466 * Media bit rate: 20Kbps 468 5. Basic Test Cases 470 5.1. Variable Available Capacity with a Single Flow 472 In this test case the bottleneck-link capacity between the two 473 endpoints varies over time. This test is designed to measure the 474 responsiveness of the candidate algorithm. This test tries to 475 address the requirements in [I-D.ietf-rmcat-cc-requirements], which 476 requires the algorithm to adapt the flow(s) and provide lower end-to- 477 end latency when there exists: 479 o an intermediate bottleneck 481 o change in available capacity (e.g., due to interface change, 482 routing change, abrupt arrival/departure of background non- 483 adaptive traffic). 485 o maximum media bit rate is greater than link capacity. In this 486 case, when the application tries to ramp up to its maximum bit 487 rate, since the link capacity is limited to a value lower, the 488 congestion control scheme is expected to stabilize the sending bit 489 rate close to the available bottleneck capacity. 491 It should be noted that the exact variation in available capacity due 492 to any of the above depends on the underlying technologies. Hence, 493 we describe a set of known factors, which may be extended to devise a 494 more specific test case targeting certain behaviors in a certain 495 network environment. 497 Expected behavior: the candidate algorithm is expected to detect the 498 path capacity constraint, converges to the bottleneck link's capacity 499 and adapt the flow to avoid unwanted media rate oscillation when the 500 sending bit rate is approaching the bottleneck link's capacity. Such 501 oscillations might occur when the media flow(s) attempts to reach its 502 maximum bit rate but overshoots the usage of the available bottleneck 503 capacity then to rectify, it reduces the bit rate and starts to ramp 504 up again. 506 Evaluation metrics : as described in Section 4.1. 508 Testbed topology: One media source S1 is connected to the 509 corresponding R1. The media traffic is transported over the forward 510 path and corresponding feedback/control traffic is transported over 511 the backward path. 513 Forward --> 514 +---+ +-----+ +-----+ +---+ 515 |S1 |=======| A |------------------------------>| B |=======|R1 | 516 +---+ | |<------------------------------| | +---+ 517 +-----+ +-----+ 518 <-- Backward 520 Figure 2: Testbed Topology for Limited Link Capacity 522 Testbed attributes: 524 o Test duration: 100s 525 o Path characteristics: as described in Section 4.2 527 o Application-related: 529 * Media Traffic: 531 + Media type: Video 533 - Media direction: forward. 535 - Number of media sources: one (1) 537 - Media timeline: 539 o Start time: 0s. 541 o End time: 99s. 543 + Media type: Audio 545 - Media direction: forward. 547 - Number of media sources: one (1) 549 - Media timeline: 551 o Start time: 0s. 553 o End time: 99s. 555 * Competing traffic: 557 + Number of sources : zero (0) 559 o Test Specific Information: 561 * One-way propagation delay: [ 50 ms, 100 ms]. on the forward 562 path direction 564 * This test uses bottleneck path capacity variation as listed in 565 Table 1 567 * When using background non-adaptive UDP traffic to induce time- 568 varying bottleneck , the physical path capacity remains at 569 4Mbps and the UDP traffic source rate changes over time as (4 - 570 (Y x r)), where r is the Reference bottleneck capacity in Mbps 571 and Y is the path capacity ratio specified in Table 1 573 +--------------------+--------------+-----------+-------------------+ 574 | Variation pattern | Path | Start | Path capacity | 575 | index | direction | time | ratio | 576 +--------------------+--------------+-----------+-------------------+ 577 | One | Forward | 0s | 1.0 | 578 | Two | Forward | 40s | 2.5 | 579 | Three | Forward | 60s | 0.6 | 580 | Four | Forward | 80s | 1.0 | 581 +--------------------+--------------+-----------+-------------------+ 583 Table 1: Path capacity variation pattern for forward direction 585 5.2. Variable Available Capacity with Multiple Flows 587 This test case is similar to Section 5.1. However in addition this 588 test will also consider persistent network load due to competing 589 traffic. 591 Expected behavior: the candidate algorithm is expected to detect the 592 variation in available capacity and adapt the media stream(s) 593 accordingly. The flows stabilize around their maximum bit rate as 594 the maximum link capacity is large enough to accommodate the flows. 595 When the available capacity drops, the flows adapt by decreasing 596 their sending bit rate, and when congestion disappears, the flows are 597 again expected to ramp up. 599 Evaluation metrics : as described in Section 4.1. 601 Testbed Topology: Two (2) media sources S1 and S2 are connected to 602 their corresponding destinations R1 and R2. The media traffic is 603 transported over the forward path and corresponding feedback/control 604 traffic is transported over the backward path. 606 +---+ +---+ 607 |S1 |===== \ / =======|R1 | 608 +---+ \\ Forward --> // +---+ 609 \\ // 610 +-----+ +-----+ 611 | A |------------------------------>| B | 612 | |<------------------------------| | 613 +-----+ +-----+ 614 // \\ 615 // <-- Backward \\ 616 +---+ // \\ +---+ 617 |S2 |====== / \ ======|R2 | 618 +---+ +---+ 620 Figure 3: Testbed Topology for Variable Available Capacity 622 Testbed attributes: 624 Testbed attributes are similar as described in Section 5.1 except the 625 test specific capacity variation setup. 627 Test Specific Information: This test uses path capacity variation as 628 listed in Table 2 with a corresponding end time of 125 seconds. The 629 reference bottleneck capacity is 2Mbps. When using background non- 630 adaptive UDP traffic to induce time-varying bottleneck for congestion 631 controlled media flows, the physical path capacity is 4Mbps and the 632 UDP traffic source rate changes over time as (4 - (Y x r)), where r 633 is the Reference bottleneck capacity in Mbps and Y is the path 634 capacity ratio specified in Table 2. 636 +--------------------+--------------+-----------+-------------------+ 637 | Variation pattern | Path | Start | Path capacity | 638 | index | direction | time | ratio | 639 +--------------------+--------------+-----------+-------------------+ 640 | One | Forward | 0s | 2.0 | 641 | Two | Forward | 25s | 1.0 | 642 | Three | Forward | 50s | 1.75 | 643 | Four | Forward | 75s | 0.5 | 644 | Five | Forward | 100s | 1.0 | 645 +--------------------+--------------+-----------+-------------------+ 647 Table 2: Path capacity variation pattern for forward direction 649 5.3. Congested Feedback Link with Bi-directional Media Flows 651 Real-time interactive media uses RTP hence it is assumed that RTCP, 652 RTP header extension or such would be used by the congestion control 653 algorithm in the backchannel. Due to asymmetric nature of the link 654 between communicating peers it is possible for a participating peer 655 to not receive such feedback information due to an impaired or 656 congested backchannel (even when the forward channel might not be 657 impaired). This test case is designed to observe the candidate 658 congestion control behavior in such an event. 660 Expected behavior: It is expected that the candidate algorithms are 661 able to cope with the lack of feedback information and adapt to 662 minimize the performance degradation of media flows in the forward 663 channel. 665 It should be noted that for this test case: logs are compared with 666 the reference case, i.e, when the backward channel has no 667 impairments. 669 Evaluation metrics : as described in Section 4.1. 671 Testbed topology: One (1) media source S1 is connected to 672 corresponding R1, but both endpoints are additionally receiving and 673 sending data, respectively. The media traffic (S1->R1) is 674 transported over the forward path and corresponding feedback/control 675 traffic is transported over the backward path. Likewise media 676 traffic (S2->R2) is transported over the backward path and 677 corresponding feedback/control traffic is transported over the 678 forward path. 680 +---+ +---+ 681 |S1 |===== \ Forward --> / =======|R1 | 682 +---+ \\ // +---+ 683 \\ // 684 +-----+ +-----+ 685 | A |------------------------------>| B | 686 | |<------------------------------| | 687 +-----+ +-----+ 688 // \\ 689 // <-- Backward \\ 690 +---+ // \\ +---+ 691 |R2 |===== / \ ======|S2 | 692 +---+ +---+ 694 Figure 4: Testbed Topology for Congested Feedback Link 696 Testbed attributes: 698 o Test duration: 100s 700 o Path characteristics: 702 * Reference bottleneck capacity: 1Mbps. 704 o Application-related: 706 * Media Source: 708 + Media type: Video 710 - Media direction: forward and backward 712 - Number of media sources: two (2) 714 - Media timeline: 716 o Start time: 0s. 718 o End time: 99s. 720 + Media type: Audio 722 - Media direction: forward and backward 724 - Number of media sources: two (2) 726 - Media timeline: 728 o Start time: 0s. 730 o End time: 99s. 732 * Competing traffic: 734 + Number of sources : zero (0) 736 o Test Specific Information: this test uses path capacity variations 737 to create congested feedback link. Table 3 lists the variation 738 patterns applied to the forward path and Table 4 lists the 739 variation patterns applied to the backward path. When using 740 background non-adaptive UDP traffic to induce time-varying 741 bottleneck for congestion controlled media flows, the physical 742 path capacity is 4Mbps for both directions and the UDP traffic 743 source rate changes over time as (4-x)Mbps in each direction, 744 where x is the bottleneck capacity specified in Table 4. 746 +--------------------+--------------+-----------+-------------------+ 747 | Variation pattern | Path | Start | Path capacity | 748 | index | direction | time | ratio | 749 +--------------------+--------------+-----------+-------------------+ 750 | One | Forward | 0s | 2.0 | 751 | Two | Forward | 20s | 1.0 | 752 | Three | Forward | 40s | 0.5 | 753 | Four | Forward | 60s | 2.0 | 754 +--------------------+--------------+-----------+-------------------+ 756 Table 3: Path capacity variation pattern for forward direction 758 +--------------------+--------------+-----------+-------------------+ 759 | Variation pattern | Path | Start | Path capacity | 760 | index | direction | time | ratio | 761 +--------------------+--------------+-----------+-------------------+ 762 | One | Backward | 0s | 2.0 | 763 | Two | Backward | 35s | 0.8 | 764 | Three | Backward | 70s | 2.0 | 765 +--------------------+--------------+-----------+-------------------+ 767 Table 4: Path capacity variation pattern for backward direction 769 5.4. Competing Media Flows with same Congestion Control Algorithm 771 In this test case, more than one media flow share the bottleneck link 772 and each of them uses the same congestion control algorithm. This is 773 a typical scenario where a real-time interactive application sends 774 more than one media flow to the same destination and these flows are 775 multiplexed over the same port. In such a scenario it is likely that 776 the flows will be routed via the same path and need to share the 777 available bandwidth amongst themselves. For the sake of simplicity 778 it is assumed that there are no other competing traffic sources in 779 the bottleneck link and that there is sufficient capacity to 780 accommodate all the flows individually. While this appears to be a 781 variant of the test case defined in Section 5.2, it focuses on the 782 capacity sharing aspect of the candidate algorithm. The previous 783 test case, on the other hand, measures adaptability, stability, and 784 responsiveness of the candidate algorithm. 786 Expected behavior: It is expected that the competing flows will 787 converge to an optimum bit rate to accommodate all the flows with 788 minimum possible latency and loss. Specifically, the test introduces 789 three media flows at different time instances, when the second flow 790 appears there should still be room to accommodate another flow on the 791 bottleneck link. Lastly, when the third flow appears the bottleneck 792 link should be saturated. 794 Evaluation metrics : as described in Section 4.1. 796 Testbed topology: Three media sources S1, S2, S3 are connected to R1, 797 R2, R3 respectively. The media traffic is transported over the 798 forward path and corresponding feedback/control traffic is 799 transported over the backward path. 801 +---+ +---+ 802 |S1 |===== \ Forward --> / =======|R1 | 803 +---+ \\ // +---+ 804 \\ // 805 +---+ +-----+ +-----+ +---+ 806 |S2 |=======| A |------------------------------>| B |=======|R2 | 807 +---+ | |<------------------------------| | +---+ 808 +-----+ +-----+ 809 // \\ 810 // <-- Backward \\ 811 +---+ // \\ +---+ 812 |S3 |====== / \ ======|R3 | 813 +---+ +---+ 815 Figure 5: Testbed Topology for Multiple congestion controlled media 816 Flows 818 Testbed attributes: 820 o Test duration: 120s 822 o Path characteristics: 824 * Reference bottleneck capacity: 3.5Mbps 826 * Path capacity ratio: 1.0 828 o Application-related: 830 * Media Source: 832 + Media type: Video 834 - Media direction: forward. 836 - Number of media sources: three (3) 838 - Media timeline: new media flows are added sequentially, 839 at short time intervals. See test specific setup below. 841 + Media type: Audio 843 - Media direction: forward. 845 - Number of media sources: three (3) 847 - Media timeline: new media flows are added sequentially, 848 at short time intervals. See test specific setup below. 850 * Competing traffic: 852 + Number of sources : zero (0) 854 o Test Specific Information: Table 5 defines the media timeline for 855 both media type. 857 +---------+------------+------------+----------+ 858 | Flow ID | Media type | Start time | End time | 859 +---------+------------+------------+----------+ 860 | 1 | Video | 0s | 119s | 861 | 2 | Video | 20s | 119s | 862 | 3 | Video | 40s | 119s | 863 | 4 | Audio | 0s | 119s | 864 | 5 | Audio | 20s | 119s | 865 | 6 | Audio | 40s | 119s | 866 +---------+------------+------------+----------+ 868 Table 5: Media Timeline for Video and Audio media sources 870 5.5. Round Trip Time Fairness 872 In this test case, multiple media flows share the bottleneck link, 873 but the one-way propagation delay for each flow is different. For 874 the sake of simplicity it is assumed that there are no other 875 competing traffic sources in the bottleneck link and that there is 876 sufficient capacity to accommodate all the flows. While this appears 877 to be a variant of test case 5.2, it focuses on the capacity sharing 878 aspect of the candidate algorithm under different RTTs. 880 Expected behavior: It is expected that the competing flows will 881 converge to bit rates to accommodate all the flows with minimum 882 possible latency and loss. Specifically, the test introduces five 883 media flows at the same time instance. 885 Evaluation metrics : as described in Section 4.1. 887 Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to 888 their corresponding media sinks R1,R2,..,R5. The media traffic is 889 transported over the forward path and corresponding feedback/control 890 traffic is transported over the backward path. The topology is the 891 same as in Section 5.4. 893 Testbed attributes: 895 o Test duration: 300s 897 o Path characteristics: 899 * Reference bottleneck capacity: 4Mbps 901 * Path capacity ratio: 1.0 903 * One-Way propagation delay for each flow: 10ms for S1-R1, 25ms 904 for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms S5-R5. 906 o Application-related: 908 * Media Source: 910 + Media type: Video 912 - Media direction: forward 914 - Number of media sources: five (5) 916 - Media timeline: new media flows are added sequentially, 917 at short time intervals. See test specific setup below. 919 + Media type: Audio 921 - Media direction: forward. 923 - Number of media sources: five (5) 925 - Media timeline: new media flows are added sequentially, 926 at short time intervals. See test specific setup below. 928 * Competing traffic: 930 + Number of sources : zero (0) 932 o Test Specific Information: Table 6 defines the media timeline for 933 both media type. 935 +---------+------------+------------+----------+ 936 | Flow IF | Media type | Start time | End time | 937 +---------+------------+------------+----------+ 938 | 1 | Video | 0s | 299s | 939 | 2 | Video | 10s | 299s | 940 | 3 | Video | 20s | 299s | 941 | 4 | Video | 30s | 299s | 942 | 5 | Video | 40s | 299s | 943 | 6 | Audio | 0 | 299s | 944 | 7 | Audio | 10s | 299s | 945 | 8 | Audio | 20s | 299s | 946 | 9 | Audio | 30s | 299s | 947 | 10 | Audio | 40s | 299s | 948 +---------+------------+------------+----------+ 950 Table 6: Media Timeline for Video and Audio media sources 952 5.6. Media Flow Competing with a Long TCP Flow 954 In this test case, one or more media flows share the bottleneck link 955 with at least one long lived TCP flow. Long lived TCP flows download 956 data throughout the session and are expected to have infinite amount 957 of data to send and receive. This is a scenario where a multimedia 958 application co-exists with a large file download. The test case 959 measures the adaptivity of the candidate algorithm to competing 960 traffic. It addresses the requirement 3 in 961 [I-D.ietf-rmcat-cc-requirements]. 963 Expected behavior: depending on the convergence observed in test case 964 5.1 and 5.2, the candidate algorithm may be able to avoid congestion 965 collapse. In the worst case, the media stream will fall to the 966 minimum media bit rate. 968 Evaluation metrics : following metrics in addition to as described in 969 Section 4.1. 971 1. Flow level: 973 A. TCP throughput. 975 B. Loss for the TCP flow 977 Testbed topology: One (1) media source S1 is connected to the 978 corresponding media sink, R1. In addition, there is a long-live TCP 979 flow sharing the same bottleneck link. The media traffic is 980 transported over the forward path and corresponding feedback/control 981 traffic is transported over the backward path. The TCP traffic goes 982 over the forward path from, S_tcp with acknowledgment packets go over 983 the backward path from, R_tcp. 985 +--+ +--+ 986 |S1|===== \ Forward --> / =======|R1| 987 +--+ \\ // +--+ 988 \\ // 989 +-----+ +-----+ 990 | A |---------------------------->| B | 991 | |<----------------------------| | 992 +-----+ +-----+ 993 // \\ 994 // <-- Backward \\ 995 +-----+ // \\ +-----+ 996 |S_tcp|=== / \ ===|R_tcp| 997 +-----+ +-----+ 999 Figure 6: Testbed Topology for TCP vs congestion controlled media 1000 Flows 1002 Testbed attributes: 1004 o Test duration: 120s 1006 o Path characteristics: 1008 * Reference bottleneck capacity: 2Mbps 1010 * Path capacity ratio: 1.0 1012 * Bottleneck queue size: [300ms, 1000ms] 1014 o Application-related: 1016 * Media Source: 1018 + Media type: Video 1020 - Media direction: forward 1022 - Number of media sources: one (1) 1024 - Media timeline: 1026 o Start time: 5s. 1028 o End time: 119s. 1030 + Media type: Audio 1032 - Media direction: forward 1033 - Number of media sources: one (1) 1035 - Media timeline: 1037 o Start time: 5s. 1039 o End time: 119s. 1041 * Additionally, implementers are encouraged to run the experiment 1042 with multiple media sources. 1044 * Competing traffic: 1046 + Number and Types of sources : one (1) and long-lived TCP 1048 + Traffic direction : forward 1050 + Congestion control: default TCP congestion control[RFC5681]. 1052 + Traffic timeline: 1054 - Start time: 0s. 1056 - End time: 119s. 1058 o Test Specific Information: none 1060 5.7. Media Flow Competing with Short TCP Flows 1062 In this test case, one or more congestion controlled media flow 1063 shares the bottleneck link with multiple short-lived TCP flows. 1064 Short-lived TCP flows resemble the on/off pattern observed in the web 1065 traffic, wherein clients (for example, browsers) connect to a server 1066 and download a resource (typically a web page, few images, text 1067 files, etc.) using several TCP connections. This scenario shows the 1068 performance of a multimedia application when several browser windows 1069 are active. The test case measures the adaptivity of the candidate 1070 algorithm to competing web traffic, it addresses the requirements 1.E 1071 in [I-D.ietf-rmcat-cc-requirements]. 1073 Depending on the number of short TCP flows, the cross-traffic either 1074 appears as a short burst flow or resembles a long TCP flow. The 1075 intention of this test is to observe the impact of short-term burst 1076 on the behavior of the candidate algorithm. 1078 Expected behavior: The candidate algorithm is expected to avoid flow 1079 starvation during the presence of short and bursty competing TCP 1080 flows, streaming at least at the minimum media bit rate. After 1081 competing TCP flows terminate, the media streams are expected to be 1082 robust enough to eventually recover to previous steady state 1083 behavior, and at the very least, avoid persistent starvation. 1085 Evaluation metrics : following metrics in addition to as described in 1086 Section 4.1. 1088 1. Flow level: 1090 A. Variation in the sending rate of the TCP flow. 1092 B. TCP throughput. 1094 Testbed topology: The topology described here is same as the one 1095 described in Figure 6. 1097 Testbed attributes: 1099 o Test duration: 300s 1101 o Path characteristics: 1103 * Reference bottleneck capacity: 2.0Mbps 1105 * Path capacity ratio: 1.0 1107 o Application-related: 1109 * Media source: 1111 + Media type: Video 1113 - Media direction: forward 1115 - Number of media sources: two (2) 1117 - Media timeline: 1119 o Start time: 5s. 1121 o End time: 299s. 1123 + Media type: Audio 1125 - Media direction: forward 1127 - Number of media sources: two (2) 1128 - Media timeline: 1130 o Start time: 5s. 1132 o End time: 299s. 1134 * Competing traffic: 1136 + Number and Types of sources : ten (10), short-lived TCP 1137 flows. 1139 + Traffic direction : forward 1141 + Congestion algorithm: default TCP Congestion control 1142 [RFC5681]. 1144 + Traffic timeline: each short TCP flow is modeled as a 1145 sequence of file downloads interleaved with idle periods. 1146 Not all short TCP flows start at the same time, 2 of them 1147 start in the ON state while rest of the 8 flows start in an 1148 OFF state. For description of short TCP flow model see test 1149 specific information below. 1151 o Test Specific Information: 1153 * Short-TCP traffic model: The short TCP model to be used in this 1154 test is described in [I-D.ietf-rmcat-eval-criteria]. 1156 5.8. Media Pause and Resume 1158 In this test case, more than one real-time interactive media flows 1159 share the link bandwidth and all flows reach to a steady state by 1160 utilizing the link capacity in an optimum way. At this stage one of 1161 the media flows is paused for a moment. This event will result in 1162 more available bandwidth for the rest of the flows as they are on a 1163 shared link. When the paused media flow resumes it would no longer 1164 have the same bandwidth share on the link. It has to make it's way 1165 through the other existing flows in the link to achieve a fair share 1166 of the link capacity. This test case is important specially for 1167 real-time interactive media which consists of more than one media 1168 flows and can pause/resume media flows at any point of time during 1169 the session. This test case directly addresses the requirement 1170 number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a 1171 variation of test case defined in Section 5.4. However, it is 1172 different as the candidate algorithms can use different strategies to 1173 increase its efficiency, for example in terms of fairness, 1174 convergence time, reduce oscillation etc, by capitalizing the fact 1175 that they have previous information of the link. 1177 Expected behavior: During the period where the third stream is 1178 paused, the two remaining flows are expected to increase their rates 1179 and reach the maximum media bit rate. When the third stream resumes, 1180 all three flows are expected to converge to the same original fair 1181 share of rates prior to the media pause/resume event. 1183 Evaluation metrics : following metrics in addition to as described in 1184 Section 4.1. 1186 1. Flow level: 1188 A. Variation in sending bit rate and goodput. Mainly observing 1189 the frequency and magnitude of oscillations. 1191 Testbed Topology: Same as test case defined in Section 5.4 1193 Testbed attributes: The general description of the testbed parameters 1194 are same as Section 5.4 with changes in the test specific setup as 1195 below- 1197 o Other test specific setup: 1199 * Media flow timeline: 1201 + Flow ID: one (1) 1203 + Start time: 0s 1205 + Flow duration: 119s 1207 + Pause time: not required 1209 + Resume time: not required 1211 * Media flow timeline: 1213 + Flow ID: two (2) 1215 + Start time: 0s 1217 + Flow duration: 119s 1219 + Pause time: at 40s 1221 + Resume time: at 60s 1223 * Media flow timeline: 1225 + Flow ID: three (3) 1227 + Start time: 0s 1229 + Flow duration:119s 1231 + Pause time: not required 1233 + Resume time: not required 1235 6. Other potential test cases 1237 It has been noticed that there are other interesting test cases 1238 besides the basic test cases listed above. In many aspects, these 1239 additional test cases can help further evaluation of the candidate 1240 algorithm. They are listed as below. 1242 6.1. Media Flows with Priority 1244 In this test case media flows will have different priority levels. 1245 This will be an extension of Section 5.4 where the same test will be 1246 run with different priority levels imposed on each of the media 1247 flows. For example, the first flow (S1) is assigned a priority of 2 1248 whereas the remaining two flows (S2 and S3) are assigned a priority 1249 of 1. The candidate algorithm MUST reflect the relative priorities 1250 assigned to each media flow. In the previous example, the first flow 1251 (S1) MUST arrive at a steady-state rate approximately twice of that 1252 of the other two flows (S2 and S3). 1254 The candidate algorithm can use a coupled congestion control 1255 mechanism or use a weighted priority scheduler for the bandwidth 1256 distribution according to the respective media flow priority or use. 1258 6.2. Explicit Congestion Notification Usage 1260 This test case requires to run all the basic test cases with the 1261 availability of Explicit Congestion Notification (ECN) [RFC6679] 1262 feature enabled. The goal of this test is to exhibit that the 1263 candidate algorithms do not fail when ECN signals are available. 1264 With ECN signals enabled the algorithms are expected to perform 1265 better than their delay based variants. 1267 6.3. Multiple Bottlenecks 1269 In this test case one congestion controlled media flow, S1->R1, 1270 traverses a path with multiple bottlenecks. As illustrated in 1271 Figure 7, the first flow (S1->R1) competes with the second congestion 1272 controlled media flow (S2->R2) over the link between A and B which is 1273 close to the sender side; again, that flow (S1->R1) competes with the 1274 third congestion controlled media flow (S3->R3) over the link between 1275 C and D which is close to the receiver side. The goal of this test 1276 is to ensure that the candidate algorithms work properly in the 1277 presence of multiple bottleneck links on the end to end path. 1279 Expected behavior: the candidate algorithm is expected to achieve 1280 full utilization at both bottleneck links without starving any of the 1281 three congestion controlled media flows. 1283 Forward ----> 1285 +---+ +---+ +---+ +---+ 1286 |S2 | |R2 | |S3 | |R3 | 1287 +---+ +---+ +---+ +---+ 1288 | | | | 1289 | | | | 1290 +---+ +-----+ +-----+ +-----+ +-----+ +---+ 1291 |S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 | 1292 +---+ | |<------| |<-----| |<----| | +---+ 1293 +-----+ +-----+ +-----+ +-----+ 1295 1st 2nd 1296 Bottleneck (A->B) Bottleneck (C->D) 1298 <------ Backward 1300 Figure 7: Testbed Topology for Multiple Bottlenecks 1302 Testbed topology: Three media sources S1, S2, and S3 are connected to 1303 respective destinations R1, R2, and R3. For all three flows the 1304 media traffic is transported over the forward path and corresponding 1305 feedback/control traffic is transported over the backward path. 1307 Testbed attributes: 1309 o Test duration: 300s 1311 o Path characteristics: 1313 * Reference bottleneck capacity: 2Mbps. 1315 * Path capacity ratio between A and B: 1.0 1317 * Path capacity ratio between B and C: 4.0. 1319 * Path capacity ratio between C and D: 0.75. 1321 * One-Way propagation delay: 1323 1. Between S1 and R1: 100ms 1325 2. Between S2 and R2: 40ms 1327 3. Between S3 and R3: 40ms 1329 o Application-related: 1331 * Media Source: 1333 + Media type: Video 1335 - Media direction: Forward 1337 - Number of media sources: Three (3) 1339 - Media timeline: 1341 o Start time: 0s. 1343 o End time: 299s. 1345 + Media type: Audio 1347 - Media direction: Forward 1349 - Number of media sources: Three (3) 1351 - Media timeline: 1353 o Start time: 0s. 1355 o End time: 299s. 1357 * Competing traffic: 1359 + Number of sources : Zero (0) 1361 7. Wireless Access Links 1363 Additional wireless network (both cellular network and WiFi network) 1364 specific test cases are defined in [I-D.ietf-rmcat-wireless-tests]. 1366 8. Security Considerations 1368 The security considerations in [I-D.ietf-rmcat-eval-criteria] and the 1369 relevant congestion control algorithms apply. The principles for 1370 congestion control are described in [RFC2914], and in particular any 1371 new method MUST implement safeguards to avoid congestion collapse of 1372 the Internet. 1374 The evaluation of the test cases are intended to be run in a 1375 controlled lab environment. Hence, the applications, simulators and 1376 network nodes ought to be well-behaved and should not impact the 1377 desired results. It is important to take appropriate caution to 1378 avoid leaking non-responsive traffic from unproven congestion 1379 avoidance techniques onto the open Internet. 1381 9. IANA Considerations 1383 There are no IANA impacts in this memo. 1385 10. Acknowledgements 1387 Much of this document is derived from previous work on congestion 1388 control at the IETF. 1390 The content and concepts within this document are a product of the 1391 discussion carried out in the Design Team. 1393 11. References 1395 11.1. Normative References 1397 [I-D.ietf-rmcat-eval-criteria] 1398 Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion 1399 Control for Interactive Real-time Media", draft-ietf- 1400 rmcat-eval-criteria-08 (work in progress), November 2018. 1402 [I-D.ietf-rmcat-video-traffic-model] 1403 Zhu, X., Cruz, S., and Z. Sarker, "Video Traffic Models 1404 for RTP Congestion Control Evaluations", draft-ietf-rmcat- 1405 video-traffic-model-06 (work in progress), November 2018. 1407 [I-D.ietf-rmcat-wireless-tests] 1408 Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and 1409 M. Ramalho, "Evaluation Test Cases for Interactive Real- 1410 Time Media over Wireless Networks", draft-ietf-rmcat- 1411 wireless-tests-05 (work in progress), June 2018. 1413 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1414 Requirement Levels", BCP 14, RFC 2119, 1415 DOI 10.17487/RFC2119, March 1997, 1416 . 1418 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1419 Jacobson, "RTP: A Transport Protocol for Real-Time 1420 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 1421 July 2003, . 1423 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 1424 Video Conferences with Minimal Control", STD 65, RFC 3551, 1425 DOI 10.17487/RFC3551, July 2003, 1426 . 1428 [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., 1429 "RTP Control Protocol Extended Reports (RTCP XR)", 1430 RFC 3611, DOI 10.17487/RFC3611, November 2003, 1431 . 1433 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 1434 "Extended RTP Profile for Real-time Transport Control 1435 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, 1436 DOI 10.17487/RFC4585, July 2006, 1437 . 1439 [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size 1440 Real-Time Transport Control Protocol (RTCP): Opportunities 1441 and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 1442 2009, . 1444 [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., 1445 and K. Carlberg, "Explicit Congestion Notification (ECN) 1446 for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 1447 2012, . 1449 11.2. Informative References 1451 [HEVC-seq] 1452 HEVC, "Test Sequences", 1453 http://www.netlab.tkk.fi/~varun/test_sequences/ . 1455 [I-D.ietf-rmcat-cc-requirements] 1456 Jesup, R. and Z. Sarker, "Congestion Control Requirements 1457 for Interactive Real-Time Media", draft-ietf-rmcat-cc- 1458 requirements-09 (work in progress), December 2014. 1460 [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, 1461 RFC 2914, DOI 10.17487/RFC2914, September 2000, 1462 . 1464 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1465 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1466 . 1468 [xiph-seq] 1469 Xiph.org, "Video Test Media", 1470 http://media.xiph.org/video/derf/ . 1472 Authors' Addresses 1474 Zaheduzzaman Sarker 1475 Ericsson AB 1476 Luleae, SE 977 53 1477 Sweden 1479 Phone: +46 10 717 37 43 1480 Email: zaheduzzaman.sarker@ericsson.com 1482 Varun Singh 1483 Nemu Dialogue Systems Oy 1484 Runeberginkatu 4c A 4 1485 Helsinki 00100 1486 Finland 1488 Email: varun.singh@iki.fi 1489 URI: http://www.callstats.io/ 1491 Xiaoqing Zhu 1492 Cisco Systems 1493 12515 Research Blvd 1494 Austing, TX 78759 1495 USA 1497 Email: xiaoqzhu@cisco.com 1498 Michael A. Ramalho 1499 Cisco Systems, Inc. 1500 6310 Watercrest Way Unit 203 1501 Lakewood Ranch, FL 34202-5211 1502 USA 1504 Phone: +1 919 476 2038 1505 Email: mramalho@cisco.com