idnits 2.17.1 draft-ietf-rmcat-eval-test-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 231: '...enerated on the end-to-end path. MUST...' RFC 2119 keyword, line 319: '...These attributes MUST be well defined,...' RFC 2119 keyword, line 323: '...ue of such a set MUST be treated as di...' RFC 2119 keyword, line 424: '...test case, bitrate variation SHOULD be...' RFC 2119 keyword, line 436: '...didate proposals MAY show the effectiv...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 14, 2014) is 3541 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: '20ms' is mentioned on line 973, but not defined == Missing Reference: '300ms' is mentioned on line 973, but not defined == Missing Reference: '1000ms' is mentioned on line 973, but not defined == Unused Reference: 'I-D.ietf-rtcweb-use-cases-and-requirements' is defined on line 1379, but no explicit reference was found in the text == Unused Reference: 'RFC5033' is defined on line 1385, but no explicit reference was found in the text == Unused Reference: 'RFC5166' is defined on line 1388, but no explicit reference was found in the text == Unused Reference: 'RFC5681' is defined on line 1391, but no explicit reference was found in the text == Unused Reference: 'SA4-EVAL' is defined on line 1394, but no explicit reference was found in the text == Unused Reference: 'SA4-LR' is defined on line 1398, but no explicit reference was found in the text == Unused Reference: 'TCP-eval-suite' is defined on line 1409, but no explicit reference was found in the text == Outdated reference: A later version (-18) exists of draft-ietf-avtcore-rtp-circuit-breakers-05 == Outdated reference: A later version (-14) exists of draft-ietf-rmcat-eval-criteria-00 == Outdated reference: A later version (-09) exists of draft-ietf-rmcat-cc-requirements-02 == Outdated reference: A later version (-02) exists of draft-sarker-rmcat-cellular-eval-test-cases-01 == Outdated reference: A later version (-16) exists of draft-ietf-rtcweb-use-cases-and-requirements-14 Summary: 1 error (**), 0 flaws (~~), 16 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group Z. Sarker 3 Internet-Draft Ericsson AB 4 Intended status: Informational V. Singh 5 Expires: February 15, 2015 Aalto University 6 X. Zhu 7 M. Ramalho 8 Cisco Systems 9 August 14, 2014 11 Test Cases for Evaluating RMCAT Proposals 12 draft-ietf-rmcat-eval-test-00 14 Abstract 16 The Real-time Transport Protocol (RTP) is used to transmit media in 17 multimedia telephony applications, these applications are typically 18 required to implement congestion control. The RMCAT working group is 19 currently working on candidate algorithms for such interactive real- 20 time multimedia applications. This document describes the test cases 21 to be used in the performance evaluation of those candidate 22 algorithms. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on February 15, 2015. 41 Copyright Notice 43 Copyright (c) 2014 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 61 4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 7 62 4.1. Evaluation metircs . . . . . . . . . . . . . . . . . . . 8 63 4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 64 4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 65 5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 66 5.1. Variable Available Capacity with Single RMCAT flow . . . 10 67 5.2. Variable Available Capacity with Multiple RMCAT flows . . 12 68 5.3. Congested Feedback Link with Bi-directional RMCAT flows . 14 69 5.4. Competing Flows with Same RMCAT Algorithm . . . . . . . . 16 70 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 18 71 5.6. RMCAT Flow competing with a long TCP Flow . . . . . . . . 20 72 5.7. RMCAT Flow competing with short TCP Flows . . . . . . . . 22 73 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 24 74 6. Other potential test cases . . . . . . . . . . . . . . . . . 26 75 6.1. Explicit Congestion Notification Usage . . . . . . . . . 26 76 6.2. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 26 77 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 28 78 7.1. Cellular Network Specific Test Cases . . . . . . . . . . 28 79 7.2. Wi-Fi Network Specific Test Cases . . . . . . . . . . . . 28 80 8. Security Considerations . . . . . . . . . . . . . . . . . . . 29 81 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 82 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 29 83 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 29 84 11.1. Normative References . . . . . . . . . . . . . . . . . . 29 85 11.2. Informative References . . . . . . . . . . . . . . . . . 30 86 Appendix A. List of Network Parameters . . . . . . . . . . . . . 31 87 A.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 31 88 A.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 31 89 A.3. DropTail Router Queue Length . . . . . . . . . . . . . . 32 90 Appendix B. Models . . . . . . . . . . . . . . . . . . . . . . . 32 91 B.1. Jitter models . . . . . . . . . . . . . . . . . . . . . . 32 92 B.2. Loss generation model . . . . . . . . . . . . . . . . . . 35 93 B.3. TCP taffic model . . . . . . . . . . . . . . . . . . . . 35 94 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35 96 1. Introduction 98 This memo describes a set of test cases for evaluating candidate 99 RMCAT congestion control algorithm proposals, it is based on the 100 guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] and the 101 requirements discussed in [I-D.ietf-rmcat-cc-requirements]. The test 102 cases cover basic usage scenarios and are described using a common 103 structure, which allows for additional test cases to be added to 104 those described herein to accommodate other topologies and/or the 105 modeling of different path characteristics. It is the intention of 106 this work to capture the consensus of the RMCAT working group 107 participants regarding the test cases upon which the performance of 108 the candidate RMCAT proposals should be evaluated. 110 2. Terminology 112 The terminology defined in RTP [RFC3550], RTP Profile for Audio and 113 Video Conferences with Minimal Control [RFC3551], RTCP Extended 114 Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback 115 (RTP/AVPF) [RFC4585], Support for Reduced-Size RTCP [RFC5506], and 116 RTP Circuit Breaker algorithm [I-D.ietf-avtcore-rtp-circuit-breakers] 117 apply. 119 3. Structure of Test cases 121 All test cases in this document follow a basic structure allowing 122 implementers to describe a new test scenario without repeatedly 123 explaining common attributes. The structure includes a general 124 description section that describes the test case and its motivation. 125 Additionally the test case defines a set of attributes that 126 characterize the testbed, i.e., the network path between 127 communicating peers and the diverse traffic sources. 129 o Define the test case: 131 * General description: describes the motivation and the goals of 132 the test case. 134 * Expected behavior: describe the desired rate adaptation 135 behaviour. 137 * Define a check-list to evaluate the desired behaviour: this 138 indicates the minimum set of metrics (e.g., link utilization, 139 media sending rate) that a proposed algorithm needs to measure 140 to validate the expected rate adaptation behaviour. It should 141 also indicate the time granularity (e.g., averaged over 10ms, 142 100ms, or 1s) for measuring certain metrics. Typical 143 measurement interval is 200ms. 145 o Define testbed topology: every test case needs to define an 146 evaluation testbed topology. Figure 1 shows such an evaluation 147 topology. In this evaluation topology, S1..Sn are traffic 148 sources. These sources generate media traffic and use either an 149 RMCAT candidate congestion control algorithm or other congestion 150 control algorithm designed for media, such as TFRC. R1..Rn are 151 the corresponding receivers. A test case can have one or more 152 such traffic sources (S) and corresponding receivers (R). The 153 path from the source to destination is denoted as forward and the 154 path from a destination to a source is denoted as backward. The 155 following basic structure of test case has been described from the 156 perspective of media generating endpoints attached on the left- 157 hand side of Figure 1. In this setup, media flows in forward 158 direction and corresponding feedback/control messages flow in the 159 backward direction. However, it is also possible to set up the 160 test with media flowing in both forward and backward directions. 161 In that case, unless otherwise specified by the test case, it is 162 expected that the backward path does not introduce any congestion 163 related impairments and has enough capacity to accommodate both 164 media and feedback/control messages. It should be noted that 165 depending on the test cases it is possible to have different path 166 characteristics in of the either directions. 168 o 170 +---+ +---+ 171 |S1 |====== \ Forward --> / =======|R1 | 172 +---+ \\ // +---+ 173 \\ // 174 +---+ +-----+ +-----+ +---+ 175 |S2 |=======| A |------------------------------>| B |=======|R2 | 176 +---+ | |<------------------------------| | +---+ 177 +-----+ +-----+ 178 (...) // \\ (...) 179 // <-- Backward \\ 180 +---+ // \\ +---+ 181 |Sn |====== / \ ======|Rn | 182 +---+ +---+ 184 Figure 1: Example of A Testbed Topology 186 In a laboratory testbed environment there may exist a significant 187 amount of traffic on portions of the network path between the 188 endpoints that is not desired for the purposes of these RMCAT 189 tests. Some of this traffic may be generated by other processes 190 on the endpoints themselves (e.g., discovery protocols) or by 191 other endpoints not presently under test. It is recommended not 192 to route traffic generated by endpoints that are not under test 193 through the test bed. Additionally, it is recommended to route 194 non-RMCAT traffic generated by the endpoints under test around the 195 bottleneck links specified herein. 197 o Define testbed attributes: 199 * Duration: defines the duration of the test. 201 * Path characteristics: defines the end-to-end transport level 202 path characteristics of the testbed in a particular test case. 203 Two sets of attributes describe the path characteristics, one 204 for the forward path and the other for the backward path. The 205 path characteristics for a particular path direction is 206 applicable to all the Sources "S" sending traffic on that path. 207 If only one attribute is specified, it is used for both path 208 directions, however, unless specified the reverse path has no 209 capacity restrictions and no path loss. 211 + Path direction: forward or backward. 213 + Bottleneck-link capacity: defines minimum capacity of the 214 end-to-end path 216 + One-way propagation delay: describes the end-to-end latency 217 along the path when network queues are empty, i.e., the time 218 it takes for a packet to go from the sender to the receiver 219 without encountering any queuing delay. 221 + Maximum end-to-end jitter: defines the maximum jitter that 222 can be observed along the path. 224 + Bottleneck queue type: for example, Droptail, FQ-CoDel, or 225 PIE. 227 + Bottleneck queue size: defines size of queue in terms of 228 queuing time when the queue is full (in milliseconds). 230 + Path loss ratio: characterizes the non-congested, additive, 231 losses to be generated on the end-to-end path. MUST 232 describe the loss pattern or loss model used to generate the 233 losses. 235 * Application-related: defines the traffic source behaviour for 236 implementing the test case 238 + Media traffic Source: defines the characteristics of the 239 media sources. When using more than one media source, the 240 different attributes are enumerated separately for each 241 different media source. 243 - Media type: Video/Voice/Application/Text 245 - Media flow direction: forward, backward or both. 247 - Number of media sources: defines the total number of 248 media sources 250 - Media codec: Constant Bit Rate (CBR) or Variable Bit Rate 251 (VBR) 253 - Media source behaviour: describes the media encoder 254 behavior. It defines the main parameters that affect the 255 adaptation behaviour. This may include but not limited 256 to: 258 o Adaptability: describes the adaptation options. For 259 example, in the case of video it defines the following 260 ranges of adaptation: bit rate, frame rate, video 261 resolution. Similarly, in the case of voice, it 262 defines the range of bit rate adaptation, the sampling 263 rate variation, and the variation in packetization 264 interval. 266 o Output variation : for a VBR encoder it defines the 267 encoder output variation from the average target rate 268 over a particular measurement interval. For example, 269 on average the encoder output may vary between 5% to 270 15% above or below the average target bit rate when 271 measured over a 100 ms time window. The time interval 272 over which the variation is specified must be 273 provided. 275 o Responsiveness to a new bit rate request: the lag in 276 time between a new bit rate request and actual rate 277 changes in encoder output. Depending on the encoder, 278 this value may be specified in absolute time (e.g. 279 10ms to 1000ms) or other appropriate metric (next 280 frame interval time). 282 - Media content: describes the chosen media sequences; For 283 example, test sequences are available at: [xiph-seq] and 284 [HEVC-seq]. 286 - Media timeline: describes the point when the media source 287 is introduced and removed from the testbed. For example, 288 the media source may start transmitting immediately when 289 the test case begins, or after a few seconds. 291 - Startup behaviour: the media starts at a defined bit 292 rate, which may be the minimum, maximum bit rate, or a 293 value in between (in Kbps). 295 + Competing traffic source: describes the characteristics of 296 the competing traffic source, the different types of 297 competing flows are enumerated in 298 [I-D.ietf-rmcat-eval-criteria]. 300 - Traffic direction: forward, backward or both. 302 - Type of sources: defines the types of competing traffic 303 sources. Types of competing traffic flows are listed in 304 [I-D.ietf-rmcat-eval-criteria]. For example, the number 305 of TCP flows connected to a web browser, the mean size 306 and distribution of the content downloaded. 308 - Number of sources: defines the total number of competing 309 sources of each media type. 311 - Congestion control: enumerates the congestion control 312 used by each type of competing traffic. 314 - Traffic timeline: describes when the competing traffic 315 starts and ends in the test case. 317 * Additional attributes: describes attributes essential for 318 implementing a test case which are not included in the above 319 structure. These attributes MUST be well defined, so that 320 other implementers are able to implement it. 322 Any attribute can have a set of values (enclosed within "[]"). Each 323 member value of such a set MUST be treated as different value for the 324 same attribute. It is desired to run separate tests for each such 325 attribute value. 327 The test cases described in this document follow the above structure. 329 4. Recommended Evaluation Settings 331 This section describes recommended test case settings and could be 332 overwritten by the respective test cases. 334 4.1. Evaluation metircs 336 To evaluate the performance of the candidate algorithms it is 337 expected to log enough information to visualize the following metrics 338 at a fine enough time granularity: 340 1. Flow level: 342 A. End-to-end delay for the RMCAT flow. 344 B. Variation in sending bit rate and goodput. Mainly observing 345 the frequency and magnitude of oscillations. 347 C. Packet losses observed at the receiving endpoint 349 D. Feedback message overhead 351 E. Convergence time. 353 2. Transport level: 355 A. Bandwidth utilization 357 B. Queue length (milliseconds at specified path capacity): 359 + average over the length of the session 361 + 5 and 95 percentile 363 + median, maximum, minimum 365 4.2. Path characteristics 367 Each path between a sender and receiver as described in Figure 1 have 368 the following characteristics unless otherwise specified in the test 369 case. 371 o Path direction: forward and backward. 373 o Bottleneck-link capacity: 4Mbps. 375 o One-Way propagation delay: 50ms. It is encouraged to test with 376 additional propagation delays mentioned in Appendix A.1 378 o Maximum end-to-end jitter: 30ms. Jitter models are described in 379 Appendix B.1 381 o Bottleneck queue type: Drop tail. It is encouraged to test with 382 other AQM schemes, such as FQ-CoDel and PIE. 384 o Bottleneck queue size: 300ms. 386 o Path loss ratio: 0%. 388 Examples of additional network parameters are discussed in 389 Appendix A. 391 4.3. Media source 393 Unless otherwise specified, each test case will include one or more 394 media sources as described below. 396 o Media type: Video 398 * Media codec: VBR 400 * Media source behaviour: 402 + Adaptability: 404 - Bit rate range: 150 Kbps - 1.5 Mbps. In real-life 405 applications the bitrate range can vary a lot depending 406 on the provided service, for example, the maximum bitrate 407 can be up to 4Mbps. However, for running tests to 408 evaluate the congestion control algorithms it is more 409 important to have a look at how they are reacting to 410 certain amount of bandwidth change. Also it is possible 411 that the media traffic generator used in a particular 412 simulator or testbed if not capable of generating higher 413 bitrate. Hence we have selected a suitable bitrate range 414 typical of consumer-grade video conferencing applications 415 in designing the test case. If a different bitrate range 416 is used in the test cases, the end-to-end path capacity 417 values will also need to be scaled accordingly. 419 - Frame resolution: 144p - 720p (or 1080p) 421 - Frame rate: 10fps - 30fps 423 + Variation from target bitrate: +/-5%. Unless otherwise 424 specified in the test case, bitrate variation SHOULD be 425 calculated over one (1) second period of time. 427 + Responsiveness to new bit rate request: 100ms 429 * Media content: The media content should represent a typical 430 video conversational scenario with head and shoulder movement. 431 We recommend to use Foreman video sequence. 433 * Media startup behaviour: 150Kbps. It should be noted that 434 applications can use smart ways to select an optimal startup 435 bitrate values for a certain network condition. In such cases 436 the candidate proposals MAY show the effectiveness of such 437 smart approach as an additional information for the evaluation 438 process. 440 o Media type: Audio 442 * Media codec: CBR 444 * Media bitrate: 20Kbps 446 5. Basic Test Cases 448 5.1. Variable Available Capacity with Single RMCAT flow 450 In this test case the bottleneck-link capacity between the two 451 endpoints varies over time. This test is designed to measure the 452 responsiveness of the candidate algorithm. This test tries to 453 address the requirements in [I-D.ietf-rmcat-cc-requirements], which 454 requires the algorithm to adapt the flow(s) and provide lower end-to- 455 end latency when there exists: 457 o an intermediate bottleneck 459 o change in available capacity (e.g., due to interface change, 460 routing change). 462 o maximum Media Bit Rate is Greater than Link Capacity. In this 463 case, the application will attempt to ramp up to its maximum bit 464 rate, since the link capacity is limited to a value lower, the 465 congestion control scheme is expected to stabilize the sending bit 466 rate close to the available bottleneck capacity. This situation 467 can occur when the endpoints are connected via thin long networks 468 even though the advertised capacity of the access network may be 469 higher. 471 It should be noted that the exact variation in available capacity due 472 to any of the above depends on the under-lying technologies. Hence, 473 we describe a set of known factors, which may be extended to devise a 474 more specific test case targeting certain behaviour in a certain 475 network environment. 477 Expected behavior: the candidate algorithm is expected to detect the 478 path capacity constraint, converges to bottleneck link's capacity and 479 adapt the flow to avoid unwanted oscillation when the sending bit 480 rate is approaching the bottleneck link's capacity. The oscillations 481 occur when the media flow(s) attempts to reach its maximum bit rate, 482 overshoots the usage of the available bottleneck capacity, to rectify 483 it reduces the bit rate and starts to ramp up again. 485 Testbed topology: One media source S1 is connected to corresponding 486 R1. The media traffic is transported over the forward path and 487 corresponding feedback/control traffic is transported over the 488 backward path. 490 Forward --> 491 +---+ +-----+ +-----+ +---+ 492 |S1 |=======| A |------------------------------>| B |=======|R1 | 493 +---+ | |<------------------------------| | +---+ 494 +-----+ +-----+ 495 <-- Backward 497 Figure 2: Testbed Topology for Limited Link Capacity 499 To evaluate the performance of the candidate algorithms it is 500 expected to log enough information to visualize the metrics described 501 in Section 4.1 at a fine enough time granularity. 503 Testbed attributes: 505 o Test duration: 100s 507 o Path characteristics: as described in Section 4.2 509 o Application-related: 511 * Media Traffic: 513 + Media type: Video 515 - Media direction: forward. 517 - Number of media sources: One (1) 519 - Media timeline: 521 o Start time: 0s. 523 o End time: 99s. 525 + Media type: Audio 527 - Media direction: forward. 529 - Number of media sources: One (1) 531 - Media timeline: 533 o Start time: 0s. 535 o End time: 99s. 537 * Competing traffic: 539 + Number of sources : Zero (0) 541 o Test Specific Information: 543 * This test uses the following one way propagation delays of 50 544 ms and 100 ms. 546 * This test uses bottleneck path capacity variation as listed in 547 Table 1 549 +---------------------+----------------+------------+---------------+ 550 | Variation pattern | Path direction | Start time | Path capacity | 551 | index | | | | 552 +---------------------+----------------+------------+---------------+ 553 | One | Forward | 0s | 1Mbps | 554 | Two | Forward | 40s | 2.5Mbps | 555 | Three | Forward | 60s | 600Kbps | 556 | Four | Forward | 80s | 1Mbps | 557 +---------------------+----------------+------------+---------------+ 559 Table 1: Path capacity variation pattern for forward direction 561 5.2. Variable Available Capacity with Multiple RMCAT flows 563 This test case is similar to Section 5.1. However in addition this 564 test will also consider persistent network load due to competing 565 traffic. 567 Expected behavior: the candidate algorithms is expected to detect the 568 variation in available capacity and adapt the media stream(s) 569 accordingly. The flows stabilize around their maximum bitrate as the 570 as the maximum link capacity is large enough to accommodate the 571 flows. When the available capacity drops, the flow(s) adapts by 572 decreasing its sending bit rate, and when congestion disappears, the 573 flow(s) are again expected to ramp up. 575 To evaluate the performance of the candidate algorithms it is 576 expected to log enough information to visualize the metrics described 577 in Section 4.1 at a fine enough time granularity: 579 Testbed Topology: Two (2) media sources S1 and S2 are connected to 580 their corresponding destinations R1 and R2. The media traffic is 581 transported over the forward path and corresponding feedback/control 582 traffic is transported over the backward path. 584 +---+ +---+ 585 |S1 |===== \ / =======|R1 | 586 +---+ \\ Forward --> // +---+ 587 \\ // 588 +-----+ +-----+ 589 | A |------------------------------>| B | 590 | |<------------------------------| | 591 +-----+ +-----+ 592 // \\ 593 // <-- Backward \\ 594 +---+ // \\ +---+ 595 |S2 |====== / \ ======|R2 | 596 +---+ +---+ 598 Figure 3: Testbed Topology for Variable Available Capacity 600 Testbed attributes: 602 Testbed attributes are similar as described in Section 5.1 except the 603 test specific capacity variation setup. 605 Test Specific Information: This test uses path capacity variation as 606 listed in Table 2 with a corresponding end time of 125 seconds. 608 +---------------------+----------------+------------+---------------+ 609 | Variation pattern | Path direction | Start time | Path capacity | 610 | index | | | | 611 +---------------------+----------------+------------+---------------+ 612 | One | Forward | 0s | 4Mbps | 613 | Two | Forward | 25s | 2Mbps | 614 | Three | Forward | 50s | 3.5Mbps | 615 | Four | Forward | 75s | 1Mbps | 616 | Five | Forward | 100s | 2Mbps | 617 +---------------------+----------------+------------+---------------+ 619 Table 2: Path capacity variation pattern for forward direction 621 5.3. Congested Feedback Link with Bi-directional RMCAT flows 623 RMCAT WG has been chartered to define algorithms for RTP hence it is 624 assumed that RTCP, RTP header extension or such would be used by the 625 congestion control algorithm in the backchannel. Due to asymmetric 626 nature of the link between communicating peers it is possible for a 627 participating peer to not receive such feedback information due to an 628 impaired or congested backchannel (even when the forward channel 629 might not be impaired). This test case is designed to observe the 630 candidate congestion control behaviour in such an event. 632 It is expected that the candidate algorithms is able to cope with the 633 lack of feedback information and adapt to minimize the performance 634 degradation of media flows in the forward channel. 636 It should be noted that for this test case: logs are compared with 637 the reference case, i.e, when the backward channel has no impairments 639 To evaluate the performance of the candidate algorithms it is 640 expected to log enough information to visualize the metrics described 641 in Section 4.1 at a fine-grained time intervals: 643 Testbed topology: One (1) media source S1 is connected to 644 corresponding R1, but both endpoints are additionally receiving and 645 sending data, respectively. The media traffic (S1->R1) is 646 transported over the forward path and corresponding feedback/control 647 traffic is transported over the backward path. Likewise media 648 traffic (S2->R2) is transported over the backward path and 649 corresponding feedback/control traffic is transported over the 650 forward path. 652 +---+ +---+ 653 |S1 |===== \ Forward --> / =======|R1 | 654 +---+ \\ // +---+ 655 \\ // 656 +-----+ +-----+ 657 | A |------------------------------>| B | 658 | |<------------------------------| | 659 +-----+ +-----+ 660 // \\ 661 // <-- Backward \\ 662 +---+ // \\ +---+ 663 |R2 |===== / \ ======|S2 | 664 +---+ +---+ 666 Figure 4: Testbed Topology for Congested Feedback Link 668 Testbed attributes: 670 o Test duration: 100s 672 o Path characteristics: 674 * Bottleneck-link capacity: 2Mbps. 676 o Application-related: 678 * Media Source: 680 + Media type: Video 682 - Media direction: forward and backward 684 - Number of media sources: Two (2) 686 - Media timeline: 688 o Start time: 0s. 690 o End time: 99s. 692 + Media type: Audio 694 - Media direction: forward and backward 696 - Number of media sources: Two (2) 698 - Media timeline: 700 o Start time: 0s. 702 o End time: 99s. 704 * Competing traffic: 706 + Number of sources : Zero (0) 708 o Test Specific Information: This test uses path capacity variations 709 to create congested feedback link. Table 3 lists the variation 710 patterns applied to the forward path and Table 4 lists the 711 variation patterns applied to the backward path. 713 +---------------------+----------------+------------+---------------+ 714 | Variation pattern | Path direction | Start time | Path capacity | 715 | index | | | | 716 +---------------------+----------------+------------+---------------+ 717 | One | Forward | 0s | 2Mbps | 718 | Two | Forward | 20s | 1Mbps | 719 | Three | Forward | 40s | 500Kbps | 720 | Four | Forward | 60s | 2Mbps | 721 +---------------------+----------------+------------+---------------+ 723 Table 3: Path capacity variation pattern for forward direction 725 +---------------------+----------------+------------+---------------+ 726 | Variation pattern | Path direction | Start time | Path Capacity | 727 | index | | | | 728 +---------------------+----------------+------------+---------------+ 729 | One | Backward | 0s | 2Mbps | 730 | Two | Backward | 35s | 800Kbps | 731 | Three | Backward | 70s | 2Mbps | 732 +---------------------+----------------+------------+---------------+ 734 Table 4: Path capacity variation pattern for backward direction 736 5.4. Competing Flows with Same RMCAT Algorithm 738 In this test case, more than one RMCAT media flow shares the 739 bottleneck link and each of them uses the same congestion control 740 algorithm. This is a typical scenario where a real-time interactive 741 application sends more than one media flows to the same destination 742 and these flows are multiplexed over the same port. In such a 743 scenario it is likely that the flows will be routed via the same path 744 and need to share the available bandwidth amongst themselves. For 745 the sake of simplicity it is assumed that there are no other non- 746 RMCAT competing traffic sources in the bottleneck link and that there 747 is sufficient capacity to accommodate all the flows individually. 748 While this appears to be a variant of the test case defined in 749 Section 5.2, it focuses on the capacity sharing aspect of the 750 candidate algorithm. The previous test case, on the other hand, 751 measures adaptability, stability, and responsiveness of the candidate 752 algorithm. 754 Expected behavior: It is expected that the competing flows will 755 converge to an optimum bit rate to accommodate all the flows with 756 minimum possible latency and loss. Specifically, the test introduces 757 three media flows at different time instances, when the second flow 758 appears there should still be room to accommodate another flow on the 759 bottleneck link. Lastly, when the third flow appears the bottleneck 760 link should be saturated. 762 To evaluate the performance of the candidate algorithms it is 763 expected to log enough information to visualize the metrics described 764 in Section 4.1 at a fine enough time granularity: 766 Testbed topology: Three media sources S1, S2, S3 are connected to 767 respective R1, R2, R3. The media traffic is transported over the 768 forward path and corresponding feedback/control traffic is 769 transported over the backward path. 771 +---+ +---+ 772 |S1 |===== \ Forward --> / =======|R1 | 773 +---+ \\ // +---+ 774 \\ // 775 +---+ +-----+ +-----+ +---+ 776 |S2 |=======| A |------------------------------>| B |=======|R2 | 777 +---+ | |<------------------------------| | +---+ 778 +-----+ +-----+ 779 // \\ 780 // <-- Backward \\ 781 +---+ // \\ +---+ 782 |S3 |====== / \ ======|R3 | 783 +---+ +---+ 785 Figure 5: Testbed Topology for Multiple RMCAT Flows 787 Testbed attributes: 789 o Test duration: 120s 791 o Path characteristics: 793 * Bottleneck-link capacity: 3.5Mbps 795 o Application-related: 797 * Media Source: 799 + Media type: Video 801 - Media direction: forward. 803 - Number of media sources: Three (3) 805 - Media timeline: New media flows are added sequentially, 806 at short time intervals. See test specific setup below. 808 + Media type: Audio 809 - Media direction: forward. 811 - Number of media sources: Three (3) 813 - Media timeline: New media flows are added sequentially, 814 at short time intervals. See test specific setup below. 816 * Competing traffic: 818 + Number of sources : Zero (0) 820 o Test Specific Information: 822 * Media flow timeline: 824 + Flow ID: One (1) 826 + Start time: 0s 828 + End time: 119s 830 * Media flow timeline: 832 + Flow ID: Two (2) 834 + Start time: 20s 836 + End time: 119s 838 * Media flow timeline: 840 + Flow ID: Three (3) 842 + Start time: 40s 844 + End time: 119s 846 5.5. Round Trip Time Fairness 848 In this test case, multiple RMCAT media flows share the bottleneck 849 link, but the end-to-end path latency for each RMCAT flow is 850 different. For the sake of simplicity it is assumed that there are 851 no other non-RMCAT competing traffic sources in the bottleneck link 852 and that there is sufficient capacity to accommodate all the flows. 853 While this appears to be a variant of test case 5.2, it focuses on 854 the capacity sharing aspect of the candidate algorithm under 855 different RTTs. 857 It is expected that the competing flows will converge to bit rates to 858 accommodate all the flows with minimum possible latency and loss. 859 Specifically, the test introduces five media flows at the same time 860 instance. 862 To evaluate the performance of the candidate algorithms it is 863 expected to log enough information to visualize the metrics described 864 in Section 4.1 at a fine enough time granularity: 866 Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to 867 their corresponding media sinks R1,R2,..,R5. The media traffic is 868 transported over the forward path and corresponding feedback/control 869 traffic is transported over the backward path. The topology is the 870 same as in Section 5.4. The end-to-end path delays are: 10ms for 871 S1-R1, 25ms for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms 872 S5-R5, respectively. 874 Testbed attributes: 876 o Test duration: 120s 878 o Path characteristics: 880 * One-Way propagation delay for each flow: 10ms, 25ms, 50ms, 881 100ms, 150ms. 883 o Application-related: 885 * Media Source: 887 + Media type: Video 889 - Media direction: forward 891 - Number of media sources: Five (5) 893 - Media timeline: 895 o Start time: 0s. 897 o End time: 119s. 899 + Media type: Audio 901 - Media direction: forward. 903 - Number of media sources: Five (5) 904 - Media timeline: 906 o Start time: 0s. 908 o End time: 119s. 910 * Competing traffic: 912 + Number of sources : Zero (0) 914 o Test Specific Information: None 916 5.6. RMCAT Flow competing with a long TCP Flow 918 In this test case, one or more RMCAT media flows share the bottleneck 919 link with at least one long lived TCP flows. Long lived TCP flows 920 download data throughout the session and are expected to have 921 infinite amount of data to send and receive. This is a scenario 922 where a multimedia application co-exists with a large file download. 923 The test case measures the adaptivity of the candidate algorithm to 924 competing traffic. It addresses the requirement 3 in 925 [I-D.ietf-rmcat-cc-requirements]. 927 Expected behavior: depending on the convergence observed in test case 928 5.1 and 5.2, the candidate algorithm may be able to avoid congestion 929 collapse. In the worst case, the media stream will fall to the 930 minimum media bit rate. 932 To evaluate the performance of the candidate algorithms it is 933 expected to log enough information to visualize the following metrics 934 in addition to the metrics described in Section 4.1 at a fine enough 935 time granularity: 937 1. Flow level: 939 A. TCP throughput. 941 Testbed topology: One (1) media source S1 is connected to 942 corresponding media sink, R1. In addition, there is a long-live TCP 943 flow sharing the same bottleneck link. The media traffic is 944 transported over the forward path and corresponding feedback/control 945 traffic is transported over the backward path. The TCP traffic goes 946 over the forward path from, S_tcp with acknowledgement packets 947 flowing along the backward path from, R_tcp. 949 +--+ +--+ 950 |S1|===== \ Forward --> / =======|R1| 951 +--+ \\ // +--+ 952 \\ // 953 +-----+ +-----+ 954 | A |---------------------------->| B | 955 | |<----------------------------| | 956 +-----+ +-----+ 957 // \\ 958 // <-- Backward \\ 959 +-----+ // \\ +-----+ 960 |S_tcp|=== / \ ===|R_tcp| 961 +-----+ +-----+ 963 Figure 6: Testbed Topology for TCP vs RMCAT Flows 965 Testbed attributes: 967 o Test duration: 120s 969 o Path characteristics: 971 * Bottleneck-link capacity: 2Mbps 973 * Bottleneck queue size: [20ms, 300ms, 1000ms] 975 o Application-related: 977 * Media Source: 979 + Media type: Video 981 - Media direction: forward 983 - Number of media sources: One (1) 985 - Media timeline: 987 o Start time: 5s. 989 o End time: 119s. 991 + Media type: Audio 993 - Media direction: forward 995 - Number of media sources: One (1) 996 - Media timeline: 998 o Start time: 5s. 1000 o End time: 119s. 1002 * Additionally, implementers are encouraged to run the experiment 1003 with multiple media sources. 1005 * Competing traffic: 1007 + Number and Types of sources : one (1), long-lived TCP 1009 + Traffic direction : forward 1011 + Congestion control: Default TCP congestion control. 1013 + Traffic timeline: 1015 - Start time: 0s. 1017 - End time: 119s. 1019 o Test Specific Information: None 1021 5.7. RMCAT Flow competing with short TCP Flows 1023 In this test case, one or more RMCAT media flow shares the bottleneck 1024 link with multiple short-lived TCP flows. Short-lived TCP flows 1025 resemble the on/off pattern observed in the web traffic, wherein 1026 clients (browsers) connect to a server and download a resource 1027 (typically a web page, few images, text files, etc.) using several 1028 TCP connections (up to 4). This scenario shows the performance of 1029 the multimedia application when several browser windows are active. 1030 The test case measures the adaptivity of the candidate algorithm to 1031 competing web traffic, it addresses the requirements 1.E in 1032 [I-D.ietf-rmcat-cc-requirements]. 1034 Depending on the number of short TCP flows, the cross-traffic either 1035 appears as a short burst flow or resembles a long TCP flow. The 1036 intention of this test is to observe the impact of short-term burst 1037 on the behaviour of the candidate algorithm. 1039 To evaluate the performance of the candidate algorithms it is 1040 expected to log enough information to visualize the following metrics 1041 in addition to the metrics described in Section 4.1 at a fine enough 1042 time granularity: 1044 1. Flow level: 1046 A. Variation in the sending rate of the TCP flow. 1048 B. TCP throughput. 1050 Testbed topology: The topology described here is same as the one 1051 described in Figure 6. 1053 Testbed attributes: 1055 o Test duration: 300s 1057 o Path characteristics: 1059 * Bottleneck-link capacity: 2.0Mbps 1061 o Application-related: 1063 * Media Source: 1065 + Media type: Video 1067 - Media direction: forward 1069 - Number of media sources: two (2) 1071 - Media timeline: 1073 o Start time: 5s. 1075 o End time: 299s. 1077 + Media type: Audio 1079 - Media direction: forward 1081 - Number of media sources: two (2) 1083 - Media timeline: 1085 o Start time: 5s. 1087 o End time: 299s. 1089 * Competing traffic: 1091 + Number and Types of sources : Ten (10), short-lived TCP 1092 flows. 1094 + Traffic direction : forward 1096 + Congestion algorithm: Default TCP Congestion control. 1098 + Traffic timeline: Each short TCP flow is modeled as a 1099 sequence of file downloads interleaved with idle periods. 1100 See test specific setup. Not all short TCPs start at the 1101 same time, 2 start in the ON state while 8 start in an OFF 1102 stats. The model for the idle times for the OFF state is 1103 discussed in the Short-TCP model. 1105 o Test Specific Information: 1107 * Short-TCP traffic model: 1109 + File sizes: uniform distribution between 100KB to 1MB 1111 + Idle period: the duration of the OFF state is derived from 1112 an exponential distribution with the mean value of 10 1113 seconds. 1115 5.8. Media Pause and Resume 1117 In this test case, more than one real-time interactive media flows 1118 share the link bandwidth and all flows reach to a steady state by 1119 utilizing the link capacity in an optimum way. At these stage one of 1120 the media flow is paused for a moment. This event will result in 1121 more available bandwidth for the rest of the flows and as they are on 1122 a shared link. When the paused media flow will resume it would no 1123 longer have the same bandwidth share on the link. It has to make 1124 it's way through the other existing flows in the link to achieve a 1125 fair share of the link capacity. This test case is important 1126 specially for real-time interactive media which consists of more than 1127 one media flows and can pause/resume media flow at any point of time 1128 during the session. This test case directly addresses the 1129 requirement number 5 in [I-D.ietf-rmcat-cc-requirements]. One can 1130 think it as a variation of test case defined in Section 5.4. 1131 However, it is different as the candidate algorithms can use 1132 different strategies to increase its efficiency, for example the 1133 fairness, convergence time, reduce oscillation etc, by capitalizing 1134 the fact that they have previous information of the link. 1136 To evaluate the performance of the candidate algorithms it is 1137 expected to log enough information to visualize the following metrics 1138 in addition to the metrics described in Section 4.1 at a fine enough 1139 time granularity: 1141 1. Flow level: 1143 A. Variation in sending bit rate and goodput. Mainly observing 1144 the frequency and magnitude of oscillations. 1146 Testbed Topology: Same as test case defined in Section 5.4 1148 Testbed attributes: The general description of the testbed parameters 1149 are same as Section 5.4 with changes in the test specific setup as 1150 below- 1152 o Other test specific setup: 1154 * Media flow timeline: 1156 + Flow ID: One (1) 1158 + Start time: 0s 1160 + Flow duration: 119s 1162 + Pause time: not required 1164 + Resume time: not required 1166 * Media flow timeline: 1168 + Flow ID: Two (2) 1170 + Start time: 0s 1172 + Flow duration: 119s 1174 + Pause time: at 40s 1176 + Resume time: at 60s 1178 * Media flow timeline: 1180 + Flow ID: Three (3) 1182 + Start time: 0s 1184 + Flow duration:119s 1185 + Pause time: not required 1187 + Resume time: not required 1189 6. Other potential test cases 1191 It has been noticed that there are other interesting test cases 1192 besides the basis test cases listed above. In many aspects, these 1193 additional test cases can help to further evaluate the candidate 1194 algorithm. They are listed as below. 1196 6.1. Explicit Congestion Notification Usage 1198 This test case requires to run all the basic test cases with the 1199 availability of Explicit Congestion Notification (ECN) [RFC6679] 1200 feature enabled. The goal of this test is to exhibit that the 1201 candidate algorithms does not fail when ECN signals are available. 1202 With ECN signals enabled the algorithms are expected to perform 1203 better than their delay based variants. 1205 6.2. Multiple Bottlenecks 1207 In this test case one RMCAT flow, S1->R2 traverse a path with 1208 multiple bottlenecks. As illustrated in Figure 7, the first flow 1209 (S1->R1) competes with the second RMCAT flow (S2->R2) over the link 1210 between A and B which is close to the sender side; again, that flow 1211 (S1->R1) competes with the third RMCAT flow (S3->R3) over the link 1212 between C and D which is close to the receiver side. The goal of 1213 this test is to ensure that the candidate algorithms work properly in 1214 the presence of multiple bottleneck links on the end to end path. 1216 Expected behavior: the candidate algorithm is expected to achieve 1217 full utilization at both bottleneck links without starving any of the 1218 three RMCAT flows. 1220 Forward ----> 1222 +---+ +---+ +---+ +---+ 1223 |S2 | |R2 | |S3 | |R3 | 1224 +---+ +---+ +---+ +---+ 1225 | | | | 1226 | | | | 1227 +---+ +-----+ +-----+ +-----+ +-----+ +---+ 1228 |S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 | 1229 +---+ | |<------| |<-----| |<----| | +---+ 1230 +-----+ +-----+ +-----+ +-----+ 1232 1st 2nd 1233 Bottleneck (A->B) Bottleneck (C->D) 1235 <------ Backward 1237 Figure 7: Testbed Topology for Multiple Bottlenecks 1239 Testbed topology: Three media sources S1, S2, and S3 are connected to 1240 respective destinations R1, R2, and R3. For all three flows the 1241 media traffic is transported over the forward path and corresponding 1242 feedback/control traffic is transported over the backward path. 1244 Testbed attributes: 1246 o Test duration: 120s 1248 o Path characteristics: 1250 * Path capacity between A and B = 2Mbps. 1252 * Path capacity between B and C = 8Mbps. 1254 * Path capacity between C and D = 1.5Mbps. 1256 * One-Way propagation delay: 1258 1. Between S1 and R1 : 100ms 1260 2. Between S2 and R2: 40ms 1262 3. Between S3 and R3: 40ms 1264 o Application-related: 1266 * Media Source: 1268 + Media type: Video 1270 - Media direction: Forward 1272 - Number of media sources: Three (3) 1274 - Media timeline: 1276 o Start time: 0s. 1278 o End time: 119s. 1280 + Media type: Audio 1282 - Media direction: Forward 1284 - Number of media sources: Three (3) 1286 - Media timeline: 1288 o Start time: 0s. 1290 o End time: 119s. 1292 * Competing traffic: 1294 + Number of sources : Zero (0) 1296 7. Wireless Access Links 1298 7.1. Cellular Network Specific Test Cases 1300 Additional cellular network specific test cases are define in 1301 [I-D.draft-sarker-rmcat-cellular-eval-test-cases] 1303 7.2. Wi-Fi Network Specific Test Cases 1305 TBD 1307 [Editor's Note: We should encourage people to come up with possible 1308 WiFi Network specific test cases] 1310 8. Security Considerations 1312 Security issues have not been discussed in this memo. 1314 9. IANA Considerations 1316 There are no IANA impacts in this memo. 1318 10. Acknowledgements 1320 Much of this document is derived from previous work on congestion 1321 control at the IETF. 1323 The content and concepts within this document are a product of the 1324 discussion carried out in the Design Team. 1326 11. References 1328 11.1. Normative References 1330 [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., 1331 and K. Carlberg, "Explicit Congestion Notification (ECN) 1332 for RTP over UDP", RFC 6679, August 2012. 1334 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1335 Jacobson, "RTP: A Transport Protocol for Real-Time 1336 Applications", STD 64, RFC 3550, July 2003. 1338 [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and 1339 Video Conferences with Minimal Control", STD 65, RFC 3551, 1340 July 2003. 1342 [RFC3611] Friedman, T., Caceres, R., and A. Clark, "RTP Control 1343 Protocol Extended Reports (RTCP XR)", RFC 3611, November 1344 2003. 1346 [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, 1347 "Extended RTP Profile for Real-time Transport Control 1348 Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July 1349 2006. 1351 [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size 1352 Real-Time Transport Control Protocol (RTCP): Opportunities 1353 and Consequences", RFC 5506, April 2009. 1355 [I-D.ietf-avtcore-rtp-circuit-breakers] 1356 Perkins, C. and V. Singh, "Multimedia Congestion Control: 1357 Circuit Breakers for Unicast RTP Sessions", draft-ietf- 1358 avtcore-rtp-circuit-breakers-05 (work in progress), 1359 February 2014. 1361 [I-D.ietf-rmcat-eval-criteria] 1362 Singh, V. and J. Ott, "Evaluating Congestion Control for 1363 Interactive Real-time Media", draft-ietf-rmcat-eval- 1364 criteria-00 (work in progress), January 2014. 1366 [I-D.ietf-rmcat-cc-requirements] 1367 Jesup, R., "Congestion Control Requirements For RMCAT", 1368 draft-ietf-rmcat-cc-requirements-02 (work in progress), 1369 February 2014. 1371 [I-D.draft-sarker-rmcat-cellular-eval-test-cases] 1372 Sarker, Z. and I. Johansson, "Evaluation Test Cases for 1373 Interactive Real-Time Media over Cellular Networks", 6 1374 2014, . 1377 11.2. Informative References 1379 [I-D.ietf-rtcweb-use-cases-and-requirements] 1380 Holmberg, C., Hakansson, S., and G. Eriksson, "Web Real- 1381 Time Communication Use-cases and Requirements", draft- 1382 ietf-rtcweb-use-cases-and-requirements-14 (work in 1383 progress), February 2014. 1385 [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion 1386 Control Algorithms", BCP 133, RFC 5033, August 2007. 1388 [RFC5166] Floyd, S., "Metrics for the Evaluation of Congestion 1389 Control Mechanisms", RFC 5166, March 2008. 1391 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1392 Control", RFC 5681, September 2009. 1394 [SA4-EVAL] 1395 R1-081955, 3GPP., "LTE Link Level Throughput Data for SA4 1396 Evaluation Framework", 3GPP R1-081955, 5 2008. 1398 [SA4-LR] S4-050560, 3GPP., "Error Patterns for MBMS Streaming over 1399 UTRAN and GERAN", 3GPP S4-050560, 5 2008. 1401 [xiph-seq] 1402 Xiph.org, , "Video Test Media", 1403 http://media.xiph.org/video/derf/ , . 1405 [HEVC-seq] 1406 HEVC, , "Test Sequences", 1407 http://www.netlab.tkk.fi/~varun/test_sequences/ , . 1409 [TCP-eval-suite] 1410 Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier, 1411 R., Gang, W., Eggert, L., Ha, S., and I. Rhee, "Towards a 1412 Common TCP Evaluation Suite", Proc. PFLDnet. 2008, August 1413 2008. 1415 Appendix A. List of Network Parameters 1417 In addition to the recommended evaluation settings in Section 4, the 1418 implemntors can extend their tests by choosing from the following 1419 values: 1421 A.1. One-way Propagation Delay 1423 Experiments are expected to verify that the congestion control is 1424 able to work in challenging situations, for example over trans- 1425 continental and/or satellite links. Typical values are: 1427 1. Very low latency: 0-1ms 1429 2. Low latency: 50ms 1431 3. High latency: 150ms 1433 4. Extreme latency: 300ms 1435 A.2. End-to-end Loss 1437 To model lossy links, the experiments can choose one of the following 1438 loss rates, the fractional loss is the ratio of packets lost and 1439 packets sent. 1441 1. no loss: 0% 1443 2. 1% 1445 3. 5% 1447 4. 10% 1448 5. 20% 1450 A.3. DropTail Router Queue Length 1452 The router queue length is measured as the time taken to drain the 1453 FIFO queue. It has been noted in various discussions that the queue 1454 length in the current deployed Internet varies significantly. While 1455 the core backbone network has very short queue length, the home 1456 gateways usually have larger queue length. Those various queue 1457 lengths can be categorized in the following way: 1459 1. QoS-aware (or short): 70ms 1461 2. Nominal: 300-500ms 1463 3. Buffer-bloated: 1000-2000ms 1465 Here the size of the queue is measured in bytes or packets and to 1466 convert the queue length measured in seconds to queue length in 1467 bytes: 1469 QueueSize (in bytes) = QueueSize (in sec) x Throughput (in bps)/8 1471 Appendix B. Models 1473 B.1. Jitter models 1475 This section defines jitter model for the purposes of this document. 1476 When jitter is to be applied to both the RMCAT flow and any competing 1477 flow (such as a TCP competing flow), the competing flow will use the 1478 jitter definition below that does not allow for re-ordering of 1479 packets on the competing flow (see NR-RBPDV definition below). 1481 Jitter is an overloaded term in communications. Its meaning is 1482 typically associated with the variation of a metric (e.g., delay) 1483 with respect to some reference metric (e.g., average delay or minimum 1484 delay). For example, RFC 3550 jitter is a smoothed estimate of 1485 jitter which is particularly meaningful if the underlying packet 1486 delay variation was caused by a Gaussian random process. 1488 Because jitter is an overloaded term, we instead use the term Packet 1489 Delay Variation (PDV) to describe the variation of delay of 1490 individual packets in the same sense as the IETF IPPM WG has defined 1491 PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has 1492 defined IP Packet Delay Variation (IPDV) in their documents (e.g., 1493 Y.1540). 1495 Most PDV distributions in packet network systems are one-sided 1496 distributions (the measurement of which with a finite number of 1497 measurement samples result in one-sided histograms). In the usual 1498 packet network transport case there is typically one packet that 1499 transited the network with the minimum delay, then a majority of 1500 packets also transit the system within some variation from this 1501 minimum delay, and then a minority of the packets transits the 1502 network with delays higher than the median or average transit time 1503 (these are outliers). Although infrequent, outliers can cause 1504 significant deleterious operation in adaptive systems and should be 1505 considered in RMCAT adaptation designs. 1507 In this section we define two different bounded PDV characteristics, 1508 1) Random Bounded PDV and 2) Approximately Random Subject to No- 1509 Reordering Bounded PDV. 1511 Random Bounded PDV (RBPDV): 1513 The RBPDV probability distribution function (pdf) is specified to be 1514 of some mathematically describable function which includes some 1515 practical minimum and maximum discrete values suitable for testing. 1516 For example, the minimum value, x_min, might be specified as the 1517 minimum transit time packet and the maximum value, x_max, might be 1518 idefined to be two standard deviations higher than the mean. 1520 Since we are typically interested in the distribution relative to the 1521 mean delay packet, we define the zero mean PVD sample, z(n), to be 1522 z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random 1523 variable x and x_mean is the mean of x. 1525 We assume here that s(n) is the original source time of packet n and 1526 the post-jitter induced emmission time, j(n), for packet n is j(n) = 1527 {[z(n) + x_mean] + s(n)}. It follows that the separation in the post- 1528 jitter time of packets n and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}. 1529 Since the first term is always a positive quantity, we note that 1530 packet reordering at the receiver is possible whenever the second 1531 term is greater than the first. Said another way, whenever the 1532 difference in possible zero mean PDV sample delays (i.e., [x_max- 1533 x_min]) exceeds the inter-departure time of any two sent packets, we 1534 have the possibility of packet re-ordering. 1536 There are important use cases in real networks where packets can 1537 become re-ordered such as in load balancing topologies and during 1538 route changes. However, for the vast majority of cases there is no 1539 packet re-ordering because most of the time packets follow the same 1540 path. Due to this, if a packet becomes overly delayed, the packets 1541 after it on that flow are also delayed. This is especially true for 1542 mobile wireless links where there are per-flow queues prior to base 1543 station scheduling. Owing to this important use case, we define 1544 another PDV profile similar to the above, but one that does not allow 1545 for re-ordering within a flow. 1547 Approximately Random Subject to No-Reordering Bounded PDV (NR-RPVD): 1549 No Reordering RPDV, NR-RPVD, is defined similarly to the above with 1550 one important exception. Let serial(n) be defined as the 1551 serialization delay of packet n at the lowest bottleneck link rate 1552 (or other appropriate rate) in a given test. Then we produce all the 1553 post-jitter values for j(n) for n = 1, 2, ... N, where N is the 1554 length of the source sequence s to be jittered. The exception can be 1555 stated as follows: We revisit all j(n) beginning from index n=2, and 1556 if j(n) is determined to be less than [j(n-1)+serial(n-1)], we 1557 redefine j(n) to be equal to [j(n-1)+serial(n-1)] and continue for 1558 all remaining n (i.e., n = 3, 4, .. N). This models the case where 1559 the packet n is sent immediately after packet (n-1) at the bottleneck 1560 link rate. Although this is generally the theoretical minimum in 1561 that it assumes that no other packets from other flows are in-between 1562 packet n and n+1 at the bottleneck link, it is a reasonable 1563 assumption for per flow queuing. 1565 We note that this assumption holds for some important exception 1566 cases, such as packets immediately following outliers. There are a 1567 multitude of software controlled elements common on end-to-end 1568 Internet paths (such as firewalls, ALGs and other middleboxes) which 1569 stop processing packets while servicing other functions (e.g., 1570 garbage collection). Often these devices do not drop packets, but 1571 rather queue them for later processing and cause many of the 1572 outliers. Thus NR-RPVD models this particular use case (assuming 1573 serial(n+1) is defined appropriately for the device causing the 1574 outlier) and thus is believed to be important for adaptation 1575 development for RMCAT. 1577 [Editor's Note: It may require to define test distributions as well. 1578 Example test distrubution may include- 1580 1 - Two-sided: Uniform PDV Distribution. Two quantities to define: 1581 x_min and x_max. 1583 2 - Two-sided: Truncated Gaussian PDV Distribution. Four quantities 1584 to define: the appropriate x_min and x_max for test (e.g., +/- two 1585 sigma values), the standard deviation and the mean. 1587 3 - One Sided: TBD] 1589 B.2. Loss generation model 1591 [Editor's note : Describes the model for generating packet losses, 1592 for example, losses can be generated using traces, or using the 1593 Gilbert-Elliot model, or randomly (uncorrelated loss).] 1595 B.3. TCP taffic model 1597 Long-lived TCP flows will download data throughout the session and 1598 are expected to have infinite amount of data to send or receive. 1600 Each short TCP flow is modeled as a sequence of file downloads 1601 interleaved with idle periods. Not all short TCPs start at the same 1602 time, i.e., some start in the ON state while others start in the OFF 1603 state. 1605 The short TCP flows can be modelled in two ways, 1) 100s of flows 1606 fetching small (5-20 KB) amounts of data, or 2) 10s of flows fetching 1607 slightly larger (100-1000KB) amounts of data. 1609 The idle period is typically derived from an exponential distribution 1610 with the mean value of 10 seconds. 1612 [Open issue: short-lived/bursty TCP cross-traffic parameters are 1613 still to be agreed upon]. 1615 Authors' Addresses 1617 Zaheduzzaman Sarker 1618 Ericsson AB 1619 Luleae, SE 977 53 1620 Sweden 1622 Phone: +46 10 717 37 43 1623 Email: zaheduzzaman.sarker@ericsson.com 1625 Varun Singh 1626 Aalto University 1627 School of Electrical Engineering 1628 Otakaari 5 A 1629 Espoo, FIN 02150 1630 Finland 1632 Email: varun@comnet.tkk.fi 1633 URI: http://www.netlab.tkk.fi/~varun/ 1634 Xiaoqing Zhu 1635 Cisco Systems 1636 510 McCarthy Blvd 1637 Milpitas, CA 95134 1638 USA 1640 Email: xiaoqzhu@cisco.com 1642 Michael A. Ramalho 1643 Cisco Systems, Inc. 1644 8000 Hawkins Road 1645 Sarasota, FL 34241 1646 USA 1648 Phone: +1 919 476 2038 1649 Email: mramalho@cisco.com