idnits 2.17.1 draft-ietf-rmcat-wireless-tests-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 290 has weird spacing: '...nternet fix...' -- The document date (March 13, 2020) is 1505 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: '40ms' is mentioned on line 325, but not defined == Missing Reference: '150ms' is mentioned on line 325, but not defined == Missing Reference: 'Uplink' is mentioned on line 345, but not defined == Missing Reference: 'Downlink' is mentioned on line 345, but not defined -- Looks like a reference, but probably isn't: '4' on line 826 -- Looks like a reference, but probably isn't: '8' on line 826 -- Looks like a reference, but probably isn't: '12' on line 826 -- Looks like a reference, but probably isn't: '16' on line 826 -- Looks like a reference, but probably isn't: '20' on line 826 == Unused Reference: 'RFC8174' is defined on line 993, but no explicit reference was found in the text == Outdated reference: A later version (-14) exists of draft-ietf-rmcat-eval-criteria-13 Summary: 0 errors (**), 0 flaws (~~), 8 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group Z. Sarker 3 Internet-Draft Ericsson AB 4 Intended status: Informational X. Zhu 5 Expires: September 14, 2020 J. Fu 6 Cisco Systems 7 March 13, 2020 9 Evaluation Test Cases for Interactive Real-Time Media over Wireless 10 Networks 11 draft-ietf-rmcat-wireless-tests-11 13 Abstract 15 The Real-time Transport Protocol (RTP) is a common transport choice 16 for interactive multimedia communication applications. The 17 performance of these applications typically depends on a well- 18 functioning congestion control algorithm. To ensure a seamless and 19 robust user experience, a well-designed RTP-based congestion control 20 algorithm should work well across all access network types. This 21 document describes test cases for evaluating performances of 22 candidate congestion control algorithms over cellular and Wi-Fi 23 networks. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on September 14, 2020. 42 Copyright Notice 44 Copyright (c) 2020 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 2. Cellular Network Specific Test Cases . . . . . . . . . . . . 3 61 2.1. Varying Network Load . . . . . . . . . . . . . . . . . . 6 62 2.1.1. Network Connection . . . . . . . . . . . . . . . . . 6 63 2.1.2. Simulation Setup . . . . . . . . . . . . . . . . . . 7 64 2.1.3. Expected behavior . . . . . . . . . . . . . . . . . . 9 65 2.2. Bad Radio Coverage . . . . . . . . . . . . . . . . . . . 9 66 2.2.1. Network connection . . . . . . . . . . . . . . . . . 9 67 2.2.2. Simulation Setup . . . . . . . . . . . . . . . . . . 9 68 2.2.3. Expected behavior . . . . . . . . . . . . . . . . . . 10 69 2.3. Desired Evaluation Metrics for cellular test cases . . . 10 70 3. Wi-Fi Networks Specific Test Cases . . . . . . . . . . . . . 10 71 3.1. Bottleneck in Wired Network . . . . . . . . . . . . . . . 12 72 3.1.1. Network topology . . . . . . . . . . . . . . . . . . 12 73 3.1.2. Test/simulation setup . . . . . . . . . . . . . . . . 13 74 3.1.3. Typical test scenarios . . . . . . . . . . . . . . . 14 75 3.1.4. Expected behavior . . . . . . . . . . . . . . . . . . 15 76 3.2. Bottleneck in Wi-Fi Network . . . . . . . . . . . . . . . 15 77 3.2.1. Network topology . . . . . . . . . . . . . . . . . . 15 78 3.2.2. Test/simulation setup . . . . . . . . . . . . . . . . 16 79 3.2.3. Typical test scenarios . . . . . . . . . . . . . . . 17 80 3.2.4. Expected behavior . . . . . . . . . . . . . . . . . . 18 81 3.3. Other Potential Test Cases . . . . . . . . . . . . . . . 19 82 3.3.1. EDCA/WMM usage . . . . . . . . . . . . . . . . . . . 19 83 3.3.2. Effect of heterogeneous link rates . . . . . . . . . 19 84 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 85 5. Security Considerations . . . . . . . . . . . . . . . . . . . 20 86 6. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 20 87 7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 21 88 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 89 8.1. Normative References . . . . . . . . . . . . . . . . . . 21 90 8.2. Informative References . . . . . . . . . . . . . . . . . 22 91 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 93 1. Introduction 95 Wireless networks (both cellular and Wi-Fi [IEEE802.11]) are an 96 integral and increasingly more significant part of the Internet. 97 Typical application scenarios for interactive multimedia 98 communication over wireless include from video conferencing calls in 99 a bus or train as well as live media streaming at home. It is well 100 known that the characteristics and technical challenges for 101 supporting multimedia services over wireless are very different from 102 those of providing the same service over a wired network. Although 103 the basic test cases as defined in [I-D.ietf-rmcat-eval-test] have 104 covered many common effects of network impairments for evaluating 105 RTP-based congestion control schemes, they remain to be tested over 106 characteristics and dynamics unique to a given wireless environment. 107 For example, in cellular networks, the base station maintains 108 individual queues per radio bearer per user hence it leads to a 109 different nature of interactions between traffic flows of different 110 users. This contrasts with a typical wired network setting where 111 traffic flows from all users share the same queue at the bottleneck. 112 Furthermore, user mobility patterns in a cellular network differ from 113 those in a Wi-Fi network. Therefore, it is important to evaluate the 114 performance of proposed candidate RTP-based congestion control 115 solutions over cellular mobile networks and over Wi-Fi networks 116 respectively. 118 The draft [I-D.ietf-rmcat-eval-criteria] provides the guideline for 119 evaluating candidate algorithms and recognizes the importance of 120 testing over wireless access networks. However, it does not describe 121 any specific test cases for performance evaluation of candidate 122 algorithms. This document describes test cases specifically 123 targeting cellular and Wi-Fi networks. 125 2. Cellular Network Specific Test Cases 127 A cellular environment is more complicated than its wireline 128 counterpart since it seeks to provide services in the context of 129 variable available bandwidth, location dependencies and user 130 mobilities at different speeds. In a cellular network, the user may 131 reach the cell edge which may lead to a significant amount of 132 retransmissions to deliver the data from the base station to the 133 destination and vice versa. These radio links will often act as a 134 bottleneck for the rest of the network and will eventually lead to 135 excessive delays or packet drops. An efficient retransmission or 136 link adaptation mechanism can reduce the packet loss probability but 137 there will remain some packet losses and delay variations. Moreover, 138 with increased cell load or handover to a congested cell, congestion 139 in the transport network will become even worse. Besides, there 140 exist certain characteristics that distinguish the cellular network 141 from other wireless access networks such as Wi-Fi. In a cellular 142 network -- 144 o The bottleneck is often a shared link with relatively few users. 146 * The cost per bit over the shared link varies over time and is 147 different for different users. 149 * Leftover/unused resources can be consumed by other greedy 150 users. 152 o Queues are always per radio bearer hence each user can have many 153 such queues. 155 o Users can experience both Inter and Intra Radio Access Technology 156 (RAT) handovers (see [HO-def-3GPP] for the definition of 157 "handover"). 159 o Handover between cells or change of serving cells (as described in 160 [HO-LTE-3GPP] and [HO-UMTS-3GPP]) might cause user plane 161 interruptions which can lead to bursts of packet losses, delay 162 and/or jitter. The exact behavior depends on the type of radio 163 bearer. Typically, the default best-effort bearers do not 164 generate packet loss, instead, packets are queued up and 165 transmitted once the handover is completed. 167 o The network part decides how much the user can transmit. 169 o The cellular network has variable link capacity per user. 171 * It can vary as fast as a period of milliseconds. 173 * It depends on many factors (such as distance, speed, 174 interference, different flows). 176 * It uses complex and smart link adaptation which makes the link 177 behavior ever more dynamic. 179 * The scheduling priority depends on the estimated throughput. 181 o Both Quality of Service (QoS) and non-QoS radio bearers can be 182 used. 184 Hence, a real-time communication application operating over a 185 cellular network needs to cope with a shared bottleneck link and 186 variable link capacity, events like handover, non-congestion related 187 loss, abrupt changes in bandwidth (both short term and long term) due 188 to handover, network load and bad radio coverage. Even though 3GPP 189 has defined QoS bearers [QoS-3GPP] to ensure high-quality user 190 experience, it is still preferable for real-time applications to 191 behave in an adaptive manner. 193 Different mobile operators deploy their own cellular networks with 194 their own set of network functionalities and policies. Usually, a 195 mobile operator network includes a range of radio access technologies 196 such as 3G and 4G/LTE. Looking at the specifications of such radio 197 technologies it is evident that only the more recent radio 198 technologies can support the high bandwidth requirements from real- 199 time interactive video applications. The future real-time 200 interactive application will impose even greater demand on cellular 201 network performance which makes 4G (and beyond) radio technologies 202 more suitable for such genre of application. 204 The key factors in defining test cases for cellular networks are: 206 o Shared and varying link capacity 208 o Mobility 210 o Handover 212 However, these factors are typically highly correlated in a cellular 213 network. Therefore, instead of devising separate test cases for 214 individual important events, we have divided the test case into two 215 categories. It should be noted that the goal of the following test 216 cases is to evaluate the performance of candidate algorithms over the 217 radio interface of the cellular network. Hence it is assumed that 218 the radio interface is the bottleneck link between the communicating 219 peers and that the core network does not introduce any extra 220 congestion along the path. Consequently, this draft has kept as out 221 of scope the combination of multiple access technologies involving 222 both cellular and Wi-Fi users. In this latter case the shared 223 bottleneck is likely at the wired backhaul link. These test cases 224 further assume a typical real-time telephony scenario where one real- 225 time session consists of one voice stream and one video stream. 227 Even though it is possible to carry out tests over operational 228 cellular networks (e.g., LTE/5G), and actually such tests are already 229 available today, these tests cannot in general be carried out in a 230 deterministic fashion to ensure repeatability. The main reason is 231 that these networks are controlled by cellular operators and there 232 exist various amounts of competing traffic in the same cell(s). In 233 practice, it is only in underground mines that one can carry out near 234 deterministic testing. Even there, it is not guaranteed either as 235 workers in the mines may carry with them their personal mobile 236 phones. Furthermore, the underground mining setting may not reflect 237 typical usage patterns in an urban setting. We, therefore, recommend 238 that a cellular network simulator is used for the test cases defined 239 in this document, for example -- the LTE simulator in [NS-3]. 241 2.1. Varying Network Load 243 The goal of this test is to evaluate the performance of the candidate 244 congestion control algorithm under varying network load. The network 245 load variation is created by adding and removing network users a.k.a. 246 User Equipments (UEs) during the simulation. In this test case, each 247 user/UE in the media session is an endpoint following RTP-based 248 congestion control. User arrivals follow a Poisson distribution 249 proportional to the length of the call, to keep the number of users 250 per cell fairly constant during the evaluation period. At the 251 beginning of the simulation, there should be enough time to warm-up 252 the network. This is to avoid running the evaluation in an empty 253 network where network nodes are having empty buffers, low 254 interference at the beginning of the simulation. This network 255 initialization period should be excluded from the evaluation period. 256 Typically, the evaluation period starts 30 seconds after test 257 initialization. 259 This test case also includes user mobility and some competing 260 traffic. The latter includes both the same types of flows (with same 261 adaptation algorithms) and different types of flows (with different 262 services and congestion control schemes). 264 2.1.1. Network Connection 266 Each mobile user is connected to a fixed user. The connection 267 between the mobile user and fixed user consists of a cellular radio 268 access, an Evolved Packet Core (EPC) and an Internet connection. The 269 mobile user is connected to the EPC using cellular radio access 270 technology which is further connected to the Internet. At the other 271 end, the fixed user is connected to the Internet via wired connection 272 with sufficiently high bandwidth, for instance, 10 Gbps, so that the 273 system bottleneck is on the cellular radio access interface. The 274 wired connection to in this setup does not introduce any network 275 impairments to the test; it only adds 10 ms of one-way propagation 276 delay. 278 The path from the fixed user to the mobile users is defined as 279 "Downlink" and the path from the mobile users to the fixed user is 280 defined as "Uplink". We assume that only uplink or downlink is 281 congested for mobile users. Hence, we recommend that the uplink and 282 downlink simulations are run separately. 284 uplink 285 ++))) +--------------------------> 286 ++-+ ((o)) 287 | | / \ +-------+ +------+ +---+ 288 +--+ / \----+ +-----+ +----+ | 289 / \ +-------+ +------+ +---+ 290 UE BS EPC Internet fixed 291 <--------------------------+ 292 downlink 294 Figure 1: Simulation Topology 296 2.1.2. Simulation Setup 298 The values enclosed within "[ ]" for the following simulation 299 attributes follow the same notion as in [I-D.ietf-rmcat-eval-test]. 300 The desired simulation setup is as follows -- 302 1. Radio environment: 304 A. Deployment and propagation model: 3GPP case 1 (see 305 [HO-deploy-3GPP]) 307 B. Antenna: Multiple-Input and Multiple-Output (MIMO), 2D or 3D 308 antenna pattern. 310 C. Mobility: [3km/h, 30km/h] 312 D. Transmission bandwidth: 10MHz 314 E. Number of cells: multi-cell deployment (3 Cells per Base 315 Station (BS) * 7 BS) = 21 cells 317 F. Cell radius: 166.666 Meters 319 G. Scheduler: Proportional fair with no priority 321 H. Bearer: Default bearer for all traffic. 323 I. Active Queue Management (AQM) settings: AQM [on,off] 325 2. End-to-end Round Trip Time (RTT): [40ms, 150ms] 327 3. User arrival model: Poisson arrival model 329 4. User intensity: 331 * Downlink user intensity: {0.7, 1.4, 2.1, 2.8, 3.5, 4.2, 4.9, 332 5.6, 6.3, 7.0, 7.7, 8.4, 9,1, 9.8, 10.5} 334 * Uplink user intensity : {0.7, 1.4, 2.1, 2.8, 3.5, 4.2, 4.9, 335 5.6, 6.3, 7.0} 337 5. Simulation duration: 91s 339 6. Evaluation period: 30s-60s 341 7. Media traffic: 343 1. Media type: Video 345 a. Media direction: [Uplink, Downlink] 347 b. Number of Media source per user: One (1) 349 c. Media duration per user: 30s 351 d. Media source: same as defined in Section 4.3 of 352 [I-D.ietf-rmcat-eval-test] 354 2. Media Type: Audio 356 a. Media direction: Uplink and Downlink 358 b. Number of Media source per user: One (1) 360 c. Media duration per user: 30s 362 d. Media codec: Constant Bit Rate (CBR) 364 e. Media bitrate: 20 Kbps 366 f. Adaptation: off 368 8. Other traffic models: 370 * Downlink simulation: Maximum of 4Mbps/cell (web browsing or 371 FTP traffic following default TCP congestion control 372 [RFC5681]) 374 * Unlink simulation: Maximum of 2Mbps/cell (web browsing or FTP 375 traffic following default TCP congestion control [RFC5681]) 377 2.1.3. Expected behavior 379 The investigated congestion control algorithms should result in 380 maximum possible network utilization and stability in terms of rate 381 variations, lowest possible end to end frame latency, network latency 382 and Packet Loss Rate (PLR) at different cell load levels. 384 2.2. Bad Radio Coverage 386 The goal of this test is to evaluate the performance of candidate 387 congestion control algorithm when users visit part of the network 388 with bad radio coverage. The scenario is created by using a larger 389 cell radius than that in the previous test case. In this test case, 390 each user/UE in the media session is an endpoint following RTP-based 391 congestion control. User arrivals follow a Poisson distribution 392 proportional to the length of the call, to keep the number of users 393 per cell fairly constant during the evaluation period. At the 394 beginning of the simulation, there should be enough amount of time to 395 warm-up the network. This is to avoid running the evaluation in an 396 empty network where network nodes are having empty buffers, low 397 interference at the beginning of the simulation. This network 398 initialization period should be excluded from the evaluation period. 399 Typically, the evaluation period starts 30 seconds after test 400 initialization. 402 This test case also includes user mobility and some competing 403 traffic. The latter includes the same kind of flows (with same 404 adaptation algorithms). 406 2.2.1. Network connection 408 Same as defined in Section 2.1.1 410 2.2.2. Simulation Setup 412 The desired simulation setup is the same as the Varying Network Load 413 test case defined in Section 2.1 except the following changes: 415 1. Radio environment: Same as defined in Section 2.1.2 except the 416 following: 418 A. Deployment and propagation model: 3GPP case 3 (see 419 [HO-deploy-3GPP]) 421 B. Cell radius: 577.3333 Meters 423 C. Mobility: 3km/h 425 2. User intensity = {0.7, 1.4, 2.1, 2.8, 3.5, 4.2, 4.9, 5.6, 6.3, 426 7.0} 428 3. Media traffic model: Same as defined in Section 2.1.2 430 4. Other traffic models: 432 * Downlink simulation: Maximum of 2Mbps/cell (web browsing or 433 FTP traffic following default TCP congestion control 434 [RFC5681]) 436 * Unlink simulation: Maximum of 1Mbps/cell (web browsing or FTP 437 traffic following default TCP congestion control [RFC5681]) 439 2.2.3. Expected behavior 441 The investigated congestion control algorithms should result in 442 maximum possible network utilization and stability in terms of rate 443 variations, lowest possible end to end frame latency, network latency 444 and Packet Loss Rate (PLR) at different cell load levels. 446 2.3. Desired Evaluation Metrics for cellular test cases 448 The evaluation criteria document [I-D.ietf-rmcat-eval-criteria] 449 defines the metrics to be used to evaluate candidate algorithms. 450 Considering the nature and distinction of cellular networks we 451 recommend that at least the following metrics be used to evaluate the 452 performance of the candidate algorithms: 454 o Average cell throughput (for all cells), shows cell utilizations. 456 o Application sending and receiving bitrate, goodput. 458 o Packet Loss Rate (PLR). 460 o End-to-end Media frame delay. For video, this means the delay 461 from capture to display. 463 o Transport delay. 465 o Algorithm stability in terms of rate variation. 467 3. Wi-Fi Networks Specific Test Cases 469 Given the prevalence of Internet access links over Wi-Fi, it is 470 important to evaluate candidate RTP-based congestion control 471 solutions over test cases that include Wi-Fi access links. Such 472 evaluations should highlight the inherently different characteristics 473 of Wi-Fi networks in contrast to their wired counterparts: 475 o The wireless radio channel is subject to interference from nearby 476 transmitters, multipath fading, and shadowing. These effects lead 477 to fluctuations in the link throughput and sometimes an error- 478 prone communication environment. 480 o Available network bandwidth is not only shared over the air 481 between concurrent users but also between uplink and downlink 482 traffic due to the half-duplex nature of the wireless transmission 483 medium. 485 o Packet transmissions over Wi-Fi are susceptible to contentions and 486 collisions over the air. Consequently, traffic load beyond a 487 certain utilization level over a Wi-Fi network can introduce 488 frequent collisions over the air and significant network overhead, 489 as well as packet drops due to buffer overflow at the 490 transmitters. This, in turn, leads to excessive delay, 491 retransmissions, packet losses and lower effective bandwidth for 492 applications. Note further that the collision-induced delay and 493 loss patterns are qualitatively different from those caused by 494 congestion over a wired connection. 496 o The IEEE 802.11 standard (i.e., Wi-Fi) supports multi-rate 497 transmission capabilities by dynamically choosing the most 498 appropriate modulation and coding scheme (MCS) for the given 499 received signal strength. A different choice in the MCS Index 500 leads to different physical-layer (PHY-layer) link rates and 501 consequently different application-layer throughput. 503 o The presence of legacy devices (e.g., ones operating only in IEEE 504 802.11b) at a much lower PHY-layer link rate can significantly 505 slow down the rest of a modern Wi-Fi network. As discussed in 506 [Heusse2003], the main reason for such anomaly is that it takes 507 much longer to transmit the same packet over a slower link than 508 over a faster link, thereby consuming a substantial portion of air 509 time. 511 o Handover from one Wi-Fi Access Point (AP) to another may lead to 512 excessive packet delays and losses during the process. 514 o IEEE 802.11e has introduced the Enhanced Distributed Channel 515 Access (EDCA) mechanism to allow different traffic categories to 516 contend for channel access using different random back-off 517 parameters. This mechanism is a mandatory requirement for the Wi- 518 Fi Multimedia (WMM) certification in Wi-Fi Alliance. It allows 519 for prioritization of real-time application traffic such as voice 520 and video over non-urgent data transmissions (e.g., file 521 transfer). 523 In summary, the presence of Wi-Fi access links in different network 524 topologies can exert different impact on the network performance in 525 terms of application-layer effective throughput, packet loss rate, 526 and packet delivery delay. These, in turn, will influence the 527 behavior of end-to-end real-time multimedia congestion control. 529 Unless otherwise mentioned, the test cases in this section choose the 530 PHY- and MAC-layer parameters based on the IEEE 802.11n Standard. 531 Statistics collected from enterprise Wi-Fi networks show that the two 532 dominant physical modes are 802.11n and 802.11ac, accounting for 41% 533 and 58% of connected devices. As Wi-Fi standards evolve over time -- 534 for instance, with the introduction of the emerging Wi-Fi 6 (based on 535 IEEE 802.11ax) products -- the PHY- and MAC-layer test case 536 specifications need to be updated accordingly to reflect such 537 changes. 539 Typically, a Wi-Fi access network connects to a wired infrastructure. 540 Either the wired or the Wi-Fi segment of the network can be the 541 bottleneck. The following sections describe basic test cases for 542 both scenarios separately. The same set of performance metrics as in 543 [I-D.ietf-rmcat-eval-test]) should be collected for each test case. 545 We recommend to carry out the test cases as defined in this document 546 using a simulator, such as [NS-2] or [NS-3]. When feasible, it is 547 encouraged to perform testbed-based evaluations using Wi-Fi access 548 points and endpoints running up-to-date IEEE 802.11 protocols, such 549 as 802.11ac and the emerging Wi-Fi 6, so as to verify the viability 550 of the candidate schemes. 552 3.1. Bottleneck in Wired Network 554 The test scenarios below are intended to mimic the setup of video 555 conferencing over Wi-Fi connections from the home. Typically, the 556 Wi-Fi home network is not congested and the bottleneck is present 557 over the wired home access link. Although it is expected that test 558 evaluation results from this section are similar to those as in 559 [I-D.ietf-rmcat-eval-test], it is still worthwhile to run through 560 these tests as sanity checks. 562 3.1.1. Network topology 564 Figure 2 shows the network topology of Wi-Fi test cases. The test 565 contains multiple mobile nodes (MNs) connected to a common Wi-Fi 566 access point (AP) and their corresponding wired clients on fixed 567 nodes (FNs). Each connection carries either a RTP-based media flow 568 or a TCP traffic flow. Directions of the flows can be uplink (i.e., 569 from mobile nodes to fixed nodes), downlink (i.e., from fixed nodes 570 to mobile nodes), or bi-directional. The total number of 571 uplink/downlink/bi-directional flows for RTP-based media traffic and 572 TCP traffic are denoted as N and M, respectively. 574 Uplink 575 +----------------->+ 576 +------+ +------+ 577 | MN_1 |)))) /=====| FN_1 | 578 +------+ )) // +------+ 579 . )) // . 580 . )) // . 581 . )) // . 582 +------+ +----+ +-----+ +------+ 583 | MN_N | ))))))) | | | |========| FN_N | 584 +------+ | | | | +------+ 585 | AP |=========| FN0 | 586 +----------+ | | | | +----------+ 587 | MN_tcp_1 | )))) | | | |======| FN_tcp_1 | 588 +----------+ +----+ +-----+ +----------+ 589 . )) \\ . 590 . )) \\ . 591 . )) \\ . 592 +----------+ )) \\ +----------+ 593 | MN_tcp_M |))) \=====| FN_tcp_M | 594 +----------+ +----------+ 595 +<-----------------+ 596 Downlink 598 Figure 2: Network topology for Wi-Fi test cases 600 3.1.2. Test/simulation setup 602 o Test duration: 120s 604 o Wi-Fi network characteristics: 606 * Radio propagation model: Log-distance path loss propagation 607 model (see [NS3WiFi]) 609 * PHY- and MAC-layer configuration: IEEE 802.11n 611 * MCS Index at 11: 16-QAM 1/2, Raw Data Rate at 52Mbps 613 o Wired path characteristics: 615 * Path capacity: 1Mbps 617 * One-Way propagation delay: 50ms. 619 * Maximum end-to-end jitter: 30ms 621 * Bottleneck queue type: Drop tail. 623 * Bottleneck queue size: 300ms. 625 * Path loss ratio: 0%. 627 o Application characteristics: 629 * Media Traffic: 631 + Media type: Video 633 + Media direction: See Section 3.1.3 635 + Number of media sources (N): See Section 3.1.3 637 + Media timeline: 639 - Start time: 0s. 641 - End time: 119s. 643 * Competing traffic: 645 + Type of sources: long-lived TCP or CBR over UDP 647 + Traffic direction: See Section 3.1.3 649 + Number of sources (M): See Section 3.1.3 651 + Congestion control: Default TCP congestion control [RFC5681] 652 or constant-bit-rate (CBR) traffic over UDP. 654 + Traffic timeline: See Section 3.1.3 656 3.1.3. Typical test scenarios 658 o Single uplink RTP-based media flow: N=1 with uplink direction and 659 M=0. 661 o One pair of bi-directional RTP-based media flows: N=2 (i.e., one 662 uplink flow and one downlink flow); M=0. 664 o One pair of bi-directional RTP-based media flows: N=2; one uplink 665 on-off CBR flow over UDP: M=1 (uplink). The CBR flow has ON time 666 at t=0s-60s and OFF time at t=60s-119s. 668 o One pair of bi-directional RTP-based media flows: N=2; one uplink 669 off-on CBR flow over UDP: M=1 (uplink). The CBR flow has OFF time 670 at t=0s-60s and ON time at t=60s-119s. 672 o One RTP-based media flow competing against one long-live TCP flow 673 in the uplink direction: N=1 (uplink) and M = 1(uplink). The TCP 674 flow has start time at t=0s and end time at t=119s. 676 3.1.4. Expected behavior 678 o Single uplink RTP-based media flow: the candidate algorithm is 679 expected to detect the path capacity constraint, to converge to 680 the bottleneck link capacity, and to adapt the flow to avoid 681 unwanted oscillations when the sending bit rate is approaching the 682 bottleneck link capacity. No excessive oscillations in the media 683 rate should be present. 685 o Bi-directional RTP-based media flows: the candidate algorithm is 686 expected to converge to the bottleneck capacity of the wired path 687 in both directions despite the presence of measurement noise over 688 the Wi-Fi connection. In the presence of background TCP or CBR 689 over UDP traffic, the rate of RTP-based media flows should adapt 690 promptly to the arrival and departure of background traffic flows. 692 o One RTP-based media flow competing with long-live TCP flow in the 693 uplink direction: the candidate algorithm is expected to avoid 694 congestion collapse and to stabilize at a fair share of the 695 bottleneck link capacity. 697 3.2. Bottleneck in Wi-Fi Network 699 The test cases in this section assume that the wired segment along 700 the media path is well-provisioned whereas the bottleneck exists over 701 the Wi-Fi access network. This is to mimic the application scenarios 702 typically encountered by users in an enterprise environment or at a 703 coffee house. 705 3.2.1. Network topology 707 Same as defined in Section 3.1.1 709 3.2.2. Test/simulation setup 711 o Test duration: 120s 713 o Wi-Fi network characteristics: 715 * Radio propagation model: Log-distance path loss propagation 716 model (see [NS3WiFi]) 718 * PHY- and MAC-layer configuration: IEEE 802.11n 720 * MCS Index at 11: 16-QAM 1/2, Raw Data Rate at 52Mbps 722 o Wired path characteristics: 724 * Path capacity: 100Mbps. 726 * One-Way propagation delay: 50ms. 728 * Maximum end-to-end jitter: 30ms. 730 * Bottleneck queue type: Drop tail. 732 * Bottleneck queue size: 300ms. 734 * Path loss ratio: 0%. 736 o Application characteristics: 738 * Media Traffic: 740 + Media type: Video 742 + Media direction: See Section 3.2.3. 744 + Number of media sources (N): See Section 3.2.3. 746 + Media timeline: 748 - Start time: 0s. 750 - End time: 119s. 752 * Competing traffic: 754 + Type of sources: long-lived TCP or CBR over UDP. 756 + Number of sources (M): See Section 3.2.3. 758 + Traffic direction: See Section 3.2.3. 760 + Congestion control: Default TCP congestion control [RFC5681] 761 or constant-bit-rate (CBR) traffic over UDP. 763 + Traffic timeline: See Section 3.2.3. 765 3.2.3. Typical test scenarios 767 This section describes a few test scenarios that are deemed as 768 important for understanding the behavior of a candidate RTP-based 769 congestion control scheme over a Wi-Fi network. 771 a. Multiple RTP-based media flows sharing the wireless downlink: 772 N=16 (all downlink); M = 0. This test case is for studying the 773 impact of contention on the multiple concurrent media flows. For 774 an 802.11n network, given the MCS Index of 11 and the 775 corresponding link rate of 52Mbps, the total application-layer 776 throughput (assuming reasonable distance, low interference and 777 infrequent contentions caused by competing streams) is around 778 20Mbps. A total of N=16 RTP-based media flows (with a maximum 779 rate of 1.5Mbps each) are expected to saturate the wireless 780 interface in this experiment. Evaluation of a given candidate 781 scheme should focus on whether the downlink media flows can 782 stabilize at a fair share of the total application-layer 783 throughput. 785 b. Multiple RTP-based media flows sharing the wireless uplink: N = 786 16 (all uplink); M = 0. When multiple clients attempt to 787 transmit media packets uplink over the Wi-Fi network, they 788 introduce more frequent contentions and potential collisions. 789 Per-flow throughput is expected to be lower than that in the 790 previous downlink-only scenario. Evaluation of a given candidate 791 scheme should focus on whether the uplink flows can stabilize at 792 a fair share of the total application-layer throughput. 794 c. Multiple bi-directional RTP-based media flows: N = 16 (8 uplink 795 and 8 downlink); M = 0. The goal of this test is to evaluate the 796 performance of the candidate scheme in terms of bandwidth 797 fairness between uplink and downlink flows. 799 d. Multiple bi-directional RTP-based media flows with on-off CBR 800 traffic over UDP: N = 16 (8 uplink and 8 downlink); M = 5 801 (uplink). The goal of this test is to evaluate the adaptation 802 behavior of the candidate scheme when its available bandwidth 803 changes due to the departure of background traffic. The 804 background traffic consists of several (e.g., M=5) CBR flows 805 transported over UDP. These background flows are ON at time 806 t=0-60s and OFF at time t=61-120s. 808 e. Multiple bi-directional RTP-based media flows with off-on CBR 809 traffic over UDP: N = 16 (8 uplink and 8 downlink); M = 5 810 (uplink). The goal of this test is to evaluate the adaptation 811 behavior of the candidate scheme when its available bandwidth 812 changes due to the arrival of background traffic. The background 813 traffic consists of several (e.g., M=5) parallel CBR flows 814 transported over UDP. These background flows are OFF at time 815 t=0-60s and ON at times t=61-120s. 817 f. Multiple bi-directional RTP-based media flows in the presence of 818 background TCP traffic: N=16 (8 uplink and 8 downlink); M = 5 819 (uplink). The goal of this test is to evaluate how RTP-based 820 media flows compete against TCP over a congested Wi-Fi network 821 for a given candidate scheme. TCP flows have start time at t=40s 822 and end time at t=80s. 824 g. Varying number of RTP-based media flows: A series of tests can be 825 carried out for the above test cases with different values of N, 826 e.g., N = [4, 8, 12, 16, 20]. The goal of this test is to 827 evaluate how a candidate scheme responds to varying traffic load/ 828 demand over a congested Wi-Fi network. The start times of the 829 media flows are randomly distributes within a window of t=0-10s; 830 their end times are randomly distributed within a window of 831 t=110-120s. 833 3.2.4. Expected behavior 835 o Multiple downlink RTP-based media flows: each media flow is 836 expected to get its fair share of the total bottleneck link 837 bandwidth. Overall bandwidth usage should not be significantly 838 lower than that experienced by the same number of concurrent 839 downlink TCP flows. In other words, the behavior of multiple 840 concurrent TCP flows will be used as a performance benchmark for 841 this test scenario. The end-to-end delay and packet loss ratio 842 experienced by each flow should be within an acceptable range for 843 real-time multimedia applications. 845 o Multiple uplink RTP-based media flows: overall bandwidth usage by 846 all media flows should not be significantly lower than that 847 experienced by the same number of concurrent uplink TCP flows. In 848 other words, the behavior of multiple concurrent TCP flows will be 849 used as a performance benchmark for this test scenario. 851 o Multiple bi-directional RTP-based media flows with dynamic 852 background traffic carrying CBR flows over UDP: the media flows 853 are expected to adapt in a timely fashion to the changes in 854 available bandwidth introduced by the arrival/departure of 855 background traffic. 857 o Multiple bi-directional RTP-based media flows with dynamic 858 background traffic over TCP: during the presence of TCP background 859 flows, the overall bandwidth usage by all media flows should not 860 be significantly lower than those achieved by the same number of 861 bi-directional TCP flows. In other words, the behavior of 862 multiple concurrent TCP flows will be used as a performance 863 benchmark for this test scenario. All downlink media flows are 864 expected to obtain similar bandwidth as each other. The 865 throughput of each media flow is expected to decrease upon the 866 arrival of TCP background traffic and, conversely, increase upon 867 their departure. Both reactions should occur in a timely fashion, 868 for example, within 10s of seconds. 870 o Varying number of bi-directional RTP-based media flows: the test 871 results for varying values of N -- while keeping all other 872 parameters constant -- is expected to show steady and stable per- 873 flow throughput for each value of N. The average throughput of 874 all media flows is expected to stay constant around the maximum 875 rate when N is small, then gradually decrease with increasing 876 value of N till it reaches the minimum allowed rate, beyond which 877 the offered load to the Wi-Fi network exceeds its capacity (i.e., 878 with a very large value of N). 880 3.3. Other Potential Test Cases 882 3.3.1. EDCA/WMM usage 884 The EDCA/WMM mechanism defines prioritized QoS for four traffic 885 classes (or Access Categories). RTP-based real-time media flows 886 should achieve better performance in terms of lower delay and fewer 887 packet losses with EDCA/WMM enabled when competing against non- 888 interactive background traffic such as file transfers. When most of 889 the traffic over Wi-Fi is dominated by media, however, turning on WMM 890 may degrade performance since all media flows now attempt to access 891 the wireless transmission medium more aggressively, thereby causing 892 more frequent collisions and collision-induced losses. This is a 893 topic worthy of further investigation. 895 3.3.2. Effect of heterogeneous link rates 897 As discussed in [Heusse2003], the presence of clients operating over 898 slow PHY-layer link rates (e.g., a legacy 802.11b device) connected 899 to a modern network may adversely impact the overall performance of 900 the network. Additional test cases can be devised to evaluate the 901 effect of clients with heterogeneous link rates on the performance of 902 the candidate congestion control algorithm. Such test cases, for 903 instance, can specify that the PHY-layer link rates for all clients 904 span over a wide range (e.g., 2Mbps to 54Mbps) for investigating its 905 effect on the congestion control behavior of the real-time 906 interactive applications. 908 4. IANA Considerations 910 This memo includes no request to IANA. 912 5. Security Considerations 914 The security considerations in [I-D.ietf-rmcat-eval-criteria] and the 915 relevant congestion control algorithms apply. The principles for 916 congestion control are described in [RFC2914], and in particular, any 917 new method must implement safeguards to avoid congestion collapse of 918 the Internet. 920 Given the difficulty of deterministic wireless testing, it is 921 recommended and expected that the tests described in this document 922 would be done via simulations. However, in the case where these test 923 cases are carried out in a testbed setting, the evaluation should 924 take place in a controlled lab environment. In the testbed, the 925 applications, simulators and network nodes ought to be well-behaved 926 and should not impact the desired results. It is important to take 927 appropriate caution to avoid leaking non-responsive traffic with 928 unproven congestion avoidance behavior onto the open Internet. 930 6. Contributors 932 The following individuals contributed to the design, implementation, 933 and verification of the proposed test cases during earlier stages of 934 this work. They have helped to validate and substantially improve 935 this specification. 937 Ingemar Johansson, of Ericsson AB 938 contributing to the description and validation of cellular test cases 939 during the earlier stage of this draft. 941 Wei-Tian Tan, , of Cisco Systems designed and set up 942 a Wi-Fi testbed for evaluating parallel video conferencing streams, 943 based upon which proposed Wi-Fi test cases are described. He also 944 recommended additional test cases to consider, such as the impact of 945 EDCA/WMM usage. 947 Michael A. Ramalho, of AcousticComms Consulting 948 (previously at Cisco Systems) applied learnings from Cisco's internal 949 experimentation to the early versions of the draft. He also worked 950 on validating the proposed test cases in a VM-based lab setting. 952 7. Acknowledgments 954 The authors would like to thank Tomas Frankkila, Magnus Westerlund, 955 Kristofer Sandlund, Sergio Mena de la Cruz, and Mirja Kuehlewind for 956 their valuable inputs and review comments regarding this draft. 958 8. References 960 8.1. Normative References 962 [HO-deploy-3GPP] 963 TS 25.814, 3GPP., "Physical layer aspects for evolved 964 Universal Terrestrial Radio Access (UTRA)", October 2006, 965 . 968 [I-D.ietf-rmcat-eval-criteria] 969 Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion 970 Control for Interactive Real-time Media", draft-ietf- 971 rmcat-eval-criteria-13 (work in progress), March 2020. 973 [I-D.ietf-rmcat-eval-test] 974 Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test 975 Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- 976 eval-test-10 (work in progress), May 2019. 978 [IEEE802.11] 979 IEEE, "Standard for Information technology-- 980 Telecommunications and information exchange between 981 systems Local and metropolitan area networks--Specific 982 requirements Part 11: Wireless LAN Medium Access Control 983 (MAC) and Physical Layer (PHY) Specifications", 2012. 985 [NS3WiFi] "Wi-Fi Channel Model in ns-3 Simulator", 986 . 989 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 990 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 991 . 993 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 994 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 995 May 2017, . 997 8.2. Informative References 999 [Heusse2003] 1000 Heusse, M., Rousseau, F., Berger-Sabbatel, G., and A. 1001 Duda, "Performance anomaly of 802.11b", in Proc. 23th 1002 Annual Joint Conference of the IEEE Computer and 1003 Communications Societies, (INFOCOM'03), March 2003. 1005 [HO-def-3GPP] 1006 TR 21.905, 3GPP., "Vocabulary for 3GPP Specifications", 1007 December 2009, . 1010 [HO-LTE-3GPP] 1011 TS 36.331, 3GPP., "E-UTRA- Radio Resource Control (RRC); 1012 Protocol specification", December 2011, 1013 . 1016 [HO-UMTS-3GPP] 1017 TS 25.331, 3GPP., "Radio Resource Control (RRC); Protocol 1018 specification", December 2011, 1019 . 1022 [NS-2] "ns-2", December 2014, 1023 . 1025 [NS-3] "ns-3 Network Simulator", . 1027 [QoS-3GPP] 1028 TS 23.203, 3GPP., "Policy and charging control 1029 architecture", June 2011, . 1032 [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, 1033 RFC 2914, DOI 10.17487/RFC2914, September 2000, 1034 . 1036 Authors' Addresses 1037 Zaheduzzaman Sarker 1038 Ericsson AB 1039 Laboratoriegraend 11 1040 Luleae 97753 1041 Sweden 1043 Phone: +46 107173743 1044 Email: zaheduzzaman.sarker@ericsson.com 1046 Xiaoqing Zhu 1047 Cisco Systems 1048 12515 Research Blvd., Building 4 1049 Austin, TX 78759 1050 USA 1052 Email: xiaoqzhu@cisco.com 1054 Jiantao Fu 1055 Cisco Systems 1056 771 Alder Drive 1057 Milpitas, CA 95035 1058 USA 1060 Email: jianfu@cisco.com