idnits 2.17.1 draft-ietf-mops-streaming-opcons-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (12 July 2020) is 1381 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MOPS J. Holland 3 Internet-Draft Akamai Technologies, Inc. 4 Intended status: Informational A. Begen 5 Expires: 13 January 2021 Networked Media 6 S. Dawkins 7 Tencent America LLC 8 12 July 2020 10 Operational Considerations for Streaming Media 11 draft-ietf-mops-streaming-opcons-02 13 Abstract 15 This document provides an overview of operational networking issues 16 that pertain to quality of experience in delivery of video and other 17 high-bitrate media over the internet. 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at https://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on 13 January 2021. 36 Copyright Notice 38 Copyright (c) 2020 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 43 license-info) in effect on the date of publication of this document. 44 Please review these documents carefully, as they describe your rights 45 and restrictions with respect to this document. Code Components 46 extracted from this document must include Simplified BSD License text 47 as described in Section 4.e of the Trust Legal Provisions and are 48 provided without warranty as described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Notes for Contributors and Reviewers . . . . . . . . . . 3 54 1.1.1. Venues for Contribution and Discussion . . . . . . . 3 55 1.1.2. Template for Contributions . . . . . . . . . . . . . 3 56 1.1.3. History of Public Discussion . . . . . . . . . . . . 4 57 2. Bandwidth Provisioning . . . . . . . . . . . . . . . . . . . 5 58 2.1. Scaling Requirements for Media Delivery . . . . . . . . . 5 59 2.1.1. Video Bitrates . . . . . . . . . . . . . . . . . . . 5 60 2.1.2. Virtual Reality Bitrates . . . . . . . . . . . . . . 5 61 2.2. Path Requirements . . . . . . . . . . . . . . . . . . . . 6 62 2.3. Caching Systems . . . . . . . . . . . . . . . . . . . . . 6 63 2.4. Predictable Usage Profiles . . . . . . . . . . . . . . . 6 64 2.5. Unpredictable Usage Profiles . . . . . . . . . . . . . . 7 65 2.6. Extremely Unpredictable Usage Profiles . . . . . . . . . 8 66 3. Adaptive Bitrate . . . . . . . . . . . . . . . . . . . . . . 9 67 3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 9 68 3.2. Segmented Delivery . . . . . . . . . . . . . . . . . . . 9 69 3.2.1. Idle Time between Segments . . . . . . . . . . . . . 9 70 3.2.2. Head-of-Line Blocking . . . . . . . . . . . . . . . . 10 71 3.3. Unreliable Transport . . . . . . . . . . . . . . . . . . 10 72 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 73 5. Security Considerations . . . . . . . . . . . . . . . . . . . 11 74 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 11 75 7. Informative References . . . . . . . . . . . . . . . . . . . 11 76 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 13 78 1. Introduction 80 As the internet has grown, an increasingly large share of the traffic 81 delivered to end users has become video. Estimates put the total 82 share of internet video traffic at 75% in 2019, expected to grow to 83 82% by 2022. What's more, this estimate projects the gross volume of 84 video traffic will more than double during this time, based on a 85 compound annual growth rate continuing at 34% (from Appendix D of 86 [CVNI]). 88 In many contexts, video traffic can be handled transparently as 89 generic application-level traffic. However, as the volume of video 90 traffic continues to grow, it's becoming increasingly important to 91 consider the effects of network design decisions on application-level 92 performance, with considerations for the impact on video delivery. 94 This document aims to provide a taxonomy of networking issues as they 95 relate to quality of experience in internet video delivery. The 96 focus is on capturing characteristics of video delivery that have 97 surprised network designers or transport experts without specific 98 video expertise, since these highlight key differences between common 99 assumptions in existing networking documents and observations of 100 video delivery issues in practice. 102 Making specific recommendations for mitigating these issues is out of 103 scope, though some existing mitigations are mentioned in passing. 104 The intent is to provide a point of reference for future solution 105 proposals to use in describing how new technologies address or avoid 106 these existing observed problems. 108 1.1. Notes for Contributors and Reviewers 110 Note to RFC Editor: Please remove this section and its subsections 111 before publication. 113 This section is to provide references to make it easier to review the 114 development and discussion on the draft so far. 116 1.1.1. Venues for Contribution and Discussion 118 This document is in the Github repository at: 120 https://github.com/ietf-wg-mops/draft-ietf-mops-streaming-opcons 122 Readers are welcome to open issues and send pull requests for this 123 document. 125 Substantial discussion of this document should take place on the MOPS 126 working group mailing list (mops@ietf.org). 128 * Join: https://www.ietf.org/mailman/listinfo/mops 130 * Search: https://mailarchive.ietf.org/arch/browse/mops/ 132 1.1.2. Template for Contributions 134 Contributions are solicited regarding issues and considerations that 135 have an impact on media streaming operations. 137 Please note that contributions may be merged and substantially 138 edited, and as a reminder, please carefully consider the Note Well 139 before contributing: https://datatracker.ietf.org/submit/note-well/ 141 Contributions can be emailed to mops@ietf.org, submitted as issues to 142 the issue tracker of the repository in Section 1.1.1, or emailed to 143 the document authors at draft-ietf-mops-streaming-opcons@ietf.org. 145 Contributors describing an issue not yet addressed in the draft are 146 requested to provide the following information, where applicable: 148 * a suggested title or name for the issue 150 * a long-term pointer to the best reference describing the issue 152 * a short description of the nature of the issue and its impact on 153 media quality of service, including: 155 - where in the network this issue has root causes 157 - who can detect this issue when it occurs 159 * an overview of the issue's known prevalence in practice. pointers 160 to write-ups of high-profile incidents are a plus. 162 * a list of known mitigation techniques, with (for each known 163 mitigation): 165 - a name for the mitigation technique 167 - a long-term pointer to the best reference describing it 169 - a short description of the technique: 171 o what it does 173 o where in the network it operates 175 o an overview of the tradeoffs involved-how and why it's 176 helpful, what it costs. 178 - supplemental information about the technique's deployment 179 prevalence and status 181 1.1.3. History of Public Discussion 183 Presentations: 185 * IETF 105 BOF: 187 https://www.youtube.com/watch?v=4G3YBVmn9Eo&t=47m21s 189 * IETF 106 meeting: 191 https://www.youtube.com/watch?v=4_k340xT2jM&t=7m23s 193 2. Bandwidth Provisioning 195 2.1. Scaling Requirements for Media Delivery 197 2.1.1. Video Bitrates 199 Video bitrate selection depends on many variables. Different 200 providers give different guidelines, but an equation that 201 approximately matches the bandwidth requirement estimates from 202 several video providers is given in [MSOD]: 204 Kbps = (HEIGHT * WIDTH * FRAME_RATE) / (15 * 1024) 206 Height and width are in pixels, and frame rate is in frames per 207 second. The actual bitrate required for a specific video will also 208 depend on the codec used, fidelity desired and some other 209 characteristics of the video itself, such as the amount and frequency 210 of high-detail motion, which may influence the compressability of the 211 content, but this equation provides a rough estimate. 213 Here are a few common resolutions used for video content, with their 214 typical per-user bandwidth requirements according to this formula: 216 +============+================+===============================+ 217 | Name | Width x Height | Approximate Bitrate for 60fps | 218 +============+================+===============================+ 219 | DVD | 720 x 480 | 1.3 Mbps | 220 +------------+----------------+-------------------------------+ 221 | 720p (1K) | 1280 x 720 | 3.6 Mbps | 222 +------------+----------------+-------------------------------+ 223 | 1080p (2K) | 1920 x 1080 | 8.1 Mbps | 224 +------------+----------------+-------------------------------+ 225 | 2160p (4k) | 3840 x 2160 | 32 Mbps | 226 +------------+----------------+-------------------------------+ 228 Table 1 230 2.1.2. Virtual Reality Bitrates 232 Even the basic virtual reality (360-degree) videos (that allow users 233 to look around freely, referred to as three degrees of freedom - 234 3DoF) require substantially larger bitrates when they are captured 235 and encoded as such videos require multiple fields of view of the 236 scene. The typical multiplication factor is 8 to 10. Yet, due to 237 smart delivery methods such as viewport-based or tiled-based 238 streaming, we do not need to send the whole scene to the user. 239 Instead, the user needs only the portion corresponding to its 240 viewpoint at any given time. 242 In more immersive applications, where basic user movement (3DoF+) or 243 full user movement (6DoF) is allowed, the required bitrate grows even 244 further. In this case, the immersive content is typically referred 245 to as volumetric media. One way to represent the volumetric media is 246 to use point clouds, where streaming a single object may easily 247 require a bitrate of 30 Mbps or higher. Refer to [PCC] for more 248 details. 250 2.2. Path Requirements 252 The bitrate requirements in Section 2.1 are per end-user actively 253 consuming a media feed, so in the worst case, the bitrate demands can 254 be multiplied by the number of simultaneous users to find the 255 bandwidth requirements for a router on the delivery path with that 256 number of users downstream. For example, at a node with 10,000 257 downstream users simultaneously consuming video streams, 258 approximately up to 80 Gbps would be necessary in order for all of 259 them to get 1080p resolution at 60 fps. 261 However, when there is some overlap in the feeds being consumed by 262 end users, it is sometimes possible to reduce the bandwidth 263 provisioning requirements for the network by performing some kind of 264 replication within the network. This can be achieved via object 265 caching with delivery of replicated objects over individual 266 connections, and/or by packet-level replication using multicast. 268 To the extent that replication of popular content can be performed, 269 bandwidth requirements at peering or ingest points can be reduced to 270 as low as a per-feed requirement instead of a per-user requirement. 272 2.3. Caching Systems 274 TBD: pros, cons, tradeoffs of caching designs at different locations 275 within the network? 277 Peak vs. average provisioning, and effects on peering point 278 congestion under peak load? 280 Provisioning issues for caching systems? 282 2.4. Predictable Usage Profiles 284 Historical data shows that users consume more video and videos at 285 higher bitrates than they did in the past on their connected devices. 286 Improvements in the codecs that help with reducing the encoding 287 bitrates with better compression algorithms could not have offset the 288 increase in the demand for the higher quality video (higher 289 resolution, higher frame rate, better color gamut, better dynamic 290 range, etc.). In particular, mobile data usage has shown a large 291 jump over the years due to increased consumption of entertainement as 292 well as conversational video. 294 TBD: insert charts showing historical relative data usage patterns 295 with error bars by time of day in consumer networks? 297 Cross-ref vs. video quality by time of day in practice for some case 298 study? Not sure if there's a good way to capture a generalized 299 insight here, but it seems worth making the point that demand 300 projections can be used to help with e.g. power consumption with 301 routing architectures that provide for modular scalability. 303 2.5. Unpredictable Usage Profiles 305 Although TCP/IP has been used with a number of widely used 306 applications that have symmetric bandwidth requirements (similar 307 bandwidth requirements in each direction between endpoints), many 308 widely-used Internet applications operate in client-server roles, 309 with asymmetric bandwidth requirements. A common example might be an 310 HTTP GET operation, where a client sends a relatively small HTTP GET 311 request for a resource to an HTTP server, and often receives a 312 significantly larger response carrying the requested resource. When 313 HTTP is commonly used to stream movie-length video, the ratio between 314 response size and request size can become quite large. 316 For this reason, operators may pay more attention to downstream 317 bandwidth utilization when planning and managing capacity. In 318 addition, operators have been able to deploy access networks for end 319 users using underlying technologies that are inherently asymetric, 320 favoring downstream bandwidth (e.g. ADSL, cellular technologies, 321 most IEEE 802.11 variants), assuming that users will need less 322 upstream bandwidth than downstream bandwidth. This strategy usually 323 works, except when it does not, because application bandwidth usage 324 patterns have changed. 326 One example of this type of change was when peer-to-peer file sharing 327 applications gained popularity in the early 2000s. To take one well- 328 documented case ([RFC5594]), the Bittorrent application created 329 "swarms" of hosts, uploading and downloading files to each other, 330 rather than communicating with a server. Bittorrent favored peers 331 who uploaded as much as they downloaded, so that new Bittorrent users 332 had an incentive to significantly increase their upstream bandwidth 333 utilization. 335 The combination of the large volume of "torrents" and the peer-to- 336 peer characteristic of swarm transfers meant that end user hosts were 337 suddenly uploading higher volumes of traffic to more destinations 338 than was the case before Bittorrent. This caused at least one large 339 ISP to attempt to "throttle" these transfers, to mitigate the load 340 that these hosts placed on their network. These efforts were met by 341 increased use of encryption in Bittorrent, similar to an arms race, 342 and set off discussions about "Net Neutrality" and calls for 343 regulatory action. 345 Especially as end users increase use of video-based social networking 346 applications, it will be helpful for access network providers to 347 watch for increasing numbers of end users uploading significant 348 amounts of content. 350 2.6. Extremely Unpredictable Usage Profiles 352 The causes of unpredictable usage described in Section 2.5 were more 353 or less the result of human choices, but we were reminded during a 354 post-IETF 107 meeting that humans are not always in control, and 355 forces of nature can cause enormous fluctuations in traffic patterns. 357 In his talk, Sanjay Mishra [Mishra] reported that after the CoViD-19 358 pandemic broke out in early 2020, 360 * Comcast's streaming and web video consumption rose by 38%, with 361 their reported peak traffic up 32% overall between March 1 to 362 March 30 [Comcast], 364 * AT&T reported a 28% jump in core network traffic (single day in 365 April, as compared to pre stay-at-home daily average traffic), 366 with video accounting for nearly half of all mobile network 367 traffic, while social networking and web browsing remained the 368 highest percentage (almost a quarter each) of overall mobility 369 traffic [ATT], and 371 * Verizon reported similar trends with video traffic up 36% over an 372 average day (pre COVID-19) [Verizon]. 374 We note that other operators saw similar spikes during this time 375 period. Craig Labowitz [Labovitz] reported 377 * Weekday peak traffic increases over 45%-50% from pre-lockdown 378 levels, 380 * A 30% increase in upstream traffic over their pre-pandemic levels, 381 and 383 * A steady increase in the overall volume of DDoS traffic, with 384 amounts exceeding the pre-pandemic levels by 40%. (He attributed 385 this increase to the significant rise in gaming-related DDoS 386 attacks ([LabovitzDDoS]), as gaming usage also increased.) 388 3. Adaptive Bitrate 390 3.1. Overview 392 Adaptive BitRate (ABR) is a sort of application-level response 393 strategy in which the receiving media player attempts to detect the 394 available bandwidth of the network path by experiment or by observing 395 the successful application-layer download speed, then chooses a video 396 bitrate (among the limited number of available options) that fits 397 within that bandwidth, typically adjusting as changes in available 398 bandwidth occur in the network or changes in capabilities occur in 399 the player (such as available memory, CPU, display size, etc.). 401 The choice of bitrate occurs within the context of optimizing for 402 some metric monitored by the video player, such as highest achievable 403 video quality, or lowest rate of expected rebuffering events. 405 3.2. Segmented Delivery 407 ABR playback is commonly implemented by video players using HLS 408 [RFC8216] or DASH [DASH] to perform a reliable segmented delivery of 409 video data over HTTP. Different player implementations and receiving 410 devices use different strategies, often proprietary algorithms 411 (called rate adaptation or bitrate selection algorithms), to perform 412 available bandwidth estimation/prediction and the bitrate selection. 413 Most players only use passive observations, i.e., they do not 414 generate probe traffic to measure the available bandwidth. 416 This kind of bandwidth-measurement systems can experience trouble in 417 several ways that can be affected by networking design choices. 419 3.2.1. Idle Time between Segments 421 When the bitrate selection is successfully chosen below the available 422 capacity of the network path, the response to a segment request will 423 typically complete in less absolute time than the duration of the 424 requested segment. The resulting idle time within the connection 425 carrying the segments has a few surprising consequences: 427 * Mobile flow-bandwidth spectrum and timing mapping. 429 * TCP slow-start when restarting after idle requires multiple RTTs 430 to re-establish a throughput at the network's available capacity. 431 On high-RTT paths or with small enough segments, this can produce 432 a falsely low application-visible measurement of the available 433 network capacity. 435 A detailed investigation of this phenomenon is available in 436 [NOSSDAV12]. 438 3.2.2. Head-of-Line Blocking 440 In the event of a lost packet on a TCP connection with SACK support 441 (a common case for segmented delivery in practice), loss of a packet 442 can provide a confusing bandwidth signal to the receiving 443 application. Because of the sliding window in TCP, many packets may 444 be accepted by the receiver without being available to the 445 application until the missing packet arrives. Upon arrival of the 446 one missing packet after retransmit, the receiver will suddenly get 447 access to a lot of data at the same time. 449 To a receiver measuring bytes received per unit time at the 450 application layer, and interpreting it as an estimate of the 451 available network bandwidth, this appears as a high jitter in the 452 goodput measurement. 454 Active Queue Management (AQM) systems such as PIE [RFC8033] or 455 variants of RED [RFC2309] that induce early random loss under 456 congestion can mitigate this by using ECN [RFC3168] where available. 457 ECN provides a congestion signal and induce a similar backoff in 458 flows that use Explicit Congestion Notification-capable transport, 459 but by avoiding loss avoids inducing head-of-line blocking effects in 460 TCP connections. 462 3.3. Unreliable Transport 464 In contrast to segmented delivery, several applications use UDP or 465 unreliable SCTP to deliver RTP or raw TS-formatted video. 467 Under congestion and loss, this approach generally experiences more 468 video artifacts with fewer delay or head-of-line blocking effects. 469 Often one of the key goals is to reduce latency, to better support 470 applications like videoconferencing, or for other live-action video 471 with interactive components, such as some sporting events. 473 Congestion avoidance strategies for this kind of deployment vary 474 widely in practice, ranging from some streams that are entirely 475 unresponsive to using feedback signaling to change encoder settings 476 (as in [RFC5762]), or to use fewer enhancement layers (as in 477 [RFC6190]), to proprietary methods for detecting quality of 478 experience issues and cutting off video. 480 4. IANA Considerations 482 This document requires no actions from IANA. 484 5. Security Considerations 486 This document introduces no new security issues. 488 6. Acknowledgements 490 Thanks to Mark Nottingham, Glenn Deen, Dave Oran, Aaron Falk, Kyle 491 Rose, and Leslie Daigle for their very helpful reviews and comments. 493 7. Informative References 495 [ATT] AT&T, "Tuesday (March 24, 2020) Network Insights", March 496 2020, . 498 [Comcast] CNBC, "Comcast sees network traffic surge amid coronavirus 499 outbreak", March 2020, 500 . 503 [CVNI] Cisco Systems, Inc., "Cisco Visual Networking Index: 504 Forecast and Trends, 2017-2022 White Paper", 27 February 505 2019, . 509 [DASH] "Information technology -- Dynamic adaptive streaming over 510 HTTP (DASH) -- Part 1: Media presentation description and 511 segment formats", ISO/IEC 23009-1:2019, 2019. 513 [Labovitz] Labovitz, C. and Nokia Deepfield, "Network traffic 514 insights in the time of COVID-19: April 9 update", April 515 2020, . 518 [LabovitzDDoS] 519 Takahashi, D. and Venture Beat, "Why the game industry is 520 still vulnerable to DDoS attacks", May 2018, 521 . 525 [Mishra] Mishra, S. and J. Thibeault, "An update on Streaming Video 526 Alliance", 2020, . 531 [MSOD] Akamai Technologies, Inc., "Media Services On Demand: 532 Encoder Best Practices", 2019, . 537 [NOSSDAV12] 538 al., S.A.e., "What Happens When HTTP Adaptive Streaming 539 Players Compete for Bandwidth?", June 2012, 540 . 542 [PCC] al., S.S.e., "Emerging MPEG Standards for Point Cloud 543 Compression", March 2019, 544 . 546 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 547 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 548 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 549 S., Wroclawski, J., and L. Zhang, "Recommendations on 550 Queue Management and Congestion Avoidance in the 551 Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998, 552 . 554 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 555 of Explicit Congestion Notification (ECN) to IP", 556 RFC 3168, DOI 10.17487/RFC3168, September 2001, 557 . 559 [RFC5594] Peterson, J. and A. Cooper, "Report from the IETF Workshop 560 on Peer-to-Peer (P2P) Infrastructure, May 28, 2008", 561 RFC 5594, DOI 10.17487/RFC5594, July 2009, 562 . 564 [RFC5762] Perkins, C., "RTP and the Datagram Congestion Control 565 Protocol (DCCP)", RFC 5762, DOI 10.17487/RFC5762, April 566 2010, . 568 [RFC6190] Wenger, S., Wang, Y.-K., Schierl, T., and A. 569 Eleftheriadis, "RTP Payload Format for Scalable Video 570 Coding", RFC 6190, DOI 10.17487/RFC6190, May 2011, 571 . 573 [RFC8033] Pan, R., Natarajan, P., Baker, F., and G. White, 574 "Proportional Integral Controller Enhanced (PIE): A 575 Lightweight Control Scheme to Address the Bufferbloat 576 Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017, 577 . 579 [RFC8216] Pantos, R., Ed. and W. May, "HTTP Live Streaming", 580 RFC 8216, DOI 10.17487/RFC8216, August 2017, 581 . 583 [Verizon] Rorbuck, M. and Fierce Telecom, "Verizon: U.S. network 584 usage starts to normalize as subscribers settle into new 585 routines", April 2020, 586 . 590 Authors' Addresses 592 Jake Holland 593 Akamai Technologies, Inc. 594 150 Broadway 595 Cambridge, MA 02144, 596 United States of America 598 Email: jakeholland.net@gmail.com 600 Ali Begen 601 Networked Media 602 Turkey 604 Email: ali.begen@networked.media 606 Spencer Dawkins 607 Tencent America LLC 608 United States of America 610 Email: spencerdawkins.ietf@gmail.com