idnits 2.17.1 draft-feamster-livingood-iab-covid19-workshop-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (28 October 2020) is 1276 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Architecture Board F. Bronzino 3 Internet-Draft Universite Savoie Mont Blanc 4 Intended status: Informational E. Culley 5 Expires: 1 May 2021 Comcast 6 N. Feamster 7 S. Liu 8 University of Chicago 9 J. Livingood 10 Comcast 11 P. Schmitt 12 Princeton University 13 28 October 2020 15 IAB COVID-19 Workshop: Interconnection Changes in the United States 16 draft-feamster-livingood-iab-covid19-workshop-00 18 Abstract 20 During the early weeks and months of the COVID-19 pandemic, 21 significant changes to Internet usage occurred as a result of a 22 sudden global shift to people working, studying and quarantining at 23 home. One aspect that this affected was interconnection between 24 networks, which this paper studies. This paper explores some of the 25 effects of these changes on Internet interconnection points, in terms 26 of utilization, traffic ratios, and other performance characteristics 27 such as latency. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at https://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on 1 May 2021. 46 Copyright Notice 48 Copyright (c) 2020 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 53 license-info) in effect on the date of publication of this document. 54 Please review these documents carefully, as they describe your rights 55 and restrictions with respect to this document. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 2. Long-Term Interconnection Data . . . . . . . . . . . . . . . 3 61 3. Comcast's COVID-19-Related Experiences . . . . . . . . . . . 4 62 4. Detailed Statistical Observations . . . . . . . . . . . . . . 6 63 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 6 64 6. Security Considerations . . . . . . . . . . . . . . . . . . . 6 65 7. Normative References . . . . . . . . . . . . . . . . . . . . 6 66 8. Informative References . . . . . . . . . . . . . . . . . . . 6 67 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7 69 1. Introduction 71 During the early weeks and months of the COVID-19 pandemic 72 [WHO-Declaration], significant changes to Internet usage occurred as 73 a result of a sudden global shift to people working, studying and 74 quarantining at home. One aspect that this affected was 75 interconnection between networks, which this paper studies. 77 In 2016, the Interconnection Measurement Project [Tinker-Blog] was 78 launched. The IMP platform initially collected interconnection- 79 related data from seven U.S.-based cable-based Internet Service 80 Providers (ISPs) and an [ArXiv-Paper] shared some of the details and 81 findings. The initial focus of the project was to explore 82 utilization characteristics of interconnect links during a period of 83 time when video traffic was steadily increasing. The project 84 concluded that there was ample aggregate capacity on interconnect 85 links between ISPs and peers (including content providers), and that 86 ISPs continually added capacity to their interconnects to keep pace 87 with the growth in traffic. 89 This IAB paper looks specifically at the long-term interconnection 90 data from one of those ISPs, Comcast. We examine the long-term pre- 91 COVID-19 trend as well as what occurred as COVID-19 impacted the 92 Internet from February 2020 through August 2020. We also include 93 observations from Comcast concerning interconnection changes during 94 this timeframe. 96 We hope that this information will be useful to the IAB workshop and 97 the Internet community more broadly. It may serve as an interesting 98 and useful historical reference in the future. 100 2. Long-Term Interconnection Data 102 The IMP platform collected interconnection data starting in 2016, 103 roughly four years prior to the COVID-19-driven shift in Internet 104 usage. This provides an interesting capability for a before and 105 after view of interconnection. A full explanation of the data can be 106 found in Section 3 of the [ArXiv-Paper]. At a high level, the 107 maindata collected encompasses: 109 * Timestamp (representing a five-minute interval) 111 * Region (representing an aggregated link group) 113 * Anonymized partner network 115 * Access ISP 117 * Total ingress bytes 119 * Total egress bytes 121 * Capacity 123 Utilization is captured based on sampled IPFIX records, with a packet 124 sampling rate of 1/1,000. SNMP polling data yields information about 125 the capacity on each link. The IMP platform does not have direct 126 access to partner network identities---this dataset only includes an 127 anonymous identifier corresponding to that particular partner. 128 However, in collaboration with Comcast, IMP has worked to identify 129 specific partner networks in the dataset for specific peers in the 130 interest of detail COVID-19-related study. 132 Focusing on Comcast within this dataset reveals several trends, in 133 both aggregate capacity and utilization, as well as how utilization 134 and capacity changed during the period of the COVID-19 pandemic. The 135 data also reveals how traffic volumes changed for specific peers 136 during this time period. 138 The IMP data that tracks capacity shows Comcast steadily adding 139 capacity from mid-2018 to present, with a significant increase in the 140 rate of additional capacity from the beginning of 2020, increasing 141 further in the second and third quarters of 2020. Specifically, we 142 analyzed the rate at which capacity was added during these periods, 143 on a month-by-month basis. We found that Comcast was adding 144 aggregate capacity on its interconnects at *nearly twice* the rate as 145 it was being added during 2019. 147 Over a long timeframe, Comcast's daily peak hour interconnection port 148 utilization has remained consistently around 90%. During the period 149 at the end of March and the beginning of April 2020, daily peak 150 utilization briefly increased to about 97% but steadily returned to 151 normal levels in a matter of weeks by the end of April 2020, as 152 Comcast quickly increased the rate at which it added capacity to keep 153 pace with growing traffic demands. 155 We also explored traffic volumes associated with each Comcast peer, 156 comparing the ranks and volumes of each individual peer as measured 157 on September 1, 2020 as compared to January 1, 2020, in both the 158 upstream and downstream directions. Doing so allowed us to 159 understand both the magnitude of changes in traffic volumes and 160 ratios, as well as how specific individual peers deviated from normal 161 baseline behavior, in terms of both upstream and downstream traffic 162 rates and ratios. The data suggests that both the upstream and 163 downstream directions saw some changes, although the deviations in 164 upstream traffic patterns were greater: the Pearson coefficient for 165 downstream traffic was 0.977, whereas for upstream traffic the 166 Pearson correlation coefficient was only 0.935, indicating a more 167 significant shift in upstream traffic ratios during this timeframe. 169 3. Comcast's COVID-19-Related Experiences 171 Comcast observed a wide range of significant changes in Internet 172 usage as residential users remained at home and shifted to working 173 and studying from home. Changes in usage patterns observed in the 174 access network in sum led to changes in the traffic flowing to 175 interconnected networks. As the pandemic developed, there was a wide 176 variety of changes in traffic volumes. At some locations in the 177 network little change was detected while other locations saw a huge 178 growth in the volume of traffic. 180 At the peak of the surge, in March and April 2020, the average amount 181 of growth observed across locations and types was roughly 33%. Voice 182 & video conferencing (conferencing hereafter) jumped as much as 285% 183 and Wi-Fi use increased 36% among our Xfinity Mobile (MVNO) customers 184 [June-Blog] [July-Blog]. As this continued into May 2020 we observed 185 conferencing remained up roughly 210-285%, VPN traffic up 30-40% and 186 gaming downloads up 20-80%, and web-based streaming video consumption 187 up 20-40% [May-Blog]. In this several week period, traffic 188 essentially grew at or more than it had in the prior year, which was 189 significant growth in a short period of time. 191 In the months following the onset of the pandemic Comcast observed: 193 * Overall average downstream peak growth is up 13% (up as much as 194 20% at times). 196 * Overall average upstream peak growth is up 36%. 198 * In the access network, an average of 771 network augments per week 199 were performed, peaking at over 1,800 in a single week and over 200 with over 7 weeks with more than 1,000 per week. For comparison, 201 the average earlier in the year was roughly 350 per week. 203 * In the core network, over 500 augments were made in order to add 204 146 Tbps in capacity. 206 * On a daily basis roughly 700,000 automated speed tests from 207 customer homes were conducted in order to gauge the customer 208 experience during this time. Average speeds to customers (both 209 downstream and upstream) have generally remained at or above 105% 210 of advertised speeds since March 1, 2020 in all regions. National 211 average speeds have remained between 110% - 115% of advertised 212 speeds over the same period. [NF-Paper-1] and [NF-Paper-2] 214 * The share of streaming video as a percentage of total traffic is 215 declined slightly from 67% to 63%. Despite strong growth, 216 conferencing occupies a small share of total traffic and grew from 217 1% share to 4%. But gaming software released have driven 218 significant download spikes since late April 2020. 220 * For interconnection, peering coordinators across operators worked 221 cooperatively and quickly to cut through any red tape and add new 222 capacity as quickly as possible. 224 * In 2019, settlement free interconnection capacity [SFI-Policy] - a 225 subset of overall interconnect types - grew by 15%. Between 226 January and August 2020, driven by COVID-19 changes, there was an 227 overall 37% increase in capacity from that prior 2019 level. And 228 between March and October 2020 one Settlement Free Peer alone 229 increased 115%. 231 * Between March and October 2020 from Comcast observed other notable 232 per-peer traffic increases of 245% and 3,900%. 234 4. Detailed Statistical Observations 236 As briefly mentioned in previous sections, downstream traffic rates 237 from many partners remained stable---the Pearson correlation 238 coefficient for peak download rates between January 1, 2020 and 239 September 2, 2020 is 0.977, indicating that the peak download rates 240 to most peers was similar between these two time periods. On the 241 other hand, certain peers experienced either a significant increase 242 or decrease in peak download rates---often by two or three orders of 243 magnitude. Similarly, other peers experienced a decrease in peak 244 downstream rates by several orders of magnitude. 246 On the other hand, upstream traffic rates were far less stable: In 247 contrast, the Pearson coefficient for upload rates between January 1, 248 2020 and September 2, 2020 was only 0.935, suggesting more a more 249 significant deviation in peak upstream rates. As with peak download 250 rates, some peers experienced significant decreases, as well: in one 251 outlier case, peak rate decreased by almost five orders of magnitude. 252 A small handful of peers saw similar decreases. Yet, a far greater 253 number of peers saw increases in peak upload rates by two to three 254 orders of magnitude. 256 5. IANA Considerations 258 This document includes no request to IANA. 260 6. Security Considerations 262 This document includes no security considerations. 264 7. Normative References 266 8. Informative References 268 [ArXiv-Paper] 269 Feamster, NF., "Revealing Utilization at Internet 270 Interconnection Points", 5 September 2016, 271 . 273 [July-Blog] 274 Nafshi, EN., "COVID-19 Network Report: How A Smart Network 275 Delivered Speed and Stability When it Mattered", 13 July 276 2020, . 279 [June-Blog] 280 Werner, TW., "Cresting the Wave: The Factors that Powered 281 our Network Through the COVID-19 Surge", 15 June 2020, 282 . 285 [May-Blog] Comcast, "COVID-19 Network Update", 20 May 2020, 286 . 289 [NF-Paper-1] 290 Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast Design 291 Audit Report of Comcast's Network Performance Measurement 292 System", April 2020, . 296 [NF-Paper-2] 297 Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast's 298 Report on Comcast's Network Performance Measurement System 299 Results Data", May 2020, . 303 [SFI-Policy] 304 Comcast Cable Communications Management, LLC, "Comcast 305 Settlement-Free Interconnection (SFI) Policy", October 306 2013, . 308 [Tinker-Blog] 309 Feamster, NF., "Interconnection Measurement Project 310 Website", 9 May 2016, . 314 [WHO-Declaration] 315 Adhanom Ghebreyesus, TAG., "WHO Director-General's opening 316 remarks at the media briefing on COVID-19 - 11 March 317 2020", 11 March 2020, 318 . 322 Authors' Addresses 323 Francesco Bronzino 324 Universite Savoie Mont Blanc 325 Annecy-le-Vieux 326 France 328 Email: francesco.bronzino@univ-smb.fr 330 Elizabeth Culley 331 Comcast 332 Mount Laurel, NJ 333 United States of America 335 Email: elizabeth_culley@comcast.com 337 Nick Feamster 338 University of Chicago 339 Chicago, IL 340 United States of America 342 Email: feamster@uchicago.edu 344 Shinan Liu 345 University of Chicago 346 Chicago, IL 347 United States of America 349 Email: shinanliu@uchicago.edu 351 Jason Livingood 352 Comcast 353 Philadelphia, PA 354 United States of America 356 Email: jason_livingood@comcast.com 358 Paul Schmitt 359 Princeton University 360 Princeton, NJ 361 United States of America 363 Email: pschmitt@cs.princeton.edu