idnits 2.17.1 draft-feamster-livingood-iab-covid19-workshop-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (28 October 2020) is 1275 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Architecture Board F. Bronzino 3 Internet-Draft Universite Savoie Mont Blanc 4 Intended status: Informational E. Culley 5 Expires: 1 May 2021 Comcast 6 N. Feamster 7 S. Liu 8 University of Chicago 9 J. Livingood 10 Comcast 11 P. Schmitt 12 Princeton University 13 28 October 2020 15 IAB COVID-19 Workshop: Interconnection Changes in the United States 16 draft-feamster-livingood-iab-covid19-workshop-01 18 Abstract 20 During the early weeks and months of the COVID-19 pandemic, 21 significant changes to Internet usage occurred as a result of a 22 sudden global shift to people working, studying and quarantining at 23 home. One aspect that this affected was interconnection between 24 networks, which this paper studies. This paper explores some of the 25 effects of these changes on Internet interconnection points, in terms 26 of utilization, traffic ratios, and other performance characteristics 27 such as latency. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at https://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on 1 May 2021. 46 Copyright Notice 48 Copyright (c) 2020 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 53 license-info) in effect on the date of publication of this document. 54 Please review these documents carefully, as they describe your rights 55 and restrictions with respect to this document. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 2. Long-Term Interconnection Data . . . . . . . . . . . . . . . 3 61 3. Comcast's COVID-19-Related Experiences . . . . . . . . . . . 4 62 4. Detailed Statistical Observations . . . . . . . . . . . . . . 6 63 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 6 64 6. Security Considerations . . . . . . . . . . . . . . . . . . . 6 65 7. Normative References . . . . . . . . . . . . . . . . . . . . 6 66 8. Informative References . . . . . . . . . . . . . . . . . . . 6 67 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7 69 1. Introduction 71 During the early weeks and months of the COVID-19 pandemic 72 [WHO-Declaration], significant changes to Internet usage occurred as 73 a result of a sudden global shift to people working, studying and 74 quarantining at home. One aspect that this affected was 75 interconnection between networks, which this paper studies. 77 In 2016, the Interconnection Measurement Project [Tinker-Blog] was 78 launched. The IMP platform initially collected interconnection- 79 related data from seven U.S.-based cable-based Internet Service 80 Providers (ISPs) and an [ArXiv-Paper] shared some of the details and 81 findings. The initial focus of the project was to explore 82 utilization characteristics of interconnect links during a period of 83 time when video traffic was steadily increasing. The project 84 concluded that there was ample aggregate capacity on interconnect 85 links between ISPs and peers (including content providers), and that 86 ISPs continually added capacity to their interconnects to keep pace 87 with the growth in traffic. 89 This IAB paper looks specifically at the long-term interconnection 90 data from one of those ISPs, Comcast. We examine the long-term pre- 91 COVID-19 trend as well as what occurred as COVID-19 impacted the 92 Internet from February 2020 through August 2020. We also include 93 observations from Comcast concerning interconnection changes during 94 this timeframe. 96 We hope that this information will be useful to the IAB workshop and 97 the Internet community more broadly. It may serve as an interesting 98 and useful historical reference in the future. 100 2. Long-Term Interconnection Data 102 The IMP platform collected interconnection data starting in 2016, 103 roughly four years prior to the COVID-19-driven shift in Internet 104 usage. This provides an interesting capability for a before and 105 after view of interconnection. A full explanation of the data can be 106 found in Section 3 of the [ArXiv-Paper]. At a high level, the 107 maindata collected encompasses: 109 * Timestamp (representing a five-minute interval) 111 * Region (representing an aggregated link group) 113 * Anonymized partner network 115 * Access ISP 117 * Total ingress bytes 119 * Total egress bytes 121 * Capacity 123 Utilization is captured based on sampled IPFIX records, with a packet 124 sampling rate of 1/1,000. SNMP polling data yields information about 125 the capacity on each link. The IMP platform does not have direct 126 access to partner network identities---this dataset only includes an 127 anonymous identifier corresponding to that particular partner. 128 However, in collaboration with Comcast, IMP has worked to identify 129 specific partner networks in the dataset for specific peers in the 130 interest of detail COVID-19-related study. 132 Focusing on Comcast within this dataset reveals several trends, in 133 both aggregate capacity and utilization, as well as how utilization 134 and capacity changed during the period of the COVID-19 pandemic. The 135 data also reveals how traffic volumes changed for specific peers 136 during this time period. 138 The IMP data that tracks capacity shows Comcast steadily adding 139 capacity from mid-2018 to present, with a significant increase in the 140 rate of additional capacity from the beginning of 2020, increasing 141 further in the second and third quarters of 2020. Specifically, we 142 analyzed the rate at which capacity was added during these periods, 143 on a month-by-month basis. We found that Comcast was adding 144 aggregate capacity on its interconnects at *nearly twice* the rate as 145 it was being added during 2019. 147 Over a long timeframe, Comcast's daily peak hour interconnection port 148 utilization on its busiest links has remained consistently around 149 90%. For our analysis, we consider the links that represent the 99th 150 percentile in terms of utilization. During the period at the end of 151 March and the beginning of April 2020, daily peak utilization briefly 152 increased to about 97% but steadily returned to normal levels in a 153 matter of weeks by the end of April 2020, as Comcast quickly 154 increased the rate at which it added capacity to keep pace with 155 growing traffic demands. 157 We also explored traffic volumes associated with each Comcast peer, 158 comparing the ranks and volumes of each individual peer as measured 159 on September 1, 2020 as compared to January 1, 2020, in both the 160 upstream and downstream directions. Doing so allowed us to 161 understand both the magnitude of changes in traffic volumes and 162 ratios, as well as how specific individual peers deviated from normal 163 baseline behavior, in terms of both upstream and downstream traffic 164 rates and ratios. The data suggests that both the upstream and 165 downstream directions saw some changes, although the deviations in 166 upstream traffic patterns were greater: the Pearson coefficient for 167 downstream traffic was 0.977, whereas for upstream traffic the 168 Pearson correlation coefficient was only 0.935, indicating a more 169 significant shift in upstream traffic ratios during this timeframe. 171 3. Comcast's COVID-19-Related Experiences 173 Comcast observed a wide range of significant changes in Internet 174 usage as residential users remained at home and shifted to working 175 and studying from home. Changes in usage patterns observed in the 176 access network in sum led to changes in the traffic flowing to 177 interconnected networks. As the pandemic developed, there was a wide 178 variety of changes in traffic volumes. At some locations in the 179 network little change was detected while other locations saw a huge 180 growth in the volume of traffic. 182 At the peak of the surge, in March and April 2020, the average amount 183 of growth observed across locations and types was roughly 33%. Voice 184 & video conferencing (conferencing hereafter) jumped as much as 285% 185 and Wi-Fi use increased 36% among our Xfinity Mobile (MVNO) customers 187 [June-Blog] [July-Blog]. As this continued into May 2020 we observed 188 conferencing remained up roughly 210-285%, VPN traffic up 30-40% and 189 gaming downloads up 20-80%, and web-based streaming video consumption 190 up 20-40% [May-Blog]. In this several week period, traffic 191 essentially grew at or more than it had in the prior year, which was 192 significant growth in a short period of time. 194 In the months following the onset of the pandemic Comcast observed: 196 * Overall average downstream peak growth is up 13% (up as much as 197 20% at times). 199 * Overall average upstream peak growth is up 36%. 201 * In the access network, an average of 771 network augments per week 202 were performed, peaking at over 1,800 in a single week and over 203 with over 7 weeks with more than 1,000 per week. For comparison, 204 the average earlier in the year was roughly 350 per week. 206 * In the core network, over 500 augments were made in order to add 207 146 Tbps in capacity. 209 * On a daily basis roughly 700,000 automated speed tests from 210 customer homes were conducted in order to gauge the customer 211 experience during this time. Average speeds to customers (both 212 downstream and upstream) have generally remained at or above 105% 213 of advertised speeds since March 1, 2020 in all regions. National 214 average speeds have remained between 110% - 115% of advertised 215 speeds over the same period. [NF-Paper-1] and [NF-Paper-2] 217 * The share of streaming video as a percentage of total traffic is 218 declined slightly from 67% to 63%. Despite strong growth, 219 conferencing occupies a small share of total traffic and grew from 220 1% share to 4%. But gaming software released have driven 221 significant download spikes since late April 2020. 223 * For interconnection, peering coordinators across operators worked 224 cooperatively and quickly to cut through any red tape and add new 225 capacity as quickly as possible. 227 * In 2019, settlement free interconnection capacity [SFI-Policy] - a 228 subset of overall interconnect types - grew by 15%. Between 229 January and August 2020, driven by COVID-19 changes, there was an 230 overall 37% increase in capacity from that prior 2019 level. And 231 between March and October 2020 one Settlement Free Peer alone 232 increased 115%. 234 * Between March and October 2020 from Comcast observed other notable 235 per-peer traffic increases of 245% and 3,900%. 237 4. Detailed Statistical Observations 239 As briefly mentioned in previous sections, downstream traffic rates 240 from many partners remained stable---the Pearson correlation 241 coefficient for peak download rates between January 1, 2020 and 242 September 2, 2020 is 0.977, indicating that the peak download rates 243 to most peers was similar between these two time periods. On the 244 other hand, certain peers experienced either a significant increase 245 or decrease in peak download rates---often by two or three orders of 246 magnitude. Similarly, other peers experienced a decrease in peak 247 downstream rates by several orders of magnitude. 249 On the other hand, upstream traffic rates were far less stable: In 250 contrast, the Pearson coefficient for upload rates between January 1, 251 2020 and September 2, 2020 was only 0.935, suggesting more a more 252 significant deviation in peak upstream rates. As with peak download 253 rates, some peers experienced significant decreases, as well: in one 254 outlier case, peak rate decreased by almost five orders of magnitude. 255 A small handful of peers saw similar decreases. Yet, a far greater 256 number of peers saw increases in peak upload rates by two to three 257 orders of magnitude. 259 5. IANA Considerations 261 This document includes no request to IANA. 263 6. Security Considerations 265 This document includes no security considerations. 267 7. Normative References 269 8. Informative References 271 [ArXiv-Paper] 272 Feamster, NF., "Revealing Utilization at Internet 273 Interconnection Points", 5 September 2016, 274 . 276 [July-Blog] 277 Nafshi, EN., "COVID-19 Network Report: How A Smart Network 278 Delivered Speed and Stability When it Mattered", 13 July 279 2020, . 282 [June-Blog] 283 Werner, TW., "Cresting the Wave: The Factors that Powered 284 our Network Through the COVID-19 Surge", 15 June 2020, 285 . 288 [May-Blog] Comcast, "COVID-19 Network Update", 20 May 2020, 289 . 292 [NF-Paper-1] 293 Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast Design 294 Audit Report of Comcast's Network Performance Measurement 295 System", April 2020, . 299 [NF-Paper-2] 300 Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast's 301 Report on Comcast's Network Performance Measurement System 302 Results Data", May 2020, . 306 [SFI-Policy] 307 Comcast Cable Communications Management, LLC, "Comcast 308 Settlement-Free Interconnection (SFI) Policy", October 309 2013, . 311 [Tinker-Blog] 312 Feamster, NF., "Interconnection Measurement Project 313 Website", 9 May 2016, . 317 [WHO-Declaration] 318 Adhanom Ghebreyesus, TAG., "WHO Director-General's opening 319 remarks at the media briefing on COVID-19 - 11 March 320 2020", 11 March 2020, 321 . 325 Authors' Addresses 326 Francesco Bronzino 327 Universite Savoie Mont Blanc 328 Annecy-le-Vieux 329 France 331 Email: francesco.bronzino@univ-smb.fr 333 Elizabeth Culley 334 Comcast 335 Mount Laurel, NJ 336 United States of America 338 Email: elizabeth_culley@comcast.com 340 Nick Feamster 341 University of Chicago 342 Chicago, IL 343 United States of America 345 Email: feamster@uchicago.edu 347 Shinan Liu 348 University of Chicago 349 Chicago, IL 350 United States of America 352 Email: shinanliu@uchicago.edu 354 Jason Livingood 355 Comcast 356 Philadelphia, PA 357 United States of America 359 Email: jason_livingood@comcast.com 361 Paul Schmitt 362 Princeton University 363 Princeton, NJ 364 United States of America 366 Email: pschmitt@cs.princeton.edu