idnits 2.17.1 draft-livingood-woundy-congestion-mgmt-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (September 1, 2010) is 4980 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force C. Bastian 3 Internet-Draft T. Klieber 4 Intended status: Informational J. Livingood 5 Expires: March 5, 2011 J. Mills 6 R. Woundy 7 Comcast 8 September 1, 2010 10 Comcast's Protocol-Agnostic Congestion Management System 11 draft-livingood-woundy-congestion-mgmt-09 13 Abstract 15 This document describes the congestion management system of Comcast 16 Cable, a large cable broadband Internet Service Provider (ISP) in the 17 U.S. Comcast completed deployment of this congestion management 18 system on December 31, 2008. 20 Status of this Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at http://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on March 5, 2011. 37 Copyright Notice 39 Copyright (c) 2010 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (http://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Applicability to Other Types of Networks . . . . . . . . . . . 3 56 3. Key Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 57 4. Historical Overview . . . . . . . . . . . . . . . . . . . . . 7 58 5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 59 6. Relationship Between Managing Congestion and Adding 60 Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 61 7. Implementation and Configuration . . . . . . . . . . . . . . . 10 62 7.1. Thresholds For Determining When a CMTS Port Is in a 63 Near Congestion State . . . . . . . . . . . . . . . . . . 14 64 7.2. Thresholds For Determining When a User Is in an 65 Extended High Consumption State and for Release from 66 that Classification . . . . . . . . . . . . . . . . . . . 15 67 7.3. Effect of BE Quality of Service on Users' Broadband 68 Experience . . . . . . . . . . . . . . . . . . . . . . . . 19 69 7.4. Equipment/Software Used and Location . . . . . . . . . . . 21 70 8. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 23 71 9. Exceptional Network Utilization Considerations . . . . . . . . 23 72 10. Limitations of This Congestion Management System . . . . . . . 24 73 11. Low Extra Delay Background Transport and Other 74 Possibilities . . . . . . . . . . . . . . . . . . . . . . . . 24 75 12. Security Considerations . . . . . . . . . . . . . . . . . . . 24 76 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 25 77 14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 25 78 15. Change Log . . . . . . . . . . . . . . . . . . . . . . . . . . 25 79 16. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 26 80 17. Notes for RFC Editor . . . . . . . . . . . . . . . . . . . . . 26 81 18. Informative References . . . . . . . . . . . . . . . . . . . . 27 82 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 29 84 1. Introduction 86 Comcast Cable is a large broadband Internet Service Provider (ISP), 87 based in the U.S., serving the majority of its customers via cable 88 modem technology. During the late part of 2008, and completing on 89 December 31, 2008, Comcast deployed a new congestion management 90 system across its entire network. This new system was developed in 91 response to dissatisfaction in the Internet community as well as 92 complaints to the U.S. Federal Communications Commission (FCC) 93 regarding Comcast's old system, which targeted specific peer-to-peer 94 (P2P) applications. This new congestion management system is 95 protocol-agnostic, meaning that it does not examine or impact 96 specific user applications or network protocols, which is perceived 97 as a more fair system for managing network resources at limited times 98 when congestion may occur. 100 It is important for readers to note that congestion can occur in any 101 IP network, and, when it does, packets can be delayed or dropped. As 102 Bob Briscoe has pointed out on an IETF mailing list, some amount of 103 packet loss can be normal and/or tolerable, noting "But a single TCP 104 flow with a round trip time (RTT) of 80ms can attain 50 Mbps with a 105 loss fraction of 0.0013% (1 in ~74,000 packets) so there's no need to 106 try to achieve loss figures much lower than this. And indeed, if 107 flows aren't bottlenecked elsewhere, TCP will drive the system until 108 it gets such loss levels. If, instead, a customer is downloading 109 five separate 10 Mbps TCP flows still with an 80ms RTT, TCP will 110 drive losses up to 1 in ~3,000, or 0.03%, and any lower loss rates 111 won't be able to improve performance." As a result, applications and 112 protocols have been designed to deal with the reality that congestion 113 can occur in any IP network, the mechanics of which we explain in 114 detail later in this document. 116 The purpose of this document is to describe how this example of a 117 large scale congestion management system functions. This is 118 partially in response to questions from other ISPs as well as 119 solution developers, who are interested in learning from and/or 120 deploying similar systems in other networks. In addition, it is 121 hoped that such a document may help inform new work in the IETF, in 122 the hope that better systems and protocols may be possible in the 123 future. Lastly, the authors wish to transparently and openly 124 document this system, so that there could be no doubt about how the 125 system functioned. 127 2. Applicability to Other Types of Networks 129 Several document reviewers and other IETF participants have pointed 130 out that, though we refer to functional elements that are specific to 131 a DOCSIS-based network implementation, this type of congestion 132 management system could be generally applied to nearly any type of 133 network. Thus, it is important for readers to take note of this and 134 take into consideration that this sort of protocol-agnostic 135 congestion management system could certainly fit in a wide variety of 136 network types and implementations. 138 3. Key Terminology 140 This section defines the key terms used in this document. Some terms 141 below refer to elements of the Comcast network. As a result, it may 142 be helpful to refer to Figure 1 when reviewing some of these terms. 144 3.1. Cable Modem 146 A device located at the customer premise used to access the Comcast 147 High Speed Internet (HSI) network. In some cases, the cable modem is 148 owned by the customer, and in other cases it is owned by the cable 149 operator. This device has an interface (i.e., someplace to plug in a 150 cable) for connecting the coaxial cable provided by the cable company 151 to the modem, as well as one or more interfaces for connecting the 152 modem to a customer's PC or home gateway device (e.g., home gateway, 153 router, firewall, access point, etc.). In some cases, the cable 154 modem function, i.e., the ability to access the Internet, is 155 integrated into a home gateway device or Embedded Multimedia Terminal 156 Adapter (eMTA). Once connected, the cable modem links the customer 157 to the HSI network and ultimately the broader Internet. 159 3.2. Cable Modem Termination System (CMTS) 161 A piece of hardware located in a cable operator's local network 162 (generally in a "headend", Section 3.10) that acts as the gateway to 163 the Internet for cable modems in a particular geographic area. A 164 simple way to think of the CMTS is as a router with interfaces on one 165 side leading to the Internet and interfaces on the other connecting 166 to Optical Nodes and then customers, in a so-called "last mile" 167 network. 169 3.3. Cable Modem Termination System (CMTS) Port 171 Also referred to simply as a "port". A port is a physical interface 172 on a device used to connect cables in order to connect with other 173 devices for transferring information/data. An example of a physical 174 port is a CMTS port. A CMTS has both upstream and downstream network 175 interfaces to serve the local access network, which are referred to 176 as upstream or downstream ports. A port generally serves a 177 neighborhood of hundreds of homes. Over time, CMTS ports tend to 178 serve fewer and fewer homes, as the network is segmented for capacity 179 growth purposes. Prior to DOCSIS version 3, a single CMTS physical 180 port was used for either transmitting or receiving data downstream or 181 upstream to a given neighborhood. With DOCSIS version 3, and the 182 channel bonding feature, multiple CMTS physical ports can be combined 183 to create a virtual port. A CMTS is also briefly defined in Section 184 2.6 of [RFC3083]. 186 3.4. Channel Bonding 188 A technique for combining multiple downstream and/or upstream 189 channels to increase customers' download and/or upload speeds, 190 respectively. Multiple channels from the Hybrid Fiber Coax (HFC, 191 Section 3.11) network can be bonded into a single virtual port 192 (called a bonded group), which acts as a large single channel or port 193 to provide increased speeds for customers. Channel bonding is a 194 feature of Data Over Cable Service Interface Specification (DOCSIS) 195 version 3, as described in the [CableLabs DOCSIS 3.0 MAC and Upper 196 Layer Protocols Interface Specification]. 198 3.5. Coaxial Cable (Coax) 200 A type of cable used by a cable operator to connect customer premise 201 equipment (CPE) -- such as TVs, cable modems (including eMTAs), and 202 Set Top Boxes -- to the HFC network. This cable may be used both 203 within the home as well as in segments of the last mile network 204 running to a home or customer premise location. There are many 205 grades of coaxial cable that are used for different purposes. 206 Different types of coaxial cable are used for different purposes on 207 the network. 209 3.6. Comcast High Speed Internet (HSI) 211 A service/product offered by Comcast for delivering Internet service 212 over a broadband connection. 214 3.7. Customer Premise Equipment (CPE) 216 Any device that resides at the customer's residence, connected to the 217 Comcast network, whether controlled by Comcast or not. 219 3.8. Data Over Cable Service Interface Specification (DOCSIS) 221 A reference standard developed by CableLabs that specifies how 222 components on cable networks need to be built to enable HSI service 223 over an HFC network, as noted in the [CableLabs DOCSIS 3.0 Cable 224 Modem to Customer Premise Equipment Interface Specification], 225 [CableLabs DOCSIS 3.0 Physical Layer Specification], [CableLabs 226 DOCSIS 3.0 MAC and Upper Layer Protocols Interface Specification], 227 [CableLabs DOCSIS 3.0 Security Specification], and [CableLabs DOCSIS 228 3.0 Operations Support System Interface Specification]. These 229 standards define the specifications for the cable modem and the CMTS 230 such that any DOCSIS certified cable modem will work on any DOCSIS- 231 certified CMTS, independent of the selected vendor. The 232 interoperability of cable modems and CMTSs allows customers to 233 purchase a DOCSIS-certified modem from a retail outlet and use it on 234 their cable-networked home. All DOCSIS-related standards are 235 available to the public at the CableLabs website, at 236 http://www.cablelabs.com. 238 3.9. Downstream 240 Description of the direction in which a signal travels, in this case 241 from the network to a user. Downstream traffic occurs when users are 242 downloading something from the Internet, such as watching a web-based 243 video, reading web pages, or downloading software updates. 245 3.10. Headend 247 A cable facility responsible for receiving TV signals for 248 distribution over the HFC network to the end customers. This 249 facility typically also houses one or more CMTSs. This is sometimes 250 also called a "hub". 252 3.11. Hybrid Fiber Coax (HFC) 254 A network architecture used primarily by cable companies, comprised 255 of fiber optic and coaxial cables that currently deliver Voice, 256 Video, and Internet services to customers, as defined in Section 1.2 257 of the [CableLabs DOCSIS 3.0 MAC and Upper Layer Protocols Interface 258 Specification]. 260 3.12. Internet Protocol Detail Record (IPDR) 262 Standardized technology for monitoring and/or recording subscribers' 263 upstream and downstream Internet usage data based on their cable 264 modem. The data is collected from the CMTS and sent to a server for 265 further processing. Additional information is available at 266 http://www.ipdr.org, as well as [IPDR Standard] and [CableLabs DOCSIS 267 IPDR Support]. 269 3.13. Optical Node 271 A component of the HFC network generally located in customers' local 272 neighborhoods which is used to convert the optical signals sent over 273 fiber-optic cables to electrical signals that can be sent over 274 coaxial cable to customers' cable modems, or vice versa. A fiber 275 optic cable connects the Optical Node, through distribution hubs, to 276 the CMTS and coaxial cable connects the Optical Node to customers' 277 cable modems. 279 3.14. Provisioned Bandwidth 281 The peak speed associated with a tier of service purchased by a 282 customer. For example, a customer with a 105 Mbps downstream and 10 283 Mbps upstream speed tier would be said to be provisioned with 105 284 Mbps of downstream bandwidth and 10 Mbps of upstream bandwidth. This 285 is often referred to as 105/10 service in industry parlance. 287 The Provisioned Bandwidth is the speed that a customer's modem is 288 configured (and the network is engineered) to deliver on a regular 289 basis (which is not the same as a "Committed Information Rate" or a 290 guaranteed rate). Internet speeds are generally a best effort 291 service that are dependent on a number of variables, many of which 292 are outside the control of an Internet Service Provide (ISP). In 293 general, speeds do not typically exceed a customer's provisioned 294 speed. Comcast, however, invented a technology called "PowerBoost" 295 [PowerBoost Specification] that, for example, enables users to 296 experience brief boosts above their provisioned speeds while they 297 transfer large files over the Internet, by utilizing excess capacity 298 which may be available in the network at that time. 300 3.15. Quality of Service (QoS) 302 A set of techniques to manage network resources to ensure a level of 303 performance to specific data flows, as described in [RFC1633] and 304 [RFC2475]. One method for providing QoS to a network is by 305 differentiating the type of traffic by class or flow and assigning 306 priorities to each type. When the network becomes congested, the 307 data packets that are marked as having higher priority will have 308 higher likelihood of being serviced. 310 3.16. Upstream 312 Description of the direction in which a signal travels, in this case 313 from the user to the network. Upstream traffic occurs when users are 314 uploading something to the network, such as sending email, sending 315 files to another computer, or uploading photos to a digital photo 316 website. 318 4. Historical Overview 320 Comcast began the engineering project to develop a new congestion 321 management system in March 2008, the same month that Comcast hosted 322 the 71st meeting of the IETF in Philadelphia, PA, USA. On May 28, 323 2008, Comcast participated in an IETF Peer-to-Peer Infrastructure 324 Workshop [RFC5594], hosted by the Massachusetts Institute of 325 Technology (MIT) in Cambridge, MA, USA. 327 In order to participate in this workshop, interested attendees were 328 asked to submit a paper to a technical review team, which Comcast did 329 on May 9, 2008, in the [Comcast P2Pi Position Paper]. Comcast 330 subsequently attended and participated in this valuable workshop. 331 During the workshop, Comcast outlined the high-level design for a new 332 congestion management system [Comcast IETF P2P Infrastructure 333 Workshop Presentation] and solicited comments and other feedback from 334 attendees and other members of the Internet community (presentations 335 were also posted to the IETF's P2Pi mailing list). The congestion 336 management system outlined in that May 2008 workshop was later tested 337 in trial markets and is in essence what was then deployed by Comcast 338 later in 2008. 340 Following an August 2008 FCC document [FCC Memorandum and Opinion - 341 August 2008] regarding how Comcast managed congestion on its High- 342 Speed Internet ("HSI") network, Comcast disclosed to the FCC [FCC 343 Network Management Response - September 2008] and the public 344 additional technical details of the congestion management system that 345 it intended to and did implement by the end of 2008 [FCC Congestion 346 Management Deployment Completion Letter - January 2009], including 347 the thresholds involved in this new system. While the description of 348 how this system is deployed in the Comcast network is necessarily 349 specific to the various technologies and designs specific to that 350 network, a similar system could be deployed on virtually any large 351 scale ISP network or other IP network. 353 5. Summary 355 Comcast's HSI network has elements which are shared across many 356 subscribers. This means that Comcast's HSI customers share upstream 357 and downstream bandwidth with their neighbors. Although the 358 available bandwidth is substantial, so, too, is the demand. Thus, 359 when a relatively small number of customers in a neighborhood place 360 disproportionate demands on network resources, this can cause 361 congestion that degrades their neighbors' Internet experience. The 362 goal of Comcast's new congestion management system is to enable all 363 users of our network resources to access a "fair share" of that 364 bandwidth, in the interest of ensuring a high-quality online 365 experience for all of Comcast's HSI customers. 367 Importantly, the new approach is protocol-agnostic; that is, it does 368 not manage congestion by focusing on the use of the specific 369 protocols that place a disproportionate burden on network resources, 370 or any other protocols. Rather, the new approach focuses on managing 371 the traffic of those individuals who are using the most bandwidth at 372 times when network congestion threatens to degrade subscribers' 373 broadband experience and who are contributing disproportionately to 374 such congestion at those points in time. 376 Specific details about these practices, including relevant threshold 377 information, the type of equipment used, and other particulars, are 378 discussed at some length later in this document. At the outset, 379 however, we present a very high-level, simplified overview of how 380 these practices work. Despite all the detail provided further below, 381 the fundamentals of this approach can be summarized succinctly: 383 1. Software installed in the Comcast network continuously examines 384 aggregate traffic usage data for individual segments of Comcast's 385 HSI network. If overall upstream or downstream usage on a 386 particular segment of Comcast's HSI network reaches a pre- 387 determined level, the software moves on to step two. 389 2. At step two, the software examines bandwidth usage data for 390 subscribers in the affected network segment to determine which 391 subscribers are using a disproportionate share of the bandwidth. 392 If the software determines that a particular subscriber or 393 subscribers have been the source of high volumes of network 394 traffic during a recent period of minutes, traffic originating 395 from that subscriber or those subscribers temporarily will be 396 assigned a lower priority status. 398 3. During the time that a subscriber's traffic is assigned the lower 399 priority status, their packets will not be delayed or dropped so 400 long as the network segment is not actually congested. If, 401 however, the network segment becomes congested, their packets 402 could be intermittently delayed or dropped. 404 4. The subscriber's traffic returns to normal priority status once 405 his or her bandwidth usage drops below a set threshold over a 406 particular time interval. 408 Comcast undertook considerable effort, over the course of many 409 months, to formulate our plans for this congestion management 410 approach, adjusting them, and subjecting them to real-world trials. 411 Market trials were conducted in Chambersburg, PA; Warrenton, VA; Lake 412 City, FL; East Orange, FL; and Colorado Springs, CO, between June and 413 September, 2008. This enabled us to validate the utility of the 414 general approach and collect substantial trial data to test multiple 415 variations and alternative formulations. 417 6. Relationship Between Managing Congestion and Adding Capacity 419 Many people have questioned whether congestion should ever exist at 420 all, if an ISP was adding sufficient capacity. There is certainly a 421 relationship between capacity and congestion. But there are two 422 types of congestion that generally present themselves in a network. 423 The first general type of congestion is regularly occurring and is 424 the result of gradually increasing traffic levels up to a point where 425 typical usage peaks cause congestion on a regular basis, and the 426 other which is unpredictable. Comcast, like many ISPs, has a set 427 capacity management process by which capacity additions are 428 automatically triggered based on certain usage trends, which is 429 geared towards bringing additional capacity to the network prior to 430 the onset of regularly occurring congestion. As such, capacity is 431 added when needed and before it presents noticeable effects. This 432 process is in place since capacity additions are not instantaneous 433 and in many cases require significant physical work. 435 The second general type of congestion is unpredictable congestion, 436 which can occur for a wide range of reasons. One example may be due 437 to current events, where users may be all rushing to access specific 438 content at the exact same time, and where the systems serving that 439 content may not be able to keep up with demand. Another example may 440 be due to a localized disaster, where some network paths have been 441 destroyed or otherwise impaired, and where many users are attempting 442 to communicate with one another at traffic levels significantly above 443 normal. 445 Thus in both cases, even with continuous upgrades and constant 446 investment in additional capacity, the fact remains that network 447 capacity is not unlimited. A congestion management system, absent 448 superior protocol-based solutions which do not currently exist, can 449 therefore help manage the effects of congestion on users, improving 450 their Internet experience. 452 7. Implementation and Configuration 454 It is important to note that the implementation details below and the 455 overall design of the system are matched to traffic patterns that 456 exist on the Internet today and which the authors believe will exist 457 in the near future. While the authors desired to make the system 458 highly adaptable and a good long-term network investment, significant 459 changes in such traffic patterns may necessitate a change in the 460 configuration of the system or, in extreme cases, a different type of 461 system altogether. 463 To understand exactly how these new congestion management practices 464 work, it is helpful to have a general understanding of how Comcast's 465 HSI network is designed. Comcast's HSI network is what is commonly 466 referred to as a hybrid fiber-coax network, with coaxial cable 467 connecting each subscriber's cable modem to an Optical Node, and 468 fiber optic cables connecting the Optical Node, through distribution 469 hubs, to the Cable Modem Termination System (CMTS), which is also 470 known as a "data node". The CMTSes are then connected to higher- 471 level routers, which in turn are connected to Comcast's Internet 472 backbone facilities. Today, Comcast has over 3,200 CMTSes deployed 473 throughout our network, serving over 15 million HSI subscribers. 475 Each CMTS has multiple "ports" that handle traffic coming into and 476 leaving the CMTS. In particular, each cable modem deployed on the 477 Comcast HSI network is connected to the CMTS through the ports on the 478 CMTS. These ports can be either "downstream" ports or "upstream" 479 ports, depending on whether they send information to cable modems 480 (downstream) or receive information from cable modems (upstream) 481 attached to the port. (Note that the term "port" as used here 482 generally contemplates single channels on a CMTS, but these 483 statements will apply to virtual channels, also known as "bonded 484 groups", in a DOCSIS 3.0 environment.) Even without channel bonding, 485 multiple channels are usually configured to come out of each physical 486 port. Said another way, there is generally a mapping of multiple 487 channels to each physical port. 489 Currently, on average, approximately 275 cable modems share the same 490 downstream port and about 100 cable modems share the same upstream 491 port, however this is constantly changing (both numbers generally 492 become smaller over time, based on current DOCSIS technology). Both 493 types of ports can experience congestion that could degrade the 494 broadband experience of our subscribers and, unlike with the previous 495 congestion management practices, both upstream and downstream traffic 496 are subject to management in this new congestion management system. 498 Based upon the design of the network and traffic patterns observed, 499 the most likely place for congestion to occur is on these CMTS ports. 500 As a result, the congestion management system measures the traffic 501 conditions of CMTS ports, and applies any policy actions to traffic 502 on those ports (rather than some other, more distant segment of the 503 network). 505 To implement Comcast's new protocol-agnostic congestion management 506 practices, Comcast purchased new hardware and software that was 507 deployed near the Regional Network Routers ("RNRs") that are further 508 upstream in Comcast's network. This new hardware consists of 509 Internet Protocol Detail Record ("IPDR") servers, Congestion 510 Management servers, and PacketCable Multimedia ("PCMM") servers. 511 Further details about each of these pieces of equipment can be found 512 below, in Section 7.4. It is important to note here, however, that 513 even though the physical location of these servers is at the RNR, the 514 servers communicate with -- and manage individually -- multiple ports 515 on multiple CMTSes to effectuate the practices described in this 516 document. That is to say, bandwidth usage on one CMTS port will have 517 no effect on whether the congestion management practices described 518 herein are applied to a subscriber on a different CMTS port. 520 Figure 1 provides a simplified graphical depiction of the network 521 architecture just described: 523 Figure 1: Simplified Network Diagram Showing High-Level Comcast 524 Network and Servers Relevant to Congestion Management 526 ------------------------- 527 / \ 528 | Comcast Internet Backbone | 529 \ ---- 530 +------------+ --------------------/ \ 531 | Congestion | / \ 532 | Management |<+++GigE++++ +---->| Internet | 533 | Server | + | | Backbone | 534 +------------+ + | \ Router / 535 + Fiber \ / 536 +------------+ + | ----- 537 | QoS | + | 538 | Server |<+++GigE++++ \/ 539 | | + ----- 540 +------------+ + / \ 541 + / \ 542 +------------+ + | Regional | 543 | Statistics | +++++++>| Network | 544 | Collection |<+++GigE++++ | Router | 545 | Server | \ / 546 +------------+ +---Fiber------>\ /<------Fiber----+ 547 | ----- | 548 \/ \/ 549 ----- ----- 550 / \ / \ 551 / Local \ / Local \ 552 | Market | | Market | 553 \ Router / \ Router / 554 +--------->\ /<------------+ \ / 555 | ----- | ------ 556 | /\ | /\ 557 Fiber | Fiber | 558 | Fiber | Fiber 559 | | | | 560 \/ \/ \/ \/ 561 /------\ /------\ /------\ /------\ 562 | CMTS | | CMTS | | CMTS | | CMTS | 563 \------/ \------/ \------/ \------/ 564 /\ /\ /\ /\ 565 | | | | 566 Fiber Fiber Fiber Fiber 567 | | | | 568 \/ \/ \/ \/ 569 +---------+ +---------+ +---------+ +---------+ 570 | Optical | | Optical | | Optical | | Optical | 571 | Node | | Node | | Node | | Node | 572 +---------+ +---------+ +---------+ +---------+ 573 /\ /\ /\ /\ /\ /\ 574 || || ||______ || _____|| || 575 Coax Coax |__Coax| Coax |Coax__| Coax 576 || || || || || || 577 \/ \/ \/ \/ \/ \/ 578 +=======+ +=======+ +=======+ +=======+ +=======+ +=======+ 579 = Cable = = Cable = = Cable = = Cable = = Cable = = Cable = 580 = Modem = = Modem = = Modem = = Modem = = Modem = = Modem = 581 +=======+ +=======+ +=======+ +=======+ +=======+ +=======+ 583 ================================================================ 584 + Note: This diagram is a simplification of the actual network + 585 + and servers, which in actuality includes significant + 586 + redundancy and other details too complex to represent here. + 587 ================================================================ 589 Figure 1 591 Each Comcast HSI subscriber's cable modem has a "bootfile", which is 592 essentially a configuration file that contains certain pieces of 593 information about the subscriber's service to ensure that the service 594 functions properly. (Note: No personal information is included in 595 the bootfile; it only includes information about the service that the 596 subscriber has purchased.) For example, the bootfile contains 597 information about the maximum speed (what we refer to in this 598 document as the "provisioned bandwidth") that a particular modem can 599 achieve based on the tier (personal/residential, commercial, etc.) 600 the customer has purchased. Bootfiles are generally reset from time 601 to time to account for changes in the network and other updates, and 602 this is usually done through a command sent from the network and 603 without the subscriber noticing. In preparation for the transition 604 to this new congestion management system, Comcast sent new bootfiles 605 to our HSI customers' cable modems that created two Quality of 606 Service ("QoS") levels for Internet traffic going to and from the 607 cable modem: (1) "Priority Best Effort" traffic ("PBE"); and (2) 608 "Best Effort" traffic ("BE"). As with previous changes to cable 609 modem bootfiles, the replacement of the old bootfile with the new 610 bootfile requires no active participation by Comcast customers. 612 Thereafter, all traffic going to or coming from cable modems on the 613 Comcast HSI network is designated as either PBE or BE. PBE is the 614 default status for all Internet traffic coming from or going to a 615 particular cable modem. Traffic is designated BE for a particular 616 cable modem only when both of two conditions are met: 618 o First, the usage level of a particular upstream or downstream port 619 of a CMTS, as measured over a particular period of time, must be 620 nearing the point where congestion could degrade users' 621 experience. We refer to this as the "Near Congestion State" and, 622 based on the technical trials we have conducted (further validated 623 in our full deployment), we have established a threshold, 624 described in more detail below, for when a particular CMTS port 625 enters that state. 627 o Second, a particular subscriber must be making an extended, high 628 contribution to the bandwidth usage on the particular port, 629 relative to the service tier they purchased, as measured over a 630 particular period of time. We refer to this as the "Extended High 631 Consumption State" and, based on the technical trials we have 632 conducted (further validated in our full deployment), we have 633 established a threshold, described in more detail below, for when 634 a particular user enters that state. 636 When, and only when, both conditions are met, a user's upstream or 637 downstream traffic (depending on which type of port is in the Near 638 Congestion State) is designated as BE. Then, to the extent that 639 actual congestion occurs, any delay resulting from the congestion 640 will affect BE traffic before it affects PBE traffic. 642 We now explain the foregoing in greater detail in the following 643 sections. 645 7.1. Thresholds For Determining When a CMTS Port Is in a Near 646 Congestion State 648 For a CMTS port to enter the Near Congestion State, traffic flowing 649 to or from that CMTS port must exceed a specified level (the "Port 650 Utilization Threshold") for a specific period of time (the "Port 651 Utilization Duration"). The Port Utilization Threshold on a CMTS 652 port is measured as a percentage of the total aggregate upstream or 653 downstream bandwidth for the particular port during the relevant 654 timeframe. The Port Utilization Duration on the CMTS is measured in 655 minutes. 657 Values for each of the thresholds which are used as part of this 658 congestion management technique have been tentatively established 659 after an extensive process of lab tests, simulations, technical 660 trials, vendor evaluations, customer feedback, and a third-party 661 consulting analysis. In the same way that specific anti-spam or 662 other network management practices are adjusted to address new issues 663 that arise, it is a near certainty that these values will change over 664 time, as Comcast gathers more data and performs additional analysis 665 resulting from wide-scale use of the new technique. Moreover, as 666 with any large network or software system, software bugs and/or 667 unexpected errors may arise, requiring software patches or other 668 corrective actions. As always, Comcast's decisions on these matters 669 are driven by the marketplace imperative that we deliver the best 670 possible experience to our HSI subscribers. 672 Given our experience as described above, we determined that a 673 starting point for the upstream Port Utilization Threshold should be 674 70 percent and the downstream Port Utilization Threshold should be 80 675 percent. For the Port Utilization Duration, we determined that the 676 starting point should be approximately 15 minutes (although some 677 technical limitations in some newer CMTSes deployed on Comcast's 678 network may make this time period vary slightly). Thus, over any 15- 679 minute period, if an average of more than 70 percent of a port's 680 upstream bandwidth capacity or more than 80 percent of a port's 681 downstream bandwidth capacity is utilized, that port is determined to 682 be in a Near Congestion State. 684 Based on the trials conducted and operational experience to date, a 685 typical CMTS port on our HSI network is in a Near Congestion State 686 only for relatively small portions of the day, if at all, though 687 there is no way to forecast what will be the busiest time on a 688 particular port on a particular day. Moreover, the trial data and 689 operational experience indicate that, even when a particular port is 690 in a Near Congestion State, the instances where the network actually 691 becomes congested during the Port Utilization Duration are few, and 692 managed users whose packets may be intermittently delayed or dropped 693 during those congested periods perceive little, if any, effect, as 694 discussed below. 696 7.2. Thresholds For Determining When a User Is in an Extended High 697 Consumption State and for Release from that Classification 699 Once a particular CMTS port is in a Near Congestion State, the 700 software examines whether any cable modems are consuming bandwidth 701 disproportionately. (Note: Although each cable modem is typically 702 assigned to a particular household, the software does not and cannot 703 actually identify individual users, the number of users sharing a 704 cable modem, or analyze particular users' traffic.) For purposes of 705 this document, we use "cable modem", "user", and "subscriber" 706 interchangeably to mean a subscriber account or user account and not 707 an individual person. For a user to enter an Extended High 708 Consumption State, he or she must consume greater than a certain 709 percentage of his or her provisioned upstream or downstream bandwidth 710 (the "User Consumption Threshold") for a specific length of time (the 711 "User Consumption Duration"). The User Consumption Threshold is 712 measured as a user's consumption of a particular percentage of his or 713 her total provisioned upstream or downstream bandwidth. That 714 bandwidth is the maximum speed that a particular modem can achieve 715 based on the tier (personal/residential, commercial, etc.) the 716 customer has purchased. For example, if a user buys a service with 717 speeds of 50 Mbps downstream and 10 Mbps upstream, then his or her 718 provisioned downstream speed is 50 Mbps and provisioned upstream 719 speed is 10 Mbps. It is also important to note that because the User 720 Consumption Threshold is a percentage of provisioned bandwidth for a 721 particular user account, and not a static value, users of higher 722 speed tiers have correspondingly higher User Consumption Thresholds. 723 Lastly, the User Consumption Duration is measured in minutes. 725 Following lab tests, simulations, technical trials, customer 726 feedback, vendor evaluations, and an independent third-party 727 consulting analysis, we have determined that the appropriate starting 728 point for the User Consumption Threshold is 70 percent of a 729 subscriber's provisioned upstream or downstream bandwidth, and that 730 the appropriate starting point for the User Consumption Duration is 731 15 minutes (this has been further validated in our full deployment). 732 That is, when a subscriber uses an average of 70 percent or more of 733 his or her provisioned upstream or downstream bandwidth over a 734 particular 15-minute period, that user is then in an Extended High 735 Consumption State. Therefore, this is a consumption-based threshold 736 and not a peak-speed-based threshold. Thus, the Extended High 737 Consumption State is not tied to whether a user has bursted once or 738 more above this 70% threshold for a brief moment. Instead, it is 739 consumption-based, meaning that a certain bitrate must be exceeded 740 over at least the entire User Consumption Duration. 742 The User Consumption Thresholds have been set sufficiently high that 743 using the HSI connection for VoIP, gaming, web surfing, or most 744 streaming video cannot alone cause subscribers to our standard-level 745 HSI service to exceed the User Consumption Threshold. For example, 746 while one of Comcast's common HSI service tiers has a provisioned 747 downstream bandwidth of 22 Mbps today, streaming video (even some HD 748 video) from Hulu uses less than 2.5 Mbps, a Vonage or Skype VoIP call 749 uses less than 131 Kbps, and streaming music uses less than 128 kbps 750 (in this example, 70 percent of 22 Mbps is 15.4 Mbps). As noted 751 above, these values are subject to change as necessary in the same 752 way that specific anti-spam or other network management practices are 753 adjusted to address new issues that arise, or should unexpected 754 software bugs or other problems arise. 756 Based on data collected from the trial markets where the new 757 congestion management practices were tested (further validated in our 758 full deployment), on average less than one-third of one percent of 759 subscribers have had their traffic priority status changed to the BE 760 state on any given day. For example, in Colorado Springs, CO, the 761 largest test market, on any given day in August 2008, an average of 762 22 users out of 6,016 total subscribers in the trial had their 763 traffic priority status changed to BE at some point during the day. 765 A user's traffic is released from a BE state when the user's 766 bandwidth consumption drops below 50 percent of his or her 767 provisioned upstream or downstream bandwidth for a period of 768 approximately 15 minutes. These release criteria are intended to 769 minimize (and hopefully prevent) user QoS oscillation, i.e., a 770 situation in which a particular user could cycle repeatedly between 771 BE and PBE. Thus, without this lower release criteria, we were 772 concerned that certain users would oscillate between BE and PBE 773 states for an extended period, without clear benefit to the system 774 and other users, and placing an unnecessary signaling burden on the 775 system. NetForecast, Inc., an independent consultant retained to 776 provide analysis and recommendations regarding Comcast's trials and 777 related congestion management work, suggested this approach, which 778 has worked well in our trials, lab testing, and subsequent national 779 deployment. 781 Simply put, there are four steps to determining whether the traffic 782 associated with a particular cable modem is designated as PBE or BE: 784 1. Determine if the CMTS port is in a Near Congestion State. 786 2. If yes, determine whether any users are in an Extended High 787 Consumption State. 789 3. If yes, change those users' traffic to BE from PBE. If the 790 answer at either step one or step two is no, no action is taken. 792 4. If a user's traffic has been designated BE, check user 793 consumption at next interval. If user consumption has declined 794 below predetermined threshold, reassign the user's traffic as 795 PBE. If not, recheck at next interval. 797 In cases where a CMTS regularly enters a Near Congestion State, and 798 where congestion subsequently does occur, but where no users match 799 the criteria to be classified in an Extended High Consumption State, 800 this may indicate the congestion observed is regularly occurring, 801 rather than unpredictable congestion. As such, this may be an 802 additional data point in favor of considering whether and when to add 803 capacity. 805 Figure 2 graphically depicts how this congestion management process 806 works, using an example of a situation where upstream port 807 utilization may be reaching a Near Congestion State (the same 808 diagram, with different values in the appropriate places, could be 809 used to depict the management process for downstream ports, as well): 811 Figure 2: Upstream Congestion Management Decision Flowchart 813 /\ 814 +------------+ / \ +---------+ +---------+ 815 | Start | / \ | | / / 816 | Congestion | / \ | | / / 817 | Management +-->+ Question +--YES-->| Result |--THEN-->/ ACTION / 818 | Process | \ #1 / | #1 | / #1 / 819 | | \ / | | / / 820 +------------+ \ / +---------+ +---------+ 821 \/ | 822 | THEN 823 NO | 824 | \/ 825 \/ /\ 826 +---------+ / \ 827 | | / \ 828 | | / \ 829 | Result |<-------------NO------------+ Question + 830 | #2 | \ #2 / 831 | | \ / 832 +---------+ \ / 833 \/ 834 | 835 YES 836 | 837 /\ \/ 838 +---------+ / \ +---------+ 839 | | / \ | | 840 | | / \ THEN, AT | | 841 | Result |<--YES--+ Question + <---NEXT ANALYSIS------+ Result | 842 | #4 | \ #3 / POINT /\ | #3 | 843 | | \ / | | | 844 +---------+ \ / | +---------+ 845 \/ | 846 | | 847 +------------NO-------------+ 848 KEY TO FIGURES ABOVE: 850 Question #1: Is the CMTS Upstream Port Utilization at an average 851 of OVER 70% for OVER 15 minutes? 852 Result #1: CMTS marked in a Near Congestion State, indicating 853 congestion *may* occur soon. 854 Action #1: Search most recent analysis timeframe (approx. 15 mins.) 855 of IPDR usage data. 856 Question #2: Are any users consuming an average of OVER 70% of 857 provisioned upstream bandwidth for OVER 15 minutes? 858 Result #2: No action taken. 859 Result #3: Change user's upstream traffic from Priority Best Effort 860 (PBE) to Best Effort (BE). 861 Question #3: Is the user in Best Effort (BE) consuming an average 862 of LESS THAN 50% of provisioned upstream bandwidth 863 over a period of 15 minutes? 864 Result #4: Change user's upstream traffic back to Priority Best 865 Effort (PBE) from Best Effort (BE). 867 Figure 2 869 7.3. Effect of BE Quality of Service on Users' Broadband Experience 871 When a CMTS port is in a Near Congestion State and a cable modem 872 connected to that port is in an Extended High Consumption State, that 873 cable modem's traffic is designated as BE. Depending upon the level 874 of utilization on the CMTS port, this designation may or may not 875 result in the user's traffic being delayed or, in extreme cases, 876 dropped before PBE traffic is dropped. This is because of the way 877 that the CMTS handles traffic. Specifically, CMTS ports have what is 878 commonly called a "scheduler" that puts all the packets coming from 879 or going to cable modems on that particular port in a queue and then 880 handles them in turn. A certain number of packets can be processed 881 by the scheduler in any given moment; for each time slot, PBE traffic 882 is given priority access to the available capacity, and BE traffic is 883 processed on a space-available basis. 885 A rough analogy would be to busses that empty and fill up at 886 incredibly fast speeds. As empty busses arrive at the figurative 887 "bus stop" -- every two milliseconds in this case -- they fill up 888 with as many packets as are waiting for "seats" on the bus, to the 889 limits of the bus' capacity. During non-congested periods, the bus 890 will usually have several empty seats, but, during congested periods, 891 the bus will fill up and packets will have to wait for the next bus. 892 It is in the congested periods that BE packets will be affected. If 893 there is no congestion, packets from a user in a BE state should have 894 little trouble getting on the bus when they arrive at the bus stop. 895 If, on the other hand, there is congestion in a particular instance, 896 the bus may become filled by packets in a PBE state before any BE 897 packets can get on. In that situation, the BE packets would have to 898 wait for the next bus that is not filled by PBE packets. In reality, 899 this all takes place in two-millisecond increments, so even if the 900 packets miss 50 "busses", the delay will only be about one-tenth of a 901 second. 903 During times of actual network congestion, when packets from BE 904 traffic might be intermittently delayed, there is a variety of 905 effects that could be experienced by a user whose traffic is delayed, 906 depending upon what applications he or she is using. Typically, a 907 user whose traffic is in a BE state during actual congestion may find 908 that a webpage loads sluggishly, a peer-to-peer upload takes somewhat 909 longer to complete, or a VoIP call sounds choppy. Of course, the 910 same thing could happen to the customers on a port that is congested 911 in the absence of any congestion management; the difference here is 912 that the effects of any such delays are shifted toward those who have 913 been placing the greatest burden on the network, instead of being 914 distributed randomly among the users of that port without regard to 915 their consumption levels. As a matter of fact, our studies concluded 916 that the experience of the PBE subscribers improves when this 917 congestion management system is enabled. This conclusion is based on 918 network measurements, such as latency. 920 NetForecast explored the potential risk of a worst-case scenario for 921 users whose traffic is in a BE state: the possibility of "bandwidth 922 starvation" in the theoretical case where 100 percent of the CMTS 923 bandwidth is taken up by PBE traffic for an extended period of time. 924 In theory, such a condition could mean that a given user whose 925 traffic is designated BE would be unable to effectuate an upload or 926 download (as noted above, both are managed separately) for some 927 period of time. However, when these management techniques were 928 tested, first in company testbeds and then in our real-world trials 929 conducted in the five markets (further validated in our full 930 deployment), such a theoretical condition did not occur. In 931 addition, our experience with the system as fully deployed in our 932 production network demonstrates that these management practices have 933 very modest real-world impacts. In addition, Comcast did not receive 934 a single customer complaint in any of the trial markets that could be 935 traced to this congestion management system, despite having broadly 936 publicized these trials. In our subsequent national deployment into 937 our production network, we still have yet to find a specific 938 complaint that can be traced back to the effect of this congestion 939 management system. 941 Comcast continues to monitor how user traffic is affected by these 942 new congestion management techniques and will make the adjustments 943 necessary to ensure that all Comcast HSI customers have a high- 944 quality Internet experience. 946 7.4. Equipment/Software Used and Location 948 The above-mentioned functions are carried out using three different 949 types of application servers, supplied by three different vendors. 950 As mentioned above, these servers are installed near Comcast's 951 regional network routers. The exact locations of these servers is 952 not particularly relevant to this document, as this does not change 953 the fact that they manage individual CMTS ports. 955 The first application server is an IPDR server, which collects 956 relevant cable modem volume usage information from the CMTS, such as 957 how many aggregate upstream or downstream bytes a subscriber uses 958 over a particular period of time. IPDR has been adopted as a 959 standard by many industry organizations and initiatives, such as 960 CableLabs, ATIS, ITU, and 3GPP, among others. The IPDR software 961 deployed was developed by Active Broadband Networks, and is noted as 962 the Statistics Collection Server in Figure 3. 964 The second application server is the Congestion Management server, 965 which uses Simple Network Management Protocol ("SNMP") [RFC3410] to 966 measure CMTS port utilization and detect when a port is in a Near 967 Congestion State. When this happens, the Congestion Management 968 server then queries the relevant IPDR data for a list of cable modems 969 meeting the criteria set forth above for being in an Extended High 970 Consumption State. The Congestion Management Server software 971 deployed was developed by Sandvine. 973 If one or more users meet the criteria to be managed, then the 974 Congestion Management server notifies a third application server, the 975 PCMM application server, as to which users have been in an Extended 976 High Consumption State and whose traffic should be treated as BE. 977 The PCMM servers are responsible for signaling a given CMTS to set 978 the traffic for specific cable modems with a BE QoS, and for tracking 979 and managing the state of such CMTS actions. If no users meet the 980 criteria to be managed, no users will have their traffic managed. 981 The PCMM software deployed was developed by Camiant, and is noted as 982 the QoS Server in Figure 3. 984 Figure 3 graphically depicts the high-level management flows among 985 the congestion management components on Comcast's network, as 986 described above: 988 Figure 3: Simplified Diagram Showing High-Level Management Flows 989 Relevant to the System 991 +---------------+ +---------------+ 992 | Congestion | Instruct QoS Server | QoS | 993 | Management |******to Change QoS for****>| Server | 994 | Server | a Device | | 995 +----+---+------+ +-------+-------+ 996 /\ /\ * 997 | | Relay Selected * 998 X +---Statistics: Bytes---+ QoS Action: 999 | Up/Down by Device | Change from PBE 1000 X +-------+-------+ to BE, or from 1001 | | Statistics | BE to PBE 1002 X | Collection | * 1003 Periodic SNMP | Server | * 1004 Requests to +---------------+ * 1005 Check CMTS Port /\ * 1006 Utilization | * 1007 Levels Statistics Sent * 1008 | Periodically From CMTS * 1009 X | * 1010 | +-----------+-----------+ * 1011 +-X-X-X-X-X-X->| CMTS in Headend |<******** 1012 +-----------------------+ 1013 H /\ /\ H 1014 H Internet Traffic H 1015 H to/from User H 1016 H \/ \/ H 1017 /+---------------------+\ 1018 / | User's +---------+ |\ 1019 / | Home | Cable | | \ 1020 | | Modem | | 1021 ============ | +---------+ | 1022 = Notes: = +----------------------+ 1023 = ====================================================== 1024 = 1-Statistics Collection Servers use IP Detail Records (IPDR). = 1025 = 2-QoS Servers use PacketCable Multimedia (PCMM) = 1026 = to set QoS gates on the CMTS. = 1027 = 3-This figure is a simplification of the actual network and = 1028 = servers, which included redundancies and other complexities = 1029 = not necessary to depict the functional design. = 1030 ================================================================= 1032 Figure 3 1034 8. Conclusion 1036 Comcast started design and development of this new protocol-agnostic 1037 congestion management system in March 2008. Comcast shared the 1038 design with the IETF and others in the Internet community, as well as 1039 with an independent consultant, incorporating feedback we received 1040 into the final design. Following lab testing, the system was tested 1041 in Comcast's production network in trial markets between June and 1042 September 2008. Comcast's production network transition to this new 1043 protocol-agnostic congestion management system began in October 2008 1044 and was completed on December 31, 2008. 1046 As described herein, the new approach does not manage congestion by 1047 focusing on managing the use of specific protocols. Nor does this 1048 approach use TCP "reset packets" [RFC3360]. Rather, the system acts 1049 such that (1) during periods when a CMTS port is in a Near Congestion 1050 State, (2) it identifies the subscribers on that port who have 1051 consumed a disproportionate amount of bandwidth over the preceding 15 1052 minutes, (3) it lowers the priority status of those subscribers' 1053 traffic to BE status until those subscribers meet the release 1054 criteria, and (4) during periods of actual congestion, handles PBE 1055 traffic before BE traffic. Comcast's trials and subsequent national 1056 deployment indicate that this new congestion management system 1057 ensures a quality online experience for all of Comcast's HSI 1058 customers. 1060 9. Exceptional Network Utilization Considerations 1062 This system was developed to cope with somewhat "normal" occurrences 1063 of congestion that could occur on virtually any IP network. It 1064 should also be noted, however, that such a system could also prove 1065 particularly useful in the case of "exceptional network utilization" 1066 events which existing network usage models do not or cannot 1067 accurately predict. Some network operators refer to these 1068 exceptional events as 'surges' in utilization, similar to sudden 1069 surges in demand in electrical power grids, with which many people 1070 may be familiar. 1072 For example, in the case of a severe global pandemic, it may be 1073 expected that large swaths of the population may need to work 1074 remotely, via their Internet connection. In such a case, a largely 1075 unprecedented level of utilization may occur. In such cases, it may 1076 be helpful to have a flexible congestion management system that could 1077 adapt to this situation and help allocate network resources while 1078 additional capacity is being brought online or while a temporary 1079 condition persists. 1081 10. Limitations of This Congestion Management System 1083 The main limitations of the system include: 1085 o The system is not an end-to-end congestion management system, nor 1086 does it enable one. 1088 o The system does not signal the presence of congestion to user 1089 applications or to all devices on the network path. 1091 o The system does not explicitly enable additional user and/or 1092 application responses to congestion. 1094 o The system does not enable DDoS mitigation or other capabilities. 1096 11. Low Extra Delay Background Transport and Other Possibilities 1098 There are several new IETF working group efforts which are focused on 1099 the question of congestion and its effects, avoiding congestion, 1100 managing congestion, and communicating congestion information. This 1101 includes the Congestion Exposure (CONEX) working group, the 1102 Application Layer Transport Optimization (ALTO) working group, and 1103 the Low Extra Delay Background Transport (LEDBAT) working group. 1104 Should one or more of these working groups be successful in producing 1105 useful work, it is possible that the design or configuration of the 1106 system documented here may need to change. For example, this 1107 congestion management system does not currently have a way take into 1108 accounts differing classes of data transfer, such as that which 1109 LEDBAT may specify, which may better yield to other traffic than 1110 existing transport protocols. In addition, CONEX may specify methods 1111 for this or other systems to signal congestion state or expected 1112 congestion to other parts of the network, and/or hosts on either end 1113 of a particular network flow. Furthermore, it is conceivable that 1114 the result of current or future IETF work could obviate the need for 1115 such a congestion management system entirely. 1117 12. Security Considerations 1119 It is important that an ISP secure access to the Congestion 1120 Management Servers and the QoS Servers, as well as QoS signaling to 1121 the CMTSes, so that unauthorized users and/or hosts cannot make 1122 unauthorized changes to QoS settings in the network. 1124 It is also important to secure access to the Statistics Collection 1125 Server since this contains IPDR-based byte transfer data which is 1126 considered private by end users on an individual basis. In addition 1127 this data is considered ISP-proprietary traffic data on an aggregate 1128 basis. Access to the Statistics Collection Server should also be 1129 secured so that false usage statistics cannot be fed into the system. 1130 It is important to note that IPDR data contain a count of bytes sent 1131 and bytes received, by cable modem MAC address, over a given interval 1132 of time. This data does not contain things such as the source and/or 1133 destination Internet address of that data, nor does it contain the 1134 protocols used, ports used, etc. 1136 13. IANA Considerations 1138 There are no IANA considerations in this document. 1140 NOTE TO RFC EDITOR: PLEASE REMOVE THIS NULL SECTION PRIOR TO 1141 PUBLICATION. 1143 14. Acknowledgements 1145 The authors wish to acknowledge the hard work of the many people who 1146 helped to develop and/or review this document, as well as the people 1147 who helped deploy the system in such a short period of time. 1149 The authors also wish to acknowledge the following individuals for 1150 performing a detailed review of this document and/or providing 1151 comments and feedback that helped to improve and evolve this 1152 document: 1154 - Kris Bransom 1156 - Bob Briscoe 1158 - Lars Eggert 1160 - Ari Keranen 1162 - Tero Kivinen 1164 - Matt Mathis 1166 - Stanislav Shalunov 1168 15. Change Log 1170 NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION. 1172 o -09 - Based on an IESG review comment from Ari Keranen, via Jari 1173 Arkko, resolved some repetition in Section 7. 1175 o -08 - expanded definition of CPE in 3.7 to address an IESG comment 1176 from David Harrington 1178 o -07 - fixed nits identified by Pete McCann in the gen-art review 1180 o -06 - fixed nits identified by Lars Eggert and addressed concerns 1181 raised by Tero Kivinen in the security review 1183 o -05 - minor edits from Jason's final proof-reading of the document 1185 o -04 - various changes based on the recommendations of several IETF 1186 participants, especially Bob Briscoe (thanks, Bob!), made ASCII 1187 art shorter vertically (closing an open item), updated Mbps again 1188 (closing an open item), retained IPDR reference (closing an open 1189 item), added section on applicability to other networks (as 1190 suggested by Matt as well as Bob), completed security section, 1191 updated intro, closed ALL open issues 1193 o -03 - changed FCC doc reference to a more stable ref point, 1194 closing an open item. updated Mbps numbers to reflect current 1195 Internet speeds, closing an open item. also, closed out all 1196 editorial notes, closing an open item. additional tweaks following 1197 homework assignments send to co-authors. 1199 o -02 - lots of minor tweaks based on homework assignments that 1200 Jason handed out to co-authors, and closed out several open items 1202 o -01 - closed item on the open issue list - updated references 1204 o -00 - first version published 1206 16. Open Issues 1208 NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION. 1210 1. All open issues have been closed! 1212 17. Notes for RFC Editor 1214 NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION. 1216 ALSO: Please try to adjust page breaks to keep various ASCII art on 1217 one page, as some may break across two pages now. 1219 18. Informative References 1221 [CableLabs DOCSIS 3.0 Cable Modem to Customer Premise Equipment 1222 Interface Specification] 1223 CableLabs, "Data-Over-Cable Service Interface 1224 Specifications - DOCSIS 3.0 - Cable Modem to Customer 1225 Premise Equipment Interface Specification", DOCSIS 3.0 CM- 1226 SP-CMCIv3-I01-080320, March 2008, . 1230 [CableLabs DOCSIS 3.0 MAC and Upper Layer Protocols Interface 1231 Specification] 1232 CableLabs, "Data-Over-Cable Service Interface 1233 Specifications - DOCSIS 3.0 - MAC and Upper Layer 1234 Protocols Interface Specification", DOCSIS 3.0 CM-SP- 1235 MULPIv3.0-I11-091002, October 2009, . 1239 [CableLabs DOCSIS 3.0 Operations Support System Interface 1240 Specification] 1241 CableLabs, "Data-Over-Cable Service Interface 1242 Specifications - DOCSIS 3.0 - Operations Support System 1243 Interface Specification", DOCSIS 3.0 CM-SP-OSSIv3.0-I10- 1244 091002, October 2009, . 1247 [CableLabs DOCSIS 3.0 Physical Layer Specification] 1248 CableLabs, "Data-Over-Cable Service Interface 1249 Specifications - DOCSIS 3.0 - Physical Layer 1250 Specification", DOCSIS 3.0 CM-SP-PHYv3.0-I08-090121, 1251 January 2009, . 1254 [CableLabs DOCSIS 3.0 Security Specification] 1255 CableLabs, "Data-Over-Cable Service Interface 1256 Specifications - DOCSIS 3.0 - Security Specification", 1257 DOCSIS 3.0 CM-SP-SECv3.0-I11-091002, March 2008, . 1261 [CableLabs DOCSIS IPDR Support] 1262 Yassini, R., "Data-Over-Cable Service Interface 1263 Specifications - DOCSIS 2.0 - Operations Support System 1264 Interface Specification", DOCSIS 2.0 CM-SP-OSSIv2.0-C01- 1265 081104, November 2008, . 1268 [Comcast IETF P2P Infrastructure Workshop Presentation] 1269 Livingood, J. and R. Woundy, "Comcast's IETF P2P 1270 Infrastructure Workshop Presentation on May 28, 2008", FCC 1271 Filings Comcast Network Management Proceedings, May 2008, 1272 . 1275 [Comcast P2Pi Position Paper] 1276 Livingood, J. and R. Woundy, "Comcast's IETF P2P 1277 Infrastructure Workshop Position Paper", FCC 1278 Filings Comcast Network Management Proceedings, May 2008, 1279 . 1283 [FCC Congestion Management Deployment Completion Letter - January 1284 2009] 1285 Zachem, K., "Letter to the FCC Advising of Successful 1286 Deployment of Comcast's New Congestion Management System", 1287 FCC Filings Comcast Network Management Proceedings, 1288 January 2009, . 1291 [FCC Memorandum and Opinion - August 2008] 1292 Martin, K., Copps, M., Adelstein, J., Tate, D., and R. 1293 McDowell, "FCC Memorandum and Opinion Regarding Reasonable 1294 Network Management", File No. EB-08-IH-1518 WC Docket No. 1295 07-52, August 2008, . 1298 [FCC Network Management Response - September 2008] 1299 Zachem, K., "Letter to the FCC Regarding Comcast's Network 1300 Management Practices", FCC Filings Comcast Network 1301 Management Proceedings, September 2008, . 1304 [IPDR Standard] 1305 Cotton, S., Cockrell, B., Walls, P., and T. Givoly, 1306 "Network Data Management - Usage (NDM-U) For IP-Based 1307 Services Service Specification - Cable Labs DOCSIS 2.0 1308 SAMIS", IPDR Service Specifications NDM-U, November 2004, 1309 . 1312 [PowerBoost Specification] 1313 Comcast Cable Communications Management LLC, "Comcast 1314 PowerBoost Specification", Website Comcast.com, June 2010, 1315 . 1319 [RFC1633] Braden, B., Clark, D., and S. Shenker, "Integrated 1320 Services in the Internet Architecture: an Overview", 1321 RFC 1633, June 1994. 1323 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1324 and W. Weiss, "An Architecture for Differentiated 1325 Services", RFC 2475, December 1998. 1327 [RFC3083] Woundy, R., "Baseline Privacy Interface Management 1328 Information Base for DOCSIS Compliant Cable Modems and 1329 Cable Modem Termination Systems", RFC 3083, March 2001. 1331 [RFC3360] Floyd, S., "Inappropriate TCP Resets Considered Harmful", 1332 BCP 60, RFC 3360, August 2002. 1334 [RFC3410] Case, J., Mundy, R., Partain, D., and B. Stewart, 1335 "Introduction and Applicability Statements for Internet- 1336 Standard Management Framework", RFC 3410, December 2002. 1338 [RFC5594] Peterson, J. and A. Cooper, "Report from the IETF Workshop 1339 on Peer-to-Peer (P2P) Infrastructure, May 28, 2008", 1340 RFC 5594, July 2009. 1342 Authors' Addresses 1344 Chris Bastian 1345 Comcast Cable Communications 1346 One Comcast Center 1347 1701 John F. Kennedy Boulevard 1348 Philadelphia, PA 19103 1349 US 1351 Email: chris_bastian@cable.comcast.com 1352 URI: http://www.comcast.com 1353 Tom Klieber 1354 Comcast Cable Communications 1355 One Comcast Center 1356 1800 Bishops Gate Drive 1357 Mount Laurel, NJ 08054 1358 US 1360 Email: tom_klieber@cable.comcast.com 1361 URI: http://www.comcast.com 1363 Jason Livingood 1364 Comcast Cable Communications 1365 One Comcast Center 1366 1701 John F. Kennedy Boulevard 1367 Philadelphia, PA 19103 1368 US 1370 Email: jason_livingood@cable.comcast.com 1371 URI: http://www.comcast.com 1373 Jim Mills 1374 Comcast Cable Communications 1375 One Comcast Center 1376 1800 Bishops Gate Drive 1377 Mount Laurel, NJ 08054 1378 US 1380 Email: jim_mills@cable.comcast.com 1381 URI: http://www.comcast.com 1383 Richard Woundy 1384 Comcast Cable Communications 1385 27 Industrial Avenue 1386 Chelmsford, MA 01824 1387 US 1389 Email: richard_woundy@cable.comcast.com 1390 URI: http://www.comcast.com