< draft-livingood-woundy-congestion-mgmt-02.txt   draft-livingood-woundy-congestion-mgmt-03.txt >
Internet Engineering Task Force C. Bastian Internet Engineering Task Force C. Bastian
Internet-Draft T. Klieber Internet-Draft T. Klieber
Intended status: Informational J. Livingood, Ed. Intended status: Informational J. Livingood, Ed.
Expires: August 14, 2010 J. Mills Expires: August 14, 2010 J. Mills
R. Woundy R. Woundy
Comcast Comcast
February 10, 2010 February 10, 2010
Comcast's Protocol-Agnostic Congestion Management System Comcast's Protocol-Agnostic Congestion Management System
draft-livingood-woundy-congestion-mgmt-02 draft-livingood-woundy-congestion-mgmt-03
Abstract Abstract
This document describes the congestion management system of Comcast This document describes the congestion management system of Comcast
Cable, a large cable broadband Internet Service Provider (ISP) in the Cable, a large cable broadband Internet Service Provider (ISP) in the
U.S. Comcast completed deployment of this congestion management U.S. Comcast completed deployment of this congestion management
system on December 31, 2008. system on December 31, 2008.
Status of this Memo Status of this Memo
skipping to change at page 2, line 19 skipping to change at page 2, line 19
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the BSD License. described in the BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Key Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 2. Key Terminology . . . . . . . . . . . . . . . . . . . . . . . 3
3. Historical Overview . . . . . . . . . . . . . . . . . . . . . 7 3. Historical Overview . . . . . . . . . . . . . . . . . . . . . 7
4. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 4. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5. Implemetation and Configuration . . . . . . . . . . . . . . . 9 5. Implementation and Configuration . . . . . . . . . . . . . . . 9
5.1. Thresholds For Determining When a CMTS Port Is in a 5.1. Thresholds For Determining When a CMTS Port Is in a
Near Congestion State . . . . . . . . . . . . . . . . . . 13 Near Congestion State . . . . . . . . . . . . . . . . . . 13
5.2. Thresholds For Determining When a User Is in an 5.2. Thresholds For Determining When a User Is in an
Extended High Consumption State and for Release from Extended High Consumption State and for Release from
that Classification . . . . . . . . . . . . . . . . . . . 14 that Classification . . . . . . . . . . . . . . . . . . . 14
5.3. Effect of BE Quality of Service on Users&apos 5.3. Effect of BE Quality of Service on Users&apos
Broadband Experience . . . . . . . . . . . . . . . . . . . 17 Broadband Experience . . . . . . . . . . . . . . . . . . . 17
5.4. Equipment/Software Used and Location . . . . . . . . . . . 19 5.4. Equipment/Software Used and Location . . . . . . . . . . . 19
6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 21 6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 21
7. Exceptional Network Utilization Considerations . . . . . . . . 21 7. Exceptional Network Utilization Considerations . . . . . . . . 21
8. Security Considerations . . . . . . . . . . . . . . . . . . . 22 8. Security Considerations . . . . . . . . . . . . . . . . . . . 22
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22
11. Change Log . . . . . . . . . . . . . . . . . . . . . . . . . . 22 11. Change Log . . . . . . . . . . . . . . . . . . . . . . . . . . 22
12. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 22 12. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 23
13. Informative References . . . . . . . . . . . . . . . . . . . . 23 13. Informative References . . . . . . . . . . . . . . . . . . . . 23
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26
1. Introduction 1. Introduction
Comcast Cable is a large broadband Internet Service Provider (ISP), Comcast Cable is a large broadband Internet Service Provider (ISP),
based in the U.S., serving the majority of its customers via cable based in the U.S., serving the majority of its customers via cable
modem technology. During the late part of 2008, and completing on modem technology. During the late part of 2008, and completing on
December 31, 2008, Comcast deployed a new congestion management December 31, 2008, Comcast deployed a new congestion management
system across its entire network. This new system was developed in system across its entire network. This new system was developed in
skipping to change at page 6, line 49 skipping to change at page 6, line 49
general, speeds do not typically exceed a customer's provisioned general, speeds do not typically exceed a customer's provisioned
speed. Comcast, however, invented a technology called "PowerBoost" speed. Comcast, however, invented a technology called "PowerBoost"
[PowerBoost Specification] that, for example, enables users to [PowerBoost Specification] that, for example, enables users to
experience brief boosts above their provisioned speeds while they experience brief boosts above their provisioned speeds while they
transfer large files over the Internet, by utilizing excess capacity transfer large files over the Internet, by utilizing excess capacity
which may be available in the network at that time. which may be available in the network at that time.
2.15. Quality of Service (QoS) 2.15. Quality of Service (QoS)
A set of techniques to manage network resources to ensure a level of A set of techniques to manage network resources to ensure a level of
performance to specific data flows, as descrived in [RFC1633] and performance to specific data flows, as described in [RFC1633] and
[RFC2475]. One method for providing QoS to a network is by [RFC2475]. One method for providing QoS to a network is by
differentiating the type of traffic by class or flow and assigning differentiating the type of traffic by class or flow and assigning
priorities to each type. When the network becomes congested, the priorities to each type. When the network becomes congested, the
data packets that are marked as having higher priority will have data packets that are marked as having higher priority will have
higher likelihood of being serviced. higher likelihood of being serviced.
2.16. Upstream 2.16. Upstream
Description of the direction in which a signal travels, in this case Description of the direction in which a signal travels, in this case
from the user to the network. Upstream traffic occurs when users are from the user to the network. Upstream traffic occurs when users are
skipping to change at page 7, line 31 skipping to change at page 7, line 31
2008, Comcast participated in an IETF Peer-to-Peer Infrastructure 2008, Comcast participated in an IETF Peer-to-Peer Infrastructure
Workshop [RFC5594], hosted by the Massachusetts Institute of Workshop [RFC5594], hosted by the Massachusetts Institute of
Technology (MIT) in Cambridge, MA, USA. Technology (MIT) in Cambridge, MA, USA.
In order to participate in this workshop, interested attendees were In order to participate in this workshop, interested attendees were
asked to submit a paper to a technical review team, which Comcast did asked to submit a paper to a technical review team, which Comcast did
on May 9, 2008, in the [Comcast P2Pi Position Paper]. Comcast on May 9, 2008, in the [Comcast P2Pi Position Paper]. Comcast
subsequently attended and participated in this valuable workshop. subsequently attended and participated in this valuable workshop.
During the workshop, Comcast outlined the high-level design for a new During the workshop, Comcast outlined the high-level design for a new
congestion management system [Comcast IETF P2P Infrastructure congestion management system [Comcast IETF P2P Infrastructure
Workshop Presentation] and solicitied comments and other feedback Workshop Presentation] and solicited comments and other feedback from
from attendees and other members of the Internet community attendees and other members of the Internet community (presentations
(presentations were also posted to the IETF's P2Pi mailing list). were also posted to the IETF's P2Pi mailing list). The congestion
The congestion management system outlined in that May 2008 workshop management system outlined in that May 2008 workshop was later tested
was later tested in trial markets and is in essence what was then in trial markets and is in essence what was then deployed by Comcast
deployed by Comcast later in 2008. later in 2008.
Following an August 2008 [FCC Memorandum and Opinion] regarding how Following an August 2008 [FCC Memorandum and Opinion - August 2008]
Comcast managed congestion on its High-Speed Internet ("HSI") regarding how Comcast managed congestion on its High-Speed Internet
network, Comcast disclosed to the FCC and the public additional ("HSI") network, Comcast disclosed to the FCC [FCC Network Management
technical details of the congestion management system that it Response - September 2008] and the public additional technical
intended to and did implement by the end of 2008, including the details of the congestion management system that it intended to and
thresholds involved in this new system. That system is detailed in did implement by the end of 2008 [FCC Congestion Management
this document. While the description of how this system is deployed Deployment Completion Letter - January 2009], including the
in the Comcast network is necessarily specific to the various thresholds involved in this new system. While the description of how
technologies and designs specific to that network, a similar system this system is deployed in the Comcast network is necessarily
could be deployed on virtually any large scale ISP network or other specific to the various technologies and designs specific to that
IP network. network, a similar system could be deployed on virtually any large
scale ISP network or other IP network.
4. Summary 4. Summary
Comcast's HSI network has elements which are shared across many Comcast's HSI network has elements which are shared across many
subscribers. This means that Comcast's HSI customers share upstream subscribers. This means that Comcast's HSI customers share upstream
and downstream bandwidth with their neighbors. Although the and downstream bandwidth with their neighbors. Although the
available bandwidth is substantial, so, too, is the demand. Thus, available bandwidth is substantial, so, too, is the demand. Thus,
when a relatively small number of customers in a neighborhood place when a relatively small number of customers in a neighborhood place
disproportionate demands on network resources, this can cause disproportionate demands on network resources, this can cause
congestion that degrades their neighbors' Internet experience. The congestion that degrades their neighbors' Internet experience. The
skipping to change at page 9, line 19 skipping to change at page 9, line 19
Comcast undertook considerable effort, over the course of many Comcast undertook considerable effort, over the course of many
months, to formulate our plans for this congestion management months, to formulate our plans for this congestion management
approach, adjusting them, and subjecting them to real-world trials. approach, adjusting them, and subjecting them to real-world trials.
Market trials were conducted in Chambersburg, PA; Warrenton, VA; Lake Market trials were conducted in Chambersburg, PA; Warrenton, VA; Lake
City, FL; East Orange, FL; and Colorado Springs, CO, between June and City, FL; East Orange, FL; and Colorado Springs, CO, between June and
September, 2008. This enabled us to validate the utility of the September, 2008. This enabled us to validate the utility of the
general approach and collect substantial trial data to test multiple general approach and collect substantial trial data to test multiple
variations and alternative formulations. variations and alternative formulations.
5. Implemetation and Configuration 5. Implementation and Configuration
To understand exactly how these new congestion management practices To understand exactly how these new congestion management practices
work, it is helpful to have a general understanding of how Comcast's work, it is helpful to have a general understanding of how Comcast's
HSI network is designed. Comcast's HSI network is what is commonly HSI network is designed. Comcast's HSI network is what is commonly
referred to as a hybrid fiber-coax network, with coaxial cable referred to as a hybrid fiber-coax network, with coaxial cable
connecting each subscriber's cable modem to an Optical Node, and connecting each subscriber's cable modem to an Optical Node, and
fiber optic cables connecting the Optical Node, through distribution fiber optic cables connecting the Optical Node, through distribution
hubs, to the Cable Modem Termination System (CMTS), which is also hubs, to the Cable Modem Termination System (CMTS), which is also
known as a "data node". The CMTSes are then connected to higher- known as a "data node". The CMTSes are then connected to higher-
level routers, which in turn are connected to Comcast's Internet level routers, which in turn are connected to Comcast's Internet
skipping to change at page 10, line 6 skipping to change at page 10, line 6
multiple channels are usually configured to come out each physical multiple channels are usually configured to come out each physical
port. Said another way, there is generally a mapping of multiple port. Said another way, there is generally a mapping of multiple
channels to each physical port. channels to each physical port.
Currently, on average, approximately 275 cable modems share the same Currently, on average, approximately 275 cable modems share the same
downstream port and about 100 cable modems share the same upstream downstream port and about 100 cable modems share the same upstream
port, however this is constantly changing (both numbers generally port, however this is constantly changing (both numbers generally
become smaller over time). Both types of ports can experience become smaller over time). Both types of ports can experience
congestion that could degrade the broadband experience of our congestion that could degrade the broadband experience of our
subscribers and, unlike with the previous congestion management subscribers and, unlike with the previous congestion management
practices, both upstream and downstream traffic are be subject to practices, both upstream and downstream traffic are subject to
management in this new congestion management system. management in this new congestion management system.
To implement Comcast's new protocol-agnostic congestion management To implement Comcast's new protocol-agnostic congestion management
practices, Comcast purchased new hardware and software that was practices, Comcast purchased new hardware and software that was
deployed near the Regional Network Routers ("RNRs") that are further deployed near the Regional Network Routers ("RNRs") that are further
upstream in Comcast's network. This new hardware consists of upstream in Comcast's network. This new hardware consists of
Internet Protocol Detail Record ("IPDR") servers, Congestion Internet Protocol Detail Record ("IPDR") servers, Congestion
Management servers, and PacketCable Multimedia ("PCMM") servers. Management servers, and PacketCable Multimedia ("PCMM") servers.
Further details about each of these pieces of equipment can be found Further details about each of these pieces of equipment can be found
below, in Section 5.4. It is important to note here, however, that below, in Section 5.4. It is important to note here, however, that
skipping to change at page 12, line 40 skipping to change at page 12, line 40
cable modem only when both of two conditions are met: cable modem only when both of two conditions are met:
o First, the usage level of a particular upstream or downstream port o First, the usage level of a particular upstream or downstream port
of a CMTS, as measured over a particular period of time, must be of a CMTS, as measured over a particular period of time, must be
nearing the point where congestion could degrade users' nearing the point where congestion could degrade users'
experience. We refer to this as the "Near Congestion State" and, experience. We refer to this as the "Near Congestion State" and,
based on the technical trials we have conducted, we have based on the technical trials we have conducted, we have
established a threshold, described in more detail below, for when established a threshold, described in more detail below, for when
a particular CMTS port enters that state. a particular CMTS port enters that state.
o Second, a particular subscriber must be making a significant o Second, a particular subscriber must be making an extended, high
contribution to the bandwidth usage on the particular port, as contribution to the bandwidth usage on the particular port,
measured over a particular period of time. We refer to this as relative to the service tier they purchased, as measured over a
the "Extended High Consumption State" and, based on the technical particular period of time. We refer to this as the "Extended High
trials we have conducted, we have established a threshold, Consumption State" and, based on the technical trials we have
described in more detail below, for when a particular user enters conducted, we have established a threshold, described in more
that state. detail below, for when a particular user enters that state.
When, and only when, both conditions are met, a user's upstream or When, and only when, both conditions are met, a user's upstream or
downstream traffic (depending on which type of port is in the Near downstream traffic (depending on which type of port is in the Near
Congestion State) is designated as BE. Then, to the extent that Congestion State) is designated as BE. Then, to the extent that
actual congestion occurs, any delay resulting from the congestion actual congestion occurs, any delay resulting from the congestion
will affect BE traffic before it affects PBE traffic. will affect BE traffic before it affects PBE traffic.
We now explain the foregoing in greater detail in the following We now explain the foregoing in greater detail in the following
sections. sections.
skipping to change at page 14, line 16 skipping to change at page 14, line 16
managed users whose traffic is delayed during those congested periods managed users whose traffic is delayed during those congested periods
perceive little, if any, effect, as discussed below. perceive little, if any, effect, as discussed below.
5.2. Thresholds For Determining When a User Is in an Extended High 5.2. Thresholds For Determining When a User Is in an Extended High
Consumption State and for Release from that Classification Consumption State and for Release from that Classification
Once a particular CMTS port is in a Near Congestion State, the Once a particular CMTS port is in a Near Congestion State, the
software examines whether any cable modems are consuming bandwidth software examines whether any cable modems are consuming bandwidth
disproportionately. (Note: Although each cable modem is typically disproportionately. (Note: Although each cable modem is typically
assigned to a particular household, the software does not and cannot assigned to a particular household, the software does not and cannot
actually identify individual users or analyze particular users' actually identify individual users, the number of users sharing a
traffic.) For purposes of this document, we use "cable modem", cable modem, or analyze particular users' traffic.) For purposes of
"user", and "subscriber" interchangeably to mean a subscriber account this document, we use "cable modem", "user", and "subscriber"
or user account and not an individual person.). For a user to enter interchangeably to mean a subscriber account or user account and not
an Extended High Consumption State, he or she must consume greater an individual person. For a user to enter an Extended High
than a certain percentage of his or her provisioned upstream or Consumption State, he or she must consume greater than a certain
downstream bandwidth (the "User Consumption Threshold") for a percentage of his or her provisioned upstream or downstream bandwidth
specific length of time (the "User Consumption Duration"). The User (the "User Consumption Threshold") for a specific length of time (the
Consumption Threshold is measured as a user's consumption of a "User Consumption Duration"). The User Consumption Threshold is
particular percentage of his or her total provisioned upstream or measured as a user's consumption of a particular percentage of his or
downstream bandwidth. That bandwidth is the maximum speed that a her total provisioned upstream or downstream bandwidth. That
particular modem can achieve based on the tier (personal/residential, bandwidth is the maximum speed that a particular modem can achieve
commercial, etc.) the customer has purchased. For example, if a user based on the tier (personal/residential, commercial, etc.) the
buys a service with speeds of 8 Mbps downstream and 1 Mbps upstream, customer has purchased. For example, if a user buys a service with
then his or her provisioned downstream speed is 8 Mbps and speeds of 50 Mbps downstream and 10 Mbps upstream, then his or her
provisioned upstream speed is 1 Mbps. It is also important to note provisioned downstream speed is 50 Mbps and provisioned upstream
that because the User Consumption Threshold is a percentage of speed is 10 Mbps. It is also important to note that because the User
provisioned bandwidth for a particular user account, and not a static Consumption Threshold is a percentage of provisioned bandwidth for a
value, users of higher speed tiers have correspondingly higher User particular user account, and not a static value, users of higher
Consumption Thresholds. Lastly, the User Consumption Duration is speed tiers have correspondingly higher User Consumption Thresholds.
measured in minutes. Lastly, the User Consumption Duration is measured in minutes.
Following lab tests, simulations, technical trials, customer Following lab tests, simulations, technical trials, customer
feedback, vendor evaluations, and a third-party consulting analysis, feedback, vendor evaluations, and a third-party consulting analysis,
we have determined that the appropriate starting point for the User we have determined that the appropriate starting point for the User
Consumption Threshold is 70 percent of a subscriber's provisioned Consumption Threshold is 70 percent of a subscriber's provisioned
upstream or downstream bandwidth, and that the appropriate starting upstream or downstream bandwidth, and that the appropriate starting
point for the User Consumption Duration is 15 minutes. That is, when point for the User Consumption Duration is 15 minutes. That is, when
a subscriber uses an average of 70 percent or more of his or her a subscriber uses an average of 70 percent or more of his or her
provisioned upstream or downstream bandwidth over a particular 15- provisioned upstream or downstream bandwidth over a particular 15-
minute period, that user is then in an Extended High Consumption minute period, that user is then in an Extended High Consumption
skipping to change at page 15, line 10 skipping to change at page 15, line 10
peak-speed-based threshold. Thus, the Extended High Consumption peak-speed-based threshold. Thus, the Extended High Consumption
State is not tied to whether a user has bursted once or more above State is not tied to whether a user has bursted once or more above
this 70% threshold for a brief moment. Instead, it is consumption- this 70% threshold for a brief moment. Instead, it is consumption-
based, meaning that a certain bitrate must be exceeded over at least based, meaning that a certain bitrate must be exceeded over at least
the entire User Consumption Duration. the entire User Consumption Duration.
The User Consumption Thresholds have been set sufficiently high that The User Consumption Thresholds have been set sufficiently high that
using the HSI connection for VoIP, web surfing, or most streaming using the HSI connection for VoIP, web surfing, or most streaming
video cannot alone cause subscribers to our standard-level HSI video cannot alone cause subscribers to our standard-level HSI
service to exceed the User Consumption Threshold. For example, while service to exceed the User Consumption Threshold. For example, while
Comcast's standard-level HSI service provisions downstream bandwidth one of Comcast's common HSI service tiers has a provisioned
at 12 Mbps, today, streaming video (even some HD video) from Hulu downstream bandwidth of 15 Mbps today, streaming video (even some HD
uses less than 2.5 Mbps, a Vonage or Skype VoIP call uses less than video) from Hulu uses less than 2.5 Mbps, a Vonage or Skype VoIP call
131 Kbps, and streaming music uses less than 128 kbps (in this uses less than 131 Kbps, and streaming music uses less than 128 kbps
example, 70 percent of 12 Mbps is 8.4 Mbps). As noted above, these (in this example, 70 percent of 15 Mbps is 10.5 Mbps). As noted
values are subject to change as necessary in the same way that above, these values are subject to change as necessary in the same
specific anti-spam or other network management practices are adjusted way that specific anti-spam or other network management practices are
to address new issues that arise, or should unexpected software bugs adjusted to address new issues that arise, or should unexpected
or other problems arise. software bugs or other problems arise.
Based on data collected from the trial markets where the new Based on data collected from the trial markets where the new
management practices are being tested, on average less than one-third management practices are being tested, on average less than one-third
of one percent of subscribers have had their traffic priority status of one percent of subscribers have had their traffic priority status
changed to the BE state on any given day. For example, in Colorado changed to the BE state on any given day. For example, in Colorado
Springs, CO, the largest test market, on any given day in August Springs, CO, the largest test market, on any given day in August
2008, an average of 22 users out of 6,016 total subscribers in the 2008, an average of 22 users out of 6,016 total subscribers in the
trial had their traffic priority status changed to BE at some point trial had their traffic priority status changed to BE at some point
during the day. during the day.
A user's traffic is released from a BE state when the user's A user's traffic is released from a BE state when the user's
bandwidth consumption drops below 50 percent of his or her bandwidth consumption drops below 50 percent of his or her
provisioned upstream or downstream bandwidth for a period of provisioned upstream or downstream bandwidth for a period of
approximately 15 minutes. These release criteria are intended to approximately 15 minutes. These release criteria are intended to
minimize (and hopefully prevent) user QoS oscillation (hysteresis), minimize (and hopefully prevent) user QoS oscillation (hysteresis),
i.e., a situation in which a particular user could cycle repeatedly i.e., a situation in which a particular user could cycle repeatedly
between BE and PBE. NetForecast, Inc., an independent consultant between BE and PBE. NetForecast, Inc., an independent consultant
retained to provide analysis and recommendations regarding Comcast's retained to provide analysis and recommendations regarding Comcast's
trials and related congestion management work, suggested this trials and related congestion management work, suggested this
approach, which has worked well in our ongoing trials and lab approach, which has worked well in our ongoing trials and lab
testing. In trials, we have observed that user traffic rarely testing.
remains in a managed state longer than the initial 15-minute period.
Simply put, there are four steps to determining whether the traffic Simply put, there are four steps to determining whether the traffic
associated with a particular cable modem is designated as PBE or BE: associated with a particular cable modem is designated as PBE or BE:
1. Determine if the CMTS port is in a Near Congestion State. 1. Determine if the CMTS port is in a Near Congestion State.
2. If yes, determine whether any users are in an Extended High 2. If yes, determine whether any users are in an Extended High
Consumption State. Consumption State.
3. If yes, change those users' traffic to BE from PBE. If the 3. If yes, change those users' traffic to BE from PBE. If the
skipping to change at page 17, line 50 skipping to change at page 17, line 50
that the CMTS handles traffic. Specifically, CMTS ports have what is that the CMTS handles traffic. Specifically, CMTS ports have what is
commonly called a "scheduler" that puts all the packets coming from commonly called a "scheduler" that puts all the packets coming from
or going to cable modems on that particular port in a queue and then or going to cable modems on that particular port in a queue and then
handles them in turn. A certain number of packets can be processed handles them in turn. A certain number of packets can be processed
by the scheduler in any given moment; for each time slot, PBE traffic by the scheduler in any given moment; for each time slot, PBE traffic
is given priority access to the available capacity, and BE traffic is is given priority access to the available capacity, and BE traffic is
processed on a space-available basis. (It is important to note that processed on a space-available basis. (It is important to note that
congestion can occur in any IP network, and, when it does, packets congestion can occur in any IP network, and, when it does, packets
can be delayed or dropped. As a result, applications and protocols can be delayed or dropped. As a result, applications and protocols
have been designed to deal with this reality. Our congestion have been designed to deal with this reality. Our congestion
management systems attempts to ensure that, in those rare cases where management system attempts to ensure that, in those rare cases where
packets may be dropped, BE packets are dropped before PBE packets are packets may be dropped, BE packets are dropped before PBE packets are
dropped.) dropped.)
A rough analogy would be to busses that empty and fill up at A rough analogy would be to busses that empty and fill up at
incredibly fast speeds. As empty busses arrive at the figurative incredibly fast speeds. As empty busses arrive at the figurative
"bus stop" -- every two milliseconds in this case -- they fill up "bus stop" -- every two milliseconds in this case -- they fill up
with as many packets as are waiting for "seats" on the bus, to the with as many packets as are waiting for "seats" on the bus, to the
limits of the bus' capacity. During non-congested periods, the bus limits of the bus' capacity. During non-congested periods, the bus
will usually have several empty seats, but, during congested periods, will usually have several empty seats, but, during congested periods,
the bus will fill up and packets will have to wait for the next bus. the bus will fill up and packets will have to wait for the next bus.
It is in the congested periods that BE packets will be affected. If It is in the congested periods that BE packets will be affected. If
there is no congestion, packets from a user in a BE state should have there is no congestion, packets from a user in a BE state should have
little trouble getting on the bus when they arrive at the bus stop. little trouble getting on the bus when they arrive at the bus stop.
If, on the other hand, there is congestion in a particular instance, If, on the other hand, there is congestion in a particular instance,
the bus may become filled by packets in a PBE state before any BE the bus may become filled by packets in a PBE state before any BE
packets can get on. In that situation, the BE packets would have to packets can get on. In that situation, the BE packets would have to
wait for the next bus that is not filled by PBE packets. In reality, wait for the next bus that is not filled by PBE packets. In reality,
this all takes place in two-millisecond increments, so even if the this all takes place in two-millisecond increments, so even if the
packets miss 50 "busses", the delay only will be about one-tenth of a packets miss 50 "busses", the delay will only be about one-tenth of a
second. second.
During times of actual network congestion, when BE traffic might be During times of actual network congestion, when BE traffic might be
delayed, there are a variety of effects that could be experienced by delayed, there are a variety of effects that could be experienced by
a user whose traffic is delayed, depending upon what applications he a user whose traffic is delayed, depending upon what applications he
or she is using. Typically, a user whose traffic is in a BE state or she is using. Typically, a user whose traffic is in a BE state
during actual congestion may find that a webpage loads sluggishly, a during actual congestion may find that a webpage loads sluggishly, a
peer-to-peer upload takes somewhat longer to complete, or a VoIP call peer-to-peer upload takes somewhat longer to complete, or a VoIP call
sounds choppy. Of course, the same thing could happen to the sounds choppy. Of course, the same thing could happen to the
customers on a port that is congested in the absence of any customers on a port that is congested in the absence of any
congestion management; the difference here is that the effects of any congestion management; the difference here is that the effects of any
such delays are shifted toward those who have been placing the such delays are shifted toward those who have been placing the
greatest burden on the network, instead of being distributed randomly greatest burden on the network, instead of being distributed randomly
among the users of that port without regard to their consumption among the users of that port without regard to their consumption
levels. levels. As a matter of fact, our studies concluded that the
experience of the PBE subscribers improves when this congestion
management system is enabled. This conclusion is based on network
measurements, such as latency, and not the actual subscriber
experience.
NetForecast, Inc. explored the potential risk of a worst-case NetForecast, Inc. explored the potential risk of a worst-case
scenario for users whose traffic is in a BE state: the possibility of scenario for users whose traffic is in a BE state: the possibility of
"bandwidth starvation" in the theoretical case where 100 percent of "bandwidth starvation" in the theoretical case where 100 percent of
the CMTS bandwidth is taken up by PBE traffic for an extended period the CMTS bandwidth is taken up by PBE traffic for an extended period
of time. In theory, such a condition could mean that a given user of time. In theory, such a condition could mean that a given user
whose traffic is designated BE would be unable to effectuate an whose traffic is designated BE would be unable to effectuate an
upload or download (as noted above, both are managed separately) for upload or download (as noted above, both are managed separately) for
some period of time. However, when these management techniques were some period of time. However, when these management techniques were
tested, first in company testbeds and then in our real-world trials tested, first in company testbeds and then in our real-world trials
conducted in the five markets, such a theoretical condition did not conducted in the five markets, such a theoretical condition did not
occur. In addition, trial results demonstrated that these management occur. In addition, our experience with the system as fully deployed
in our production network demonstrates that these management
practices have very modest real-world impacts. In addition, Comcast practices have very modest real-world impacts. In addition, Comcast
did not receive a single customer complaint in any of the trial did not receive a single customer complaint in any of the trial
markets that could be traced to the new congestion management system, markets that could be traced to this congestion management system,
despite having broadly publicized these trials. despite having broadly publicized these trials. In our subsequent
national deployment into our production network, we still have yet to
find a specific complaint that can be traced back to the effected of
this congestion management system.
Comcast continues to monitor how user traffic is affected by these Comcast continues to monitor how user traffic is affected by these
new congestion management techniques and will make the adjustments new congestion management techniques and will make the adjustments
necessary to ensure that all Comcast HSI customers have a high- necessary to ensure that all Comcast HSI customers have a high-
quality Internet experience. quality Internet experience.
5.4. Equipment/Software Used and Location 5.4. Equipment/Software Used and Location
The above-mentioned functions are carried out using three different The above-mentioned functions are carried out using three different
types of application servers, supplied by three different vendors. types of application servers, supplied by three different vendors.
As mentioned above, these servers are installed near Comcast's As mentioned above, these servers are installed near Comcast's
regional network routers. The exact locations of various servers is regional network routers. The exact locations of these servers is
not particularly relevant to this document, as this does not change not particularly relevant to this document, as this does not change
the fact that they manage individual CMTS ports. the fact that they manage individual CMTS ports.
The first application server is an IPDR server, which collects The first application server is an IPDR server, which collects
relevant cable modem volume usage information from the CMTS, such as relevant cable modem volume usage information from the CMTS, such as
how many aggregate upstream or downstream bytes a subscriber uses how many aggregate upstream or downstream bytes a subscriber uses
over a particular period of time. IPDR has been adopted as a over a particular period of time. IPDR has been adopted as a
standard by many industry organizations and initiatives, such as standard by many industry organizations and initiatives, such as
CableLabs, ATIS, ITU, and 3GPP, among others. CableLabs, ATIS, ITU, and 3GPP, among others.
skipping to change at page 21, line 24 skipping to change at page 21, line 32
transition on December 31, 2008. As described above, the new transition on December 31, 2008. As described above, the new
approach does not manage congestion by focusing on managing the use approach does not manage congestion by focusing on managing the use
of specific protocols. Nor will this approach use TCP "reset of specific protocols. Nor will this approach use TCP "reset
packets" [RFC3360]. Rather, the new system acts such that (1) during packets" [RFC3360]. Rather, the new system acts such that (1) during
periods when a CMTS port is in a Near Congestion State, (2) it periods when a CMTS port is in a Near Congestion State, (2) it
identifies the subscribers on that port who have consumed a identifies the subscribers on that port who have consumed a
disproportionate amount of bandwidth over the preceding 15 minutes, disproportionate amount of bandwidth over the preceding 15 minutes,
(3) it lowers the priority status of those subscribers' traffic to BE (3) it lowers the priority status of those subscribers' traffic to BE
status until those subscribers meet the release criteria, and (4) status until those subscribers meet the release criteria, and (4)
during periods of congestion, delay BE traffic before PBE traffic is during periods of congestion, delay BE traffic before PBE traffic is
delayed. Our trials and our network-wide deployment indicate that delayed. Our trials and our subsequent network-wide deployment
this new congestion management system ensures a quality online indicate that this new congestion management system ensures a quality
experience for all of our HSI customers. online experience for all of our HSI customers.
7. Exceptional Network Utilization Considerations 7. Exceptional Network Utilization Considerations
This system was developed to cope with somewhat "normal" occurrences This system was developed to cope with somewhat "normal" occurrences
of congestion that could occur on virtually any IP network. It of congestion that could occur on virtually any IP network. It
should also be noted, however, that such a system could also prove should also be noted, however, that such a system could also prove
particularly useful in the case of "exceptional network utilization" particularly useful in the case of "exceptional network utilization"
events which existing network usage models do not or cannot events which existing network usage models do not or cannot
accurately predict. Some network operators refer to these accurately predict. Some network operators refer to these
exceptional events as 'surges' in utilization, similar to sudden exceptional events as 'surges' in utilization, similar to sudden
skipping to change at page 22, line 26 skipping to change at page 22, line 32
10. Acknowledgements 10. Acknowledgements
The authors wish to acknowledge the hard work of the many people who The authors wish to acknowledge the hard work of the many people who
helped to develop and/or review this document, as well as the people helped to develop and/or review this document, as well as the people
who helped deploy the system in such a short period of time. who helped deploy the system in such a short period of time.
11. Change Log 11. Change Log
NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION. NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION.
o v03 - changed FCC doc reference to a more stable ref point,
closing an open item. updated Mbps numbers to reflect current
Internet speeds, closing an open item. also, closed out all
editorial notes, closing an open item. additional tweaks following
homework assignments send to co-authors.
o v02 - lots of minor tweaks based on homework assignments that o v02 - lots of minor tweaks based on homework assignments that
Jason handed out to co-authors, and closed out several open items Jason handed out to co-authors, and closed out several open items
o v01 - closed item on the open issue list - updated references o v01 - closed item on the open issue list - updated references
o v00 - first version published o v00 - first version published
12. Open Issues 12. Open Issues
NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION. NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION.
1. Need to add more stable URL references for FCC docs in 1. MAJOR: Add some content to the Security Considerations section
Informative Refs - change from Comcast file locations to FCC file
locations.
2. Make the ASCII art a bit smaller (vertically)
3. Double-check IPDR references - ensure most recent version of 2. MINOR: Make the ASCII art a bit smaller (vertically)
standard is referenced (incl. possibly DOCSIS 3.0 reference w/r/t
IPDR)
4. Close out any editorial notes in the document 3. MINOR: Double-check IPDR references - ensure most recent version
5. Add some content to the Security Considerations section of standard is referenced (incl. possibly DOCSIS 3.0 reference
w/r/t IPDR)
6. Change the reference URL for PowerBoost to something more stable, 4. MINOR: Change the reference URL for PowerBoost to something more
perhaps from the USPTO stable, perhaps from the USPTO
7. Update all Mbps numbers to reflect more current broadband speeds 5. MINOR: Before final update of document, update Mbps speeds to
reflect then-current speeds.
13. Informative References 13. Informative References
[CableLabs DOCSIS 3.0 Cable Modem to Customer Premise Equipment [CableLabs DOCSIS 3.0 Cable Modem to Customer Premise Equipment
Interface Specification] Interface Specification]
CableLabs, "Data-Over-Cable Service Interface CableLabs, "Data-Over-Cable Service Interface
Specifications - DOCSIS 3.0 - Cable Modem to Customer Specifications - DOCSIS 3.0 - Cable Modem to Customer
Premise Equipment Interface Specification", DOCSIS 3.0 CM- Premise Equipment Interface Specification", DOCSIS 3.0 CM-
SP-CMCIv3-I01-080320, March 2008, <http:// SP-CMCIv3-I01-080320, March 2008, <http://
www.cablelabs.com/specifications/ www.cablelabs.com/specifications/
skipping to change at page 24, line 28 skipping to change at page 24, line 39
wiki/PeerToPeerInfrastructure/02-Comcast-IETF-P2Pi.pdf>. wiki/PeerToPeerInfrastructure/02-Comcast-IETF-P2Pi.pdf>.
[Comcast P2Pi Position Paper] [Comcast P2Pi Position Paper]
Livingood, J. and R. Woundy, "Comcast's IETF P2P Livingood, J. and R. Woundy, "Comcast's IETF P2P
Infrastructure Workshop Position Paper", FCC Infrastructure Workshop Position Paper", FCC
Filings Comcast Network Management Proceedings, May 2008, Filings Comcast Network Management Proceedings, May 2008,
<http://trac.tools.ietf.org/area/rai/trac/raw-attachment/ <http://trac.tools.ietf.org/area/rai/trac/raw-attachment/
wiki/PeerToPeerInfrastructure/ wiki/PeerToPeerInfrastructure/
16%20ietf-p2pi-comcast-20080509.pdf>. 16%20ietf-p2pi-comcast-20080509.pdf>.
[FCC Filing - Attachment A: Old Congestion Management System] [FCC Congestion Management Deployment Completion Letter - January
Zachem, K., "Attachment A: Comcast Corporation Description 2009]
of Current Network Management Practices", FCC
Filings Comcast Network Management Proceedings,
September 2008, <http://downloads.comcast.net/docs/
Attachment_A_Current_Practices.pdf>.
[FCC Filing - Attachment B: New Congestion Management System]
Zachem, K., "Attachment B: Comcast Corporation Description
of Planned Network Management Practices to be Deployed
Following the Termination of Current Practices", FCC
Filings Comcast Network Management Proceedings,
September 2008, <http://downloads.comcast.net/docs/
Attachment_B_Future_Practices.pdf>.
[FCC Filing - Attachment C: Implementation and Compliance Plan]
Zachem, K., "Attachment C: Comcast Corporation Network
Management Transition Compliance Plan", FCC
Filings Comcast Network Management Proceedings,
September 2008, <http://downloads.comcast.net/docs/
Attachment_C_Compliance_Plan.pdf>.
[FCC Filing - Congestion Management Deployment Completion Letter]
Zachem, K., "Letter to the FCC Advising of Successful Zachem, K., "Letter to the FCC Advising of Successful
Deployment of Comcast's New Congestion Management System", Deployment of Comcast's New Congestion Management System",
FCC Filings Comcast Network Management Proceedings, FCC Filings Comcast Network Management Proceedings,
January 2009, <http://downloads.comcast.net/docs/ January 2009, <http://fjallfoss.fcc.gov/ecfs/document/
comcast-nm-transition-notification.pdf>. view?id=6520192582>.
[FCC Filing - Cover Letter]
Zachem, K., "Cover Letter to the FCC Regarding Comcast's
Network Management Techniques", FCC Filings Comcast
Network Management Proceedings, September 2008,
<http://downloads.comcast.net/docs/Cover_Letter.pdf>.
[FCC Memorandum and Opinion] [FCC Memorandum and Opinion - August 2008]
Martin, K., Copps, M., Adelstein, J., Tate, D., and R. Martin, K., Copps, M., Adelstein, J., Tate, D., and R.
McDowell, "FCC Memorandum and Opinion Regarding Reasonable McDowell, "FCC Memorandum and Opinion Regarding Reasonable
Network Management", File No. EB-08-IH-1518 WC Docket No. Network Management", File No. EB-08-IH-1518 WC Docket No.
07-52, August 2008, <http://hraunfoss.fcc.gov/ 07-52, August 2008, <http://hraunfoss.fcc.gov/
edocs_public/attachmatch/FCC-08-183A1.pdf>. edocs_public/attachmatch/FCC-08-183A1.pdf>.
[FCC Network Management Response - September 2008]
Zachem, K., "Letter to the FCC Regarding Comcast's Network
Management Practices", FCC Filings Comcast Network
Management Proceedings, September 2008, <http://
fjallfoss.fcc.gov/ecfs/document/view?id=6520169715>.
[IPDR Standard] [IPDR Standard]
Cotton, S., Cockrell, B., Walls, P., and T. Givoly, Cotton, S., Cockrell, B., Walls, P., and T. Givoly,
"Network Data Management - Usage (NDM-U) For IP-Based "Network Data Management - Usage (NDM-U) For IP-Based
Services Service Specification - Cable Labs DOCSIS 2.0 Services Service Specification - Cable Labs DOCSIS 2.0
SAMIS", IPDR Service Specifications NDM-U, November 2004, SAMIS", IPDR Service Specifications NDM-U, November 2004,
<http://www.ipdr.org/public/Service_Specifications/3.X/ <http://www.ipdr.org/public/Service_Specifications/3.X/
DOCSIS(R)3.5-A.0.pdf>. DOCSIS(R)3.5-A.0.pdf>.
[PowerBoost Specification] [PowerBoost Specification]
Comcast Cable Communications Management LLC, "Comcast Comcast Cable Communications Management LLC, "Comcast
 End of changes. 29 change blocks. 
117 lines changed or deleted 106 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/