< draft-linsner-lmap-use-cases-02.txt   draft-linsner-lmap-use-cases-03.txt >
INTERNET-DRAFT Marc Linsner INTERNET-DRAFT Marc Linsner
Intended Status: Informational Cisco Systems Intended Status: Informational Cisco Systems
Expires: August 29, 2013 Philip Eardley Expires: January 16, 2014 Philip Eardley
Trevor Burbridge Trevor Burbridge
BT BT
February 25, 2013 July 15, 2013
Large-Scale Broadband Measurement Use Cases Large-Scale Broadband Measurement Use Cases
draft-linsner-lmap-use-cases-02 draft-linsner-lmap-use-cases-03
Abstract Abstract
Measuring broadband performance on a large scale is important for Measuring broadband performance on a large scale is important for
network diagnostics by providers and users, as well for as public network diagnostics by providers and users, as well for as public
policy. To conduct such measurements, user networks gather data, policy. To conduct such measurements, user networks gather data,
either on their own initiative or instructed by a measurement either on their own initiative or instructed by a measurement
controller, and then upload the measurement results to a designated controller, and then upload the measurement results to a designated
measurement server. Understanding the various scenarios and users of measurement server. Understanding the various scenarios and users of
measuring broadband performance is essential to development of the measuring broadband performance is essential to development of the
skipping to change at page 2, line 27 skipping to change at page 2, line 27
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1 Internet Service Provider (ISP) Use Case . . . . . . . . . . 3 2.1 Internet Service Provider (ISP) Use Case . . . . . . . . . . 3
2.2 End User Network Diagnostics . . . . . . . . . . . . . . . . 4 2.2 End User Network Diagnostics . . . . . . . . . . . . . . . . 4
2.3 Multi-provider Network Measurements . . . . . . . . . . . . 4 2.3 Regulators . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4 Over the Top Providers . . . . . . . . . . . . . . . . . . . 4
2.5 Regulators . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 Details of ISP Use Case . . . . . . . . . . . . . . . . . . . . 6 3 Details of ISP Use Case . . . . . . . . . . . . . . . . . . . . 6
3.1 Existing Capabilities and Shortcomings . . . . . . . . . . . 6 3.1 Existing Capabilities and Shortcomings . . . . . . . . . . . 6
3.2 Understanding the quality experienced by customers . . . . . 6 3.2 Understanding the quality experienced by customers . . . . . 6
3.3 Benchmarking and competitor insight . . . . . . . . . . . . 8 3.3 Benchmarking and competitor insight . . . . . . . . . . . . 8
3.4 Understanding the impact and operation of new devices and 3.4 Understanding the impact and operation of new devices and
technology . . . . . . . . . . . . . . . . . . . . . . . . . 8 technology . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.5 Design and planning . . . . . . . . . . . . . . . . . . . . 9 3.5 Design and planning . . . . . . . . . . . . . . . . . . . . 9
3.6 Identifying, isolating and fixing network problems . . . . . 11 3.6 Identifying, isolating and fixing network problems . . . . . 11
3.7 Comparison with the regulator use case . . . . . . . . . . . 12 3.7 Comparison with the regulator use case . . . . . . . . . . . 12
3.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . 13 3.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . 13
4 Security Considerations . . . . . . . . . . . . . . . . . . . . 14 4 Security Considerations . . . . . . . . . . . . . . . . . . . . 14
5 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 14 5 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 14
6 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 14 6 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 14
7 References . . . . . . . . . . . . . . . . . . . . . . . . . . 14 7 References . . . . . . . . . . . . . . . . . . . . . . . . . . 15
7.1 Normative References . . . . . . . . . . . . . . . . . . . 14 7.1 Normative References . . . . . . . . . . . . . . . . . . . 15
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 15 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 15
1 Introduction 1 Introduction
Large-scale measurement efforts in [LMAP-REQ] describe three use Large-scale measurement efforts in [LMAP-REQ] describe three use
cases to be considered in deriving the requirements to be used in cases to be considered in deriving the requirements to be used in
developing the solution. This documents attempts to describe those developing the solution. This documents attempts to describe those
use cases in further detail and include additional use cases. use cases in further detail and include additional use cases.
1.1 Terminology 1.1 Terminology
skipping to change at page 4, line 31 skipping to change at page 4, line 31
End users may want to determine whether their network is performing End users may want to determine whether their network is performing
according to the specifications (e.g., service level agreements) according to the specifications (e.g., service level agreements)
offered by their Internet service provider, or they may want to offered by their Internet service provider, or they may want to
diagnose whether components of their network path are impaired. End diagnose whether components of their network path are impaired. End
users may perform measurements on their own, using the measurement users may perform measurements on their own, using the measurement
infrastructure they provide or infrastructure offered by a third infrastructure they provide or infrastructure offered by a third
party, or they may work directly with their network or application party, or they may work directly with their network or application
provider to diagnose a specific performance problem. Depending on provider to diagnose a specific performance problem. Depending on
the circumstances, measurements may occur at specific pre-defined the circumstances, measurements may occur at specific pre-defined
intervals, or may be triggered manually. A system administrator may intervals, or may be triggered manually. A system administrator may
perform such measurements on behalf of the user. perform such measurements on behalf of the user. Example use cases
of end user initiated performance measurements include:
2.3 Multi-provider Network Measurements
As an extension of the first use case, multiple network providers and
third parties, such as a regulatory body, may collaborate to gather
network performance data on a one-time or recurring basis, using a
subset of customers of the service providers. The form of
collaboration is beyond the scope of this paper, however it should be
understood that a data collection platform must serve multiple
stakeholder interests.
The main consumer of this use case is someone other than the 'last
mile' provider.
2.4 Over the Top Providers
Possibly an extension to the Multi-Provider use case, OTT providers o An end user may wish to perform diagnostics prior to calling
have an interest to ensure Quality of Experience (QOE) associated their ISP to report a problem. Hence, the end user could connect
with content consumption. The uniqueness to this use case compared a MA to different points of their home network and trigger manual
to those mentioned above is the feature that the measurement client tests. Different attachment points could include their in-home
will run on software based apps and embedded apps such as those found 802.11 network or an Ethernet port on the back of their BB modem.
in set-top boxes or disc players. It is assumed that measurement
tests run by OTT providers would only include the metrics associated
with layer 3 and up.
The main consumer of this use case are content providers. o An OTT or ISP service provider may deploy a MA within an their
service platform to provide the end user a capability to diagnose
service issues. For instance a video streaming service may
include a manually initiated MA within their platform that has the
Controller and Collector predefined. The end user could initiate
performance tests manually, with results forwarded to both the
provider and the end user via other means, like UI, email, etc.
2.5 Regulators 2.3 Regulators
Regulators in jurisdictions around the world are responding to Regulators in jurisdictions around the world are responding to
consumers' adoption of broadband technology solution for traditional consumers' adoption of broadband technology solution for
telecommunications and media services by reviewing the historical traditional telecommunications and media services by reviewing the
approaches to regulating these industries and services and in some historical approaches to regulating these industries and services
cases modifying existing approaches or developing new solutions. and in some cases modifying existing approaches or developing new
solutions.
Some jurisdictions have responded to a perceived need for greater Some jurisdictions have responded to a perceived need for greater
information about broadband performance in the development of information about broadband performance in the development of
regulatory policies and approaches for broadband technologies by regulatory policies and approaches for broadband technologies by
developing large-scale measurement programs. Programs such as the developing large-scale measurement programs. Programs such as the
U.S. Federal Communications Commission's Measuring Broadband America, U.S. Federal Communications Commission's Measuring Broadband
U.K. Ofcom's UK Broadband Speeds reports and a growing list of other America, U.K. Ofcom's UK Broadband Speeds reports and a growing
programs employ a diverse set of operational and technical approaches list of other programs employ a diverse set of operational and
to gathering data in scientifically and statistical robust ways to technical approaches to gathering data in scientifically and
perform analysis and reporting on diverse aspects of broadband statistical robust ways to perform analysis and reporting on
performance. diverse aspects of broadband performance.
While each jurisdiction responds to distinct consumer, industry, and While each jurisdiction responds to distinct consumer, industry,
regulatory concerns, much commonality exists in the need to produce and regulatory concerns, much commonality exists in the need to
datasets that are able to compare multiple broadband providers, produce datasets that are able to compare multiple broadband
diverse technical solutions, geographic and regional distributions, providers, diverse technical solutions, geographic and regional
and marketed and provisioned levels and combinations of broadband distributions, and marketed and provisioned levels and
services. combinations of broadband services.
Regulators role in the development and enforcement of broadband Regulators role in the development and enforcement of broadband
policies also require that the measurement approaches meet a high policies also require that the measurement approaches meet a high
level of verifiability, accuracy and fairness to support valid and level of verifiability, accuracy and fairness to support valid and
meaningful comparisons of broadband performance meaningful comparisons of broadband performance
LMAP standards could answer regulators shared needs by providing LMAP standards could answer regulators shared needs by providing
scalable, cost-effective, scientifically robust solutions to the scalable, cost-effective, scientifically robust solutions to the
measurement and collection of broadband performance information. measurement and collection of broadband performance information.
The main consumer of this use case are regulators The main consumer of this use case are regulators
3 Details of ISP Use Case 3 Details of ISP Use Case
3.1 Existing Capabilities and Shortcomings 3.1 Existing Capabilities and Shortcomings
In order to get reliable benchmarks some ISPs use vendor provided In order to get reliable benchmarks some ISPs use vendor provided
hardware measurement platforms that connect directly to the home hardware measurement platforms that connect directly to the home
gateway. These devices typically perform a continuous test schedule, gateway. These devices typically perform a continuous test
allowing the operation of the network to be continually assessed schedule, allowing the operation of the network to be continually
throughout the day. Careful design ensures that they do not assessed throughout the day. Careful design ensures that they do
detrimentally impact the home user experience or corrupt the test not detrimentally impact the home user experience or corrupt the
results by testing when the user is also using the Broadband line. test results by testing when the user is also using the Broadband
While the test capabilities of such probes are good, they are simply line. While the test capabilities of such probes are good, they
too expensive to deploy on mass scale to enable detailed are simply too expensive to deploy on mass scale to enable
understanding of network performance (e.g. to the granularity of a detailed understanding of network performance (e.g. to the
single backhaul or single user line). In addition there is no easy granularity of a single backhaul or single user line). In addition
way to operate similar tests on other devices (eg set top box) or to there is no easy way to operate similar tests on other devices (eg
manage application level tests (such as IPTV) using the same control set top box) or to manage application level tests (such as IPTV)
and reporting framework. using the same control and reporting framework.
ISPs also use speed and other diagnostic tests from user owned ISPs also use speed and other diagnostic tests from user owned
devices (such as PCs, tablets or smartphones). These often use devices (such as PCs, tablets or smartphones). These often use
browser related technology to conduct tests to servers in the ISP browser related technology to conduct tests to servers in the ISP
network to confirm the operation of the user BB access line. These network to confirm the operation of the user BB access line. These
tests can be helpful for a user to understand whether their BB line tests can be helpful for a user to understand whether their BB
has a problem, and for dialogue with a helpdesk. However they are not line has a problem, and for dialogue with a helpdesk. However they
able to perform continuous testing and the uncontrolled device and are not able to perform continuous testing and the uncontrolled
home network means that results are not comparable. Producing device and home network means that results are not comparable.
statistics across such tests is very dangerous as the population is Producing statistics across such tests is very dangerous as the
self-selecting (e.g. those who think they have a problem). population is self-selecting (e.g. those who think they have a
problem).
Faced with a gap in current vendor offerings some ISPs have taken the Faced with a gap in current vendor offerings some ISPs have taken
approach of placing proprietary test capabilities on their home the approach of placing proprietary test capabilities on their
gateway and other consumer device offerings (such as Set Top Boxes). home gateway and other consumer device offerings (such as Set Top
This also means that different device platforms may have different Boxes). This also means that different device platforms may have
and largely incomparable tests, developed by different company sub- different and largely incomparable tests, developed by different
divisions managed by different systems. company sub-divisions managed by different systems.
3.2 Understanding the quality experienced by customers 3.2 Understanding the quality experienced by customers
Operators want to understand the quality of experience (QoE) of their Operators want to understand the quality of experience (QoE) of
broadband customers. The understanding can be gained through a their broadband customers. The understanding can be gained through
"panel", ie a measurement probe is deployed to a few 100 or 1000 of a "panel", ie a measurement probe is deployed to a few 100 or 1000
its customers. The panel needs to be a representative sample for each of its customers. The panel needs to be a representative sample
of the operator's technologies (FTTP, FTTC, ADSL...) and broadband for each of the operator's technologies (FTTP, FTTC, ADSL...) and
options (80Mb/s, 20Mb/s, basic...), ~100 probes for each. The broadband options (80Mb/s, 20Mb/s, basic...), ~100 probes for
operator would like the end-to-end view of the service, rather than each. The operator would like the end-to-end view of the service,
(say) just the access portion. So as well as simple network rather than (say) just the access portion. So as well as simple
statistics like speed and loss rates they want to understand what the network statistics like speed and loss rates they want to
service feels like to the customer. This involves relating the pure understand what the service feels like to the customer. This
network parameters to something like a 'mean opinion score' which involves relating the pure network parameters to something like a
will be service dependent (for instance web browsing QoE is largely 'mean opinion score' which will be service dependent (for instance
determined by latency above a few Mb/s). web browsing QoE is largely determined by latency above a few
Mb/s).
An operator will also want compound metrics such as "reliability", An operator will also want compound metrics such as "reliability",
which might involve packet loss, DNS failures, re-training of the which might involve packet loss, DNS failures, re-training of the
line, video streaming under-runs etc. line, video streaming under-runs etc.
The operator really wants to understand the end-to-end service The operator really wants to understand the end-to-end service
experience. However, the home network (Ethernet, wifi, powerline) is experience. However, the home network (Ethernet, wifi, powerline)
highly variable and outside its control. To date, operators (and is highly variable and outside its control. To date, operators
regulators) have instead measured performance from the home gateway. (and regulators) have instead measured performance from the home
However, mobile operators clearly must include the wireless link in gateway. However, mobile operators clearly must include the
the measurement. wireless link in the measurement.
Active measurements are the most obvious approach, ie special Active measurements are the most obvious approach, ie special
measurement traffic is sent by - and to - the probe. In order not to measurement traffic is sent by - and to - the probe. In order not
degrade the service of the customer, the measurement data should only to degrade the service of the customer, the measurement data
be sent when the user is silent, and it shouldn't reduce the should only be sent when the user is silent, and it shouldn't
customer's data allowance. The other approach is passive measurements reduce the customer's data allowance. The other approach is
on the customer's real traffic; the advantage is that it measures passive measurements on the customer's real traffic; the advantage
what the customer actually does, but it creates extra variability is that it measures what the customer actually does, but it
(different traffic mixes give different results) and especially it creates extra variability (different traffic mixes give different
raises privacy concerns. results) and especially it raises privacy concerns.
From an operator's viewpoint, understanding customers better enables From an operator's viewpoint, understanding customers better
it to offer better services. Also, simple metrics can be more easily enables it to offer better services. Also, simple metrics can be
understood by senior managers who make investment decisions and by more easily understood by senior managers who make investment
sales and marketing. decisions and by sales and marketing.
The characteristics of large scale measurements that emerge from The characteristics of large scale measurements that emerge from
these examples: these examples:
1. Averaged data (over say 1 month) is generally ok 1. Averaged data (over say 1 month) is generally ok
2. A panel (subset) of only a few customers is OK 2. A panel (subset) of only a few customers is OK
3. Both active and passive measurements are possible, though the 3. Both active and passive measurements are possible, though the
former seems easier former seems easier
4. Regularly scheduled tests are fine (providing active tests 4. Regularly scheduled tests are fine (providing active tests
back off if the customer is using the line). Scheduling can be back off if the customer is using the line). Scheduling can be
skipping to change at page 14, line 29 skipping to change at page 14, line 29
analysed in the presence of topology, line, product or contract analysed in the presence of topology, line, product or contract
information it is critical that the test points are accurately information it is critical that the test points are accurately
defined and authenticated. defined and authenticated.
Finally the test data, along with any associated network, product or Finally the test data, along with any associated network, product or
contract data is commercial or private information and needs to be contract data is commercial or private information and needs to be
protected. protected.
4 Security Considerations 4 Security Considerations
TBD The transport of Controller to MA and MA to Collector traffic must be
protected both in-flight and such that each entity is known and
trusted to each other.
It is imperative that end user identifying data is protected.
Identifying data includes, end user name, time and location of the
MA, and any attributes about a service such as service location,
including IP address that could be used to re-construct physical
location.
5 IANA Considerations 5 IANA Considerations
TBD TBD
6 Contributors 6 Contributors
The information in this document is partially derived from text The information in this document is partially derived from text
written by the following contributors: written by the following contributors:
 End of changes. 25 change blocks. 
131 lines changed or deleted 129 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/