< draft-linsner-lmap-use-cases-03.txt   draft-linsner-lmap-use-cases-04.txt >
INTERNET-DRAFT Marc Linsner INTERNET-DRAFT Marc Linsner
Intended Status: Informational Cisco Systems Intended Status: Informational Cisco Systems
Expires: January 16, 2014 Philip Eardley Expires: April 5, 2014 Philip Eardley
Trevor Burbridge Trevor Burbridge
BT BT
July 15, 2013 October 2, 2013
Large-Scale Broadband Measurement Use Cases Large-Scale Broadband Measurement Use Cases
draft-linsner-lmap-use-cases-03 draft-linsner-lmap-use-cases-04
Abstract Abstract
Measuring broadband performance on a large scale is important for Measuring broadband performance on a large scale is important for
network diagnostics by providers and users, as well for as public network diagnostics by providers and users, as well for as public
policy. To conduct such measurements, user networks gather data, policy. To conduct such measurements, user networks gather data,
either on their own initiative or instructed by a measurement either on their own initiative or instructed by a measurement
controller, and then upload the measurement results to a designated controller, and then upload the measurement results to a designated
measurement server. Understanding the various scenarios and users of measurement server. Understanding the various scenarios and users of
measuring broadband performance is essential to development of the measuring broadband performance is essential to development of the
skipping to change at page 2, line 26 skipping to change at page 2, line 26
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1 Internet Service Provider (ISP) Use Case . . . . . . . . . . 3 2.1 Internet Service Provider (ISP) Use Case . . . . . . . . . . 3
2.2 End User Network Diagnostics . . . . . . . . . . . . . . . . 4 2.2 Regulators . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Regulators . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2.1 Measurement Providers . . . . . . . . . . . . . . . . . 5
2.2.2 Benchmarking and competitor insight . . . . . . . . . . 5
2.3 Fixed and Mobile Service . . . . . . . . . . . . . . . . . . 6
3 Details of ISP Use Case . . . . . . . . . . . . . . . . . . . . 6 3 Details of ISP Use Case . . . . . . . . . . . . . . . . . . . . 6
3.1 Existing Capabilities and Shortcomings . . . . . . . . . . . 6 3.1 Existing Capabilities and Shortcomings . . . . . . . . . . . 6
3.2 Understanding the quality experienced by customers . . . . . 6 3.2 Understanding the quality experienced by customers . . . . . 7
3.3 Benchmarking and competitor insight . . . . . . . . . . . . 8 3.3 Understanding the impact and operation of new devices and
3.4 Understanding the impact and operation of new devices and
technology . . . . . . . . . . . . . . . . . . . . . . . . . 8 technology . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.5 Design and planning . . . . . . . . . . . . . . . . . . . . 9 3.4 Design and planning . . . . . . . . . . . . . . . . . . . . 9
3.6 Identifying, isolating and fixing network problems . . . . . 11 3.5 Identifying, isolating and fixing network problems . . . . . 10
3.7 Comparison with the regulator use case . . . . . . . . . . . 12 3.6 Comparison with the regulator use case . . . . . . . . . . . 12
3.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . 13 3.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . 13
4 Security Considerations . . . . . . . . . . . . . . . . . . . . 14 4 Security Considerations . . . . . . . . . . . . . . . . . . . . 14
5 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 14 5 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 14
6 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 14 Appendix A. End User Use Case . . . . . . . . . . . . . . . . . . 14
7 References . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
7.1 Normative References . . . . . . . . . . . . . . . . . . . 15 Normative References . . . . . . . . . . . . . . . . . . . . . . . 15
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 15 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 15
1 Introduction 1 Introduction
Large-scale measurement efforts in [LMAP-REQ] describe three use Large-scale measurement efforts in [LMAP-REQ] describe three use
cases to be considered in deriving the requirements to be used in cases to be considered in deriving the requirements to be used in
developing the solution. This documents attempts to describe those developing the solution. This documents attempts to describe those
use cases in further detail and include additional use cases. use cases in further detail and include additional use cases.
1.1 Terminology 1.1 Terminology
skipping to change at page 4, line 19 skipping to change at page 4, line 19
The end-to-end perspective matters, across home /enterprise The end-to-end perspective matters, across home /enterprise
networks, peering points, CDNs etc. networks, peering points, CDNs etc.
o Understanding the impact and operation of new devices and o Understanding the impact and operation of new devices and
technology. As a new product is deployed, or a new technology technology. As a new product is deployed, or a new technology
introduced into the network, it is essential that its operation introduced into the network, it is essential that its operation
and impact on other services is measured. This also helps to and impact on other services is measured. This also helps to
quantify the advantage that the new technology is bringing and quantify the advantage that the new technology is bringing and
support the business case for larger roll-out. support the business case for larger roll-out.
2.2 End User Network Diagnostics 2.2 Regulators
End users may want to determine whether their network is performing
according to the specifications (e.g., service level agreements)
offered by their Internet service provider, or they may want to
diagnose whether components of their network path are impaired. End
users may perform measurements on their own, using the measurement
infrastructure they provide or infrastructure offered by a third
party, or they may work directly with their network or application
provider to diagnose a specific performance problem. Depending on
the circumstances, measurements may occur at specific pre-defined
intervals, or may be triggered manually. A system administrator may
perform such measurements on behalf of the user. Example use cases
of end user initiated performance measurements include:
o An end user may wish to perform diagnostics prior to calling
their ISP to report a problem. Hence, the end user could connect
a MA to different points of their home network and trigger manual
tests. Different attachment points could include their in-home
802.11 network or an Ethernet port on the back of their BB modem.
o An OTT or ISP service provider may deploy a MA within an their
service platform to provide the end user a capability to diagnose
service issues. For instance a video streaming service may
include a manually initiated MA within their platform that has the
Controller and Collector predefined. The end user could initiate
performance tests manually, with results forwarded to both the
provider and the end user via other means, like UI, email, etc.
2.3 Regulators
Regulators in jurisdictions around the world are responding to
consumers' adoption of broadband technology solution for
traditional telecommunications and media services by reviewing the
historical approaches to regulating these industries and services
and in some cases modifying existing approaches or developing new
solutions.
Some jurisdictions have responded to a perceived need for greater
information about broadband performance in the development of
regulatory policies and approaches for broadband technologies by
developing large-scale measurement programs. Programs such as the
U.S. Federal Communications Commission's Measuring Broadband
America, U.K. Ofcom's UK Broadband Speeds reports and a growing
list of other programs employ a diverse set of operational and
technical approaches to gathering data in scientifically and
statistical robust ways to perform analysis and reporting on
diverse aspects of broadband performance.
While each jurisdiction responds to distinct consumer, industry,
and regulatory concerns, much commonality exists in the need to
produce datasets that are able to compare multiple broadband
providers, diverse technical solutions, geographic and regional
distributions, and marketed and provisioned levels and
combinations of broadband services.
Regulators role in the development and enforcement of broadband
policies also require that the measurement approaches meet a high
level of verifiability, accuracy and fairness to support valid and
meaningful comparisons of broadband performance
LMAP standards could answer regulators shared needs by providing
scalable, cost-effective, scientifically robust solutions to the
measurement and collection of broadband performance information.
The main consumer of this use case are regulators
3 Details of ISP Use Case
3.1 Existing Capabilities and Shortcomings
In order to get reliable benchmarks some ISPs use vendor provided
hardware measurement platforms that connect directly to the home
gateway. These devices typically perform a continuous test
schedule, allowing the operation of the network to be continually
assessed throughout the day. Careful design ensures that they do
not detrimentally impact the home user experience or corrupt the
test results by testing when the user is also using the Broadband
line. While the test capabilities of such probes are good, they
are simply too expensive to deploy on mass scale to enable
detailed understanding of network performance (e.g. to the
granularity of a single backhaul or single user line). In addition
there is no easy way to operate similar tests on other devices (eg
set top box) or to manage application level tests (such as IPTV)
using the same control and reporting framework.
ISPs also use speed and other diagnostic tests from user owned
devices (such as PCs, tablets or smartphones). These often use
browser related technology to conduct tests to servers in the ISP
network to confirm the operation of the user BB access line. These
tests can be helpful for a user to understand whether their BB
line has a problem, and for dialogue with a helpdesk. However they
are not able to perform continuous testing and the uncontrolled
device and home network means that results are not comparable.
Producing statistics across such tests is very dangerous as the
population is self-selecting (e.g. those who think they have a
problem).
Faced with a gap in current vendor offerings some ISPs have taken
the approach of placing proprietary test capabilities on their
home gateway and other consumer device offerings (such as Set Top
Boxes). This also means that different device platforms may have
different and largely incomparable tests, developed by different
company sub-divisions managed by different systems.
3.2 Understanding the quality experienced by customers
Operators want to understand the quality of experience (QoE) of
their broadband customers. The understanding can be gained through
a "panel", ie a measurement probe is deployed to a few 100 or 1000
of its customers. The panel needs to be a representative sample
for each of the operator's technologies (FTTP, FTTC, ADSL...) and
broadband options (80Mb/s, 20Mb/s, basic...), ~100 probes for
each. The operator would like the end-to-end view of the service,
rather than (say) just the access portion. So as well as simple
network statistics like speed and loss rates they want to
understand what the service feels like to the customer. This
involves relating the pure network parameters to something like a
'mean opinion score' which will be service dependent (for instance
web browsing QoE is largely determined by latency above a few
Mb/s).
An operator will also want compound metrics such as "reliability",
which might involve packet loss, DNS failures, re-training of the
line, video streaming under-runs etc.
The operator really wants to understand the end-to-end service
experience. However, the home network (Ethernet, wifi, powerline)
is highly variable and outside its control. To date, operators
(and regulators) have instead measured performance from the home
gateway. However, mobile operators clearly must include the
wireless link in the measurement.
Active measurements are the most obvious approach, ie special
measurement traffic is sent by - and to - the probe. In order not
to degrade the service of the customer, the measurement data
should only be sent when the user is silent, and it shouldn't
reduce the customer's data allowance. The other approach is
passive measurements on the customer's real traffic; the advantage
is that it measures what the customer actually does, but it
creates extra variability (different traffic mixes give different
results) and especially it raises privacy concerns.
From an operator's viewpoint, understanding customers better
enables it to offer better services. Also, simple metrics can be
more easily understood by senior managers who make investment
decisions and by sales and marketing.
The characteristics of large scale measurements that emerge from Regulators in jurisdictions around the world are responding to
these examples: consumers' adoption of broadband technology solution for traditional
telecommunications and media services by reviewing the historical
approaches to regulating these industries and services and in some
cases modifying existing approaches or developing new solutions.
1. Averaged data (over say 1 month) is generally ok Some jurisdictions have responded to a perceived need for greater
information about broadband performance in the development of
regulatory policies and approaches for broadband technologies by
developing large-scale measurement programs. Programs such as the
U.S. Federal Communications Commission's Measuring Broadband America,
U.K. Ofcom's UK Broadband Speeds reports and a growing list of other
programs employ a diverse set of operational and technical approaches
to gathering data in scientifically and statistical robust ways to
perform analysis and reporting on diverse aspects of broadband
performance.
2. A panel (subset) of only a few customers is OK While each jurisdiction responds to distinct consumer, industry, and
regulatory concerns, much commonality exists in the need to produce
datasets that are able to compare multiple broadband providers,
diverse technical solutions, geographic and regional distributions,
and marketed and provisioned levels and combinations of broadband
services.
3. Both active and passive measurements are possible, though the Regulators role in the development and enforcement of broadband
former seems easier policies also require that the measurement approaches meet a high
level of verifiability, accuracy and fairness to support valid and
meaningful comparisons of broadband performance
4. Regularly scheduled tests are fine (providing active tests LMAP standards could answer regulators shared needs by providing
back off if the customer is using the line). Scheduling can be scalable, cost-effective, scientifically robust solutions to the
done some time ahead ('starting tomorrow, run the following test measurement and collection of broadband performance information.
every day').
5. The operator needs to devise metrics and compound measures 2.2.1 Measurement Providers
that represent the QoE
6. End-to-end service matters, and not (just) the access link In some jurisdictions, the role of measuring is provided by a
performance measurement provider. Measurement providers measure a network
performance from users to multiple content providers to show a
performance of the actual network. Users need to know a performance
that are using. In addition, they need to know a performance of other
ISP of same location as information for selecting the network.
Measurement providers will show the measurement result with
measurement methods and measurement parameters.
3.3 Benchmarking and competitor insight 2.2.2 Benchmarking and competitor insight
An operator may want to check that the results reported by the An operator may want to check that the results reported by the
regulator match its own belief about how its network is performing. regulator match its own belief about how its network is performing.
There is quite a lot of variation in underlying line performance for There is quite a lot of variation in underlying line performance for
customers on (say) a nominal 20Mb/s service, so it is possible for customers on (say) a nominal 20Mb/s service, so it is possible for
two panels of ~100 probes to produce different results. two panels of ~100 probes to produce different results.
An operator may also want more detailed understanding of its An operator may also want more detailed understanding of its
competitors, beyond that reported by the regulator - probably by competitors, beyond that reported by the regulator - probably by
getting a third party to establish a panel of probes in its rival getting a third party to establish a panel of probes in its rival
skipping to change at page 8, line 51 skipping to change at page 6, line 9
done some time ahead ('starting tomorrow, run the following test done some time ahead ('starting tomorrow, run the following test
every day'). every day').
5. The performance metrics are whatever the operator wants to 5. The performance metrics are whatever the operator wants to
benchmark. As well as QoE measures, it may want to measure some benchmark. As well as QoE measures, it may want to measure some
network-specific parameters. network-specific parameters.
6. As well as the performance of the access link, the performance 6. As well as the performance of the access link, the performance
of different network segments, including end-to-end. of different network segments, including end-to-end.
3.4 Understanding the impact and operation of new devices and technology 2.3 Fixed and Mobile Service
From a consumer perspective, the differentiation between fixed
broadband and mobile (cellular) service is blurring as the
applications used are very similar. Hence, similar measurements will
take place on both fixed and mobile broadband services.
3 Details of ISP Use Case
3.1 Existing Capabilities and Shortcomings
In order to get reliable benchmarks some ISPs use vendor provided
hardware measurement platforms that connect directly to the home
gateway. These devices typically perform a continuous test schedule,
allowing the operation of the network to be continually assessed
throughout the day. Careful design ensures that they do not
detrimentally impact the home user experience or corrupt the test
results by testing when the user is also using the Broadband line.
While the test capabilities of such probes are good, they are simply
too expensive to deploy on mass scale to enable detailed
understanding of network performance (e.g. to the granularity of a
single backhaul or single user line). In addition there is no easy
way to operate similar tests on other devices (eg set top box) or to
manage application level tests (such as IPTV) using the same control
and reporting framework.
ISPs also use speed and other diagnostic tests from user owned
devices (such as PCs, tablets or smartphones). These often use
browser related technology to conduct tests to servers in the ISP
network to confirm the operation of the user BB access line. These
tests can be helpful for a user to understand whether their BB line
has a problem, and for dialogue with a helpdesk. However they are not
able to perform continuous testing and the uncontrolled device and
home network means that results are not comparable. Producing
statistics across such tests is very dangerous as the population is
self-selecting (e.g. those who think they have a problem).
Faced with a gap in current vendor offerings some ISPs have taken the
approach of placing proprietary test capabilities on their home
gateway and other consumer device offerings (such as Set Top Boxes).
This also means that different device platforms may have different
and largely incomparable tests, developed by different company sub-
divisions managed by different systems.
3.2 Understanding the quality experienced by customers
Operators want to understand the quality of experience (QoE) of their
broadband customers. The understanding can be gained through a
"panel", ie a measurement probe is deployed to a few 100 or 1000 of
its customers. The panel needs to be a representative sample for each
of the operator's technologies (FTTP, FTTC, ADSL...) and broadband
options (80Mb/s, 20Mb/s, basic...), ~100 probes for each. The
operator would like the end-to-end view of the service, rather than
(say) just the access portion. So as well as simple network
statistics like speed and loss rates they want to understand what the
service feels like to the customer. This involves relating the pure
network parameters to something like a 'mean opinion score' which
will be service dependent (for instance web browsing QoE is largely
determined by latency above a few Mb/s).
An operator will also want compound metrics such as "reliability",
which might involve packet loss, DNS failures, re-training of the
line, video streaming under-runs etc.
The operator really wants to understand the end-to-end service
experience. However, the home network (Ethernet, wifi, powerline) is
highly variable and outside its control. To date, operators (and
regulators) have instead measured performance from the home gateway.
However, mobile operators clearly must include the wireless link in
the measurement.
Active measurements are the most obvious approach, ie special
measurement traffic is sent by - and to - the probe. In order not to
degrade the service of the customer, the measurement data should only
be sent when the user is silent, and it shouldn't reduce the
customer's data allowance. The other approach is passive measurements
on the customer's real traffic; the advantage is that it measures
what the customer actually does, but it creates extra variability
(different traffic mixes give different results) and especially it
raises privacy concerns.
From an operator's viewpoint, understanding customers better enables
it to offer better services. Also, simple metrics can be more easily
understood by senior managers who make investment decisions and by
sales and marketing.
The characteristics of large scale measurements that emerge from
these examples:
1. Averaged data (over say 1 month) is generally ok
2. A panel (subset) of only a few customers is OK
3. Both active and passive measurements are possible, though the
former seems easier
4. Regularly scheduled tests are fine (providing active tests
back off if the customer is using the line). Scheduling can be
done some time ahead ('starting tomorrow, run the following test
every day').
5. The operator needs to devise metrics and compound measures
that represent the QoE
6. End-to-end service matters, and not (just) the access link
performance
3.3 Understanding the impact and operation of new devices and technology
Another type of measurement is to test new capabilities and services Another type of measurement is to test new capabilities and services
before they are rolled out. For example, the operator may want to: before they are rolled out. For example, the operator may want to:
check whether a customer can be upgraded to a new broadband option; check whether a customer can be upgraded to a new broadband option;
understand the impact of IPv6 before it makes it available to its understand the impact of IPv6 before it makes it available to its
customers (will v6 packets get through, what will the latency be to customers (will v6 packets get through, what will the latency be to
major websites, what transition mechanisms will be most is major websites, what transition mechanisms will be most is
appropriate?); check whether a new capability can be signaled using appropriate?); check whether a new capability can be signaled using
TCP options (how often it will be blocked by a middlebox? - along the TCP options (how often it will be blocked by a middlebox? - along the
lines of some existing experiments) [Extend TCP]; investigate a lines of some existing experiments) [Extend TCP]; investigate a
quality of service mechanism (eg checking whether Diffserv markings quality of service mechanism (eg checking whether Diffserv markings
skipping to change at page 9, line 32 skipping to change at page 9, line 5
2. Most of the tests are probably simply: "send one packet and 2. Most of the tests are probably simply: "send one packet and
record what happens", so an occasional one-off test is sufficient. record what happens", so an occasional one-off test is sufficient.
3. A panel (subset) of only a few customers is probably OK, to 3. A panel (subset) of only a few customers is probably OK, to
gain an understanding of the impact of a new technology, but it gain an understanding of the impact of a new technology, but it
may be necessary to check an individual line where the roll-out is may be necessary to check an individual line where the roll-out is
per customer. per customer.
4. An active measurement is needed. 4. An active measurement is needed.
3.5 Design and planning 3.4 Design and planning
Operators can use large scale measurements to help with their network Operators can use large scale measurements to help with their network
planning - proactive activities to improve the network. planning - proactive activities to improve the network.
For example, by probing from several different vantage points the For example, by probing from several different vantage points the
operator can see that a particular group of customers has performance operator can see that a particular group of customers has performance
below that expected during peak hours, which should help capacity below that expected during peak hours, which should help capacity
planning. Naturally operators already have tools to help this - a planning. Naturally operators already have tools to help this - a
network element reports its individual utilisation (and perhaps other network element reports its individual utilisation (and perhaps other
parameters). However, making measurements across a path rather than parameters). However, making measurements across a path rather than
at a point may make it easier to understand the network. There may at a point may make it easier to understand the network. There may
also be parameters like bufferbloat that aren't currently reported by also be parameters like bufferbloat that aren't currently reported by
equipment and/or that are intrinsically path metrics. equipment and/or that are intrinsically path metrics.
With better information, capacity planning and network design can be
more effective. Such planning typically uses simulations to emulate
the measured performance of the current network and understand the
likely impact of new capacity and potential changes to the topology.
It may also be possible to run stress tests for risk analysis, for It may also be possible to run stress tests for risk analysis, for
example 'if whizzy new application (or device) becomes popular, which example 'if whizzy new application (or device) becomes popular, which
parts of my network would struggle, what would be the impact on other parts of my network would struggle, what would be the impact on other
services and how many customers would be affected'. services and how many customers would be affected'. What-if
simulations could help quantify the advantage that a new technology
brings and support the business case for larger roll-out. This
approach should allow good results with measurements from a limited
panel of customers.
Another example is that the operator may want to monitor performance Another example is that the operator may want to monitor performance
where there is a service level agreement. This could be with its own where there is a service level agreement. This could be with its own
customers, especially enterprises may have an SLA. The operator can customers, especially enterprises may have an SLA. The operator can
proactively spot when the service is degrading near to the SLA limit, proactively spot when the service is degrading near to the SLA limit,
and get information that will enable more informed conversations with and get information that will enable more informed conversations with
the customer at contract renewal. the customer at contract renewal.
An operator may also want to monitor the performance of its An operator may also want to monitor the performance of its
suppliers, to check whether they meet their SLA or to compare two suppliers, to check whether they meet their SLA or to compare two
suppliers if it is dual-sourcing. This could include its transit suppliers if it is dual-sourcing. This could include its transit
operator, CDNs, peering, video source, local network provider (for a operator, CDNs, peering, video source, local network provider (for a
global operator in countries where it doesn't have its own network), global operator in countries where it doesn't have its own network),
even the whole network for a virtual operator. even the whole network for a virtual operator.
Through a better understanding of its own network and its suppliers, Through a better understanding of its own network and its suppliers,
the operator should be able to focus investment more effectively - in the operator should be able to focus investment more effectively - in
the right place at the right time with the right technology. What-if the right place at the right time with the right technology.
tests could help quantify the advantage that a new technology brings
and support the business case for larger roll-out.
The characteristics of large scale measurements emerging from these The characteristics of large scale measurements emerging from these
examples: examples:
1. A key challenge is how to integrate results from measurements 1. A key challenge is how to integrate results from measurements
into existing network planning and management tools into existing network planning and management tools
2. New tests may need to be devised for the what-if and risk 2. New tests may need to be devised for the what-if and risk
analysis scenarios. analysis scenarios.
skipping to change at page 11, line 5 skipping to change at page 10, line 31
are needed, in order to refine understanding, as well as end-to- are needed, in order to refine understanding, as well as end-to-
end measurements. end measurements.
6. The primary interest is in measuring specific network 6. The primary interest is in measuring specific network
performance parameters rather than QoE. performance parameters rather than QoE.
7. Regularly scheduled tests are fine 7. Regularly scheduled tests are fine
8. Active measurements are needed; passive ones probably aren't 8. Active measurements are needed; passive ones probably aren't
3.6 Identifying, isolating and fixing network problems 3.5 Identifying, isolating and fixing network problems
Operators can use large scale measurements to help identify a fault Operators can use large scale measurements to help identify a fault
more rapidly and decide how to solve it. more rapidly and decide how to solve it.
Operators already have Test and Diagnostic tools, where a network Operators already have Test and Diagnostic tools, where a network
element reports some problem or failure to a management system. element reports some problem or failure to a management system.
However, many issues are not caused by a point failure but something However, many issues are not caused by a point failure but something
wider and so will trigger too many alarms, whilst other issues will wider and so will trigger too many alarms, whilst other issues will
cause degradation rather than failure and so not trigger any alarm. cause degradation rather than failure and so not trigger any alarm.
Large scale measurements can help provide a more nuanced view that Large scale measurements can help provide a more nuanced view that
helps network management to identify and fix problems more rapidly helps network management to identify and fix problems more rapidly
and accurately. and accurately. The network management tools may use simulations to
emulate the network and so help identify a fault and assess possible
solutions.
One example was described in [IETF85-Plenary]. The operator was One example was described in [IETF85-Plenary]. The operator was
running a measurement panel for reasons discussed in sub use case #1. running a measurement panel for reasons discussed in sub use case #1.
It was noticed that the performance of some lines had unexpectedly It was noticed that the performance of some lines had unexpectedly
degraded. This led to a detailed (off-line) investigation which degraded. This led to a detailed (off-line) investigation which
discovered that a particular home gateway upgrade had caused a discovered that a particular home gateway upgrade had caused a
(mistaken!) drop in line rate. (mistaken!) drop in line rate.
Another example is that occasionally some internal network management Another example is that occasionally some internal network management
event (like re-routing) can be customer-affecting (of course this is event (like re-routing) can be customer-affecting (of course this is
skipping to change at page 12, line 22 skipping to change at page 12, line 4
2. Results from the tests shouldn't be averaged 2. Results from the tests shouldn't be averaged
3. Tests are generally run on an ad hoc basis, ie specific 3. Tests are generally run on an ad hoc basis, ie specific
requests for immediate action requests for immediate action
4. "End-to-middle" measurements, ie across a specific network 4. "End-to-middle" measurements, ie across a specific network
segment, are very relevant segment, are very relevant
5. The primary interest is in measuring specific network 5. The primary interest is in measuring specific network
performance parameters and not QoE performance parameters and not QoE
6. New tests are needed for example to check the home network (ie 6. New tests are needed for example to check the home network (ie
the connection from the home hub to the set top boxes or to a the connection from the home hub to the set top boxes or to a
tablets on wifi) tablets on wifi)
7. Active measurements are critical. Passive ones may be useful 7. Active measurements are critical. Passive ones may be useful
to help understand exactly what the customer is experiencing. to help understand exactly what the customer is experiencing.
3.7 Comparison with the regulator use case 3.6 Comparison with the regulator use case
Today an increasing number of regulators measure the performance of Today an increasing number of regulators measure the performance of
broadband operators. Typically they deploy a few 1000 probes, each of broadband operators. Typically they deploy a few 1000 probes, each of
which is connected directly to the broadband customer's home gateway which is connected directly to the broadband customer's home gateway
and periodically measures the performance of that line. The regulator and periodically measures the performance of that line. The regulator
ensures they have a set of probes that covers the different ISPs and ensures they have a set of probes that covers the different ISPs and
their different technology types and contract speeds, so that they their different technology types and contract speeds, so that they
can publish statistically-reasonable average performances. can publish statistically-reasonable average performances.
Publicising the results stimulates competition and so pressurises Publicising the results stimulates competition and so pressurises
ISPs to improve broadband service. ISPs to improve broadband service.
skipping to change at page 13, line 49 skipping to change at page 13, line 30
'control' will be via negotiation with its contractor. 'control' will be via negotiation with its contractor.
o Politics: A regulator has to take account of government targets o Politics: A regulator has to take account of government targets
(eg UK government: "Our ambition (by 2015) is to provide superfast (eg UK government: "Our ambition (by 2015) is to provide superfast
broadband (24Mbps) to at least 90 per cent of premises in the UK broadband (24Mbps) to at least 90 per cent of premises in the UK
and to provide universal access to standard broadband with a speed and to provide universal access to standard broadband with a speed
of at least 2Mbps.") This may affect the metrics the regulator of at least 2Mbps.") This may affect the metrics the regulator
wants to measure and certainly affects how they interpret results. wants to measure and certainly affects how they interpret results.
The operator is more focused on winning market share. The operator is more focused on winning market share.
3.8 Conclusions 3.7 Conclusions
There is a clear need from an ISP point of view to deploy a single There is a clear need from an ISP point of view to deploy a single
coherent measurement capability across a wide number of heterogeneous coherent measurement capability across a wide number of heterogeneous
devices both in their own networks and in the home environment. These devices both in their own networks and in the home environment. These
tests need to be able to operate from a wide number of locations to a tests need to be able to operate from a wide number of locations to a
set of interoperable test points in their own network as well as set of interoperable test points in their own network as well as
spanning supplier and competitor networks. spanning supplier and competitor networks.
Regardless of the tests being operated, there needs to be a way to Regardless of the tests being operated, there needs to be a way to
demand or schedule the tests and critically ensure that such tests do demand or schedule the tests and critically ensure that such tests do
skipping to change at page 14, line 43 skipping to change at page 14, line 25
It is imperative that end user identifying data is protected. It is imperative that end user identifying data is protected.
Identifying data includes, end user name, time and location of the Identifying data includes, end user name, time and location of the
MA, and any attributes about a service such as service location, MA, and any attributes about a service such as service location,
including IP address that could be used to re-construct physical including IP address that could be used to re-construct physical
location. location.
5 IANA Considerations 5 IANA Considerations
TBD TBD
6 Contributors Appendix A. End User Use Case
The information in this document is partially derived from text End users may want to determine whether their network is performing
written by the following contributors: according to the specifications (e.g., service level agreements)
offered by their Internet service provider, or they may want to
diagnose whether components of their network path are impaired. End
users may perform measurements on their own, using the measurement
infrastructure they provide or infrastructure offered by a third
party, or they may work directly with their network or application
provider to diagnose a specific performance problem. Depending on
the circumstances, measurements may occur at specific pre-defined
intervals, or may be triggered manually. A system administrator may
perform such measurements on behalf of the user. Example use cases
of end user initiated performance measurements include:
James Miller jamesmilleresquire@gmail.com o An end user may wish to perform diagnostics prior to calling
their ISP to report a problem. Hence, the end user could connect
a MA to different points of their home network and trigger manual
tests. Different attachment points could include their in-home
802.11 network or an Ethernet port on the back of their BB modem.
7 References o An OTT or ISP service provider may deploy a MA within an their
service platform to provide the end user a capability to diagnose
service issues. For instance a video streaming service may
include a manually initiated MA within their platform that has the
Controller and Collector predefined. The end user could initiate
performance tests manually, with results forwarded to both the
provider and the end user via other means, like UI, email, etc.
7.1 Normative References Contributors
The information in this document is partially derived from text
written by the following contributors:
James Miller jamesmilleresquire@gmail.com
Rachel Huang rachel.huang@huawei.com
Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[LMAP-REQ] Schulzrinne, H., "Large-Scale Measurement of Broadband [LMAP-REQ] Schulzrinne, H., "Large-Scale Measurement of Broadband
Performance: Use Cases, Architecture and Protocol Performance: Use Cases, Architecture and Protocol
Requirements", draft-schulzrinne-lmap-requirements, Requirements", draft-schulzrinne-lmap-requirements,
September, 2012 September, 2012
[IETF85 Plenary] Crawford, S., "Large-Scale Active Measurement of [IETF85 Plenary] Crawford, S., "Large-Scale Active Measurement of
 End of changes. 31 change blocks. 
194 lines changed or deleted 216 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/